I'm going to assume this is a GM starter. The problem normally is excessive voltage drop to the "stock solenoid" on the starter because of the long wiring typical in most boats to the dash and back (ign switch wire). A trick we've used for years is to install a Ford type slave solenoid in the wire to the GM stock starter solenoid (back near the starter). The key switch then is working only the "slave" solenoid and the GM starter solenoid receives full voltage. As I recall it takes about 10+ volts to pull a GM solenoid and some boats just wont supply that much with the long run from the dash. Heat makes the problem worse so many time you'll have a situation where it will start when cold but then won't after it's warmed up. This is an age old problem with many cars as well. Used to encounter the same thing years ago with old VW bugs. There's other problems that can be what you're describing but this is the most common one.
If you need the wiring diagram for this install I can fax it or scan it for you. Just give a hollar.