Is the speaker dvc or svc? what is the ohm rating of the sub? How do you have the subwoofer wired?
10ga is definitely not large enough power or ground. Voltage drops in long wire runs. 8ga. is what I would have run in the car where it's not as much an issue because you're usually listening to the stereo while the car is running thus the alternator is keeping your voltage up above 13.5 volts. Your boat is making out at 12v ouput when your motor is not runing while beach. your amp will shut down due to low voltage at 11v. As you can see, you don't have very much room there so maintaining the least amount of resistance is key. Your power and ground wire should be changed to atleast a 8ga wire..
Where is your subwoofer located, and is it in the proper enclosure?
Also, your 500.4 kenwood is likely rated at 14.4 volts at 2ohms. 12 volts at 4ohms will be considerable less power.
Remeber that the system in your car will be louder because among other variables, it's inside the vehicle. The boat is open air, so it will take more to get the same output.