Could be your cavitating, and losing thrust on the pump from lack of bowl pressure? Don't see any hardware hanging down, other than the ski loader. Whats the face of your impeller looking like, the condition of it, may help determine whats happening, if it's the pump. And yes the damaged fin can cause grief at your target speeds.
My buddies got a 2oft Taylor BR, his wants to turn right, when he is alone in the boat, but when there is a passenger it's nice and straight. He does have an add-a-shoe installed, and that's most likely the cause, winter came too soon to do any more testing.