Did you try test fitting the impeller(and checking the front clearance) WITHOUT the key on the shaft? I'm thinking as you are pushing the impeller on, it may be catching the key and driving it forward. This will make it wedge tight as the key climbs the ramp at the forward end of the keyway.BTW, I wouldn't force the impeller on. It will just result in a very difficult to remove impeller, or even a permanently damaged impeller.
I've heard the same conflicting reports on the stuffer. Why not try it without to see what the new impeller gains you and then install the stuffer to see what you gain from that.