Air leaking will only sound bad if it whistles, but the performance of the sub will not be affected.
I really, really, really hate to do this but I'm going to agree with ShockwaveBob and disagree with HH.
Air leaks, even small ones, will absolutely kill the performance of a bass reflex enclosure and dramatically reduce the performance of an acoustic suspension enclosure. It's more than just noticeable. It's dramatic.
BTW, the best way to seal a sub to an enclosure is with plumber's putty. You roll it up like a rope, put it on, and squish the driver into it. Just peel off the excess and it's clean and sealed perfectly. Silicone is shit to work with and seals poorly, by comparison.
Did I mention how much posting this pains me?