Unless you are running a business from the house or you have a kid that is downloading all of the Bearshare Porn each night, I don't think most residents will have an issue.
Also, it depends on well your VOIP carrier handles quality of service to deal with latency issues within the WAN or LAN.
You may want to check with your local ISP, you may find that they don't even have QOS (quality of service) running, if it is not enabled you will all ways have voice quality issues.
This is a big issue with most ISP's because there main concern is data not voice.