That's according to a new Swedish survey of nearly 28,000 people. Researchers say road noise had the biggest effect on young or middle-aged people.
But people between 60 and 80 years old showed no negative health effects.
Researchers say this could be because their hearing is not as sharp as younger people.
The study will be published in Environmental Health.