I do not believe this is paranoia in this case. That is, I would not be worried about having done this once, but over a long period of time, bicycle wind noise will probably cause hearing loss:
View attachment 7519
At 20 miles per hour, the wind noise on a bicycle is nearly as loud as the engine noise from my motorcycle at 2-lane speeds.