Well I could have agreed with you on this 4 years ago. You see I've had tinnitus since May of 2006 and the first couple of years I was extremely cautious and used protection when ever I felt there was a chance that there could be loud sounds. This meant me using custom made earplugs at work, using heavy foam earplugs the extremely few times I went out to a pub or similar (after all I was 22 when I got this thing and couldn't as well isolate my self).
After a few years though I started using protections less and less. And by that I mean I didn't use earplugs at work anymore (I worked extra as a bus driver at the time while studying at a University). I still wouldn't dream of going out without plugs though. And between 2009-2015 I pretty much never used earplugs at all. I didn't go to clubs or concerts or anything like that so I even stopped bringing plugs with me. I actually didn't even have a set of plugs in my pocket for years.
My tinnitus was stable for all this time with a few milder spikes that never lasted more than a day. But to be honest I probably wouldn't even know because I didn't even think about my tinnitus for years. I was "cured". I didn't hear it unless I listened for it. If tinnitus came up in a conversation I would listen for it and go "yep there it is" and then it would go back to the oblivion.
Then in January 2015 I had a major spike that became permanent. After that my tinnitus is not as stable anymore. Recently I went to a wedding reception and even though I wore SNR:35db earplugs the whole time I am having a pretty bad spike at the moment that has been lasting a good couple of weeks that I'm praying is not going to become permanent. This would never have happened only a few years ago!
So what I'm trying to say is that it's not as clear cut as you might think. Your tinnitus might be stable for now but there can come a time, like it seems to have for me, where something that shouldn't be harmful according to all of the sound level charts just for some reason is. I am going to have to come to terms with the fact that I can never ever go to a loud venue again, even with the heaviest protection, and that is sadly the fact for many others. It's not unreasonable fear. It's simply common sense.