I'm going to say "not that dangerous"; on the other hand, I still operate firearms myself sometimes as rural life sometimes demands. Fox cornered in the henhouse? Better borrow a gun, since you can't reason with them and killing them with hand tools is just messy and awful for everyone (I heard the results of my dad killing a rodent by repeatedly bashing it with a shovel in our chicken coop, when I was 8 or so, and it made a strong impression!)
Let's assume we're talking about centerfire rifle rounds, since that's the loudest thing; .308 is reported at as high as 167db at the muzzle. I'm going to be generous and say you get the same 167db at 6". You said "quite close" to your house, so I'll pretend that was 8' from your exterior wall since that's easy math.
167db at 6" gives us 161 at 12", 155 at 24", 149 at 48" and 142 at 8'. Building materials are rated by "Sound Transmission Class", and the worst rating, "very poor" is 25, representing a 25db reduction.
That would give an actual interior peak of 117db. That's loud enough to be startling as hell, roughly twice as loud as a really loud bang from crashing plates together, but well under the threshold for instant damage. I do not at all assume that us people with damaged hearing are as resilient as healthy people, but this still wouldn't worry me very much. I've been around .223 centerfire which is only a little quieter than .308, and while those weapons are too loud for me to want to have anything to do with them, the difference in how loud they are from holding them while they go off vs standing even 5 or 10 feet away is very significant.
I'm a little curious why you apparently have idiots discharging lethal weapons within the vicinity of a structure which almost certainly wouldn't protect you if they shot inside it by accident, if the answer to that is "really bad neighbors", then, on the one hand, yikes, you have my deepest sympathy -- on the other, you also have my empathy, because I grew up next door to idiots and my parents once had a bullet pass between them when they were standing like 6' apart, absolutely terrifying.