There's less risk associated with lower frequency sounds causing damage, but that risk isn't eliminated. There are rare reported cases of infrasound causing people problems, but this is usually with those who work around certain air conditioners all day, for example. You can look this up on google.
We hear because the frequencies involved in a broadband sound directly resonate with the hair cells inside our cochlea at specific pitches. This is from circa 20 Hz to 20 kHz. All objects on earth have a resonant frequency at which they will vibrate when exposed to their resonant pitch. There are various representations of this such as playing a G note on a guitar and hearing a door or other object rattle, for example.
Our pinnas collect sound in a specific way; they guide certain frequencies into our ear canal which then hits our eardrums causing the sound to travel mechanically towards the cochlea via the ossicles. The sound will resonate certain hair cells but not others depending upon which frequencies are present in the broadband sound one is being exposed to. These vibrations are then electrically converted into sound as they travel up the auditory nerve and into the brain. This is where a significant amount of processing takes place which involves other parts of the brain such as the hippocampus and amygdala. The analysis of the sound is based upon previous experiences (memories) and if danger is interpreted, your body will go into fight or flight, for example, but if you hear your wedding song or something you associate with a happy memory, then you may have a rush of endorphins and feel happy, depending on the context. There's a huge amount of complexity involved in hearing a sound and how it is interpreted. This is the domain of psychoacoustics.
Getting back to low frequencies: anything lower than 20 Hz can't be heard by a human ear which means there are no hair cells that will be resonated by those particular sounds. This is why we have the HL (hearing level) standard in audiometry, and the A-weighted decibel scale for measuring pressure waves, as they are tailored to the human ear.
Very low frequencies will just pass right through your entire body, brick walls, and other large objects as they are so long in wavelength. They can't be heard because they aren't resonating your cells, but some of them can be felt. The reason people consider lower frequencies (that can't be heard) as less dangerous is because of the lack of activation of the signal process directly at the ear itself. These waves still carry energy though, and if loud enough, then damage is a possibility. However, when sound is this low it could also damage your internal organs. It would have to be hideously loud, though. This is what makes rocket launches so interesting. A lot of thought has to go into the location of such things as the low frequency sound waves can literally demolish buildings within a given range. They can also kill a person which is why you have to stay at least 3-4 miles away.
Higher frequency sounds are generally much riskier when it comes to your average noise exposure scenario. Firstly because your pinnas actually guide these frequencies into your ear canal, and secondly, there are more hair cells that may violently resonant if the sounds are loud. This can cause free radical damage and oxidative death. This is why people take NAC on this site. The theory is that the antioxidants fight the free radical damage in your inner ear when exposed to noise.