Something that really crystallized my view on this was reading an account of a horrific home invasion, captured on video, where people entered from multiple places at the same time, and shot all the occupants in a matter of seconds.
In that scenario, even if you had a loaded gun on your person, it might not do you much good unless you were in a state of chronic awareness, constantly scanning your environment for threats, even when you were relaxing in your own home. Otherwise, by the time you think "oh shit it's happening, better start shooting", you're already dead.
So, for me it's a question of: do I want to live my whole life in a constant state of fear, being prepared for the absolute worst at any moment, or do I want to accept the very small risk of a sudden violent injury in exchange for being able to live my life normally without constantly assuming every small sudden noise is the sound of a glock being aimed at my head?
I also don't think the risk calculus that a lot of hardcore gun owners subscribe to, is very accurate. That is, violent crime is lower on the list of things that typically kill you in the US compared to cancer, heart disease, automobiles, etc. Yet, many people will keep a loaded gun around, but also continue to smoke, eat red meat, drive cars in city traffic, etc. I fully believe (and this shows my bias and rationalization) that for some people, the illusion of being able to protect oneself with a gun, is a cognitive trick they are employing to try to feel in control of their own inevitable death...