If your house never burns down then you aren't paranoid, you are smart !
Whatever you feel is "safe" will be. It's kind of a thing where you are the one who defines the term for yourself.
I have various chargers (home made and purchased) that I trust to function normally BUT that doesn't mean that I plug them in and forget about them. (Which is getting much harder lately...)
It's just one of those things that has ingrained into me over the years, like respecting electricity. Don't stand in a puddle when working with AC wiring. Likewise don't leave a heavy-duty battery charging on a counter in a small room with a door closed overnight. (Can you say >BOOM< !)
Adequate ventilation - for fumes or cooling,
proper surfaces - non-reactive or fireproof,
reliable equipment - non-damaged, good condition, functioning within ALL parametres,
all are considerations when charging.
And I do it all so automatically and instinctively that it was hard to break it down for writing it here. Usually it takes only a couple of seconds to assess it all under the circumstances and just do it the right way.
So do what you think is right.
As you become more familiar with your equipment and how it responds to the cycles the better you will be able to know what to do and when to do it.
Are you paying attention to what is happening when you "check" it now ? Are the batteries cool, warm, or getting almost hot ? Is the charger making more noise at some times versus the others ? Is your safety enclosure working as a heat sink or is it a potential trap for explosive fumes ? Are there even any fumes to worry about ?
If you can answer all questions easily (even ones not asked here) then you have everything under control.
If you don't understand what the questions pertain to then you might want to look into the actual processes involved and find out what should be happening and what could happen. (in a worse case scenario for example.)
Different battery types made from differing materials will require different processes to be recharged. What works fine for one may cause complete breakdown for another.
As always, it comes down to knowing what you are doing. How you get to know is up to you.
Sorry for the bog block answer. Kind of slow here at work at the moment.