I'll tell you back in 1986 i put together my first laser, a 1mW HeNe. I accidentally caught my eye with the beam and it was so bright it instantly gave me a spot that didn't go away for about 30 mins. That was 1mW.
I think the rule of thumb is that anything 5mW or over is an eye sight hazard meaning it can cause permanent damage. You are not alone as a person who is fascinated by lasers and wants get one but is concerned about the hazards.
I commend you for searching out advice on this rather than just buying a laser and finding out the hard way. There is a famous post somewhere from a guy that was cleaning the lens of his, i think 75mW or so pointer, with the batteries in it and accidentally engaged the beam straight into his eye. It took months for his eyesight to recover and I don't think it fully recovered.
So yes, the truth is that in order to "enjoy" a laser you will have to get one that will be an eye hazard and you should observe all safety protocols when using it otherwise accept the risks.
But, if you do decided to a laser that is a hazard, no matter what the power, if you observe smart recommended safety protocols, you will still enjoy the amazing virtues of the laser.
One suggestion is this: start out with a 5mW pointer until you are familiar and comfortable with it. then graduate to say 50mW and up from there. you can always sell your old lower power laser to recoup some costs.