Honestly, I don't think it's that dangerous if you know how to be safe. It already seems that you have enough common sense to not look directly into the beam of a laser so that's an extra point for you.
A lot of people disagree with me because I don't use safety goggles and I don't think IR leakage is dangerous, but I personally think the dangers of these lasers are exaggerated.
I won't tell you not to be safe, but I will say you don't need goggles with a laser below 400mW if you have common sense to protect you.
Looking at a laser's dot may give you after images after a few minutes, but hell, after watching TV in the dark I get after images. As long as you are careful not to get a reflection or direct exposure, you should be fine.
If you think about it, when you go outside during the day your eyes are taking in MUCH MUCH more light from diffused reflections than your little laser can possibly hope to output. People don't go blind from looking at the ground on a sunny day, unless the ground happens to be a polished or mirrored surface. Even a 2 inch diameter magnifying glass used to focus sunlight to burn ants, roaches, what have you, will focus at least a WATT of the full visible spectrum. I'm sure you've done this. Did you use goggles? Are you blind? (I wouldn't recommend staring at the dot of 2 watts of focused sunlight for an extended period though...)
Short exposure to diffused reflections will not hurt you. Looking at the beam will not hurt you.
Get goggles if you want to be safe or have no natural sense of safety.
If you can't afford them, just play safe until you can.
I've played with a few lasers from 400-5mW (the only one I still have is my +100mW green) with no goggles and I still have 20/20 vision. I even removed the IR filter in my greens (OH NO!!!) and I haven't been killed by IR leakage... yet.
: