What do you mean by looked at it? Looking at the beam is always safe, taking a direct hit to the eye is not safe. Looking at the dot up close or while burning stuff isn't safe, the dot can be brighter than the sun and so it can cause damage. I wouldn't be to worried about 50mW though as long as it doesn't reflect off something and hit you in the eye.
To answer your question, <1mW is considered safe however I still wouldn't point one in my eye, <5mW is reasonably safe because your blink reflex is fast enough to prevent damage. Any laser can cause eye damage if you take a direct hit if the power density is high enough (this depends on the area of the beam). Lasers in the visible spectrum will damage your retina while an IR laser that you can't even see can damage your lens or cornea. Any damage is permanent and cumulative, so slight damage that you don't notice will become noticeable with slightly more damage. So it's necessary to be very careful with lasers, someone too careless with lasers over a long period of time could end up with some eye damage before they know it. Also with any class 4 laser (500mW and up) you must avoid any direct contact with the beam, not just your eyes.
Alan