Yes if a beam is ever visible when wearing the goggles then you're wearing the wrong goggles. You can use that laser without goggles but you must be certain there is no chance of an accident, watch out for mirrors, glass, etc. Anything over 5mW can do permanent eye damage faster than you can blink your eye. Never view the dot of a laser up close, and never do any burning without the goggles, looking at the beam alone won't harm you, but if it slips out of your hand or rolls off the table and flashes you in the eye you can have permanent eye damage, so use caution.
Alan