2m, very diffused surface or if you are trying to pop a balloon and you mentioned you always shine around in the dark ie pupils dilated.....there is still a very minimal risk. (1) Some balloons have very smooth/almost shiny surfaces, you focus it to a small dot. 2-3m takes quite a number of seconds to pop for some colours like green, you might be trying to concentrate on the really focused laser spot. You know our field of vision is pretty narrow and the retina portion which is often used is really a very small spot. The damage is actually accumulative at that pretty small region on the retina, pop enough dark balloons at 0.5m to even 5m and it will happen one fine day. The keyword is always really focused spot with very high surface brightness, it may not seem to be that bright if you look at the amount of light falling on surfaces 2m away, but check it out with a small magnifying glass and focus the reflected laser light on a so-called diffused surface, that light can be refocused down to a very very small dot with very high brightness and it will fall on your retina.
In fact flashaholics would tell you not to stare at a XR-E R2 LED at 2m for long periods of time in a dark room. Pop enough balloons at 2-3m in the dark without glasses, and you might just get some visual damage.
Its usually safe most cases, but like i said in the firearms cases, why assume and take whatever 0.001% risk? Same goes for rear seatbelts in cars, the risk of death is still there even if one drives at only 50mph on a country road. (eg unexpected animal dashes across and you swerve). The point about safety glasses is to make it a habit, so that the screw-ups are covered. (ie new surface on a brand of black balloons which are far shinier and you are trying out for the first time @ 2m, but you thought that its same as the old brand)