I, too, was curious about this subject in my early days on the forum:

http://laserpointerforums.com/f53/laser-dot-danger-43384.html
Well, let me preface this by saying that you should always wear safety glasses when operating a laser indoors. You'll never encounter a situation as I describe below.

For the purpose of the math, let's assume you have an ideal, 100% reflective matte surface that scatters incident light evenly in a hemisphere radiating from the light source.

First, you want to calculate the area of that hemisphere. Assuming you're about 3m from the laser dot, the area of that hemisphere will be 2*pi*r^2 - 56,548,667.8mm^2.

Next, divide your laser power into that to determine the irradiance per square millimeter. This will be a fairly small number for something like a ~2W laser.

Then multiply that number by the area of the opening your iris allows into your eye to determine what amount of laser radiation would actually reach your eye.

For a 2W laser at a distance of 3m assuming a PERFECT surface, that would be on the order of 0.011mW. If you moved close, like 30cm, that number jumps to over 1mW.

**That being said, the hazard doesn't lie in the diffuse scatter - the hazard lies in specular reflections coming from whatever you point the laser at. So disregard the false sense of security gained from pretending there will be ideal conditions with all matte surfaces (no specular reflections or irregular scatter), and ***just buy some goggles*.
Trevor