What you see from a few meters away is not the real dot, it's a sort of ghost. That's because for light near the low-wavelength limit of visibility, normal eyes are effectively quite shortsighted, so you can't focus on it (unless you're farsighted). Also the eyes sensitivity for that light is quite low - unlike green which is near the sensitivity maximum (the apparent intensity difference between BR and green is on the order of 1:100).
How I focus a BR: point the beam at a wall a few meters distant. Out-of-focus, you'll see a hazy blob. Focusing in, it'll become smaller. When reaching the focus point, it will NOT contract to a point (as with green or red), instead it will change appearance and appear to develop a rather well-defined boundary with some internal structure, looking quite different from the out-of-focus picture. Try it.
Put your laser down on a table then and walk towards the wall and have a close-up look at the dot - you'll see it really is just a few mm's across, but you have to be pretty close to really see it like that (less than a meter or so, depending on your eyes).