There is always going to be a hazard of reflection from some angle. Most surfaces, especially indoors, can reflect at least some of the beam. With proper goggles I have been protected the times that I notice the dimmed dot get brighter for a moment, which was at least a partial reflection. The dot itself is a very real danger too. Like Constandinos97 said, the diffuse reflection can be damaging. A dark surface might absorb a lot of it, but still be bright enough to do damage nearby. Also, a surface like plastic can be dangerous when it starts to melt. I've noticed that even black matte surfaces quickly become reflective if they start to melt or bead up.
Moral being, that is a dangerous instrument as you already know. Since you already have proper protection, you should use it, yes. It doesn't take a direct hit or even much diffuse light to cause at least some damage. If you're ever looking to do something with the beam holding still, you could also build a beam stop to put behind it and safely absorb the dot. Cool laser, by the way!