I got my 200 mW Sanwu pocket in 405 nm today, and I don't think shining it on my wall, or wooden door seems to be an issue (at least not for a few minutes at a time). If I focus the beam a foot or two away from the device, by the time it hits the wall, it's a large enough spot to not really do anything. I used a thermal camera and it only heats up a few degrees compared to the surrounding material.
OTOH, I also have a blue 445 coming soon that's 2W. That might be another story.