Depends on diode .. IR are very sensitive, but 1W 445nm are rather tough. Not that you CANT kill it. But I certainly did a lot of bad things to mine and it i fine. It can survive several seconds of reverse polarity for example (with battery, current driver would kill it).
Then it is all question of obtaining correct voltage. For example 1W 445nm diode works perfectly with 3 AA or AAA batteries, as it's working voltage is around 4.5V. Check my sig. for such direct drive build. It even has advantages - it's more effective as there is no driver and no power loses. On the other hand, the current will get low as the batteries get depleted and you won't be able to squeeze all the juice out of them.
If you feel lucky, go for it. It can be done, and it was done. If you don't feel like blowing the diode, don't do it.