Maybe the driver backs off right away when you put two in because that is twice the voltage that it was made to run off of. If it dims with twice the voltage and then operates normally with just one battery, or half the voltage, my best guess is the driver isn't designed for two, just one.
Some drivers, when they get too hot, will back off and nearly shut down, explaining why your output dims but I'm surprised it will still function with one battery afterwards, it could be damaged right away if two batteries in series doubles the voltage it was designed to run off of.
I have a laser which won't run with one battery, but will with two because it was designed for a higher voltage than one can supply, your situation appears to be the opposite. In my case no damage will happen but yours, likely.