I cant see how the capacitor is connected, but somehow it looks like its in backwards
Also its on the output and not on the input.
[edit:] wait... is this capacitor connected in series?
Also it looks like you changed the polarity of the breadboard, so negative is positive while postitive is negative. The ceramic capacitors are also missing, but thats not the problem now...
So did you measure the voltage before you applied it to the driver? like connecting a DMM and playing with the pot to see how it changes from 0 to 5V? Was everything correct? (Most DMMs could show some peaks when changing the pot postition, thats normal)
It should be impossible to kill a diode with modulation since the driver is never putting out more then 100%. You already tried it with 5V on the modulation input? Maybe you were overdriving your diode in general and it just survived this far.
edit: Just made an quick setup, here is what it could look like:
Works fine, applied 12V to the input and got a clean 0-5V at the output...
When testing with the DMM the pot resistance is not critical. So also check how your voltage behaves when connecting the driver.