Yeah, that can be annoying. If it's really stuck in there, you might have no choice but to crush the attachment a little so that you can make a connection with the internal part of it.
One thing you could try... although not really that good an idea, wrap the tip/laser in plastic, run under hot water for 3-5 minutes, than stick it in the freezer for 10-15 minutes... Basic expansion/contraction used same approach with plenty of things, but... idk how well that would play with the diode, definitely don't get it wet.
Good luck.