That's the downside of securing things in place with plenty of adhesive compounds. Expoxy resin is especially hard to get rid off, and doesnt dissolve well in whatever solvent either.
I've been playing with the mentioned DX thermal compound. Doenst seem to be that good of a thermal conductor, but it does have nice other properties. It bonds very nicely with many surfaces (glass, aluminium, galvanized steel, pcb), but remains easy to remove. It sets to a sort of rubbery state that isnt that hard to peel off even with just fingernails in case you make a mistake.
Even though advertised as a thermal compound, i find the stuff much more useful as a semi-permantant 'tack'... it can surely bond/glue surfaces together, but also be removed with limited mechanical force if required. Also, it peels off with very little residue, much nicer than most silicone kits and similar substances.