You are entirely right. I have a solid silver host, known to have great thermal capabilities, and it clearly gets "warm to the touch" after a short 20 seconds or so.
However, sometimes, if the heatsink is properly designed and made out of a clearly effective thermal conductor (like copper or...