There will be some capacitance downstream of the pot, whether it be a physical component, or an incorporation into the IC's input pin, or outright parasitic. The resistance of the pot itself will form an RC filter with this, which is low pass, and negate most negative effects.
If there's nothing to drive the output high or low when the wiper loses contact for a microsecond, there's nothing to worry about. If you're irrationally worried about it regardless of the facts, add your own capacitance on the input.
There are a few topologies where the input is indeed pulled high or low with no input, but In this case, pot noise is just another groupthink boogeyman.