Ahh, I forgot you'd be using an analog input. The analog inputs are noisy, so they always change a little, triggering the change detector all the time. As you can see the patch works with the dial since that has no noise.
Here's a change detector with noise suppression, but I don't know if that's enough. You can change the mov_avg filter length and you can replace the div_64 quantizer with another division.
If you could use an encoder instead of a potentiometer, the noise problem would go away.