as you wrote, noise is likely to be the issue. every time you read that pot value it is slightly different and therefore the boolean is always true.
try to decrease the resolution of your analog reading before you connect it to the logic/change or smooth the output of the reading. i would bitshift the reading by 2 or 3 (effectively dividing by 2 or 4) and try this way.
i have no axoloti here with me right now, so cannot test.