If the same pot works fine with PB0, then the problem must be something special about PB1.
Something seems to pull the voltage up, so the pot has difficulties reaching the lower ADC values. That could be:
a) outside of the CPU. There coulkd be a tiny solder bridge or short circuit between the PB1 line and VDDA or some other positive signal.
b) inside the CPU: some processors can turn on different functions of the same pin at the same time, useless as that is. I am not sure if the Axoloti CPU can do this. So maybe, if somewhere in your patch some object used PB1 as an output, that would explain the problem.
c) damage to the CPU chip. Sometimes, when handling electronic parts and circuits without proper ESD protection, a pin gets zapped, which damages the chip. That could totally kill the chip, but more often does only damage to the one pin. This has happened to me twice with CPUs (not Axoloti though), and each time the pin afterwards behaved as if it had an additional resistor connected from the pin to either supply voltage or ground. If that has happened to you, that is awfully bad luck, because it is impossible to repair.