In my experience most 24/96kHz capable ADC/DAC's sound good at lower sample rates too.
Sorry for the necro, but doesn't this depend on whether the anti-aliasing/reconstruction filters follow the sample rate change? Nyquist for 96kHz is obviously 48kHz, so if the ADC doesn't properly support a 48kHz mode and is effectively sampling at 96kHz and then throwing away every other sample then stuff above 24kHz will alias. Obviously that is above the range of human hearing, but obviously stuff above that can alias down into range of our hearing. There was a problem someone was having on the Mutable Instruments forums where their audio interface didn't have proper anti-aliasing filters when running at 48kHz instead of 96kHz and the ~39kHz PWM carrier frequency from their Shruthi was aliasing (39-24=15, 24-15= 9kHz which is definitely audible).
I'm just curious, did you account for that sort of stuff in your design when you elected to use 96kHz running at 48kHz?