Lots of interesting replies here! The effect I'm seeking is different from a vocoder. Toneburst described it well. SirSickSik's granular objects are very cool and I've tinkered around with them a little bit - but my experiments with them fall short of a MIDI controlled polyphonic granular pitch shifter.
They break down how Manipulator works in this video (10 min video but they jump right into it, watch the first 50 seconds for basic explanation).
Some of the controls are labeled Pitch Shift, Formant Shift, Harmonic Shift, Frequency Modulate, Alternator, Smear (time stretch), Detune...
I suspect it is possible to do in Axoloti, but will take some sophisticated patching.