I think the pitch shifting objects that are in Axo library are all delay based, which can be fairly "wild" as you describe it. And obviously also add some delay.
More precise pitch shifting can be done in FFT domain, which is a lot more complicated and cpu hungry.
So I think in Axo world some trade offs is to be expected. There are probably people in here that could make something better, but personally I am not able to.
If you want to play around with delay based pitch shifting yourself, take a look at the tutorial called "22_overlap_add_shifter.axp". It's not directly pitchshifting as it is, but the idea is very close. Experiment a bit and see what you can come up with