I am not sure about the specific numbers, but I would say that the less objects you have in these polyphonic subpatches the better, because everything you put inside will be cloned as many voice as you set.
But then again, some things just have to be inside the poly subpatch for it to work properly.
About using many table/object references, I am not sure if it is more heavy than using inlets/outlets, someone else needs to chime in here. But I think we are in an area where it tit makes very little difference, without really knowing.
If you want to measure if you add more cycles to the patch, you can try something like this: