What do you mean "very distorted" ? I am assuming it's audio rate clicks from rapid vca triggering?
My first guess is that it has to do with the live input from the envelope follower producing a very jagged output. Try a smooth object in between that and your math/> to see if that helps.
My other thought is that the bool output of math/> isn't putting out long enough trigs -- try a timer/pulselength on the output before going into the envelope.
2 wild guesses that may not help at all. Let us know how it works out!