ah, I see your issue....
the issue is that ctrl/hook needs to be unhooked when you switch to it.
this would be best done with updating the ctrl/hook with a new inlet, to allow the unhook to happen, under user control.
(currently it only unhooks when the parameter changes - as far as i can tell with a quick look)
@johannes any objection to adding a unhook on rising edge?
without this, you would end up duplicating much of the logic in ctrl/hook... not hard, but messy if your doing it graphically.