Non-Audio Power / Groud Loop / Performance Issues

[weasel79]

hey guys
so i've been running into this problem more and more often lately, where my hardware or lets say the whole prototype system of my diy syth starts behaving really weird, depending on power supply/grounding situation. I'm not talking about the "usual" digital SD etc noise that you get sometimes, but let me start with explaining my current setup:

• i have a midibox sending midi on DIN going into an arduino midi shield with midi through, from there into a control axoloti which does poly voice handling and then forwards midi to several axo synth voice boards.
• midibox and arduino are powered through an usb hub with probably shitty power supply.
• axolotis are either poweredd through the same usb hub or sometimes via 12v from a linear eurorack power supply
• some analog gear is also powered by that linear supply
• midi/data lines between all boards, serial only. between midibox/arduio/axxo only via optocoupled DIN midi
• audio connections w ground between axolotis and audio gear
• random extra ground connection here and there but not comprehensive or even systematic
• macbook pro conected to usb hub and to macbook power supply
• all the AC converters go to the same power outlet

now i always had issues here and there but in general it worked quite stable and the random issues were nothing a system power cycle couldn't fix. but two days ago it started really fucking up:

i was working on changing the axo to axo communication, from DIN midi first to serial, then to usb. at some point i had the latter working, 4 axolotis in a usb chain, the last one i was patching also was connected to the hub. i think about 3 of the 4 had addtl power conenctions to linear 12v. everything was working as expected, super snappy and i was super happy. it was already acting kind of weird where i got heavy humming whenever i unplugged the macbook power supply or the macbook to hub connection. but it was working. now here comes the fuckup. i don't remember what exactly i changed but i was just starting to clean the dev code and wire patching up a bit and everything stopped working.

• the control axo seemed to receive an endless loop of the last received midi message, with data1 and data2 set to 0. like a half assed midi feedback loop.
• this made the whole system unresponive and lag/crash/deconnect all the time.
• it got better after a power cycle and the control axo now is working when the usb-to-hub-to-macbook connection is intact. as soon as i unplug that on either port the midi hammering starts and it will stop working. half of the time this oly depends on the physical connection, but sometimes i have to start the patcher and connect in order to make it run. most times i can hot-plug the usb in and out, and the axoloti will hang/run accordingly
• i tried going back to very basic midi i/o forwarding patches to debug and diagnose but i barely get anything. like, i have the midi poly forwarding on the control axo, and it works fine (as long as usb connected...), log shows the sent midi data, but the receiving axo doesn't get any input anymore, no matter what, not on usb, not on DIN midi
• tried swapping around the axolotis to make sure its not a faulty unit, same behaviour
• i rarely do get the control board to work properly with the macbook not connected so its not a osx/usb midi issue
• tried disconnecting all analog gear and connections. problem persist with just midibox, arduino and one axoloti.

i know these symptoms i described are pretty erratic and probably just a weird showing of some fundamental problem. i strongly suspect a big fuckup in my powersupply/groud loop situation. so, whats to do here? anybody has specific advice or even some general (electric? or axo specific) guidelines and rules how to power-connect a system like this properly? obvisouly the macbook wont be a part of the system after developing so keeping that plugged in is not really an option....

i was already tempted to find workarounds on the software/pather side but the whole thing was running perfectly for half an hour before it started acting up. also after writing all this up i am pretty sure i will come back tomorrow, just fire everythig up and it will maggically work again.

[weasel79]

ok it keeps getting worse and worse.

this morning for debugging, i put my keyboard into the basic midi splitter patch on one axo, DIN in, and went through all midi OUT options, din cable, USB device etc, it sends data perfectly fine and i can receive it on other monitoring devices. it works somewhat standalone without USB connected most of the time, still goes ito hanging mode every now and then when i plug/unplug usb devices. or sometimes it loggs out after working for a couple secconds, at some point i noticed midi feedback loops in the debugger again.

then i take a second axo, just put a basic midi monitor with LEDs on, and i get NOTHING, no matter which midi device i try.

to top it off nicely, one of the voice boards just seems to have died - it just blinks on power up and then ggoes black, and i think two other boards are currently not connecting to the patcher, the patcher just dies away on trying to connect.

so yeay. anybody please? i'm about to throw this whole thing out the window.

edit. ok yeah i've had it. just one axoloti conencted to my midibox board directly, with just a midi monitor and it goes hanging after 10sec when i unplug USB

oops also i think i just killed the secod axoloti today... blinking then black
edit2. the blinking then black issues seems to be resolvable by holding s2 on power, will investigagte

fwiw it was also the first day of me using two patcher instances parallel when all the problems started..

[lokki]

this sounds like a corrupted startup patch to me. can you try and upload a new one to the SdCard/flash and see if this improves the situation. do you have any custom edited objects that are maybe "faulty"? i.e. bad memory access etc.

[MattilynMattroe]

I had similar problems with an axo that kept crashing and thought it was a power issue. Turned out to be a custom object in a patch I had as the startup patch. The problem object was for sending midi back to a controller for led feedback... It was sending the same midi message every k cycle I think, and I didn't see the midi when monitoring on another axo or the laptop because it was only going to the controller. I tweaked the object and the problem disappeared.

I doubt it's the parallel patcher thing - I often have 3 open with no issues

[weasel79]

I will, right now def two axos are corrupted but i had most of these issues before that happened. The corrupting part only started today

I’m bakc in the lab in 1h will post some more test results

The weird part tho is that all my patches work perfect when connect to macbook. Shit only starts when i unplug that.

I also already tried rolling back to 100% stable patches from 3 days ago, same weirdo hangung and presumably midi loops

edit. unbricked the first axoloti via S2-boot then re-flash. will investigate further.

is there a chance that i burned maybe a little capacitor in the power suuply circtuiry or something like that? by (hot)plugging usb and 12v poewr back and forth?

ok new test result:
simple midi moitor with LED conected on receiving "voice" axoloti. simple midi thru with LED on "control" board. midi din ggoing MIDIBOX->CONTROL BRD->VOICE BRD. voice board is running standalone on 12v, receiving midi fine. next i connect the ctrl board to the patcher, go live with the midi thru project. EVERYTHING WORKS FINE. leds on both axolotis for every keystroke. now. i upload as startup. disconnect USB. connect ctrl board to 12v. play some keys. first 5 come through. and then. boom. ctrl board is dead again.
now funny enough, i reconnect USB, patcher NOT live, the ctrl board starts working again but not forwarding midi to voice board anymore. i'm dying really.
no custom objects, no complicated wiring, just straight up din midi between a proven sender and 2 axo boards. and a 12v power supply...
its like whenever i disconnect USB there is some weirde midi feedback active. i only have din connectors one way though and no usb connections. the axolotis are only connected to din midi and 12v poewr.

sometimes boards wont connect to the patcher anymore. it just times out for a while trying to connect, then gives me a million java errors, and then gives me a million log messages of what i assume is the last patch stored as startup. its really like 3 days ago EVERYTHING went to shit and for the life of me i can't figure out what plug i pulled

edit 14: ok i have it back running now with DIN midi. still some hickups here and there. like sometimes when i connect my midi thru or the led-arduino via midi, the looping and hanging will return...

and the latest twist - now the arduino started having similar behaviour. when its only connected to the usb hub and the usb hub NOT connected to the macbook, i get midi hangers and loops and disconnects on the axos again. back to normal as soon as the usb connector goes into macbook. wtf. whats wrong with my system. must be a power/ground issue that somehow affects my midi messages?

....aaaand now unplugging the macbook power supply gives me the same issue lol i'm lost. do i just run 5 extra ground wires from every single board straight into my power outlet? and 5 more between each board? i am somewhat suspecting super dirty power supply from the usb hub i am using.

i know these issues will be better once i finalize my power situation, but still i am sooo bummed. usb midi chaining was working perfectly for 20-30 mniutes and now its all gone back to sluggy din midi.

[weasel79]

haha so now.
i connected the midibox MCU, arduino and 4 axolotis to the same linear PSU. other than that only din midi connections. not usb hubs or connections at all. everything running again like 10 days ago. until..... i plug the 1/8" jack of the active speaker into the summing output of the 3 voice axolotis. introducing this new power circuit, via a single audio line, makes the system have, wait for it, midi hicckups again. i get missing noteoffs, and random extra note ons at what sounds like note 0 and note 128, LFO clicking or bat noises. not as bad as before, no total unresponsiveness, i can mute the hanging notes by exhausting my polyphony.

i am at total loss here and actually scared. i knew one day it would come bite my ass that i totally refused to understand basic electronic principles.

how can this one 3.5mm audio connection mess with my midi stream? as soon as i unplug the audio jack, it resumes normal work. plug it back in, midi hangers.

should i just grgound-connect the speaker amp circuit or the power supply with my main PSU circuit?

i feel so dumb cause everything was working fine until a couple days ago and i know i am just fighting the symptoms and not the cause of what started this fuckery...

---

oh and here's an extra side story for flavour: my control board does polyhandling of note-on note-off, splitting 6 voices to 3x2 vocies on 3 seperate midi channels. but. i always had CCs and all other midi data remain on channel one so i dont have to send triplets. now, on the voice axos, i have the patch and subpatch set to the according midi channel for that voice. so technically CCs on channel 1 shouldn't be read. but miraculously, it always was working until this mishap started. i 100% was receiving and controlling axo voice parameters with midi CCs. now, since i had to reset everything, rewire, re flash, bla bla bla, and still work around my ground issues, i dont receive any CCs anymore! i know i am not supposed to receive them, bt i did not change anything in that part of the patch and it definately was working before. call me crazy already.

[MattilynMattroe]

Dude! No idea on a solution, but I feel your pain! Hope you work it out!

[lokki]

sorry, are you saying you are plugging the monitors into the headphone output (3.5mm seems to indicate just that) you should NEVER do that! the Headphone out runs at a virtual ground and you cannot ground it, so never plug a device in there that is connected to mains power

[weasel79]

No i’m not. I am running the 6.35mm outs of 3 voice axos to a summing stage, and from there via 3.5 to the amp/speaker. Summing is on same power circuit as axos and doesnt cause any issues

I was about to add anorher common ground connection btwn both leftover power circuits but then i fried my midibox MCU with 12v real quick. So thats the new thing on top of the to do list for the next 2h...

[lokki]

ah good. phew...what does the summing stage look like? what happens if you unplug two of the axolotis from there while you have midi issues?

[weasel79]

Its a buffered multiple from eurorack world, so active. I will try, good call, but at the current setup i doubt that changes anything, cause all axos are equal..