Lots of good information so far.
The Raspberry Pi is complete overkill for your sequencer. Although its price is lower, the processor that it's based on is designed for running full-blown GUI Applications that run on top of Linux. That's going to add a layer between your code and the hardware that will reduce performance (timing tightness) unless you'll willing to learn Linux device driver development.
In truth, something as lowly as the 8-bit PIC processor would be more than powerful enough to develop a very high performance sequencer that is as tight as any music hardware you might compare it to. Remember that devices like the venerable TR-808 were designed around an 8-bit processor that only clocked at a few megahertz. A PIC probably has about 10 times that processing power.
There are some options out there like the MIDIbox, It is open source and even has a sequencer.
However, I think you're right that the building-block environment of the axoloti is offering something that other systems do not have. You might be able to find a unique way to take advantage of it. The axoloti is slight overkill, but it will probably be way more enjoyable than rolling your own PIC-based sequencer.
Here you're getting at the power of embedded programming. Those Timer peripherals are where all the magic happens. You might even want to dig deeper than the Teensy FrequencyTimer library and code directly to the hardware for minimal latency (then again, constant latency isn't so bad if you're playing from a pre-recorded sequence).
Here is where you want to avoid Linux and instead work with the Timer peripherals directly. You can either do bare-metal coding or use a RTOS. I'm not totally familiar with whether the axoloti library allows access to the Timer hardware (like Teensy FrequencyTimer), so I'll let others speak to that. Usually, the operating system scheduler clock runs at 1 ms per tick accuracy, where the Timer hardware can operate down to nanoseconds if you need that accuracy. I've implemented 512-level resolution LED brightness PWM with Timer interrupts in the nanosecond range and it is rock solid.
p.s. I agree that sticking with MIDI coding and buying a finished product to add MIDI-to-CV would give you the most bang per buck, both in terms of dollars and in terms of development effort.