Working with DSP (Digital Signal Processors) is a bit of a lab experiment. When it comes to audio, it’s also important to be creative. In this article, I’m going to walk you through my approach to experimenting with DSP processors written in CMajor using Python and Jupyter notebooks.
With a few tweaks, this method can be used to experiment with processors written in other languages and frameworks.
Inspirations
DSP testing is a topic on which I’ve had a lot of brooding. Coming from the backend development world, I used to use test-driven development technique and write test before writing logic. But what I found is that this approach is not suitable for DSP. Audio programming is similar to computer graphics, shaders, rendering etc. It’s hard to judge whether the processor is good without seeing or hearing the output. Also, the output is not fully deterministic and doesn’t have to be exactly the same every time. Depending on the CPU/GPU, it can vary a bit, which is totally fine.
This led me to the conclusion that it’s not worth the effort to create automated tests in advance, because it’s not possible to make good assertions without involving our perception.
I also found that DSP is a bit like design and front-end development, where (visual) perception is important. So I decided to take some inspiration from frontend development and found two concepts that can be adapted and used in audio:
- Visual regression testing is an approach where we keep an image rendered by our application and verify that the future versions of our application won’t change it. We render the views to images with the first test run and after approving them, we treat them as our reference. If the rendering changes according to our intentions, we simply update the reference images with new ones. If they change when they shouldn’t, it means something in our application is broken. When comparing rendered views, we should remember not to compare them exactly, but to allow for a certain level of error. Thanks to this, the tests will not fail when we run them on a different machine, with a different GPU, or after an OS or browser update.
- Storybook is a tool used by frontend developers to visualize UI components in isolation. It allows you to create a catalog of components so that you can check their “look & feel” - how they look like in different scenarios and allow you to interact with them.
Idea
What I like about Storybook is the ability to work on components in isolation. I can see them in different contexts, in different scenarios, and see what they look like in edge cases. Most importantly, I can actually see them and even interact with them without being distracted by other components. Storybook also allows for visual regression testing. This allows me to check how changes I have made in the component have changed it visually. Thanks to this, I can be sure that I didn’t change the design unintentionally.
This inspired me to recreate some of the Storybook features in the DSP world. I decided to use Jupyter (I personally use Jupyter in VScode) and Python because they cover a lot of building blocks out of the box. I also needed the ability to visualize the waveforms, spectograms, filter characteristics, etc. I decided to use Python because it is known as a good tool for analysis purposes with a lot of handy libraries for such work.
So the concept is simple. I have several Jupyter notebooks, which I treat like storybooks. When working on a notebook, I keep it simple and focus on one component or different versions of the same component. I usually create a separate section for each testing scenario. In each section, I start by running the processor with certain inputs, then I visualize and/or add a play button to listen to the output.
I could also do “audio regression testing” to check if I haven’t broken already created processors, but for the moment I decided to use the CMajor test file format for that. It provides maxDiffDb, which allows us to set the level of differences we can tolerate. I’ll say more about this in a future post.
Running the processor
The first thing I need is the ability to run my processor. Let’s start with a minimal implementation. I have a common class that I use in all my notebooks in cmajor_processor.py.
import subprocess
import librosa
import os
cmajor_path = '[YOUR PATH TO CMAJOR]/bin/cmaj'
sample_rate = 44100
class Processor:
def __init__(self, directory):
self.directory = directory
# Length in seconds
def render(self, length):
patch_file = self.directory + "/patch.cmajorpatch"
output_file = self.directory + "/_temp-output.wav"
frames = length * sample_rate
if os.path.exists(output_file): os.remove(output_file)
try:
subprocess.check_output([
cmajor_path,
'render',
'--rate=' + str(sample_rate),
'--length=' + str(frames),
'--output=' + output_file,
patch_file
])
except subprocess.CalledProcessError as e:
print(e.output)
print(e.stderr)
raise e
out = librosa.load(output_file, sr=sample_rate)[0]
os.remove(output_file)
return out
It executes the processor using a CLI tool and renders the output to a temporary file. Then the output file is loaded by librosa and removed from the filesystem.
It’s a trivial and naive implementation, but I like it for its stability. I could create C++ bindings of CMajor JIT engine and use them in Python instead of using CLI, but it would require more maintenance in the future and extra compilation step. Right, the advantage would be faster execution time, but in my opinion CMajor is not mature and adopted technology yet, so there is a risk of API changes that would force me to make adjustments with new versions. Still, it’s worth monitoring the situation and maybe changing this approach in the future.
Example processor and test scenario
For the example purposes, I’ll use a simple ADSR envelope processor from CMajor examples:
processor ADSR
{
input event (std::notes::NoteOn, std::notes::NoteOff) eventIn;
input event float attack;
input event float decay;
input event float sustain;
input event float release;
output stream float32 out;
event eventIn (std::notes::NoteOn e) { noteReceived = true; triggered = true; }
event eventIn (std::notes::NoteOff e) { triggered = false; }
event attack (float seconds) { attackRamp = 1.0 / (processor.frequency * max (minTime, seconds)); }
event decay (float seconds) { decayMultiplier = pow (0.0001, 1.0 / (processor.frequency * max (minTime, seconds))); }
event sustain (float levelPercent) { sustainLevel = levelPercent * 0.01; }
event release (float seconds) { releaseMultiplier = pow (0.0001, 1.0 / (processor.frequency * max (minTime, seconds))); }
float32 minTime = 0.001f;
float64 attackRamp = 0.01;
float64 decayMultiplier = 0.99;
float64 sustainLevel = 1;
float64 releaseMultiplier = 0.999;
bool noteReceived = false;
bool triggered = false;
void main()
{
float64 currentValue;
loop
{
if (noteReceived)
{
noteReceived = false;
// Attack
while (currentValue < 1.0 && triggered && ! noteReceived)
{
currentValue = min (currentValue + attackRamp, 1.0);
out <- float (currentValue);
advance();
}
// Decay
while (currentValue >= sustainLevel && triggered && ! noteReceived)
{
currentValue *= decayMultiplier;
out <- float (currentValue);
advance();
}
// Sustain
while (triggered && ! noteReceived)
{
out <- float (currentValue);
advance();
}
}
// Release
currentValue *= releaseMultiplier;
out <- float (currentValue);
advance();
}
}
}
I keep this processor in the simple-adsr/processor.cmajor file.
To visualize how this envelope looks like, we need to create a test scenario that would set parameters of the processor and emit NoteOn and NoteOff events. Let’s create a simple-adsr/test-scenario.cmajor file:
processor TestScenario[[main]]
{
output stream float out;
node adsr = ADSR;
// ADSR parameters
const float attack = 1.0f;
const float decay = 1.0f;
const float sustain = 50.0f;
const float release = 1.0f;
// Test parameters
const float timeToRelease = 2;
void main()
{
adsr.attack <- attack;
adsr.decay <- decay;
adsr.sustain <- sustain;
adsr.release <- release;
std::notes::NoteOn noteOn;
noteOn.channel = 0;
noteOn.pitch = 0;
noteOn.velocity = 1;
adsr.eventIn <- noteOn;
for (int i = 0; i < timeToRelease * processor.frequency; i++)
{
out <- adsr.out;
adsr.advance();
advance();
}
std::notes::NoteOff noteOff;
noteOff.channel = 0;
noteOff.pitch = 0;
noteOff.velocity = 1;
adsr.eventIn <- noteOff;
loop
{
out <- adsr.out;
adsr.advance();
advance();
}
}
}
To complete the CMajor part of our experiments, we need to create a manifest file. I saved it as simple-adsr/patch.cmajorpatch:
{
"CmajorVersion": 1,
"ID": "dev.miszczak.ADSR",
"version": "1.0",
"name": "ADSR test",
"description": "",
"category": "fx",
"manufacturer": "Patryk Miszczak-Malesiński",
"isInstrument": true,
"source": ["processor.cmajor", "test-scenario.cmajor"]
}
Running the test
Then you can render the result using the class we wrote earlier:
import processor
time_in_seconds = 4
simpleADSR = Processor("simple-adsr")
simpleADSRoutput = simpleADSR.render(time_in_seconds)
To visualize the output I use the waveshow function which is part of librosa:
librosa.display.waveshow(simpleADSRoutput, sr=sample_rate)
Alternatively, you can use matplotlib directly, but for simple waveform plotting, I prefer to use the librosa helpers, as they automatically add an x-axis scale.
Next steps
In upcoming posts, I’ll show you my approach to passing input parameters and streams to experiment with effects processors. You’ll also learn how to identify aliasing problems and analyze filters.