A Recipe For Learning Audio Programming

A Recipe For Learning Audio Programming

Overview

This document aims to provide a loose guide for those interested in learning about audio programming, specificially low-level audio DSP development. It is particular geared for musically inclined individuals. At one point, I was a kid in music school trying to figure this stuff out for myself. This is the guide I wish I had starting out.

Code is a terrible way to learn DSP

Looking at code is a terrible way to learn about DSP. Trust me, I tried it with soundpipe. Didn't work.

By the time an algorithm finds its way into a C/C++ implementation, most of the bits that would have helped you learn about it have stripped away. Many components are usually pre-derived or optimized away. Programming syntaxes like C/C++ also notate DSP in a very obfuscated way. Even audio DSLs like FAUST or SOUL are like this to an extent.

This isn't to say avoid looking at code. In fact, quite the opposite. Find as much existing code as possible, use it to explore and interact. But don't expect things to suddenly make sense with just code alone.

The best way to learn this kind of stuff is to learn the theory, and how to study the mathematical notation behind it. I hate to say it because I'm terrible at math, but I've found this to the true. Concepts didn't start making sense until I was enrolled in a DSP class during my masters program, surrounded by engineers instead of musicians.

Learning theory is beyond the scope of this document, and I'm not comfortable giving advice about that. All I'll say is find a good mentor, and DO THE PROBLEM SETS. Also, a little goes a very long way. If you're anything like me, you're interested in musical applications, and not theory for the sake of theory. Make sure your mentor understands this.

Audio programming is mostly math

There's very little programming in an audio program. It mostly comes down to a glob of arithmetic operations in a loop. "It's all just multiplies and adds", as a friend of mine says. There's very little for newbies to grab on to. If you have a background in CS or software engineering, it can be challenging because there's structures or algorithms to cling on to.

Most of the audio programming skill comes from taking mathematical models and representations of algorithms, and carefully implementing them to efficiently work in code. You'll want to be mindful of the constraints involved with developing software in realtime.

Choose a good programming language for realtime audio

Most programming languages are not suitable for realtime audio. Usually this comes down to how memory is managed, but other things can come into play too.

So, if you're gonna seriously commit to learning this stuff, you might as well commit to something that won't burn you in the end. This is especially true if you want to do real-time.

(To be clear: You can do audio DSP in just about any language, as long as it is can do math stuff. I've seen plenty of amazing music DSP stuff done in languages like Python, Go, Lua, and Scheme. However, algorithms implemented in these languages will objectively perform worse than if they were written in a more realtime-friendly language. Realtime computing isn't about being fast. It's about being consistently fast enough. These languages may perform well enough for your realtime needs, in which case, carry on. Maybe you aren't even interested in doing realtime at all (it certainly would make things easier!), in which case this is all non-applicable.)

Here are some highly opinionated choices that are all suitable for realtime audio:

C is a classic. I personally use C. I know, it's a footgun. But it's also extremly portable, and there's a lot of existing code and tooling available.

Rust is also a suitable choice, and it seems to be gaining momentum. I can't say much more on that, as I don't use it. But I'll probably be paid to write Rust code someday.

Zig could also be a good choice for audio programming, too. I am interested in project, and may even try to learn it some day for free someday.

Lots of people in the "audio programming community" will tell you that you need to learn C++. You don't needto learn C++, but learning it will help you land employment in the industry, if you're into that sort of stuff. Personally, I don't know or use much C++ in my work, and this me cut off from the audio programming community at large. I consider this a feature, not a bug.

Offline First, then Realtime

One of the (few) things I did right was to avoid trying to make realtime audio apps, and instead focus on writing offline programs that generate audio files. You get started so much faster, it's less boilerplate, easier to configure, easier to debug, etc. Stick to command line apps, no GUI stuff.

libsndfile is a great starting point. The first program I studied was this sine wave. The real game changer though was sfprocess.c. Learning how this example worked was the catalyst that would eventually bring soundpipe into existence.

Write stupid programs that generate audio files, then bring those audio files into your existing workflow samplers and DAWs! Always do things in the direction of music. Always.

I should also mention that going from offline to realtime is much easier than going from realtime to offline. Both my Soundpipe and sporth projects were originally offline only and I wasn't even interested in making either work in realtime. But, the transition to realtime was mostly a smooth one, mostly because I wasn't doing anything fancy. Meanwhile, LMMS, a DAW built with realtime at it's inception, and over a decade later, there are still issues related to offline rendering. Mark my words, it's because they tried to do fancy things like use a multi-threaded engine (yikes) and utilize oversampling.

Avoid Plugins (for a while)

Everyone wants to make things like VST/AU plugins so they can use them in their DAWs. I think they can wait. Personally, I've never done any plugin development besides some minor dabbling. At this point, I don't really think it's a skill I need to learn. I'll be the first to admit that it made several things harder, but it also simplified many things to the point where I could focus on things without getting too frustrated.

Plugins are massive balls of complexity, and they just add weight to the learning process. It makes most things harder than they should be. The only thing that newbies should be thinking about is that core inner DSP loop Sliders, knobs, GUIS, live input, etc can wait.

Plugins have so much boilerplate and extra cruft to worry about that has nothing to do with you. They are also much harder to debug. In my limited experience with plugin development, I ended up putting in the time to structure things so that there was an offline commandline test program in addition to the actual plugin, which allowed me to debug code much faster and with less friction. This idea only came to me because of all the time I spent building simple offline utilities.

I should mention: while it makes plugin development exceptionally easier, JUCE is not for newbies. It's built for experienced C++ developers who write audio plugins professionally. It's also a ball of complexity masking many other balls of complexity.

Seriously, just stick to writing tiny command line programs that make WAV files for a bit. It's a lot of fun, and you'll be amazed how far it will take you!

BUT, if you're going to learn about how to make plugins, I recommend studying LADSPA, one of the simplest audio plugin architectures I've come across. The SDK is tiny, and comes with sample C code for not only a plugin, but for a plugin loader as well. I never really used LADSPA myself, but playing around with that SDK helped me better understand what a plugin actually was.

Learn FAUST (eventually)

Learn FAUST at some point. It's great. But it can be learned later. To me, FAUST is best appreciated after you've done some work in a lower level language like C/C++. Usually you'll need to be using glue C++ code with FAUST anyways. For the simpler DSP algorithms, sometimes FAUST can be overkill. Simple filters, delays, and oscillators are quite manageable in C/C++. Things start paying off in FAUST when you start to build more complex things like reverbs and physical models.

home | index