How to make #neuromusic

The other day I found out that the FENS conference has a call for artists to showcase their neuroscience-related art. While I am not an artist, I thought I could come up with something for the occasion. Inspired by Alain Destexhe and his Spikiss project, I set out to make some electronic music using the neural data I analyze. My idea was to make an audio artistic impression of what it would sound like to be inside the brain of an animal. This was the result (check out the response to a visual stimulus happening at 00:10!):

For this piece, I wanted the music to closely represent spikes and oscillations in the brain, and at the same time to be somewhat cool and interesting musically. In an electrophysiological dataset we have the local field potential (LFP, a broadband signal which aggregates electrical activity from a small area around the electrode from which it is recorded) and the spike trains (i.e. for every neuron, the times at which it emitted an action potential, which are extracted from the LFP). I wanted to use both the continuous LFP signal and the discrete spike times to generate the music.

A side note on spike-phase coding¶

While a lot of neuroscientific research focuses on spike coding (i.e. the idea that the brain represents information only with the amount and temporal arrangement of spikes), the timing of spikes relative to some baseline oscillation in the LFP could offer a powerful additional resource. This is sometimes referred to as spike-phase coding. Studies have shown how spike times relative to the phase of the LFP in low frequencies carry more information about visual and auditory stimuli (compared to spike times alone). Relating spike times to this LFP 'clock' could expand the coding repertoire of the brain, and be used for example to memorize and disambiguate objects. Furthermore, it could provide a mechanism to temporally organize activity across the brain, and it could work even when oscillations are highly irregular, as is the case in bats and humans. Inspired by this research, I wanted the sounds representing the spikes to be affected by the LFP, and came up with the following plan.

The idea¶

• Each neuron should play only one specific note. Since every neuron has its own waveform (the shape of the action potential), for me it makes sense to have that correspond to a specific note, a sort of signature of a specific neuron. This will also the listener to identify and follow the firing pattern of a neuron over time.
• The sound that each spike makes should be modulated by the energy of the local field potential (LFP) in a certain frequency band, for reasons outlined above.
• Each LFP frequency band should correspond to one long note (playing throughout the piece), a sort of omnipresent background tone.
• The LFP sound should be modulated by the values of the filtered LFP at each point in time, and by the energy in that frequency band.

The plan¶

• STEP 0: Prepare the neural data.
• STEP 1: Generate MIDI files for our spikes, encoding also LFP information which we can use to modulate our sound. Generate MIDI files for our LFP.
• STEP 2: Import the files into Ableton, assign each neuron/LFP band to a synthesizer, and modulate the sounds using the LFP data we have added to our MIDI files.

What you should already know¶

To understand the technical part, you should be a little bit familiar with MIDI files and Ableton.

STEP 0: Preparing the neural data¶

For this example, I am using data recorded in mouse V1 by my colleague Matthijs. Animals are exposed to drifting gratings and to a continuous tone. Every trial, either the grating will suddenly shift orientation, or the audio tone will change frequency, or both changes will happen simultaneously.

I took a random chunk of about 10 consecutive trials, and binned spikes in bins of 1 ms, separately for putative pyramidal cells and putative interneurons. I then took the LFP of a single channel during the same period, and filtered in the delta, theta, alpha, beta, and gamma bands using a narrow Kaiser filter. For each band, I save both the filtered signal and the power over time (square of the Hilbert transform of the filtered signal).

Let's have a look at our data (I'm plotting only 2 seconds).

STEP 1: Making MIDI files from spikes and LFP¶

The first thing we need to do is to convert spike times and LFP signal into MIDI files, which we can then import into Ableton and play through various synthesizers. To do this, I used a Python package called MIDIUtil. If you have worked with VSTs inside Ableton or other DAWs, what I'm doing here should be relatively straightforward. I'll briefly mention a few things which were not 100% clear to me before I started out:

• MIDI track: A MIDI file can have multiple tracks, which will correspond to different MIDI tracks in Ableton. This means we don't need to make a separate MIDI file per neuron, but we can make one file with multiple tracks. Tracks should not be confused with MIDI channels. Essentially, a single MIDI instrument can receive multiple channels of data (remember those old keyboards where you select a sound for the left hand and a sound for the right hand? those are different channels). I'm not gonna use channels here, and just make one track per neuron (or per LFP band).
• MIDI CC (Control Change) is essentially a way to expand the functionality of MIDI. There are 127 MIDI CC, each can transmit a number between 0 and 127. In Ableton, you can edit these using clip envelopes. Some of these controllers are by default mapped to things like velocity and panning, others are empty. I'll put LFP data in the CCs and we'll see how to map that to Ableton parameters later on.
• MIDI timing, a.k.a. how will time be translated between spikes and MIDI notes? The way I'll write spike-notes in MIDI is by using ticks, which are the smallest time unit in MIDI. We'll need to specify a time resolution of our MIDI file, given by the ticks per quarter note parameter (or pulses per quarter, PPQ). Then there is the beat, which typically corresponds to one quarter note. The last ingredient is the tempo of our MIDI file, which in MIDI files is specified in microseconds per beat, and not in beats per minute (BPM). A common tempo of 120 beats per minute is 500000 microseconds per beat in MIDI-world. The duration of a tick is then (depending on whether you want to compute it using microseconds per beat of beats per minute):

\begin{equation} \textrm{tick duration } (\mu s) = \frac{\mu s \textrm{ per beat}}{\textrm{ticks per quarter note}} \end{equation}

\begin{equation} \textrm{tick duration } (\mu s) = \frac{6\times 10^7}{\textrm{beats per minute }\times \textrm{ticks per quarter note}} \end{equation}

Note that $6 \times 10^7$ is just how many microseconds there are in a minute. Since we'll write a tick for every spike (binned in 1 ms bins), if we wanted to play spikes at their original speed, we should set ticks per quarter note to be 480, so that the duration of a tick is roughly 1 ms. However, that's definitely going to be really unpleasant to listen to, we want something slower. I'll set ticks per quarter note to 24, meaning that 1 tick will be ~21 ms, so we're making our track roughly 21 times slower. That means that since we had 56155 time points, each of 1 ms, our total track duration will be $(56155 \times 21) / 60000 \approx 19$ minutes.

Ok maybe that's a bit long, but it's ok for now, we can later play with the BPM setting in Ableton to adjust playback speed, and also simply cut it. Before we continue, since the MIDI controller data needs to be between 0 and 127, I'll quickly scale the LFP data to be in that range.

We initialize a MIDIFile object with one track per neuron. For details on the parameters, check out the docs.

We then add the spike times of each neuron to the corresponding track, together with the LFP data encoded in the MIDI CCs.

For the LFP, I generated a MIDI file which has one track per frequency band. Each track only has one note playing through the full time period, and the actual values of the filtered LFP and its power are again stored as controller events, which we will use to modulate the sound over time.

STEP 2: Playing our MIDI files in Ableton¶

Once we have created the MIDI files, let's drag the interneuron spikes into Ableton. In the envelope section, we can find the LFP data we wrote in the MIDI CC. But seeing is easier than reading, so:

Now we need a way to use the controller data to modulate our sounds in Ableton. Max for Live has a device for this, called Espression Control, but it doesn't provide the flexibility we need. Luckily, there is Espression Control Plus, which you can download for free. Once downloaded, you simply need to drag the .amxd file onto your track. You can then map MIDI CC 20 - that's our LFP in the delta band - to any parameter, like for example the release of your synthesizers. In this way, the LFP signal modulates how long your spikes-notes are. I follow these steps in this example:

This is the basic of the method, so now you're equipped to experiment with how you modulate your sounds using data in the MIDI CCs.

How I did it¶

In my first example, I created two different sounds, a shorter round tone for interneurons and a brassy tone with a bit of attack for pyramidal neurons. Interneurons are modulated by power in the delta frequency (MIDI CC 30), with the MIDI CC mapped to a macro which in turns changes a couple of things (opens the cutoff, increases the amplifier and filter decay, and increases the filter resonance). Pyramidal neurons are modulated by power in the theta frequency, where higher power essentially makes the sound 'longer' by increasing the attack of the filter and the amplifier and filter decay. It is also mapped to a delay, and higher power increases how much of the sound is sent to the delay. The notes I chose were solely based on my taste, I mostly have A, B, C, D, E, and G in there.

For the LFP, each frequency band plays one continuous note, the amplitude of the signal modulates filter frequency and resonance, whereas the power modulates the volume (so that we only hear the note when there's some power in the band).

Lastly, I made a MIDI file containing the times at which the orientation and frequency (of the gratings/tone that the mouse is exposed to) change, and mapped those to percussion sounds, so we can hear how the brain responds to change in visual and audio stimuli. When you hear a longer noise tone panned to the right, it means the visual stimulus has changed orientation (e.g. at 00:11 seconds), whereas a short tone panned to the left (e.g. at 02:38) signals that the frequency of the auditory tone has shifted. The two can also play at the same time if both stimuli change. I'll put the piece I made below if you'd like to have another listen.

This was just an example, I plan to make more tracks in the future, using data from different experiments, to see what the different parts of the brain sound like under different conditions!