Letting the System Play With You: Notes on Procedural Music Game Design

It feels good to put this out in this state. Conceptually I think this is like 75% complete in terms of core mechanics and a procedural music system. There’s been so much upheaval in the possibilities for creative code. The spark for this project came in 2020 with Moonlight Flight, a mobile game that I developed with a few students in NERDLab. While locked down, I drifted back to something I haven’t done in a really long time: make music. 

The act of making improvisational music is packed with emotions. I started learning to play the guitar when I was in the 5th grade. My mom played a lot when I was in kindergarten but then stopped. Her guitar became a thing to play with. So one day a few of my friends tried to jam. We also had a Casio SK-10, with that amazing demo button. We convinced ourselves that we should perform for our class.

I lugged the instruments to class the next day. We asked our teacher for permission to play them at recess instead of going outside to the playground. She told us as long as we played something for the class afterwards, we could stay inside with the instruments. It felt like the shortest recess ever. We had nothing. Hit the demo button… and I think I might have hit a few strings on the guitar? Silence. Then the hot sting of embarrassment. 

I usually took the bus home, but with the instruments I sat on the curb waiting for my mom to pick me up from school. She had recently gone back to work as a real estate agent. Work happened for my mom, and I sat there with that huge guitar case for about an hour. It seemed like every person that walked by me had the same question “do you play the guitar?” No, no, no. When my mom finally arrived, I told her I wanted to learn to play the guitar.

After about a year of instruction, I had a level of confidence that if someone asked me “do you play guitar?” that I could say yes. My first instructor was a graduate student at a local college. He lived in a guest house in someone’s backyard, which I thought was pretty cool. He had turned it into a recording studio, so when he taught me a blues scale and riff he was able to produce an instrumental track that I could practice with. At lessons, my instructor and I would take turns with rhythm and lead. I loved picking apart a scale and being a part of that creative energy. It carried over into playing with friends in high school. My technophilia merged with music in high school, leading me to experiment with music production software like Acid Pro. I would record my band playing songs and then chop up a lot of it into loops. As a band, we rarely performed. The act of creation was more important and the possibility of capturing that through technology really spoke to me.

I enjoy the experimentation aspect of making music. Long ago I thought that it was a universal feeling, but I realized as I tried to “spark up a jam” with people that knew how to play an instrument, I saw that some people would prefer to stick to music that’s already been written. My soul wants to believe that it’s just anxiety and that there’s some primal element to music that’s deeply human - especially the collaborative creation.

So this game. Between now and 2020, I had some epiphanies. The first is that I’m done with mobile. When I started making digital games, the app store was the best way to get something in front of an audience. Today without a marketing budget, your game will never be seen. But also, I just don’t like mobile phones. I thought they were so cool in 2008. Then they became attention magnets. And when was the last time I enjoyed playing a mobile game? I’m not saying that there aren’t good mobile games, but when was the last time you played a mobile game that could only be played on a phone because of a specific design affordance? And why would I want someone to play my game due to a few minutes of separation or fear of boredom related anxiety?

My favorite games are the ones I played with friends on a couch. That shared sense of time and place with people is deeply meaningful. Around the time that I was feeling this, the technical solution I had used for Moonlight Flight to patch loops together wasn’t going to work as part of a game mechanic. Synchronizing the loops of a bunch of rendered wav files was a memory hog. I decided to explore MIDI. I started playing with the MPTK asset and it seemed like the best approach was to export everything I had composed into MIDIs.

This was a bit of a nosedive. I tried using MIDI clips as loops that queued for specific instruments. Ableton’s MIDI export options were limiting my process. I really liked the idea of someone playing what I wrote, but for my sanity I had to let that particular idea go. I embraced procedural music generation which now feels more like the system is playing with you. That co-creative vibe feels right when I think about my own enjoyment of making improvisational music.

The first experiment was to let the player collect randomized notes that came from a list of allowed notes and steps. Each of these notesets were tied to multiple instruments and the player would play a series of sequences. The drums are MIDI clips that I’m looping and queueing up.

The second experiment is fixing bugs and making the code behave more like what was in my head. The first experiment made this cool soup of notes but the time sequencing, while synchronized, still had a few problems with the rules for what constituted a step. After a few back and forths with ChatGPT, it seemed like the best path forward was using rendered wav files for drum loops.

In the current experiment, I changed the allowed notes and steps to be generated as a scale and style for an instrument. I set how notes should behave (lead, harmony, bass, drone) and their style (syncopated, swing, sparse, constant), the root note, the lowest note, the highest note, and the scale. These values comprise a noteset and the player progresses through a long loop of these. The drums are controlled through another system. Evolving obstacles are generated as part of a grid as each drum loop comes to an end. If an obstacle evolves past a hazardous state it becomes a collectable star. When the player collects 4 of these stars, the drums progress to the next pattern.

It feels like the right time for feedback. This version can be played with Nintendo controls on a Mac or any gamepad on a Windows machine. More than one player can join, just hit start on any connected controller to join at the beginning, select a ship with the arrows and fly around. I haven’t done any tests with multiplayer and the latest version yet. If you happen to be able to play this with a friend, I’d love especially to hear about your experience.