Around November of last year I was contacted by the Toronto International Film Festival. They have an annual exhibit called the DigiPlaySpace, which is an interactive technology exhibit for kids. The curator for this year's show had stumbled upon my website and asked if the Joytone could be shipped up to Toronto to be on display in this year's show.
The Joytone 1.0 was built for my senior design project at the University of Pennsylvania. It was my first real electronics project, and there were a number of time and cost constraints that led to a lot of quick and dirty design decisions. I told the curator that the Joytone 1.0 would break, but that I could build another one, even bigger and badder.
So I signed up for a membership at the Techshop in San Francisco and got to work. I taught myself how to layout PCBs, got custom circuit boards printed, bought a whole lot of LEDs, and took some woodworking classes. With the time and resources I always wanted to put into the Joytone, I was able to build a much more fully realized version of it. There are a ton of build pics below, but first - some glamour shots of the final product:
The Design Challenge
If you're a newcomer to the blog, some of the older posts have more information about the thoughts and design goals behind the first Joytone, but I'll summarize them briefly here. I wanted to build a new electronic musical instrument, and I had two design goals. The first was to improve on the physical interface of musical instruments. Many acoustic musical instruments are designed around the physical phenomena that produce sound - a cello is larger than a violin because longer strings make lower notes, not because the larger form factor is advantageous for the player. The constraints of physics and a great deal of history have developed the physical interfaces we're familiar with today, and most electronic instruments seek to emulate their acoustic counterparts. Since electronic instruments aren't constrained by the mechanism of sound production, there's a really awesome opportunity to improve on the user experience of playing an instrument. The second design challenge I tackled was the expressiveness of electronic musical instruments. Many electronic instruments lack the same expressive depth as the acoustic instruments they're designed after, so I wanted to build an interface that would allow really rich, flexible control over the sound.
The Joytone's interface consists of a hexagonal layout of 72 joysticks, just like the ones you'd find on an Xbox 360 controller. The hexagonal arrangement of identical sensors creates an "isomorphic" layout. This has a distinct advantage over the non-isomorphic layouts of many musical instruments. As an example, a piano has white keys and black keys, which are different sizes and shapes. To play a C Major scale is very easy - simply play all the white keys in order from left to right. To play a C# Major scale requires using three of the black keys. Even though they're both Major scales (the same pattern of musical intervals), they're played in different ways. There are 12 Major scales, and each one requires memorizing a specific pattern of white and black keys - and that's not counting the various other flavors of musical scales. An isomorphic layout like the one on the Joytone removes all that bias. Once you know the pattern of jumps around the hexagonal grid required to play a major scale, that pattern is consistent no matter where you start from. The pattern for playing a C Major scale is the same as the pattern for a C# Major scale, they just start on different notes. Similarly, all major chords have the same shape, as do all minor chords.
The joysticks give the instrument huge expressive capability. I chose joysticks because they offer two dimensions of analog controls in a familiar and inviting physical interface. Each joystick controls one note. The vertical axis mixes between two different waveforms - a warm soft triangle wave, and a bright reverse sawtooth wave. The horizontal axis varies the tuning between a pair of oscillators - at one extreme they're exactly in tune, providing a narrow, flat sound, and at the other extreme the oscillators are out of sync by a couple hertz, producing a fat, wide sound. The distance the joystick has been moved from its resting position determines the volume of the note. With a really simple interaction, the player can control three distinct parameters. The Joytone can play six notes at once, and can control each of the three parameters separately for each note, allowing the player to emphasize specific notes in a chord, or use a rich palette of sounds all at once.
How It All Began
If you look back at some of the older posts you can see some of the refinement work I did with the interface design. Not documented on the blog is the whole design process leading up to that. I walked into my senior design project with the idea to build an isomorphic musical instrument, and walked out with the Joytone - there's a good design story there, but I had spent the six months or so leading up to my senior design project reading and sketching and learning about music theory. Someday I'd like to document all the ideas that weren't used. After taking a music theory class I began thinking about the design of instruments because we learned about the patterns behind music - for example, how every major chord is composed of the same set of musical intervals. As a complete beginner I found it easy to learn all these patterns but was frustrated by the enormous effort required to translate what was in my head to the physical interface of a piano, and I started to wonder if that interface could be improved.
For version two I had an opportunity to revisit some design decisions I made previously. I wanted to have custom printed circuit boards (PCBs) made so it would be more reliable and easier to fix, which meant that I had to have a repeatable pattern that I could break out into a number of identical PCBs. I ended up choosing to make nine rows of eight joysticks each for a grand total of 72, in a rough square pattern.
I also decided to use an Arduino Mega with a special piece of firmware on it called HIDUINO. This allowed the Joytone to act as a class-compliant MIDI device when plugged into a computer via USB. My laptop was running the Pure Data patch, taking the place of the Raspberry Pi from Joytone V1.
The first thing I did was buy lights. The lights were far and away the most annoying part of the first Joytone, so instead of buying individual LEDs I opted for pre-wired strands of 25 (after all, I had some funding now). I would have tried to build the LEDs directly into the circuit boards, but I wanted to try to keep the circuit boards simple as it was my first time designing one.
I used a piece of software called Express PCB to do the board layout. It once I got the components and vias placed correctly based on the physical constraints, routing the traces was kind of like a game. It took a couple of attempts, but I ended up with this as the design. It's a two-layer PCB with a ground plane in the rear and all the signal traces on top.
The PCBs came back from the factory looking perfect! I got 25 of them printed because the marginal cost of an additional copy is pretty small, and I figured I might want some spares for repairs / other instruments. I spent a lot of time soldering in the Techshop - doing the multiplexers was the most nerve-wracking part since they're so small.
A multiplexer is like a big multi-position switch. The ones I used were 16-channel analog multiplexers - they have 16 inputs and one output, and can connect any of the input signals to the output based on the state of four channel-select wires. So each board needed power, ground, four channel-select lines, and a signal output line; seven wires total. Every wire but the signal output could be common between all nine rows of thumbsticks, which meant I could build a neat little custom Arduino shield to handle all the routing. Below is the paper design and the final implementation. The power, ground, and channel select wires are the long horizontal rows, and the signals from the selected channels come out the bottom of each of the nine columns and get routed to the Arduino's analog inputs. There are also connections for the screen, the three layout buttons, and the power for the lights.
The screen was easy to hook up also. Below is a picture of the first successful test of the screen. The color of the backlight is controllable via PWM from the Arduino.
While I was waiting on some parts to come to help with the electronics I started building the enclosure. It had to be really sturdy to survive six weeks in a kids' museum, and I wanted it to be easily serviceable in case it needed repairs. I ended up with this wedge-shaped design that would allow me to slide the electronics out easily. It also has a nice 22 degree angle to lift the play surface up to a comfortable position.
The screen and buttons are mounted to a black faceplate that has a frosted acrylic window for the joysticks to poke through (it also diffuses the light nicely). Below that is another sheet of acrylic that holds the PCBs, and between the two layers is a honeycomb structure that separates the cells and stops light from bleeding between them.
Laser cutting the frosted acrylic was stressful - I only had enough acrylic for two tries, and the insert had to be sized correctly within a couple thousandths of an inch. I messed up the first time, but the second one fit perfectly!
This is when the magic really started to happen. The lights worked marvelously, and the clear plastic joystick caps conducted the light beautifully. The picture to the left is a shot of a light test with the honeycomb. The top layer is dark blue acrylic for more dramatic borders between the cells when viewed through the acrylic. In the picture to the right, the instrument is highlighting a C Major scale. The purple joysticks are the Cs across various octaves, and the green joysticks are other in-key notes.
I wanted the instrument to be serviceable, so I designed the enclosure with a removable back. The acrylic sandwich holding the electronics could be slid back, tilted, and lifted out easily.
This is a shot of the back. There are an awful lot of wires, but they're all color coded and organized, so it's a little more manageable than last time. Here you can see the cables for the screen in front, then along the right side are the nine sets of cables coming from the PCBs. To the left are two holes through the side for power and the USB cable.
And here's the final product. The instrument is in "rainbow mode" here, which is better for free play. The joysticks are color coded so that a note is colored consistently across octaves. Since there are 12 musical notes in an octave, 12 colors were used in the palette. It's a little hard to tell them apart, so I'm looking into alternate color schemes.