Edit: For newcomers to the blog, I'm Dave Sharples and featured in the photo above and video below is David Glanzman, who was my intrepid teammate for the project. This project was my senior design project for the Digital Media Design major at the University of Pennsylvania. It doubled as the final project for an Embedded Systems class David and I were in together.
The Joytone is more or less finished! We didn't manage to finish writing the GUI for the screen, but we'll chalk that up to a future project. For now we have polyphony, smooth timbre control, even an intelligent pitch bend. The pitch bend ranges from -2 semitones to +2 semitones, and dynamically figures out what the corresponding distance in Hertz should be based on what note you're playing (this even works over several notes, with a different interpolation for each one).
There are more details about the core concept behind this project in the first post on the blog, but I'll quickly summarize it here. We wanted to invent an entirely new electronic musical instrument, and there were two things we wanted to focus on in the design of the interface. The first is that we wanted to improve upon the physical design of musical instruments. Most acoustic instruments are designed around physical phenomena that make sound rather than convenience for the user (violins are smaller than cellos because shorter strings make higher notes). This instrument uses a hexagonally isomorphic layout, which means that the notes are distributed on a hexagonal grid, and they all have the same physical size and shape. They also have the same musical relationships to each other - if you move to the right by one thumbstick, that corresponds to going up a perfect fourth musically, and this is true no matter where you start on the grid. This means that musical structures like a major chord or a minor scale are always the same shape, no matter which note you start on, which is pretty rad. The second thing we wanted to focus on was making the instrument really expressive. Lots of synthesizers only measure how hard you strike the key and other timbre controls are available on knobs that you have to remove your hands from the keyboard to use. We're using little thumbsticks, which give you two dimensions of analog control in a familiar physical interface.
Below is a video of our final product! We just finished our last demo, wrapping up my senior design project. Playing the instrument is my teammate for the project, David Glanzman, who is a talented musician.
I thought it would be cool to show some of the final build process too! Here's the rough sketch of my plan for the physical enclosure. The wires coming from the thumbsticks were going to need some headroom, so there's a 3" vertical space at the bottom for all the wire routing. The thumbsticks would need to be mounted on some sort of board (we looked at perfboards originally, but couldn't find one large enough and ended up using acrylic). We also needed a wall structure to divide up the lights and prevent bleed, and a top sheet with holes sized correctly to prevent over-rotation of the thumbsticks, but still allow a large enough range of motion to be subtle and musical.
These are the acrylic pieces, fresh out of the laser cutter. In black is the top piece, with a press-fit hole for the frosted acrylic panel. At the top is the wiring plate, to which the thumbsticks were glued. Each thumbstick has three slots for wires to come through - one for each joystick and another for the click action. There is also a small circular hole cut for the lights. At the right is the stability plate, which is designed to slide over the thumbstick bodies and provide extra grip on the sides to prevent them from breaking loose. Unfortunately the thumbstick I was using to test the size of these holes had weird dimensions that differed from the rest, so the stability plate didn't fit over most of the thumbsticks. We ended up scrapping it because it would have been too much work to file down each of the slots, and we didn't have enough acrylic to cut another sheet. Luckily the glue ended up being strong enough to hold the thumbsticks.
Here's a shot of some of the honeycomb structure. Since we wanted to be able to control the light for each thumbstick to give user feedback, we wanted to have good color separation when the lights were on. I designed this hexagonal honeycomb that fits around the thumbsticks and stretches from the wiring plate up to the frosted top sheet. This is only two layers of the six required to fill that vertical distance. The very top layer is black acrylic for extra visual impact, while the lower ones are white for maximum light bounce. The layers are all glued together with methylene chloride, which is a really neat kind of acrylic glue. It wicks into the gap and chemically welds the plastic together in the space of about 10 seconds. Nifty stuff. Below are the hexagonal by-products of cutting the honeycomb layers. We ended up with 342 of these. Some are translucent acrylic rather than white because that's all we could find. They're close enough.
The next step was to take the frosted sheet with the thumbstick holes and glue it into the black top sheet. I cut the pieces with a 0.003" overlap for a press fit, but I glued it together with the methylene chloride again for extra strength since it was surface people would be interacting with. To the right, the top sheet is resting on top of the partially finished honeycomb (note that the black layer of the honeycomb hasn't been added yet).
Here's the top assembly being tested on the wooden enclosure (a simple box with a ledge about an inch down on the inside to act as a shelf for the wiring plate).
Next was the wiring. This was far and away the most tedious part of this build. Each of the 57 thumbsticks needed six wires - both potentiometers needed power and ground, and each one had a signal wire. We came up with this common bus scheme where we daisy chained all the power and ground connections along diagonal rows, then linked up all the rows in a zig-zag pattern. The power and ground wires have little nicks in the wire where we soldered the thumbsticks on. Since each of the eight multiplexers we used had 16 channels (eight on each side, labeled Top and Bottom), we grouped the thumbsticks' signal wires into bunches of eight, and soldered on a male header so we could plug them in more easily to the ADC board. We also installed the lights at this stage. Each of the 8mm through-hole NeoPixels has 4 leads, for power, ground, data in, and data out. These were also daisy chained along diagonal rows in a big zig zag, and the lights were close enough together that we were able to just solder the leads together.
Next we powered up the lights to test them. The clear thumbstick caps were a good choice - they conducted light really well. Unfortunately, that rainbow pattern means it doesn't really work. In theory, since these lights were all daisy chained together, we should be able to send the first light 57 lights' worth of color information. It should then strip off the first packet and push the rest down the line, where the next one strips off one packet and pushes the rest of the data, until all the lights are updated. However, after about four lights, our signal turned into noise and produced random, flickering color. The lights seemed to work fine on a breadboard, but something about this configuration doesn't work (perhaps too much distance between the lights, or bad solder joints, or cross talk). We decided to leave the lights with the default blue color that comes up when they power on and make this a future project. The lights do look pretty though.
We also tested the lights with the cover in place, just to see how it looked. The honeycomb structure works well! There's nice separation between the colors, and the frosted acrylic diffuses the light for a more even glow. In an ideal world, we'd be able to control all these lights with an Arduino Micro and show the user which notes are in or out of key, or guide them through a song by lighting up the next note to play, or give them suggestions to help complete chords based on what note they're currently playing.
Next we started installing the processing components. We screwed the components into the box for extra stability. Closest to the camera is our ADC board (at the top of it are the 8 multiplexers we had to hook the thumbsticks into). Towards the back is the Raspberry Pi, which is connected to a little GPIO breakout we wired up, and a custom power board. We found a 5 volt 5 amp power supply, which would be able to run all 57 LEDs (which draw 60 milliamps at full brightness, for 3.42 amps total) and the Pi (which in turn powers the ADC board).
Here's the first assembly of the entire system. This photo was taken minutes before my senior design final presentation. We hadn't finished wiring up the lights at this point, so we demoed without the extra wow factor. We also hadn't finished writing the code to make it polyphonic, so we were playing in monophonic mode (one note at a time) during the demo. It's actually a miracle this worked at all, considering we'd been awake for about 48 hours.
The very next day was our final demo for Embedded Systems, which we were also using this project for. We managed to get the lights to turn on with the rest of the system by this point, but we realized we left the light data wire exposed, which would make them flash in random patterns when you touched it. Also it was still monophonic.
At the end of the class demos, our teacher announced that we would all be demoing again for the faculty in a week. So we had a little more time to add polish to the project. We wired up some of the panel-mount buttons, added the touch ribbon for pitch bend, and finished the code for polyphony. We didn't manage to get the screen working, but we figured it was a low priority compared to more prominent features like pitch bend. This is what we had for our very last demo - where faculty from the School of Engineering and Applied Science and the School of Design stopped by to see all the cool projects the class had produced this year.
As a visual aid, here's a map of how things are wired up on the inside
The design of the system is covered in much greater detail in the paper (linked at the top of this post), but I'll give a quick overview. The thumbsticks are connected to an array of multiplexers that feed the signals into 8 analog input pins on the Cypress PSoC 4. The PSoC then uses an internal analog multiplexer to feed the signals into the onboard ADC, then communicates the data via SPI to the Raspberry Pi. A python script running on the Pi processes the data and communicates with the synthesis program, Pure Data via OSC. Our goal was to also have color data sent to an Arduino Micro to control the RGB LEDs, but we didn't get that component working.
Overall, I'm really happy with how it came out. Looking back at the original goals for the project, I think it hits the mark on all of them. It gives control over loudness, pitch, and timbre with efficient gestures. It's easy to play accurate pitches, but still allows for subtle variations and vibrato. It offers several dimensions of timbre control. It's polyphonic, intuitive, and ergonomic. I could never have expected to end up with such a rad object, and I'm thankful for all the help and advice I've received along the way (especially for my teammate David Glanzman, without whom this instrument wouldn't exist, and Maggie for being supportive and patient with me).