For alpha reviews last week I managed to get the chain of communication working from physical input to sound output. With that success I decided to focus on further planning, hunting for hardware, and design work this week. I mentioned earlier that I'm going to use this project as the final project for my Embedded Systems class as well, allowing me to spend twice as much time on it. The project divides pretty neatly - for Embedded Systems I'm going to focus on the large scale I/O problem, the sensor technology, and the physical design of the instrument (most of this work will be done on the STM32F4 microcontroller), for Senior Design I'm going to focus on the communication between the STM and the RaspberryPi, the sophistication of the sound output, the experience design of the instrument, and the visual feedback to the user (most of this development will happen on the RaspberryPi).
Rahul ordered two STM32F4 boards for this project, and I also recently found out that the PennHacks group has 10 of them lying around that were donated to the event. I also read online the that the microcontrollers are now compatible with mbed - a popular online IDE and compiler that makes it much easier to write code for these boards. When I get back from break I can start really developing for this platform. My first goal is to port over the existing setup and extend it to use multiple thumbsticks.
One of the reasons I wanted to use thumbstick mechanisms as the physical input device was that it would allow me to take advantage of the vibrant game system modding community. One of the most interesting mods is this little guy:
The mod kit comes with two clear plastic caps that sit on top of the thumbstick mechanism, and a tiny PCB with rings of LEDs that fit around the thumbstick. The clear plastic gets illuminated by the LEDs to great effect. I think it would be awesome if each of the 36 thumbsticks were fitted with a ring of lights and a clear cap; there are some really excellent user interaction opportunities here. If the instrument could use colors to differentiate between what notes are in and out of the currently selected key, it could level out the learning curve. If a user is trying to learn a song, the instrument could illuminate the next note the user should play, guiding him through the song. By changing the color of there thumbstick as you actuate the note, the user could gain a much more intuitive understanding of the instrument (e.g. as a filter is applied gradually, the light fades to red, indicating that the effect is active). The little boards in these mod kits can cycle through 7 colors - this likely corresponds to cycling through the different combinations of turning the red, green, and blue LEDs on or off. Of course, for 36 keys, we now have 108 digital outputs to wrangle, but that problem isn't intractable. One big issue is price. Each of these kits costs $15, some wholesalers will sell them for as low as $8 ish, but that's still nearly $300 and we'd be throwing out all the little built-in controllers we don't need. It might be possible to build a custom illumination system for the thumbsticks, provided the LEDs are bright enough. I've done a little scrounging around the resources at school and found some RGB LEDs to test for this purpose.
There's also another bit of useful hardware I've found.
That's a $50 3.5" touchscreen for the RaspberryPi. It would make a great interface for the instrument, allowing the user to cycle through synthesis presets, select light colors, view the internal state of the software, or activate FX. There is a little synth called the OP-1 made by Teenage Engineering that has an amazing little display with great retro-looking graphics. Below is a series of 8 screenshots of different info displays on the synth.
This is going to be my design inspiration going forward. The displays above represent some incredibly detailed and intricate behaviors of the synth engine. My instrument likely won't be as complex but I think a screen like this would be useful to indicate what note you're playing, what effects are active, and the system status of the raspberry pi. I've started doing some sketches of these display already.
The last thing I've been working on and thinking about is an auxiliary input mechanism for broad effects, much like the relationship a stomp box has to a guitar. Little touch strips and knobs on the side of the instrument could control the volume or a filter or the master delay for all the notes, in addition to selecting transposition modes for the notes or the base octave of the instrument. The construction of the instrument and placement of these controls will be determined by the ergonomics of the instrument - something I'm working on but haven't fully defined yet.