November 24, 2007
Reading for November 27th, are now posted. Enjoy!

October 2, 2007
To upload your thoughtless acts, create a new assignment page like any other lab. You'll see "Thoughtless Acts" listed as one of the assignment options.

May 24, 2008
This site has been archived and is no longer editable. Stay tuned for the next version, coming in the fall!

Bubblegum update

Project Members: 
Andrew McDiarmid
Hannes Hesse
Sooyeon Han

Physical structure

Physical Interface

This sketch provides a more realistic estimate of the size of the gumballs. We plan to obtain a box or old wooden chest and transform it into a container for our bubblegum sequencer. We will drill a grid of 16 x 16 holes in the surface of the chest, and place the camera inside. Since it will be necessary (see below) for the camera to recognize color, we will also light the underside of the grid from below.

Computer vision (Detecting where the balls are)

We acquired a Logitech webcam, whose input stream is available to applications as a Quicktime stream. We found some third-party code that makes this stream available to ImageJ as a series of still images. ImageJ is an open-source image processing library (originally for medical image processing, but now widely used in other areas) for Java.

Using some ImageJ base classes, we process the pixels of each frame separately in the HSB color space. We have an adjustable grid that needs to be aligned with the holes of the sequencer surface as part of the calibration process. Around the center of each grid area, we measure an area of 10x10 pixels and compute the average hue of all pixels whose saturation and brightness levels are above a threshold that can be varied as part of the calibration process.

The mean hue will then be quantized to match the colors of our gumballs, or more abstractly, to generate discrete values between 0 and N (where N is the number of colors or instruments used). The measured and quantized values are written into a two-dimensional data structure which represents the current state of the gumballs on the surface.

Calibration software GUI

The software seems to perform fairly robustly when used on test images printed on paper. A white background proved to be easier to process than a black background. Sufficient and even lighting is important, so we will most likely need to place a diffused light source in the box.

The colors of the gumballs need to be carefully chosen: It may, for example, prove problematic to use colors too close to one another on the HSB color circle (like yellow, orange, red and pink).

Test images

white backgroundblack background

These two images approximate the appearance of well-lit gumballs against a white and black background. We tested the above computer vision system with each, first with the fully saturated colored circles (left), then with the images of gumballs (right).  As stated above, our test system detected color better against the white background, thus we will likely paint the underside of the grid white.

Making music

Once we have a data structure that represents the current state of the physical interface, this needs to be transformed into music. We plan to run a timer thread separate from the computer vision thread that loops generates a drumloop by repeatedly reading the gumball datastructure and generates MIDI events from this structure. These events will essentially consist of the color of each gumball encoded as pitch and possibly the vertical position encoded as velocity or something else.

These MIDI messages should be available systemwide to other applications or even other hardware. The goal is to make the sequencer act like someone is playing notes on a keyboard.


Comments from TUI Teaching Team

From Dave:
Guys, I really like this idea and looking forward to seeing it in person. I know you've been spending a lot of time on the implementation, and I think the idea is pretty solid as well. Nice work. 

From Kimiko:
Looks great. It would be nice to get a feel for how the music is created using this sequencer. Would a musician place one ball at a time? Multiple balls could be spread at the same time? Move multiple rows or columns of balls at the same time? How does the affordance of bubble gum sized balls help the musician or create unique opportunity for making music? In other words, for your presentation, be prepared to discuss how the “tangibility” of your system contributes to the music making. Good luck!

Powered by Drupal - Design by Artinet