Physical structure

This sketch provides a more realistic estimate of the size of the gumballs. We plan to obtain a box or old wooden chest and transform it into a container for our bubblegum sequencer. We will drill a grid of 16 x 16 holes in the surface of the chest, and place the camera inside. Since it will be necessary (see below) for the camera to recognize color, we will also light the underside of the grid from below.

Computer vision (Detecting where the balls are)

We acquired a Logitech webcam, whose input stream is available to applications as a Quicktime stream. We found some third-party code that makes this stream available to ImageJ as a series of still images. ImageJ is an open-source image processing library (originally for medical image processing, but now widely used in other areas) for Java.

Using some ImageJ base classes, we process the pixels of each frame separately in the HSB color space. We have an adjustable grid that needs to be aligned with the holes of the sequencer surface as part of the calibration process. Around the center of each grid area, we measure an area of 10x10 pixels and compute the average hue of all pixels whose saturation and brightness levels are above a threshold that can be varied as part of the calibration process.

The mean hue will then be quantized to match the colors of our gumballs, or more abstractly, to generate discrete values between 0 and N (where N is the number of colors or instruments used). The measured and quantized values are written into a two-dimensional data structure which represents the current state of the gumballs on the surface.

The software seems to perform fairly robustly when used on test images printed on paper. A white background proved to be easier to process than a black background. Sufficient and even lighting is important, so we will most likely need to place a diffused light source in the box.

The colors of the gumballs need to be carefully chosen: It may, for example, prove problematic to use colors too close to one another on the HSB color circle (like yellow, orange, red and pink).

Test images

These two images approximate the appearance of well-lit gumballs against a white and black background. We tested the above computer vision system with each, first with the fully saturated colored circles (left), then with the images of gumballs (right).  As stated above, our test system detected color better against the white background, thus we will likely paint the underside of the grid white.

Making music

Once we have a data structure that represents the current state of the physical interface, this needs to be transformed into music. We plan to run a timer thread separate from the computer vision thread that loops generates a drumloop by repeatedly reading the gumball datastructure and generates MIDI events from this structure. These events will essentially consist of the color of each gumball encoded as pitch and possibly the vertical position encoded as velocity or something else.

These MIDI messages should be available systemwide to other applications or even other hardware. The goal is to make the sequencer act like someone is playing notes on a keyboard.