Physical structure

This sketch provides a more realistic estimate of the size of the gumballs. We plan to obtain a box or old wooden chest and transform it into a container for our bubblegum sequencer. We will drill a grid of 16 x 16 holes in the surface of the chest, and place the camera inside. Since it will be necessary (see below) for the camera to recognize color, we will also light the underside of the grid from below.

Computer vision (Detecting where the balls are)

We acquired a Logitech webcam, whose input stream is available to
applications as a Quicktime stream. We found some third-party code that
makes this stream available to ImageJ as a series of still images. ImageJ is an open-source image processing
library (originally for medical image processing, but now widely used
in other areas) for Java.

Using some ImageJ base classes, we process the pixels of each frame
separately in the HSB color space. We have an adjustable grid that
needs to be aligned with the holes of the sequencer surface as part of
the calibration process. Around the center of each grid area, we
measure an area of 10x10 pixels and compute the average hue of all
pixels whose saturation and brightness levels are above a threshold
that can be varied as part of the calibration process.

The mean hue will then be quantized to match the colors of our
gumballs, or more abstractly, to generate discrete values between 0 and
N (where N is the number of colors or instruments used). The measured
and quantized values are written into a two-dimensional data structure
which represents the current state of the gumballs on the surface.

The software seems to perform fairly robustly when used on test
images printed on paper. A white background proved to be easier to
process than a black background. Sufficient and even lighting is
important, so we will most likely need to place a diffused light source
in the box.

The colors of the gumballs need to be carefully chosen: It may, for
example, prove problematic to use colors too close to one another on
the HSB color circle (like yellow, orange, red and pink).

Test images

These two images approximate the appearance of well-lit gumballs against a white and black background. We tested the above computer vision system with each, first with the fully saturated colored circles (left), then with the images of gumballs (right).

As stated above, our test system detected color better against the white background, thus we will likely paint the underside of the grid white.

Making music

Once we have a data structure that represents the current state of
the physical interface, this needs to be transformed into music. We
plan to run a timer thread separate from the computer vision thread
that loops generates a drumloop by repeatedly reading the gumball
datastructure and generates MIDI events from this structure. These
events will essentially consist of the color of each gumball encoded as
pitch and possibly the vertical position encoded as velocity or
something else.

These MIDI messages should be available systemwide to other
applications or even other hardware. The goal is to make the sequencer
act like someone is playing notes on a keyboard.