Problem statement

Since some time in the 1980's, step sequencers have been a popular
tool to create and edit music, particularly drumloops. Drumcomputers
like the Roland TR-808 have relied on this method heavily.

A step sequencer consists of a number of positions (typically 16)
that represent points in time in a soundloop (in this case 16 note positions
comprising one measure). By turning on or off these discrete positions
(often represented as buttons with LEDs to indicate their status), the
user places a sample at that particular point in the loop.

If there is only one physical row of buttons available, a selector
typically changes the track so that the user can edit several
instruments just by switching tracks.

When played back, a running light often indicates the current playback position of the loop.

However, only being able to directly manipulate one track at a time
can be limiting. Software sequencers have long solved this by simply
adding additional tracks as rows in the sequencing grid. Only recently, hardware like the Mononome were developed to address this. The Mononome consists of several rows of LED-lit buttons that can be used to control software step sequencers.

Proposed solution

We aim to make step sequencers more physical by using tangible
objects to represent samples. Our project builds on the work of Durrell
Bishop's Marble Answering Machine and Patten, Recht, and Ishii's AudioPad by mapping musical samples to colored marbles (or likely gumballs),
which can be set into holes arranged in a grid on a tabletop. As in
software sequencers, columns in the grip map to beats (time) in the
loop. A camera placed below the surface of the table reads the colors
in each column to generate the loop. Rows in the grid, however, no
longer needed for sample assignment, can be mapped to other MIDI data,
for example velocity, so that placing a marble higher on the grid will
generate a louder sound.

While the mapping between samples and colored marbles is not ideal,
such physical embodiment does provide differentiation that arbitrary
assignment of samples to rows in a GUI or LED grid lacks. Users can
assign samples to colors of their own choice (provided each color
chosen is distinguishable by the camera from the others), and remember,
for example, that black marbles are kick drums, and red ones snare
drums. Mapping to color rather than some recognizable iconography
requires some arbitrary memorization, but provides much more
flexibility for customization.

Sketches / designs

Technical implementation

We plan to implement this system through a combination of computer vision and MIDI messages.

The table consists of a board with holes in it, so that the marbles
are visible from below. Below the surface, a vertically mounted camera
takes a continuous picture of the table. The image is fed into a
computer vision system which divides the image into a grid and
(possibly after some segmentation) computes average the average hue of
each hole (proper lighting will be required). The system transforms the
state of the board into a two-dimensional data structure and sends it
to the next software layer to process.

The next layer can either consist of an improvised step sequencer or
of a transformer that forwards the state of the board as MIDI messages
to another music application, thus acting as a plugin for existing
software.