Problem statement

Since some time in the 1980's, step sequencers have been a popular tool to create and edit music, particularly drumloops. Drumcomputers like the Roland TR-808 have relied on this method heavily.

A step sequencer consists of a number of positions (typically 16) that represent points in time in a soundloop (in this case 16 note positions comprising one measure). By turning on or off these discrete positions (often represented as buttons with LEDs to indicate their status), the user places a sample at that particular point in the loop.

If there is only one physical row of buttons available, a selector typically changes the track so that the user can edit several instruments just by switching tracks.

When played back, a running light often indicates the current playback position of the loop.

However, only being able to directly manipulate one track at a time can be limiting. Software sequencers have long solved this by simply adding additional tracks as rows in the sequencing grid. Only recently, hardware like the Mononome were developed to address this. The Mononome consists of several rows of LED-lit buttons that can be used to control software step sequencers.

Proposed solution

We aim to make step sequencers more physical by using tangible objects to represent samples. Our project builds on the work of Durrell Bishop's Marble Answering Machine and Patten, Recht, and Ishii's AudioPad by mapping musical samples to colored marbles (or likely gumballs), which can be set into holes arranged in a grid on a tabletop. As in software sequencers, columns in the grip map to beats (time) in the loop. A camera placed below the surface of the table reads the colors in each column to generate the loop. Rows in the grid, however, no longer needed for sample assignment, can be mapped to other MIDI data, for example velocity, so that placing a marble higher on the grid will generate a louder sound.

While the mapping between samples and colored marbles is not ideal, such physical embodiment does provide differentiation that arbitrary assignment of samples to rows in a GUI or LED grid lacks. Users can assign samples to colors of their own choice (provided each color chosen is distinguishable by the camera from the others), and remember, for example, that black marbles are kick drums, and red ones snare drums. Mapping to color rather than some recognizable iconography requires some arbitrary memorization, but provides much more flexibility for customization.

Sketches / designs

Technical implementation

We plan to implement this system through a combination of computer vision and MIDI messages.

The table consists of a board with holes in it, so that the marbles are visible from below. Below the surface, a vertically mounted camera takes a continuous picture of the table. The image is fed into a computer vision system which divides the image into a grid and (possibly after some segmentation) computes average the average hue of each hole (proper lighting will be required). The system transforms the state of the board into a two-dimensional data structure and sends it to the next software layer to process.

The next layer can either consist of an improvised step sequencer or of a transformer that forwards the state of the board as MIDI messages to another music application, thus acting as a plugin for existing software.