Assignment: Synthesis: Music Instrument (group work)

Collaborators:

Collaborators: nick, agreiner, sohyeong

Description

We have created a collaborative storytelling music mixer for children. Using the score of Prokofiev's Peter and the Wolf, the system allows users to manipulate music tracks by moving pieces on a playing surface. Users have two types of pieces that they can manipulate. First is a set of plastic statuettes representing the main characters in the Peter and the Wolf story: Peter, the wolf, the duck, the hunter, and the cat. We also have location pieces that invoke tracks of background "presence" sounds. The locations are Grandfather's house, the meadow, the lake, and the forest.

Initially, no tracks play until someone places a character or location piece in the active area of the table (a rectangular area in the center of the table in which the camera tracks movements). To encourage children to play with the pieces collaboratively, we have set up the system so that the nearer a given piece is to the center of the play area, the louder the piece's associated track plays. The tracks for the various characters play simultaneously, yielding a surprisingly pleasing combination of sounds that can be varied infinitely. When a piece is moved to the side, its track ceases to play. By putting character pieces together in the middle, essentially on stage, where everyone can reach, children can make the characters interact with each other and reenact the story. Children hear the associated theme music (played by its designated orchestral instrument) for a given character while focusing attention on that character. The location pieces can evoke a specific locale in which the scenes from the story take place. We chose to use background sounds rather than music for locations so that they are distinctive but do not interfere with the "action" of the game. The location pieces are also shaped differently, to make it easy to predict their function.

The system takes advantage of the open-source ReacTIVision library for tracking movements on a surface with a camera. Each piece has a printed paper "amoeba" shape attached to the underside of the base, which ReacTIVision recognizes as a specific object. We use the Minim library in Processing to handle audio manipulation and to integrate the tracking and audio into one application.

Materials

Support structure from old Bubblegum Sequencer
2 foot by 4 foot sheet of clear acrylic
toy statuettes
printed location images
printed "amoeba" shapes (from ReacTIVision PDF file)
foamcore posterboard (support for lamp, and bases for statuettes)
fluorescent lamp
webcam
clear packing tape
laptop computer running Processing (v155) and ReacTIVision
sound samples

Processing Code

/**
* Peter and the Wolf
*/

import ddf.minim.*;
import ddf.minim.effects.*;
import processing.serial.*;
import tuio.*;

Minim minim;
AudioPlayer[] grooves;
Character[] characters;
TableTracker tracker;
TuioClient tuio;
int maxX = 640;
int maxY = 480;

void setup(){
size(maxX, maxY);
minim = new Minim(this);
frameRate(12);
PFont myFont = createFont("Verdana", 12);
textFont(myFont);
// make character list
characters = new Character[11];
characters[0] = new Character(-1, "bird.aif");
characters[1] = new Character(0, "catmidi.aif");
characters[2] = new Character(3, "ducknoisered.aif");
characters[3] = new Character(-1, "grandfather.aif");
characters[4] = new Character(4, "hunters.aif");
characters[5] = new Character(2, "peter.aif");
characters[6] = new Character(1, "wolf.aif");
characters[7] = new Character(6, "forestquiet.aif");
characters[8] = new Character(10, "grandpashouse.aif");
characters[9] = new Character(5, "lakequiet.aif");
characters[10] = new Character(9, "meadowcrickets.aif");
// instantiate objects
grooves = new AudioPlayer[characters.length];
minim = new Minim(this);
tuio = new TuioClient(this);
tracker = new TableTracker(characters, tuio);
tracker.maxX = maxX;
tracker.maxY = maxY;
}

void draw()
{
background(0);
tracker.update();
updateAudio();
updateVis();
}

void updateAudio(){
for(int i=0; i<characters.length; i++){
if (characters[i].onStage){
//the piece is in the active area
if(grooves[i] == null || (grooves[i] != null && !grooves[i].isPlaying()) ){ //the song isn't playing already
//start the song
grooves[i] = minim.loadFile(characters[i].getFileName(), 1024);grooves[i].loop();
grooves[i].loop();
}
else{
//set the volume;
//calculate the distance from the center
double distance = Math.sqrt(Math.pow((maxX/2) - characters[i].getX(), 2) + Math.pow((maxY/2) - characters[i].getY(), 2));
//scale the distance to the gain range (-40 to 0)
double maxDist = Math.sqrt(Math.pow(maxX, 2) + Math.pow(maxY, 2));
int scaledDist = (int)(0 - (40 * distance/maxDist));
grooves[i].setGain(scaledDist);
}
}
else{
//the character is not on the surface
if(grooves[i] != null && grooves[i].isPlaying()){
//stop the song
grooves[i].close();
grooves[i] = null;
}
}
}//end of the for loop
}

void updateVis() {
Character c;
for(int i=0; i<characters.length; i++){
c = characters[i];
if (c.onStage){
fill(255);
ellipse(c.getX(), c.getY(), 10, 10);
fill(204,0,0);
if (c.getX() > (width/2)) textAlign(RIGHT);
else textAlign(LEFT);
text(c.getFileName() + " (" + (int)grooves[i].getGain() + ")", c.getX() + 10, c.getY());
// println(c.getId() + " is at " + c.getX() + ", " + c.getY());
}
}
}

void stop()
{
// always close Minim audio classes when you finish with them
// always stop Minim before exiting
minim.stop();

super.stop();
}

/*
*class for Character objects
*/

public class Character{
private int id;
private int x;
private int y;
private String soundFile;
public boolean onStage;

public Character(int id, String soundFile){
this.id = id;
this.soundFile = soundFile;
x = 0;
y = 0;
}

public int getX(){
return x;
}

public int getY(){
return y;
}

public String getFileName(){
return soundFile;
}

public int getId(){
return id;
}

public void setX(int newX){
x = newX;
}

public void setY(int newY){
y = newY;
}

}

/*
* Tracker for Peter and the Wolf table.
*/

public class TableTracker {
private TuioClient _tuio;
private Character[] _cList;
public int maxX;
public int maxY;

public TableTracker(Character[] cList, TuioClient tuio) {
_cList = cList;
_tuio  = tuio;
}

public void update() {
// zero-out the character positions
for (int j=0; j<_cList.length; j++) {
_cList[j].onStage = false;
}
// get the table objects
TuioObject[] tuioObjectList = _tuio.getTuioObjects();
for (int i=0; i<tuioObjectList.length; i++) {
TuioObject tobj = tuioObjectList[i];
// find the appropriate character and update
for (int j=0; j<_cList.length; j++) {
if (_cList[j].getId() == tobj.getFiducialID()) {
_cList[j].setX(tobj.getScreenX(maxX));
_cList[j].setY(tobj.getScreenY(maxY));
_cList[j].onStage = true;
}
}
}
}

}