08-09 IUAV Interaction Design Lab 2 Projects

This page explains the main functions of our codes. We divided it in two parts: Processing code and Arduino code.

Processing code
Interaction: The code controls when a blade has to play. It was not enough programming it to play when the flex sensor is bent over than a certain value, because in that case you would hear a sound played as long as the flex is bent.
The note has to play just once, and it will play again only after the blade has turned back to its default position. We defined some boolean variables called stoSuonando, stoPiegando, stoContando, to know the blade current status and calculate when we could activate the sounds.
Moreover, we also had to know the amount of bending to regulate the volume. Thus, we set a counter that starts when a blade is bent and stops either after 30 milliseconds or when the blade gets back to the default position. We then map the obtained value to make a proportion such this:
absolute maximum bending : bending value reached = maximum volume : current volume.

// FLEX IN DEFAULT POSITION
// reset sounds, bending value and delay counter
if(arduino.analogRead(i) > flexDefault[i]-variazioneFlex
&& arduino.analogRead(i) < flexDefault[i]+variazioneFlex) {
  stoSuonando[i] = false;
  stoPiegando[i] = false;
  stoContando[i] = false;
  conta[i] = 0;
  bend[i] = 0;
}

// DELAY COUNTER
// counts till millisDelay value, then stops
if(stoContando[i] = true) {
  if(conta[i] < millisDelay) conta[i]++;
  else {
    stoContando[i] = false;
  }
}

// FLEX BENT
// detects that it's active, and starts counting
if(arduino.analogRead(i) <= flexDefault[i]-variazioneFlex
|| arduino.analogRead(i) >= flexDefault[i]+variazioneFlex) {
  stoPiegando[i] = true;
  if(conta[i] < millisDelay) stoContando[i] = true;
}

// Calculates max bending achieved before the delay counter stops
if(stoPiegando[i] == true && stoContando[i] == true) {
  newBend[i] = arduino.analogRead(i);
  if(newBend[i] > bend[i]) bend[i] = newBend[i];
}

// PLAY SOUND
if(stoSuonando[i] == false && stoContando[i] == false) {
  if(bend[i] > 0) {
    stoSuonando[i] = true;
    volume[i] = map(bend[i], flexDefault[i], maxFlex, 0, 1);
    client.write(i + " and " + volume[i]);
    suono[i].volume(volume[i]);

    if (suono[i+10] != null){
      if (suono[i+10].state==Ess.PLAYING) {
        suono[i+10].stop();
      }
    }      

    suono[i+10].play();
    conta[i] = 0;
  }
}

Sounds: The computer plays a sound file each time a blade is bent in the opposite installation. It receives, from the server, which note it has to play, and at what loudness. Then, according to the microphone value, a reverberation effect will be added.
We used ESS library for Processing to work with sounds. A problem with this library is that, once it applies an effect to a file, that file is saved into Processing memory as filtered. So, if you want to play it once again, you will not “overwrite” the new reverberation effect, but you will add it to the previous one. To avoid this, we had to re-associate the original sound file each time we played a sound.

void playSounds(int whichSound, float whatVolume){

  // REVERBERATION
  // uses the amount of wind blowing: the more wind, the more riverberd
  reverberation = map(absWind, 0, 200, -0.2, 2);
  if(reverberation < 0) reverberation = 0;

  // if it was playing, it stops to allow the same sample to start again
  if (suono[whichSound] != null){
    if (suono[whichSound].state==Ess.PLAYING) {
      suono[whichSound].stop();
    }
  }

  // resets the sound file, to eliminate the filtering
  suono[whichSound] = new AudioChannel(soundFiles[whichSound]);

  // reverberation effect
  if(reverberation > 0.2) {
    env_lv0[0] = new EPoint(0,0);
  }
  else {
    env_lv0[0] = new EPoint(0,1);
  }
  env_lv0[1] = new EPoint(reverberation,3);
  envelope_lv0.points = env_lv0;

  envelope_lv0.filter(suono[whichSound]);

  suono[whichSound].volume(whatVolume);      // sets volume
  suono[whichSound].play();                  // play sounds
}

Communication Processing-Arduino board: Since we needed to elaborate the values given by our sensors, we chose to control everything through Processing, that communicates with Arduino thanks to the Firmata library (to know more have a look below).
In order to do it, we wrote in the declaration part:

import processing.serial.*;
import cc.arduino.*;

Arduino arduino;
Serial port;

Then we needed to declare which pins on the Arduino board would receive the INPUTS and which ones would be set for the OUTPUTS.

// ARDUINO   /////////////////////////////////////////////////////////////////////////
  arduino = new Arduino(this, Arduino.list()[0], 115200);
  arduino.pinMode(0, Arduino.INPUT);

  for (int i = 7; i < 12; i++){
    arduino.pinMode(i, Arduino.OUTPUT);
  }

The values coming from one of the flex sensors attached to the Arduino board (e.g. the one connected to the pin number 4), could be read with:

arduino.analogRead(4);

“.analogRead” because the flex are connected to the analog pins.
On the other side, to send an output (e.g. to turning on our LEDs), we used:

arduino.digitalWrite(7, Arduino.HIGH); //this command turns on the LEDs connected to the pin 7

arduino.digitalWrite(ledPins[i], Arduino.HIGH); // this turns them off

Communication client-server: Our two computers were connected to the same Wifi net in order to communicate to each other.
Basically each computer has to read the sensors values, process and then send them to the other computer, being a client for it. At the same time they both receive the values from the other one, so they must behave at the same time as servers.
To create a server we wrote in the declaration part:

import processing.net.*;

Client client;
Server server;
...

Then in the setup:

// SERVER / CLIENT  ///////////////////////////////////////////////////////////////////

  // Starts a server on port 10002
  // Please note: only clients need to know a target IP address, the server
  // just listens to a given port
  server = new Server(this, 10002);
  //client = new Client(this, "172.16.248.227", 10002);
  serverRunning = true;

  println("server starting");

The last line (println(“server starting”);) is useful to know if the server starts to work when you run the code.
In the void draw(), then, we wrote the instructions to check if there is a message and in that case to read it and send the value at the right function (playSounds(getFlex, getVolume);).

// SERVER   /////////////////////////////////////////////////////////////////////////
  if(client != null) {
    if (client.available() > 0) {
      String message = client.readString();
      println(message);
    }

  if(serverRunning == true) {
      //check for the next client in line with a new message
      Client thisClient = server.available();

      // is there a client?
      if(thisClient != null) {
        // check to see if there is a message from the client
        if(thisClient.available() > 0) {
          // read in the message
          String message = thisClient.readString();

          if(message.length() > 8 ) {
            getFlex = int(message.substring(0,1));
            getVolume = float(message.substring(7));

            // plays the sample sounds of given note and volume when a message is received
            playSounds(getFlex, getVolume);
          }

          println(message);

          // this is a very simple server - it's write method it just broadcasts to
          // every client connected
          server.write(message);
        }
      }

The first three lines check if there is a client set for the server.
The problem of having a server and a client running on the same code, is that the server function should be started before the client one. That’s why we decided to run the client manually, by pressing the key “c” on our computers’ keyboard. In that way we were sure that both the two server started before the client and so the communication could happen regularly.

The function that makes the client starting is:

void keyPressed() {
  if (key == 'c'){
    startClient();
  }
}

void startClient(){

  //create a new client
  //this takes an IP address (localhost means connect to myself)
  //and a port number
  client = new Client(this, "172.16.248.227", 10002);
  println("client starting");
  client.write("ciao Vale");
}

The function that sends the right value to the other computer is in the flexActive() function and it is simply this:

  client.write(i + " and " + volume[i]);

Arduino code
As we said before, we control our Arduino board directly from Processing.
What allowed us to do it is the Firmata library (that you can download here), which is contained in the Arduino library for Processing.
Firmata library contains some useful examples, like Simple_Analog_Firmata or Simple_Digital_Firmata. In our case we needed to use both analog and digital sensors or actuators, so we combined those two examples together creating our Arduino analog-digital.

The good thing of controlling everything from Processing is that once you uploaded this code inside the Arduino board, you will never need to change it.
The code settings can be made only on your computer, in the Processing code!

Arduino code:
Steli | Arduino analog-digital (ZIP | 84 KB) (PDF | 44 KB)

08-09 IUAV Interaction Design Lab 2 Projects

… code