2014-02-18

Co.Labs

Hacking House Plants To Make Music When They’re Touched

As we discovered in our lab, not every touch interface needs to be glass or plastic.



All of our gadgets are glass, metal, and plastic. Wouldn’t it be nicer to interact with a computing device that was alive and real?

After watching this video we went to see a talk by Ivan Poupyrev, former head of Disney Research, at Maker Faire in NYC.

When we got back, we whipped up four touch-sensitive plants that can be played like musical instruments. Real plants. Ones that need water and the sun for food and grow out of the dirt from little seeds. Our goal was to someday have people say things like “I’m mellow, I play the bamboo.” Or, “I am hardcore so I play the cactus.”

How The Musical Plant Works

The plants we hacked play different notes depending how they are grabbed. For example, when someone touches the stem the note is different than when they touch its outer leaves. When grabbed with two fingers instead of one, the sounds are different; same goes for other hand positions.

We reproduced a version of Disney’s “Touche,” the technology Poupyrev and his team built to encode the frequencies that conductive materials like water, human bodies, and plants, among other materials carry whenever they are touched by a human. The process was much like the one that allows your smartphone to be touch sensitive.

We build a Touche from a tutorial by Mads HoBye, Instructables' artist-in-residence, who hacked his own version using a small Arduino.

A Breakdown Of Touche With Arduino

Before jumping into the gritty detail of our project, we break down the code from the Instructable you will use to build a Touche shield.

The “Touche Arduino” is made of three components--the circuit board that amplifies the current caused by human touch, the Arduino which encodes this current as raw data, and Processing, which receives the data and parses it as an array of 160 frequencies via serial connection. A serial message sends sequential data one bit at a time, over a communication channel like a USB port.

Going Wireless

We wanted to keep things organic and keep the plants as far from the computer as possible. Our goal was to use the Yún, a new Wi-Fi-enabled Linux board combined with Arduino, to allow the plants to speak to the computer wirelessly.

Due to several challenges, we had to use a standard Arduino Uno hooked up to an iMac instead of our original plans with the Arduino Yún. We will explain the golden knowledge we learned in the process of failing.

Let’s Get Real Time: A Story Of Trial And Error

We explored different ways to have the Yun inside the plants talk to the computer over Wi-Fi. The sole idea of playing an instrument is that you generate sound in real time. When you pull a string on a guitar you immediately hear the chord out loud. We could not afford to have our office musicians wait longer than half a second before they heard the first note and the next after.

Trial 1: Access Pin Via REST API

The data that’s generated when you touch your plant is available via REST API because the Arduino Yún supports an Internet connection. The size of each encoded frequency is 160, so we needed to make a rest call 160 times just to get the value of a single frequency.

You can read about making REST API calls to an Arduino Yún here.

Trial 2: JSON

Our second option was making the Arduino structure the data before it returned it. When the user touches the plant, the Arduino Yún has the ability to structure an array of size 160 that stacks the data from the selected analog pin which the user calls.

The client makes the call:

myArduinoYun.local/arduino/plants.json

And the array that is created from that is returned:

[{1:153},{2:186}, …. {159:78}]

Although this reduced our original 160 calls down to 1, there was still a three-second delay because the Arduino was internally structuring that data into a stacked array.

Trial 3. Bridge / {Key:Value}

The Bridge library is a great predefined function on the Yún which allows communication between Arduino and Linux.

By invoking the Bridge.put method in your code, the data is saved to the Arduino captures as a 160 key-value pair array in the internal storage of the Linux. Then you can simply access each value by calling: http://myArduinoYun.local/data/get/KEY

And the Yún, or server side, returns the value inside the key slot in an array:

For example, invoking the call: http://myArduinoYun.local/data/get/159 would return 78.

This was still not good enough. There was a delay in the auditory response averaging 1.6 seconds and 5 in its worst-case scenario.

Trial 4: Shell Command

Since the Linux side of the Yún supports the shell command, this allowed us to write a lump of data to an micro SD card inside the Arduino.

We used a single line of code:

p.runShellCommand(echo "data" >> /mnt/sda1/arduino/www/logger/plant.json).

This was by far the longest running time we encountered, taking from 10 to 27 seconds before we got auditory response.

What We Learned

Although we did not end up using the Arduino Yún, we learned it is a great tool with many capabilities. It can be fast but not optimal for a project that requires real-time interaction. The connection will always be limited to the speed of the network. It did not help that our project dealt with such large amounts of data for each individual reaction to a frequency.

We are still hatching out a way to incorporate the Yún into our project because we believe there is still a lot of room for exploration with its capabilities and will write an update soon. In the meantime, here are the instructions to re-create our project with an Arduino Uno and a computer.

What You Will Need

Circuit board from Mads Hobye

Computer

Speakers

Copper Wire

Arduino

Processing

Minim Library for Processing

A Windowfarms stand for the plants (optional)

Plants

1) Install Arduino, Processing, and the Minim Library onto your computer.

2) Download our code from the Co.Labs GitHub page, an adaptation of Mads Hobye’s Arduino Touche.

3) Build the circuit board from the Arduino Touche Instructable following the instructions here. We looked around and found the best shopping choice for the components you’ll need:

4) Plug the USB cable to the Arduino board and your computer to run the Arduino code you downloaded.

5) Open and run the Processing code from our GitHub.

6) The paper clip should be hooked to your copper wire attached to your circuit board. Place the paper clip inside the dirt of your plant.

7) Check to see what values your plant is sending to the Processing console window. If you are not getting any value, check your circuit board and make sure your paper clip is securely inside the dirt.

8) Optional: If you want to pick the sounds your plant makes when it is touched in different places, add your own MP3 files inside the data folder inside your Processing sketch. We got our sounds from this open source website.

9) Compose your own symphony with your new musical plant!

Update: A Way To Make The Project Independent With The Yun

We found a way to cut out the Mac component for making this DIY. This time, you will need an Arduino Yun instead of the Uno. The solution was simple. All we needed was a little USB Audio Adapter that can be purchased at any Best Buy or online retailer like Adafruit for under $15 dollars.

Since the Arduino Yún has its own Linux support, Linino, it can pretty much be treated as its own independent computer. By writing the music to the SD card, and adding a simple line of code, brought to our attention by Mikamai developers,

p.runShellCommand("madplay /mnt/sda1/guitar1.mp3");

we had music playing on your Yún within seconds, no Mac necessary. Be sure that after you install the opkg update and the kmod-usb audio mentioned in the tutorial above, you also install the opkg madplay. This is what will allow you to play music. Also, note that since we no longer have the mac to power the Yun, now you’ll need to attach any compatible external batteries.






Add New Comment

2 Comments