Social Sequencer - interactive music device
Social Sequencer is an interactive music device that allows you to create and compose music by simply arranging cubes on a sleek and intuitive board. Each cube represents a unique note or percussion sound, allowing you to build and layer intricate melodies and beats with ease. Here is a video of the device in action:
The full installation consists of three boards representing different instruments, the idea here is to allow people to play together and make music collaboratively creating a social experience. We created this together with my good friend and electronics enthusiast Donatas Valiulis.
What’s next?
I want Social Sequencer to travel around all sorts of art and technology events and spark people’s interest in electronic music. I’m also looking for a more long term placements in galleries or cultural spaces. If you are interested or know a place where this would fit nicely, please let me know. I am also interested in experimenting with adding video component to the device, so that not only sound, but also video would be controlled via it.
Appearances
Social Sequencer has already traveled through numerous locations and events. It was shown in big industrial spaces, festivals, corporate gatherings, swampy forests and private events. Here are some memorable appearances.
Social Sequencer was a proud participant of Vilnius TechPark second birthday festivities welcoming people from business and IT sectors for some talks, socializing and a big party.
No Trolls Allowed 2018 hacker camp was another memorable location where Social Sequencer participated. It’s an amazing event held near a beautiful lake close to Vilnius, Lithuania where people enthusiastic about building stuff gathers for two days to share their projects, participate in workshops and have a late night heated beer driven discussions about which minor version of some obscure framework is better. It was really interesting to show it to a very tech savvy crowd, discuss various technical details and dive deep into implementation nuances.
One of the first places where Social Sequencer was fully assembled and demonstrated was it’s birthplace: LinkMenų Fabrikas. It was one of the attractions alongside a myriad of other devices, prototypes and inventions during one of their events.
The most memorable place for me is still The hiccup cure of Braille Satellite 2018 music festival where we set everything up around the huge old oak tree at the edge of the forest. People could play with it standing around the tree and the sounds were broadcasted via radio waves to the receivers scattered throughout the area.
Festival goers were also the perfect crowd for testing out this device, during night time some serious jamming was happening around the tree. People even called it “Oak Stage”.
The initial plan was to build Social Sequencer in time for “The summit of Braille Satellite 2017” music festival. We heavily underestimated the required effort, and had to work late nights before the festival to finish the minimal version with just one board, but when we went to the spot to set the installation up, everything started not to work, and we ended up spending half a night looking at the computer screen sitting on the wet ground under the tree being bitten by swarms of mosquitoes. It definitely doesn’t help to fix software bugs when you are constantly being bitten by the biological ones. We ended up leaving it with just the lights on and some cryptic writings around it to make it look more like some kind of weird conceptual art installation.
The idea
The idea of Social Sequencer was born during Vilnius Light Festival 2016. We were roaming around the city and looking at all kinds of light installations. Many of them looked really impressive and beautiful, but they were all static. That made me think how nice it would be to create something that was interactive and allow participants to be fully immersed in the experience. That’s when the idea of creating the board with cubes representing musical notes struck me.
It was one of these rare cases where the original concept didn’t really change much during the production process and the initial idea turned out to be pretty close to how the final product ended up.
The core mechanic is based on sequencer or drum machine, the devices that allow one to create repeating patterns of notes or drum sounds and play them in a loop. First analog sequencers date back to as early as 1940s and were widely adopted during 70s and 80s. Robust and feature rich versions of them are an integral part of every modern music creation software like Ableton or Cubase. Here is a nice video walking you through the history of drum machines .
Developing the concept
It was clear that building the prototype just by ourselves would be too expensive and time consuming, so I started looking for various funds and organizations that might support prototyping the idea. I had to invent quite a lot of stories describing the device as well as things like it’s “cultural importance”, “unique value proposition”, etc. that were needed for all sorts of applications. It was quite a tedious and boring process but the good things that came from there were that I had a much clearer idea about how the thing might works as a whole, not just from technical perspective, and also that I made this video showing how the final installation will look like.
I had to learn how to do some 3D modeling with Blender to make this video and it turned out to be much more exhausting and swearword intensive experience than I expected, but that’s another story.
One more concept that came up during this process was using cubes made from different materials so the way the cube feels in your hand would correlate with what sounds that cube produces. So that for example cubes made from metal would produce “cold” and metallic sounds and the ones made from soft materials produce “soft” and “fluffy” ones.
We didn’t implement this part as it proved a bit too involved, costly and having many problems, like metal ones being very heavy, glass ones being sharp, etc. We ended up adding different ornaments to indicate different type of sounds.
The technology
It was an extremely interesting working on Social Sequencer as we needed to do lots of different stuff starting from choosing the right approaches and chips, soldering electronics, programming Arduino micro-controller, using CDC machines to cut out physical boards, designing and developing our own PCB boards, making it work via MIDI with Ableton and eventually migrating all audio set-up to headless mode running on Raspberry Pi and PureData. This section is all about nitty gritty details of the process.
Sanity check
From the technical perspective, the first problem we had to solve was to determine if there is a simple and robust enough solution to “read” the arrangement of cubes that are on the board so we could turn them into notes. Few different approaches were considered including each cube having distinct QR code and camera being placed inside the device, using RFID tags inside each cube and some weird stuff with lasers or ultra sound proximity detectors or more primitive materials like tin foil, but all of them were either too complicated or too prone for errors.
We ended up choosing the approach of using the matrix of Hall effect sensors inside the board and magnets inside each cube. This approach seemed to make most sense as Hall effect sensors were cheap, small and easy to work with and the only thing that needed to be inside the cube is a simple magnet. We also saw potential to have cubes representing different variations of same note by using magnets of different strengths. But most importantly it seemed like a solution that we could prototype easily in a reasonable time frame.
After buying few Hall effect sensors and magnets, we constructed a first prototype using Arduino micro-controller and cardboard box from tomato juice. To our surprise everything went pretty smoothly and we were able to detect when the object with magnet is placed in the spot and when it is removed. There were some issues with the strength of the magnets being a bit too similar to cover many different variations, but it was clear that just using the polarity of magnets would allow to detect two different variations reliably. So we decided that the approach is good enough.
First prototype
We started working on the first prototype after we struck a deal with LinkMenų Fabrikas, a creativity and innovation space run by Vilnius Tech university. We approached them and they were interested and offered to allow us to use their premises, equipment and consultations from their engineers working in various fields. We were very happy with this collaboration as LinkMenų Fabrikas turned out to be this very cool place with lots of very enthusiastic people and every sort of machine you might want from big soldering stations and all sorts of electronic components to huge laser engraver cutter machines and 3D printers. Many other cool projects were being built there at the time including Dance For The Dawn installation for Burning Man 2017 .
Electronics: PCB board
After a round of discussions with their engineers we all agreed that the approach of using Hall effect sensors still makes most sense and that we gonna go with that. So the next thing we had to figure out is how to feed the signals from three matrixes of 40 Hall effect sensors each into single Arduino controller. We decided to go with the cascade of multiplexer chips. Multiplexers are chips that have multiple inputs and allow you to read any of them through single wire. This way if we have one multiplexer for each row of sensors we can read data from all 8 of them only using two wires (one for reading data and the other for switching between the inputs). With five rows of these and the fact that they can all share control line (as we are reading the whole column at the time) it was enough to put one more multiplexer in front of the initial ones to be able to read the data from whole board of 40 Hall effect sensors via two wires.
After verifying the approach on a prototyping board we designed the PCB board using Sprint Layout software. That was the first time I have made my own PCB board from scratch so it was really interesting to see how the process looks like. Apparently you just print the layout on a normal paper, use laminating machine to transfer it to the special copper covered plate, wash the paper layer and then keep the plate in a ferric chloride solution for a few hours. This removes the unwanted copper and leaves just the designed circuit layout.
After that was done we just soldered all the chips in and the board was ready to be used, this is how the finished PCB board with multiplexer set-up ended up looking:
Electronics: LEDs
We also knew we want to have programmable LED strip inside each board to be able to illuminate and mark the current tick. This was I think the first time I tried hooking up LED strip to Arduino. I got instructions from my electronics mentor Ričardas on how to power the LEDs up from one of the big boxes that provide AC current that I have never seen before. He said to set the right voltage and turn on the device. So I turned on the device and then tried setting the right voltage. Of course that resulted in all LEDs flashing bright for a brief moment followed by darkness. There were quite a few incidents like this during the process, we were regulars at the local electronics store.
After some playing around we found the right approach on how to place these LEDs and what materials to use in between them to get an effect of the whole column lighting up. Initially we thought of just lighting up the column that is being active, but after accidentally running the “rotating rainbow color” code on the board we decided that it is too cool to not be used. It decreased usability a bit as it was harder to spot the active column marker, especially in daylight, but the wow effect greatly out-weighted the concerns. We ended up using LED strip with WS2811 driver chips and SMD5050 LEDs that is controllable from Arduino via SPI protocol. It was actually very interesting to learn how this protocol works by using just a single control wire with any number of chained LEDs.
Software: Arduino
To be able to read the sensor data, update LEDs and send MIDI note messages we hooked it all to Arduino micro-controller. The implementation was pretty straight forward in theory but we ran into many problems while trying to make it work in actual settings. Most issues arose when trying to read the sensor data. Apparently things like temperature and nearby objects emitting electromagnetic radiation significantly affect the output of the Hall Effect sensors. We ended up implementing calibration routines that have to be run every time the device is installed in new place. After we resolved this issue, other things went pretty smoothly. Here is how the final code looks like:
/***************************************************************************
This is the main code for Social Sequencer device, full info about the project:
https://tamulaitis.lt/project/social-sequencer
Author: Giedrius Tamulaitis, giedrius@tamulaitis.lt
Version: 0.8
***************************************************************************/
#include <Adafruit_NeoPixel.h>
#ifdef __AVR__
#include <avr/power.h>
#endif
// Tempo of an internal clock controlling the speed of the sequencer (in BPM)
#define INTERNAL_CLOCK_BPM 240L
// Use external MIDI clock signal instead of an internal clock (not yet implemented)
#define USE_EXTERNAL_MIDI_CLOCK false
// If debug mode is on various info is printed to Serial port
#define DEBUG_MODE true
// If sketch is started with CALIBRATION_MODE set to true it will wait for two seconds, perform
// calibration once, print matrixNoFieldVoltages array to Serial port and brick
#define CALIBRATION_MODE false
// Duration of calibration procedure (in milliseconds)
#define CALIBRATION_DURATION 8000
// Number of boards connected to the brain
#define BOARD_COUNT 2
// Number of supported cube types
#define CUBE_TYPE_COUNT 4
// Number of columns in the board
#define COL_COUNT 8
// Number of rows in the board
#define ROW_COUNT 5
// Analog output with no applied field, enable CALIBRATION_MODE and paste results here
long matrixNoFieldVoltages[][ROW_COUNT][COL_COUNT] = {
// Board 0
{
{48196, 48900, 49300, 48292, 49600, 49000, 48800, 49611},
{49143, 48299, 49362, 49014, 48298, 48894, 49023, 48600},
{48811, 48208, 48499, 49473, 49466, 48808, 49426, 49400},
{49354, 49065, 48599, 49124, 48892, 48869, 47599, 49649},
{49095, 48903, 49203, 49176, 49097, 49189, 49498, 49125}
},
// Board 1
{
{49156, 49424, 48097, 49601, 48698, 48534, 48299, 48696},
{48309, 49193, 48594, 48993, 48600, 48499, 48102, 48799},
{48399, 48698, 48596, 48903, 48702, 48500, 48829, 48699},
{49298, 48793, 49298, 48799, 48597, 48100, 49076, 49568},
{48802, 48597, 49159, 49299, 48744, 48700, 49301, 48532}
}
};
// MIDI note mappings for different cube types
const byte midiNotes[][CUBE_TYPE_COUNT][ROW_COUNT] = {
// Board 0
{
{0, 1, 2, 3, 4},
{5, 6, 7, 8, 9},
{10, 11, 12, 13, 14},
{15, 16, 17, 18, 19}
},
// Board 1
{
{20, 21, 22, 23, 24},
{25, 26, 27, 28, 29},
{30, 31, 32, 33, 34},
{35, 36, 37, 38, 39}
}
};
// Pins to which LED strip data channels of the boards are connected
const int LED_STRIP_CONTROL_PINS[] = {22, 24};
// Pins to which main multiplexer outputs of each board are connected
const int BOARD_INPUT_PINS[] = {A0, A1, A2};
// First dimention is board index, second is control line (0 = A, 1 = B, 2 = C)
const int MAIN_MUX_CONTROL_PINS[][3] = {
{37, 39, 41}, // Board 0
{36, 38, 40} // Board 1
};
// First dimention is board index, second is control line (0 = A, 1 = B, 2 = C)
const int SUB_MUX_CONTROL_PINS[][3] = {
{31, 33, 35}, // Board 0
{30, 32, 34} // Board 1
};
// Aliases for multiplexer line names just for comfort
#define LINE_A 0
#define LINE_B 1
#define LINE_C 2
// If Hall sensor voltage jumps by more than VOLTAGE_SPIKE_TRESHOLD and stays above
// for MAX_VOLTAGE_SPIKE_DURATION measuring cycles we no longer count it as a random
// spike and consider that this is a legit change in magnetic field
#define MAX_VOLTAGE_SPIKE_DURATION 3
// Amount (in voltage * 100L) the voltage has to jump for it to be considered a spike
#define VOLTAGE_SPIKE_TRESHOLD 2000L
// Maximum number of samples from which the average voltage is calculated
#define VOLTAGE_AVERAGING_SAMPLE_LIMIT 200
#define VOLTAGE_MEASURING_TIMEOUT_AFTER_TICK 20 // In milliseconds
#define VOLTAGE_MEASURING_AFTER_MAIN_MUX_SWITCH_TIMEOUT 2 // In milliseconds
// Number of LEDs in LED strip of each board
const int LED_STRIP_PIXEL_COUNT = 48;
// LED strip objects of each board
Adafruit_NeoPixel ledStrips[] = {
Adafruit_NeoPixel(LED_STRIP_PIXEL_COUNT, LED_STRIP_CONTROL_PINS[0], NEO_GRB + NEO_KHZ800),
Adafruit_NeoPixel(LED_STRIP_PIXEL_COUNT, LED_STRIP_CONTROL_PINS[1], NEO_GRB + NEO_KHZ800)
};
// How often to update LED strip colors (in milliseconds)
#define LED_STRIP_UPDATE_FREQUENCY 20
// Last LED strip color update time
unsigned long lastLEDStripUpdateTime;
// LED strip iterator used for achieving rainbow cycle effect
int ledStripIterator = 0;
// FX strip is scraped for now, will need to sort this mess out later
#define FX_STRIP_LED_FADE_SPEED 20
#define FX_STRIP_SEGMENT_LED_WHEEL_DIFFERENCE 20
#define FX_STRIP_SEGMENT_COUNT 4
#define FX_STRIP_LEDS_PER_SEGMENT 3
const int FX_STRIP_LED_COUNT = FX_STRIP_SEGMENT_COUNT * FX_STRIP_LEDS_PER_SEGMENT;
int fxStripLEDBrightness[FX_STRIP_LED_COUNT];
uint8_t fxStripLEDColors[FX_STRIP_LED_COUNT][3];
long fxStripLEDLastChangeTime[FX_STRIP_LED_COUNT];
int fxStripLastSegment = 0;
Adafruit_NeoPixel strip = Adafruit_NeoPixel(FX_STRIP_LED_COUNT, 26, NEO_GRB + NEO_KHZ800);
// Signal definitions for multiplexer control lines, first dimention is channel (0-7), second is a control line (0 = A, 1 = B, 2 = C)
const bool MUX_CONTROL_SIGNALS[][3] = {
{false, false, false}, // Channel 0
{true, false, false}, // Channel 1
{false, true, false}, // Channel 2
{true, true, false}, // Channel 3
{false, false, true}, // Channel 4
{true, false, true}, // Channel 5
{false, true, true}, // Channel 6
{true, true, true} // Channel 7
};
// Array for storing number of samples no field voltage of each Hall sensor is calculated from
long matrixNoFieldVoltageSampleCount[BOARD_COUNT][ROW_COUNT][COL_COUNT];
// Array for storing how many times the voltage spike was encoutered in a row
int matrixVoltageSpikeCount[BOARD_COUNT][ROW_COUNT][COL_COUNT];
// Array for storing number of samples averaged voltage of each Hall sensor is calculated from
long matrixAverageVoltageSampleCount[BOARD_COUNT][ROW_COUNT][COL_COUNT];
// Array that holds the averaged current voltages across the boards
long matrixAverageVoltages[BOARD_COUNT][ROW_COUNT][COL_COUNT];
// Array that is holding current state of the matrix (what cube types are placed where)
int matrixStates[BOARD_COUNT][ROW_COUNT][COL_COUNT];
// This is used to convert the analog voltage reading to milliGauss
#define TOMILLIGAUSS 3756L // For A1302: 1.3mV = 1Gauss, and 1024 analog steps = 5V, so 1 step = 3756mG
// Index of currently active column
int activeCol = COL_COUNT - 1;
// Index of column that was active before current one
int prevCol = COL_COUNT - 2;
// Index of column that will be active next
int nextCol = 0;
int activeRow = 0;
// Time last tick was invoked
unsigned long lastTickTime;
// Duration of one tick
unsigned long tickDuration = 60L * 1000L / INTERNAL_CLOCK_BPM;
unsigned long lastMainMuxRowSwitchTime;
unsigned long mainMuxSwitchDuration;
bool isFirstIterationAfterVoltageMeasuringTimeout = true;
uint32_t activeColor;
uint32_t colorWheel[256];
int colorWheelConstant = 256 / LED_STRIP_PIXEL_COUNT;
#define CALIBRATION_BUTTON_PIN 53
#define CALIBRATION_DURATION_IN_CYCLES 2
bool doStartCalibrationModeInNextCycle = false;
bool inCalibrationMode = false;
int currentCalibrationModeCycle;
#define INDOCATOR_BLINK_DURATION_DURING_CALIBRATION 20
unsigned long indicatorLEDLastChngeTime;
bool indicatorLEDState = false;
unsigned long lastFrameTime;
void initColorConstants() {
for (int i=0; i<256; i++) {
colorWheel[i] = Wheel(i);
}
activeColor = ledStrips[0].Color(255, 255, 255);
}
void initIndicatorLED() {
pinMode(13, OUTPUT);
digitalWrite(13, LOW);
}
void initCalibrationButton() {
pinMode(CALIBRATION_BUTTON_PIN, INPUT_PULLUP);
}
void updateIndicatorLED() {
if (inCalibrationMode || doStartCalibrationModeInNextCycle) {
if (millis() - indicatorLEDLastChngeTime > INDOCATOR_BLINK_DURATION_DURING_CALIBRATION) {
indicatorLEDState = !indicatorLEDState;
digitalWrite(13, indicatorLEDState ? HIGH : LOW);
indicatorLEDLastChngeTime = millis();
}
}
}
void setup() {
Serial.begin(9600);
Serial1.begin(31250);
lastTickTime = millis();
lastLEDStripUpdateTime = millis();
initBoards();
initFXStrip();
initColorConstants();
initIndicatorLED();
initCalibrationButton();
initMatrixVoltageArray();
if (CALIBRATION_MODE) {
delay(2000);
performCalibration();
printNoFieldVoltageArray();
while (1);
}
lastFrameTime = millis();
}
void loop() {
// Serial.println(millis() - lastFrameTime);
lastFrameTime = millis();
updateMatrixVoltageArray();
processLEDStrips();
if (USE_EXTERNAL_MIDI_CLOCK)
processExternalMIDIClock();
else
processInternalClock();
// processFXStrip();
// strip.show();
updateIndicatorLED();
}
/******************************************************************************
CORE FUNCTIONS
******************************************************************************/
int getCubeTypeByRelativeVoltage(long voltage) {
if (voltage > 2000)
return 3;
else if (voltage < -2000)
return 2;
return 0;
}
int _t_measureCount = 0;
void onTick() {
// Serial.print(activeRow);
// Serial.print(" - ");
// Serial.println(_t_measureCount);
incrementColIndexes();
activeRow = 0;
bool noteBroadcasted = false;
int currentBoardNr;
int currentRow;
int cubeType;
for (currentBoardNr = 0; currentBoardNr < BOARD_COUNT; currentBoardNr++) {
setActiveSubMuxCol(currentBoardNr, nextCol);
setActiveMainMuxRow(currentBoardNr, activeRow);
silencePreviousColNotes(currentBoardNr);
for (currentRow = 0; currentRow < ROW_COUNT; currentRow++) {
cubeType = getCubeTypeByRelativeVoltage(matrixAverageVoltages[currentBoardNr][currentRow][activeCol] - matrixNoFieldVoltages[currentBoardNr][currentRow][activeCol]);
// _debugPrintCRV(activeCol, currentRow, matrixAverageVoltageSampleCount[currentBoardNr][currentRow][activeCol]);
matrixAverageVoltageSampleCount[currentBoardNr][currentRow][activeCol] = 0;
if (inCalibrationMode) {
matrixNoFieldVoltageSampleCount[currentBoardNr][currentRow][activeCol] = 0;
}
if (!inCalibrationMode && !doStartCalibrationModeInNextCycle) {
if (cubeType != 0 && matrixStates[currentBoardNr][currentRow][activeCol] == 0) {
onNoteOn(currentBoardNr, currentRow, activeCol, cubeType);
}
if (DEBUG_MODE) {
_debugPrintCRV(activeCol, currentRow, (matrixAverageVoltages[currentBoardNr][currentRow][activeCol] - matrixNoFieldVoltages[currentBoardNr][currentRow][activeCol]) / 100);
// _debugPrintCRV(activeCol, currentRow, gaussValue);
}
} else {
if (DEBUG_MODE)
Serial.println("Calibrating...");
}
}
if (DEBUG_MODE)
Serial.println();
}
updateLEDStrips();
processCalibrationButton();
isFirstIterationAfterVoltageMeasuringTimeout = true;
}
void updateMatrixVoltageArray() {
if (millis() - lastTickTime > VOLTAGE_MEASURING_TIMEOUT_AFTER_TICK) {
if (isFirstIterationAfterVoltageMeasuringTimeout) {
mainMuxSwitchDuration = (tickDuration - (millis() - lastTickTime)) / ROW_COUNT;
// Serial.println(mainMuxSwitchDuration);
isFirstIterationAfterVoltageMeasuringTimeout = false;
lastMainMuxRowSwitchTime = millis();
_t_measureCount = 0;
}
bool doSwitchRow = ((millis() - lastMainMuxRowSwitchTime) >= mainMuxSwitchDuration) && activeRow != (ROW_COUNT - 1);
int nextRow = (activeRow + 1) % ROW_COUNT;
if (millis() - lastMainMuxRowSwitchTime > VOLTAGE_MEASURING_AFTER_MAIN_MUX_SWITCH_TIMEOUT || activeRow == 0) {
_t_measureCount++;
for (int currentBoardNr = 0; currentBoardNr < BOARD_COUNT; currentBoardNr++) {
long tempVoltage = analogRead(BOARD_INPUT_PINS[currentBoardNr]) * 100L;
if (!inCalibrationMode) {
if (matrixAverageVoltageSampleCount[currentBoardNr][activeRow][nextCol] < VOLTAGE_AVERAGING_SAMPLE_LIMIT)
matrixAverageVoltageSampleCount[currentBoardNr][activeRow][nextCol]++;
if (matrixAverageVoltageSampleCount[currentBoardNr][activeRow][nextCol] == 1)
matrixAverageVoltages[currentBoardNr][activeRow][nextCol] = tempVoltage;
matrixAverageVoltages[currentBoardNr][activeRow][nextCol] = (tempVoltage + matrixAverageVoltages[currentBoardNr][activeRow][nextCol] * matrixAverageVoltageSampleCount[currentBoardNr][activeRow][nextCol]) / (matrixAverageVoltageSampleCount[currentBoardNr][activeRow][nextCol] + 1);
} else {
if (matrixNoFieldVoltageSampleCount[currentBoardNr][activeRow][nextCol] < VOLTAGE_AVERAGING_SAMPLE_LIMIT)
matrixNoFieldVoltageSampleCount[currentBoardNr][activeRow][nextCol]++;
if (matrixNoFieldVoltageSampleCount[currentBoardNr][activeRow][nextCol] == 1)
matrixNoFieldVoltages[currentBoardNr][activeRow][nextCol] = tempVoltage;
matrixNoFieldVoltages[currentBoardNr][activeRow][nextCol] = (tempVoltage + matrixNoFieldVoltages[currentBoardNr][activeRow][nextCol] * matrixNoFieldVoltageSampleCount[currentBoardNr][activeRow][nextCol]) / (matrixNoFieldVoltageSampleCount[currentBoardNr][activeRow][nextCol] + 1L);
}
if (doSwitchRow)
setActiveMainMuxRow(currentBoardNr, nextRow);
}
}
if (doSwitchRow) {
// Serial.print(activeRow);
// Serial.print(" - ");
// Serial.println(_t_measureCount);
activeRow = nextRow;
lastMainMuxRowSwitchTime = millis();
_t_measureCount = 0;
}
} else {
// Serial.println(millis() - lastTickTime);
}
}
void onNoteOn(int boardIndex, int row, int col, int cubeType) {
noteOn(1, midiNotes[boardIndex][cubeType - 1][row], 0x7F);
matrixStates[boardIndex][row][col] = cubeType;
}
void silencePreviousColNotes(int boardIndex) {
for (int row = 0; row < ROW_COUNT; row++) {
if (matrixStates[boardIndex][row][prevCol] > 0) {
noteOff(1, midiNotes[boardIndex][matrixStates[boardIndex][row][prevCol] - 1][row]);
matrixStates[boardIndex][row][prevCol] = 0;
}
}
}
void processExternalMIDIClock() {
}
void processInternalClock() {
if (millis() - lastTickTime > tickDuration) {
// Serial.print(millis() - (lastTickTime + tickDuration));
// Serial.print(" - ");
onTick();
lastTickTime = lastTickTime + tickDuration;
// Serial.println(millis() - lastTickTime);
}
}
/******************************************************************************
INITIALIZATION FUNCTIONS
******************************************************************************/
void initMatrixStates(int boardIndex) {
for (int currentCol = 0; currentCol < COL_COUNT; currentCol++) {
for (int currentRow = 0; currentRow < ROW_COUNT; currentRow++) {
matrixStates[boardIndex][currentRow][currentCol] = 0;
}
}
}
void initBoards() {
for (int currentBoardNr = 0; currentBoardNr < BOARD_COUNT; currentBoardNr++) {
initSubMuxes(currentBoardNr);
initMainMuxes(currentBoardNr);
initMatrixStates(currentBoardNr);
ledStrips[currentBoardNr].begin();
}
}
void initMatrixVoltageArray() {
for (int currentBoardNr = 0; currentBoardNr < BOARD_COUNT; currentBoardNr++) {
for (int currentCol = 0; currentCol < COL_COUNT; currentCol++) {
for (int currentRow = 0; currentRow < ROW_COUNT; currentRow++) {
matrixAverageVoltageSampleCount[currentBoardNr][currentRow][currentCol] = 0;
matrixAverageVoltages[currentBoardNr][currentRow][currentCol] = matrixNoFieldVoltages[currentBoardNr][currentRow][currentCol];
matrixVoltageSpikeCount[currentBoardNr][currentRow][currentCol] = 0;
}
}
}
}
/******************************************************************************
LED STRIP FUNCTIONS
******************************************************************************/
void setPixelBatchColor(int boardIndex, int batchIndex, uint32_t color) {
for (int i = batchIndex * 3; i < (batchIndex + 1) * 3; i++) {
ledStrips[boardIndex].setPixelColor(i, color);
}
for (int i = 47 - batchIndex * 3; i < 47 - (batchIndex + 1) * 3; i++) {
ledStrips[boardIndex].setPixelColor(i, color);
}
}
void updateLEDStrips() {
for (int stripIndex = 0; stripIndex < BOARD_COUNT; stripIndex++) {
for (int i=0; i< LED_STRIP_PIXEL_COUNT; i++) {
ledStrips[stripIndex].setPixelColor(i, colorWheel[((i * colorWheelConstant) + ledStripIterator) & 255]);
}
setPixelBatchColor(stripIndex, activeCol, activeColor);
}
for (int stripIndex = 0; stripIndex < BOARD_COUNT; stripIndex++) {
ledStrips[stripIndex].show();
}
}
void processLEDStrips() {
if (millis() - lastLEDStripUpdateTime >= LED_STRIP_UPDATE_FREQUENCY) {
ledStripIterator = ledStripIterator + 1 % (255 * 5);
updateLEDStrips();
lastLEDStripUpdateTime = millis();
}
}
/******************************************************************************
FX STRIP FUNCTIONS
******************************************************************************/
void initFXStrip() {
for (int i=0; i<FX_STRIP_LED_COUNT; i++) {
fxStripLEDBrightness[i] = 0;
fxStripLEDColors[i][0] = 0;
fxStripLEDColors[i][1] = 0;
fxStripLEDColors[i][2] = 0;
fxStripLEDLastChangeTime[i] = 0;
}
strip.begin();
strip.show(); // Initialize all pixels to 'off'
}
uint8_t splitColor ( uint32_t c, char value ) {
switch ( value ) {
case 'r': return (uint8_t)(c >> 16);
case 'g': return (uint8_t)(c >> 8);
case 'b': return (uint8_t)(c >> 0);
default: return 0;
}
}
void lightUpFXStripSegment(int segmentIndex, byte wheelPosition) {
for(uint8_t j=FX_STRIP_LEDS_PER_SEGMENT * segmentIndex; j < FX_STRIP_LEDS_PER_SEGMENT * (segmentIndex + 1); j++) {
uint32_t newColor = Wheel((wheelPosition + (FX_STRIP_SEGMENT_LED_WHEEL_DIFFERENCE * j - FX_STRIP_LEDS_PER_SEGMENT * segmentIndex)) % 255);
fxStripLEDBrightness[j] = 255;
fxStripLEDColors[j][0] = splitColor(newColor, 'r');
fxStripLEDColors[j][1] = splitColor(newColor, 'g');
fxStripLEDColors[j][2] = splitColor(newColor, 'b');
fxStripLEDLastChangeTime[j] = millis();
// strip.setPixelColor(j, newColor);
strip.setPixelColor(j, fxStripLEDColors[j][0], fxStripLEDColors[j][1], fxStripLEDColors[j][2]);
}
}
void onTickFXStrip() {
int randomSegmentIndex = random(0, FX_STRIP_SEGMENT_COUNT);
if (randomSegmentIndex == fxStripLastSegment) {
randomSegmentIndex = (randomSegmentIndex + 1) % FX_STRIP_SEGMENT_COUNT;
}
byte randomWheelPosition = random(0, 255);
lightUpFXStripSegment(randomSegmentIndex, randomWheelPosition);
fxStripLastSegment = randomSegmentIndex;
}
void processFXStrip() {
for (int i=0; i<FX_STRIP_LED_COUNT; i++) {
if (fxStripLEDBrightness[i] > 0) {
if (1) {
fxStripLEDBrightness[i] -= FX_STRIP_LED_FADE_SPEED;
if (fxStripLEDBrightness[i] < 0)
fxStripLEDBrightness[i] = 0;
fxStripLEDLastChangeTime[i] = millis();
strip.setPixelColor(i, (int)((float)fxStripLEDColors[i][0] / 255 * fxStripLEDBrightness[i]), (int)((float)fxStripLEDColors[i][1] / 255 * fxStripLEDBrightness[i]), (int)((float)fxStripLEDColors[i][2] / 255 * fxStripLEDBrightness[i]));
}
}
}
}
/******************************************************************************
MUX CONTROL FUNCTIONS
******************************************************************************/
void initSubMuxes(int boardIndex) {
pinMode(SUB_MUX_CONTROL_PINS[boardIndex][LINE_A], OUTPUT);
pinMode(SUB_MUX_CONTROL_PINS[boardIndex][LINE_B], OUTPUT);
pinMode(SUB_MUX_CONTROL_PINS[boardIndex][LINE_C], OUTPUT);
setActiveSubMuxCol(boardIndex, 0);
}
void setSubMuxes(int boardIndex, bool lineA, bool lineB, bool lineC) {
digitalWrite(SUB_MUX_CONTROL_PINS[boardIndex][LINE_A], lineA ? HIGH : LOW);
digitalWrite(SUB_MUX_CONTROL_PINS[boardIndex][LINE_B], lineB ? HIGH : LOW);
digitalWrite(SUB_MUX_CONTROL_PINS[boardIndex][LINE_C], lineC ? HIGH : LOW);
}
void setActiveSubMuxCol(int boardIndex, int colNr) {
setSubMuxes(boardIndex, MUX_CONTROL_SIGNALS[colNr][LINE_A], MUX_CONTROL_SIGNALS[colNr][LINE_B], MUX_CONTROL_SIGNALS[colNr][LINE_C]);
}
void initMainMuxes(int boardIndex) {
pinMode(MAIN_MUX_CONTROL_PINS[boardIndex][LINE_A], OUTPUT);
pinMode(MAIN_MUX_CONTROL_PINS[boardIndex][LINE_B], OUTPUT);
pinMode(MAIN_MUX_CONTROL_PINS[boardIndex][LINE_C], OUTPUT);
setActiveMainMuxRow(boardIndex, 0);
}
void setMainMuxes(int boardIndex, bool lineA, bool lineB, bool lineC) {
digitalWrite(MAIN_MUX_CONTROL_PINS[boardIndex][LINE_A], lineA ? HIGH : LOW);
digitalWrite(MAIN_MUX_CONTROL_PINS[boardIndex][LINE_B], lineB ? HIGH : LOW);
digitalWrite(MAIN_MUX_CONTROL_PINS[boardIndex][LINE_C], lineC ? HIGH : LOW);
}
void setActiveMainMuxRow(int boardIndex, int rowNr) {
setMainMuxes(boardIndex, MUX_CONTROL_SIGNALS[rowNr][LINE_A], MUX_CONTROL_SIGNALS[rowNr][LINE_B], MUX_CONTROL_SIGNALS[rowNr][LINE_C]);
}
/******************************************************************************
HELPER FUNCTIONS
******************************************************************************/
void onCycleEnded() {
if (doStartCalibrationModeInNextCycle) {
doStartCalibrationModeInNextCycle = false;
inCalibrationMode = true;
currentCalibrationModeCycle = 0;
}
if (inCalibrationMode) {
if (currentCalibrationModeCycle < CALIBRATION_DURATION_IN_CYCLES) {
currentCalibrationModeCycle++;
} else {
inCalibrationMode = false;
digitalWrite(13, LOW);
}
}
}
void incrementColIndexes() {
prevCol = activeCol;
activeCol = (activeCol + 1) % COL_COUNT;
nextCol = (activeCol + 1) % COL_COUNT;
if (activeCol == 0)
onCycleEnded();
}
int getPreviousCol() {
int previousCol = activeCol - 1;
if (previousCol < 0)
previousCol = COL_COUNT - 1;
return previousCol;
}
long convertToGauss (int voltageOffset) {
return voltageOffset * TOMILLIGAUSS / 1000 / 100;
}
uint32_t Wheel(byte WheelPos) {
WheelPos = 255 - WheelPos;
if (WheelPos < 85) {
return ledStrips[0].Color(255 - WheelPos * 3, 0, WheelPos * 3);
}
if (WheelPos < 170) {
WheelPos -= 85;
return ledStrips[0].Color(0, WheelPos * 3, 255 - WheelPos * 3);
}
WheelPos -= 170;
return ledStrips[0].Color(WheelPos * 3, 255 - WheelPos * 3, 0);
}
/******************************************************************************
MIDI FUNCTIONS
******************************************************************************/
// Send a MIDI note on message
void noteOn(byte channel, byte pitch, byte velocity) {
// 0x90 is the first of 16 note on channels. Subtract one to go from MIDI's 1-16 channels to 0-15
channel += 0x90 - 1;
// Ensure we're between channels 1 and 16 for a note on message
if (channel >= 0x90 && channel <= 0x9F) {
Serial1.write(channel);
Serial1.write(pitch);
Serial1.write(velocity);
}
}
// Send a MIDI note off message
void noteOff(byte channel, byte pitch) {
// 0x80 is the first of 16 note off channels. Subtract one to go from MIDI's 1-16 channels to 0-15
channel += 0x80 - 1;
// Ensure we're between channels 1 and 16 for a note off message
if (channel >= 0x80 && channel <= 0x8F) {
Serial1.write(channel);
Serial1.write(pitch);
Serial1.write((byte)0x00);
}
}
// Send a MIDI control change message
void controlChange(byte channel, byte control, byte value) {
// 0xB0 is the first of 16 control change channels. Subtract one to go from MIDI's 1-16 channels to 0-15
channel += 0xB0 - 1;
// Ensure we're between channels 1 and 16 for a CC message
if (channel >= 0xB0 && channel <= 0xBF) {
Serial1.write(channel);
Serial1.write(control);
Serial1.write(value);
}
}
/******************************************************************************
SERVICE FUNCTIONS
******************************************************************************/
void processCalibrationButton() {
if (digitalRead(CALIBRATION_BUTTON_PIN) == LOW && !inCalibrationMode) {
doStartCalibrationModeInNextCycle = true;
}
}
void performCalibration() {
if (DEBUG_MODE)
Serial.println("Performing calibration. Make sure no cubes are placed on any of the boards.");
long startTime = millis();
long sampleCounts[BOARD_COUNT][ROW_COUNT][COL_COUNT];
long iterationCounter = 1;
do {
for (int currentBoardNr = 0; currentBoardNr < BOARD_COUNT; currentBoardNr++) {
for (int currentCol = 0; currentCol < COL_COUNT; currentCol++) {
setActiveSubMuxCol(currentBoardNr, currentCol);
for (int currentRow = 0; currentRow < ROW_COUNT; currentRow++) {
setActiveMainMuxRow(currentBoardNr, currentRow);
long tempVoltage = (long)analogRead(BOARD_INPUT_PINS[currentBoardNr]) * 100L;
if (iterationCounter == 1) {
sampleCounts[currentBoardNr][currentRow][currentCol] = 0;
matrixNoFieldVoltages[currentBoardNr][currentRow][currentCol] = tempVoltage;
} else {
if (sampleCounts[currentBoardNr][currentRow][currentCol] < VOLTAGE_AVERAGING_SAMPLE_LIMIT)
sampleCounts[currentBoardNr][currentRow][currentCol]++;
matrixNoFieldVoltages[currentBoardNr][currentRow][currentCol] = (tempVoltage + matrixNoFieldVoltages[currentBoardNr][currentRow][currentCol] * sampleCounts[currentBoardNr][currentRow][currentCol]) / (sampleCounts[currentBoardNr][currentRow][currentCol] + 1L);
}
}
}
}
if (DEBUG_MODE) {
Serial.print(".");
if (iterationCounter % 40 == 0)
Serial.println();
}
iterationCounter++;
} while (millis() - startTime < CALIBRATION_DURATION);
if (DEBUG_MODE) {
Serial.println();
Serial.println("Calibration complete.");
}
}
void printNoFieldVoltageArray() {
Serial.println();
Serial.println();
for (int currentBoardNr = 0; currentBoardNr < BOARD_COUNT; currentBoardNr++) {
Serial.print("\n // Board ");
Serial.println(currentBoardNr);
Serial.println(" {");
for (int currentRow = 0; currentRow < ROW_COUNT; currentRow++) {
Serial.print(" {");
for (int currentCol = 0; currentCol < COL_COUNT; currentCol++) {
Serial.print(matrixNoFieldVoltages[currentBoardNr][currentRow][currentCol]);
if (currentCol < COL_COUNT - 1)
Serial.print(", ");
}
Serial.print("}");
Serial.println(currentRow == ROW_COUNT - 1 ? "" : ",");
}
Serial.print(" }");
Serial.println(currentBoardNr == BOARD_COUNT - 1 ? "" : ",");
}
}
/******************************************************************************
DEBUG FUNCTIONS
******************************************************************************/
void _debugPrintCRV(int col, int row, long theValue) {
Serial.print(col);
Serial.print(",");
Serial.print(row);
Serial.print(": ");
// if (cubeType > 0)
// Serial.print(cubeType);
// else
// Serial.print(" ");
Serial.print(theValue == 0 ? " " : (theValue > 0 ? "+" : ""));
Serial.print(theValue);
Serial.print(abs(theValue) < 10 ? " " : "");
Serial.print(abs(theValue) < 100 ? " " : "");
Serial.print(abs(theValue) < 1000 ? " " : "");
Serial.print(abs(theValue) < 10000 ? " " : "");
Serial.print(" ");
}
Software: PureData
The first prototype needed a decent laptop to run as playback of the sounds was handled by Ableton software. It was fastest to implement and try the boards out this way, but the ultimate goal was to run it fully automatically and without a need for any paid software. After some research I settled on using PureData , a Max/MSP based audio generator and processor that allows you to build complex audio processing pipelines via UI, that can then be run headless on any Linux based device including RaspberryPi. It’s a great and extremely flexible tool, but the way it works is one hell of a mess to understand. It is based around the concept of Bang, which is kind of a signal that can trigger some action. The problem is this Bang can behave in several different ways depending on situation, and this inconsistency makes it really hard to understand. In the end I achieved what I needed with it, but whenever I had a choice whether to do something with PureData or move it to Arduino code, I always chose the latter.
Most complex parts were actually not audio related at all. I needed a way to easily switch through multiple sound banks and that proved to be a nightmare to do with Pure Data, the below portion is doing just that: loading files from different folders based on what number of bank is selected. In any programming languages this would be just a few lines of code.
Hardware
In parallel with prototyping the electronics we were working on the cases for the boards that would house all the components. It started from detailed 3D model that we were able to feed into the huge CDC cutter machine layer by layer.
There was tons of work to make it work, we ruined quite some materials, had to adjust and re-adjust CDC cutter settings, play with temperatures of laser cutter, test multiple painting options, but I’ll spare you the details and leave you with two animated images implying that the process was easy and smooth.
After this step we had all the individual parts of the device ready to be assembled into one single entity.
Putting it all together
Our designs proved to be good and assembling everything into one working entity didn’t bring any unexpected problems, but was extremely time intensive. We had to solder tons of wires, tune all the sensors, test out connectors and of course use lots of duct tape, because all decent technology is held together by the duct tape.
Above you can see the photos taken during the process of assembling the boards. Each board is fully detachable and can be connected to the “brain” of the device via Ethernet cable. The cable or the protocol doesn’t have anything in common with internet or network, we just chose Ethernet cable, because it packs 8 wires in a compact connector and is easy to work with.
Prototype of the “brain” was made by using power supply from PC that was enough to power Arduino and LEDs in all the boards. The only thing missing from the picture is a sound card with 8 outputs (1 x stereo for mixed output and 3 x stereo for sound s from each board separately).
Logo
Finally when all was done I wanted to make a nice logo for the device so we can have it on social media and make some stickers with it. I ended up making it in half an hour and was really happy with the result.
We made some stickers with this logo and left them lying around the device during the events. Interestingly enough a lot of people thought that these are the instructions on how to “correctly” place the cubes on the board and started arranging cubes into the “S” shape just to hear the arhythmical weird mess of sounds followed by empty pauses.
Funding
Given the scope of the project we really wanted to get some funding, but due to non-traditional and artsy nature of the product we needed to be very creative with who to approach.
We tried applying for grants from Lithuanian Ministry of Culture and several other European Union institutions but they were not too impressed. If I remember correctly the response was more or less: we don’t see any cultural value in this project, and actually we don’t even understand what are you planning to do here at all.
Another attempt was to get funding for Social Sequencer from Burning Man, a legendary festival taking place in Nevada desert in USA. They are famous for all sorts of art installations and our device would be a good fit there. In theory. Boy I am happy that they ended up rejecting our project. The amount of money we were asking for was ridiculously low, timelines we provided were ten fold unrealistic, the amount of logistical problems we never thought about was huge. We did consult a guy who was a frequent Burning Man attendee and helped to set up various projects there and and after hearing our grand idea his advice was: “guys, have you considered to forget about this project and just go to the festival and have a good time”. This of course didn’t stop us from applying to Burning Man and still trying to pull it of, but looking back, he was definitely right. On the other hand us being absolutely terrible at estimating the amount of resources and time that will go into making the device was the only reason we actually ended up building it. Worth mentioning is that the same year one Lithuanian installation did reach Burning man festival, it was “Dance for the Dawn” by Karolis Misevičius and his team, you can see a photo of it below.
Next we tried different organizations that fund art projects. They are very niche and there aren’t a lot of them, but there are a few. Most we applied to were not interested, but we got some nice traction from an Awesome Foundation in San Francisco. Frankly it was mostly a single person from the foundation who really understood and absolutely loved our project. His name was Mitch Altman. He is a very interesting person himself who back in the day invented and manufactured the device called “TV be Gone”, a remote control allowing you to turn off any TV that has a remote control, including the ones in public spaces like bars, etc. Also was an early developer of virtual reality technologies, co-founder of one of the first hackerspaces in US called Noisebridge and a prominent figure in the whole maker movement.
Unfortunately Awesome Foundation had a rule that they are not funding projects outside San Francisco area and Mitch wasn’t able to convince others to break this rule in the end. But he pointed us to The Pollination Project a fund that is giving out grants for “driving transformational change in communities, cultures and countries”. It was a bit weird, because we didn’t see our device as designed to help diminishing poverty or providing equal opportunities for minorities in any shape or form. But they saw something in our project and we got the grant. In the end they wrote the whole article about how Social Sequencer is helping prison population to re-integrate into society. The article made us cringe a bit, but we accepted the honors nevertheless.
Version 2.0
By the time the first version became functional LinkMenų Fabrikas and it’s director Adas became very interested in the project. Together we started looking for more ways to get funding and expanding on the project. One thing we ended up pulling off was getting a grant from MITA, an agency for supporting science, innovation and technology. They are mostly funding projects where scientists and entrepreneurs collaborate to create innovative products. Again, we had to be creative how we present the whole project. But that’s where the guys from LinkMenų Fabrikas shined. They wrote a 60 page document describing the project in in very formal and bureaucratic language. Fortunately this time the effort was on LinkMenų Fabrikas side. I have to admit when they sent me the final draft and I read the abstract, I couldn’t understand that it is actually about our Social Sequencer. But they knew exactly what they are doing because we did get the grant.
Above you can see the first page from that application and translated to english is says: Scientific research paper “Technical feasibility study of the application of new advanced technologies to develop non-traditional thinking”. I was really glad I was not the one who had to come up with this name and I was also very happy that with this grant we were able to build second version that was more polished, had various improvements made by LinkMenų engineers, much nicer 3D printed cubes and was all white!
They are still bringing trimmed down version of Social Sequencer to various exhibitions to present Vilnius Tech University and to show what cool projects you could be working on if you choose to join Vilnius Tech. That’s really nice, but I still want to find new spots where it would be possible to demonstrate the whole installation and allow people to create an ever evolving piece of music by randomly collaborating together, if you are interested or know somebody who is, please let me know.