As part of the coursework assessments for the iOS Programming and Personal Professional Practitioner modules for my MSc in Audio and Music Technology Course at the University of York, I was tasked to develop an audio or music application and develop a business model for it. This resulted in the creation of Coolintang, which is a musical instrument app meant to help reduce stress and anxiety. Essentially, the app is designed to turn iPads into a touch-enabled Kulintang. A Kulintang is an indigenous musical instrument widely known in the Philippines to be associated with the island of Mindanao, particularly in the provinces of Maguindanao and Lanao. The Kulintang is made up of 5-9 gongs arranged in ascending order of pitch. More details about Coolintang and its purpose can be read in the technical report below. A PDF file can also be downloaded here.
The video below explains what the app is and provides a demonstration of how the app can be used:
A copy of the full business model file that shows how the app can be developed further into a commercially viable application can be downloaded here.
The app was created using Xcode and Swift. We were taught the basics of it in our classes. With the help of my supervisor as well as online tutorials, I was able to build the app despite being a beginner at coding. Swift templates were provided in our labs and these were tweaked by myself to suit the intended outcome for this project. Though there are some minor bugs or issues at the end, the creation of the app can still be considered successful, since it is working and is able to display the core concept.
In the future, I hope to revisit this app and develop it further so that anyone can download it on the app store. Especially those who would like to ease their stress and anxiety with soothing sounds and music, accompanied by beautiful backgrounds.
In this TechTalk, we will look at an example of a technological solution for music therapy or special needs music and relate it to the design of the app for my final MSc project.
Transcript:
If you’re interested in audio, music, and technology, then you’ve come to the right place.
Hi! I’m Francesca, and you’re watching my TechTalk.
In the last TechTalk, I talked about inclusive immersive technology. In this video, we’ll discuss app design considerations, available AR development software, and the MR hardware I plan to use. Let’s get started!
In 1987, one of my supervisors worked on a project called MidiGrid. It is an interactive computer-based musical instrument developed primarily for people with physical disabilities. The concept, as mentioned in a paper he co-wrote, is that “MidiGrid allows users to trigger musical material freely in real-time using the computer’s mouse… the screen shows a grid of boxes, each of which contains a nugget of music that is triggered as the mouse cursor moves over it. Hand gestures are thus converted, via the mouse into notes, chords and musical sequences.” To simply put, MidiGrid allows users to play the stored music by moving a mouse cursor over a box. In addition to that, since it is created primarily for people with physical disabilities, the mouse cursor movement range is designed to be customizable to cater to the user. With features like this, MidiGrid is quite useful for music therapy improvisation sessions. This is just scratching the surface of what MidiGrid is, but with this information, it made me wonder about how I should approach my final project. Since I aim to develop an app that’s geared towards people with limited movement, the design of the interface should be well thought out.
Currently, these are the design considerations that I have:
For the app’s interface:
1. What should be the layout of the menu?
2. How big should the buttons be?
For the hardware:
1. What are the limitations of using a certain MR hardware?
2. What types of mobilities should I take into account? Will the addition of hand-held controllers make it difficult for people with limited movement to use the app?
3.Is it a must to add haptic feedback or vibrations?
After I sort out how the app should be designed, the next stage would be the app development process. There are several software options for creating an AR iOS app. At the moment, I am looking at Apple’s ARKit and Unity’s AR Foundation plus XR Interaction Toolkit. ARKit is a framework for Swift that that allows developers to create AR experiences for apple devices. Similarly, Unity’s AR Foundation is the same, then the XR toolkit is an easy way to add interactivity to virtual elements. Initially, I would like to use ARKit because it’s accessible and I think it’s more user-friendly for someone like me who is still learning the ropes of coding. Then again, I feel like Unity would be user-friendly too, especially with the addition of the XR Interaction Toolkit. I’m still undecided at this point, so I really need to do a more in-depth comparison between the two development tools. Maybe in a future video, I’ll try to do a comparison. What do you think?
Anyway, as for the MR hardware, I’ve been looking at ZapBox. It’s an affordable mixed reality kit for designing and developing mixed reality experiences. It comes with a cardboard headset, pointcodes that acts us room anchors to help workout the position of surfaces in a room, trackable cardboard controllers for interacting with virtual elements, a lens adapter for smartphone cameras, and an app for building a map of codes and browsing other content. Moreover, they provide documentation on how to integrate the codes to both Swift and Unity, which I think is very convenient for my project. I’m looking for other options too as I’m still in the research and planning stage, but ZapBox is definitely the top option for now.
There’s still a long way to go before my app comes to fruition, but I can see the exciting journey ahead. I’ll try to post more TechTalk videos about the project, especially when it’s in the app development process already. Stay tuned for that!
Once again, I’m Francesca, and thank you for listening to my TechTalk.
In this TechTalk, we will look at examples of inclusive immersive technology.
Transcript:
If you’re interested in audio, music, and technology, then you’ve come to the right place.
Hi! I’m Francesca, and you’re watching my TechTalk.
In the last TechTalk, I talked about XR. In this video, we’ll look at how immersive technology can be used in special needs music. Let’s get started!
Immersive technology is becoming more and more popular nowadays thanks to the gaming and entertainment industry. We can see a lot of immersive reality headsets like Oculus Rift, Google Cardboard, HTC Vive, and many more are now available on the market. Although these are mostly used for gaming and media consumption, more industries are trying to take advantage of this technology too. In the healthcare sector, this technology is used in different ways. For the purpose of this video, we will talk about its context in special needs education and music.
After a brief search on the internet, it seems that there are quite a number of projects utilizing immersive technology for use in special needs already, whether it be for educational or therapeutic purposes. In the United States, their Department of Education invested $2.5 Million in a project called “VOISS: Virtual Reality Opportunities to Implement Social Skills.” Through the project they aim to give people with high-functioning autism and disability, a safe virtual space where they can learn social skills and other behaviors at their own pace. The project showed promising results and this further proves that the possibilities for how we can make the lives of people with autism and disabilities easier using this technology are endless.
Another example of inclusive immersive technology is a project called “Performance Without Barriers.” This project aims to design virtual reality instruments for musicians with disabilities. The research team partnered with a software development company to develop “Infinite Instrument” for the HTC Vive headset. Although the virtual instrument has hand-held controllers, it is designed in such a way where different types of mobilities were taken into consideration. The controllers are a good addition because they make the whole experience more immersive; since musicians are able to engage not only their sense of sight and hearing, but also their sense of touch.
The Performance Without Barriers team mentioned that most immersive technologies are to marketed to able-bodied users, and I think this is true. With the ever-growing popularity of immersive technologies, it is important to create immersive experiences that are inclusive too.
Through the examples mentioned, the design of any immersive tool really needs to be well thought out, especially if it’s going to be used in special needs learning or music settings. Since I aim to develop a mixed reality musical instrument app for my final MSc project, I really need to research the best way to design my app’s interface so that it will be easy for people with limited movement to use it.
In my next TechTalk, I will discuss design considerations, available AR app development software, and the MR hardware I plan to use.
Once again, I’m Francesca, and thank you for listening to my TechTalk.
In this TechTalk, we will talk about the different types of extended realities– what they are and how they came to be.
Transcript:
If you’re interested in audio, music, and technology, then you’ve come to the right place.
Hi! I’m Francesca, and you’re watching my TechTalk.
In the last TechTalk, I gave a brief description of AR, MR and special needs music. In this video, we will be discussing virtual technology in more depth. Let’s get started!
When talking about augmented reality and mixed reality, it’s hard not to mention virtual reality too. These technologies as well as other types of immersive experiences fall under the term XR, otherwise known as Extended Reality. The X in XR is also viewed as a stand-in variable which means it can be replaced with other letters representing a specific technology like VR, AR, MR. Often though, the term XR is used to group VR, AR, and MR together. The earliest XR concept can be traced back to Sir Charles Wheatstone, a British philosopher and inventor who contributed in various scientific fields particularly in the area of sound and vision. In 1838, he invented the stereoscope, a device that plays around with our visual perception or how our brain sees things. Basically, viewing through a stereoscope produces a three-dimensional image. This illusion is created by presenting the same image in both the left and the right eye, but with different angles. The different perspectives are then interpreted by the brain as a three-dimensional image. In the 1840s, Scottish Physicist Sir David Brewster, improved the design of the stereoscope by making it more portable. By then, photographs were a thing already, so aside from drawings and artwork, people were now able to view three-dimensional images of the real world.
Through the years, people have built upon the idea of three-dimensional images and in 1935, a concrete concept of virtual reality first came to light. Stanley G. Weinbaum an American sci-fi writer, wrote the Pygmalion’s Spectacles. Long story short, it involves the creation of spectacles that excites all of the five senses to fully immerse a user in a movie or story so that they can be a part of it. Something similar to this was invented by Morton Heilig in 1957. The machine is called Sensorama and it offered users an immersive movie experience by stimulating their senses such as sight, hearing, and smell. This is the basis of the VR goggles we know today.
The invention of virtual reality gave birth to the development of many other realities. Since then, immersive technology has seen massive improvements. The year 1990 was when the term augmented reality appeared, and in 1992 the first operational mixed reality system was created at the USAF Armstrong Labs at Brooks Air Force Base in San Antonio, Texas. Louis Rosenburg created a system called Virtual Fixtures, which was made up of binocular magnifiers and robot arms. The device overlaid graphical information to a real-world environment and had simple haptic surface combinations. This device was used in military training to increase work efficiency especially for physical tasks. The robot arms were seen in the exact position as the user’s physical arms and the generated visuals were barriers, fields, guides, etc.
Although 1992 saw the first mixed reality system, it wasn’t until 1994 when the term was officially defined. Paul Milgram and Fumio Kishino, in their paper called “A Taxonomy of Mixed Reality Visual Displays,” tackled the virtuality continuum concept, which is sometimes referred to as the reality-virtuality continuum. According to the paper, MR is said to be “…anywhere between the extrema of the virtuality continuum.” In simple terms, MR is anywhere between the real world and the virtual world spectrum. So, it could either be more towards the side of augmented reality or augmented virtuality. Based on this definition, my MSc project is considered to be on the augmented reality side, as I hope to place virtual musical instruments in the real world.
So the question now is how can MR, or any XR technology for that matter, be used for developing technological solutions for music therapy or special needs music? The answer? Well, you have to watch my next TechTalk to find out.
Once again, I’m Francesca, and thank you for listening to my TechTalk.
In this vlog, which I will refer to as TechTalk from now on, I introduce my final project topic for my MSc in Audio and Music Technology. Spoiler alert! The topic is Mixed Reality Musical Instruments iOS App for Special Needs Music.
Transcript:
If you’re interested in audio, music, and technology, then you’ve come to the right place.
Hi! I’m Francesca, and you’re watching my TechTalk
I’m Francesca Lorico, a graduate student at the University of York pursuing my master of science degree in audio and music technology. This is the first of many TechTalks (maybe just 4?) documenting my final project for my program. In this TechTalk, I will give you an overview of my project topic, goals, research considerations, reason for choosing this topic, and my hopes for the project. Let’s get started!
The idea for my final project is to create a Mixed Reality Musical Instruments iOS App for Special Needs Music. That’s a bit of a mouthful, I know, so let’s break it down.
To better understand what mixed reality or MR is, we need to talk about augmented reality or AR first. According to Intel, “AR overlays digital information on real-world elements.” The perfect example of this technology can be seen in the very popular mobile game, Pokemon Go. When it was first released in 2016, a lot of people started swarming outside– to parks, beaches, churches, and many other places trying to locate, capture, and train pokemon using their mobile device’s GPS. But more than that, the app allowed them to see the pokemon in the world around them through their mobile phone’s camera. This made the whole gameplay more enjoyable for the players because they got to experience a unique combination of the real world and the game they were playing. So, in simple terms, AR lets us see virtual elements in the real world through a device. This is the foundation of Mixed Reality. But, in addition to being able to see virtual elements in the real world, MR also allows us to move and manipulate virtual elements in the real world using our own hands. Keep in mind, this is only possible if you have the gear and software to run it.
Now that you have a general idea of what MR is, you may already know what I mean by… Mixed Reality Musical Instruments iOS App. But to give you a better sense… Try to picture yourself wearing regular eyeglasses and holding sticks. You’ll be able to see your surroundings and the sticks, right? But imagine if these eyeglasses could let you see virtual things too. Look! the sticks are now wooden rods and there’s a virtual musical instrument in front of you!
Now let’s move on to the last part of the project topic. What does it mean when we add the words ‘for Special Needs Music’? The Merriam-Webster dictionary defines special needs as “any of various difficulties (such as a physical, emotional, behavioral, or learning disability or impairment) that causes an individual to require additional or specialized services or accommodations (such as in education or recreation).” There are numerous specialized services that enrich the lives of individuals with special needs, and one way is through music therapy. According to the British Association For Music Therapy or BAMT, “Everyone has the ability to respond to music, and Music Therapy uses this connection to facilitate positive changes in emotional wellbeing and communication through the engagement in live musical interaction between client and therapist.”
So, going back to my project, the mixed reality app I plan to develop is geared towards becoming a possible medium that can be used in special needs music therapy sessions, particularly for people with limited movement.
To sum it all up, the ultimate goal for my final MSc project is to develop an augmented reality musical instruments iOS app using an AR development software, paired or interfaced with a specific device for a mixed reality experience, that is especially beneficial for people with limited movement. With that goal in mind, the research will focus on the following:
1. What are the possibilities for developing an MR application for special needs music?
2. How is MR technology beneficial for special needs music?
3. What are the limitations of MR for special needs music?
Finally, the reason I chose to research this is because I truly believe in the power of music and how it can help and heal people. Through this project, I really hope to produce an app that would give people with special needs, specifically those with limited movement, the chance to experience playing a musical instrument in an effortless way using virtual means.
In my next videos, I will discuss the project in more detail. Specifically, I will talk more about MR, existing technological solutions for special needs music, and many more. So, stay tuned!
Once again, I’m Francesca, and thank you for listening to my TechTalk.
But first, let’s talk about vocal effects processors and loopers!
What is a Vocal Effects Processor?
As stated in this article written by Andy Jones, the earliest notable vocal effects processor can be attributed to Alvino Rey’s Talk Box, which was created in the 1930s. Thereafter, came the vocoder, then auto-tune, then harmony effects, and so on. Over the years, a lot of vocal effects processors have been created. These electronic devices, which often have multiple built-in effects, enhance vocals by adding color and depth as one sings in real-time. Thus, such devices are used by singers especially for live musical performances. Some of the most common effects that can be seen in processors nowadays are echo, harmony, reverb, and many more. A lot of well-known vocal effects processors also come with loopers.
What is a Looper?
A looper, also referred to as a looper pedal or loop pedal, is a device for real-time recording, layering of sounds, and playback of the recorded sounds. In this article by Emma Madden, the looper is said to have originated from Terry Riley. In the ’60s, Riley incorporated tape loops to the live performance of his third album ‘A Rainbow in Curved Air’. The result of the creative layered repetition in his performance sparked what is now known as the art of looping. But, what exactly are loopers like nowadays? Based on my experience, the looper I’ve used is able to record 4 bars and have more than one loop channel. Having more than one loop channel, lets a singer or musician store different loop phrases, and then play them back at different parts of a song. Recordings can also be overdubbed multiple times with just a push of a button, creating textured backing tracks for a more engaging and full-sounding live performance. It is very clear that looper devices have definitely come a long way since the 60s.
What are some of the most popular vocal effects processors and loopers?
What’s common with all the vocal effects processors and loopers available in the market?
Yes, you’ve guessed it! They are either a pedal or a tabletop vocal effects processor and looper.
Now, imagine a wearable vocal effects processor and looper that allows you to control a vocal effect, just by changing the position of your hand…
Introducing, MINI VOX!
With vocalists in mind, MINI VOX will be the perfect device for enhancing and manipulating one’s vocals during any part of their performance– whether it be adding more delay effects to just the last word of a musical phrase or cutting out the reverb abruptly for artistic effect. Whatever the intention is, this device will allow vocalists to have total control plus real-time enhancement and manipulation of vocals using gestural movement, at any point of a song.
Why aren’t their photos or videos of MINI VOX, you ask? Well, it’s because it is just in the prototype stage. It should also be noted that due to lack of time and resources, this will not be available in the near future. Nonetheless, this is the big idea:
A pair of electronic devices that communicate with one another. The board on the left side is the main controller, which will have inputs for microphones, and outputs for connections to a computer and speaker systems. The image on the right side is a seamless wearable device that has four modifiable buttons. In its initial state, there will be 2 buttons for toggling between vocal effects and other settings, 1 button for starting and stopping a loop phrase recording, and 1 button for clearing the recorded loop phrase. Ideally, these buttons will be modifiable to suit the needs of the user. In any case, further research is needed in order to fully realize this idea.
So, what’s the purpose of MINI VOX and why was it created then?
MINI VOX was created for my Interfacing for Audio and Music module coursework assessment at the University of York. The task for the assignment is to research the history and current state of my chosen area, and decide what challenge I want to address. Then, to demonstrate the challenge being addressed, a system which interfaces electronic sensors to a microcontroller using common electronic components must be designed and built. This system should also be capable of producing sounds and/or visuals in PD, Max/MSP, or other external synthesis system.
As you may have noticed, the challenge I have chosen is the absence of wearable gestural controlled vocal effects processors and loopers in the market. I have done a bit of searching and have not seen a device like this yet. So, it makes me wonder why this hasn’t been done yet. Is there no need for this? Is it too difficult to mass produce? Or are there companies developing such a device already and we just don’t know about it yet? But, instead of dwelling on these questions, I’ve decided to put matters into my own hands by trying to design and build my own wearable vocal effects processor and looper for the coursework assessment. As someone who is new to electronics and programming, my goal for this is to see what the possibilities are for designing and building such a device. The result of this attempt is the creation of the MINI VOX patch and circuit.
So, what is the prototype like, you ask? See for yourself…
This video gives an overview of the patch and circuit.
This is the demonstration video of the prototype.
As you can see from the video, the prototype is definitely working. The buttons are able to trigger functions in the patch, and some effects can be controlled by gestural movements. Obviously, there’s still a lot of room for improvement. However, the prototype surely proves that the big idea is not impossible.
If YOU also want to take matters into your own hands by creating your own prototype, follow my how-to guide here.
This guide walks you through how to create the MINI VOX patch and circuit. There are five steps in total. All of the patches and codes used are provided in this guide. To download these, simply click on the links. By the way, if you haven’t read what this is all about and you haven’t seen the MINI VOX videos yet, then please check out the main page here.
All set? Let’s get started!
STEP 1: Create the vocal effects processor patch
MINI VOX Max 7 Patch
The first step is to create the MINI VOX patch. This was created in Max 7, but this can be created in PD too. The patch is designed in a simple yet detailed manner for ease of use and navigation. It has a color scheme that is cool and easy on the eyes for aesthetic purposes. Everything is labeled and useful notes were placed where appropriate. The patch is divided into 2 main sections– Audio Settings and Vocal Effects. The Audio Settings section is divided into two parts– Microphone and Main Out. The Microphone part contains a mic on/off button, a meter, and a dry gain slider. The mic on/off button is an ezadc~ interface object that converts the analog signal to a digital signal. Essentially, this is for both audio input and turning the audio on/off. Below this button is a meter~ interface object which shows the peak amplitude of the microphone’s audio signal. Then, on the right, there is a dry gain slider object for listening only to the dry signal or raw sound coming from the mic. Below the microphone settings is the Main Out settings. As seen in the image above, there is a main out on/off button on the left side and a live gain slider on the right side. The main out on/off button is an ezdac~ interface object that converts the digital signal to an analog signal. This is for both the audio output and turning the audio on/off. The live gain slider, on the other hand, is for controlling the overall gain.
The vocal effects section contains 3 vocal effects and a looper. The vocal effects are– Delay, Reverb, and Harmony. The Delay effect allows the user to play around with the delay time and the delay amount. The reverb effect allows the user to input the liveness amount and output level, as well as to adjust the overall reverb volume. For the harmony effect, the voices are fixed, but users can increase or decrease the volume of the voices by adjusting the volume slider. Below are screenshots of the subpatches containing the effects:
Delay:
Reverb:
Harmony:
The looper has an on/off toggle at the top. To start recording a loop, users hold down the space bar while recording a musical phrase. To stop recording the loop, users need to release the space bar. A CLEAR LOOP button is included for deleting the recorded loop. Below these are a pair of sliders. The first slider is for increasing the volume of the recorded loop. The second slider is for hearing the click track, if the user would like to use one. At the moment, there is only one bpm. Future versions of this patch would most likely see a modifiable click track. Below is a screenshot of the looper subpatch:
Please note that the patches or codes used for the delay, reverb, harmony, and looper are only slightly modified by me. The original creators of the codes used are as follows:
Delay: Multi-tapout~ delay code by Jacob Sachs-Mishalanie as seen here
Reverb: Audio Processing Reverb code by David Tinapple and Sharon McCaman as seen here
Harmony: Pitch Shifting code by Sam Tarakajian (aka dude837) as seen here
Looper: Boomerang! code by Sam Tarakajian (aka dude837) as seen here
In its simplest form, the diagram below shows how the main patch works:
MINI VOX Patch Connection Diagram
Looking at the diagram above, it shows that the microphone signal goes into the delay, reverb, harmony, looper, meter, and dry gain. The dry gain goes into the live gain, then into the main out. For the effects, all three are linked to one another to cover all possible effects combinations that users would like to use. Before going into the live gain and then to the main out, all three effects are passed through the looper.
The built-in A and B buttons on the micro:bit are connected to pins 5 and 11, respectively. The additional button C, is connected to pin 1. All three buttons are also connected to GND for grounding them. For the MIDI, pin 0 is connected to the MIDI connector pin 5 with a 10 ohm resistor in between. Additionally, pin 3V is connected to the MIDI connector pin 4 with with a 330 ohm resistor in between. Lastly, GND is connected to MIDI connector pin 2 for grounding it.
Now that you have an idea, we’re now ready to assemble the components!
Step 3.1: Detach the edge connector breakout board from the prototyping kit by using the screwdriver to unscrew the screws. This is necessary so that later on, only the edge connector breakout board and the micro:bit are attached to the hand.
3.1 (a)
3.1 (b)
Step 3.2: Place the protective case on the micro:bit, then attach the micro:bit to the edge connector breakout board.
3.2 (a)
3.2 (b)
3.2 (c)
Step 3.3: Mount the MIDI PCB mount DIN on the breadboard. Keep in mind that pins should be on separate rows.
3.3 (a)
3.3 (b)
Step 3.4: Take 2 female-to-male breadboard leads, connect them to one another. Connect the female end to pin 0 of the breakout board and connect the male end to the row aligned with MIDI connector pin 5. After, attach the 10 ohm resistor in between pin 0 and pin 5 on the same row.
3.4 (a)
3.4 (b)
3.4 (c)
3.4 (d)
Step 3.5: Take 2 female-to-male breadboard leads, connect them to one another. Connect the female end to pin 3V of the breakout board and connect the male end to the row aligned with MIDI connector pin 4. After, attach the 330 ohm resistor in between pin 3V and pin 4 on the same row.
3.5 (a)
3.5 (b)
3.5 (c)
Step 3.6: Take 2 female-to-male breadboard leads, connect them to one another. Connect the female end to pin 0V of the breakout board and connect the male end to the row aligned with MIDI connector pin 2 to ground the circuit. The MIDI OUT circuit, should be working now.
3.6 (a)
3.6 (b)
3.6 (c)
Step 3.7: Mount the PCB push button in the middle of the breadboard, on the first few rows. Take 2 female-to-male breadboard leads, connect them to one another. Connect the female end to pin 1 of the breakout board and connect the male end to the row aligned with the top terminal of the button. Take another pair of female-to-male breadboard leads, connect them to one another. Connect the female end to pin 0V of the breakout board and connect the male end to the row aligned with the bottom terminal of the button.
3.7 (a)
3.7 (b)
3.7 (c)
3.7 (d)
3.7 (e)
Step 3.8: Take the velcro then cut out a strip that fits around your hand. Secure the velcro strip to the back of the micro:bit and breakout board using some tape.
3.8 (a)
3.8 (b)
3.8 (e)
Step 3.9: Finally, attach one end of the USB cable to the micro:bit, and the other end to your computer. Then, attach the MIDI IN of the MIDI-to-USB cable to your circuit, and the USB to your computer.
3.9 (a)
3.9 (b)
STEP 4: Programming the prototype
After assembling the circuit, the next step is programming the micro:bit. This can be done in mu-editor or in BBC micro:bit’s online Python Editor. The code used can be seen below, but you can also download the .hex and .py files here.
from microbit import *
import math
# Codes from the I-AM Module Labs of the University of York
# Slightly modified by Francesca Lorico
# MIDI ON triggered by micro:bit buttons
def midiNoteOn(chan, n, vel):
MIDI_NOTE_ON = 0x90
if chan > 15:
return
if n > 127:
return
if vel > 127:
return
msg = bytes([MIDI_NOTE_ON | chan, n, vel])
uart.write(msg)
# MIDI OFF triggered by micro:bit buttons
def midiNoteOff(chan, n, vel):
MIDI_NOTE_OFF = 0x80
if chan > 15:
return
if n > 127:
return
if vel > 127:
return
msg = bytes([MIDI_NOTE_OFF | chan, n, vel])
uart.write(msg)
# MIDI CC triggered by micro:bit accelerometer
def midiControlChange(chan, n, value):
MIDI_CC = 0xB0
if chan > 15:
return
if n > 127:
return
if value > 127:
return
msg = bytes([MIDI_CC | chan, n, value])
uart.write(msg)
# Initialization
def Start():
uart.init(baudrate=31250, bits=8, parity=None, stop=1, tx=pin0)
# Main Program
Start()
# Most recent values of buttons and Accelerometer
lastA = False
lastB = False
lastC = False
BUTTON_A_NOTE = 36
BUTTON_B_NOTE = 37
BUTTON_C_NOTE = 38
last_tilt_x = 0
last_tilt_y = 0
# While loop of the program
while True:
# buttons
a = button_a.was_pressed()
b = button_b.was_pressed()
c = pin1.is_touched()
if a is True and lastA is False:
midiNoteOn(0, BUTTON_A_NOTE, 0)
elif a is False and lastA is True:
midiNoteOff(0, BUTTON_A_NOTE, 0)
if b is True and lastB is False:
midiNoteOn(1, BUTTON_B_NOTE, 0)
elif b is False and lastB is True:
midiNoteOff(1, BUTTON_B_NOTE, 0)
if c is True and lastC is False:
midiNoteOn(2, BUTTON_C_NOTE, 0)
elif c is False and lastC is True:
midiNoteOff(2, BUTTON_C_NOTE, 0)
# Current X Y of accelerometer
current_tilt_x = accelerometer.get_x()
current_tilt_y = accelerometer.get_y()
# X Y of accelerometer to MIDI (0 - 127)
if current_tilt_x != last_tilt_x:
mod_x = math.floor(math.fabs((((current_tilt_x + 1024) / 2048) * 127)))
midiControlChange(0, 20, mod_x)
last_tilt_x = current_tilt_x
sleep(10)
if current_tilt_y != last_tilt_y:
mod_y = math.floor(math.fabs((((current_tilt_y + 1024) / 2048) * 127)))
midiControlChange(0, 21, mod_y)
last_tilt_y = current_tilt_y
sleep(10)
Before programming the micro:bit, it is first important to think about how you want the circuit to control the patch. For this prototype, it has been decided that the buttons on the micro:bit, as well as the additional button on the breadboard will trigger looper controls of the patch. The A button triggers the on/off button of the looper, the B button triggers the start and stop loop recording function, and the additional button triggers the clear loop function. Each button is labeled as a, b, and c respectively in the code above. On the other hand, the X and Y accelerometer coordinates of the micro:bit is dedicated to changing the values of the delay amount and reverb output level of the patch.
In order for the micro:bit to trigger controls on the patch, communication between the two must be established. This is where MIDI ON, OFF, and CC come into play. MIDI ON and OFF are needed for letting the patch know when a button is pressed or released, whereas MIDI CC is used for sending control change events. After these are defined in the program, an initialization statement needs to be added. The next step after this is to code the main program. In this part, it is necessary to define the most recent values of the variables first, then write a while loop after. The loop contains newly defined variables for the buttons that need to be set to have specific functions. For instance, button a was assigned the function was_pressed to indicate that the button was indeed pressed. More information on button functions can be found here. After the variables, an if-elif statement for the buttons follows. Below this is an if statement for the accelerometer.
After flashing or saving the code to the micro:bit, the next step is to add objects to the patch which will receive the MIDI messages sent out by the micro:bit. Below are images indicating the objects added in the patch:
A Button
B Button
C Button
X Accelerometer
Y Accelerometer
To receive the button messages, a notein object needs to be added to the patch. The argument that follows notein is the channel where the message is being received. 1 is for the A button, 2 is for the B button, 3 is for the C button. The messages are then sent to the intended functions in the patch. For the accelerometer message, a ctlin object needs to be added to the patch. The arguments following ctlin are the controller number and channel where the data from X and Y coordinates are received. After this object, a scale object is added in order to scale the values from the accelerometer so as to fit the range of the sliders. Then, these are sent to the intended functions in the patch as well.
STEP 5: Enjoy the prototype!
The final step in this how-to guide is to run the patch, test out the prototype, and just have fun with it! Once you’re more familiar with how it works, why not try adding more effects or functions?
That’s all to it. Have fun!
Thoughts on possible future work:
Programming the same buttons to have multiple functions. For example, instead of having a separate button for clearing the recording, it would be more efficient if pressing A+B could do this. Doing so would allow other functions to be programmed to the additional push button.
Programming dedicated buttons or a potentiometer for switching between effects.
Adding potentiometers for controlling the volume sliders of effects.
Adding more effects, multiple loop channels, and a modifiable click track.
Francesca Lorico 