Tag Archives: Special Needs Music

TechTalk 4: The Master(s) Plan! Get it?

In this TechTalk, we will look at an example of a technological solution for music therapy or special needs music and relate it to the design of the app for my final MSc project.

Transcript:

If you’re interested in audio, music, and technology, then you’ve come to the right place.
Hi! I’m Francesca, and you’re watching my TechTalk.
In the last TechTalk, I talked about inclusive immersive technology. In this video, we’ll discuss app design considerations, available AR development software, and the MR hardware I plan to use. Let’s get started!

In 1987, one of my supervisors worked on a project called MidiGrid. It is an interactive computer-based musical instrument developed primarily for people with physical disabilities. The concept, as mentioned in a paper he co-wrote, is that “MidiGrid allows users to trigger musical material freely in real-time using the computer’s mouse… the screen shows a grid of boxes, each of which contains a nugget of music that is triggered as the mouse cursor moves over it. Hand gestures are thus converted, via the mouse into notes, chords and musical sequences.” To simply put, MidiGrid allows users to play the stored music by moving a mouse cursor over a box. In addition to that, since it is created primarily for people with physical disabilities, the mouse cursor movement range is designed to be customizable to cater to the user. With features like this, MidiGrid is quite useful for music therapy improvisation sessions. This is just scratching the surface of what MidiGrid is, but with this information, it made me wonder about how I should approach my final project. Since I aim to develop an app that’s geared towards people with limited movement, the design of the interface should be well thought out. 
Currently, these are the design considerations that I have:

For the app’s interface:
1. What should be the layout of the menu?
2. How big should the buttons be?

For the hardware:
1. What are the limitations of using a certain MR hardware?
2. What types of mobilities should I take into account? Will the addition of hand-held controllers make it difficult for people with limited movement to use the app?
3.Is it a must to add haptic feedback or vibrations?

After I sort out how the app should be designed, the next stage would be the app development process. There are several software options for creating an AR iOS app. At the moment, I am looking at Apple’s ARKit and Unity’s AR Foundation plus XR Interaction Toolkit. ARKit is a framework for Swift that that allows developers to create AR experiences for apple devices. Similarly, Unity’s AR Foundation is the same, then the XR toolkit is an easy way to add interactivity to virtual elements. Initially, I would like to use ARKit because it’s accessible and I think it’s more user-friendly for someone like me who is still learning the ropes of coding. Then again, I feel like Unity would be user-friendly too, especially with the addition of the XR Interaction Toolkit. I’m still undecided at this point, so I really need to do a more in-depth comparison between the two development tools. Maybe in a future video, I’ll try to do a comparison. What do you think?

Anyway, as for the MR hardware, I’ve been looking at ZapBox. It’s an affordable mixed reality kit for designing and developing mixed reality experiences. It comes with a cardboard headset, pointcodes that acts us room anchors to help workout the position of surfaces in a room, trackable cardboard controllers for interacting with virtual elements, a lens adapter for smartphone cameras, and an app for building a map of codes and browsing other content. Moreover, they provide documentation on how to integrate the codes to both Swift and Unity, which I think is very convenient for my project. I’m looking for other options too as I’m still in the research and planning stage, but ZapBox is definitely the top option for now.
There’s still a long way to go before my app comes to fruition, but I can see the exciting journey ahead. I’ll try to post more TechTalk videos about the project, especially when it’s in the app development process already. Stay tuned for that! 


Once again, I’m Francesca, and thank you for listening to my TechTalk.

References:

Want to connect or collaborate? Send me an email here.

TechTalk 3: Inclusive Immersive Technology

In this TechTalk, we will look at examples of inclusive immersive technology.

Transcript:

If you’re interested in audio, music, and technology, then you’ve come to the right place.

Hi! I’m Francesca, and you’re watching my TechTalk.

In the last TechTalk, I talked about XR. In this video, we’ll look at how immersive technology can be used in special needs music. Let’s get started!

Immersive technology is becoming more and more popular nowadays thanks to the gaming and entertainment industry. We can see a lot of immersive reality headsets like Oculus Rift, Google Cardboard, HTC Vive, and many more are now available on the market. Although these are mostly used for gaming and media consumption, more industries are trying to take advantage of this technology too. In the healthcare sector, this technology is used in different ways. For the purpose of this video, we will talk about its context in special needs education and music.

After a brief search on the internet, it seems that there are quite a number of projects utilizing immersive technology for use in special needs already, whether it be for educational or therapeutic purposes. In the United States, their Department of Education invested $2.5 Million in a project called “VOISS: Virtual Reality Opportunities to Implement Social Skills.” Through the project they aim to give people with high-functioning autism and disability, a safe virtual space where they can learn social skills and other behaviors at their own pace. The project showed promising results and this further proves that the possibilities for how we can make the lives of people with autism and disabilities easier using this technology are endless.

Another example of inclusive immersive technology is a project called “Performance Without Barriers.” This project aims to design virtual reality instruments for musicians with disabilities. The research team partnered with a software development company to develop “Infinite Instrument” for the HTC Vive headset. Although the virtual instrument has hand-held controllers, it is designed in such a way where different types of mobilities were taken into consideration. The controllers are a good addition because they make the whole experience more immersive; since musicians are able to engage not only their sense of sight and hearing, but also their sense of touch.

The Performance Without Barriers team mentioned that most immersive technologies are to marketed to able-bodied users, and I think this is true. With the ever-growing popularity of immersive technologies, it is important to create immersive experiences that are inclusive too. 
Through the examples mentioned, the design of any immersive tool really needs to be well thought out, especially if it’s going to be used in special needs learning or music settings. Since I aim to develop a mixed reality musical instrument app for my final MSc project, I really need to research the best way to design my app’s interface so that it will be easy for people with limited movement to use it.

In my next TechTalk, I will discuss design considerations, available AR app development software, and the MR hardware I plan to use. 

Once again, I’m Francesca, and thank you for listening to my TechTalk.

References:

Want to connect or collaborate? Send me an email here.

TechTalk 1: Me, Myself, and iOS Development Project Topic

In this vlog, which I will refer to as TechTalk from now on, I introduce my final project topic for my MSc in Audio and Music Technology. Spoiler alert! The topic is Mixed Reality Musical Instruments iOS App for Special Needs Music.

Transcript:

If you’re interested in audio, music, and technology, then you’ve come to the right place.

Hi! I’m Francesca, and you’re watching my TechTalk

I’m Francesca Lorico, a graduate student at the University of York pursuing my master of science degree in audio and music technology. This is the first of many TechTalks (maybe just 4?) documenting my final project for my program. In this TechTalk, I will give you an overview of my project topic, goals, research considerations, reason for choosing this topic, and my hopes for the project. Let’s get started!

The idea for my final project is to create a Mixed Reality Musical Instruments iOS App for Special Needs Music. That’s a bit of a mouthful, I know, so let’s break it down.

To better understand what mixed reality or MR is, we need to talk about augmented reality or AR first. According to Intel, “AR overlays digital information on real-world elements.” The perfect example of this technology can be seen in the very popular mobile game, Pokemon Go. When it was first released in 2016, a lot of people started swarming outside– to parks, beaches, churches, and many other places trying to locate, capture, and train pokemon using their mobile device’s GPS. But more than that, the app allowed them to see the pokemon in the world around them through their mobile phone’s camera. This made the whole gameplay more enjoyable for the players because they got to experience a unique combination of the real world and the game they were playing. So, in simple terms, AR lets us see virtual elements in the real world through a device. This is the foundation of Mixed Reality. But, in addition to being able to see virtual elements in the real world, MR also allows us to move and manipulate virtual elements in the real world using our own hands. Keep in mind, this is only possible if you have the gear and software to run it.

Now that you have a general idea of what MR is, you may already know what I mean by… Mixed Reality Musical Instruments iOS App. But to give you a better sense… Try to picture yourself wearing regular eyeglasses and holding sticks. You’ll be able to see your surroundings and the sticks, right? But imagine if these eyeglasses could let you see virtual things too. Look! the sticks are now wooden rods and there’s a virtual musical instrument in front of you!

Now let’s move on to the last part of the project topic. What does it mean when we add the words ‘for Special Needs Music’? The Merriam-Webster dictionary defines special needs as “any of various difficulties (such as a physical, emotional, behavioral, or learning disability or impairment) that causes an individual to require additional or specialized services or accommodations (such as in education or recreation).” There are numerous specialized services that enrich the lives of individuals with special needs, and one way is through music therapy. According to the British Association For Music Therapy or BAMT, “Everyone has the ability to respond to music, and Music Therapy uses this connection to facilitate positive changes in emotional wellbeing and communication through the engagement in live musical interaction between client and therapist.” 

So, going back to my project, the mixed reality app I plan to develop is geared towards becoming a possible medium that can be used in special needs music therapy sessions, particularly for people with limited movement. 

To sum it all up, the ultimate goal for my final MSc project is to develop an augmented reality musical instruments iOS app using an AR development software, paired or interfaced with a specific device for a mixed reality experience, that is especially beneficial for people with limited movement. With that goal in mind, the research will focus on the following:

1. What are the possibilities for developing an MR application for special needs music? 
2. How is MR technology beneficial for special needs music?
3. What are the limitations of MR for special needs music?

Finally, the reason I chose to research this is because I truly believe in the power of music and how it can help and heal people. Through this project, I really hope to produce an app that would give people with special needs, specifically those with limited movement, the chance to experience playing a musical instrument in an effortless way using virtual means.

In my next videos, I will discuss the project in more detail. Specifically, I will talk more about MR, existing technological solutions for special needs music, and many more. So, stay tuned!

Once again, I’m Francesca, and thank you for listening to my TechTalk.

References:

Want to connect or collaborate? Send me an email here.