In this TechTalk, we will look at an example of a technological solution for music therapy or special needs music and relate it to the design of the app for my final MSc project.
Transcript: If you’re interested in audio, music, and technology, then you’ve come to the right place. Hi! I’m Francesca, and you’re watching my TechTalk. In the last TechTalk, I talked about inclusive immersive technology. In this video, we’ll discuss app design considerations, available AR development software, and the MR hardware I plan to use. Let’s get started! In 1987, one of my supervisors worked on a project called MidiGrid. It is an interactive computer-based musical instrument developed primarily for people with physical disabilities. The concept, as mentioned in a paper he co-wrote, is that “MidiGrid allows users to trigger musical material freely in real-time using the computer’s mouse… the screen shows a grid of boxes, each of which contains a nugget of music that is triggered as the mouse cursor moves over it. Hand gestures are thus converted, via the mouse into notes, chords and musical sequences.” To simply put, MidiGrid allows users to play the stored music by moving a mouse cursor over a box. In addition to that, since it is created primarily for people with physical disabilities, the mouse cursor movement range is designed to be customizable to cater to the user. With features like this, MidiGrid is quite useful for music therapy improvisation sessions. This is just scratching the surface of what MidiGrid is, but with this information, it made me wonder about how I should approach my final project. Since I aim to develop an app that’s geared towards people with limited movement, the design of the interface should be well thought out. Currently, these are the design considerations that I have: For the app’s interface: 1. What should be the layout of the menu? 2. How big should the buttons be? For the hardware: 1. What are the limitations of using a certain MR hardware? 2. What types of mobilities should I take into account? Will the addition of hand-held controllers make it difficult for people with limited movement to use the app? 3.Is it a must to add haptic feedback or vibrations? After I sort out how the app should be designed, the next stage would be the app development process. There are several software options for creating an AR iOS app. At the moment, I am looking at Apple’s ARKit and Unity’s AR Foundation plus XR Interaction Toolkit. ARKit is a framework for Swift that that allows developers to create AR experiences for apple devices. Similarly, Unity’s AR Foundation is the same, then the XR toolkit is an easy way to add interactivity to virtual elements. Initially, I would like to use ARKit because it’s accessible and I think it’s more user-friendly for someone like me who is still learning the ropes of coding. Then again, I feel like Unity would be user-friendly too, especially with the addition of the XR Interaction Toolkit. I’m still undecided at this point, so I really need to do a more in-depth comparison between the two development tools. Maybe in a future video, I’ll try to do a comparison. What do you think? Anyway, as for the MR hardware, I’ve been looking at ZapBox. It’s an affordable mixed reality kit for designing and developing mixed reality experiences. It comes with a cardboard headset, pointcodes that acts us room anchors to help workout the position of surfaces in a room, trackable cardboard controllers for interacting with virtual elements, a lens adapter for smartphone cameras, and an app for building a map of codes and browsing other content. Moreover, they provide documentation on how to integrate the codes to both Swift and Unity, which I think is very convenient for my project. I’m looking for other options too as I’m still in the research and planning stage, but ZapBox is definitely the top option for now. There’s still a long way to go before my app comes to fruition, but I can see the exciting journey ahead. I’ll try to post more TechTalk videos about the project, especially when it’s in the app development process already. Stay tuned for that! Once again, I’m Francesca, and thank you for listening to my TechTalk.
References:
- MidiGrid: Past, Present and Future by Andy Hunt and Ross Kirk- https://www.researchgate.net/publication/2949294_MidiGrid_Past_Present_and_Future
- MidiGrid Images: http://midigrid.com/
- ARKit: https://developer.apple.com/augmented-reality/arkit/
- AR Foundation and XR Interaction Toolkit: https://unity.com/unity/features/ar
- ZapBox: https://www.zappar.com/zapbox/
Want to connect or collaborate? Send me an email here.
Francesca Lorico 