Apple Vision Pro

Investigating Emerging Technology with an Accessibility Focus

Discovery of the capabilities of Apple Vision Pro to create an app proposal that puts users with mobile disabilities as the primary focus.

App Research & Ideation Hackathon – Created in Association with Dogstudio/DEPT

✏️

Research

🗺

Product Strategy

🎨

Interface Design

✏️ Research 🗺 Product Strategy 🎨 Interface Design

Brief

Goal

We’re looking for an experience that leverages AI for the Apple Vision Pro headset and provides an accessible, entertaining experience to users with low mobility.

Full Deck Proposal

We have a lot of nitty-gritty stuff to say on this one – check out the full deck for the finer details

view the deck

Observation

We need more information to understand how to build with the tech, who the users are, and what these users want to experience 🤔

Research Phase
To guide us in the right direction, we get acquainted with the need-to-know topics

Audience Focus

Low-Mobility Functionality – What features does VisionOS provide for specific motor needs?
Immersive Technology on User Mentality – How does an immersive environment transform the user’s mind and impact what they do next?
Feeling Included is Important – How can we create for low-mobility users while inviting everyone to the table?

Technology Focus

Visuals – How do you display elements in the virtual space?
Interactions – How do you interact with those elements?
Audio – How does sound interact with virtual elements and real elements?

Ideation Phase
With enough research to know what we’re covering, we move right into churning out ideas with action

First, we utilized ChatGPT to spitball possible app concepts – creating tangents of ideas strings we didn’t even consider.

Next, we spend a bit more time on concepts that hit all our focus points – checking their validity with the who, what, why how format to cover our bases and expand the idea.

Problem

What else can we provide users to keep continued app engagement?

Solution

Activities focused on the mind through learning, creativity, and emotion.

What is the product?

Who is the product for?

Youths Interested in Starting Music

Empathy Focus

To set the scene, imagine yourself as Ace. He’s an only child with a stay-at-home single mom and he’s just doing his best with the circumstances his short life has given him. Being bound to a wheelchair since childhood has made life difficult to navigate, and making friends is not any easier. Especially now with being home-schooled to help with his autism and easing the added stress from classmates, he feels alone and out of place in this world. Music has always been an escape for him – whether to de-stress or get happy, he feels a connection with it. Deep down, he’s always wanted to get more musically involved, but with his life the way it is, he’s never been able to pursue it further.

Adults with Musical Know-How

Empathy Focus

To set the scene, imagine yourself as Claire. She works remotely as a Sr Customer Service Rep. She likes her job, as she’s always enjoyed people, but as her Multiple Sclerosis (MS) progresses doing an on-site job becomes more and more challenging. So she working with what she’s able to do to keep her independence. Claire has a handful of close friends who support her and are able to assist her when she needs it. But even with the support, having ways to unwind, and entertain herself is a struggle. Before her MS diagnosis, she used to play clarinet in the school band… and it’s something she misses a lot.

Why make this product?

By visualizing sounds, musical concepts are more accessible to any mobility and mentality. Music encompasses learning, creativity, and emotion, effortlessly – even without words.

Emotional ❤️‍🔥

Music alters the body’s response to physical tasks… aka gets us feeling any which way – and that’s the best part of it.

Creative 🎨

Music provides an expressive outlet to help relieve stress and anxiety while developing imagination and creative thinking.

Educational 📚

Understanding music can help children visualize various elements that should go together like they would do when solving a math problem.

What does the product do?

It’s learning through focused mini-games

Education Focus

For those not knowing where to begin, a series of 5-15 minute long mini-games are available for a variety of topics. Completing games leads to accomplishments, badges, and streaks that reward the user on their learnings – as well as, encouraging them to come back and play.

It’s creating layered sounds into visual compositions

Creative Focus

Users will have access to a library of sounds that they can apply to their space. When multiple sounds become layered, they will auto-adjust their parameters to follow the same time signature. In addition to creating as many layers of sounds as the user likes, they will also be able to play with the tempo and tone of the entity they are seeing.

It’s enjoying visualized sounds in your personal space

Emotional Focus

For those who don’t want to create music but still engage, the app provides a visual oasis for the user to enjoy. Users can choose to enjoy with a mood or instrument focus – and then watch the world build around them. We also give the users the ability to generate new visuals and sounds within the chosen mood or instrument.

It’s a way to share with family and friends

Community Focus

Users can invite loved ones to enjoy the app with them. Whether it’s relaxing in a tranquil piano melody, or showcasing your latest layered creation – users can get the sense of community that music is all about.

How does the product function?

👌

Indirect tough for everything

🎸

Support from GarageBand for the composing software

✨

AI, but with parameters for sounds and colors

🎵

Support from Apple Music for the sound library

Four Ways to Play

Create – layering sounds to create visual soundscapes
Vibe – choose a mood or instrument and enjoy the generated soundscape
Learn – select a mini-game to hone your musical knowledge
Share – sharing and inviting others to enjoy soundscape together

Create – User Flow

Vibe – User Flow

AI, AR & 3D Strategy

Visualizing Sounds

When visualizing sound, we need to map certain parameters. We use different audio parameters like timbre, level, pitch, and instrument types as the basic elements of music. Each note played is like adding a stroke to an imaginary canvas.

By carefully linking visual elements to their corresponding audio qualities, we create a seamless connection between what you hear and what you see. Colors come alive, expressing the emotions of the melody. Heights rise and fall, following the changes in pitch and rhythm.

Colored Tones

Color theory in sound is the idea of connecting specific colors with different aspects of music. For example, a high piano note might be seen as blue, while a low guitar note could be red. This adds visual richness to the music, allowing you to see colors as you play different notes. It brings an extra layer of visual expression to your musical journey.