Project 2
Studio IV, spring 2023 -audio-based mixed-reality interactions
figma(continued process):
https://www.figma.com/file/SyWZmyudnsZ5ejDrtggrH1/Audio-Odyssey?t=0hVbkLZGpUrTFLXO-1
Brief:
In multisensory interactions, visual and haptic inputs and outputs have been mainly used while spatial audio has been underutilized. How do we leverage spatial audio in augmenting our environments? How can we use sound as a main channel of our interactions with others, with objects or/and with our environment?
Considerations:
Consider novel ways for interacting with your environment:
What are the interactions that you are introducing in your project?
What are the inspirations for your project?
Prototype by making early versions to test the overall form of your project and its relation to its environment.
OBJECTIVES
- Consider different levels of interactions in an environment.
- Ability to critically map human behavior and shaping it in an environment
- Research through Design and Making
- Master multisensory interactions
Process:
3/26
Initial Ideas:
- Jbl speaker system; branding, environment; party mode; spatial audio; embedded speakers; JBL pulse 4-reference
- Jbl branded experience to bring your room to life(Speaker brands are all about the nuanced detail, material, and high quality renders); high end speaker brands(unique, clear, minimalistic)
- Jbl as a brand is younger, more youthful, an early 20 year old, ambitious(still young relative to other, high-end speaker brands)
- E-boys or Bro-Dudes(design group name)
- 3d location- mapping of where the sound is produced
- JBL speaker/ luxury car experience/branded pop-up
- Shoes that make a custom track for you based on your walk(But in this version you get to pick the genre and it generates/recommends a song for you); this is based on what?-temperature-time of day
- AI spotify DJ(recommends song of the moment);Can’t decide what to listen to? take a quick 5 question quiz to give you the perfect song to be in the moment; could be for a special occasion; more memorable moments, hanging with friends
3/27
Initial Concepts:
- 3d location mapping
- JBL branded experience
- JBL speaker system(i.e. Turn any living room into a party environment); Adjusting the visuals, music/vibration engineering
- Creating a futuristic based design that is an extension of the JBL product line; Goal: Being able to instantly change an environment(study to party mode); IOT of devices, connects with all light and devices in the room; Connecting other rooms with the same experience/vibe; The amount of people changes the experience; visuals/LEDS that connect to the speakers/part of the JBL system; Ability to feel the sound, bass embedded into the furniture(more accessible); JBL visuals: organic growth; designing a customizable space through 3D organic visuals
- Shoes that make a custom track for you(recommending songs; based on AI Spotify DJ)
chatGpT exploration
JBL System
- Multi-Room Setup: Create a multi-room setup with the JBL speakers so that the music and lighting can be seamlessly enjoyed throughout the entire environment.
- Live Music Integration: Incorporate live music into the JBL speaker system by allowing musicians to connect their instruments directly to the speakers and synchronize with the lighting, visual displays, and haptic feedback technology(creating a portable recording studio)
- Haptic Feedback: Integrate haptic feedback technology into the speakers to create vibrations and physical sensations that enhance the music listening experience.
- -Visual Displays: Add visual displays to the JBL speaker system, such as projection mapping or LED screens, to complement the music with stunning visuals.
3D location mapping w/ audio
- Personalized Sound Experiences: Use 3D sound mapping to create personalized sound experiences for each user. For example, the branded speaker could use machine learning algorithms to analyze each user’s preferences and create customized soundscapes based on their individual tastes and preferences.
- Interactive Soundscapes: Use 3D sound mapping to create interactive soundscapes that allow users to explore different sounds and audio effects. For example, users could move around a room and hear different sounds as they approach or move away from different speakers.
- Immersive Audio-Visual Experiences: Integrate 3D sound mapping with visual displays, such as projection mapping or LED screens, to create immersive audio-visual experiences that showcase the unique qualities of the branded speaker. For example, the visual displays could be synchronized with the sound maps to create a dynamic and engaging experience.
Precedents:
How to map a room using only a sound: https://www.kurzweilai.net/how-to-map-a-room-using-only-a-sound
Mapp 3D: https://meyersound.com/product/mapp-3d/
Brainstorming:
- JBL — connecting people through sound. People’s location affects the sounds created in the room. Could possibly base this on Conway’s Game of Life. Distance in between two people affects the sound.
- Spotify system + AI music generator -shoes that make a custom soundtrack based on your walk/run. Building off of the Nike Music Shoe concept and using the help of Spotify AI DJ to recommend songs or playlists.
- Spotify Song of the Moment-generate a song based on a quick quiz to help make this moment more memorable
- 3D location mapping -simple interactions, embedded speakers, insane visuals
Story for JBL experience:
- Bring the JBL system into the room
- Press the button and power it up
- Test room space
- Snap fingers or press button to digitally map room
- Pairing app shows where to locate additional JBL add ons
- Customizable haptic feedback + visuals
- Access to a variety of preset modes/themes
- People moving in the spaces changes
Continued Stories:
- JBL branded experience to bring any room to life
- Spotify Song of the Moment(Spotify AI DJ)
- 3D location mapping
- Design Firms + Soundscapes: Arup, Sonos Studio, Harman, 4DSound, SonicMinds
Research Question:
How might we…
-use spatial audio to unify a space
-connect people across the world; connect with family
-use spatial audio to bring any room to life
-turn any room into the ultimate customizable party space
-use spatial audio to create a portable recording studio
Use spatial audio to add to mundane activities to create more enjoyable experiences?
3/28
Connect physical(sound) w/ interface, Can music help enhance the activity you’re doing?-walking-waking up-showering-studying-partying-eating food- cooking (cooking movements, cooking times)-game-based-acoustic mapping, lidar scanning, sonar sensors-talk to Daniel Rosenberg about 3D mapping-making Spotify more accessible
Cooking Experience
- Customized Spotify playlist for cooking. Enter a photo of the thing you want to cook, AI analyzes it and tells how long you need to cook it for. The duration and dish you want to make is sent to Spotify, which generates a playlist that is as long as the cooking duration and is related to the dish you are cooking. Put each ingredient / song into a playlist. -cooking movements -cooking times
- enchanted spatula
- wearable for chef
- Making the cooking experience easier, beginner, and more novel
Proximity with your friends:
Travel + Walk-by + Music + Apple music/Spotify -find music nearby and pick up (catch) songs that people near you are listening to; connecting people through music; pick-up new songs on a walk; Take a selfie every time you catch a song to remember how you came across this song; could gamify it and create a leaderboard-pokemon go; Catching songs through AR — pokemon go?
Refining Concept:
Spotify Song Catch + Song of the Moment
3/29
3D visual mixer; depending on how close you are to different objects, it changes the intensity of the different instruments… can you create your own songs through gestures / walking closer to certain parts of the song?
-You can save certain parts / elements of the song. These are live animated objects in real space.
-Become your own studio conductor
-experiencing a live concert
Overarching Themes:
-branded experience with apple music/apple watch
-Our service enhances travel and allows the user to experience a new place through a musical lens. Top 10 songs in that city.
User flow 1:
- Download the app and enable location
- As Luca is walking while listening to music on this app, Luca comes across MAX who is listening to a different new song.
- Luca gets a notification “somebody near you is listening to Bad Blood by Taylor Swift”. Capture this song?
- If the captured song is in the top 10, Luca can choose to visualize this song in real space. Luca presses the “visualize / live concert” button
- Luca gets to decide which part of the song he wants to experience the live concert of
- AR experience begins where different instruments of the song are put in physical space. Luca can walk near an instrument to increase its volume and pay close attention to it.
- Luca has the option to save specific instruments of the song to the app that he can explore with later.
User flow 2 — Live Concert:
- Download the app and enable location
- You get 10 new songs on a daily basis that you are supposed to find and experience in AR.
- You are walking and get a notification “song near you”. You capture that song and press the “live concert button”
- AR experience begins where different instruments of the song are put in physical space. You can walk near an instrument to increase its volume and pay close attention to it.
- You can save specific instruments of the song to the app
- You can share the song card on social media “I found Bad Blood near the Exchange”
User flow 3 — Product for smaller artists in a neighborhood .. uplifting the smaller community:
Target audience — small local artists.
- Download the app and enable location
- You get songs from local artists in the region on a daily basis that you are supposed to find and experience in AR.
- You are walking and get a notification “song near you”. You capture that song and press the “live concert button”
- AR experience begins where different instruments / tempo / pitch / frequency of the song are put in physical space. You can walk near an instrument to increase its volume and pay close attention to it.
- You can save specific instruments of the song to the app
- You can share the song card on social media “I found “Bad Blood” near the Exchange”
Questions:
What are the interactions you are introducing with your environment?
What are the inspirations for your project?
What’s the process or method to address this question or problem?
Who are your users? What is the context?
Precedent: https://www.noussonic.com/showcases/pulse-louvre
Research Question:
- How might visualize a spatial audio experience to help local, smaller musicians standout and gain exposure?
- How might we create an immersive song experience to visualize the complexity of music?
- How can we create multi-sensorial music experiences?
- Lyrics shown in AR too?
References: https://rd.nytimes.com/projects/key-concepts-in-spatial-audio
Figma Process:
