Synthesia
TYPE
Google Labs x USC Makeathon
ROLE
Visual Engineer & Designer
DURATION
3 Days
ORGANIZATION
USC
An interactive AI-powered music visualization system that uses Google's Mediapipe for motion tracking to control and visualize music, creating a dynamic audio-visual experience.
The Challenge
The Google Labs x USC Makeathon challenged teams to create innovative projects that leverage AI for creative expression. Our team set out to explore how AI could bridge the gap between movement, music, and visual art by creating an interactive system that responds to physical gestures.
Project Goals
- Create an interactive system that translates movement into sound and visuals
- Use Google's AI tools to track and interpret physical gestures
- Generate dynamic music that responds to user movements
- Visualize the relationship between movement, sound, and color
The Tinkerer's Guide to AI session at the Google Labs x USC Makeathon
Try the Interactive Demo
Experience Synthesia for yourself! Move your hands in front of your camera to control the music and visuals. The left hand manipulates the vocal track while the right hand controls the instrumental track. Vertical movement adjusts volume, and horizontal movement shapes reverb and delay effects.
Note: This demo requires camera access to track hand movements. For the best experience, ensure you're in a well-lit environment with your hands clearly visible to the camera.
Our Interdisciplinary Team
Synthesia was created by a diverse team of students with backgrounds in design, development, music production, and performance. My role as Visual Engineer involved designing the visual elements and performing movement for the system to track.
Aryan demonstrating the Touch Designer interface during development
The makeathon venue at USC where teams collaborated
Team Members & Roles
- Sonya Chiang & Belissima Arriaga: Graphic and Visual Design, AI animation
- Aryan Kapoor: Developer, Producer, Motion Tracking Machine Learning
- Jericho Sanchez & Meghna Nair: Visual Engineer, Dancer, Stable Diffusion Designer
- Seraph Kumar: Project Management, Music Production
- Karen Cho: Graphic Design, Singing, Music Production
Technology & Implementation
Synthesia integrates multiple AI technologies to create a seamless interaction between movement, sound, and visuals. The core of the system uses Google's Mediapipe for real-time hand tracking, which then controls parameters in our custom audio engine and visual generator.
Touch Designer interface showing real-time motion tracking and visual generation
AI Tools Used
- Google Mediapipe: For real-time hand tracking and gesture recognition
- NotebookLM: Used to learn and implement Touch Designer functionality
- Touch Designer: For creating real-time visual responses to audio and movement
- Runway: For generating AI-powered video content
- ImageFX: For creating visual elements and mood boards
How It Works
The interactive music is controlled through hand movements:
- Left hand manipulates the vocal track
- Right hand controls the instrumental track
- Vertical movement adjusts volume (higher = louder)
- Horizontal movement shapes reverb and delay effects
- The system generates visuals that respond to both the music and movement
The Synthesia Experience
Synthesia creates a unique multi-sensory experience where users can literally "see" and "shape" music through their movements. The system responds in real-time, creating a feedback loop between the user's gestures, the evolving soundscape, and the dynamic visuals.
The Google Makeathon event space where we developed and presented Synthesia
Behind the scenes of our team working on the Synthesia project
User Journey
- User approaches the installation and stands in front of the camera
- System detects their hands and begins tracking movements
- Initial sounds and visuals appear, responding to even slight movements
- User explores how different gestures affect the audio-visual experience
- The system adapts to the user's movement style, creating a personalized experience
- Multiple users can interact simultaneously, creating collaborative compositions
Results & Recognition
Synthesia was well-received at the Google Labs x USC Makeathon, with judges and participants appreciating the innovative use of AI for creative expression. The project successfully demonstrated how AI tools can enhance artistic experiences and create new forms of interactive entertainment.
Key Achievements
- Successfully integrated multiple AI technologies into a cohesive interactive system
- Created an intuitive interface that allowed users with no musical training to create expressive sounds
- Developed a real-time visual system that responded dynamically to both movement and audio
- Demonstrated the potential of AI as a creative collaborator rather than just a tool
- Created a publicly accessible web version that allows anyone to experience the project
Reflections
Working on Synthesia taught me valuable lessons about the intersection of technology and creative expression. The project challenged our team to think beyond traditional interfaces and explore how AI can create more intuitive and embodied interactions with digital systems.
Key Learnings
- The importance of cross-disciplinary collaboration when creating AI-powered creative tools
- How to design systems that feel responsive and "alive" through real-time feedback
- Techniques for making complex technology accessible through intuitive physical interactions
- The potential of AI to augment rather than replace human creativity
- The value of rapid prototyping and iteration in developing interactive experiences
See Synthesia in Action
Watch our project demonstration video to see how Synthesia translates movement into music and visuals:
You can also watch on YouTube for a full-screen experience.