Bee the Beat | Devpost

Inspiration

We love music, it's fun, and it connects people. But not everyone has an instrument, and sometimes you just want to make quirky beats or splashes of colour without needing formal training. We wanted to create something fun and accessible, so we built a website where anyone can play music with their hands in mid-air, save or download their creations, and share them with friends to spread joy and a little pop of colour!

What it does

Bee the Beat transforms simple hand movements into music and visuals. By using your webcam, the app tracks your hands in real time and maps each finger movement to specific notes, scales, or sounds, allowing you to “play” music in mid-air like a virtual instrument.

Every movement also generates colourful, paintball-like visual splatters on the screen, making the experience both auditory and visual. Users can choose between different sound modes and instruments, experiment with pitch shifting, and even switch scales with just a pinch. Once a recording is finished, the app allows users to save their creation, download it locally, or upload it to a gallery where it can be viewed again and again.

How we built it

Frontend: React + Vite for UI and gallery submission.
Computer Vision: MediaPipe for hand tracking in real-time.
Sound Creation: Sounddevice (Python) to generate audio waveforms.
Backend: Flask API to handle uploads, gallery management, and AWS integration.
Database: AWS for storing recordings and metadata.

Challenges we ran into

Full-stack integration: Connecting React, Flask, and AWS while keeping everything smooth.
Low latency: Ensuring real-time response between hand motion, sound generation, and visuals.
Gesture recognition: Differentiating between subtle movements like pinches, taps, and swipes.

Accomplishments that we're proud of

We were able to combine multiple hand gestures (pinches, taps, sweeps, and finger taps) to make the experience both intuitive and expressive. We were also able to integrate the frontend, backend, and database to make a fully functioning website. We also added a gallery feature where users can save to and show their creations with others.

What we learned

How to integrate computer vision, sound processing, and full-stack web tech into one project.
Hands-on experience with real-time audio generation and MediaPipe gesture recognition.
Careful API design when linking frontend and backend in hackathon time constraints.
Focusing on visuals and the UI for a better user experience.

What's next for Bee the Beat

Add more instruments (realistic piano, guitar, violin, etc.) via soundfonts.
Expand accessibility by supporting gesture-based chords and one-hand play or maybe using facial expressions.
Integrate with a mobile app so you can play anywhere with just your phone camera.
Add a collaborative gallery where people can remix and share performances.
Improve visuals into a full audio-reactive stage experience (like a mini virtual concert).