Dubbify

Our conference call app provides real-time, live translation for every participant, allowing your team to speak their native language while everyone else hears them in their own.

Comment

Inspiration

When looking at the theme of this hackathon of helping Canadians. Canadians are a very diverse group of people with many different languages. Some people don’t speak English, which can make it difficult for them to make phone calls, because of the language barrier or even studders. Dubify is the solution to this as we add a translator in the call.

What it does

Dubify is a call center web app that accepts inbound calls. The process of these calls is: The agent first deduces which language the person is speaking. If the language is english, the agent doesn’t interfere, If the language is not english, the agent translates between english and the foreign language.

How we built it

Frontend Next.js: A React framework for building the user interface React: For component-based UI development TailwindCSS: For styling with utility classes Supabase: For cookie-based authentication Shadcn and Framer Motion: For styling and animating components Backend Python and FastAPI: For SQL parsing and optimization Supabase: For storing user data and made queries ElevenLabs: For the agent that translates between languages Twilio: To manage phone numbers, so real phone numbers can call our app.

Challenges we ran into

ElevenLab: the agents' behaviours can be very unpredictable, which makes it very annoying when constructing a system prompt. Our app used a lot of tokens, before we started learning how to close agents and calls. Twilio: We got banned on Twilio when we had two accounts working on it. We got a second account. Twilio is also pretty expensive.

Accomplishments that we're proud of

We are proud to have both subtitles and real translations happen during the call, which allows for clearer communication. We are proud that the operator is quickly able to join the call after the language is detected. The twilio conference room management was impressive along with the webhooks that the agent is able to use.

What we learned

In this hackathon, we learned how to use popular AI tools like Elevenlabs to build something impressive. We also learned the capabilities of twilio and how we can make calls and conferences. We also learned about Websockets and a practical use case for them.

What's next for Dubbify

Logging calls, so you have the history of your phone conversations and the phone numbers that called. Support for different languages as the main language like French, Spanish, Mandarin.

Built With

Share this project:

Updates