Healthbee

Healthbee is a real-time AI assistant that simplifies complex medical conversations, suggests questions, and summarizes consultations, helping patients better understand their healthcare

Comment

Inspiration

For many people, medical terminology is difficult to understand. When visiting a doctor or picking up medication, patients often hear explanations from doctors or pharmacists that are filled with complex clinical language. Because this language is highly technical, many people struggle to fully understand their condition, treatment plan, or how to properly use their medication.

This gap in understanding can cause patients to miss important details, forget medical advice, or even make poor health decisions.

The problem is even more serious for:

  • New immigrants to Canada
  • People whose first language is not English
  • Anyone without a medical background

With the rise of Generative AI and Agentic AI, we saw an opportunity: to use AI as a real-time communication assistant during everyday medical conversations. Our goal is to help people understand complex medical information and make better health decisions.

That’s why we designed Healthbee — a medical communication assistant powered by Agentic AI.

Healthbee helps users clearly understand medical information and manage their health with confidence. By leveraging generative AI’s real-time comprehension, summarization, and translation capabilities, Healthbee acts as an intelligent assistant during medical conversations, making complex medical information easy to understand, review, and act upon.

What It Does

Healthbee is a real-time AI prompt tool that helps users understand complex medical information while communicating with doctors, pharmacists, or insurance providers.

It offers three core features:

1. Real-Time Medical Language Simplification

When a doctor explains a condition, Healthbee instantly translates complex medical terminology into simple, easy-to-understand language. It can also translate explanations into other languages, helping users immediately grasp the doctor’s meaning.

2. Smart Question Suggestions & Conversation Summaries

During conversations, patients may forget important questions. Healthbee analyzes the discussion and automatically generates relevant question suggestions.

After the appointment, Healthbee generates a clear structured summary, including:

  • Diagnosis information
  • Medication recommendations
  • Suggested next steps for health management

3. Medication and Health Information Explanations

When picking up prescriptions, Healthbee instantly explains:

  • The medication’s purpose
  • Dosage instructions
  • Frequency of use
  • Potential side effects

All instructions are simplified into easy language and can be translated into the user’s preferred language (for example, Chinese).

Privacy and Security

Healthbee places strong emphasis on privacy and security:

  • All medical conversations are encrypted
  • Data access is controlled by the user
  • Information is shared only with explicit authorization

How We Built It

Real-Time On-Device Audio Pipeline

Healthbee’s entire audio processing pipeline is deployed locally on-device, supporting both iOS and Android.

The system performs:

  • Real-time speech-to-text (STT)
  • Speaker diarization

These processes operate with millisecond-level latency, allowing Healthbee to instantly understand conversations between patients and healthcare professionals.

On-Device Inference

We use Sherpa-ONNX, an open-source inference runtime optimized for mobile hardware.

Processing speech locally provides two key benefits:

  • Lower latency
  • Improved privacy, since raw audio never leaves the device

Multi-Agent AI System

On the backend, we use LangGraph to coordinate a multi-agent AI system.

Once speech is transcribed, a scheduler distributes the conversation content to specialized agents, each designed for a specific medical task.

Examples include:

Medical Terminology Agent

  • Detects complex medical terms
  • Translates them into clear explanations

Medication Information Agent

  • Provides medication purpose, dosage, and side effects
  • Generates simplified summaries
  • Translates explanations into the user’s native language

Question Suggestion Agent

  • Analyzes the conversation
  • Generates potential questions the patient may want to ask

Consultation Summary Agent

After the consultation ends, Healthbee generates a structured summary including:

  • Diagnosis information
  • Medication guidance
  • Recommended next steps

Proactive AI Assistance

Healthbee uses a proactive push mechanism.
When important information is detected, recommendations are instantly presented to the user—similar to a real-time teleprompter, staying one step ahead of the patient’s thoughts.

Personalized User Memory

We implemented a user memory layer that stores:

  • User preferences
  • Communication styles
  • Historical health data

This ensures Healthbee provides personalized and context-aware assistance.

Security Architecture

Healthbee protects medical data through:

  • End-to-end encryption
  • User-controlled permissions
  • Secure cloud infrastructure

All data access requires explicit user authorization.

Challenges We Ran Into

1. Real-Time On-Device Audio Pipeline

Running speech-to-text and speaker diarization simultaneously on-device was challenging. These models operate at different speeds and produce outputs at different intervals.

We designed a custom synchronization algorithm to merge both output streams in real time, ensuring speech is correctly attributed to the right speaker without introducing noticeable latency.

Supporting this across both iOS and Android devices with varying hardware performance was one of our toughest engineering challenges.

2. Building a Truly Proactive Multi-Agent System

Traditional AI systems respond to user prompts. Healthbee needed to anticipate user needs in real time.

To achieve this, we built a gateway agent that continuously analyzes conversations and schedules specialized sub-agents in the background.

These agents retrieve information such as:

  • Medical explanations
  • Medication details
  • Insurance information
  • Health advice

The challenge was ensuring that responses appeared at the right moment, without delays or disorganized outputs.

3. Security and Account Management

Healthbee supports users connecting third-party systems such as:

  • Insurance platforms
  • Health records
  • Medication information databases

Each user session must remain secure and isolated.

We redesigned the backend architecture to support independent authentication and permission-based data access, ensuring personalized and secure AI assistance.

Accomplishments We’re Proud Of

End-to-End Real-Time Audio System

We successfully launched a fully functional local real-time audio pipeline. Seeing Healthbee explain medical terms, suggest questions, and summarize consultations during live conversations proved that the product is both viable and valuable.

Multi-System Integration

We integrated five major technical components into a cohesive system rather than stacking separate tools. The result is a platform that genuinely improves communication efficiency in healthcare settings.

Strong Privacy Protection

Healthbee implements end-to-end encryption and user-controlled access, ensuring sensitive medical information remains secure.

What We Learned

Real-Time Speech Systems Are Complex

Streaming audio processing requires careful management of:

  • Buffering strategies
  • Streaming APIs
  • Latency budgets

We had to design Healthbee differently from traditional request-response AI applications.

AI Can Improve Everyday Life

Healthbee showed us how AI can meaningfully improve daily experiences by helping people better understand health information and communicate more effectively with healthcare professionals.

Security Must Be Built In From the Start

Protecting sensitive medical data requires security considerations at every stage of development.

Breaking Language Barriers Matters

Healthbee helps remove language barriers that prevent people from accessing healthcare information—an especially important feature for immigrants and non-native English speakers.

What’s Next for Healthbee

Personalized AI Assistance

As users interact with the app, Healthbee will learn their communication patterns and knowledge gaps, enabling more personalized recommendations.

Pre-Appointment Preparation

We plan to integrate calendar and meeting information so that before a medical appointment, Healthbee can:

  • Load relevant health background information
  • Generate suggested questions
  • Prepare health summaries for doctors

Cross-Platform Expansion

Healthbee will expand to additional platforms, including Android desktop environments, allowing users to access the assistant across multiple devices.

Built With

Share this project:

Updates