# agent
Source: https://docs.pipecat.ai/api-reference/cli/cloud/agent
Manage agent deployments
The `agent` command provides sub-commands for managing your deployed agents. These commands allow you to view status, start agents, see logs, and manage deployments.
## start
Start a deployed agent instance, creating an active session.
**Usage:**
```shell theme={null}
pipecat cloud agent start [ARGS] [OPTIONS]
```
**Arguments:**
Unique string identifier for the agent deployment. Must not contain spaces.
**Options:**
Path to an alternate deploy config file. Defaults to `pcc-deploy.toml`.
Public API key to authenticate the agent deployment. Will default to any key
set in your config.
For more information, see [API keys](/pipecat-cloud/fundamentals/accounts-and-organizations#api-keys).
Stringified JSON object to pass to the agent deployment. This data will be
available to the agent as a `data` parameter in your `bot()` method.
More information [here](/pipecat-cloud/fundamentals/active-sessions#running-an-agent).
Skip summary confirmation before issuing start request.
Create a Daily WebRTC session for the agent.
Stringified JSON object with Daily room properties to customize the WebRTC
session. Only used when `--use-daily` is set to true.
See [Daily API
documentation](https://docs.daily.co/reference/rest-api/rooms/config) for
available properties.
Organization to start the agent for. If not provided, uses the current
organization from your configuration.
## stop
Stop an active agent session and clean up its resources.
**Usage:**
```shell theme={null}
pcc agent stop [ARGS] [OPTIONS]
```
**Arguments:**
Name of the agent. Must not contain spaces.
**Options:**
Path to an alternate deploy config file. Defaults to `pcc-deploy.toml`.
ID of the session to stop.
Organization which the agent belongs to. If not provided, uses the current
organization from your configuration.
Bypass prompt for confirmation before stopping the session.
## status
Shows the current status of an agent deployment, including health and conditions.
**Usage:**
```shell theme={null}
pipecat cloud agent status [ARGS]
```
**Arguments:**
Unique string identifier for the agent deployment. Must not contain spaces.
## deployments
Lists deployment history for an agent, including image versions and timestamps.
**Usage:**
```shell theme={null}
pipecat cloud agent deployments [ARGS]
```
**Arguments:**
Unique string identifier for the agent deployment. Must not contain spaces.
## logs
Displays combined logs from all agent instances, useful for debugging issues.
**Usage:**
```shell theme={null}
pipecat cloud agent logs [ARGS] [OPTIONS]
```
**Arguments:**
Unique string identifier for the agent deployment. Must not contain spaces.
**Options:**
Filter logs by severity: `ALL`, `DEBUG`, `INFO`, `WARNING`, `ERROR`,
`CRITICAL`.
Limit the number of log lines to display.
Filter results for specific agent deployment ID (obtainable from `pipecat
cloud agent deployments [agent-name]`).
Filter results for specific session ID (obtainable from `pipecat cloud agent
sessions [agent-name]`).
## list
Lists all agents in an organization with their details.
**Usage:**
```shell theme={null}
pipecat cloud agent list [OPTIONS]
```
**Options:**
Organization to list agents for. If not provided, uses the current
organization from your configuration.
Filter agents by region. Only agents deployed in the specified region will be
shown. If not provided, agents from all regions are listed.
## sessions
Lists active sessions for a specified agent. When there are no active sessions, it suggests how to start a new session.
When used with the `--id` option, displays detailed information about a specific session including CPU and memory usage with sparkline visualizations and percentile summaries.
**Usage:**
```shell theme={null}
pipecat cloud agent sessions [ARGS] [OPTIONS]
```
**Arguments:**
Name of the agent to list active sessions for.
**Options:**
Path to an alternate deploy config file. Defaults to `pcc-deploy.toml`.
Session ID to view detailed metrics for. When provided, displays CPU and
memory usage statistics including sparkline visualizations and percentile
summaries (p50, p90, p99).
Organization to list sessions for. If not provided, uses the current
organization from your configuration.
## delete
Deletes an agent deployment. This will prevent starting new agents and remove all associated data.
This action is irreversible. All data will be lost.
**Usage:**
```shell theme={null}
pipecat cloud agent delete [ARGS] [OPTIONS]
```
**Arguments:**
Unique string identifier for the agent deployment. Must not contain spaces.
**Options:**
Do not prompt for confirmation before deleting the agent.
# auth
Source: https://docs.pipecat.ai/api-reference/cli/cloud/auth
Authentication and authorization commands
The `auth` command group manages authentication with Pipecat Cloud, allowing you to login, logout, and check your current account identity.
## login
Begins an authorization flow to authenticate with Pipecat Cloud.
**Usage:**
```shell theme={null}
pipecat cloud auth login [OPTIONS]
```
**Options:**
Skip opening a browser window for authentication and print the URL instead.
Useful for remote or containerized environments.
This command initiates the authentication process by:
1. Generating a unique authentication URL
2. Opening your default web browser (if available)
3. Waiting for you to complete the sign-in process in the browser
4. Retrieving and storing your access token locally
If the browser doesn't open automatically, or if you use the `--headless` option, you can copy and paste the displayed URL into your browser manually.
On successful login, the CLI will store the access token in the local configuration file (defaults to `~/.config/pipecatcloud/pipecatcloud.toml`). This token will be used for all subsequent requests to the Pipecat Cloud API.
You can override the default location by setting the `PIPECAT_CONFIG_PATH` environment variable.
## logout
Signs out of the current Pipecat Cloud account by removing the access token from your local configuration file.
**Usage:**
```shell theme={null}
pipecat cloud auth logout
```
## use-pat
Authenticates with a [Personal Access Token](/pipecat-cloud/guides/personal-access-tokens) instead of interactive OAuth login. Validates the token against the API and stores it in your local config file.
**Usage:**
```shell theme={null}
pipecat cloud auth use-pat
```
**Arguments:**
Personal Access Token (must start with `pcc_pat_`).
You can also set the `PIPECAT_TOKEN` environment variable instead of storing
the token locally. See the [PAT guide](/pipecat-cloud/guides/personal-access-tokens)
for details.
## whoami
Displays information about the currently authenticated user, including user ID, active organization, auth method, and Daily API key.
**Usage:**
```shell theme={null}
pipecat cloud auth whoami
```
## Configuration
CLI configuration is stored in `~/.config/pipecatcloud/pipecatcloud.toml`. You can override this location by setting the `PIPECAT_CONFIG_PATH` environment variable.
The configuration stores your access token, active organization, and default API keys.
**View current configuration:**
```shell theme={null}
pipecat cloud --show-cli-config
```
We do not recommend manually editing the configuration file. If the file becomes malformed, run `pipecat cloud auth login` to regenerate it.
```toml theme={null}
token = "..."
org = "user-namespace"
[another-user-org]
default_public_key = "pk_..."
default_public_key_name = "Test Key"
[futher-user-org]
default_public_key = "pk_..."
default_public_key_name = "Pipecat Cloud Public Key"
```
***
Managing your account and collaborating on agents as part of a team
# build
Source: https://docs.pipecat.ai/api-reference/cli/cloud/build
Manage cloud builds
The `build` command provides tools for managing cloud builds on Pipecat Cloud. You can view build logs, check build status, and list recent builds.
Cloud builds are typically triggered automatically when you run [`pipecat cloud deploy`](/api-reference/cli/cloud/deploy) without specifying an image. These commands help you monitor and manage those builds.
## logs
View logs for a cloud build.
**Usage:**
```shell theme={null}
pipecat cloud build logs [OPTIONS] BUILD_ID
```
**Arguments:**
The ID of the build to get logs for. You can find build IDs using `pipecat cloud build list`.
**Options:**
Number of log lines to retrieve. Maximum is 10000.
Organization to use. If not provided, uses the current organization from your configuration.
## status
Get the current status of a cloud build.
**Usage:**
```shell theme={null}
pipecat cloud build status [OPTIONS] BUILD_ID
```
**Arguments:**
The ID of the build to check.
**Options:**
Organization to use. If not provided, uses the current organization from your configuration.
The status command displays detailed build information including:
* Current status (`pending`, `building`, `success`, `failed`, `timeout`)
* Region, context hash, and Dockerfile path
* Timestamps for creation, start, and completion
* Build duration and context/image sizes
* Error messages (if the build failed)
## list
List recent cloud builds.
**Usage:**
```shell theme={null}
pipecat cloud build list [OPTIONS]
```
**Options:**
Number of builds to list. Maximum is 100.
Filter by build status. Valid values: `pending`, `building`, `success`, `failed`, `timeout`.
Filter by region.
Organization to use. If not provided, uses the current organization from your configuration.
## Examples
**List all recent builds:**
```shell theme={null}
pipecat cloud build list
```
**List only failed builds:**
```shell theme={null}
pipecat cloud build list --status failed
```
**View logs for a specific build:**
```shell theme={null}
pipecat cloud build logs abc12345-6789-0abc-def0-123456789abc
```
**Check status of a build:**
```shell theme={null}
pipecat cloud build status abc12345-6789-0abc-def0-123456789abc
```
# deploy
Source: https://docs.pipecat.ai/api-reference/cli/cloud/deploy
Create or modify an agent deployment
The `deploy` command creates a new agent deployment or updates an existing one. It builds a deployment manifest with the provided parameters and monitors the deployment status until the agent is ready.
When no image is specified, the CLI will offer to build your agent using [Pipecat Cloud Build](/pipecat-cloud/guides/cloud-builds). This handles building and deploying without requiring you to manage a container registry. A `Dockerfile` must be present in your build context directory.
If the agent name already exists, you'll be prompted to confirm the update. Use `--force` to skip this prompt and also force a new deployment even if the configuration hasn't changed (replacing all running pods).
This command will wait for the active deployment / revision to enter a ready state before returning. If the deployment fails, the command will exit with an [error](/pipecat-cloud/fundamentals/error-codes) with more information.
## Usage
```shell theme={null}
pipecat cloud deploy [ARGS] [OPTIONS]
```
**Arguments:**
Unique string identifier for the agent deployment. Must not contain spaces.
URL of the Docker image to deploy. Must be a valid Docker image URL. For
example: `docker.io/my-repo/my-image:latest`. Not required when using [cloud
builds](/pipecat-cloud/guides/cloud-builds) or `--build-id`.
**Options:**
Path to an alternate deploy config file. Defaults to `pcc-deploy.toml` in the
current directory. Can also be set via the `PIPECAT_DEPLOY_CONFIG_PATH`
environment variable, with the flag taking precedence.
Name of the image pull secret to use for accessing private repositories. The
secret must be previously created using the `pipecat cloud secrets
image-pull-secret` command, and must be in the
[`ready`](/pipecat-cloud/fundamentals/secrets#provisioning-and-readiness)
state at deploy time.
Organization to deploy the agent to. If not provided, uses the current
organization from your configuration.
Name of the secret set to use for the deployment. The secret set must exist in
the specified organization and must be in the
[`ready`](/pipecat-cloud/fundamentals/secrets#provisioning-and-readiness)
state — deploys that bind a `pending` or `failed` set are rejected.
Minimum number of agent instances to keep warm at all times. Default is 0,
which means the agent will scale down to zero when not in use. Setting this to
1 or higher avoids cold starts.
Maximum number of allowed agent instances. Must be between 1 and 50. If you
need more agents, please contact us at [help@daily.co](mailto:help@daily.co) or via
[Discord](https://discord.gg/dailyco).
Enable Krisp VIVA noise cancellation with the specified audio filter model.
Valid values are:
* `tel`: Telephony model (up to 16kHz)
* `pro`: WebRTC model (up to 32kHz)
In addition to this flag, you also need to enable the `KrispVivaFilter()` for your transport. See the [Krisp VIVA](/api-reference/server/utilities/audio/krisp-viva-filter) documentation for more information.
Maximum session duration in seconds. When a session reaches this limit, the
agent's connection is forcibly closed — the session is cut off mid-flight
with no notice to the bot. Valid range: `60` to `14400` (4 hours). Defaults
to `7200` (2 hours) when unset.
See [Session duration limits](/pipecat-cloud/fundamentals/active-sessions#session-duration-limits)
for more information.
The agent profile to use for resource allocation. Valid values are:
`agent-1x`, `agent-2x`, `agent-3x`.
See [Agent Profiles](/pipecat-cloud/fundamentals/deploy#agent-profiles) for more information.
Region where the agent will be deployed. If not specified, uses your
organization's default region (typically `us-west`). Choose a region close to
your users for optimal latency.
Force a new deployment even if the configuration hasn't changed, replacing all
running pods. Useful for picking up updated container images when using
mutable tags like `latest`, or for refreshing modified secret values.
Skip all confirmation prompts, including cloud build prompts. Useful for CI/CD
pipelines. When used without an image, automatically triggers a cloud build.
Deploy using an existing cloud build ID instead of building a new image.
Cannot be used together with `--image`. You can find build IDs using `pipecat
cloud build list`.
Build context directory for cloud builds. Defaults to the current directory.
Path to Dockerfile for cloud builds. Defaults to `Dockerfile` in the build
context directory.
## Examples
**Deploy a new agent:**
```shell theme={null}
pipecat cloud deploy my-first-agent your-docker-repository/my-first-agent:0.1
```
**Update an existing agent with a new image:**
```shell theme={null}
pipecat cloud deploy my-first-agent your-docker-repository/my-first-agent:0.2
```
**Deploy with a specific secret set:**
```shell theme={null}
pipecat cloud deploy my-first-agent your-docker-repository/my-first-agent:0.1 --secrets my-secret-set
```
**Deploy a private image using image pull credentials:**
```shell theme={null}
pipecat cloud deploy my-first-agent your-docker-repository/my-first-agent:0.1 --credentials dockerhub-creds
```
**Keep one instance always warm to avoid cold starts:**
```shell theme={null}
pipecat cloud deploy my-first-agent your-docker-repository/my-first-agent:0.1 --min-agents 1
```
**Limit the maximum number of agent instances:**
```shell theme={null}
pipecat cloud deploy my-first-agent your-docker-repository/my-first-agent:0.1 --max-agents 5
```
**Deploy to a specific region:**
```shell theme={null}
pipecat cloud deploy my-first-agent your-docker-repository/my-first-agent:0.1 --region eu-central
```
**Deploy with Krisp VIVA noise cancellation:**
```shell theme={null}
pipecat cloud deploy my-first-agent your-docker-repository/my-first-agent:0.1 --krisp-viva-audio-filter tel
```
**Cap session duration (useful as a safety net for short-interaction agents):**
```shell theme={null}
pipecat cloud deploy my-first-agent your-docker-repository/my-first-agent:0.1 --max-session-duration 300
```
**Deploy using cloud build (interactive):**
```shell theme={null}
pipecat cloud deploy
```
**Deploy using cloud build in CI/CD (no prompts):**
```shell theme={null}
pipecat cloud deploy --yes
```
**Deploy using an existing build:**
```shell theme={null}
pipecat cloud deploy --build-id abc12345-6789-0abc-def0-123456789abc
```
**Deploy using an alternate config file:**
```shell theme={null}
pipecat cloud deploy --config-file pcc-deploy.staging.toml
```
## Configuration File (pcc-deploy.toml)
The `deploy` command supports a configuration file for repeatable deployments. Create a `pcc-deploy.toml` file in your project root to define deployment settings that can be shared across your team and version controlled.
### File Location
Place `pcc-deploy.toml` in the same directory where you run the `pipecat cloud deploy` command. The CLI will automatically detect and use this file.
To use an alternate config file (e.g. for different environments), pass `--config-file`:
```shell theme={null}
pipecat cloud deploy --config-file pcc-deploy.staging.toml
```
You can also set the `PIPECAT_DEPLOY_CONFIG_PATH` environment variable, with the `--config-file` flag taking precedence.
### Precedence
Values are applied with the following order of precedence:
1. CLI arguments (highest priority)
2. `pcc-deploy.toml` values
3. Default values (lowest priority)
This allows you to define defaults in the config file while still overriding specific values via CLI flags when needed.
### Configuration Options
#### Required Fields
Name of the agent to deploy. Must start with a lowercase letter or number, can
include hyphens, and must end with a lowercase letter or number. Maximum 54 characters.
```toml theme={null}
agent_name = "my-voice-agent"
```
Docker image URL with tag. Required when not using cloud builds or `build_id`.
```toml theme={null}
image = "your-dockername/my-agent:0.1"
```
You must specify either `image` or `build_id`, or omit both to trigger a cloud
build at deploy time.
#### Optional Fields
An existing cloud build ID to deploy. Cannot be used together with `image`.
```toml theme={null}
build_id = "abc12345-6789-0abc-def0-123456789abc"
```
Region where the agent will be deployed. If not specified, uses your
organization's default region (typically `us-west`).
```toml theme={null}
region = "us-east"
```
Name of the secret set to use for environment variables. The secret set must
exist in the same region as the agent and must be in the
[`ready`](/pipecat-cloud/fundamentals/secrets#provisioning-and-readiness)
state — deploys that bind a `pending` or `failed` set are rejected.
```toml theme={null}
secret_set = "my-agent-secrets"
```
Name of the image pull secret for private registries. The image pull secret must exist in the same region as the agent and must be in the [`ready`](/pipecat-cloud/fundamentals/secrets#provisioning-and-readiness) state.
```toml theme={null}
image_credentials = "dockerhub-credentials"
```
Agent profile for resource allocation. Valid values: `agent-1x`, `agent-2x`, `agent-3x`.
```toml theme={null}
agent_profile = "agent-2x"
```
Maximum session duration in seconds. When a session reaches this limit, the
agent's connection is forcibly closed — the session is cut off mid-flight
with no notice to the bot. Valid range: `60` to `14400` (4 hours). See
[Session duration limits](/pipecat-cloud/fundamentals/active-sessions#session-duration-limits)
for more information.
```toml theme={null}
max_session_duration = 300
```
#### Scaling Configuration
Define auto-scaling behavior in a `[scaling]` section:
Minimum number of agent instances to keep warm. Setting to 0 allows scaling to zero but may result in cold starts.
```toml theme={null}
[scaling]
min_agents = 1
```
Maximum number of agent instances allowed.
```toml theme={null}
[scaling]
max_agents = 20
```
#### Krisp VIVA Configuration
Configure Krisp VIVA noise cancellation in a `[krisp_viva]` section:
Krisp VIVA audio filter model. Valid values: `tel` (telephony, up to 16kHz) or `pro` (WebRTC, up to 32kHz). Omit or set to `null` to disable.
```toml theme={null}
[krisp_viva]
audio_filter = "tel"
```
#### Cloud Build Configuration
Configure cloud build behavior in a `[build]` section:
Directory to use as the build context. Defaults to the current directory.
```toml theme={null}
[build]
context_dir = "."
```
Path to the Dockerfile within the build context.
```toml theme={null}
[build]
dockerfile = "Dockerfile"
```
Additional file patterns to exclude from the build context. Common patterns like `.git`, `.env`, `__pycache__`, and `.venv` are excluded automatically.
```toml theme={null}
[build.exclude]
patterns = ["*.md", "tests/", "docs/"]
```
### Complete Example
```toml theme={null}
# Basic agent configuration
agent_name = "my-voice-agent"
region = "us-west"
# Secrets
secret_set = "my-agent-secrets"
# Resource allocation
agent_profile = "agent-1x"
# Auto-scaling configuration
[scaling]
min_agents = 1
max_agents = 20
# Cloud build configuration (optional, shown with defaults)
[build]
context_dir = "."
dockerfile = "Dockerfile"
[build.exclude]
patterns = ["*.md", "tests/"]
```
```toml theme={null}
# Basic agent configuration
agent_name = "my-voice-agent"
image = "your-dockername/my-voice-agent:0.1"
region = "us-west"
# Secrets and credentials
secret_set = "my-agent-secrets"
image_credentials = "dockerhub-credentials"
# Resource allocation
agent_profile = "agent-1x"
# Auto-scaling configuration
[scaling]
min_agents = 1
max_agents = 20
# Krisp VIVA noise cancellation
[krisp_viva]
audio_filter = "tel"
```
### Using the Configuration File
Once you have a `pcc-deploy.toml` file, simply run:
```shell theme={null}
pipecat cloud deploy
```
The CLI will automatically load your configuration. You can still override any value using CLI flags:
```shell theme={null}
# Use config file but override the region
pipecat cloud deploy --region eu-central
# Use config file but force update without confirmation
pipecat cloud deploy --force
```
# docker
Source: https://docs.pipecat.ai/api-reference/cli/cloud/docker
Build and push Docker images for agent deployments
The `docker` command provides utilities for building, tagging, and pushing Docker images to container registries. This command automatically parses registry information from your deployment configuration and supports both Docker Hub and custom registries.
## build-push
Build, tag, and push a Docker image for your agent deployment. This command reads configuration from your `pcc-deploy.toml` file to automatically determine registry settings, image names, and versions.
**Usage:**
```shell theme={null}
pipecat cloud docker build-push [ARGS] [OPTIONS]
```
**Arguments:**
Name of the agent to build image for. If not provided, uses the `agent_name`
from your `pcc-deploy.toml` file.
**Options:**
Path to an alternate deploy config file. Defaults to `pcc-deploy.toml`.
**Registry Configuration:**
Registry type to push to. Supported values: `dockerhub`, `custom`. When not
specified, automatically detected from the `image` field in your
`pcc-deploy.toml` file.
Registry username for authentication. When not specified, automatically parsed
from the `image` field in your `pcc-deploy.toml` file (e.g., `myusername` from
`myusername/app:1.0`).
Custom registry URL (required for custom registries). When not specified,
automatically parsed from the `image` field for custom registries (e.g.,
`gcr.io` from `gcr.io/project/app:1.0`).
**Build Configuration:**
Version tag for the image. When not specified, automatically extracted from
the `image` field in your `pcc-deploy.toml` file (e.g., `1.0` from
`myusername/app:1.0`).
Build and tag only, do not push to registry. Useful for local testing or when
you want to push manually later.
Do not tag the image as `latest`. By default, images are tagged with both the
specified version and `latest`.
## Configuration
The `docker build-push` command reads configuration from your `pcc-deploy.toml` file to minimize required command-line arguments. Here's how different registry setups work:
### Docker Hub (Default)
For Docker Hub repositories, the minimal configuration is:
```toml theme={null}
agent_name = "my-agent"
image = "myusername/my-agent:1.0"
```
This automatically configures:
* Registry: `dockerhub`
* Username: `myusername`
* Agent name: `my-agent`
* Version: `1.0`
### Custom Registry
For custom registries like Google Container Registry, AWS ECR, or private registries:
```toml theme={null}
agent_name = "my-agent"
image = "gcr.io/my-project/my-agent:1.0"
```
This automatically configures:
* Registry: `custom`
* Registry URL: `gcr.io`
* Username/Project: `my-project`
* Agent name: `my-agent`
* Version: `1.0`
### Docker Configuration Section
For advanced configuration, add a `[docker]` section:
```toml theme={null}
agent_name = "my-agent"
image = "myusername/my-agent:1.0"
[docker]
auto_latest = false # Don't tag as 'latest'
```
Available `[docker]` options:
Whether to automatically tag the image as `latest` in addition to the
specified version.
## Examples
### Basic Usage (Recommended)
With a properly configured `pcc-deploy.toml`:
```shell theme={null}
# Build and push using all configuration from pcc-deploy.toml
pipecat cloud docker build-push
```
### Override Version
```shell theme={null}
# Use a different version than what's in pcc-deploy.toml
pipecat cloud docker build-push --version 2.0
```
### Build Only
```shell theme={null}
# Build and tag locally without pushing
pipecat cloud docker build-push --no-push
```
### Different Registry
```shell theme={null}
# Override registry settings for one-time builds
pipecat cloud docker build-push --registry custom --registry-url my-registry.com --username myuser
```
### Skip Latest Tag
```shell theme={null}
# Only tag with the specific version, not 'latest'
pipecat cloud docker build-push --no-latest
```
## Platform Support
All images are built for the `linux/arm64` platform, which is required for Pipecat Cloud deployments. This is automatically configured and cannot be changed.
## Error Handling
The command provides helpful error messages for common issues:
* **Authentication errors**: Suggests the appropriate `docker login` command
* **Missing Dockerfile**: Indicates that a Dockerfile must be present in the current directory
* **Registry access issues**: Provides guidance on checking permissions and authentication
# organizations
Source: https://docs.pipecat.ai/api-reference/cli/cloud/organizations
Organization and API key management commands
The `organizations` command group helps you manage your Pipecat Cloud organizations and API keys. You can list and select organizations, as well as create, list, and manage API keys for use with the platform.
Organization and user management is not available via the CLI. Please use the
[Pipecat Cloud Dashboard](https://pipecat.daily.co) to manage organizations
and users.
## list
Lists all organizations that your account has access to, highlighting the currently active one used for CLI operations.
**Usage:**
```shell theme={null}
pipecat cloud organizations list
```
## select
Changes your active organization for CLI operations.
This command either presents an interactive selection menu or directly sets a specified organization as your default. The selection is stored in your local configuration file (defaults to `~/.config/pipecatcloud/pipecatcloud.toml`) and used for all subsequent CLI commands.
**Usage:**
```shell theme={null}
pipecat cloud organizations select [OPTIONS]
```
**Options:**
Bypass prompt by directly specifying a namespace / organization string.
## keys
The `keys` sub-commands manage API keys for authenticating with Pipecat Cloud services.
### keys list
Lists all API keys for the current organization.
**Usage:**
```shell theme={null}
pipecat cloud organizations keys list [OPTIONS]
```
**Options:**
Organization ID to list keys for. If not provided, the default organization
will be used.
### keys create
Create a new public API key for account / organization. Command will prompt the user to enter
a human-readable name for the key.
**Usage:**
```shell theme={null}
pipecat cloud organizations keys create [OPTIONS]
```
**Options:**
Organization ID to create key for. If not provided, the default organization
will be used.
### keys delete
Delete an API key from your organization. Command will prompt the user to select which key they wish to delete.
**Usage:**
```shell theme={null}
pipecat cloud organizations keys delete [OPTIONS]
```
**Options:**
Organization ID to delete key for. If not provided, the default organization
will be used.
### keys use
Sets a specific API key as your default for CLI operations.
The selected key is stored in the local configuration file (defaults to `~/.config/pipecatcloud/pipecatcloud.toml`) and will be used for all subsequent requests to the Pipecat Cloud API.
Please note that the key must be associated with the same user account or organization as you are making requests to.
If the public key is revoked or deleted via the dashboard, the user will need to re-run this command to select a new key.
**Usage:**
```shell theme={null}
pipecat cloud organizations keys use [OPTIONS]
```
**Options:**
Organization ID to select default key from. If not provided, the default
organization will be used.
## properties
The `properties` sub-commands manage organization properties such as default region settings.
### properties list
Lists all current property values for your organization.
**Usage:**
```shell theme={null}
pipecat cloud organizations properties list [OPTIONS]
```
**Options:**
Organization ID to list properties for. If not provided, the default
organization will be used.
### properties schema
Shows available properties with detailed metadata including type information, current values, default values, and available values.
**Usage:**
```shell theme={null}
pipecat cloud organizations properties schema [OPTIONS]
```
**Options:**
Organization ID to show properties schema for. If not provided, the default
organization will be used.
### properties set
Updates a specific organization property.
**Usage:**
```shell theme={null}
pipecat cloud organizations properties set PROPERTY_NAME VALUE [OPTIONS]
```
**Arguments:**
Name of the property to set (e.g., `defaultRegion`)
Value to set for the property
**Options:**
Organization ID to update property for. If not provided, the default
organization will be used.
**Example:**
```shell theme={null}
pipecat cloud organizations properties set defaultRegion eu-central
```
## default-region
Convenience command to get or set the default region for your organization.
**Usage:**
```shell theme={null}
pipecat cloud organizations default-region [REGION] [OPTIONS]
```
**Arguments:**
Region to set as default. If omitted, displays the current default region and
available regions.
**Options:**
Organization ID to configure. If not provided, the default organization will
be used.
**Examples:**
View current default region:
```shell theme={null}
pipecat cloud organizations default-region
```
Set default region:
```shell theme={null}
pipecat cloud organizations default-region eu-central
```
***
Managing your account and collaborating on agents as part of a team
# regions
Source: https://docs.pipecat.ai/api-reference/cli/cloud/regions
View available deployment regions
The `regions` command helps you discover which regions are available for deploying agents and storing secrets in Pipecat Cloud.
## list
List all available regions with their codes and display names.
**Usage:**
```shell theme={null}
pipecat cloud regions list
```
This command displays a table of all regions where you can deploy agents and store secrets. Use the region codes shown in this list when specifying the `--region` flag in other commands.
**Example output:**
```
Code Name
us-west US West (Oregon)
us-east US East (Virginia)
eu-central Europe (Frankfurt)
ap-south Asia Pacific (Mumbai)
```
## Using regions
Once you know the available region codes, you can use them with other commands:
**Deploy an agent to a specific region:**
```shell theme={null}
pipecat cloud deploy my-agent my-image:latest --region eu-central
```
**Create secrets in a specific region:**
```shell theme={null}
pipecat cloud secrets set my-secrets API_KEY=abc123 --region eu-central
```
**List agents in a specific region:**
```shell theme={null}
pipecat cloud agent list --region us-east
```
Secrets and image pull secrets must be in the same region as the agents that
use them. When deploying multi-region applications, create separate secret
sets for each region.
Choose a region close to your users for optimal latency and performance.
# secrets
Source: https://docs.pipecat.ai/api-reference/cli/cloud/secrets
Secret sets and secret management commands
The `secrets` command group helps you manage sensitive information for your agent deployments. You can create and manage secret sets (key-value pairs) and image pull secrets (for private Docker registries).
## list
List secret sets and image pull secrets for active namespace / organization.
Each row includes a
[readiness status](/pipecat-cloud/fundamentals/secrets#provisioning-and-readiness)
(`ready`, `pending`, or `failed`).
If provided with a valid secret set name, will show the keys of that set
alongside its status (values are hidden).
**Usage:**
```shell theme={null}
pipecat cloud secrets list [ARGS] [OPTIONS]
```
**Arguments:**
Name of the secret set list keys for. Must be a valid string identifier.
**Options:**
Show secret sets only. Filter out image pull secrets from the results.
Organization to list secrets for. If not provided, uses the current
organization from your configuration.
Filter secrets by region. Only secrets in the specified region will be shown.
If not provided, secrets from all regions are listed.
## set
Create or update a secret set with the given name and key-value pairs. Secrets can be passed directly as key value pairs or loaded from a file.
**Usage:**
```shell theme={null}
pipecat cloud secrets set [ARGS] [OPTIONS]
```
**Arguments:**
Name of the secret set to create or modify. Must be a valid string identifier
containing only characters, numbers, and hyphens.
List of secret key-value pairs e.g. `KEY1=value1 KEY2="value with spaces"`.
See [this note](/pipecat-cloud/fundamentals/secrets#special-characters) on using special
characters in secret values.
Example:
```shell theme={null}
pipecat cloud secrets set my-secrets 'API_KEY=123 API_KEY_2="value with spaces"'
```
**Options:**
Relative path to a file with a list of secret key-value pairs. Each line in
the file should be in the format `KEY=value`.
Example:
```shell theme={null}
pipecat cloud secrets set my-secrets --file .env
```
Skip confirmations and proceed with the operation.
Organization to create/update the secret set in. If not provided, uses the
current organization from your configuration.
Region where the secret set will be stored. If not specified, uses your
organization's default region (typically `us-west`). Secrets must be in the
same region as the agents that use them.
**Example:**
Create a secret set in a specific region:
```shell theme={null}
pipecat cloud secrets set my-secrets API_KEY=abc123 --region eu-central
```
## unset
Removes a specific secret key from a secret set.
**Usage:**
```shell theme={null}
pipecat cloud secrets unset [ARGS] [OPTIONS]
```
**Arguments:**
Name of the secret set to remove the secret from.
The key of the secret to remove from the set.
Example:
```shell theme={null}
pipecat cloud secrets unset my-secret-set SOME_KEY
```
**Options:**
Skip confirmations and proceed with the operation.
Organization containing the secret set. If not provided, uses the current
organization from your configuration.
## delete
Deletes an entire secret set.
**Usage:**
```shell theme={null}
pipecat cloud secrets delete [ARGS] [OPTIONS]
```
**Arguments:**
Name of the secret set to delete. This action is irreversible.
**Options:**
Skip confirmations and proceed with the operation.
Organization containing the secret set. If not provided, uses the current
organization from your configuration.
## image-pull-secret
Creates or updates credentials for pulling images from private Docker registries.
This command accepts image repository credentials as `username:password`,
base64-encodes them, and securely stores them. These credentials are used with
the [deploy](./deploy) command when pulling images from private repositories. If
a secret with the same name already exists, its credentials will be updated in
place. If you don't provide credentials directly, the command will prompt you for
input.
**Usage:**
```shell theme={null}
pipecat cloud secrets image-pull-secret [ARGS] [OPTIONS]
```
**Arguments:**
Name of the credentials set to create or modify. Must be a valid string
identifier.
Host address of the image repository e.g. `https://index.docker.io/v1/`.
Credentials for the image repository in the form of `username:password`.
Will prompt you for the value if not provided.
Example:
```shell theme={null}
pipecat cloud secrets image-pull-secret my-registry-creds https://index.docker.io/v1/ my-username:my-password
```
**Options:**
Encode the credentials in base64 format. This is enabled by default.
Organization to create the image pull secret in. If not provided, uses the
current organization from your configuration.
Region where the image pull secret will be stored. If not specified, uses your
organization's default region (typically `us-west`). Image pull secrets must
be in the same region as the agents that use them.
**Examples:**
Create an image pull secret in a specific region:
```shell theme={null}
pipecat cloud secrets image-pull-secret my-registry-creds https://index.docker.io/v1/ --region eu-central
```
***
Learn more about managing application secrets
# eval
Source: https://docs.pipecat.ai/api-reference/cli/eval
Run behavioral evals against a Pipecat agent, individually or as a suite
Run scenario-based behavioral evals. `pipecat eval run` tests scenarios against an already-running agent; `pipecat eval suite` spawns the agents listed in a manifest and runs their scenarios concurrently. Both exit `0` when everything passes and `1` otherwise.
If `pipecat-ai[cli]` is a dependency of your project, run these commands with `uv run pipecat eval`. They're also available as `python -m pipecat.evals`.
See the [Pipecat Evals guide](/pipecat/evals/overview) for concepts, the scenario format, and manifests.
## eval run
Run one or more scenarios against an already-running agent (started with `-t eval`).
**Usage:**
```shell theme={null}
pipecat eval run [OPTIONS] SCENARIOS...
```
**Arguments:**
One or more scenario YAML files.
**Options:**
WebSocket URL of the agent's eval transport.
Print a line for each turn and expectation as it resolves.
Record each scenario's conversation audio (audio-mode scenarios).
Directory for `--audio` recordings: `/.wav`.
Directory for cached synthesized user audio. Defaults to
`/pipecat/tts`.
Disable the user-audio cache: re-synthesize every turn (no reads or writes).
Default per-expectation timeout in seconds, for expectations without their own
`within_ms`.
Directory for each scenario's logs: `/.eval.log` (plus
`.debug.log` under `--debug`).
Also save `.debug.log` with the harness's full per-pipeline logs.
Cancel the agent's pipeline (exit it) after the run. By default the agent is
left running so it can serve more scenarios.
Fire the bot's `on_client_disconnect` callback when the eval client
disconnects. Bots often cancel their pipeline there, so it's off by default. A
scenario's `trigger_disconnect:` field opts in on its own.
## eval suite
Spawn the agents in a manifest and run their scenarios concurrently. Everything except the `suite:` list can be set in the manifest or overridden on the command line (the command line wins).
**Usage:**
```shell theme={null}
pipecat eval suite [OPTIONS] MANIFEST_PATH
```
**Arguments:**
Manifest YAML listing agents and their scenarios.
**Options:**
Only run bots whose path contains this substring.
Only run this scenario name.
Run subdirectory name under `runs_dir`. Defaults to a timestamp.
Output base, overriding the manifest's `runs_dir`. A `/` subdirectory
with `logs/` and `recordings/` is created under it. Defaults to `eval-runs`.
Override the manifest's `bots_dir` (bot paths are relative to it).
Override the manifest's `scenarios_dir`.
Override the manifest's `concurrency` (how many runs execute at once).
Override the manifest's `base_port` (default `7900`). Each run gets \`base\_port
* index\`.
Override the manifest's `cache_dir` for cached synthesized user audio.
Disable the user-audio cache: re-synthesize every turn (no reads or writes).
Default per-expectation timeout in seconds, for expectations without their own
`within_ms`.
Override the manifest's spawn template. Default: `"{python} {bot} -t eval --port {port}"`.
Override the Python interpreter used to spawn each agent.
Record conversation audio.
Also save `.debug.log` with the harness's full per-pipeline logs.
## Examples
```shell theme={null}
# Run one scenario against a running agent
pipecat eval run scenarios/capital_question.yaml
# Run a batch of scenarios, verbosely
pipecat eval run scenarios/*.yaml -v
# Run a full suite
pipecat eval suite manifest.yaml
# Only the support agent, 8 runs at a time, named output dir
pipecat eval suite manifest.yaml -p support -c 8 -n nightly
```
# init
Source: https://docs.pipecat.ai/api-reference/cli/init
Initialize and scaffold a new Pipecat project
`pipecat init` initializes a new Pipecat app. It's the single entry point for building with Pipecat.
It gives you a choice about how to scaffold your project:
* Using a coding agent like Claude Code or Codex
* Using an interactive wizard
No matter how you choose to scaffold, the coding-agent files `AGENTS.md` and `CLAUDE.md` are provided so your favorite coding agent works well with Pipecat.
**Usage:**
```shell theme={null}
pipecat init [TARGET_DIR] [OPTIONS]
```
**Arguments:**
Directory to initialize. `pipecat init my-bot` targets `./my-bot`; `pipecat
init .` targets the current directory. With no argument and no scaffold
options, the CLI prompts for a directory. When scaffold options are present,
the bot is generated **in-place** in this directory (the project name is
derived from it); a missing `TARGET_DIR` then defaults to the current
directory. The special value `quickstart` (`pipecat init quickstart`)
scaffolds the canned quickstart bot into `./pipecat-quickstart` (see
[Quickstart Preset](#quickstart-preset)).
**Guide options:**
Overwrite existing `AGENTS.md`, `CLAUDE.md`, and `GETTING_STARTED.md` files. By
default existing guide files are kept, so your edits are never clobbered.
**Scaffold options:** Passing any of these (or `--config`) scaffolds a bot non-interactively, in-place in `TARGET_DIR`.
Project name. Defaults to the target directory name; pass this to override it.
Bot type: `web` or `telephony`. Optional — inferred from `--transport` when
omitted (telephony if any transport is a telephony provider, otherwise web).
Transport provider. Repeatable for multiple transports (e.g. `-t daily -t
smallwebrtc`). Valid values: `daily`, `smallwebrtc`, `twilio`, `telnyx`,
`plivo`, `exotel`, `daily_pstn`, `twilio_daily_sip`.
Pipeline mode: `cascade` or `realtime`.
Speech-to-Text service (cascade mode). e.g. `deepgram_stt`, `openai_stt`.
Language model service (cascade mode). e.g. `openai_llm`, `anthropic_llm`.
Text-to-Speech service (cascade mode). e.g. `cartesia_tts`, `elevenlabs_tts`.
Realtime service (realtime mode). e.g. `openai_realtime`,
`gemini_live_realtime`.
Video avatar service (web bots only). e.g. `heygen_video`, `tavus_video`,
`simli_video`.
Client framework (web bots only): `react`, `vanilla`, or `none`.
Client dev server (when using `--client-framework react`): `vite` or `nextjs`.
Daily PSTN mode (required when transport is `daily_pstn`): `dial-in` or
`dial-out`.
Twilio + Daily SIP mode (required when transport is `twilio_daily_sip`):
`dial-in` or `dial-out`.
Enable audio recording.
Enable transcription logging.
Enable video input (web bots only).
Enable video output (web bots only).
Generate Pipecat Cloud deployment files (Dockerfile, pcc-deploy.toml).
Enable Krisp noise cancellation (requires cloud deployment).
Enable observability.
Make the generated bot eval-ready: add an `eval` transport entry and starter
scenarios in `server/evals/`, plus the dependencies to run them. See the
[evals docs](/api-reference/cli/eval) for the verification workflow.
Path to a JSON config file. Triggers non-interactive scaffolding. CLI flags
override file values.
Print the resolved scaffold configuration as JSON without writing any files.
Print all available service options as JSON and exit. Useful for CI scripts
and coding agents that need to discover valid values at runtime.
## Behavior
* **Existing guide files are kept.** Re-running `pipecat init` never overwrites an existing `AGENTS.md`, `CLAUDE.md`, or `GETTING_STARTED.md`, so your edits are safe. If a guide was written by an older Pipecat version, an interactive run offers to refresh it on the spot; a non-interactive run prints how.
* **`--overwrite-guide` refreshes them**, overwriting the existing guide files with the current templates (for example, after upgrading Pipecat).
* `GETTING_STARTED.md` is written on the coding-agent path only, not when scaffolding a bot.
## Scaffolding a Bot
`pipecat init` scaffolds a complete project — `bot.py`, dependencies, config, and an optional client — in-place in the target directory, so the coding-agent guide and a runnable bot live together.
### Interactive
With no scaffold options, an interactive `pipecat init` asks whether to build with a coding agent or scaffold a bot now. Choosing to scaffold runs a wizard for bot type, transport, AI services, and deployment options.
### Non-Interactive
Passing any scaffold option (or `--config`) skips the prompts and builds the project from your flags and/or a config file — the path coding agents and automation use. The typical agent loop is:
```shell theme={null}
# 1. A developer initializes the project (writes the coding-agent guide):
pipecat init my-bot
# 2. They open ./my-bot in a coding agent and describe the bot they want.
# 3. The agent scaffolds it in place, mapping the requirements to flags:
pipecat init . --bot-type web --transport daily --mode cascade \
--stt deepgram_stt --llm openai_llm --tts cartesia_tts --eval
```
All required fields must be specified or the command exits with a list of all missing/invalid fields. Bot type is not required — it's inferred from `--transport` when omitted. Run `pipecat init --list-options` to discover valid service and transport values.
## Quickstart Preset
```shell theme={null}
pipecat init quickstart
```
Scaffolds the canned quickstart bot (SmallWebRTC + Daily, Deepgram, OpenAI, Cartesia) into `./pipecat-quickstart` **and** writes `AGENTS.md` + `CLAUDE.md` there (the generated README carries the Context Hub setup). It's the fastest way to a running project that your coding agent can work with.
## Examples
### Initialize for a coding agent
```shell theme={null}
pipecat init my-bot
```
### Scaffold in place, non-interactively
```shell theme={null}
pipecat init . --bot-type web --transport daily --mode cascade \
--stt deepgram_stt --llm openai_llm --tts cartesia_tts
```
Scaffolds into the current directory, deriving the project name from it. Bot type is inferred from the transport (`web` here), so `--bot-type` is optional.
### Realtime bot
```shell theme={null}
pipecat init rt-bot --bot-type web --transport smallwebrtc \
--mode realtime --realtime openai_realtime
```
### Multiple transports
```shell theme={null}
pipecat init my-bot --bot-type web \
--transport daily --transport smallwebrtc \
--mode cascade --stt deepgram_stt --llm openai_llm --tts cartesia_tts
```
### With a React client
```shell theme={null}
pipecat init my-bot --bot-type web --transport daily \
--mode cascade --stt deepgram_stt --llm openai_llm --tts cartesia_tts \
--client-framework react --client-server vite
```
### Telephony
```shell theme={null}
pipecat init call-bot --bot-type telephony --transport twilio \
--mode cascade --stt deepgram_stt --llm openai_llm --tts cartesia_tts
```
### Eval-ready bot
```shell theme={null}
pipecat init my-bot --bot-type web --transport daily \
--mode cascade --stt deepgram_stt --llm openai_llm --tts cartesia_tts \
--eval
```
### Refresh the guide files
```shell theme={null}
pipecat init my-bot --overwrite-guide
```
### Scaffold the quickstart project
```shell theme={null}
pipecat init quickstart
```
### Discover available options
```shell theme={null}
pipecat init --list-options
```
Output:
```json theme={null}
{
"bot_type": ["web", "telephony"],
"transports": {
"web": ["daily", "smallwebrtc"],
"telephony": ["twilio", "twilio_daily_sip_dialin", "twilio_daily_sip_dialout", ...]
},
"stt": ["deepgram_stt", "mistral_stt", "openai_stt", "xai_stt", ...],
"llm": ["openai_llm", "anthropic_llm", ...],
"tts": ["cartesia_tts", "elevenlabs_tts", "soniox_tts", ...],
"realtime": ["openai_realtime", "gemini_live_realtime", ...],
"video": ["heygen_video", "tavus_video", "simli_video"]
}
```
This is useful for scripting — for example, to pick a TTS provider:
```shell theme={null}
options=$(pipecat init --list-options)
tts=$(echo "$options" | jq -r '.tts[0]')
```
### Dry run
```shell theme={null}
pipecat init . --bot-type web --transport daily \
--mode cascade --stt deepgram_stt --llm openai_llm --tts cartesia_tts \
--dry-run
```
Prints the resolved configuration as JSON and writes nothing.
### From a config file
```shell theme={null}
pipecat init my-bot --config project-config.json
```
Sample `project-config.json`:
```json theme={null}
{
"project_name": "my-bot",
"bot_type": "web",
"transports": ["daily"],
"mode": "cascade",
"stt_service": "deepgram_stt",
"llm_service": "openai_llm",
"tts_service": "cartesia_tts",
"recording": false,
"transcription": false,
"deploy_to_cloud": true,
"enable_krisp": false,
"enable_observability": false
}
```
CLI flags override any values in the file, so you can use a base config and customize per-run:
```shell theme={null}
pipecat init my-bot --config base-config.json --name custom-name --no-deploy-to-cloud
```
## Generated Project Structure
```
my-bot/
├── AGENTS.md # Coding-agent guide
├── CLAUDE.md # @AGENTS.md import for Claude Code
├── server/ # Python bot server
│ ├── bot.py # Main bot implementation
│ ├── evals/ # Starter eval scenarios (with --eval)
│ ├── pyproject.toml # Python dependencies
│ ├── .env.example # Environment variables template
│ ├── Dockerfile # Container image (if cloud enabled)
│ └── pcc-deploy.toml # Deployment config (if cloud enabled)
├── client/ # Web client (if generated)
│ ├── src/
│ ├── package.json
│ └── ...
├── .gitignore
└── README.md # Project setup instructions
```
The generated `README.md` includes a **"Building with an AI coding agent"** section with [Pipecat Context Hub](/api-reference/context-hub) setup, so a scaffolded project is ready to extend with Claude Code or Codex.
## Next Steps
The full flow: initialize a project, then build with a coding agent or
scaffold a bot
# CLI Overview
Source: https://docs.pipecat.ai/api-reference/cli/overview
Command-line tool for scaffolding and deploying Pipecat bots
Initialize a project, then build with a coding agent or scaffold a bot
Test your agents with scripted scenarios and an LLM judge
Push your bots to production with one command
## Requirements
* Python 3.11 or later
## Installation
The CLI ships with `pipecat-ai` as the optional `cli` extra. Install it globally with [uv](https://docs.astral.sh/uv/):
```bash theme={null}
uv tool install "pipecat-ai[cli]"
```
Verify installation:
```bash theme={null}
pipecat --version
```
All commands can use either `pipecat` or the shorter `pc` alias.
This gives you the built-in `pipecat init` and `pipecat eval` commands. The `cloud` command is provided by a separate package that you co-install with `--with`:
```bash theme={null}
# Add deploy commands
uv tool install "pipecat-ai[cli]" --with pipecatcloud
```
## Commands
* **[`pipecat init`](/api-reference/cli/init)** - Initialize and scaffold a new Pipecat project, the single entry point for new projects (built in)
* **[`pipecat eval`](/api-reference/cli/eval)** - Run behavioral evals against your agents
* **[`pipecat cloud`](/api-reference/cli/cloud/auth)** - Deploy and manage bots on Pipecat Cloud (requires `pipecatcloud`)
## Getting Help
View help for any command:
```bash theme={null}
pipecat --help
pipecat init --help
pipecat eval --help
pipecat cloud --help
```
## Next Steps
Initialize your project and start building with pipecat init
# API Reference
Source: https://docs.pipecat.ai/api-reference/client/android/api-reference
# Android SDK Overview
Source: https://docs.pipecat.ai/api-reference/client/android/overview
Build Android applications with Pipecat's Kotlin client library
The Pipecat Android SDK provides a Kotlin implementation for building voice and multimodal AI applications on Android. It handles:
* Real-time audio and video streaming
* Bot communication and state management
* Media device handling
* Event handling
## Installation
Add the dependency for your chosen transport to your `build.gradle` file. For example, to use the Daily transport:
```gradle theme={null}
implementation "ai.pipecat:daily-transport:1.0.3"
```
## Example
Here's a simple example using Daily as the transport layer.
```kotlin theme={null}
val callbacks = object : PipecatEventCallbacks() {
override fun onBackendError(message: String) {
Log.e(TAG, "Error from backend: $message")
}
// ...
}
val options = PipecatClientOptions(callbacks = callbacks)
val client: PipecatClientDaily = PipecatClient(DailyTransport(context), options)
// Kotlin coroutines:
client.startBotAndConnect(startBotParams).await()
// Or, callbacks:
client.startBotAndConnect(startBotParams).withCallback {
// ...
}
```
## Documentation
SDK API documentation
Daily, Gemini, OpenAI WebRTC, and SmallWebRTC transports
# Daily WebRTC Transport
Source: https://docs.pipecat.ai/api-reference/client/android/transports/daily
WebRTC implementation for Android using Daily
The Daily transport implementation enables real-time audio and video communication in your Pipecat Android applications using [Daily's](https://www.daily.co/) WebRTC infrastructure.
## Installation
Add the Daily transport dependency to your `build.gradle`:
```gradle theme={null}
implementation "ai.pipecat:daily-transport:1.0.3"
```
## Usage
Create a client using the Daily transport:
```kotlin theme={null}
val callbacks = object : PipecatEventCallbacks() {
override fun onBackendError(message: String) {
Log.e(TAG, "Error from backend: $message")
}
// ...
}
val options = PipecatClientOptions(callbacks = callbacks)
val client: PipecatClientDaily = PipecatClient(DailyTransport(context), options)
// Kotlin coroutines
client.startBotAndConnect(startBotParams).await()
// Callbacks
client.startBotAndConnect(startBotParams).withCallback {
// ...
}
```
## Configuration
Your server endpoint should return Daily-specific configuration:
```json theme={null}
{
"dailyRoom": "https://your-domain.daily.co/room-name",
"dailyToken": "your-daily-token"
}
```
## Resources
Simple Chatbot Demo
Client Transports
Complete API documentation for the Daily transport implementation
# Gemini Live Websocket Transport
Source: https://docs.pipecat.ai/api-reference/client/android/transports/gemini-websocket
Websocket implementation for Android using Gemini
The Gemini Live Websocket transport implementation enables real-time audio communication with the Gemini Live service, using a direct websocket connection.
Transports of this type are designed primarily for development and testing
purposes. For production applications, you will need to build a server
component with a server-friendly transport, like the
[DailyTransport](./daily), to securely handle API keys.
## Installation
Add the transport dependency to your `build.gradle`:
```gradle theme={null}
implementation "ai.pipecat:gemini-live-websocket-transport:0.3.7"
```
## Usage
Create a client:
```kotlin theme={null}
val transport = GeminiLiveWebsocketTransport.Factory(context)
val options = RTVIClientOptions(
params = RTVIClientParams(
baseUrl = null,
config = GeminiLiveWebsocketTransport.buildConfig(
apiKey = "",
generationConfig = Value.Object(
"speech_config" to Value.Object(
"voice_config" to Value.Object(
"prebuilt_voice_config" to Value.Object(
"voice_name" to Value.Str("Puck")
)
)
)
),
initialUserMessage = "How tall is the Eiffel Tower?"
)
)
)
val client = RTVIClient(transport, callbacks, options)
client.start().withCallback {
// ...
}
```
## Resources
Simple Chatbot Demo
Client Transports
Complete API documentation for the Pipecat Android client.
# OpenAI Realtime WebRTC Transport
Source: https://docs.pipecat.ai/api-reference/client/android/transports/openai-webrtc
WebRTC implementation for Android using OpenAI
The OpenAI Realtime WebRTC transport implementation enables real-time audio communication with the OpenAI Realtime service, using a direct WebRTC connection.
## Installation
Add the transport dependency to your `build.gradle`:
```gradle theme={null}
implementation "ai.pipecat:openai-realtime-webrtc-transport:0.3.7"
```
## Usage
Create a client:
```kotlin theme={null}
val transport = OpenAIRealtimeWebRTCTransport.Factory(context)
val options = RTVIClientOptions(
params = RTVIClientParams(
baseUrl = null,
config = OpenAIRealtimeWebRTCTransport.buildConfig(
apiKey = apiKey,
initialMessages = listOf(
LLMContextMessage(role = "user", content = "How tall is the Eiffel Tower?")
),
initialConfig = OpenAIRealtimeSessionConfig(
voice = "ballad",
turnDetection = Value.Object("type" to Value.Str("semantic_vad")),
inputAudioNoiseReduction = Value.Object("type" to Value.Str("near_field")),
inputAudioTranscription = Value.Object("model" to Value.Str("gpt-4o-transcribe"))
)
)
)
)
val client = RTVIClient(transport, callbacks, options)
client.start().withCallback {
// ...
}
```
## Resources
Simple Chatbot Demo
Client Transports
Complete API documentation for the Pipecat Android client.
# Small WebRTC Transport
Source: https://docs.pipecat.ai/api-reference/client/android/transports/small-webrtc
WebRTC implementation for Android
The Small WebRTC transport implementation enables real-time audio communication with the Small WebRTC Pipecat transport, using a direct WebRTC connection.
## Installation
Add the transport dependency to your `build.gradle`:
```gradle theme={null}
implementation "ai.pipecat:small-webrtc-transport:0.3.7"
```
## Usage
Create a client:
```kotlin theme={null}
val transport = SmallWebRTCTransport.Factory(context, baseUrl)
val options = RTVIClientOptions(
params = RTVIClientParams(baseUrl = null),
enableMic = true,
enableCam = true
)
val client = RTVIClient(transport, callbacks, options)
client.start().withCallback {
// ...
}
```
## Resources
Demo App
Client Transports
Complete API documentation for the Pipecat Android client.
# C++ SDK Overview
Source: https://docs.pipecat.ai/api-reference/client/cpp/overview
Build native applications with Pipecat’s C++ client library
The Pipecat C++ SDK provides a native implementation for building voice and multimodal AI applications. It supports:
* Linux (`x86_64` and `aarch64`)
* macOS (`aarch64`)
* Windows (`x86_64`)
## Dependencies
### libcurl
The SDK uses [libcurl](https://curl.se/libcurl/) for HTTP requests.
```bash theme={null}
sudo apt-get install libcurl4-openssl-dev
```
On macOS `libcurl` is already included so there is nothing to install.
On Windows we use [vcpkg](https://vcpkg.io/en/) to install dependencies. You
need to set it up following one of the
[tutorials](https://learn.microsoft.com/en-us/vcpkg/get_started/get-started).
The `libcurl` dependency will be automatically downloaded when building.
## Installation
Clone the SDK:
```bash theme={null}
git clone https://github.com/pipecat-ai/pipecat-client-cxx
cd pipecat-client-cxx
```
Build the SDK using CMake:
```bash theme={null}
cmake . -G Ninja -Bbuild -DCMAKE_BUILD_TYPE=Release
ninja -C build
```
```bash theme={null}
# Initialize Visual Studio environment
"C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Auxiliary\Build\vcvarsall.bat" amd64
# Configure and build
cmake . -Bbuild --preset vcpkg
cmake --buildbuild --config Release
```
### Cross-compilation
For Linux aarch64:
```bash theme={null}
cmake . -G Ninja -Bbuild -DCMAKE_TOOLCHAIN_FILE=aarch64-linux-toolchain.cmake -DCMAKE_BUILD_TYPE=Release
ninja -C build
```
## Documentation
Complete SDK API documentation
WebRTC implementation using Daily
# Daily WebRTC Transport
Source: https://docs.pipecat.ai/api-reference/client/cpp/transport
WebRTC implementation for C++ using Daily
The Daily transport implementation enables real-time audio and video communication in your Pipecat C++ applications using [Daily's](https://www.daily.co/) WebRTC infrastructure.
## Dependencies
### Daily Core C++ SDK
Download the [Daily Core C++ SDK](https://github.com/daily-co/daily-core-sdk) from the [available releases](https://github.com/daily-co/daily-core-sdk/releases) for your platform and set:
```bash theme={null}
export DAILY_CORE_PATH=/path/to/daily-core-sdk
```
### Pipecat C++ SDK
Build the base [Pipecat C++ SDK](https://github.com/pipecat-ai/pipecat-client-cxx-daily) first and set:
```bash theme={null}
export PIPECAT_SDK_PATH=/path/to/pipecat-client-cxx
```
## Building
First, set a few environment variables:
```bash theme={null}
PIPECAT_SDK_PATH=/path/to/pipecat-client-cxx
DAILY_CORE_PATH=/path/to/daily-core-sdk
```
Then, build the project:
```bash theme={null}
cmake . -G Ninja -Bbuild -DCMAKE_BUILD_TYPE=Release
ninja -C build
```
```bash theme={null}
# Initialize Visual Studio environment
"C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Auxiliary\Build\vcvarsall.bat" amd64
# Configure and build
cmake . -Bbuild --preset vcpkg
cmake --build build --config Release
```
## Examples
Simple C++ implementation example
C++ client with PortAudio support
Example Node.js proxy implementation
# API Reference
Source: https://docs.pipecat.ai/api-reference/client/ios/api-reference
# iOS SDK Overview
Source: https://docs.pipecat.ai/api-reference/client/ios/overview
Build iOS applications with Pipecat’s Swift client library
The Pipecat iOS SDK provides a Swift implementation for building voice and multimodal AI applications on iOS. It handles:
* Real-time audio streaming
* Bot communication and state management
* Media device handling
* Configuration management
* Event handling
## Installation
Add the SDK to your project using Swift Package Manager:
```swift theme={null}
// Core SDK
.package(url: "https://github.com/pipecat-ai/pipecat-client-ios.git", from: "1.0.0"),
// Daily transport implementation
.package(url: "https://github.com/pipecat-ai/pipecat-client-ios-daily.git", from: "1.0.0"),
```
Then add the dependencies to your target:
```swift theme={null}
.target(name: "YourApp", dependencies: [
.product(name: "PipecatClientIOS", package: "pipecat-client-ios")
.product(name: "PipecatClientIOSDaily", package: "pipecat-client-ios-daily")
]),
```
## Example
Here's a simple example using Daily as the transport layer:
```swift theme={null}
import PipecatClientIOS
import PipecatClientIOSDaily
let pipecatClientOptions = PipecatClientOptions.init(
transport: DailyTransport.init(),
enableMic: currentSettings.enableMic,
enableCam: false,
)
self.pipecatClientIOS = PipecatClient.init(
options: pipecatClientOptions
)
let startBotParams = APIRequest.init(endpoint: URL(string: $PIPECAT_API_URL + "/connect")!)
self.pipecatClientIOS?.startBotAndConnect(startBotParams: startBotParams) { (result: Result) in
switch result {
case .failure(let error):
// handle error
case .success(_):
// handle success
}
}
```
## Documentation
SDK API documentation
Daily, Gemini, OpenAI WebRTC, and SmallWebRTC transports
Pipecat Client iOS on GitHub
Simple Chatbot Demo
# Daily WebRTC Transport
Source: https://docs.pipecat.ai/api-reference/client/ios/transports/daily
WebRTC implementation for iOS using Daily
The Daily transport implementation enables real-time audio and video communication in your Pipecat iOS applications using [Daily's](https://www.daily.co/) WebRTC infrastructure.
## Installation
Add the Daily transport package to your project:
```swift theme={null}
.package(url: "https://github.com/pipecat-ai/pipecat-client-ios-daily.git", from: "1.0.0")
// Add to your target dependencies
.target(name: "YourApp", dependencies: [
.product(name: "PipecatClientIOSDaily", package: "pipecat-client-ios-daily")
])
```
## Usage
Create a client using the Daily transport:
```swift theme={null}
import PipecatClientIOS
import PipecatClientIOSDaily
let pipecatClientOptions = PipecatClientOptions.init(
transport: DailyTransport.init(),
enableMic: currentSettings.enableMic,
enableCam: false,
)
self.pipecatClientIOS = PipecatClient.init(
options: pipecatClientOptions
)
let startBotParams = APIRequest.init(endpoint: URL(string: $PIPECAT_API_URL + "/connect")!)
self.pipecatClientIOS?.startBotAndConnect(startBotParams: startBotParams) { (result: Result) in
switch result {
case .failure(let error):
// handle error
case .success(_):
// handle success
}
}
```
## Configuration
Your server endpoint should return Daily-specific configuration:
```swift theme={null}
// Example server response
{
"url": "https://your-domain.daily.co/room-name",
"token": "your-daily-token"
}
```
## API Reference
Simple Chatbot Demo
Daily Transport
Complete API documentation for the Daily transport implementation
# Gemini Live Websocket Transport
Source: https://docs.pipecat.ai/api-reference/client/ios/transports/gemini-websocket
Websocket implementation for iOS using Gemini
The Gemini Live Websocket transport implementation enables real-time audio communication with the Gemini Live service, using a direct websocket connection.
Transports of this type are designed primarily for development and testing
purposes. For production applications, you will need to build a server
component with a server-friendly transport, like the
[DailyTransport](./daily), to securely handle API keys.
## Installation
Add the Gemini transport package to your project:
```swift theme={null}
.package(url: "https://github.com/pipecat-ai/pipecat-client-ios-gemini-live-websocket.git", from: "0.3.1"),
// Add to your target dependencies
.target(name: "YourApp", dependencies: [
.product(name: "PipecatClientIOSGeminiLiveWebSocket", package: "pipecat-client-ios-gemini-live-websocket")
],
```
## Usage
Create a client:
```swift theme={null}
let options: RTVIClientOptions = .init(
params: .init(config: [
.init(
service: "llm",
options: [
.init(name: "api_key", value: .string("")),
.init(name: "initial_messages", value: .array([
.object([
"role": .string("user"), // "user" | "system"
"content": .string("I need your help planning my next vacation.")
])
])),
.init(name: "generation_config", value: .object([
"speech_config": .object([
"voice_config": .object([
"prebuilt_voice_config": .object([
"voice_name": .string("Puck") // "Puck" | "Charon" | "Kore" | "Fenrir" | "Aoede"
])
])
])
]))
]
)
])
)
let client = GeminiLiveWebSocketVoiceClient(options: options)
try await client.start()
```
## API Reference
Simple Chatbot Gemini Demo
iOS Gemini Live WebSocket
Complete API documentation for the Gemini transport implementation
# OpenAIRealTimeWebRTCTransport
Source: https://docs.pipecat.ai/api-reference/client/ios/transports/openai-webrtc
## Overview
The OpenAI Realtime WebRTC transport implementation enables real-time audio communication directly with the [OpenAI Realtime API using WebRTC](https://platform.openai.com/docs/guides/realtime-webrtc) voice-to-voice service.
It handles media device management, audio/video streams, and state management for the connection.
## Installation
Add the OpenAI transport package to your project:
```swift theme={null}
.package(url: "https://github.com/pipecat-ai/pipecat-client-ios-openai-realtime.git", from: "0.0.1"),
// Add to your target dependencies
.target(name: "YourApp", dependencies: [
.product(name: "PipecatClientIOSOpenAIRealtimeWebrtc", package: "pipecat-client-ios-openai-realtime")
],
```
## Usage
Create a client:
```swift theme={null}
let rtviClientOptions = RTVIClientOptions.init(
enableMic: currentSettings.enableMic,
enableCam: false,
params: .init(config: [
.init(
service: "llm",
options: [
.init(name: "api_key", value: .string(openaiAPIKey)),
.init(name: "initial_messages", value: .array([
.object([
"role": .string("user"), // "user" | "system"
"content": .string("Start by introducing yourself.")
])
])),
.init(name: "session_config", value: .object([
"instructions": .string("You are Chatbot, a friendly and helpful assistant who provides useful information, including weather updates."),
"voice": .string("echo"),
"input_audio_noise_reduction": .object([
"type": .string("near_field")
]),
"turn_detection": .object([
"type": .string("semantic_vad")
])
])),
]
)
])
)
self.rtviClientIOS = RTVIClient.init(
transport: OpenAIRealtimeTransport.init(options: rtviClientOptions),
options: rtviClientOptions
)
try await rtviClientIOS.start()
```
Currently, invalid session configurations will result in the OpenAI connection
being failed.
## API Reference
Simple Chatbot OpenAI Demo
iOS OpenAI Realtime WebRTC
Complete API documentation for the OpenAI transport implementation
# SmallWebRTCTransport
Source: https://docs.pipecat.ai/api-reference/client/ios/transports/small-webrtc
A lightweight WebRTC transport for peer-to-peer connections with Pipecat for iOS
`SmallWebRTCTransport` enables peer-to-peer WebRTC connections between clients and your Pipecat application. It implements bidirectional audio and video streaming using WebRTC for real-time communication.
This transport is intended for lightweight implementations. It expects your Pipecat server to include the corresponding [`SmallWebRTCTransport` server-side](/api-reference/server/services/transport/small-webrtc) implementation.
## Installation
Add the `SmallWebRTCTransport` package to your project:
```swift theme={null}
.package(url: "https://github.com/pipecat-ai/pipecat-client-ios-small-webrtc.git", from: "0.0.1")
// Add to your target dependencies
.target(name: "YourApp", dependencies: [
.product(name: "PipecatClientIOSSmallWebrtc", package: "pipecat-client-ios-small-webrtc")
])
```
## Usage
Create a client using the `SmallWebRTCTransport`:
```swift theme={null}
import PipecatClientIOS
import PipecatClientIOSSmallWebrtc
let rtviClientOptions = RTVIClientOptions.init(
enableMic: currentSettings.enableMic,
enableCam: currentSettings.enableCam,
params: RTVIClientParams(
config: [
.init(
service: SmallWebRTCTransport.SERVICE_NAME,
options: [
.init(name: "server_url", value: .string($PIPECAT_SERVER_URL))
]
)
]
)
)
self.rtviClientIOS = RTVIClient.init(
transport: SmallWebRTCTransport.init(options: rtviClientOptions),
options: rtviClientOptions
)
self.rtviClientIOS?.start() { result in
switch result {
case .failure(let error):
// handle error
case .success(_):
// handle success
}
}
```
## API Reference
Video transform Demo
Small WebRTC transport
Complete API documentation for the Small WebRTC transport implementation
# Callbacks and events
Source: https://docs.pipecat.ai/api-reference/client/js/callbacks
The Pipecat JavaScript client listens for messages and events from the bot via the transport layer. This allows you to respond to changes in state, errors, and other events. The client implements the RTVI standard for these communications.
## Event Handling Options
You can handle events in two ways:
### 1. Callbacks
Define handlers in the client constructor:
```typescript theme={null}
const pcClient = new PipecatClient({
callbacks: {
onBotReady: () => console.log("Bot ready via callback"),
// ... other callbacks
},
});
```
### 2. Event Listeners
Add handlers using the event emitter pattern:
```typescript theme={null}
pcClient.on(RTVIEvent.BotReady, () => console.log("Bot ready via event"));
```
Events and callbacks provide the same functionality. Choose the pattern that
best fits your application's architecture.
## Callbacks
### State and connectivity
Local user successfully established a connection to the transport.
Local user disconnected from the transport, either intentionally by calling
`pcClient.disconnect()` or due to an error.
Provides a `TransportState` string representing the connectivity state of the
local client. See [transports](./transports/transport) for state explanation.
A call to [`startBot()`](./client-methods#startbot) (i.e. a pre-connection
REST endpoint) was successful and the bot should now be started or in the
process of starting. The callback receives any data returned from your
endpoint.
The bot has been instantiated, its pipeline is configured, and it is receiving
user media and interactions. This method is passed a `BotReadyData` object,
which contains the RTVI `version` number. Since the bot is remote and may be
using a different version of RTVI than the client, you can use the passed
`version` string to check for compatibility.
Bot connected to the transport and is configuring. Note: bot connectivity does
not infer that its pipeline is yet ready to run. Please use `onBotReady`
instead.
Bot disconnected from the transport. This may occur due to session expiry, a
pipeline error or for any reason the server deems the session over. By
default, the client will also disconnect when this fires. Set
`disconnectOnBotDisconnect: false` in the constructor to keep the client
connected.
A participant joined the session. Fires for all participants, including the
bot and the local user.
A participant left the session. Fires for all participants, including the bot
and the local user.
### Messages and errors
Receives custom messages sent from the server to the client. This provides a
generic channel for server-to-client communication. The data structure is
flexible and defined by the server implementation.
Response error when an action fails or an unknown message type is sent from
the client.
Error signalled by the bot. This could be due to a malformed config update or
an unknown action dispatch or the inability to complete a client request. The
message parameter is of type `error` and matches [the RTVI
standard](/client/rtvi-standard#error-%F0%9F%A4%96). Its `data` field includes
a `message` string that describes the error and a `fatal` boolean indicating
if the error is unrecoverable and resulted in a bot disconnection. If `fatal`
is true, the client will automatically disconnect.
### Media and devices
Lists available local media microphone devices. Triggered when a new device
becomes available, a device is removed, or in response to
`pcClient.initDevices()`.
Lists available local media camera devices. Triggered when a new device
becomes available, a device is removed, or in response to
`pcClient.initDevices()`.
Lists available local speaker devices. Triggered when a new device becomes
available, a device is removed, or in response to `pcClient.initDevices()`.
User selected a new microphone as their selected/active device.
User selected a new camera as their selected/active device.
User selected a new speaker as their selected/active device.
Error related to media devices, such as camera or microphone issues. This
could be due to permissions, device unavailability, or other related problems.
See the [DeviceError](./errors#deviceerror) section for more details about the
return type.
Media track from a local or remote participant/bot was started and playable.
Can be either an audio or video track.
Media track from a local or remote participant/bot was stopped and no longer
playable.
Media track from a local or remote participant's screenshare was started and
playable. Can be either an audio or video track.
Media track from a local or remote participant's screenshare was stopped and
no longer playable.
### Audio and Voice Activity
Local audio gain level (0 to 1).
Remote audio gain level (0 to 1). Note: if more than one participant is
connected to the transport, the `participant` property details the associated
peer/bot.
The bot started speaking/sending speech audio.
The bot stopped speaking/sending speech audio.
The local user started speaking. This method is more reliable than using audio
gain and is the result of the bot's VAD (voice activity detection) model. This
provides a more accurate result in noisy environments.
The local user stopped speaking, indicated by the VAD model.
The server has started ignoring audio from the client (server-side muting).
The client should continue sending audio normally but may want to show an
indication to the user that their input is not being processed. See [User
Input Muting](/pipecat/fundamentals/user-input-muting) for more details.
The server has stopped ignoring audio from the client (server-side muting
ended). The client can update its UI to indicate that the user's input is
being processed again.
### Transcription
Transcribed local user input (both partial and final).
Callback receives a `TranscriptData` object:
The transcribed text.
Indicates if the text is final (true) or partial (false).
The timestamp of the transcription.
The ID of the user the transcription is for.
A best-effort stream of the bot's output text, including both spoken and unspoken content. This callback is triggered as the bot aggregates the LLM's response into sentences or other logical text blocks as well as word-by-word during TTS synthesis. The callback receives a `BotOutputData` object:
The aggregated output text from the bot.
Indicates if the text has been spoken by the bot.
The aggregation type used for this output (e.g., "sentence", "code").
"sentence" and "word" are reserved aggregation types defined by the RTVI
standard. Other aggregation types may be defined by custom text aggregators
used by the server. "word" aggregated outputs are sent at the time of TTS
synthesis for real-time word-level streaming and can be used in lieu of
`onBotTtsText` if desired.
DEPRECATED in favor of `onBotOutput` in Pipecat version 0.0.95 and client-js
version 1.5.0
Finalized bot output text generated by the LLM. Sentence aggregated.
### Service-specific Events
Bot LLM search response text generated by the LLM service. This is typically
used for search or retrieval tasks.
Search capabilities are currently only supported by Google Gemini. To take
advantage of this event, your pipeline must include a
[`GoogleLLMService`](/api-reference/server/services/llm/google) and your
pipeline worker should include the
[`GoogleRTVIObserver`](/api-reference/server/rtvi/google-rtvi-observer) in
lieu of the typical `RTVIObserver`.
The search result text.
The rendered content of the search result.
The origins of the search result.
The URI of the site where the search result was found.
The title of the site where the search result was found.
The individual search results.
The text of the search result.
The confidence scores for the search result.
Streamed LLM token response text generated by the LLM service.
The text of the LLM response.
LLM service inference started.LLM service inference concluded.
If your TTS service supports streamed responses over sockets, the text
parameter contains the words from TTS service as they are spoken. If you are
using a HTTP based TTS service, the text parameter will contain the full text
of the TTS response.
The text of the LLM response.
TTS service started inference.TTS service inference concluded.
### Function Calling
A function call has been initiated by the LLM. The metadata included depends
on the server's [`function_call_report_level`](/api-reference/server/rtvi/rtvi-observer#configuration) configuration.
Name of the function being called. Only included if report level is `NAME` or `FULL`.
A function call is in progress. This replaces the deprecated `onLLMFunctionCall`
callback and is the event that triggers registered
[`FunctionCallHandler`s](/api-reference/client/js/client-methods#registerfunctioncallhandler)
when a `function_name` is present.
Name of the function being called. Only included if report level is `NAME` or `FULL`.
Unique identifier for this function call.
Arguments passed to the function. Only included if report level is `FULL`.
A function call has completed or been cancelled.
Name of the function that was called. Only included if report level is `NAME` or `FULL`.
Identifier matching the original function call.
Whether the function call was cancelled before completing.
The result of the function call, if available. Only included if report level is `FULL`.
DEPRECATED in favor of `onLLMFunctionCallInProgress` in Pipecat version
0.0.102 and client-js version 1.6.0
A function call request from the LLM.
### User interface
The bot sent a UI command for your app to act on. `data` is `{ command, payload }`;
standard commands are `scroll_to`, `highlight`, `select_text`, `click`, `focus`,
`set_input_value`, `toast`, and `navigate` (the [React
SDK](/api-reference/client/react/hooks#user-interface) ships default handlers).
See [the RTVI standard](/client/rtvi-standard#user-interface) for payload shapes.
A lifecycle envelope for background work the bot dispatched, for rendering a
progress card. `data.kind` is one of `group_started`, `job_update`,
`job_completed`, `group_completed`. Cancel a group with
[`cancelUIJobGroup()`](/api-reference/client/js/client-methods#user-interface).
### Other
Pipeline mterics data provided by Pipecat. [Learn
more](/pipecat/fundamentals/metrics).
## Events
Each callback described above has a corresponding event that can be listened for using the `.on()` method. This allows you to handle the same functionality using either callbacks or event listeners, depending on your preferred architecture.
Here's the complete reference mapping events to their corresponding callbacks:
### State and connectivity Events
| Event Name | Callback Name | Data Type |
| ----------------------- | ------------------------- | ---------------- |
| `Connected` | `onConnected` | - |
| `Disconnected` | `onDisconnected` | - |
| `TransportStateChanged` | `onTransportStateChanged` | `TransportState` |
| `BotReady` | `onBotReady` | `BotReadyData` |
| `BotConnected` | `onBotConnected` | - |
| `BotDisconnected` | `onBotDisconnected` | `Participant` |
| `ParticipantConnected` | `onParticipantJoined` | `Participant` |
| `ParticipantLeft` | `onParticipantLeft` | `Participant` |
### Message and Error Events
| Event Name | Callback Name | Data Type |
| --------------- | ----------------- | ------------- |
| `ServerMessage` | `onServerMessage` | `any` |
| `MessageError` | `onMessageError` | `RTVIMessage` |
| `Error` | `onError` | `RTVIMessage` |
### Media Events
| Event Name | Callback Name | Data Type |
| ---------------------- | ------------------------ | ------------------------------- |
| `TrackStarted` | `onTrackStarted` | `MediaStreamTrack, Participant` |
| `TrackStopped` | `onTrackStopped` | `MediaStreamTrack, Participant` |
| `AvailableMicsUpdated` | `onAvailableMicsUpdated` | `MediaDeviceInfo[]` |
| `AvailableCamsUpdated` | `onAvailableCamsUpdated` | `MediaDeviceInfo[]` |
| `MicUpdated` | `onMicUpdated` | `MediaDeviceInfo` |
| `CamUpdated` | `onCamUpdated` | `MediaDeviceInfo` |
| `SpeakerUpdated` | `onSpeakerUpdated` | `MediaDeviceInfo` |
| `DeviceError` | `onDeviceError` | `DeviceError` |
### Audio Activity Events
| Event Name | Callback Name | Data Type |
| --------------------- | ----------------------- | --------------------- |
| `LocalAudioLevel` | `onLocalAudioLevel` | `number` |
| `RemoteAudioLevel` | `onRemoteAudioLevel` | `number, Participant` |
| `BotStartedSpeaking` | `onBotStartedSpeaking` | - |
| `BotStoppedSpeaking` | `onBotStoppedSpeaking` | - |
| `UserStartedSpeaking` | `onUserStartedSpeaking` | - |
| `UserStoppedSpeaking` | `onUserStoppedSpeaking` | - |
| `UserMuteStarted` | `onUserMuteStarted` | - |
| `UserMuteStopped` | `onUserMuteStopped` | - |
### Text and Transcription Events
| Event Name | Callback Name | Data Type |
| ------------------- | --------------------- | ---------------- |
| `UserTranscript` | `onUserTranscript` | `TranscriptData` |
| `BotOutput` | `onBotOutput` | `BotOutputData` |
| ~~`BotTranscript`~~ | ~~`onBotTranscript`~~ | `BotLLMTextData` |
| `BotLlmText` | `onBotLlmText` | `BotLLMTextData` |
| `BotTtsText` | `onBotTtsText` | `BotTTSTextData` |
### Service State Events
| Event Name | Callback Name | Data Type |
| ---------------------- | ------------------------ | -------------------------- |
| `BotLlmSearchResponse` | `onBotLlmSearchResponse` | `BotLLMSearchResponseData` |
| `BotLlmStarted` | `onBotLlmStarted` | - |
| `BotLlmStopped` | `onBotLlmStopped` | - |
| `BotTtsStarted` | `onBotTtsStarted` | - |
| `BotTtsStopped` | `onBotTtsStopped` | - |
### Function Call Events
| Event Name | Callback Name | Data Type |
| --------------------------- | ----------------------------- | ------------------------------- |
| `LLMFunctionCallStarted` | `onLLMFunctionCallStarted` | `LLMFunctionCallStartedData` |
| `LLMFunctionCallInProgress` | `onLLMFunctionCallInProgress` | `LLMFunctionCallInProgressData` |
| `LLMFunctionCallStopped` | `onLLMFunctionCallStopped` | `LLMFunctionCallStoppedData` |
| ~~`LLMFunctionCall`~~ | ~~`onLLMFunctionCall`~~ | `LLMFunctionCallData` |
### User Interface Events
| Event Name | Callback Name | Data Type |
| ------------ | -------------- | ---------------- |
| `UICommand` | `onUICommand` | `UICommandData` |
| `UIJobGroup` | `onUIJobGroup` | `UIJobGroupData` |
Streaming snapshots and sending UI events are [client methods](/api-reference/client/js/client-methods#user-interface); see [the RTVI standard](/client/rtvi-standard#user-interface) for payload types.
### Other Events
| Event Name | Callback Name | Data Type |
| ---------- | ------------- | -------------------- |
| `Metrics` | `onMetrics` | `PipecatMetricsData` |
## Usage Example
```typescript theme={null}
import { PipecatClient, RTVIEvent } from "@pipecat-ai/client-js";
// Using callbacks
const pcClient = new PipecatClient({
callbacks: {
onBotReady: () => console.log("Bot ready via callback"),
onUserTranscript: (data) => console.log("Transcript:", data.text),
},
});
// Alternate approach: Using event listeners
pcClient.on(RTVIEvent.BotReady, () => {
console.log("Bot ready via event");
});
```
## Transport Compatibility
# PipecatClient Constructor
Source: https://docs.pipecat.ai/api-reference/client/js/client-constructor
Setting up the PipecatClient
```javascript theme={null}
import { PipecatClient } from "@pipecat-ai/client-js";
```
`PipecatClient` is the primary component for building the client-side portion of a client-bot interaction. It is designed to work with various transport layers, such as WebRTC, WebSockets, or HTTP, allowing you to pick and choose the communication layer that best suits your application while maintaining a consistent API.
When initializing the `PipecatClient`, you must provide a transport instance
to the constructor for your chosen protocol or provider. See
[Transport](./transports/transport) for more information. For the purpose of
this guide, we'll demonstrate using the [Daily WebRTC
transport](/api-reference/client/js/transports/daily).
## Example
```typescript theme={null}
import { RTVIEvent, RTVIMessage, PipecatClient } from "@pipecat-ai/client-js";
import { DailyTransport } from "@pipecat-ai/daily-transport";
const PipecatClient = new PipecatClient({
transport: new DailyTransport(),
enableMic: true,
enableCam: false,
enableScreenShare: false,
timeout: 15 * 1000,
callbacks: {
onConnected: () => {
console.log("[CALLBACK] User connected");
},
onDisconnected: () => {
console.log("[CALLBACK] User disconnected");
},
onTransportStateChanged: (state: string) => {
console.log("[CALLBACK] State change:", state);
},
onBotConnected: () => {
console.log("[CALLBACK] Bot connected");
},
onBotDisconnected: () => {
console.log("[CALLBACK] Bot disconnected");
},
onBotReady: () => {
console.log("[CALLBACK] Bot ready to chat!");
},
},
});
```
***
## API reference
### transport
An instance of the `Transport` type you will use to connect to your bot service (`PipecatClient.connect()`). Transports implement the underlying device management, connectivity, media transmission, and state logic that manage the lifecycle of your session.
As a best practice, we recommend you construct the transport inline in the
client constructor, as opposed to holding a reference to it. Access to the
transport is typically unnecessary. For advanced use cases that do require
access to the transport, we recommend doing so via the
`PipecatClient.transport` property, which provides some additional safeguards.
```typescript theme={null}
import { PipecatClient } from "@pipecat-ai/client-js";
import { DailyTransport } from "@pipecat-ai/daily-transport";
const pcClient = new PipecatClient({
transport: new DailyTransport(),
});
```
### callbacks
Map of callback functions. See [callbacks](./callbacks).
### Media Initialization
Enable user's local microphone device.
Enable user's local webcam device. Note: Not all transports support video.
Setting this value in that case will have no effect.
Enable user's local screen share. Note: Not all transports support screen
sharing. Setting this value in that case will have no effect.
### Session behavior
When `true` (default), the client will automatically disconnect when the bot
disconnects. Set to `false` to keep the client transport connected after the
bot leaves — useful if you want to handle reconnection or show a waiting state
without tearing down the transport. Introduced in client-js 1.7.0.
# Client Methods
Source: https://docs.pipecat.ai/api-reference/client/js/client-methods
The Pipecat JavaScript client provides a comprehensive set of methods for managing bot interactions and media handling. These core methods are documented below.
## Session connectivity
### startBot()
`async startBot(startBotParams: APIEndpoint): Promise`
This method hits your server endpoint to start the bot and optionally obtain the connection parameters needed for `connect()` to connect the `Transport`. It returns a Promise that resolves with the response from the server.
The `APIEndpoint` object should have the following shape:
The URL of the endpoint to connect to. This should be a valid REST endpoint.
Optional headers to include in the request to the endpoint. This can be used to pass authentication tokens or other necessary headers.
Optional request data to include in the request to the endpoint. This can be used to pass additional data to your server-side endpoint. Oftentimes, this is used to pass the initial prompt or other configuration data to initialize the bot.
Optional timeout in milliseconds for the request to the endpoint.
During the `startBot()` process, the transport state will transition through the states: "authenticating" and "authenticated".
```javascript theme={null}
try {
await pcClient.startBot({
endpoint: "/api/start", // Your server endpoint to start the bot
requestData: {
initial_prompt: "You are a pirate captain",
llm_provider: "openai",
},
});
} catch (error) {
console.error("Error starting the bot:", error);
}
```
### connect()
`async connect(connectParams): Promise`
This method initiates the connection process, optionally passing parameters that your transport class requires to establish a connection or an endpoint to your server for obtaining those parameters.
An object containing the `TransportConnectionParams` your Transport expects.
Check your transport class documentation for the expected shape of `TransportConnectionParams`. For example, the DailyTransport expects a `url` and `token`.
In 1.2.0 we deprecated support for passing a `ConnectionEndpoint` object directly to `connect()`. Instead, you should use the `startBot()` or `startBotAndConnect()` methods to fetch connection parameters from your server endpoint and then pass those parameters directly to `connect()`.
This method can be try / catched to handle errors at startup:
```typescript theme={null}
try {
await pcClient.connect({
webrtcUrl: "http://my-server/api/offer",
});
} catch (error) {
console.error("Error connecting to the bot:", error);
}
```
During the connection process, the transport state will transition through the following states: "connecting", "connected", "ready".
Calling `connect()` asynchronously will resolve when the bot and client signal
that they are ready. If you want to call `connect()` without `await`, you can
use the `onBotReady` callback or `BotReady` event to know when you can
interact with the bot.
Attempting to call `connect()` when the transport is already in a 'connected'
or 'ready' state will throw an error. You should [disconnect](#disconnect)
from a session first before attempting to connect again.
### startBotAndConnect()
`async startBotAndConnect(startBotParams: APIEndpoint): Promise`
This method combines the functionality of `startBot()` and `connect()`. It first starts the bot by hitting your server endpoint and then connects the transport passing the response from the endpoint to the transport as connection parameters.
```javascript theme={null}
try {
await pcClient.startBotAndConnect({
endpoint: "/api/start", // Your server endpoint to start the bot
requestData: {
initial_prompt: "You are a pirate captain",
llm_provider: "openai",
},
});
} catch (error) {
console.error("Error starting up:", error);
}
```
It's equivalent to: `pcClient.startBot(...).then((resp) => pcClient.connect(resp))`.
### disconnect()
`async disconnect(): Promise`
Disconnects from the active session. The transport state will transition to "disconnecting" and then "disconnected".
It is common practice for bots to exit and cleanup when the client disconnects.
```typescript theme={null}
await pcClient.disconnect();
```
### disconnectBot()
`disconnectBot(): void`
Triggers the bot to disconnect from the session, leaving the client connected.
```typescript theme={null}
await pcClient.disconnectBot();
```
## Messages
Custom messaging between the client and the bot. This is useful for sending data to the bot, triggering specific actions, reacting to server events, or querying the server.
For more, see the [Custom Messaging](/client/guides/custom-messaging) guide.
### sendClientMessage()
`sendClientMessage(msgType: string, data?: unknown): void`
Sends a custom message to the bot and does not expect a response. This is useful for sending data to the bot or triggering specific actions.
A string identifying the message.
Optional data to send with the message. This can be any JSON-serializable
object.
### sendClientRequest()
`async sendClientRequest(msgType: string, data: unknown, timeout?: number): Promise`
Sends a custom request to the bot and expects a response. This is useful for querying the server or triggering specific actions that require a response. The method returns a Promise that resolves with the data from response.
A string identifying the message.
Optional data to send with the message. This can be any JSON-serializable
object.
Optional timeout in milliseconds for the request. If the request does not
receive a response within this time, it will reject with an RTVIMessage of
type `'error-response'`.
## Devices
### initDevices()
`async initDevices(): Promise`
Initializes the media device selection machinery, based on `enableCam`/`enableMic` selections and defaults (i.e. turns on the local cam/mic). This method can be called before `connect()` to test and switch between camera and microphone sources.
```typescript theme={null}
await pcClient.initDevices();
```
### getAllMics()
`async getAllMics(): Promise`
Returns a list of available microphones in the form of [`MediaDeviceInfo[]`](https://developer.mozilla.org/en-US/docs/Web/API/MediaDeviceInfo).
```typescript theme={null}
mic_device_list = pcClient.getAllMics();
```
### getAllCams()
`async getAllCams(): Promise`
Returns a list of available cameras in the form of [`MediaDeviceInfo[]`](https://developer.mozilla.org/en-US/docs/Web/API/MediaDeviceInfo).
```typescript theme={null}
cam_device_list = pcClient.getAllCams();
```
### getAllSpeakers()
`async getAllSpeakers(): Promise`
Returns a list of available speakers in the form of [`MediaDeviceInfo[]`](https://developer.mozilla.org/en-US/docs/Web/API/MediaDeviceInfo).
```typescript theme={null}
speaker_device_list = pcClient.getAllSpeakers();
```
### selectedMic
`selectedMic: MediaDeviceInfo | {}`
The currently selected microphone, represented as a `MediaDeviceInfo` object. If no microphone is selected, it returns an empty object.
```typescript theme={null}
current_mic = pcClient.selectedMic;
```
### selectedCam
`selectedCam: MediaDeviceInfo | {}`
The currently selected camera, represented as a `MediaDeviceInfo` object. If no camera is selected, it returns an empty object.
```typescript theme={null}
current_cam = pcClient.selectedCam;
```
### selectedSpeaker
`selectedSpeaker: MediaDeviceInfo | {}`
The currently selected speaker, represented as a `MediaDeviceInfo` object. If no speaker is selected, it returns an empty object.
```typescript theme={null}
current_speaker = pcClient.selectedSpeaker;
```
### updateMic()
`updateMic(micId: string): void`
Switches to the microphone identified by the provided `micId`, which should match a `deviceId` in the list returned from [`getAllMics()`](#getallmics).
deviceId
```typescript theme={null}
pcClient.updateMic(deviceId);
```
### updateCam()
`updateCam(camId: string): void`
Switches to the camera identified by the provided `camId`, which should match a `deviceId` in the list returned from [`getAllCams()`](#getallcams).
deviceId
```typescript theme={null}
pcClient.updateCam(deviceId);
```
### updateSpeaker()
`updateSpeaker(speakerId: string): void`
Switches to the speaker identified by the provided `speakerId`, which should match a `deviceId` in the list returned from [`getAllSpeakers()`](#getallspeakers).
deviceId
```typescript theme={null}
pcClient.updateSpeaker(deviceId);
```
### enableMic(enable: boolean)
`enableMic(enable: boolean): void`
Turn on or off (unmute or mute) the client mic input.
A boolean indicating whether to enable (`true`) or disable (`false`) the
microphone.
```typescript theme={null}
pcClient.enableMic(true);
```
### enableCam(enable: boolean)
`enableCam(enable: boolean): void`
Turn on or off the client cam input.
A boolean indicating whether to enable (`true`) or disable (`false`) the
camera.
```typescript theme={null}
pcClient.enableCam(true);
```
### enableScreenShare(enable: boolean)
`enableScreenShare(enable: boolean): void`
Start a screen share from the client's device.
A boolean indicating whether to enable (`true`) or disable (`false`) screen
sharing.
```typescript theme={null}
pcClient.enableScreenShare(true);
```
### isMicEnabled
`isMicEnabled: boolean`
An accessor to determine if the client's microphone is enabled.
```typescript theme={null}
mic_enabled = pcClient.isMicEnabled;
```
### isCamEnabled
`isCamEnabled: boolean`
An accessor to determine if the client's camera is enabled.
```typescript theme={null}
cam_enabled = pcClient.isCamEnabled;
```
### isSharingScreen
An accessor to determine if the client is sharing their screen.
```typescript theme={null}
screen_sharing = pcClient.isSharingScreen;
```
## Tracks (audio and video)
### tracks()
`tracks(): Tracks`
Returns a `Tracks` object with available `MediaStreamTrack` objects for both the client and the bot.
```typescript theme={null}
live_tracks_list = pcClient.tracks();
```
**Tracks Type**
```typescript theme={null}
{
local: {
audio?: MediaStreamTrack;
video?: MediaStreamTrack;
},
bot?: {
audio?: MediaStreamTrack;
video?: MediaStreamTrack;
}
}
```
## Advanced LLM Interactions
### sendText()
`async sendText(content: string, options?: SendTextOptions = {}): void`
A method to append text to the user's context. This is useful for providing text input as an alternative to audio input for the user.
The text content to send to the bot.
An optional set of options for how the bot should handle the text input.
Whether to immediately run the bot with the updated context. If `false`,
the context will be updated but the bot will not be run until the next
message or action that triggers the bot to run (like the user speaking).
Whether the bot should respond with audio. If `true`, the bot's response
will be processed by TTS and be spoken. If `false`, the bot will bypass
the TTS and respond with text only.
### registerFunctionCallHandler()
`registerFunctionCallHandler(functionName: string, callback: FunctionCallCallback): void`
Registers a function call handler that will be called when the bot requests a function call. This is useful for when the server-side function handler needs information from the client to execute the function call or when the client needs to perform some action based on the running of function call.
The name of the function to handle. This should match the function name in the bot's context.
`type FunctionCallCallback = (fn: FunctionCallParams) => Promise`
The callback function to call when the bot sends a function call request. This function should accept the following parameters:
The name of the function being called. It should always match the name you
registered the handler under.
The arguments passed to the function call. This is a key-value object where
the keys are the argument names and the values are the argument values.
The callback should return a Promise that resolves with the result of the function call or void if no result is needed. If returning a result, it should be a `string` or `Record`.
## Other
### transport
`transport: Transport`
A safe accessor for the transport instance used by the client. This is useful for accessing transport-specific methods or properties that are not exposed directly on the client.
```typescript theme={null}
const transport = pcClient.transport as DailyTransport;
transport.getSessionInfo();
```
### setLogLevel()
`setLogLevel(level: LogLevel): void`
Sets the log level for the client. This is useful for debugging and controlling the verbosity of logs. The log levels are defined in the `LogLevel` enum:
```typescript theme={null}
export enum LogLevel {
NONE = 0,
ERROR = 1,
WARN = 2,
INFO = 3,
DEBUG = 4,
}
```
By default, the log level is set to `LogLevel.DEBUG`.
```typescript theme={null}
pcClient.setLogLevel(LogLevel.INFO);
```
## User interface
These methods voice-enable a UI: the client streams the screen to a server-side [`UIWorker`](/api-reference/server/workers/ui-worker) and forwards UI events, while the bot drives the page with commands (handled via the [`UICommand` event](/api-reference/client/js/callbacks#user-interface-events)). See [the RTVI standard](/client/rtvi-standard#user-interface) for the wire payloads and [Controlling the UI](/pipecat/learn/ui-worker) for the patterns.
### startUISnapshotStream()
`startUISnapshotStream(options?: A11ySnapshotStreamerOptions): void`
Streams accessibility snapshots of the page to the bot as `ui-snapshot` messages. Emits an initial snapshot, then re-emits on DOM, form-control, focus, scroll, resize, and visibility changes, coalesced by `debounceMs`. Calling it again replaces the existing stream.
Coalesce window before emitting a snapshot.
Tag off-screen nodes `offscreen` so the bot scrolls before acting on them.
Log each emitted snapshot to the console.
### stopUISnapshotStream()
`stopUISnapshotStream(): void`
Stops the managed snapshot stream and tears down its observers. Idempotent.
### sendUIEvent()
`sendUIEvent(event: string, payload?: unknown): void`
Sends an app-defined UI event to the bot (routed to the worker's `@ui_event` handlers).
App-defined event name (e.g. `"note_click"`).
Optional schemaless payload.
### cancelUIJobGroup()
`cancelUIJobGroup(jobId: string, reason?: string): void`
Asks the bot to cancel an in-flight job group (only honored for groups the server marked `cancellable`). The `jobId` comes from the `group_started` envelope on the [`UIJobGroup` event](/api-reference/client/js/callbacks#user-interface-events).
The job group id.
Optional human-readable reason.
To act on the bot's commands, resolve its snapshot refs against the live DOM
with the exported helpers `findElementByRef(ref)`, `findRefForElement(el)`,
`snapshotDocument(root?, options?)`, and `serializeSelection()`.
## Transport Compatibility
# Errors
Source: https://docs.pipecat.ai/api-reference/client/js/errors
## RTVIError Type
Base `PipecatClient` error type, extends [`Error`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Error) and primarily introduces the `status` field. Most methods will try to throw an error of this type when something goes wrong. This is different from the RTVI error event and its corresponding [`onError` callback](./callbacks#param-on-error), which are used for communicating errors that are sent by the bot.
A unique identifier (or HTTP code if applicable) for the error.
A human-readable message describing the error.
## Pre-defined RTVIErrors
### ConnectionTimeoutError
Emitted when the bot does not enter a ready state within the specified timeout period during the `connect()` method call.
### StartBotError
Emitted from `startBot()` or `startBotAndConnect()` when the endpoint responds with an error or the `fetch` itself fails. This may be due to the endpoint being unavailable, or the server failing to parse the provided data.
All `StartBotError` instances will have an `error` field set to
`invalid-request-error`.
HTTP status code returned by the endpoint, if applicable.
Verbose error message returned by the endpoint, if provided. To take advantage
of this, the endpoint should return an error response with a JSON object with
an `info` field containing the error message.
### TransportStartError
Emitted when the Transport is not able to connect. You may need to check the connection parameters provided or returned from you endpoint.
### BotNotReadyError
Emitted when the client attempts to perform an action or method that requires the bot to be in a ready state, but the bot is not ready. You must call `connect()` first and wait for the bot to be ready before performing such actions.
### DeviceError
Emitted when there is an issue with acquiring or using a media device, such as a camera or microphone. This could be due to permissions issues, device unavailability, or other related problems.
List of media devices, `'cam'`, `'mic'`, and/or `'speaker'`, that are
unavailable or could not be accessed.
The `type` field will indicate what type of device error occurred. Options include:
* `"in-use"`: A device is currently in use by another application and cannot be accessed. *windows only*
* `"permissions"`: The user has not granted permission to access the media device.
* `"undefined-mediadevices"`: `getUserMedia()` is not an available API on the current platform or browser.
* `"not-found"`: The specified media device could not be found.
* `"constraints"`: The media device could not be configured with the specified constraints.
* `"unknown"`: An unknown error occurred while accessing the media device.
Additional details about the device error, if available.
### UnsupportedFeatureError
Not all Transports are created equal, and some may not support certain features. This error is thrown when a feature is requested that the current Transport does not support.
This custom field will contain the name of the unsupported feature.
# JavaScript SDK Overview
Source: https://docs.pipecat.ai/api-reference/client/js/overview
Build web applications with Pipecat’s JavaScript client library
The Pipecat JavaScript SDK provides a lightweight client implementation that handles:
* Device and media stream management
* Connecting to Pipecat bots
* Messaging with Pipecat bots and handling responses using the RTVI standard
* Managing session state and errors
## Installation
Install the SDK and a transport implementation (e.g. Daily for WebRTC):
```bash theme={null}
npm install @pipecat-ai/client-js
npm install @pipecat-ai/[daily-transport, small-webrtc-transport, etc.]
```
## Example
Here's a simple example using Daily as the transport layer:
```javascript theme={null}
import { PipecatClient } from "@pipecat-ai/client-js";
import { DailyTransport } from "@pipecat-ai/daily-transport";
// Handle incoming audio from the bot
function handleBotAudio(track, participant) {
if (participant.local || track.kind !== "audio") return;
const audioElement = document.createElement("audio");
audioElement.srcObject = new MediaStream([track]);
document.body.appendChild(audioElement);
audioElement.play();
}
// Create and configure the client
const pcClient = new PipecatClient({
transport: new DailyTransport(),
enableMic: true,
callbacks: {
onTrackStarted: handleBotAudio,
},
});
// Connect to your bot
pcClient.connect({
url: "https://your-daily-room-url",
token: "your-daily-token",
});
```
## Explore the SDK
Configure your client instance with transport and callbacks
Core methods for interacting with your bot
Handle bot events, messages, and state changes
Daily, SmallWebRTC, WebSocket, and other transports
The Pipecat JavaScript SDK implements the [RTVI standard](/client/rtvi-standard) for real-time AI inference, ensuring compatibility with any RTVI-compatible server and transport layer.
# Daily WebRTC Transport
Source: https://docs.pipecat.ai/api-reference/client/js/transports/daily
The DailyTransport class provides a WebRTC transport layer using [Daily.co's](https://daily.co) infrastructure. It wraps a Daily-JS call client to handle audio/video device management, WebRTC connections, and real-time communication between clients and bots. For complete documentation on Daily's API, see the [Daily API Reference](https://docs.daily.co/reference/daily-js).
This transport is designed for production use cases, leveraging Daily's global infrastructure for low-latency, high-quality audio and video streaming. It expects your Pipecat server to include the corresponding [`DailyTransport` server-side](/api-reference/server/services/transport/daily) implementation.
## Usage
### Basic Setup
```javascript theme={null}
import { PipecatClient } from "@pipecat-ai/client-js";
import { DailyTransport } from "@pipecat-ai/daily-transport";
const pcClient = new PipecatClient({
transport: new DailyTransport({
// DailyTransport constructor options
bufferLocalAudioUntilBotReady: true, // Optional, defaults to false
inputSettings: { video: { processor: { type: "background-blur" } } },
}),
enableCam: false, // Default camera off
enableMic: true, // Default microphone on
callbacks: {
// Event handlers
},
// ...
});
await pcClient.connect({
url: "https://your-domain.daily.co/room",
token: "your-daily-token", // Optional, if your room requires authentication
});
```
## API Reference
### Constructor Options
```typescript theme={null}
interface DailyTransportConstructorOptions extends DailyFactoryOptions {
bufferLocalAudioUntilBotReady?: boolean;
}
```
If set to `true`, the transport will buffer local audio until the bot is ready. This is useful for ensuring that bot gets any audio from the user that started before the bot is ready to process it.
The `DailyTransportConstructorOptions` extends the `DailyFactoryOptions` type that is accepted by the underlying Daily instance. These options are passed directly through to the Daily constructor. See the [Daily API Reference](https://docs.daily.co/reference/daily-js/daily-call-client/properties) for a complete list of options.
While you can provide the room url and optional token as part of your
constructor options, the typical pattern is to provide them via a connection
endpoint with `startBot()` or directly as part of `connect()`. See below.
### TransportConnectionParams
On `connect()`, the `DailyTransport` optionally takes a set of [`DailyCallOptions`](https://docs.daily.co/reference/daily-js/daily-call-client/properties) to connect to a Daily room. This can be provided directly to the `PipecatClient`'s `connect()` method or via a starting endpoint passed to the `PipecatClient`'s `startBotAndConnect()` method. If using an endpoint, your endpoint should return a JSON object matching the `DailyCallOptions` type. See the [client connect()](/api-reference/client/js/client-methods#connect) documentation for more information.
```typescript client theme={null}
pcClient.connect({
url: 'https://your.daily.co/room'
});
// OR...
pcClient.startBotAndConnect({
endpoint: '/api/start', // Your server endpoint to start the bot
});
```
```python server theme={null}
@app.post("/api/start")
async def start(request: Request) -> Dict[Any, Any]:
print("Creating room and token for RTVI connection")
room_url, token = await create_room_and_token()
# Start the bot process
print("Starting bot subprocess")
try:
subprocess.Popen(
[f"python3 -m bot.py -u {room_url} -t {token}"],
shell=True,
bufsize=1,
cwd=os.path.dirname(os.path.abspath(__file__)),
)
except Exception as e:
raise HTTPException(status_code=500, detail=f"Failed to start subprocess: {e}")
# Return the Daily call options in format expected by DailyTransport/Daily Call Object
return {"url": room_url, "token": token}
```
### Methods
For most operations, you will not interact with the transport directly. Most methods have an equivalent in the `PipecatClient` and should be called from the `PipecatClient`. However, there are a few transport-specific methods that you may need to call directly. When doing so, be sure to access your transport via the `transport` property of the `PipecatClient` instance.
* `preAuth()`
This is the one method meant to be called directly, which is used to allow you to gather information about the Daily room prior to connecting. As a Daily-specific action, it is not exposed through the `PipecatClient`. This method must be called prior to `connect()` and use the same `room_url` and `token` (optional) as what will be used on `connect()`.
```typescript theme={null}
pcClient.transport.preAuth({
url: "https://your.daily.co/room",
token: "your_token",
});
const roomInfo = pcClient.transport.dailyCallClient.room();
```
## Events
The transport implements the various [`PipecatClient` event handlers](/api-reference/client/js/callbacks). For Daily-specific events, you can attach listeners to the underlying Daily call client. For a list of available events, see the [Daily API Reference](https://docs.daily.co/reference/daily-js/events).
```typescript theme={null}
pcClient.transport.dailyCallClient.on('recording-started', (ev) => {...});
```
## Advanced
### Accessing the Daily Call
For advanced use cases, where you may need to work with the Daily call client directly, you can access it via the `dailyCallClient` property.
```javascript theme={null}
const dailyCall = pcClient.transport.dailyCallClient;
```
The Daily call client returned is safe-guarded to not allow you to call
functions which affect the call's lifecycle and will redirect you to use
either a Transport method or the `PipecatClient` to perform the equivalent
action.
## More Information
Simple Chatbot Demo
`DailyTransport`
`@pipecat-ai/daily-transport`
# GeminiLiveWebSocketTransport
Source: https://docs.pipecat.ai/api-reference/client/js/transports/gemini
## Overview
The `GeminiLiveWebsocketTransport` class implements a fully functional [Pipecat `Transport`](./transport), providing a framework for implementing real-time communication directly with the [Gemini Live](https://ai.google.dev/gemini-api/docs/live-api) service. Like all transports, it handles media device management, audio/video streams, and state management for the connection.
Transports of this type are designed primarily for development and testing
purposes. For production applications, you will need to build a server
component with a server-friendly transport, like the
[DailyTransport](./daily), to securely handle API keys.
## Usage
### Basic Setup
```javascript theme={null}
import {
GeminiLiveWebsocketTransport,
GeminiLLMServiceOptions,
} from "@pipecat-ai/gemini-live-websocket-transport";
import { PipecatClient } from "@pipecat-ai/client-js";
const options: GeminiLLMServiceOptions = {
api_key: "YOUR_API_KEY",
initial_messages: [
// Set up initial system and user messages.
// Without the user message, the bot will not respond immediately
// and wait for the user to speak first.
{
role: "model",
content: "You are a confused jellyfish.",
},
{ role: "user", content: "Blub blub!" },
],
settings: {
temperature: 0.7,
maxOutput_tokens: 1000,
},
};
const transport = new GeminiLiveWebsocketTransport(options);
let pcClient = new PipecatClient({
transport: new GeminiLiveWebsocketTransport(options),
callbacks: {
// Event handlers
},
});
pcClient.connect();
```
## API Reference
### Constructor Options
#### `GeminiLLMServiceOptions`
```typescript theme={null}
interface GeminiLLMServiceOptions {
api_key: string; // Required: Your Gemini API key
initial_messages?: Array<{
// Optional: Initial conversation context
content: string;
role: string;
}>;
settings?: {
// Optional: Generation parameters
candidate_count?: number;
max_output_tokens?: number;
temperature?: number;
top_p?: number;
top_k?: number;
presence_penalty?: number;
frequency_penalty?: number;
response_modalities?: string;
speech_config?: {
voice_config?: {
prebuilt_voice_config?: {
voice_name: "Puck" | "Charon" | "Kore" | "Fenrir" | "Aoede";
};
};
};
};
}
```
### TransportConnectionParams
The `GeminiLiveWebsocketTransport` does not take connection parameters. It connects directly to the Gemini Live service using the API key provided as part of the initial configuration.
### Events
The GeminiLiveWebSocketTransport implements the various [PipecatClient event handlers](/api-reference/client/js/callbacks). Check out the docs or samples for more info.
## More Information
Gemini Live Basic Demo
`GeminiLiveWebsocketTransport`
`@pipecat-ai/gemini-live-websocket-transport`
# OpenAIRealTimeWebRTCTransport
Source: https://docs.pipecat.ai/api-reference/client/js/transports/openai-webrtc
## Overview
The `OpenAIRealTimeWebRTCTransport` is a fully functional [Pipecat `Transport`](/api-reference/client/js/transports/transport). It provides a framework for implementing real-time communication directly with the [OpenAI Realtime API using WebRTC](https://platform.openai.com/docs/guides/realtime-webrtc) voice-to-voice service. It handles media device management, audio/video streams, and state management for the connection.
Transports of this type are designed primarily for development and testing
purposes. For production applications, you will need to build a server
component with a server-friendly transport, like the
[DailyTransport](./daily), to securely handle API keys.
## Usage
### Basic Setup
```javascript theme={null}
import { OpenAIRealTimeWebRTCTransport, OpenAIServiceOptions } from '@pipecat-ai/openai-realtime-webrtc-transport';
import { PipecatClient } from '@pipecat-ai/client-js';
const options: OpenAIServiceOptions = {
api_key: 'YOUR_API_KEY',
settings: {
instructions: 'You are a confused jellyfish.',
},
initial_messages: [{ role: "user", content: "Blub blub!" }],
};
let pcClient = new PipecatClient({
transport: new OpenAIRealTimeWebRTCTransport (options),
...
});
pcClient.connect();
```
## API Reference
### Constructor Options
Below is the transport's type definition for the OpenAI Session configuration you need to pass in to the `create()` method. See the [OpenAI Realtime API documentation](https://platform.openai.com/docs/api-reference/realtime-client-events/session/update) for more details on each of the options and their defaults.
```typescript theme={null}
export type OpenAIFunctionTool = {
type: "function";
name: string;
description: string;
parameters: JSONSchema;
};
export type OpenAIServerVad = {
type: "server_vad";
create_response?: boolean;
interrupt_response?: boolean;
prefix_padding_ms?: number;
silence_duration_ms?: number;
threshold?: number;
};
export type OpenAISemanticVAD = {
type: "semantic_vad";
eagerness?: "low" | "medium" | "high" | "auto";
create_response?: boolean; // defaults to true
interrupt_response?: boolean; // defaults to true
};
export type OpenAISessionConfig = Partial<{
modalities?: string;
instructions?: string;
voice?:
| "alloy"
| "ash"
| "ballad"
| "coral"
| "echo"
| "sage"
| "shimmer"
| "verse";
input_audio_noise_reduction?: {
type: "near_field" | "far_field";
} | null; // defaults to null/off
input_audio_transcription?: {
model: "whisper-1" | "gpt-4o-transcribe" | "gpt-4o-mini-transcribe";
language?: string;
prompt?: string[] | string; // gpt-4o models take a string
} | null; // we default this to gpt-4o-transcribe
turn_detection?: OpenAIServerVad | OpenAISemanticVAD | null; // defaults to server_vad
temperature?: number;
max_tokens?: number | "inf";
tools?: Array;
}>;
export interface OpenAIServiceOptions {
api_key: string;
model?: string;
initial_messages?: LLMContextMessage[];
settings?: OpenAISessionConfig;
}
```
### TransportConnectionParams
The `OpenAIRealTimeWebRTCTransport` does not take connection parameters. It connects directly to the OpenAI Realtime API using the API key provided as part of the initial configuration.
### Events
The transport implements the various [`PipecatClient` event handlers](/api-reference/client/js/callbacks). Check out the docs or samples for more info.
## More Information
OpenAI Realtime Basic Demo
`OpenAIRealTimeWebRTCTransport`
`@pipecat-ai/openai-realtime-webrtc-transport`
# SmallWebRTCTransport
Source: https://docs.pipecat.ai/api-reference/client/js/transports/small-webrtc
A lightweight WebRTC transport for peer-to-peer connections with Pipecat
`SmallWebRTCTransport` enables peer-to-peer WebRTC connections between clients and your Pipecat application. It implements bidirectional audio and video streaming using WebRTC for real-time communication.
This transport is intended for lightweight implementations, particularly for local development and testing. It expects your Pipecat server to include the corresponding [`SmallWebRTCTransport` server-side](/api-reference/server/services/transport/small-webrtc) implementation.
## Usage
### Basic Setup
```javascript theme={null}
import { PipecatClient } from "@pipecat-ai/client-js";
import { SmallWebRTCTransport } from "@pipecat-ai/small-webrtc-transport";
const pcClient = new PipecatClient({
transport: new SmallWebRTCTransport({
// Optional configuration for the transport
iceServers: [{ urls: "stun:stun.l.google.com:19302" }],
}),
enableCam: false, // Default camera off
enableMic: true, // Default microphone on
callbacks: {
// Event handlers
},
});
await pcClient.connect({
webrtcUrl: "/api/offer", // Your WebRTC signaling server endpoint
});
```
## API Reference
### Constructor Options
```typescript theme={null}
interface SmallWebRTCTransportConstructorOptions {
iceServers?: RTCIceServer[];
waitForICEGathering?: boolean;
webrtcUrl?: string;
audioCodec?: string;
videoCodec?: string;
mediaManager?: MediaManager;
}
```
#### Properties
Array of ICE server configurations for connection establishment. Default is `[{ urls: "stun:stun.l.google.com:19302" }]`.
```javascript theme={null}
// Set custom ICE servers
transport.iceServers = [
{ urls: "stun:stun.l.google.com:19302" },
{ urls: "stun:stun1.l.google.com:19302" },
];
```
If `true`, the transport will wait for ICE gathering to complete before being
considered `'connected'`.
URL of the WebRTC signaling server's offer endpoint. This endpoint may also be provided as part of `connect()`.
Note: This field used to be called `connectionUrl` in versions prior to
`1.2.0`.
Preferred audio codec to use. If not specified, your browser default will be
used.
Preferred video codec to use. If not specified, your browser default will be
used.
The media manager to use for handling local audio and video streams. This
should not be overridden unless you have a specific reason to use a different
media manager. The default is `DailyMediaManager`, which is suitable for most
use cases. Note that the `DailyMediaManager` does not use any of Daily's
services, it simply takes advantage of vast media support provided by the
Daily library.
### TransportConnectionParams
```typescript theme={null}
export type SmallWebRTCTransportConnectionOptions = {
webrtcUrl?: string;
};
```
On `connect()`, the `SmallWebRTCTransport` optionally takes a set of connection parameters. This can be provided directly to the `PipecatClient`'s `connect()` method or via a starting endpoint passed to the `PipecatClient`'s `startBotAndConnect()` method. If using an endpoint, your endpoint should return a JSON object matching the `SmallWebRTCTransportConnectionOptions` type, which currently expects a single `webrtcUrl` property.
```typescript client theme={null}
pcClient.startBotAndConnect({
endpoint: '/api/start', // Your server endpoint to start the bot and return the webrtcUrl
});
// OR...
pcClient.connect({
webrtcUrl: '/api/offer', // Your WebRTC offer/answer endpoint
});
```
```python server theme={null}
# See
# https://github.com/pipecat-ai/pipecat-examples/blob/main/p2p-webrtc/video-transform/server/server.py
# for a complete example of how to implement the server-side endpoint.
@app.post("/api/offer")
async def offer(request: dict, background_tasks: BackgroundTasks):
pipecat_connection = SmallWebRTCConnection(ice_servers)
await pipecat_connection.initialize(sdp=request["sdp"], type=request["type"])
@pipecat_connection.event_handler("closed")
async def handle_disconnected(webrtc_connection: SmallWebRTCConnection):
logger.info(f"Discarding peer connection for pc_id: {webrtc_connection.pc_id}")
pcs_map.pop(webrtc_connection.pc_id, None)
background_tasks.add_task(run_bot, pipecat_connection)
answer = pipecat_connection.get_answer()
return answer
```
### Methods
For most operations, you will not interact with the transport directly. Most methods have an equivalent in the `PipecatClient` and should be called from the `PipecatClient`. However, there are a few transport-specific methods that you may need to call directly. When doing so, be sure to access your transport via the `transport` property of the `PipecatClient` instance.
Sets the preferred audio codec.
```javascript theme={null}
transport.setAudioCodec("opus");
```
Sets the preferred video codec.
```javascript theme={null}
transport.setVideoCodec("VP8");
```
## Events
The transport implements the various [`PipecatClient` event handlers](/api-reference/client/js/callbacks).
## Connection Process
The connection process follows these steps:
1. The transport negotiates a WebRTC connection with the corresponding pipecat transport, complete with transceivers for the media and a data channel for messaging.
2. The transport sends a message to the pipecat transport to let it know it's ready.
3. The Pipecat transport sends a message letting the client know it is ready.
## Reconnection Handling
The transport includes automatic reconnection logic:
* Up to 3 reconnection attempts after connection failures
* Detection of ICE connection state changes
* Graceful recovery from temporary disconnections
* Graceful disconnect when reconnection attempts fail
## More Information
Real-time video transformation example
`@pipecat-ai/small-webrtc-transport`
# Transport Overview
Source: https://docs.pipecat.ai/api-reference/client/js/transports/transport
Transports are the means by which `PipecatClient`s communicate with their bot services. Transports implement the underlying device management, connectivity, media transmission, and state logic that manage the lifecycle of your session.
All transport packages (such as `DailyTransport`) extend from the `Transport` base class defined in the `client-js` library. You can extend this class if you are looking to implement your own or add additional functionality.
## Transport lifecycle
Each Pipecat client instance is associated with a transport instance. The instance will re-use the transport instance across multiple calls to `connect()`, allowing you to connect to different bot services without needing to create a new transport or client each time.
```typescript theme={null}
import { PipecatClient } from "@pipecat-ai/client-js";
import { DailyTransport } from "@pipecat-ai/daily-transport";
const pcClient = new PipecatClient({
transport: new DailyTransport(),
...
});
await pcClient.startBotAndConnect({ endpoint: "/api/start" });
await pcClient.disconnect();
await pcClient.connect(); // re-uses url returned from previous startBotAndConnect call, skipping the endpoint
```
## Transport states
`TransportState`
Your transport instance goes through a series of states during its lifecycle. These states are:
Transport is idle and has not yet been initialized (default state).
Transport is being initialized. This occurs in response to a
`pcClient.initDevices()` call, where the transport is being set up in order
to enumerate local media devices. If you call `connect()` and bypass
`initDevices()`, the transport will skip this state and go directly to
`Connecting`.
Transport has been initialized and is ready to connect. This state is
reached after a successful `pcClient.initDevices()` call and skipped if
`initDevices()` is not used.
Your client has called `pcClient.startBot()` or
`pcClient.startBotAndConnect()` and is waiting for a response from your
server containing connection details for your transport (such as a session
URL and token). Note: If you provide the `TransportConnectionParams`
directly to `connect()` without calling either `startBot` methods, the
transport will skip this state and go directly to `Connecting`.
Your client has called `pcClient.startBot()` or
`pcClient.startBotAndConnect()` and has successfully received a response. If
using `startBotAndConnect()`, it will quickly move into the `Connecting`
state. Note: If you provide the `TransportConnectionParams` directly to
`connect()` without calling either `startBot` methods, the transport will
skip this state and go directly to `Connecting`.
The transport is connecting to the server.
The transport has successfully connected to the session and is awaiting a
client-ready signal (indicated audio and video tracks are ready to be sent
and received).
Transport is ready and the session can begin.
Transport is disconnecting from the session.
An error occurred during the transport lifecycle. This indicates a fatal
error and the transport should move quickly into the `Disconnected` state.
You can access the current transport state via `pcClient.state`, or by defining a callback or event:
```typescript theme={null}
// Callback
const pcClient = new PipecatClient({
transport: new DailyTransport(),
callbacks: {
onTransportStateChange: (state) => {
console.log(state);
}
//...
});
// Event
pcClient.on(RTVIEvent.TransportStateChanged, (e) => console.log(e));
// Client getter
console.log(pcClient.state); // Disconnected
```
# WebSocketTransport
Source: https://docs.pipecat.ai/api-reference/client/js/transports/websocket
A lightweight transport for WebSocket based connections with Pipecat
`WebSocketTransport` enables a purely WebSocket based connection between clients and your Pipecat application. It implements bidirectional audio and video streaming using a WebSocket for real-time communication.
This transport is intended for lightweight implementations, particularly for local development and testing. It expects your Pipecat server to include the corresponding [`WebSocketTransport` server-side](/api-reference/server/services/transport/websocket-server) implementation.
The `WebSocketTransport` is best suited for server-server applications and prototyping client/server apps.
For client/server production applications, we strongly recommend using a WebRTC-based transport for robust network and media handling. For more on WebRTC vs. Websocket communication, check out [this article](https://voiceaiandvoiceagents.com/#websockets-webrtc).
## Usage
### Basic Setup
```javascript theme={null}
import { PipecatClient } from "@pipecat-ai/client-js";
import {
WebSocketTransport,
ProtobufFrameSerializer,
} from "@pipecat-ai/websocket-transport";
const pcClient = new PipecatClient({
transport: new WebSocketTransport({
serializer: new ProtobufFrameSerializer(),
recorderSampleRate: 8000,
playerSampleRate: 8000,
}),
enableCam: false, // Default camera off
enableMic: true, // Default microphone on
callbacks: {
// Event handlers
},
});
await pcClient.connect({
wsUrl: "ws://localhost:7860/ws", // Your WebSocket server URL
});
```
## API Reference
### Constructor Options
```typescript theme={null}
type WebSocketTransportOptions = {
wsUrl?: string;
serializer?: WebSocketSerializer;
recorderSampleRate?: number;
playerSampleRate?: number;
};
export interface WebSocketTransportConstructorOptions extends WebSocketTransportOptions {
mediaManager?: MediaManager;
}
```
#### Properties
URL of the WebSocket server. This is the endpoint your client will connect to
for WebSocket communication.
The serializer to use for encoding/decoding messages sent over the WebSocket
connection. The websocket-transport package provides two serializer options: -
`ProtobufFrameSerializer`: Uses Protocol Buffers for serialization. -
`TwilioSerializer`: Uses Twilio's serialization format. The main purpose of
the TwilioSerializer is to allow testing the bots built to work with Twilio
without having to make phone calls.
Sample rate for which to encode the audio input. Default is `16000`.
Sample rate for which to decode the incoming audio for output. Default is
`24000`.
The media manager to use for handling local audio and video streams. This
should not be overridden unless you have a specific reason to use a different
media manager. The default is `DailyMediaManager`, which is suitable for most
use cases. Note that the `DailyMediaManager` does not use any of Daily's
services, it simply takes advantage of vast media support provided by the
Daily library.
### TransportConnectionParams
The `WebSocketTransport` takes the same options as the constructor; `WebSocketTransportOptions`. Anything provided here will override the defaults set in the constructor. The `wsUrl` is required to establish a connection.
```typescript client theme={null}
pcClient.connect({
wsUrl: 'http://localhost:7860/ws'
});
// OR...
pcClient.startBotAndConnect({
endpoint: '/api/start', // returns { wsUrl }
});
```
```python server theme={null}
# See
# https://github.com/pipecat-ai/pipecat-examples/blob/main/websocket/server/server.py
# for a complete example of how to implement the server-side endpoint.
@app.websocket("/ws")
async def websocket_endpoint(websocket: WebSocket):
await websocket.accept()
print("WebSocket connection accepted")
try:
await run_bot(websocket)
except Exception as e:
print(f"Exception in run_bot: {e}")
@app.post("/api/start")
async def start(request: Request) -> Dict[Any, Any]:
ws_url = "ws://localhost:7860/ws"
return {"wsUrl": ws_url}
```
```python bot theme={null}
# See
# https://github.com/pipecat-ai/pipecat-examples/blob/main/websocket/server/bot_websocket_server.py
# for a complete example of a bot script using the WebSocketTransport.
from pipecat.serializers.protobuf import ProtobufFrameSerializer
from pipecat.transports.websocket.fastapi import (
FastAPIWebsocketParams,
FastAPIWebsocketTransport,
)
async def run_bot(websocket_client):
ws_transport = FastAPIWebsocketTransport(
websocket=websocket_client,
params=FastAPIWebsocketParams(
audio_in_enabled=True,
audio_out_enabled=True,
add_wav_header=False,
serializer=ProtobufFrameSerializer(),
),
)
llm = ... # Initialize your LLM here, e.g., OpenAI, HuggingFace, etc.
messages = [{ role: "system", content: "You are a helpful assistant." }]
context = LLMContext(messages)
user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
context,
user_params=LLMUserAggregatorParams(
vad_analyzer=SileroVADAnalyzer(),
),
)
pipeline = Pipeline(
[
ws_transport.input(),
user_aggregator,
llm, # LLM
ws_transport.output(),
assistant_aggregator,
]
)
worker = PipelineWorker(
pipeline,
params,
)
...
```
### Methods
For most operations, you will not interact with the transport directly. Most methods have an equivalent in the `PipecatClient` and should be called from the `PipecatClient`. However, there is one transport-specific methods that you may need to call directly. When doing so, be sure to access your transport via the `transport` property of the `PipecatClient` instance.
If implementing your own serializer, you will need to pass the user audio
stream to the transport via this method, which takes an `ArrayBuffer` of audio
data.
```javascript theme={null}
transport.handleUserAudioStream(chunk.data);
```
## Events
The transport implements the various [`PipecatClient` event handlers](/api-reference/client/js/callbacks).
## Reconnection Handling
The WebSocketTransport does provide reconnection handling. If the WebSocket connection is lost, it will attempt to reconnect twice. If all reconnection attempts fail, the transport will gracefully disconnect.
## More Information
Basic Agent example using a WebSocket transport
Example using a WebSocket transport to simulate a Twilio connection to a bot
`WebSocketTransport`
`@pipecat-ai/websocket-transport`
# API Reference
Source: https://docs.pipecat.ai/api-reference/client/react-native/api-reference
API reference for the Pipecat React Native SDK
The Pipecat React Native SDK leverages the Pipecat JavaScript SDK for seamless integration with React Native applications.
For detailed information, please reference to the [Javascript SDK docs](/api-reference/client/js/client-constructor).
**Just ensure you use the appropriate transport layer for React Native.**
# React Native SDK Overview
Source: https://docs.pipecat.ai/api-reference/client/react-native/overview
Build React Native applications with Pipecat's React Native client library
The Pipecat React Native SDK leverages the [Pipecat JavaScript SDK](/api-reference/client/js/overview) and its `PipecatClient` to provide seamless integration for React Native applications.
Since the JavaScript SDK is designed to work across both web and React Native platforms, the core functionalities remain the same:
* Device and media stream management
* Connecting to Pipecat bots
* Messaging with Pipecat bots and handling responses using the RTVI standard
* Managing session state and errors
The primary difference lies in the transport layer, which is tailored to support the unique requirements of the React Native environment.
For example, when using the SDK with React Native, you would install `RNDailyTransport` instead of `DailyTransport`.
## Installation
Install the SDK and a transport implementation. Follow the appropriate docs for each transport:
* [Daily](https://github.com/pipecat-ai/pipecat-client-react-native-transports/tree/main/transports/daily#installation)
* [SmallWebRTC](https://github.com/pipecat-ai/pipecat-client-react-native-transports/tree/main/transports/smallwebrtc#installation)
Installing the React Native automatically includes the corresponding version
of the JavaScript SDK.
## Requirements
This package introduces some constraints on what OS/SDK versions your project can support:
* iOS: Deployment target >= 15
* Android: `minSdkVersion` >= 24
## Quick start
Here's a simple example using Daily as the transport layer:
```tsx theme={null}
import { RNDailyTransport } from "@pipecat-ai/react-native-daily-transport";
import { PipecatClient } from "@pipecat-ai/client-js";
// Create and configure the client
let pipecatClient = new PipecatClient({
transport: new RNDailyTransport(),
enableMic: true,
enableCam: false,
});
// Connect to your bot
await pipecatClient.startBotAndConnect({
endpoint: `${process.env.PIPECAT_API_URL || "/start"}`,
});
```
### More Examples
A basic example demonstrating how to integrate an RNDailyTransport with a
React Native project.
A more comprehensive Daily example showcasing a more feature-rich React
Native application along with a server-side bot component.
A more comprehensive SmallWebRTC example showcasing a more feature-rich
React Native application along with a server-side bot component.
## Explore the SDK
The Pipecat React Native SDK leverages the Pipecat JavaScript SDK for seamless integration with React Native applications. For detailed information, refer to our JavaScript documentation.
> Just ensure you use the appropriate transport layer for React Native.
React Native-specific API documentation
Daily and SmallWebRTC transports for React Native
Configure your client instance with transport and callbacks
Core methods for interacting with your bot
# Daily WebRTC Transport
Source: https://docs.pipecat.ai/api-reference/client/react-native/transports/daily
The DailyTransport class provides a WebRTC transport layer using [Daily.co's](https://daily.co) infrastructure. It wraps a React-Native-Daily-JS call client to handle audio/video device management, WebRTC connections, and real-time communication between clients and bots. For complete documentation on Daily's API, see the [Daily RN API Reference](https://docs.daily.co/reference/rn-daily-js).
This transport is designed for production use cases, leveraging Daily's global infrastructure for low-latency, high-quality audio and video streaming. It expects your Pipecat server to include the corresponding [`DailyTransport` server-side](/api-reference/server/services/transport/daily) implementation.
## Usage
### Basic Setup
```javascript theme={null}
import { RNDailyTransport } from "@pipecat-ai/react-native-daily-transport";
import { PipecatClient } from "@pipecat-ai/client-js";
// Create and configure the client
let pipecatClient = new PipecatClient({
transport: new RNDailyTransport(),
enableMic: true,
enableCam: false,
});
// Connect to your bot
await pipecatClient.startBotAndConnect({
endpoint: `${process.env.PIPECAT_API_URL || "/start"}`,
});
```
## API Reference
### Constructor Options
The `DailyTransportConstructorOptions` extends the `DailyFactoryOptions` type that is accepted by the underlying Daily instance. These options are passed directly through to the Daily constructor. See the [Daily RN API Reference](https://docs.daily.co/reference/rn-daily-js/daily-call-client/properties) for a complete list of options.
The Pipecat React Native SDK leverages the Pipecat JavaScript SDK for seamless integration with React Native applications.
For detailed information, please reference to the [Javascript SDK docs](/api-reference/client/js/transports/daily).
**Just ensure you use the appropriate transport layer for React Native.**
## More Information
Simple Chatbot Demo
`RNDailyTransport`
`@pipecat-ai/react-native-daily-transport`
# SmallWebRTCTransport
Source: https://docs.pipecat.ai/api-reference/client/react-native/transports/small-webrtc
A lightweight WebRTC transport for peer-to-peer connections with Pipecat
`SmallWebRTCTransport` enables peer-to-peer WebRTC connections between clients and your Pipecat application. It implements bidirectional audio and video streaming using WebRTC for real-time communication.
This transport is intended for lightweight implementations, particularly for local development and testing. It expects your Pipecat server to include the corresponding [`SmallWebRTCTransport` server-side](/api-reference/server/services/transport/small-webrtc) implementation.
## Usage
### Basic Setup
```javascript theme={null}
import { PipecatClient } from "@pipecat-ai/client-js";
import {
RNSmallWebRTCTransport,
SmallWebRTCTransportConstructorOptions,
} from '@pipecat-ai/react-native-small-webrtc-transport';
import { DailyMediaManager } from '@pipecat-ai/react-native-daily-media-manager/src';
const options: SmallWebRTCTransportConstructorOptions = {
mediaManager: new DailyMediaManager()
};
const pcClient = new PipecatClient({
transport: new RNSmallWebRTCTransport(options),
enableCam: false, // Default camera off
enableMic: true, // Default microphone on
callbacks: {
// Event handlers
},
});
const connectParams: APIRequest = {
endpoint: baseUrl + '/start'
};
await client?.startBotAndConnect(connectParams);
```
## API Reference
The Pipecat React Native SDK leverages the Pipecat JavaScript SDK for seamless integration with React Native applications.
For detailed information, please reference to the [Javascript SDK docs](/api-reference/client/js/transports/small-webrtc).
**Just ensure you use the appropriate transport layer for React Native.**
## More Information
Real-time video transformation example
`@pipecat-ai/small-webrtc-transport`
# Components
Source: https://docs.pipecat.ai/api-reference/client/react/components
Ready-to-use React components for Pipecat applications
The Pipecat React SDK provides several components for handling audio, video, and visualization in your application.
## PipecatClientProvider
The root component for providing Pipecat client context to your application. It also includes built-in conversation state management, so any descendant component can use the [`usePipecatConversation`](/api-reference/client/react/hooks#usepipecatconversation) hook to access messages without adding a separate provider.
```jsx theme={null}
{/* Child components can use usePipecatConversation, usePipecatClient, etc. */}
```
**Props**
A singleton instance of `PipecatClient`
## PipecatClientAudio
Creates a new `