Build conversational AI voice agents with ElevenLabs Platform using React, JavaScript, React Native, or Swift SDKs. Configure agents, tools (client/server/MCP), RAG knowledge bases, multi-voice, and...
ElevenLabs Agents Platform is a comprehensive solution for building production-ready conversational AI voice agents. The platform coordinates four core components:
ElevenLabs migrated to new scoped packages in August 2025:
DEPRECATED (Do not use):
@11labs/react → DEPRECATED@11labs/client → DEPRECATEDCurrent packages:
npm install @elevenlabs/react@0.9.1 # React SDK
npm install @elevenlabs/client@0.9.1 # JavaScript SDK
npm install @elevenlabs/react-native@0.5.2 # React Native SDK
npm install @elevenlabs/elevenlabs-js@2.21.0 # Base SDK
npm install -g @elevenlabs/agents-cli@0.2.0 # CLI
If you have old packages installed, uninstall them first:
npm uninstall @11labs/react @11labs/client
Use this skill when:
Design & Configure:
Connect & Deploy:
@elevenlabs/react)@elevenlabs/client)@elevenlabs/react-native)Operate & Optimize:
For building voice chat interfaces in React applications.
Installation:
npm install @elevenlabs/react zod
Basic Example:
import { useConversation } from '@elevenlabs/react';
import { z } from 'zod';
export default function VoiceChat() {
const { startConversation, stopConversation, status } = useConversation({
// Public agent (no API key needed)
agentId: 'your-agent-id',
// OR private agent (requires API key)
apiKey: process.env.NEXT_PUBLIC_ELEVENLABS_API_KEY,
// OR signed URL (server-generated, most secure)
signedUrl: '/api/elevenlabs/auth',
// Client-side tools (browser functions)
clientTools: {
updateCart: {
description: "Update the shopping cart",
parameters: z.object({
item: z.string(),
quantity: z.number()
}),
handler: async ({ item, quantity }) => {
console.log('Updating cart:', item, quantity);
return { success: true };
}
}
},
// Event handlers
onConnect: () => console.log('Connected'),
onDisconnect: () => console.log('Disconnected'),
onEvent: (event) => {
switch (event.type) {
case 'transcript':
console.log('User said:', event.data.text);
break;
case 'agent_response':
console.log('Agent replied:', event.data.text);
break;
}
},
// Regional compliance (GDPR, data residency)
serverLocation: 'us' // 'us' | 'global' | 'eu-residency' | 'in-residency'
});
return (
<div>
<button onClick={startConversation}>Start Conversation</button>
<button onClick={stopConversation}>Stop</button>
<p>Status: {status}</p>
</div>
);
}
For managing agents via code with version control and CI/CD.
Installation:
npm install -g @elevenlabs/agents-cli
# or
pnpm install -g @elevenlabs/agents-cli
Workflow:
# 1. Authenticate
elevenlabs auth login
# 2. Initialize project (creates agents.json, tools.json, tests.json)
elevenlabs agents init
# 3. Create agent from template
elevenlabs agents add "Support Agent" --template customer-service
# 4. Configure in agent_configs/support-agent.json
# 5. Push to platform
elevenlabs agents push --env dev
# 6. Test
elevenlabs agents test "Support Agent"
# 7. Deploy to production
elevenlabs agents push --env prod
Project Structure Created:
your_project/
├── agents.json # Agent registry
├── tools.json # Tool configurations
├── tests.json # Test configurations
├── agent_configs/ # Individual agent files
├── tool_configs/ # Tool configuration files
└── test_configs/ # Test configuration files
For creating agents dynamically (multi-tenant, SaaS platforms).
Installation:
npm install elevenlabs
Example:
import { ElevenLabsClient } from 'elevenlabs';
const client = new ElevenLabsClient({
apiKey: process.env.ELEVENLABS_API_KEY
});
// Create agent
const agent = await client.agents.create({
name: 'Support Bot',
conversation_config: {
agent: {
prompt: {
prompt: "You are a helpful customer support agent.",
llm: "gpt-4o",
temperature: 0.7
},
first_message: "Hello! How can I help you today?",
language: "en"
},
tts: {
model_id: "eleven_turbo_v2_5",
voice_id: "your-voice-id"
}
}
});
console.log('Agent created:', agent.agent_id);
ElevenLabs recommends structuring agent prompts using 6 components:
Define the agent's identity, role, and character traits.
Example:
You are Alex, a friendly and knowledgeable customer support specialist at TechCorp.
You have 5 years of experience helping customers solve technical issues.
You're patient, empathetic, and always maintain a positive attitude.
Describe the communication context (phone, web chat, video call).
Example:
You're speaking with customers over the phone. Communication is voice-only.
Customers may have background noise or poor connection quality.
Speak clearly and occasionally use thoughtful pauses for emphasis.
Specify formality, speech patterns, humor, and verbosity.
Example:
Tone: Professional yet warm. Use contractions ("I'm" instead of "I am") to sound natural.
Avoid jargon unless the customer uses it first. Keep responses concise (2-3 sentences max).
Use encouraging phrases like "I'll be happy to help with that" and "Let's get this sorted for you."
Define objectives and success criteria.
Example:
Primary Goal: Resolve customer technical issues on the first call.
Secondary Goals:
- Verify customer identity securely
- Document issue details accurately
- Offer proactive solutions
- End calls with confirmation that the issue is resolved
Success Criteria: Customer verbally confirms their issue is resolved.
Set boundaries, prohibited topics, and ethical constraints.
Example:
Guardrails:
- Never provide medical, legal, or financial advice
- Do not share confidential company information
- If asked about competitors, politely redirect to TechCorp's offerings
- Escalate to a human supervisor if customer becomes abusive
- Never make promises about refunds or credits without verification
Describe available external capabilities and when to use them.
Example:
Available Tools:
1. lookup_order(order_id) - Fetch order details from database. Use when customer mentions an order number.
2. transfer_to_supervisor() - Escalate to human agent. Use when issue requires manager approval.
3. send_password_reset(email) - Trigger password reset email. Use when customer can't access account.
Always explain to the customer what you're doing before calling a tool.
Complete Template:
{
"agent": {
"prompt": {
"prompt": "Personality:\nYou are Alex, a friendly customer support specialist.\n\nEnvironment:\nYou're speaking with customers over the phone.\n\nTone:\nProfessional yet warm. Keep responses concise.\n\nGoal:\nResolve technical issues on the first call.\n\nGuardrails:\n- Never provide medical/legal/financial advice\n- Escalate abusive customers\n\nTools:\n- lookup_order(order_id) - Fetch order details\n- transfer_to_supervisor() - Escalate to human",
"llm": "gpt-4o",
"temperature": 0.7,
"max_tokens": 500
}
}
}
Controls when the agent interrupts or waits for the user to finish speaking.
3 Modes:
| Mode | Behavior | Best For |
|---|---|---|
| Eager | Responds quickly, jumps in at earliest opportunity | Fast-paced support, quick orders |
| Normal | Balanced, waits for natural conversation breaks | General customer service (default) |
| Patient | Waits longer, allows detailed user responses | Information collection, therapy, tutoring |
Configuration:
{
"conversation_config": {
"turn": {
"mode": "patient" // "eager" | "normal" | "patient"
}
}
}
Use Cases:
Gotchas:
Create branching conversation flows with subagent nodes and conditional routing.
Node Types:
Configuration:
{
"workflow": {
"nodes": [
{
"id": "node_1",
"type": "subagent",
"config": {
"system_prompt": "You are now a technical support specialist. Ask detailed diagnostic questions.",
"turn_eagerness": "patient",
"voice_id": "tech_support_voice_id"
}
},
{
"id": "node_2",
"type": "tool",
"tool_name": "transfer_to_human"
}
],
"edges": [
{
"from": "node_1",
"to": "node_2",
"condition": "user_requests_escalation"
}
]
}
}
Use Cases:
Gotchas:
Inject runtime data into prompts, first messages, and tool parameters using {{var_name}} syntax.
System Variables (Auto-Available):
{{system__agent_id}} // Current agent ID
{{system__conversation_id}} // Conversation ID
{{system__caller_id}} // Phone number (telephony only)
{{system__called_number}} // Called number (telephony only)
{{system__call_duration_secs}} // Call duration
{{system__time_utc}} // Current UTC time
{{system__call_sid}} // Twilio call SID (Twilio only)
Custom Variables:
// Provide when starting conversation
const conversation = await client.conversations.create({
agent_id: "agent_123",
dynamic_variables: {
user_name: "John",
account_tier: "premium",
order_id: "ORD-12345"
}
});
Secret Variables (For API Keys):
{{secret__stripe_api_key}}
{{secret__database_password}}
Important: Secret variables only used in headers, never sent to LLM providers.
Usage in Prompts:
{
"agent": {
"prompt": {
"prompt": "You are helping {{user_name}}, a {{account_tier}} customer."
},
"first_message": "Hello {{user_name}}! I see you're calling about order {{order_id}}."
}
}
Gotcha: Missing variables cause "Missing required dynamic variables" error. Always provide all referenced variables when starting conversation.
Option 1: Public Agents (No API Key)
const { startConversation } = useConversation({
agentId: 'your-public-agent-id' // Anyone can use
});
Option 2: Private Agents with API Key
const { startConversation } = useConversation({
agentId: 'your-private-agent-id',
apiKey: process.env.NEXT_PUBLIC_ELEVENLABS_API_KEY
});
⚠️ Warning: Never expose API keys in client-side code. Use signed URLs instead.
Option 3: Signed URLs (Recommended for Production)
// Server-side (Next.js API route)
import { ElevenLabsClient } from 'elevenlabs';
export async function POST(req: Request) {
const client = new ElevenLabsClient({
apiKey: process.env.ELEVENLABS_API_KEY // Server-side only
});
const signedUrl = await client.convai.getSignedUrl({
agent_id: 'your-agent-id'
});
return Response.json({ signedUrl });
}
// Client-side
const { startConversation } = useConversation({
agentId: 'your-agent-id',
signedUrl: await fetch('/api/elevenlabs/auth').then(r => r.json()).then(d => d.signedUrl)
});
Dynamically switch between different voices during a single conversation.
Use Cases:
Configuration:
{
"agent": {
"prompt": {
"prompt": "When speaking as the customer, use voice_id 'customer_voice_abc123'. When speaking as the agent, use voice_id 'agent_voice_def456'."
}
}
}
Gotchas:
Customize how the agent pronounces specific words or phrases.
Supported Formats:
Configuration:
{
"pronunciation_dictionary": [
{
"word": "ElevenLabs",
"pronunciation": "ɪˈlɛvənlæbz",
"format": "ipa"
},
{
"word": "API",
"pronunciation": "ey-pee-ay",
"format": "cmu"
},
{
"word": "AI",
"substitution": "artificial intelligence"
}
]
}
Use Cases:
Gotcha: Only Turbo v2/v2.5 models support phoneme-based pronunciation. Other models silently skip phoneme entries but still process word substitutions.
Adjust speaking speed dynamically (0.7x - 1.2x).
Configuration:
{
"voice_settings": {
"speed": 1.0 // 0.7 = slow, 1.0 = normal, 1.2 = fast
}
}
Use Cases:
Best Practices:
Create custom voices using ElevenLabs Voice Design tool.
Workflow:
voice_id in agent configurationVoice Cloning Best Practices:
Gotcha: Using English-trained voices for non-English languages causes pronunciation issues. Always use language-matched voices.
Support for 32+ languages with automatic detection and in-conversation switching.
Configuration:
{
"agent": {
"language": "en" // ISO 639-1 code
}
}
Multi-Language Presets (Different Voice Per Language):
{
"conversation_config": {
"language_presets": [
{
"language": "en",
"voice_id": "en_voice_id",
"first_message": "Hello! How can I help you today?"
},
{
"language": "es",
"voice_id": "es_voice_id",
"first_message": "¡Hola! ¿Cómo puedo ayudarte hoy?"
},
{
"language": "fr",
"voice_id": "fr_voice_id",
"first_message": "Bonjour! Comment puis-je vous aider aujourd'hui?"
}
]
}
}
Automatic Language Detection: Agent detects user's language and switches automatically.
Supported Languages: English, Spanish, French, German, Italian, Portuguese, Dutch, Polish, Arabic, Chinese, Japanese, Korean, Hindi, and 18+ more.
Enable agents to access large knowledge bases without loading entire documents into context.
How It Works:
Configuration:
{
"agent": {
"prompt": {
"knowledge_base": ["doc_id_1", "doc_id_2"]
}
}
}
Upload Documents via API:
import { ElevenLabsClient } from 'elevenlabs';
const client = new ElevenLabsClient({ apiKey: process.env.ELEVENLABS_API_KEY });
// Upload document
const doc = await client.knowledgeBase.upload({
file: fs.createReadStream('support_docs.pdf'),
name: 'Support Documentation'
});
// Compute RAG index
await client.knowledgeBase.computeRagIndex({
document_id: doc.id,
embedding_model: 'e5_mistral_7b' // or 'multilingual_e5_large'
});
Retrieval Configuration:
{
"knowledge_base_config": {
"max_chunks": 5, // Number of chunks to retrieve
"vector_distance_threshold": 0.8 // Similarity threshold
}
}
Use Cases:
Gotchas:
ElevenLabs supports 4 distinct tool types, each with different execution patterns.
Execute operations on the client side (browser or mobile app).
Use Cases:
React Example:
import { useConversation } from '@elevenlabs/react';
import { z } from 'zod';
const { startConversation } = useConversation({
clientTools: {
updateCart: {
description: "Update the shopping cart with new items",
parameters: z.object({
item: z.string().describe("The item name"),
quantity: z.number().describe("Quantity to add")
}),
handler: async ({ item, quantity }) => {
// Client-side logic
const cart = getCart();
cart.add(item, quantity);
updateUI(cart);
return { success: true, total: cart.total };
}
},
navigate: {
description: "Navigate to a different page",
parameters: z.object({
url: z.string().describe("The URL to navigate to")
}),
handler: async ({ url }) => {
window.location.href = url;
return { success: true };
}
}
}
});
Gotchas:
Make HTTP requests to external APIs from ElevenLabs servers.
Use Cases:
Configuration via CLI:
elevenlabs tools add-webhook "Get Weather" --config-path tool_configs/get-weather.json
tool_configs/get-weather.json:
{
"name": "get_weather",
"description": "Fetch current weather for a city",
"url": "https://api.weather.com/v1/current",
"method": "GET",
"parameters": {
"type": "object",
"properties": {
"city": {
"type": "string",
"description": "The city name (e.g., 'London', 'New York')"
}
},
"required": ["city"]
},
"headers": {
"Authorization": "Bearer {{secret__weather_api_key}}"
}
}
Dynamic Variables in Tools:
{
"url": "https://api.crm.com/customers/{{user_id}}",
"headers": {
"X-API-Key": "{{secret__crm_api_key}}"
}
}
Gotchas:
Connect to external MCP servers for standardized tool access.
Use Cases:
Configuration:
Approval Modes:
Gotchas:
Example: Using ElevenLabs MCP Server in Claude Desktop:
{
"mcpServers": {
"ElevenLabs": {
"command": "uvx",
"args": ["elevenlabs-mcp"],
"env": {
"ELEVENLABS_API_KEY": "<your-key>",
"ELEVENLABS_MCP_OUTPUT_MODE": "files"
}
}
}
}
Modify the internal state of the conversation without external calls.
Use Cases:
Built-in System Tools:
end_call - End the conversationdetect_language - Detect user's languagetransfer_agent - Switch to different agent/workflow nodetransfer_to_number - Transfer to external phone number (telephony only)dtmf_playpad - Display DTMF keypad (telephony only)voicemail_detection - Detect voicemail (telephony only)Configuration:
{
"system_tools": [
{
"name": "update_conversation_state",
"description": "Update the conversation context with new information",
"parameters": {
"key": { "type": "string" },
"value": { "type": "string" }
}
}
]
}
Gotchas:
@elevenlabs/react)Installation:
npm install @elevenlabs/react zod
Complete Example:
import { useConversation } from '@elevenlabs/react';
import { z } from 'zod';
import { useState } from 'react';
export default function VoiceAgent() {
const [transcript, setTranscript] = useState<string[]>([]);
const {
startConversation,
stopConversation,
status,
isSpeaking
} = useConversation({
agentId: 'your-agent-id',
// Authentication (choose one)
apiKey: process.env.NEXT_PUBLIC_ELEVENLABS_API_KEY,
// Client tools
clientTools: {
updateCart: {
description: "Update shopping cart",
parameters: z.object({
item: z.string(),
quantity: z.number()
}),
handler: async ({ item, quantity }) => {
console.log('Cart updated:', item, quantity);
return { success: true };
}
}
},
// Events
onConnect: () => {
console.log('Connected to agent');
setTranscript([]);
},
onDisconnect: () => console.log('Disconnected'),
onEvent: (event) => {
if (event.type === 'transcript') {
setTranscript(prev => [...prev, `User: ${event.data.text}`]);
} else if (event.type === 'agent_response') {
setTranscript(prev => [...prev, `Agent: ${event.data.text}`]);
}
},
onError: (error) => console.error('Error:', error),
// Regional compliance
serverLocation: 'us'
});
return (
<div>
<div>
<button onClick={startConversation} disabled={status === 'connected'}>
Start Conversation
</button>
<button onClick={stopConversation} disabled={status !== 'connected'}>
Stop
</button>
</div>
<div>Status: {status}</div>
<div>{isSpeaking && 'Agent is speaking...'}</div>
<div>
<h3>Transcript</h3>
{transcript.map((line, i) => (
<p key={i}>{line}</p>
))}
</div>
</div>
);
}
@elevenlabs/client)For vanilla JavaScript projects (no React).
Installation:
npm install @elevenlabs/client
Example:
import { Conversation } from '@elevenlabs/client';
const conversation = new Conversation({
agentId: 'your-agent-id',
apiKey: process.env.ELEVENLABS_API_KEY,
onConnect: () => console.log('Connected'),
onDisconnect: () => console.log('Disconnected'),
onEvent: (event) => {
switch (event.type) {
case 'transcript':
document.getElementById('user-text').textContent = event.data.text;
break;
case 'agent_response':
document.getElementById('agent-text').textContent = event.data.text;
break;
}
}
});
// Start conversation
document.getElementById('start-btn').addEventListener('click', async () => {
await conversation.start();
});
// Stop conversation
document.getElementById('stop-btn').addEventListener('click', async () => {
await conversation.stop();
});
ElevenLabs SDKs support two connection types with different characteristics.
Comparison Table:
| Feature | WebSocket | WebRTC |
|---|---|---|
| Authentication | signedUrl |
conversationToken |
| Audio Format | Configurable | PCM_48000 (hardcoded) |
| Sample Rate | Configurable (16k, 24k, 48k) | 48000 (hardcoded) |
| Latency | Standard | Lower |
| Device Switching | Flexible | Limited (format locked) |
| Best For | General use, flexibility | Low-latency requirements |
WebSocket Configuration (default):
import { useConversation } from '@elevenlabs/react';
const { startConversation } = useConversation({
agentId: 'your-agent-id',
// WebSocket uses signed URL
signedUrl: async () => {
const response = await fetch('/api/elevenlabs/auth');
const { signedUrl } = await response.json();
return signedUrl;
},
// Connection type (optional, defaults to 'websocket')
connectionType: 'websocket',
// Audio config (flexible)
audioConfig: {
sampleRate: 24000, // 16000, 24000, or 48000
format: 'PCM_24000'
}
});
WebRTC Configuration:
const { startConversation } = useConversation({
agentId: 'your-agent-id',
// WebRTC uses conversation token (different auth flow)
conversationToken: async () => {
const response = await fetch('/api/elevenlabs/token');
const { token } = await response.json();
return token;
},
// Connection type
connectionType: 'webrtc',
// Audio format is HARDCODED to PCM_48000 (not configurable)
// audioConfig ignored for WebRTC
});
Backend Token Endpoints:
// WebSocket signed URL (GET /v1/convai/conversation/get-signed-url)
app.get('/api/elevenlabs/auth', async (req, res) => {
const response = await fetch(
`https://api.elevenlabs.io/v1/convai/conversation/get-signed-url?agent_id=${AGENT_ID}`,
{ headers: { 'xi-api-key': ELEVENLABS_API_KEY } }
);
const { signed_url } = await response.json();
res.json({ signedUrl: signed_url });
});
// WebRTC conversation token (GET /v1/convai/conversation/token)
app.get('/api/elevenlabs/token', async (req, res) => {
const response = await fetch(
`https://api.elevenlabs.io/v1/convai/conversation/token?agent_id=${AGENT_ID}`,
{ headers: { 'xi-api-key': ELEVENLABS_API_KEY } }
);
const { conversation_token } = await response.json();
res.json({ token: conversation_token });
});
When to Use Each:
| Use WebSocket When | Use WebRTC When |
|---|---|
| Need flexible audio formats | Need lowest possible latency |
| Switching between audio devices frequently | Audio format can be locked to 48kHz |
| Standard latency is acceptable | Building real-time applications |
| Need maximum configuration control | Performance is critical |
Gotchas:
Installation:
npx expo install @elevenlabs/react-native @livekit/react-native @livekit/react-native-webrtc livekit-client
Requirements:
Example:
import { useConversation } from '@elevenlabs/react-native';
import { View, Button, Text } from 'react-native';
import { z } from 'zod';
export default function App() {
const { startConversation, stopConversation, status } = useConversation({
agentId: 'your-agent-id',
signedUrl: 'https://api.elevenlabs.io/v1/convai/auth/...',
clientTools: {
updateProfile: {
description: "Update user profile",
parameters: z.object({
name: z.string()
}),
handler: async ({ name }) => {
console.log('Updating profile:', name);
return { success: true };
}
}
}
});
return (
<View style={{ padding: 20 }}>
<Button title="Start" onPress={startConversation} />
<Button title="Stop" onPress={stopConversation} />
<Text>Status: {status}</Text>
</View>
);
}
Gotchas:
Installation (Swift Package Manager):
dependencies: [
.package(url: "https://github.com/elevenlabs/elevenlabs-swift-sdk", from: "1.0.0")
]
Requirements:
Use Cases:
Installation: Copy-paste embed code from dashboard or use this template:
<script src="https://elevenlabs.io/convai-widget/index.js"></script>
<script>
ElevenLabsWidget.init({
agentId: 'your-agent-id',
// Theming
theme: {
primaryColor: '#3B82F6',
backgroundColor: '#1F2937',
textColor: '#F9FAFB'
},
// Position
position: 'bottom-right', // 'bottom-left' | 'bottom-right'
// Custom branding
branding: {
logo: 'https://example.com/logo.png',
name: 'Support Agent'
}
});
</script>
Use Cases:
Status: Closed Beta (requires sales contact) Release: 2025
Scribe is ElevenLabs' real-time speech-to-text service for low-latency transcription.
Capabilities:
Authentication: Uses single-use tokens (not API keys):
// Fetch token from backend
const response = await fetch('/api/scribe/token');
const { token } = await response.json();
// Backend endpoint
const token = await client.scribe.getToken();
return { token };
React Hook (useScribe):
import { useScribe } from '@elevenlabs/react';
export default function Transcription() {
const {
connect,
disconnect,
startRecording,
stopRecording,
status,
transcript,
partialTranscript
} = useScribe({
token: async () => {
const response = await fetch('/api/scribe/token');
const { token } = await response.json();
return token;
},
// Commit strategy
commitStrategy: 'vad', // 'vad' (automatic) or 'manual'
// Audio format
sampleRate: 16000, // 16000 or 24000
// Events
onConnect: () => console.log('Connected to Scribe'),
onDisconnect: () => console.log('Disconnected'),
onPartialTranscript: (text) => {
console.log('Interim:', text);
},
onFinalTranscript: (text, timestamps) => {
console.log('Final:', text);
console.log('Timestamps:', timestamps); // Word-level timing
},
onError: (error) => console.error('Error:', error)
});
return (
<div>
<button onClick={connect}>Connect</button>
<button onClick={startRecording}>Start Recording</button>
<button onClick={stopRecording}>Stop Recording</button>
<button onClick={disconnect}>Disconnect</button>
<p>Status: {status}</p>
<p>Partial: {partialTranscript}</p>
<p>Final: {transcript}</p>
</div>
);
}
JavaScript SDK (Scribe.connect):
import { Scribe } from '@elevenlabs/client';
const connection = await Scribe.connect({
token: 'your-single-use-token',
sampleRate: 16000,
commitStrategy: 'vad',
onPartialTranscript: (text) => {
document.getElementById('interim').textContent = text;
},
onFinalTranscript: (text, timestamps) => {
const finalDiv = document.getElementById('final');
finalDiv.textContent += text + ' ';
// timestamps: [{ word: 'hello', start: 0.5, end: 0.8 }, ...]
},
onError: (error) => {
console.error('Scribe error:', error);
}
});
// Start recording from microphone
await connection.startRecording();
// Stop recording
await connection.stopRecording();
// Manual commit (if commitStrategy: 'manual')
await connection.commit();
// Disconnect
await connection.disconnect();
Transcribing Pre-Recorded Files:
import { Scribe } from '@elevenlabs/client';
const connection = await Scribe.connect({ token });
// Send audio buffer
const audioBuffer = fs.readFileSync('recording.pcm');
await connection.sendAudioData(audioBuffer);
// Manually commit to get final transcript
await connection.commit();
// Wait for final transcript event
Event Types:
SESSION_STARTED: Connection establishedPARTIAL_TRANSCRIPT: Interim transcription (unbuffered)FINAL_TRANSCRIPT: Complete sentence/phraseFINAL_TRANSCRIPT_WITH_TIMESTAMPS: Final + word timingERROR: Transcription errorAUTH_ERROR: Authentication failedOPEN: WebSocket openedCLOSE: WebSocket closedCommit Strategies:
| Strategy | Description | Use When |
|---|---|---|
vad (automatic) |
Voice Activity Detection auto-commits on silence | Real-time transcription |
manual |
Call connection.commit() explicitly |
Pre-recorded files, controlled commits |
Audio Formats:
PCM_16000 (16kHz, 16-bit PCM)PCM_24000 (24kHz, 16-bit PCM)Gotchas:
When to Use Scribe:
When NOT to Use: Use Agents Platform instead if you need:
Simulate full conversations and evaluate against success criteria.
Configuration via CLI:
elevenlabs tests add "Refund Request Test" --template basic-llm
test_configs/refund-request-test.json:
{
"name": "Refund Request Test",
"scenario": "Customer requests refund for defective product",
"user_input": "I want a refund for order #12345. The product was broken when it arrived.",
"success_criteria": [
"Agent acknowledges the request empathetically",
"Agent asks for order number (which was already provided)",
"Agent verifies order details",
"Agent provides refund timeline or next steps"
],
"evaluation_type": "llm"
}
Run Test:
elevenlabs agents test "Support Agent"
Verify that agents correctly use tools with the right parameters.
Configuration:
{
"name": "Account Balance Test",
"scenario": "Customer requests account balance",
"expected_tool_call": {
"tool_name": "get_account_balance",
"parameters": {
"account_id": "ACC-12345"
}
}
}
Test agent capacity under high concurrency.
Configuration:
# Spawn 100 users, 1 per second, test for 10 minutes
elevenlabs test load \
--users 100 \
--spawn-rate 1 \
--duration 600
Gotchas:
API Endpoint:
POST /v1/convai/agents/:agent_id/simulate
Example:
const simulation = await client.agents.simulate({
agent_id: 'agent_123',
scenario: 'Customer requests refund',
user_messages: [
"I want a refund for order #12345",
"I ordered it last week",
"Yes, please process it"
],
success_criteria: [
"Agent acknowledges request",
"Agent asks for order details",
"Agent provides refund timeline"
]
});
console.log('Test passed:', simulation.passed);
console.log('Criteria met:', simulation.evaluation.criteria_met);
Use Cases:
Extract structured data from conversation transcripts.
Features:
{
"evaluation_criteria": {
"resolution": "Was the customer's issue resolved?",
"sentiment": "Was the conversation tone positive?",
"compliance": "Did the agent follow company policies?"
}
}
{
"data_collection": {
"fields": [
{ "name": "customer_name", "type": "string" },
{ "name": "issue_type", "type": "enum", "values": ["billing", "technical", "other"] },
{ "name": "satisfaction", "type": "number", "range": [1, 5] }
]
}
}
Access:
Metrics:
Access: Dashboard → Analytics tab
Default: 2 years (GDPR-compliant)
Configuration:
{
"privacy": {
"transcripts": {
"retention_days": 730 // 2 years (GDPR)
},
"audio": {
"retention_days": 2190 // 6 years (HIPAA)
}
}
}
Compliance Recommendations:
Regional Configuration:
const { startConversation } = useConversation({
serverLocation: 'eu-residency' // 'us' | 'global' | 'eu-residency' | 'in-residency'
});
For maximum privacy, enable zero retention to immediately delete all conversation data.
Limitations:
Reduce costs by caching repeated inputs.
How It Works:
input_cache_write)input_cache_read)Configuration:
{
"llm_config": {
"caching": {
"enabled": true,
"ttl_seconds": 3600 // 1 hour
}
}
}
Use Cases:
Savings: Up to 90% on cached inputs
Switch between models based on cost/performance needs.
Available Models:
Configuration:
{
"llm_config": {
"model": "gpt-4o-mini" // Swap anytime via dashboard or API
}
}
Temporarily exceed concurrency limits during high-demand periods.
How It Works:
Configuration:
{
"call_limits": {
"burst_pricing_enabled": true
}
}
Use Cases:
Gotchas:
Real-time event streaming for live transcription, agent responses, and tool calls.
Event Types:
audio - Audio stream chunkstranscript - Real-time transcriptionagent_response - Agent's text responsetool_call - Tool execution statusconversation_state - State updatesExample:
const { startConversation } = useConversation({
onEvent: (event) => {
switch (event.type) {
case 'transcript':
console.log('User said:', event.data.text);
break;
case 'agent_response':
console.log('Agent replied:', event.data.text);
break;
case 'tool_call':
console.log('Tool called:', event.data.tool_name);
break;
}
}
});
Use your own OpenAI API key or custom LLM server.
Configuration:
{
"llm_config": {
"custom": {
"endpoint": "https://api.openai.com/v1/chat/completions",
"api_key": "{{secret__openai_api_key}}",
"model": "gpt-4"
}
}
}
Use Cases:
Gotchas:
Receive notifications when a call ends and analysis completes.
Configuration:
{
"webhooks": {
"post_call": {
"url": "https://api.example.com/webhook",
"headers": {
"Authorization": "Bearer {{secret__webhook_auth_token}}"
}
}
}
}
Payload:
{
"conversation_id": "conv_123",
"agent_id": "agent_456",
"transcript": "...",
"duration_seconds": 120,
"analysis": {
"sentiment": "positive",
"resolution": true,
"extracted_data": {
"customer_name": "John Doe",
"issue_type": "billing"
}
}
}
Security (HMAC Verification):
import crypto from 'crypto';
export async function POST(req: Request) {
const signature = req.headers.get('elevenlabs-signature');
const payload = await req.text();
const hmac = crypto
.createHmac('sha256', process.env.WEBHOOK_SECRET!)
.update(payload)
.digest('hex');
if (signature !== hmac) {
return new Response('Invalid signature', { status: 401 });
}
// Process webhook
const data = JSON.parse(payload);
console.log('Conversation ended:', data.conversation_id);
// MUST return 200
return new Response('OK', { status: 200 });
}
Gotchas:
Disable voice, use text-only conversations.
Configuration:
{
"conversation_config": {
"chat_mode": true // Disables audio input/output
}
}
Benefits:
Use Cases:
SIP Trunking:
SIP Endpoint: sip-static.rtc.elevenlabs.io
TLS Transport: Recommended for production
SRTP Encryption: Supported
Supported Providers: Twilio, Vonage, RingCentral, Sinch, Infobip, Telnyx, Exotel, Plivo, Bandwidth
Native Twilio Integration:
{
"telephony": {
"provider": "twilio",
"phone_number": "+1234567890",
"account_sid": "{{secret__twilio_account_sid}}",
"auth_token": "{{secret__twilio_auth_token}}"
}
}
Use Cases:
# Install globally
npm install -g @elevenlabs/cli
# Authenticate
elevenlabs auth login
# Set residency (for GDPR compliance)
elevenlabs auth residency eu-residency # or 'in-residency' | 'global'
# Check current user
elevenlabs auth whoami
Environment Variables (For CI/CD):
export ELEVENLABS_API_KEY=your-api-key
Initialize Project:
elevenlabs agents init
Directory Structure Created:
your_project/
├── agents.json # Agent registry
├── tools.json # Tool configurations
├── tests.json # Test configurations
├── agent_configs/ # Individual agent files (.json)
├── tool_configs/ # Tool configuration files
└── test_configs/ # Test configuration files
# Create agent
elevenlabs agents add "Support Agent" --template customer-service
# Deploy to platform
elevenlabs agents push
elevenlabs agents push --agent "Support Agent"
elevenlabs agents push --env prod
elevenlabs agents push --dry-run # Preview changes
# Import existing agents
elevenlabs agents pull
# List agents
elevenlabs agents list
# Check sync status
elevenlabs agents status
# Delete agent
elevenlabs agents delete <agent_id>
# Create webhook tool
elevenlabs tools add-webhook "Get Weather" --config-path tool_configs/get-weather.json
# Create client tool
elevenlabs tools add-client "Update Cart" --config-path tool_configs/update-cart.json
# Deploy tools
elevenlabs tools push
# Import existing tools
elevenlabs tools pull
# Delete tools
elevenlabs tools delete <tool_id>
elevenlabs tools delete --all
# Create test
elevenlabs tests add "Refund Test" --template basic-llm
# Deploy tests
elevenlabs tests push
# Import tests
elevenlabs tests pull
# Run test
elevenlabs agents test "Support Agent"
Pattern:
# Development
elevenlabs agents push --env dev
# Staging
elevenlabs agents push --env staging
# Production (with confirmation)
elevenlabs agents push --env prod --dry-run
# Review changes...
elevenlabs agents push --env prod
Environment-Specific Configs:
agent_configs/
├── support-bot.json # Base config
├── support-bot.dev.json # Dev overrides
├── support-bot.staging.json # Staging overrides
└── support-bot.prod.json # Prod overrides
GitHub Actions Example:
name: Deploy Agent
on:
push:
branches: [main]
paths:
- 'agent_configs/**'
- 'tool_configs/**'
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Install CLI
run: npm install -g @elevenlabs/cli
- name: Test Configs
run: elevenlabs agents push --dry-run --env prod
env:
ELEVENLABS_API_KEY: ${{ secrets.ELEVENLABS_API_KEY_PROD }}
- name: Deploy
run: elevenlabs agents push --env prod
env:
ELEVENLABS_API_KEY: ${{ secrets.ELEVENLABS_API_KEY_PROD }}
Commit:
agent_configs/*.jsontool_configs/*.jsontest_configs/*.jsonagents.json, tools.json, tests.jsonIgnore:
# .gitignore
.env
.elevenlabs/
*.secret.json
Symptom: "Missing required dynamic variables" error, no transcript generated
Cause: Dynamic variables referenced in prompts/messages but not provided at conversation start
Solution:
const conversation = await client.conversations.create({
agent_id: "agent_123",
dynamic_variables: {
user_name: "John",
account_tier: "premium",
// Provide ALL variables referenced in prompts
}
});
Symptom: Tool not executing, agent says "tool not found"
Cause: Tool name in config doesn't match registered name (case-sensitive)
Solution:
// agent_configs/bot.json
{
"agent": {
"prompt": {
"tool_ids": ["orderLookup"] // Must match exactly
}
}
}
// tool_configs/order-lookup.json
{
"name": "orderLookup" // Match case exactly
}
Symptom: Webhook auto-disabled after failures
Cause:
Solution:
// Always verify HMAC signature
import crypto from 'crypto';
const signature = req.headers['elevenlabs-signature'];
const payload = JSON.stringify(req.body);
const hmac = crypto
.createHmac('sha256', process.env.WEBHOOK_SECRET)
.update(payload)
.digest('hex');
if (signature !== hmac) {
return res.status(401).json({ error: 'Invalid signature' });
}
// Process webhook
// ...
// MUST return 200
res.status(200).json({ success: true });
Symptom: Generated audio varies in volume/tone
Cause:
Solution:
Symptom: Unpredictable pronunciation, accent issues
Cause: Using English-trained voice for non-English language
Solution:
{
"language_presets": [
{
"language": "es",
"voice_id": "spanish_trained_voice_id" // Must be Spanish-trained
}
]
}
Symptom: CLI authentication fails
Cause: Using restricted API key (not currently supported)
Solution: Use unrestricted API key for CLI operations
Symptom: Changes not reflected after push
Cause: Hash-based change detection missed modification
Solution:
# Force re-sync
elevenlabs agents init --override
elevenlabs agents pull # Re-import from platform
# Make changes
elevenlabs agents push
Symptom: Tool called but parameters empty or incorrect
Cause: Schema definition doesn't match actual usage
Solution:
// tool_configs/order-lookup.json
{
"parameters": {
"type": "object",
"properties": {
"order_id": {
"type": "string",
"description": "The order ID to look up (format: ORD-12345)" // Clear description
}
},
"required": ["order_id"]
}
}
Symptom: Agent doesn't use knowledge base
Cause: RAG index still computing (can take minutes for large documents)
Solution:
// Check index status before using
const index = await client.knowledgeBase.getRagIndex({
document_id: 'doc_123'
});
if (index.status !== 'ready') {
console.log('Index still computing...');
}
Symptom: Intermittent "protocol error" when using WebSocket connections
Cause: Network instability or incompatible browser
Solution:
Symptom: Works locally but fails in production
Cause: Agent visibility settings or API key configuration
Solution:
Symptom: "Host elevenlabs.io is not allowed to connect to this agent"
Cause: Agent has allowlist enabled but using shared link
Solution:
Symptom: Agent gets stuck in workflow, never completes
Cause: Edge conditions creating loops
Solution:
Symptom: Calls rejected during traffic spikes
Cause: Burst pricing not enabled in agent settings
Solution:
{
"call_limits": {
"burst_pricing_enabled": true
}
}
Symptom: MCP tools not responding
Cause: MCP server slow or unreachable
Solution:
Symptom: First message from agent gets cut off on Android devices (works fine on iOS/web)
Cause: Android devices need time to switch to correct audio mode after connection
Solution:
import { useConversation } from '@elevenlabs/react';
const { startConversation } = useConversation({
agentId: 'your-agent-id',
// Add connection delay for Android
connectionDelay: {
android: 3_000, // 3 seconds (default)
ios: 0, // No delay needed
default: 0 // Other platforms
},
// Rest of config...
});
Explanation:
Testing:
# Test on Android device
npm run android
# First message should now be complete
Symptom: "Refused to load the script because it violates the following Content Security Policy directive" errors in browser console
Cause: Applications with strict Content Security Policy don't allow data: or blob: URLs in script-src directive. ElevenLabs SDK uses Audio Worklets that are loaded as blobs by default.
Solution - Self-Host Worklet Files:
Step 1: Copy worklet files to your public directory:
# Copy from node_modules
cp node_modules/@elevenlabs/client/dist/worklets/*.js public/elevenlabs/
Step 2: Configure SDK to use self-hosted worklets:
import { useConversation } from '@elevenlabs/react';
const { startConversation } = useConversation({
agentId: 'your-agent-id',
// Point to self-hosted worklet files
workletPaths: {
'rawAudioProcessor': '/elevenlabs/rawAudioProcessor.worklet.js',
'audioConcatProcessor': '/elevenlabs/audioConcatProcessor.worklet.js',
},
// Rest of config...
});
Step 3: Update CSP headers to allow self-hosted scripts:
# nginx example
add_header Content-Security-Policy "
default-src 'self';
script-src 'self' https://elevenlabs.io;
connect-src 'self' https://api.elevenlabs.io wss://api.elevenlabs.io;
worker-src 'self';
" always;
Worklet Files Location:
node_modules/@elevenlabs/client/dist/worklets/
├── rawAudioProcessor.worklet.js
└── audioConcatProcessor.worklet.js
Gotchas:
@elevenlabs/clientWhen You Need This:
blob: URLsThis skill composes well with:
Official Documentation:
Examples:
Community:
Production Tested: WordPress Auditor, Customer Support Agents Last Updated: 2025-11-03 Package Versions: elevenlabs@1.59.0, @elevenlabs/cli@0.2.0