Azure AI Voicelive TS

MicrosoftOfficialCreated by Microsoft

Skill
Tool integration
Data and AI
MIT
No issues found

From the author

Azure AI Voice Live SDK for JavaScript/TypeScript. Build real-time voice AI applications with bidirectional WebSocket communication. Use for voice assistants, conversational AI, real-time speech-to-speech, and voice-enabled chatbots in Node.js or browser environments. Triggers: "voice live", "real-time voice", "VoiceLiveClient", "VoiceLiveSession", "voice assistant TypeScript", "bidirectional audio", "speech-to-speech JavaScript".

— the description by Microsoft

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

The Azure AI Voice Live SDK for JavaScript and TypeScript enables developers to create voice-enabled applications that support real-time, bidirectional communication. It provides a client-based interface for managing WebSocket sessions, handling audio streaming, and integrating function calling capabilities within Node.js or browser-based environments.

What it does

Establishes bidirectional WebSocket sessions for voice interaction.
Supports streaming audio input and output in various formats.
Provides event-based subscription patterns for handling conversation lifecycle.
Enables function calling to allow AI assistants to execute external actions.
Supports multiple voice configuration options and turn detection settings.
Facilitates integration with Azure identity services for authentication.

When to use it

When building conversational voice assistants in TypeScript or JavaScript.
When implementing real-time speech-to-speech features.
When integrating voice-enabled chatbots into web or server-side applications.
When you need to define custom tools for an AI assistant to perform tasks.

Advantages

Provides a structured client hierarchy for session management.
Supports both Node.js and modern browser environments.
Includes built-in support for function calling and tool definitions.
Offers flexible audio format and turn detection configuration.
Uses standard Azure SDK authentication patterns.

Usage examples

“How do I start a voice session with the SDK?”: Initialize a VoiceLiveClient with credentials, then call client.startSession() with the desired model name to establish a WebSocket connection.
“How can I add a custom tool to my voice assistant?”: Define the tool using JSON Schema in the session configuration's tools array, then handle the onResponseFunctionCallArgumentsDone event to execute your function logic.

Install

Download the skill as a ZIP:
https://autogineer.com/agent-skills/microsoft/azure-ai-voicelive-ts/download

Unzip it into the cross-client skills folder of your project:

mkdir -p .agents/skills && unzip azure-ai-voicelive-ts.zip -d .agents/skills

Documentation by Microsoft

@azure/ai-voicelive (JavaScript/TypeScript)

Real-time voice AI SDK for building bidirectional voice assistants with Azure AI in Node.js and browser environments.

Installation

npm install @azure/ai-voicelive @azure/identity
# TypeScript users
npm install @types/node

Current Version: 1.0.0-beta.3

Supported Environments:

Node.js LTS versions (20+)
Modern browsers (Chrome, Firefox, Safari, Edge)

Environment Variables

AZURE_VOICELIVE_ENDPOINT=https://<resource>.cognitiveservices.azure.com
# Optional: API key if not using Entra ID
AZURE_VOICELIVE_API_KEY=<your-api-key>
# Optional: Logging
AZURE_LOG_LEVEL=info
AZURE_TOKEN_CREDENTIALS=prod # Required only if DefaultAzureCredential is used in production

Authentication

Microsoft Entra Token Credential (Recommended)

import { DefaultAzureCredential, ManagedIdentityCredential } from "@azure/identity";
import { VoiceLiveClient } from "@azure/ai-voicelive";

// Local dev: DefaultAzureCredential. Production: set AZURE_TOKEN_CREDENTIALS=prod or AZURE_TOKEN_CREDENTIALS=<specific_credential>
const credential = new DefaultAzureCredential({requiredEnvVars: ["AZURE_TOKEN_CREDENTIALS"]});
// Or use a specific credential directly in production:
// See https://learn.microsoft.com/javascript/api/overview/azure/identity-readme?view=azure-node-latest#credential-classes
// const credential = new ManagedIdentityCredential();
const endpoint = "https://your-resource.cognitiveservices.azure.com";

const client = new VoiceLiveClient(endpoint, credential);

API Key

import { AzureKeyCredential } from "@azure/core-auth";
import { VoiceLiveClient } from "@azure/ai-voicelive";

const endpoint = "https://your-resource.cognitiveservices.azure.com";
const credential = new AzureKeyCredential("your-api-key");

const client = new VoiceLiveClient(endpoint, credential);

Client Hierarchy

VoiceLiveClient
└── VoiceLiveSession (WebSocket connection)
    ├── updateSession()      → Configure session options
    ├── subscribe()          → Event handlers (Azure SDK pattern)
    ├── sendAudio()          → Stream audio input
    ├── addConversationItem() → Add messages/function outputs
    └── sendEvent()          → Send raw protocol events

Quick Start

import { DefaultAzureCredential } from "@azure/identity";
import { VoiceLiveClient } from "@azure/ai-voicelive";

const credential = new DefaultAzureCredential({requiredEnvVars: ["AZURE_TOKEN_CREDENTIALS"]});
const endpoint = process.env.AZURE_VOICELIVE_ENDPOINT!;

// Create client and start session
const client = new VoiceLiveClient(endpoint, credential);
const session = await client.startSession("gpt-4o-mini-realtime-preview");

// Configure session
await session.updateSession({
  modalities: ["text", "audio"],
  instructions: "You are a helpful AI assistant. Respond naturally.",
  voice: {
    type: "azure-standard",
    name: "en-US-AvaNeural",
  },
  turnDetection: {
    type: "server_vad",
    threshold: 0.5,
    prefixPaddingMs: 300,
    silenceDurationMs: 500,
  },
  inputAudioFormat: "pcm16",
  outputAudioFormat: "pcm16",
});

// Subscribe to events
const subscription = session.subscribe({
  onResponseAudioDelta: async (event, context) => {
    // Handle streaming audio output
    const audioData = event.delta;
    playAudioChunk(audioData);
  },
  onResponseTextDelta: async (event, context) => {
    // Handle streaming text
    process.stdout.write(event.delta);
  },
  onInputAudioTranscriptionCompleted: async (event, context) => {
    console.log("User said:", event.transcript);
  },
});

// Send audio from microphone
function sendAudioChunk(audioBuffer: ArrayBuffer) {
  session.sendAudio(audioBuffer);
}

Session Configuration

await session.updateSession({
  // Modalities
  modalities: ["audio", "text"],
  
  // System instructions
  instructions: "You are a customer service representative.",
  
  // Voice selection
  voice: {
    type: "azure-standard",  // or "azure-custom", "openai"
    name: "en-US-AvaNeural",
  },
  
  // Turn detection (VAD)
  turnDetection: {
    type: "server_vad",      // or "azure_semantic_vad"
    threshold: 0.5,
    prefixPaddingMs: 300,
    silenceDurationMs: 500,
  },
  
  // Audio formats
  inputAudioFormat: "pcm16",
  outputAudioFormat: "pcm16",
  
  // Tools (function calling)
  tools: [
    {
      type: "function",
      name: "get_weather",
      description: "Get current weather",
      parameters: {
        type: "object",
        properties: {
          location: { type: "string" }
        },
        required: ["location"]
      }
    }
  ],
  toolChoice: "auto",
});

Event Handling (Azure SDK Pattern)

The SDK uses a subscription-based event handling pattern:

const subscription = session.subscribe({
  // Connection lifecycle
  onConnected: async (args, context) => {
    console.log("Connected:", args.connectionId);
  },
  onDisconnected: async (args, context) => {
    console.log("Disconnected:", args.code, args.reason);
  },
  onError: async (args, context) => {
    console.error("Error:", args.error.message);
  },
  
  // Session events
  onSessionCreated: async (event, context) => {
    console.log("Session created:", context.sessionId);
  },
  onSessionUpdated: async (event, context) => {
    console.log("Session updated");
  },
  
  // Audio input events (VAD)
  onInputAudioBufferSpeechStarted: async (event, context) => {
    console.log("Speech started at:", event.audioStartMs);
  },
  onInputAudioBufferSpeechStopped: async (event, context) => {
    console.log("Speech stopped at:", event.audioEndMs);
  },
  
  // Transcription events
  onConversationItemInputAudioTranscriptionCompleted: async (event, context) => {
    console.log("User said:", event.transcript);
  },
  onConversationItemInputAudioTranscriptionDelta: async (event, context) => {
    process.stdout.write(event.delta);
  },
  
  // Response events
  onResponseCreated: async (event, context) => {
    console.log("Response started");
  },
  onResponseDone: async (event, context) => {
    console.log("Response complete");
  },
  
  // Streaming text
  onResponseTextDelta: async (event, context) => {
    process.stdout.write(event.delta);
  },
  onResponseTextDone: async (event, context) => {
    console.log("\n--- Text complete ---");
  },
  
  // Streaming audio
  onResponseAudioDelta: async (event, context) => {
    const audioData = event.delta;
    playAudioChunk(audioData);
  },
  onResponseAudioDone: async (event, context) => {
    console.log("Audio complete");
  },
  
  // Audio transcript (what assistant said)
  onResponseAudioTranscriptDelta: async (event, context) => {
    process.stdout.write(event.delta);
  },
  
  // Function calling
  onResponseFunctionCallArgumentsDone: async (event, context) => {
    if (event.name === "get_weather") {
      const args = JSON.parse(event.arguments);
      const result = await getWeather(args.location);
      
      await session.addConversationItem({
        type: "function_call_output",
        callId: event.callId,
        output: JSON.stringify(result),
      });
      
      await session.sendEvent({ type: "response.create" });
    }
  },
  
  // Catch-all for debugging
  onServerEvent: async (event, context) => {
    console.log("Event:", event.type);
  },
});

// Clean up when done
await subscription.close();

Function Calling

// Define tools in session config
await session.updateSession({
  modalities: ["audio", "text"],
  instructions: "Help users with weather information.",
  tools: [
    {
      type: "function",
      name: "get_weather",
      description: "Get current weather for a location",
      parameters: {
        type: "object",
        properties: {
          location: {
            type: "string",
            description: "City and state or country",
          },
        },
        required: ["location"],
      },
    },
  ],
  toolChoice: "auto",
});

// Handle function calls
const subscription = session.subscribe({
  onResponseFunctionCallArgumentsDone: async (event, context) => {
    if (event.name === "get_weather") {
      const args = JSON.parse(event.arguments);
      const weatherData = await fetchWeather(args.location);
      
      // Send function result
      await session.addConversationItem({
        type: "function_call_output",
        callId: event.callId,
        output: JSON.stringify(weatherData),
      });
      
      // Trigger response generation
      await session.sendEvent({ type: "response.create" });
    }
  },
});

Voice Options

Voice Type	Config	Example
Azure Standard	`{ type: "azure-standard", name: "..." }`	`"en-US-AvaNeural"`
Azure Custom	`{ type: "azure-custom", name: "...", endpointId: "..." }`	Custom voice endpoint
Azure Personal	`{ type: "azure-personal", speakerProfileId: "..." }`	Personal voice clone
OpenAI	`{ type: "openai", name: "..." }`	`"alloy"`, `"echo"`, `"shimmer"`

Supported Models

Model	Description	Use Case
`gpt-4o-realtime-preview`	GPT-4o with real-time audio	High-quality conversational AI
`gpt-4o-mini-realtime-preview`	Lightweight GPT-4o	Fast, efficient interactions
`phi4-mm-realtime`	Phi multimodal	Cost-effective applications

Turn Detection Options

// Server VAD (default)
turnDetection: {
  type: "server_vad",
  threshold: 0.5,
  prefixPaddingMs: 300,
  silenceDurationMs: 500,
}

// Azure Semantic VAD (smarter detection)
turnDetection: {
  type: "azure_semantic_vad",
}

// Azure Semantic VAD (English optimized)
turnDetection: {
  type: "azure_semantic_vad_en",
}

// Azure Semantic VAD (Multilingual)
turnDetection: {
  type: "azure_semantic_vad_multilingual",
}

Audio Formats

Format	Sample Rate	Use Case
`pcm16`	24kHz	Default, high quality
`pcm16-8000hz`	8kHz	Telephony
`pcm16-16000hz`	16kHz	Voice assistants
`g711_ulaw`	8kHz	Telephony (US)
`g711_alaw`	8kHz	Telephony (EU)

Key Types Reference

Type	Purpose
`VoiceLiveClient`	Main client for creating sessions
`VoiceLiveSession`	Active WebSocket session
`VoiceLiveSessionHandlers`	Event handler interface
`VoiceLiveSubscription`	Active event subscription
`ConnectionContext`	Context for connection events
`SessionContext`	Context for session events
`ServerEventUnion`	Union of all server events

Error Handling

import {
  VoiceLiveError,
  VoiceLiveConnectionError,
  VoiceLiveAuthenticationError,
  VoiceLiveProtocolError,
} from "@azure/ai-voicelive";

const subscription = session.subscribe({
  onError: async (args, context) => {
    const { error } = args;
    
    if (error instanceof VoiceLiveConnectionError) {
      console.error("Connection error:", error.message);
    } else if (error instanceof VoiceLiveAuthenticationError) {
      console.error("Auth error:", error.message);
    } else if (error instanceof VoiceLiveProtocolError) {
      console.error("Protocol error:", error.message);
    }
  },
  
  onServerError: async (event, context) => {
    console.error("Server error:", event.error?.message);
  },
});

Logging

import { setLogLevel } from "@azure/logger";

// Enable verbose logging
setLogLevel("info");

// Or via environment variable
// AZURE_LOG_LEVEL=info

Browser Usage

// Browser requires bundler (Vite, webpack, etc.)
import { VoiceLiveClient } from "@azure/ai-voicelive";
import { InteractiveBrowserCredential } from "@azure/identity";

// Use browser-compatible credential
const credential = new InteractiveBrowserCredential({
  clientId: "your-client-id",
  tenantId: "your-tenant-id",
});

const client = new VoiceLiveClient(endpoint, credential);

// Request microphone access
const stream = await navigator.mediaDevices.getUserMedia({ audio: true });
const audioContext = new AudioContext({ sampleRate: 24000 });

// Process audio and send to session
// ... (see samples for full implementation)

Best Practices

Use DefaultAzureCredential for local development; use ManagedIdentityCredential or WorkloadIdentityCredential for production — Never hardcode API keys
Set both modalities — Include ["text", "audio"] for voice assistants
Use Azure Semantic VAD — Better turn detection than basic server VAD
Handle all error types — Connection, auth, and protocol errors
Clean up subscriptions — Call subscription.close() when done
Use appropriate audio format — PCM16 at 24kHz for best quality

Reference Links

Resource	URL
npm Package	https://www.npmjs.com/package/@azure/ai-voicelive
GitHub Source	https://github.com/Azure/azure-sdk-for-js/tree/main/sdk/ai/ai-voicelive
Samples	https://github.com/Azure/azure-sdk-for-js/tree/main/sdk/ai/ai-voicelive/samples
API Reference	https://learn.microsoft.com/javascript/api/@azure/ai-voicelive

Files

ATTRIBUTION.mdAdded by AutoGineer982 B
LICENSELicence file added by AutoGineer1.1 KiB
SKILL.md13.5 KiB
references/audio-streaming.md12.3 KiB
references/function-calling.md13.5 KiB

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Needs an API key, Needs Node.js

Security

No issues found by our static scan· No issues found

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Created by

Microsoft

Publisher

Microsoft (Official)

Source

microsoft/skills · .github/plugins/azure-sdk-typescript/skills/azure-ai-voicelive-ts · commit 0696600

Licence

MIT License (redistribution allowed)

LICENSE (added by AutoGineer from LICENSE)
ATTRIBUTION.md (written by AutoGineer)

Synced

2026-11-09

AutoGineer is not affiliated with or endorsed by Microsoft or Microsoft.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download azure-ai-voicelive-ts.zip