Table of Contents

Enabling Real-Time Speech for an Agent

This guide walks through the end-to-end process of configuring real-time voice conversations for an agent, from setting up the Azure OpenAI API endpoint to enabling speech on the agent itself.

Overview

Real-time speech allows users to have spoken conversations with an agent through the Chat User Portal. The feature requires three configuration layers:

  1. API Endpoint — Connection to an Azure OpenAI resource that has a GPT-Realtime model deployed
  2. AI Model — A realtime speech model definition that references the endpoint and deployment
  3. Agent Settings — Speech enabled on the agent with a reference to the AI model

Note: The agent editor shows the Realtime Speech Configuration block only when the FoundationaLLM.Preview feature flag is enabled. Turn it on under Platform > Configuration (Feature Flags tab).

Prerequisites

  • Access to the Management Portal with permissions to manage API endpoints, AI models, and agents
  • An Azure OpenAI resource with a GA realtime model deployed (for example gpt-realtime)
  • The Azure OpenAI resource's endpoint URL, and either an API key stored in Key Vault (referenced by an App Configuration key) or an identity that can call the resource
  • The deployment name of the realtime model in your Azure OpenAI resource

Important: FoundationaLLM uses the Azure OpenAI GA Realtime API (wss://{resource}.openai.azure.com/openai/v1/realtime?model={deployment}). The preview protocol (/openai/realtime with api-version and deployment query parameters) is not supported and there is no backwards compatibility with it. Preview deployments such as gpt-4o-realtime-preview cannot be used; deploy a GA realtime model.

Coordinated deployment: the Core API forwards the GA event names (for example response.output_audio.delta) to browsers unchanged. Deploy the Core API together with a User Portal (and Agent Portal) release that handles the GA event names; a portal built for the preview names does not play audio against a Core API deployed from this version.

Step 1: Configure the Azure OpenAI API Endpoint

If you already have an Azure OpenAI API endpoint configured without an API version and with Category General and Subcategory AIModel, you can skip to Step 2. If your existing endpoint has an API version because other models (for example chat completions) need it, create a dedicated endpoint for realtime speech.

Creating the Endpoint

  1. In the sidebar, select Platform > API Endpoints, and then select Create API Endpoint.
  2. Configure the following settings:
Field Value Example
Resource name / Display name A descriptive name AzureOpenAI-Realtime
Provider Microsoft (Azure OpenAI)
Category General
Subcategory AIModel. The model editor lists only endpoints with this category and subcategory.
Endpoint URL Your Azure OpenAI resource URL https://my-openai.openai.azure.com/
API version Leave blank. The GA Realtime WebSocket does not accept api-version, and the platform never sends it for realtime connections. Do not reuse an endpoint whose API version is required by other models. (blank)
Authentication type API Key or Azure Identity
Authentication parameters For API Key, add api_key_configuration_name with the App Configuration key that holds the API key (optionally api_key_header_name; the default header is api-key). For Azure Identity, add scope, for example https://cognitiveservices.azure.com/.default.
  1. Select Create API Endpoint.

Important: The URL should be the base endpoint of your Azure OpenAI resource (for example https://my-openai.openai.azure.com/). Do not include /openai/realtime, /openai/v1, or other path segments — the platform constructs the full URL automatically and logs a warning when it has to strip such segments.

Verifying the Endpoint

After saving, confirm the endpoint appears in the Platform > API Endpoints list with the correct provider and category.

Step 2: Create a Realtime Speech AI Model

  1. In the sidebar, select Agents > Models, and then select Create Model.
  2. Configure the following settings:
Field Value Description
Display name / Name Descriptive name For example GPT-Realtime-Speech
Model Type Realtime Speech
Model Endpoint The endpoint created in Step 1
Model deployment name Azure OpenAI deployment name of a GA realtime model For example gpt-realtime
Model version Model version (required) For example 2025-08-28
  1. Select Create Model.

Model-level audio defaults

A realtime speech model also stores audio and turn-detection defaults. The model editor does not show these fields; the platform uses the defaults below unless you change them in the model's JSON through the Management API. The portal preserves them when you save the model.

Property Default Description
voice alloy Default voice. The agent's Voice setting overrides it.
input_audio_format pcm16 pcm16 is required for the User Portal and the Agent Portal, which record PCM16 audio. The backend also accepts g711_ulaw and g711_alaw, but only for custom clients that encode those formats; selecting them for the portals sends Azure the wrong codec. Any other value is rejected.
output_audio_format pcm16 pcm16 is required for the User Portal and the Agent Portal, which decode PCM16 audio. The backend also accepts g711_ulaw and g711_alaw, but only for custom clients that decode those formats; selecting them for the portals makes playback unintelligible. Any other value is rejected.
input_audio_transcription_enabled true When false, user speech is not transcribed and transcript-based conversation naming does not run; with Show Transcriptions enabled the user message shows the fixed text Voice input (transcription disabled).. Agent transcription is unaffected.
input_audio_transcription_model whisper-1 Model used to transcribe user speech when transcription is enabled
turn_detection.type server_vad Voice activity detection mode
turn_detection.threshold 0.5 Default speech detection threshold. The agent's setting overrides it.
turn_detection.prefix_padding_ms 300 Audio captured before speech is detected
turn_detection.silence_duration_ms 500 Default silence duration. The agent's setting overrides it.
turn_detection.interrupt_response true Default barge-in behavior. The agent's setting overrides it.

Step 3: Enable Real-Time Speech on the Agent

  1. In the sidebar, select Agents > Agents, and then select the pencil icon for the agent (or select Create Agent).
  2. Select the Agent Configuration tab.
  3. Select the Realtime Speech Configuration block to open it.
  4. Configure the settings below, and then select Done.
  5. Select Save Changes (or Create Agent).

Speech Settings

Setting Default Description
Enabled No Set to Yes to enable realtime speech. The other settings appear when it is on.
Model — The realtime speech AI model from Step 2. Only models of type Realtime Speech are listed. Required when speech is enabled.
Stop Words (comma-separated) stop, end conversation, goodbye Examples of clear hangup phrasing included in the end_conversation tool description. They do not end the session by themselves.
Voice alloy The voice for agent responses: alloy, ash, ballad, coral, echo, sage, shimmer, or verse. Each option shows a short description.
Speech Detection Threshold 0.50 0.00–0.99. Higher values reduce noise-triggered interruptions.
Silence Duration (ms) 500 100–3000. Longer values reduce choppy end-of-turn behavior.
Allow Barge-in Yes Whether the user's speech can interrupt an in-progress response
Show Transcriptions Yes Whether spoken words appear as text messages in the chat thread
Include Conversation History Yes Whether prior text messages are sent as context to the speech model

The agent's Voice, Speech Detection Threshold, Silence Duration (ms), and Allow Barge-in values are always saved with the agent, so they take precedence over the model-level defaults.

Important: Do not give any agent tool the name end_conversation. That name is reserved for the session hangup tool. If an agent with realtime speech enabled has a tool with that name (in any letter case), the Core API rejects the voice session and asks you to rename the tool. See Tools.

Setting Recommended Value Why
Show Transcriptions Yes Provides a written record of voice conversations
Include Conversation History Yes Gives the agent context from prior text messages
Stop Words stop, goodbye, end conversation Examples the model can treat as a clear request to hang up

Verification

Testing from the User Portal

  1. Open the Chat User Portal
  2. Select the agent you just configured
  3. Verify the microphone button appears to the right of the chat input
  4. Click the microphone button and grant microphone access if prompted
  5. Speak a question and verify:
    • The sound visualizer shows your audio activity
    • The agent responds with spoken audio
    • Transcriptions appear in the chat thread (if enabled)
  6. Click the stop button, or ask the agent to hang up, to end the session

Testing from the API

Use the configuration endpoint to verify the agent's speech settings:

curl -s "https://{core-api-url}/instances/{instance-id}/agents/{agent-name}/realtime-speech/config" \
  -H "Authorization: Bearer {token}" | python -m json.tool

Expected response:

{
  "enabled": true,
  "stop_words": ["stop", "goodbye", "end conversation"],
  "voice": "alloy"
}

Agent Configuration JSON Reference

For agents configured via API, the real-time speech settings are stored in the realtime_speech_settings property:

{
  "name": "voice-assistant",
  "realtime_speech_settings": {
    "enabled": true,
    "realtime_speech_ai_model_object_id": "/instances/{instanceId}/providers/FoundationaLLM.AIModel/aiModels/GPT-Realtime-Speech",
    "stop_words": ["stop", "end conversation", "goodbye"],
    "voice": "alloy",
    "threshold": 0.5,
    "silence_duration_ms": 500,
    "interrupt_response": true,
    "max_session_duration_seconds": 0,
    "show_transcriptions": true,
    "include_conversation_history": true
  }
}
Field Type Description
enabled boolean Whether speech is active
realtime_speech_ai_model_object_id string Resource path to the realtime speech AI model
stop_words string[] Examples of clear hangup phrasing for the end_conversation tool. They do not end the session by themselves.
voice string Agent override for the model voice. When null, the model voice is used.
threshold number Agent override for the speech detection threshold
silence_duration_ms number Agent override for the silence duration
interrupt_response boolean Agent override for barge-in
max_session_duration_seconds number Maximum session length (0 = unlimited)
show_transcriptions boolean Save transcriptions to the chat thread
include_conversation_history boolean Send prior messages as context

Troubleshooting

Realtime Speech Configuration Does Not Appear in the Agent Editor

  • Enable the FoundationaLLM.Preview feature flag under Platform > Configuration, and then reload the agent editor.

The Model List Is Empty

  • Only AI models of type Realtime Speech are listed. Check the model type on Agents > Models.

Microphone Button Does Not Appear in User Portal

  • Verify Enabled is set to Yes in the agent's Realtime Speech Configuration
  • Verify a Model is selected
  • Refresh the User Portal page after saving agent changes

Voice Session Is Rejected Because of end_conversation

  • An agent tool is named end_conversation. Rename it on the agent's Tools tab and save the agent.

"Voice Conversation Error: Not properly configured"

  • The API endpoint is missing or has an invalid API key
  • Verify the API key is stored correctly in Key Vault
  • Confirm the Key Vault secret name matches the App Configuration reference
  • Check that the CoreAPI service has the AzureOpenAI:Essentials configuration section loaded

401 Authentication Error in Server Logs

  • The API key stored in Key Vault may be incorrect or expired
  • Rotate the key in your Azure OpenAI resource and update the Key Vault secret
  • Verify the Azure OpenAI resource is accessible from your deployment

Agent Responds but No Transcriptions Appear

  • Verify Show Transcriptions is set to Yes in the agent's speech settings
  • The transcription model (whisper-1) must be available on the Azure OpenAI endpoint

Voice Sounds Robotic or Choppy

  • Check network latency between your deployment and the Azure OpenAI resource
  • Consider deploying the Azure OpenAI resource in the same region as your FoundationaLLM instance
  • Increase the agent's Silence Duration (ms) to give voice activity detection more buffer