Table of Contents

AI Models

Learn how to configure and manage AI model definitions in the Management Portal.

Overview

AI models define the language models and other AI models available to your FoundationaLLM deployment. Each model points to an API endpoint and a deployment on that endpoint. Models power agent conversations, embeddings, tools such as image generation, and realtime speech.

To open the page, in the sidebar select Agents > Models.

Model List

The Models page lists models 20 at a time. Scroll down to load more; "Loading more models..." appears while the next page loads.

Column Description
Name The display name. When it differs from the resource name, the resource name is shown below it in parentheses.
Source Type The model type, for example completion or embedding
Actions View JSON definition, edit, clone, and delete

Searching and refreshing

  • Type in Search models to filter. The search runs on the server and matches model names and display names.
  • Select the refresh button (Refresh models) to reload the list.
  • Select the Name column header to sort the loaded rows.

Actions

Icon Action
</> View JSON definition opens the model resource JSON
Pencil Opens Edit Model. Disabled without write permission on the model.
Clone Opens Create Model prefilled with the model's type, endpoint, deployment, version, and parameters, with an empty name
Trash Deletes the model after confirmation. Disabled without delete permission.

Model Types

Type (label in the editor) Value Use Cases
Basic basic General model definitions
Embedding embedding Vector embeddings for semantic search and knowledge
Completion completion Agent conversations and tool completions. The agent's Main model list shows only completion models.
ImageGeneration image-generation Image generation tools
Realtime Speech realtime-speech Realtime voice conversations. The agent's realtime speech Model list shows only these models.

Creating a Model

  1. On the Models page, select Create Model. To start from an existing model, select its clone icon instead.
  2. Fill in the fields below.
  3. Select Create Model. The portal saves the model and returns to the list.
Field Description
Display name Shown in lists and pickers. Required.
Name The resource name. The portal derives it from the display name and checks availability (✔️ available, ❌ taken). Letters, numbers, dashes, and underscores only. Cannot be changed later.
Model Type One of the types above. Required.
Model Endpoint The API endpoint the model calls. Required. Only endpoints with Category General and Subcategory AIModel are listed.
Model deployment name The deployment name or model ID on the endpoint (for TypeSafe Jev, jev-latest or a pinned ID such as jev-1.13.0). Required.
Model version The model version. Required.
Model parameters Default parameters for every workflow and tool that uses the model. Select Add Property, and enter the Property Key, Property Type (String, Number, Boolean, or Object / Array), and Property Value. Which keys the runtime applies depends on the endpoint provider.

Select Cancel and confirm to leave without saving.

Note: API endpoints are managed on Platform > API Endpoints. See API Endpoints.

Editing a Model

  1. Select the pencil icon for the model.
  2. On Edit Model, change the settings. The resource name is shown under the display name and cannot be changed.
  3. Select Save Changes. The page stays open after saving.

Saving keeps properties that the editor does not show, such as the audio defaults on realtime speech models.

Deleting a Model

  1. Select the trash icon for the model.
  2. Confirm in the Delete Model dialog.

Warning: Deleting a model affects any agents and tools that use it. Check dependencies first.

Provider Guidance

OpenAI Responses and GPT-6 Astra

Endpoints with the Microsoft or OpenAI provider have an Operation type setting on the API endpoint: Chat Completions, Completions (legacy), or Responses. Choose Responses for GPT-6 Astra and other reasoning models that require the Responses API. For Responses endpoints, the portal stores the URL as the /openai/v1/ surface, hides the API version, and recommends a timeout of 300 seconds or more.

Astra is a completion model. Point it at an endpoint whose operation type is Responses; saving an Astra model on another endpoint is rejected. Do not assign Astra to an Azure OpenAI Assistants workflow.

Recommended model parameters:

Key Value
reasoning_effort low, medium (default), high, xhigh, or max. Do not use none.
max_new_tokens 32000 or at least 25000 so reasoning plus visible output has room.
model_family gpt-6-astra when the Foundry deployment name is not gpt-6-astra.
store Leave unset (false). Set true only if you accept server-side response storage.

temperature and top_p are invalid on Astra. If they are present on the model or as agent overrides they are dropped, recorded on the workflow execution artifact as dropped_model_parameters, and never sent to the API.

If a completion returns "The model returned an incomplete response (reason: max_output_tokens)", increase max_new_tokens or lower reasoning_effort.

Conversation history is replayed as plain text. Prior-turn reasoning items are not sent back to the model. Gateway completions, user-prompt rewrite, and Azure OpenAI direct orchestration remain on Chat Completions and must not use Astra.

TypeSafe Jev

Jev is a decision model, not a chat model. It is hosted by TypeSafe and is not in the Azure or Microsoft Foundry catalogs. Create an API endpoint with the TypeSafe provider (URL https://api.typesafe.ai; calls go to POST /v1/systemone), and point a Completion model at it.

Field Value
Model deployment name jev-latest or a pinned ID such as jev-1.13.0
Prompt JSON map of choice, score, and noul questions. This is the question set Jev evaluates.
User message The state Jev evaluates.
Tools None. Jev does not call tools.

The optional model parameter questions overrides the prompt when both are set. The parameter value is the same JSON map.

Example prompt:

{
  "department": {
    "type": "choice",
    "instructions": "Which team should handle this?",
    "criteria": {
      "billing": "Payment or subscription issues",
      "technical": "Bugs or integration problems",
      "sales": "Pricing or account questions"
    }
  },
  "is_urgent": {
    "type": "noul",
    "instructions": "The message conveys urgency or time-sensitivity"
  }
}

The User Portal shows the decision as text (department: technical (confidence 0.78)). Jev does not write a conversation title; the portal uses a short title taken from the user message.

Claude

Workflow and tool completions can use Claude through a Bedrock API endpoint (ChatBedrockConverse) or an Anthropic API endpoint (ChatAnthropic, the Anthropic Messages API). The Claude Agent tool is separate: it uses the Claude Agent SDK, not these AI model parameters. See Configure the Claude Agent Tool.

Anthropic Messages (direct API and Azure AI Foundry)

Point the AI model at an API endpoint whose provider is Anthropic, and set Model deployment name to the Claude model or Foundry deployment name.

Host Authentication
https://api.anthropic.com API key. The runtime reads the configuration setting named by authentication_parameters.api_key_configuration_name or authentication_parameters.api_key.
Azure AI Foundry (*.services.ai.azure.com) and other Azure hosts (*.openai.azure.com, *.cognitiveservices.azure.com) API key or Azure Identity (Entra ID).

An Azure resource root that omits /anthropic is called as {resource}/anthropic. A URL that ends in /v1 or /v1/messages, and an inference path that contains /models, is trimmed back to that base before the call. An API key on an Azure host is also sent as the api-key header.

Azure Identity uses authentication_parameters.scope when that value is set. Otherwise Foundry hosts (*.services.ai.azure.com) use https://ai.azure.com/.default, and other Azure hosts use https://cognitiveservices.azure.com/.default.

max_new_tokens on the model or on a workflow main-model override is sent as max_tokens. The override wins over the model value. Claude Fable accepts only the default sampling settings; other temperature, top_p, and top_k values are omitted.

Claude on Amazon Bedrock

Claude Sonnet 5 changed an important default: adaptive thinking is on by default, and its default effort is high. Anthropic also states that max_tokens is a hard limit across thinking + visible response text. If thinking output is not configured to be displayed, those tokens may be invisible to your application.

To use Bedrock Claude, point the AI model at a Bedrock API endpoint and set Model deployment name to the Bedrock model ID (for example anthropic.claude-sonnet-5 or the inference-profile ID your account uses).

Set Bedrock-specific request fields on the AI model (preferred) so every workflow and tool that uses the model receives them. You can also set the same keys under an agent's Workflow main model parameters (Workflow tab); those apply to the workflow LLM only. A top-level thinking property is ignored. Core API settings.model_parameters does not accept additional_model_request_fields.

additional_model_request_fields

This key is a LangChain ChatBedrockConverse field. FoundationaLLM copies it onto the Bedrock client when the model's API endpoint provider is Bedrock. Azure OpenAI and OpenAI models ignore it.

  1. On Agents > Models, open the Claude completion model.
  2. Under What are the model parameters?, select Add Property and enter:
    • Property Key: additional_model_request_fields
    • Property Type: Object / Array
    • Property Value:
{
  "thinking": {
    "type": "adaptive"
  },
  "output_config": {
    "effort": "high"
  }
}
  1. Select Save in the dialog, and then Save Changes.

effort must sit in output_config, not inside thinking. Putting it in thinking returns a Bedrock ValidationException.

Goal additional_model_request_fields
Default Sonnet 5 depth Omit the object, or "effort": "high"
Cheaper / faster "output_config": { "effort": "low" } or "medium"
Hard agent or coding work "xhigh" or "max"
Turn thinking off "thinking": { "type": "disabled" } (Anthropic API allows this; confirm on your Bedrock model card)
Old extended thinking "type": "enabled" with budget_tokens returns 400 on Sonnet 5

Adaptive thinking is already on if you omit thinking. Set this object when you need a non-default effort or an explicit thinking config.

Token budget and sampling

Thinking tokens count toward max output even when the thinking text is not returned. On Bedrock, set max_tokens (not max_new_tokens; max_new_tokens is not applied to ChatBedrockConverse):

Key Type Guidance
max_tokens number Leave headroom for thinking plus the visible reply. Use 16000 or higher at high / xhigh / max effort.
temperature, top_p, top_k — Do not set these on Claude Sonnet 5. Non-default sampling returns 400.

If a completion is almost entirely thinking and then truncated, raise max_tokens or lower effort.

Google Gemini

Gemini models use an API endpoint with the Google Agent Platform provider (or the older Google VertexAI provider), Google Service Account authentication, and a service_account_credentials authentication parameter. The endpoint's optional Project and Location are passed to Gemini. See Configuring Gemini Image Generation and Configuring Gemini Web Grounding.

Realtime speech

Realtime speech models use an Azure OpenAI endpoint without an API version. See Enabling Real-Time Speech.

Model Parameters

Common keys:

Parameter Description Typical Range
temperature Response randomness. Invalid on Claude Sonnet 5 and GPT-6 Astra. 0.0 - 2.0
max_tokens Maximum output (thinking + visible text) for Bedrock Claude and for the Anthropic Messages provider. 1 - model limit
max_new_tokens Maximum response length for OpenAI-family models. On the Anthropic Messages provider this is sent as max_tokens. Bedrock ChatBedrockConverse does not apply it. 1 - model limit
top_p Nucleus sampling. Invalid on Claude Sonnet 5 and GPT-6 Astra. 0.0 - 1.0
additional_model_request_fields Bedrock-only object passed as Converse additionalModelRequestFields (thinking, effort) See Claude on Amazon Bedrock
reasoning_effort OpenAI Responses / GPT-6 Astra reasoning depth low, medium, high, xhigh, max

Using Models

Where Setting
Agent Basic Settings tab Main model — the workflow model for conversations and tool routing (completion models only)
Agent Workflow tab Workflow main model parameters — overrides for the main model on this agent
Agent Agent Configuration tab Feature-specific models, such as the Semantic Cache and Realtime Speech Configuration models
Tool editor Resource References Main Model and other model slots on tools, each with a Temperature setting
Platform > Configuration (Self-Service tab) Models offered for self-service agents. See Self-Service AI Models.

Access Control

Permission Description
FoundationaLLM.AIModel/aiModels/read View models
FoundationaLLM.AIModel/aiModels/write Create and edit models (enables the edit icon)
FoundationaLLM.AIModel/aiModels/delete Delete models (enables the delete icon)

Best Practices

  • Use display names that describe the model and its purpose, and resource names that include the model family, for example GPT-4o-Chat-Main or Text-Embedding-3-Large.
  • Use the clone action to create variants (for example the same deployment with different default parameters).
  • Set reasonable defaults on the model and override them per agent or tool only when needed.

Troubleshooting

Model not available in a dropdown

  • Check the model type. The agent Main model list shows only Completion models, and the realtime speech Model list shows only Realtime Speech models.
  • Check your permissions on the model.

Endpoint not listed in Model Endpoint

The model editor lists only API endpoints with Category General and Subcategory AIModel. Update the endpoint on Platform > API Endpoints.

Model responses failing

  • Verify the API endpoint configuration and authentication.
  • Check the deployment status on the provider.
  • For Claude Sonnet 5 on Bedrock: a 400 often means temperature / top_p / top_k is set, or thinking uses "type": "enabled" with budget_tokens. Remove sampling parameters and use adaptive thinking with output_config.effort as described in Claude on Amazon Bedrock.
  • For GPT-6 Astra: make sure the endpoint's operation type is Responses.