Table of Contents

API and Token Limits

This reference describes the request, token, and resource limits that FoundationaLLM enforces, where they are configured, and what clients receive when a limit is exceeded.

Overview

FoundationaLLM limits usage in three ways:

Limit Purpose Where it is configured
Quotas Rate limits on Core API requests, monthly token usage limits, and limits on the number of agents or sandboxes a user can own. Quotas > Quotas in the Management Portal (FoundationaLLM.Quota/quotas resources).
Model token parameters The maximum size of each model response. AI model parameters (Agents > Models) and the agent's Workflow main model parameters.
Platform limits Fixed or App Configuration-based limits on request sizes, uploads, and Gateway API token budgets. Code defaults and App Configuration keys.

Quotas

Quotas are resources of the FoundationaLLM.Quota resource provider. Create and edit them under Quotas > Quotas, and review quota metrics and events under Quotas > Quotas Analysis. For step-by-step instructions, see Configuring Quotas.

Quota type (type) quota_type values Context What it limits
rate-limit-quota APIControllerRequestRate, AgentRequestRate CoreAPI:{ControllerName}, or CoreAPI:Completions:{agentName} for one agent. Requests within metric_window_seconds. When the limit is exceeded, the caller is locked out for lockout_duration_seconds.
usage-limit-quota TokenUsage Agent:__All__ (all agents) or Agent:{agentName}. Token usage per calendar month (metric_window_type is Month).
resource-limit-quota Agents, Sandboxes Agents or Sandboxes. The number of agents or sandboxes that a user owns.

Rate limit quotas can target these Core API controllers: Branding, Completions, CompletionsStatus, Configuration, Files, Management, OneDriveWorkSchool, RealtimeSpeech, Sessions, Status, and UserProfiles.

Scope

metric_partition Shown in the editor as Behavior
None None (Shared) A single limit shared by all callers. Not available for resource limit quotas.
UserPrincipalName User Principal Name — UPN (Individual) A separate limit for each user principal name.
UserIdentifier User Identifier (Individual) A separate limit for each user identifier.

A quota can also apply only to a specific security principal (principal_type and security_principal_id): a user, a group, a service principal, or a single agent access token. Agent access token scoping is available for rate limit and usage limit quotas that target a specific agent.

Rate Limit Enforcement

Rate limits are evaluated over 20-second intervals. For predictable enforcement, set metric_window_seconds to a multiple of 20 (60 is typical) and metric_limit to a multiple of metric_window_seconds / 20. When distributed_enforcement is true, Core API instances share their request counts so the limit applies across all instances.

Platform Quotas Package

The platform quotas package defines these rate limit quotas:

Quota name Context Limit
Platform-Rate-Limit-Completion-Per-User CoreAPI:Completions 120 requests per 60 seconds per UPN
Platform-Rate-Limit-Completion-Status-Per-User CoreAPI:CompletionsStatus 1,200 requests per 60 seconds per UPN
Platform-Rate-Limit-Conversation-Per-User CoreAPI:Sessions 120 requests per 60 seconds per UPN
Platform-Rate-Limit-File-Per-User CoreAPI:Files 60 requests per 60 seconds per UPN

All four use a 60-second lockout and distributed enforcement. The quotas that are active in your instance are the ones listed under Quotas > Quotas. If no quotas are defined, no quota limits are enforced.

Important

The Core API loads quota definitions when it starts. Changes to quotas take effect after the Core API restarts.

Quota Exceeded Responses

When a rate limit or usage limit quota is exceeded, the Core API returns HTTP 429 Too Many Requests with a quota evaluation result:

{
  "quota_type": "rate-limit-quota",
  "quota_details": {
    "Platform-Rate-Limit-Completion-Per-User": [ "CoreAPI:Completions" ]
  },
  "quota_exceeded": true,
  "time_until_retry_seconds": 45
}
Property Description
quota_type rate-limit-quota or usage-limit-quota.
quota_details The exceeded quotas, keyed by quota name, with the contexts that caused them to be exceeded.
quota_exceeded true when at least one quota was exceeded.
time_until_retry_seconds Rate limits only. The number of seconds after which the request can be retried.

Usage limit quotas are evaluated for completion requests and realtime speech sessions. Resource limit quotas are evaluated when a user creates an agent or a sandbox, or when agent ownership is transferred; the request fails with 429 and a message such as "The resource limit quota has been exceeded. You have reached the maximum number of agents allowed."

Model Token Limits

Tokens

Each completion consumes:

Category Description
Prompt tokens Tokens in the input: workflow prompts, conversation history, file content, and the user message.
Completion tokens Tokens in the model response, including reasoning or thinking tokens where the model reports them as output.
Total tokens The sum of prompt and completion tokens.

Users see token counts on messages in the User Portal when the agent's Would you like to show the message tokens? setting is on. Usage limit quotas count total token usage per month.

Response Length

The maximum response length is set with model parameters:

Parameter Applies to
max_new_tokens OpenAI-family models. On the Anthropic Messages provider, it is sent as max_tokens.
max_tokens Claude on Amazon Bedrock and the Anthropic Messages provider. On Bedrock, max_new_tokens is not applied.

Set these parameters on the AI model, or override them for an agent in Workflow main model parameters on the Workflow tab. Callers can also override model parameters for a single Core API completion request:

{
  "agent_name": "my-agent",
  "session_id": "{sessionId}",
  "user_prompt": "Explain quantum computing",
  "settings": {
    "model_parameters": {
      "max_new_tokens": 2000
    }
  }
}

For thinking and reasoning models, the output limit covers both the thinking tokens and the visible response. If responses are truncated, raise the limit or lower the reasoning effort. For model-specific guidance, see AI Models and Claude on Amazon Bedrock.

The context window size depends on the model and deployment. Check the model provider's documentation for the limits of your deployment.

Agent Settings That Affect Token Use

Setting Effect
Conversation History > Max Messages (conversation_history_settings.max_history) Limits how many previous conversation turns are sent with each request.
Max rounds (workflow.properties.max_rounds) Limits how many model rounds the FoundationaLLM function-calling workflow runs for one request.
Enable tool escalation (workflow.properties.enable_tool_escalation) Adds a lightweight first pass that binds only the tools the request needs.

Platform Limits

Limit Value Configuration
Maximum HTTP request body size (Core API, Management API, Context API, Gateway API) 512 MiB Fixed.
Files per conversation message 10 (default) FoundationaLLM:APIEndpoints:CoreAPI:Configuration:MaxUploadsPerMessage
File types allowed for upload Configured list of extensions FoundationaLLM:APIEndpoints:CoreAPI:Configuration:AllowedUploadFileExtensions
Maximum size of files used directly in the completion context, by extension Configured per extension FoundationaLLM:APIEndpoints:ContextAPI:Configuration:FileService:KnowledgeSearchContextFileMaxSizeBytes
Core API Branding endpoint 100 requests per minute per client IP address Fixed.
Gateway API completion request size 20,000 tokens (default) FoundationaLLM:APIEndpoints:GatewayAPI:Configuration:CompletionRequestMaxSizeTokens
Gateway API embedding request size 6,400 tokens (default) FoundationaLLM:APIEndpoints:GatewayAPI:Configuration:EmbeddingRequestMaxSizeTokens
Gateway API token rate budget The Azure OpenAI deployment's token rate limit multiplied by this factor FoundationaLLM:APIEndpoints:GatewayAPI:Configuration:TokenRateLimitMultiplier

The Gateway API brokers calls to Azure OpenAI deployments for platform operations such as embedding in data pipelines. Agent workflows in the LangChain API call their models through the AI model's API endpoint configuration, not through the Gateway API.

Agent Access Tokens

Constraint Description
Agent scope Each token is bound to a single agent.
Expiration Each token has an expiration date, set when it is created. Expired tokens are rejected.
Permissions Token holders get the roles assigned to the agent's virtual security group (by default, Reader on the agent).
Quotas Rate limit and usage limit quotas can apply to an individual token.

See Agent Access Tokens.

Handling Limits in Client Applications

  1. Retry 429 responses after time_until_retry_seconds, or with exponential backoff when the value is not present.
  2. Use asynchronous completions (async-completions) for long-running requests, and poll the status endpoint at a moderate interval. Status polling counts against CoreAPI:CompletionsStatus rate limits.
  3. Watch for usage limit errors, which do not clear until the next month or until an administrator changes the quota.
import time
import requests

def post_with_retry(url, headers, body, max_retries=3):
    for attempt in range(max_retries):
        response = requests.post(url, headers=headers, json=body)
        if response.status_code != 429:
            return response
        retry_after = response.json().get('time_until_retry_seconds') or 2 ** attempt
        time.sleep(retry_after)
    raise Exception('Max retries exceeded')