Table of Contents

Dynamic Compute

This document provides a detailed reference for the dynamic compute capabilities in FoundationaLLM. Dynamic compute enables agents to execute arbitrary code in isolated, sandboxed environments as part of their orchestration workflows.

Overview

FoundationaLLM supports multiple dynamic compute backends through a provider-based architecture. For example, when an agent's tool requires code execution (e.g., the Code tool), the platform creates a code session — an isolated container environment where code can be executed, files can be uploaded and downloaded, and results are returned to the orchestration pipeline.

The code session lifecycle is managed by the Context API (ContextAPI), which coordinates between the orchestration layer and the underlying container infrastructure.

Architecture

Request Flow

Agent Tool → OrchestrationAPI → ContextServiceClient → ContextAPI → CodeSessionService → Code Session Provider → Container Infrastructure

Key Components

Component Path Description
ICodeSessionProviderService ContextEngine/Interfaces/ Interface defining the contract for all code session providers
CodeSessionService ContextEngine/Services/ Central service that routes requests to the appropriate provider
ContextServiceClient Common/Clients/ HTTP client used by the orchestration layer to communicate with the Context API
CodeSessionsController ContextAPI/Controllers/ REST controller exposing code session endpoints
DedicatedCustomContainerProvider ContextEngine/Services/ Resolves per-user dedicated container environments

Code Session Providers

FoundationaLLM includes four code session provider implementations. Each implements the ICodeSessionProviderService interface.

1. Azure Container Apps Code Interpreter

Property Value
Provider name AzureContainerAppsCodeInterpreter
Class AzureContainerAppsCodeInterpreterService
Use case Production workloads using Azure's managed Python code interpreter
Authentication Azure Managed Identity (https://dynamicsessions.io/.default scope)

Uses Azure Container Apps Dynamic Sessions with the built-in code interpreter. The platform obtains a bearer token via ServiceContext.AzureCredential and communicates with the Azure-managed session pool endpoint. Session identifiers are derived from the conversation ID and context (tool name) to enable session reuse within the same conversation.

Session ID format: {conversationId}-{context}-{HHmmssfff} (truncated to 128 characters)

2. Azure Container Apps Custom Container

Property Value
Provider name AzureContainerAppsCustomContainer
Class AzureContainerAppsCustomContainerService
Use case Production workloads using custom Docker images on Azure Container Apps Dynamic Sessions
Authentication Azure Managed Identity (https://dynamicsessions.io/.default scope)
API version 2024-10-02-preview

Runs user-provided container images in the Azure Container Apps Dynamic Sessions infrastructure. The custom container must expose the standard code session HTTP API (see Container API Protocol). This provider uses the same Azure authentication and session management as the code interpreter, but routes requests through the custom container API format.

3. Dedicated Custom Container

Property Value
Provider name DedicatedCustomContainer
Class DedicatedCustomContainerCodeSessionService
Use case Per-user persistent container environments for development and testing
Authentication None (direct HTTP)

Targets a pre-provisioned, per-user container instance. The DedicatedCustomContainerProvider resolves the correct environment based on the environment type and the user's UPN. This enables scenarios where specific developers have their own container instances for debugging and iteration.

Session ID format: __dedicated_code_session_{base64Guid}__

4. Local Custom Container

Property Value
Provider name LocalCustomContainer
Class LocalCustomContainerCodeSessionService
Use case Local development and testing against a container running on the developer's machine
Authentication None (direct HTTP)

Connects to a container running locally (e.g., via Docker). This provider is not registered in the DI container by default — it is instantiated inline by CodeSessionService when an EndpointProviderOverride is present and enabled in the request.

Session ID format: __local_code_session_{base64Guid}__

Container API Protocol

Custom containers (used by the AzureContainerAppsCustomContainer, DedicatedCustomContainer, and LocalCustomContainer providers) must expose the following HTTP endpoints. All endpoints accept an identifier query parameter that identifies the code session.

Execute Code

POST {endpoint}/code/execute?identifier={codeSessionId}
Content-Type: application/json

{
    "code": "<code_to_execute>"
}

Response:

{
    "detail": {
        "output": "<stdout>",
        "error": "<stderr>",
        "results": "<execution_result>"
    }
}

Upload File

POST {endpoint}/files/upload?identifier={codeSessionId}
Content-Type: multipart/form-data

(form field "file" with binary content)

List Files

GET {endpoint}/files?identifier={codeSessionId}

Returns a JSON array of file store items, each with name, type ("file" or "directory"), and path information. The service recursively traverses directories to build a flat file listing.

Download File

POST {endpoint}/files/download?identifier={codeSessionId}
Content-Type: application/json

{
    "file_name": "<file_name>"
}

Returns the binary file content in the response body.

Delete Files

POST {endpoint}/files/delete?identifier={codeSessionId}

Deletes all files in the code session.

Note: The Azure Container Apps Code Interpreter provider uses a different API contract (the Azure-native Dynamic Sessions API at version 2024-10-02-preview) and does not follow this custom container protocol.

Configuration

App Configuration Keys

All configuration is managed through Azure App Configuration.

Azure Container Apps Dynamic Sessions — Code Interpreter

Key: FoundationaLLM:Code:CodeExecution:AzureContainerAppsDynamicSessions:CodeInterpreter

{
    "Endpoints": {
        "python": [
            "https://<region>.dynamicsessions.io/subscriptions/<sub>/resourceGroups/<rg>/sessionPools/<pool>"
        ]
    }
}

The Endpoints dictionary maps environment type names to lists of Azure Container Apps Dynamic Sessions pool endpoints.

Azure Container Apps Dynamic Sessions — Custom Container

Key: FoundationaLLM:Code:CodeExecution:AzureContainerAppsDynamicSessions:CustomContainer

{
    "Endpoints": {
        "python": [
            "https://<region>.dynamicsessions.io/subscriptions/<sub>/resourceGroups/<rg>/sessionPools/<custom-pool>"
        ]
    }
}

Same structure as the code interpreter settings, but pointing to session pools configured with custom container images.

Dedicated Custom Containers

Key: FoundationaLLM:Code:DedicatedCustomContainers

{
    "Environments": {
        "python": {
            "UserEnvironments": {
                "user@domain.com": [
                    {
                        "Endpoint": "https://my-dedicated-container.example.com",
                        "Enabled": true
                    }
                ]
            }
        }
    }
}
Field Description
Environments Top-level dictionary keyed by environment type name
UserEnvironments Dictionary keyed by user principal name (UPN, case-insensitive)
Endpoint The base URL of the dedicated container instance
Enabled Whether this environment is active for the user

Code Tool Configuration

The Code tool is configured as part of an agent's tool definition. The following properties control how the tool interacts with the code session infrastructure.

Code Tool Properties

The following properties can be set on the Code tool configuration:

Property JSON Key Type Required Description
Code Session Required code_session_required bool Yes Indicates whether the agent tool requires a code session. For the Code tool, this must always be set to true.
Endpoint Provider code_session_endpoint_provider string Yes The name of the code session provider to use (see the available values in the Endpoint Provider Values section below).
Environment Type code_session_environment_type string Yes The environment type (used to resolve the correct endpoint from the provider's configuration). Possible values are Python and ClaudeAgent (coming soon).
Endpoint Provider Overrides code_session_endpoint_provider_overrides array No Per-user overrides that redirect code execution to a custom endpoint (see more details in the Endpoint Provider Override section below).
Code Session ID code_session_id string — Set at runtime after session creation; not configured by the user.
Code Session Endpoint code_session_endpoint string — Set at runtime after session creation; not configured by the user.

[!NOTE] The code_session_language property is deprecated. Use code_session_environment_type instead, which supports arbitrary environment types beyond programming languages.

[!NOTE] The code_session_required property is used by other tools that may optionally leverage code sessions. For the Code tool, this property must always be set to true since it relies on code execution. The value of the flag is used by the Orchestration API to determine which tools require code session setup. For each such tool, a dedicated code session will be created and its identifier added to the runtime configuration passed to the tool.

Endpoint Provider Values

Value Description
AzureContainerAppsCodeInterpreter Azure Container Apps Dynamic Sessions with built-in Python interpreter
AzureContainerAppsCustomContainer Azure Container Apps Dynamic Sessions with a custom container image
DedicatedCustomContainer Per-user dedicated container instance (resolved via App Configuration)

Endpoint Provider Override

The code_session_endpoint_provider_overrides property allows redirecting code execution to a different endpoint on a per-user basis. When an override is present and enabled, the platform bypasses the configured provider and instantiates a LocalCustomContainerCodeSessionService targeting the override endpoint.

{
    "code_session_endpoint_provider_overrides": [
        {
            "upn": "developer@domain.com",
            "endpoint": "http://localhost:8080",
            "enabled": true
        }
    ]
}
Field Type Description
upn string The user principal name this override applies to
endpoint string The base URL of the custom container endpoint
enabled bool Whether the override is active

Security note: Override endpoints should only point to trusted, internal, and appropriately secured services. In production environments, use this feature sparingly and under governance controls to avoid accidental or malicious redirection of code sessions.

Code Session Lifecycle

  1. Session Creation — The orchestration layer calls ContextServiceClient.CreateCodeSession with the agent name, conversation ID, context (tool name), endpoint provider, and environment type. The Context API creates a session via the resolved provider and persists a ContextCodeSessionRecord to Cosmos DB.

  2. File Upload — Before code execution, files from the conversation context can be uploaded to the session. The platform retrieves file content from the file service and uploads each file to the container. Existing files are deleted before new uploads (clean-slate approach).

  3. Code Execution — Code is sent to the container's execution endpoint. The response includes standard output, standard error, and an execution result string.

  4. File Download — Output files generated by the code execution can be downloaded from the container's file store.

  5. Session Tracking — All code session records and file upload records are persisted to the Context Cosmos DB container for auditing and session recovery.

Provider Resolution Logic

The CodeSessionService.GetCodeSessionProviderService method resolves the provider in the following order:

  1. Endpoint Provider Override — If the request includes an EndpointProviderOverride with Enabled = true, a LocalCustomContainerCodeSessionService is instantiated targeting the override endpoint. This takes precedence over all other providers.

  2. Named Provider — If no override is present, the provider is resolved by the endpoint_provider name:

    • AzureContainerAppsCodeInterpreter → AzureContainerAppsCodeInterpreterService
    • AzureContainerAppsCustomContainer → AzureContainerAppsCustomContainerService
    • DedicatedCustomContainer → Resolved via DedicatedCustomContainerProvider using environment type and UPN

  3. Fallback — If the provider name is not recognized, a 400 Bad Request error is returned.