Table of Contents

Upgrade a v1-only sandbox environment

This guide is the cutover checklist for an instance that already has sandbox v1 (Kata pods, a guest image, a Sandbox Configuration, maybe an older MCP) and needs the current platform stack.

It is assembled from the work that brought Lemon Desert and SNT Dev forward. Role assignment, MCP identity, and first-time portal resources are in Sandbox Deployment. This page is the delta: images, App Config, cluster wiring, API rolls, and class/policy republish.

The product contract is still major version 1 (Secure Sandbox Contract v1). “v1-only environment” here means old runtime and defaults, not a different public API.

What a v1-only environment is missing

A v1 cluster can often create a Kata pod. It still fails once you roll current Management / Context, because those builds require pieces v1 never shipped:

Gap Symptom on create or first use
No FilesSidecarImage AKS sandbox defaults configuration is missing FilesSidecarImage
No WorkspaceQuotaImage AKS sandbox defaults configuration is missing WorkspaceQuotaImage (ephemeral and shared-SSD)
Old guest image (DisallowUnknownFields) json: unknown field "proc_mode" on ExecuteCommand
Files sidecar with no ingress exception download_file hangs or fails; sandbox-default-deny blocks port 8081
Context not in the cluster Sidecar URL http://<sandbox>.<ns>.svc.cluster.local:8081 is unreachable (Lemon Desert ACA Context)
Stale AKSSandboxDefaults.NodePoolName Create fails: sandbox workloads cannot schedule on general-purpose pools
Policy / class still on v1 revisions New APIs send proc_mode and FUSE quota; old snapshot does not match

Existing sandboxes keep the enforcement snapshot from create time (class, policy, digest, secret references, node pool). After you fix App Config or publish a new policy revision, users must start a new conversation (or delete the old sandbox). create_or_get_sandbox is per conversation. Idle timeout and maximum lifetime are not frozen: they follow the current Sandbox Configuration after resource-provider caches refresh.

Two repositories

Do not treat one repo as sufficient.

Repository What you build and deploy from it
foundationallm (this repo) Context API, Management API, Authorization API, Orchestration API, portals. Go images: sandbox-files-api (src/go/SandboxFilesAPI), sandbox-workspace-quota (src/go/SandboxWorkspaceQuota). NetworkPolicy example: docs/docs/development/sandbox-files-networkpolicy.yaml.
foundationallm-aks-pods-sandbox Guest images (sandbox-image/Dockerfile: executor + fllm-proxy-bridge). Publish script: scripts/Publish-SandboxImage.ps1. MCP adapter image akssandbox-mcp-server and deploy-aks.ps1. Policy-domain contract and verifications.

Branch that matches this stack: zst-security-sandbox-v2 in both repos (FoundationaLLM implementation also used zst-security-sandbox-v2-core).

Product ACR for shared images is usually foundationallm.azurecr.io (subscription FoundationaLLM Shared Services). Lemon Desert also used environment ACR acrakspodssbx.azurecr.io. Pin digests, not floating tags. The cluster kubelet identity must be able to pull every digest you put in App Config or a class.

  1. Publish sidecar and quota images; confirm the cluster can pull them.
  2. Merge the new fields into AKSSandboxDefaults (do not replace the JSON blindly).
  3. Apply NetworkPolicy in the actual sandbox namespace; confirm Context is in-cluster.
  4. Point NodePoolName (and storage classes) at live Kata pools / StorageClasses.
  5. Roll Context API (required), then Management API. Restart Context after App Config changes.
  6. Publish new class revisions that pin a current guest-image digest.
  7. Publish new policy revisions if the current validator would reject a clone of v1 (timeout, destinations).
  8. Retarget configurations at those revisions. Recreate sandboxes.
  9. Confirm MCP adapter settings (Management API URL, no impersonation headers).
  10. Smoke-test create, execute, download, and (if used) proxy egress.

1. Images from foundationallm

Files sidecar (fllm-files)

Used for download_file / upload_file so large files do not travel as execute_command stdout.

az acr build --registry foundationallm --image sandbox-files-api:<tag> "src/go/SandboxFilesAPI"

Record the digest:

az acr repository show --name foundationallm --image sandbox-files-api:<tag> --query digest -o tsv

App Config field: FilesSidecarImage = foundationallm.azurecr.io/sandbox-files-api@sha256:<digest>. Port is 8081 (FilesSidecarPort).

Lemon Desert and SNT Dev both used the same product digest for the sidecar (sandbox-files-api@sha256:c8880e92…). New pods pick it up at create; old pods stay on the previous snapshot.

Workspace quota (fllm-workspace-fuse)

Current Context requires this for ephemeral and shared-SSD creates. Loop-device quota is hardcoded off (UsesLoopWorkspaceQuota returns false); both modes use FUSE.

az acr build --registry foundationallm --image sandbox-workspace-quota:1.0.2 "src/go/SandboxWorkspaceQuota"

App Config field: WorkspaceQuotaImage = foundationallm.azurecr.io/sandbox-workspace-quota@sha256:<digest>.

SNT Dev was missing this image entirely in the product ACR until it was built there. Rolling Management/Context to poc118-stream / poc118-images without this field made every create fail in 1–3 seconds.

The sidecar runs privileged with SYS_ADMIN (Bidirectional FUSE mount). The user and files containers stay unprivileged.

Platform API images

Roll at least:

Service Why
Context API Builds the pod (sidecars, quota, proc_mode, files URL). This is the service that throws the missing-image 500s.
Management API Sandbox resources, policy validator, configuration ETag, execute-command.
Authorization API Sandbox actions and PBAC (empty UPN for MCP managed identity).

A v1 Management API with a current Context (or the reverse) produces confusing 400/500s. Roll them together from the same sandbox-v2 build.

2. Images from foundationallm-aks-pods-sandbox

Guest image (class pin)

Current Context/Management always send enforcement.filesystem.proc_mode (masked or pid-namespace) and include it in bundle_digest. A v1 guest binary that uses DisallowUnknownFields rejects the request:

invalid request JSON: json: unknown field "proc_mode"

On SNT Dev this hit ephemeral (sandbox-image-basic tag poc110, August 16) but not VirtualPC (sandbox-image-preloaded was already newer).

Build from the coordination repo repo root (needs executor/ and proxy-bridge/):

# Standard Python / executor image (Dockerfile target fllm)
az acr build --registry foundationallm --image sandbox-image-basic:<tag> `
  --file sandbox-image/Dockerfile --target fllm .

Or scripts/Publish-SandboxImage.ps1 (SBOM / Cosign when you need a signed class pin).

Publish a new class revision with container_image_digest = registry/repo@sha256:…. Images live on the class, not the policy. Then point the configuration at that class revision.

Preloaded / WeasyPrint / Node variants are extra class images on top of the same executor. They still need a current fllm-sandbox-exec. Lemon Desert’s weasyprint-69 work was an incremental preloaded image; it does not replace sandbox-image-basic for ephemeral SKUs.

MCP adapter

Image: foundationallm.azurecr.io/akssandbox-mcp-server:<version>.

The adapter must call Management API with its managed identity. It must not have Kubernetes credentials, FLLM_AGENT_NAME, or SANDBOX_USER_IDENTITY_*. Those impersonate a person and ignore X-FLLM-SANDBOX-CONFIGURATION-ID.

Deployment procedure: foundationallm-aks-pods-sandbox/docs/plans/aks-deployment-plan.md and deploy-aks.ps1.

SNT Dev MCP stayed akssandbox-mcp-server:poc110 while APIs moved to poc118; that was fine for routing. Creates still failed in Context until WorkspaceQuotaImage was set.

3. App Config: AKSSandboxDefaults

Key (content type application/json):

FoundationaLLM:APIEndpoints:ContextAPI:Configuration:SandboxCodeSessionService:AKSSandboxDefaults

Merge. Read the existing object, add or update fields, write it back. Replacing the value drops cluster id, namespace, storage classes, and gateway settings.

Minimum fields current Context expects beyond a v1 blob:

Field Purpose
FilesSidecarImage Required always
FilesSidecarPort 8081
WorkspaceQuotaImage Required for ephemeral and shared-SSD
Namespace Must match the live sandbox namespace (fllm-sandbox on SNT Dev, sandboxes on Lemon Desert)
NodePoolName Fallback Kata pool (SNT Dev: live pool was sandbox3 while defaults still said sandbox2)
AKSClusterResourceId Cluster Context is allowed to use
ContainerRegistryLoginServer Extra ACR for the Management Portal image catalog (optional)
Storage class names Required for the persistence SKUs you publish (see below)
WorkspaceServiceAccountName Azure Files mount identity; do not automount the token on the user container

Example merge (PowerShell), then restart Context:

$key = "FoundationaLLM:APIEndpoints:ContextAPI:Configuration:SandboxCodeSessionService:AKSSandboxDefaults"
$store = "<app-config-name>"
$raw = az appconfig kv show --name $store --key $key --auth-mode login --query value -o tsv
$obj = $raw | ConvertFrom-Json
$obj.FilesSidecarImage = "foundationallm.azurecr.io/sandbox-files-api@sha256:<digest>"
$obj.FilesSidecarPort = 8081
$obj.WorkspaceQuotaImage = "foundationallm.azurecr.io/sandbox-workspace-quota@sha256:<digest>"
$obj.NodePoolName = "<live-kata-pool>"
az appconfig kv set --name $store --key $key --value ($obj | ConvertTo-Json -Compress) `
  --content-type application/json --yes --auth-mode login

Storage classes used in the Lemon Desert / SNT work (names may differ; they must exist in the cluster):

App Config field Typical SKU
WorkspaceStorageClassName Standard Azure Files (SMB)
WorkspaceSharedSsdStorageClassName Shared Premium SSD Files (workroom)
WorkspaceSsdStorageClassName Dedicated Premium Files
WorkspaceDiskStorageClassName Dedicated Premium SSD disk
WorkspacePremiumV2DiskStorageClassName Premium SSD v2 (zonal pools)
WorkspaceUltraDiskStorageClassName Ultra Disk (dedicated Ultra pool only)

Do not enable Ultra on a general-purpose pool.

4. Cluster

Namespace and deny/allow

v1 often has sandbox-default-deny (no ingress, no egress). The files sidecar needs an exception.

Example in this repo targets namespace fllm-sandbox and Context label app=platform-context-api in namespace fllm. Edit both before apply.

Lemon Desert used namespace sandboxes and had no allow-policy; the repo YAML as-is would have missed it.

kubectl apply -f docs/docs/development/sandbox-files-networkpolicy.yaml
# or a copy with the environment's namespace and Context selector

Context must be in-cluster

Sidecar downloads use http://<sandbox-id>.<namespace>.svc.cluster.local:8081.

Environment Context location Sidecar reachability
SNT Dev platform-context-api in fllm Works after NetworkPolicy
Lemon Desert Container App cacontextapi… with no VNet; laptop hybrid Context Cannot reach cluster DNS. Files sidecar process in the pod was fine; Context could not use it

If Context stays on ACA or a laptop, download_file will not work no matter how correct App Config is. Put Context in the AKS cluster (or give it equivalent cluster DNS + NetworkPolicy).

Node pools

Sandbox pods use nodeSelector: kubernetes.azure.com/agentpool=<name>, Kata (kata-vm-isolation), and sandbox taints. Configuration node_pool_name wins; otherwise AKSSandboxDefaults.NodePoolName.

On SNT Dev, Sandbox Tester / virtualpc-sandbox still named sandbox2 after the live pool moved to sandbox3. Fix both the default and each configuration you care about.

Pull secrets / AcrPull

Quota and files images are new repositories. AKS node identity needs AcrPull on every registry you reference. First create after a new image often looks like a hang, then ImagePullBackOff.

5. Portal resources (immutable revisions)

Classes

Publish a new revision (for example secure-sandbox-basic r3) with the new guest digest. Copy CPU / memory / workspace / tmp from the v1 revision unless you intend to change them.

SNT Dev ephemeral class r3 used sandbox-image-basic@sha256:db5a5e33… (executor that understands proc_mode), same 500m / 512Mi / 1 GiB workspace / 64 MiB tmp as r2.

The Management Portal image picker may only catalog sandbox-image-preloaded in Shared Services ACR. Basic-image pins can be done through the Management API with the portal token. Lemon Desert local tags do not appear in that catalog until they are copied to the catalog ACR.

Policies

Policies are immutable. You cannot edit r1 in place.

Current Management API validator (SandboxPolicyDefinitionValidator):

  • timeout_seconds 1–3600. v1 workroom policies with 7200s are grandfathered; a new revision must use 3600 (normalizer copies execution.timeout_seconds to the top-level field).
  • proxy-only requires a non-empty destination allowlist; none requires an empty one.
  • Destinations are host + port + http/https. Wildcards like *.blob.core.windows.net are allowed.

Guest commands have no DNS. Egress is only HTTP_PROXY / HTTPS_PROXY → 127.0.0.1:3128 → fllm-proxy-bridge. If a workload must reach a host (PyPI, Anthropic, tenant blob), add it to Allowed destinations on a new revision, then retarget the configuration.

SNT Dev sentinel-workroom-policy r1 allowed Anthropic + PyPI only. build_deliverable DNS to sentineldevtenant.blob.core.windows.net failed in the guest until r2 added that host (and *.blob.core.windows.net).

Configurations

Update in place with ETag. Re-GET before POST. Point sandbox_class and sandbox_policy at the new revisions. Optionally set node_pool_name.

Configurations you do not retarget keep the old class/policy (SNT Dev sentinel-workroom-x stayed on policy r1).

6. MCP (if the environment uses it)

After APIs and defaults work, the adapter is usually only:

  • Management API base URL and instance ID
  • Managed identity with Sandboxes Contributor + Reader on the allowed configuration + UPN-match policies (Sandbox Deployment)
  • Client headers: X-FLLM-API-Key, X-FLLM-SESSION-ID, X-FLLM-SANDBOX-CONFIGURATION-ID
  • No X-USER-IDENTITY on the external path

Reference URLs from the two environments (yours will differ):

Environment MCP URL
Lemon Desert https://mcp-server-ld.foundationallm.ai/akssandbox/mcp
SNT Dev https://mcp-dev.sentinel.unentangled.org/akssandbox/mcp

7. Smoke tests

Use a new conversation after each config/class/policy change.

  1. Create — pod Running, Kata, 3/3 containers when files + FUSE apply (user, fllm-files, fllm-workspace-fuse).
  2. Quota image present — fllm-workspace-fuse Ready (not CrashLoop). CrashLoop on an old pod can be a leftover v1 quota container; recreate.
  3. Execute — echo ok or df -B1 /workspace. Must not mention proc_mode.
  4. Download — write >32 KB under /workspace, then download_file. Confirm file_record and no content_base64 for large files. This is the Lemon Desert Phase 0 check.
  5. Proxy (proxy-only policies) — curl --noproxy '*' https://<allowed-host> fails DNS (expected). The same curl with default HTTPS_PROXY reaches the host or is denied with CONNECT 403 if not on the list.

8. What is not part of this upgrade

A host outside the sandbox (tenant web app, Azure Function, laptop) resolving *.blob.core.windows.net is a VNet / private DNS problem, not a sandbox class or policy problem.

On SNT Dev, after workroom policy r2, the guest proxy path to blob worked (HEAD → HTTP 400). Emma’s getaddrinfo ENOTFOUND was Node on sentinel-dev-tenant-app: storage public access disabled, private endpoint present, but privatelink.blob.core.windows.net had no A record for sentineldevtenant. That fix is an A record (or PE DNS zone group) plus an app restart. Do not treat it as a sandbox rollback.

Reference environments

Facts from the Lemon Desert and SNT Dev cutovers. Copy the pattern, not the names, unless you are on that instance.

Lemon Desert SNT Dev
App Config appcs-l43jljq2i5ox6 fllm-dev-app-config
Cluster aks-pods-sbx / sumatra-code-sessions fllm-dev-aks
Sandbox namespace sandboxes fllm-sandbox
Quota / files ACR acrakspodssbx.azurecr.io (also product ACR) foundationallm.azurecr.io
Context ACA, no VNet (sidecar unused until Context is in-cluster) In-cluster platform-context-api
Files NetworkPolicy Missing; default-deny allow-context-sandbox-files already present
First create-blocker after API roll Quota image + Context placement Missing WorkspaceQuotaImage in App Config (image not in product ACR)
First execute-blocker — Old sandbox-image-basic vs proc_mode
Instance ID (example) 8ac6074c-bdde-43cb-a140-ec0002d96d2b b9725462-2a46-411d-ad91-15513e60b01d

Troubleshooting

Symptom Likely cause
Create 500 in 1–3s, missing WorkspaceQuotaImage / FilesSidecarImage App Config not merged, or Context not restarted
ImagePullBackOff on fllm-workspace-fuse or fllm-files Image not in the registry the node pulls; missing AcrPull
unknown field "proc_mode" Class still pins a v1 guest digest
Could not resolve host inside the guest Expected without proxy. Add destinations and use HTTPS_PROXY
destination is not allowed / CONNECT 403 Host not on the current policy snapshot
download_file fails, sidecar Ready NetworkPolicy namespace/selector wrong, or Context not in-cluster
Create works, wrong node / Pending Configuration or defaults still name a retired pool
Policy POST 400, timeout 7200 New revision must use ≤ 3600
Configuration POST 412 Stale ETag; re-GET
MCP create ignores configuration header Adapter is sending X-USER-IDENTITY or has a fallback user identity
403 blank user name, userId = MCP object id Authorization / Management build still rejects empty UPN
Same conversation after a policy change Old snapshot; new session required