Table of Contents

Khalil completion latency (User Portal)

Date: 2026-08-22
Environment:

  • Cohorts A–C: User Portal http://localhost:3000 → deployed Core API (cacoreapil43jljq2i5ox6, instance 8ac6074c-bdde-43cb-a140-ec0002d96d2b)
  • Cohort D: same portal → local Core (https://127.0.0.1:63279) → local Orchestration (https://127.0.0.1:7324) → local LangChain (http://127.0.0.1:8765) on this branch (parallel title, full tool schemas)
  • Cohort E: same local stack after two-pass tool escalation (enable_tool_escalation) Agents: Khalil (featured, unchanged) and KhalilZoomies (clone used to measure the prompt-policy change)
    Method: V001-style User Portal chat (new conversation, send, wait for completion). Timings from the message timestamp tooltip (processingTime) and from the MAA-Workflow content-artifact timer metadata.

Khalil is a full-featured function-calling agent (MAA-Workflow, max_rounds = 5) with MCP memory and other tools. Production Khalil was not edited. The prompt-policy experiment lives on KhalilZoomies.

Executive summary

Plain V001 prompts such as “What is the capital of France?” do not stay tool-free on Khalil. The agent prompt requires at least one tool on every turn and requires GetMemory at the start of every conversation. Measured user-visible latency for those turns is 3.9–18.5 s (median ~7.1 s).

When the documented exception is used (Do not use a tool. …), Khalil complies. Those turns complete in 2.2–3.9 s, with a single LLM call of 1.4–2.0 s. About 0.7–1.0 s of the remaining time is platform overhead (async start, polling, Cosmos/status hops). On the first message of a new chat, conversation-title generation adds ~1.0 s on the critical path when LangChain is sequential (deployed host, Cohorts B–C). This branch starts that call in parallel; Cohort D workflow time was 1.44 s for a 1.38 s answer + 1.30 s title.

KhalilZoomies applies that prompt change (answer trivia/greetings/math directly; tools only when they add value; no mandatory GetMemory / Ask). Same V001 prompts, no “do not use a tool” prefix, completed in 1.95–4.40 s on the measured trivia/greeting turns — matching Cohort B without the exception. Memory write still used MCP when the user asked to remember something.

Cohort E (this PR) adds two-pass tool escalation on Zoomies. Pass 1 binds only synthetic request_tools plus the short main prompt. Trivia answers immediately (tool_escalation=answered); memory/search turns name the tools and Pass 2 binds that subset (fail_open loads every tool). A warm 10/10 replay of every Cohort A prompt confirmed the lift: no-tool workflows 0.91–1.43 s and 815–903 tokens vs Khalil’s 4.62–18.54 s / 14k–49k. “Remember this” escalated to AppendMemory only; in-thread recall answers from history (not GetMemory).

The largest controllable costs are therefore:

  1. Forced tool use (extra LLM round + MCP/Ask hops + 14k–51k prompt tokens).
  2. Binding the full tool surface even when no tool runs (~6.2k prompt tokens) — addressed for Zoomies in Cohort E.
  3. Sequential conversation naming after the answer.
  4. Async completion + 250 ms polling + per-poll token-usage refresh.

How completions actually run

User Portal chat does not use the unused 2 s pollForCompletion helper. The live path is:

  1. POST /instances/{id}/async-completions (api.sendMessage).
  2. Core API starts a long-running operation: load agent + conversation, prepare history, write user/agent/prompt rows to Cosmos, explode the agent (workflow, models, every tool, optional code sessions), create a State API operation, then hand off to Orchestration → LangChain.
  3. LangChain runs FoundationaLLMFunctionCallingWorkflow (MAA-Workflow): bind all tools, tool_choice='auto', up to max_rounds (Khalil: 5). Each round can call tools, then another LLM turn.
  4. On a new conversation, get_conversation_name makes a second LLM call after the answer, still inside the operation.
  5. The portal polls GET .../async-completions/{operationId}/status every 250 ms (backs off after 20 s / 60 s) and also calls token-usage on every poll.
  6. User-visible processingTime is agent.timeStamp - user.timeStamp. Core stamps the agent message when ProcessCompletionResponse finishes.

Relevant code:

  • Portal send/poll: src/ui/UserPortal/js/api.ts, src/ui/UserPortal/stores/appStore.ts
  • Core start/status: src/dotnet/CoreAPI/Controllers/CompletionsController.cs, src/dotnet/Common/Services/CompletionService.cs
  • Object explosion: src/dotnet/Orchestration/Orchestration/OrchestrationBuilder.cs
  • Workflow: src/python/plugins/agent_core/pkg/foundationallm_agent_plugins/workflows/foundationallm_function_calling_workflow.py

Benchmark results

Cohort A — V001-style prompts (no “do not use a tool”)

Khalil used tools on every turn (GetMemory always; Ask on the France question; memory write on “remember this”).

# Prompt (V001 analog) New chat Tooltip (s) Workflow (s) Prompt / completion tokens Tools Reply
1 Reply with only the word OK. Yes 6.31 (GetMemory 0.72; LLM 1.71 + 1.29) 14,756 / 35 GetMemory OK
2 What is the capital of France? (3.1) Yes 9.45 8.38 24,070 / 65 GetMemory 0.34, Ask 1.00 Paris 🇫🇷
3 What is 2+2? No 7.08 — 24,426 / 66 Yes (MCP) 4
4 Name the three primary colors. No 18.54 — 48,772 / 104 Yes (MCP) red/yellow/blue (+ RGB note)
5 V001 merge smoke: reply with OK (11) Yes 4.62 — 14,758 / 34 GetMemory OK
6 My favorite color is blue. Remember this. (3.6) Yes 11.01 — 51,678 / 104 Memory write remembered blue
7 What is my favorite color? (3.6) No 3.92 — 14,883 / 31 GetMemory Blue
8 Reply with only the word hello. Yes 7.86 — 14,856 / 34 GetMemory hello
9 In two sentences, explain what a compiler does. No 7.66 — 24,390 / 105 Yes (MCP) two-sentence definition
10 Reply with only the word yes. No 6.50 5.51 14,967 / 23 GetMemory 2.40 yes

Cohort A: min 3.92 s, median ~7.1 s, mean ~8.3 s, max 18.54 s.

France workflow breakdown (9.45 s tooltip / 8.38 s workflow):

Step Seconds
workflow_llm_call_1_5 (tool selection) 2.02
tool_call_GetMemory 0.34
workflow_llm_call_2_5 1.64
tool_call_Ask 1.00
workflow_llm_call_3_5 (final answer) 1.88
conversation_name_generation 1.05
workflow total 8.38
Platform / poll gap (tooltip − workflow) ~0.83

“yes” breakdown (6.50 s tooltip / 5.51 s workflow): LLM 1.31 + GetMemory 2.40 + LLM 1.73. No title generation (not the first message).

Cohort B — explicit no-tool (prompt exception)

Khalil’s prompt allows skipping tools only when the user says so. These turns produced only an MAA-Workflow artifact (no mcp pills, no “Running tools…”).

# Prompt New chat Tooltip (s) Workflow (s) LLM call (s) Title gen (s) Prompt / completion tokens Reply
1 Do not use a tool. Reply with only the word OK. Yes 3.89 2.85 1.77 1.05 6,230 / 9 OK
2 Do not use a tool. What is the capital of France? No 2.83 1.44 1.40 — 6,254 / 10 Paris.
3 Do not use a tool. What is 2+2? No 2.16 1.45 1.43 — 6,279 / 5 4
4 Do not use any tools. In two sentences, explain what a compiler does. Yes 3.94 2.93 1.95 0.95 6,233 / 51 two sentences

Cohort B: min 2.16 s, median ~3.4 s, max 3.94 s.

Follow-up no-tool turns (no title generation): ~2.2–2.8 s user-visible, ~1.4 s in the workflow, ~0.7–1.4 s outside it.

Cohort C — KhalilZoomies (prompt policy only; no “do not use a tool”)

KhalilZoomies keeps Khalil’s workflow/model (FoundationaLLMFunctionCallingWorkflow, GPT-5-4, max_rounds = 5) and tools. The main system prompt was rewritten so the model answers directly for facts, greetings, math, and short replies, and uses tools only when they add value. Production Khalil was left unchanged.

That prompt is instance data (KhalilZoomies-Workflow-main_prompt in the Prompt resource provider), not a compiled artifact. The exact text used for Cohorts C–E and the 10/10 replay is checked in at khalil-zoomies-main-prompt.txt. Restore it with a Management API POST to that prompt resource (parent FoundationaLLM.Agent|agents|KhalilZoomies) if the child prompt is missing again. Escalation runs only when the workflow property enable_tool_escalation is set.

# Prompt (V001 analog) New chat Tooltip (s) Workflow Prompt / completion tokens Tools Reply
1 Reply with only the word OK. Yes 4.40 LLM 1.49 + title 1.04 6,175 / 17 (workflow 6,122 / 9) none (MAA only) OK
2 What is the capital of France? Yes 3.40 LLM 1.32 + title 0.71 6,181 / 22 (workflow 6,122 / 15) none (MAA only) Paris 🇫🇷
3 What is 2+2? No 1.95 single LLM (no title) 12,292 / 26 none (MAA only) 4 ✨
4 My favorite color is blue. Remember this. Yes ~7–8 (completed between 6–8 s polls) MCP + MAA 14,669 / 65 mcp + MAA-Workflow remembered blue
5 What is my favorite color? No 2.51 single LLM (history; no GetMemory) 12,312 / 34 none (MAA only) blue 💙

Comparison on the same prompts:

Prompt Khalil A (forced tools) Khalil B (explicit no-tool) KhalilZoomies (policy)
OK (new chat) 6.31 s 3.89 s 4.40 s
France (new chat) 9.45 s ~3.9 s (new-chat floor) 3.40 s
2+2 (follow-up) 7.08 s 2.16 s 1.95 s
Remember color 11.01 s — ~7–8 s (tool still used)
Recall color 3.92 s (GetMemory) — 2.51 s (history only)

The prompt change delivers most of the Cohort B win without asking the user to opt out of tools. France dropped from 9.45 s to 3.40 s because GetMemory + Ask + two extra LLM rounds no longer run. First-turn title generation is still sequential on the deployed LangChain host (~0.7–1.0 s). This PR starts that title request in parallel with the answer; Cohort D measured that path on local LangChain. Cohort E (this PR) then replaces the 14-tool bind with a Pass 1 request_tools call so no-tool turns drop from ~6.2k to ~740 prompt tokens.

Cohort D — KhalilZoomies on local LangChain (title in parallel)

Same France prompt on KhalilZoomies, first user message of a new chat, against the local stack built from this branch (FOUNDATIONALLM_URL_OVERRIDE_* and portal LOCAL_API_URL pointed at 127.0.0.1). LangChain logs show the title LLM and the answer LLM starting 51 ms apart (17:57:19.745 and 17:57:19.796).

Metric Seconds
workflow_llm_call_1_5_execution_time (answer) 1.38
conversation_name_generation_execution_time 1.30
workflow_execution_time 1.44
Tooltip processingTime 58.11
Prompt / completion tokens 6,122 / 15 (chip 6,168 / 22)
Tools none (MAA only)
Reply Paris 🇫🇷
Sidebar title Capital of France

Parallel is proven: workflow total is max(1.38, 1.30) + 0.06, not 1.38 + 1.30. Sequential title would have been ~2.68 s inside the workflow. The title call finished first (LangChain [llm/end] 1.29s); the await after the answer added almost nothing.

The 58.11 s tooltip is not the workflow. Local Core and Orchestration spent ~50–56 s on cold Azure SDK calls before LangChain accepted POST /async-completions at 17:57:16 (many Azure.Core “Response … took 10.0s” warnings). Deployed Core does not pay that tax; Cohort C France was 3.40 s user-visible with sequential title. On a warm deployed host the expected first-turn saving from this change is the title time that no longer sits on the critical path (~0.7–1.3 s).

During “Thinking” / “Planning my next moves…” both token chips stayed at 0 and only updated when the operation completed, which matches the portal change (token-usage refresh on terminal status only).

Cohort E — KhalilZoomies two-pass tool escalation (local LangChain)

Same local stack as Cohort D after this PR’s escalation path loaded. Zoomies workflow prompts that had gone missing (KhalilZoomies-Workflow-main_prompt and siblings) were recreated via Management API before the run; production Khalil was not edited. Completions were started through local Core (POST /async-completions) and confirmed in the User Portal.

# Prompt New chat Escalation Bound tools Workflow (s) Escalation LLM (s) Title (s) Prompt / completion tokens (MAA) Reply
1 What is the capital of France? Yes answered (none) 1.42 1.42 1.15 (parallel) 739 / 11 Paris 🇫🇷
2 What is 2+2? No answered (none) 1.22 1.22 — 763 / 9 4 ✨
3 My favorite color is blue. Remember this. Yes escalated AppendMemory 4.95 1.93 1.21 (parallel) 5,745 / 89 (Pass 1 741; Pass 2 first LLM 1,284) remembered blue

France vs prior Zoomies local run (same prompt, first turn):

Metric Cohort D (full tool bind) Cohort E (escalate)
tool_escalation (not present) answered
Bound tools on the answer call all 14 none (request_tools only on Pass 1)
Answer / Pass 1 prompt tokens 6,122 739 (−88%)
workflow_llm_call_1_5 1.38 s skipped (Pass 1 answered)
workflow_execution_time 1.44 s 1.42 s
Title generation 1.30 s parallel 1.15 s parallel
OpenAI tools on the answer request 14 agent tools request_tools only

LangChain logs for the three turns show Pass 1 requests with only request_tools. The remember turn then called request_tools with AppendMemory and Pass 2 bound only that tool (not fail-open to 14). Pass 2 first LLM was 1,284 tokens vs Cohort D’s 6,122. After the MCP write, a second selection call grew to 3,720 tokens because the tool result entered the thread — still far below Cohort A’s 14k–51k.

Workflow wall on no-tool turns did not drop in proportion to tokens. GPT-5.4 still spent ~1.2–1.4 s on a 9–11 token answer. The win is payload size (and headroom as more tools are added), plus cheaper escalate turns. User-visible first-turn France was again dominated by cold local Azure SDK (~46 s Core timestamps / ~49 s poll); warm follow-up 2+2 was 3.37 s between user and agent timestamps (workflow 1.22 s + ~2.1 s platform). Remember user-visible 7.36 s / workflow 4.95 s (Pass 1 + AppendMemory 0.84 s + two Pass 2 LLMs). That matches Cohort C’s ~7–8 s memory write, but with a much smaller first bind.

Portal confirmation: Khalil Zoomies chat Capital of France shows Paris 🇫🇷 and 4 ✨ with MAA-Workflow only (no mcp pills). Favorite Color Preference shows the remember reply with mcp + MAA-Workflow, which is the intended escalate path.

Cohort A replay — all 10 original Khalil prompts on Zoomies + escalation

Warm local stack (Core / Orchestration / LangChain already exercised). Same 10 Cohort A prompts, same new-chat / follow-up grouping. First replay: 9/10. Recall (#7) escalated to GetMemory (4.13 s / 5,898 tokens) and was not faster than Khalil A’s 3.92 s tooltip — a regression vs Cohort C’s history-only 2.51 s. Pass 1 now prefers in-thread facts over memory tools. Second replay after that prompt change: 10/10.

# Prompt New chat Khalil A tooltip (s) Khalil A tokens Zoomies E workflow (s) Zoomies E tokens Escalation vs Khalil A
1 Reply with only the word OK. Yes 6.31 14,756 1.19 815 answered −5.12 s workflow vs tooltip; −94% tokens
2 What is the capital of France? Yes 9.45 24,070 0.99 815 answered −8.46 s; −97% tokens; no GetMemory/Ask
3 What is 2+2? No 7.08 24,426 1.09 839 answered −5.99 s; −97% tokens
4 Name the three primary colors. No 18.54 48,772 1.18 862 answered −17.36 s; −98% tokens
5 V001 merge smoke: reply with OK Yes 4.62 14,758 1.04 815 answered −3.58 s; −94% tokens
6 My favorite color is blue. Remember this. Yes 11.01 51,678 5.74 5,996 escalated → AppendMemory −5.27 s; still a tool turn
7 What is my favorite color? No 3.92 14,883 1.02 849 answered (history) −2.90 s; no GetMemory
8 Reply with only the word hello. Yes 7.86 14,856 1.07 815 answered −6.79 s
9 In two sentences, explain what a compiler does. No 7.66 24,390 1.43 836 answered −6.23 s
10 Reply with only the word yes. No 6.50 14,967 0.91 903 answered −5.59 s

No-tool turns: 0.91–1.43 s workflow, 815–903 prompt tokens, MAA only. Khalil A on the same lines was 4.62–18.54 s and 14k–49k tokens with GetMemory/Ask/MCP. Remember stayed a memory write (5.74 s, AppendMemory only) and still beat Khalil A 11.01 s. Recall used conversation history (1.02 s / 849 tokens) instead of Khalil’s GetMemory.

No fail-open to all 14 tools. No Ask on France. No GetMemory on trivia, greetings, or in-thread recall. Pass 1 tokens rose 739 → 815 after the history-first instruction; still ~87% below Cohort D’s 6,122.

Compare workflow seconds to Khalil tooltips only as an upper bound on lift: deployed Khalil also had ~0.7–1.0 s of platform time on top of the workflow. Local first-turn poll walls are still Azure-SDK cold starts and are not used here.

Token chips vs. user text

The portal token chip on the user bubble is the full completion prompt size, not the typed sentence. On Khalil / Cohorts B–D a six-word no-tool prompt still shows ~6,200–6,300 tokens. After Cohort E, Zoomies no-tool MAA prompt tokens are ~815; the chip can read higher because the token-usage API currently double-counts workflow rows. Forced-tool Khalil turns still show ~14,700–24,000+ because memory tool results enter later LLM calls.

What contributes to the latency

1. Khalil’s prompt forces tools (largest, config-only)

View Prompt on a live Khalil completion includes:

  • “NEVER answer a question without using at least one tool”
  • “At the START of every conversation, … call the memory tool and read what is stored. Do this every time.”
  • Exception only if the user explicitly says not to use a tool

So V001’s “simple completions that do not use a tool” cannot happen on Khalil unless the tester adds that exception. The extra cost is typically one MCP GetMemory hop (0.3–2.4 s) + one extra LLM round (1.3–2.0 s), and sometimes Ask (~1 s) plus a third LLM round.

2. Full tool schemas on every request (~6.2k tokens even with no tools)

FoundationaLLMFunctionCallingWorkflow historically always did self.workflow_llm.bind_tools(self.tools) and tool_choice='auto' when any tools exist. The no-tool cohort still sent ~6,230 prompt tokens for “OK”. That is almost entirely system prompt + tool definitions, not the user sentence. Large prompts increase TTFT even when the model returns 5–10 tokens.

Done in this PR (opt-in via enable_tool_escalation): a Pass 1 call binds only request_tools. Cohort E France used 739 tokens and never entered the workflow_llm_call_*_5 loop. Files, [ROUTER], missing tools, or an untrusted request_tools payload fail open to the full bind. Pass 1 instructions are the platform prompt Platform-Agent-Workflow-Escalation. A missing prompt also fail-opens to every tool.

The UI still shows “Planning my next moves…” / “Deciding whether tools are needed...” on the first LLM call.

3. Multi-round function calling (max_rounds = 5)

Timer keys are workflow_llm_call_{n}_5. Each round:

  • Updates State API status (“Checking if I need to use tools…”, “Running tools…”)
  • Invokes the bound LLM
  • Runs selected tools sequentially
  • Feeds tool results back for another LLM call

Trivial facts therefore pay for a ReAct-style loop even when the answer is “Paris”.

4. Conversation title generation (~1.0 s; sequential on deployed, parallel on this branch)

On deployed LangChain, get_conversation_name still runs after the answer. First-message no-tool turns: 1.77 s answer + 1.05 s title ≈ 2.85 s workflow vs 1.44 s on the next turn.

This PR starts title generation from the user prompt in parallel with the main completion (start_conversation_name_generation → asyncio.create_task). Cohort D (local LangChain): answer 1.38 s + title 1.30 s inside 1.44 s workflow.

5. Memory MCP over streamable_http

GetMemory hits https://mcp-server-ld.foundationallm.ai/memory/mcp. Observed 0.34–2.40 s. Retrieved memory is then appended into later LLM calls, which is why prompt tokens jump from ~15k to ~24k to ~49k in a three-turn chat.

6. Platform + polling gap (~0.7–1.0 s)

Tooltip time is consistently ~0.7–1.0 s above workflow_execution_time. That gap covers:

  • Core: agent + conversation RP, Cosmos history, three-item session write, LRO context upsert
  • OrchestrationBuilder: sequential GetResourceAsync for workflow, models, endpoints, every tool (and CreateCodeSession if any tool sets CodeSessionRequired, even when unused)
  • State API create + LangChain accept
  • Portal: 250 ms poll interval, plus token-usage fetch on every poll, plus conversation RP + downstream status on every poll
  • Final Cosmos patches (message, prompt, optional rename)

There is no token streaming on this path. The user waits for the whole operation.

7. History and attachment prep

PrepareCompletionRequest loads the full Cosmos conversation and, when history is enabled, sends it downstream. Combined with injected memory/tool transcripts, later turns get slower (18.54 s on the third default-path message).

Recommendations (highest impact first)

Prompt and agent configuration (no platform change)

  1. Stop requiring a tool on every Khalil turn. Allow a direct answer for greetings, trivia, and “reply with OK”. Keep “use a tool when you need current/personal/external data”.
  2. Make GetMemory conditional. Call it when the user refers to themselves, prior work, or preferences — not before “What is 2+2?”. Or load memory once per conversation in Core/Orchestration and inject a short summary, instead of an MCP round trip every turn.
  3. Do not send trivia to Ask. “Capital of France” should not pay for a knowledge tool + a third LLM call.
  4. For V001-style latency tests, either prefix Do not use a tool. or use a chat-only agent with no MCP tools. Khalil is the wrong baseline for “simple completion” if the prompt forbids tool-free answers.
  5. Shrink the main prompt and tool descriptions. ~6.2k tokens of always-on context is the floor of every Khalil call.

Workflow / LangChain

  1. Move conversation naming off the critical path. Done in this PR for FoundationaLLMFunctionCallingWorkflow: start title generation from the user prompt in parallel with the answer. Cohort D: title 1.30 s + answer 1.38 s inside a 1.44 s workflow (not 2.68 s).
  2. Avoid binding unused tools. Done in this PR as two-pass escalation (tool_escalation.py): Pass 1 binds request_tools; Pass 2 binds the named subset or fail-opens to every tool. Cohort E: France 6,122 → 739 tokens; remember bound AppendMemory only.
  3. Cache workflow LLM + bound tools per agent process instead of reconstructing them on every request.
  4. Truncate or summarize tool results before they enter conversation history so prompt tokens do not 2–3× by turn three.

Core / Orchestration / portal (platform)

  1. Stream tokens (SSE/WebSocket) instead of async + 250 ms poll. Improves perceived latency even if total time is unchanged; removes poll jitter and most status hops.
  2. Refresh token usage only when the operation completes, not on every poll (appStore.startPolling).
  3. Skip the conversation RP Get on every status poll unless authorization actually requires it.
  4. Parallelize OrchestrationBuilder resource loads. Today tool/model/endpoint explosion is sequential.
  5. Create code sessions only when a code tool is selected, not while exploding every tool on a no-tool chat turn.
  6. Cache exploded agent objects for a short TTL (Khalil’s tool graph does not change per “OK”).

Observability

  1. Surface MAA-Workflow timer fields in a first-class “request timing” view (they are already stored). The timestamp tooltip alone hides whether time was LLM, MCP, naming, or poll.
  2. Add a server span for “page → first token” and “page → complete” so portal poll delay is visible in App Insights.

Expected impact if the top items ship

Using Cohort B as the “Khalil answers without tools” floor:

Change Expected saving on a first-turn “capital of France”
Stop forced GetMemory + Ask (prompt only) ~5–7 s (9.45 s → ~3–4 s, matching Cohort B new-chat)
Name the chat after complete ~1.0 s on first turn (~3.9 s → ~2.8 s)
Stream + stop per-poll token-usage ~0.3–0.8 s perceived; much better TTFT
Smaller bound tool set / prompt Measured (Cohort E): 6,122 → 739 tokens; workflow wall still ~1.4 s on GPT-5.4

Measured on KhalilZoomies (prompt only, deployed APIs): France 9.45 s → 3.40 s; follow-up 2+2 7.08 s → 1.95 s.

Measured on KhalilZoomies (this branch, local LangChain, Cohort D): France workflow 1.44 s with title 1.30 s in parallel (answer LLM 1.38 s, 6,122 tokens). Sequential title would have been ~2.68 s in the workflow. User-visible tooltip was 58.11 s because local Core/Orchestration Azure calls were cold (~10 s each); that gap is environment overhead, not the workflow.

Measured on KhalilZoomies (this branch, local LangChain, Cohort E escalation): France answered in 1.42 s workflow / 739 tokens (title 1.15 s parallel). Follow-up 2+2 1.22 s / 763 tokens. Remember escalated to AppendMemory in 4.95 s workflow (Pass 1 741 tokens, Pass 2 first LLM 1,284). No-tool TTFT did not shrink much; the payload did.

A dedicated no-tool chat agent would still skip Pass 1 entirely. Escalation keeps tools available without paying for their schemas on every trivia turn.

Reproduction

  1. Start User Portal (deployed Core API is enough):
    pwsh -File ./devtools/scripts/powershell/Manage-LocalServices.ps1 -Action Start -Services UserPortal
  2. Open http://localhost:3000, sign in, select Khalil.
  3. Cohort A: select Khalil, send the V001 prompts as written (tools will run).
  4. Cohort B: prefix Do not use a tool.
  5. Cohort C: select Khalil Zoomies (KhalilZoomies) and send the same V001 prompts without the exception. The agent main prompt must match khalil-zoomies-main-prompt.txt (NEVER require a tool on every turn; no “NEVER answer without a tool”).
  6. Cohort D (parallel title): start local Core, Orchestration, LangChain, and User Portal after az login, then repeat a first-turn Zoomies prompt. Confirm workflow_execution_time ≈ max(answer, title), not the sum.
  7. Hover the agent timestamp for processingTime; open the MAA-Workflow artifact for *_execution_time and token counts.
  8. Cohort E (escalation): same local stack with this branch’s LangChain plugins loaded (restart LangChain after plugin edits; uvicorn --reload does not watch them). New Zoomies chat, “What is the capital of France?”. Expect tool_escalation=answered, empty tool_escalation_bound_tools, no workflow_llm_call_1_5, prompt tokens ~815. A “remember this” turn should show tool_escalation=escalated and tool_escalation_bound_tools=AppendMemory (or fail open to all tools if the model names a missing tool).
  9. Full Cohort A replay: run the ten original Khalil prompts on Zoomies (same new-chat grouping). Expect 8/10 answered with no tools; #6 escalated to AppendMemory; #7 answered from history (not GetMemory).