Completion telemetry & log correlation

This document describes the end-to-end telemetry/log correlation flow for FoundationaLLM completion requests and provides a starter pack of Log Analytics / Application Insights queries to investigate them.

The goal of the design is simple:

Given a single OperationId, retrieve every span and log line produced by every service involved in that completion — across CoreAPI, OrchestrationAPI, GatewayAPI, StateAPI, and LangChainAPI.

1. Starter pack: Log Analytics queries

All queries assume the FoundationaLLM services export their telemetry to a single Application Insights / Log Analytics workspace via OpenTelemetry + Azure Monitor.

The correlation identifiers (OperationId, InstanceId, ConversationId) are surfaced as both span attributes (on requests / dependencies) and log custom dimensions (on traces / exceptions). Either source can be used for filtering.

1.1 All telemetry for a single completion

let opId = "<operation-id>";
union AppTraces, AppDependencies, AppRequests, AppExceptions
| where Properties.["FoundationaLLM-OperationId"] == opId
| project TimeGenerated, Type, AppRoleName, SeverityLevel, Name, Message,
          OperationId, ParentId, Properties
| order by TimeGenerated asc

This is the primary query. It returns the chronological timeline of every record produced by every service for a given OperationId.

1.2 Same query, but excluding noisy "infrastructure" traffic

The two main sources of noise when investigating a completion are:

Status / health-check endpoints (GET /status, GET /instances/{id}/status).
Status polling traffic — GET .../async-completions/{operationId}/status is called every few seconds while a long-running completion is in flight, producing many AppRequests / AppDependencies / AppTraces entries that share the same OperationId but are not interesting on their own.

let opId = "<operation-id>";
let statusPollPath = @"/async-completions/[^/]+/status$";
let statusHealthPath = @"^(/status|/instances/[^/]+/status)$";
union AppTraces, AppDependencies, AppRequests, AppExceptions
| where Properties.["FoundationaLLM-OperationId"] == opId
// Drop health-check requests/dependencies.
| where not(Type in ("AppRequest", "AppDependency")
            and tostring(coalesce(Url, Name)) matches regex statusHealthPath)
// Drop status-polling requests/dependencies.
| where not(Type in ("AppRequest", "AppDependency")
            and tostring(coalesce(Url, Name)) matches regex statusPollPath)
| project TimeGenerated, Type, AppRoleName, SeverityLevel, Name, Message,
          OperationId, ParentId, Properties
| order by TimeGenerated asc

Tip: tweak the two regex variables (statusPollPath, statusHealthPath) to match any additional polling/health endpoints you want to exclude.

1.3 Errors and warnings only, for a single completion

let opId = "<operation-id>";
union AppTraces, AppExceptions, AppRequests, AppDependencies
| where Properties.["FoundationaLLM-OperationId"] == opId
| where (Type == "AppTrace"      and SeverityLevel >= 2)    // Warning+
     or (Type == "AppException")
     or (Type == "AppRequest"    and Success == false)
     or (Type == "AppDependency" and Success == false)
| project TimeGenerated, Type, AppRoleName, SeverityLevel, Name, Message,
          ResultCode, Properties
| order by TimeGenerated asc

1.4 All completions for a single conversation (session)

let convId = "<conversation-id>";
AppTraces
| where Properties.["FoundationaLLM-ConversationId"] == convId
| summarize firstLog = min(TimeGenerated),
            lastLog  = max(TimeGenerated),
            services = make_set(AppRoleName)
          by tostring(Properties.["FoundationaLLM-OperationId"])
| order by firstLog desc

1.5 All completions for a single user

let upn = "<user@contoso.com>";
AppRequests
| where Properties.["FoundationaLLM-UPN"] == upn
| where Name has_any ("CoreAPI_Completions_GetCompletion",
                      "CoreAPI_AsyncCompletions_StartCompletionOperation")
| project TimeGenerated, Name, ResultCode, DurationMs,
          operationId    = tostring(Properties.["FoundationaLLM-OperationId"]),
          conversationId = tostring(Properties.["FoundationaLLM-ConversationId"]),
          instanceId     = tostring(Properties.["FoundationaLLM-InstanceId"])
| order by TimeGenerated desc

1.6 End-to-end latency per service for a completion

let opId = "<operation-id>";
union AppRequests, AppDependencies
| where Properties.["FoundationaLLM-OperationId"] == opId
| summarize totalDurationMs = sum(DurationMs),
            count           = count()
          by AppRoleName, Type
| order by totalDurationMs desc

1.7 Completions that timed out or failed in the last 24h

union AppRequests, AppExceptions
| where TimeGenerated > ago(24h)
| where Properties.["FoundationaLLM-OperationId"] != ""
| where (Type == "AppRequest"   and Success == false)
     or (Type == "AppException")
| project TimeGenerated, AppRoleName, Name, ResultCode,
          operationId    = tostring(Properties.["FoundationaLLM-OperationId"]),
          conversationId = tostring(Properties.["FoundationaLLM-ConversationId"]),
          message        = coalesce(OuterMessage, Message)
| order by TimeGenerated desc

1.8 Drop status-polling noise globally (reusable snippet)

If you frequently want a "no polling" view, save the following as a function (e.g. fnNoPolling) in your Log Analytics workspace:

let statusPollPath   = @"/async-completions/[^/]+/status$";
let statusHealthPath = @"^(/status|/instances/[^/]+/status)$";
union AppRequests, AppDependencies, AppTraces, AppExceptions
| where not(Type in ("AppRequest", "AppDependency")
            and tostring(coalesce(Url, Name)) matches regex statusPollPath)
| where not(Type in ("AppRequest", "AppDependency")
            and tostring(coalesce(Url, Name)) matches regex statusHealthPath)

You can then write fnNoPolling | where Properties.["FoundationaLLM-OperationId"] == "..." for any investigation.

2. Correlation model

2.1 Trace boundary

CoreAPI is the only trace boundary for completion requests. Every inbound completion call (/completions, /async-completions, /async-completions/{operationId}/status) starts a fresh root activity in CoreAPI, regardless of what the caller sent in traceparent.

If the caller did send a traceparent, the value is preserved as an ActivityLink on the new root, so the original caller trace is still discoverable from the CoreAPI trace.

This decision keeps:

the completion trace tree clean and bounded (no risk of inheriting arbitrarily deep upstream traces),
caller traceability intact via the link,
sampling behavior unchanged (sampling is evaluated on the new root).

2.2 Business correlation via baggage

Three identifiers are propagated as OpenTelemetry baggage:

Key	Source	Meaning
`OperationId`	`CompletionRequest.OperationId`	The single id that ties everything together. Auto-generated by CoreAPI if missing.
`InstanceId`	`{instanceId}` route parameter	The FoundationaLLM instance.
`ConversationId`	`CompletionRequest.SessionId`	The conversation/session, when present.

Baggage is set once in CoreAPI by CompletionTelemetryScope.BeginCoreAPIBoundary(...) and is then automatically propagated via the standard W3C baggage header to every downstream HTTP call (Orchestration, Gateway, State, LangChain).

Identity-bearing values (UPN, UserId) are not added to baggage. They are attached as activity tags only on the boundary span, to avoid sending PII over network headers to downstream services.

2.3 Span enrichment

Both runtimes register a baggage enricher that copies the current baggage onto every started span as attributes:

.NET: FoundationaLLM.Common.Telemetry.BaggageSpanEnricher (a BaseProcessor<Activity>), registered centrally in FoundationaLLM.Common.Services.DependencyInjection.AddOpenTelemetry.
Python: foundationallm.telemetry.BaggageSpanProcessor (an OpenTelemetry SDK SpanProcessor), attached to the global TracerProvider from Telemetry.configure_monitoring.

As a result, every span (whether produced by CoreAPI, by the ASP.NET Core / FastAPI instrumentation, by HttpClient instrumentation, by LangChain spans, etc.) carries OperationId, InstanceId, and ConversationId as attributes.

2.4 Log enrichment

Spans alone are not enough — we also want log records to carry the same correlation identifiers so they appear as customDimensions in Application Insights. Two mechanisms achieve this:

.NET — ILogger.BeginScope is opened in CompletionTelemetryScope.BeginCoreAPIBoundary / BeginInbound / BeginLinked and inside PollingHttpClient<,>.ExecuteOperationAsync. The OpenTelemetry logging provider is configured with IncludeScopes = true, so all scope keys are emitted as custom dimensions on every log.
Python — foundationallm.telemetry.BaggageLoggingFilter is attached to all logging handlers (console, azure, default). It reads the well-known baggage keys from the current OTel context and copies them onto every LogRecord, where the Azure Monitor LoggingHandler picks them up as custom dimensions.

2.5 Long-running operations

For async completions, the original boundary trace ends as soon as CoreAPI returns 202 Accepted. Subsequent work happens on a separate trace tree, started by:

the polling client (CoreAPI → Orchestration/State for status),
and the LangChainAPI background task that actually produces the result.

To preserve cross-trace navigability, when an async operation is started, the W3C trace context of the boundary activity is captured and persisted on LongRunningOperationContext:

LongRunningOperationContext.TraceParent  // W3C traceparent of the start activity
LongRunningOperationContext.TraceState   // W3C tracestate, if any

Every status poll then reads these values back, parses them into an ActivityContext, and starts its own activity with an ActivityLink to the start trace via CompletionTelemetryScope.BeginLinked(...).

The OperationId baggage carries through both trace trees, so the OperationId queries in section 1 still return everything regardless of the polling boundary.

3. Per-service behavior

Service	Inbound boundary	Outbound propagation
CoreAPI	New root activity via `CompletionTelemetryScope.BeginCoreAPIBoundary`. Sets baggage + logger scope. Inbound `traceparent` preserved as `ActivityLink`.	W3C `traceparent` + `baggage` headers (HttpClient OTel).
OrchestrationAPI	`CompletionTelemetryScope.BeginInbound` on completion / async-completion / status endpoints.	W3C `traceparent` + `baggage` headers.
StateAPI	`CompletionTelemetryScope.BeginInbound` on every operation route (`/operations/{operationId}/...`).	n/a (terminal in this hop).
LangChainAPI	FastAPI/OTel instrumentation extracts `traceparent`. The completion router calls `set_completion_correlation_baggage(...)` to seed `OperationId` / `InstanceId` / `ConversationId` baggage on the current OTel context, and again inside the `asyncio.create_task` background coroutine that runs the actual completion (so the correlation survives the request scope ending).	n/a (terminal in this hop, but downstream OTel calls already carry baggage).
PollingHttpClient	n/a (client). Opens `ILogger.BeginScope` with `OperationId` so each polling log carries it as a custom dimension.	Standard W3C propagation.

4. Quick troubleshooting checklist

A query by OperationId returns nothing. Confirm the workspace is actually receiving telemetry for the time range; then confirm the request really hit CoreAPI (CoreAPI is what generates the OperationId if the caller did not supply one).
OperationId shows up on requests but not on traces. That usually means the relevant service is missing the inbound scope wiring (in .NET: CompletionTelemetryScope.BeginInbound(...) at the top of the controller method; in Python: set_completion_correlation_baggage(...)).
OperationId stops appearing after 202 Accepted. That is expected on the original trace tree — switch to querying by customDimensions.OperationId instead of operation_Id to pick up the polling/background work as well.
Status-polling noise dominates the timeline. Use the regex filters in section 1.2 (or the fnNoPolling snippet in section 1.8) to drop health-check and status-polling requests/dependencies.

Table of Contents