Table of Contents

Azure Data Lake as a Knowledge Source

Learn how to configure Azure Data Lake Storage Gen2 as a data source and use it in a data pipeline.

Overview

An Azure Data Lake data source points FoundationaLLM at a storage account and a set of folders. A data pipeline then reads files from those folders, extracts text, and indexes it so agents can search it.

Prerequisites

  • An Azure Data Lake Storage Gen2 account with the containers and folders that hold your documents.
  • Network access from the FoundationaLLM services to the storage account.
  • The Data Sources Contributor role at the instance scope, so the Create Data Source button is shown.

Note: If you don't see the Create Data Source button, your account most likely doesn't have the Data Sources Contributor role at the instance scope. In the sidebar, select Security > Access Control to add the role assignment, or ask an administrator to do so.

Create the Data Source

  1. In the sidebar, select Data > Data Sources, then select Create Data Source.

  2. Enter a Data source name (for example, corporate-docs-adls) and an optional Data description.

  3. Under What is the type of the data source?, select Azure Data Lake.

  4. Select an Authentication type and fill in the fields it shows:

    Authentication type Fields
    Azure Identity Account name: the storage account name (for example, mystorageaccount).
    Account Key API Key: a storage account access key. Endpoint: the storage endpoint URL (for example, https://mystorageaccount.dfs.core.windows.net).
    Connection String Connection string: the full storage connection string.

  5. Under Folder(s), type each folder path and press Enter or , to add it. The first segment of a path is the container name, for example:

    • mycontainer
    • mycontainer/documents
    • mycontainer/departments/hr/policies

  6. Optionally enter a cost center.

  7. Select Create Data Source.

Connection strings and account keys are stored as Key Vault-backed values.

Authentication Options

With Azure Identity, FoundationaLLM uses its managed identity to read the storage account, so no secret is stored. Grant the FoundationaLLM managed identity a data-plane role on the storage account, such as Storage Blob Data Reader:

az role assignment create \
  --assignee <managed-identity-object-id> \
  --role "Storage Blob Data Reader" \
  --scope /subscriptions/<sub-id>/resourceGroups/<rg>/providers/Microsoft.Storage/storageAccounts/<storage-account>

Account Key or Connection String

In the Azure portal, open the storage account and select Access keys to copy a key or connection string. Account keys grant full access to the account, so rotate them regularly and prefer Azure Identity in production.

Use the Data Source in a Data Pipeline

  1. In the sidebar, select Data > Data Pipelines, then select Create Pipeline.
  2. On the Stages tab, select the Data Source node on the canvas.
  3. In the Data Source dialog, select your data source in the Data source dropdown, then select a Data source plugin. The standard package provides Azure Data Lake Data Source (FoundationaLLM).
  4. Set the plugin's Folders parameter to the folders this pipeline should process (the first part of each value is the container name).
  5. Add stages such as text extraction, partitioning, embedding, and indexing, then add a trigger.

See Creating Data Pipelines for the full editor walkthrough.

Supported File Types

The standard Text Extraction Data Pipeline Stage handles:

Category File types
Documents (through content text extraction dependencies) PDF, DOCX, PPTX, XLSX
Plain text, code, and data (read directly) TXT, MD, HTML, XML, JSON, JSONL, YAML, CSV, TSV, RTF, TEX, TOML, INI, CSS, and common source code files (C, CPP, CS, JAVA, JS, TS, PY, PHP, RB, SH)

Other file types, including images, legacy Office formats (DOC, PPT, XLS), and archives, are reported as unsupported for that work item.

Troubleshooting

  • Authentication failures: Confirm the managed identity role assignment, or check that the key or connection string hasn't been rotated.
  • Access denied or no content found: Check that the container and folder paths are correct and that the storage firewall allows FoundationaLLM services.
  • Network errors: If you use private endpoints, confirm the virtual network and DNS configuration.
  • Run failures: Open Data > Data Pipeline Runs and use the Analyze run action to see errors per stage. See Data Pipeline Runs.