Data Pipeline Plugins Reference

This document provides comprehensive technical specifications for all data pipeline plugins available in FoundationaLLM. These plugins are defined in PluginPackageManager.cs in the DataPipelinePlugins project.

Overview

Data pipeline plugins are modular components that enable data processing workflows in FoundationaLLM. Plugins are organized into the following categories:

Data Source Plugins: Define where and how data is read from
Data Pipeline Stage Plugins: Process data through various transformation stages
Content Text Extraction Plugins: Extract text from specific file formats
Content Text Partitioning Plugins: Split text into manageable chunks

Plugin Categories

Category Hierarchy

FoundationaLLM Data Pipeline Plugins
├── Data Source Plugins (3)
├── Data Pipeline Stage Plugins (11)
├── Content Text Extraction Plugins (6)
└── Content Text Partitioning Plugins (2)

Data Source Plugins

Data source plugins define how the pipeline reads input data from various sources.

Azure Data Lake Data Source

Plugin Name: AzureDataLakeDataSource
Category: DataSource
Description: Provides the FoundationaLLM standard implementation for Azure Data Lake data sources.

Parameters:

Parameter	Type	Required	Description
`Folders`	Array	Yes	List of strings defining data lake folders (the first part identifies the container name).

Example Configuration:

{
  "plugin_object_id": "/instances/{instanceId}/providers/FoundationaLLM.Plugin/plugins/AzureDataLakeDataSource",
  "plugin_parameters": [
    {
      "parameter_metadata": {
        "name": "Folders",
        "type": "array"
      },
      "default_value": ["container/folder1", "container/folder2"]
    }
  ]
}

Use Cases:

Reading documents from Azure Data Lake Storage
Processing files from multiple containers or folders
Batch processing of stored documents

Context File Data Source

Plugin Name: ContextFileDataSource
Category: DataSource
Description: Provides the FoundationaLLM standard implementation for Context File data sources.

Parameters:

Parameter	Type	Required	Description
`ContextFileObjectId`	ResourceObjectId	Yes	The FoundationaLLM object identifier of the context file.
`ContentAction`	String	Yes	The action to be performed on the content item. Can be 'add-or-update' or 'remove'.

Example Configuration:

{
  "plugin_object_id": "/instances/{instanceId}/providers/FoundationaLLM.Plugin/plugins/ContextFileDataSource",
  "plugin_parameters": [
    {
      "parameter_metadata": {
        "name": "ContextFileObjectId",
        "type": "resource-object-id"
      },
      "default_value": "/instances/{instanceId}/providers/FoundationaLLM.Context/contextFiles/{contextFileId}"
    },
    {
      "parameter_metadata": {
        "name": "ContentAction",
        "type": "string"
      },
      "default_value": "add-or-update"
    }
  ]
}

Use Cases:

Processing individual context files
Adding or updating specific content items
Removing content from indexes

SharePoint Online Data Source

Plugin Name: SharePointOnlineDataSource
Category: DataSource
Description: Provides the FoundationaLLM standard implementation for SharePoint Online data sources.

Parameters:

Parameter	Type	Required	Description
`DocumentLibraries`	Array	Yes	List of strings defining SharePoint Online document libraries.

Example Configuration:

{
  "plugin_object_id": "/instances/{instanceId}/providers/FoundationaLLM.Plugin/plugins/SharePointOnlineDataSource",
  "plugin_parameters": [
    {
      "parameter_metadata": {
        "name": "DocumentLibraries",
        "type": "array"
      },
      "default_value": ["Shared Documents", "Archive"]
    }
  ]
}

Use Cases:

Reading documents from SharePoint document libraries
Processing shared team documents
Indexing collaborative content

Data Pipeline Stage Plugins

Data pipeline stage plugins perform specific transformations on data as it flows through the pipeline.

Text Extraction Stage

Plugin Name: TextExtractionDataPipelineStage
Category: DataPipelineStage
Description: Provides the FoundationaLLM standard implementation for text extraction data pipeline stages.

Parameters:

Parameter	Type	Required	Description
`MaxContentSizeCharacters`	Int	No	The maximum allowed content item size in characters.

Dependencies: Requires one or more content text extraction plugins (PDF, DOCX, PPTX, XLSX, Image, ImageMetadata)

Example Configuration:

{
  "plugin_object_id": "/instances/{instanceId}/providers/FoundationaLLM.Plugin/plugins/TextExtractionDataPipelineStage",
  "plugin_parameters": [
    {
      "parameter_metadata": {
        "name": "MaxContentSizeCharacters",
        "type": "int"
      },
      "default_value": 1000000
    }
  ],
  "plugin_dependencies": [
    {
      "selection_type": "multiple",
      "dependency_plugin_names": [
        "PDFContentTextExtraction",
        "DOCXContentTextExtraction",
        "PPTXContentTextExtraction",
        "XLSXContentTextExtraction",
        "ImageContentTextExtraction",
        "ImageMetadataTextExtraction"
      ]
    }
  ]
}

Use Cases:

Extracting text from various document formats
Converting binary files to text for further processing
Handling multiple file types in a single pipeline

Text Partitioning Stage

Plugin Name: TextPartitioningDataPipelineStage
Category: DataPipelineStage
Description: Provides the FoundationaLLM standard implementation for text partitioning data pipeline stages.

Parameters:

Parameter	Type	Required	Description
`PartitioningStrategy`	String	Yes	Strategy used to partition text (can be Token or Semantic).

Dependencies: Requires exactly one content text partitioning plugin (Token or Semantic)

Example Configuration:

{
  "plugin_object_id": "/instances/{instanceId}/providers/FoundationaLLM.Plugin/plugins/TextPartitioningDataPipelineStage",
  "plugin_parameters": [
    {
      "parameter_metadata": {
        "name": "PartitioningStrategy",
        "type": "string"
      },
      "default_value": "Token"
    }
  ],
  "plugin_dependencies": [
    {
      "selection_type": "single",
      "dependency_plugin_names": [
        "TokenContentTextPartitioning",
        "SemanticContentTextPartitioning"
      ]
    }
  ]
}

Use Cases:

Splitting large documents into manageable chunks
Creating overlapping text segments for better context
Preparing text for embedding models

Azure AI Content Safety Shielding Stage

Plugin Name: AzureAIContentSafetyShieldingDataPipelineStage
Category: DataPipelineStage
Description: Provides the FoundationaLLM standard implementation for content shielding data pipeline stages.

Parameters: None

Dependencies: None

Example Configuration:

{
  "plugin_object_id": "/instances/{instanceId}/providers/FoundationaLLM.Plugin/plugins/AzureAIContentSafetyShieldingDataPipelineStage",
  "plugin_parameters": []
}

Use Cases:

Filtering inappropriate content
Content safety validation
Compliance with content policies

Gateway Text Embedding Stage

Plugin Name: GatewayTextEmbeddingDataPipelineStage
Category: DataPipelineStage
Description: Provides the FoundationaLLM standard implementation for text embedding data pipeline stages that perform text embedding using the FoundationaLLM Gateway API.

Parameters:

Parameter	Type	Required	Description
`KnowledgeUnitObjectId`	ResourceObjectId	Yes	The FoundationaLLM resource identifier of the Knowledge Unit that provides the embedding configuration.
`EmbeddingRequestSizeTokens`	Int	Yes	The maximum size in tokens of each embedding request sent to the Gateway API.

Parameter Selection Hints:

KnowledgeUnitObjectId: Filter by providers/FoundationaLLM.Context/knowledgeUnits

Example Configuration:

{
  "plugin_object_id": "/instances/{instanceId}/providers/FoundationaLLM.Plugin/plugins/GatewayTextEmbeddingDataPipelineStage",
  "plugin_parameters": [
    {
      "parameter_metadata": {
        "name": "KnowledgeUnitObjectId",
        "type": "resource-object-id"
      },
      "default_value": "/instances/{instanceId}/providers/FoundationaLLM.Context/knowledgeUnits/{knowledgeUnitId}"
    },
    {
      "parameter_metadata": {
        "name": "EmbeddingRequestSizeTokens",
        "type": "int"
      },
      "default_value": 8000
    }
  ]
}

Use Cases:

Generating vector embeddings for semantic search
Creating embeddings using configured models
Preparing data for vector database indexing

Azure AI Search Indexing Stage

Plugin Name: AzureAISearchIndexingDataPipelineStage
Category: DataPipelineStage
Description: Provides the FoundationaLLM standard implementation for indexing data pipeline stages that use Azure AI Search.

Parameters:

Parameter	Type	Required	Description
`KnowledgeUnitObjectId`	ResourceObjectId	Yes	The FoundationaLLM resource identifier of the Knowledge Unit that provides the embedding configuration.
`VectorStoreId`	String	Yes	The name of the Vector Store resource that identifies the logical partition of the Azure AI Search index.

Parameter Selection Hints:

KnowledgeUnitObjectId: Filter by providers/FoundationaLLM.Context/knowledgeUnits

Example Configuration:

{
  "plugin_object_id": "/instances/{instanceId}/providers/FoundationaLLM.Plugin/plugins/AzureAISearchIndexingDataPipelineStage",
  "plugin_parameters": [
    {
      "parameter_metadata": {
        "name": "KnowledgeUnitObjectId",
        "type": "resource-object-id"
      },
      "default_value": "/instances/{instanceId}/providers/FoundationaLLM.Context/knowledgeUnits/{knowledgeUnitId}"
    },
    {
      "parameter_metadata": {
        "name": "VectorStoreId",
        "type": "string"
      },
      "default_value": "default"
    }
  ]
}

Use Cases:

Indexing embeddings in Azure AI Search
Creating searchable content for agents
Partitioning index data by vector store

Azure AI Search Removal Stage

Plugin Name: AzureAISearchRemovalDataPipelineStage
Category: DataPipelineStage
Description: Provides the FoundationaLLM standard implementation for content removal data pipeline stages that use Azure AI Search.

Parameters:

Parameter	Type	Required	Description
`KnowledgeUnitObjectId`	ResourceObjectId	Yes	The FoundationaLLM resource identifier of the Knowledge Unit that provides the vector database configuration.
`VectorStoreId`	String	Yes	The name of the Vector Store resource that identifies the logical partition of the Azure AI Search index.

Parameter Selection Hints:

KnowledgeUnitObjectId: Filter by providers/FoundationaLLM.Context/knowledgeUnits

Example Configuration:

{
  "plugin_object_id": "/instances/{instanceId}/providers/FoundationaLLM.Plugin/plugins/AzureAISearchRemovalDataPipelineStage",
  "plugin_parameters": [
    {
      "parameter_metadata": {
        "name": "KnowledgeUnitObjectId",
        "type": "resource-object-id"
      },
      "default_value": "/instances/{instanceId}/providers/FoundationaLLM.Context/knowledgeUnits/{knowledgeUnitId}"
    },
    {
      "parameter_metadata": {
        "name": "VectorStoreId",
        "type": "string"
      },
      "default_value": "default"
    }
  ]
}

Use Cases:

Removing content from Azure AI Search indexes
Cleaning up deleted documents
Managing index lifecycle

Knowledge Extraction Stage

Plugin Name: KnowledgeExtractionDataPipelineStage
Category: DataPipelineStage
Description: Provides the FoundationaLLM standard implementation for knowledge extraction data pipeline stages.

Parameters:

Parameter	Type	Required	Description
`EntityExtractionPromptObjectId`	ResourceObjectId	Yes	The FoundationaLLM resource identifier of the Prompt resource that is used for entity extraction.
`EntityExtractionCompletionModel`	String	Yes	The completion model used for entity extraction.
`EntityExtractionEntityTypes`	String	Yes	List of comma-separated entity types to extract (e.g., PERSON, ORGANIZATION, LOCATION).
`EntityExtractionModelTemperature`	Float	Yes	The temperature used for the entity extraction model (0.0 for deterministic output, 1.0 for more creative output).
`EntityExtractionMaxOutputTokenCount`	Int	Yes	The maximum number of output tokens for the entity extraction model.

Parameter Selection Hints:

EntityExtractionPromptObjectId: Filter by providers/FoundationaLLM.Prompt/prompts where category = "DataPipeline"

Example Configuration:

{
  "plugin_object_id": "/instances/{instanceId}/providers/FoundationaLLM.Plugin/plugins/KnowledgeExtractionDataPipelineStage",
  "plugin_parameters": [
    {
      "parameter_metadata": {
        "name": "EntityExtractionPromptObjectId",
        "type": "resource-object-id"
      },
      "default_value": "/instances/{instanceId}/providers/FoundationaLLM.Prompt/prompts/{promptId}"
    },
    {
      "parameter_metadata": {
        "name": "EntityExtractionCompletionModel",
        "type": "string"
      },
      "default_value": "gpt-4"
    },
    {
      "parameter_metadata": {
        "name": "EntityExtractionEntityTypes",
        "type": "string"
      },
      "default_value": "PERSON,ORGANIZATION,LOCATION,DATE,EVENT"
    },
    {
      "parameter_metadata": {
        "name": "EntityExtractionModelTemperature",
        "type": "float"
      },
      "default_value": 0.0
    },
    {
      "parameter_metadata": {
        "name": "EntityExtractionMaxOutputTokenCount",
        "type": "int"
      },
      "default_value": 4000
    }
  ]
}

Use Cases:

Extracting entities from unstructured text
Building knowledge graphs
Identifying key information in documents

Knowledge Graph Consolidation Stage

Plugin Name: KnowledgeGraphConsolidationDataPipelineStage
Category: DataPipelineStage
Description: Provides the FoundationaLLM standard implementation for knowledge graph consolidation data pipeline stages.

Parameters: None

Dependencies: None

Example Configuration:

{
  "plugin_object_id": "/instances/{instanceId}/providers/FoundationaLLM.Plugin/plugins/KnowledgeGraphConsolidationDataPipelineStage",
  "plugin_parameters": []
}

Use Cases:

Merging duplicate entities
Consolidating knowledge graph data
Normalizing extracted information

Knowledge Graph Summarization Stage

Plugin Name: KnowledgeGraphSummarizationDataPipelineStage
Category: DataPipelineStage
Description: Provides the FoundationaLLM standard implementation for knowledge graph summarization data pipeline stages.

Parameters:

Parameter	Type	Required	Description
`SummarizationPromptObjectId`	ResourceObjectId	Yes	The FoundationaLLM resource identifier of the Prompt resource that is used for entity summarization.
`SummarizationCompletionModel`	String	Yes	The completion model used for entity summarization.
`SummarizationCompletionModelTemperature`	Float	Yes	The temperature used for the entity summarization model (recommended value is 0.7).
`SummarizationCompletionMaxOutputTokenCount`	Int	Yes	The maximum number of output tokens for the entity summarization model.

Parameter Selection Hints:

SummarizationPromptObjectId: Filter by providers/FoundationaLLM.Prompt/prompts where category = "DataPipeline"

Example Configuration:

{
  "plugin_object_id": "/instances/{instanceId}/providers/FoundationaLLM.Plugin/plugins/KnowledgeGraphSummarizationDataPipelineStage",
  "plugin_parameters": [
    {
      "parameter_metadata": {
        "name": "SummarizationPromptObjectId",
        "type": "resource-object-id"
      },
      "default_value": "/instances/{instanceId}/providers/FoundationaLLM.Prompt/prompts/{promptId}"
    },
    {
      "parameter_metadata": {
        "name": "SummarizationCompletionModel",
        "type": "string"
      },
      "default_value": "gpt-4"
    },
    {
      "parameter_metadata": {
        "name": "SummarizationCompletionModelTemperature",
        "type": "float"
      },
      "default_value": 0.7
    },
    {
      "parameter_metadata": {
        "name": "SummarizationCompletionMaxOutputTokenCount",
        "type": "int"
      },
      "default_value": 2000
    }
  ]
}

Use Cases:

Creating entity summaries
Generating descriptions for knowledge graph nodes
Summarizing extracted information

Knowledge Graph Embedding Stage

Plugin Name: KnowledgeGraphEmbeddingDataPipelineStage
Category: DataPipelineStage
Description: Provides the FoundationaLLM standard implementation for knowledge graph embedding data pipeline stages.

Parameters:

Parameter	Type	Required	Description
`KnowledgeUnitObjectId`	ResourceObjectId	Yes	The FoundationaLLM resource identifier of the Knowledge Unit that provides the embedding configuration.

Parameter Selection Hints:

KnowledgeUnitObjectId: Filter by providers/FoundationaLLM.Context/knowledgeUnits

Example Configuration:

{
  "plugin_object_id": "/instances/{instanceId}/providers/FoundationaLLM.Plugin/plugins/KnowledgeGraphEmbeddingDataPipelineStage",
  "plugin_parameters": [
    {
      "parameter_metadata": {
        "name": "KnowledgeUnitObjectId",
        "type": "resource-object-id"
      },
      "default_value": "/instances/{instanceId}/providers/FoundationaLLM.Context/knowledgeUnits/{knowledgeUnitId}"
    }
  ]
}

Use Cases:

Creating embeddings for knowledge graph entities
Enabling semantic search over entities
Preparing graph data for vector operations

Knowledge Graph Indexing Stage

Plugin Name: KnowledgeGraphIndexingDataPipelineStage
Category: DataPipelineStage
Description: Provides the FoundationaLLM standard implementation for knowledge graph indexing data pipeline stages.

Parameters:

Parameter	Type	Required	Description
`KnowledgeUnitObjectId`	ResourceObjectId	Yes	The FoundationaLLM resource identifier of the Knowledge Unit that provides the indexing configuration.

Parameter Selection Hints:

KnowledgeUnitObjectId: Filter by providers/FoundationaLLM.Context/knowledgeUnits

Example Configuration:

{
  "plugin_object_id": "/instances/{instanceId}/providers/FoundationaLLM.Plugin/plugins/KnowledgeGraphIndexingDataPipelineStage",
  "plugin_parameters": [
    {
      "parameter_metadata": {
        "name": "KnowledgeUnitObjectId",
        "type": "resource-object-id"
      },
      "default_value": "/instances/{instanceId}/providers/FoundationaLLM.Context/knowledgeUnits/{knowledgeUnitId}"
    }
  ]
}

Use Cases:

Indexing knowledge graph data
Making entities searchable
Storing graph structure in search index

Knowledge Graph Publishing Stage

Plugin Name: KnowledgeGraphPublishingDataPipelineStage
Category: DataPipelineStage
Description: Provides the FoundationaLLM standard implementation for knowledge graph publishing data pipeline stages.

Parameters:

Parameter	Type	Required	Description
`KnowledgeUnitObjectId`	ResourceObjectId	Yes	The FoundationaLLM resource identifier of the Knowledge Unit that provides the publishing configuration.

Parameter Selection Hints:

KnowledgeUnitObjectId: Filter by providers/FoundationaLLM.Context/knowledgeUnits

Example Configuration:

{
  "plugin_object_id": "/instances/{instanceId}/providers/FoundationaLLM.Plugin/plugins/KnowledgeGraphPublishingDataPipelineStage",
  "plugin_parameters": [
    {
      "parameter_metadata": {
        "name": "KnowledgeUnitObjectId",
        "type": "resource-object-id"
      },
      "default_value": "/instances/{instanceId}/providers/FoundationaLLM.Context/knowledgeUnits/{knowledgeUnitId}"
    }
  ]
}

Use Cases:

Publishing processed knowledge graphs
Making graph data available to applications
Finalizing graph structure

Content Text Extraction Plugins

Content text extraction plugins handle extracting text from specific file formats. These plugins are used as dependencies by the Text Extraction Stage.

PDF Content Text Extraction

Plugin Name: PDFContentTextExtraction
Category: ContentTextExtraction
Subcategory: PDF
Description: Provides the FoundationaLLM standard implementation for content text extraction from PDF files.

Parameters: None

Supported File Types: .pdf

Example Configuration:

{
  "plugin_object_id": "/instances/{instanceId}/providers/FoundationaLLM.Plugin/plugins/PDFContentTextExtraction",
  "plugin_parameters": []
}

DOCX Content Text Extraction

Plugin Name: DOCXContentTextExtraction
Category: ContentTextExtraction
Subcategory: DOCX
Description: Provides the FoundationaLLM standard implementation for content text extraction from DOCX files.

Parameters: None

Supported File Types: .docx

Example Configuration:

{
  "plugin_object_id": "/instances/{instanceId}/providers/FoundationaLLM.Plugin/plugins/DOCXContentTextExtraction",
  "plugin_parameters": []
}

PPTX Content Text Extraction

Plugin Name: PPTXContentTextExtraction
Category: ContentTextExtraction
Subcategory: PPTX
Description: Provides the FoundationaLLM standard implementation for content text extraction from PPTX files.

Parameters: None

Supported File Types: .pptx

Example Configuration:

{
  "plugin_object_id": "/instances/{instanceId}/providers/FoundationaLLM.Plugin/plugins/PPTXContentTextExtraction",
  "plugin_parameters": []
}

XLSX Content Text Extraction

Plugin Name: XLSXContentTextExtraction
Category: ContentTextExtraction
Subcategory: XLSX
Description: Provides the FoundationaLLM standard implementation for content text extraction from XLSX files.

Parameters: None

Supported File Types: .xlsx

Example Configuration:

{
  "plugin_object_id": "/instances/{instanceId}/providers/FoundationaLLM.Plugin/plugins/XLSXContentTextExtraction",
  "plugin_parameters": []
}

Image Content Text Extraction

Plugin Name: ImageContentTextExtraction
Category: ContentTextExtraction
Subcategory: JPG,JPEG,GIF,PNG,WEBP,TIFF,BMP
Description: Provides the FoundationaLLM standard implementation for text extraction from image files.

Parameters:

Parameter	Type	Required	Description
`AIModelObjectId`	ResourceObjectId	Yes	The FoundationaLLM resource identifier of the AI Model resource that identifies the LLM model used to extract text from images.

Parameter Selection Hints:

AIModelObjectId: Filter by providers/FoundationaLLM.AIModel/aiModels

Supported File Types: .jpg, .jpeg, .gif, .png, .webp, .tiff, .bmp

Example Configuration:

{
  "plugin_object_id": "/instances/{instanceId}/providers/FoundationaLLM.Plugin/plugins/ImageContentTextExtraction",
  "plugin_parameters": [
    {
      "parameter_metadata": {
        "name": "AIModelObjectId",
        "type": "resource-object-id"
      },
      "default_value": "/instances/{instanceId}/providers/FoundationaLLM.AIModel/aiModels/{aiModelId}"
    }
  ]
}

Use Cases:

Extracting text from scanned documents
OCR processing of images
Converting visual content to searchable text

Image Metadata Text Extraction

Plugin Name: ImageMetadataTextExtraction
Category: ContentTextExtraction
Subcategory: JPG,JPEG,GIF,PNG,WEBP,TIFF,BMP
Description: Provides the FoundationaLLM standard implementation for metadata (description) extraction from image files.

Parameters:

Parameter	Type	Required	Description
`AIModelObjectId`	ResourceObjectId	Yes	The FoundationaLLM resource identifier of the AI Model resource that identifies the LLM model used to extract metadata (description) from images.

Parameter Selection Hints:

AIModelObjectId: Filter by providers/FoundationaLLM.AIModel/aiModels

Supported File Types: .jpg, .jpeg, .gif, .png, .webp, .tiff, .bmp

Example Configuration:

{
  "plugin_object_id": "/instances/{instanceId}/providers/FoundationaLLM.Plugin/plugins/ImageMetadataTextExtraction",
  "plugin_parameters": [
    {
      "parameter_metadata": {
        "name": "AIModelObjectId",
        "type": "resource-object-id"
      },
      "default_value": "/instances/{instanceId}/providers/FoundationaLLM.AIModel/aiModels/{aiModelId}"
    }
  ]
}

Use Cases:

Generating image descriptions
Creating alt text for images
Extracting semantic meaning from images

Content Text Partitioning Plugins

Content text partitioning plugins handle splitting text into chunks. These plugins are used as dependencies by the Text Partitioning Stage.

Token Content Text Partitioning

Plugin Name: TokenContentTextPartitioning
Category: ContentTextPartitioning
Subcategory: Token
Description: Provides the FoundationaLLM standard implementation for content text partitioning using a token-based strategy.

Parameters:

Parameter	Type	Required	Description
`PartitionSizeTokens`	Int	Yes	The size of the text partition in tokens.
`PartitionOverlapTokens`	Int	Yes	The size of the text partition overlaps in tokens.

Example Configuration:

{
  "plugin_object_id": "/instances/{instanceId}/providers/FoundationaLLM.Plugin/plugins/TokenContentTextPartitioning",
  "plugin_parameters": [
    {
      "parameter_metadata": {
        "name": "PartitionSizeTokens",
        "type": "int"
      },
      "default_value": 400
    },
    {
      "parameter_metadata": {
        "name": "PartitionOverlapTokens",
        "type": "int"
      },
      "default_value": 100
    }
  ]
}

Use Cases:

Creating fixed-size text chunks
Ensuring consistent partition sizes
Optimizing for embedding model token limits

Recommended Values:

Standard documents: 400 tokens partition, 100 tokens overlap
Technical documents: 500-600 tokens partition, 100-150 tokens overlap
Short content: 200-300 tokens partition, 50-75 tokens overlap

Semantic Content Text Partitioning

Plugin Name: SemanticContentTextPartitioning
Category: ContentTextPartitioning
Subcategory: Semantic
Description: Provides the FoundationaLLM standard implementation for content text partitioning using a semantic strategy.

Parameters: None

Example Configuration:

{
  "plugin_object_id": "/instances/{instanceId}/providers/FoundationaLLM.Plugin/plugins/SemanticContentTextPartitioning",
  "plugin_parameters": []
}

Use Cases:

Preserving semantic meaning in chunks
Creating naturally-bounded text segments
Handling complex document structures

Characteristics:

Variable-sized partitions based on content
Respects semantic boundaries (paragraphs, sections)
Better for documents with clear structure

Parameter Types

The following parameter types are used across plugins:

Type	Description	Example
`String`	Text value	"gpt-4", "Token"
`Int`	Integer number	400, 8000
`Float`	Decimal number	0.7, 1.0
`Array`	List of strings	["folder1", "folder2"]
`ResourceObjectId`	FoundationaLLM resource reference	"/instances/{id}/providers/..."

Plugin Selection Types

When plugins have dependencies, the selection type determines how many can be chosen:

Selection Type	Description
`Single`	Exactly one dependency plugin must be selected
`Multiple`	One or more dependency plugins can be selected

Common Pipeline Patterns

Basic Document Processing Pipeline

Azure Data Lake Data Source
  ↓
Text Extraction Stage
  ├─ PDF Content Text Extraction
  ├─ DOCX Content Text Extraction
  └─ XLSX Content Text Extraction
  ↓
Text Partitioning Stage
  └─ Token Content Text Partitioning (400 tokens, 100 overlap)
  ↓
Gateway Text Embedding Stage
  ↓
Azure AI Search Indexing Stage

Knowledge Graph Pipeline

SharePoint Online Data Source
  ↓
Text Extraction Stage
  ↓
Text Partitioning Stage
  ↓
Knowledge Extraction Stage
  ↓
Knowledge Graph Consolidation Stage
  ↓
Knowledge Graph Summarization Stage
  ↓
Knowledge Graph Embedding Stage
  ↓
Knowledge Graph Indexing Stage
  ↓
Knowledge Graph Publishing Stage

Content Removal Pipeline

Context File Data Source (ContentAction: "remove")
  ↓
Azure AI Search Removal Stage

Best Practices

Parameter Configuration

Token Partitioning:
- Use 300-500 tokens for standard documents
- Keep overlap at 20-30% of partition size
- Adjust based on embedding model limits
Knowledge Extraction:
- Use temperature 0.0 for consistent entity extraction
- Set appropriate max output tokens (typically 2000-4000)
- Define specific entity types relevant to your domain
Embedding Configuration:
- Match embedding request size to model capabilities
- Consider rate limits when setting batch sizes
- Use appropriate Knowledge Unit references

Performance Optimization

Data Source Selection:
- Filter folders/libraries to only necessary content
- Use specific paths rather than broad scans
- Consider data volume when choosing source
Extraction Plugins:
- Only include file type plugins you need
- Set appropriate content size limits
- Monitor extraction performance
Partitioning Strategy:
- Token-based for speed and consistency
- Semantic for quality and context preservation
- Test both with your specific content

Troubleshooting

Common Issues

Issue: Plugin not found error
Solution: Verify plugin name matches exactly (case-sensitive)

Issue: Missing required parameter
Solution: Check parameter requirements table and ensure all required parameters are provided

Issue: Invalid ResourceObjectId
Solution: Verify the resource exists and the path format is correct

Issue: Dependency plugin not selected
Solution: Ensure required dependency plugins are included in configuration

Source Reference

All plugin specifications are defined in:

/src/dotnet/DataPipelinePlugins/PluginPackageManager.cs

This source file is the authoritative reference for all plugin metadata, parameters, and dependencies.

Table of Contents

Data Pipeline Plugins Reference

Overview

Plugin Categories

Category Hierarchy

Data Source Plugins

Azure Data Lake Data Source

Context File Data Source

SharePoint Online Data Source

Data Pipeline Stage Plugins

Text Extraction Stage

Text Partitioning Stage

Azure AI Content Safety Shielding Stage

Gateway Text Embedding Stage

Azure AI Search Indexing Stage

Azure AI Search Removal Stage

Knowledge Extraction Stage

Knowledge Graph Consolidation Stage

Knowledge Graph Summarization Stage

Knowledge Graph Embedding Stage

Knowledge Graph Indexing Stage

Knowledge Graph Publishing Stage

Content Text Extraction Plugins

PDF Content Text Extraction

DOCX Content Text Extraction

PPTX Content Text Extraction

XLSX Content Text Extraction

Image Content Text Extraction

Image Metadata Text Extraction

Content Text Partitioning Plugins

Token Content Text Partitioning

Semantic Content Text Partitioning

Parameter Types

Plugin Selection Types

Common Pipeline Patterns

Basic Document Processing Pipeline

Knowledge Graph Pipeline

Content Removal Pipeline

Best Practices

Parameter Configuration

Performance Optimization

Troubleshooting

Common Issues

Source Reference

Related Documentation