1. What Is Chunking?
Chunking is the process of splitting a document into smaller segments, called chunks, that an AI knowledge base can index, search, and retrieve individually. When a user asks a question, the system does not search entire documents. Instead, it searches across thousands of chunks and returns the ones that are most semantically relevant to the query.
Think of it like an index in the back of a textbook. Rather than reading the whole book, you look up a topic and go directly to the relevant page. Chunking builds a similar index, except the AI creates it automatically and searches it using meaning rather than exact keywords.
The quality of these chunks directly affects retrieval accuracy. A well-chunked knowledge base surfaces precise, relevant answers. A poorly chunked one returns fragments that are too vague, too broad, or missing critical context.
Learn more: For a broader overview of preparing documents for AI, see AI Knowledge Preparation.
2. Respect Natural Boundaries in Your Content
The most effective chunking strategies work with the natural structure of your documents rather than applying arbitrary character or word limits. Headings, section breaks, and topic transitions are signals that a new chunk should begin.
When preparing documents, these boundaries matter:
- Heading-based breaks: Each H2 or H3 section typically maps well to a single chunk. If a section is very long, sub-sections provide natural split points.
- Topic transitions: When the subject changes mid-document, that transition is a natural chunk boundary, even if there is no formal heading.
- Paragraph clusters: Groups of paragraphs that discuss the same concept should stay together in a single chunk rather than being split mid-thought.
Documents with clear headings and well-organized sections produce better chunks with less manual intervention. This is one of the strongest reasons to invest in document structure before uploading. For practical steps, see How to Prepare Documents for AI.
3. Keep Procedures and Instructions Together
Procedural content, such as step-by-step instructions, troubleshooting guides, or setup processes, is especially sensitive to chunking. If a five-step procedure is split across two chunks, the AI may return only steps 1 through 3, leaving the user with an incomplete answer.
Best practices for procedural content:
- Keep all steps of a single procedure within one chunk where possible
- Include prerequisites and expected outcomes alongside the steps
- If a procedure is too long for a single chunk, split at logical sub-task boundaries (e.g., "Part 1: Installation" and "Part 2: Configuration") rather than at arbitrary step numbers
The goal is to preserve enough local context that a retrieved chunk can be interpreted correctly. Procedures may require more than one chunk, so retrieval and context assembly should preserve sequence, scope, and related steps where possible.
4. Handle Tables, Images, and Structured Data
Tables and images present unique challenges for chunking. A pricing table split in half loses its meaning. A diagram separated from its explanatory text becomes an orphaned image with no retrievable context.
Guidance for structured and visual content:
- Tables: Keep tables with their headers. If a table is too large to fit in a single chunk, consider splitting it by logical grouping (e.g., by product category or region) and ensuring each sub-table retains its column headers.
- Images: Associate images with descriptive captions or alt text. When configured, FAQ Ally can index image captions for retrieval, allowing visual content to be surfaced alongside text. See multimodal image embedding for details.
- Code blocks: Keep code snippets with their surrounding explanation. A code block without context is rarely useful as a standalone chunk.
5. Use Overlap to Preserve Context Across Chunk Boundaries
Even with careful splitting, some context is lost at chunk boundaries. Overlap, where a small amount of text from the end of one chunk is repeated at the beginning of the next, helps preserve continuity.
How overlap helps:
- A sentence at the end of chunk A that introduces a concept expanded in chunk B can appear in both, giving each chunk enough context to stand alone
- Overlap reduces the chance that a query falls into the "gap" between two chunks where the answer spans the boundary
Most modern chunking systems apply overlap automatically. The important thing for document authors is to write content that flows logically so that overlapping regions contain meaningful transitional text rather than filler.
6. Attach Metadata to Chunks
Metadata can give retrieval systems and reviewers additional signals beyond raw text. A chunk that carries its document title, section heading, and publication date is often easier to interpret and verify than one that carries only body text.
Common metadata dimensions:
- Source document title and file name
- Section heading the chunk was extracted from
- Document date (publication or last review)
- Tags or categories if your documents use them
- Audience (internal, customer-facing)
FAQ Ally can retain source metadata with training content, and agents can use document tags where configured. Descriptive file names, headings, and document-level metadata make retrieval and review more effective.
7. Avoid Chunks That Are Too Small or Too Large
Chunk size is a trade-off. Chunks that are too small lose context and produce vague answers. Chunks that are too large dilute relevance and may include information unrelated to the query.
Symptoms of chunks that are too small:
- AI answers are technically correct but lack enough detail to be useful
- Follow-up questions are needed to get a complete answer
- Single-sentence chunks dominate retrieval results
Symptoms of chunks that are too large:
- AI answers include irrelevant information mixed in with the relevant content
- Multiple unrelated topics appear in a single response
- Confidence scores may be lower because the chunk is not a precise match
Aim for chunks that contain a single, complete idea or procedure. The exact size in tokens varies depending on the content type and the platform configuration. Focus on completeness and coherence rather than hitting a specific token count.
8. Practical Examples
To illustrate the difference, consider how the same content might be chunked well versus poorly.
Example: Return Policy
Poor chunking:
- Chunk 1: "Our return policy allows returns within 30 days."
- Chunk 2: "Exceptions apply for clearance items and custom orders. Contact support for details."
A user who asks "Can I return a clearance item?" might only retrieve Chunk 1, which says nothing about exceptions.
Better chunking:
- Chunk 1: "Our return policy allows returns within 30 days of purchase. Exceptions apply for clearance items and custom orders, which are final sale. For all other items, initiate a return through your account dashboard or contact support. Items must be in original packaging."
This single chunk answers the question completely, regardless of how the user phrases it.
Example: Multi-step Setup Procedure
Poor chunking:
- Chunk 1: "Step 1: Download the installer. Step 2: Run the installer as administrator."
- Chunk 2: "Step 3: Enter your license key. Step 4: Restart the application."
Better chunking:
- Chunk 1: "Setup Procedure: Prerequisites: Windows 10 or later, administrator access. Step 1: Download the installer from the portal. Step 2: Run the installer as administrator. Step 3: Enter your license key when prompted. Step 4: Restart the application. Expected result: The dashboard loads with your account details."
Final Thoughts
Chunking is an important factor in AI knowledge base quality, yet it is largely invisible to end users. The work happens during document preparation and training, while its effects appear when retrieval selects and assembles context. Respect natural content boundaries, keep procedures intact where possible, handle visual content thoughtfully, and test representative questions.
Related: AI Knowledge Preparation | How to Prepare Documents for AI | Citations vs. Hallucinations | Multimodal image embedding | Home