IVYXSTUDIO · IVY NODE

Text Chunker

ivy.node.text-chunker · v0.0.0

ivyx✓

Splits a long text document into overlapping chunks suitable for embedding and retrieval-augmented generation (RAG). Each chunk is a contiguous slice of the source text bounded by chunk_size with a configurable overlap to preserve context across boundaries. The output chunks list feeds text-vectorizer for embedding before milvus-insert. This node is pure-compute and performs no network or file access.

#rag#text#chunking

Inputs

Field	Type	Description
textrequired	string	The source document text to split into chunks.
chunk_size	integer	Maximum number of characters per chunk.
overlap	integer	Number of characters each chunk overlaps with the previous one to preserve context.
features	object	Features
labels	object	Labels

Outputs

Field	Type	Description
chunksrequired	array	Ordered list of overlapping text chunks.
features	object	Features
labels	object	Labels

Source

python

# Input preparation
inp = __ivy_ctx__["nodes"][__ivy_node_id__]["input"]
text = inp["text"]
chunk_size = int(inp.get("chunk_size", 512))
overlap = int(inp.get("overlap", 64))

# Compute
from typing import List

class TextChunker:
    """
    Splits text into overlapping character-bounded chunks for RAG ingestion.
    """

    def __init__(self, chunk_size: int = 512, overlap: int = 64):
        if chunk_size <= 0:
            raise ValueError("chunk_size must be a positive integer")
        if overlap < 0:
            raise ValueError("overlap must be zero or a positive integer")
        if overlap >= chunk_size:
            raise ValueError("overlap must be smaller than chunk_size")
        self.chunk_size = chunk_size
        self.overlap = overlap

    def chunk(self, text: str) -> List[str]:
        """
        Split text into overlapping chunks.

        :param text: Source document text (str).
        :return: Ordered list of chunk strings (List[str]).
        """
        if text is None:
            raise ValueError("text is not provided")
        stripped = text.strip()
        if not stripped:
            return []
        step = self.chunk_size - self.overlap
        chunks: List[str] = []
        start = 0
        length = len(stripped)
        while start < length:
            end = start + self.chunk_size
            chunk = stripped[start:end].strip()
            if chunk:
                chunks.append(chunk)
            start += step
        return chunks

chunker = TextChunker(chunk_size=chunk_size, overlap=overlap)
chunks = chunker.chunk(text)

# Output collection (runner reads __ivy_ctx__)
out = __ivy_ctx__["nodes"][__ivy_node_id__]["output"]
out["chunks"] = chunks

Tests

Requires: python:3.11

basic-chunking
Split a multi-sentence document into overlapping chunks (should succeed).
ragchunkingbasic
empty-text
Empty text yields an empty chunk list (no error).
ragchunkingedge