IVYXSTUDIO · IVY NODE

PDF Processor

ivy.node.pdf-processor · v0.0.0

ivyx✓

Extracts and splits text content from PDF documents into manageable chunks. This node processes PDF files stored in BytesIO buffers, extracts all text content, and splits it into chunks of 1000 characters with 200 character overlap. The output text_lines array is ideal for vectorization and storage in vector databases. Typically used with text-vectorizer and milvus-insert for building document search systems. The buffer can come from minio-file-reader or other file sources.

#PDF

Inputs

Field	Type	Description
bufferrequired	object	The content of the PDF document as a BytesIO object. (Node reference)
features	object	Features
labels	object	Labels

Outputs

Field	Type	Description
text_linesrequired	array	Not declared
features	object	Features
labels	object	Labels

Source

python

# Input preparation
inp = __ivy_ctx__["nodes"][__ivy_node_id__]["input"]
buffer = inp["buffer"]

# Compute
from langchain_community.document_loaders import PyPDFLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
import tempfile
import os
import io
from typing import List  # Import for List type hint

class PDFProcessor:
    """
    A class to process PDF documents and extract text chunks.
    """

    def __init__(self, pdf_content: io.BytesIO):
        """
        Initialize the PDFProcessor with PDF content in BytesIO format.

        :param pdf_content: The content of the PDF document as a BytesIO object.
        """
        self.pdf_content = pdf_content
        self.text_lines: List[str] = []  # text_lines is a list of strings (List[str])

    def process(self) -> List[str]:
        """
        Load and split the PDF document into manageable text chunks.

        :return: A list of text chunks (List[str]).
        """
        # Create a temporary file to store the PDF content
        with tempfile.NamedTemporaryFile(delete=False, suffix=".pdf") as temp_file:
            temp_file.write(self.pdf_content.getvalue())
            temp_file_path = temp_file.name

        try:
            # Load the PDF content using PyPDFLoader
            loader = PyPDFLoader(temp_file_path)
            docs = loader.load()

            # Split the text into chunks
            text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200)
            chunks = text_splitter.split_documents(docs)
            self.text_lines = [chunk.page_content for chunk in chunks]
        finally:
            # Clean up the temporary file
            os.remove(temp_file_path)

        return self.text_lines

pdf_processor = PDFProcessor(pdf_content=buffer)

text_lines = pdf_processor.process()

# Output collection (runner reads __ivy_ctx__)
out = __ivy_ctx__["nodes"][__ivy_node_id__]["output"]
out["text_lines"] = text_lines

Tests

Requires: python:3.11

basic-pdf-processing
Process valid PDF buffer (should succeed).
pdfprocessingbasic
error-invalid-pdf
Invalid PDF format should handle error gracefully.
error-handlinginvalid-input