← All ivy-nodes
IVYXSTUDIO · IVY NODE
P
PDF Processor
ivy.node.pdf-processor · v0.1.0
ivyx✓
Extracts the text of a PDF and returns it in overlapping chunks with the page count. Reads with pypdf, so a scanned PDF with no text layer returns no chunks.
#PDF
Inputs
| Field | Type | Description |
|---|---|---|
| bufferrequired | Not declared | The PDF's bytes. |
| chunk_size | integer | Characters per chunk. |
| chunk_overlap | integer | Characters two neighbouring chunks share. |
Outputs
| Field | Type | Description |
|---|---|---|
| text_linesrequired | array | The document's text, in chunks. |
| page_countrequired | integer | How many pages the PDF has. |
Source
python
inp = __ivy_ctx__["nodes"][__ivy_node_id__]["input"]
buffer = inp["buffer"]
chunk_size = int(inp.get("chunk_size", 1000))
chunk_overlap = int(inp.get("chunk_overlap", 200))
# pypdf directly. The inherited cell loaded langchain_community and
# langchain_text_splitters to make the same two calls, through a temp file.
from pypdf import PdfReader
if chunk_size <= 0 or not 0 <= chunk_overlap < chunk_size:
raise ValueError("chunk_size must be positive and chunk_overlap smaller than it.")
buffer.seek(0)
reader = PdfReader(buffer)
text = "\n".join((page.extract_text() or "").strip() for page in reader.pages).strip()
step = chunk_size - chunk_overlap
text_lines = [text[i:i + chunk_size] for i in range(0, len(text), step)] if text else []
text_lines = [t for i, t in enumerate(text_lines) if i == 0 or len(t) > chunk_overlap]
out = __ivy_ctx__["nodes"][__ivy_node_id__]["output"]
out["text_lines"] = text_lines
out["page_count"] = len(reader.pages)Tests
Requires: python:3.9, pypdf
- one-page
A one-page PDF yields its sentence as one chunk.
- not-a-pdf
Bytes that are not a PDF are refused.