IVYXSTUDIO · IVY NODE

File Reader

ivy.node.minio-file-reader · v0.0.0

ivyx✓

Reads files from MinIO object storage and returns them as BytesIO buffers. This node downloads files from a specified bucket and file name, making them available for processing. The output buffer can be used with pdf-processor for PDFs, image-processor for images, or other file processing nodes. Requires minio-provider for the connection. Use this node to retrieve files from object storage for processing pipelines.

#MinIO#Object Storage#S3 Compatible Storage#Private Cloud#Distributed Storage#High Performance Storage#Cloud-Native Storage#S3 API#Open Source Storage

Inputs

Field	Type	Description
minio_providerrequired	object	An instance of MinioProvider. (Node reference)
file_namerequired	string	The name of the file to read from the bucket.
bucket_namerequired	string	The name of the bucket to read from.
features	object	Features
labels	object	Labels

Outputs

Field	Type	Description
filerequired	object	Custom type: io.BytesIO
features	object	Features
labels	object	Labels

Source

python

# Input preparation
inp = __ivy_ctx__["nodes"][__ivy_node_id__]["input"]
minio_provider = inp["minio_provider"]
file_name = inp["file_name"]
bucket_name = inp["bucket_name"]

# Compute
import io
from minio import Minio
from minio.error import S3Error

class MinioFileReader:
    """
    A class to handle file downloads from MinIO.
    """

    def __init__(self, provider: MinioProvider):
        """
        Initialize the MinioFileReader with a MinioProvider instance.

        :param provider: An instance of Minio.
        """
        self.provider = provider

    def read_file(self, bucket_name: str, file_name: str) -> io.BytesIO:
        """
        Read a file from a specified bucket and return it as a BytesIO object.

        :param bucket_name: The name of the bucket to read from.
        :param file_name: The name of the file to read.
        :return: The file content as a BytesIO object.
        """
        try:
            response = self.provider.get_client().get_object(bucket_name, file_name)
            buffer = io.BytesIO(response.read())
            response.close()
            response.release_conn()
            return buffer
        except S3Error as exc:
            print(f"Error occurred while reading the file: {exc}")
            return None

minio_file_reader = MinioFileReader(provider=minio_provider)

file = minio_file_reader.read_file(
    bucket_name=bucket_name,
    file_name=file_name
)

# Output collection (runner reads __ivy_ctx__)
out = __ivy_ctx__["nodes"][__ivy_node_id__]["output"]
out["file"] = file

Tests

Requires: python:3.11

basic-file-read
Read file with valid inputs (should succeed).
miniofile-readbasic