IVYXSTUDIO · IVY NODE

Text Vectorizer

ivy.node.text-vectorizer · v0.0.0

ivyx✓

Creates a text vectorizer instance using a pre-trained tokenizer and model. This node is essential for ML pipelines that require text embeddings. The vectorizer can convert text into numerical vectors for similarity search, classification, or clustering. Requires tokenizer and model from tokenizer-model-loader node. The output vectorizer is used with question node for query vectorization or milvus-insert for storing embeddings.

#embedding#generator

Inputs

Field	Type	Description
tokenizerrequired	object	Pre-trained tokenizer instance from tokenizer-model-loader node. (Node reference)
modelrequired	object	The model used for generating text embeddings (Node reference)
features	object	Features
labels	object	Labels

Outputs

Field	Type	Description
text_vectorizerrequired	object	Text vectorizer instance
features	object	Features
labels	object	Labels

Source

python

# Input preparation
inp = __ivy_ctx__["nodes"][__ivy_node_id__]["input"]
tokenizer = inp["tokenizer"]
model = inp["model"]

# Compute
from typing import List, Dict, Any
import torch
from tqdm import tqdm
from transformers import PreTrainedTokenizer, PreTrainedModel

class TextVectorizer:
    """
    A class to generate embeddings from text using a pre-trained model and tokenizer.
    """

    def __init__(self, tokenizer: PreTrainedTokenizer, model: PreTrainedModel):
        """
        Initialize the TextVectorizer with a tokenizer and model.

        :param tokenizer: The tokenizer for the language model (PreTrainedTokenizer).
        :param model: The model used for generating text embeddings (PreTrainedModel).
        """
        self.tokenizer = tokenizer
        self.model = model

    def vectorizer_texts(self, text_lines: List[str] = None, show_progress: bool = True) -> Dict[str, int | List[float] | str]:
        """
        Generate embeddings for the provided text lines.

        :param text_lines: List of text lines to generate embeddings for (List[str]).
        :param show_progress: Whether to show a progress bar (bool).
        :return: List of dictionaries containing IDs, vectors, and text (List[Dict[str, int | List[float] | str]]).
        """
        if text_lines is None:
            raise ValueError("text_lines is not provided. Please provide text_lines during initialization or set it later.")

        embeddings = []
        iterable = tqdm(text_lines, bar_format="{n_fmt}/{total_fmt}") if show_progress else text_lines
        for i, line in enumerate(iterable):
            try:
                vector = self.vectorizer_text(line)
                embeddings.append({"id": i, "vector": vector, "text": line})
            except Exception as e:
                print(f"Error generating embedding for text line {i}: {e}")
                continue  # Skip this line and continue with the next one

        return embeddings

    def vectorizer_text(self, text: str) -> List[float]:
        """
        Encode the given text into an embedding vector.

        :param text: The text to encode (str).
        :return: Normalized embedding vector as a list of floats (List[float]).
        """
        try:
            inputs = self.tokenizer(text, return_tensors="pt", padding=True, truncation=True)
            with torch.no_grad():
                outputs = self.model(**inputs)
            embeddings = outputs.last_hidden_state[:, 0, :]
            return torch.nn.functional.normalize(embeddings, p=2, dim=1).squeeze().tolist()
        except Exception as e:
            print(f"Error encoding text: {e}")
            raise

text_vectorizer = TextVectorizer(tokenizer=tokenizer, model=model)

# Output collection (runner reads __ivy_ctx__)
out = __ivy_ctx__["nodes"][__ivy_node_id__]["output"]
out["text_vectorizer"] = text_vectorizer

Tests

Requires: python:3.11

basic-vectorizer-creation
Create vectorizer with valid tokenizer and model (should succeed).
vectorizerbasicembedding