towhee

What it is

Towhee is a Python framework designed to streamline processing of unstructured data through LLM-based pipeline orchestration. It handles data types including text, images, audio, and video, transforming them into formats like text, images, or embeddings for storage in vector databases. It offers a Pythonic, method-chaining API to build data processing pipelines and supports deployment options via Triton-based engines.

How it works

Towhee is organized around four building blocks: Operators, Pipelines, DataCollection API, and Engine. Operators are reusable processing units (models, data methods, or functions). Pipelines connect operators in a DAG to implement complex tasks such as embedding extraction or cross-modal analysis. The DataCollection API provides a fluent interface for map, filter, and other transformations, enabling processing of unstructured data (video, audio, text, images). The Engine coordinates data flow and resource usage, with a basic engine for single-machine runs and a Triton-based engine for containerized deployments. It supports converting pipelines to Nvidia Triton server deployments and includes built-in pipelines for tasks like sentence embedding and image embedding.

Getting started

Getting started requires Python 3.7+. Install Towhee via:

pip install towhee towhee.models

The README also describes pre-defined pipelines (sentence embedding, image embedding, video deduplication, retrieval-augmented generation) accessible via Towhee Hub, and provides example code for using the sentence_embedding pipeline and building custom CLIP-based pipelines.

Usage examples (from README)

from towhee import AutoPipes, AutoConfig
# get the built-in sentence_similarity pipeline
config = AutoConfig.load_config('sentence_embedding')
config.model = 'paraphrase-albert-small-v2'
config.device = 0
sentence_embedding = AutoPipes.pipeline('sentence_embedding', config=config)

# generate embedding for one sentence
embedding = sentence_embedding('how are you?').get()
# batch generate embeddings for multi-sentences
embeddings = sentence_embedding.batch(['how are you?', 'how old are you?'])
embeddings = [e.get() for e in embeddings]

Custom pipelines

from towhee import ops, pipe, DataCollection
p = (
    pipe.input('file_name')
    .map('file_name', 'img', ops.image_decode.cv2())
    .map('img', 'vec', ops.image_text_embedding.clip(model_name='clip_vit_base_patch32', modality='image'))
    .map('vec', 'vec', ops.towhee.np_normalize())
    .map(('vec', 'file_name'), (), ops.ann_insert.faiss_index('./faiss', 512))
    .output()
)

for f_name in ['https://raw.githubusercontent.com/towhee-io/towhee/main/assets/dog1.png',
               'https://raw.githubusercontent.com/towhee-io/towhee/main/assets/dog2.png',
               'https://raw.githubusercontent.com/towhee-io/towhee/main/assets/dog3.png']:
    p(f_name)

# Flush faiss data into disk. 
p.flush()
# search image by text
decode = ops.image_decode.cv2('rgb')
p = (
    pipe.input('text')
    .map('text', 'vec', ops.image_text_embedding.clip(model_name='clip_vit_base_patch32', modality='text'))
    .map('vec', 'vec', ops.towhee.np_normalize())
    # faiss op result format:  [[id, score, [file_name], ...]
    .map('vec', 'row', ops.ann_search.faiss_index('./faiss', 3))
    .map('row', 'images', lambda x: [decode(item[2][0]) for item in x])
    .output('text', 'images')
)

DataCollection(p('puppy Corgi')).show()

Core Concepts

Towhee comprises: Operators, Pipelines, DataCollection API, Engine. Operators are individual processing blocks; Pipelines connect operators in a DAG; DataCollection API provides map, filter, flat_map, etc.; Engine drives dataflow and can run locally or via a Triton-based engine in containers.

Resource

• TowheeHub: https://towhee.io/
• docs: https://towhee.readthedocs.io/en/latest/
• examples: https://github.com/towhee-io/examples

Contributing

Contributions include issues, questions, and documentation improvements. See CONtributing.md for details.

Traction

Stars: 3454 Forks: 260 Open issues: 0