OmAgent

What it is

OmAgent is a python library for building multimodal language agents with ease. It aims to wrap complex engineering (worker orchestration, task queue, node optimization) behind a simple interface to define agents, with abstractions for reusable components and native support for video, image, and audio inputs. It includes features for multimodal interaction, local model deployment, and a distributed architecture with Lite mode.

How it works

The project offers a graph-based workflow orchestration engine and memory types to enable contextual reasoning. It provides native multimodal interaction support, including VLM models, real-time API access, computer vision models, and mobile connectivity. It also includes algorithms for agent reasoning beyond LLMs (e.g., ReAct, CoT, SC-Cot) and supports running models locally via Ollama or LocalAI. The architecture supports distributed deployment and scaling, with a Lite mode option to reduce middleware needs.

Getting started

• python >= 3.10
• Install omagent_core Use pip to install omagent_core latest release.

  pip install omagent-core
  

  Or install the latest version from the source code like below.

  pip install -e omagent-core
  

Quick Start (Example commands from README)

1. Generate a container configuration:

   cd examples/step1_simpleVQA
   python compile_container.py
   

   This creates a container.yaml with default settings under examples/step1_simpleVQA.
2. Configure LLM settings in configs/llms/gpt.yml:
   • Set your OpenAI API key or compatible endpoint through environment variable or by directly modifying the yml file

   export custom_openai_key="your_openai_api_key"
   export custom_openai_endpoint="your_openai_endpoint"
   

   You can use a locally deployed Ollama to call your own language model. The tutorial is here.

Run the demo

1. Run the simple VQA demo with webpage GUI: For WebpageClient usage: Input and output are in the webpage

   cd examples/step1_simpleVQA
   python run_webpage.py
   

   Open the webpage at http://127.0.0.1:7860, you will see the interface.

🤖 Example Projects

1. Video QA Agents: Builds a system that can answer questions about uploaded videos using a Gradio-based application. See details in the repository and linked docs.
2. Mobile Personal Assistant: Example of a multimodal assistant with a tutorial available in the docs.
3. Agentic Operators: Reusable agentic workflows (CoT, ReAct, etc.) with comparisons and datasets described in the docs.

💻 Documentation

More detailed documentation is available at https://om-ai-lab.github.io/OmAgentDocs/.

🤝 Contributing

For more information on how to contribute, see here (CONTRIBUTING.md).

🔔 Follow us

Follow on X, Discord and WeChat for updates (links in README).