Declarative Data Infrastructurefor Multimodal AI

Unified Storage, Orchestration, and Retrieval — from Data, Apps, to Agents

Start Building in 2 Minutes

85% lessdata pipeline code|

Shipped in 3 days(not 3 weeks)|

Open Source • From the creator of Apache Parquet and Impala and engineers who worked at:

1# Replace complex pipelines with simple declarations
2import pixeltable as pxt
3from pixeltable.functions import openai, yolox
4from pixeltable.iterators import FrameIterator
5
6# Create table - no infrastructure code needed
7videos = pxt.create_table('content', {
8    'video': pxt.Video,
9    'title': pxt.String
10})
11
12# Automatic frame extraction
13frames = pxt.create_view('frames', videos,
14    iterator=FrameIterator.create(video=videos.video, fps=1)
15)
16
17# Define what you want computed - runs automatically
18frames.add_computed_column(
19    objects=yolox(frames.frame, model_id='yolox_s')
20)
21frames.add_computed_column(
22    description=openai.vision(
23        prompt='Describe what's happening in this frame',
24        image=frames.frame,
25        model='gpt-4o-mini'
26    )
27)
28
29# Pixeltable handles orchestration, caching, incremental updates

pxt.create_table

videos

├─ video: pxt.Video

└─ title: pxt.String

line 41

pxt.create_view

auto

frames

├─ frame: pxt.Image

└─ FrameIterator @ 1fps

line 46

add_computed_column

objects← yolox()

description← openai.vision()

Database, Orchestration, and AIForged into One Data Infrastructure

Pixeltable replaces the complex multi-system architecture needed for AI applications with a single declarative table interface that natively handles multimodal data like images, videos, and documents.

Loading architecture diagram...

Replace Complexitywith Declarative Simplicity

See how Pixeltable eliminates the need for separate storage, orchestration, and retrieval systems.

traditional_pipeline.py

47 lines

1# Separate systems, manual wiring
2import boto3
3import pinecone
4import psycopg2
5from airflow import DAG
6
7# 1. Download from S3
8s3 = boto3.client('s3')
9s3.download_file(bucket, key, local_path)
10
11# 2. Extract frames manually
12frames = extract_frames(local_path, fps=1)
13
14# 3. Run model, handle errors
15for frame in frames:
16    try:
17        embedding = model.encode(frame)
18        objects = yolox.detect(frame)
19    except Exception as e:
20        log_error(e)
21        continue
22    
23    # 4. Store in Pinecone
24    pinecone_index.upsert([(id, embedding)])
25    
26    # 5. Store metadata in Postgres
27    cursor.execute(
28        "INSERT INTO frames ...",
29        (id, objects, timestamp)
30    )
31
32# 6. Set up Airflow DAG for updates...

Systems:S3PineconePostgresAirflowCustom ETL

Traditional Approach

storage_orchestration_retrieval.py

12 lines74% less

1import pixeltable as pxt
2from pixeltable.functions import openai, yolox
3
4# One table, automatic orchestration
5videos = pxt.create_table('content', {
6    'video': pxt.Video
7})
8
9# Computed columns run automatically
10frames = pxt.create_view('frames', videos,
11    iterator=FrameIterator.create(fps=1))
12
13frames.add_computed_column(
14    objects=yolox(frames.frame))
15frames.add_computed_column(
16    embedding=openai.embed(frames.frame))
17frames.add_embedding_index('embedding')
18
19# Insert triggers full pipeline
20videos.insert([{'video': 'new.mp4'}])

Built-in:Auto-orchestrationIncrementalVersionedQueryable

With Pixeltable

1 System

vs 5+ separate tools

75% Less

code to maintain

Zero

orchestration config

See the full quickstart guide

The Anatomy of a Multimodal AI Agent

Explore the code behind Pixelbot, built on Pixeltable. See how declarative data infrastructure simplifies workflows like RAG, tool use, and multimodal data handling.

Multimodal AI Agent/

Try Live Now View Code

pixelbot

setup_pixeltable.py

Storage & Orchestration

endpoint.py

Web Server & API Endpoints

functions.py

UDFs and tool definitions

config.py

Model settings and configuration

requirements.txt

Dependencies

.env

Environment variables

Standard web app files (not Pixeltable-specific)

data

static

templates

logs

6 Core Files

Lineage, Versioning, Caching

Built-in

setup_pixeltable.py

GitHub Project

1pxt.create_dir("agents", if_exists="ignore")
2
3# === DOCUMENT PROCESSING ===
4documents = pxt.create_table(
5    "agents.collection",
6    {
7        "document": pxt.Document, 
8        "uuid": pxt.String, 
9        "timestamp": pxt.Timestamp, 
10        "user_id": pxt.String
11    },
12)
13chunks = pxt.create_view(
14    "agents.chunks",
15    documents,
16    iterator=DocumentSplitter.create(
17        document=documents.document,
18        separators="paragraph",
19        metadata="title, heading, page"
20    ),
21)
22chunks.add_embedding_index(
23    "text",
24    string_embed=sentence_transformer.using(
25        model_id=config.EMBEDDING_MODEL_ID
26    ),
27)
28
29@pxt.query
30def search_documents(query_text: str, user_id: str):
31    sim = chunks.text.similarity(query_text)
32    return (
33        chunks
34        .where((chunks.user_id == user_id) & (sim > 0.5))
35        .order_by(sim, asc=False)
36        .select(
37            chunks.text, 
38            source_doc=chunks.document, 
39            sim=sim
40        )
41        .limit(20)
42    )
43
44# === IMAGE PROCESSING ===
45images = pxt.create_table(
46    "agents.images",
47    {
48        "image": pxt.Image, 
49        "uuid": pxt.String, 
50        "timestamp": pxt.Timestamp, 
51        "user_id": pxt.String
52    },
53)
54images.add_computed_column(
55    thumbnail=pxt_image.b64_encode(
56        pxt_image.resize(images.image, size=(96, 96))
57    ),
58)
59images.add_embedding_index(
60    "image",
61    embedding=clip.using(model_id=config.CLIP_MODEL_ID),
62)
63
64# ... and so on for Video, Audio, Memory, Chat History, Personas, Image Generation ...
65
66# === AGENT WORKFLOW DEFINITION ===
67tools = pxt.tools(
68    functions.get_latest_news,
69    functions.fetch_financial_data,
70    search_video_transcripts,
71)
72
73tool_agent = pxt.create_table(
74    "agents.tools",
75    {
76        "prompt": pxt.String,
77        "timestamp": pxt.Timestamp,
78        "user_id": pxt.String,
79        "initial_system_prompt": pxt.String,
80        "final_system_prompt": pxt.String,
81        "max_tokens": pxt.Int,
82        "temperature": pxt.Float,
83    },
84)
85
86# === DECLARATIVE WORKFLOW WITH COMPUTED COLUMNS ===
87# Step 1: Initial LLM Reasoning (Tool Selection)
88tool_agent.add_computed_column(
89    initial_response=messages(
90        model=config.CLAUDE_MODEL_ID,
91        system=tool_agent.initial_system_prompt,
92        messages=[{"role": "user", "content": tool_agent.prompt}],
93        tools=tools,
94        tool_choice=tools.choice(required=True),
95    ),
96)
97
98# Step 2: Tool Execution
99tool_agent.add_computed_column(
100    tool_output=invoke_tools(tools, tool_agent.initial_response)
101)
102
103# Step 3: Context Retrieval (Parallel RAG)
104tool_agent.add_computed_column(
105    doc_context=search_documents(tool_agent.prompt, tool_agent.user_id)
106)
107# ... other context retrieval steps ...
108
109# Step 7: Final LLM Reasoning (Answer Generation)
110tool_agent.add_computed_column(
111    final_response=messages(
112        model=config.CLAUDE_MODEL_ID,
113        system=tool_agent.final_system_prompt,
114        messages=tool_agent.final_prompt_messages,
115    ),
116)
117
118# Step 8: Extract Final Answer Text
119tool_agent.add_computed_column(
120    answer=tool_agent.final_response.content[0].text
121)

How Pixeltable Works

Store multimodal data in tables, define transformations as computed columns, and query everything together. Pixeltable handles all the orchestration, caching, and model execution automatically.

Build

Transform

Retrieve

Serve

1. Connect to Your Data

Connect directly to your existing object stores (S3, local) without moving or duplicating data. Pixeltable organizes your files into queryable, versioned tables, eliminating the need for separate databases or vector stores.

Connect to Your Data In-Place (S3 & Local)

Zero-Duplication Design (References your files)

Persistent & Versioned Tables

Native Multimodal Types (Video, Image, Audio, etc.)

Explore Datasets View Docs

pixeltable_workflow.py

Docs

1import pixeltable as pxt
2
3# Create a directory for your tables
4pxt.create_dir('demo_project')
5
6# Define table with image and text columns
7img_table = pxt.create_table(
8    'demo_project.images', 
9    {
10        'input_img': pxt.Image,
11        'raw_text': pxt.String  # For UDF example
12    }
13)
14
15# Insert data (paths or URLs and text)
16img_table.insert([
17    {
18        'input_img': 'image1.jpg', 
19        'raw_text': 'Text for image 1'
20    },
21    {
22        'input_img': 'image2.png', 
23        'raw_text': 'Text for image 2'
24    }
25])

Your Backend for Multimodal AI

pip install pixeltableYour entire AI data stack

90%

Reduction in pipeline complexity

Simplify your AI data pipelines with declarative processing

75%

Faster development cycles

Accelerate your ML development with automated workflows

60%

Lower infrastructure costs

Deploy serverless functions when you need them, all without leaving pixeltable

Use Cases

What Can You Build?

Pixeltable's primitives compose into any multimodal AI workflow

Data Wrangling for ML

Curate, augment, and export training datasets. Pre-annotate with models, integrate Label Studio, export to PyTorch.

example.py

# Extract frames, run detection, export
frames.add_computed_column(
  objects=yolox(frames.frame)
)
pxt.io.export_parquet(frames)

Curate & AugmentPre-annotateExport PyTorchVersion Control

Learn more

Backend for AI Apps

Build RAG systems, semantic search, and multimodal APIs. Pixeltable handles storage, retrieval, and orchestration.

example.py

# Add embedding index for search
docs.add_embedding_index(
  'content', 
  embedding=openai.embeddings()
)

RAG PipelinesVector SearchMultimodal APIsAuto-sync

Learn more

Agents & MCP

Tool-calling agents with persistent memory, MCP server integration, and automatic conversation history.

example.py

# LLM with tools, Pixeltable executes
t.add_computed_column(
  response=openai.chat_completions(
    messages=t.msgs, tools=tools
))

Tool CallingPersistent MemoryMCP IntegrationState Management

Learn more

Get Started in 5 Minutes Explore Public Datasets

Everything You Need to Know

Common questions about building with Pixeltable