Beam College

Apache Beam: Powering the Future of Event-Driven Agents at Scale

Mon, 09 Mar 2026 14:02:09 -0600

Where did Apache Beam start, and where is it taking us? Join us as we dive into the origins of the Beam model and examine the critical shift toward a streaming-first data landscape. We’ll break down why—in today’s expansive landscape that demands real-time context—robust streaming has become the essential nervous system for event-driven agents.

Discover how the unified Beam model enables autonomous agents to react, reason, and act in real-time, turning raw, unbounded data streams into intelligent, decentralized action.

Getting Started with Apache Beam (2023)

Thu, 13 Feb 2025 21:22:35 -0600

Beam in the Data Analytics Lifecycle (2022)

Thu, 13 Feb 2025 21:08:48 -0600

How Apache Beam sets you up for a generative AI world

Wed, 14 Feb 2024 09:50:04 -0600

Overview of Beam Quest

Wed, 14 Feb 2024 09:48:13 -0600

Project Shield: How we use Beam to defend democracy

Wed, 14 Feb 2024 09:48:13 -0600

AI-Powered Data Processing

Mon, 01 Jan 0001 00:00:00 +0000

Build a data pipeline that leverages AI techniques to process and derive insights from data.

Apache Beam: Powering the Future of Event-Driven Agents at Scale

Mon, 01 Jan 0001 00:00:00 +0000

Where did Apache Beam start, and where is it taking us?

Join us as we dive into the origins of the Beam model and examine the critical shift toward a streaming-first data landscape. We’ll break down why—in today’s expansive landscape that demands real-time context—robust streaming has become the essential nervous system for event-driven agents.

Discover how the unified Beam model enables autonomous agents to react, reason, and act in real-time, turning raw, unbounded data streams into intelligent, decentralized action.

Beam College 2021 Opening Remarks

Mon, 01 Jan 0001 00:00:00 +0000

Welcome to the opening remarks for Beam College 2021 where Evren provided a brief overview of the importance of Apache Beam and why it is a great moment to learn.

Learning Resources for Apache Beam

Thu, 13 Feb 2025 21:23:57 -0600

Beam Overview (2022)

Thu, 13 Feb 2025 21:10:42 -0600

Background About Data Processing Systems

Mon, 13 Feb 2023 21:23:57 -0600

Apache Beam in the Data Analytics Lifecycle

Mon, 01 Jan 0001 00:00:00 +0000

Authoring your first pipeline

Mon, 01 Jan 0001 00:00:00 +0000

This hands-on session guides beginners through creating their first Apache Beam pipeline from scratch. We’ll start with core Beam concepts—PCollections, PTransforms, and the Pipeline object—then walk through a practical example building a data processing pipeline step by step. You’ll learn how to read data from sources, apply transformations like Map, FlatMap, and GroupByKey, and write results to sinks. The session covers common patterns, debugging techniques, and best practices for structuring your pipeline code. We’ll also explore how these foundational concepts translate to real-world MLOps scenarios like feature engineering pipelines and batch inference workflows. Whether you’re new to Beam or looking to integrate it into your ML platform, you’ll leave with the confidence to start building production-ready pipelines on runners like Dataflow.

Build a data pipeline without coding

Mon, 01 Jan 0001 00:00:00 +0000

Prototype common data processing tasks using Apache Beam YAML within a Google Colab environment.

Making the Jump from Batch to Streaming

Mon, 09 Feb 2026 16:47:05 -0600

This session dives into Apache Beam’s streaming primitives, focusing on reading from unbounded sources using Splittable DoFn and Unbounded Source, windowing strategies, and triggers with accumulation modes.

From Click to Chart: Building a Real-Time Analytics Engine

Mon, 09 Feb 2026 16:43:32 -0600

This session, “From Click to Chart,” demystifies the engineering behind real-time analytics. We will trace the lifecycle of a data point as it travels through a modern Google Cloud Platform (GCP) architecture, moving from ingestion to visualization in seconds. Using Apache Beam, we will explore how to build a unified pipeline that handles massive streams of data without the need to manage servers.

Key Points Addressed:

The “Why” of Streaming: Understanding the shift from traditional Batch processing to Real-Time Streaming.
The Architecture: A deep dive into the “Golden Path” stack: Cloud Pub/Sub (Ingest) $\rightarrow$ Dataflow (Process) $\rightarrow$ BigQuery (Store).
Apache Beam Fundamentals: Introduction to the unified programming model, including Windowing (how to group infinite data) and Watermarks (handling late data).

Real-Time Semantic Enrichment & Clustering

Thu, 13 Feb 2025 21:11:58 -0600

How to be Involved with Beam Community

Mon, 13 Feb 2023 21:34:45 -0600

Industry Data Management 4.0

Mon, 01 Jan 0001 00:00:00 +0000

Real-time Anomaly Detection

Mon, 01 Jan 0001 00:00:00 +0000

Build a data pipeline that enables real-time anomaly detection.

YAML: a new SDK to author your pipelines

Mon, 09 Feb 2026 16:30:58 -0600

In this talk, we explore a new way of authoring and running your Beam pipelines; via the YAML SDK! Learn how you can split your pipeline infrastructure from your complex processing logic.

Tutorial: My First Pipeline with Beam

Thu, 13 Feb 2025 21:13:46 -0600

Give Me Back My Data

Thu, 13 Feb 2025 20:25:43 -0600

Leverage Managed I/O

Mon, 01 Jan 0001 00:00:00 +0000

Build a pipeline that integrates with Iceberg | Kafka | BigQuery via Managed I/O.

The Dataflow Job Builder

Mon, 09 Feb 2026 16:31:55 -0600

Learn how you can create low-code and no-code Beam YAML jobs in the Cloud Dataflow UI.

Exception Management Inside Beam

Thu, 13 Feb 2025 21:15:55 -0600

Apache Beam Overview

Thu, 13 Feb 2025 20:28:30 -0600

Slides available at https://github.com/griscz/beam-college/blob/main/day1/A2-Apache_Beam_Overview.pdf

CI/CD with Dataflow Templates

Wed, 14 Feb 2024 09:39:28 -0600

Introducing Managed IO, the New Era of Beam Connectors

Mon, 09 Feb 2026 16:45:01 -0600

Discover what makes Managed IO a major leap towards a more unified, flexible, and upgrade-friendly connector ecosystem. This talk dives into the motivation and design behind Managed IO, highlighting a few key goals:

• Provide a consistent API across all connectors
• Enable runners to seamlessly upgrade IOs, pulling in bug fixes and new features — all with zero user effort
• Allow runners to fine-tune connector behavior for their environment

Part 3: Intro to RunInference

Thu, 13 Feb 2025 21:27:01 -0600

Multi-language Pipelines with Beam (2022)

Thu, 13 Feb 2025 21:18:08 -0600

Describing a Pipeline Declaratively

Thu, 13 Feb 2025 20:30:12 -0600

Slides available at https://github.com/griscz/beam-college/blob/main/day1/A3-Describe_pipeline_declaratively.pdf

Open Challenge: Innovate with Beam

Mon, 01 Jan 0001 00:00:00 +0000

Build any solution you want using Apache Beam.

Part 4: From Speech to Classifier

Thu, 13 Feb 2025 21:27:01 -0600

Dataflow Templates

Thu, 13 Feb 2025 21:19:40 -0600

Runner Architecture, Management & Autotuning

Thu, 13 Feb 2025 20:31:54 -0600

Beam YAML Bootcamp

Wed, 14 Feb 2024 09:46:58 -0600

Scaling Iceberg Ingestion with Apache Beam

Mon, 01 Jan 0001 00:00:00 +0000

This session explores the technical evolution of Apache Iceberg integration within the Apache Beam ecosystem. We dive into a suite of recent performance enhancements designed to streamline data lake ingestion at scale. Key topics include the adoption of table-defined compression for improved processing and storage efficiency, and the implementation of metadata caching to minimize lookups and prevent metadata service quota exhaustion. We also examine direct write capabilities that bypass expensive processing for large bundles, and autosharding mechanisms that optimize file sizes and ensure horizontal scalability.

Part 5: LLM to Speech Output

Thu, 13 Feb 2025 21:27:01 -0600

Apache Beam Demo & Dataflow SQL

Thu, 13 Feb 2025 20:31:54 -0600

Getting Started with Remote ML Inference in Beam Java

Mon, 01 Jan 0001 00:00:00 +0000

This session introduces the new Remote ML Inference transform in the Apache Beam Java SDK and shows how Java pipelines can run inference using external model services such as OpenAI. We’ll walk through how the transform works, how to use the OpenAI model handler with practical examples, and how to implement custom model handlers for other remote ML providers. The talk covers common usage patterns, framework extensibility, and includes a live demo so developers can quickly add remote ML inference capabilities to their Beam Java pipelines.

Real-Time Anomaly Detection with Apache Beam

Mon, 09 Feb 2026 16:45:55 -0600

Real-time anomaly detection is essential for identifying unexpected patterns and critical events in streaming data. This talk addresses the unique algorithmic challenges of anomaly detection in streaming environments and introduces a new feature within Apache Beam designed for this purpose. We will demonstrate how to seamlessly integrate both online and pre-trained offline anomaly detection models into Beam pipelines, empowering users to build robust, scalable, and real-time anomaly detection systems.

Part 6: Recap and How to Extend the Example

Thu, 13 Feb 2025 21:27:01 -0600

Advanced Patterns for Windows & Triggers

Thu, 13 Feb 2025 20:36:13 -0600

Building Scalable Semantic Search and RAG Pipelines

Mon, 09 Feb 2026 16:43:32 -0600

This presentation introduces vector-based semantic search and Retrieval Augmented Generation (RAG), demonstrating how to build scalable pipelines for using Apache Beam. We’ll start by explaining fundamental concepts like chunking, embeddings and vector similarity. Then we’ll explore semantic search applications before extending to full RAG systems.

The presentation walks through implementing both semantic search and RAG pipelines using Apache Beam’s ML components, covering data ingestion, chunking, embedding generation, vector database integration, and similarity search. By the end, students will understand the theoretical foundations of both systems and have practical knowledge of how to implement them at scale using Apache Beam’s distributed processing capabilities.

DoFn Lifecycle and User Code Requirements

Thu, 13 Feb 2025 20:38:18 -0600

Video Data Processing with Apache Beam

Mon, 09 Feb 2026 16:43:32 -0600

This architecture leverages a Splittable DoFn for parallelized video ingestion, distributing frame extraction across workers before applying Sliding Window logic to generate temporal 3D tensors. It utilizes Beam’s RunInference API with a KeyedModelHandler for GPU-accelerated inference, ensuring robust state management. Finally, CoGroupByKey synchronizes asynchronous feature vectors with metadata, serializing the aligned dataset into TFRecord SequenceExamples for downstream training.

Branching & Merging PCollections

Thu, 13 Feb 2025 20:55:06 -0600

Real-Time Stateful Processing of Video Data

Mon, 09 Feb 2026 16:45:55 -0600

You will learn how to build a pipeline that processes video data in real time to enable efficient analysis and event or anomaly detection.

Reads video streams or recorded footage from the source
Applies sliding-window analysis to examine activity over short intervals
Uses stateful processing to track patterns and detect anomalies over time

Finally, it raises alerts or stores annotated events in a structured format that can be used for monitoring dashboards or downstream analysis.

Advanced Group & Aggregation

Thu, 13 Feb 2025 20:56:51 -0600

Assembling the Puzzle: High-Performance Entity Building streaming Beam pipeline using a Two-Tiered State Architecture

Mon, 09 Feb 2026 16:43:32 -0600

When source systems emit only partial updates to conserve network bandwidth, Data Engineers face the complex task of reconstructing complete entities in real-time. In this session, we will deep dive into a high-performance, SCD-like streaming pipeline that dynamically reconstructs full entities from partial data before sinking them to the data warehouse.

The core of our solution is a custom two-tiered state backend architecture. By intelligently combining Apache Beam’s native, low-latency state API (Tier 1) with an external third-party data store (Tier 2), we overcome standard memory and throughput limitations. Join us to explore how this two-tiered design, alongside Beam timers, drastically reduces external database lookups, minimizes network latency, and unlocks unparalleled performance for stateful streaming pipelines.

State & Timers Patterns

Thu, 13 Feb 2025 20:58:07 -0600

Beyond Vectors: Building Scalable GraphRAG with Apache Beam and Cloud Spanner

Mon, 09 Feb 2026 16:43:32 -0600

Retrieval-Augmented Generation (RAG) has revolutionized how we interact with LLMs, but standard vector search often fails to capture the complex, multi-hop relationships hidden in data. “GraphRAG” solves this by grounding answers in a structured Knowledge Graph, but building these graphs from raw unstructured data at scale presents a massive data engineering challenge.

In this session, we will demonstrate how to build a production-grade GraphRAG ingestion and retrieval pipeline using Apache Beam and Google Cloud Spanner Graph.

Custom Containers

Thu, 13 Feb 2025 20:59:26 -0600

Multi-language Pipelines

Thu, 13 Feb 2025 21:00:34 -0600

Build a Source with Splittable DoFns

Thu, 13 Feb 2025 21:02:34 -0600

Dataflow Templates

Thu, 13 Feb 2025 21:04:04 -0600

GCP Dataflow Architecture

Thu, 13 Feb 2025 21:05:31 -0600

Hackathon

Mon, 28 Apr 2025 02:13:32 -0600

What is the Schedule?

The hackathon will be May 17 & 18, 2025. For more details, check out the schedule page

Who can participate?

In order to participate in Beam College 2025 you need to fulfill all of these requirements:
be 18 years or older not be a Google employee not a resident of Cuba, Iran, North Korea, Syria, Crimea, Donetsk People’s Republic, Luhansk People’s Republic, Belarus, or Russia.

What are the hackathon prizes?

1st place: $1,500 USD
2nd place: $1,000 USD
3rd place: $500 USD
Prize will be split evenly among team members.

How many people can be in a team?

Teams may have 2-6 participants.

Stateful processing In Apache Beam

Tue, 01 Apr 2025 13:10:03 -0500

The stateful processing interface in Apache Beam serves as a versatile tool for data processing, empowering users with advanced capabilities to handle complex workflows. This session will delve into the diverse functionalities provided by stateful processing, illustrating their practical applications through clear and concise code examples.

Real-Time Streaming with Kafka

Tue, 01 Apr 2025 13:08:32 -0500

Explore real-time streaming pipelines with Kafka I/O. This session will share best practices for optimizing Kafka I/O performance and cost-efficiency, including strategies like redistribute transforms and offset-based deduplication. We will also cover integrating Dataflow with Google Managed Kafka for scalable data processing.

Real-Time Anomaly Detection with Apache Beam

Tue, 01 Apr 2025 12:10:28 -0500

Building Scalable Semantic Search and RAG Pipelines

Tue, 01 Apr 2025 12:09:13 -0500

Introducing Managed IO, the New Era of Beam Connectors

Tue, 01 Apr 2025 12:07:40 -0500

YAML: a new SDK to author your pipelines.

Tue, 01 Apr 2025 12:07:38 -0500

In this talk, we explore a new way of authoring and running your Beam pipelines; via the YAML SDK! Learn how you can split your pipeline infrastructure from your complex processing logic.

The Dataflow Job Builder

Fri, 14 Mar 2025 09:46:58 -0600

Learn how you can create low-code and no-code Beam YAML jobs in the Cloud Dataflow UI.

Getting Started: Intro to Creating a Beam Pipeline

Thu, 13 Feb 2025 21:22:35 -0600

In this introductory session, Sascha provides a technical overview of Apache Beam and how it enables developers to build data pipelines that run on various frameworks like Spark and Flink. It supports both batch and stream processing, providing flexibility for diverse data needs. Beam’s SDKs are available in Go, Python, and Java, allowing developers to choose their preferred language.

How Apache Beam sets you up for a generative AI world

Wed, 14 Feb 2024 09:50:04 -0600

This session provides an overview of Apache Beam and its place in the generative AI space. What problem does Beam solve? What are its advantages? Why and how are companies using it?

Implementing a Complex ML Pipeline in Beam

Wed, 14 Feb 2024 09:45:14 -0600

In this session we explain how to implement a complex ML pipeline with Apache Beam. The pipeline we will build takes audio data, convert it to text, classify it to identify the topic or subject, feed it to a LLM and then take the output of the model and turn it back to voice.

Implementing a ML Pipeline with Google AI Studio

Wed, 14 Feb 2024 09:45:14 -0600

This tutorial demonstrates how to perform streaming inference with Apache Beam and Google AI Studio’s Gemini model, focusing on a geography-based example to get country capitals.

Making the Jump from Batch to Streaming

Wed, 14 Feb 2024 09:42:24 -0600

About Beam College

Mon, 01 Jan 0001 00:00:00 +0000

Beam College is an online, free educational program that provides hands-on training to solve data processing use cases using Apache Beam®.

Topics

Beam College 2026 sessions are organized in the following 3 tracks:

Beginner: Overview of what is Apache Beam and how it fits on your data processing stack, building blocks and fundamentals for building your first pipeline.
Tips & Tricks: Sessions focused on solving specific use cases with Apache Beam including processing video data, processing time series data, migrating from batch to streaming, etc.
New Features: Sessions focused on reviewing some of the latest features in Apache Beam covering topics like remote inference for AI, RAG pipelines, managed I/O with Iceberg, etc.

Sessions are provided as a live online conference. We encourage participants to join live in order to maximize their learning and engagement opportunities. However, all sessions are recorded and published for future reference.

Additional Learning Resources

Mon, 01 Jan 0001 00:00:00 +0000

Besides the archive of previous sessions that you can find in this website, Apache Beam has these other very useful learning resources:

Tour of Beam is an interactive way of learning to write Beam code with a sandbox, where you can write and run pipelines while walking through various concepts.
Beam Playground is an interactive environment to try out Beam transforms and examples without having to install Apache Beam in your environment.
Beam Quest is a series of labs that teach you how to write and test Apache Beam pipelines. Each lab takes about 1.5 hours to complete. When you complete the quest, you’re granted a badge that you can use to show your Beam expertise.

For an extensive list of learning resources for Apache Beam we recommend the learning resources page in the Apache Beam website.

Code of Conduct

Mon, 01 Jan 0001 00:00:00 +0000

Contributor Covenant Code of Conduct

Our Pledge

We as members, contributors, and leaders pledge to make participation in our
community a harassment-free experience for everyone, regardless of age, body
size, visible or invisible disability, ethnicity, sex characteristics, gender
identity and expression, level of experience, education, socio-economic status,
nationality, personal appearance, race, caste, color, religion, or sexual
identity and orientation.

We pledge to act and interact in ways that contribute to an open, welcoming,
diverse, inclusive, and healthy community.

Contact

Mon, 01 Jan 0001 00:00:00 +0000

Contact

Hello. :)

Frequently Asked Questions

Mon, 01 Jan 0001 00:00:00 +0000

Frequently asked questions about Beam College.

Previous Instructors

Mon, 01 Jan 0001 00:00:00 +0000

Meet our archive of instructors from all editions of Beam College.

Privacy Policy

Mon, 01 Jan 0001 00:00:00 +0000

Beam College collects some personal data from its users. We take great care in collecting only the minimum data that we need to provide the service and to manage this data adequately. This document provides more details about the data we gather and how it is used.

Data controller

The legal entity responsible for Beam College is:

Nearshore Link, Inc.
2940 Thousand Oaks Dr.
Austin, TX 78746

Contact: Pedro Galvan pedro@sg.com.mx

Register

Mon, 01 Jan 0001 00:00:00 +0000

Schedule

Mon, 01 Jan 0001 00:00:00 +0000

Beam College 2026 will be on April 21-23, 2026.

All times in UTC. Click on the calendar link to add to your calendar in your local time.

Terms and Conditions

Mon, 01 Jan 0001 00:00:00 +0000

Version 0.9. March 12th, 2025.

By registering for this Hackathon event or participating in any way, you fully and unconditionally agree to comply with all of the terms and conditions below. If you do not agree with any of these terms and conditions, do not register for and participate in this event and do not submit an entry.

This event is aimed at promoting the development of solutions that involve the use of Apache Beam by giving awards to the winner/s as consideration of his/her/their works.