Scalable Python Data Analysis with BigQuery DataFrames (BigFrames)#

BigQuery DataFrames (bigframes) is an open-source Python library that brings the power of distributed computing to your data science workflow. By providing a familiar pandas and scikit-learn compatible API, BigFrames allows you to analyze and model massive datasets where they live—directly in BigQuery.

Why Choose BigQuery DataFrames?#

BigFrames eliminates the “data movement bottleneck.” Instead of downloading large datasets to a local environment, BigFrames translates your Python code into optimized SQL, executing complex transformations across the BigQuery fleet.

Petabyte-Scale Scalability: Effortlessly process datasets that far exceed local memory limits.
Familiar Python Ecosystem: Use the same read_gbq, groupby, merge, and pivot_table functions you already know from pandas.
Generative AI and Machine Learning: Seamlessly leverage Gemini models, AI functions, and vector search with bigframes.bigquery.ai, alongside BigQuery ML’s powerful algorithms via a scikit-learn-compatible interface (bigframes.ml).
Enterprise-Grade Security: Maintain data governance and security by keeping your data within the BigQuery perimeter.
Hybrid Flexibility: Easily move between distributed BigQuery processing and local pandas analysis with to_pandas().

Core Components of BigFrames#

BigQuery DataFrames is organized into specialized modules designed for the modern data stack:

bigframes.pandas: A high-performance, pandas-compatible API for scalable data exploration, cleaning, and transformation.
bigframes.bigquery: Specialized utilities for direct BigQuery resource management, including integrations with Gemini and other AI models in the bigframes.bigquery.ai submodule.

Quickstart: Scalable Data Analysis in Seconds#

Install BigQuery DataFrames via pip:

pip install --upgrade bigframes

The following example demonstrates how to perform a distributed aggregation on a public dataset with millions of rows using just a few lines of Python:

import bigframes.pandas as bpd

# If running in your local environment or Colab, uncomment these lines and add your GCP project ID
# PROJECT_ID = "bigframes-dev"
# bpd.options.bigquery.project = PROJECT_ID

# Initialize BigFrames and load a public dataset
df = bpd.read_gbq("bigquery-public-data.usa_names.usa_1910_2013")

# Perform familiar pandas operations that execute in the cloud
top_names = (
    df.groupby("name")
    .agg({"number": "sum"})
    .sort_values("number", ascending=False)
    .head(10)
)

# Bring the final, aggregated results back to local memory if needed
print(top_names.to_pandas())

Sample Notebooks and Interactive Demos#

Explore sample notebooks demonstrating end-to-end workflows across analytics, GenAI, and machine learning. Each notebook can be launched directly in BigQuery Studio or Consumer Colab:

Notebook	BigQuery Studio	Consumer Colab
Getting started with BigFrames	Open in BQ Studio	Run in Colab
AI functions (Gemini & GenAI)	Open in BQ Studio	Run in Colab
Data visualization	Open in BQ Studio	Run in Colab
Analyzing posters with AI functions	Open in BQ Studio	Run in Colab
DataFrame operations	Open in BQ Studio	Run in Colab
Multimodal DataFrames	Open in BQ Studio	Run in Colab
SQL interoperability with bqsql magic	Open in BQ Studio	Run in Colab
Timedelta operations	Open in BQ Studio	Run in Colab
Remote Functions	Open in BQ Studio	Run in Colab

Browse the complete notebooks catalog on GitHub for additional tutorials and community examples.

Articles and Guides#

To learn more about BigQuery DataFrames architecture, features, and best practices, check out the following articles and documentation guides:

BigQuery DataFrames: Generally Available: Official announcement detailing petabyte-scale pandas and scikit-learn on BigQuery.
Generative AI in BigQuery with DataFrames: How to leverage Gemini models and AI functions directly in Python.
Analyze Multimodal Data with BigQuery DataFrames: End-to-end unstructured data analysis across images, audio, and PDF documents.
Scalable Data Science with BigQuery DataFrames: Best practices for scaling analytics and machine learning workflows in the cloud.
BigQuery DataFrames Introduction (Cloud Docs): Official documentation covering data manipulation, sessions, and deployment.
Try BigQuery DataFrames (Quickstart Guide): Step-by-step tutorial in BigQuery Studio and Colab.

Explore the Documentation#

User Documentation

User Guide
- BigQuery DataFrames (BigFrames)

Community & Updates

Changelog