Cloud Storage overview

This page describes how Cloud Storage works and the use cases it supports.

How Cloud Storage works

Cloud Storage is a scalable and managed storage service offered by Google Cloud that lets you store data as objects in containers called buckets.

All buckets are associated with a project, and you group your projects under an organization. After you create a project, you can create buckets, upload objects to your buckets, and download objects from your buckets. You can also grant permissions to make your data accessible to principals you specify or accessible to everyone on the public internet. Directory capabilities let you utilize Cloud Storage more similarly to a hard drive or Network Attached Storage (NAS): folders let you organize objects in a directory structure, and managed folders let you simplify access control to your objects.

Each project, bucket, object, folder, and managed folder is a resource in Google Cloud, as are things such as Compute Engine instances.

The Google Cloud hierarchy

Here's how the Cloud Storage structure can apply to a real-world case:

  • Organization: Your company, called Example Inc., creates a Google Cloud organization called exampleinc.org.

  • Project: Example Inc. is building several applications, and each one is associated with a project. Each project has its own set of Cloud Storage APIs, as well as other resources.

  • Bucket: Each project can contain multiple buckets, which are containers to store your objects. For example, you might create a photos bucket for all the image files your app generates and a separate videos bucket. Cloud Storage offers different storage classes and locations for your buckets, letting you choose the durability and availability of your data to suit the needs of your workloads.

    Buckets serve as a primary data foundation in the broader Google Cloud ecosystem. You can connect your buckets as storage backends for other Google Cloud services, such as AI Hypercomputer, Vertex AI, and Google Kubernetes Engine.

  • Object: Buckets contain objects, such as an image called puppy.png. An object is an immutable piece of data consisting of a file of any format. Each bucket can contain essentially unlimited individual objects.

  • Folder: Buckets with hierarchical namespace enabled can contain folders. Folders enable a real file system for storing objects, as opposed to a simulated file system. You can atomically rename a folder and all the objects within it in one operation.

  • Managed folder: Each bucket can also contain managed folders, which grant or revoke additional access beyond the IAM permissions set on the bucket. Managed folders don't use a true directory tree structure; rather, a managed folder is a resource overlay used only for permission checking.

Tools for Cloud Storage

You can interact with Cloud Storage by using the following tools:

  • Google Cloud console: The Google Cloud console provides a visual interface for you to manage your data in a browser.

  • Google Cloud CLI: The gcloud CLI lets you interact with Cloud Storage through a terminal using gcloud storage commands.

  • Client libraries: The Cloud Storage client libraries allow you to manage your data using one of your preferred languages, including C++, C#, Go, Java, Node.js, PHP, Python, and Ruby.

  • REST APIs: Manage your data using the JSON or XML API.

  • Terraform: Terraform is an infrastructure-as-code (IaC) tool that you can use to provision the infrastructure for Cloud Storage. For more information, see Provision resources with Cloud Storage.

  • gRPC: gRPC lets you interact with Cloud Storage. gRPC is a high performance, open source universal RPC framework developed by Google that you can use to define your services using Protocol Buffers.

  • Cloud Storage FUSE: Cloud Storage FUSE lets you mount Cloud Storage buckets to your local file system. This enables your applications to read from a bucket or write to a bucket by using standard file system semantics.

  • GCSFS Python library: GCSFS is a Python library that provides a Pythonic file-system interface to Cloud Storage.

Use cases and key features

Cloud Storage provides durable, globally available object storage that's adaptable to a wide range of enterprise workloads, from high-performance Artificial Intelligence (AI) and Machine Learning (ML) training to regulatory archiving.

Store and access data for AI/ML and data analytics

Accelerate your AI/ML pipelines and analytical insights with high-throughput, low-latency data access designed for large-scale processing.

Capability Capability, product, or feature
Cloud Storage Rapid High-performance storage colocated with compute for sub-millisecond access and multi-TB/s aggregate throughput. Includes:
  • Rapid Bucket: Zonal object storage in the Rapid storage class designed for data-intensive AI/ML and analytics workloads.
  • Rapid Cache: Fully managed, SSD-backed zonal read cache that automatically scales capacity and bandwidth to accelerate read performance.
Hierarchical namespace Organizes data into a logical directory structure with atomic folder operations and up to 8x higher initial QPS limits than buckets without hierarchical namespace. Hierarchical namespace can only be enabled at the time of bucket creation and can't be enabled on an existing bucket.
Cloud Storage FUSE Mounts buckets to your local file system, which enables your applications to read from a bucket or write to a bucket by using standard file system semantics.

For more details on optimizing performance, see Optimizing storage for AI/ML and data analytics.

Securing your data and compliance

Protect your assets and meet global regulatory standards with built-in encryption, access controls, and immutable data retention policies.

Security layer Capability, product, or feature
Access control Enforce least-privilege access and restrict traffic to trusted networks. Products and features include:
  • Identity and Access Management: Authorize specific users, groups, or service accounts to perform actions on Cloud Storage resources through role-based access controls.
  • Uniform bucket-level access: Unify access control for your Cloud Storage resources by disabling object-level Access Control Lists (ACLs) and exclusively using bucket-level IAM policies.
  • Bucket IP filtering: Restrict access to a bucket based on the source IP address of the request and secure your data from unauthorized access from specific IP addresses or Virtual Private Cloud (VPC).
Data encryption Encrypt data at rest and in transit. Products and features include:
  • Google-managed keys: Encrypt your Cloud Storage data at rest automatically by default by using server-side keys owned and maintained by Google.
  • CMEK (Cloud KMS): Control the rotation schedules, access permissions, and lifecycles of your encryption keys to protect your Cloud Storage resources.
  • CSEK: Secure your Cloud Storage data at rest by providing your own cryptographic keys with each individual API request, instead of Google storing or recovering your key material.
Data retention and immutability Prevent accidental deletion and help satisfy data compliance. Products and features include:
  • Bucket Lock: Govern how long objects in buckets must be retained by specifying a retention policy for the bucket to enforce regulatory and compliance data retention rules.
  • Object Retention: Govern the lifespan of individual objects by applying an immutable retain-until date and time to specific objects, preventing them from being deleted or replaced until that duration expires.
  • Object Versioning: Preserve historical copies of your data by automatically archiving overwritten or deleted live objects as noncurrent versions, enabling you to restore past object states at any time.
  • Soft delete: Prevent permanent loss of data against accidental or malicious deletion by retaining recently deleted objects and buckets. By default, Cloud Storage enables soft delete for all buckets with a seven day retention period.

Backup, disaster recovery, and business continuity

Protect your business against data loss, disaster events, and regional outages with multi-region availability and fast cross-region replication.

Capability, product, or feature Description
Dual-regions and multi-regions Automatically replicate data across geographically separated Google Cloud regions for 99.99% availability.
Turbo replication Replicate 100% of newly written objects between dual-region pairs within 15 minutes.
Object Versioning and soft delete Retain previous iterations and recently deleted objects for rollback after accidental overwrites or ransomware events.

Cost optimization and data lifecycle management

Reduce storage spend across the data lifecycle by matching access frequency to cost-effective storage tiers.

Storage class, product, or feature Description
Storage classes Choose between Standard storage, Nearline storage, Coldline storage, Archive storage, and Rapid storage based on data access frequency.
Autoclass Automatically transition objects between storage classes based on object access patterns.
Object Lifecycle Management Delete or transition objects by using custom rule-based policies.

Content distribution and web serving

Distribute digital assets, downloads, and media files globally with low latency and high availability.

Capability, product, or feature Description
Cloud CDN and Cloud Load Balancing Cache static assets at Google's global edge locations for ultra-low latency web delivery.
Signed URLs Provide time-limited read or write access to private objects without requiring Google credentials.
Static website hosting Serve static HTML, CSS, JavaScript, and media assets directly from a public bucket.

Data migration and event-driven ingestion

Streamline the movement of external data into Google Cloud and trigger automated processing workflows upon object upload.

Integration Description
Storage Transfer Service Managed, high-scale transfer of online data from Amazon S3, Microsoft Azure Blob Storage, HTTP sources, or on-premises storage.
Transfer Appliance Ruggedized hardware appliances that ship up to hundreds of petabytes of offline data to Google Cloud.
Eventarc and Pub/Sub Emit real-time event notifications on object creation, deletion, or metadata updates to trigger Cloud Functions or Cloud Run services.

Resource names

Each resource has a unique name that identifies it, much like a filename. Buckets have a resource name in the form of projects/_/buckets/BUCKET_NAME, where BUCKET_NAME is the ID of the bucket. Objects have a resource name in the form of projects/_/buckets/BUCKET_NAME/objects/OBJECT_NAME, where OBJECT_NAME is the ID of the object.

A #NUMBER appended to the end of the resource name indicates a specific generation of the object. #0 is a special identifier for the most recent version of an object. #0 is useful to add when the name of the object ends in a string that would otherwise be interpreted as a generation number.

Quickstart guides

To learn the fundamentals of using Cloud Storage, visit the following guides:

Looking for other products?

If Cloud Storage is not the right storage solution for you, see more information about the following storage services:

What's next