Cloud Data Platforms
Modern data engineering is increasingly cloud-native. Snowflake, Google BigQuery, Amazon Redshift, and Databricks are the major platforms that power analytics at scale. Each offers different trade-offs in compute/storage separation, pricing, and ecosystem.
Snowflake
Snowflake is a fully managed cloud data warehouse with complete separation of compute and storage. It runs on AWS, Azure, and GCP.
Key Features
- Virtual warehouses - Independent compute clusters that can be resized, paused, or auto-scaled
- Zero-copy cloning - Instant, storage-free copies of databases for dev/test
- Time travel - Query and restore data up to 90 days in the past
- Data sharing - Share live data across Snowflake accounts without copying
- Snowpipe - Serverless, continuous data ingestion from cloud storage
Pricing Model
Snowflake charges separately for storage (compressed, per TB/month) and compute (per warehouse-hour based on warehouse size). Compute can be paused to zero cost.
-- Create a virtual warehouse
CREATE WAREHOUSE analytics_wh
WAREHOUSE_SIZE = 'MEDIUM' -- XSMALL through 6X-LARGE
AUTO_SUSPEND = 300 -- seconds of inactivity
AUTO_RESUME = TRUE
INITIALLY_SUSPENDED = TRUE;
-- Clone a database for development
CREATE DATABASE dev_analytics
CLONE prod_analytics;
-- Query historical data (time travel)
SELECT *
FROM orders AT(TIMESTAMP => '2025-01-15 12:00:00'::TIMESTAMP);Google BigQuery
BigQuery is a serverless data warehouse on GCP. It automatically manages compute resources, scaling transparently to petabytes.
Key Features
- Serverless - No clusters to manage; compute scales automatically
- Columnar storage - Capacitor columnar format with high compression
- BigLake - Query external data in GCS, S3, or Azure Blob with unified access control
- Omni - Run BigQuery queries across multi-cloud (AWS, Azure) from a single interface
- Partitioning + clustering - Fine-grained data organization for cost and performance
-- Create a partitioned, clustered table CREATE TABLE my_dataset.events ( event_id STRING, event_name STRING, user_id INT64, event_time TIMESTAMP, payload JSON ) PARTITION BY DATE(event_time) CLUSTER BY user_id OPTIONS( partition_expiration_days = 365, require_partition_filter = TRUE );Amazon Redshift
Redshift is AWS's fully managed data warehouse, known for its columnar storage and massively parallel processing (MPP) architecture.
Key Features
- RA3 nodes - Separate compute and storage with managed storage (Redshift Managed Storage)
- AQUA - Advanced Query Accelerator for faster scans on compressed data
- Spectrum - Query data directly in S3 without loading into Redshift
- Auto WLM - Automatic workload management for concurrent queries
Databricks
Databricks provides a unified analytics platform built on Apache Spark, offering both data engineering and data science workflows.
Key Features
- Delta Lake - ACID transactions, schema enforcement, and time travel on data lakes
- Unity Catalog - Fine-grained governance across workspaces and clouds
- Serverless SQL - Auto-scaling SQL warehouses with per-second billing
- MLflow - Machine learning lifecycle management built in
- Delta Sharing - Open protocol for sharing data across platforms
Platform Comparison
| Feature | Snowflake | BigQuery | Redshift | Databricks |
|---|---|---|---|---|
| Serverless | Yes (auto-resume) | Fully | RA3 reduces ops | Serverless SQL |
| Compute/storage separation | Full | Full | RA3 nodes | Via Delta Lake |
| Open formats | Proprietary | Proprietary | Proprietary | Delta, Parquet, Iceberg |
| Multi-cloud | AWS, Azure, GCP | GCP + Omni | AWS only | AWS, Azure, GCP |
| Python/notebooks | Snowpark | BigQuery DataFrames | Limited | Native |