Reflective Data Catalog

Reflective Data Catalog#

Reflective’s unified Python interface for accessing SAI (Stratospheric Aerosol Injection) climate model data across cloud providers (S3, GCS, Azure, Cloudflare R2) for use on the Reflective Cloud Hub. Note: At this time, most non ESGF and ESM sources will not be accessible outside of the Reflective Cloud Hub due to technical limitations. We are working on storing the data in new locations and will update when it’s ready.

This is an Intake-like interface for browsing, searching, and loading SRM related datasets in a unified manner. All available datasets can be seen here with more infomration in the Reflective Cloud Hub documentation. We’ve also included an example Jupyter Notebook showing how to use the tool.

Installation#

pip install reflective-data-catalog

For development:

git clone https://github.com/ReflectiveCloud/reflective-data-catalog.git
cd reflective-data-catalog
pip install -e ".[dev]"
pre-commit install

This installs a pre-commit hook that automatically runs Ruff linting (with auto-fix) and formatting on every commit.

Quick Start#

from reflective_data_catalog import ReflectiveCatalog

rdc = ReflectiveCatalog()

# Load a dataset lazily with dask
ds = rdc.cesm2_waccm_g6_1p5k_hilla(variable='T').to_dask()

# Load a dataset into memory
ds = rdc.miroc_es2h_g6_1p5k_sai(variable='SurfT').read()

Available Sources#

Source	Description
`cesm2_waccm_g6_1p5k_hilla`	CESM2-WACCM G6-1.5K-HiLLA
`cesm2_waccm_historical`	CESM2-WACCM Historical
`cesm2_waccm_ssp245`	CESM2-WACCM SSP2-4.5
`cesm2_waccm6_g6_1p5k_hilla`	CESM2-WACCM6 G6-1.5K-HiLLA
`e3smv3_g6_1p5k_hilla`	E3SMv3 G6-1.5K-HiLLA
`miroc_es2h_g6_1p5k_hilla`	MIROC-ES2H G6-1.5K-HiLLA
`miroc_es2h_g6_1p5k_sai`	MIROC-ES2H G6-1.5K-SAI
`ukesm1_g6_1p5k_hilla`	UKESM1.1 G6-1.5K-HiLLA
`ukesm1_ssp245`	UKESM1.1 SSP2-4.5

Usage#

Selecting Parameters#

Each source accepts keyword arguments to select the table, variable, ensemble member, and other parameters:

# Specify variable, table, and ensemble
ds = rdc.cesm2_waccm_g6_1p5k_hilla(
    variable='T',
    table='AMON',
    ensemble='r2'
).to_dask()

# MIROC: HiLLA vs SAI are different experiment prefixes in storage — use the matching source
ds = rdc.miroc_es2h_g6_1p5k_hilla(
    table='Amon',
    variable='tas',
    variant='baseline',
    ensemble='r01',
).to_dask()
ds = rdc.miroc_es2h_g6_1p5k_sai(
    table='Mon',
    variable='SurfT',
    ensemble='r01',
).to_dask()  # default variant is G6-1.5K-SAI

Discovering Available Data#

Each source provides discovery methods to explore what data is available:

source = rdc.ukesm1_g6_1p5k_hilla()

# List available variables, ensembles, or tables
source.list_variables()
source.list_ensembles()
source.list_tables()

# Print a full summary
source.discover()

Google Cloud CMIP6 / GeoMIP (intake-esm)#

Access cloud-optimized Zarr data from the Google Cloud CMIP6 catalog:

# Search and load in one step
datasets = rdc.esm.load(
    experiment_id=['G6sulfur', 'ssp245', 'ssp585'],
    variable_id='tas',
    table_id='Amon',
    require_all_on=['source_id', 'institution_id'],
)

# Or use the GeoMIP convenience helper
datasets = rdc.geomip_cloud.load_ensemble(
    experiments=['G6sulfur', 'ssp245', 'ssp585'],
    variable='tas',
)

# Quick single-experiment load
ds_dict = rdc.geomip_cloud.g6sulfur(variable='tas')

# Explore what's available
rdc.geomip_cloud.list_models()
rdc.geomip_cloud.list_variables(experiment_id='G6sulfur')
rdc.geomip_cloud.summary()

# Advanced: direct search then load
subset = rdc.esm.search(
    experiment_id='G6sulfur',
    variable_id=['tas', 'pr'],
    table_id='Amon',
)
datasets = subset.to_dataset_dict()

ESGF Data#

The catalog also provides access to ESGF (Earth System Grid Federation) data:

ds = rdc.esgf.geomip.g6sulfur(model='UKESM1-0-LL', variable='tas')

Running Tests#

Run the full test suite:

pytest

Run with coverage report:

pytest --cov=reflective_data_catalog --cov-report=term-missing

Run a specific test file:

pytest tests/test_flexible_sources.py

Tests mock all external services (S3, ESGF, intake-esm) so no network access or cloud credentials are required.

Requirements#

License#

Apache 2.0