v0.12.2

Changelog

Bug Fixes

• Improved conda error logging by capturing stderr in solve logs and avoiding unnecessary stacktraces. (#580)

Trivial/Internal Changes

• #580

Changes

• 7bc2bff Bump version: 0.12.1 → 0.12.2
• 02c2e0c chore: add second EOF
• 78e84db Bump version: 0.12.0 → 0.12.1
• fac866b chore: support mip_id
• d09556c fix: update test to expect CondaCommandError instead of CalledProcessError
• ba662d2 docs: add changelog entries for PR #580
• 29bba46 fix: Log conda stderr to solve logs
• 7d269b6 chore: add extra cvs

v0.12.0

Changelog

Features

• Added ESMValTool ozone diagnostics. (#473)

• Metric for diagnostics of double ITCZ was added: Spatial corrleation of simulated DJF precipitation climatology against reference dataset over the area of 20S to 0 latitude and 100 to 210 longitude. (#557)

• Added lazy loading and finalisation support for CMIP7 datasets via DRS and complete parsers, matching the existing CMIP6 pattern.

  This removes the realm filter for CMIP7 executions as it cannot be parsed from the DRS.
  We strongly recommend the use of branded variables when filtering CMIP7 datasets to properly constrain the expected variables. (#571)

• Update ilamb3, remove the ohc-noaa diagnostic and add the evspsbl-pr diagnostic. (#573)

• Added time_units and calendar metadata to CMIP6 and CMIP7 datasets, enabling proper handling of non-standard CF calendars such as 360_day and noleap. Time values are now stored as cftime.datetime objects instead of datetime.datetime. (#574)

• Added CMIP7 data catalog and ESMValTool recipe variants for CMIP7 diagnostics. (#577)

Improvements

• Implemented a memory constraint using the environment variable MEMORY_LIMIT_PARSL_JOB_GB to set the memory limit (units: GB) for a PARSL worker. (#464)
• Extracted shared finalisation logic into FinaliseableDatasetAdapterMixin, reducing code duplication between CMIP6 and CMIP7 adapters. (#571)
• Use the metadata from the data catalog instead of reading netCDF files when determining ESMValTool branch times (#577)

Bug Fixes

• Fixed fire diagnostic CMIP7 data selection and recipe writing. (#540)
• Migrated to use a different post-processed JSON file for mapping CMIP6 compound names to CMIP7.
  This fixes some errors when dealing with tasmax/tasmin and the removes the out_name attribute which was correctly included in #530. (#547)
• Apply same version filtering logic to the regression tests as loading the data catalog (#570)

Changes

• cb55f38 Bump version: 0.11.1 → 0.12.0
• 7aefb3d chore(deps): bump the github-actions group with 2 updates
• 38a3866 chore: fix test
• d1f47f2 chore: Add regression outputs
• 4e01e96 chore: update catalog
• 46a12ea chore: add obs4mips requirement
• a0d92c6 test: Fix failing tests
• 27bb39f fix(solver): re-finalise group after constraints that add unfinalised rows
• 7bd41af chore: Add empty solves
• 5591831 chore: Support cmip6/7
• a3cddcd fix: sanitize non-YAML-safe values in save_datasets_to_yaml
• 055c3a8 chore: add test_specs
• 9777aa8 chore: ruff
• 1632d39 chore: run all solves
• 380d911 chore: add a cmip7 data catalog and regenerate the ESMValTool tests
• 126781b chore: add CMIP7 data catalog
• 003a2c6 chore: Use a subset of cmip6 source ids
• efc2461 chore: update catalog
• 4aa128e chore: add ozone
• 6f0f36c refactor: add a esgf_data_catalog_trimmed fixture
• dd95c00 chore: add back the catch
• ff5dd49 chore: cleanup is_na checks
• a6c1c91 chore: fetch obs4ref datasets too
• e5c68fb fix: add postgresql_using cast for String->DateTime downgrade
• 3697ab9 fix: use FALSE instead of 0 in migration for PostgreSQL boolean column
• 01c93c9 fix: use enum names in migration UPDATE for PostgreSQL compatibility
• e51352d chore: remove test
• 116520f chore: skip the diagnostic
• fa17b8e chore: Add regression file
• da8cbee docs: add changelog entry for time_units and calendar feature
• 98649d4 fix: handle mixed-calendar cftime comparison in contiguity constraint
• a403a21 chore: rename changelog to match PR #573
• 063e7cb chore: add lossy compression as the thetaO files are large
• 404b25b chore: add other missing variable
• 5e29107 chore: add missing variable
• c55b29c revert: Undo the changes due to new datasets
• b0f18d7 test: update tests
• 3d9560a chore: add migration
• 5a82ac6 feat: parse into cftime at boundary
• 981cb4f chore: add time_units and calendar
• 6741964 chore: read mandatory variables
• 09d9efc fix: Better handling of nan values
• 1b85499 docs: Update documenation about DRS parser
• 78f46df docs: changelog update
• e43f358 fix: correct the order of the constraints
• b044e53 chore: add changelog entries for PR #571
• 761022e chore: Clean up CMIP6 and CMIP7 test overlap
• 866be11 chore: refactor tests
• f90ca0d chore: update tests
• 9a600f8 chore: remove realm from filtering
• b78c97c feat: Add CMIP7 lazy loading
• 650dd6f chore: Fix regression outputs
• 88b9d5b chore: changelog
• 019f870 fix: apply version filtering to the regression analysis
• 911c87e chore: Use variable_name
• c81d78b chore: fix pre-commit
• 999d944 add change log
• 5c69847 rename double_ITCZ to doubleITCZ for consistency in output file naming
• 5e54e72 chore: add test case for double ITCZ
• 8ecfcc3 chore(deps-dev): bump jupyterlab in the python-dependencies group
• e9ad222 chore: clean up datarequirements
• e8fab3e chore: loosen facet filter type
• 5393d00 chore: remove ohc dataset
• 8815a50 chore: fix mypy
• 55ccbca add changelog
• 5fb2f31 add tests for memory limit
• 67d0378 expand ilamb3 ref configure files to allow for dictionaries with data requirement facets
• e5b3ca9 replace the hard-coded memory limit with an env variable
• da7bade initial commit
• 3cc6603 fix: lockfile
• 9f66c29 fix: Explicitly handle extra spurious extra coord
• 9aed0cf fix: correctly parse variant indexes in CMIP6-to-CMIP7 conversion
• 14f1cb3 chore: remove physical_parameter_name as it isn't needed
• 2cf846b changelog
• d6d3b9c bump ilamb3 version
• fe0b9f9 add ilamb model color selector, only 1 model but at least won't be black
• 55553f8 add the evspsbl-pr metric only over land as requested
• c77383e append the obs4REF dataset registry to the ilamb so these keys can be used in configure files
• ade4287 change how models are named in the plots and output filenames
• 2a6f5c8 changes to configure files, data references updated where possible
• 2deee16 Temporarily remove the ocean heat content diagnostic as per email from Forrest (on behalf of the MBTT) on 17-Feb
• 7123b57 fix: test
• 9386b77 chore: use the common cache dir
• abbf776 chore: use correct drs
• 38952dd chore: add missing attributes
• cd33274 Fix fire diagnostic CMIP7 data selection and recipe writing
• 166b744 docs: Changelog
• c713489 refactor: Refactor DReq variable handling and filename generation
• 91df83f added 473 to change log
• 5501551 automatic formatting fixes
• 7402cb7 cleaned up ozone.py
• b16785d update ozone-zonal diagnostic
• cb81694 add ESACCI-OZONE data (variable o3) for ESMValTool
• f1e8588 first draft of esmvaltool ozone diagnostics
• 003cde9 memory limit to a parsl worker in HPCExecutor

v0.11.1

Changelog

Bug Fixes

• Fixed DRS re-ingestion from crashing or regressing already-finalised datasets. Previously, re-ingesting the same directory with the DRS parser would either crash with a TypeError due to pd.NA comparisons, or overwrite finalised metadata with empty values. Finalised datasets are now skipped during DRS ingestion while still adding any new files.

  Reduced memory usage during dataset ingestion by releasing ORM objects from the SQLAlchemy session after each dataset commit, preventing unbounded memory growth on large archives. (#567)

Changes

• 271f0a8 Bump version: 0.11.0 → 0.11.1
• f96176c docs: add changelog for #567
• 56fe6ee fix: reduce memory during ingestion and add solve logging
• 9f57e32 fix: prevent DRS re-ingestion from regressing finalised datasets

v0.11.0

Announcements

This release brings large performance improvements for ingesting datasets, particularly for low IOPs filesystems. The previous ingestion process opened each NetCDF file to extract metadata which is slow for large archives.

A new "drs" CMIP6 parser has been introduced which uses the CMIP6 DRS for the initial ingest. The remaining metadata is then loaded at solve time if a dataset might be needed for a diagnostic. This allows users to ingests a large swath of data and only the datasets needed for diagnostics will be materialised.

This can be enabled by setting the cmip6_parser to drs.

Changelog

Breaking Changes

• Changed get_branding_suffix, get_realm, and get_cmip7_compound_name to require a table_id parameter in addition to variable_id, enabling Data Request compound name lookups. (#530)

• Failed diagnostic executions now clear the execution group's dirty flag,
  preventing automatic retry on subsequent solves.

  Previously, failed executions were retried indefinitely.
  Use ref solve --rerun-failed or ref executions flag-dirty to explicitly retry failed diagnostics.
  The solver also now skips duplicate submissions when an execution with the same dataset hash is already in progress. (#552)

Features

• Added a constraint to add the parent experiment. (#214)

• Added lazy dataset ingestion with two-phase finalisation. Datasets are now bootstrapped from directory structure metadata only (no file I/O), with full metadata extracted lazily at solve time after filtering narrows candidates. This dramatically reduces ingestion time for large CMIP6 archives on HPC parallel file systems. (#515)

• Added CMIP7 support to all ESMValTool diagnostics using OR-logic data requirements, enabling automatic evaluation of CMIP7 datasets alongside existing CMIP6 support. (#519)

• Added CMIP7 data requirements and test data specifications for all PMP diagnostics (annual cycle, ENSO, and variability modes). (#526)

• Added esgf_data_catalog test fixture and per-provider solver regression baselines using pre-generated parquet catalogs, enabling solver regression testing without requiring sample data downloads. (#529)

• Added structured CMIP6-to-CMIP7 variable mappings from the CMIP7 Data Request, with a typed DReqVariableMapping class for reliable branding suffix, realm, and output name lookups. (#530)

• TOML file for QAQC requirement from REF added. (#532)

• Added dimensions to files produced by ESMValTool diagnostics. (#534)

• Added CMIP7 data support to ILAMB diagnostics, enabling dual CMIP6/CMIP7 data requirements with branded variable name lookups and dynamic source type detection. (#535)

• Distinguished system errors (OOM, disk full, worker crash) from diagnostic logic errors when handling execution failures.
  System errors leave the execution group dirty so they are automatically retried on the next solve,
  while diagnostic errors clear the dirty flag to prevent retrying indefinitely with the same data.

  The solver also now skips duplicate submissions when an execution is already in progress for the same dataset hash.

  Added --rerun-failed and --no-wait flags to ref solve, and a new ref executions fail-running command for marking stuck executions as failed. (#552)

• Added validation that prevents DataRequirements from filtering or grouping on columns that require dataset finalisation, raising a clear error instead of silently producing empty results. (#561)

• Added --limit flag to the solve command to cap the number of executions, and --dataset-filter option to both solve and datasets list commands to filter input datasets by facet values before solving. (#562)

Improvements

• Improved solver performance by caching slug column lookups and avoiding expensive DataFrame string representation in debug logging. (#533)
• Removed the ecgtools dependency and 16 transitive dependencies (intake, intake-esm, joblib, zarr, etc.) by replacing it with a focused internal catalog builder module. This also eliminates the pydantic v1 deprecation warning that ecgtools was causing. File parsing now shows a tqdm progress bar. (#558)
• Replaced xarray with netCDF4 for metadata-only reads during dataset ingestion, significantly reducing per-file parsing overhead. (#559)
• Parallelised the finalise_datasets operation for CMIP6 datasets, mirroring the threaded approach used during ingest.
  The number of worker threads is controlled by the existing n_jobs parameter on CMIP6DatasetAdapter. (#564)

Bug Fixes

• Removed deprecated mix_stderr parameter from CliRunner in the test fixture, fixing compatibility with Click 8.3+. (#528)
• Fixed a ValueError in AddSupplementaryDataset when the data catalog contained duplicate index labels for supplementary datasets. (#537)
• Fixed confusion between variable_id and out_name in fake CMIP7 data. (#539)
• Improved the resiliance of the celery worker configuration to failures (#550)
• Mount a celeryconfig.py via ConfigMap for Flower so accept_content is read correctly by Celery's config loader. The CELERY_ACCEPT_CONTENT env var is not picked up by Flower/Kombu directly. The config is user-configurable via flower.celeryConfig in Helm values. (#556)
• Fixed REF_CMIP6_PARSER and REF_LOG_FORMAT environment variables not being applied because the Config class was missing the post-init hook for environment variable overrides. (#561)
• Resolved pandas FutureWarnings to support both pandas 2 and 3, including fixes for DataFrame concatenation with empty or all-NA entries and null-type mismatches in parquet round-trips. (#565)

Trivial/Internal Changes

• #538, #555

Changes

• 3bfd343 Bump version: 0.10.0 → 0.11.0
• ca8d500 chore: Update comment
• 4ffeaba chore: upgrade pins for ilamb
• 4396148 fix: revert compat=override on open_mfdataset
• a2b86e3 docs: add changelog for #565
• 4f5d1b0 chore: Upgrade lockfile and fix some errors
• e1a3b27 chore: add coverage
• 6d25567 chore: add default separator in alembic
• eef3980 fix: time_coder warning
• bc4e6ed chore: Pin to use tas
• d96d0fb fix(solver): preserve DataCatalog wrapper in apply_dataset_filters
• 95ae318 fix(tests): use to_frame() when accessing DataCatalog in solver tests
• ca4f8e2 docs: Changelog
• d89b359 chore: run the finalise in threads
• a8cf780 chore: clean up
• 51f8443 chore: add fix changelog entry for PR #561
• 4cb0432 feat(cli): add --dataset-filter option to datasets list command
• 22085b5 chore: add changelog entry for PR #561
• 17c47bd feat(solver): add --dataset-filter option to filter input datasets when solving
• 3142e2b chore: Support env variables for parser
• d335227 feat(solver): add optional --limit flag to solve command
• 23ca55b feat(solver): validate DataRequirement columns against adapter finalisation
• 30bd383 test: improve test coverage
• 1fff6b9 test: add an integration test comparing the lazy loading to complete runs
• 58e2044 chore: fix regression output
• b49ae0f chore: enforce known converted file
• 3e97a10 chore: log exception
• 8dfc89e perf(datasets): replace xarray with netCDF4 for metadata-only reads
• 10be217 chore: extract dates directly
• 445a7fc docs: add changelog entry for ecgtools removal
• 2056c2a refactor: use tqdm for parsing progress in catalog builder
• 0e6aa7f feat: add progress logging to catalog builder
• b3dae9e refactor: remove ecgtools dependency
• ef366f5 test: Fix failing test
• b425c42 chore: Move the check location
• 8a4c3d7 chore: actually run the tests...
• bca97bf chore: Remove checks
• 084ac07 fix(testing): raise clear error when dataset paths are missing
• 41102b2 chore(helm): add init container to wait for Dragonfly before starting Flower
• 32a45e8 fix: the fetch has to happen in the test-cases job to set the paths
• 52c54f0 chore: add more logging
• 5df80d7 chore: add workflow
• 41b14cc chore: don't show tqdm on downloads
• fdab615 chore: add esgf cache info
• 80b183c chore: retry checking if the flower instance is healthy
• 3cbf143 chore: remove previous fetches
• 4e415ed chore: wait for healthcheck
• 0e9c24a docs: add changelog for flower celeryconfig fix
• 5d1c98c feat(helm): add Flower celeryconfig and CI worker health check
• 7af141a doc: update docs for in...

v0.10.0

Announcements

This release add initial support for CMIP7 datasets.
This is still a work in progress so the list of the attributes that are tracked may change as we integrate the providers.

Changelog

Features

• Added database support for CMIP7 datasets based on the CMIP7 Global Attributes v1.0 specification. (#503)
• Added CMIP7 data requirements support, enabling providers to fetch CMIP6 data from ESGF and translate it to CMIP7 format using the CMIP7 CV converter. (#510)
• Added diagnostic summary introspection and auto-generated documentation for all providers. The ref providers show command now defaults to detailed list format and supports --columns for filtering table output. (#518)

Improvements

• Separated model from observation runs for regional historical diagnostics. (#460)
• Made listing the changed files from a regression test faster. (#514)
• Prepared the monorepo for splitting diagnostic provider packages into independent repositories. Extracted shared test fixtures into a climate-ref[test] pytest plugin, decoupled climate-ref-core from application-level types, and added API surface documentation, versioning strategy, provider compatibility CI, and a copier template for bootstrapping new provider repositories. (#520)

Bug Fixes

• Fixed CMEC bundle dimension validation to use a subset check instead of exact equality, allowing diagnostics with multiple data requirements to have varying output dimensions. (#523)

Trivial/Internal Changes

• #516

Changes

• b31c8a5 Bump version: 0.9.1 → 0.10.0
• 2ff4f58 chore: add changelog entry for PR #523
• 9b809ee fix: use subset check for CMEC bundle dimension validation
• ce56f34 fix(helm): use distinct chart version tags per build context
• 9a04820 chore: remove unnecessary conftest.py from climate-ref-core tests
• 38454c9 fix: address PR review comments
• 0ea9510 fix: improve diff coverage and restrict CI permissions
• a461179 fix(ci): use coverage run for proper entry-point plugin tracking
• 603dae7 fix(ci): skip conftest_plugin in imports-without-extras check
• d22d9d8 chore: add changelog entry for #520
• 9feddb4 ci: add provider compatibility workflow and copier template
• 491912a docs: add API surface, versioning strategy, and contributor guide
• 2121d1c refactor: extract shared test fixtures into climate-ref[test] pytest plugin
• 0953583 chore: add main_log_debug.txt to gitignore and remove existing files
• 57ee249 Apply suggestion from @lewisjared
• f6b876b docs: add changelog entry for PR #518
• 52fdcf4 feat(cli): make list format default for providers show and add --columns option
• 8145dee feat: improve diagnostic docs with admonitions, tabs, and overview by source type
• 7f087aa feat: add diagnostic summary introspection and documentation generation
• 8a758f6 test: improve coverage for CMIP7, solver, and test_cases CLI
• 887242e fix: resolve merge conflict in solver OR-logic after InvalidDiagnosticException removal
• ebd8a0f fix(tests): set up context manager mocks for xr.open_dataset in CMIP7 tests
• 04b7e3f Add changelog
• e81fdef Add codecov.yml with relaxed coverage thresholds
• 7aa967c feat(cmip7): add license_id and external_variables to CMIP7 model
• ce39225 Also update output collection of regional historical trend diagnostic
• f42fac2 Add changelog
• f942b93 Faster listing of regression data
• 9e334b9 Add another variable to test case
• a8fd54b Do not try to push to container registry from forks
• 2bb22bc Update regression test output
• fda40b4 Add changelog
• 3a2083b Split trends recipe and fix the other two regional recipes
• d787258 Update recipes
• c83a5c5 Update regional historical annual cycle and timeseries
• 97242bf fix: address PR review comments
• 32a9b7f test: increase test coverage for CMIP7 data requirements
• 5d09eba docs: add changelog for PR #510
• bcc1754 feat: add CMIP7 data requirements support for Example provider
• 8377a0a style: reorganize import statements for clarity
• 8e47b3e refactor: address PR review comments for CMIP7 support
• 624f40e Add tests for CMIP7 adapter with CMIP6-converted files
• d70e0a2 Add changelog for PR #503
• 0614379 Add CMIP7 database support

v0.9.1

Changelog

Features

• Added ingest_data() lifecycle hook to providers, enabling automatic dataset ingestion during ref providers setup. PMP climatology data is now ingested automatically, eliminating the need for a separate manual ingestion step. (#508)

Improvements

• Implemented coupled versioning for Helm chart: chart version, appVersion, and default image tag now stay in sync with the application version and are updated automatically by bump-my-version. (#507)
• Improved CLI performance by skipping database backup for read-only commands like config list and datasets list. (#511)
• Improved CLI startup time by deferring heavy imports until needed. (#512)

Bug Fixes

• Fixed Helm chart CI to use correct image tag override path (defaults.image.tag instead of invalid climate-ref.image.tag). (#507)

Improved Documentation

• Updated getting started documentation with clearer configuration and dataset download instructions. (#508)

Changes

• b06b4cc Bump version: 0.9.0 → 0.9.1
• c579e31 docs: fix admonition syntax and typo in getting-started guides
• 0d71679 fix(tests): update patch paths for lazy imports
• 6311473 chore: add changelog and fix lint after merge
• 4d500fd test: add unit tests for providers, datasets, config, and models
• ec2352f ci: only run helm jobs when helm/ changes or on main
• ab47504 perf: move SourceDatasetType to lightweight module and add more lazy imports
• 5100872 perf(cli): optimize startup time with lazy imports
• 5b546b0 chore: add changelog for PR #511
• bb10a39 feat(cli): skip database backup for read-only commands
• 758e672 fix: Use the right path
• 8df2043 docs: add changelog entries for PR #508
• c371c79 feat(providers): add ingest_data hook for provider-level dataset ingestion
• 4335793 docs: update configuration and dataset download instructions; remove empty tutorials section
• ba9be9d chore: add skip-validate
• 0086ade docs: add changelog entries for Helm versioning fixes
• a368352 chore: update Helm chart versioning and defaults to 0.9.0
• f2ed7dd chore: bump to latest image
• 7189d4a test(packaging): use the setup command

v0.9.0

Announcements

This release brings some upgrades to the testing framework we use and a streamlined setup process.
This should make it easier to add new diagnostics.

We have temporarily pinned pandas<3 until we can verify that it everything works as expected.

Changelog

Features

• Added test data management infrastructure for diagnostic development:

  • New ref test-cases CLI commands (fetch, list, run) for managing and running diagnostic test cases.
  • ESGF data fetching utilities with support for CMIP6 and obs4MIPs datasets.
  • TestDataSpecification and TestCase classes for defining reproducible test scenarios.

  (#475)

• Add functionality to translate a CMIP6 dataset to the new CMIP7 conventions (#484)

• Added test automation infrastructure for diagnostic testing using test-cases (#485)

• Added CMIP6 to CMIP7 format conversion command-line script to translate CMIP6 datasets into CMIP7-compatible format. (#489)

• Added RegistryRequest class for fetching datasets from pooch registries (pmp-climatology, obs4ref) instead of ESGF. (#490)

• Added Helm chart for Kubernetes deployment with automated CI/CD pipeline for building and publishing the chart to GitHub Container Registry, including deployment templates for provider workloads (ESMValTool, PMP, ILAMB), Flower monitoring UI, Dragonfly Redis dependency, and comprehensive integration testing in minikube. (#492)

• Add CI workflow to verify solve works without network access (ci-offline-solve.yaml).
  This test uses Docker with --network none to block all network access including
  subprocesses. Runs every other day and can be triggered manually. (#497)

• Added provider lifecycle hooks for offline execution setup. Providers can now implement setup_environment(), fetch_data(), and post_setup() methods to prepare for execution on HPC compute nodes without internet access. A new ref providers setup CLI command runs all provider setup hooks, fetching required reference data to the local cache before offline solving. (#498)

Improvements

• Clean up the open database connections in the test suite (#482)
• Improved catalog handling with hash-based change detection and multi-file dataset support. Enhanced CLI test-cases commands with new flags: --only-missing, --force, --dry-run, --if-changed, and --clean. (#490)
• Updated ESMValTool to v2.13.0 (#500)

Trivial/Internal Changes

• #480, #506

Changes

• 2eb538f Bump version: 0.8.1 → 0.9.0
• c435cb4 docs: Changelog
• f8caea7 chore: pin pandas to <3
• 6d3d884 chore(deps-dev): bump the python-dependencies group across 1 directory with 9 updates
• 6e98652 chore: add more test coverage
• eb655a5 Add changelog
• db4ca16 Regenerate example output
• d7ed855 Update ESMValTool to v2.13.0
• b8f7ce6 refactor: address PR review feedback for provider lifecycle
• 2368a2d Update docs/how-to-guides/hpc_executor.md
• 5dcead9 refactor: address PR review feedback for provider lifecycle hooks
• 690f759 docs: update provider setup documentation
• 439a85d revert: remove blocking
• a6c47ae docs: add changelog for provider lifecycle hooks
• 0892d14 fix(config): handle network errors when fetching ignore datasets file
• 665ccd6 feat(providers): add lifecycle hooks for offline execution setup
• c4323d4 revert: makefile
• e477331 test: add Docker-based offline solve test workflow
• 524ea64 chore: don't download all data
• 089de09 fix: add conda environment setup before offline tests
• 2c1a436 fix: fetch test data before running offline solve tests
• 28f1070 test: add CI job to verify solve works without network access
• b0dc7e2 chore: ignore all .ref
• 850c823 chore(deps): bump the github-actions group with 4 updates
• 0d7e81b docs: add documentation for the helm chart
• 62e7768 fix: Set HOME dir to a writable location
• c9aedb0 revert: undo livliness checks for now
• 34b2e16 feat: add liviness probe
• 950b17f chore(deps-dev): bump the python-dependencies group with 3 updates
• 2002e40 chore: Re-enable CI/CD workflows
• 51b163e feat: Build and publish Helm chart in CI
• fd730d3 chore: Add pragma: no cover for display-only CLI code
• 7322e25 refactor: Extract utilities from test_cases.py and add tests
• f878721 docs: Add changelog entries for PR #490
• 149e302 feat: Add RegistryRequest for pooch registries and improve catalog handling
• 724be90 docs: Changelog
• b3d0135 feat: added script to translate CMIP6 to CMIP7
• 22d1eab fix: Updates to understanding from Zeb
• 06a119e chore(deps-dev): bump the python-dependencies group with 4 updates
• d407a89 chore: remove the unneeded files
• 018b53c test: Fix udpates to paths
• a8c5f44 chore: Updates after rereading the global attributes file
• 69b8ec1 chore: use parameterised tests instead
• cb246c0 test: improve coverage
• 0beb3fa Update packages/climate-ref-core/src/climate_ref_core/cmip6_to_cmip7.py
• bb8fefc doc: Changelog
• b2bae88 chore: run test-cases separately
• a4ee861 chore: ignore pytest warning
• bd408dc chore: add example regression output
• 58fd3c9 feat: Add a helper class for tracking test case paths
• 3ea8367 test: offload current testing to the validate cmec bundles routine
• 683fdee chore: run the example diagnostic
• a75e074 chore: add example catalog
• 7ac134a chore: move test case tests to their respective packages
• ce706b4 chore: move catalog to a provider-specific path
• a5e2659 chore: remove doctest as the datafile isn't available
• 83147da docs: Changelog
• daf3c8a feat: Add cmip6 to cmip7 translation
• 7227336 fix: Allow fetching multiple variables
• 5a8b72e fix: typos in docs
• 2e4a7d3 fix: remove unused function
• 236c0b3 chore: improve coverage
• 281c2aa chore: clean up tests
• a27870f feat: Generate a data catalog on fetch
• 6be929d chore: Move ESGF_DATA_DIR definition to testing
• 1f0e89b Update packages/climate-ref/tests/unit/datasets/test_cmip6.py
• 779f974 chore: Fix more warnings
• 51c9b49 chore: allow to fetch and run
• 39d0ef7 chore: Clean up db in the cli
• d9dac70 chore: use pytest 9's strict mode
• e84660c test: Fix old testing value
• 20aeda6 docs: Changelog
• edb2965 chore: Fix sqlite resource errors
• bb2f52a refactor: rename commands
• 7777f24 test: Add more coverage
• 6b850b1 test: test testing tests
• 3b39fab docs: Add documentationf for the new diagnostics
• 77202fc test: improve coverage
• f365d06 doc: Changelog
• 27dde70 revert: Go back to separating stderr and stdout
• 57ea206 chore: pin xarray for ilamb3 due to issue
• 3dc9ca1 Potential fix for pull request finding 'Explicit returns mixed with implicit (fall through) returns'
• 6f70658 fix: handle a regression in xarray
• 84ec4fe fix: Clean up some type hints from updating mypy
• 5326654 docs: fix typo
• e3b2541 feat: Add a harness to run a set of specs
• aafd16d chore(deps-dev): bump the python-dependencies group with 31 updates
• 3b6d9e1 fix: use uv not pip
• 4084648 docs: Changelog
• b7384fb ci: Ignore dependabot PRs
• a1e0121 chore(deps): bump the github-actions group with 5 updates
• bccb710 chore: add ipython dep to fix CI
• 35f8a05 test: Add coverage for the new modules
• d189868 feat: add test spec
• 1a3894c feat: Add esgf fetching utlities from sample repo
• 50ca6bc chore: add required packages

v0.8.1

Changelog

Bug Fixes

• Add a pin for fastprogress (dependency of intake-esm) to work around bug in newer versions (#476)

Trivial/Internal Changes

• #477

Changes

• 7a6c236 Bump version: 0.8.0 → 0.8.1
• 50933b8 docs: Clean up missing changelogs
• 5e51f8f ci: Update uv in CI
• 3f70a7c chore: Add dependabot
• 7caeda5 chore: change to pin for fastprogress
• ef060b7 fix: Bump a yanked version of deprecated
• 9ea1da7 fix: Add IPython to depedencies temporarily

v0.8.0

Changelog

Features

• Enable spatial 3-d variables that has levels for PMP annual cycle. (#411)

• Added ignore datasets constraint and configuration file.

  If no path is configured for the ignore_datasets_file in the configuration file,
  the default ignore datasets file is downloaded from the Climate-REF GitHub repository
  if it does not exist or is older than 6 hours. (#447)

Improvements

• Validated the slurm configurations of the HPCExecutor using pydantic (#375)
• Added reference values to ESMValTool series output. (#452)

Bug Fixes

• General fixes found when rerunning, including handling an edge case where no log output is written,
  ignoring empty input directories and increased logging of the number of executions. (#444)
• Worked around pydata/xarray#2742
  by always replacing the default fillvalue for the data type with NaN in arrays
  read with Xarray. (#454)
• Excluded piControl from PMP annual cycle and variability metrics owing to non-overlapping periods with observations. (#463)

Improved Documentation

• Add a Jupyter notebook showing how to use the CMIP7 Assessment Fast Track website OpenAPI. (#466)

Changes

• 31aa63f Bump version: 0.7.0 → 0.8.0
• f6fe0e8 Fix formatting
• 8b11814 Apply suggestions from code review
• 1229263 Another attempt at having colored output in notebooks
• 478762c Use color in cells
• 364e6d4 More updates
• d5a49f1 rerun ci
• 7aca1de Rename subtitle
• 6d53277 Fix dataset selection notebook
• e49d967 Update dependencies
• 0b66a39 Add changelog
• 0ac7809 Rename
• 0f170d5 Add more examples
• f0a240d Run notebooks
• a408315 Add metric values to the notebook
• 6e8fda5 Add API demo notebook
• 4f24e01 add change log
• 5479e51 removed picontrol
• c3a6874 Add ignore datasets configuration file
• c1cb343 Update reference output
• 735a246 Convert default fillvalues to NaNs
• 218d374 chore: Changelog
• 8b09184 chore: Rename changelog
• 9bbf327 Update packages/climate-ref-pmp/src/climate_ref_pmp/diagnostics/annual_cycle.py
• e589b46 Update reference output
• d58312f Add reference values to series output
• dc2f6ed minor fixes
• 75067cd add docs for the configrations of HPCExecutor validation
• ed442d1 minor cleanup on hpc.py
• 738a68c add NPGO regression test output files
• 5f15381 fix broken test for annual cycle
• 03fd7d3 Lee1043 pmp mov bugfix (#450)
• a5e0623 chore: Catch exception in local
• 515909e add test results
• 6fa4ce8 chore: Minor tweaks to config
• 137b886 test: Fix test
• 0329781 fix: Skip missing directories and continue
• be476cb chore: Add logs for dry-mode
• 7d5190f chore: Add log message for the number of solves
• eb3d33c clean up
• d6d10b7 clean up
• 9d06a32 clean up
• aa81f80 clean up
• 6edda7b add change log
• 6afc821 clean up
• 30688bf update
• 47ba229 update
• aca5dfd in progress
• 3f2ff0f clean up
• 9c3f374 in progress -- push for archiving the progress
• a5c9281 in progress
• a02adda update
• 41a25a5 initial commit
• dfff9d0 add changelog
• ce9e42f validate hpc slurm config using pydantic

v0.7.0

Changelog

Breaking Changes

• Use the directory structure template from the obs4MIPs specification to define
  the instance_id. (#383)

• Used logical or instead of logical and to combine multiple facet filters.

  This means that the selected input datasets will include items that match any of
  the specified FacetFilters, rather than only those that match all of them.
  This change was made to improve the usability of the option to use multiple facet
  filters, allowing users to be more specific with their filtering criteria. (#414)

• Changed all constraints so they return a filtered dataframe instead of a boolean.

  This allows using the constraints in multi-model diagnostics. (#416)

Features

• Add a timerange constraint to ensure the required data is available. (#399)
• Add a human readable representation for climate_ref_core.datasets.ExecutionDatasetCollection. (#401)
• Allow execution outputs to have dimensions.
  These dimensions allow the outputs to be filtered in the same manner as the metric values (#434)
• Adds filter options to the ref executions list-groups command.
  This allows users to filter by facet, provider and diagnostic,
  as well as whether the latest executions was successful
  or if the exection group was has been marked as dirty (#438)
• Added a ref executions delete-groups command for removing unneeded results from the database.
  This will delete the execution group and any executions and outputs associated with the group. (#441)

Improvements

• Updated the sample data to v0.7.3. (#402)
• PMP version updated to v3.9.2 (#404)
• Specified the "table_id" for all ESMValTool diagnostics to improve data selection. (#407)
• Added a groupby operation to the RequireFacets constraint to reduce failed runs because of missing data. (#408)
• Adds extra ESMValTool dimensions to the output series (#410)
• The dataset ingestion process now supports updating datasets if new files become available (#412)
• Added series output for ESMValTool diagnostics. (#413)
• Gracefully handles empty directories when ingesting.
  The default behaviour is now to skip any invalid datasets as that is a sane default when ingesting many datasets. (#419)
• Include obs4MIPs and thetao in the ESGF download script and turn into a CLI tool.
  This script was renamed to fetch-esgf.py to fetch-esgf.py as it now includes more than just CMIP6 data. (#420)
• Improved support for concatenating historical and future experiments in the ESMValTool recipe (#431)
• Log the ESMValTool recipe and configuration for easier debugging (#432)
• Adds output dimensions to ILAMB outputs.
  This adds additional dimensions and attributes to the series values (#435)
• Replaced instance_id by individual facets in group_by of ESMValTool diagnostics. (#439)

Bug Fixes

• Resolves missing index and index names for series (#410)
• Added missing column to the PMP dataset model (#418)
• Removes the ERA-5 ta data Obs4REF in preference for Obs4MIPs (#421)
• Avoid selecting the areacella variable when filling the recipe for cloud scatterplots. (#427)
• Fixes to the ocean heat content IOMB diagnostic (#433)

Trivial/Internal Changes

• #440

Changes

• e146ffb Bump version: 0.6.6 → 0.7.0
• 1864139 chore: Increase coverage
• 7403638 chore: Test other models
• f25a960 feat: Support removing outputs
• 2364218 docs: Changelog
• f290d13 doc: Changelog
• be1f2d2 feat: Add delete command
• 9819515 chore: Improve coverage
• f8ef3fa Remove outdated example output
• 5cbea59 chore: Fix ruff issues
• 68d48a8 Add changelog
• fcf956f Update example output
• 642c0d7 chore: Update ruff and uv
• 2a5ecb3 Replace instance_id by individual facets
• e2dcdfb docs: Changelog
• 85d4485 chore: add dirty and successful fields
• 624d1ac feat: Add filtering by facets to the executions list-groups
• 3a5fe94 chore: centralise console management
• 0f06b2b chore: Update regression outputs
• 1742005 chore: Handle more captions
• b4ce2b0 chore: Fix tests
• 8f42895 docs: Rest of changelog
• 85a9f37 docs: Changelog
• 86ed15a chore: rerun regression tests
• e08ebf3 chore: More clean ups
• 97a8424 feat: Update to include dimensions for the outputs
• 8627491 feat: Add dimensions to the CMEC format
• 279cdd1 chore: Fix tests
• 268d1d9 test: Add tests
• bb437db docs: Changelog
• 90ba329 chore: the _cv_dimension is required
• d7afee5 feat: Add dimension columns to the figure outputs
• 8959c8d changelog
• ad88acc ensure that volcello is associated with the run if needed
• 0c26d40 changed argument names weren't updated
• aee3064 Log ESMValTool recipe and config
• 522a2ee Add changelog
• 5099b36 Improve conversion from dataframe to recipe
• e91c97d Add changelog
• 37e3e06 Avoid selecting the areacella variable when filling the recipe
• 4f66d39 Re-order ESMValTool constraints
• 83e2fcc Use only apply style filters
• c4fb933 chore: Broad noqa to fix linting issue
• 4f03894 chore: Fix last test
• d5efa9e Rename some remaining metric occurrences to diagnostic
• 74c8ba7 Use logical or to combine multiple facet filters
• 03ac049 chore: don't include table id
• db44814 chore: Remove obs4REF data from the catalog
• 436a465 chore: Bump sample data to exclude the ta data
• 53997c3 chore: Remove the ta data that isn't needed any more.
• 0b56445 chore: Add more constraints
• 1654f0a chore: Update the ocean data
• 8985572 chore: Changelog
• e532605 Revert "Update number of files"
• 6e41089 Revert "Add ERA-5 obs4MIPs data to obs4REF"
• e50b691 docs: Update documentation
• 00f8c59 feat: Make the fetch script more flexible
• 6476797 feat: Make script more flexible
• f9a1b29 chore: Changelog
• 1251d75 fix: Gracefully handle empty directories
• f0dc00a chore: Changelog
• 3297a73 fix: Add PMP version to db
• 46e43f4 fix: Add PMP version to db
• ab406f6 Update example output
• d9f15f3 Improve ESMValTool series output
• 572ce23 docs: Changelog
• f9276a1 test: Add tests for database queries
• f77ef6b chore: Add logging to the ingest command
• 9ef2013 feat: Support adding/updating datasets
• 02db324 fix: no remove_emsembles for obs4mips
• fd40c67 chore: Add pmp lock file
• 46f283f chore: Include a statistic column
• 0922827 chore: Refactor out the series extraction to a separate file
• 385d378 fix: Added index and index_name into the db
• 70c6e0f Improve docstring
• f27f16c Improve docstring
• f065884 Use None as default value and add example to docstring
• 9af3e93 Apply more constraints to ESMValTool diagnostics
• 45722d5 Add groupby operation to require facets constraint
• 90519a4 Add constraint that areacello variables are available
• 1779166 Add changelog
• 5708c5f Specify the table_id for all recipes
• bdc269d Update example output but not pmp/annual-cycle
• 64b7bec Use "version" as the obs4MIPs version facet
• d310461 clean up
• e47dbb6 add change log
• 2073446 version update
• 6b30569 Update ESMValTool diagnostics to use timerange constraint
• 3b9fd47 Add a timerange constraint
• 8d30802 Add changelog
• 85de978 Update reference output
• 5e57968 Update sample data to v0.7.3
• dfef55c Add changelog
• 014401a Add test and rename existing tests
• 34a53ea Improve string representation of ExecutionDatasetCollection
• 99ded7a chore: Upgrade cmip6 dataset definitions
• 93e5e4a chore: Fix links in changelog