Skip to content

Enhanced support in weather-dl for downloading data across month ranges spanning multiple years. #372

@dabhicusp

Description

@dabhicusp

I have a suggestion for an enhancement to the weather-dl tool. Currently, when downloading data for a month range that spans multiple years, the process requires manual splitting of the month range into separate configurations and executing multiple jobs for each part. This process can be tedious and time-consuming.

For instance, if we wish to download data from April 1994 to August 2012, and the target path follows this format: (gs://xyz/{year}/{year}{month:02d}_hres_var.grb2), we need to split the dates into three parts as follows:

1st part :
year=1994
month=04/to/12
day=all

For the 2nd part:
year=1995/to/2011
month=01/to/12
day=all

And for the last part:
year=2012
month=01/to/08
day=all

I propose an enhancement in the weather-dl parser by introducing a new keyword, such as month_year, in the configuration.
The value of this keyword would look like 08-2012 or 01-2022. Additionally, we can add a script to the parser of weather-dl that can correctly interpret this keyword.

P.S.: A similar use-case can be found in the Arco-era5 dataset. Please refer to any file containing sfc in the filename for more context.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions