ERA5 Download error - only for latest date range, as of new year

[bnubald]

• IceNet version: 0.2.x (0.2.9_dev)

Description

Attempting to download latest ERA5 data results in error.

This doesn't occur for different date ranges, only the latest one (i.e, 2025-01-01 to 2025-01-06). i.e., up to today.

What I Did

$ icenet_data_era5 -w 10 -v --vars tas --levels '' south 2025-1-1 2025-1-6

[06-01-25 02:44:15 :INFO    ] - ERA5 Data Downloading
[06-01-25 02:44:15 :DEBUG   ] - ./data/era5 already exists
[06-01-25 02:44:15 :DEBUG   ] - ./data/masks already exists
2025-01-06 02:44:21,148 INFO [2024-09-28T00:00:00] **Welcome to the New Climate Data Store (CDS)!** This new system is in its early days of full operations and still undergoing enhancements and fine tuning. Some disruptions are to be expected. Your 
[feedback](https://jira.ecmwf.int/plugins/servlet/desk/portal/1/create/202) is key to improve the user experience on the new CDS for the benefit of everyone. Thank you.
[06-01-25 02:44:21 :INFO    ] - [2024-09-28T00:00:00] **Welcome to the New Climate Data Store (CDS)!** This new system is in its early days of full operations and still undergoing enhancements and fine tuning. Some disruptions are to be expected. Your 
[feedback](https://jira.ecmwf.int/plugins/servlet/desk/portal/1/create/202) is key to improve the user experience on the new CDS for the benefit of everyone. Thank you.
2025-01-06 02:44:21,149 INFO [2024-09-26T00:00:00] Watch our [Forum](https://forum.ecmwf.int/) for Announcements, news and other discussed topics.
[06-01-25 02:44:21 :INFO    ] - [2024-09-26T00:00:00] Watch our [Forum](https://forum.ecmwf.int/) for Announcements, news and other discussed topics.
2025-01-06 02:44:21,149 INFO [2024-09-16T00:00:00] Remember that you need to have an ECMWF account to use the new CDS. **Your old CDS credentials will not work in new CDS!**
[06-01-25 02:44:21 :INFO    ] - [2024-09-16T00:00:00] Remember that you need to have an ECMWF account to use the new CDS. **Your old CDS credentials will not work in new CDS!**
2025-01-06 02:44:21,150 WARNING [2024-06-16T00:00:00] CDS API syntax is changed and some keys or parameter names may have also changed. To avoid requests failing, please use the "Show API request code" tool on the dataset Download Form to check you are using the correct syntax for your API request.
[06-01-25 02:44:21 :WARNING ] - [2024-06-16T00:00:00] CDS API syntax is changed and some keys or parameter names may have also changed. To avoid requests failing, please use the "Show API request code" tool on the dataset Download Form to check you are using the correct syntax for your API request.
[06-01-25 02:44:21 :INFO    ] - Building request(s), downloading and daily averaging from ERA5 API
[06-01-25 02:44:21 :INFO    ] - Processing single download for tas @ None with 6 dates
[06-01-25 02:44:21 :DEBUG   ] - Got year filenames: ./data/era5/south/tas/latlon_2025.nc and ./data/era5/south/tas/2025.nc
[06-01-25 02:44:21 :DEBUG   ] - Excluding 0 dates already existing from 6 dates requested.
[06-01-25 02:44:21 :DEBUG   ] - Processing 6 dates
[06-01-25 02:44:21 :INFO    ] - Downloading data for tas...
2025-01-06 02:44:21,823 WARNING [2024-10-10T00:00:00] From 1 July to 17 November 2024, the final ERA5 product is different to ERA5T due to the correction of [the assimilation of incorrect snow observations on the Alps](https://confluence.ecmwf.int/x/USuXGw)
[06-01-25 02:44:21 :WARNING ] - [2024-10-10T00:00:00] From 1 July to 17 November 2024, the final ERA5 product is different to ERA5T due to the correction of [the assimilation of incorrect snow observations on the Alps](https://confluence.ecmwf.int/x/USuXGw)
2025-01-06 02:44:21,824 INFO Request ID is 0ad4cf85-4656-49db-b246-178377694867
[06-01-25 02:44:21 :INFO    ] - Request ID is 0ad4cf85-4656-49db-b246-178377694867
2025-01-06 02:44:21,887 INFO status has been updated to accepted
[06-01-25 02:44:21 :INFO    ] - status has been updated to accepted
2025-01-06 02:44:29,309 INFO status has been updated to running
[06-01-25 02:44:29 :INFO    ] - status has been updated to running
2025-01-06 02:44:42,156 INFO status has been updated to successful
[06-01-25 02:44:42 :INFO    ] - status has been updated to successful
[06-01-25 02:44:42 :INFO    ] - Downloading https://object-store.os-api.cci2.ecmwf.int:443/cci2-prod-cache/546d0c7fbb8d54389955f57ab901be31.nc
[06-01-25 02:44:42 :DEBUG   ] - HTTP headers {
    "accept-ranges": "bytes",
    "content-length": "2180908",
    "content-type": "application/netcdf",
    "date": "Mon, 06 Jan 2025 02:44:42 GMT",
    "etag": "\"5250ac56a84b1121b7ab3f822b0efec5\"",
    "last-modified": "Mon, 06 Jan 2025 02:44:35 GMT",
    "x-amz-request-id": "tx000007a20ae158689183a-00677b439a-10cdad10-default",
    "x-rgw-object-type": "Normal"
}
[06-01-25 02:44:42 :DEBUG   ] - url estimate_size size=2180908 mode=wb skip=0 trust_size=True
[06-01-25 02:44:42 :DEBUG   ] - Issue request for https://object-store.os-api.cci2.ecmwf.int:443/cci2-prod-cache/546d0c7fbb8d54389955f57ab901be31.nc
[06-01-25 02:44:42 :DEBUG   ] - Headers: {}
[06-01-25 02:44:42 :INFO    ] - Download completed: /tmp/tmp0d89545c/latlon_2025.nc.download
[06-01-25 02:44:42 :INFO    ] - Downloaded to ./data/era5/south/tas/latlon_2025.nc
[06-01-25 02:44:42 :INFO    ] - Postprocessing CDS API data at ./data/era5/south/tas/latlon_2025.nc
[06-01-25 02:44:42 :DEBUG   ] - Moving to ./data/era5/south/tas/latlon_2025.bak.nc
[06-01-25 02:44:43 :DEBUG   ] - Could not import boto3, continuing with reduced functionality.
[06-01-25 02:44:43 :DEBUG   ] - GDAL data found in package: path='/data/hpcdata/users/username/miniconda3/envs/icenet0.2.9_dev/lib/python3.11/site-packages/rasterio/gdal_data'.
[06-01-25 02:44:43 :ERROR   ] - Thread failure: The indexing operation you are attempting to perform is not valid on netCDF4.Variable object. Try loading your data into memory first by calling .load().
Traceback (most recent call last):
  File "/data/hpcdata/users/username/miniconda3/envs/icenet0.2.9_dev/lib/python3.11/site-packages/xarray/backends/netCDF4_.py", line 104, in _getitem
    array = getitem(original_array, key)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "src/netCDF4/_netCDF4.pyx", line 4716, in netCDF4._netCDF4.Variable.__getitem__
  File "/data/hpcdata/users/username/miniconda3/envs/icenet0.2.9_dev/lib/python3.11/site-packages/netCDF4/utils.py", line 461, in _out_array_shape
    c = count[..., i].ravel()[0] # All elements should be identical.
        ~~~~~~~~~~~~~~~~~~~~~^^^
IndexError: index 0 is out of bounds for axis 0 with size 0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/hpcdata/users/username/git/icenet/icenet/icenet/data/interfaces/downloader.py", line 226, in download
    future.result()
  File "/data/hpcdata/users/username/miniconda3/envs/icenet0.2.9_dev/lib/python3.11/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/data/hpcdata/users/username/miniconda3/envs/icenet0.2.9_dev/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/data/hpcdata/users/username/miniconda3/envs/icenet0.2.9_dev/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/hpcdata/users/username/git/icenet/icenet/icenet/data/interfaces/downloader.py", line 302, in _single_download
    self.postprocess(var, latlon_path)
  File "/data/hpcdata/users/username/git/icenet/icenet/icenet/data/interfaces/cds.py", line 207, in postprocess
    da = da.where(da.time < pd.Timestamp(strip_dates_before), drop=True)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/hpcdata/users/username/miniconda3/envs/icenet0.2.9_dev/lib/python3.11/site-packages/xarray/core/common.py", line 1134, in where
    return ops.where_method(self, cond, other)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/hpcdata/users/username/miniconda3/envs/icenet0.2.9_dev/lib/python3.11/site-packages/xarray/core/ops.py", line 177, in where_method
    return apply_ufunc(
           ^^^^^^^^^^^^
  File "/data/hpcdata/users/username/miniconda3/envs/icenet0.2.9_dev/lib/python3.11/site-packages/xarray/core/computation.py", line 1208, in apply_ufunc
    return apply_dataarray_vfunc(
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/data/hpcdata/users/username/miniconda3/envs/icenet0.2.9_dev/lib/python3.11/site-packages/xarray/core/computation.py", line 315, in apply_dataarray_vfunc
    result_var = func(*data_vars)
                 ^^^^^^^^^^^^^^^^
  File "/data/hpcdata/users/username/miniconda3/envs/icenet0.2.9_dev/lib/python3.11/site-packages/xarray/core/computation.py", line 683, in apply_variable_ufunc
    input_data = [
                 ^
  File "/data/hpcdata/users/username/miniconda3/envs/icenet0.2.9_dev/lib/python3.11/site-packages/xarray/core/computation.py", line 684, in <listcomp>
    broadcast_compat_data(arg, broadcast_dims, core_dims)
  File "/data/hpcdata/users/username/miniconda3/envs/icenet0.2.9_dev/lib/python3.11/site-packages/xarray/core/computation.py", line 599, in broadcast_compat_data
    data = variable.data
           ^^^^^^^^^^^^^
  File "/data/hpcdata/users/username/miniconda3/envs/icenet0.2.9_dev/lib/python3.11/site-packages/xarray/core/variable.py", line 434, in data
    return self.values
           ^^^^^^^^^^^
  File "/data/hpcdata/users/username/miniconda3/envs/icenet0.2.9_dev/lib/python3.11/site-packages/xarray/core/variable.py", line 607, in values
    return _as_array_or_item(self._data)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/hpcdata/users/username/miniconda3/envs/icenet0.2.9_dev/lib/python3.11/site-packages/xarray/core/variable.py", line 313, in _as_array_or_item
    data = np.asarray(data)
           ^^^^^^^^^^^^^^^^
  File "/data/hpcdata/users/username/miniconda3/envs/icenet0.2.9_dev/lib/python3.11/site-packages/xarray/core/indexing.py", line 658, in __array__
    self._ensure_cached()
  File "/data/hpcdata/users/username/miniconda3/envs/icenet0.2.9_dev/lib/python3.11/site-packages/xarray/core/indexing.py", line 655, in _ensure_cached
    self.array = NumpyIndexingAdapter(np.asarray(self.array))
                                      ^^^^^^^^^^^^^^^^^^^^^^
  File "/data/hpcdata/users/username/miniconda3/envs/icenet0.2.9_dev/lib/python3.11/site-packages/xarray/core/indexing.py", line 628, in __array__
    return np.asarray(self.array, dtype=dtype)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/hpcdata/users/username/miniconda3/envs/icenet0.2.9_dev/lib/python3.11/site-packages/xarray/core/indexing.py", line 529, in __array__
    return np.asarray(array[self.key], dtype=None)
                      ~~~~~^^^^^^^^^^
  File "/data/hpcdata/users/username/miniconda3/envs/icenet0.2.9_dev/lib/python3.11/site-packages/xarray/backends/netCDF4_.py", line 91, in __getitem__
    return indexing.explicit_indexing_adapter(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/hpcdata/users/username/miniconda3/envs/icenet0.2.9_dev/lib/python3.11/site-packages/xarray/core/indexing.py", line 820, in explicit_indexing_adapter
    result = raw_indexing_method(raw_key.tuple)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/hpcdata/users/username/miniconda3/envs/icenet0.2.9_dev/lib/python3.11/site-packages/xarray/backends/netCDF4_.py", line 114, in _getitem
    raise IndexError(msg)
IndexError: The indexing operation you are attempting to perform is not valid on netCDF4.Variable object. Try loading your data into memory first by calling .load().
[06-01-25 02:44:43 :INFO    ] - 0 daily files downloaded
[06-01-25 02:44:43 :INFO    ] - No regrid batches to processing, moving on...
[06-01-25 02:44:43 :INFO    ] - Rotating wind data prior to merging
/data/hpcdata/users/username/miniconda3/envs/icenet0.2.9_dev/lib/python3.11/site-packages/iris/__init__.py:354: FutureWarning: Ignoring a datum in netCDF load for consistency with existing behaviour. In a future version of Iris, this datum will be applied. To apply the datum when loading, use the iris.FUTURE.datum_support flag.
  cubes = _load_collection(uris, constraints, callback).cubes()
[06-01-25 02:44:43 :INFO    ] - Rotating wind data in ./data/era5/south/uas ./data/era5/south/vas
[06-01-25 02:44:43 :INFO    ] - 0 files for uas
[06-01-25 02:44:43 :INFO    ] - 0 files for vas