Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CFchecker does not find a coordinate variable #37

Open
senesis opened this issue Feb 28, 2018 · 20 comments
Open

CFchecker does not find a coordinate variable #37

senesis opened this issue Feb 28, 2018 · 20 comments

Comments

@senesis
Copy link

senesis commented Feb 28, 2018

I am quite newcomer to CFChecker, so please forgive me for possible irrelevance of this issue

When provided with a file showing this structure (here abridged), CFchecker produces the output reported further below, which seems to be a misbehaviour

netcdf ch4global_Amon_CNRM-CM6-1_amip_r99i1p1f2_gr_197901-217812 {
dimensions:
	axis_nbounds = 2 ;
	str_len = 255 ;
	time = UNLIMITED ; // (1 currently)
variables:
	char sector(str_len) ;
		sector:name = "sector" ;
		sector:standard_name = "region" ;
	float ch4global(time) ;
	        ....
	        ch4global:coordinates = "sector" ;
	....

Output :

Using CF Checker Version 3.0.5
Checking against CF Version CF-1.6
Using Standard Name Table Version 49 (2018-02-13T08:44:33Z)
Using Area Type Table Version 6 (22 February 2017)

ERROR: (2.6.1): This netCDF file does not appear to contain CF Convention data.
ERROR: (5): coordinates attribute referencing non-existent variable

this is also true when the main variable has more (space) dimensions
@RosalynHatcher
Copy link
Contributor

Hmmm. I've tried your snippet above and it works ok for me. Could you send me the header of your netcdf file please (ncdump -h) as I think there must be something elsewhere in the file that is causing this error. I will also improve that error message so that it indicates which variable is the problem.

@senesis
Copy link
Author

senesis commented Feb 28, 2018

Here it is. The file is small, I send it separately.

netcdf ch4global_Amon_CNRM-CM6-1_amip_r99i1p1f2_gr_197901-217812 {
dimensions:
	axis_nbounds = 2 ;
	str_len = 255 ;
	time = UNLIMITED ; // (1 currently)
variables:
	char sector(str_len) ;
		sector:name = "sector" ;
		sector:standard_name = "region" ;
	double time(time) ;
		time:axis = "T" ;
		time:standard_name = "time" ;
		time:long_name = "Time axis" ;
		time:calendar = "gregorian" ;
		time:units = "days since 1850-01-01 00:00:00" ;
		time:time_origin = "1850-01-01 00:00:00" ;
		time:bounds = "time_bounds" ;
	double time_bounds(time, axis_nbounds) ;
	float ch4global(time) ;
		ch4global:online_operation = "average" ;
		ch4global:cell_methods = "area: time: mean" ;
		ch4global:interval_operation = "900 s" ;
		ch4global:interval_write = "1 month" ;
		ch4global:_FillValue = 1.e+20f ;
		ch4global:missing_value = 1.e+20f ;
		ch4global:coordinates = "sector" ;
		ch4global:standard_name = "mole_fraction_of_methane_in_air" ;
		ch4global:description = "Global Mean Mole Fraction of CH4" ;
		ch4global:long_name = "Global Mean Mole Fraction of CH4" ;
		ch4global:history = "none" ;
		ch4global:units = "1e-09" ;

// global attributes:
		:name = "/scratch/utmp/ftdir/senesi/eclis/transfers/toy_atm/iox/ch4global_Amon_CNRM-CM6-1_amip_r99i1p1f2_gr_%start_date%-%end_date%" ;
		:Conventions = "CF-1.7 CMIP-6.2" ;
		:creation_date = "2018-02-28T08:45:37Z" ;
		:tracking_id = "hdl:21.14100/835f2c5e-469d-4fd6-9e3c-4edd6ff4fc20" ;
		:description = "DECK: AMIP" ;
		:title = "CNRM-CM6-1 model output prepared for CMIP6 / CMIP amip" ;
		:activity_id = "CMIP" ;
		:contact = "[email protected]" ;
		:data_specs_version = "01.00.21" ;
		:dr2xml_version = "0.99" ;
		:CMIP6_CV_experiments_latest_tag = "3.3.0 (22; g8906e28)" ;
		:experiment_id = "amip" ;
		:experiment = "AMIP" ;
		:external_variables = "areacella" ;
		:forcing_index = 2 ;
		:frequency = "mon" ;
		:further_info_url = "https://furtherinfo.es-doc.org/CMIP6.CNRM-CERFACS.CNRM-CM6-1.amip.none.r99i1p1f2" ;
		:grid = "data regridded to a T127 gaussian grid (128x256 latlon) from a native atmosphere T127l reduced gaussian grid" ;
		:grid_label = "gr" ;
		:nominal_resolution = "250 km" ;
		:history = "none" ;
		:initialization_index = 1 ;
		:institution_id = "CNRM-CERFACS" ;
		:institution = "CNRM (Centre National de Recherches Meteorologiques, Toulouse 31057, France), CERFACS (Centre Europeen de Recherche et de Formation Avancee en Calcul Scientifique, Toulouse 31057, France)" ;
		:license = "CMIP6 model data produced by CNRM-CERFACS is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License (https://creativecommons.org/licenses). Consult https://pcmdi.llnl.gov/CMIP6/TermsOfUse for terms of use governing CMIP6 output, including citation requirements and proper acknowledgment. Further information about this data, including some limitations, can be found via the further_info_url (recorded as a global attribute in this file) and at http://www.umr-cnrm.fr/cmip6/. The data producers and data providers make no warranty, either express or implied, including, but not limited to, warranties of merchantability and fitness for a particular purpose. All liabilities arising from the supply of the information (including any liability arising in negligence) are excluded to the fullest extent permitted by law." ;
		:mip_era = "CMIP6" ;
		:physics_index = 1 ;
		:product = "model-output" ;
		:realization_index = 99 ;
		:realm = "atmos atmosChem" ;
		:references = "A character string containing a list of published or web-based references that describe the data or the methods used to produce it.Typically, the user should provide references describing the modelformulation here" ;
		:source = "CNRM-CM6-1 (2017):  aerosol: prescribed monthly fields computed by TACTIC_v2 scheme atmos: Arpege 6.3 (T127; Gaussian Reduced with 24572 grid points in total distributed over 128 latitude circles (with 256 grid points per latitude circle between 30degN and 30degS reducing to 20 grid points per latitude circle at 88.9degN and 88.9degS); 91 levels; top level 78.4 km) atmosChem: OZL_v2 land: Surfex 8.0c ocean: Nemo 3.6 (eORCA1, tripolar primarily 1deg; 362 x 294 longitude/latitude; 75 levels; top grid cell 0-1 m) seaIce: Gelato 6.1" ;
		:source_id = "CNRM-CM6-1" ;
		:source_type = "AGCM" ;
		:sub_experiment_id = "none" ;
		:sub_experiment = "none" ;
		:table_id = "Amon" ;
		:variable_id = "ch4global" ;
		:variant_info = ". Information provided by this attribute may in some cases be flawed. Users can find more comprehensive and up-to-date documentation via the further_info_url global attribute." ;
		:variant_label = "r99i1p1f2" ;
		:EXPID = "toy_atm" ;
		:CMIP6_CV_version = "cv=3.2.7-89-gac4b169" ;
		:xios_commit = "1442" ;
		:nemo_gelato_commit = "417012b853ea50e_c28a02af91d965f" ;
		:arpege_minor_version = "6.3.1" ;
}

@RosalynHatcher
Copy link
Contributor

RosalynHatcher commented Feb 28, 2018 •
edited
Loading

The file you sent me works absolutely fine through the head of master. I will leave this issue open for the moment so that once I've got a release candidate available for community testing you can confirm that it is ok for you.

Whilst testing this I did find one non-related bug - where the checker fell over due to not liking the one dimensional character array for sector. This has been fixed.

@senesis
Copy link
Author

senesis commented Feb 28, 2018

Thanks

@RosalynHatcher RosalynHatcher mentioned this issue Feb 28, 2018
Merged
@durack1
Copy link

durack1 commented Feb 4, 2019

@RosalynHatcher we're also getting ERROR (2.6.1): This netCDF file does not appear to contain CF Convention data. is there an obvious fix for this?

@RobertPincus @taylor13

@RosalynHatcher
Copy link
Contributor

@durack1 @RobertPincus @taylor13
Without seeing the file I can't tell you what the problem is or even if it is related to this ticket. Could you either send me or make available the troublesome file please and I'll take a look.

@martinjuckes
Copy link

@RosalynHatcher : I've just come across this. The problem may be that @senesis is using a version of the checker which defaults to CF-1.6 and doesn't support CF-1.7. In any case, the output above says that it is using CF-1.6, and in CF-1.6 the Conventions setting CF-1.7 CMIP-6.2 is invalid.

@RobertPincus
Copy link

Here's an ncdump of the file:
netcdf rld {
dimensions:
expt = 18 ;
site = 100 ;
level = 61 ;
variables:
float lat(site) ;
lat:long_name = "ERA-Interim latitude" ;
lat:units = "degree_north" ;
lat:standard_name = "latitude" ;
float lon(site) ;
lon:long_name = "ERA-Interim longitude" ;
lon:units = "degree_east" ;
lon:standard_name = "longitude" ;
float time(site) ;
time:long_name = "ERA-Interim fractional day of the year 2014" ;
time:units = "days since 2014-1-1 0:0:0" ;
time:standard_name = "time" ;
time:calendar = "gregorian" ;
float plev(site, level) ;
plev:standard_name = "air_pressure" ;
plev:units = "Pa" ;
plev:coordinates = "lon lat" ;
plev:long_name = "Pressure at layer edge" ;
float rld(expt, site, level) ;
rld:variable_id = "rld" ;
rld:standard_name = "downwelling_longwave_flux_in_air" ;
rld:units = "W m-2" ;
rld:coordinates = "lon lat time" ;
float profile_weight(site) ;
profile_weight:title = "profile weight to recover global mean" ;
profile_weight:units = "1" ;
profile_weight:coordinates = "lon lat" ;
profile_weight:long_name = "profile weight to recover global mean" ;

// global attributes:
:activity_id = "RFMIP" ;
:product = "model-output" ;
:experiment_id = "rad-irf" ;
:table_id = "Efx" ;
:frequency = "fx" ;
:sub_experiment_id = "none" ;
:data_specs_version = "01.00.12" ;
:physics_index = 1LL ;
:Conventions = "CF-1.7 CMIP-6.0" ;
:mip_era = "CMIP6" ;
:experiment = "rad_irf" ;
:sub_experiment = "none" ;
:realization_index = 1LL ;
:initialization_index = 1LL ;
:source_type = "RAD" ;
:nominal_resolution = "10 km" ;
:realm = "atmos" ;
:grid_label = "gn" ;
:grid = "columns sampled from ERA-Interim, radiative fluxes computed independently" ;
:institution_id = "AER" ;
:institution = "Research and Climate Group, Atmospheric and Environmental Research, 131 Hartwell Avenue, Lexington, MA 02421, USA" ;
:source_id = "LBLRTM-12-8" ;
:version = "12.8" ;
:source = "LBLRTM 12.8 (2017): \naerosol: none\natmos: none\natmosChem: none\nland: none\nlandIce: none\nocean: none\nocnBgchem: none\nseaIce: none" ;
:further_info_url = "https://furtherinfo.es-doc.org/CMIP6.AER.LBLRTM-12-8.rad-irf.none.r1i1p1f1" ;
:forcing_index = 1LL ;
:license = "CMIP6 model data produced by AER is licensed under a Creative Commons Attribution "Share Alike 4.0" International License (https://creativecommons.org/licenses). Consult https://pcmdi.llnl.gov/CMIP6/TermsOfUse for terms of use governing CMIP6 output, including citation requirements and proper acknowledgment. Further information about this data, including some limitations, can be found via the further_info_url (recorded as a global attribute in this file). The data producers and data providers make no warranty, either express or implied, including, but not limited to, warranties of merchantability and fitness for a particular purpose. All liabilities arising from the supply of the information (including any liability arising in negligence) are excluded to the fullest extent permitted by law." ;
:creation_date = "2019-02-04T23:18:42Z" ;
:tracking_id = "hdl:21.14100/3f590ab5-4d3f-4a42-b2a2-0edc7a3d35b8" ;
:variant_label = "r1i1p1f1" ;
:variable_id = "rld" ;
}

@durack1
Copy link

durack1 commented Feb 5, 2019

@RosalynHatcher I'd note that for the global_attribute :Conventions = "CF-1.7 CMIP-6.0" ; this can have a valid sub-index of CMIP-6.0, CMIP-6.1 or the current revision CMIP-6.2 following the specs doc (findable at https://goo.gl/v1drZl, and current version which is 6.2.7)

@taylor13
Copy link

taylor13 commented Feb 5, 2019

I hadn't seen integer values expressed as 1LL. Is this o.k.? what does "LL" mean in this context?

@RosalynHatcher
Copy link
Contributor

Once I'd removed the LL's Karl mentions above in order to generate the netcdf. (I've not come across this either.) The file is read absolutely fine by cf-checker-3.1.0. The CF Checker doesn't care about the sub-index of anything in the Conventions global attribute other than CF-. Can you please confirm that you were using the latest version of the checker?

@RobertPincus
Copy link

@RosalynHatcher I am using the web interface at http://puma.nerc.ac.uk/cgi-bin/cf-checker.pl. I've just confirmed that the checker doesn't seem to be parsing the Conventions attribute correctly:
WARNING: Cannot determine CF version from the Conventions attribute; checking against latest CF version: CF-1.6
Using CF Checker Version 2.0.9
Checking against CF Version CF-1.6
Using Standard Name Table Version 63 (2019-02-05T13:57:21Z)
Using Area Type Table Version 9 (07 August 2018)

@RobertPincus
Copy link

@taylor13 Might LL refer to long ints? These are set as part of a Python dictionary and written using netcdf4:
expt_attrs = {
"Conventions" :"CF-1.7 CMIP-6.0",
"realization_index" :1,
"initialization_index":1,...

@taylor13
Copy link

taylor13 commented Feb 5, 2019

certainly possible. But @RosalynHatcher thinks that's causing problems with the checker. Rosalyn, does the checker accept long ints?

@RobertPincus
Copy link

RobertPincus commented Feb 5, 2019 via email

@RosalynHatcher
Copy link
Contributor

RosalynHatcher commented Feb 6, 2019 •
edited
Loading

The version you are using is an old version that only goes up to CF-1.6. As Martin said "CF-1.6 CMIP-6.0" was invalid at CF-1.6, hence the failure. Your file contains CF-1.7 data so you should be using the latest checker on our test server: http://pumatest.nerc.ac.uk/cgi-bin/cf-checker.pl

Please try running it through this version. I don't know whether it will handle the LL though, I suspect not. The reason I had to remove them from the CDL you posted was because ncgen was giving me syntax errors because of it and I thus couldn't generate the netCDF file with LL in it.

The current puma.nerc.ac.uk server will be retired in the next couple of months sorry for the confusion, it's taken us much longer than anticipated to move across.

If you still have problems running it through the CF-1.7 version of the checker, can you please create a new issue as this isn't related to the original poster's one. Thanks.

@RosalynHatcher
Copy link
Contributor

Not sure if edits of messages get communicated out, but I just updated my mistake in the above that it isn't a bug rather that "CF-1.7 CMIP-6.0" is not valid at CF-1.6.

@RobertPincus
Copy link

RobertPincus commented Feb 6, 2019 via email

@taylor13
Copy link

taylor13 commented Feb 6, 2019

If the "LL" is still included next to integers, I would try to remove them. It's not good if ncgen doesn't know how to handle them. That would lead me to believe that they are not standard netCDF. (Even if the CF-checker doesn't object.)

@durack1
Copy link

durack1 commented Feb 6, 2019

@RobertPincus what software are you using to write these files? netcdf4-python no?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@durack1 @taylor13 @RosalynHatcher @martinjuckes @senesis @RobertPincus