$Header$ -*-text-*-

The netCDF Operators NCO version 4.7.7 have hatched.

http://nco.sf.net (Homepage, Mailing lists, Help)
http://github.com/nco (Source Code, Issues, Releases, Developers)

What's new? Version 4.7.7 has modest new features everywhere:
Improved forward and backward compatibility with newer and older
versions of ESMF_RegridWeightGen; Generation and inferral of grids
running north->south (à la ECMWF); exact symmetry for Gaussian grid
interfaces; an ncpdq compression map that converts doubles to floats;
production of "skeleton files" for new grids; equivalent treatment of
MPAS and CESM grid variables in arithmetic; and reversion of the
filename whitelist security. 

Work on NCO 5.0.0 has commenced "under the hood". The key leap in that 
release will be support for netCDF4 user-defined types. Printing of
netCDF4 user-defined types ENUM and VLEN is ready now (though
unsupported) with the --udt flag. 5.0.0 will contain the finished
version of that, and progress on native weight generation by ncremap.

Enjoy,
Charlie

NEW FEATURES (full details always in ChangeLog):

A. ncpdq has a new packing map: dbl_flt
   The map converts to double-precision to single-precision,
   so it is technically a precision change, not packing.
   It is in ncpdq because it reduces space by ~50%, like packing.  
   Like packing the precision loss is irreversible.
   Unlike packing, no attributes are created, modified, or deleted.
   ncpdq -M dbl_flt in.nc out.nc
   http://nco.sf.net/nco.html#dbl_flt
   http://nco.sf.net/nco.html#ncpdq

B. Filename sanitization via character whitelists, first introduced in
   NCO 4.7.3 to increase security against malicious filenames, has
   been turned-off, possibly permanently. Points passionately raised
   by a user led us to turn-off this feature until lack of it is shown
   to lead to a real-world exploit.
   https://github.com/nco/nco/issues/104
   http://nco.sf.net/nco.html#wht_lst

C. The CCM/CAM/EAM family of atmospheric models does not output a
   "bounds" variable or attribute corresponding to its "lev"
   coordinate. This prevents NCO from activating some desirable CF
   machinery. To workaround this, NCO now outputs the ilev coordinate
   (and hyai, hybi) whenever the lev coordinate is also output.
   ncks -v T in.nc out.nc
   http://nco.sf.net/nco.html#bnd   
   
D. Equivalent treatment of MPAS and CESM grid variables.
   NCO does not arithmetically alter grid-related variables.
   Who wants to subtract latitude coordinates from eachother?
   Previously it only skipped arithmetic on CESM'ish grid variables.
   Now NCO has an extensive (pun intended!) list of MPAS and other
   grid variables that it will not subtract. For the most part,
   ncremap will not regrid these variables either.
   http://nco.sf.net/nco.html#cnv_CCSM
   http://nco.sf.net/nco.html#cnv_MPAS
   http://nco.sf.net/nco.html#ncremap

E. I greatly expanded the field manual to regridding with ncremap.
   The documentation is currently available to E3SM users at
   https://acme-climate.atlassian.net/wiki/spaces/SIM/pages/754286611/Regridding+E3SM+Data+with+ncremap
   and to others via email request (it will be in Users Guide soon).

F. ncremap adds --ignore_degenerate to default ESMF_RegridWeightGen
   (ERWG) options when ERWG version >= 7.0. This is done to preserve
   backwards compatibility since ERWG 7.1.0r and later require
   --ignore_degenerate to successfully regrid some datasets (e.g.,
   CICE) that previous ERWG versions handle fine.  
   ncremap -m map.nc in.nc out.nc
   http://nco.sf.net/nco.html#wgt_opt

G. Not only do the British drive on the wrong side of the road, they
   prefer North-to-South (n2s) grids over South-to-North (s2n) grids. 
   NCO now implements a latitude direction keyword lat_drc.
   Grid generation implements n2s ordering via --rgr lat_drc=n2s.
   Grid diagnosis now recognizes n2s ordering for all rectangular grids.
   http://nco.sf.net/nco.html#lat_drc

H. ncremap now generates exactly symmetric interface latitudes for
   Gaussian grids. To our knowledge only NCO generates accurate
   Gaussian interfaces (via a Newton-Raphson iteration of boundaries 
   to enclose the area indicated by the Gaussian weight). However,
   previously the interface latitudes could accumulate rounding errors
   during Newton-Raphson iteration differ between the hemispheres by
   ~1.0e-14. Now symmetry properties are used to ensure exact symmetry. 
   http://nco.sf.net/nco.html#gss

I. ncremap now produces a "skeleton file" of grids it generates
   when requested with --skl_fl.
   ncremap -G latlon=2560,5136#lat_typ=gss#lon_typ=grn_ctr#lat_drc=n2s -g grd_ecmwf.nc --skl=skl_ecmwf.nc
   http://nco.sf.net/nco.html#skl_fl
   http://nco.sf.net/nco.html#skeleton

BUG FIXES:

A. ncks -r could incorrectly report that CDF5 was enabled.
   This occurred when linked to netCDF 4.5.x-4.6.1.
   This has been fixed.

B. ncremap previously broke on weight files that lacked a num_wgts
   dimension, even if that dimension was unused (orphaned). 
   Dependence on num_wgts has been eliminated and fixes this.
   This also enables compressed maps to work correctly.

C. Fixed a bug in the convergence test to detect Gaussian grids

Full release statement at http://nco.sf.net/ANNOUNCE

KNOWN PROBLEMS DUE TO NCO:

   This section of ANNOUNCE reports and reminds users of the
   existence and severity of known, not yet fixed, problems. 
   These problems occur with NCO 4.7.7 built/tested under
   MacOS 10.13.6 with netCDF 4.6.1 on HDF5 1.10.2 and with
   Linux with netCDF 4.6.2-development (20180515) on HDF5 1.8.19.

A. NOT YET FIXED (NCO problem)
   Correctly read arrays of NC_STRING with embedded delimiters in ncatted arguments

   Demonstration:
   ncatted -D 5 -O -a new_string_att,att_var,c,sng,"list","of","str,ings" ~/nco/data/in_4.nc ~/foo.nc
   ncks -m -C -v att_var ~/foo.nc

   20130724: Verified problem still exists
   TODO nco1102
   Cause: NCO parsing of ncatted arguments is not sophisticated
   enough to handle arrays of NC_STRINGS with embedded delimiters.

B. NOT YET FIXED (NCO problem?)
   ncra/ncrcat (not ncks) hyperslabbing can fail on variables with multiple record dimensions

   Demonstration:
   ncrcat -O -d time,0 ~/nco/data/mrd.nc ~/foo.nc

   20140826: Verified problem still exists
   20140619: Problem reported by rmla
   Cause: Unsure. Maybe ncra.c loop structure not amenable to MRD?
   Workaround: Convert to fixed dimensions then hyperslab

KNOWN PROBLEMS DUE TO BASE LIBRARIES/PROTOCOLS:

A. NOT YET FIXED (netCDF4 or HDF5 problem?)
   Specifying strided hyperslab on large netCDF4 datasets leads
   to slowdown or failure with recent netCDF versions.

   Demonstration with NCO <= 4.4.5:
   time ncks -O -d time,0,,12 ~/ET_2000-01_2001-12.nc ~/foo.nc
   Demonstration with NCL:
   time ncl < ~/nco/data/ncl.ncl   
   20140718: Problem reported by Parker Norton
   20140826: Verified problem still exists
   20140930: Finish NCO workaround for problem
   Cause: Slow algorithm in nc_var_gets()?
   Workaround #1: Use NCO 4.4.6 or later (avoids nc_var_gets())
   Workaround #2: Convert file to netCDF3 first, then use stride

B. NOT YET FIXED (netCDF4 library bug)
   Simultaneously renaming multiple dimensions in netCDF4 file can corrupt output

   Demonstration:
   ncrename -O -d lev,z -d lat,y -d lon,x ~/nco/data/in_grp.nc ~/foo.nc # Completes but file is unreadable
   ncks -v one ~/foo.nc

   20150922: Confirmed problem reported by Isabelle Dast, reported to Unidata
   20150924: Unidata confirmed problem
   20160212: Verified problem still exists in netCDF library
   20160512: Ditto
   20161028: Verified problem still exists with netCDF 4.4.1
   20170323: Verified problem still exists with netCDF 4.4.2-development
   20170323: https://github.com/Unidata/netcdf-c/issues/381
   20171102: Verified problem still exists with netCDF 4.5.1-development
   20171107: https://github.com/Unidata/netcdf-c/issues/597
   Bug tracking: https://www.unidata.ucar.edu/jira/browse/fxm
   More details: http://nco.sf.net/nco.html#ncrename_crd

C. NOT YET FIXED (would require DAP protocol change?)
   Unable to retrieve contents of variables including period '.' in name
   Periods are legal characters in netCDF variable names.
   Metadata are returned successfully, data are not.
   DAP non-transparency: Works locally, fails through DAP server.

   Demonstration:
   ncks -O -C -D 3 -v var_nm.dot -p http://thredds-test.ucar.edu/thredds/dodsC/testdods in.nc # Fails to find variable

   20130724: Verified problem still exists. 
   Stopped testing because inclusion of var_nm.dot broke all test scripts.
   NB: Hard to fix since DAP interprets '.' as structure delimiter in HTTP query string.

   Bug tracking: https://www.unidata.ucar.edu/jira/browse/NCF-47

D. NOT YET FIXED (would require DAP protocol change)
   Correctly read scalar characters over DAP.
   DAP non-transparency: Works locally, fails through DAP server.
   Problem, IMHO, is with DAP definition/protocol

   Demonstration:
   ncks -O -D 1 -H -C -m --md5_dgs -v md5_a -p http://thredds-test.ucar.edu/thredds/dodsC/testdods in.nc

   20120801: Verified problem still exists
   Bug report not filed
   Cause: DAP translates scalar characters into 64-element (this
   dimension is user-configurable, but still...), NUL-terminated
   strings so MD5 agreement fails 

"Sticky" reminders:

A. Reminder that NCO works on most HDF4 and HDF5 datasets, e.g., 
   HDF4: AMSR MERRA MODIS ...
   HDF5: GLAS ICESat Mabel SBUV ...
   HDF-EOS5: AURA HIRDLS OMI ...

B. Pre-built executables for many OS's at:
   http://nco.sf.net#bnr

