$Header$ -*-text-*-

Happy Valentine's Day! Love, the netCDF Operators NCO version 4.9.2.

http://nco.sf.net (Homepage, Mailing lists, Help)
http://github.com/nco (Source Code, Issues, Releases, Developers)

What's new?

We have re-licensed NCO source code from GPL3 to 3-Clause BSD
starting with Version 4.9.2. Why? To encourage wider adoption
and support. Without institutional (i.e., federal agency) support,
NCO development would crawl to a standstill. We are acceding
to the preferences of our occasional sponsors (DOE, NASA, NSF)
for permissive OpenSource licenses.

Version 4.9.2 contains many regridding fixes and features: 
Improved adoption has made the NCO weight-generator more robust.
Its signature strengths are speed and ultra-high resolution support.
It can create ~3 km global maps from in under an hour to and from
some (not all) grids including FV physics grids (ne1024pg2), MPAS
grids, and RLL grids. Problems remain, though ncremap also supports
ESMF and TempestRemap so take your pick. Moreover the map checker,
ncks --chk_map, produces improved metrics with fewer false alarms.

Work on NCO 4.9.3 has commenced and will improve NCO weight-generator  
accuracy, reduce vertical interpolation memory use, and supply more
accurate weight generation options for rectangular lat-lon grids.

Work on NCO 5.0.0 has commenced "under the hood". A key leap in that 
release will be support for netCDF4 user-defined types. Printing of
netCDF4 user-defined types ENUM and VLEN is ready now (though
unsupported) with the --udt flag. 5.0.0 will contain the finished
version of that, and include options for invoking mbtempest in place
of tempest. 

Enjoy,
Charlie

NEW FEATURES (full details always in ChangeLog):

A. ncremap now supports vertical interpolation of pure-pressure
   files to hybrid grids when the surface PS is included in the
   pure-pressure data-file rather than the vertical grid-file.
   Moreover, the PS may be time-varying. This feature permits
   full decoupling of PS from the vertical grid-file which allows
   for more elegant manipulation of vertical grid-files.
   ncremap --vrt=vrt.nc ncep.nc hybrid.nc
   http://nco.sf.net/nco.html#vrt

B. ncks --chk_map checks for and reports NaN weights.
   Previously chk_map generally ignored NaNs in the weight array.
   The format of the analysis report has also been improved, and gives
   the locations of extrema, i.e., max/min for weights and errors.
   The checker takes better account of masks and regional grids and
   so now reports fewer false-positive map errors.
   ncks --chk_map map.nc
   ncks -D 2 --chk_map map.nc # Increased verbosity
   http://nco.sf.net/nco.html#chk_map

C. ncremap and ncclimo both now export HDF5_USE_FILE_LOCKING='FALSE'
   unless that environment variable already exists. This proves
   useful at preventing mysterious crashes, especially in batch mode.
   Let us know if this setting causes problems for your workflow.
   http://nco.sf.net/nco.html#HDF5_USE_FILE_LOCKING

D. NCO's weight generator can now read unstructured SCRIP gridfiles
   created by TempestRemap utilities that do not include the
   (supposedly standard) "grid_dims" dimension when converting Exodus
   files to SCRIP. The unstructured dimension size is used instead of
   the contents of the "grid_dims" variable. 
   
E. ncremap --devnull option controls whether internal (and sometimes
   noisy) output from NCO weight-generator is sent to terminal or to
   /dev/null. The default is to send internal messages to /dev/null.
   This only works with the NCO weight-generator (--alg_typ=nco).
   ncremap --devnull=Yes ... # Send internal messages to /dev/null
   ncremap --devnull=No  ... #  Send internal messages to terminal
   http://nco.sf.net/nco.html#devnull
   http://nco.sf.net/nco.html#ncremap

F. ncpdq -U now always unpacks all variables, including packed
   grid variables (like "lat"). Previously it would not unpack 
   variables with specially reserved grid names ("lat", "lon", etc.).
   Thanks to Susan Rennie for reporting this issue.  
   This fixes a bug introduced in NCO 4.7.7. Solution is to use
   earlier versions of NCO or to upgrade.
   http://nco.sf.net/nco.html#ncpdq

BUG FIXES:

A. ncks --chk_map statistics no longer scrambled.
   Statistics of "normal" global->global conservative maps are
   unaffected. Some other types of map-files are.
   There is no workaround. Solution is to upgrade.
   http://nco.sf.net/nco.html#chk_map

B. ncremap --alg_typ=fv2fv now implements the same TempestRemap
   options as --alg_typ=fv2fv_stt. The previous fv2fv options
   were not actually as described. Solution is to upgrade.
   http://nco.sf.net/nco.html#fv2fv
   http://nco.sf.net/nco.html#fv2fv_stt

C. ncremap now passes all NCO options through to ncks.
   Previously these options, such as file formats were not passed
   and ncks used defaults instead.
   There is no workaround. Solution is to upgrade.

Full release statement at http://nco.sf.net/ANNOUNCE

KNOWN PROBLEMS DUE TO NCO:

   This section of ANNOUNCE reports and reminds users of the
   existence and severity of known, not yet fixed, problems. 
   These problems occur with NCO 4.9.2 built/tested under
   MacOS 10.15.2 with netCDF 4.7.3 on HDF5 1.10.2 and with
   Linux with netCDF 4.7.4-development (20200102) on HDF5 1.8.19.

A. NOT YET FIXED (NCO problem)
   Correctly read arrays of NC_STRING with embedded delimiters in ncatted arguments

   Demonstration:
   ncatted -D 5 -O -a new_string_att,att_var,c,sng,"list","of","str,ings" ~/nco/data/in_4.nc ~/foo.nc
   ncks -m -C -v att_var ~/foo.nc

   20130724: Verified problem still exists
   TODO nco1102
   Cause: NCO parsing of ncatted arguments is not sophisticated
   enough to handle arrays of NC_STRINGS with embedded delimiters.

B. NOT YET FIXED (NCO problem?)
   ncra/ncrcat (not ncks) hyperslabbing can fail on variables with multiple record dimensions

   Demonstration:
   ncrcat -O -d time,0 ~/nco/data/mrd.nc ~/foo.nc

   20140826: Verified problem still exists
   20140619: Problem reported by rmla
   Cause: Unsure. Maybe ncra.c loop structure not amenable to MRD?
   Workaround: Convert to fixed dimensions then hyperslab

KNOWN PROBLEMS DUE TO BASE LIBRARIES/PROTOCOLS:

A. NOT YET FIXED (netCDF4 or HDF5 problem?)
   Specifying strided hyperslab on large netCDF4 datasets leads
   to slowdown or failure with recent netCDF versions.

   Demonstration with NCO <= 4.4.5:
   time ncks -O -d time,0,,12 ~/ET_2000-01_2001-12.nc ~/foo.nc
   Demonstration with NCL:
   time ncl < ~/nco/data/ncl.ncl   
   20140718: Problem reported by Parker Norton
   20140826: Verified problem still exists
   20140930: Finish NCO workaround for problem
   20190201: Possibly this problem was fixed in netCDF 4.6.2 by https://github.com/Unidata/netcdf-c/pull/1001
   Cause: Slow algorithm in nc_var_gets()?
   Workaround #1: Use NCO 4.4.6 or later (avoids nc_var_gets())
   Workaround #2: Convert file to netCDF3 first, then use stride
   Workaround #3: Compile NCO with netCDF >= 4.6.2

B. NOT YET FIXED (netCDF4 library bug)
   Simultaneously renaming multiple dimensions in netCDF4 file can corrupt output

   Demonstration:
   ncrename -O -d lev,z -d lat,y -d lon,x ~/nco/data/in_grp.nc ~/foo.nc # Completes but produces unreadable file foo.nc
   ncks -v one ~/foo.nc

   20150922: Confirmed problem reported by Isabelle Dast, reported to Unidata
   20150924: Unidata confirmed problem
   20160212: Verified problem still exists in netCDF library
   20160512: Ditto
   20161028: Verified problem still exists with netCDF 4.4.1
   20170323: Verified problem still exists with netCDF 4.4.2-development
   20170323: https://github.com/Unidata/netcdf-c/issues/381
   20171102: Verified problem still exists with netCDF 4.5.1-development
   20171107: https://github.com/Unidata/netcdf-c/issues/597
   20190202: Progress has recently been made in netCDF 4.6.3-development
   More details: http://nco.sf.net/nco.html#ncrename_crd

C. NOT YET FIXED (would require DAP protocol change?)
   Unable to retrieve contents of variables including period '.' in name
   Periods are legal characters in netCDF variable names.
   Metadata are returned successfully, data are not.
   DAP non-transparency: Works locally, fails through DAP server.

   Demonstration:
   ncks -O -C -D 3 -v var_nm.dot -p http://thredds-test.ucar.edu/thredds/dodsC/testdods in.nc # Fails to find variable

   20130724: Verified problem still exists. 
   Stopped testing because inclusion of var_nm.dot broke all test scripts.
   NB: Hard to fix since DAP interprets '.' as structure delimiter in HTTP query string.

   Bug tracking: https://www.unidata.ucar.edu/jira/browse/NCF-47

D. NOT YET FIXED (would require DAP protocol change)
   Correctly read scalar characters over DAP.
   DAP non-transparency: Works locally, fails through DAP server.
   Problem, IMHO, is with DAP definition/protocol

   Demonstration:
   ncks -O -D 1 -H -C -m --md5_dgs -v md5_a -p http://thredds-test.ucar.edu/thredds/dodsC/testdods in.nc

   20120801: Verified problem still exists
   Bug report not filed
   Cause: DAP translates scalar characters into 64-element (this
   dimension is user-configurable, but still...), NUL-terminated
   strings so MD5 agreement fails 

"Sticky" reminders:

A. Reminder that NCO works on most HDF4 and HDF5 datasets, e.g., 
   HDF4: AMSR MERRA MODIS ...
   HDF5: GLAS ICESat Mabel SBUV ...
   HDF-EOS5: AURA HIRDLS OMI ...

B. Pre-built executables for many OS's at:
   http://nco.sf.net#bnr

