From 70111d0995d224065e3f108c3a27b5db2ae78bc9 Mon Sep 17 00:00:00 2001 From: Dennis Heimbigner Date: Sun, 29 Sep 2024 20:25:50 -0600 Subject: [PATCH] Extend the netcdf API to support programmatic changes to the plugin search path re: https://github.com/Unidata/netcdf-c/issues/2753 As suggested by Ed Hartnett, This PR extends the netcdf.h API to support programmatic control over the search path used to locate plugins. I created several different APIs, but finally settled on the following API as being the simplest possible. It does have the disadvantage that it requires use of a global lock (not implemented) if used in a threaded environment. Specifically, note that modifying the plugin path must be done "atomically". That is, in a multi-threaded environment, it is important that the sequence of actions involved in setting up the plugin path must be done by a single processor or in some other way as to guarantee that two or more processors are not simultaneously accessing the plugin path get/set operations. As an example, assume there exists a mutex lock called PLUGINLOCK. Then any processor accessing the plugin paths should operate as follows: ```` lock(PLUGINLOCK); nc_plugin_path_get(...); nc_plugin_path_set(...); unlock(PLUGINLOCK); ```` ## Internal Architecture It is assumed here that there only needs to be a single set of plugin path directories that is shared by all filter code and is independent of any file descriptor; it is global in other words. This means, for example, that the path list for NCZarr and for HDF5 will always be the same. However internally, processing the set of plugin paths depends on the particular NC_FORMATX value (NC_FORMATX_NC_HDF5 and NC_FORMATX_NCZARR, currently). So the *nc_plugin_path_set* function, will take the paths it is given and propagate them to each of the NC_FORMATX dispatchers to store in a way that is appropriate to the given dispatcher. There is a complication with respect to the *nc_plugin_path_get* function. It is possible for users to bypass the netcdf API and modify the HDF5 plugin paths directly. This can result in an inconsistent plugin path between the value used by HDF5 and the global value used by netcdf-c. Since there is no obvious fix for this, we warn the user of this possibility and otherwise ignore it. ## Test Changes * A new test -- unit_test/run_pluginpaths.sh -- was created to test this new capability. * A new test utility has been added as *nczarr_test/ncpluginpath* to return information about the default plugin path list. ## Documentation * A new file -- docs/pluginpath.md -- provides documentation of the new API. ## Misc. Changes 1. Add some path manipulation utilities to netcf_aux.h 2. Fix the construction of netcdf_json.h as a BUILT_SOURCE 3. Fix some minor bugs in netcdf_json.h 4. Convert netcdf_json.h and netcdf_proplist.h to BUILT_SOURCE. 5. Add NETCDF_ENABLE_HDF5 as synonym for USE_HDF5 6. Fix some size_t <-> int conversion warnings. 7. Encountered and fixed the Windows \r\n problem in tst_pluginpaths.c. 8. Cleanup some minor CMakeLists.txt problems. ## Addendum: Proposed API The API makes use of a counted vector of strings representing the sequence of directories in the path. The relevant type definition is as follows. ```` typedef struct NCPluginList {size_t ndirs; char** dirs;} NCPluginList; ```` The API proposed in this PR looks like this (from netcdf-c/include/netcdf_filter.h). ````int nc_plugin_path_ndirs(size_t* ndirsp);```` This function returns the number of directories in the sequence if internal directories of the internal plugin path list. The argument is as follows: - *ndirsp* store the number of directories in this memory. * ````int nc_plugin_path_get(NCPluginList* dirs);```` This function returns the current sequence of directories from the internal plugin path list. Since this function does not modify the plugin path, it does not need to be locked; it is only when used to get the path to be modified that locking is required. The argument is as follows: - *dirs* counted vector for storing the sequence of directies in the internal path list. If the value of *dirs.dirs is NULL (the normal case), then memory is allocated to hold the vector of directories. Otherwise, use the memory of *dirs.dirs* to hold the vector of directories. * ````int nc_plugin_path_set(const NCPluginList* dirs);```` This function empties the current internal path sequence and replaces it with the sequence of directories argument. Using an *ndirs* argument of 0 will clear the set of plugin paths. The argument are as follows: - *dirs* counted vector for storing the sequence of directies in the internal path list. --- .github/workflows/main-cmake.yml | 2 +- CMakeLists.txt | 52 +- cmake/dependencies.cmake | 8 +- cmake/netcdf_functions_macros.cmake | 14 +- config.h.cmake.in | 3 + configure.ac | 86 +- docs/Doxyfile.in | 1 + docs/Makefile.am | 2 +- docs/filters.md | 99 +- docs/footer.html | 2 +- docs/header.html | 44 +- docs/indexing.dox | 2 +- docs/pluginpath.md | 186 ++++ docs/quickstart_env.md | 6 +- docs/quickstart_paths.md | 12 +- h5_test/Makefile.am | 3 +- include/CMakeLists.txt | 29 +- include/Makefile.am | 35 +- include/nc4internal.h | 9 + include/ncdispatch.h | 2 +- include/ncjson.h | 37 +- include/ncpathmgr.h | 3 + include/ncplugins.h | 39 + include/ncproplist.h | 108 +++ include/netcdf_aux.h | 97 +- include/netcdf_filter.h | 63 +- include/netcdf_json.h | 1310 --------------------------- libdispatch/CMakeLists.txt | 3 +- libdispatch/Makefile.am | 5 +- libdispatch/daux.c | 209 ++++- libdispatch/dcopy.c | 11 +- libdispatch/ddispatch.c | 3 +- libdispatch/dfile.c | 8 + libdispatch/dinfermodel.c | 5 +- libdispatch/dinstance.c | 5 +- libdispatch/dnotnc4.c | 2 + libdispatch/dpathmgr.c | 20 + libdispatch/dplugins.c | 341 +++++++ libdispatch/ds3util.c | 67 +- libdispatch/ncbytes.c | 2 +- libdispatch/ncjson.c | 32 +- libdispatch/nclist.c | 61 +- libdispatch/ncproplist.c | 350 +++++++ libdispatch/ncs3sdk_h5.c | 9 +- libdispatch/ncuri.c | 2 +- libhdf5/CMakeLists.txt | 2 +- libhdf5/Makefile.am | 2 +- libhdf5/hdf5dispatch.c | 1 + libhdf5/hdf5internal.c | 4 +- libhdf5/hdf5plugins.c | 245 +++++ libnczarr/CMakeLists.txt | 6 +- libnczarr/Makefile.am | 3 +- libnczarr/zdispatch.c | 2 +- libnczarr/zfile.c | 2 +- libnczarr/zfilter.c | 731 +-------------- libnczarr/zfilter.h | 6 - libnczarr/zinternal.c | 12 +- libnczarr/zmap.c | 6 +- libnczarr/zplugins.c | 772 ++++++++++++++++ libnczarr/zplugins.h | 45 + libsrc4/nc4dispatch.c | 12 +- nc_test/tst_atts3.c | 2 +- nc_test4/CMakeLists.txt | 2 + nc_test4/Makefile.am | 17 +- nc_test4/ref_fixedstring.cdl | 13 +- nc_test4/ref_fixedstring.h5 | Bin 2057 -> 0 bytes nc_test4/tst_fixedstring.sh | 7 +- ncdump/CMakeLists.txt | 8 +- ncdump/Makefile.am | 4 + ncdump/echon.c | 99 ++ nczarr_test/CMakeLists.txt | 6 +- nczarr_test/Makefile.am | 2 + nczarr_test/ncpluginpath.c | 165 ++++ plugins/CMakeLists.txt | 1 - plugins/H5Zblosc.c | 2 +- plugins/H5Zblosc.h | 2 +- plugins/H5Znoop.c | 3 +- test_common.in | 9 +- unit_test/CMakeLists.txt | 6 + unit_test/Makefile.am | 21 +- unit_test/aws_config.c | 3 +- unit_test/ref_get.txt | 3 + unit_test/ref_set.txt | 6 + unit_test/run_aws_config.sh | 32 +- unit_test/run_pluginpaths.sh | 116 +++ unit_test/tst_pluginpaths.c | 349 +++++++ 86 files changed, 3720 insertions(+), 2398 deletions(-) create mode 100644 docs/pluginpath.md create mode 100644 include/ncplugins.h create mode 100644 include/ncproplist.h delete mode 100644 include/netcdf_json.h create mode 100644 libdispatch/dplugins.c create mode 100644 libdispatch/ncproplist.c create mode 100644 libhdf5/hdf5plugins.c create mode 100644 libnczarr/zplugins.c create mode 100644 libnczarr/zplugins.h delete mode 100644 nc_test4/ref_fixedstring.h5 create mode 100644 ncdump/echon.c create mode 100644 nczarr_test/ncpluginpath.c create mode 100644 unit_test/ref_get.txt create mode 100644 unit_test/ref_set.txt create mode 100755 unit_test/run_pluginpaths.sh create mode 100644 unit_test/tst_pluginpaths.c diff --git a/.github/workflows/main-cmake.yml b/.github/workflows/main-cmake.yml index 93edb15cef..4a783a6819 100644 --- a/.github/workflows/main-cmake.yml +++ b/.github/workflows/main-cmake.yml @@ -1,6 +1,6 @@ name: NetCDF-C CMake CI - Windows -on: [pull_request, workflow_dispatch] +on: [ pull_request, workflow_dispatch] env: REMOTETESTDOWN: ${{ vars.REMOTETESTDOWN }} diff --git a/CMakeLists.txt b/CMakeLists.txt index 8c3e9eeaee..4d12bb6b47 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -54,7 +54,6 @@ if (NOT DEFINED NETCDF_IS_TOP_LEVEL) endif () endif () - ################################ # The target ################################ @@ -72,8 +71,6 @@ endif() add_library(netcdf) add_library(netCDF::netcdf ALIAS netcdf) - - # Version of the dispatch table. This must match the value in # configure.ac. set(NC_DISPATCH_VERSION 5) @@ -363,12 +360,12 @@ endif() set(DEFAULT_CHUNK_SIZE 16777216 CACHE STRING "Default Chunk Cache Size.") set(DEFAULT_CHUNK_CACHE_SIZE 16777216U CACHE STRING "Default Chunk Cache Size.") set(DEFAULT_CHUNKS_IN_CACHE 1000 CACHE STRING "Default number of chunks in cache.") -set(DEFAULT_CHUNK_CACHE_PREEMPTION 0.75 CACHE STRING "Default file chunk cache preemption policy (a number between 0 and 1, inclusive.") +set(DEFAULT_CHUNK_CACHE_PREEMPTION 0.75 CACHE STRING "Default file chunk cache preemption policy (a number between 0 and 1, inclusive).") # HDF5 default cache size values set(CHUNK_CACHE_SIZE ${DEFAULT_CHUNK_CACHE_SIZE} CACHE STRING "Default HDF5 Chunk Cache Size.") set(CHUNK_CACHE_NELEMS ${DEFAULT_CHUNKS_IN_CACHE} CACHE STRING "Default maximum number of elements in cache.") -set(CHUNK_CACHE_PREEMPTION ${DEFAULT_CHUNK_CACHE_PREEMPTION} CACHE STRING "Default file chunk cache preemption policy for HDf5 files(a number between 0 and 1, inclusive.") +set(CHUNK_CACHE_PREEMPTION ${DEFAULT_CHUNK_CACHE_PREEMPTION} CACHE STRING "Default file chunk cache preemption policy for HDf5 files(a number between 0 and 1, inclusive.)") set(NETCDF_LIB_NAME "" CACHE STRING "Default name of the netcdf library.") set(TEMP_LARGE "." CACHE STRING "Where to put large temp files if large file tests are run.") @@ -660,8 +657,29 @@ if(NOT ENABLE_PLUGIN_INSTALL) unset(NETCDF_PLUGIN_INSTALL_DIR CACHE) endif() -if(ENABLE_PLUGIN_INSTALL) - if(NOT DEFINED NETCDF_PLUGIN_INSTALL_DIR) +# Internally, the variable HDF5_PLUGIN_PATH +# is always used as a surrogate for the default +# plugin path. If it was not defined by the user, +# then set it internally. +if((NOT DEFINED HDF5_PLUGIN_PATH) OR ("${HDF5_PLUGIN_PATH}" STREQUAL "")) + if(ISMSVC OR ISMINGW) + string(REPLACE "\\" "/" AUP "$ENV{ALLUSERSPROFILE}") + set(HDF5_PLUGIN_PATH "${AUP}/hdfd5/lib/plugin") + else() + set(HDF5_PLUGIN_PATH "/usr/local/hdf5/lib/plugin") + endif() + set(ENV{HDF5_PLUGIN_PATH} "${HDF5_PLUGIN_PATH}") +endif() + +# The --with-plugin-dir gives the user control of the plugin +# directory. If set to 'yes' (the default), then the last directory in +# HDF5_PLUGIN_PATH is used (see above). +# If a specific directory is provided, it will be used. +# If 'no', then plugins will not be installed. + +if(NETCDF_ENABLE_PLUGIN_INSTALL) + if(DEFINED NETCDF_PLUGIN_INSTALL_DIR) + set(NETCDF_PLUGIN_INSTALL_DIR "${HDF5_PLUGIN_PATH}") # Default to HDF5_PLUGIN_PATH or its default directories if(DEFINED ENV{HDF5_PLUGIN_PATH}) set(NETCDF_PLUGIN_INSTALL_DIR "$ENV{HDF5_PLUGIN_PATH}") @@ -682,15 +700,7 @@ endif(ENABLE_PLUGIN_INSTALL) if(ENABLE_PLUGIN_INSTALL) # Use the lowest priority dir in the path - if(NOT ISMSVC AND NOT ISMINGW) - string(REPLACE ":" ";" PATH_LIST ${NETCDF_PLUGIN_INSTALL_DIR}) - else() - set(PATH_LIST ${NETCDF_PLUGIN_INSTALL_DIR}) - endif() - - # Get last element - list(GET PATH_LIST -1 NETCDF_PLUGIN_INSTALL_DIR) - set(PLUGIN_INSTALL_DIR_SETTING "${NETCDF_PLUGIN_INSTALL_DIR}") + getlastdir("${HDF5_PLUGIN_PATH}" "NETCDF_PLUGIN_INSTALL_DIR") message(STATUS "Final value of-DPLUGIN_INSTALL_DIR=${NETCDF_PLUGIN_INSTALL_DIR}") else() # No option specified unset(NETCDF_PLUGIN_INSTALL_DIR) @@ -753,10 +763,10 @@ if(NOT WIN32) endif() # Options for S3 Support -#option(NETCDF_ENABLE_S3 "Enable S3 support." OFF) option(NETCDF_ENABLE_S3_AWS "Enable S3 support via AWS-CPP-SDK" OFF) option(NETCDF_ENABLE_S3_INTERNAL "Enable S3 Internal support." OFF) +#option(NETCDF_ENABLE_S3 "Enable S3 support." OFF) cmake_dependent_option(NETCDF_ENABLE_S3 "Enable S3 Support" ON "NETCDF_ENABLE_S3_AWS OR NETCDF_ENABLE_S3_INTERNAL" OFF) option(NETCDF_ENABLE_NCZARR_S3 "Enable NCZarr S3 support; Deprecated in favor of NETCDF_ENABLE_S3" ${NETCDF_ENABLE_S3}) @@ -877,7 +887,7 @@ if(NETCDF_ENABLE_TESTS) # See https://github.com/Unidata/netcdf-c/issues/2627 for more information. ### option(NETCDF_ENABLE_BENCHMARKS "Run benchmark Tests." OFF) - + set(BUILD_BENCHMARKS ${NETCDF_ENABLE_BENCHMARKS} CACHE BOOL "alias for NETCDF_ENABLE_BENCHMARKS") ### # End known-failures. @@ -1380,8 +1390,8 @@ endif(NETCDF_ENABLE_MMAP) # Used in the `configure_file` calls below set(ISCMAKE "yes") if(MSVC) - set(ISMSVC ON CACHE BOOL "" FORCE) - set(REGEDIT ON CACHE BOOL "" FORCE) + set(ISMSVC yes CACHE BOOL "" FORCE) + set(REGEDIT yes CACHE BOOL "" FORCE) # Get windows major version and build number execute_process(COMMAND "systeminfo" OUTPUT_VARIABLE WININFO) if(WININFO STREQUAL "") @@ -1564,8 +1574,6 @@ endif() # STATIC_DEFINE netcdf_BUILT_AS_STATIC #) - - ## # Brute force, grab all of the dlls from the dependency directory, # install them in the binary dir. Grab all of the .libs, put them diff --git a/cmake/dependencies.cmake b/cmake/dependencies.cmake index c21ac3b22b..8f5e7cb5f7 100644 --- a/cmake/dependencies.cmake +++ b/cmake/dependencies.cmake @@ -226,13 +226,13 @@ if(USE_HDF5) endif(USE_HDF5) ################################ -# Curl Libraryies +# Curl Libraries # Only needed for DAP (DAP2 or DAP4) -# and NCZARR with S3 Support +# and NCZARR S3 support +# and byterange support ################################ -if( (NETCDF_ENABLE_DAP AND (NETCDF_ENABLE_DAP2 OR NETCDF_ENABLE_DAP4 OR NETCDF_ENABLE_BYTERANGE_SUPPORT)) OR (NETCDF_ENABLE_NCZARR AND NETCDF_ENABLENCZARR_S3)) - +if( NETCDF_ENABLE_DAP2 OR NETCDF_ENABLE_DAP4 OR NETCDF_ENABLE_BYTERANGE_SUPPORT OR NETCDF_ENABLE_NCZARR_S3) # See if we have libcurl find_package(CURL) #target_compile_options(netcdf diff --git a/cmake/netcdf_functions_macros.cmake b/cmake/netcdf_functions_macros.cmake index dd2643e1fe..ea9d8ce819 100644 --- a/cmake/netcdf_functions_macros.cmake +++ b/cmake/netcdf_functions_macros.cmake @@ -324,4 +324,16 @@ function(is_disabled feature ret_val) set(${ret_val} "yes" PARENT_SCOPE) set("NC_${ret_val}" 1 PARENT_SCOPE) endif(${feature}) -endfunction() \ No newline at end of file +endfunction() + +# Extract the last element from a path string +function(getlastdir s ret_val) + if(NOT ISMSVC AND NOT ISMINGW) + string(REPLACE ":" ";" list "${s}") + else() + set(list ${s}) + endif() + list(GET list -1 last) + set(${ret_val} "${last}" PARENT_SCOPE) +endfunction() + diff --git a/config.h.cmake.in b/config.h.cmake.in index b1b0507a2b..e088afb1b2 100644 --- a/config.h.cmake.in +++ b/config.h.cmake.in @@ -124,6 +124,9 @@ are set when opening a binary file on Windows. */ #define ERANGE_FILL 1 #endif +/* if true, use hdf5 S3 virtual file reader */ +#cmakedefine NETCDF_ENABLE_HDF5 1 + /* if true, use hdf5 S3 virtual file reader */ #cmakedefine NETCDF_ENABLE_HDF5_ROS3 1 diff --git a/configure.ac b/configure.ac index fb6642281d..400b393d4c 100644 --- a/configure.ac +++ b/configure.ac @@ -1,4 +1,4 @@ -o# -*- Autoconf -*- +# -*- Autoconf -*- ## Process this file with autoconf to produce a configure script. # This is part of Unidata's netCDF package. Copyright 2005-2018, see @@ -936,8 +936,6 @@ else AC_MSG_RESULT([${have_zip}]) enable_nczarr_zip=${have_zip} # alias - - AC_MSG_CHECKING([whether nczarr zip support is enabled]) AC_MSG_RESULT([${enable_nczarr_zip}]) @@ -949,15 +947,10 @@ else AC_MSG_CHECKING([whether netcdf S3 support should be enabled]) AC_ARG_ENABLE([s3], [AS_HELP_STRING([--enable-s3], - [enable netcdf S3 support])]) + [enable netcdf S3 support (default off)])]) test "x$enable_s3" = xyes || enable_s3=no AC_MSG_RESULT($enable_s3) - if test "x$enable_remote_functionality" = xno ; then - AC_MSG_WARN([--disable-remote-functionality => --disable-s3]) - enable_s3=no - fi - # --enable-nczarr-s3 is synonym for --enable-s3 (but...) AC_MSG_CHECKING([whether netcdf NCZarr S3 support should be enabled]) AC_ARG_ENABLE([nczarr-s3], @@ -966,11 +959,16 @@ else AC_MSG_RESULT([$enable_nczarr_s3 (Deprecated) Please use --enable-s3)]) # Set enable_s3 instead of enable_nczarr_s3 - if test "x$enable_s3" = xno && test "x$enable_nczarr_s3" = xyes && test "x$enable_remote_functionality" = xyes; then + if test "x$enable_s3" = xno && test "x$enable_nczarr_s3" = xyes ; then enable_s3=yes # back compatibility fi unset enable_nczarr_s3 + if test "x$enable_remote_functionality" = xno ; then + AC_MSG_WARN([--disable-remote-functionality => --disable-s3]) + enable_s3=no + fi + # Note we check for the library after checking for enable_s3 # because for some reason this fails if we unconditionally test for sdk # and it is not available. Fix someday @@ -1166,7 +1164,7 @@ AC_ARG_ENABLE([benchmarks], are timed. We use these tests to check netCDF performance.])]) test "x$enable_benchmarks" = xyes || enable_benchmarks=no AC_MSG_RESULT($enable_benchmarks) -if test "x$enable_HDF5" = xno -a "x$enable_benchmarks" = xyes; then +if test "x$enable_hdf5" = xno -a "x$enable_benchmarks" = xyes; then AC_MSG_ERROR([Can't use benchmarks if HDF5 is disabled.]) fi AM_CONDITIONAL(BUILD_BENCHMARKS, [test x$enable_benchmarks = xyes]) @@ -1645,7 +1643,7 @@ enable_hdf5_szip=no has_hdf5_ros3=no if test "x$enable_hdf5" = xyes; then - + AC_DEFINE([NETCDF_ENABLE_HDF5], [1], [if true, use HDF5]) AC_DEFINE([USE_HDF5], [1], [if true, use HDF5]) AC_DEFINE([H5_USE_16_API], [1], [use HDF5 1.6 API]) @@ -2145,25 +2143,46 @@ fi AC_SUBST(STD_FILTERS,[$std_filters]) +# Internally, the variable HDF5_PLUGIN_PATH +# is always used as a surrogate for the default +# plugin path. If it was not defined by the user, +# then set it internally. +if test "x${HDF5_PLUGIN_PATH}" = x ; then + if test "x$ISMSVC" = xyes || test "x$ISMINGW" = xyes; then + HDF5_PLUGIN_PATH="${ALLUSERSPROFILE}\\hdfd5\\lib\\plugin" + else + HDF5_PLUGIN_PATH="/usr/local/hdf5/lib/plugin" + fi +fi + # The --with-plugin-dir gives the user control of the plugin -# directory. If set to 'yes' (the default), then first directory in -# the HDF5_PLUGIN_PATH, or /usr/local/hdf5/lib/plugin (the default -# HDF5 plugin dir) will be used. If another directory is provided, it -# will be used. If 'no', then plugins will not be installed. +# directory. If set to 'yes' (the default), then the last directory in +# HDF5_PLUGIN_PATH is used (see above). +# If a specific directory is provided, it will be used. +# If 'no', then plugins will not be installed. + AC_MSG_CHECKING([whether and where we should install plugins]) AC_ARG_WITH([plugin-dir], [AS_HELP_STRING([--with-plugin-dir=|yes|no|--without-plugin-dir], [Install selected standard filters in specified or default directory])], [],[with_plugin_dir=no]) +AC_MSG_RESULT([$with_plugin_dir]) +# Using selected filter options forces installation if test "x$have_zstd" = xyes; then if test "x$enable_plugins" = xyes && test "x$with_plugin_dir" = xno; then + AC_MSG_WARN([--enable_filter-zstd => --with-plugin-dir.]) with_plugin_dir=yes fi fi -AC_MSG_RESULT([$with_plugin_dir]) if test "x$with_plugin_dir" = xno ; then # option missing|disabled with_plugin_dir=no with_plugin_dir_setting="N.A." enable_plugin_dir=no +<<<<<<< HEAD +elif test "x$with_plugin_dir" != x ; then + if test "x$with_plugin_dir" != xyes && test "x$with_plugin_dir" != xno ; then + # Presume true directory + HDF5_PLUGIN_PATH="${with_plugin_dir}" +======= elif test "x$with_plugin_dir" = xyes ; then # --with-plugin-dir, no argument # Default to last dir (lowest search priority) in HDF5_PLUGIN_PATH PLUGIN_PATH="$HDF5_PLUGIN_PATH" @@ -2178,24 +2197,25 @@ elif test "x$with_plugin_dir" = xyes ; then # --with-plugin-dir, no argument fi fi +>>>>>>> Unidata/main fi - # Use the lowest priority dir in the path - if test "x$ISMSVC" = xyes || test "x$ISMINGW" = xyes; then - PLUGIN_DIR=`echo "$PLUGIN_PATH" | tr ';' ' '` - else - PLUGIN_DIR=`echo "$PLUGIN_PATH" | tr ':' ' '` - fi - for pp in ${PLUGIN_DIR} ; do last="$pp"; done - PLUGIN_DIR="$last" - with_plugin_dir_setting="$PLUGIN_DIR" - # canonical form is all forward slashes - with_plugin_dir=`echo "$PLUGIN_DIR" | tr '\\\\' '/'` - enable_plugin_dir=yes - AC_MSG_NOTICE([Defaulting to --with-plugin-dir=$with_plugin_dir]) -else # --with-plugin-dir= - with_plugin_dir_setting="$with_plugin_dir" - enable_plugin_dir=yes fi + +# Use the last dir (lowest search priority) in HDF5_PLUGIN_PATH +PLUGIN_PATH="$HDF5_PLUGIN_PATH" +if test "x$ISMSVC" = xyes || test "x$ISMINGW" = xyes; then + PLUGIN_DIR=`echo "$PLUGIN_PATH" | tr ';' ' '` +else + PLUGIN_DIR=`echo "$PLUGIN_PATH" | tr ':;' ' '` +fi +last= +for pp in ${PLUGIN_DIR} ; do last="$pp"; done +PLUGIN_DIR="$last" +with_plugin_dir_setting="$PLUGIN_DIR" +# canonical form is all forward slashes +with_plugin_dir=`echo "$PLUGIN_DIR" | tr '\\\\' '/'` +enable_plugin_dir=yes +AC_MSG_NOTICE([Final --with-plugin-dir=$with_plugin_dir]) AM_CONDITIONAL([ENABLE_PLUGIN_DIR], [test "x$enable_plugin_dir" = xyes]) AC_SUBST([PLUGIN_INSTALL_DIR], [$with_plugin_dir]) # Better value for libnetcdf.settings diff --git a/docs/Doxyfile.in b/docs/Doxyfile.in index e6e6c0e5ff..817778c1a7 100644 --- a/docs/Doxyfile.in +++ b/docs/Doxyfile.in @@ -965,6 +965,7 @@ INPUT = @abs_top_srcdir@/docs/mainpage.dox \ @abs_top_srcdir@/docs/all-error-codes.md \ @abs_top_srcdir@/docs/filters.md \ @abs_top_srcdir@/docs/quickstart_filters.md \ + @abs_top_srcdir@/docs/pluginpath.md \ @abs_top_srcdir@/docs/quickstart_paths.md \ @abs_top_srcdir@/docs/quickstart_env.md \ @abs_top_srcdir@/include/netcdf.h \ diff --git a/docs/Makefile.am b/docs/Makefile.am index 3bacd2d08d..5f897260ed 100644 --- a/docs/Makefile.am +++ b/docs/Makefile.am @@ -16,7 +16,7 @@ known_problems.md COPYRIGHT.md inmeminternal.dox testserver.dox \ byterange.md nczarr.md quantize.md all-error-codes.md \ quickstart_paths.md cloud.md header.html attribute_conventions.md \ file_format_specifications.md quickstart_filters.md quickstart_env.md \ -doxygen-awesome-css netcdf-50x50.png +doxygen-awesome-css netcdf-50x50.png pluginpath.md # Turn off parallel builds in this directory. .NOTPARALLEL: diff --git a/docs/filters.md b/docs/filters.md index bab3df3b9b..8a510ee3db 100644 --- a/docs/filters.md +++ b/docs/filters.md @@ -13,9 +13,8 @@ should be aware. * ***Auto Install of filters***
An option is now provided to automatically install HDF5 filters into a default location, or optionally -into a user-specified location. This is described in -[Appendix H](#filters_appendixh) -(with supporting information in [Appendix G](#filters_appendixg)). +into a user-specified location. +This is described in the *pluginpath.md* document. * ***NCZarr Filter Support***
[NCZarr filters](#filters_nczarr) are now supported. @@ -367,32 +366,8 @@ The details for writing such a filter are defined in the HDF5 documentation[1,2] ### Plugin directory {#filters_plugindir} The HDF5 loader searches for plugins in a number of directories. -This search is contingent on the presence or absence of the environment -variable named ***HDF5\_PLUGIN\_PATH***. - -As with all other "...PATH" variables, it is a sequence of absolute -directories separated by a separator character. For *nix* operating systems, -this separator is the colon (':') character. For Windows and Mingw, the -separator is the semi-colon (';') character. So for example: - -* Linux: `export HDF5_PLUGIN_PATH=/usr/lib:/usr/local/lib` -* Windows: `export HDF5_PLUGIN_PATH=c:\\ProgramData\\hdf5\\plugin;c:\\tools\\lib` - -If HDF5\_PLUGIN\_PATH is defined, then the loader will search each directory -in the path from left to right looking for shared libraries with specific -exported symbols representing the entry points into the library. - -If HDF5\_PLUGIN\_PATH is not defined, the loader defaults to using -these default directories: - -* Linux: `/usr/local/hdf5/lib/plugin` -* Windows: `%ALLUSERSPROFILE%\\hdf5\\lib\\plugin` - -It should be noted that there is a difference between the search order -for HDF5 versus NCZarr. The HDF5 loader will search only the directories -specificed in HDF5\_PLUGIN\_PATH. In NCZarr, the loader -searches HDF5\_PLUGIN\_PATH and as a last resort, -it also searches the default directory. +The netcdf-c process for installing and locating plugins is described +in detail in the *pluginpath.md* document. ### Plugin Library Naming {#filters_Pluginlib} @@ -1164,7 +1139,7 @@ When installing the netcdf library, the following other libraries must be instal 2. The HDF5 wrapper for *libzstd.so* -- There are several options for obtaining this (see [Appendix G](#filters_appendixg).) 3. (Optional) The Zarr wrapper for *libzstd.so* -- you need this if you intend to read/write Zarr datasets that were compressed using zstandard; again see [Appendix G](#filters_appendixg). -## Appendix G. Finding Filters {#filters_appendixg} +## Appendix G. Finding Filter Implementations {#filters_appendixg} A major problem for filter users is finding an implementation of an HDF5 filter wrapper and (optionally) its corresponding NCZarr wrapper. There are several ways to do this. @@ -1190,40 +1165,7 @@ so they are only usable with netcdf-4. This will change in the future. As part of the overall build process, a number of filter wrappers are built as shared libraries in the "plugins" directory. These wrappers can be installed as part of the overall netcdf-c installation process. WARNING: the installer still needs to make sure that the actual filter/compression libraries are installed: e.g. libzstd and/or libblosc. - -The target location into which libraries in the "plugins" directory are installed is specified -using a special *./configure* option - -```` ---with-plugin-dir= -or ---with-plugin-dir -```` -or its corresponding *cmake* option. - -```` --DPLUGIN_INSTALL_DIR= -or --DPLUGIN_INSTALL_DIR=YES -```` -This option defaults to the value "yes", which means that filters are -installed by default. This can be disabled by one of the following options. - -```` ---without-plugin-dir (automake) -or ---with-plugin-dir=no (automake) -or --DPLUGIN_INSTALL_DIR=NO (CMake) -```` - -If the option is specified with no argument (automake) or with the value "YES" (CMake), -then it defaults (in order) to the following directories: - -1. If the HDF5\_PLUGIN\_PATH environment variable is defined, then last directory in the list of directories in the path is used. -2. (a) `/usr/local/hdf5/lib/plugin` for linux/unix operating systems (including Cygwin)
- (b) `%ALLUSERSPROFILE%\\hdf5\\lib\\plugin` for Windows and MinGW - +See the document *pluginpaths.md* for details on the installation process. If NCZarr is enabled, then in addition to wrappers for the standard filters, additional libraries will be installed to support NCZarr access to filters. Currently, this list includes the following: @@ -1243,31 +1185,6 @@ provided by the *lib__nczh5filters.so* shared library. Note also that if you disable HDF5 support, but leave NCZarr support enabled, then all of the above filters should continue to work. -### HDF5\_PLUGIN\_PATH - -At the moment, NetCDF uses the existing HDF5 environment variable -*HDF5\_PLUGIN\_PATH* to locate the directories in which filter wrapper -shared libraries are located. This is used both for the HDF5 filter -wrappers but also the NCZarr codec wrappers. - -*HDF5\_PLUGIN\_PATH* is a typical Windows or Unix style -path-list. That is it is a sequence of absolute directory paths -separated by a specific separator character. For Windows, the -separator character is a semicolon (';') and for Unix, it is a a -colon (':'). - -So, if HDF5\_PLUGIN\_PATH is defined at build time, and -*--with-plugin-dir* is specified with no argument then the last -directory in the path will be the one into which filter wrappers are -installed. Otherwise the default directories are used. - -The important thing to note is that at run-time, there are several cases to consider: - -1. HDF5\_PLUGIN\_PATH is defined and is the same value as it was at build time -- no action needed -2. HDF5\_PLUGIN\_PATH is defined and is has a different value from build time -- the user is responsible for ensuring that the run-time path includes the same directory used at build time, otherwise this case will fail. -3. HDF5\_PLUGIN\_PATH is not defined at either run-time or build-time -- no action needed -4. HDF5\_PLUGIN\_PATH is not defined at run-time but was defined at build-time -- this will probably fail - ## Appendix I. A Warning on Backward Compatibility {#filters_appendixi} The API defined in this document should accurately reflect the @@ -1290,9 +1207,9 @@ A user may encounter an incompatibility if any of the following appears in user For additional information, see [Appendix B](#filters_appendixb). -## Point of Contact {#filters_poc} +## History {#filters_history} *Author*: Dennis Heimbigner
-*Email*: dmh at ucar dot edu
+*Email*: dennis.heimbigner@gmail.com
*Initial Version*: 1/10/2018
*Last Revised*: 5/18/2022 diff --git a/docs/footer.html b/docs/footer.html index efa3357e59..a92449b0bb 100644 --- a/docs/footer.html +++ b/docs/footer.html @@ -1,4 +1,4 @@ - +