How to use this in a makefile for CI/CD (sqlcli1.h: No such file or directory)

[ZacZelner]

Hi, we are using this for years, and recently got a issue when we build a auto CI/CD pipeline.
We are using a makefile to do the auto CI/CD, and code are:

ifeq (,$(shell go env GOBIN))
GOBIN=$(shell go env GOPATH)/bin
else
GOBIN=$(shell go env GOBIN)
endif
CGO_CFLAGS=-I$DB2HOME/include
CGO_LDFLAGS=-L$DB2HOME/lib
LD_LIBRARY_PATH=$DB2HOME/lib

install-runtime:
GO111MODULE=on go install github.com/tools/godep@latest
GO111MODULE=on go install github.com/ibmdb/go_ibm_db/[email protected]

local_setup:

sh ./deployment/local/setup.sh

go run /go/pkg/mod/github.com/ibmdb/[email protected]/installer/setup.go

#CIO CI CD
.PHONY: unit-test
unit-test:
go test tests/wc_test.go

.PHONY: lint
lint:
@echo '==> Linting...'
go fmt
go vet

However, when it runs to go vet, it pop up the error:
vendor/github.com/ibmdb/go_ibm_db/api/api_unix.go:12:11: fatal error: sqlcli1.h: No such file or directory
// #include <sqlcli1.h>

From our experience, it means the env setting is not correct.

We have already claim the env in the beginning, however it does not work.

Anyone know how to build this go_ibm_db in a makefile?

Thanks!

[vmathur12]

@ZacZelner
This error occurs when environment variables are not set properly or when some files are missing.
Could you please share the output of the following commands:
echo $IBM_DB_HOME
go env
ls $IBM_DB_HOME/include
ls $IBM_DB_HOME/lib
uname -a
Thanks

[bimalkjha]

@ZacZelner Any update? Thanks.

[ZacZelner]

Thanks. @bimalkjha @vmathur12 .

It seems there is no $IBM_DB_HOME env in the container when the makefile build.

Part of our makefile:

Development environments

-----------------------------------------------------------------------------

install-runtime:
GO111MODULE=on go install github.com/tools/godep@latest
GO111MODULE=on go install github.com/ibmdb/go_ibm_db/[email protected]

local_setup:

sh ./deployment/local/setup.sh

go run /go/pkg/mod/github.com/ibmdb/[email protected]/installer/setup.go

.PHONY: unit-test
unit-test:
go test tests/wc_test.go

.PHONY: lint
lint:
@echo '==> IBM_DB_HOME env...'
@echo IBM_DB_HOME
go env
@env
@echo '==> Linting...'
go fmt
go vet

I print all the env:
go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/opt/app-root/src/.cache/go-build"
GOENV="/opt/app-root/src/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/workspace/go-mod-cache"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/opt/app-root/src/.go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/lib/golang"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/golang/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.20.12"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/workspace/cloned-git-repository/go.mod"
GOWORK=""
CGO_CFLAGS="-O2 -g"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-O2 -g"
CGO_FFLAGS="-O2 -g"
CGO_LDFLAGS="-O2 -g"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build2109435798=/tmp/go-build -gno-record-gcc-switches"
KUBERNETES_PORT_443_TCP_PORT=443
MAKEFILE_SCRIPT=lint
SUMMARY=Platform for building and running Go Applications
_=/usr/bin/make
VERSION=1.20.12
CONTAINER_NAME=rhel8/go-toolset
NSS_SDB_USE_CACHE=no
PATH=/opt/app-root/src/bin:/opt/app-root/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/app-root/src/.go/bin
STI_SCRIPTS_PATH=/usr/libexec/s2i
STI_SCRIPTS_URL=image:///usr/libexec/s2i
GOPATH=/opt/app-root/src/.go
KUBERNETES_PORT_443_TCP=tcp://172.21.0.1:443
container=oci
PWD=/workspace/cloned-git-repository
KUBERNETES_PORT=tcp://172.21.0.1:443
HOME=/opt/app-root/src
HOSTNAME=cdm-acq-ocv-dxmjc-lint-pod
GO_MAJOR_VERSION=1
SHLVL=1
NAME=golang
MAKEFLAGS=
MFLAGS=
KUBERNETES_SERVICE_PORT_HTTPS=443
GO_PATCH_VERSION=12
PLATFORM=el8
KUBERNETES_SERVICE_HOST=172.21.0.1
APP_ROOT=/opt/app-root
KUBERNETES_SERVICE_PORT=443
BUILD_VERSION=
SSL_CERT_DIR=/tekton-custom-certs:/etc/ssl/certs:/etc/pki/tls/certs:/system/etc/security/cacerts
KUBERNETES_PORT_443_TCP_ADDR=172.21.0.1
NODEJS_VER=20
GO_MINOR_VERSION=20
GO_MOD_CACHE_PATH=/workspace/go-mod-cache
TERM=xterm
DESCRIPTION=Go Toolset available as a container is a base platform for building and running various Go applications and frameworks. Go is an easy to learn, powerful, statically typed language in the C/C++ tradition with garbage collection, concurrent programming support, and memory safety features.
KUBERNETES_PORT_443_TCP_PROTO=tcp
MAKELEVEL=1

It works well in our dockerfile before. But not work well in Makefile.

Our CI/CD developer said:
I understand that to use Db2 in a Go project you need to perform some extra steps like this. Like setting environment variables and running a setup script.
Each "step" in the pipeline is executed in its own container. Meaning that you lose the file and environment variables contents except for locations that are explicitly specified in the pipeline. Such as the location where dependencies are installed, source code is, and executable is output. And not all volumes are carried over to every container task. For example, the build (compile) step needs the source code available, but the docker image build step does not. So the volume where the source code is (from the git clone step) is made available to the compile/build task, but not the docker image build task. All of that is to say, any environment variables that you are created in one task, are likely getting lost in the next task. And almost certainly not carried over to the application runtime. Same goes for whatever "setup.go" does.

[ZacZelner]

@bimalkjha @vmathur12 any update for this?

[ZacZelner]

Or how can we skip downloading go install github.com/ibmdb/go_ibm_db/installer, but packed all related folder (like github.com/ibmdb/go_ibm_db/installer/clidriver) directly into image and make your tool work well? @bimalkjha @vmathur12

[bimalkjha]

@ZacZelner Please check install instructions for go_ibm_db in this doc: https://github.com/ibmdb/go_ibm_db?tab=readme-ov-file#how-to-install-in-linuxmac

If you already have a db2client like dsdriver or dsclient or db2server installed in the system, you can set IBM_DB_HOME env var pointing to the install directory of db2client and then install go driver. Otherwise, you need to run setup.go command so that setup.go can pull clidriver and install in the system for use by go. sqlcli1.h file comes with db2clients under include directory. The clidriver that get pulled by setup.go also as include directory with sqlcli1.h file. Then you need to set the CGO flags like in the install doc.

I would suggest to update your CI/CD pipeline script to pull https://public.dhe.ibm.com/ibmdl/export/pub/software/data/db2/drivers/odbc_cli/linuxx64_odbc_cli.tar.gz , untar it and get full path of clidriver directory. Then set IBM_DB_HOME=full path of untared clidriver. Then set below env vars:
CGO_CFLAGS=-I$IBM_DB_HOME/include
CGO_LDFLAGS=-L$IBM_DB_HOME/lib
LD_LIBRARY_PATH=$IBM_DB_HOME/lib

Replace DB2HOME with IBM_DB_HOME in your CI/CD pipeline script.
Now, run your go program, it should work. Please try and let us know all steps that you tried and location of clidriver. Share output of below commands from CI/CD pipeline if go script fails:

db2level
echo $IBM_DB_HOME
ls $IBM_DB_HOME
echo $CGO_CFLAGS
echo $CGO_LDFLAGS
echo $LD_LIBRARY_PATH
echo $PATH

Thanks.

[ZacZelner]

@bimalkjha

Thanks! We tried some solution like you said. However, due to our user access, we just can put all these clidriver into vendor folder, and set:
CGO_CFLAGS="-I/workspace/cloned-git-repository/vendor/github.com/ibmdb/go_ibm_db/installer/clidriver//include"
etc.

Our application source code folder is /workspace/cloned-git-repository.

When we build the go application, it still has this error:

Building go application using version:
go version go1.20.12 linux/amd64
Setting custom GOMODCACHE to /workspace/go-mod-cache

github.com/ibmdb/go_ibm_db/api

vendor/github.com/ibmdb/go_ibm_db/api/api_unix.go:12:11: fatal error: sqlcli1.h: No such file or directory
// #include <sqlcli1.h>
^~~~~~~~~~~
compilation terminated.

the $PATH is:/opt/app-root/src/bin:/opt/app-root/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/app-root/src/.go/bin

Our question is: can the support clidriver folder be put in vendor folder and work well?
If not, should it must be installed in the parent root folder?