SQL Server connectivity issue - 'TCP Provider: Error code 0x68'

[Paul-Eccleston-i2]

I am seeing a repeatable but slightly random error of TCP Provider: Error code 0x68 when running two MS SQL Server Docker containers in the same Docker bridge network.

The first docker container (server) is running as follows:-

  docker run \
    "-d" \
    "--rm" \
    "--platform=linux/amd64" \
    "--network-alias=server.test" \
    "--name=server" \
    "--network=test" \
    "--env=ACCEPT_EULA=Y" \
    "--env=SA_PASSWORD=StrongPassword" \
    "--env=MSSQL_PID=Developer" \
    "--env=MSSQL_AGENT_ENABLED=true" \
    "mcr.microsoft.com/mssql/rhel/server:2022-latest"

The second docker container (client) is running as follows:-

  docker run \
    "--rm" \
    "--platform=linux/amd64" \
    "--network=test" \
    "--name=client" \
    -v "<host path to scripts>:<container path to scripts>" \
    "mcr.microsoft.com/mssql/rhel/server:2022-latest" \
    bash -c "set -e; ${test_command}"

The following is the test command executed:-

test_command="${1:-"/opt/mssql-tools/bin/sqlcmd -S server.test -U sa -P StrongPassword -N -C -i '<sql scripts to run>'"}"

Things I have tried:-

• Verified RAM usage is acceptable on the host (32GB total, around 22GB in use)
• Tried different Microsoft SQL images
• Ensured no packages are being updated in the base OS (seemed to fix the issue my preventing certain packages from installing, but this no longer works)
• Tried ruling out scripts, but at this point it does not seem to be a single script causing the issue.
• Looked at the SQL Server container logs - nothing of interest is shown, no errors are logged.

So I am at a loss as to the cause of the issue at this point, it seems to me to be network connectivity related, but I see nothing obviously wrong with the above, does anybody have any suggestions on how I can try and debug this further, if you need any further information then please let me know.

[Paul-Eccleston-i2]

OK I have done some more investigation into the above issue and it is to do with the Red Hat base image used to produce the 'Microsoft SQL Server - RHEL based images' here.

Here is the results of my investigation so far, all testing is done with the option -N for sqlcmd (encrypt connection - same as -Nm) and with NO Red Hat package updates applied:-

• Specifying image FROM "mcr.microsoft.com/mssql/rhel/server:2022-CU12-rhel-9.1" - [Result] Connectivity is fine, the issue as shown in the OP above is NOT seen.
• Specifying image FROM "mcr.microsoft.com/mssql/rhel/server:2022-CU15-rhel-9.1" - [Result] Connectivity is fine, the issue as shown in the OP above is NOT seen.
• Specifying image FROM "mcr.microsoft.com/mssql/rhel/server:2022-CU16-rhel-9.1" - [Result] Connectivity is fine, the issue as shown in the OP above is NOT seen.
• Specifying image FROM "mcr.microsoft.com/mssql/rhel/server:2022-CU17-rhel-9.1" - [Result] Connectivity is broken, the issue as shown in the OP above IS seen.

Notes:-

• Tag name 2022-CU17-rhel-9.1 currently is the same as tag latest
• If updates are applied to the Red Hat base image for ANY of the above images then the issue as shown in the OP above IS seen.
• Specifying the -No (optional encryption) for sqlcmd prevents the issue from occurring, however I need connectivity to be encrypted so this is not a workaround.

So in conclusion the Red Hat base image that MS uses is broken for encryption and causes sqlcmd to error out when attempting to connect from one container to another (in the same docker network) using encryption.

[crballentine-ess]

I can confirm the issue exists for me as well with the 2022-CU17-rhel-9.1 (:latest) tagged image. It started occurring in my Podman Desktop environment when I re-built our images against the :latest tag. After reverting to the 2022-CU16-rhel-9.1 tagged image the problem goes away. When I rebuild against the :latest tagged image the issue re-occurs.

[Paul-Eccleston-i2]

I have now created a simple bash script to demonstrate the issue, this script is actually even simpler than explained in the OP requiring a single SQL Server Red Hat container running and executing a SQL script (included inline in the script) with connection encryption turned on (fully automated in the script).

Link to script:- https://gist.github.com/Paul-Eccleston-i2/b22980c317bbf04132ecdbee2743f843