Zookeeper stunnel setup

Obsolete documentation

This wiki contains various bits of information that have meanwhile been integrated into our main body of documentation, The dCache Book. These texts will be removed from here during early 2019 in order to avoid fragmentation of the documentation.

---

Secure Tunnel for dCache Domains to Zookeeper Cluster

Zookeeper 3.4.x does not accept TLS encrypted traffic from clients. The support for TLS has been added to the Zookeeper 3.5.x with ZOOKEEPER-2125. However, the 3.5.x release is still in alpha. With the absence of support for TLS in the stable Zookeeper release, we have reached for an alternate solution based on stunnel to encrypt the traffic between clients and Zookeeper cluster. The steps to configure stunnel are laid down below.

Stunnel

stunnel is a proxy designed to add TLS encryption functionality to existing clients and servers without any changes in the programs' code. Its architecture is optimized for security, portability, and scalability (including load-balancing), making it suitable for large deployments.

stunnel can be installed from standard package repositories of Linux distros as follows,

 $ sudo apt-get install stunnel4 (Ubuntu/Debian)
 $ sudo yum install stunnel  (Centos/Fedora)

For Ubuntu, a sample config file is avaliable at /usr/share/doc/stunnel4/examples/stunnel.conf-sample. stunnel can be configured in the server mode (acting as TLS termination end-point and as a load-balance) or client-mode (for non-TLS capable clients).

Server Mode

In the server mode, stunnel can be used as for TLS termination with round-robin based load-balancing. It can also be configured for peer-hostname verification to ensure that only whitelisted hosts can connect.

The host key and the host certificate (in pem format) should be appended together in a single file as follows

cat hostkey.pem hostcert.pem >> stunnel.pem

By default the configuration is loaded from /etc/stunnel/stunnel.conf. A sample snippet stunnel configuration can be found below. This configuration listens on port 2182 for TLS connections. Upon successful TLS handshake, the encrypted channel is terminated here and the plain-text payload is then tunneled to Zookeeper nodes zoocluster1 and zoocluster2 on a round-robin fashion. Hostname whitelisting can be specified here with checkHost field.

; Enable support for the insecure SSLv3 protocol
options = -NO_SSLv3

; These options provide additional security at some performance degradation
;options = SINGLE_ECDH_USE
;options = SINGLE_DH_USE

[zookeeper]
accept  = 2182
connect = zoocluster1.noname.de:2181
connect = zoocluster2.noname.de:2181
cert = /etc/stunnel/stunnel.pem
checkHost = zooclient1.noname.de
checkHost = zooclient2.noname.de
CAPath = /opt/noname/certs

Client Mode

Stunnel client configuration looks very similar to above. client = yes must be set for stunnel to run in client mode. stunnel runs on server-mode by default. In the following configuration, stunnel client listens on the port 2181. All plain-text communication to Zookeeper can be pointed to this stunnel client, which tunnels it over TLS.

; **************************************************************************
; * Service defaults may also be specified in individual service sections  *
; **************************************************************************

; Enable support for the insecure SSLv3 protocol
;options = -NO_SSLv3

; These options provide additional security at some performance degradation
;options = SINGLE_ECDH_USE
;options = SINGLE_DH_USE

; **************************************************************************
; * Include all configuration file fragments from the specified folder     *
; **************************************************************************

;include = /etc/stunnel/conf.d

; **************************************************************************
; * Service definitions (remove all services for inetd mode)               *
; **************************************************************************

; ***************************************** Example TLS client mode services

; The following examples use /etc/ssl/certs, which is the common location
; of a hashed directory containing trusted CA certificates.  This is not
; a hardcoded path of the stunnel package, as it is not related to the
; stunnel configuration in /etc/stunnel/.

[zookeeper]
client = yes
accept = 2181
connect = stunnel.noname.de:2182
;verify = 2
cert = /etc/stunnel/stunnel.pem
CApath = /opt/noname/certs
checkHost = stunnel.noname.de
;OCSPaia = yes

checkHost can also be configured as above to perform hostname whitelisting.

Tunneling Zookeeper traffic via stunnel for dCache

A typical TLS tunnel between a dCache cell (Zookeeper client) and a Zookeeper cluster would require the admins to setup stunnel in client mode for the dCache cells and in server mode for exposing the Zookeeper cluster. Figure below depicts a simple setup.