Skip to content

Commit

Permalink
fixup: extend the lifecycle of Proton config instances
Browse files Browse the repository at this point in the history
  • Loading branch information
@kgiusti
kgiusti committed Sep 25, 2024
1 parent 727eaa0 commit 9a532a9
Show file tree
Hide file tree
Showing 5 changed files with 556 additions and 178 deletions.
178 changes: 104 additions & 74 deletions src/tls/README.txt
View file
Original file line number Diff line number Diff line change
Expand Up @@ -26,12 +26,13 @@ See the router management schema for more details.

== Primary Data Structures ==

There are four primary data structures used for TLS support:
There are five primary data structures used for TLS support:

- qd_tls_context_t (private)
- qd_ssl2_profile_t
- qd_tls_profile_t
- qd_tls_config_t
- qd_tls_session_t
- qd_proton_config_t

The root structure is the qd_tls_context_t. There is a one-to-one
relationship between a qd_tls_context_t and an sslProfile management
Expand All @@ -43,17 +44,30 @@ sslProfile entity. A qd_tls_context_t has an embedded
qd_ssl2_profile_t instance for holding the current values from the
sslProfile record.

A qd_tls_profile_t represents an instance of an Openssl configuration
context. It is created by loading the certificates provided by an
sslProfile record. It is a child of the qd_tls_context_t that
maintains the sslProfile configuration used to create it. A
qd_tls_config_t is created and owned by the Listener/Connector which
is configured to use the sslProfile associated with the
qd_tls_config_t parent qd_tls_context_t.
A qd_tls_config_t represents an instance of an Openssl configuration
context. A qd_tls_config_t is created and owned by the
Listener/Connector which is configured to use an sslProfile. In order
to support management updates to an sslProfile record all
qd_tls_config_ts are held in a list in the qd_tls_context_t associated
with the qd_tls_config_t's parent sslProfile. The qd_tls_config_t
creates a qd_proton_config_t using the sslProfile and configuration
settings from its parent Listener/Connector.

A qd_proton_config_t is created by a qd_tls_config_t and wraps the
underlying Proton SSL/TLS configuration instance. This is either an
instance of a pn_ssl_domain_t (for AMQP) or a pn_tls_config_t (for raw
connection). This wrapper fufills two requirements: 1) creation and
deletion of Proton TLS sessions using the same configuration MUST be
single threaded, and 2) the Proton configuration MUST NOT be freed
until all sessions using the configuration have been freed. The
qd_proton_config_t contains a mutex and an atomic reference count to
satisfy these requirements. qd_proton_config_ts are immutable. In
order to update a configuration a new qd_proton_config_t is created to
replace the old.

A qd_tls_session_t is the per-connection TLS state. It is created
using the qd_tls_config_t associated with the connection's parent
Listener/Connector.
using the qd_tls_config_t and qd_proton_config_t associated with the
connection's parent Listener/Connector.

=== Object Lifecycle ===

Expand All @@ -64,11 +78,13 @@ qd_tls_config_t instances have been freed.

A qd_tls_config_t instance is reference-counted. The
Listener/Connector that created the qd_tls_config_t holds a reference
to it as well as all child qd_tls_session_t instances created using
that qd_tls_config_t. A qd_tls_config_t instance is created when a
Listener/Connector is instantiated via the management agent. It is
destroyed after the parent Listener/Connector is deleted and all
active child qd_tls_session_t instances have been freed.
to it. A reference is also held by the parent qd_tls_context_t. A
qd_tls_config_t instance is created when a Listener/Connector is
instantiated via the management agent. It is destroyed after the
parent Listener/Connector is deleted.

A qd_proton_config_t is reference counted by the parent
qd_tls_config_t and all qd_tls_session_ts using it.

A qd_tls_session_t is created when a new protocol connection is
opened. It is released when the associated connection is terminated.
Expand All @@ -84,58 +100,59 @@ opened. It is released when the associated connection is terminated.
+---+---------------------+
|
|
+-->+- qd_tls_config_t -+ <--> +- qd_tls_config_t -+ <--> ...
| | | |
+----> | | | |
| +-------------------+ +-------------------+
| ^ ^
| | |
| | |
| +-+- Listener -+ +-+- Listener -+
| | "Foo" | | "Bar" |
| +--+-----------+ +--------------+
| |
| |
| +--------+
| |
| |
| +- qd_tls_session_t -+
| | "Conn1" |
+-------------->+ |
| | |
| +--------------------+
| |
| |
| +- qd_tls_session_t -+
| | "Conn2" |
+---------------+ |
| +- qd_tls_config_t-+ +- qd_tls_config_t -+
+-->| | <--> | | <--> ...
+-----------------++ +-------------------+
^ | ^
| | |
| | |
+-+- Listener -+ | +-+- Listener -+
| "Foo" | | | "Bar" |
+--------------+ | +--------------+
|
|
+------------------------+
|
V
+- qd_tls_session_t -+ +- qd_proton_config_t -+
| "Conn1" | | |
| +-------->+ |
| | | |
+--------------------+ +--+-------------------+
^
|
+- qd_tls_session_t -+ |
| "Conn2" | |
| +------------+
| |
+--------------------+

In the above diagram there are two Listener instances both sharing
the sslProfile record "Profile1". Each listener maintains its own
qd_tls_config_t. Listener "Foo" has two active TLS connections. These
connections have a dedicated qd_tls_session_t instance. The
qd_tls_session_t instances also reference their partent
qd_tls_config_t.
In the above diagram there are two Listener instances both sharing the
sslProfile record "Profile1". Each listener maintains its own
qd_tls_config_t. The qd_tls_config_t's are held in a list on the
parent qd_tls_context_t. Listener "Foo" has two active TLS
connections. These connections have a dedicated qd_tls_session_t
instance. The sessions reference the qd_proton_config_t used to create
them.


== Threading

The management operations - create/update/delete - for sslProfile,
Listener, and Connector management entities occur on the management
agent thread. Therefore these operations are not multithreaded. This
allows the implementation to avoid using locks to manage
qd_tls_context_t and qd_tls_config_t instances. Debug code is present
to assert that these operations are only called via the management
agent thread.

Per-connection qd_tls_session_t operations may occur on any I/O
thread and therefore can run simultaineously with the management agent
thread. Each qd_tls_config_t instance has a mutex that must be held
when a qd_tls_session_t is created and deleted in order to prevent
races with the management agent thread. I/O operations using a
qd_tls_session_t are thread safe - there is no need to hold the
parent qd_tls_config_t lock while I/O is occurring.
qd_tls_context_t and qd_tls_config_t/qd_proton_config_t
instances. Debug code is present to assert that these operations are
only called via the management agent thread.

Per-connection qd_tls_session_t operations may occur on any I/O thread
and therefore can run simultaineously with the management agent
thread. This means sslProfile updates (and deletions) may occur while
I/O threads are using the qd_tls_sesson_t that are dependent on the
sslProfile configuration. There are mutexes in the qd_tls_config_t and
the qd_proton_config_t that prevent races between the I/O threads and
management. See below.

== sslProfile Management Update

Expand All @@ -152,22 +169,35 @@ during the create/update operation which occurs on the management
thread. Therefore the management thread assumes the cost of the
operation and I/O threads are not impacted.

However updating an sslProfile causes all child qd_tls_config_t
instances to update their run-time configuration. This means that all
qd_tls_config_ts have to re-load their certificate files. While this
is being done (on the management thread) new qd_tls_session_t
instances using that qd_tls_config_t may be created by the I/O
threads (e.g. a new connection arrives).

Note the qd_tls_config_t lock must be held when new child
qd_tls_session_t instances are created (to prevent collisions with
the management agent thread). If the qd_tls_config_t lock is held
while the certificate files are being loaded then the I/O thread would
be blocked waiting to create the qd_tls_session_t and long latencies
would result. This implementation avoids this problem by reloading the
certificates outside of the lock and only taking the lock long enough
to simply switch in the new configuration. This operation is very fast
and results in minimal I/O thread impact.
However updating an sslProfile requires all child qd_tls_config_t
instances update their run-time configuration. This means that all
qd_tls_config_ts have to re-load their certificate files and generate
new qd_proton_config_t instances. While this is being done (on the
management thread) new qd_tls_session_t instances using that
qd_tls_config_t and its current qd_proton_config_t may be created by
the I/O threads (e.g. a new connection arrives).

Two mutexes are used to prevent race conditions between the management
thread and the I/O threads.

First there is a mutex in the qd_tls_config_t that protects its
pointer to the qd_proton_config_t instance. Keep in mind that a
qd_proton_config_t is immutable and must be replaced in order to
update the TLS configuration. When an sslProfile management operation
occurs this lock is held long enough to swap out the old
qd_proton_config_t instance with a new qd_proton_config_t instance
containing the updated management configuration. This same lock must
be held by an I/O thread when the qd_proton_config_t pointer is copied
into a new qd_tls_session_t instance and the qd_proton_config_t's
reference count is incremented. This prevents the qd_proton_config_t
from being deleted during the qd_tls_session_t creation process.

Second there is a mutex in the qd_proton_config_t that must be held
when a Proton session is created or destroyed. This is necessary because
these operations are not thread safe.

Since qd_proton_config_t are reference counted and immutable there is
no need to nest these locks.

== Files

Expand Down
54 changes: 35 additions & 19 deletions src/tls/private.h
View file
Original file line number Diff line number Diff line change
Expand Up @@ -28,21 +28,40 @@
#include "qpid/dispatch/log.h"

typedef struct qd_tls_context_t qd_tls_context_t;
typedef struct qd_proton_config_t qd_proton_config_t;
typedef struct pn_tls_config_t pn_tls_config_t;
typedef struct pn_tls_t pn_tls_t;
typedef struct pn_ssl_domain_t pn_ssl_domain_t;
typedef struct pn_ssl_t pn_ssl_t;


/**
* Handle for Proton TLS configuration.
*
* A Proton TLS configuration cannot be destroyed until all related TLS sessions have closed and the parent TLS
* configuration has been deleted. A reference count is used to ensure this. Additionally a mutex is provided to
* single-thread the creation and deletion of proton sessions. This is necessary since proton sessions are
* multi-threaded.
*/
struct qd_proton_config_t {
// only one of the following Proton pointers will be set based on whether this configuration is for a raw connection
// or an AMQP connection
pn_tls_config_t *pn_raw;
pn_ssl_domain_t *pn_amqp;
sys_mutex_t lock; // held when creating and deleting sessions
sys_atomic_t ref_count; // for parent qd_tls_config_t and all children qd_tls_session_t
};

/**
* Context for a single per-connection TLS data stream
*/
struct qd_tls_session_t {
qd_tls_config_t *tls_config; // parent
qd_proton_config_t *proton_tls_cfg; // TLS Proton configuration used by session

// only one of the following Proton session pointers will be set based on whether this session is for a raw
// connection or an AMQP connection
pn_tls_t *pn_raw;
pn_ssl_t *pn_amqp;
pn_tls_t *pn_raw;
pn_ssl_t *pn_amqp;

void *user_context;
qd_tls_session_on_secure_cb_t *on_secure_cb;
Expand Down Expand Up @@ -72,22 +91,17 @@ struct qd_tls_session_t {
*/
struct qd_tls_config_t {
DEQ_LINKS(qd_tls_config_t); // for parent qd_tls_context_t list
sys_mutex_t lock;
char *ssl_profile_name;
char *uid_format; // lock must be held

// only one of the following Proton pointers will be set based on whether this configuration is for a raw connection
// or an AMQP connection
pn_tls_config_t *pn_raw; // lock must be held
pn_ssl_domain_t *pn_amqp; // lock must be held

qd_tls_type_t p_type;
sys_atomic_t ref_count;
long version; // lock must be held

bool authenticate_peer;
bool verify_hostname;
bool is_listener;
sys_mutex_t lock;
char *ssl_profile_name;
char *uid_format; // lock must be held
qd_proton_config_t *proton_tls_cfg; // lock must be held
qd_tls_type_t p_type;
sys_atomic_t ref_count;
long version; // lock must be held

bool authenticate_peer;
bool verify_hostname;
bool is_listener;
};

DEQ_DECLARE(qd_tls_config_t, qd_tls_config_list_t);
Expand Down Expand Up @@ -116,5 +130,7 @@ pn_tls_config_t *tls_private_allocate_raw_config(const char *ssl_profile_name, c
bool is_listener, bool verify_hostname, bool authenticate_peer);
pn_ssl_domain_t *tls_private_allocate_amqp_config(const char *ssl_profile_name, const qd_ssl2_profile_t *config,
bool is_listener, bool verify_hostname, bool authenticate_peer);
qd_proton_config_t *qd_proton_config(pn_tls_config_t *tls_cfg, pn_ssl_domain_t *ssl_cfg);
void qd_proton_config_decref(qd_proton_config_t *p_cfg);
#endif

Loading

0 comments on commit 9a532a9

Please sign in to comment.