Adding support for OpenSSL 3.x to IndySockets

[MWASoftware]

I have published two new packages to GitHub in support of providing OpenSSL 3.x support to Indy.

Most of my work has been on creating an OpenSSL Header file code generator including support of just in time resolution of API Calls. I have also applied this work to create an up-to-date version of Indy that includes a separate OpenSSL package and supports OpenSSL 1.0.2 through to 3.x.

Any comments appreciated.

Code Generator

You can find this at https://github.com/MWASoftware/PascalAPI4OpenSSL

This is an entirely new Pascal program that processes a set of template header units to generate the actual OpenSSL Header units supporting static and dynamic linking. Although the published version does have a few OpenSSL assumptions left in it, the code generator is potentially of wider application in support of generating Pascal APIs from 'C' API header files.

I have written an accompanying User Manual that goes into the detail.

Updated Indy Package

You can find this at https://github.com/MWASoftware/Indy.proposedUpdate

Branch: OpenSSLFinal (default)

This branch is up-to-date with the current IndySockets master but otherwise separates out all OpenSSL support into a separate package (Lib/OpenSSL). It also includes the OpenSSL header files generated by the above utility. The same source code tree supports both Delphi and Lazarus/fpc.

The test programs and my own projects confirm the OpenSSL 3.x support and I am happy with the stability of the code. See the readme file for more information.

I don't propose to do any more work on this subject at present other than respond to defect reports. Otherwise it is available for anyone who is interested.

Regards

Tony Whyman

[RDL-28]

I checked, it's working! The only thing on Windows is that it doesn't use libraries from the executable directory until you explicitly specify GetIOpenSSLDDL.SetOpenSSLPath
Otherwise, everything is fine. Thank you.

[rlebeau]

This proposal will not be mergable as-is into the main Indy library, as all traces of OpenSSL are already in progress of being moved to a completely new package with its own repo:

https://github.com/IndySockets/IndyTLS-OpenSSL

Any further efforts to add support for OpenSSL 3.x need to be directed to that project. Version 1.0 will provide backwards compatibility of existing user code for OpenSSL 1.0.2 during this transition. Version 2.0 will then add support for OpenSSL 3.x.

There is currently a branch of the main Indy library that will finalize this separation:

https://github.com/IndySockets/Indy/tree/Indy-10.7

Indy 10.7 and IndyTLSOpenSSL 1.0 are pending release.

[MWASoftware]

This proposal will not be mergable as-is into the main Indy library, as all traces of OpenSSL are already in progress of being moved to a completely new package with its own repo:

https://github.com/IndySockets/IndyTLS-OpenSSL

Remy,
This is not a pull request and hence I am not surprised when you say it is unmergable. What it is, is a demonstration of the OpenSSLHeaders package that I have put together showing how it could be integrated with Indy 10.6 (including a separate OpenSSL package). It does so in a way that a wider group of users can experiment with it and give feedback. It can also be seen as taking some of the "heat" off an Indy 10.7 release by providing an unofficial interim update to10.6 that provides full support to OpenSSL 3.x and Lazarus/fpc.

The lack of Indy support beyond the legacy 1.0.2 (and which has been out of support since 1st Jan 2020) is now a serious problem for many developers including myself and having a version of Indy available to support OpenSSL 3.x was the motivation for my work.

My proposed separate OpenSSLHeaders package also takes out of Indy all worry about support of link strategies and legacy versions. Indy need only support the OpenSSL 3.x API. Do you want me to produce a pull request to add these headers to your proposed TLS package?

As for Indy 10.7, I would look to a more radical change than just making OpenSSL support a separate package. Working on it has made me realise how unwieldy it has become supporting many obscure legacy protocols. There are also units that I could find no use for or references to - some of which seemed to be uncompilable. A code tidy up is the minimum necessary.

My personal preference would be to create an Indy Core package supporting the http/https stack only. Much more lightweight and focused on what most users (including myself) actually use it for (RESTful services plus JSON is my typical use).
I would then create an Indy "extras" package with FTP, SMTP, POP3 and IMAP and perhaps Telnet, and then look very carefully at whether any other packages are required for the rest. (e.g. how many use Gopher nowadays?)

Regards

Tony

Any further efforts to add support for OpenSSL 3.x need to be directed to that project. Version 1.0 will provide backwards compatibility of existing user code for OpenSSL 1.0.2 during this transition. Version 2.0 will then add support for OpenSSL 3.x.

There is currently a branch of the main Indy library that will finalize this separation:

https://github.com/IndySockets/Indy/tree/Indy-10.7

Indy 10.7 and IndyTLSOpenSSL 1.0 are pending release.

[rlebeau]

Remy, This is not a pull request and hence I am not surprised when you say it is unmergable.

Perhaps, but it is yet another side project to pull Indy users in another direction. There are now several separate, independent, and unrelated projects all attempting to do the same thing. I started IndyTLSOpenSSL with the intention to consolidate into 1 unified official release. No simple task.

What it is, is a demonstration of the OpenSSLHeaders package that I have put together showing how it could be integrated with Indy 10.6 (including a separate OpenSSL package).

Like earlier big-scale proposals/PRs (cough cough, #299), I simply lack the free time to review and approve them quickly. Smaller reviews are better. I will nonetheless review this one as time permits. I need to get something released sooner rather than later, and big-scale reviews keep setting me back.

The lack of Indy support beyond the legacy 1.0.2 (and which has been out of support since 1st Jan 2020) is now a serious problem for many developers

I'm well aware of that. And I'm sorry it has taken as long as it has to get Indy more up to date. But I am only 1 person, with a lot on my plate outside of Indy. That's why I was happy when JP returned, since he wrote most of Indy's original OpenSSL code. But he got impatient and went off on his own and released his own project to bring OpenSSL 3.x to Indy.

and having a version of Indy available to support OpenSSL 3.x was the motivation for my work.

Yours, and several others.

My proposed separate OpenSSLHeaders package also takes out of Indy all worry about support of link strategies and legacy versions. Indy need only support the OpenSSL 3.x API.

And that very well is a goal for IndyTLSOpenSSL v2.0. But I know there are a lot of Indy users, and a lot of legacy code out there. So there should be a stepping stone to prepare people for that migration. Hence IndyTLSOpenSSL v1.0. To make sure the package separation is clean and functional before then changing everything that is OpenSSL-related.

Do you want me to produce a pull request to add these headers to your proposed TLS package?

That would be ideal. Though there are already a few other PRs with different approaches, so weeding out what to merge is no simple task. And I really can't merge anything until Indy v10.7 and IndyTLSOpenSSL v1.0 are released first. I intended to do that over this recent holiday break, but that wasn't able to happen.

As for Indy 10.7, I would look to a more radical change than just making OpenSSL support a separate package.

That is not going to happen for 10.7, or likely any 10.x version for that matter. It's too radical a change. I'm already planning bigger reorganizational changes for 11.0. But that's been in development for a long time, so I'm starting to trickle some of the pending changes into 10.x in smaller releases.

My personal preference would be to create an Indy Core package supporting the http/https stack only.

That's not likely to happen. That would defeat the purpose of the Core package. HTTP/HTTPS are protocols. Core and Protocols should stay separate.

I would then create an Indy "extras" package with FTP, SMTP, POP3 and IMAP and perhaps Telnet, and then look very carefully at whether any other packages are required for the rest. (e.g. how many use Gopher nowadays?)

That, on the other hand, is a more viable idea, and one I've already been considering for awhile. The Protocols package is too large for its own good. Breaking it up is a goal for a future version.

[JPeterMugaas]

I have to admit that I did get impatient but I'm not the only one. I have to be honest in saying that it probably may be impossible to actually reconcile several approaches towards getting OpenSSL 3.x support in Indy at this point and part of this can be attributed to the complexity of OpenSSL itself. I just simply chose my own path. That may make me "part of the problem" but I rather think the approach I took with TaurusTLS is a very sound approach. I will toot my own horn here.

With TaurusTLS, I was able to the following:

1. I expanded the X509 object to have many more properties for displaying certificate information.
2. I used an older version of Tony Whyman's headers with my own modifications. In particular, I was able to solve the STACK_OF macros as well as make more functions available. I have not reviewed the latest work on that and at this point, I fear that I would lose some functionality if I simply adopted the new header versions, and it would be very complex to "switch" over to the newer versions of the code.
3. I was able to test the TLS code with several clients and servers. I focused on FTP since I also wrote a lot of that code.
4. TaurusTLS has an event for when the TLS negotiation is completed.
5. I simplified the API by removing the duplicate events. The TCallbackEvent is replaced with what was in TCallbackExEvent event and the TPasswordEvent is replaced with what was in the TPasswordEventEx event. The old OnStatusInfo event can be replaced with the new OnStatusInfo event and the OnSSLNegotiated as demonstrated in the TaurusTLS\demos\TaurusFTPClient\TaurusFTPClient.dproj demo project.
6. I use a feature in OpenSSL 1.1.1 called security levels. This is described at: https://docs.openssl.org/3.0/man3/SSL_CTX_set_security_level/#default-callback-behaviour . Basically, ciphers are rejected if they are too weak for a particular security level which the developer can control. If the developer wishes to accept or reject TLS attempts based on their own criterion, they can use the new OnSecurityLevel event to do this.
7. I have made a beta version of TaurusTLS at: https://github.com/JPeterMugaas/TaurusTLS/releases/tag/1.0.0.1beta1 .
8. TaurusTLS supports the new types defined in Indy 10.7 branch as well as supporting the shipping version of Indy in RAD Studio. I follow the Indy 10.7 Development and adjust TaurusTLS appropriately.

TaurusTLS is available at: https://github.com/JPeterMugaas/TaurusTLS or at https://github.com/TurboPack/indy_extras .

But there are some issues at my end. I have a limited amount of money, and I can't afford a MacIntosh, Android, and iOS devices so I'm not equipped to support those platforms although I am more than happy to accept contributions from others. I unlike other people have a lot of time I can devote to TaurusTLS.

[MWASoftware]

Peter, Can you check this. I did pick up most of the changes you sent me and incorporated several if not all the STACK_OF macros. Tony

…

On 06/01/2025 10:51, J. Peter Mugaas wrote: I used an older version of Tony Whyman's headers with my own modifications. In particular, I was able to solve the STACK_OF macros as well as make more functions available. I have not reviewed the latest work on that and at this point, I fear that I would lose some functionality if I simply adopted the new header versions, and it would be very complex to

[JPeterMugaas]

For the STACK_OF macros I went further and applied them to a number of OpenSSL "objects". Sample code is below (pasted right from TaurusTLS):

type
   PSTACK_OF_SSL_CIPHER = type pointer;

 {$IFNDEF OPENSSL_STATIC_LINK_MODEL}
type
  Tsk_SSL_CIPHER_new = function(cmp : TOPENSSL_sk_compfunc) : PSTACK_OF_SSL_CIPHER cdecl;
  Tsk_SSL_CIPHER_new_null = function : PSTACK_OF_SSL_CIPHER cdecl;
  Tsk_SSL_CIPHER_free = procedure(st : PSTACK_OF_SSL_CIPHER) cdecl;
  Tsk_SSL_CIPHER_num = function (const sk : PSTACK_OF_SSL_CIPHER) : TIdC_INT cdecl;
  Tsk_SSL_CIPHER_value = function (const sk : PSTACK_OF_SSL_CIPHER; i : TIdC_INT) : PSSL_CIPHER cdecl;
  Tsk_SSL_CIPHER_push = function (sk : PSTACK_OF_SSL_CIPHER; st : PSSL_CIPHER) : TIdC_INT cdecl;
  Tsk_SSL_CIPHER_dup = function (sk : PSTACK_OF_SSL_CIPHER) : PSTACK_OF_SSL_CIPHER cdecl;
  Tsk_SSL_CIPHER_find = function (sk : PSTACK_OF_SSL_CIPHER; _val : PSSL_CIPHER) : TIdC_INT cdecl;
  Tsk_SSL_CIPHER_pop_free = procedure (sk : PSTACK_OF_SSL_CIPHER; func: TOPENSSL_sk_freefunc) cdecl;

var
  sk_SSL_CIPHER_new: Tsk_SSL_CIPHER_new absolute sk_new;
  sk_SSL_CIPHER_new_null : Tsk_SSL_CIPHER_new_null absolute sk_new_null;
  sk_SSL_CIPHER_free : Tsk_SSL_CIPHER_free absolute sk_free;
  sk_SSL_CIPHER_num : Tsk_SSL_CIPHER_num absolute sk_num;
  sk_SSL_CIPHER_value : Tsk_SSL_CIPHER_value absolute sk_value;
  sk_SSL_CIPHER_push : Tsk_SSL_CIPHER_push absolute sk_push;
  sk_SSL_CIPHER_dup : Tsk_SSL_CIPHER_dup absolute sk_dup;
  sk_SSL_CIPHER_find : Tsk_SSL_CIPHER_find absolute sk_find;
  sk_SSL_CIPHER_pop_free :  Tsk_SSL_CIPHER_pop_free absolute sk_pop_free;
{$ELSE}
  function sk_SSL_CIPHER_new(cmp : Tsk_new_cmp) : PSTACK_OF_SSL_CIPHER cdecl; external CLibCrypto name 'OPENSSL_sk_new';
  function sk_SSL_CIPHER_new_null : PSTACK_OF_SSL_CIPHER cdecl; external CLibCrypto name 'OPENSSL_sk_new_null';
  procedure sk_SSL_CIPHER_free(st : PSTACK_OF_SSL_CIPHER) cdecl; external CLibCrypto name 'OPENSSL_sk_free';
  function sk_SSL_CIPHER_num (const sk : PSTACK_OF_SSL_CIPHER) : TIdC_INT cdecl; external CLibCrypto name 'OPENSSL_sk_num';
  function sk_SSL_CIPHER_value (const sk : PSTACK_OF_SSL_CIPHER; i : TIdC_INT): PSSL_CIPHER cdecl; external CLibCrypto name 'OPENSSL_sk_value';
  function sk_SSL_CIPHER_push (sk : PSTACK_OF_SSL_CIPHER; st : PSSL_CIPHER): TIdC_INT cdecl; external CLibCrypto name 'OPENSSL_sk_push';
  function sk_SSL_CIPHER_dup (sk : PSTACK_OF_SSL_CIPHER) : PSTACK_OF_SSL_CIPHER cdecl; external CLibCrypto name 'OPENSSL_sk_dup';
  function sk_SSL_CIPHER_find (sk : PSTACK_OF_SSL_CIPHER; val : PSSL_CIPHER) : TIdC_INT cdecl; external CLibCrypto name 'OPENSSL_sk_find';
  procedure sk_SSL_CIPHER_pop_free (sk : PSTACK_OF_SSL_CIPHER; func: Tsk_pop_free_func) cdecl; external CLibCrypto name 'OPENSSL_sk_pop_free';

{$ENDIF}

[MWASoftware]

Do you want me to produce a pull request to add these headers to your proposed TLS package?

That would be ideal. Though there are already a few other PRs with different approaches, so weeding out what to merge is no simple task. And I really can't merge anything until Indy v10.7 and IndyTLSOpenSSL v1.0 are released first. I intended to do that over this recent holiday break, but that wasn't able to happen.

I have proposed two pull requests. The first is on IndyTLSOpenSSL and copies all the OpenSSL files and Tests from my Indy.ProposedUpdate fork to IndyTLSOpenSSL.

The second is a pull request on IndySockets/Indy branch 10.7. This folds in the lazarus build files from my Indy.ProposedUpdate fork and is needed to test the IndyTLSOpenSSL pull request under lazarus/fpc - apart being generally useful.

[MWASoftware]

I did bring all of these into my OpenSSL Headers, albeit under their original names. The extract below is. taken from the template headers. The generated header contains both the var and the function call variants as well as compatibility functions.

function OPENSSL_sk_new(cmp:TOPENSSL_sk_compfunc):POPENSSL_STACK; {introduced 1.1.0}
function OPENSSL_sk_new_null:POPENSSL_STACK; {introduced 1.1.0}
function OPENSSL_sk_new_reserve(c:TOPENSSL_sk_compfunc; n:longint):POPENSSL_STACK; {introduced 1.1.0}
function OPENSSL_sk_reserve(st:POPENSSL_STACK; n:longint):longint; {introduced 1.1.0}
procedure OPENSSL_sk_free(_para1:POPENSSL_STACK); {introduced 1.1.0}
procedure OPENSSL_sk_pop_free(st:POPENSSL_STACK; func:TOPENSSL_sk_freefunc); {introduced 1.1.0}

For the STACK_OF macros I went further and applied them to a number of OpenSSL "objects". Sample code is below (pasted right from TaurusTLS):

type
   PSTACK_OF_SSL_CIPHER = type pointer;

 {$IFNDEF OPENSSL_STATIC_LINK_MODEL}
type
  Tsk_SSL_CIPHER_new = function(cmp : TOPENSSL_sk_compfunc) : PSTACK_OF_SSL_CIPHER cdecl;
  Tsk_SSL_CIPHER_new_null = function : PSTACK_OF_SSL_CIPHER cdecl;
  Tsk_SSL_CIPHER_free = procedure(st : PSTACK_OF_SSL_CIPHER) cdecl;
  Tsk_SSL_CIPHER_num = function (const sk : PSTACK_OF_SSL_CIPHER) : TIdC_INT cdecl;
  Tsk_SSL_CIPHER_value = function (const sk : PSTACK_OF_SSL_CIPHER; i : TIdC_INT) : PSSL_CIPHER cdecl;
  Tsk_SSL_CIPHER_push = function (sk : PSTACK_OF_SSL_CIPHER; st : PSSL_CIPHER) : TIdC_INT cdecl;
  Tsk_SSL_CIPHER_dup = function (sk : PSTACK_OF_SSL_CIPHER) : PSTACK_OF_SSL_CIPHER cdecl;
  Tsk_SSL_CIPHER_find = function (sk : PSTACK_OF_SSL_CIPHER; _val : PSSL_CIPHER) : TIdC_INT cdecl;
  Tsk_SSL_CIPHER_pop_free = procedure (sk : PSTACK_OF_SSL_CIPHER; func: TOPENSSL_sk_freefunc) cdecl;

var
  sk_SSL_CIPHER_new: Tsk_SSL_CIPHER_new absolute sk_new;
  sk_SSL_CIPHER_new_null : Tsk_SSL_CIPHER_new_null absolute sk_new_null;
  sk_SSL_CIPHER_free : Tsk_SSL_CIPHER_free absolute sk_free;
  sk_SSL_CIPHER_num : Tsk_SSL_CIPHER_num absolute sk_num;
  sk_SSL_CIPHER_value : Tsk_SSL_CIPHER_value absolute sk_value;
  sk_SSL_CIPHER_push : Tsk_SSL_CIPHER_push absolute sk_push;
  sk_SSL_CIPHER_dup : Tsk_SSL_CIPHER_dup absolute sk_dup;
  sk_SSL_CIPHER_find : Tsk_SSL_CIPHER_find absolute sk_find;
  sk_SSL_CIPHER_pop_free :  Tsk_SSL_CIPHER_pop_free absolute sk_pop_free;
{$ELSE}
  function sk_SSL_CIPHER_new(cmp : Tsk_new_cmp) : PSTACK_OF_SSL_CIPHER cdecl; external CLibCrypto name 'OPENSSL_sk_new';
  function sk_SSL_CIPHER_new_null : PSTACK_OF_SSL_CIPHER cdecl; external CLibCrypto name 'OPENSSL_sk_new_null';
  procedure sk_SSL_CIPHER_free(st : PSTACK_OF_SSL_CIPHER) cdecl; external CLibCrypto name 'OPENSSL_sk_free';
  function sk_SSL_CIPHER_num (const sk : PSTACK_OF_SSL_CIPHER) : TIdC_INT cdecl; external CLibCrypto name 'OPENSSL_sk_num';
  function sk_SSL_CIPHER_value (const sk : PSTACK_OF_SSL_CIPHER; i : TIdC_INT): PSSL_CIPHER cdecl; external CLibCrypto name 'OPENSSL_sk_value';
  function sk_SSL_CIPHER_push (sk : PSTACK_OF_SSL_CIPHER; st : PSSL_CIPHER): TIdC_INT cdecl; external CLibCrypto name 'OPENSSL_sk_push';
  function sk_SSL_CIPHER_dup (sk : PSTACK_OF_SSL_CIPHER) : PSTACK_OF_SSL_CIPHER cdecl; external CLibCrypto name 'OPENSSL_sk_dup';
  function sk_SSL_CIPHER_find (sk : PSTACK_OF_SSL_CIPHER; val : PSSL_CIPHER) : TIdC_INT cdecl; external CLibCrypto name 'OPENSSL_sk_find';
  procedure sk_SSL_CIPHER_pop_free (sk : PSTACK_OF_SSL_CIPHER; func: Tsk_pop_free_func) cdecl; external CLibCrypto name 'OPENSSL_sk_pop_free';

{$ENDIF}

[MWASoftware]

• I expanded the X509 object to have many more properties for displaying certificate information.

• TaurusTLS has an event for when the TLS negotiation is completed.

I can see the value of both of these additions. I carefully minimised the changes to the IdSSLOpenSSL unit in order to limit the scope of the change as well as to ensure maximum backwards compatible. However, both of the above are welcome additions including moving the X.509 code into a separate unit and I would like to fold them into my pull request. The only issue is when. I can also understand Remy's point about being overwhelmed by the size of a pull request.

I use a feature in OpenSSL 1.1.1 called security levels. This is described at: https://docs.openssl.org/3.0/man3/>SSL_CTX_set_security_level/#default-callback-behaviour . Basically, ciphers are rejected if they are too weak for a particular >security level which the developer can control. If the developer wishes to accept or reject TLS attempts based on their own >criterion, they can use the new OnSecurityLevel event to do this.

Again, this looks like a useful addition and I will need to look at it.