libtcnative w/custom OpenSSL on MacOS

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

libtcnative w/custom OpenSSL on MacOS

Christopher Schultz-2
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

All,

I'm trying to run tests locally (macos mojave) with a custom openssl
version. I have OpenSSL (installed via brew) in
/usr/local/Cellar/openssl@1.1/1.1.1, and I've copied all these files
into my Tomcat's bin/ directory, so this is what I have in
$CATALINA_HOME/bin:

> drwxr-xr-x  4       128 Sep 11 08:48 engines-1.1 -rwxr-xr-x  1
> 166112 Oct  9 16:17 libapr-1.0.dylib -rw-r--r--  1    288560 Oct  9
> 16:17 libapr-1.a lrwxr-xr-x  1        16 Oct  9 16:17
> libapr-1.dylib -> libapr-1.0.dylib -rw-r--r--  1   2432132 Sep 27
> 17:49 libcrypto.1.1.dylib -r--r--r--  1   4093208 Sep 11 08:48
> libcrypto.a lrwxr-xr-x  1        19 Sep 11 08:48 libcrypto.dylib ->
> libcrypto.1.1.dylib -rw-r--r--  1    489672 Sep 27 17:49
> libssl.1.1.dylib -r--r--r--  1    720096 Sep 11 08:48 libssl.a
> lrwxr-xr-x  1        16 Sep 11 08:48 libssl.dylib ->
> libssl.1.1.dylib -rwxr-xr-x  1    213716 Nov  5 10:50
> libtcnative-1.0.dylib -rw-r--r--  1   1097240 Nov  5 10:50
> libtcnative-1.a lrwxr-xr-x  1        21 Nov  5 10:50
> libtcnative-1.dylib -> libtcnative-1.0.dylib lrwxr-xr-x  1
> 19 Nov  5 10:50 libtcnative-1.la -> ../libtcnative-1.la -rw-r--r--
> 1      1091 Nov  5 10:50 libtcnative-1.lai

and also in engines-1.1:

- -r--r--r--  1    4240 Sep 27 17:49 capi.dylib
- -r--r--r--  1   13400 Sep 27 17:49 padlock.dylib

I have set in build.properties:

test.apr.loc=output/build/bin/

When running "ant test", the AprLifecycleListener is telling me:

>> OpenSSL successfully initialized [OpenSSL 1.1.1  11 Sep 2018]

... which looks like it's correct. But when e.g.
TestOpenSSLCipherConfigurationParser runs, I'm getting errors coming
from LibreSSL, which is the globally-installed crypto library
installed on macos:

> 4690560620:error:14FFF0B9:SSL routines:(UNKNOWN)SSL_internal:no
> cipher
> match:/BuildRoot/Library/Caches/com.apple.xbs/Sources/libressl/libress
l-22.200.4/libressl-2.6/ssl/ssl_lib.c:1324:

Obviously,
>
OpenSSL is not being used for everything.

otool tells me that everything looks okay:

> $ otool -L output/build/bin/libtcnative-1.dylib
> output/build/bin/libtcnative-1.dylib:
> /usr/local/apr/lib/libtcnative-1.0.dylib (compatibility version
> 3.0.0, current version 3.18.0)
> /usr/local/opt/openssl@1.1/lib/libssl.1.1.dylib (compatibility
> version 1.1.0, current version 1.1.0)
> /usr/local/opt/openssl@1.1/lib/libcrypto.1.1.dylib (compatibility
> version 1.1.0, current version 1.1.0)
> /usr/local/opt/apr/libexec/lib/libapr-1.0.dylib (compatibility
> version 7.0.0, current version 7.5.0) /usr/lib/libSystem.B.dylib
> (compatibility version 1.0.0, current version 1252.200.5)

What am I missing, here?

Thanks,
- -chris
-----BEGIN PGP SIGNATURE-----
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/

iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlvggYUACgkQHPApP6U8
pFh2SQ//eBdMeZ/n7BMp6OZlxEhrFr7DUwl8Q1oqJSS1g95xt7FWpBHCSa88xZGf
oKMdrG4VAJ5y64slBea2lf7Nf2QbOsVLwJBUcgJqjN+zTqVe0nVYTkwU7F7iFM4Y
pvW/cadPi1PO4WpHSynG8xKB2yUcXXzAIzCgcB6EXDZm/L9KHavCkAe+Sb6ZwQKe
ruhHYK9821OPeK9LBbOs+XG6dqGcnzycujrr+KZ88fu9/8NfA0Qxw3Y/5Zrtp39n
dxAGM2/SozLtmqhBTJI5uzd3y0HCe+m2ET1tIwbpjs/E/qwiHK+zARF7QEqsPGrw
PJsDB6BXNSTAbGeuF4QdPtAUWbemSWqHrnFlob686S8JIfGLyjuKNW9k+d2pNBjV
1l9+XK4EZ0hT1kw/FvM7DE3ICdc9LXQ0iN9urqrmOkf4W/9JiMNHEsNasQFd0GtF
yX9Q8vB93FqGK8moe6XtXnujZsv29BpO1wF40Lw4PJMQxYyY5c1GI4bv6p5aZilt
7YVy+Ox/zL0FD2GN4+vAwxsnlLAkNDNvMTzPO4BS/FuoAX5lTojVTAr/C1hWIiMR
TpE+h6P33TJ9qgLD6+dDDn08hE72+J105ZeHK725L2WXgBhnuQJprEdhYBQH5137
grp6uyV4OsXV1vNjPDw1mEiZhNsTPe9cValA1d8fkRfql3/rAY0=
=Whjs
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: libtcnative w/custom OpenSSL on MacOS

Rainer Jung-3
Hi Chris,

Am 05.11.2018 um 18:44 schrieb Christopher Schultz:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> All,
>
> I'm trying to run tests locally (macos mojave) with a custom openssl
> version. I have OpenSSL (installed via brew) in
> /usr/local/Cellar/openssl@1.1/1.1.1, and I've copied all these files
> into my Tomcat's bin/ directory, so this is what I have in
> $CATALINA_HOME/bin:
>
>> drwxr-xr-x  4       128 Sep 11 08:48 engines-1.1 -rwxr-xr-x  1
>> 166112 Oct  9 16:17 libapr-1.0.dylib -rw-r--r--  1    288560 Oct  9
>> 16:17 libapr-1.a lrwxr-xr-x  1        16 Oct  9 16:17
>> libapr-1.dylib -> libapr-1.0.dylib -rw-r--r--  1   2432132 Sep 27
>> 17:49 libcrypto.1.1.dylib -r--r--r--  1   4093208 Sep 11 08:48
>> libcrypto.a lrwxr-xr-x  1        19 Sep 11 08:48 libcrypto.dylib ->
>> libcrypto.1.1.dylib -rw-r--r--  1    489672 Sep 27 17:49
>> libssl.1.1.dylib -r--r--r--  1    720096 Sep 11 08:48 libssl.a
>> lrwxr-xr-x  1        16 Sep 11 08:48 libssl.dylib ->
>> libssl.1.1.dylib -rwxr-xr-x  1    213716 Nov  5 10:50
>> libtcnative-1.0.dylib -rw-r--r--  1   1097240 Nov  5 10:50
>> libtcnative-1.a lrwxr-xr-x  1        21 Nov  5 10:50
>> libtcnative-1.dylib -> libtcnative-1.0.dylib lrwxr-xr-x  1
>> 19 Nov  5 10:50 libtcnative-1.la -> ../libtcnative-1.la -rw-r--r--
>> 1      1091 Nov  5 10:50 libtcnative-1.lai
>
> and also in engines-1.1:
>
> - -r--r--r--  1    4240 Sep 27 17:49 capi.dylib
> - -r--r--r--  1   13400 Sep 27 17:49 padlock.dylib
>
> I have set in build.properties:
>
> test.apr.loc=output/build/bin/
>
> When running "ant test", the AprLifecycleListener is telling me:
>
>>> OpenSSL successfully initialized [OpenSSL 1.1.1  11 Sep 2018]
>
> ... which looks like it's correct. But when e.g.
> TestOpenSSLCipherConfigurationParser runs, I'm getting errors coming
> from LibreSSL, which is the globally-installed crypto library
> installed on macos:
>
>> 4690560620:error:14FFF0B9:SSL routines:(UNKNOWN)SSL_internal:no
>> cipher
>> match:/BuildRoot/Library/Caches/com.apple.xbs/Sources/libressl/libress
> l-22.200.4/libressl-2.6/ssl/ssl_lib.c:1324:
>
> Obviously,
>>
> OpenSSL is not being used for everything.
>
> otool tells me that everything looks okay:
>
>> $ otool -L output/build/bin/libtcnative-1.dylib
>> output/build/bin/libtcnative-1.dylib:
>> /usr/local/apr/lib/libtcnative-1.0.dylib (compatibility version
>> 3.0.0, current version 3.18.0)
>> /usr/local/opt/openssl@1.1/lib/libssl.1.1.dylib (compatibility
>> version 1.1.0, current version 1.1.0)
>> /usr/local/opt/openssl@1.1/lib/libcrypto.1.1.dylib (compatibility
>> version 1.1.0, current version 1.1.0)
>> /usr/local/opt/apr/libexec/lib/libapr-1.0.dylib (compatibility
>> version 7.0.0, current version 7.5.0) /usr/lib/libSystem.B.dylib
>> (compatibility version 1.0.0, current version 1252.200.5)
>
> What am I missing, here?

Try setting test.openssl.path in build.properties to the full path to
the openssl binary (.../bin/openssl).

See r1614560 and r1614587.

Regards,

Rainer

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: libtcnative w/custom OpenSSL on MacOS

Christopher Schultz-2
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256



On 11/5/18 13:05, Rainer Jung wrote:

> Hi Chris,
>
> Am 05.11.2018 um 18:44 schrieb Christopher Schultz:
>> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
>>
>> All,
>>
>> I'm trying to run tests locally (macos mojave) with a custom
>> openssl version. I have OpenSSL (installed via brew) in
>> /usr/local/Cellar/openssl@1.1/1.1.1, and I've copied all these
>> files into my Tomcat's bin/ directory, so this is what I have in
>> $CATALINA_HOME/bin:
>>
>>> drwxr-xr-x  4       128 Sep 11 08:48 engines-1.1 -rwxr-xr-x  1
>>> 166112 Oct  9 16:17 libapr-1.0.dylib -rw-r--r--  1    288560
>>> Oct  9 16:17 libapr-1.a lrwxr-xr-x  1        16 Oct  9 16:17
>>> libapr-1.dylib -> libapr-1.0.dylib -rw-r--r--  1   2432132 Sep
>>> 27 17:49 libcrypto.1.1.dylib -r--r--r--  1   4093208 Sep 11
>>> 08:48 libcrypto.a lrwxr-xr-x  1        19 Sep 11 08:48
>>> libcrypto.dylib -> libcrypto.1.1.dylib -rw-r--r--  1    489672
>>> Sep 27 17:49 libssl.1.1.dylib -r--r--r--  1    720096 Sep 11
>>> 08:48 libssl.a lrwxr-xr-x  1        16 Sep 11 08:48
>>> libssl.dylib -> libssl.1.1.dylib -rwxr-xr-x  1    213716 Nov  5
>>> 10:50 libtcnative-1.0.dylib -rw-r--r--  1   1097240 Nov  5
>>> 10:50 libtcnative-1.a lrwxr-xr-x  1        21 Nov  5 10:50
>>> libtcnative-1.dylib -> libtcnative-1.0.dylib lrwxr-xr-x  1 19
>>> Nov  5 10:50 libtcnative-1.la -> ../libtcnative-1.la
>>> -rw-r--r-- 1      1091 Nov  5 10:50 libtcnative-1.lai
>>
>> and also in engines-1.1:
>>
>> - -r--r--r--  1    4240 Sep 27 17:49 capi.dylib - -r--r--r--  1
>> 13400 Sep 27 17:49 padlock.dylib
>>
>> I have set in build.properties:
>>
>> test.apr.loc=output/build/bin/
>>
>> When running "ant test", the AprLifecycleListener is telling me:
>>
>>>> OpenSSL successfully initialized [OpenSSL 1.1.1  11 Sep
>>>> 2018]
>>
>> ... which looks like it's correct. But when e.g.
>> TestOpenSSLCipherConfigurationParser runs, I'm getting errors
>> coming from LibreSSL, which is the globally-installed crypto
>> library installed on macos:
>>
>>> 4690560620:error:14FFF0B9:SSL
>>> routines:(UNKNOWN)SSL_internal:no cipher
>>> match:/BuildRoot/Library/Caches/com.apple.xbs/Sources/libressl/libre
ss
>>
>>>
l-22.200.4/libressl-2.6/ssl/ssl_lib.c:1324:

>>
>> Obviously,
>>>
>> OpenSSL is not being used for everything.
>>
>> otool tells me that everything looks okay:
>>
>>> $ otool -L output/build/bin/libtcnative-1.dylib
>>> output/build/bin/libtcnative-1.dylib:
>>> /usr/local/apr/lib/libtcnative-1.0.dylib (compatibility
>>> version 3.0.0, current version 3.18.0)
>>> /usr/local/opt/openssl@1.1/lib/libssl.1.1.dylib (compatibility
>>> version 1.1.0, current version 1.1.0)
>>> /usr/local/opt/openssl@1.1/lib/libcrypto.1.1.dylib
>>> (compatibility version 1.1.0, current version 1.1.0)
>>> /usr/local/opt/apr/libexec/lib/libapr-1.0.dylib (compatibility
>>> version 7.0.0, current version 7.5.0)
>>> /usr/lib/libSystem.B.dylib (compatibility version 1.0.0,
>>> current version 1252.200.5)
>>
>> What am I missing, here?
>
> Try setting test.openssl.path in build.properties to the full path
> to the openssl binary (.../bin/openssl).
>
> See r1614560 and r1614587.

Aha! That was it!

I was confused because I was thinking that the version was being
properly-detected by Tomcat. But the tests were using the "openssl
ciphers" command to pull the lists of ciphers instead of doing it
using JNI.

Would it be worth it to use JNI to pull-back the list of supported
ciphers instead of running an external command?

- -chris
-----BEGIN PGP SIGNATURE-----
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/

iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlvgnnkACgkQHPApP6U8
pFhgOw/+OR102nB5cvil60WbW8k2NV3x8WlGIP6prx806XX0yFGvmeZCLHmKgvBS
aNYb1rljTjtNl71P/U3hmFoU/7wV4ocXhC/p+FmfMUh9i4dI7JXlu7fZJz4xBPUr
GkdiQP+dYpI9LRfXKUnEZu/ZoubASkmzMluQrbUwSrlvgpapplRqFCqkOk9uvshK
Yt7C+JXWtq4HAKP3jxTX4yLns9E5nQx8tbzkgeiGpedQEa7b76T7zS4mswHR6ET3
MDXsTqjy+V/8Zm9PCWUE/NLvObd5zjCYdoye5nPyew5XEwrDJhYd/c0u/jGA18kO
2u3z9sn1xAqhaX9VFsnSDgg9XGeOaogLG6lDMLZ5zEIHvdo9NDFIZ9IOSqTl4nDK
X8Z5rwnkdD4KioVPtNX4KE7n7E5xI22fwi0rFKmp5zicpRi1a6Y6Awm1uqgs4JSV
ZrtP1gopZlzM5+qoA1WMsqQ0/PwkZksAul0cOYHXA/DqURrRL2SJGuFD+5Fq/N9d
sVn2JfT3ezO5/xCdzVx/AGd8NRQbPoT1wHkKuAANhfT/gnQnZUXLqV/z+wK0aCY2
P6vmONuydMm0IrJOTzSC+5O1SYREQlFYFNI7CCBu176Tp/CritvFvNvE17e1e31P
NfpggxSO1Z7NpuezO5YKELSV41KdnOw+KTPXpkFtS2U6V3Pvfmg=
=Olxh
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: libtcnative w/custom OpenSSL on MacOS

markt
On 05/11/2018 19:48, Christopher Schultz wrote:
> On 11/5/18 13:05, Rainer Jung wrote:
>> Am 05.11.2018 um 18:44 schrieb Christopher Schultz:

<snip/>

>>> What am I missing, here?
>
>> Try setting test.openssl.path in build.properties to the full path
>> to the openssl binary (.../bin/openssl).
>
>> See r1614560 and r1614587.
>
> Aha! That was it!
>
> I was confused because I was thinking that the version was being
> properly-detected by Tomcat. But the tests were using the "openssl
> ciphers" command to pull the lists of ciphers instead of doing it
> using JNI.
>
> Would it be worth it to use JNI to pull-back the list of supported
> ciphers instead of running an external command?

The purpose of the tests is to ensure that the Tomcat code that
replicates OpenSSL's cipher selection behaves the same way as the latest
OpenSSL code. I don't see that it matters whether we determine the
OpenSSL behaviour via an external command or JNI.

The upside is more consistent tests and one less build parameter to
configure.

The downside is APR/native becomes required for those tests. Running
those tests for all three connectors is fairly pointless so only running
them with APR might be an upside.

I haven't checked to see if the API we'd need to use is accessible via
the current JNI API or whether we'd need to extend it.

Is it worth it? For me this in the the category of it looks to be a
worthwhile itch to scratch if someone wants to scratch it.

Mark

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: libtcnative w/custom OpenSSL on MacOS

Christopher Schultz-2
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Mark,

On 11/5/18 17:32, Mark Thomas wrote:

> On 05/11/2018 19:48, Christopher Schultz wrote:
>> On 11/5/18 13:05, Rainer Jung wrote:
>>> Am 05.11.2018 um 18:44 schrieb Christopher Schultz:
>
> <snip/>
>
>>>> What am I missing, here?
>>
>>> Try setting test.openssl.path in build.properties to the full
>>> path to the openssl binary (.../bin/openssl).
>>
>>> See r1614560 and r1614587.
>>
>> Aha! That was it!
>>
>> I was confused because I was thinking that the version was being
>> properly-detected by Tomcat. But the tests were using the
>> "openssl ciphers" command to pull the lists of ciphers instead of
>> doing it using JNI.
>>
>> Would it be worth it to use JNI to pull-back the list of
>> supported ciphers instead of running an external command?
>
> The purpose of the tests is to ensure that the Tomcat code that
> replicates OpenSSL's cipher selection behaves the same way as the
> latest OpenSSL code. I don't see that it matters whether we
> determine the OpenSSL behaviour via an external command or JNI.
>
> The upside is more consistent tests and one less build parameter
> to configure.

It also reduces potential confusion and unexpected failure. I didn't
realize that the unit tests were launching "openssl" using the default
shell PATH (which in my case obviously was running libressl's
utility), and so there was confusion between the JVM's view of the
world (through java.library.path) and Process.exec()'s view of the
world (through $PATH).

That may be splitting hairs mentioning that effect, but it's quite a
thick piece of hair IMO.

> The downside is APR/native becomes required for those tests.
> Running those tests for all three connectors is fairly pointless so
> only running them with APR might be an upside.
>
> I haven't checked to see if the API we'd need to use is accessible
> via the current JNI API or whether we'd need to extend it.
>
> Is it worth it? For me this in the the category of it looks to be
> a worthwhile itch to scratch if someone wants to scratch it.

I'll have a look at what is available and what must be available in
order to support it. On the up-side, the source code for the "ciphers"
command[1] is mercifully short and easy to understand.

- -chris

[1] https://github.com/openssl/openssl/blob/master/apps/ciphers.c
-----BEGIN PGP SIGNATURE-----
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/

iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlvhneoACgkQHPApP6U8
pFgBPQ//a3u1Dnl+aN7ZcHb+ziWlT31llONXr47CNj3CUXKjpDXxqcIq9LE8jJTV
rZ3IG/A5HXj5ZpiOpivgIb/95X1whNKtzStoNqNkveZLL1s4PmIgP/4sKQG0NtKc
qaqCul43Rrbss9Cd8n3ZEg056ACMTHL/h/5z5FWoJ3WABNJ5X9WMCTfayZWZq3pG
OoYh1wsKPAxDPmzhl/yPqXU7eWmuWFQGfkHx23By6Yi2EbXSZXyzAFc2RTQd6RfU
/C17R+Opo/2iCBjI/M4qCzyQPmP/IEOQzVZMPW1XZ439vp/jlPA912jCAmKiJpZD
JCt+y6RTsqSVczYxsC/5WU467WcJRqJRG8pM3cXFeGZbUL+J7Fv3x3b+cJtwX2E4
zZbM3fNlJ8SQ3AuXvCKGMtthM9djE6fqWtk33G7F9hZRIlB7Y3sFugUslmF7EbZA
DkHeOHoDb/GAWUfOq0DqJQWJp8NjV7pmFINtHmeOv3K2H1B/+ylMUFV3LxX27oCz
5GCeS36RkqUpj4DFz7O2EEzecMdwvgluAxQHKuG4/VY5QG6xtfXVv7aIkls7n6CP
qifgYypa1FUea6/++L8bjtFmSq1TfzyilCgLibttmvzkKQ7MoWHM2ih4E2BsnuzY
Ci/7E0ZVwchvCuCzPHxy+Yu/Nd7T1CWPchRVDsRLLH9ifJrg4l0=
=UYUZ
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]