Opened 3 years ago

Closed 3 years ago

#4922 closed enhancement (fixed)

OpenSSL-3.0.1

Reported by: Bruce Dubbs Owned by: lfs-book
Priority: normal Milestone: 11.1
Component: Book Version: git
Severity: normal Keywords:
Cc:

Description

New major version

OpenSSL 3.0

### Major changes between OpenSSL 1.1.1 and OpenSSL 3.0.0 [7 sep 2021]

  • Enhanced 'openssl list' with many new options.
  • Added migration guide to man7.
  • Implemented support for fully "pluggable" TLSv1.3 groups.
  • Added suport for Kernel TLS (KTLS).
  • Changed the license to the Apache License v2.0.
  • Moved all variations of the EVP ciphers CAST5, BF, IDEA, SEED, RC2, RC4, RC5, and DES to the legacy provider.
  • Moved the EVP digests MD2, MD4, MDC2, WHIRLPOOL and RIPEMD-160 to the legacy provider.
  • Added convenience functions for generating asymmetric key pairs.
  • Deprecated the OCSP_REQ_CTX type and functions.
  • Deprecated the EC_KEY and EC_KEY_METHOD types and functions.
  • Deprecated the RSA and RSA_METHOD types and functions.
  • Deprecated the DSA and DSA_METHOD types and functions.
  • Deprecated the DH and DH_METHOD types and functions.
  • Deprecated the ERR_load_ functions.
  • Remove the RAND_DRBG API.
  • Deprecated the ENGINE API.
  • Added OSSL_LIB_CTX, a libcrypto library context.
  • Added various _ex functions to the OpenSSL API that support using a non-default OSSL_LIB_CTX.
  • Interactive mode is removed from the 'openssl' program.
  • The X25519, X448, Ed25519, Ed448, SHAKE128 and SHAKE256 algorithms are included in the FIPS provider.
  • X509 certificates signed using SHA1 are no longer allowed at security level 1 or higher. The default security level for TLS is 1, so certificates signed using SHA1 are by default no longer trusted to authenticate servers or clients.
  • enable-crypto-mdebug and enable-crypto-mdebug-backtrace were mostly disabled; the project uses address sanitize/leak-detect instead.
  • Added a Certificate Management Protocol (CMP, RFC 4210) implementation also covering CRMF (RFC 4211) and HTTP transfer (RFC 6712). It is part of the crypto lib and adds a 'cmp' app with a demo configuration. All widely used CMP features are supported for both clients and servers.
  • Added a proper HTTP client supporting GET with optional redirection, POST, arbitrary request and response content types, TLS, persistent connections, connections via HTTP(s) proxies, connections and exchange via user-defined BIOs (allowing implicit connections), and timeout checks.
  • Added util/check-format.pl for checking adherence to the coding guidelines.
  • Added OSSL_ENCODER, a generic encoder API.
  • Added OSSL_DECODER, a generic decoder API.
  • Added OSSL_PARAM_BLD, an easier to use API to OSSL_PARAM.
  • Added error raising macros, ERR_raise() and ERR_raise_data().
  • Deprecated ERR_put_error(), ERR_get_error_line(), ERR_get_error_line_data(), ERR_peek_error_line_data(), ERR_peek_last_error_line_data() and ERR_func_error_string().
  • Added OSSL_PROVIDER_available(), to check provider availibility.
  • Added 'openssl mac' that uses the EVP_MAC API.
  • Added 'openssl kdf' that uses the EVP_KDF API.
  • Add OPENSSL_info() and 'openssl info' to get built-in data.
  • Add support for enabling instrumentation through trace and debug output.
  • Changed our version number scheme and set the next major release to 3.0.0
  • Added EVP_MAC, an EVP layer MAC API, and a generic EVP_PKEY to EVP_MAC bridge. Supported MACs are: BLAKE2, CMAC, GMAC, HMAC, KMAC, POLY1305and SIPHASH.
  • Removed the heartbeat message in DTLS feature.
  • Added EVP_KDF, an EVP layer KDF and PRF API, and a generic EVP_PKEY to EVP_KDF bridge. Supported KDFs are: HKDF, KBKDF, KRB5 KDF, PBKDF2, PKCS12 KDF, SCRYPT, SSH KDF, SSKDF, TLS1 PRF, X9.42 KDF and X9.63 KDF.
  • All of the low-level MD2, MD4, MD5, MDC2, RIPEMD160, SHA1, SHA224, SHA256, SHA384, SHA512 and Whirlpool digest functions have been deprecated.
  • All of the low-level AES, Blowfish, Camellia, CAST, DES, IDEA, RC2, RC4, RC5 and SEED cipher functions have been deprecated.
  • All of the low-level DH, DSA, ECDH, ECDSA and RSA public key functions have been deprecated.
  • SSL 3, TLS 1.0, TLS 1.1, and DTLS 1.0 only work at security level 0.
  • Added providers, a new pluggability concept that will replace the ENGINE API and ENGINE implementations.

Change History (37)

comment:1 by ken@…, 3 years ago

They also say (reformatted to not be on a silly superlong line) 

Most applications that worked with OpenSSL 1.1.1 will still work
unchanged and will simply need to be recompiled (although you may
see numerous compilation warnings about using deprecated APIs).

Some applications may need to make changes to compile and work
correctly, and many applications will need to be changed to avoid
the deprecations warnings. We have put together a migration guide
[https://www.openssl.org/docs/man3.0/man7/migration_guide.html]
to describe the major differences in OpenSSL 3.0 compared to
previous releases.

In other words, for anyone upgrading an existing (BLFS) system you will need to recompile everything which links to openssl before you can discard the currently-installed 1.1.1.

comment:2 by Xi Ruoyao, 3 years ago

Besides the afalg failure (we've documented in the book), 80-test_cmp_http.t also fails:

​https://github.com/openssl/openssl/issues/16546

comment:3 by Xi Ruoyao, 3 years ago

Below is a list of packages I've rebuilt with OpenSSL-3.0.0 (updating):

Now openssl-1.1 libs is moved out from /usr/lib on my system.

Last edited 3 years ago by Xi Ruoyao (previous) (diff)

comment:4 by ken@…, 3 years ago

Fedora will be documenting which of their packages have problems in ​https://bugzilla.redhat.com/show_bug.cgi?id=1825937

comment:5 by ken@…, 3 years ago

Perl module IO-Socket-SSL now fails 8 of 16 subtests in t/connectSSL-timeout.t At this stage I have no idea if it is usable.

comment:6 by ken@…, 3 years ago

Ruby-3.0.2 ftbfs 

ssl_pkey_rsa.c: In function 'Init_ossl_rsa':
ossl_pkey_rsa.c:885:58: error: 'RSA_SSLV23_PADDING' undeclared (first use in this function); did you mean 'RSA_PKCS1_PADDING'?
  885 | #define DefRSAConst(x) rb_define_const(cRSA, #x, INT2NUM(RSA_##x))
      |                                                          ^~~~

Searching led me to ​https://github.com/ruby/openssl/issues/369 (Ruby doesn't support openssl-3.0). Not a package I care greatly about, I only use it in one of my tests of the texmf scripts.

Looking at other matches which google found for me, after all the homebrew mac problems (they switched to openssl-3) was a patch at gentoo. But it doesn't fix the problem for me maybe it fixes a different problem. ​https://bugs.gentoo.org/attachment.cgi?id=739632

Last edited 3 years ago by ken@… (previous) (diff)

in reply to:  5 ; comment:7 by ken@…, 3 years ago

Replying to ken@…:

Perl module IO-Socket-SSL now fails 8 of 16 subtests in t/connectSSL-timeout.t At this stage I have no idea if it is usable.

In fact it fails other tests later in the suite. If the connection is fine, it works in normal usage, but one of my biber tests (using ​https:// to get a remote bib file, 'remote1' in my current latest-test tarball) sometimes fails (i.e. it failed once a while ago, and I guess the file was temporarily unavailable). The link seems to have provided some sort of 'not available' response or timeout with openssl-3, but that was interpreted as 'success, file downloaded' and then biber complained that a temporary file was corrupted. Unfortunately, that temporary file was deleted.

I've raised ​https://github.com/noxxi/p5-io-socket-ssl/issues/111

comment:8 by ken@…, 3 years ago

To avoid this being lost: I cannot build cbindgen (and therefore firefox). Raised as ​https://github.com/rust-lang/cargo/issues/10013 - some rust crates need to be fixed. I guess that might happen for 1.58.0 or 1.59.0.

in reply to:  7 ; comment:9 by ken@…, 3 years ago

Replying to ken@…:

Replying to ken@…: I've raised ​https://github.com/noxxi/p5-io-socket-ssl/issues/111

And there I was asked about the test in its dependency Net-SSLeay. I was allowing the tests in that to fail because we say that one external test *could* fail. In fact, with openssl-3.0.0 many tests now fail (5 tests ,37 subtests). I've now raised ​https://github.com/radiator-software/p5-net-ssleay/issues/330

in reply to:  9 ; comment:10 by ken@…, 3 years ago

Replying to ken@…:

Replying to ken@…:

Replying to ken@…: I've raised ​https://github.com/noxxi/p5-io-socket-ssl/issues/111

And there I was asked about the test in its dependency Net-SSLeay. I was allowing the tests in that to fail because we say that one external test *could* fail. In fact, with openssl-3.0.0 many tests now fail (5 tests ,37 subtests). I've now raised ​https://github.com/radiator-software/p5-net-ssleay/issues/330

Doh, I missed the patches earlier in the thread.

in reply to:  10 comment:11 by ken@…, 3 years ago

Replying to ken@…:

Replying to ken@…:

Replying to ken@…:

Replying to ken@…: I've raised ​https://github.com/noxxi/p5-io-socket-ssl/issues/111

And there I was asked about the test in its dependency Net-SSLeay. I was allowing the tests in that to fail because we say that one external test *could* fail. In fact, with openssl-3.0.0 many tests now fail (5 tests ,37 subtests). I've now raised ​https://github.com/radiator-software/p5-net-ssleay/issues/330

Doh, I missed the patches earlier in the thread.

With those, only 1 subtest fails in Net-SSLeay. No change to the failures in IO-Socket-SSL.

comment:12 by ken@…, 3 years ago

If this goes in, we should use the developer release of Net-SSLeay, currently ​https://metacpan.org/release/CHRISN/Net-SSLeay-1.91_01 which fixes all the tests in that for me.

IO-OpenSSL still broken if attempting to retrieve something returns soemthing other than 'success' (and is treated as success).

With cbindgen, if the cargo files have already been downloaded with a previous version of openssl I can build cbindgen. But not using only openssl-3.0.0 and rustc-1.56.0. I don't regard that dirty approach as suitable for the book, so as I said to Bruce, if this goes in then someone else will have to update firefox-91.3.0 on Tuesday.

comment:13 by ken@…, 3 years ago

Now that my outgoing mail *appears* to be working again (no idea what changed), I'll mention that the fix for cbindgen is upstream, and I've proved that using a sed to change the versions of the curl and curl-sys crates (to 0.4.40 and 0.4.50) in rust's src/tools/cargo/Cargo.toml does the job for me.

A while ago I mailed the editors that I didn't think updating to rustc-1.56.1 was important (although for anyone who codes in rust that is a good idea), but since the sed requires rustc to be rebuilt I think this would be a good time to update and remeasure. I assume builds are a little slower because of the extra lint. I'll also mention that although rustc becomes 1.56.1, cargo remains at 1.56.0.

Unfortunately, my fresh build to test this has ended up with an almost unusable xfce, so I can't measure this until I find out what broke (I'm guessing something using meson-0.60, but for the moment that is just a guess).

comment:14 by Bruce Dubbs, 3 years ago

Summary: OpenSSL-3.0.0 → OpenSSL-3.0.1

Now version 3.0.1.

comment:15 by ken@…, 3 years ago

I've today had a report that IO::Socket:SSL-2.073 contains small changes which should mean it works and tests fine with openssl-3.0.0. At the moment I have no time to test this.

comment:16 by Xi Ruoyao, 3 years ago

Ruby-3.1.0 works fine with OpenSSL-3.

comment:17 by Douglas R. Reno, 3 years ago

Ruby and stunnel have been upgraded.

comment:18 by Douglas R. Reno, 3 years ago

No tests failed on this build.

For anyone looking to subject their system to OpenSSL-3 *only* (no OpenSSL-1.1), run the following commands: 

rm -v /usr/lib/lib{crypto,ssl}.so*
rm -v /usr/lib/pkgconfig/{libcrypto,libssl,openssl}.pc
rm -v /usr/bin/{c_rehash,openssl}
rm -rfv /usr/lib/engines-1.1
rm -rfv /usr/include/openssl
rm -rfv /usr/share/doc/openssl-1.1.1m
rm -rfv /etc/ssl

Note that I'm doing this on a fresh system with systemd-250 coming along for the ride. I'll fix any problems that come up in BLFS too that I experience.

I think Bruce wants to drop OpenSSL-3.0.1 in on Friday

comment:19 by Douglas R. Reno, 3 years ago

Note that I'm still in chroot, just had the quick idea to run the tests for Python-3 as I rebuilt it.

This should be a non-issue, but bringing it up in case anyone else notices it. test_unicodedata fails due to me not having name resolution in chroot, but test_ssl fails because of an assertion failure when determining the OpenSSL version. I suspect that will be fixed in a later version of Python. Note that Fedora doesn't have a patch for it either (because it's just a test issue) 

404 tests OK.

2 tests failed:
    test_ssl test_unicodedata

1 test altered the execution environment:
    test_ftplib

20 tests skipped:
    test_devpoll test_gdb test_idle test_ioctl test_kqueue test_msilib
    test_nis test_ossaudiodev test_sqlite test_startfile test_tcl
    test_tix test_tk test_ttk_guionly test_ttk_textonly test_turtle
    test_winconsoleio test_winreg test_winsound test_zipfile64
0:03:17 load avg: 1.46
0:03:17 load avg: 1.46 Re-running failed tests in verbose mode
0:03:17 load avg: 1.46 Re-running test_ssl in verbose mode (matching: test_openssl_version)
test_ssl: testing with 'OpenSSL 3.0.1 14 Dec 2021' (3, 0, 0, 1, 0)
          under 'Linux-5.13.12-x86_64-with-glibc2.34'
          HAS_SNI = True
          OP_ALL = 0x80000050
          OP_NO_TLSv1_1 = 0x10000000
test_openssl_version (test.test_ssl.BasicSocketTests) ... test test_ssl failed
FAIL

======================================================================
FAIL: test_openssl_version (test.test_ssl.BasicSocketTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/sources/Python-3.10.1/Lib/test/test_ssl.py", line 543, in test_openssl_version
    self.assertTrue(
AssertionError: False is not true : ('OpenSSL 3.0.1 14 Dec 2021', (3, 0, 0, 1, 0), '0x30000010')

in reply to:  18 comment:20 by ken@…, 3 years ago

Replying to Douglas R. Reno:

I think Bruce wants to drop OpenSSL-3.0.1 in on Friday

My results with one of the biber tests to use a remote bib file (on openssl-3.0.0 with IO-Socket-SSL-2.073 were not any better than before that was fixed for openssl-3. But that test is for the few of us who build biber from source, and the temporary test files get deleted by biber so that there is nothing to examine apart from the report.

I'm inclined to remove that test from my latex tests tarball since there is no way of determining what goes wrong. So, no objection to the upgrade.

comment:21 by Douglas R. Reno, 3 years ago

wpa_supplicant does compile okay (has a ton of warnings), but has issues when connecting to some networks.

This is the error you'll get: 

OpenSSL: EVP_DigestInit_ex failed: error:0308010C:digital envelope routines::unsupported
tls_connection_set_params: Clearing pending SSL error: error:03000086:digital envelope routines::initialization error

I'll craft a patch to fix this since Fedora has fixes for it at ​https://src.fedoraproject.org/rpms/wpa_supplicant/tree/rawhide - I'll try: 

0001-OpenSSL-Allow-systemwide-secpolicy-overrides-for-TLS.patch
0001-EAP-TTLS-PEAP-peer-Fix-failure-when-using-session-ti.patch 
0001-openssl-Disable-padding-after-initializing-the-ciphe.patch 
0001-openssl-Remove-deprecated-functions-from-des_encrypt.patch

comment:22 by Douglas R. Reno, 3 years ago

My patch does not fix all of the deprecation warnings, but does allow wpa_supplicant to connect to my network again. I'll drop that in shortly, just want to build a couple of other things first.

comment:23 by Douglas R. Reno, 3 years ago

Python 2 has some major breakage with OpenSSL-3. It'll still build the _ssl module, but it thinks the SSL version is 0000000000.

An example of a test failure: 

Exception in thread Thread-219:
Traceback (most recent call last):
  File "/sources/Python-2.7.18/Python-2.7.18/Lib/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/sources/Python-2.7.18/Python-2.7.18/Lib/test/test_ftplib.py", line 245, in run
    asyncore.loop(timeout=0.1, count=1)
  File "/sources/Python-2.7.18/Python-2.7.18/Lib/asyncore.py", line 220, in loop
    poll_fun(timeout, map)
  File "/sources/Python-2.7.18/Python-2.7.18/Lib/asyncore.py", line 156, in poll
    read(obj)
  File "/sources/Python-2.7.18/Python-2.7.18/Lib/asyncore.py", line 87, in read
    obj.handle_error()
  File "/sources/Python-2.7.18/Python-2.7.18/Lib/asyncore.py", line 83, in read
    obj.handle_read_event()
  File "/sources/Python-2.7.18/Python-2.7.18/Lib/test/test_ftplib.py", line 331, in handle_read_event
    super(SSLConnection, self).handle_read_event()
  File "/sources/Python-2.7.18/Python-2.7.18/Lib/asyncore.py", line 449, in handle_read_event
    self.handle_read()
  File "/sources/Python-2.7.18/Python-2.7.18/Lib/asynchat.py", line 119, in handle_read
    self.handle_error()
  File "/sources/Python-2.7.18/Python-2.7.18/Lib/asynchat.py", line 115, in handle_read
    data = self.recv (self.ac_in_buffer_size)
  File "/sources/Python-2.7.18/Python-2.7.18/Lib/test/test_ftplib.py", line 353, in recv
    return super(SSLConnection, self).recv(buffer_size)
  File "/sources/Python-2.7.18/Python-2.7.18/Lib/asyncore.py", line 387, in recv
    data = self.socket.recv(buffer_size)
  File "/sources/Python-2.7.18/Python-2.7.18/Lib/ssl.py", line 754, in recv
    return self.read(buflen)
  File "/sources/Python-2.7.18/Python-2.7.18/Lib/ssl.py", line 641, in read
    v = self._sslobj.read(len)
error: [Errno 0] Error

Because of the 00000000, it also skips several important modules, such as _hashlib.

These are the test results: 

== Tests result: FAILURE ==

357 tests OK.

3 tests failed:
    test_ftplib test_ssl test_urllib2_localnet

44 tests skipped:
    test_aepack test_al test_applesingle test_bsddb test_bsddb185
    test_bsddb3 test_cd test_cl test_codecmaps_cn test_codecmaps_hk
    test_codecmaps_jp test_codecmaps_kr test_codecmaps_tw test_curses
    test_dl test_gdb test_gl test_idle test_imageop test_imgfile
    test_kqueue test_linuxaudiodev test_macos test_macostools
    test_msilib test_nis test_ossaudiodev test_scriptpackages
    test_smtpnet test_socketserver test_sqlite test_startfile
    test_sunaudiodev test_tcl test_timeout test_tk test_ttk_guionly
    test_ttk_textonly test_turtle test_urllib2net test_urllibnet
    test_winreg test_winsound test_zipfile64
9 skips unexpected on linux2:
    test_bsddb test_bsddb3 test_gdb test_idle test_tcl test_tk
    test_ttk_guionly test_ttk_textonly test_turtle

Total duration: 9 min 59 sec
Tests result: FAILURE

I'll continue as is, but I suspect this will probably break at least nmap.

comment:24 by Douglas R. Reno, 3 years ago

With libarchive, I can confirm that the tests segfault as Xi described: 

./build/autoconf/test-driver: line 107: 242883 Segmentation fault      (core dumped) "$@" > $log_file 2>&1
FAIL: libarchive_test

sed -i '436a if ((OSSL_PROVIDER_load(NULL, "legacy")) == NULL)\n return(ARCHIVE_FAILED);' libarchive/archive_digest.c

Seems to fix it for me. :)

comment:25 by Douglas R. Reno, 3 years ago

libevent does appear to have problems (at least if we ask the test suite): 

./test/test.sh -b EPOLL
Running tests:
EPOLL 
 test-eof: OKAY
 test-closed: OKAY
 test-weof: OKAY
 test-time: OKAY
 test-changelist: OKAY
 test-fdleak: OKAY
 test-dumpevents: OKAY (output not checked)
 regress: [warn] getaddrinfo: address family for nodename not supported
[warn] getaddrinfo: address family for nodename not supported

  FAIL test/regress_http.c:3154: assert(test_ok == 2): -2 vs 2http/https_incomplete: 
  [https_incomplete FAILED]

  FAIL test/regress_http.c:3154: assert(test_ok == 2): -2 vs 2http/https_incomplete_timeout: 
  [https_incomplete_timeout FAILED]

  FAIL test/regress_http.c:3363: assert(test_ok == 2): -1 vs 2http/https_chunk_out: 
  [https_chunk_out FAILED]

  FAIL test/regress_dns.c:2105: assert(gaic_freed != 1000): 1000 vs 1000dns/getaddrinfo_cancel_stress: 
  [getaddrinfo_cancel_stress FAILED]

  FAIL test/regress_ssl.c:538: assert(got_close == 1): 0 vs 1ssl/bufferevent_socketpair_dirty_shutdown: 
  [bufferevent_socketpair_dirty_shutdown FAILED]

  FAIL test/regress_ssl.c:538: assert(got_close == 1): 0 vs 1ssl/bufferevent_renegotiate_socketpair_dirty_shutdown: 
  [bufferevent_renegotiate_socketpair_dirty_shutdown FAILED]

  FAIL test/regress_ssl.c:538: assert(got_close == 1): 0 vs 1ssl/bufferevent_socketpair_startopen_dirty_shutdown: 
  [bufferevent_socketpair_startopen_dirty_shutdown FAILED]
7/351 TESTS FAILED. (39 skipped)
FAILED

It is known upstream at ​https://github.com/libevent/libevent/issues/1233 - whether we use the functions impacted in libevent is unknown. I'm going to press on and I'll follow the issue on Github so we can get fixes as soon as they're available.

comment:26 by Douglas R. Reno, 3 years ago

Regarding Net-SSLeay, I can definitely confirm that something's wrong in it's current state: 

Test Summary Report
-------------------
t/local/33_x509_create_cert.t        (Wstat: 256 Tests: 139 Failed: 1)
  Failed test:  37
  Non-zero exit status: 1
t/local/39_pkcs12.t                  (Wstat: 512 Tests: 19 Failed: 2)
  Failed tests:  11-12
  Non-zero exit status: 2
t/local/43_misc_functions.t          (Wstat: 512 Tests: 46 Failed: 2)
  Failed tests:  4, 9
  Non-zero exit status: 2
t/local/44_sess.t                    (Wstat: 4608 Tests: 58 Failed: 18)
  Failed tests:  2-4, 6-10, 12, 14-16, 18-22, 24
  Non-zero exit status: 18
t/local/45_exporter.t                (Wstat: 3584 Tests: 36 Failed: 14)
  Failed tests:  1-5, 7, 9-14, 16, 18
  Non-zero exit status: 14
Files=40, Tests=1805,  3 wallclock secs ( 0.10 usr  0.02 sys +  3.14 cusr  0.28 csys =  3.54 CPU)
Result: FAIL
Failed 5/40 test programs. 37/1805 subtests failed.
make: *** [Makefile:1110: test_dynamic] Error 255
renodr [ /sources/Net-SSLeay-1.90 ]$

I'm going to try those patches Xi linked above next and see how that goes.

in reply to:  26 ; comment:27 by ken@…, 3 years ago

Replying to Douglas R. Reno:

Regarding Net-SSLeay, I can definitely confirm that something's wrong in it's current state: 

Test Summary Report
-------------------
t/local/33_x509_create_cert.t        (Wstat: 256 Tests: 139 Failed: 1)
  Failed test:  37
  Non-zero exit status: 1
t/local/39_pkcs12.t                  (Wstat: 512 Tests: 19 Failed: 2)
  Failed tests:  11-12
  Non-zero exit status: 2
t/local/43_misc_functions.t          (Wstat: 512 Tests: 46 Failed: 2)
  Failed tests:  4, 9
  Non-zero exit status: 2
t/local/44_sess.t                    (Wstat: 4608 Tests: 58 Failed: 18)
  Failed tests:  2-4, 6-10, 12, 14-16, 18-22, 24
  Non-zero exit status: 18
t/local/45_exporter.t                (Wstat: 3584 Tests: 36 Failed: 14)
  Failed tests:  1-5, 7, 9-14, 16, 18
  Non-zero exit status: 14
Files=40, Tests=1805,  3 wallclock secs ( 0.10 usr  0.02 sys +  3.14 cusr  0.28 csys =  3.54 CPU)
Result: FAIL
Failed 5/40 test programs. 37/1805 subtests failed.
make: *** [Makefile:1110: test_dynamic] Error 255
renodr [ /sources/Net-SSLeay-1.90 ]$

I'm going to try those patches Xi linked above next and see how that goes.

Please use 1.91_01. That is technically a development release but all tests passed for me with openssl-3.0.0. It didn't help with the old version of the other module, so since we are still on 1.1.1 I haven't put it in the book.

in reply to:  27 comment:28 by Douglas R. Reno, 3 years ago

Replying to ken@…:

Replying to Douglas R. Reno:

Regarding Net-SSLeay, I can definitely confirm that something's wrong in it's current state: 

Test Summary Report
-------------------
t/local/33_x509_create_cert.t        (Wstat: 256 Tests: 139 Failed: 1)
  Failed test:  37
  Non-zero exit status: 1
t/local/39_pkcs12.t                  (Wstat: 512 Tests: 19 Failed: 2)
  Failed tests:  11-12
  Non-zero exit status: 2
t/local/43_misc_functions.t          (Wstat: 512 Tests: 46 Failed: 2)
  Failed tests:  4, 9
  Non-zero exit status: 2
t/local/44_sess.t                    (Wstat: 4608 Tests: 58 Failed: 18)
  Failed tests:  2-4, 6-10, 12, 14-16, 18-22, 24
  Non-zero exit status: 18
t/local/45_exporter.t                (Wstat: 3584 Tests: 36 Failed: 14)
  Failed tests:  1-5, 7, 9-14, 16, 18
  Non-zero exit status: 14
Files=40, Tests=1805,  3 wallclock secs ( 0.10 usr  0.02 sys +  3.14 cusr  0.28 csys =  3.54 CPU)
Result: FAIL
Failed 5/40 test programs. 37/1805 subtests failed.
make: *** [Makefile:1110: test_dynamic] Error 255
renodr [ /sources/Net-SSLeay-1.90 ]$

I'm going to try those patches Xi linked above next and see how that goes.

Please use 1.91_01. That is technically a development release but all tests passed for me with openssl-3.0.0. It didn't help with the old version of the other module, so since we are still on 1.1.1 I haven't put it in the book.

I will do that, but ​https://github.com/radiator-software/p5-net-ssleay/pull/297/files seemed like it might help.

in reply to:  27 ; comment:29 by Douglas R. Reno, 3 years ago

Replying to ken@…:

Replying to Douglas R. Reno:

Regarding Net-SSLeay, I can definitely confirm that something's wrong in it's current state: 

Test Summary Report
-------------------
t/local/33_x509_create_cert.t        (Wstat: 256 Tests: 139 Failed: 1)
  Failed test:  37
  Non-zero exit status: 1
t/local/39_pkcs12.t                  (Wstat: 512 Tests: 19 Failed: 2)
  Failed tests:  11-12
  Non-zero exit status: 2
t/local/43_misc_functions.t          (Wstat: 512 Tests: 46 Failed: 2)
  Failed tests:  4, 9
  Non-zero exit status: 2
t/local/44_sess.t                    (Wstat: 4608 Tests: 58 Failed: 18)
  Failed tests:  2-4, 6-10, 12, 14-16, 18-22, 24
  Non-zero exit status: 18
t/local/45_exporter.t                (Wstat: 3584 Tests: 36 Failed: 14)
  Failed tests:  1-5, 7, 9-14, 16, 18
  Non-zero exit status: 14
Files=40, Tests=1805,  3 wallclock secs ( 0.10 usr  0.02 sys +  3.14 cusr  0.28 csys =  3.54 CPU)
Result: FAIL
Failed 5/40 test programs. 37/1805 subtests failed.
make: *** [Makefile:1110: test_dynamic] Error 255
renodr [ /sources/Net-SSLeay-1.90 ]$

I'm going to try those patches Xi linked above next and see how that goes.

Please use 1.91_01. That is technically a development release but all tests passed for me with openssl-3.0.0. It didn't help with the old version of the other module, so since we are still on 1.1.1 I haven't put it in the book.

I dropped the new Net-SSLeay in, and I did also test IO-Socket-SSL (but I'd prefer you updated that). Everything looks good here

in reply to:  29 comment:30 by ken@…, 3 years ago

Replying to Douglas R. Reno:

Please use 1.91_01. That is technically a development release but all tests passed for me with openssl-3.0.0. It didn't help with the old version of the other module, so since we are still on 1.1.1 I haven't put it in the book.

I dropped the new Net-SSLeay in, and I did also test IO-Socket-SSL (but I'd prefer you updated that). Everything looks good here

I haven't given up on my IO-Socket-SSL|biber problem, just lacking time.

comment:31 by Douglas R. Reno, 3 years ago

A few quick updates:

  • OpenLDAP's client built. Server untested, will do that later (after GNOME-41).
  • Cyrus SASL builds fine with tons of compilation warnings (over 300). Seems to work well for me, but I did check Fedora's repository and they have several OpenSSL-3 related fixes... but I'm not sure if they affect us. Still though, if someone wants me to apply those patches, let me know.
  • Sendmail works fine, which is not something I was honestly expecting. It looks like they added OpenSSL-3 compatibility back when OpenSSL-3 was still in alpha.

It looks like the next package I could hit an issue with is MIT Kerberos 5. I'm almost to systemd and will hopefully have that (and meson) in tonight, pending no problems with rustc or any other packages I need to build of course!

comment:32 by Douglas R. Reno, 3 years ago

For krb5, I can confirm Xi's report regarding build failures if -Werror=discarded-qualifiers is set, and the sed does fix that problem. The test suite still exhibits the same behavior related to DejaGNU, and almost all of the TCL+Expect tests are broken due to slight differences in output from OpenSSL-1.1.1 to OpenSSL-3.0.

I was able to successfully test the package using the configuration in the book.

If we do encounter any more issues with this particular package, both Fedora and upstream have at least 8 patches for it. Most of them seem to be related to removing deprecated code and some cryptography-related refactoring, but there is also the removal of the kadm5 TCL/dejagnu tests and replacing them with python-based tests.

I have also started kdc, kadmin, and kpropd using their respective systemd units and didn't notice any difference in output when comparing to the same on my development machine (other than a message complaining about dictionaries not being available, but I don't have cracklib at this stage so I don't have anything in /usr/share/dict).

In terms of the python tests, I do get an extra failure on OpenSSL-3.0 that I didn't receive on OpenSSL-1.1: 

renodr [ /sources/krb5-1.19.2/krb5-1.19.2/src/tests ]$ sudo PYTHONPATH=/sources/krb5-1.19.2/krb5-1.19.2/src/util /bin/python3 ./t_pkinit.py -v
*** [1] Starting: /sources/krb5-1.19.2/krb5-1.19.2/src/kdc/krb5kdc -n
Stash file /sources/krb5-1.19.2/krb5-1.19.2/src/tests/testdir/stash uses DEPRECATED enctype !
Stash file /sources/krb5-1.19.2/krb5-1.19.2/src/tests/testdir/stash uses DEPRECATED enctype !
krb5kdc: starting...
*** [1] Started with pid 1385598

====== UPN SANs ======
*** [2] Executing: /sources/krb5-1.19.2/krb5-1.19.2/src/clients/kinit/kinit -X X509_user_identity=PKCS12:/sources/krb5-1.19.2/krb5-1.19.2/src/tests/dejagnu/pkinit-certs/user-upn2.p12 user@KRBTEST.COM
Pass phrase for /sources/krb5-1.19.2/krb5-1.19.2/src/tests/dejagnu/pkinit-certs/user-upn2.p12: 
Password for user@KRBTEST.COM: 
kinit: Pre-authentication failed: Preauthentication failed while getting initial credentials
*** [2] Completed with return code 1
*** Failure: /sources/krb5-1.19.2/krb5-1.19.2/src/clients/kinit/kinit failed with code 1.
*** Last mark: UPN SANs
*** Last command (#2): /sources/krb5-1.19.2/krb5-1.19.2/src/clients/kinit/kinit -X X509_user_identity=PKCS12:/sources/krb5-1.19.2/krb5-1.19.2/src/tests/dejagnu/pkinit-certs/user-upn2.p12 user@KRBTEST.COM
*** Output of last command:
Pass phrase for /sources/krb5-1.19.2/krb5-1.19.2/src/tests/dejagnu/pkinit-certs/user-upn2.p12: 
Password for user@KRBTEST.COM: 
kinit: Pre-authentication failed: Preauthentication failed while getting initial credentials
Use --debug=NUM to run a command under a debugger.  Use
--stop-after=NUM to stop after a daemon is started in order to
attach to it with a debugger.  Use --help to see other
options.

This is indicative of a problem, however I'm unable to reproduce it using the configuration in the book. For example: 

renodr [ /sources/krb5-1.19.2/krb5-1.19.2/src/tests/dejagnu/pkinit-certs ]$ sudo systemctl start krb5-kdc
renodr [ /sources/krb5-1.19.2/krb5-1.19.2/src/tests/dejagnu/pkinit-certs ]$ sudo systemctl start krb5-kadmind
renodr [ /sources/krb5-1.19.2/krb5-1.19.2/src/tests/dejagnu/pkinit-certs ]$ sudo systemctl start krb5-kpropd
renodr [ /sources/krb5-1.19.2/krb5-1.19.2/src/tests/dejagnu/pkinit-certs ]$ kinit
Password for renodr@renospecialties.net: 
renodr [ /sources/krb5-1.19.2/krb5-1.19.2/src/tests/dejagnu/pkinit-certs ]$ klist
Ticket cache: FILE:/tmp/krb5cc_1000
Default principal: renodr@renospecialties.net

Valid starting       Expires              Service principal
12/29/2021 18:35:54  12/30/2021 18:35:54  krbtgt/renospecialties.net@renospecialties.net

But at the same time, this behavior may only exhibit itself if you're passing a certificate to Kerberos as an authentication mechanism - something which other packages in the book could do (such as Evolution, gnome-online-accounts, pidgin, or Wireshark).

Further investigation shows that the certificate does exist, but the way Kerberos generated it *is not valid and uses a removed algorithm*: 

renodr [ /sources/krb5-1.19.2/krb5-1.19.2/src/tests/dejagnu/pkinit-certs ]$ openssl pkcs12 -info -nodes -in user-upn2.p12 
Enter Import Password:
MAC: sha1, Iteration 2048
MAC length: 20, salt length: 8
PKCS7 Encrypted data: pbeWithSHA1And40BitRC2-CBC, Iteration 2048
Error outputting keys and certificates
4017C5FE057F0000:error:0308010C:digital envelope routines:inner_evp_generic_fetch:unsupported:crypto/evp/evp_fetch.c:349:Global default library context, Algorithm (RC2-40-CBC : 0), Properties ()

The generic file: 

renodr [ /sources/krb5-1.19.2/krb5-1.19.2/src/tests/dejagnu/pkinit-certs ]$ openssl pkcs12 -info -nodes -in generic.p12 
Enter Import Password:
MAC: sha1, Iteration 2048
MAC length: 20, salt length: 8
PKCS7 Encrypted data: pbeWithSHA1And40BitRC2-CBC, Iteration 2048
Error outputting keys and certificates
4067339F287F0000:error:0308010C:digital envelope routines:inner_evp_generic_fetch:unsupported:crypto/evp/evp_fetch.c:349:Global default library context, Algorithm (RC2-40-CBC : 0), Properties ()

Because of that, I'm inclined to add the patches in, but without the removal of the TCL tests. I'll do that after dinner here.

comment:33 by Douglas R. Reno, 3 years ago

Over at Fedora, I'm going to try the patches in the following order: 

Patch21: Fix-softpkcs11-build-issues-with-openssl-3.0.patch
Patch22: Remove-deprecated-OpenSSL-calls-from-softpkcs11.patch
Patch23: Fix-k5tls-module-for-OpenSSL-3.patch
Patch32: Add-buildsystem-detection-of-the-OpenSSL-3-KDF-inter.patch
Patch33: Use-OpenSSL-s-SSKDF-in-PKINIT-when-available.patch
Patch34: Use-OpenSSL-s-KBKDF-and-KRB5KDF-for-deriving-long-te.patch
Patch35: Handle-OpenSSL-3-s-providers.patch

I suspect the providers patch may be the only one that is actually required, but it doesn't hurt to have the others taken care of as well. Maybe we can also do without the sed to configure (since we'll need to run an autoreconf anyway).

comment:34 by Douglas R. Reno, 3 years ago

Ouch, that took a lot longer than I was expecting and was painful.

I ended up needing to apply all seven of those patches from Fedora. Note that this does get rid of the compiler errors/warnings that Xi's sed silenced too, and fixes tons of deprecation issues.

After creating the patch, I ended up having to make several modifications to get Kerberos to build again. Fedora assumes that you're going to be using the OpenSSL backend for *tls* and *crypto* instead of Kerberos' internal backend. The advantage for using OpenSSL for these backends is that, if you have FIPS enabled (which is very uncommon and is barred from export in some countries), Kerberos can use it. However, because their patches assume that you're using those backends in Kerberos, it throws a compiler error because of undefined variables in a function that is only supposed to be used on systems that have those backends. To work around that, I #ifdef-ed it out because we're using Kerberos' internal backends for TLS and cryptography (which is also what Arch and most other distributions do).

Tests appear to be much more stable throughout, but that one Python test is still having issues. It appears to be an issue with the Dejagnu/TCL section of the test suite failing and not creating users like it is supposed to (it even notes that 'lib.t' isn't in the search path for dejagnu). However, this patch is long enough - I don't think we really need to worry about it. We already have our bases covered with the line "Some tests may fail with the latest version of dejagnu and glibc.", and removing the TCL tests (and modifying the others) would add over 18,000 lines to the patch.

comment:35 by Douglas R. Reno, 3 years ago

Attempting to build subversion will result in an undefined reference to ERR_GET_FUNC in libserf-1.so: 

/bin/sh "/sources/subversion-1.14.1/subversion-1.14.1/libtool" --tag=CC --silent --mode=compile gcc -std=c90  -DLINUX -D_REENTRANT -D_GNU_SOURCE   -g -O2  -g -O2   -I./subversion/include -I./subversion -I/usr/include/apr-1   -I/usr/include/apr-1 -I/usr/include -I/usr/include/serf-1   -o subversion/svn/util.lo -c subversion/svn/util.c
cd subversion/svn && /bin/sh "/sources/subversion-1.14.1/subversion-1.14.1/libtool" --tag=CC --silent --mode=link gcc -shared  -g -O2  -g -O2    -rpath /usr/lib  -o svn  add-cmd.lo auth-cmd.lo blame-cmd.lo cat-cmd.lo changelist-cmd.lo checkout-cmd.lo cl-conflicts.lo cleanup-cmd.lo commit-cmd.lo conflict-callbacks.lo copy-cmd.lo delete-cmd.lo deprecated.lo diff-cmd.lo export-cmd.lo file-merge.lo filesize.lo help-cmd.lo import-cmd.lo info-cmd.lo list-cmd.lo lock-cmd.lo log-cmd.lo merge-cmd.lo mergeinfo-cmd.lo mkdir-cmd.lo move-cmd.lo notify.lo patch-cmd.lo propdel-cmd.lo propedit-cmd.lo propget-cmd.lo proplist-cmd.lo props.lo propset-cmd.lo relocate-cmd.lo resolve-cmd.lo resolved-cmd.lo revert-cmd.lo shelf-cmd.lo shelf2-cmd.lo similarity.lo status-cmd.lo status.lo svn.lo switch-cmd.lo unlock-cmd.lo update-cmd.lo upgrade-cmd.lo util.lo ../../subversion/libsvn_client/libsvn_client-1.la ../../subversion/libsvn_wc/libsvn_wc-1.la ../../subversion/libsvn_ra/libsvn_ra-1.la ../../subversion/libsvn_delta/libsvn_delta-1.la ../../subversion/libsvn_diff/libsvn_diff-1.la ../../subversion/libsvn_subr/libsvn_subr-1.la -laprutil-1 -lapr-1 
/bin/ld: /usr/lib/libserf-1.so: undefined reference to `ERR_GET_FUNC'
collect2: error: ld returned 1 exit status
make: *** [build-outputs.mk:900: subversion/svn/svn] Error 1
81.0 Elasped Time - subversion-1.14.1
renodr [ /sources ]$

This appears to be due to serf using an OpenSSL function for non-SSL-related things. To work around it, we need to add a copy of ERR_GET_FUNC to buckets/ssl_buckets.c in serf. Note that the bio_bucket_ctrl() function is broken in libserf-1.so as well with OpenSSL-3.

Patches are available and I will begin working on crafting one now. I'll drop it in once I can confirm that Subversion builds and tests properly.

comment:36 by Douglas R. Reno, 3 years ago

With the patch for serf installed, subversion both builds and tests successfully (no failed tests!)

Dropping the patch in now.

comment:37 by Bruce Dubbs, 3 years ago

Resolution: → fixed
Status: new → closed

Fixed at commit ad89f7b0d63fac3c1bca0e5d11880d97c511f84e 

Package updates.
    Update to e2fsprogs-1.46.5.
    Update to zstd-1.5.1.
    Update to expat-2.4.2.
    Update to shadow-4.10.
    Update to sysvinit-3.01.
    Update to linux-5.15.12.
    Update to iana-etc-20211224.
    Update to openssl-3.0.1.
    Update to eudev-3.2.11.
    Update lfs-latest-git.php currency for new eudev location.
Note: See TracTickets for help on using tickets.