dovecot-2.3.18

[Bruce Dubbs]

New point version.

[Tim Tassonis]

* Removed mail_cache_lookup_finished event. This event wasn't especially
  useful, but it increased CPU usage significantly.
* fts: Don't index inline base64 encoded content in FTS indexes using
  the generic tokenizer. This reduces the FTS index sizes by removing
  input that is very unlikely to be searched for. See
  https://doc.dovecot.org/configuration_manual/fts/tokenization for
  details on how base64 is detected. Only applies when using libfts.
* lmtp: Session IDs are now preserved through proxied connections, so
  LMTP sessions can be tracked. This slightly changes the LMTP session
  ID format by appending ":Tn" (transaction), ":Pn" (proxy connection)
  and ":Rn" (recipient) counters after the session ID prefix.
+ Events now have "reason_code" field, which can provide a list of
  reasons why the event is happening. See
  https://doc.dovecot.org/admin_manual/event_reasons/
+ New events are added. See https://doc.dovecot.org/admin_manual/list_of_events/
+ fts: Added fts_header_excludes and fts_header_includes settings to
  specify which headers to index. See
  https://doc.dovecot.org/settings/plugin/fts-plugin#plugin-fts-setting-fts-header-excludes
  for configuration details.
+ fts: Initialize the textcat language detection library only once per
  process. This can reduce CPU usage if fts_languages setting has multiple
  languages listed and service indexer-worker { service_count } isn't 1.
  Only applies when using libfts.
+ lib-storage: Reduced CPU usage significantly for some operations that
  accessed lots of emails (e.g. fetching all flags in a folder, SORT, ...)
+ lib: DOVECOT_PREREQ() - Add micro version which enables compiling
  external plugins against different versions of Dovecot.
+ lmtp: Added new lmtp_verbose_replies setting that makes errors sent to
  the LMTP client much more verbose with details about why exactly
  backend proxy connections or commands are failing.
+ submission: Support implicit SASL EXTERNAL with
  submission_client_workarounds=implicit-auth-external. This allows
  automatically logging in when SSL client certificate is present.
- *-login: Statistics were disabled if stats process connection was lost.
- auth: Authentication master user login fails with SCRAM-* SASL mechanisms.
- auth: With auth_cache_verify_password_with_worker=yes, passdb extra
  fields in the auth cache got lost.
- doveadm: Fixed crash if zlib_save_level setting was specified,
  but zlib_save was unset. v2.3.15 regression.
- doveadm: Proxying can panic when flushing print output. v2.3.17
  regression. Fixes:
  Panic: file ioloop.c: line 865 (io_loop_destroy): assertion failed:
  (ioloop == current_ioloop)
- doveadm: stats add --group-by parameter didn't work.
- fts: Using email-address fts tokenizer could result in excessive memory
  usage with garbage email input. This could cause the indexer-worker
  processes to fail due to reaching the VSZ memory size limit.
  Only applies when using libfts.
- imap: A SEARCH command timing out while fts returns indexes may timeout
  returning "NO [SERVERBUG]", while it should return "NO [INUSE]" instead.
- imap: LIST-EXTENDED doesn't return STATUS for all folders. Sending
  LIST .. RETURN (SUBSCRIBED STATUS (...)) did not return STATUS for
  folders that are not subscribed when they have a child folder that is
  subscribed as mandated by IMAP RFCs.
- imapc: Mailbox vsize calculation crashed with
  Panic: file index-mailbox-size.c: line 344 (index_mailbox_vsize_hdr_add_missing):
  assertion failed: (mails_left > 0)
- indexer: If indexer-worker crashes, the request it was processing gets
  stuck in the indexer process. This stops indexing for the folder until
  indexer process is restarted. v2.3.14 regression.
- indexer: Process was slowly leaking memory for each indexing request.
- lib-event: Unnamed events were wrongly filtered out for event/metric
  filters like "event=abc OR something_independent_of_event_name".
- lib-index: 64-bit big endian CPUs handle last_used field in
  dovecot.index.cache wrong.
- lib-ssl-iostream: Fix buggy OpenSSL error handling without assert-crashing.
  If there is no error available, log it as an error instead of crashing.
  The previous fix for this in v2.3.11 was incomplete. Fixes
  Panic: file istream-openssl.c: line 51 (i_stream_ssl_read_real):
  assertion failed: (errno != 0)
- lmtp: Out-of-memory issues can happen when proxying large messages to
  LMTP backend servers that accept the message data too slow.
- master: HAProxy header parsing has read buffer overflow if provided
  header size is invalid. This happens only if inet_listener
  { haproxy=yes } is configured and only if the remote IP address is in
  haproxy_trusted_networks.
- old_stats: Plugin kept increasing memory usage, which became
  noticeable with long-running imap sessions.
- stats: Dynamically adding same metric multiple times causes multiple stats.
- submission-login: Authentication does not accept OAUTH2 token (or
  other very long credentials) because it considers the line to be too long.
- submission-login: Process can crash if HELO is pipelined with an
  invalid domain.
- submission-proxy: Don't use SASL-IR if it would make the AUTH command
  line longer than 512 bytes.
- submission: Service would crash if relay server authentication failed.
- virtual: FTS search in a virtual folder could crash if there are
  duplicate mailbox GUIDs. This mainly happened when user had both INBOX
  and INBOX/INBOX folders and the namespace prefix was INBOX/. Fixes
  Panic: file hash.c: line 252 (hash_table_insert_node):
  assertion failed: (opcode == HASH_TABLE_OP_UPDATE)
- virtual: If mailbox opening fails, the backend mailbox is leaked and
  process crashes when client disconnects. Fixes
  Panic: file mail-user.c: line 232 (mail_user_deinit):
  assertion failed: ((*user)->refcount == 1)
- virtual: Searching headers in virtual folders didn't always use
  full-text search indexes, if fts_enforced=no or body.

[Douglas R. Reno]

We seem to have an OpenSSL-3 related problem here. On my development system (which runs OpenSSL-1.1), this failure is not present:

make[2]: Leaving directory '/sources/dovecot-2.3.18/dovecot-2.3.18/src/lib-ssl-iostream'
Making check in lib-dcrypt
make[2]: Entering directory '/sources/dovecot-2.3.18/dovecot-2.3.18/src/lib-dcrypt'
make  check-local
make[3]: Entering directory '/sources/dovecot-2.3.18/dovecot-2.3.18/src/lib-dcrypt'
for bin in test-crypto test-stream; do \
  if ! /bin/sh ../../libtool execute /bin/sh ../../run-test.sh ../.. ./$bin; then exit 1; fi; \
done
test_cipher_test_vectors ............................................. : ok
test_cipher_aead_test_vectors ........................................ : ok
test_hmac_test_vectors ............................................... : ok
--
test_gen_and_get_info_rsa_pem ........................................ : ok
test_get_info_rsa_private_key ........................................ : ok
test_get_info_invalid_keys ........................................... : ok
test_get_info_key_encrypted .......................................... : ok
test_get_info_pw_encrypted ........................................... : ok
test-crypto.c:827: Assert failed: ret == TRUE
make[3]: Leaving directory '/sources/dovecot-2.3.18/dovecot-2.3.18/src/lib-dcrypt'
make[2]: Leaving directory '/sources/dovecot-2.3.18/dovecot-2.3.18/src/lib-dcrypt'
Making check in lib-dict
make[2]: Entering directory '/sources/dovecot-2.3.18/dovecot-2.3.18/src/lib-dict'
make  check-local

However, on my OpenSSL-3 machine (seen above), we're getting an assertion failure, and I'm also unable to connect to Dovecot if it's running on that machine (assertion failure followed by crash).

I'm going to try the patch found in ​https://src.fedoraproject.org/rpms/dovecot/blob/rawhide/f/dovecot-2.3.14-opensslv3.patch first, since I suspect it's an issue with the deprecated EVP* functions in OpenSSL-3 which also carried behavior changes along with them.

[Douglas R. Reno]

test_get_info_key_encrypted .......................................... : ok
test_get_info_pw_encrypted ........................................... : ok
test-crypto.c:827: Assert failed: ret == TRUE
Panic: file dcrypt-openssl.c: line 2639 (dcrypt_openssl_private_to_public_key): assertion failed: (priv_key != NULL && pub_key_r != NULL)
Error: Raw backtrace: ./test-crypto() [0x45d3ed] -> ./test-crypto(backtrace_append+0x18) [0x45d578] -> ./test-crypto(backtrace_get+0x26) [0x45d5a0] -> ./test-crypto() [0x426d6c] -> ./test-crypto(default_fatal_handler+0) [0x426e3a] -> ./test-crypto(default_error_handler+0) [0x426e84] -> ./test-crypto(i_fatal+0) [0x42710e] -> .libs/libdcrypt_openssl.so(+0xd8e0) [0x485e8e0] -> ./test-crypto(dcrypt_key_convert_private_to_public+0x67) [0x41621e] -> ./test-crypto() [0x41f964] -> ./test-crypto() [0x421daf] -> ./test-crypto(test_run+0x1d) [0x42216b] -> ./test-crypto(main+0x72) [0x420fbb] -> /usr/lib/libc.so.6(+0x291ca) [0x48bb1ca] -> /usr/lib/libc.so.6(__libc_start_main+0x78) [0x48bb278] -> ./test-crypto(_start+0x21) [0x4138f1]
../../run-test.sh: line 39: 25135 Aborted                 (core dumped) /bin/valgrind -q $trace_children --error-exitcode=213 --leak-check=full --gen-suppressions=all --suppressions="$supp_path" --log-file=$test_out $noundef $*
Failed to run: ./test-crypto
make[3]: *** [Makefile:1136: check-local] Error 1
make[3]: Leaving directory '/sources/dovecot-2.3.18/dovecot-2.3.18/src/lib-dcrypt'
make[2]: *** [Makefile:983: check-am] Error 2
make[2]: Target 'check' not remade because of errors.

Hmm, I wonder if this is Valgrind related. I'll try a suppression.

[Douglas R. Reno]

Fixed at 9ef5be6e79c649741393a99e766208845cf19ffe