curl-7.71.0

[Douglas R. Reno]

New minor version

[Douglas R. Reno]

Hi friends!

I'm happy to annonce curl 7.71.0. This time shipped in sync with two security advisories you should be aware of. As always, get curl and all info from:

  https://curl.haxx.se/

curl and libcurl 7.71.0

 Public curl releases:         192
 Command line options:         232
 curl_easy_setopt() options:   277
 Public functions in libcurl:  82
 Contributors:                 2202

This release includes the following changes:

 o CURLOPT_SSL_OPTIONS: optional use of Windows' CA store (with openssl) [10]
 o setopt: add CURLOPT_PROXY_ISSUERCERT(_BLOB) for coherency [31]
 o setopt: support certificate options in memory with struct curl_blob [41]
 o tool: Add option --retry-all-errors to retry on any error [27]

This release includes the following bugfixes:

 o CVE-2020-8177: curl overwrite local file with -J [111]
 o CVE-2020-8169: Partial password leak over DNS on HTTP redirect [48]
 o *_sspi: fix bad uses of CURLE_NOT_BUILT_IN [21]
 o all: fix codespell errors [75]
 o altsvc: bump to h3-29 [114]
 o altsvc: fix 'dsthost' may be used uninitialized in this function
 o altsvc: fix parser for lines ending with CRLF [74]
 o altsvc: remove the num field from the altsvc struct [109]
 o appveyor: add non-debug plain autotools-based build [90]
 o appveyor: disable flaky test 1501 and ignore broken 1056
 o appveyor: disable test 1139 instead of ignoring it
 o asyn-*: remove support for never-used NULL entry pointers [19]
 o azure: use matrix strategy to avoid configuration redundancy [83]
 o build: disable more code/data when built without proxy support [84]
 o buildconf: remove -print from the find command that removes files
 o checksrc: enhance the ASTERISKSPACE and update code accordingly [52]
 o CI/macos: fix 'is already installed' errors by using bundle [94]
 o cirrus: disable SFTP and SCP tests [7]
 o CMake: add ENABLE_ALT_SVC option
 o CMake: add HTTP/3 support (ngtcp2+nghttp3, quiche) [34]
 o CMake: add libssh build support [37]
 o CMake: do not build test programs by default [30]
 o CMake: fix runtests.pl with CMake, add new test targets [29]
 o CMake: ignore INTERFACE_LIBRARY targets for pkg-config file [112]
 o CMake: rebuild Makefile.inc.cmake when Makefile.inc changes [58]
 o CODE_REVIEW.md: how to do code reviews in curl [108]
 o configure: fix pthread check with static boringssl
 o configure: for wolfSSL, check for the DES func needed for NTLM
 o configure: only strip first -L from LDFLAGS [89]
 o configure: repair the check if argv can be written to [47]
 o configure: the wolfssh backend does not provide SCP [57]
 o connect: improve happy eyeballs handling [118]
 o connect: make happy eyeballs work for QUIC (again) [16]
 o curl.1: Quote globbed URLs [51]
 o curl: remove -J "informational" written on stdout [36]
 o Curl_addrinfo: use one malloc instead of three [97]
 o CURLINFO_ACTIVESOCKET.3: clarify the description [87]
 o doc: add missing closing parenthesis in CURLINFO_SSL_VERIFYRESULT.3 [5]
 o doc: Rename VERSIONS to VERSIONS.md as it already has Markdown syntax [20]
 o docs/HTTP3: add qlog to the quiche build instruction
 o docs/options-in-versions: which version added each cmdline option [53]
 o docs: unify protocol lists [54]
 o dynbuf: introduce internal generic dynamic buffer functions [17]
 o easy: fix dangling pointer on easy_perform fail [26]
 o examples/ephiperfifo: turn off interval when setting timerfd [79]
 o examples/http2-down/upload: add error checks [78]
 o examples: remove asiohiper.cpp [4]
 o FILEFORMAT: add more features that tests can depend on
 o FILEFORMAT: describe verify/stderr
 o ftp: make domore_getsock() return the secondary socket properly
 o ftp: mark return-ignoring calls to Curl_GetFTPResponse with (void) [64]
 o ftp: shut down the secondary connection properly when SSL is used [43]
 o GnuTLS: Backend support for CURLINFO_SSL_VERIFYRESULT [9]
 o hostip: make Curl_printable_address not return anything [63]
 o hostip: on macOS avoid DoH when given a numerical IP address [69]
 o http2: keep trying to send pending frames after req.upload_done [40]
 o http2: simplify and clean up trailer handling [6]
 o HTTP3.md: clarify cargo build directory [77]
 o http: move header storage to Curl_easy from connectdata [107]
 o libcurl.pc: Merge Libs.private into Libs for static-only builds [28]
 o libssh2: improved error output for wrong quote syntax [39]
 o libssh2: keep sftp errors as 'unsigned long' [103]
 o libssh2: set the expected total size in SCP upload init [2]
 o libtest/cmake: Remove commented code [13]
 o list-only.d: this option existed already in 4.0
 o manpage: add three missing environment variables [121]
 o multi: add defensive check on data->multi->num_alive [96]
 o multi: implement wait using winsock events [120]
 o ngtcp2: cleanup memory when failing to connect [70]
 o ngtcp2: fix build with current ngtcp2 master implementing draft 28 [76]
 o ngtcp2: fix happy eyeballs quic connect crash [118]
 o ngtcp2: introduce qlog support [23]
 o ngtcp2: never call fprintf() in lib code in release version
 o ngtcp2: update with recent API changes [100]
 o ntlm: enable NTLM support with wolfSSL [81]
 o OpenSSL: have CURLOPT_CRLFILE imply CURLSSLOPT_NO_PARTIALCHAIN [55]
 o openssl: set FLAG_TRUSTED_FIRST unconditionally [105]
 o projects: Add crypt32.lib to dependencies for all OpenSSL configs [93]
 o quiche: clean up memory properly when failing to connect [71]
 o quiche: enable qlog output [14]
 o quiche: update SSLKEYLOGFILE support [98]
 o Revert "buildconf: use find -execdir" [38]
 o Revert "ssh: ignore timeouts during disconnect" [67]
 o runtests: remove sleep calls [18]
 o runtests: show elapsed test time with higher precision (ms)
 o select: always use Sleep in Curl_wait_ms on Win32 [82]
 o select: fix overflow protection in Curl_socket_check [22]
 o sendf: make failf() use the mvsnprintf() return code [62]
 o server/sws: fix asan warning on use of uninitialized variable
 o server/util: fix logmsg format using curl_off_t argument [106]
 o sha256: fixed potentially uninitialized variable [61]
 o share: don't set the share flag it something fails [116]
 o sockfilt: make select_ws stop waiting on exit signal event
 o socks: detect connection close during handshake [95]
 o socks: fix expected length of SOCKS5 reply [68]
 o socks: remove unreachable breaks in socks.c and mime.c [101]
 o source cleanup: remove all custom typedef structs [42]
 o test1167: fixes in badsymbols.pl [73]
 o test1177: look for curl.h in source directory [1]
 o test1238: avoid tftpd being busy for tests shortly following [33]
 o test613.pl: make tests 613 and 614 work with OpenSSH for Windows [8]
 o test75: Remove precheck test
 o tests: add https-proxy support to the test suite [49]
 o tests: add support for SSH server variant specific transfer paths [24]
 o tests: add two simple tests for --login-options [99]
 o tests: make test 1248 + 1249 use %NOLISTENPORT [3]
 o tests: pick a random port number for SSH [12]
 o tests: run stunnel for HTTPS and FTPS on dynamic ports [11]
 o timeouts: change millisecond timeouts to timediff_t from time_t [86]
 o timeouts: move ms timeouts to timediff_t from int and long [104]
 o tool: fixup a few --help descriptions [56]
 o tool: support UTF-16 command line on Windows [46]
 o tool_cfgable: free login_options at exit [102]
 o tool_getparam: fix memory leak in parse_args
 o tool_operate: fixed potentially uninitialized variables [60]
 o tool_paramhlp: fixed potentially uninitialized strtol() variable [59]
 o transfer: close connection after excess data has been read [66]
 o travis: add "qlog" as feature in the quiche build
 o travis: Add ngtcp2 and quiche tests for CMake
 o travis: upgrade to bionic, clang-9, improve readability [35]
 o typecheck-gcc.h: CURLINFO_PRIVATE does not need a 'char *' [44]
 o unit1604.c: fix implicit conv from 'SANITIZEcode' to 'CURLcode' [88]
 o url: accept "any length" credentials for proxy auth [72]
 o url: alloc the download buffer at transfer start [85]
 o url: reject too long input when parsing credentials [25]
 o url: sort the protocol schemes in rough popularity order [32]
 o urlapi: accept :: as a valid IPv6 address [15]
 o urldata: leave the HTTP method untouched in the set.* struct [45]
 o urlglob: treat literal IPv6 addresses with zone IDs as a host name [115]
 o user-agent.d: spell out what happens given a blank argument [80]
 o vauth/cleartext: fix theoretical integer overflow [50]
 o version.d: expanded and alpha-sorted [110]
 o vtls: Extract and simplify key log file handling from OpenSSL
 o wolfssl: add SSLKEYLOGFILE support [65]
 o wording: avoid blacklist/whitelist stereotypes [92]
 o write-out.d: added "response_code" 

Security advisories in next comment

[Douglas R. Reno]

Partial password leak over DNS on HTTP redirect
===============================================

Project curl Security Advisory, June 24th 2020 -
[Permalink](https://curl.haxx.se/docs/CVE-2020-8169.html)

VULNERABILITY
-------------

libcurl can be tricked to prepend a part of the password to the host name
before it resolves it, potentially leaking the partial password over the
network and to the DNS server(s).

libcurl can be given a username and password for HTTP authentication when
requesting an HTTP resource - used for HTTP Authentication such as Basic,
Digest, NTLM and similar. The credentials are set, either together with
`CURLOPT_USERPWD` or separately with `CURLOPT_USERNAME` and
`CURLOPT_PASSWORD`. Important detail: these strings are given to libcurl as
plain C strings and they are not supposed to be URL encoded.

In addition, libcurl also allows the credentials to be set in the URL, using
the standard RFC 3986 format: `http://user:password@host/path`. In this case,
the name and password are URL encoded as that's how they appear in URLs.

If the options are set, they override the credentials set in the URL.

Internally, this is handled by storing the credentials in the "URL object" so
that there is only a single set of credentials stored associated with this
single URL.

When libcurl handles a relative redirect (as opposed to an absolute URL
redirect) for an HTTP transfer, the server is only sending a new path to the
client and that path is applied on to the existing URL. That "applying" of the
relative path on top of an absolute URL is done by libcurl first generating a
full absolute URL out of all the components it has, then it applies the
redirect and finally it deconstructs the URL again into its separate
components.

This security vulnerability originates in the fact that curl did not correctly
URL encode the credential data when set using one of the `curl_easy_setopt`
options described above. This made curl generate a badly formatted full URL
when it would do a redirect and the final re-parsing of the URL would then go
bad and wrongly consider a part of the password field to belong to the host
name.

The wrong host name would then be used in a name resolve lookup, potentially
leaking the host name + partial password in clear text over the network (if
plain DNS was used) and in particular to the used DNS server(s).

The password leak is triggered if an at sign (`@`) is used in the password
field, like this: `passw@rd123`. If we also consider a user `dan`, curl would
generate a full URL like:

 `https://dan:passw@rd123@example.com/path`

... while a correct one should have been:

 `https://dan:passw%40rd123@example.com/path`

... when parsing the wrongly generated URL, libcurl would end up with user
name `dan` and password `passw` talking to the host `rd123@example.com`. That
bad host name would then be passed on to the name resolver function in use
(and for all typical cases return a "cannot resolve host name" error).

There's no hint in the name resolve as to how large portion of the password
that is actually prepended to the host name (ie an observer won't know how
much data there was on the left side of the `@`), but it can of course be a
significant enough clue for an attacker to figure out the rest.

We are not aware of any exploit of this flaw.

INFO
----

Requirements to trigger this flaw.

 1. a password set with a `@` in it
 2. an HTTP transfer
 3. a *relative* redirect that curl follows (`CURLOPT_FOLLOWLOCATION` enabled)

This bug was brought in commit
[46e164069d](https://github.com/curl/curl/commit/46e164069d), first shipped in
curl 7.62.0.

This flaw can happen to users of the curl tool as well as for applications
using libcurl.

This bug was reported and inadvertently fixed and pushed to the public source
respository before anyone realized its security impact.

The effects of this flaw is somewhat reduced if DNS-over-HTTPS is used, since
then at least the name won't be observable on the network by a passive
by-stander but only by the DoH server.

The Common Vulnerabilities and Exposures (CVE) project has assigned the name
CVE-2020-8169 to this issue.

CWE-200: Exposure of Sensitive Information to an Unauthorized Actor

Severity: 5.5 (Medium)

AFFECTED VERSIONS
-----------------

- Affected versions: libcurl 7.62.0 to and including 7.70.0
- Not affected versions: libcurl < 7.62.0

libcurl is used by many applications, but not always advertised as such.

THE SOLUTION
------------

A [fix for CVE-2020-8169](https://github.com/curl/curl/commit/600a8cded447cd)

RECOMMENDATIONS
--------------

We suggest you take one of the following actions immediately, in order of
preference:

 A - Upgrade curl to version 7.71.0

 B - Apply the patch on your libcurl version and rebuild

 C - Disable `CURLOPT_FOLLOWLOCATION` or redirects to HTTP(S).

TIMELINE
--------

This issue was first reported to the curl project on May 14, 2020. The initial
fix was done, verified and pushed to git on the same day. (As a regular
non-security related fix.)

On May 15, 2020, the bug was reported again but then with the security impact
highlighted.

This advisory was posted on June 24th 2020.

CREDITS
-------

The security issue was reported by Marek Szlagor. The initial bug report was
done by Gregory Jefferis and Jeroen Ooms. Patched by Daniel Stenberg.

Thanks a lot!

Partial password leak on DNS over HTTP redirects

I suggest anyone that's running 7.62.0 or higher should upgrade to 7.71.0

[Douglas R. Reno]

curl overwrite local file with -J
=================================

Project curl Security Advisory, June 24th 2020 -
[Permalink](https://curl.haxx.se/docs/CVE-2020-8177.html)

VULNERABILITY
-------------

curl can be tricked my a malicious server to overwrite a local file when using
`-J` (`--remote-header-name`) and `-i` (`--head`) in the same command line.

The command line tool offers the `-J` option that saves a remote file using
the file name present in the `Content-Disposition:` response header. curl then
refuses to overwrite an existing local file using the same name, if one
already exists in the current directory.

The `-J` flag is designed to save a response body, and so it doesn't work
together with `-i` and there's logic that forbids it. However, the check is
flawed and doesn't properly check for when the options are used in the
reversed order: first using `-J` and then `-i` were mistakenly accepted.

The result of this mistake was that incoming HTTP headers could overwrite a
local file if one existed, as the check to avoid the local file was done first
when body data was received, and due to the mistake mentioned above, it could
already have received and saved headers by that time.

The saved file would only get response headers added to it, as it would abort
the saving when the first body byte arrives. A malicious server could however
still be made to send back virtually anything as headers and curl would save
them like this, until the first CRLF-CRLF sequence appears.

(Also note that `-J` needs to be used in combination with `-O` to have any
effect.)

We are not aware of any exploit of this flaw.

INFO
----

Users should be aware and *never* run curl with the `-J` option in their
`$HOME` or other sensitive directories, independently of this flaw. Using curl
that way allows curl to create any file name it likes (i.e. what the remote
server suggests) and it can confuse or trick users if allowed to save files
that can mistakenly be assumed to be "locally made" or part of the system
rather than provided by a potentially malicious remote party.

This bug was brought in commit
[80675818e0417b](https://github.com/curl/curl/commit/80675818e0417b) when `-J`
was introduced to curl, first shipped in curl 7.20.0.

This flaw can happen to users of the curl tool but **not** for applications
using libcurl.

The Common Vulnerabilities and Exposures (CVE) project has assigned the name
CVE-2020-8177 to this issue.

CWE-641: Improper Restriction of Names for Files and Other Resources

Severity: 4.7 (Medium)

AFFECTED VERSIONS
-----------------

- Affected versions: curl 7.20.0 to and including 7.70.0
- Not affected versions: curl < 7.20.0 and curl >= 7.71.0

THE SOLUTION
------------

A [fix for CVE-2020-8177](https://github.com/curl/curl/commit/8236aba58542c5f.patch)

RECOMMENDATIONS
--------------

We suggest you take one of the following actions immediately, in order of
preference:

 A - Upgrade curl to version 7.71.0

 B - Apply the patch on your curl version and rebuild

 C - Do not use `-J` (in a directory with pre-existing files)

TIMELINE
--------

This issue was first reported to the curl project on May 30, 2020.

This advisory was posted on June 24th 2020.

CREDITS
-------

This issue was reported by sn on hackerone. Patched by Daniel Stenberg.

Thanks a lot!

Local file overwrite with the -J option. Affects 7.20.0 to 7.70.0

[Douglas R. Reno]

Fixed at r23324

[Bruce Dubbs]

Milestone renamed

[Bruce Dubbs]

Milestone renamed