Imported Upstream version 1.19.2upstream/1.19.2

1 files changed, 555 insertions, 0 deletions

diff --git a/ChangeLog b/ChangeLog
index fa5c084..0bac795 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,558 @@
+2017-10-26  Tim Rühsen  <tim.ruehsen@gmx.de>
+
+	Fix heap overflow in HTTP protocol handling (CVE-2017-13090)
+	* src/retr.c (fd_read_body): Stop processing on negative chunk size
+
+	Reported-by: Antti Levomäki, Christian Jalio, Joonas Pihlaja from Forcepoint
+	Reported-by: Juhani Eronen from Finnish National Cyber Security Centre
+
+2017-10-26  Tim Rühsen  <tim.ruehsen@gmx.de>
+
+	Fix stack overflow in HTTP protocol handling (CVE-2017-13089)
+	* src/http.c (skip_short_body): Return error on negative chunk size
+
+	Reported-by: Antti Levomäki, Christian Jalio, Joonas Pihlaja from Forcepoint
+	Reported-by: Juhani Eronen from Finnish National Cyber Security Centre
+
+2017-10-26  Tim Rühsen  <tim.ruehsen@gmx.de>
+
+	Update for release 1.19.2
+	* NEWS: Add news for 1.19.2
+	* doc/wget.texi: Fix 1.20 to 1.19.2
+
+2017-10-25  YX Hao  <lifenjoiner@163.com>
+
+	Avoid unnecessary UTF-8 encoded fallback (trivial change)
+	* src/retr.c (retrieve_url): Check for changed URL on redirect
+
+2017-09-27  Tim Rühsen  <tim.ruehsen@gmx.de>
+
+	Add GNU extensions to .netrc parsing
+	src/netrc.c (parse_netrc): Add 'port' and 'force' extensions
+
+	Reported-by: September 20Tim Landscheidt
+
+2017-09-18  Josef Moellers  <jmoellers@suse.de>
+
+	Bail out on unexpected 416 server errors
+	* src/http.c (gethttp): Stop on 416 if file is incomplete
+
+2017-08-04  Tim Schlueter  <schlueter.tim@linux.com>
+
+	Add gzip Content-Encoding decompression
+	* src/http.c (struct http_stat): Add remote_encoding field.
+	(read_response_body): Enable gzip decompression.
+	(initialize_request): Send gzip Accept-Encoding header.
+	(gethttp): Decompress files with gzip Content-Encoding.
+	* src/retr.c: include zlib.h.
+	(zalloc): New function.
+	(zfree): New function.
+	(fd_read_body): Decompress gzip data.
+	* src/retr.h (fd_read_body enum): Add rb_compressed_gzip flag.
+
+	Add --compression option
+	* doc/wget.texi: Add --compression documentation.
+	* src/init.c (cmd_spec_compression): New function.
+	(commands[]): Add opt.compression.
+	(defaults): Set default opt.compression value.
+	* src/main.c (option_data[]): Add struct for --compression.
+	(print_help, help[]): Add description for --compression.
+	(main): Add incompatibility checks for --compression.
+	* src/options.h (struct options): Add compression enum and field.
+
+	Adjust Extension based on Content-Encoding
+	* doc/wget.texi (--adjust-extension, adjust_extension): Updated documentation.
+	* src/http.c (encoding_t): New enum.
+	(struct http_stat): Add local_encoding field.
+	(gethttp): --adjust-extension based on Content-Encoding.
+
+2017-07-31  Darshit Shah  <darnir@gnu.org>
+
+	Document gperf as a requirement
+
+2017-07-28  Tim Rühsen  <tim.ruehsen@gmx.de>
+
+	* src/url.c (url_scheme): Use ASCII version of strncasecmp
+
+2017-07-28  Tim Rühsen  <tim.ruehsen@gmx.de>
+
+	Fix misuse of strncasecmp
+	* src/http.c (set_content_type): Use c_strcasecmp instead of strncasecmp
+
+	See issue bug #51576
+
+2017-07-09  Tim Rühsen  <tim.ruehsen@gmx.de>
+
+	Fix python test suite for GnuTLS 3.5.12+
+	* testenv/Test-*.py: Replace 127.0.0.1 by localhost
+	* testenv/certs/server-template.cfg: Likewise
+	* testenv/certs/server-cert.pem: Regenerate
+	* testenv/certs/server-crl.pem: Likewise
+	* testenv/test/base_test.py: Hardcode 'localhost' as server domain
+
+	Reported-by: Ludovic Courtès <ludo@gnu.org>
+
+2017-06-13  Tim Rühsen  <tim.ruehsen@gmx.de>
+
+	Check for 304 response before applying --adjust-extension
+	* src/http.c (gethttp): Move 304 code before --adjust-extension code
+
+	This fixes applying --adjust-extension in combination with 304
+	HTTP responses. It could lead to .html extensions to arbitrary
+	files.
+
+	Reported-by: anfractuosity
+
+2017-06-13  Tim Rühsen  <tim.ruehsen@gmx.de>
+
+	Fix buffer overflow in Public Key Pinning
+	* src/utils.c (wget_base64_decode): Add param for destination size,
+	  (wg_pubkey_pem_to_der): Amend call to wget_base64_decode(),
+	  (wg_pin_peer_pubkey): Likewise and fix code style.
+	* src/utils.h: Add param to wget_base64_decode()
+	* src/http-ntlm.c (ntlm_input): Amend call to wget_base64_decode()
+	* src/http.c (skip_content_type): Likewise
+
+	Fixes #51227
+
+2017-06-02  Tim Rühsen  <tim.ruehsen@gmx.de>
+
+	* doc/wget.texi: Mention --no-config
+
+2017-05-16  Tim Rühsen  <tim.ruehsen@gmx.de>
+
+	* testenv/Test-recursive-basic.py: Check crawled files
+
+	Fix two Metalink tests if $HOME is changed
+	* conf/expected_files.py (gen_local_fs_snapshot): Skip processing
+	  of 'pubring.kbx'
+
+2017-05-15  Tomas Hozza  <thozza@redhat.com>
+
+	Add command line option to disable use of .netrc
+	Although internally code uses option for (not) reading .netrc for
+	credentials, it was not possible to turn this behavior off on command
+	line. Note that it was possible to turn it off using wgetrc.
+
+	Idea for this change came from Bruce Jerrick (bmj001@gmail.com).
+	Reference: https://bugzilla.redhat.com/show_bug.cgi?id=1425097
+
+2017-05-15  Tomas Hozza  <thozza@redhat.com>
+
+	Fixed getting of credentials from .netrc
+	There seemed to be a copy&paste error in http.c code, which decides
+	whether to get credentials from .netrc. In ftp.c "user" and "pass"
+	variables are char*, while in http.c, these are char**. For this reason
+	they should be dereferenced when determining if password and user login
+	is set to some value.
+
+	Also since both variables are dereferenced on lines above the changed
+	code, it does not really make sense to check if they are NULL.
+
+	This patch is based on fix from Bruce Jerrick <bmj001@gmail.com>.
+	Fedora bug: https://bugzilla.redhat.com/show_bug.cgi?id=1425097
+
+2017-05-15  Tomas Hozza  <thozza@redhat.com>
+
+	Added tests for HTTP authentication using credentials from .netrc
+	Getting credentials from .netrc has been broken from time to time, thus
+	adding a test coverage to prevent regressions.
+
+	Also added setting of "HOME" environment variable when executing wget,
+	to make sure LocalFiles like .netrc, which are created just for the
+	test, are actually used.
+
+2017-05-15  Tim Rühsen  <tim.ruehsen@gmx.de>
+
+	Fix Test-https-badcerts.px
+	* util/createcerts.sh: Set date of invalid cert to +20 years
+	* tests/certs/invalid.crt: Re-generated
+	* tests/certs/invalid.key: Re-generated
+
+2017-05-14  Tim Rühsen  <tim.ruehsen@gmx.de>
+
+	* util/createcerts.sh: Fix double equal (syntax-check)
+
+	Fix Test-https-badcerts.px to work with GnuTLS
+	* tests/Test-https-badcerts.px: Remove OpenSSL requirement
+	* tests/certs/expired.crt: Re-created with valid dates
+	* tests/certs/expired.key: Likewise
+	* tests/certs/invalid.crt: Likewise
+	* tests/certs/invalid.key: Likewise
+
+2017-05-14  Vijo Cherian  <codervijo@gmail.com>
+
+	New shell script to create the certs and keys required for TLS tests
+	* util/createcerts.sh: New file
+
+2017-05-11  Tim Rühsen  <tim.ruehsen@gmx.de>
+
+	Fix HTTPS testing for stricter OpenSSL
+	* testenv/certs/README: Amend cert creation extensions
+	* testenv/certs/ca-cert.pem: Created without OCSP signing purpose
+
+	Having the OCSP signing purpose set made newer versions of OpenSSL
+	fail due to stricter checking. Test version of OpenSSL was 1.1.0e.
+
+2017-05-11  Tim Rühsen  <tim.ruehsen@gmx.de>
+
+	* src/iri.c (idn_encode): Better IDNA 2003 compatibility
+
+2017-05-10  Tim Rühsen  <tim.ruehsen@gmx.de>
+
+	* .gitlab-ci.yml: Also test OpenSSL build
+
+	* .gitlab-ci.yml: Remove installation of texlive
+
+2017-05-10  Tim Rühsen  <tim.ruehsen@gmx.de>
+
+	Add certs/wotca.pem to avoid temp. file creation
+	* tests/certs/wotca.pem: New file
+	* tests/Test-https-weboftrust.px: Remove creation of wotca.pem
+
+	This change let Test-https-weboftrust.px survive VPATH builds.
+
+2017-05-10  Tim Rühsen  <tim.ruehsen@gmx.de>
+
+	* cfg.mk: Exclude ^tests/certs/.* from syntax-check
+
+	* po/POTFILES.in: Remove spider.c (syntax-check)
+
+	* tests/Test-https-selfsigned.px: Add newline at EOF (syntax-check)
+
+2017-05-10  Vijo Cherian  <codervijo@gmail.com>
+
+	Removed all uses of temp files. Added needed files to GIT
+
+	Made CRL related files in the repo, instead of trying to generate them
+
+2017-05-09  Tim Rühsen  <tim.ruehsen@gmx.de>
+
+	Cleanup on exit in Test-https-*.px
+	* tests/Test-https-*.px: Cleanup on exit
+
+	Auto-generate interca.conf and rootca.conf
+	* configure.ac: Add interca.conf.in and rootca.conf.in to AC_CONFIG_FILES
+	* tests/certs/interca.conf: Removed
+	* tests/certs/rootca.conf: Removed
+	* tests/certs/interca.conf.in: New file
+	* tests/certs/rootca.conf.in: New file
+
+	Fix path and VPATH issues of new https/TLS tests
+	* tests/SSLTest.pm: Use $srcdir in read-only files,
+	  (_setup_server): Remove unneeded path fixation code
+	* tests/Test-https-*.px: Use $srcdir for read-only files,
+	  use $cdir for writable files
+	* tests/certs/interca.conf: Adjust paths
+	* tests/certs/rootca.conf: Adjust paths
+
+	Check for test server name resolution in tests
+	* tests/Test-https-*.px: Skip if test server name resolution fails
+
+	* tests/Makefile.am: Enable Test-https-badcerts again
+
+	Fix WgetFeature.pm to allow multiple required features
+	* tests/WgetFeature.cfg: Remove file
+	* tests/WgetFeature.pm: Extend to multiple features, cleanup
+
+	* .gitlab-ci.yml: Add wgettestingserver to /etc/hosts
+
+	* tests/SSLServer.pm: Check for IO::Socket::SSL
+
+	* tests/Test-https-*: Change server port to <= 32767
+
+	Move https test server ports from >32767 to <= 32767
+	* Test-https-badcerts.px: Change port
+	* Test-https-crl.px: Likewise
+	* Test-https-weboftrust.px: Likewise
+
+	* tests/Makefile.am: Add SSLTest.pm and SSLServer.pm to EXTRA_DIST
+
+	* tests//Makefile.am: Disable Test-https-badcerts.px
+
+	Add Gitlab CI (Debian)
+	* .gitlab-ci.yml: New file
+
+2017-05-06  Tim Rühsen  <tim.ruehsen@gmx.de>
+
+	* .travis.yml: Use trusty for libidn2-dev
+
+2017-05-04  Tim Rühsen  <tim.ruehsen@gmx.de>
+
+	* tests/certs/test-ca-key.pem: Add newline at EOF
+
+	Add static HOSTSALIAS file
+	* tests/certs/wgethosts: New file
+	* tests/Test-https-*.px: Remove creation of wgethosts file
+
+2017-05-04  Vijo Cherian  <codervijo@gmail.com>
+
+	Added new tests for SSL
+	* tests/Test-https-badcerts.px : New file
+	* tests/Test-https-clientcert.px : New file
+	* tests/Test-https-crl.px : New file
+	* tests/Test-https-weboftrust.px : New file
+	* tests/certs/interca.conf : New file
+	* tests/certs/rootca.conf : New file
+	* tests/certs/test-ca-key.pem : New file
+
+	Added all new SSL / HTTPS tests to make check
+	Added Test for SSL Web of Trust, accept only if CA chain of trust is intact.
+	Added a test script for client certificate
+	Added Test for crlfile option of wget
+	Added test to make sure that wget doesn't accept expired or invalid certs
+
+	Some clean up : 1, Removed cause of warnings from perl & other cosmetic changes
+	                2, Fix  make -j 4 check such that it passes all tests
+
+2017-05-04  Tomas Hozza  <thozza@redhat.com>
+
+	Mention TLSv1_1 and TLSv1_2 as secure-protocol values in help
+	* src/main.c: The --secure-protocol option accepts also values TLSv1_1
+	and TLSv1_2, as mentioned in the man page. However the help message
+	doesn't mention these two values. This patch adds TLSv1_1 and TLSv1_2 as
+	possible values to the help message.
+
+2017-05-03  Tim Rühsen  <tim.ruehsen@gmx.de>
+
+	* src/http.c (gethttp): Support Wayback Machine's X-Archive-Orig-last-modified
+
+2017-04-28  Vijo Cherian  <codervijo@gmail.com>
+
+	Added new tests for SSL
+	* tests/Test-https-badcerts.px : New file
+	* tests/Test-https-clientcert.px : New file
+	* tests/Test-https-crl.px : New file
+	* tests/Test-https-weboftrust.px : New file
+	* tests/certs/interca.conf : New file
+	* tests/certs/rootca.conf : New file
+	* tests/certs/test-ca-key.pem : New file
+
+	Added all new SSL / HTTPS tests to make check
+	Added Test for SSL Web of Trust, accept only if CA chain of trust is intact.
+	Added a test script for client certificate
+	Added Test for crlfile option of wget
+	Added test to make sure that wget doesn't accept expired or invalid certs
+
+	Some clean up : Removed cause of warnings from perl & other cosmetic changes
+
+2017-04-19  Vijo Cherian  <codervijo@gmail.com>
+
+	Added new tests for SSL
+	* tests/SSLServer.pm: New file
+	* tests/SSLTest.pm: New file
+	* tests/Test-https-pfs.px: New file
+	* tests/Test-https-selfsigned.px: New file
+	* tests/Test-https-tlsv1.px: New file
+	* tests/Test-https-tlsv1x.px: New file
+	* tests/certs/server.crt: New file
+	* tests/certs/server.key: New file
+	* tests/certs/test-ca-cert.pem: New file
+
+	Added 4 new test scripts all for SSL.
+	Added base pm for SSL testing.
+	Added SSL tests for TLSv1, TLSv1_1 and PFS.
+	Added test for self signed cert : check that it fails without
+	--no-check-certificate and passes with that flag.
+
+2017-04-18  Tim Rühsen  <tim.ruehsen@gmx.de>
+
+	* src/utils.c: Remove non-portable __builtin_unreachable()
+
+2017-04-18  Tim Rühsen  <tim.ruehsen@gmx.de>
+
+	Skip iconv() usage if HAVE_ICONV is not defined
+	This helps on broken iconv implementations, e.g. Solaris.
+
+	Reported-by: Mojca Miklavec
+
+2017-04-18  Tim Rühsen  <tim.ruehsen@gmx.de>
+
+	Mention 'bash' for executing ./bootstrap
+	Reported-by: Mojca Miklavec
+
+2017-04-18  Tim Rühsen  <tim.ruehsen@gmx.de>
+
+	* bootstrap.conf: Make 'sed' options more portable
+	'sed -i' works differently on FreeBSD.
+
+	Reported-by: Mojca Miklavec
+
+2017-04-16  Tim Rühsen  <tim.ruehsen@gmx.de>
+
+	Fix charset transcoding issue for non-reversible codepoints
+	* src/url.c: Check iconv() against 0, not -1
+
+	On some libiconv implementations, unknown codepoints become
+	encoded as ?, e.g. when converting a non-ascii codepoint to ASCII.
+	This results in ambigious file names which also fails our tests.
+
+2017-04-16  Tim Rühsen  <tim.ruehsen@gmx.de>
+
+	* src/iri.c: Fix WIN32 idn2_free, forgotten code
+
+2017-04-14  Darshit Shah  <darnir@gnu.org>
+
+	* src/init.c: Set flstats correctly when using WGETRC env var
+
+2017-04-08  Tim Rühsen  <tim.ruehsen@gmx.de>
+
+	Fix use of idn2_free()
+	* src/connect.c (connect_to_ip): Use xfree() instead of idn2_free()
+	* src/host.c (lookup_host): Use xfree() instead of idn2_free()
+	* src/iri.h: Do not include idn2.h
+	* src/url.c (url_free): Use xfree() instead of idn2_free()
+	* src/url.h (struct url): Remove 'idn_allocated' from struct
+
+	Reported-by: Gisle Vanem
+
+2017-04-05  Anton Yuzhaninov  <citrin+github@citrin.ru>
+
+	Fix perl warnings in tests
+	* tests/FTPServer.pm: Escape '{' in RE to fix warnings
+	* tests/FTPTest.pm: Likewise
+	* tests/HTTPServer.pm: Likewise
+	* tests/HTTPTest.pm: Likewise
+	* tests/Test-proxied-https-auth-keepalive.px: Likewise
+	* tests/Test-proxied-https-auth.px: Likewise
+	Escape '{' in RE to fix warnings:
+	Unescaped left brace in regex is deprecated, passed through in regex;
+	marked by <-- HERE in m/{{ <-- HERE port}}/
+
+2017-04-01  klemens  <ka7@github.com>
+
+	Fix typos in comments
+
+2017-03-31  Tim Rühsen  <tim.ruehsen@gmx.de>
+
+	* src/metalink.c (retrieve_from_metalink): Fix len in memset()
+
+2017-03-24  Tim Rühsen  <tim.ruehsen@gmx.de>
+
+	Add gnulib module group-member
+
+2017-03-24  Vijo Cherian  <codervijo@gmail.com>
+
+	Safeguards against TOCTTOU
+	* src/utils.h: Add struct file_stat_s declaration,
+	  change prototypes of file_exists_p(),
+	  add prototypes for fopen_stat() and open_stat().
+	* src/utils.c: Extend file_exists_p(),
+	  new function fopen_stat() and open_stat(),
+	  add new param for file_exists_p().
+	* src/init.h: Add param file_stats_t to run_wgetrc().
+	* src/ftp.c: Amend calls to extended functions.
+	* src/hsts.c: Likewise.
+	* src/http.c: Likewise.
+	* src/init.c: Likewise.
+	* src/main.c: Likewise.
+	* src/metalink.c: Likewise.
+	* src/retr.c: Likewise.
+	* src/url.c: Likewise.
+
+	Added fopen_stat() and open_stat() that checks to makes sure the file didn't
+	change underneath us.
+	Return error from file_exists_p().
+	Added a way to return error from this file without major surgery to the
+	callers.
+
+	Fixes: #20369
+
+2017-03-24  Tim Rühsen  <tim.ruehsen@gmx.de>
+
+	Update gnulib
+
+2017-03-22  Christof Horschitz  <christof@nimbusec.com>
+
+	* src/warc.c (warc_write_cdx_record): Escape URLs
+
+2017-03-20  Mike Frysinger  <vapier@gentoo.org>
+
+	Include libunistring headers only when used
+	* src/iri.c: Check for libidn2 < 0.14 to include libunistring headers
+
+	The unistring functions are used only when an older version of libidn2
+	is used, so don't include its headers either w/newer libdin2 versions.
+
+2017-03-18  Tim Rühsen  <tim.ruehsen@gmx.de>
+
+	Fix links to www.robotstxt.org
+	* NEWS: Fix links
+	* doc/wget.texi: Likewise
+	* src/res.c: Likewise
+
+	Reported-by: Noël Köthe
+
+2017-03-13  Tim Rühsen  <tim.ruehsen@gmx.de>
+
+	* tests/WgetTests.pm: Add -d to Wget test options
+
+2017-03-08  Tim Rühsen  <tim.ruehsen@gmx.de>
+
+	Include <arpa/inet.h> for Windows
+	Reported-by: Gisle Vanem
+
+2017-03-08  Tim Rühsen  <tim.ruehsen@gmx.de>
+
+	Fix updating HSTS entries
+	* src/hsts.c (hsts_store_entry): Always update 'created' field
+
+	Fixes: #50490
+	Reported-by: Deian Stefan, Atyansh Jaiswal, Jonathan Luck
+
+2017-03-06  Tim Rühsen  <tim.ruehsen@gmx.de>
+
+	Fix CRLF injection in Wget host part
+	* src/url.c (url_parse): Reject control characters in host part of URL
+
+	Reported-by: Orange Tsai
+
+2017-03-04  Benjamin Esham  <benjamin@esham.io>
+
+	* src/warc.c: Use warc_write_header_uri for all WARC-Target-URI fields
+	The WARC spec requires that all URIs be enclosed in angle brackets. This
+	was being done in most cases, but not for "WARC-Target-URI" fields in
+	WARC blocks of type "response", "resource", "revisit", and "metadata".
+
+2017-02-16  Tim Rühsen  <tim.ruehsen@gmx.de>
+
+	Fix 504 status handling
+	* src/http.c (gethttp): Move 504 handling to correct place.
+	  (http_loop): Fix memeory leak.
+	* testenv/server/http/http_server.py: Add Content-Length header on non-2xx
+	  status codes with a body
+
+	Reported-by: Adam Sampson
+
+2017-02-16  YX Hao  <lifenjoiner@163.com>
+
+	* src/url.c (url_file_name): Do not charset convert local directory
+	In a non-ASCII environment, the local path may contain non-ASCII
+	characters. The server responded file name must be converted before
+	it is concatenated to the local path. Conversion after concatenation
+	may result in 'iconv' errors.
+
+2017-02-15  Tim Rühsen  <tim.ruehsen@gmx.de>
+
+	* configure.ac: Remove manually resetting of LIBICONV variable
+	Fixes: #48193
+	Reported-by: ilovezfs
+	Reported-by: Charles
+
+	* bootstrap.conf: Call gperf to create lib/unicase/special-casing-table.h
+
+	* bootstrap.conf: Fix latest gnulib to work with gperf < 3.1
+
+2017-02-14  Darshit Shah  <darnir@gnu.org>
+
+	Pull GNULib to latest.
+	* gnulib: Pull to latest commit. This fixes an issue with Gperf 3.1
+
+2017-02-12  Tim Rühsen  <tim.ruehsen@gmx.de>
+
+	* src/main.c: Remove double 'verbose' option
+	Fixes: #50290
+
 2017-02-11  Tim Rühsen  <tim.ruehsen@gmx.de>
 
 	* NEWS: update