diff options
Diffstat (limited to 'docs/TODO')
| -rw-r--r-- | docs/TODO | 467 |
1 files changed, 303 insertions, 164 deletions
@@ -12,16 +12,16 @@ All bugs documented in the KNOWN_BUGS document are subject for fixing! 1. libcurl - 1.1 Zero-copy interface 1.2 More data sharing 1.3 struct lifreq 1.4 signal-based resolver timeouts + 1.5 get rid of PATH_MAX + 1.6 Happy Eyeball dual stack connect + 1.7 Modified buffer size approach 2. libcurl - multi interface 2.1 More non-blocking - 2.2 Remove easy interface internally - 2.3 Avoid having to remove/readd handles - 2.4 Fix HTTP Pipelining for PUT + 2.2 Fix HTTP Pipelining for PUT 3. Documentation 3.1 More and better @@ -38,6 +38,7 @@ 5.1 Better persistency for HTTP 1.0 5.2 support FF3 sqlite cookie files 5.3 Rearrange request header order + 5.4 HTTP2/SPDY 6. TELNET 6.1 ditch stdin @@ -45,71 +46,84 @@ 6.3 feature negotiation debug data 6.4 send data in chunks - 7. SSL - 7.1 Disable specific versions - 7.2 Provide mutex locking API - 7.3 Evaluate SSL patches - 7.4 Cache OpenSSL contexts - 7.5 Export session ids - 7.6 Provide callback for cert verification - 7.7 Support other SSL libraries - 7.8 Support SRP on the TLS layer - 7.9 improve configure --with-ssl - - 8. GnuTLS - 8.1 SSL engine stuff - 8.2 SRP - 8.3 check connection - 8.4 non-gcrypt - - 9. Other protocols - - 10. New protocols - 10.1 RSYNC - - 11. Client - 11.1 sync - 11.2 glob posts - 11.3 prevent file overwriting - 11.4 simultaneous parallel transfers - 11.5 provide formpost headers - 11.6 url-specific options - 11.7 metalink support - 11.8 warning when setting an option - - 12. Build - 12.1 roffit - - 13. Test suite - 13.1 SSL tunnel - 13.2 nicer lacking perl message - 13.3 more protocols supported - 13.4 more platforms supported - - 14. Next SONAME bump - 14.1 http-style HEAD output for ftp - 14.2 combine error codes - 14.3 extend CURLOPT_SOCKOPTFUNCTION prototype - - 15. Next major release - 15.1 cleanup return codes - 15.2 remove obsolete defines - 15.3 size_t - 15.4 remove several functions - 15.5 remove CURLOPT_FAILONERROR - 15.6 remove CURLOPT_DNS_USE_GLOBAL_CACHE - 15.7 remove progress meter from libcurl + 7. SMTP + 7.1 Pipelining + 7.2 Graceful base64 decoding failure + 7.3 Enhanced capability support + + 8. POP3 + 8.1 Pipelining + 8.2 Graceful base64 decoding failure + 8.3 Enhanced capability support + + 9. IMAP + 9.1 Graceful base64 decoding failure + 9.2 Enhanced capability support + + 10. LDAP + 10.1 SASL based authentication mechanisms + + 11. New protocols + 11.1 RSYNC + + 12. SSL + 12.1 Disable specific versions + 12.2 Provide mutex locking API + 12.3 Evaluate SSL patches + 12.4 Cache OpenSSL contexts + 12.5 Export session ids + 12.6 Provide callback for cert verification + 12.7 Support other SSL libraries + 12.8 improve configure --with-ssl + 12.9 Support DANE + + 13. GnuTLS + 13.1 SSL engine stuff + 13.2 check connection + + 14. SASL + 14.1 Other authentication mechanisms + + 15. Client + 15.1 sync + 15.2 glob posts + 15.3 prevent file overwriting + 15.4 simultaneous parallel transfers + 15.5 provide formpost headers + 15.6 url-specific options + 15.7 warning when setting an option + 15.8 IPv6 addresses with globbing + + 16. Build + 16.1 roffit + + 17. Test suite + 17.1 SSL tunnel + 17.2 nicer lacking perl message + 17.3 more protocols supported + 17.4 more platforms supported + + 18. Next SONAME bump + 18.1 http-style HEAD output for ftp + 18.2 combine error codes + 18.3 extend CURLOPT_SOCKOPTFUNCTION prototype + + 19. Next major release + 19.1 cleanup return codes + 19.2 remove obsolete defines + 19.3 size_t + 19.4 remove several functions + 19.5 remove CURLOPT_FAILONERROR + 19.6 remove CURLOPT_DNS_USE_GLOBAL_CACHE + 19.7 remove progress meter from libcurl + 19.8 remove 'curl_httppost' from public + 19.9 have form functions use CURL handle argument + 19.10 Add CURLOPT_MAIL_CLIENT option ============================================================================== 1. libcurl -1.1 Zero-copy interface - - Introduce another callback interface for upload/download that makes one less - copy of data and thus a faster operation. - [http://curl.haxx.se/dev/no_copy_callbacks.txt] - 1.2 More data sharing curl_share_* functions already exist and work, and they can be extended to @@ -134,6 +148,49 @@ Also, alarm() provides timeout resolution only to the nearest second. alarm ought to be replaced by setitimer on systems that support it. +1.5 get rid of PATH_MAX + + Having code use and rely on PATH_MAX is not nice: + http://insanecoding.blogspot.com/2007/11/pathmax-simply-isnt.html + + Currently the SSH based code uses it a bit, but to remove PATH_MAX from there + we need libssh2 to properly tell us when we pass in a too small buffer and + its current API (as of libssh2 1.2.7) doesn't. + +1.6 Happy Eyeball dual stack connect + + In order to make alternative technologies not suffer when transitioning, like + when introducing IPv6 as an alternative to IPv4 and there are more than one + option existing simultaneously there are reasons to reconsider internal + choices. + + To make libcurl do blazing fast IPv6 in a dual-stack configuration, this needs + to be addressed: + + http://tools.ietf.org/html/rfc6555 + +1.7 Modified buffer size approach + + Current libcurl allocates a fixed 16K size buffer for download and an + additional 16K for upload. They are always unconditionally part of the easy + handle. If CRLF translations are requested, an additional 32K "scratch + buffer" is allocated. A total of 64K transfer buffers in the worst case. + + First, while the handles are not actually in use these buffers could be freed + so that lingering handles just kept in queues or whatever waste less memory. + + Secondly, SFTP is a protocol that needs to handle many ~30K blocks at once + since each need to be individually acked and therefore libssh2 must be + allowed to send (or receive) many separate ones in parallel to achieve high + transfer speeds. A current libcurl build with a 16K buffer makes that + impossible, but one with a 512K buffer will reach MUCH faster transfers. But + allocating 512K unconditionally for all buffers just in case they would like + to do fast SFTP transfers at some point is not a good solution either. + + Dynamically allocate buffer size depending on protocol in use in combination + with freeing it after each individual transfer? Other suggestions? + + 2. libcurl - multi interface 2.1 More non-blocking @@ -143,7 +200,6 @@ - Name resolves on non-windows unless c-ares is used - NSS SSL connections - - Active FTP connections - HTTP proxy CONNECT operations - SOCKS proxy handshakes - file:// transfers @@ -151,33 +207,7 @@ - The "DONE" operation (post transfer protocol-specific actions) for the protocols SFTP, SMTP, FTP. Fixing Curl_done() for this is a worthy task. -2.2 Remove easy interface internally - - Make curl_easy_perform() a wrapper-function that simply creates a multi - handle, adds the easy handle to it, runs curl_multi_perform() until the - transfer is done, then detach the easy handle, destroy the multi handle and - return the easy handle's return code. This will thus make everything - internally use and assume the multi interface. The select()-loop should use - curl_multi_socket(). - -2.3 Avoid having to remove/readd handles - - curl_multi_handle_control() - this can control the easy handle (while) added - to a multi handle in various ways: - - o RESTART, unconditionally restart this easy handle's transfer from the - start, re-init the state - - o RESTART_COMPLETED, restart this easy handle's transfer but only if the - existing transfer has already completed and it is in a "finished state". - - o STOP, just stop this transfer and consider it completed - - o PAUSE? - - o RESUME? - -2.4 Fix HTTP Pipelining for PUT +2.2 Fix HTTP Pipelining for PUT HTTP Pipelining can be a way to greatly enhance performance for multiple serial requests and currently libcurl only supports that for HEAD and GET @@ -252,6 +282,24 @@ headers use a default value so only headers that need to be moved have to be specified. +5.4 HTTP2/SPDY + + The first drafts for HTTP2 have been published + (http://tools.ietf.org/html/draft-ietf-httpbis-http2-03) and is so far based + on SPDY (http://www.chromium.org/spdy) designs and experiences. Chances are + it will end up in that style. Chrome and Firefox already support SPDY and + lots of web services do. + + It would make sense to implement SPDY support now and later transition into + or add HTTP2 support as well. + + We should base or HTTP2/SPDY work on a 3rd party library for the protocol + fiddling. The Spindy library (http://spindly.haxx.se/) was an attempt to make + such a library with an API suitable for use by libcurl but that effort has + more or less stalled. spdylay (https://github.com/tatsuhiro-t/spdylay) may + be a better option, either used directly or wrapped with a more spindly-like + API. + 6. TELNET @@ -277,25 +325,92 @@ to provide the data to send. use, but inefficient for any other. Sent data should be sent in larger chunks. -7. SSL +7. SMTP + +7.1 Pipelining + + Add support for pipelining emails. + +7.2 Graceful base64 decoding failure + + Rather than shutting down the session and returning an error when the + decoding of a base64 encoded authentication response fails, we should + gracefully shutdown the authentication process by sending a * response to the + server as per RFC4954. + +7.3 Enhanced capability support + + Add the ability, for an application that uses libcurl, to obtain the list of + capabilities returned from the EHLO command. + +8. POP3 + +8.1 Pipelining + + Add support for pipelining commands. + +8.2 Graceful base64 decoding failure + + Rather than shutting down the session and returning an error when the + decoding of a base64 encoded authentication response fails, we should + gracefully shutdown the authentication process by sending a * response to the + server as per RFC5034. + +8.3 Enhanced capability support + + Add the ability, for an application that uses libcurl, to obtain the list of + capabilities returned from the CAPA command. + +9. IMAP -7.1 Disable specific versions +9.1 Graceful base64 decoding failure + + Rather than shutting down the session and returning an error when the + decoding of a base64 encoded authentication response fails, we should + gracefully shutdown the authentication process by sending a * response to the + server as per RFC3501. + +9.2 Enhanced capability support + + Add the ability, for an application that uses libcurl, to obtain the list of + capabilities returned from the CAPABILITY command. + +10. LDAP + +10.1 SASL based authentication mechanisms + + Currently the LDAP module only supports ldap_simple_bind_s() in order to bind + to an LDAP server. However, this function sends username and password details + using the simple authentication mechanism (as clear text). However, it should + be possible to use ldap_bind_s() instead specifing the security context + information ourselves. + +11. New protocols + +11.1 RSYNC + + There's no RFC for the protocol or an URI/URL format. An implementation + should most probably use an existing rsync library, such as librsync. + +12. SSL + +12.1 Disable specific versions Provide an option that allows for disabling specific SSL versions, such as SSLv2 http://curl.haxx.se/bug/feature.cgi?id=1767276 -7.2 Provide mutex locking API +12.2 Provide mutex locking API Provide a libcurl API for setting mutex callbacks in the underlying SSL library, so that the same application code can use mutex-locking independently of OpenSSL or GnutTLS being used. -7.3 Evaluate SSL patches +12.3 Evaluate SSL patches Evaluate/apply Gertjan van Wingerde's SSL patches: http://curl.haxx.se/mail/lib-2004-03/0087.html -7.4 Cache OpenSSL contexts +12.4 Cache OpenSSL contexts "Look at SSL cafile - quick traces look to me like these are done on every request as well, when they should only be necessary once per ssl context (or @@ -305,7 +420,7 @@ to provide the data to send. style connections are re-used. It will make us use slightly more memory but it will libcurl do less creations and deletions of SSL contexts. -7.5 Export session ids +12.5 Export session ids Add an interface to libcurl that enables "session IDs" to get exported/imported. Cris Bailiff said: "OpenSSL has functions which can @@ -313,67 +428,54 @@ to provide the data to send. the state from such a buffer at a later date - this is used by mod_ssl for apache to implement and SSL session ID cache". -7.6 Provide callback for cert verification +12.6 Provide callback for cert verification OpenSSL supports a callback for customised verification of the peer certificate, but this doesn't seem to be exposed in the libcurl APIs. Could it be? There's so much that could be done if it were! -7.7 Support other SSL libraries +12.7 Support other SSL libraries Make curl's SSL layer capable of using other free SSL libraries. Such as MatrixSSL (http://www.matrixssl.org/). -7.8 Support SRP on the TLS layer - - Peter Sylvester's patch for SRP on the TLS layer. Awaits OpenSSL support for - this, no need to support this in libcurl before there's an OpenSSL release - that does it. - -7.9 improve configure --with-ssl +12.8 improve configure --with-ssl make the configure --with-ssl option first check for OpenSSL, then GnuTLS, then NSS... -8. GnuTLS +12.9 Support DANE -8.1 SSL engine stuff + DNS-Based Authentication of Named Entities (DANE) is a way to provide SSL + keys and certs over DNS using DNSSEC as an alternative to the CA model. + http://www.rfc-editor.org/rfc/rfc6698.txt - Is this even possible? + An initial patch was posted by Suresh Krishnaswamy on March 7th 2013 + (http://curl.haxx.se/mail/lib-2013-03/0075.html) but it was a too simple + approach. See Daniel's comments: + http://curl.haxx.se/mail/lib-2013-03/0103.html . libunbound may be the + correct library to base this development on. + +13. GnuTLS -8.2 SRP +13.1 SSL engine stuff - Work out a common method with Peter Sylvester's OpenSSL-patch for SRP on the - TLS to provide name and password. GnuTLS already supports it... + Is this even possible? -8.3 check connection +13.2 check connection Add a way to check if the connection seems to be alive, to correspond to the SSL_peak() way we use with OpenSSL. -8.4 non-gcrypt - - libcurl assumes that there are gcrypt functions available when - GnuTLS is. - - GnuTLS can be built to use libnettle instead as crypto library, - which breaks the previously mentioned assumption - - The correct fix would be to detect which crypto layer that is in use and - adapt our code to use that instead of blindly assuming gcrypt. - -9. Other protocols +14. SASL -10. New protocols +14.1 Other authentication mechanisms -10.1 RSYNC + Add support for GSSAPI to SMTP, POP3 and IMAP. - There's no RFC for protocol nor URI/URL format. An implementation should - most probably use an existing rsync library, such as librsync. +15. Client -11. Client - -11.1 sync +15.1 sync "curl --sync http://example.com/feed[1-100].rss" or "curl --sync http://example.net/{index,calendar,history}.html" @@ -382,12 +484,12 @@ to provide the data to send. remote file is newer than the local file. A Last-Modified HTTP date header should also be used to set the mod date on the downloaded file. -11.2 glob posts +15.2 glob posts Globbing support for -d and -F, as in 'curl -d "name=foo[0-9]" URL'. This is easily scripted though. -11.3 prevent file overwriting +15.3 prevent file overwriting Add an option that prevents cURL from overwriting existing local files. When used, and there already is an existing file with the target file name @@ -395,14 +497,14 @@ to provide the data to send. existing). So that index.html becomes first index.html.1 and then index.html.2 etc. -11.4 simultaneous parallel transfers +15.4 simultaneous parallel transfers The client could be told to use maximum N simultaneous parallel transfers and then just make sure that happens. It should of course not make more than one connection to the same remote host. This would require the client to use the multi interface. http://curl.haxx.se/bug/feature.cgi?id=1558595 -11.5 provide formpost headers +15.5 provide formpost headers Extending the capabilities of the multipart formposting. How about leaving the ';type=foo' syntax as it is and adding an extra tag (headers) which @@ -416,7 +518,7 @@ to provide the data to send. which should overwrite the program reasonable defaults (plain/text, 8bit...) -11.6 url-specific options +15.6 url-specific options Provide a way to make options bound to a specific URL among several on the command line. Possibly by letting ':' separate options between URLs, @@ -430,55 +532,57 @@ to provide the data to send. The example would do a POST-GET-POST combination on a single command line. -11.7 metalink support - - Add metalink support to curl (http://www.metalinker.org/). This is most useful - with simultaneous parallel transfers (11.6) but not necessary. - -11.8 warning when setting an option +15.7 warning when setting an option Display a warning when libcurl returns an error when setting an option. This can be useful to tell when support for a particular feature hasn't been compiled into the library. -12. Build +15.8 IPv6 addresses with globbing + + Currently the command line client needs to get url globbing disabled (with + -g) for it to support IPv6 numerical addresses. This is a rather silly flaw + that should be corrected. It probably involves a smarter detection of the + '[' and ']' letters. + +16. Build -12.1 roffit +16.1 roffit Consider extending 'roffit' to produce decent ASCII output, and use that - instead of (g)nroff when building src/hugehelp.c + instead of (g)nroff when building src/tool_hugehelp.c -13. Test suite +17. Test suite -13.1 SSL tunnel +17.1 SSL tunnel Make our own version of stunnel for simple port forwarding to enable HTTPS and FTP-SSL tests without the stunnel dependency, and it could allow us to provide test tools built with either OpenSSL or GnuTLS -13.2 nicer lacking perl message +17.2 nicer lacking perl message If perl wasn't found by the configure script, don't attempt to run the tests but explain something nice why it doesn't. -13.3 more protocols supported +17.3 more protocols supported Extend the test suite to include more protocols. The telnet could just do ftp or http operations (for which we have test servers). -13.4 more platforms supported +17.4 more platforms supported Make the test suite work on more platforms. OpenBSD and Mac OS. Remove fork()s and it should become even more portable. -14. Next SONAME bump +18. Next SONAME bump -14.1 http-style HEAD output for ftp +18.1 http-style HEAD output for ftp #undef CURL_FTP_HTTPSTYLE_HEAD in lib/ftp.c to remove the HTTP-style headers from being output in NOBODY requests over ftp -14.2 combine error codes +18.2 combine error codes Combine some of the error codes to remove duplicates. The original numbering should not be changed, and the old identifiers would be @@ -488,37 +592,44 @@ to provide the data to send. Candidates for removal and their replacements: CURLE_FILE_COULDNT_READ_FILE => CURLE_REMOTE_FILE_NOT_FOUND + CURLE_FTP_COULDNT_RETR_FILE => CURLE_REMOTE_FILE_NOT_FOUND + CURLE_FTP_COULDNT_USE_REST => CURLE_RANGE_ERROR + CURLE_FUNCTION_NOT_FOUND => CURLE_FAILED_INIT + CURLE_LDAP_INVALID_URL => CURLE_URL_MALFORMAT + CURLE_TFTP_NOSUCHUSER => CURLE_TFTP_ILLEGAL + CURLE_TFTP_NOTFOUND => CURLE_REMOTE_FILE_NOT_FOUND + CURLE_TFTP_PERM => CURLE_REMOTE_ACCESS_DENIED -14.3 extend CURLOPT_SOCKOPTFUNCTION prototype +18.3 extend CURLOPT_SOCKOPTFUNCTION prototype The current prototype only provides 'purpose' that tells what the connection/socket is for, but not any protocol or similar. It makes it hard for applications to differentiate on TCP vs UDP and even HTTP vs FTP and similar. -15. Next major release +10. Next major release -15.1 cleanup return codes +19.1 cleanup return codes curl_easy_cleanup() returns void, but curl_multi_cleanup() returns a CURLMcode. These should be changed to be the same. -15.2 remove obsolete defines +19.2 remove obsolete defines remove obsolete defines from curl/curl.h -15.3 size_t +19.3 size_t make several functions use size_t instead of int in their APIs -15.4 remove several functions +19.4 remove several functions remove the following functions from the public API: @@ -539,18 +650,18 @@ to provide the data to send. curl_multi_socket_all -15.5 remove CURLOPT_FAILONERROR +19.5 remove CURLOPT_FAILONERROR Remove support for CURLOPT_FAILONERROR, it has gotten too kludgy and weird internally. Let the app judge success or not for itself. -15.6 remove CURLOPT_DNS_USE_GLOBAL_CACHE +19.6 remove CURLOPT_DNS_USE_GLOBAL_CACHE Remove support for a global DNS cache. Anything global is silly, and we already offer the share interface for the same functionality but done "right". -15.7 remove progress meter from libcurl +19.7 remove progress meter from libcurl The internally provided progress meter output doesn't belong in the library. Basically no application wants it (apart from curl) but instead applications @@ -559,3 +670,31 @@ to provide the data to send. The progress callback should then be bumped as well to get proper 64bit variable types passed to it instead of doubles so that big files work correctly. + +19.8 remove 'curl_httppost' from public + + curl_formadd() was made to fill in a public struct, but the fact that the + struct is public is never really used by application for their own advantage + but instead often restricts how the form functions can or can't be modified. + + Changing them to return a private handle will benefit the implementation and + allow us much greater freedoms while still maintining a solid API and ABI. + +19.9 have form functions use CURL handle argument + + curl_formadd() and curl_formget() both currently have no CURL handle + argument, but both can use a callback that is set in the easy handle, and + thus curl_formget() with callback cannot function without first having + curl_easy_perform() (or similar) called - which is hard to grasp and a design + mistake. + +19.10 Add CURLOPT_MAIL_CLIENT option + + Rather than use the URL to specify the mail client string to present in the + HELO and EHLO commands, libcurl should support a new CURLOPT specifically for + specifing this data as the URL is non-standard and to be honest a bit of a + hack ;-) + + Please see the following thread for more information: + http://curl.haxx.se/mail/lib-2012-05/0178.html + |