Update to version 2.0
HAProxy 2.0.0 was released on 2019/06/16. It added 63 new commits after version 2.0-dev7.
There were a few last-minute bug reports that started to make me worry a bit but in the end these were nothing dramatic and quickly addressed. Aside the usual bug fixes, this version mostly contains some documentation and build updates. The most visible change will concern Linux users (most users in fact), do to the removal of the totally obsolete "linux22", "linux24", "linux24e" and "linux2628" targets in the Makefile. Now there is a single "linux-glibc" one provided for environments running a recent (read "still supported") Linux kernel with a similarly recent glibc. No more worries about what was still enabled in 2.6.28 or for your libc! As a side effect of this, namespaces, TCP Fast Open and getaddrinfo() are now enabled by default on this combination. The very few remaining improvements were in any order :
-
a small change on the state file to use server name only and not a more-or-less random combination of the server ID and name anymore, as clearly people were confused by this in the past and we now have everything to rely solely on names across a cluster now ;
-
some HTX updates to pass the H2 scheme between the two sides so that applications that expect to be called as "http:" will receive it ;
-
a new "http-request replace-uri" directive to more easily get rid of any remaining reqrep users may still have in their configurations and which will be removed from 2.1 ;
-
report of each process' version in "show proc" on the master process command line (nice to detect failed upgrades) ;
-
51degrees now supports HTX and provided a dummy library so that there is no more code that we cannot build in travis, and I hope we can now continue to enforce such a rule ;
I must say I am very happy with this release, especially with how smooth it went when approaching the release over the last few weeks. Contributors were reasonable in general, avoiding to send risky patches, and testers did a great coverage, being able to report interesting issues. Seeing the type of issues we've had last is very encouraging and is what makes me want to release now without waiting more. Thanks to all these improvements, I think we now have the cleanest version ever issued at the dot-zero release. I sincerely hope new versions will continue on this trend that benefits a lot from the new development cycle.
For those who haven't followed the development cycle closely, I'll try to quickly summarize the changes since the previous LTS version (1.8). As most of you know, 1.9 will not be maintained for a long time and should mostly be seen as a technological preview or technical foundation for 2.0.
Thus what was added since 1.8 :
-
new internal native HTTP representation called HTX, was already in 1.9 and is now enabled by default in 2.0 ;
-
end-to-end HTTP/2 support including trailers and continuation frames, as needed for gRPC ; HTTP/2 may also be upgraded from HTTP/1.1 using the H2 preface;
-
server connection pooling and more advanced reuse, with ALPN protocol negotiation (already in 1.9) ;
-
layer 7 retries, allowing to use 0-RTT and TCP Fast Open to the servers as well as on the frontend ;
-
much more scalable multi-threading, which is even enabled by default on platforms where it was successfully tested ; by default, as many threads are started as the number of CPUs haproxy is allowed to run on. This removes a lot of configuration burden in VMs and containers ;
-
automatic maxconn setting for the process and the frontends, directly based on the number of available FDs (easier configuration in containers and with systemd) ;
-
logging to stdout for use in containers and systemd (already in 1.9). Logs can now provide micro-second resolution for some events ;
-
peers now support SSL, declaration of multiple stick-tables directly in the peers section, and synchronization of server names, not just IDs ;
-
In master-worker mode, the master process now exposes its own CLI and can communicate with all other processes (including the stopping ones), even allowing to connect to their CLI and check their state. It is also possible to start some sidecar programs and monitor them from the master, and the master can automatically kill old processes that survived too many reloads ;
-
the incoming connections are load-balanced between all threads depending on their load to minimize the processing time and maximize the capacity (already in 1.9) ;
-
the SPOE connection load-balancing was significantly improved in order to reduce high percentiles of SPOA response time (already in 1.9) ;
-
the "random" load balancing algorithm and a power-of-two-choices variant were introduced ;
-
statistics improvements with per-thread counters for certain things, and a prometheus exporter for all our statistics;
-
lots of debugging help, it's easier to produce a core dump, there are new commands on the CLI to control various things, there is a watchdog to fail cleanly when a thread deadlock or a spinning task are detected, so overall it should provide a better experience in field and less round trips between users and developers (hence less stress during an incident).
-
all 3 device detection engines are now compatible with multi-threading and can be build-tested without any external dependencies ;
-
"do-resolve" http-request action to perform a DNS resolution on any, sample, and resolvers now support relying on /etc/resolv.conf to match the local resolver ;
-
log sampling and balancing : it's now possible to send 1 log every 10 to a server, or to spread the logging load over multiple log servers;
-
a new SPOA agent (spoa_server) allows to interface haproxy with Python and Lua programs ;
-
support for Solaris' event ports (equivalent of kqueue or epoll) which will significantly improve the performance there when dealing with numerous connections ;
-
some warnings are now reported for some deprecated options that will be removed in 2.1. Since 2.0 is long term supported, there's no emergency to convert them, however if you see these warnings, you need to understand that you're among their extremely rare users and just because of this you may be taking risks by keeping them ;
-
A new SOCKS4 server-side layer was provided ; it allows outgoing connections to be forwarded through a SOCKS4 proxy (such as ssh -D).
-
priority- and latency- aware server queues : it is possible now to assign priorities to certain requests and/or to give them a time bonus or penalty to refine control of the traffic and be able to engage on SLAs.
-
internally the architecture was significantly redesigned to allow to further improve performance and make it easier to implement protocols that span over multiple layers (such as QUIC). This work started in 1.9 and will continue with 2.1.
-
the I/O, applets and tasks now share the same multi-threaded scheduler, giving a much better responsiveness and fairness between all tasks as is visible with the CLI which always responds instantly even under extreme loads (started in 1.9) ;
-
the internal buffers were redesigned to ease zero-copy operations, so that it is possible to sustain a high bandwidth even when forwarding HTTP/1 to/from HTTP/2 (already in 1.9) ;
-
new test suite based on Vtest (formerly varnish-test) allowing each developer to quickly verify that he didn't break anything. This has already caught many bugs that could have been otherwise released by sticking to the previous process, and is directly responsible for the increase in quality of this release ;
-
continuous integration using Travis-CI allows us to improve build quality and to detect regressions much faster ;
This list is huge and I built it by reading my previous announcement e-mails so it is certain I missed some items which may be of importance to some of you, or that you have contributed yourself (sorry about that, please voice in here if so). Some of my coworkers have spent a much longer time setting up a detailed list with examples in an article posted here :
https://www.haproxy.com/blog/haproxy-2-0-and-beyond/
[disclaimer: I only quickly glanced over it for now so if you find missing items in either list, well, consider that none of us invented anything and that what matters is the union of these lists :-)]
I was also told that some of my HapTech colleagues have been working tirelessly on a dataplane API to more dynamically reconfigure HAProxy, on a Kubernetes Ingress Controller and on an SPOA agent to perform some traffic replication to a development server, sometimes called "traffic shadowing". It's not a secret that I'm clueless in these areas which is why I haven't followed these efforts closely, but my understanding is that we'll get some imminent news about them.
The development will go on with 2.1 which will not be LTS, so it will experience quite some breakage to prepare 2.2 which will be LTS and expected approximately at the same date next year.
I'm going to deploy this shiny new version right now on the haproxy.org servers (not kidding, we're already on dev3), and I don't expect any bad joke! I hope we won't need a 2.0.1 before one week or two.
Now let's see if I managed to mangle a link or an upload again, as is the tradition with about every single new major release
Please find the usual URLs below : Site index : http://www.haproxy.org/ Discourse : http://discourse.haproxy.org/ Slack channel : https://slack.haproxy.org/ Issue tracker : https://github.com/haproxy/haproxy/issues Sources : http://www.haproxy.org/download/2.0/src/ Git repository : http://git.haproxy.org/git/haproxy-2.0.git/ Git Web browsing : http://git.haproxy.org/?p=haproxy-2.0.git Changelog : http://www.haproxy.org/download/2.0/src/CHANGELOG Cyril's HTML doc : http://cbonte.github.io/haproxy-dconv/
Willy
Complete changelog from 2.0-dev7 : Baptiste Assmann (1): MEDIUM: server: server-state only rely on server name
Ben51Degrees (3): BUG/MINOR: 51d/htx: The _51d_fetch method, and the methods it calls are now HTX aware. MINOR: 51d: Added dummy libraries for the 51Degrees module for testing. BUILD/MINOR: 51d: Updated build registration output to indicate thatif the library is a dummy one or not.
Christopher Faulet (12): BUG/MINOR: fl_trace/htx: Be sure to always forward trailers and EOM BUG/MINOR: channel/htx: Call channel_htx_full() from channel_full() BUG/MINOR: http: Use the global value to limit the number of parsed headers BUG/MINOR: htx: Detect when tail_addr meet end_addr to maximize free rooms BUG/MEDIUM: htx: Don't change position of the first block during HTX analysis CLEANUP: channel: Remove channel_htx_fwd_payload() and channel_htx_fwd_all() BUG/MEDIUM: proto_htx: Introduce the state ENDING during forwarding MINOR: htx: Add 3 flags on the start-line to deal with the request schemes MINOR: h2: Set flags about the request's scheme on the start-line MINOR: mux-h1: Set flags about the request's scheme on the start-line MINOR: mux-h2: Forward clients scheme to servers checking start-line flags BUG/MINOR: mux-h1: Wake busy mux for I/O when message is fully sent
Ilya Shipitsin (2): BUILD: travis-ci: add 51Degree device detection, update openssl to 1.1.1c BUILD: travis-ci: TFO and GETADDRINFO are now enabled by default
Kazuo Yagi (1): MINOR: doc: Remove -Ds option in man page
Olivier Houchard (7): MINOR: fd: Don't use atomic operations when it's not needed. BUG/MEDIUM: h1: Don't wait for handshake if we had an error. BUG/MEDIUM: h1: Wait for the connection if the handshake didn't complete. BUG/MEDIUM: connections: Don't call shutdown() if we want to disable linger. BUG/MEDIUM: connections: Don't use ALPN to pick mux when in mode TCP. BUG/MEDIUM: connections: Don't try to send early data if we have no mux. BUG/MEDIUM: ssl: Make sure we initiate the handshake after using early data.
Tim Duesterhus (2): BUILD: Silence gcc warning about unused return value DOC: Fix typos in CONTRIBUTING
William Lallemand (7): DOC: mworker-prog: documentation for the program section MINOR: mworker: change formatting in uptime field of "show proc" MINOR: mworker: add the HAProxy version in "show proc" MINOR: doc: add master-worker in the man page MINOR: doc: mention HAPROXY_LOCALPEER in the man MINOR: doc: update the manpage and usage message about -S DOC: add some environment variables in section 2.3
Willy Tarreau (28): MINOR: http: add a new "http-request replace-uri" action CLEANUP: 51d: move the 51d dummy lib to contrib/51d/src to match the real lib BUG/MINOR: task: prevent schedulable tasks from starving under high I/O activity CLEANUP: connection: rename the wait_event.task field to .tasklet CLEANUP: tasks: rename task_remove_from_tasklet_list() to tasklet_remove_* BUILD: makefile: clarify the "help" output and list options BUG: tasks: fix bug introduced by latest scheduler cleanup BUG/MEDIUM: mux-h2: fix early close with option abortonclose BUG/MEDIUM: mux-h2: properly account for the appended data in HTX BUILD: makefile: further clarify the "help" output and list targets BUILD: makefile: rename "linux2628" to "linux-glibc" and remove older targets BUILD: travis-ci: switch to linux-glibc instead of linux2628 DOC: update few references to the linux* targets and change them to linux-glibc BUILD: makefile: detect and reject recently removed linux targets BUILD: makefile: enable linux namespaces by default on linux BUILD: makefile: enable TFO on linux platforms BUILD: makefile: enable getaddrinfo on the linux-glibc target DOC: small updates to the CONTRIBUTING file CLEANUP: removed obsolete examples an move a few to better places DOC: update the outdated ROADMAP file DOC: create a BRANCHES file to explain the life cycle DOC: mention in INSTALL haproxy 2.0 is a long-term supported stable version BUILD: makefile: make the obsolete target detection compatible with make-3.80 BUILD: tools: work around an internal compiler bug in gcc-3.4 BUILD: pattern: work around an internal compiler bug in gcc-3.4 BUILD: makefile: enable USE_RT on Solaris BUILD: makefile: do not use echo -n DOC: mention a few common build errors in the INSTALL file