Changes in Varnish 6.6¶
For information about updating your current Varnish deployment to the new version, see Upgrading to Varnish 6.6.
A more detailed and technical account of changes in Varnish, with links to issues that have been fixed and pull requests that have been merged, may be found in the change log.
varnishd¶
Arguments¶
- varnishdnow supports the- -b noneargument to start with only the builtin VCL and no backend at all.
Parameters¶
- The - validate_headersparameter has been added to control header validation.
- The - ban_cutoffparameter now refers to the overall length of the ban list, including completed bans, where before only non-completed (“active”) bans were counted towards- ban_cutoff.
- The - vary_noticeparameter has been added to control the threshold for the new Vary Notice.
feature Flags¶
- The - busy_stats_ratefeature flag has been added to ensure statistics updates (as configured using the- thread_stats_rateparameter) even in scenarios where worker threads never run out of tasks and may remain forever busy.
Accounting¶
Body bytes accounting has been fixed to always represent the number of body bytes moved on the wire, exclusive of protocol-specific overhead like HTTP/1 chunked encoding or HTTP/2 framing.
This change affects counters like
- MAIN.s_req_bodybytes,
- MAIN.s_resp_bodybytes,
- VBE.*.*.bereq_bodybytesand
- VBE.*.*.beresp_bodybytes
as well as the VSL records
- ReqAcct,
- PipeAcctand
- BereqAcct.
Session Close Reasons¶
The connection close reason has been fixed to properly report
SC_RESP_CLOSE / resp_close where previously only
SC_REQ_CLOSE / req_close was reported.
For failing PROXY connections, SessClose now provides more
detailed information on the cause of the failure.
The session close reason logging/statistics for HTTP/2 connections have been improved.
Vary Notice¶
A log (VSL) Notice record is now emitted whenever more than
vary_notice variants are encountered in the cache for a specific
hash. The new vary_notice parameter defaults to 10.
Changes to VCL¶
Header Validation¶
Unless the new validate_headers feature is disabled, all newly set
headers are now validated to contain only characters allowed by
RFC7230. A (runtime) VCL failure is triggered if not.
VCL variables¶
- The - client.identityvariable is now accessible on the backend side.
- The variables - bereq.is_hitpassand- bereq.is_hitmisshave been added to the backend side matching- req.is_hitpassand- req.is_hitmisson the client side.
- The - bereq.xidvariable is now also available in- vcl_pipe {}
- The - resp.protovariable is now read-only as it should have been for long, like the other- *.protovariables.
Other changes to VCL¶
- Long strings in VCL can now also be denoted using - """ ... """in addition to the existing- {" ... "}.
- The - ban()builtin is now deprecated and should be replaced with std.ban().
- Trying to use - std.rollback()from- vcl_pipenow results in VCL failure.
- The modulus operator - %has been added to VCL.
- return(retry)from- vcl_backend_error {}now correctly resets- beresp.statusand- beresp.reason.
- The builtin VCL has been reworked: VCL code has been split into small subroutines, which custom VCL can prepend custom code to. - This allows for better integration of custom VCL and the built-in VCL and better reuse. 
VMODs¶
directors.shard()¶
- The shard director now supports reconfiguration (adding/removing backends) of several instances without any special ordering requirement. 
- Calling the shard director - .reconfigure()method is now optional. If not called explicitly, any shard director backend changes are applied at the end of the current task.
- Shard director - Errorlog messages with- (notice)have been turned into- Noticelog messages.
- All shard - Errorand- Noticemessages now use the unified prefix- vmod_directors: shard %s.
std.set_ip_tos()¶
The set_ip_tos() function from the bundled std vmod now sets
the IPv6 Taffic Class (TCLASS) when used on an IPv6 connection.
std.ban() and std.ban_error()¶
The std.ban() and std.ban_error() functions have been added to
the std vmod, allowing VCL to check for ban errors. A typical
usage pattern with the new interface is:
if (std.ban(...)) {
  return(synth(200, "Ban added"));
} else {
  return(synth(400, std.ban_error()));
}
cookie functions¶
The filter_re, keep_re and get_re functions from the
bundled cookie vmod have been changed to take the VCL_REGEX
type. This implies that their regular expression arguments now need to
be literal, whereas before they could be taken from some other
variable or function returning VCL_STRING.
Note that these functions never actually handled dynamic regexen, the string passed with the first call was compiled to a regex, which was then used for the lifetime of the respective VCL.
varnishlog¶
- See Accounting for changes to accounting-related VSL records. 
- See Session Close Reasons for a change affecting - SessClose.
- Three new - TimestampVSL records have been added to backend request processing:- The - Processtimestamp after- return(deliver)or- return(pass(x))from- vcl_backend_response,
- the - Fetchtimestamp before a backend connection is requested and
- the - Connectedtimestamp when a connection to a regular backend (VBE) is established, or when a recycled connection was selected for reuse.
 
- The - FetchErrorlog message- Timed out reusing backend connectionhas been renamed to- first byte timeout (reused connection)to clarify that it is emit for effectively the same reason as- first byte timeout.
- ExpKilllog (VSL) records are now masked by default. See the- vsl_maskparameter.
- Comparisons of numbers in VSL queries have been improved to match better the behavior which is likely expected by users who have not read the documentation in all detail. 
- See Vary Notice for information on a newly added - Noticelog (VSL) record.
varnishncsa¶
- The - %{X}Tformat has been added to- varnishncsa, which generalizes- %Dand- %T, but also support milliseconds (- ms) output.
- The - varnishncsa- -Eargument to show ESI requests has been changed to imply- -c(client mode). This behavior is now shared by all log utilities, and- -cno longer includes ESI requests.
varnishadm¶
- The - vcl.discardCLI command can now be used to discard more than one VCL with a single command, which succeeds only if all given VCLs could be discarded (atomic behavior).
- The - vcl.discardCLI command now supports glob patterns for vcl names.
- The - vcl.depsCLI command has been added to output dependencies between VCLs (because of labels and- return(vcl)statements).
- varnishadmnow has the- -poption to disable readline support for use in scripts and as a generic CLI connector.
varnishstat¶
- See Accounting for changes to accounting-related counters. 
- See Session Close Reasons for a change affecting - MAIN.sc_*counters.
- The - MAIN.esi_reqcounter has been added as a statistic of the number of ESI sub requests created.
- The - MAIN.s_bgfetchcounter has been added as a statistic on the number of background fetches issued.
- varnishstatnow avoids display errors of gauges which previously could underflow to negative values, being displayed as extremely high positive values.- The - -roption and the- rkey binding have been added to return to the previous behavior. When raw mode is active in- varnishstatinteractive (curses) mode, the word- RAWis displayed at the right hand side in the lower status line.
varnishtest¶
Various improvements have been made to the varnishtest facility:
- the - loopkeyword now works everywhere
- HTTP/2 logging has been improved 
- Default HTTP/2 parameters have been tweaked 
- Varnish listen address information is now available by default in the macros - ${vNAME_addr},- ${vNAME_port}and- ${vNAME_sock}. Macros by the names- ${vNAME_SOCKET_*}contain the address information for each listen socket as created with the- -aargument to- varnishd.
- Synchronization points for counters (VSCs) have been added as - varnish vNAME -expect PATTERN OP PATTERN
- varnishtest now also works with IPv6 setups 
- feature ipv4and- feature ipv6can be used to control execution of test cases which require one or the other protocol.
- haproxy arguments can now be externally provided through the - HAPROXY_ARGSvariable.
- logexpect now supports alternatives with the - expect ? ...syntax and negative matches with the- fail add ...and- fail clearsyntax.
- The overall logexpect match expectation can now be inverted using the - -errargument.
- Numeric comparisons for HTTP headers have been added: - -lt,- -le,- -eq,- -ne,- -ge,- -gt
- rxdata -somehas been fixed.
Other Changes to Varnish Utilities¶
All varnish tools using the VUT library utilities for argument
processing now support the --optstring argument to return a string
suitable for use with getopts from shell scripts.
Developer: Changes for VMOD authors¶
VMOD/VCL interface¶
- The - VCL_REGEXdata type is now supported for VMODs, allowing them to use regular expression literals checked and compiled by the VCL compiler infrastructure.- Consequently, the - VRT_re_init()and- VRT_re_fini()functions have been removed, because they are not required and their use was probably wrong anyway.
- The - VCL_SUBdata type is now supported for VMODs to save references to subroutines to be called later using- VRT_call(). Calls from a wrong context (e.g. calling a subroutine accessing- reqfrom the backend side) and recursive calls fail the VCL.- See VMOD - Varnish Modules in the Reference Manual. 
VMOD functions can also return the
VCL_SUBdata type for calls from VCL as incall vmod.returning_sub();.
- VRT_check_call()can be used to check if a- VRT_call()would succeed in order to avoid the potential VCL failure in case it would not.- It returns - NULLif- VRT_call()would make the call or an error string why not.
- VRT_handled()has been added, which is now to be used instead of access to the- handlingmember of- VRT_CTX.
- vmodtool.pyhas been improved to simplify Makefiles when many VMODs are built in a single directory.
General API¶
- VRT_ValidHdr()has been added for VMODs to conduct the same check as the whatsnew_changes_6.6_header_validation feature, for example when headers are set by VMODs using the- cache_http.cFunctions like- http_ForceHeader()from untrusted input.
- Client and backend finite state machine internals ( - enum req_stepand- enum fetch_step) have been removed from- cache.h.
- The - verrno.hheader file has been removed and merged into- vas.h
- The - pdiff()function declaration has been moved from- cache.hto- vas.h.
VSA¶
- The - VSA_getsockname()and- VSA_getpeername()functions have been added to get address information of file descriptors.
Private Pointers¶
- The interface for private pointers in VMODs has been changed: - The - freepointer in- struct vmod_privhas been replaced with a pointer to- struct vmod_priv_methods, to where the pointer to the former free callback has been moved as the- finimember.
- The former free callback type has been renamed from - vmod_priv_free_fto- vmod_priv_fini_fand as gained a- VRT_CTXargument
 
- The - VRT_priv_task_get()and- VRT_priv_top_get()functions have been added to VRT to allow vmods to retrieve existing- PRIV_TASK/- PRIV_TOPprivate pointers without creating any.
Backends¶
- The VRT backend interface has been changed: - struct vrt_endpointhas been added describing a UDS or TCP endpoint for a backend to connect to.- Endpoints also support a preamble to be sent with every new connection. 
- This structure needs to be passed via the - endpointmember of- struct vrt_backendwhen creating backends with- VRT_new_backend()or- VRT_new_backend_clustered().
 
- VRT_Endpoint_Clone()has been added to facilitate working with endpoints.
Filters (VDP/VFP)¶
- Many filter (VDP/VFP) related signatures have been changed: - vdp_init_f
- vdp_fini_f
- vdp_bytes_f
- VDP_bytes()
 - as well as - struct vdp_entryand- struct vdp_ctx- VFP_Push()and- VDP_Push()are no longer intended for VMOD use and have been removed from the API.
- The VDP code is now more strict about - VDP_END, which must be sent down the filter chain at most once. Care should be taken to send- VDP_ENDtogether with the last payload bytes whenever possible.
Stevedore API¶
- The stevedore API has been changed: - OBJ_ITER_FINALhas been renamed to- OBJ_ITER_END
- ObjExtend()signature has been changed to also cover the- ObjTrimStore()use case and
- ObjTrimStore()has been removed.
 
Developer: Changes for Authors of Varnish Utilities¶
libvarnishapi¶
- The - VSC_IsRaw()function has been added to- libvarnishapito query if a gauge is being returned raw or adjusted (see varnishstat -r option).
eof