Varnish Cache

Introduction

VCL (Varnish Configuration Language) is a domain-specific language designed for defining request handling logic in Varnish Cache. Unlike traditional configuration files, VCL is a procedural language that gets compiled into C code and then into native machine code for optimal performance.

Varnish Cache is a high-performance HTTP accelerator that sits between your application servers and clients, caching content to dramatically reduce server load and improve response times.

VCL Architecture Overview

Key Components

1. libvcc: VCL Compiler library located in lib/libvcc/
2. VCL Runtime: Execution engine in bin/varnishd/cache/
3. Management Process: VCL lifecycle management in bin/varnishd/mgt/
4. VMODs: Extension modules that extend VCL functionality

VCL Processing Flow

Client Request
    ↓
vcl_recv → vcl_hash → [Cache Lookup]
    ↓                       ↓
    ├─ vcl_hit ──────────────┘
    ├─ vcl_miss → vcl_backend_fetch → vcl_backend_response
    └─ vcl_pass → vcl_backend_fetch → vcl_backend_response
                                            ↓
                                      vcl_deliver
                                            ↓
                                    Client Response

VCL Compilation Process

Compilation Pipeline

Compilation Steps

1. Source Processing (vcc_source.c)

The VCL source code is read and prepared for compilation:

// Location: lib/libvcc/vcc_source.c
struct source *vcc_new_source(const char *src, const char *kind, const char *name);

• Handles both inline VCL strings and file-based sources
• Tracks source location for error reporting
• Supports include directives for modular VCL files

2. Lexical Analysis (vcc_token.c)

The tokenizer breaks down VCL source into tokens:

Token Types:

• Keywords: sub, if, else, return, set, unset
• Identifiers: Variable names, function names
• Literals: Strings, numbers, durations, bytes
• Operators: ==, !=, ~, !~, &&, ||
• Special tokens: {, }, ;, ,, .

3. Parsing (vcc_parse.c)

The parser builds an Abstract Syntax Tree (AST) from tokens:

Parsing Components:

• Subroutine definitions: sub vcl_recv { ... }
• Backend definitions: backend default { ... }
• ACL definitions: acl purge_acl { ... }
• Probe definitions: probe health_check { ... }
• Conditional statements: if, else, elsif
• Actions: set, unset, return, call

VCL State Machine

VCL processing is organized as a state machine where each method represents a state. The flow through states is controlled by return() actions.

Key VCL Methods

Client-Side Methods

Backend-Side Methods

Request Processing Flow

Client Request Flow

Pass Mode Flow

Common VCL Variables

Client-side (vcl_recv, vcl_deliver)

• req.url - Request URL
• req.method - HTTP method
• req.http.* - Request headers
• client.ip - Client IP address
• resp.http.* - Response headers
• resp.status - Response status code

Backend-side (vcl_backend_*)

• bereq.url - Backend request URL
• bereq.http.* - Backend request headers
• beresp.status - Backend response status
• beresp.ttl - Time to live
• beresp.http.* - Backend response headers

Object (vcl_hit, vcl_deliver)

• obj.ttl - Object TTL
• obj.grace - Grace period
• obj.hits - Number of hits
• obj.http.* - Cached headers

VCL Type System

Built-in Types

Type Conversion Examples

# String to boolean (implicit)
if (req.http.X-Forwarded-For) {  # Non-empty string = true
    # ...
}

# Int to string (implicit)
set resp.http.X-Count = 42;  # 42 converted to "42"

# Duration to string
set resp.http.X-TTL = obj.ttl;  # "120.000" for 120s TTL

# Explicit conversions via std VMOD
import std;
std.integer(req.http.X-Count, 0);  # String to int

VMOD System

VMODs (Varnish Modules) extend VCL functionality through shared libraries.

VMOD Loading Process

1. Import Declaration: import std;
2. Symbol Resolution: VCC loads VMOD descriptor
3. Type Checking: Validate function calls
4. Dynamic Loading: Load .so at runtime
5. Initialization: Call VMOD event handlers

Grace Mode and Stale Content

Grace Configuration

sub vcl_backend_response {
    # TTL for fresh content
    set beresp.ttl = 10m;

    # Grace period for stale content
    set beresp.grace = 1h;

    # Keep for additional time
    set beresp.keep = 2h;
}

sub vcl_hit {
    if (obj.ttl >= 0s) {
        # Fresh object
        return (deliver);
    } else if (obj.ttl + obj.grace > 0s) {
        # Stale but within grace
        return (deliver);
    } else {
        # Too old
        return (miss);
    }
}

Synthetic Responses

Example Implementation

sub vcl_recv {
    if (req.url ~ "^/health") {
        return (synth(200, "OK"));
    }
}

sub vcl_synth {
    if (resp.status == 200) {
        set resp.http.Content-Type = "application/json";
        set resp.body = {"{"}"status":"healthy"{"}"}";
        return (deliver);
    }
}

VCL Variable Hierarchy

Variable Accessibility by Method

Legend:

• R = Read only
• R/W = Read and Write
• - = Not accessible

Performance Best Practices

Good Practices

# GOOD: Minimize string operations
sub vcl_recv {
    # Only set if needed
    if (req.url ~ "^/api/") {
        set req.http.X-API = "true";
    }
}

# GOOD: Use simpler checks first
if (req.url ~ "^/very/complex/") {
    if (req.url ~ "with|multiple|alternatives") {
        # ...
    }
}

Avoid These Patterns

# BAD: String concatenation in hot path
sub vcl_recv {
    set req.http.X-Custom = "prefix-" + req.url + "-suffix";
}

# BAD: Complex regex in vcl_recv
if (req.url ~ "^/very/complex/regex/pattern/(with|multiple|alternatives)/.*$") {
    # ...
}

Debugging and Introspection

VCL Logging

import std;

sub vcl_recv {
    std.log("vcl_recv: " + req.url);
    std.log("Client IP: " + client.ip);
    std.log("Method: " + req.method);
}

VSL (Varnish Shared Log)

View VCL execution for debugging:

# View VCL execution
varnishlog -g request -i VCL_call,VCL_return
 
# Example output:
# VCL_call       RECV
# VCL_return     hash
# VCL_call       HASH
# VCL_return     lookup
# VCL_call       MISS
# VCL_return     fetch

VCL File Locations

Summary

VCL's internal architecture demonstrates sophisticated compiler design and runtime engineering:

1. Multi-phase compilation: Lexing → Parsing → Type checking → C generation → Native code
2. State machine execution: Predictable request flow through well-defined states
3. Type safety: Strong typing with implicit conversions where sensible
4. Extensibility: VMOD system for custom functionality
5. Performance: Compiled to native code with optimizations
6. Flexibility: Dual-layer VCL (user + builtin) for sensible defaults

Key Takeaways

• VCL is compiled, not interpreted, for maximum performance
• The state machine provides clear separation of concerns
• Built-in VCL ensures sane defaults while allowing overrides
• VMODs extend functionality without core changes
• Type system prevents common errors at compile time
• Event system manages VCL lifecycle cleanly

Additional Resources

• Official Varnish Documentation (opens in a new tab)
• VCL Reference (opens in a new tab)
• Varnish Book (opens in a new tab)

Version: Varnish 7.7.0 Last Updated: 2025