Lua Rules

Lua rules

Lua rules provide advanced parsing and enrichment capabilities for complex data transformation scenarios. They execute custom logic to extract fields, apply conditional processing, and enrich events with contextual information.

When to use Lua rules

• Vendor-specific formats that require multiple extractions.
• Conditional logic to interpret message variants.
• Field mappings, lookups, or normalization across sources.
• Message reformatting to create consistent, readable content.
• Context enrichment that adds durable, searchable tags.

Event object API

Lua rules operate on an event object with these key properties:

• event.message - The original log message text
• event.host - Source hostname or IP address
• event.program - Program or service name
• event.severity - Numeric severity level (0-7)
• event.facility - Syslog facility code
• event.user_tags - Table for custom searchable tags
• event.extra_fields - Table for additional structured data

Available helper functions

LogZilla provides helper functions for common parsing tasks:

• get_kv_parser(sep, delim, quote) - Creates key-value parser
• starts_with(str, prefix) - Check string prefix
• ends_with(str, suffix) - Check string suffix
• get_port_name(port) - Convert port number to service name
• push_event(event) - Generate extra event from current processing context

Format examples

Lua rules are stored as .lua files in the rule directory. The following example shows a basic structure for parsing key-value pairs:

lua
Wrap: OnCopyFullscreen
-- Parse key-value pairs from messages like: srcip="192.168.1.10" dstip="8.8.8.8" action="accept"
local kv_parser = get_kv_parser(' ', '=', '"')

function process(event)
    -- Check if this looks like a key-value format
    if string.find(event.message, '="') then
        -- Set program name for identification
        event.program = "CustomApp"
        
        -- Parse all key-value pairs from the message
        local kvpairs = kv_parser:match(event.message)
        
        -- Convert parsed pairs to searchable user tags
        for key, value in pairs(kvpairs) do
            if key == "srcip" then
                event.user_tags["SrcIP"] = value
            elseif key == "dstip" then
                event.user_tags["DstIP"] = value
            elseif key == "action" then
                event.user_tags["Action"] = value
            end
        end
    end
end

For conditional tagging based on message content:

lua
Wrap: OnCopyFullscreen
function process(event)
    -- Only process events from specific sources
    if event.program == "MyApp" then
        -- Add severity tags based on message content
        if string.find(event.message, "ERROR") then
            event.user_tags["Severity"] = "High"
            event.user_tags["Alert"] = "Error Detected"
        elseif string.find(event.message, "WARN") then
            event.user_tags["Severity"] = "Medium"
            event.user_tags["Alert"] = "Warning"
        end
        
        -- Extract and tag IP addresses
        local ip_pattern = "(%d+%.%d+%.%d+%.%d+)"
        local ip = string.match(event.message, ip_pattern)
        if ip then
            event.user_tags["IP_Address"] = ip
        end
    end
end

How Lua rules work

• The parsing engine evaluates installed rules in a defined order.
• Each rule inspects the event and decides whether to act.
• Rules can enrich events by writing into structured fields and tags.
• Rules can perform controlled rewrites for consistency and clarity.
• Rules can skip events that do not meet criteria without overhead.
• Rules can generate new events if needed for complex scenarios.

The result is a normalized event with useful fields for search, dashboards, and triggers.

Common enrichment patterns

• Add normalized tags, such as device role, location, or application.
• Extract identifiers (for example, user, session, interface) into dedicated fields.
• Convert vendor codes into readable labels.
• Reconstruct terse messages into human-readable summaries when needed.

Creating new events from rules

Since version 6.39, Lua rules can create additional events derived from the original event or created completely from scratch. This capability enables complex processing scenarios such as splitting message into multiple events.

The push_event(new_event) helper function creates new events. The new_event object passed to this function will be processed and indexed as any other event.

The Event:new() constructor creates new, empty events. All fields must be populated with appropriate values:

lua
Wrap: OnCopyFullscreen
local new_event = Event:new()

new_event:set_timestamp_now() -- this updates event timestamp to current ts
new_event.message = "Sample log message"
new_event.host = "localhost"
new_event.program = "MyApp"
new_event.user_tags["CustomTag"] = "Value"

push_event(new_event)

Cloning existing events and modifying specific fields provides a more convenient approach. The clone() method supports this pattern:

lua
Wrap: OnCopyFullscreen
local cloned_event = event:clone()

cloned_event.message = "Modified log message"

push_event(cloned_event)

Generated events are not subject to rule processing to prevent infinite loops.

The event passed as an argument to the process() function will be processed and indexed as usual after the rule completes. Rules that intend to create new events and skip the original should return Result.DROP. Modifying the original event is more efficient than creating a new one in most cases.

Authoring guidelines

• Keep rule intent focused; split distinct concerns into separate rules.
• Prefer stable, low-cardinality tags for dashboards and filters.
• Use consistent field names across apps to improve reuse.
• Gate rules to the intended sources using dedicated inputs where applicable; see Syslog pipeline customization.

Managing rules via CLI

Refer to the CLI section for full syntax and options: Command Line Tools --- Data Commands.

Typical operations include:

bash
Wrap: OnCopyFullscreen
# List rules and review status
logzilla rules list

# Test a rule file before installing it (requires a companion
# <rule>.tests.yaml next to the file; see Rule Test Fixtures)
logzilla rules test --path /path/to/rule.lua

# Install the rule: validates the file, runs its tests (a Lua rule
# cannot be added without a .tests.yaml), enables the rule, and reloads
logzilla rules add /path/to/rule.lua --name "My Lua Rule"

# Re-check installed rules (name filter, or --all)
logzilla rules validate "My Lua Rule"

# Enable, disable, or reload rules after manual changes
logzilla rules enable "My Lua Rule"
logzilla rules disable "My Lua Rule"
logzilla rules reload

# Review recent runtime errors for rules
logzilla rules errors

Runtime errors and automatic disabling

A Lua error raised inside process() (for example calling string.upper() on a field that a message variant does not carry) does not stop the event. The parser logs the failure with the Lua traceback, counts it against the rule, and the event continues through the remaining rules and into storage with whatever changes the rule made before the error. Partially tagged events in search results are the first visible symptom.

Errors are counted per rule over a rolling one-hour window. When the count reaches the Rule Error Limit setting (Settings → System Settings → Parser; RULE_ERROR_LIMIT, default 5), the parser disables the rule and reloads the rule set without it. From that moment none of the events that rule handled are parsed: they are stored with their original program (often Unknown), no tags, and dashboards and triggers that depend on those tags go quiet. The rule stays disabled until an administrator re-enables it. Rule status and re-enabling have no Settings screen; both are handled from the command line.

Detect it:

bash
Wrap: OnCopyFullscreen
# Status column shows "disabled"; Errors column shows recent count
logzilla rules list

# Full error text and traceback for one rule (last hour)
logzilla rules errors "Custom Rule"

# Platform log signatures
grep -E "Failure of rule|Reached limit of errors" /var/log/logzilla/logzilla.log

Recover:

bash
Wrap: OnCopyFullscreen
# Re-enable the rule; this clears its error log and reloads rules
logzilla rules enable "Custom Rule"

Then fix the cause. The traceback names the rule file and line; the logged event shows the exact message that triggered it. Typical fixes are guarding optional keys before string operations (if kv.protocol then ... end) and adding a test fixture for every message variant the device emits.

Caveats:

• Raising Rule Error Limit only delays the disable. A burst of the same unhandled message variant will still reach any limit.
• Re-enabling without fixing the rule restores parsing only until the next burst of errors.

Performance and reliability

• Narrow rule scope early to minimize unnecessary processing.
• Avoid creating many high-cardinality tags; prefer normalized keys.
• Monitor parser metrics and rule errors during rollouts.
• Validate and test rules prior to enabling in production.

bash
Wrap: OnCopyFullscreen
# Parser overview and throughput
logzilla events parser-stats

# Field and tag insights
logzilla events values --scope fields --limit 50
logzilla events values --scope tags --limit 50

Safety and governance

• Treat rules as configuration-as-code; review changes and track history.
• Favor readability and stable behavior over clever parsing tricks.
• Ensure sensitive data is not emitted as tags or message content.

Related reading

• Overview and decision guide
• Rewrite rules
• LogZilla Apps
• Syslog pipeline customization
• Command Line Tools --- Data Commands
• Rule test fixtures