OpenTelemetry Filter Implementation Review
======================================================================

1  Overview
----------------------------------------------------------------------

The OpenTelemetry (OTel) filter for HAProxy provides distributed tracing,
metrics and logging capabilities.  It creates, propagates and exports spans,
metric instruments and log records that follow the OpenTelemetry specification.
The filter hooks into the HAProxy stream processing pipeline through the
filter API and maps HAProxy channel analyzer events to OpenTelemetry span
lifecycle operations, metric recordings and log-record emissions.

The implementation lives at the top of this addon checkout, consisting of header
files, C source files, a Makefile.mk fragment, and a set of test configurations
with runner scripts.


2  Directory Structure
----------------------------------------------------------------------

  haproxy-opentelemetry/
  |-- Makefile.mk        Build integration (loaded via EXTRA_MAKE)
  |-- include/
  |   |-- include.h      Master include (pulls all headers)
  |   |-- config.h       Build-time tunables (pool sizes, limits)
  |   |-- define.h       Utility macros (memory, strings, lists)
  |   |-- debug.h        Debug/logging infrastructure
  |   |-- filter.h       Filter return codes, alert macros
  |   |-- parser.h       Configuration keyword definitions
  |   |-- conf.h         Configuration data structures
  |   |-- conf_funcs.h   Generated init/free function macros
  |   |-- event.h        Event enumeration and data table
  |   |-- scope.h        Runtime span/context structures
  |   |-- pool.h         Memory pool helpers
  |   |-- http.h         HTTP header manipulation
  |   |-- otelc.h        Span context inject/extract wrappers
  |   |-- vars.h         HAProxy variable integration
  |   |-- sample.h       Sample fetch names (otel.context, otel.bytes_*)
  |   |-- util.h         String conversion, sample helpers
  |   |-- group.h        Group action (HAProxy rule integration)
  |   `-- cli.h          CLI command interface
  |-- src/
  |   |-- filter.c       Filter lifecycle and channel callbacks
  |   |-- parser.c       Configuration file parser
  |   |-- conf.c         Configuration structure init/free
  |   |-- event.c        Scope/span execution engine
  |   |-- scope.c        Runtime context and span management
  |   |-- http.c         HTTP header get/set/remove
  |   |-- otelc.c        C wrapper inject/extract bridge
  |   |-- vars.c         HAProxy variable read/write
  |   |-- sample.c       Sample fetch for span-context ACL checks
  |   |-- pool.c         Pool alloc/free, trash buffers
  |   |-- util.c         Argument handling, sample conversion
  |   |-- group.c        Group action parsing and execution
  |   `-- cli.c          CLI command handlers
  `-- test/
      |-- copy-yml.sh    YAML configuration transformer
      |-- test-speed.sh  Performance benchmarking runner
      |-- run-test-config.sh  Single-instance runner (real file)
      |-- run-sa.sh      Standalone test runner (symlink)
      |-- run-full.sh    Full event-coverage test runner (symlink)
      |-- run-fe-be.sh   Frontend-backend chain runner
      |-- run-ctx.sh     Context propagation test runner (symlink)
      |-- run-cmp.sh     Comparison test runner (symlink)
      |-- run-tcp.sh     TCP-mode test runner (symlink)
      |-- run-updown.sh  Up-down counter test runner (symlink)
      |-- run-err.sh     Error-logging test runner (symlink)
      |-- run-empty.sh   Empty configuration test runner (symlink)
      |-- otlp_http-recorder.py  OTLP/HTTP traffic recorder
      |-- otlp_http-replay.py    OTLP/HTTP traffic replayer
      |-- haproxy-common.cfg     Shared HAProxy configuration part
      |-- index.html     Static page served by the origin server
      |-- sa/            Standalone test configs
      |-- full/          Full event-coverage test configs
      |-- fe/            Frontend-only test configs
      |-- be/            Backend-only test configs
      |-- ctx/           Context propagation test configs
      |-- cmp/           Comparison test configs
      |-- tcp/           TCP-mode test configs
      |-- updown/        Up-down counter test configs
      |-- err/           Error-logging test configs
      `-- empty/         Minimal/empty configuration test


3  Build System
----------------------------------------------------------------------

The Makefile.mk fragment is pulled in by the HAProxy top-level Makefile via the
EXTRA_MAKE variable, which lists addon directories whose Makefile.mk fragments
are included unconditionally.  The fragment detects the opentelemetry-c-wrapper
library via pkg-config or manual OTEL_INC/OTEL_LIB paths.

Build options:

  EXTRA_MAKE=<path> Addon directory containing Makefile.mk.  Listing this
                    directory is what enables the filter.  Multiple directories
                    may be passed, space-separated.
  OTEL_DEBUG=1      Compile with DEBUG_OTEL; links the _dbg variant of the
                    wrapper library and enables the debug-only flt_ops callbacks
                    deinit_per_thread, http_payload and http_reset in filter.c.
  OTEL_USE_VARS=1   Define USE_OTEL_VARS (and USE_OTEL_VARS_NAME when struct
                    var has a 'name' member), which allows span context
                    propagation via HAProxy transaction variables in addition
                    to HTTP headers; vars.c itself is always compiled.
  OTEL_INC=<path>   Manual include path for the C wrapper.
  OTEL_LIB=<path>   Manual library path for the C wrapper.
  OTEL_RUNPATH=1    Embed RPATH to the wrapper library.
  OTEL_STATIC=1     Pass --static to pkg-config so OTEL_LDFLAGS picks up the
                    full Libs.private chain; needed for static linking.

Compiled objects (vars.o is built unconditionally):

  cli.o  conf.o  event.o  filter.o  group.o  http.o  otelc.o  parser.o
  pool.o  sample.o  scope.o  util.o  vars.o

OTEL_USE_VARS does not gate whether vars.o is compiled; it only adds the
-DUSE_OTEL_VARS / -DUSE_OTEL_VARS_NAME defines that select the code paths
compiled inside vars.c.


4  Configuration Parsing
----------------------------------------------------------------------

Configuration parsing is driven by parser.c.  The filter is declared in the
HAProxy configuration with:

  filter opentelemetry [id <id>] config <file> [<name>]

The flt_otel_parse() function (parser.c) handles the "filter" line, creates an
flt_otel_conf structure, and delegates to parse_cfg() which loads the referenced
YAML/CFG file.  That file is parsed using temporary section registrations for
the section types:

  otel-instrumentation   ->  flt_otel_parse_cfg_instr()
  otel-group             ->  flt_otel_parse_cfg_group()
  otel-scope             ->  flt_otel_parse_cfg_scope()

After each section is fully parsed, a post-parse function validates the section
(e.g., flt_otel_post_parse_cfg_scope() checks that context injection is only
used on events that support it).

An optional trailing name on the filter line selects the [<name>] section of
the file: flt_otel_parse_check_scope() compares it, or the filter id when it
is absent, against HAProxy's cfg_scope while the file is parsed.

4.1  Instrumentation Section

  otel-instrumentation <name>
      config <file> [context]
      log <target>
      debug-level <value>
      rate-limit <value>
      option { disabled | hard-errors | dontlog-normal }
      groups <name> ...
      scopes <name> ...
      acl <aclname> <criterion> ...

The instrumentation block defines global filter parameters: the YAML exporter
configuration file, logging, rate limiting, and references to groups and scopes.
Exactly one instrumentation block is allowed per filter instance.

4.2  Group Section

  otel-group <name>
      scopes <name> ...

Groups bundle multiple scopes under a single name for use with HAProxy
http-request/http-response rules via the "otel-group" action.  The group action
(group.c) parses the rule, resolves the scope references at check time, and
executes all referenced scopes when the rule fires.

4.3  Scope Section

  otel-scope <name>
      otel-event <event-name> [{ if | unless } <condition>]
      idle-timeout <time>
      extract <name-prefix> [use-vars | use-headers]
      span <name> [parent <ref>] [link <ref>] [root] [kind <kind>]
        link { <ref> ... | <ref> attr <key> <sample> ... }
        attribute <key> <sample> ... [{ if | unless } <condition>]
        event <name> [time [<unit>] <sample>] <key> <sample> ... [{ if | unless } <condition>]
        baggage <key> <sample> ... [{ if | unless } <condition>]
        status <code> [<sample> ...] [{ if | unless } <condition>]
        exception <type> [message <sample> ...] [attr <key> <sample> ...] [{ if | unless } <condition>]
        inject <name-prefix> [use-vars] [use-headers]
      finish <name> ...
      instrument <type> <name> ... / instrument update <name> ...
      log-record <severity> [id <integer> event <name>] [time [<unit>] <sample>] [span <ref>] [attr <key> <sample>] ... <sample> ...
      set-var <var-name> <sample> ... [{ if | unless } <condition>]
      set-var-ctx <var-name> <ref> <field> [{ if | unless } <condition>]
      unset-var <var-name> ... [{ if | unless } <condition>]
      acl <aclname> <criterion> ...
      otel-stop [{ if | unless } <condition>]

Each scope ties to a single HAProxy analyzer event (or none, if used only
through groups).  Scopes contain context extraction directives, span
definitions, metric instruments, log records, finish and otel-stop directives.

A span may specify:
  - A parent reference (another span or extracted context).
  - One or more links to other spans/contexts.  Inline link syntax allows one
    link on the span line; the standalone "link" keyword allows multiple.
  - The "root" flag marking it as the trace root.
  - Attributes, events, baggages and status evaluated from HAProxy sample
    expressions at runtime.
  - An inject directive to propagate the span context via HTTP headers and/or
    HAProxy variables.

4.4  Configuration Structure Initialization

All configuration structures are allocated and freed using macro-generated
functions from conf_funcs.h:

  FLT_OTEL_CONF_FUNC_INIT(type, id_field, extra_init)
  FLT_OTEL_CONF_FUNC_FREE(type, id_field, extra_free)

These macros produce flt_otel_conf_<type>_init() and _free() functions.
The init function:
  - Checks the identifier length against FLT_OTEL_ID_MAXLEN (64).
  - Checks for duplicate identifiers in the target list.
  - Allocates the structure with OTELC_CALLOC.
  - Copies the identifier with OTELC_STRDUP.
  - Appends to the head list.
  - Executes any extra initialization (e.g., LIST_INIT for sub-lists in the
    span structure).

The free function:
  - Executes any extra cleanup (e.g., destroying sub-lists).
  - Frees the identifier string.
  - Removes the node from its list.
  - Frees the structure.

The full init/free chain for all structures:

  flt_otel_conf          flt_otel_conf_init() / flt_otel_conf_free()
    flt_otel_conf_instr  generated via macro
      flt_otel_conf_ph   generated (for ph_groups, ph_scopes)
    flt_otel_conf_group  generated
      flt_otel_conf_ph   generated (for ph_scopes)
    flt_otel_conf_scope  generated
      flt_otel_conf_context    generated
      flt_otel_conf_span       generated
        flt_otel_conf_link       generated
        flt_otel_conf_sample     generated + _init_ex()
          flt_otel_conf_sample_expr  generated
        flt_otel_conf_exception  generated
          flt_otel_conf_sample     generated (message, attributes)
      flt_otel_conf_str        generated (for spans_to_finish)
      flt_otel_conf_instrument generated
      flt_otel_conf_log_record generated
        flt_otel_conf_sample     generated + _init_ex()
          flt_otel_conf_sample_expr  generated
      flt_otel_conf_sample     generated (for set_vars)
      flt_otel_conf_set_var_ctx  generated
      flt_otel_conf_unset_var  generated
        flt_otel_conf_str      generated (for the vars sublist)

  The conf_funcs.h macros also generate the leaf flt_otel_conf_str and
  flt_otel_conf_hdr structures (a plain string node and a header name/value
  pair, respectively), reused wherever a scope needs such a list -- for
  instance spans_to_finish and the unset-var directive's variable names.


5  Filter Lifecycle
----------------------------------------------------------------------

The filter registers its operations in the flt_otel_ops structure (filter.c)
and the keyword parser via INITCALL1 (parser.c).

5.1  Proxy-Level Initialization

  flt_otel_ops_init():
    - Registers CLI commands via flt_otel_cli_init().
    - Initializes the OpenTelemetry library via flt_otel_lib_init(): verifies
      the C wrapper version, resolves the absolute path of the YAML
      configuration file, calls otelc_init() to set up exporters, creates the
      tracer, meter and logger objects, and registers custom memory allocation
      and thread-id callbacks with the wrapper via otelc_ext_init().
    - Sets the FLT_CFG_FL_HTX capability flag only for an HTTP-mode proxy, so
      HAProxy attaches the filter to HTX streams; a TCP-mode proxy leaves it
      unset and the filter runs on the raw stream.

  flt_otel_ops_check():
    - Validates that filter IDs are unique across all proxies.
    - Resolves group->scope and instrumentation->scope/group placeholder
      references to actual configuration structures (setting the ptr field
      and flag_used).
    - Warns about unused scopes, missing root spans, or multiple root spans.
    - Validates metric instruments: resolves update-form references to their
      matching create-form definitions, and rejects duplicate create-form names
      across scopes.
    - Computes the aggregated analyzer bitmask from all used scopes.

  flt_otel_ops_init_per_thread():
    - Starts the tracer, meter and logger background threads on first call.
    - Uses an atomic claim on instr->flag_started to guarantee that start runs
      on a single thread only; the FLT_CFG_FL_HTX flag is set elsewhere, in
      flt_otel_ops_init(), not here.

  flt_otel_ops_deinit():
    - Destroys the tracer, meter and logger.
    - Frees the entire configuration tree.
    - Calls otelc_deinit() to shut down the wrapper library.

5.2  Stream-Level Callbacks

  flt_otel_ops_attach():
    - Checks if the filter is globally disabled; returns IGNORE.
    - Applies rate limiting via ha_random32(); returns IGNORE if the random
      value exceeds the configured rate_limit.
    - Creates the runtime context (flt_otel_runtime_context_init) with a
      generated UUID and initialized span/context lists.
    - Sets pre_analyzers and post_analyzers bitmasks from the instrumentation's
      aggregated analyzer flags.  AN_REQ_WAIT_HTTP and AN_RES_WAIT_HTTP are
      placed in post_analyzers because those analyzers can only be used in the
      post_analyze callback.  AN_REQ_HTTP_TARPIT stays in pre_analyzers; it is
      left out only when channel_start_analyze force-injects the pre_analyzers
      into the channel, so the tarpit event fires only when a tarpit rule has
      armed that analyzer itself.

  flt_otel_ops_detach():
    - Frees the runtime context, which finishes all remaining active spans and
      destroys all remaining contexts.

  flt_otel_ops_check_timeouts():
    - Disarms the idle timer when the filter is disabled for the stream.
    - Checks whether the idle-timeout timer has expired; if so, fires the
      on-idle-timeout event and reschedules the timer for the next interval
      (disarming instead when the event itself disabled the filter).
    - Sets STRM_EVT_MSG on the stream's pending_events to ensure the filter is
      re-evaluated after a timeout.
    - Re-asserts the idle wake-up on the stream task expiry on every pass.

5.3  Error Handling

Helper functions manage errors:

  flt_otel_return_int() / flt_otel_return_void():
    - If the result indicates an error or an error string is set: in hard-error
      mode, the filter is disabled for the current stream (flag_disabled = 1)
      and the disabled counter is incremented atomically.  In soft-error mode,
      the error is merely logged.
    - The error string is always freed.
    - For int returns, FLT_OTEL_RET_OK is returned regardless, so the stream
      continues processing even after an error.


6  Event Processing (Channel Analyzers)
----------------------------------------------------------------------

The filter maps HAProxy channel analyzer callbacks to a table of named events
defined in event.h (FLT_OTEL_EVENT_DEFINES).

6.1  Event Table

Each event entry carries:
  - an_bit:           the HAProxy analyzer bit (AN_REQ_*, AN_RES_*)
  - an_name:          the analyzer bit name (e.g. "AN_REQ_HTTP_PROCESS_FE")
  - smp_opt_dir:      sample fetch direction (REQ or RES)
  - smp_val_fe/be:    SMP_VAL_FE_*/SMP_VAL_BE_* fetch-location masks for the
                      event's processing point (mirrors the SPOE filter's
                      per-event mapping); flt_otel_ops_check() ORs the two into
                      the location a bound scope's fetches and conditions are
                      validated against
  - flag_http_inject: whether span context can be injected into HTTP headers
                      at this point
  - flag_http_only:   whether the event fires only on an HTTP-mode proxy
  - name:             configuration event name (e.g. "on-frontend-http-request")

Events with an_bit == 0 are pseudo-events not tied to any channel
analyzer.  The stream lifecycle callbacks fire:
  - on-stream-start  (flt_otel_ops_stream_start, before channel processing)
  - on-stream-stop   (flt_otel_ops_stream_stop, after channel processing)

The check_timeouts callback fires periodically:
  - on-idle-timeout  (flt_otel_ops_check_timeouts, when stream is idle)

The stream_set_backend callback fires:
  - on-backend-set   (flt_otel_ops_stream_set_backend, when backend is assigned)

The HTTP lifecycle callbacks fire:
  - on-http-headers-request / on-http-headers-response (flt_otel_ops_http_headers)
  - on-http-end-request / on-http-end-response         (flt_otel_ops_http_end)
  - on-http-reply                                      (flt_otel_ops_http_reply)

But on-http-reply never actually fires on current HAProxy: nothing has invoked
its flt_http_reply() dispatcher since 2.2-dev8 (2020), so the filter callback
above is dead.  See README and README-configuration for the full explanation.

The remaining pseudo-events fire from channel start/end callbacks:
  - on-client-session-start / on-client-session-end
  - on-server-session-start / on-server-session-end
  - on-server-unavailable

The stream lifecycle events pass NULL for the channel argument, so
context injection/extraction via HTTP headers cannot be used.  Their
sample fetch direction is unconstrained (0xff), allowing both request
and response fetches.

6.2  Callback Flow

  stream_start(s, f):
    - Fires on-stream-start with chn=NULL.
    - Called when a new stream begins, before any channel processing.
    - Initializes the idle timer from the precomputed minimum idle_timeout in
      the instrumentation configuration and schedules the first idle wake-up
      on the stream task expiry.

  stream_set_backend(s, f, be):
    - Fires on-backend-set with chn=&s->req.
    - Called when a backend is assigned, the frontend itself included.

  stream_stop(s, f):
    - Fires on-stream-stop with chn=NULL.
    - Called when a stream is destroyed, after all channel processing.

  check_timeouts(s, f):
    - Checks whether the idle-timeout timer has expired.
    - If expired, fires on-idle-timeout and reschedules the timer.
    - Keeps the idle wake-up on the stream task expiry; disarms the timer
      when the filter is disabled for the stream.

  channel_start_analyze(chn):
    - Enables the per-channel analyzers from pre_analyzers (except
      AN_REQ_HTTP_TARPIT, which only an armed tarpit rule may enable).
    - Fires on-client-session-start (request) or on-server-session-start
      (response).

  channel_pre_analyze(chn, an_bit):
    - Looks up the event by an_bit in the event table.
    - Calls flt_otel_event_run() for the matching event.

  channel_post_analyze(chn, an_bit):
    - Same as pre_analyze but for post-analyzers (AN_REQ_WAIT_HTTP,
      AN_RES_WAIT_HTTP).

  channel_end_analyze(chn):
    - Fires on-client-session-end (request) or on-server-session-end (response).
    - For the request channel: if response analyzers were configured but
      none executed (server was unreachable), fires on-server-unavailable.

  http_headers(s, f, msg):
    - Fires on-http-headers-request or on-http-headers-response depending on
      msg->chn direction.

  http_end(s, f, msg):
    - Fires on-http-end-request or on-http-end-response depending on
      msg->chn direction.

  http_reply(s, f, status, msg):
    - Fires on-http-reply with chn=&s->res.
    - Never reached on current HAProxy: HAProxy dropped the flt_http_reply()
      call in 2.2-dev8 (commit 8dfeccf6d, 2020), so this callback has no caller.

6.3  Scope Execution

  flt_otel_event_run() (event.c):
    - Captures timestamps (CLOCK_MONOTONIC + CLOCK_REALTIME).
    - Updates the runtime context's executed-analyzers bitmask.
    - Iterates all scopes matching the event; calls flt_otel_scope_run() for
      each used scope.

  flt_otel_scope_run() (event.c):
    1. Evaluates the scope's ACL condition; if it fails:
       - If the scope contains a root span, disables the stream.
       - Returns without processing.
    2. Extracts contexts: for each configured extract directive, reads
       the span context from HTTP headers or HAProxy variables via
       flt_otel_scope_context_init().
    3. Runs "set-var" directives via flt_otel_scope_run_set_var(), setting
       HAProxy variables from sample expressions (before any span runs).
    4. Processes spans: for each configured span:
       a. Calls flt_otel_scope_span_init() which either returns an existing
          scope_span (by name) or creates a new one with resolved parent
          reference.
       b. Resolves span links against the runtime context -- first searching
          active spans, then extracted contexts.  Unresolved links are skipped
          (a debug build traces them; a release build stays silent).
       c. Evaluates attributes, events, baggages, and status from sample
          expressions via flt_otel_sample_add().
       d. Calls flt_otel_scope_run_span() which:
          - Creates the OTel span via tracer->start_span_with_options()
            (if not already started).
          - Adds all resolved links via span->add_link().
          - Sets baggage, attributes, events, and status.
          - Optionally injects the span context into HTTP headers and/or
            HAProxy variables.
    5. Runs "set-var-ctx" directives via flt_otel_scope_run_set_var_ctx(),
       storing a field of a referenced span or context into a HAProxy
       variable (after the spans have run, so their contexts are available).
    6. Processes metric instruments via flt_otel_scope_run_instrument(), which
       runs two passes: the first lazily creates create-form instruments using
       HA_ATOMIC_CAS for thread-safe one-time initialization; the second records
       measurements for update-form instruments, waiting for any still being
       created (PENDING) and skipping any never created (UNSET).
    7. Emits log records via flt_otel_scope_run_log_record(), which iterates
       the scope's log-record list, skips entries below the logger's severity
       threshold, evaluates sample expressions into a body string, resolves
       the optional span reference, and emits the record via the logger.
    8. Runs "unset-var" directives, removing the named HAProxy variables via
       flt_otel_var_unset_byname() (each gated by its optional condition).
    9. Marks spans listed in "finish" directives; a fired "otel-stop" marks
       every open span and context for completion.
   10. Calls flt_otel_scope_finish_marked() to end marked spans/contexts.
   11. Calls flt_otel_scope_free_unused() to remove finished and destroyed
       scope_span/scope_context entries from the runtime lists.
   12. If "otel-stop" fired, sets rt_ctx->flag_disabled = 1; the guard at the
       top of flt_otel_scope_run() then skips the stream's remaining scopes.


7  Runtime Data Structures
----------------------------------------------------------------------

7.1  Runtime Context (per stream)

  flt_otel_runtime_context:
    stream         Owning stream pointer.
    filter         Owning filter pointer.
    uuid[40]       Generated UUID v4 for the session.
    flag_harderr   Copied from instrumentation config.
    flag_disabled  Set when the filter encounters a hard error or ACL disables
                   processing.
    logging        Logging flags.
    analyzers      Bitmask of analyzers that have actually executed.
    idle_timeout   Idle timeout interval in milliseconds (0 = off).
    idle_exp       Tick at which the next idle timeout fires.
    bytes_in       Raw payload bytes seen on the request channel (TCP mode).
    bytes_out      Raw payload bytes seen on the response channel (TCP mode).
    spans          Linked list of flt_otel_scope_span.
    contexts       Linked list of flt_otel_scope_context.

7.2  Scope Span

  flt_otel_scope_span:
    id / id_len    Span operation name (borrowed from config).
    smp_opt_dir    Direction in which the span was created.
    flag_finish    Set by finish directives, cleared after ending.
    span           The OTel span object (NULL before start, NULL after
                   end_with_options).
    ref_span       Parent span pointer (resolved at init).
    ref_ctx        Parent context pointer (resolved at init).
    list           Chain in runtime_context.spans.

  flt_otel_scope_span_init() performs memoization: if a span with the same name
  already exists in rt_ctx->spans, it returns the existing entry.  This allows
  multiple scopes to contribute attributes/events to the same logical span.

7.3  Scope Context

  flt_otel_scope_context:
    id / id_len    Context name (borrowed from config).
    smp_opt_dir    Direction in which the context was extracted.
    flag_finish    Marks the context for destruction.
    context        The OTel span_context object.
    baggage        The inbound baggage carrier, or NULL (used by set-var-ctx).
    list           Chain in runtime_context.contexts.

  Similarly memoized: duplicate extraction of the same context name returns the
  existing entry.

7.4  Scope Data (per span per scope run, stack-allocated)

  flt_otel_scope_data:
    baggage        Key-value array for baggage items.
    attributes     Key-value array for span attributes.
    events         Linked list of flt_otel_scope_data_event (each with name,
                   an optional per-event timestamp in ts / ts_set, and a
                   key-value array).
    links          Linked list of flt_otel_scope_data_link (each with span
                   and/or context pointer).
    status         Status code and description string.

  Initialized at the start of each span processing block and freed at the end.
  The link entries hold borrowed pointers to the OTel span/context objects
  owned by the runtime context, so those are not freed; each link's own
  attribute key-value array is owned by the link, however, and is destroyed
  via otelc_kv_destroy() before the link node itself is freed.

7.5  Span Finishing

  finish <name> / finish * / finish *req* / finish *res*

  The "finish" directive marks spans and contexts for completion:
    - "*" marks all.
    - "*req*" / "*res*" marks those created in the request/response direction
      respectively.
    - Otherwise, marks by exact name.

  flt_otel_scope_finish_marked() iterates all marked entries:
    - Spans are ended via span->end_with_options() which NULLs the span pointer.
    - Contexts are destroyed via context->destroy() which NULLs the context
      pointer.

  flt_otel_scope_free_unused() then removes entries with NULL span/context
  pointers from the runtime lists.  For contexts, associated HTTP headers
  and variables are also cleaned up.

  On stream detach (flt_otel_runtime_context_free), any remaining active spans
  are force-ended and all entries are freed.


8  Span Links
----------------------------------------------------------------------

Span links associate a span with other spans or contexts without establishing
a parent-child relationship.

8.1  Configuration

Two syntaxes are supported:

  Inline (one link per span declaration):
    span <name> [parent <ref>] link <ref> [root]

  Standalone (multiple links, or one link with attributes):
    link { <ref> [<ref> ...] | <ref> attr <key> <sample> ... }

The flt_otel_conf_link structure stores each link target name and, for the
single-link form, a list of attribute samples.  Duplicate link names within
the same span are rejected by the init macro's duplicate check.
The links list is initialized in flt_otel_conf_span_init() and destroyed in
flt_otel_conf_span_free().

8.2  Runtime Resolution

At scope execution time (event.c, flt_otel_scope_run), for each configured link:
  1. The name is searched in rt_ctx->spans (active scope_span entries).
     If found, the OTel span pointer is captured.
  2. If not found in spans, the name is searched in rt_ctx->contexts (extracted
     scope_context entries).  If found, the OTel span_context pointer is
     captured.
  3. If neither is found, the link is skipped (a debug build traces the
     miss; a release build stays silent).
  4. A flt_otel_scope_data_link node is allocated and appended to the scope
     data's links list.

In flt_otel_scope_run_span(), all resolved links are applied via
span->add_link(span, link_span, link_context, link->attributes.attr,
link->attributes.cnt).  The last two arguments carry the link's resolved
attribute array and count, so a link can carry its own attributes (from the
"link <ref> attr <key> <sample> ..." form); they are empty when the link was
declared without attributes.


9  Context Propagation
----------------------------------------------------------------------

9.1  Extraction

  extract <name-prefix> [use-vars | use-headers]

Extracts an incoming trace context.  The prefix identifies the header name
pattern (for HTTP) or variable name pattern (for vars).

  - use-headers (default): flt_otel_http_headers_get() iterates HTX headers
    matching the prefix and builds an otelc_text_map.
  - use-vars: flt_otel_vars_get() reads HAProxy variables matching the prefix
    pattern.

The text map is passed to flt_otel_extract_http_headers() which uses the
C wrapper to reconstruct an otelc_span_context.

9.2  Injection

  inject <name-prefix> [use-vars] [use-headers]

Injects the current span's context into outgoing data.  Both storage types can
be used simultaneously.

  flt_otel_inject_http_headers() serializes the span context into an
  otelc_http_headers_writer which produces a text_map.  For each key-value pair:
    - use-headers: flt_otel_http_header_set() adds/replaces the header with the
      prefixed name.
    - use-vars: flt_otel_var_register() + flt_otel_var_set() stores the value
      in a HAProxy transaction variable with normalized name (dashes replaced
      with 'D', spaces with 'S', uppercase lowered; dots serve as component
      separators).


10  HTTP Header Manipulation
----------------------------------------------------------------------

  http.c provides these operations:

  flt_otel_http_headers_get(chn, prefix, prefix_len, err):
    Iterates the HTX message headers.  Headers whose name starts with the given
    prefix are collected into an otelc_text_map.  The prefix is stripped from
    the names in the returned map.

  flt_otel_http_header_set(chn, prefix, name, value, err):
    Removes any existing header matching "prefix" + "name", then adds a new
    header with the given value.  If name is NULL, all headers with the prefix
    are removed (bulk delete).

  flt_otel_http_headers_remove(chn, prefix, err):
    Convenience wrapper; removes all headers matching the prefix.


11  HAProxy Variable Integration
----------------------------------------------------------------------

Enabled with OTEL_USE_VARS=1.  Provides an alternative propagation mechanism
using HAProxy transaction-scoped variables.

Variable names are normalized: dashes and spaces are replaced with special
characters to comply with HAProxy variable naming rules.  In a build without
USE_OTEL_VARS_NAME, a meta-variable tracks the list of context variable names
so they can be enumerated for extraction; with it, the variable store is scanned
directly by name prefix.

Key functions:
  flt_otel_var_register()   Registers a variable with HAProxy.
  flt_otel_var_set()        Sets a variable value.
  flt_otel_vars_get()       Reads all context variables into a text_map for
                            extraction.
  flt_otel_vars_unset()     Removes all context variables.


12  Group Action Integration
----------------------------------------------------------------------

The "otel-group" HAProxy action allows triggering trace scopes from
tcp-request, tcp-response, http-request, http-response and
http-after-response rules:

  tcp-request         otel-group <filter-id> <group-name>
  tcp-response        otel-group <filter-id> <group-name>
  http-request        otel-group <filter-id> <group-name>
  http-response       otel-group <filter-id> <group-name>
  http-after-response otel-group <filter-id> <group-name>

  group.c implements:
    flt_otel_group_parse():   Parses the action arguments.
    flt_otel_group_check():   Resolves filter and group references.
    flt_otel_group_action():  At runtime, finds the OTel filter in the stream,
                              iterates all scopes in the group, and calls
                              flt_otel_scope_run() for each.


13  Memory Management
----------------------------------------------------------------------

  pool.c provides wrappers around HAProxy memory pools and standard
  allocation:

  flt_otel_pool_alloc()   Allocates from a pool (when non-NULL) or the heap.
  flt_otel_pool_free()    Returns memory to the pool or frees it.
  flt_otel_pool_strndup() Duplicates a string via pool allocation.
  flt_otel_trash_alloc()  Acquires a trash buffer chunk.
  flt_otel_trash_free()   Releases a trash buffer chunk.

Pool heads are registered for hot-path structures:
  - otel_scope_span       (scope.c)
  - otel_scope_context    (scope.c)
  - otel_runtime_context  (scope.c)
  - otel_span_context     (filter.c, used by the C wrapper via otelc_ext_init
                          callback)

The wrapper library's memory allocations are redirected through
flt_otel_mem_malloc() / flt_otel_mem_free() which use the otel_span_context
pool.  This ensures OTel objects benefit from HAProxy's pool allocator.


14  CLI Interface
----------------------------------------------------------------------

  cli.c registers commands under "flt-otel" for runtime control:
  - Introspection: status, instruments and scopes reports.
  - Runtime switches: enable/disable, hard-errors/soft-errors, logging, rate,
    reset-errors, and -- in a debug build -- the debug level.
  - Data control: flush, which force-exports the buffered telemetry.

Logging can be independently controlled via the instrumentation's logging
flags (ON, NOLOGNORM).  Log output goes to the log servers configured in the
instrumentation block.


15  Debug Infrastructure
----------------------------------------------------------------------

When compiled with OTEL_DEBUG=1 (DEBUG_OTEL defined), the filter enables:

  - The debug-only flt_ops callbacks deinit_per_thread, http_payload and
    http_reset are gated via OTELC_DBG_IFDEF() and set to NULL in non-debug
    builds.  (Note: stream_set_backend, http_headers, http_end and http_reply
    are registered unconditionally and back the on-backend-set,
    on-http-headers-*, on-http-end-* and on-http-reply events; tcp_payload is
    likewise unconditional and counts the bytes forwarded on a TCP-mode stream.
    None of these is debug-only.)

  - The OTELC_DBG() macro produces debug output at various levels.

  - flt_otel_scope_data_dump() dumps the complete scope data (baggage,
    attributes, events, links, status) for inspection.

  - Event usage counters (per-event htx_is_empty statistics) are maintained and
    printed at deinit.

  - Pool size information is printed at startup.

The debug level is a bitmask that can be adjusted at runtime via the CLI.


16  Test Infrastructure
----------------------------------------------------------------------

16.1  Test Scenarios

  sa    Standalone: comprehensive test exercising most request and response
        events (six are left to the 'full' configuration), span links (both
        inline and standalone syntax), events with data capture, baggage, and
        the full span hierarchy from client session start to server session
        end.

  full  Full event coverage: extends sa with the remaining lifecycle events,
        covering every filter event except on-http-tarpit-request.

  fe    Frontend-only: tests the request-side span chain with context injection
        into HTTP headers.

  be    Backend-only: tests context extraction from HTTP headers and
        response-side processing.  Designed to run as the backend of
        the fe/ test.

  ctx   Context propagation: deep nesting test that verifies context propagation
        via both HTTP headers and HAProxy variables.

  cmp   Comparison: simplified configuration made for comparison with other
        tracing implementations.

  tcp   TCP mode: traces the raw connection on a TCP-mode proxy and counts the
        forwarded payload via the otel.bytes_in / otel.bytes_out fetches.

  updown Up-down counter: a single udcnt_int instrument records a signed
         per-session delta, +1 on client session start and -1 on session end,
         so its value tracks the number of active client sessions.

  err   Error logging: the response scope parents its span under an 'orphan
        span' created only by on-server-unavailable, so span creation fails
        on every response and drives the rate-limited error/warning logging
        and the CLI-reported counters.

  empty Minimal: validates that an empty configuration (only the
        instrumentation block, no scopes) does not crash.

16.2  Test Runners

All runners are POSIX shell scripts (/bin/sh) meant to be run from the test/
directory.  The run-*.sh runners accept an optional HAProxy binary path, a
pidfile and a log file that replaces their timestamped default name under
test/_logs/ (run-fe-be.sh appends the -fe and -be suffixes to the given
name); test-speed.sh takes [-b backend] [-d duration] [-r rate-limits] and
a configuration name instead.

  run-test-config.sh  Single-instance runner: derives the configuration
                directory from the name it is invoked under (run-<dir>.sh).
                The per-configuration runners below are symlinks to it.
  run-sa.sh     Runs a single HAProxy instance with sa/ config.
  run-full.sh   Runs a single HAProxy instance with full/ config.
  run-cmp.sh    Runs a single HAProxy instance with cmp/ config.
  run-ctx.sh    Runs a single HAProxy instance with ctx/ config.
  run-tcp.sh    Runs a single HAProxy instance with tcp/ config.
  run-updown.sh Runs a single HAProxy instance with updown/ config.
  run-err.sh    Runs a single HAProxy instance with err/ config.
  run-empty.sh  Runs a single HAProxy instance with empty/ config.
  run-fe-be.sh  Launches two HAProxy instances (frontend on port 10080, backend
                on port 11080) forming a trace propagation chain.  Handles
                graceful shutdown via SIGUSR1.  A separate script, not a
                symlink.
  test-speed.sh Runs performance benchmarks for one or all configurations.

  copy-yml.sh   Transforms a template YAML configuration by replacing
                placeholders with test-specific values (service names, file
                suffixes, etc.).
  otlp_http-recorder.py  Records incoming OTLP/HTTP requests to files, with
                optional forwarding to an upstream collector.
  otlp_http-replay.py    Replays previously recorded OTLP/HTTP requests to a
                collector.

16.3  Exporter Configuration

Each test directory contains an otel.yml file configuring exporter types:
  - OTLP file exporter (writes traces to local files).
  - OTLP gRPC exporter (sends to localhost:4317).
  - OTLP HTTP exporter (sends to localhost:4318 in JSON format).


17  Notable Design Decisions
----------------------------------------------------------------------

  - Span memoization: flt_otel_scope_span_init() and
    flt_otel_scope_context_init() return existing entries if one with the
    same name already exists.  This allows multiple scopes to contribute data
    (attributes, events) to the same logical span across different analyzer
    events.

  - Lazy span creation: the OTel span object is created on first use in
    flt_otel_scope_run_span(), not at scope_span_init time.  This separates
    the span identity (name, parent reference) from the actual OTel resource.

  - Soft/hard error modes: in soft mode, errors are logged but the stream
    continues with tracing effectively abandoned for that span.  In hard mode,
    the filter disables itself for the rest of the stream.  Either way, stream
    processing is never interrupted by a tracing failure (FLT_OTEL_RET_OK is
    always returned).

  - Rate limiting uses a uint32 representation of a percentage
    (FLT_OTEL_FLOAT_U32), compared against ha_random32() for uniform
    distribution without floating-point at runtime.

  - Server-unavailable fallback: if the backend was never reached (no response
    analyzers executed), the on-server-unavailable event is fired at client
    session end to ensure all spans are properly closed.

  - Custom memory allocator: the C wrapper's allocations are routed through
    HAProxy memory pools via otelc_ext_init(), keeping OTel objects in the
    same allocation domain as the rest of the filter.

  - Thread integration: flt_otel_thread_id() returns the HAProxy tid, ensuring
    the wrapper's thread-local operations map to HAProxy worker threads.

  - Single dispatch stage per event: each scope event fires from one callback
    -- channel_pre_analyze or channel_post_analyze for analyzer-bound events,
    or a lifecycle/data callback otherwise.  Exposing both stages of every
    analyzer (an on-X-pre and an on-X-post) was considered and rejected: about
    half the events are not analyzer-bound and have no stage to split; the
    tarpit analyzer never finishes so it has no post stage, and the
    wait-for-HTTP analyzers have no parsed message in the pre stage; and the
    rest complete in a single pass, so their two stages fire microseconds
    apart.  Where a later view is genuinely needed -- the request body, for
    example, is buffered only after AN_REQ_HTTP_BODY -- an optional per-scope
    stage qualifier on the existing event name would be preferable to doubling
    the event table.  The otel-event directive would take an optional 'pre' or
    'post' keyword:

        otel-event on-http-body-request       # default stage (pre, as today)
        otel-event on-http-body-request post  # opt into the post stage

    defaulting to the current stage when omitted.  The filter would list such an
    analyzer in both the pre and post masks and run the scope only from the
    stage it asked for.


18  Tracer, Span and Metrics Internals
----------------------------------------------------------------------

This chapter describes the end-to-end lifecycle of the tracer and meter
objects, the runtime span management model, and the metric instrument
recording pipeline.

18.1  Tracer Provider Initialization

The tracer provider is set up during the proxy-level flt_otel_ops_init()
callback, which delegates to flt_otel_lib_init() (filter.c).
The initialization sequence is as follows:

  1. Version check: OTELC_IS_VALID_VERSION() verifies that the
     OpenTelemetry C wrapper library version matches the header files.

  2. Configuration path: the relative path from the "config" keyword in
     the instrumentation section is resolved to an absolute path using
     getcwd() + snprintf().

  3. SDK initialization: otelc_init(path, err) loads the YAML
     configuration file and sets up the SDK exporters, samplers,
     processors and metric readers.

  4. Tracer creation: otelc_tracer_create(err) allocates the tracer
     handle and stores it in instr->tracer.

  5. Meter creation: otelc_meter_create(err) allocates the meter handle
     and stores it in instr->meter.

  6. Logger creation: otelc_logger_create(err) allocates the logger
     handle and stores it in instr->logger.

  7. Extension callbacks: on success, otelc_ext_init() registers custom
     memory allocation (flt_otel_mem_malloc / flt_otel_mem_free) and
     thread-id (flt_otel_thread_id) callbacks so that OTel SDK objects
     use HAProxy memory pools and thread numbering.

  8. Log handler: otelc_log_set_handler() installs a callback that
     counts SDK diagnostic messages via the flt_otel_drop_cnt counter.

The tracer, meter and logger handles are stored in the flt_otel_conf_instr
structure (conf.h):

  struct flt_otel_conf_instr {
      ...
      struct otelc_tracer *tracer;  /* The OpenTelemetry tracer handle. */
      struct otelc_meter  *meter;   /* The OpenTelemetry meter handle. */
      struct otelc_logger *logger;  /* The OpenTelemetry logger handle. */
      ...
  };

18.2  Per-Thread Tracer, Meter and Logger Startup

The flt_otel_ops_init_per_thread() callback (filter.c) starts the
tracer, meter and logger background threads on the first call:

  if (HA_ATOMIC_BTS(&(conf->instr->flag_started), 0) == 0) {
      retval = OTELC_OPS(conf->instr->tracer, start);
      ...
      if (retval != OTELC_RET_ERROR) {
          retval = OTELC_OPS(conf->instr->meter, start);
          ...
      }
      if (retval != OTELC_RET_ERROR) {
          retval = OTELC_OPS(conf->instr->logger, start);
          ...
      }
  } else {
      retval = FLT_OTEL_RET_OK;
  }

The atomic bit-test-and-set on instr->flag_started ensures that start is
called only once, even when this callback runs on every worker thread and
multiple proxies share the same filter configuration.  Threads that lose the
claim return success without repeating the start.  If any start operation
fails, the error string from the failing handle is forwarded via
FLT_OTEL_ALERT.

18.3  Tracer, Meter and Logger Shutdown

At proxy deinit (flt_otel_ops_deinit, filter.c), the tracer, meter
and logger handles are first saved into local variables, then the
configuration tree is freed, and only afterwards are the handles
destroyed:

  otel_tracer = (*conf)->instr->tracer;
  otel_meter  = (*conf)->instr->meter;
  otel_logger = (*conf)->instr->logger;

  flt_otel_conf_free(conf);
  flt_otel_pool_destroy();
  otelc_deinit(&otel_tracer, &otel_meter, &otel_logger);

The ordering matters: flt_otel_conf_free() must run while the wrapper's
ext callbacks still point to the HAProxy pool allocator, and otelc_deinit()
resets those callbacks, so it runs last.  Saving the handles into locals
first avoids dereferencing the already-freed instr structure.  The
otelc_deinit() call flushes any pending spans, metric data and log records
to the configured exporters, then releases the SDK resources.

18.4  Span Lifecycle

Spans progress through four phases: identity allocation, OTel span
creation, data population, and completion.

18.4.1  Span Identity Allocation

When a scope containing a span definition executes for the first time,
flt_otel_scope_span_init() (scope.c) allocates a scope_span
entry from the otel_scope_span pool and inserts it into the runtime
context's spans list:

  retptr = flt_otel_pool_alloc(pool_head_otel_scope_span, ...);
  retptr->id          = id;       /* Borrowed from config. */
  retptr->id_len      = id_len;
  retptr->smp_opt_dir = dir;
  retptr->ref_span    = ref_span; /* Resolved parent span. */
  retptr->ref_ctx     = ref_ctx;  /* Resolved parent context. */
  LIST_INSERT(&(rt_ctx->spans), &(retptr->list));

The parent reference (ref_id) is resolved at this point by searching the
runtime context's spans list first, then the contexts list.  If the
parent name cannot be found in either list, an error is returned and the
span is not created.

Memoization: if a span with the same name already exists in
rt_ctx->spans, the existing entry is returned without allocation.  This
allows multiple scopes (across different analyzer events) to contribute
attributes, events and other data to the same logical span.

18.4.2  OTel Span Creation (Lazy)

The actual OTel span object is created lazily on first use in
flt_otel_scope_run_span() (event.c):

  if (span->span == NULL) {
      span->span = OTELC_OPS(conf->instr->tracer,
          start_span_with_options, span->id,
          span->ref_span, span->ref_ctx,
          ts_steady, ts_system, conf_span->kind, NULL, 0);
  }

The arguments are:

  span->id        The operation name (string identifier from config).
  span->ref_span  The parent span pointer (NULL if root or no parent).
  span->ref_ctx   The parent span context (from extracted context).
  ts_steady       Monotonic timestamp (CLOCK_MONOTONIC) for duration.
  ts_system       Wall-clock timestamp (CLOCK_REALTIME) for events.
  conf_span->kind The per-span configured span kind (default server,
                  overridable via "span ... kind <kind>").
  NULL, 0         Initial attribute array and count (none at creation).

This separation between identity allocation and OTel creation means the
span name, parent references and pool entry exist before the OTel
resource is allocated.  Subsequent scope executions that reference the
same span name find the existing entry (via memoization) and add their
data to the already-created OTel span.

18.4.3  Span Data Population

After creation, flt_otel_scope_run_span() (event.c) populates
the span with data collected during scope execution:

  Links (event.c):
    Each resolved link is added via span->add_link(span, link_span,
    link_context, link->attributes.attr, link->attributes.cnt).  Links
    associate the span with other spans or contexts without establishing
    a parent-child relationship.  The last two arguments carry the link's
    own resolved attributes (from the "link <ref> attr ..." form), or an
    empty array when the link has none.

  Baggage (event.c):
    span->set_baggage_kv_n(data->baggage.attr, data->baggage.cnt)
    sets key-value baggage items propagated across service boundaries.

  Attributes (event.c):
    span->set_attribute_kv_n(data->attributes.attr, data->attributes.cnt)
    sets key-value span attributes evaluated from HAProxy sample
    expressions.

  Events (event.c):
    For each event in data->events (iterated in reverse insertion order):
    span->add_event_kv_n(event->name, ts, event->attr, event->cnt) adds a
    named event with key-value attributes; ts is the event's own evaluated
    'time' value when one was set (the node's ts / ts_set fields), the
    wall-clock timestamp otherwise.

  Exceptions (event.c):
    Each configured 'exception' whose condition holds is emitted through
    span->record_exception() as a span event named 'exception', carrying
    the type, the evaluated message and any additional attributes.

  Status (event.c):
    span->set_status(data->status.code, data->status.description)
    sets the span's status code and description string.  The configured
    status lines are tried in order and the first whose condition holds
    provides the one status that is set.

18.4.4  Span Context Injection

After populating the span, if the configuration contains an "inject"
directive (conf_span->ctx_id is non-NULL), the span context is
serialized for downstream propagation (event.c).

  flt_otel_inject_http_headers() serializes the span context into an
  otelc_http_headers_writer, producing a text_map of key-value pairs.
  For each pair, depending on the ctx_flags:

    FLT_OTEL_CTX_USE_HEADERS:
      flt_otel_http_header_set() writes the header into the HTX message.

    FLT_OTEL_CTX_USE_VARS (requires OTEL_USE_VARS=1):
      flt_otel_var_register() + flt_otel_var_set() store the value
      in a HAProxy transaction variable.

  Both storage types can be used simultaneously on the same span.

18.4.5  Span Completion

Spans are ended through the marking mechanism described in chapter 7.5.
The actual end call in flt_otel_scope_finish_marked() (scope.c) is:

  OTELC_OPSR(span->span, end_with_options,
             ts_finish, OTELC_SPAN_STATUS_IGNORE, NULL);

The arguments are the monotonic timestamp, a status hint (IGNORE means
"do not override the status already set on the span"), and NULL for
error string.  After end_with_options returns, the OTELC_OPSR macro
NULLs the span pointer, making the entry eligible for removal by
flt_otel_scope_free_unused().

On stream detach, flt_otel_runtime_context_free() (scope.c)
force-ends any remaining active spans with the current monotonic
timestamp and frees all pool entries.

18.5  Metric Instruments

The filter supports the full set of OpenTelemetry metric instrument
types through a two-form configuration model: "create" form instruments
define the instrument, and "update" form instruments record measurements
against it.

The OpenTelemetry data model itself works this way: an instrument is created
once with a fixed identity, and measurements are recorded against it repeatedly.
The create form is that one-time creation; the update form is the recording.
What the create form pins is not only the identity -- name, description, unit,
kind, aggregation and any histogram bounds -- but also the sample expression
that produces the value.  The update form has no value expression of its own; it
contributes only the recording attributes, an optional if/unless condition and,
through the scope that it lives in, the event that drives the recording.

The expression is fixed, the recorded number is not.  Each update re-evaluates
the create form's value expression against the current stream, so a live fetch
such as fe_conn or lat_ns_tot yields a fresh reading on every run, whereas a
constant such as int(1) yields a steady increment.  That value is then applied
through the instrument's native operation: an additive delta for counters and
up-down counters, a distribution sample for histograms, and a last-value reading
for gauges.  A fixed expression is therefore a valid source for the model's
recording side.

A create form emits no data point on its own; a measurement appears only when an
update form for that instrument runs.  Within a scope a name may carry at most
one create form and one update form, but update forms in other scopes may target
the same create form, so one instrument can be recorded from several sites, each
with its own attributes, condition and driving event.  Across those sites the
one thing that cannot vary is the value expression; to record another quantity,
declare a separate instrument instead.

18.5.1  Instrument Types

The following instrument types are available (parser.h):

    cnt_int         Counter (uint64)
    hist_int        Histogram (uint64)
    udcnt_int       UpDownCounter (int64)
    gauge_int       Gauge (int64)

Observable (asynchronous) instruments are not supported.  The OTel SDK invokes
their callbacks from an external background thread that is not a HAProxy
thread.  HAProxy sample fetches rely on internal per-thread-group state and
return incorrect results when called from a non-HAProxy thread.

Double-precision types are not supported because HAProxy sample fetches do not
return double values.

  Special:
    update          Update-form instrument (records measurements)

Each create-form instrument carries a description, unit, aggregation type,
sample expression list, and optional histogram bucket boundaries.  Each
update-form instrument carries a reference to its create-form counterpart
and an attribute key-value array for per-scope dimensions.

18.5.2  Instrument Configuration Structure

The flt_otel_conf_instrument structure (conf.h) holds:

  idx          Meter instrument index.  Initially set to
               OTELC_METRIC_INSTRUMENT_UNSET (-1).  Transitions to
               OTELC_METRIC_INSTRUMENT_PENDING (-2) during creation,
               then to the positive meter index on success.
  type         The otelc_metric_instrument_t type constant, or
               OTELC_METRIC_INSTRUMENT_UPDATE (0xff) for update-form.
  aggr_type    The otelc_metric_aggregation_type_t constant.
               Initially OTELC_METRIC_AGGREGATION_UNSET (-1).
  description  Instrument description string (create-form only).
  unit         Instrument unit string (create-form only).
  samples      List of sample expressions for the instrument value.
  bounds       Histogram bucket boundaries array (create-form only).
  bounds_num   Number of histogram bucket boundaries.
  attributes   List of flt_otel_conf_sample entries (update-form only).
  ref          Pointer to the create-form instrument (update-form only).
  cond         Optional if/unless ACL condition gating the recording.

18.5.3  Meter Initialization and Startup

The meter handle is created alongside the tracer in flt_otel_lib_init()
(filter.c) via otelc_meter_create(err) and started per-thread
in flt_otel_ops_init_per_thread() (filter.c) via
OTELC_OPS(conf->instr->meter, start).  The meter background thread
handles periodic collection and export of metric data.

18.5.4  Instrument Creation and Recording

Metric instrument processing is performed by
flt_otel_scope_run_instrument() (event.c), which runs in two
passes during scope execution.

  Pass 1 -- Create-form instruments (event.c):

    Iterates all instruments in the scope.  For each create-form
    instrument whose idx is OTELC_METRIC_INSTRUMENT_UNSET:

    a. Thread-safe one-time creation: HA_ATOMIC_CAS transitions the idx
       from UNSET to PENDING.  If the CAS fails (another thread is
       already creating this instrument), the current thread skips it.

    b. Instrument creation: meter->create_instrument() is called with
       the instrument name, description, unit, type and callback data.
       On success, the returned index is stored atomically; on failure,
       the idx is reset to UNSET.  If the instrument carries histogram
       bucket boundaries, meter->add_view() is called before instrument
       creation to register a view with the configured aggregation
       strategy and those bounds; with bounds present and no explicit
       aggregation type, histogram aggregation is used automatically.
       An 'aggr' type given without bounds registers no view.

  Pass 2 -- Update-form instruments (event.c):

    Iterates all instruments again.  For each update-form instrument:

    a. Reference validation: the ref pointer must be non-NULL (resolved
       at check time to the create-form instrument).

    b. Index check: if the create-form instrument's idx is PENDING, the
       thread waits with ha_thread_relax() until the creating thread
       resolves it; an idx that is (or becomes) UNSET means creation
       never happened or failed, and the measurement is skipped.

    c. Recording: flt_otel_scope_run_instrument_record() evaluates the
       sample expression, converts it to an otelc_value, and calls
       meter->update_instrument_kv_n(idx, &value, attr, attr_len).

18.5.5  Sample Evaluation for Metrics

The recording function flt_otel_scope_run_instrument_record()
(event.c) supports two evaluation paths:

  Standard path: evaluates sample_process() on the first expression in
  the create-form instrument's samples list, using the stream's backend,
  session and direction context.

  Log-format path: if sample->lf_used is set, allocates a temporary
  buffer of global.tune.bufsize, calls build_logline() to evaluate the
  log-format expression, and presents the result as an SMP_T_STR sample.

Both paths converge on flt_otel_sample_to_value(), which converts the HAProxy
sample data to an otelc_value.  A metric instrument value is always int64, so
a string result (for example a value read from a variable, which the filter
stores as a string) is coerced through otelc_value_strtonum() with
OTELC_VALUE_INT64.  A numeric string is coerced and recorded; the measurement
is rejected, with a warning that names the value, only when that conversion
fails.


18.5.6  Instrument Lifecycle Summary

  Configuration time (flt_otel_conf_instrument_init):
    idx = OTELC_METRIC_INSTRUMENT_UNSET (-1)
    type = OTELC_METRIC_INSTRUMENT_UNSET (the actual type constant is
           assigned later, during keyword parsing)

  First scope execution (any thread):
    idx transitions:  UNSET -> PENDING -> meter_index  (success)
                      UNSET -> PENDING -> UNSET       (failure)

  Subsequent scope executions:
    Create-form: skipped (idx is already a valid meter index).
    Update-form: evaluates samples and records via meter API.

  Shutdown:
    otelc_deinit() flushes and destroys tracer, meter and logger,
    including all registered instruments and their callbacks.


18.6  Log Records

The filter supports OpenTelemetry log records via the "log-record"
keyword inside otel-scope sections.  Each log record is emitted through
the OTel logger at a configured severity level, with an evaluated body,
optional span correlation and optional key-value attributes.

18.6.1  Log Record Configuration Structure

The flt_otel_conf_log_record structure (conf.h) holds:

  severity     The otelc_log_severity_t severity level.
  event_id     Optional numeric event identifier (int64).
  event_name   Optional event name string.
  span         Optional span reference name (resolved at runtime).
  time         Optional timestamp expression (single-entry conf_sample list).
  attributes   List of flt_otel_conf_sample entries for attributes.
  samples      List of sample expressions for the body.
  cond         Optional if/unless ACL condition gating the record.

The attributes list contains flt_otel_conf_sample entries, one per "attr"
keyword.  Each entry's key field holds the attribute name and its sample
expressions are evaluated at runtime, following the same two-path model
(bare sample or log-format) as span attributes.

The samples list contains exactly one flt_otel_conf_sample entry, which holds
either a list of bare sample expressions or a single log-format expression.  The
log-format path is selected when the value begins with the "%[" sequence; an
explicit '%[ ... ]' wrapper is then stripped before parsing so the inner string
may use HAProxy log-format aliases such as %ci, %t or %ft.

The time list, when populated, holds a single flt_otel_conf_sample produced by
the "time [<unit>] <sample>" keyword.  The unit selector (one of s, ms, us or
ns) is stored in the sample's extra field as an OTELC_VALUE_INT32; a default
of FLT_OTEL_TIME_UNIT_S is recorded when no unit is given.  The sample itself
follows the same parsing rules as any other flt_otel_conf_sample.

18.6.2  Log Record Emission

Log record processing is performed by flt_otel_scope_run_log_record()
(event.c), called from flt_otel_scope_run() after metric instrument
processing and before span finishing.

For each configured log record the function performs:

  1. Severity check: OTELC_OPS(logger, enabled, severity) tests whether
     the logger accepts records at this severity.  If not, the entry is
     skipped.  The threshold is controlled by the "min_severity" option
     in the YAML logs signal configuration.

  2. Attribute evaluation: each entry in the attributes list is evaluated via
     flt_otel_sample_add() into a temporary flt_otel_scope_data structure.
     The evaluated key-value array is passed to logger->log_span() and freed
     after emission.

  3. Body evaluation: the single sample entry is evaluated using one of
     two paths:

     Log-format path (sample->lf_used is true):
       A temporary buffer of global.tune.bufsize is allocated and
       build_logline() evaluates the log-format expression into it.

     Bare sample expression path:
       Each expression in sample->exprs is evaluated via
       sample_process() and converted to a string via
       flt_otel_sample_to_str().  Results are concatenated into a
       single buffer.

  4. Span resolution: if conf_log->span is non-NULL, the runtime
     context's spans list is searched for a scope_span with a matching
     name.  If found, the OTel span pointer is captured for correlation.
     A missing span is non-fatal: the record is emitted without span
     correlation (a debug build traces the miss).

  5. Event timestamp selection: when conf_log->time is non-empty, its single
     sample is evaluated via flt_otel_sample_eval() in native mode.  A string
     result is coerced to int64 with otelc_value_strtonum(); on conversion
     failure the wall-clock ts is kept.  A successful int64 result is split
     into a struct timespec by the unit packed in the sample's extra field
     (s, ms, us or ns), and that timespec becomes the event timestamp.

  6. Emission: logger->log_span() is called with the severity, event_id,
     event_name, resolved span (or NULL), the event timestamp, the wall-clock ts
     as the observed timestamp, the evaluated attributes and the evaluated body
     string.

18.6.3  Logger Lifecycle Summary

  Proxy init (flt_otel_lib_init):
    otelc_logger_create() allocates the logger handle.

  Per-thread init (flt_otel_ops_init_per_thread):
    logger->start() launches the logger background thread.

  Scope execution (flt_otel_scope_run):
    flt_otel_scope_run_log_record() emits records via logger->log_span().

  Shutdown (flt_otel_ops_deinit):
    otelc_deinit() flushes pending log records and destroys the logger.
