CHANGES

1.13.0-dev.2 | 2024-12-19 17:08:09 +0100

  * GH-1949: Fix codegen for string literals containing null bytes. (Benjamin Bannier, Corelight)

    Our strings are UTF8 which can contain literal null bytes. We were
    previously generating incorrect C++ for such literals. This was due to
    us constructing C++ value via constructors taking `const char*` which is
    expected to be null-terminated, i.e., everything after the terminator
    was ignored even though it was emitted.

    With this patch we switch to C++ literals for generating literal and
    runtime strings.

    Closes #1949.

1.13.0-dev.0 | 2024-12-13 09:26:36 +0100

  * GH-1901: Add consistent validation for attributes. (Evan Typanski, Corelight)

    This updates attribute validation to see if attributes are in the
    right places. The goal here is to have one place (ie a big set) to
    answer the question "Can I use X attribute on Y node?"

    There are a lot more moving parts with attribute validation, but those
    generally have to do with behavior. A lot of that requires extra context
    in the validator, which is exactly what the validator is meant to do.
    Much of that is pretty ad-hoc, so it could get cleaned up as well.

1.12.0-dev.240 | 2024-12-12 13:09:05 +0100

  * Make sure autogen-docs pre-commit hook can always run in CI. (Benjamin Bannier, Corelight)

1.12.0-dev.238 | 2024-12-10 14:53:29 +0100

  * Update tutorial to use spicy-driver's batch mode. (Robin Sommer, Corelight)

  * Allow defining parser alias names when running spicy-driver. (Robin Sommer, Corelight)

    ``--parser-alias <alias>=<name>`` now makes the parser of name
    `<name>` accessible under an additional name `<alias>`. The option can
    be used multiple times.

    This is particularly useful when using input batches: by adding an
    alias of the form `PORT/PROTOCOL` (e.g., `--parser-alias
    80/tcp=MyHTTP::Message`), this allows to define a parser to use on for
    connections on that port (because the batch processing interprets any
    ports in the trace as strings and tries to lookup a parser of the
    corresponding name). Adding `%orig` or `%resp` to the alias name
    limits it to the corresponding direction (e.g., ``--parser-alias
    80/tcp%orig=MyHTTP::Request``).

    See the `spicy-driver` documentation for more details on how to use
    this with batches.

  * Let `spicy-{dump,driver}` print out parser alias names with ``-ll``. (Robin Sommer, Corelight)

    ``-list-parsers`` prints the direct names of all known parsers.
    However, internally, we sometimes register additional alias names that
    can also be used to lookup a parser (e.g., if there's a `%mime-type`
    or `%port` property). If one specifies ``-list-parsers`` twice now,
    those aliases are printed as well.
  * GH-1829: Catch integer shifts exceeding the width of the operand. (Robin Sommer, Corelight)

    For constants, we now reject such shifts at compile time. At runtime,
    we catch them by having the forked version of `SafeInt` throw an
    overflow exception.

  * Change `SafeInt` submodule to a fork including a fix. (Robin Sommer, Corelight)

    We now use a fork of `SafeInt` that includes
    https://github.com/dcleblanc/SafeInt/pull/64 (as well as general bump
    to upstream `master`).

  * Add `max-lines` option to Sphinx' `spicy-output` directive. (Robin Sommer, Corelight)

    This limits the lines of output shown to the given number, adding a
    `...` marker if truncated.

  * GH-1928: Deprecate `&anchor` with regular expression constructors. (Robin Sommer, Corelight)

  * Update auto-generated architecture diagram. (Robin Sommer, Corelight)

    Looks like some dependency upgrade changed the content slightly.

1.12.0-dev.226 | 2024-12-09 17:28:27 +0100

  * Fix `HILTI_CXX_FLAGS` for when multiple flags are passed. (Benjamin Bannier, Corelight)

  * Add helper function to perform shell-like string splitting. (Benjamin Bannier, Corelight)

1.12.0-dev.223 | 2024-12-09 11:54:03 +0100

  * Fix ruff 'ISC' lints. (Benjamin Bannier, Corelight)

  * Fix ruff 'I' lints. (Benjamin Bannier, Corelight)

  * Fix ruff 'C4' lints. (Benjamin Bannier, Corelight)

  * Migrate Python linting to ruff. (Benjamin Bannier, Corelight)

  * Bump pre-commit hooks. (Benjamin Bannier, Corelight)

  * Upgrade Python to 3.9 with pyupgrade. (Benjamin Bannier, Corelight)

1.12.0-dev.216 | 2024-12-06 10:34:31 +0100

  * Introduce `deprecated` helper function in Spicy validator. (Benjamin Bannier, Corelight)

  * Add test for using both `[]` and `&count`. (Benjamin Bannier, Corelight)

  * GH-1938: Deprecate `&count` attribute. (Benjamin Bannier, Corelight)

    The preferred way to indicate how many elements should be parsed for a
    vector has been vector syntax over `&count` for a long time, e.g.,

    ```
    type X = unit {
        : uint8[] &count=42; # AVOID.
        : uint8[42];         # PREFER.
    };
    ```

    With this patch `&count` is now deprecated and we emit a warning. We
    will remove support for `&count` in a future release.

    Closes #1938.

1.12.0-dev.212 | 2024-12-05 10:31:19 +0100

  * Fix doc code snippet that won't compile. (Evan Typanski, Corelight)

1.12.0-dev.210 | 2024-12-04 12:08:31 +0100

  * Fix issue with type inference for `result` constructor. (Robin Sommer, Corelight)

    `global x = result("foo");` would end up having type `string` instead
    of `result<string>`.

  * GH-1856: Disallow dereferencing a `result<void>` value. (Robin Sommer, Corelight)

    Closes #1856.

  * GH-1856: Teach `&requires` to accept condition-tests expressions. (Robin Sommer, Corelight)

    This now allows creating custom error messages when a `&require`
    condition fails. Example:

        type Foo = unit {
            x: uint8 &requires=($$ == 1 : error"Deep trouble!'");

            # or, shorter:
            y: uint8 &requires=($$ == 1 : "Deep trouble!'");
        };

    Internally, `&requires` now always wants a condition-test expression.
    That remains transparent to the user, however, because the resolver
    knows how to coerce a pure boolean `&requires` expression into a
    condition test, crafting an error message that happens to look like
    what we used to produce in the past.

    Closes #1856.

  * Teach `assert` to accept an `result<void>` value. (Robin Sommer, Corelight)

    This allows to use the new condition-test operator with assert:

        assert 3 == 4 : error"not equal"

    Because of the RHS of a condition test coercing a `string` to `error`
    automatically, this now also covers the existing syntax for `assert`.
    So the above is now (externally and internally) the same as:

        assert 3 == 4 : "not equal"

  * Add new "condition-test" operator. (Robin Sommer, Corelight)

    This adds a new expression operator to both HILTI and Spicy:

        `COND : ERROR`

    where `COND` is a boolean expression, and `ERROR` is an expression of
    type `error`. When evaluated, the new operator yields a value of type
    `result<void>`, which will be true if the `COND` is true, and set to
    `ERROR` if `COND` is false. In other words, this is a short-cut to
    both test a condition and provide an error message in case it fails.

1.12.0-dev.204 | 2024-11-28 13:57:57 +0100

  * Spell all `Attribute::Kind` values in Pascal case. (Benjamin Bannier, Corelight)

    We are not 100% consistent, but we seem to prefer PascalCase over
    SNAKE_UPPERCASE for enum values. Rename the recently introduced
    `Attribute::Kind` values to that style.

    An intended side effect of this is that we now avoid collisions with a
    possibly defined `DEBUG` macro (e.g., Zeek defines it).

1.12.0-dev.202 | 2024-11-26 18:31:48 +0100

  * GH-1901: Prefer enum over string for attributes. (Evan Typanski, Corelight)

1.12.0-dev.200 | 2024-11-25 16:14:56 +0100

  * GH-1920: Fix some inconsistencies in random access docs. (Evan Typanski, Corelight)

  * GH-1914: Make `$$` documentation more precise. (Evan Typanski, Corelight)

    The documentation would refer to `$$` in an attribute and hook with
    similar language (the "parsed" value). But, in an attribute, it refers
    to the value before any conversions. In a hook, it refers to the value
    after any conversions.

    Now, "parsed" is used to refer to the pre-conversion value, and "final"
    is used to refer to the post-conversion value. Those terms were chosen
    because they make sense when no conversions are applied (or when `$$` is
    used *in* a conversion). But, "final" can still be a little confusing,
    since you can write to `$$`.

1.12.0-dev.196 | 2024-11-25 08:30:19 +0100

  * GH-1918: Fix potential segfault with stream iterators. (Robin Sommer, Corelight)

    When trimming off the beginning of a stream, an existing iterator
    could end up dereferencing its internal chunk pointer even if the
    chunk now no longer existed. The issue was inside the
    increment/decrement operations, which didn't check if the *current*
    iterator offset was still valid (because only then the current chunk
    is guaranteed to be valid too).

  * GH-1918: Add regression tests triggering #1918. (Robin Sommer, Corelight)

1.12.0-dev.193 | 2024-11-21 14:48:06 +0100

  * GH-1919: Validate that sets are sortable. (Evan Typanski, Corelight)

1.12.0-dev.191 | 2024-11-14 11:18:29 +0100

  * Locally disable some warnings for >=gcc-13. (Benjamin Bannier, Corelight)

    fedora-40 and fedora-41 come with GCC versions which trigger new
    warnings.

    - Some uses of `visitor::range` trigger dangling-reference false
      positives, https://gcc.gnu.org/bugzilla/show_bug.cgi?id=107532.
    - The dtor of `Location` triggers maybe-uninitialized false positives in
      `hilti::rt::filesystem::path` for `_file`.

    Disable these warnings locally if we are building with >=gcc-13 for now.

  * Bump Fedora versions in CI. (Benjamin Bannier, Corelight)

1.12.0-dev.188 | 2024-11-07 15:19:35 +0100

  * Make units encompass attributes in location. (Evan Typanski, Corelight)

  * GH-1898: Move type alias attribute validation to parser. (Evan Typanski, Corelight)

    Closes #1898

    This was originally in a validator, but there's some sense that this is
    a parsing issue since attributes should not be allowed at all on most
    constructs of that form.

    Note that a lot of the cleanup here was meant to span removing the
    `opt_attributes` field in `type_decl` altogether, but the fallout from
    that is pretty large. It ended up being less cleanup and more
    rearchitecting. This way still avoids the hacky test for type aliases in
    the validator while not changing the AST structure.

1.12.0-dev.182 | 2024-11-06 14:19:55 +0100

  * GH-1859: Improve error message when a unit parameter is used as a
    field. (Robin Sommer, Corelight)

1.12.0-dev.180 | 2024-11-06 08:41:55 +0100

  * GH-1913: Avoid redundant computation during literal parsing. (Robin Sommer, Corelight)

  * GH-1910: Optimize parsing of literal bytes. (Robin Sommer, Corelight)

    We now create the bytes instance representing the literal as a global
    singleton to avoid instantiating it over and over again.

    Closes #1910.

  * Introduce `UnsafeConstIterator` for bytes instances. (Robin Sommer, Corelight)

    An unsafe iterator offers fast but unchecked access to the data. We
    also rename `bytes::Iterator` to `bytes::SafeConstIterator` so that
    for bytes we now follow the same two-tiered iterator structure as for
    streams. We then switch some library code over to now use unsafe
    iterators, gaining a noticeable speed-up in some cases.

  * Add infrastructure to create and cache global constants. (Robin Sommer, Corelight)

    So far we had replicated low-level code a few times that cached
    constants through a global declaration for reuse. This now factors out
    that logic into a central method inside the Spicy code generator.

    This isn't full interning, which I'm not sure we want/need, but covers
    the performance use case that has now come up a few times.

  * Replace our poor hash function with `std::hash()`. (Robin Sommer, Corelight)

1.12.0-dev.173 | 2024-10-28 10:22:29 +0100

  * GH-1908: Fix performance regression when parsing bytes. (Robin Sommer, Corelight)

    Turns out our improved error messages were adding additional overhead
    because we were now constructing them through `fmt()` each time we
    needed more data, independent of whether there was actually going to
    be an error reported.

    This adds a second version of `waitForInput()` that doesn't receive an
    already prepared error message, but just returns false on error so
    that the caller can throw itself.

  * GH-1857: Support `&requires`for individual vector items. (Robin Sommer, Corelight)

1.12.0-dev.169 | 2024-10-25 10:09:50 +0200

  * GH-1895: Do no longer escape backslashes when printing strings or bytes. (Robin Sommer, Corelight)

    Escaping backslashes in `print` output seems both unnecessary and
    confusing.

  * Rename `expandEscapes()` to `expandUTF8Escapes`. (Robin Sommer, Corelight)

    For better readability to avoid confusion about the semantics of the
    function.

  * Introduce style flags for functions rendering values into strings. (Robin Sommer, Corelight)

    This is just cleanup for better readability. I also fixed a couple of
    cases where we now escape control characters where previously we
    wouldn't, but I believe should have. Otherwise no functional change.

  * GH-1893: Encompass child node's location in parent. (Evan Typanski, Corelight)

    When a child is added with `addChild`, the parent's location should
    (generally) span over that child as well. This primarily helps for cases
    where a node doesn't have much of a location until it gets children
    added - like `AttributeSet`. The locations for those should encompass
    all of the attributes within the set. That logic applies for any node
    with a child: if it's the child, then its location should reflect that.

1.12.0-dev.163 | 2024-10-24 09:59:35 +0200

  * GH-1803: Fix namespacing of `hilti` IDs in Spicy-side diagnostic output. (Robin Sommer, Corelight)

    We now show them with a `spicy` prefix, which makes more sense for
    users.

  * Add printer plugin hook to customize ID printing. (Robin Sommer, Corelight)

    Because IDs are no longer AST nodes, we did not have a way for a
    compiler plugin's printing code to modify how it would like them to be
    printed. This adds a corresponding hook. It's not used anywhere yet,
    but will be soon.

    In addition, we add a notion of "user-visibility" to the printer API
    so that printing code knows whether the resulting output is something
    that will be shown to the user (e.g., in diagnostics), or remains
    internal (e..g, raw code output, debugging output). Likewise not used
    yet, but will be soon.

    We also clean up the printer API a little bit.

  * Add scope guard utility class. (Robin Sommer, Corelight)

1.12.0-dev.159 | 2024-10-23 17:33:28 +0200

  * Bump pre-commit hooks (Benjamin Bannier, Corelight)

  * Remove stray removal of directory in `/tmp` in test. (Benjamin Bannier, Corelight)

1.12.0-dev.156 | 2024-10-21 15:50:46 +0200

  * GH-1063: Document arguments to `new` operator. (Robin Sommer, Corelight)

  * GH-1858: Fix the literals parsers not following coercions. (Robin Sommer, Corelight)

1.12.0-dev.152 | 2024-10-18 15:31:40 +0200

  * GH-1891: Fix GCC 11 compilation failure. (Evan Typanski, Corelight)

1.12.0-dev.150 | 2024-10-17 10:16:57 +0200

  * GH-1792: Prioritize error message reporting unknown field. (Robin Sommer, Corelight)

    This suppresses some non-interesting follow-up errors.

  * GH-1790: Provide proper error message when trying access an unknown unit field. (Robin Sommer, Corelight)

    Returning `type::Unknown` instead of `type::Auto` will let the
    resolver process stop, allowing an already existing error message to
    kick in later. We were doing it this way already for structs, but not
    for units.

    This also includes a fix for a bug with finding bitfield ranges by ID,
    which was triggered by the change.

  * GH-1791: Fix usage of `&convert` with unit's requiring parameters. (Robin Sommer, Corelight)

    The generated code needs to create a temporary instance of the type,
    but doesn't have any arguments to provide to it. But that's ok, and we
    now let the validation pass and just instantiate a default-constructed
    instance.

    However, this change now requires an additional validator check on the
    Spicy side to ensure fields giving arguments to types do so correctly
    Before we happened to check that implicitly on the HILTI-side through
    code that now would let it pass if no arguments were given.

  * Factor out logic to validate arguments given to a type. (Robin Sommer, Corelight)

    This will allow Spicy's validator to use it as well.

    We also add two options to (1) accept usages where no argument is given at
    all even when the type would normally require it,;and (2) skip any
    actual type checking, and just confirm argument count. In a subsequent
    change, we'll use (1) to fall-back to a type's default constructor,
    and (2) to check type usage inside unit fields where we haven't
    fully coerced arguments yet.

1.12.0-dev.144 | 2024-10-16 16:10:40 +0200

  * GH-1861: Disallow ignored attributes on type aliases. (Evan Typanski, Corelight)

    This may make previously fine code invalid. Since that code would likely
    be hiding a bug, that seems okay to me.

    Closes #1861

1.12.0-dev.140 | 2024-10-15 16:21:58 +0200

  * GH-1871: Fix `&max-size` on unit containing a `switch`. (Robin Sommer, Corelight)

    We would advance our input too early, letting the subsequent
    `&max-size` check fail.

1.12.0-dev.138 | 2024-10-14 16:16:08 +0200

  * GH-1868: Associate source code locations with current fiber instead of current thread. (Robin Sommer, Corelight)

    This fixes potential location mix-ups when switching between fibers.
    Note that we still need a context-wide fallback location as well
    because we're not always running inside a fiber.

    I ran a performance comparison before/after and couldn't measure a
    difference. Looks like using TLS storage was a case of premature
    optimization.

1.12.0-dev.136 | 2024-10-14 14:20:16 +0200

  * Disable cpplint iwyu check. (Benjamin Bannier, Corelight)

  * Replace a few C-style casts with `static_cast`. (Benjamin Bannier, Corelight)

1.12.0-dev.133 | 2024-10-08 21:50:53 +0200

  * Document `continue` statements. (Evan Typanski, Corelight)

1.12.0-dev.131 | 2024-10-08 17:09:30 +0200

  * Fix "it's" used as a possessive. (Evan Typanski, Corelight)

1.12.0-dev.129 | 2024-10-07 13:02:38 +0200

  * Stop using deprecated pre-commit stage names. (Benjamin Bannier, Corelight)

    The names we used have been deprecated since some time, see
    https://github.com/pre-commit/pre-commit/issues/2732.

1.12.0-dev.127 | 2024-10-05 13:21:27 +0200

  * GH-1867: Fix infinite loops with recursive types. (Evan Typanski, Corelight)

    Closes #1867

    There are two different cases where infinite loops happen with recursive
    types. First, a type may reference itself (`type Data = Data`). Second,
    a type may reference itself inside some other type (`type Data =
    vector<Data>`).

    The first is fixed with a recursion limit. Since the type simply cannot
    resolve, it doesn't get anywhere near codegen. You could detect cycles,
    but that introduces some extra overhead and complexity that shouldn't
    be needed in a "simple" function.

    The second is fixed with an ad-hoc "occurs" check in type unification.
    That just detects cycles and aborts if one is present. This could be
    placed at some other place in the "resolve until convergence" loop, but
    it seems best put closest to the source of the issue.

1.12.0-dev.124 | 2024-09-30 17:43:41 +0200

  * GH-1875: Fix potential nullptr dereference when comparing streams. (Robin Sommer, Corelight)

    Because we are operating on unsafe iterators, need to catch when one
    goes out of bounds.

1.12.0-dev.122 | 2024-09-30 14:18:09 +0200

  * GH-1874: Add new library function `spicy::bytes_to_mac`. (Robin Sommer, Corelight)

    ```
    ## Returns a bytes value rendered as a MAC address string (i.e., colon-separated hex bytes).
    public function bytes_to_mac(value: bytes) : string;
    ```

  * Optimize `spicy::bytes_to_hexstring` and `spicy::bytes_to_mac`. (Robin Sommer, Corelight)

1.12.0-dev.119 | 2024-09-30 13:37:10 +0200

  * GH-1846: Fix bug with captures groups. (Robin Sommer, Corelight)

    When extracting the data matching capture groups we'd take it from the
    beginning of the stream, not the beginning of the current view, even
    though the latter is what we are matching against.

  * Add missing trim after matching a regular expression. (Robin Sommer, Corelight)

1.12.0-dev.116 | 2024-09-30 13:35:49 +0200

  * GH-1842: Fix when input redirection becomes visible. (Robin Sommer, Corelight)

    With `&parse-at/from` we were updating the internal state on our
    current position immediately, meaning they were visible already when
    evaluating other attributes on the same field afterwards, which is
    unexpected.

1.12.0-dev.114 | 2024-09-30 10:24:42 +0200

  * GH-1844: Fix nested look-ahead parsing. (Robin Sommer, Corelight)

    When parsing nested vectors all using look-ahead, we need to return
    control back to upper level when an inner look-ahead isn't found.

    This may change the error message for "normal" look-ahead parsing (see
    test baseline), but the new one seems fine and potentially even
    better.

  * Apply compiler suggestion. (Robin Sommer, Corelight)

1.12.0-dev.111 | 2024-09-26 20:00:42 +0200

  * Remove `ast-stats` output from tests baselines. (Robin Sommer, Corelight)

    These are too noisy as they update with every AST change. Originally I
    included `ast-stats` into some test baselines to ensure the new AST
    infrastructure isn't doing weird stuff, but seems that's working out
    ok.

1.12.0-dev.109 | 2024-09-26 17:29:17 +0200

  * Fix parsing ambiguity with properties. (Robin Sommer, Corelight)

    In the past, we had special-cased properties in our Flex/Bison parser
    so that when parsing an expression, they wouldn't be recognized.
    However, that now led an error field hook of the form `x: bytes
    &size=42 %error` to be parsed as `&size=(42 % error)`. We now switch
    to white-listing all known properties, just as we already do for
    attributes. That way conflicts should be extremely rare.

  * Redo error handling docs (Robin Sommer, Corelight)

    The old text was very outdated. This extends the content, documents
    the new per-field `%error` handler, and moves it all into the
    "parsing" section to have it closer to the error recovery content.

  * GH-1824: Add support for field-local `%error` handlers. (Robin Sommer, Corelight)

    We now support attaching an `%error` handler to an individual field:

        type Test = unit {
            a: b"A";
            b: b"B" %error { print "field B %error", self; }
            c: b"C";
        };

    With input `AxC`, that handler will trigger, whereas with `ABx` it
    won't. If the unit had a unit-wide `%error` handler as well, that one
    would trigger in both cases (i.e., for `b`, in addition to its field
    local handler).

    The handler can also be provided separately from the field:

        on b %error { ... }

    In that separate version, one can receive the error message as well by
    declaring a corresponding string parameter:

        on b(msg: string) %error { ... }

    This works externally, from outside the unit, as well:

        on Test::b(msg: string) %error { ... }

    This is eebased on top of `topic/robin/optimize-type-parsing` so that
    we get the peephole optimizer.

  * Refactor field hook attributes. (Robin Sommer, Corelight)

    Currently we only have two types of field hooks: standard hooks and
    `foreach` hooks. To prepare for more types, this refactors the code
    to represent the type with an `enum` instead of a `foreach` boolean. It
    also moves validation of permitted attributes from the parser to
    the validator.

1.12.0-dev.104 | 2024-09-26 14:14:36 +0200

  * Fine-tune `bytes` literal parsing further. (Robin Sommer, Corelight)

    This now optimizes the generated code for parsing a `bytes` literal
    based on what we're going to use it for, or not (require literal vs.
    look-ahead vs skip).

    The standard case of a parsing an expected literal now looks like
    this:

    ```
        # Begin parsing production: Ctor: b1  -> b"abc" (const bytes)
        (*self).b1 = spicy_rt::expectBytesLiteral(__data, __cur, b"abc", "../tests/spicy/types/bytes/parse-length.spicy:20:10-20:15", Null);
        __cur = __cur.advance(3);

        if ( __trim )
            (*__data).trim(begin(__cur));

        # End parsing production: Ctor: b1  -> b"abc" (const bytes)
    ```

  * Add peephole optimization simplifying rethrowing catch blocks. (Robin Sommer, Corelight)

    The optimization turns this:

    ```
    function void foo() {
         try {
            ...
         } catch {
             throw;
         }
     }
    ```

    into

    ```
    function void foo() {
         {
            ...
         }
     }
    ```

    It would be even nicer if we didn't need the braces around the
    remaining block, but it's generally not safe to remove them because if
    the block declares any locals their life-times and visibility would
    change.

  * GH-1592: Add peephole optimizer for final AST tuning. (Robin Sommer, Corelight)

    We use this to remove two statement constructs that the main optimizer
    may leave behind:

        1. `default<void>()`
        2. `(*self).__error = __error; __error = (*self).__error;`

    The second case is a quite specific situation that eventually, once we
    have CFG/DFG tracking, the main optimizer should be able to cover more
    generically. However, for now, it's just not nice to always have
    these blocks in the generated C++ code, so adding this special case
    seems useful.

    Couples notes on (2):

        - Per #1592, case 2 may also have overhead. Closes #1592.
        - Technically, this optimization isn't always correct: subsequent
          code could assume that `(*self).__error` is set, whereas after
          removal it's not (or not to the expected value). However,
          `__error` is strictly-internal state, and we know that we don't
          use it any different, so this seems ok until we have more
          general optimizer logic.

  * Disable check for reserved IDs with `--skip-validation`. (Robin Sommer, Corelight)

    This allows to write tests that use internal IDs.

  * Simplify parsing of literals. (Robin Sommer, Corelight)

    There are two changes in here:

    1. For grammars that don't use look-ahead, we skip the runtime check for a
    pending look-ahead symbol, because we know we will never have one. This removes
    generated code of the form `if ( _lah ) { ... }`. This change needs a bit of
    machinery unfortunately because we need to get the information about look-ahead
    usage over into the codegen for literal parsing.

    2. For bytes literals, we now outsource their parsing to a runtime function to
    make the generated code simpler. As a side effect this also provides more
    informative error messages when the literal isn't found.

    Taking the two together means that the code for parsing a plain `b"Foo"`
    literal may look like this now:

        spicy_rt::expectBytesLiteral(__data, __cur, b"Foo", "b.spicy:4:10-4:15", Null);
        __cur = __cur.advance(3);

        if ( __trim )
            (*__data).trim(begin(__cur));

        (*self).foo = b"Foo";

  * Optimize parsing for `bytes &size=N`. (Robin Sommer, Corelight)

  * Add mode for optimize types parsing. (Robin Sommer, Corelight)

    This provides the surrounding infrastructure, but does not yet
    implement it for any type.

  * Refactor type parsing modes. (Robin Sommer, Corelight)

    So far we had a boolean flag to differentiate between "normal" and
    "try" parsing. We turn this flag into an enum now so that we can more
    easily extend the set of modes later. No functional change otherwise.

  * Refactor pre/post parsing logic. (Robin Sommer, Corelight)

    This splits out the `&size/&max-size` handling so that we can
    special-case that later.

  * Cleanup: replace optionals with pointers. (Robin Sommer, Corelight)

    For AST nodes, we can/should use nullptrs instead of unset optionals.
    The versions using `optional` were still a left-over from the old AST
    code.

1.12.0-dev.93 | 2024-09-26 13:07:41 +0200

  * Disallow unit variables nested into other items. (Robin Sommer, Corelight)

    Initialization would be ill-defined.

  * GH-1839: Support `if`-blocks in unit. (Robin Sommer, Corelight)

    We now support `if` around a block of unit items:

        type X = unit {
             x: uint8;

             if ( self.x == 1 ) {
                 a1: bytes &size=2;
                 a2: bytes &size=2;
             };
        };

    One can also add an `else`-block:

        type X = unit {
             x: uint8;

             if ( self.x == 1 ) {
                 a1: bytes &size=2;
                 a2: bytes &size=2;
             }
             else {
                 b1: bytes &size=2;
                 b2: bytes &size=2;
             };
        };

  * Add a new unit item `Block` that stores a sequence of subitems. (Robin Sommer, Corelight)

    We now have a new unit item `Block` that represents a sequence of
    sub-items. This goes along with a new production `Block` that the item
    gets turned into inside the grammar.

    The implementation reuses logic from `unit::item::switch_::Case`. We
    then refactor `Case` to use the new `Block` instead of continuing to
    maintain its own list of items.

    We also refactor some other logic previously located with `Switch`
    to now apply to `Blocks` as well.

    Optionally, a `Block` can have a conditional expressions, an
    "else"-block as well attributes with it. We don't use them yet, but
    will soon; tests then coming as well. However, we already also
    refactor some attribute logic that previously went just with the
    `Switch` unit item to now also apply to the `Block` item, so that the
    two can support the same attributes.

  * Remove `production::Boolean`. (Robin Sommer, Corelight)

    Turns out this was neither used nor fully implemented.

  * Run a final resolver pass after optimization. (Robin Sommer, Corelight)

    Optimization may leave node state unset, such as canonical IDs if we
    created new nodes. The final resolver pass gets that back to a
    well-defined state.

    This also moves the debug output for `ast-final` out of the resolver
    loop; it now executes at the very end after optimization. That's a
    change I've been meaning to do for a while because the "final" AST
    wasn't really final. `ast-resolved` now shows what used to be
    `ast-final` (and no longer each iteration of the resolver).

    Test coming with subsequent commit, where a new test would break
    otherwise.

1.12.0-dev.87 | 2024-09-26 13:07:06 +0200

  * Compile generated C++ with `-O0` in debug mode. (Robin Sommer, Corelight)

  * Bump centos-stream in documentation. (Benjamin Bannier, Corelight)

  * Bump latest release in docs. (Benjamin Bannier, Corelight)

1.12.0-dev.83 | 2024-09-19 08:58:15 +0200

  * Fix attributes not getting validated. (Evan Typanski, Corelight)

    This particularly fixes the case where a ctor within a field will not
    have its attributes validated because it does not go through
    `isParseableType`. The "generic attribute checks" are moved to apply to
    all attributes, while type-specific ones remain in `isParseableType`

1.12.0-dev.81 | 2024-09-18 14:42:34 +0200

  * Add internal error for unvalidatable vectors. (Evan Typanski, Corelight)

    Vectors were changed to never be a vector-of-type, just a
    vector-of-items that redirect to a type. This allows validating
    attributes and whatnot properly, which only apply to fields, not types.
    The consequence is that a vector-of-types should be impossible, or else
    there is an error in the parser's workarounds to get to that state,
    which is a bug.

    Add an internal error so that case never surfaces without a big scary
    error.

  * Move legacy vector syntax check to new production. (Evan Typanski, Corelight)

  * Remove vector builders for type/ids. (Evan Typanski, Corelight)

    Currently, vectors need the inner type to actually point to a field.
    This removes the repeat from builder methods that would otherwise need a
    field but don't so that they aren't mistakenly added.

1.12.0-dev.77 | 2024-09-11 08:18:38 +0200

  * GH-1847: Fix resynchronization issue with trimmed input. (Robin Sommer, Corelight)

    When input had been trimmed, `View::advanceToNextData` could end up
    returning a view starting ahead of the valid area.

  * Add missing printer support for exception values. (Robin Sommer, Corelight)

  * GH-1860: Fix parsing for vectors of literals. (Robin Sommer, Corelight)

    This was broken in two ways:

    1. with the `(LITERAL)[]` syntax, the parser would not recognize literals
       using type constructors
    2. with the syntax `LITERAL[]`, we'd try to store the parsed value
       into a vector

  * Fix error message. (Robin Sommer, Corelight)

1.12.0-dev.71 | 2024-09-06 16:29:02 +0200

  * Move some `scoped_id` out for better locations. (Evan Typanski, Corelight)

  * Fix `vector<>` syntax missing diagnostics. (Evan Typanski, Corelight)

  * GH-1832: Fail for vectors with bytes but no stop. (Evan Typanski, Corelight)

    Cases with parentheses were caught before this change. The comments
    indicate behavior before the change, even though both cases should fail:

    public type BytesVectors = unit {
        : bytes[10] &size=5; # No diagnostic - oh no!
        : (bytes)[10] &size=5; # Error - good
    };

    That's because a new `_anon` field is created with the parentheses. This
    makes the `bytes` in an item, which can have its own `&size` and gets
    checked later. But `bytes` by itself cannot have the `&size` attribute
    since it's just a type - the vector will use its own `&size` attribute!

    The fix makes all types in vectors use the anonymous field instead, so
    their lack of attributes are properly modeled during validation.

1.12.0-dev.67 | 2024-09-04 12:44:42 +0200

  * GH-1852: Fix `skip` with units. (Robin Sommer, Corelight)

    For unit parsing with `skip`, we would create a temporary instance
    but wouldn't properly initialize it, meaning for example that
    parameters weren't available. We now generally fully initialize any
    destination, even if temporary.

  * Do not by default skip CI in update-changes cfg. (Benjamin Bannier, Corelight)

    We rely on CI to create releases, so having `[skip CI]` in our
    update-changes config was always a hassle which needed to be edited out
    when creating a release (just tagging would autocommit with the message
    template without a way to change the commit message before tagging).

    Drop this part to make releases easier and less error prone.

1.12.0-dev.63 | 2024-08-26 17:10:07 +0200

  * Bump 3rdparty/any from `7c76129` to `a05d5ad` (dependabot[bot])

1.12.0-dev.61 | 2024-08-14 14:04:27 +0200

  * Suppress warning `invalid-offsetof`. (Robin Sommer, Corelight)

    We now run into these in CI due to `-Werror`.

    Also clean up existing duplication of
    `-Wno-unused-command-line-argument`.

  * GH-1835: Fix computation of declaration dependencies. (Robin Sommer, Corelight)

    One more branch we had to follow to track down all dependencies
    between global declarations: external qualified types.

1.12.0-dev.58 | 2024-08-09 15:57:52 +0200

  * Add `string::lower` and `string::upper` (Evan Typanski, Corelight)

    These already had a whole impl in the `string` namespace, so this just
    hooks it up so you can use them reasonably on a string type in a spicy
    file. It also switches those impls to use `string_view`

  * Fix `split` and `split1` docs to be more accurate (Evan Typanski, Corelight)

    The documentation for the bytes method seemed incorrect (or, at least,
    confusing) where an empty separator did *not* split on white space. This
    correctly documents the behavior with what the actual behavior is
    (especially for split1).

    Note, the behavior is a bit confusing, unintuitive, and maybe
    inconsistent between the implementations of `split` and `split1`.
    Namely, `split1` will sometimes but the entire value in the first
    element of the tuple (if a match isn't found) or the second element (if
    the separator is just an empty string).

    For now this just updates the documentation, but I'd also welcome a
    change in behavior.

  * Add `string::split` and `string::split1` helpers (Evan Typanski, Corelight)

    This doesn't do any work to fix the TODOs that were already in the Bytes
    version. Also, there is a small unclear bit about the documentation for
    the `split` op in both Bytes and now string, namely that an empty `sep`
    argument should make the break take place at sequences of whitespace,
    but it does not do that. It may also be misreading the documentation,
    but one of the two should probably be clarified.

  * Add `string::starts_with` helper (Evan Typanski, Corelight)

1.12.0-dev.53 | 2024-08-09 15:45:05 +0200

  * GH-1831: Fix optimizer regression. (Robin Sommer, Corelight)

    We were no longer marking types as used that are referenced through a
    type name.

1.12.0-dev.51 | 2024-08-09 15:43:51 +0200

  * Fix AST dependency computation. (Robin Sommer, Corelight)

    Turns out the logic wasn't quite right yet. It worked well enough if
    all units get wrapped into `ValueReference` because then we just need
    a forward declaration to make that work; and those we would always
    output first. But it wouldn't actually track dependencies correctly to
    output them in the right order so that full struct are available where
    needed. This commit reworks that logic. Turns out this also further
    simplifies logic and state tracking, which is a good sign.

  * Provide dependency information to AST transformations. (Robin Sommer, Corelight)

    Before, we computed dependency information only once just before final
    codegen, meaning that Spicy-to-HILTI lowering would not have it
    available. This adds a separate dependency computation just before the
    transformation pass. While this isn't used yet, the change prepares
    for future functionality leveraging dependencies during lowering (like
    selective application of `&on-heap`).

  * Make AST dependency information deterministic. (Robin Sommer, Corelight)

    The dependency information returned by the context wasn't sorted, with
    order depending on pointer values. That means that users of that
    information would end up processing dependencies in an undetermined
    order, making their results non-deterministic.

  * Fix an error message. (Robin Sommer, Corelight)

1.12.0-dev.45 | 2024-08-09 14:01:49 +0200

  * Silence unused result warnings when expression has [[nodiscard]]. (Arne Welzel, Corelight)

    The code generated by assert-exception *o == "", where o is an optional
    results in an unused result warning due to operator== being
    [[nodiscard]].

  * Add -Wno-unused-command-line-argument to LD_FLAGS. (Arne Welzel, Corelight)

    Suppress warnings when flags on the command line aren't used:

        c++: error: -lspicy-rt-debug: 'linker' input unused
          [-Werror,-Wunused-command-line-argument]

  * Add Ubuntu 24.04 to CI. (Arne Welzel, Corelight)

  * Avoid maybe-uninitialized warning. (Arne Welzel, Corelight)

    Not sure, but Ubuntu 20's GCC 9.4 and OpenSUSE 15.5's GCC 7.3 are reporting

        /hilti/runtime/include/hilti/rt/types/map.h:235:21: error: '<anonymous>'
          may be used uninitialized in this function [-Werror=maybe-uninitialized]

  * Fix sys/errno.h warning. (Arne Welzel, Corelight)

    On Alpine 3.18:

        /usr/include/sys/errno.h:1:2: error: #warning redirecting incorrect
          #include <sys/errno.h> to <errno.h> [-Werror=cpp]
        1 | #warning redirecting incorrect #include <sys/errno.h> to <errno.h>

  * Avoid use-after-free warning with GCC-13.2. (Arne Welzel, Corelight)

    Not exactly sure what's going on here. I *think* calling data->begin()
    and data->end(), it sees/assumes one path freeing the underlying chain
    of the SafeConstIterator and the other then accessing the freed pointer.
    Maybe something about the evaluation of the arguments being unspecified.
    It doesn't trigger when fetching offsets sequentially beforehand, so do
    that...

        inlined from 'spicy::rt::detail::printParserState(...)::<lambda()>'
          at /src/spicy/runtime /src/parser.cc:81:19:
        /src/hilti/runtime/include/hilti/rt/intrusive-ptr.h:43:20:
          error: pointer used after 'void operator delete(void*, std::size_t)'
          -Werror=use-after-free]
        43 |     if ( m && --m->_references == 0 )
           |                 ~~~^~~~~~~~~~~$

  * Use Bytes directly instead of uint8_t array for pack(). (Arne Welzel, Corelight)

    GCC 12.2 on Debian 12 complains that raw might be unused. Replace
    with Bytes/std::string and casting data() to uint8_t* instead.

    Warning was:

        inlined from 'hilti::rt::Bytes hilti::rt::Address::pack(
          hilti::rt::ByteOrder) const'
          at /src/hilti/runtime/src/types/address.cc:132:54:
          /usr/include/c++/12/bits/char_traits.h:431:56: error:
         'raw' may be used uninitialized [-Werror=maybe-uninitialized]

  * Don't ignore alloca() return value in test. (Arne Welzel, Corelight)

    Silence warning on Alpine:

        fiber.cc:314:11: error: ignoring return value of 'void*
          __builtin_alloca(long unsigned int)' declared with attribute
          'warn_unused_result' [-Werror=unused-result]

  * Disable -Werror when precompiling libspicy.h headers (Arne Welzel, Corelight)

    https://gcc.gnu.org/bugzilla/show_bug.cgi?id=47857

  * Use operator() from OptimizerVisitor via using. (Arne Welzel, Corelight)

    This triggers -Werror=overloaded-virtual= on GCC 13.2 otherwise.

  * Fix --enable-werror werror_flags. (Arne Welzel, Corelight)

    Found while trying to test something with EXTRA_CXX_FLAGS.
    This got lost in e27cd5ac0443659e7e9fdae2f3bbe5920bd3564b.

1.12.0-dev.33 | 2024-08-07 12:38:27 +0200

  * Add an AST pass collecting dependencies between declarations. (Robin Sommer, Corelight)

  * Rework emission of C++ code. (Robin Sommer, Corelight)

    We now use the new AST dependency information to determine in which
    order to output C++ declarations, including whether to output them at
    all. Previously, we used a needed to get the desired order through
    a pretty hacky, adhoc scheme that would prioritize some types
    manually as needed.

    This includes:

    - We now only declare C++ types that are actually needed by the unit;
      no more bulk import everything from an imported unit.

    - Remove all the manual scheduling of when `cxx::declaration::*` are
      to be emitted (no more forward declarations, no more manual
      prioritization). The order of output is now generally determined
      through dependencies. For example, before we output a type, we
      output anything it depend on.

    - Merge `cxx::Function` into `cxx::declaration::Function`. This means
      we generally don't need to emit separate declaration vs
      implementation anymore; instead we decide at emission time what
      parts to output.

    - Unify treatment of cxx::declaration::*: we go through emission
      phases now where each type decides what to emit when.

    - Remove various state from cxx::declaration::* that’s no longer
      needed/used.

    - We now unconditionally forward declare structs and unions; seems
      cheap enough and makes things quite a bit easier.

    - We new logic strives to maintain the same output where possible so
      that we only get minimal baselines changes (mainly some small
      ordering changes).

  * Properly set parse extension when importing by path. (Benjamin Bannier, Corelight)

  * Rename `cxx::type::Struct`'s `inlineCode` to `code`. (Benjamin Bannier, Corelight)

    This function now generates not `inline` code, but instead code with
    bodies which we would only want to include once in the generated C++.

  * Tweak when we emit struct inline code. (Benjamin Bannier, Corelight)

    We would previously emit inline C++ code for types into each unit module
    where the type's module was imported. This meant that we potentially
    compiled identical function bodies in many files. While this could in
    theory lead to better local inlining decisions it also caused poor
    scaling as the number of modules in a Spicy grammar grows (in
    particular).

    With this patch we now emit these bodies at most once, in particular
    into the module of the type. To support this we make a number of
    declarations and definitions not `inline` anymore so they can link
    correctly. We also reduce which code gets included when importing a
    module.

  * Only emit full C++ body of functions in module of matching namespace. (Benjamin Bannier, Corelight)

  * Simplify body of generated `operator<<`. (Benjamin Bannier, Corelight)

  * Remove unused methods to retrieve module dependencies. (Robin Sommer, Corelight)

    If we ever need this, we can now use the new declaration dependencies
    instead.

  * Fix `Node::tryAs_()`. (Robin Sommer, Corelight)

    (Not used anywhere currently, but didn't work and want to keep.)

  * Fully qualify runtime identifiers in generated code. (Robin Sommer, Corelight)

    We weren't consistent doing that everywhere, which would lead to
    trouble with upcoming changes.

1.12.0-dev.21 | 2024-08-07 10:49:10 +0200

  * GH-1823: Don't qualify magic linker symbols with C++ namespace. (Robin Sommer, Corelight)

    We need them at their original values because that's what the runtime
    lbirary is hard-coded to expect.

1.12.0-dev.19 | 2024-08-06 15:01:21 +0200

  * Fix use of move'd from variable. (Benjamin Bannier, Corelight)

    Function parameters still shadown members in C++. This is a fixup of
    c3abbbe8eb28d1e2cee5b58b19e1a30710b38853.

1.12.0-dev.17 | 2024-08-06 13:49:47 +0200

  * GH-1817: Prevent null ptr dereference when looking on nodes without `Scope`. (Benjamin Bannier, Corelight)

    Closes #1817.

1.12.0-dev.15 | 2024-08-06 13:21:57 +0200

  * GH-3865: Add CMAKE_CXX_FLAGS to HILTI_CONFIG_RUNTIME_LD_FLAGS. (Arne Welzel, Corelight)

    When building Spicy with CXXFLAGS='--coverage', a subsequent
    spicy-driver invocation fails with:

        $ ./build/bin/spicy-driver ./x.spicy
        [error] failed to load library "/tmp/__library__c2d0d689ca0dbabb.hlto":
        /tmp/__library__c2d0d689ca0dbabb.hlto: undefined symbol: __gcov_merge_add

    Similarly for `spicyz` and Zeek.

    CMake adds CMAKE_CXX_FLAGS to all its linker invocations [1] which is why
    building Spicy's executables itself works out. Prepend EXTRA_CXX_FLAGS
    and CMAKE_CXX_FLAGS to HILTI_CONFIG_RUNTIME_LD_FLAGS as a fix.

    Unclear if building Spicy with `--coverage` should imply building produced
    `.hlto` files with coverage, too, but that's what is done with other CXXFLAGS
    already today.

    Relates to zeek/zeek#3865.

    [1] https://cmake.org/cmake/help/latest/variable/CMAKE_LANG_FLAGS.html

1.12.0-dev.13 | 2024-08-06 12:51:46 +0200

  * Updates <bytes>.to_int() <bytes>.to_uint() documentation. (Arne Welzel, Corelight)

    I'm going for RuntimeError for now, given that's at least not wrong
    given what's thrown in various places.

  * Fix undefined shifts of 32bit integer in toInt(). (Arne Welzel, Corelight)

    1U is 32bit on a 64bit system and shifting it by more than 31 bits
    is undefined. The following does currently produce -4294967296 instead
    of -1:

        b"\xff\xff\xff\xff".to_int(spicy::ByteOrder::Big)

  * Fix to_uint(ByteOrder) for empty byte ranges. (Arne Welzel, Corelight)

    to_uint() and to_int() for empty byte ranges throw when attempting to
    convert printable decimals to integers. Do the same for the byte order
    versions. The assumption is that it is really an error when the user
    calls to_int() or to_uint() on an empty byte range.

1.12.0-dev.9 | 2024-08-02 12:30:03 +0200

  * Simplify test. (Benjamin Bannier, Corelight)

    This test was using complicated input data and parsing it in a
    hard to follow way. This patch simplifies the test data and its
    handling.

  * GH-1815: Disallow expanding limited `View`s again with `limit`. (Benjamin Bannier, Corelight)

    The documented semantics of `View::limit` are that it creates a new view
    with equal or smaller size. In contrast to that we would still have
    allowed to expand views with more calls `limit` again as well.

    This patch changes the implementation of `View::limit` so it can only
    ever make a `View` smaller.

    We also tweak the implementation of the check for consumed `&size` when
    used together with `&eod`: if the `&size` was already nested in a
    limited view a larger `&size` value could previously extend the view so
    the `&eod` effectively was ignored. Since we now do not extend the
    `View` anymore we need to only activate the check for consumed `&size`
    if `&eod` was not specified since in this case the user communicated that
    they are fine with consuming less data.

    Closes #1815.

1.12.0-dev.6 | 2024-07-31 14:24:26 +0200

  * Start v1.12.0 development. (Benjamin Bannier, Corelight)

1.11.0-dev.287 | 2024-07-31 11:15:49 +0200

  * Remember normalized paths when checking for duplicate files in driver. (Benjamin Bannier, Corelight)

  * Remember files processed by the driver. (Benjamin Bannier, Corelight)

  * GH-1813: Fix equality implementation of module UID. (Benjamin Bannier, Corelight)

1.11.0-dev.282 | 2024-07-30 17:02:31 +0200

  * GH-1810: Fix nested look-ahead switches. (Robin Sommer, Corelight)

1.11.0-dev.279 | 2024-07-26 12:47:30 +0200

  * GH-1808: Fix non-converging optimizer pass for used functions. (Benjamin Bannier, Corelight)

1.11.0-dev.277 | 2024-07-22 14:44:40 +0200

  * Add rsync to CI Docker image. (Benjamin Bannier, Corelight)

    `rsync` is used by `doc/scripts/autogen-docs` and without it the script
    cannot run.

  * Fix running of `autogen-docs` pre-commit hook. (Benjamin Bannier, Corelight)

    If the `diagrams` package was not installed this hook would now fails.
    Since it runs for changes to any type of file this was not ideal.

    This patch adds a check for that dependency and stops the docs update
    script if it is missing.

1.11.0-dev.274 | 2024-07-22 12:01:04 +0200

  * GH-1795: Fix illegal stream content access through invalid view. (Robin Sommer, Corelight)

    When a view's beginning ended up outside of the stream's valid range
    (e.g., because the stream has been trimmed in the meantime), we
    wouldn't catch that on access and attempt to access data that was no
    longer valid. This commit extends the safety checks accordingly.

1.11.0-dev.272 | 2024-07-22 10:53:56 +0200

  * GH-1800: Add test for invoked but unimplemented sink hooks. (Benjamin Bannier, Corelight)

    This is a regression test for #1800.

  * GH-1800: Guard access to sink hooks. (Benjamin Bannier, Corelight)

    If no sink hooks were specified by the user trying to invoke them would
    raise a `std::bad_function_call` previously since we did not have
    explicitly guard the code. With #1736 we us infrastructure with less
    implicit checks so we need to handle the absence of a value ourself
    since otherwise we might end up dereferencing a nullptr.

1.11.0-dev.269 | 2024-07-22 10:53:15 +0200

  * Do not search for Python in CMake setup. (Benjamin Bannier, Corelight)

    We previously needed this since during the build we were generating
    files with Python. This is not the case anymore.

1.11.0-dev.267 | 2024-07-22 10:50:22 +0200

  * GH-1462: Defer start of feature-dependent transformations to a later point. (Benjamin Bannier, Corelight)

    We previously would evaluate whether a unit field or method was required
    by an optional feature before we had finished collecting all feature
    requirements. This behavior was fine when we were visiting individual
    modules' AST one by one, but breaks with #1462 were we changed using a
    single AST to hold all modules.

    This patch defers transformations until all feature requirements have
    been collected.

1.11.0-dev.264 | 2024-07-22 09:27:15 +0200

  * Update developer's documentation. (Robin Sommer, Corelight)

    Includes:

    - Rework and update developer documentation's architecture diagram.
    This is now generated through Python's `diagrams` package (i.e.,
    graphviz). Because the layout is automatically determined by `dot`,
    it's not perfect; but seems good enough.

    - Update debugging streams.

    - Remove benchmarking section, as this was completely outdated.

1.11.0-dev.262 | 2024-07-15 11:49:46 +0200

  * Remove unused `Engine` type and tracking. (Benjamin Bannier, Corelight)

  * Remove API customization points for `spicy::Engine`. (Benjamin Bannier, Corelight)

  * Remove support for customizable Spicy hook and field engine. (Benjamin Bannier, Corelight)

1.11.0-dev.258 | 2024-07-12 16:34:16 +0200

  * GH-1784: Fix import HILTI vs Spicy confusion. (Robin Sommer, Corelight)

    Our tracking of module dependencies could end up mixing `.spicy`
    and `.hlt` modules when the former transitioned into the latter.

1.11.0-dev.255 | 2024-07-12 13:46:57 +0200

  * Update lists of releases in docs [skip CI]. (Benjamin Bannier, Corelight)

  * Bump latest release in docs [skip CI]. (Benjamin Bannier, Corelight)

  * Drop docs around `try`/`catch` [skip CI]. (Benjamin Bannier, Corelight)

    This feature is not on the horizon and having it documented prominently
    painted an incorrect picture for users. Drop its docs for now until we
    actually provide it.

1.11.0-dev.251 | 2024-07-09 18:16:15 +0200

  * Drop unused include. (Benjamin Bannier, Corelight)

  * GH-1785: Prevent defining already forwarded constants. (Benjamin Bannier, Corelight)

    We previously would add both forward declarations as well as full
    definitions for any constant in an imported module, i.e. we emitted
    the same constant in multiple C++ files which the linker seems to have
    cleaned up for us.

    With this path we only emit a definition if we are not already emitting
    a forward declaration.

    Closes #1785.

  * Remove duplicate emission of types. (Benjamin Bannier, Corelight)

    This was a noop but still useless work.

1.11.0-dev.247 | 2024-07-09 10:38:23 +0200

  * Add missing include. (Benjamin Bannier, Corelight)

1.11.0-dev.244 | 2024-07-09 09:23:01 +0200

  * Document spicy-driver's `@gap` for batch input. (Robin Sommer, Corelight)

1.11.0-dev.242 | 2024-07-04 12:19:21 +0200

  * Fix copy-n-paste error in GH action name. (Benjamin Bannier, Corelight)

  * Fix when GH `email-failure` action sends notification. (Benjamin Bannier, Corelight)

    In order to detect whether to send out a failure email we previously
    would check that the action conclusion contained certain statuses. This
    check seems to have been nonsensical as we only notified for benign
    failure reasons (the check would have made a little more sense if
    negated, but would have then also notified on successes).

    With this patch we instead explicitly enumerate the failure conclusions
    to trigger on, see
    https://docs.github.com/en/webhooks/webhook-events-and-payloads#check_suite--check_suite-object
    for the full list of conclusions.

1.11.0-dev.239 | 2024-07-04 12:13:40 +0200

  * Drop macos versions unsupported by Cirrus. (Benjamin Bannier, Corelight)

1.11.0-dev.237 | 2024-07-04 12:13:03 +0200

  * GH-1775: Extend validation of synchronization properties. (Robin Sommer, Corelight)

    Closes #1775.

1.11.0-dev.235 | 2024-07-04 12:09:41 +0200

  * GH-1777: Fix interning of regexps for `%skip*`. (Benjamin Bannier, Corelight)

    If we used regexps for `%skip`, `%skip-pre` or `%skip-post` we would
    previously compute unique identifiers in a way which clashed with
    general regexp interning elsewhere[^1].

    With this patch we now pick a unique stem for `%skip*`-related regexps
    so they do not clash anymore.

    [^1]: https://github.com/zeek/spicy/blob/90751715212c7268ebb86276d4365dfaedd94882/spicy/toolchain/src/compiler/codegen/parsers/literals.cc#L115

1.11.0-dev.233 | 2024-07-04 12:08:15 +0200

  * GH-1774: Fix synchronization with symbol different from last lookahead token. (Benjamin Bannier, Corelight)

    If `synchronize-[at|after]` used a lookahead symbol different from the
    last lookahead symbol (e.g., normal parsing used a
    literal, recovery a regexp), synchronization would clobber the lookahead
    symbol so that after successful synchronization normal parsing could
    still not succeed.

    This patch sets up dedicated parser state for synchronization to prevent
    the clobbering.

  * Use symbol unique for each error recovery. (Benjamin Bannier, Corelight)

    By using a constant symbol name we could run into situations where
    different error recoveries on different regexps (e.g., in unrelated
    units) could accidentally use the same symbol. This is due to regexps
    being automatically interned.

    With this patch we generate unique symbols for each position where we
    recover.

  * Add debug log of parser state during synchronization. (Benjamin Bannier, Corelight)

    We have a similar debug log already when synchronizing on regexps, but
    where missing it for other literals.

1.11.0-dev.229 | 2024-07-03 12:00:00 +0200

  * Call `%sync_advance` hook also when entering sync mode. (Benjamin Bannier, Corelight)

    This allows users to control the full synchronization lifecycle, from
    entering it, to regular checks during the search (both via
    `%sync_advance`, until it is left (via their explicit `confirm`, e.g.,
    from a `%synced` hook).

1.11.0-dev.227 | 2024-06-28 13:20:55 +0200

  * Improve doc wrt. `%error` hook in sync mode. (Jan Grashoefer, Corelight)

  * Add draft release notes for spicy-1.11.0. [skip CI] (Benjamin Bannier, Corelight)

1.11.0-dev.224 | 2024-06-25 08:50:58 +0200

  * GH-3779: Add `%sync_advance` hook. (Robin Sommer, Corelight)

    This adds support for a new unit hook:

    ```
    on %sync_advance(offset: uint64) {
        ...
    }
    ```

    This hook is called regularly (see below) during error recovery when
    synchronization skips over data or gaps while searching for a valid
    synchronization point. It can be used to check in on the
    synchronization to, e.g., abort further processing if it just keeps
    failing. `offset` is the current position inside the input stream that
    synchronization just skipped to.

    By default, "called regularly" means that it's called every 4KB of
    input skipped over while searching for a synchronization point. That
    value can be changed by setting a unit property
    `%sync-advance-block-size = <number of bytes>`.

    As an additional minor tweak, this also changes the name of what used
    to be the `__gap__` profiler to now be called `__sync_advance` because
    it's profiling the time spent in skipping data, not just gaps.

  * Add stream method `statistics()` to retrieve input statistics. (Robin Sommer, Corelight)

    This returns a struct of the following type, reflecting the input
    seen so far:

    ```
    type StreamStatistics = struct {
        num_data_bytes: uint64;     ## number of data bytes processed
        num_data_chunks: uint64;    ## number of data chunks processed, excluding empty chunks
        num_gap_bytes: uint64;      ## number of gap bytes processed
        num_gap_chunks: uint64;     ## number of gap chunks processed, excluding empty chunks
    };
    ```

  * Extend HILTI's stream implementation to track data statistics. (Robin Sommer, Corelight)

  * Add unit method `stream()` to access current input stream. (Robin Sommer, Corelight)

    We implement this by storing the current input stream inside the
    internal parse struct as a new field `__stream`. Using `stream()` then
    turns into an access to that field (i.e,  `*(self.__stream)`). If that
    field never gets accessed, the optimizer will remove it, so there's no
    overhead if nobody ever uses `stream()` with a unit.

1.11.0-dev.219 | 2024-06-20 10:32:09 +0200

  * Fix Spicy's support for `network` type. `network` was not
    recognized by the scanner. (Robin Sommer, Corelight)

1.11.0-dev.217 | 2024-06-19 09:29:16 +0200

  * Fix lints reported by clang-tidy-18 (Benjamin Bannier, Corelight)

1.11.0-dev.202 | 2024-06-18 12:31:36 +0200

  * GH-1763: Only allow creation of `const` variables from literals. (Benjamin Bannier, Corelight)

    Since it is hard to enforce that `const` variables are initialized in
    correct order (order of decls usually is not significant in Spicy/HILTI
    and works e.g., for `global`s, but this is not the case for `const`s due
    to codegen), this patch disallows creating `const` values from anything
    but literals. This completely removes the ordering issue.

1.11.0-dev.200 | 2024-06-14 10:56:04 +0200

  * GH-1759: Fix `if`-condition with `switch` parsing. (Robin Sommer, Corelight)

    The parser generator was ignoring `if` conditions attached to `switch`
    constructs. While we actually had a test for this already, turns out
    we had recorded a broken baseline. Plus, we were testing only one
    variant of `switch` (expression-based, not look-ahead-based). This
    implements and tests both variants now.

  * Fix clang-tidy. (Robin Sommer, Corelight)

1.11.0-dev.197 | 2024-06-13 12:43:33 +0200

  * GH-1750: Add `to_real` method to `bytes`. (Robin Sommer, Corelight)

    This interprets the data as representing an ASCII-encoded floating
    point number and converts that into a ``real``. The data can be in
    either decimal or hexadecimal format. If it cannot be parsed as
    either, throws an `InvalidValue` exception.

  * GH-1608: Add `get_optional` method to maps. (Robin Sommer, Corelight)

    This returns an optional either containing the map's element for the
    given key if that entry exists, or an unset optional if it does not.

  * Update some debug baselines. (Robin Sommer, Corelight)

1.11.0-dev.193 | 2024-06-13 12:10:22 +0200

  * GH-1760: Fix generated code for huge `const` collections. (Benjamin Bannier, Corelight)

1.11.0-dev.191 | 2024-06-13 09:59:22 +0200

  * GH-1598: Enforce that the argument `new` is either a type or a
    ctor. (Robin Sommer, Corelight)

    So far we allowed some more generic expressions as well, but it's hard
    for the parser to support arbitrary expressions here due to parsing
    ambiguities, leaving things inconsistent. So we now limit it to what
    was pretty much the intent originally anyways.

    Note that the error message for #1598 stays the same: it's not great,
    but seems good enough. However, we now actually disallow the
    workaround shown in the ticket as well for consistency. The new
    work-around is shown in the changes to `hilti.codegen.type-info`.

  * GH-90/GH-1733: Add `result` and `spicy::Error` types to Spicy to
    facilitate error handling. (Robin Sommer, Corelight)

    The `result` and `error` types were already implemented internally
    HILTI-side, but not yet available to Spicy users. This exposes them to
    Spicy as well. To avoid name clashes with existing code, we don't
    introduce `error` as a new type keyword, but instead make it available
    as a library type ``spicy::Error``.

    Typical usage is something like this:

    ```
    function foo() : result<int64> {
        ...
        if ( everything_is_ok )
            return 42;
        else
            return error"Something went wrong.";
    }

    if ( local x = foo() )
        print "result: %d " % *x;
    else
        print "error: %s " % x.error();
    ```

    The documentation has more specifics.

  * Support `result<void>` to HILTI. (Robin Sommer, Corelight)

    This allows to capture errors even if there's no actual result
    otherwise. Example (HILTI syntax):

    ```
    function result<void> x(bool b) {
         if ( b )
             return Null; # coerces to a successful result<void>
         else
             return error("trouble...");
     }

    assert x(True);
    assert x(False).error() == error("trouble...");

    ```

  * Add `==`/`!=` operators for HILTI `error` instances. (Robin
    Sommer, Corelight)

1.11.0-dev.185 | 2024-06-11 18:47:19 +0200

  * Bump centos-stream in CI. (Benjamin Bannier, Corelight)

1.11.0-dev.183 | 2024-06-07 10:35:53 +0200

  * GH-1745: Fix C++ initialization of global constants through global functions. (Robin Sommer, Corelight)

    The changes ordering of the emitted global declarations so that
    functions now come first, allowing them be used inside subsequent
    constant initializations.

1.11.0-dev.181 | 2024-06-04 10:18:25 +0200

  * Clean up includes. (Benjamin Bannier, Corelight)

  * Fix Flex linker error when building as part of Zeek. (Benjamin Bannier, Corelight)

    When building Spicy as part of Zeek against the Homebrew flex-2.6.4 I
    saw linker errors after f52325693aad8bd7931d51c4658027a8b7d7adae,

    ```
    ld: Undefined symbols:
      HiltiFlexLexer::LexerInput(char*, unsigned long), referenced from:
          vtable for hilti::detail::parser::Scanner in driver.cc.o
      HiltiFlexLexer::LexerOutput(char const*, unsigned long), referenced from:
          vtable for hilti::detail::parser::Scanner in driver.cc.o
    ```

    It is still not clear to me how this error comes about, but the change in
    this patch seems to address the issue.

1.11.0-dev.178 | 2024-06-03 09:50:53 +0200

  * Fix a typo in packing.rst (Tanner Kvarfordt)

    Fix a typo in packing.rst where the template argument to unpack was (u)int
    instead of real.

  * Expand guidelines on improving compilation performance. [skip CI] (Benjamin Bannier, Corelight)

  * Fix documented type mapping for integers. [skip CI] (Benjamin Bannier, Corelight)

1.11.0-dev.172 | 2024-05-17 12:51:34 +0200

  * GH-1742: Unroll ctrs of big containers. (Benjamin Bannier, Corelight)

    When generating C++ code for container ctrs we previously would directly
    invoke the respective C++ ctrs taking an initializer list. For very big
    initializer lists this causes very bad C++ compiler performance, e.g.,
    compiling code constructing a vector with 10,000 elements could take
    minutes.

    With this patch we unroll such ctrs calls by calling a dedicated
    initialization function. For huge containers this causes creating of big
    functions instead of big initializer lists, but compiling functions
    seems to behave more predictively.

    Closes #1742.

1.11.0-dev.170 | 2024-05-17 12:51:06 +0200

  * GH-1743: Use a checked cast for `map`'s `in` operator. (Benjamin Bannier, Corelight)

1.11.0-dev.168 | 2024-05-15 10:19:54 +0200

  * Fix behavior for unset accept and decline hooks. (Benjamin Bannier, Corelight)

    We did not properly initialize pointer values for accept and decline
    hooks. In downstream code they then appeared to be set when they were in
    fact not.

    With this patch we initialize them with proper defaults.

1.11.0-dev.166 | 2024-05-14 11:54:16 +0200

  * Docs: Add new section with guidelines and best practices. (Robin Sommer, Corelight)

    This focuses on performance for now, but may be extended with other
    areas alter.

    Much of the content was contributed by Corelight Labs.

  * Docs: Update Custom Extensions section. (Robin Sommer, Corelight)

    The usage of `-P` wasn't up to date.

  * Docs: Update feedback section. (Robin Sommer, Corelight)

1.11.0-dev.162 | 2024-05-13 14:42:08 +0200

  * Update types.rst (Smoot)

1.11.0-dev.160 | 2024-05-13 11:45:04 +0200

  * GH-1657: Update Spicy runtime driver to use new stream features for improved performance. (Robin Sommer, Corelight)

    This does two things:

    - When adding data to a stream, we now do that without copying
      anything initially. For block input (e.g., UDP) that's always fine
      because the parser will never suspend before it's fully done
      parsing; hence we can safely delete it once the parser returns. For
      stream input (e.g., TCP), we make the stream own its data later
      if (and only if) the parser suspends.

    - For block input (e.g., UDP) we now keep reusing the same stream for
      subsequent blocks, instead of creating a new one each time. This
      allows the stream to reuse an allocated chunk that it may have still
      cached internally.

    The result of this, plus the new chunk caching introduced earlier, is
    that for a UDP flow, we never need to allocate more than one chunk,
    and never need to copy any data; and for TCP it's the same as long as
    parsing consumes all data before suspending (which should be a common
    case), plus, when we allocate new storage we only copy data that didn't
    get trimmed immediately anyways.

  * Give stream a method to reset it into freshly initialized state. (Robin Sommer, Corelight)

    This does not clear the internal chunk cache.

  * Cache previously trimmed chunks inside stream for reuse. (Robin Sommer, Corelight)

    A chain now retains one previously used but no longer needed chunk for
    reuse, so that we can avoid constant cycles of creating/destructing
    chunks (and their paylaod memory) in the common case of a parser
    consuming full chunks without yielding. The caching is also geared
    towards owning/non-owning semantics staying consistent across
    subsequent append operations.

  * Extend stream API to allow for chunks that don't own their data. (Robin Sommer, Corelight)

    By default, we still copy data when creating chunks but we add a
    parallel API that just stores pointers, assuming the data will stay
    around as long as needed. If the stream owner cannot guarantee that,
    they may at any point convert all not-owned data into owned data
    through a corresponding `makeOwning()` stream method. To make that
    method efficient even with long chains of chunks, we internally
    maintain an invariant that only the last chunk of chain can be
    non-owning: whenever we add a new chunk to a chain, we ensure that the
    previous tail become owning at that point. In other words, we amortize
    the work across all newly added chunks.

  * Remove `std::functional` from `DeferredExpression`. (Robin Sommer)

  * Revert "Remove support for deferred expressions." (Robin Sommer)
    Turns out this is actually still being used by the Zeek
    integration.

1.11.0-dev.151 | 2024-05-10 16:35:50 +0200

  * Remove unused include headers. (Robin Sommer, Corelight)

  * Remove all usage of `std::function` from toolchain. (Robin Sommer, Corelight)

  * Remove usage of `std::function` from parser and sink runtime representations. (Robin Sommer, Corelight)

    We switch to raw function pointers, which is easy enough.

  * Remove use of `std::function` from `spicy::rt::Configuration`. (Robin Sommer, Corelight)

    Callbacks are now classic function pointers.

  * Remove unused functional header. (Robin Sommer, Corelight)

  * Remove use of `std::function` from runtime vector class. (Robin Sommer, Corelight)

  * Remove support for deferred expressions. (Robin Sommer, Corelight)

    These weren't used anymore anywhere so we can remove the corresponding
    code from toolchain and runtime.

1.11.0-dev.142 | 2024-05-06 15:14:40 +0200

  * GH-1664: Fix `&convert` typing issue with bit ranges. (Robin Sommer, Corelight)

    Turns out #1664 was only indirectly related to the `&convert` itself;
    the real issue was that we couldn't assign one bitfield struct to
    another if their field types didn't match exactly, even in cases where
    at the C++ level there was no meaningful difference. In this case we
    ended up with a field that had a C++ type `rt::Bool` in one type and
    `bool` in another, leading to errors when assigning the latter to the
    former. We now allow to creating instances of the former from the
    latter through standard C++ type conversions on a per field basis.

  * Suppress new `clang-tidy` warnings. (Robin Sommer, Corelight)

  * Fix a Spicy scoping issue across imports. (Robin Sommer)

    We could get a bogus "unknown ID" error for default arguments of
    functions defined in an imported module if that default argument was
    itself referring to an identifier inside yet another imported module.
    The test case shows the exact situation that was broken.

1.11.0-dev.137 | 2024-04-25 13:38:42 +0200

  * Remove Spicy parser support for unsupported `&priority` attribute. (Benjamin Bannier, Corelight)

  * Make spelling of hook `priority` consistent across Spicy and HILTI. (Benjamin Bannier, Corelight)

    We were naming the priority attribute differently in Spicy (`priority`)
    and HILTI (`&priority`). While e.g., a Spicy `Hook` could correct
    extract its priority, this still could have lead to potential issues if
    we were attempting to access the priority of a Spicy hook from HILTI as
    we do not perform any adjustment of this attribute when lowering to
    HILTI. This distinction also made it hard to generate intended code from
    the outside using our API (e.g., from Zeek) since one needed to be aware
    at which level the attribute was injected (Spicy or HILTI).

    With this patch we internally translate a Spicy `priority` attribute to
    `&attribute` syntax.

1.11.0-dev.134 | 2024-04-25 13:36:59 +0200

  * Fix incremental skipping. (Benjamin Bannier, Corelight)

    We previously would incorrectly compute the amount of data to skip which
    could have potentially lead to the parser consuming more data than
    available. With this patch we correctly use the actually consumed amount
    to compute what the trim from the input.

  * GH-1724: Fix skipping in size-constrained units. (Benjamin Bannier, Corelight)

    We previously could skip too much data if `skip` was used in a unit with
    a global `&size`. This was due to the machinery moving the input forward
    believing that `skip` units were that really field productions. While
    this is true in a sense in this particular case it still lead to
    incorrect behavior, in particular as far is input handling is concerned
    `skip` productions largely behave like any other unit.

1.11.0-dev.131 | 2024-04-25 09:27:48 +0200

  * Fix potential internal error in port string conversion. (Robin Sommer)

  * GH-1284: GH-1693: Promote use of `-x` over `-c`, clean up `-P`. (Robin Sommer)

    This includes these pieces:

    - `-P` now requires a prefix argument that set's the C++ namespace, so
      that generated prototypes match that of `-x`. Like with `-x`, the
      prefix may be empty (`-P ""`) to get back to the old `hlt::`.

    - For both `-P` and `-x` the prefix now must be a valid C++ identifier,
      because we use it as such.

    - `spicyc` usage message now refers to `-c` and `-l` as for debugging
      use.

    - Update documentation on host applications, switching to using `-x`
      instead of `-{cl}`.

    - Scanned for other use of `-c` and `-l` in the docs as well, the
      remaining ones seem fine.

  * Remove generated, in-code linker JSON meta data. (Robin Sommer)

    This was originally to allow for compiling multiple Spicy modules
    separately, with the meta data providing what's the necessary to add
    any cross-module functionality. However, we've moved away from that
    approach and now already require the compiler to always see all code,
    so this is no longer needed/possible.

1.11.0-dev.125 | 2024-04-18 15:53:48 +0200

  * GH-1501: Improve some error messages for runtime parse errors. (Robin Sommer, Corelight)

  * GH-1586: Make skip productions behave like the production they are wrapping. (Robin Sommer, Corelight)

  * GH-1719: Fix `new` passing a unit reference to an `inout` unit parameter. (Robin Sommer, Corelight)

    One might debate whether this is something we should allow at all but
    we do permit it elsewhere when passing unit parameters, and changing
    that would probably break code, so for consistency this allows it for
    `new` as well.

    Internally, there was a more general inconsistency introduced by
    automatic derefs of Spicy-level strong references. While we do need
    that deref to happen for operator resolution, we now remove it after
    generation of HILTI code so that HILTI-level resolution can then work
    as expected on the value references introduced for units. I wouldn't
    be surprised if the prior behavior was causing more trouble than just
    #1719 and we just hadn't run into yet it.

  * Mark automatic derefs inside the operator's AST node. (Robin Sommer, Corelight)

    This allows to differentiate between explicit `deref` operations part
    of the source code and implicit `deref` operations inserted by the
    coercer. A subsequent commit will leverage this information, but I've
    been meaning to do this anyways because it could be hard to track
    where a particular `deref` was coming from.

  * Unify code generation for `new`. (Robin Sommer, Corelight)

    There's no functional change (afaict) but for consistency of
    implementation and results, `new` should go through
    `compileCallArguments()`.

  * GH-1655: Reject joint usage of filters and look-ahead. (Robin Sommer, Corelight)

    We cannot retrieve look-ahead information from units to which a filter
    is connected, because parsing for outer and inner layers will be
    operating on separate streams. While we could pass the lahead token
    number upwards, we would also need the stream position for the end of
    the lahead symbol, but we cannot tie an iterator on the filtered
    stream back to a position on the original stream. So we now reject
    this.

    This shouldn't cause any actual backwards-incompatibility because this
    wasn't working in the first place: it  would reliably abort with an
    invalid stream iterator exception during parsing.

  * Add test confirming that `&parse-{at,from}` don't interfere with outer look-ahead parsing. (Robin Sommer, Corelight)

    This makes sure fields with these attributes are ignored for
    look-ahead computation of the unit containing them. We actually have a
    related test for `&parse-from` already, but can't hurt to have it
    covered further here.

1.11.0-dev.114 | 2024-04-16 09:38:38 +0200

  * Bump baselines. (Benjamin Bannier, Corelight)

1.11.0-dev.112 | 2024-04-15 17:22:55 +0200

  * Update NEWS with a list of incompatibilities compared to previous version. (Robin Sommer)

  * Pretty print reference types in Spicy output. (Robin Sommer)

    We now render `strong_ref<T>` as `T&`, as one would expect on
    the Spicy side.

  * Do not perform automatic deref on RHS of an assignment. (Robin Sommer)

    This used to be accepted but had not the intended effect at runtime:

    ```
    function f(s: string&) {
         *s = new "xxx";
    }
    ```

  * Fix line numbers. (Robin Sommer)

    Line numbers could be off in the presence of comments including '#'
    characters.

  * Fix auto-deref of LHS references. (Robin Sommer)

    We were checking the constness of the reference, not of the wrapped
    value.

  * Improve error message. (Robin Sommer)

1.11.0-dev.105 | 2024-04-10 13:24:03 +0200

  * Add missing file to git. (Robin Sommer, Corelight)

  * Document generic operators. (Robin Sommer, Corelight)

  * Refactor output logic in `spicy-doc-to-rst`. (Robin Sommer, Corelight)

  * GH-1711: Fix forwarding of a reference unit parameter to a non-reference parameter. (Robin Sommer, Corelight)

  * GH-1599: Fix integer increment/decrement operators require mutable arguments. (Robin Sommer, Corelight)

  * Add tests checking sinks as unit parameters. (Robin Sommer, Corelight)

  * GH-1710: Improve implementation of sink type. (Robin Sommer, Corelight)

    So far we lowered Spicy's `sink` type into a `strong_ref` a HILTI
    codegen time, meaning that a unit's `sink` field would have type
    `sink` at the Spicy-level and then later `strong_ref<sink>` inside
    HILTI (actually: `strong_ref<spicy_rt::Sink>`). This approach led some
    inconsistencies because of the mismatch between the two levels, and it
    also made the implementation more complex than necessary. We now let
    `sink` fields simply have type `sink&` (i.e., `strong_ref<sink>`) in
    Spicy; everything else then falls in place more easily. From a user
    perspective, the change should remain largely invisible.

  * Fix internal assertion potentially triggering erroneously during retrieval of parent node. (Robin Sommer, Corelight)

  * GH-1618: Fix and clarify usage of references in Spicy. (Robin Sommer, Corelight)

    References weren't fully consistent in their properties and
    implementation, and they weren't documented either. This commit cleans
    that up and adds documentation.

    Changes:

    - Constness 1: For a type `T&`, which internally is `strong_ref<T>`,
      we now consistently make the inner `T` a mutable LHS type. That
      seems most natural and useful from a user perspective. In
      particular, this allows mutating objects passed into functions as
      parameters without declaring them `inout`: (`foo (x: bytes&)`). This
      is important because declaring the parameter `inout` makes the
      reference itself mutable (not the contained object), which isn't
      what one wants. Plus, the latter doesn't work for unit parameters
      anyways. Accordingly, we also adapt our built-in operators to
      use`T&` instead of `inout T&`.

    - Constness 2: For a type `T&`, we now make the outer reference type
      constant and non-mutable. This is mostly for consistency, it doesn't
      really change anything as we didn't havemutating operations on
      references anyways.

    - Feature: We add `new BASIC_TYPE` as syntax for creating default
      initialized values of basic types. So far we only had `new
      NAMED_TYPE` and `new VALUE`.

    Technical note to the reviewer: the changes to the `recreateAs*`
    methods are necessary to avoid complex types being copied into the
    newly created type, which would lead to name resolution problems. This
    is tested through existing tests which fail otherwise
    (`spicy.types.sink.filter-it.spicy`,
    `spicy.rt.base64-filter-eod.spicy`). (Well, `recreateAsLHS()` is
    tested that way, but the others would have the same issue.) This is a
    good change anyways, because it keeps the AST smaller.

  * GH-1515: Catch unsupported types for unit inout parameters. (Robin Sommer, Corelight)

    We support only types that are (internally) passed around as
    references.

  * GH-1583: Disallow coercion when passing arguments to `inout` parameters. (Robin Sommer, Corelight)

1.11.0-dev.92 | 2024-04-10 09:40:32 +0200

  * Update release version in documentation. (Robin Sommer, Corelight)

    This hadn't been updated, leading to outdated links.

  * Do not require all AST nodes to be destroyed before we begin compilation. (Robin Sommer)

    If a 3rdparty (like Zeek) still retains a pointer to a `Node`, the
    check would trigger, making usage awkward. Instead we now just
    check at context destruction time that we no longer have any live
    nodes.

1.11.0-dev.88 | 2024-04-09 09:33:03 +0200

  * Bump 3rdparty/utf8proc from `1fe43f5` to `894e810` (dependabot[bot])

1.11.0-dev.86 | 2024-04-04 15:51:17 +0200

  * Overhaul AST node memory management. (Robin Sommer)

    We switch back to reference counting nodes, but through a custom
    scheme that allows us to continue passing around raw pointers most
    of the time.

  * Remove unused class. (Robin Sommer)

  * Reduce memory usage for operators with complex argument types. (Robin Sommer)

    We now use external types for to store operands of name types, which
    reduces the number of AST nodes created for operators substantially.

  * Remove meta/location information from AST IDs. (Robin Sommer)

    Turns out these aren't used anywhere.

  * Use less memory for storing context IDs. (Robin Sommer)

    32 bits should still be plenty. Because these are stored in many
    nodes, the change is noticeable in overall memory consumption.

  * Compute an ID's internal views on demand. (Robin Sommer)

    We had previously optimized IDs by pre-computing some information for
    quick access. However, for most IDs that information isn't actually
    accessed at all, so we now compute it only on demand the first time
    it's needed. This saves both CPU and memory.

  * Move inherit-scope information into virtual method. (Robin Sommer)

    This saves space, and is also more aligned with how other information is
    managed as well.

  * Change storage for AstContext. (Robin Sommer)

    No need for `shared_ptr`, now using a `unique_ptr`.

  * Do not auto-allocate storage for errors with each AST node. (Robin Sommer)

    We now create the vector for error messages only when needed.

  * Reuse `Meta` instances across nodes. (Robin Sommer)

    We now store each `Meta` value only once globally. Saves about 15%
    memory.

1.11.0-dev.75 | 2024-04-03 11:55:16 +0200

  * Remove a few left-over unused variables. (Benjamin Bannier, Corelight)

  * Fix a few instances where codegen was non-deterministic. (Benjamin Bannier, Corelight)

1.11.0-dev.72 | 2024-04-03 09:35:23 +0200

  * Fix repeated evaluations of `&parse-at` expression. (Robin Sommer)

  * GH-1316: GH-1635: Provide better error messages for some cases of unknown unit IDs. (Robin Sommer, Corelight)

  * GH-1493: Support/fix public type aliases to units. (Robin Sommer, Corelight)

    An alias like `public type Unit1 = Unit2` used to lead to C++-side
    compiler errors, which this fixes. We also fully support this now by
    making both `Unit1` and `Unit2` available for parsing to host
    applications. Internally, the `Unit1` parser is just a small facade
    pointing to the parsing functions for `Unit2`.

  * GH-1661: Deprecate usage of `&convert` with `&chunked`. (Robin Sommer, Corelight)

    Per discussion in #1661, this combination can lead to confusion and
    can be worked around if really needed.

1.11.0-dev.64 | 2024-04-02 17:44:13 +0200

  * Fix GCC false positive around `strncpy` use. (Benjamin Bannier, Corelight)

1.11.0-dev.62 | 2024-03-25 10:39:42 +0100

  * Reimplement `IDBase` for better performance. (Robin Sommer)

    Our codegen phase had quite some overhead due to repeated ID
    operations recomputing the same information over and over again (e.g.,
    subpaths and namespaces). This reimplements the class to compute
    everything once upfront, cutting codegen time into half.

  * Add more tests for `IDBase`. (Robin Sommer)

    This includes a tiny semantic change for `IDBase::length()`: it now
    returns zero for an empty ID, which seems more consistent. (This
    doesn't seem to have an impact anywhere, all tests pass.)

1.11.0-dev.59 | 2024-03-21 12:35:06 +0100

  * Rework order of C++ codegen. (Robin Sommer)

    We used to potentially codegen modules multiple times. Now we cache a
    module's generated C++ code inside the AST node the first time we
    create it. From there, we can then easily reuse it later, in
    particular when needing to import its declarations into another
    module. Internally, we switch storage for `cxx::Unit` to shared
    pointers, and tweak the debug logging a bit for better readability in
    this new model.

  * Cleanup: Remove flag to compile implementation from higher-level codegen method. (Robin Sommer)

    Pushing down the condition that we used to pass in, to lower-level
    code.

  * Fix duplicates in module dependency tracking. (Robin Sommer)

    We now use a set instead of a vector to unique dependencies
    automatically.

  * Fix file name case for fuzzer builds. (Benjamin Bannier, Corelight)

1.11.0-dev.54 | 2024-03-18 09:56:58 +0100

  * Refresh CI platforms. (Benjamin Bannier, Corelight)

1.11.0-dev.50 | 2024-03-16 08:55:11 +0100

  * Rework memory management for AST nodes. (Robin Sommer, Corelight)

    We switch memory management of AST nodes to an arena/bump allocator
    that releases them as a whole once the AST gets destroyed, not
    individually/continiously through their own life-time scoping. This
    then allows us to also switch them from `shared_ptr` to raw pointers
    throughout the AST code. The result is a compiler speed up of about
    20% for some complex analyzer.

  * Centralize the `vector` type we use for storing AST nodes. (Robin Sommer, Corelight)

    No functional change, this just unifies the various vectors of
    `Node`-derived classes so that there's a central place where to define
    the underlying vector type. For now this is just for easier
    maintenance. In the future we could experiment with different vector
    or allocator implementations.

  * Make node tags `constexpr`. (Robin Sommer, Corelight)

1.11.0-dev.45 | 2024-03-15 15:11:26 +0100

  * Fix broken f-string. (Benjamin Bannier, Corelight)

  * Remove Moneterey Homebrew CI tasks. (Benjamin Bannier, Corelight)

  * Add Cirrus tasks running non-Homebrew macos builds and tests. (Benjamin Bannier, Corelight)

1.11.0-dev.41 | 2024-03-15 12:47:42 +0100

  * Fix fuzzer builds for reworked AST. (Benjamin Bannier, Corelight)

1.11.0-dev.39 | 2024-03-13 09:07:42 +0100

  * Bump softprops/action-gh-release from 1 to 2 (dependabot[bot])

  * Bump typos pre-commit hook. (Benjamin Bannier, Corelight)

  * Bump clang-format. (Benjamin Bannier, Corelight)

  * Modernize Python scripts with `pyupgrade`. (Benjamin Bannier, Corelight)

  * Reformat Python with ruff-format. (Benjamin Bannier, Corelight)

  * Fix generation of doc example code. (Benjamin Bannier, Corelight)

  * Fix Python lints diagnosed by ruff. (Benjamin Bannier, Corelight)

1.11.0-dev.30 | 2024-03-12 12:02:43 +0100

  * Get rid of more dynamic casts. (Robin Sommer, Corelight)

    The main use of dynamic casts that's left now (other than for
    debugging code) is inside the grammar's production hierarchy. That
    logic isn't standing out during profiling, so seems fine to leave for
    now.

  * Introduce custom RTTI system for casting safely between node
    types. (Robin Sommer, Corelight)

    Our custom system is faster than C++'s `dynamic_cast<>`, resulting in
    a noticeable speed-up for larger parsers.

    We assign a unique integer tag to  each `Node`-derived class. Each
    class' constructors pass an array through to the top-level `Node`
    class that contains a series of these integers describing the
    inheritance path from the derived class back up to `Node`. The `Node`
    class stores this path for fast type checks. In addition, each
    `Node`-derived class `T` gets a couple constants as members: (1) a
    copy of its own tag (`T::NodeTag`), and (2) a level indicating its
    distance from `Node` in the inheritance tree (`T::NodeLevel`). The
    level is used as index into an instance's tag array when performing
    type checks. The result is that we can do `isA<T>` operation with just
    a single comparison between an array element and a constant value.

    This system is optimized for our very simple Node hierarchy: not very
    deep (max. 4 entries in the tag array), single-inheritance only,
    derived class are always the same distance from `Node`, and
    all `Node`-derived classes are known upfront.

1.11.0-dev.27 | 2024-03-06 17:06:09 +0100

  * Speed up `util::toIdentifier`. (Robin Sommer)

    Turns out this is called a lot. This changes baselines due to slight
    differences in generated IDs (changes are good because they better avoid
    potential conflicts).

    This also removes the option to ensure non-keyword IDs: I don't
    think we need that because we check for that when we generate C++
    code.

  * GH-1675: Extend runtime profiling to measure parser input volume. (Robin Sommer, Corelight)

    With `--enable-profiling` the output for Spicy units/fields now
    includes a new `volume` column, like this:

    ```
    #name                                                   count       time      avg-%    total-%          volume
    [...]
    spicy/unit/test::A                                          1     285500      43.96      43.96               8
    spicy/unit/test::A/__gap__                                  4       3167       0.12       0.49               0
    spicy/unit/test::A/__synchronize__                          1      35500       5.47       5.47               4
    spicy/unit/test::A::a                                       1      74833      11.52      11.52               -
    spicy/unit/test::A::b                                       1      15333       2.36       2.36               1
    spicy/unit/test::A::c                                       1      19125       2.94       2.94               1
    spicy/unit/test::A::d                                       1       7583       1.17       1.17               1
    spicy/unit/test::A::e                                       1       8042       1.24       1.24               1
    ```

    Three different things here:

    - The `volume` column for `spicy/unit/TYPE` and
      `spicy/unit/TYPE::FIELD` augments the already existing timing
      measurement and reports the total, aggregate number of bytes that
      this unit/field got to parse over the course of the processing.

    - For units going into synchronization mode, there are now additional
      rows `spicy/unit/TYPE/__synchronize__` that report both CPU time and
      volume spent in synchronization while processing that unit.

    - For units encountering input gaps during synchronization, there are
      now additional rows 'spicy/unit/TYPE/__gap__` that report total
      aggregate gap size encountered while processing the unit.

    All the volume measurements are taken as differences of two offsets
    inside the input stream. For normal unit/field parsing, we subtract
    the final offset after parsing an instance from the initial offset
    where its parsing started.[1] For synchronization, it's the offset
    where synchronization stopped successfully minus where it started.[2]
    For gaps, it's the offset where we continued after the gap minus where
    the gap started.[3] All these differences are then added up for each
    row over the course of total input stream processing.

    Note that volume isn't counted if parsing for some reason never
    reaches the point where the end measurement would be taken (e.g., a
    parser error prevents it from being reached; in the output above
    that's the case for `spicy/unit/test::A::a`).

    Closes #1675.

    [1] This *includes any ranges that the unit spent in synchronization
    mode trying to recover from parse errors.

    [2] This does *not* include any gaps encountered because they don't
    affect stream offsets.

    [3] Little glitch: these values can currently by off by one due to some
    internal ambiguity.

  * Move inferring of a unit's context type into the resolver. (Robin Sommer)

    No functional change here, just cleanup: this is where the logic
    belongs.

  * Infer constness of `<unit>.context()` from that of `<unit>`. (Robin Sommer)

    Closes https://github.com/corelight/zeek-spicy-openvpn/issues/11.

  * Ensure constness of result for const map/vector index operator. (Robin Sommer)

    Not sure this actually changes anything, but it ensures they are correct.

  * Simplify constness check for struct field assignments. (Robin Sommer)

    Same treatment as for unit fields.

  * Bump 3rdparty/filesystem from `2fc4b46` to `42ea4fc` (dependabot[bot])

1.11.0-dev.15 | 2024-03-05 10:43:27 +0100

  * Bump 3rdparty/filesystem from `2fc4b46` to `42ea4fc` (dependabot[bot])

1.11.0-dev.11 | 2024-03-04 10:07:25 +0100

  * New AST architecture. (Robin Sommer)

    This is a large revamp of compiler internals, cleaning up and speeding
    up lots of the AST pipeline. From a user perspective, nothing changes,
    except that the new compiler is a tiny bit more strict: turns out that
    in rare cases the old compiler ended up accepting some ill-formed
    Spicy code that is now (rightfully) rejected. Specifically, two
    instances of this are known where existing Spicy code may need
    tweaking now:

    - Identifiers from the (internal) `hilti::` namespace are no longer
      accessible. Usually you can just scope them with `spicy::` instead,
      and it'll work.

    - The old compiler didn't always enforce constness as it should have.
      In particular, function parameters could end up being mutable even
      when they weren't declared as `inout`. Now `inout` is required for
      supporting any mutable operations on a parameter, so make sure to
      add it where needed.

1.11.0-dev.9 | 2024-03-01 18:10:49 +0100

  * Simplify `while` loops over constant conditions. (Benjamin Bannier, Corelight)

    While loops with constant conditions are unlikely to occur in code
    generated by us, but could appear in user code. Cleaning them up is
    simple and cheap, so this patch implements a pass which simplifies them.

1.11.0-dev.7 | 2024-02-27 17:44:22 +0100

  * GH-1624: Enable optimizations when running `spicy-build`. (Benjamin Bannier, Corelight)

    We previously would emit single C++ source files from `spicy-build`.
    This made it impossible to enable optimizations since we could not know
    whether individual code was used. Since optional features are by default
    enabled and only disabled through an optimizer pass this meant that
    `spicy-build` emitted code which performed a lot of work not done
    usually.

    This patch reworks `spicy-build` to use `spicyc`'s `-x` instead which
    has a global view and can run optimizations, and emits all source files
    into a prefix.

    Closes #1624.
    Closes #1625.
    Closes #1622.

  * Fix some shellcheck warnings in `spicy-build`. (Benjamin Bannier, Corelight)

  * Fix handling of `spicyc -x` when passed a filename. (Benjamin Bannier, Corelight)

    For the `-x` flag users can pass either the name of a directory or a
    file. If passed a filename the file name is incorporated into the
    identifier generated namespace identifier of e.g., the form
    `__hlt_FILENAME` in the emitted C++ files. Previously when passing a
    directory we would still emit a `_` separator, but since there was no
    filename nothing after it so we generated namespace identifiers like
    `__hlt_`.

    With this patch we now only emit the separator if passed a filename.

  * Do not optimize out public functions. (Benjamin Bannier, Corelight)

    Optimizing out public functions breaks the assumption we have elsewhere
    that something marked `public` is part of the public API and would not
    get optimized out. This is e.g., the behavior we have for units.

1.11.0-dev.2 | 2024-02-26 12:16:17 +0100

  * Fix stray Python escape sequence. (Benjamin Bannier, Corelight)

1.10.0-dev.149 | 2024-02-22 09:48:01 +0100

  * GH-1585: Put closing of unit sinks behind feature guard. (Benjamin Bannier, Corelight)

    This code gets emitted, regardless of whether a sink was actually
    connected or not. Put it behind a feature guard so it does not enable
    the feature on its own.

    Closes #1585.

1.10.0-dev.147 | 2024-02-21 10:30:22 +0100

  * GH-1667: Always advance input before attempting resynchronization. (Benjamin Bannier, Corelight)

    When we enter resynchronization after hitting a parse error we
    previously would have left the input alone, even though we know it fails
    to parse. We then relied fully on resynchronization to advance the
    input.

    While this just pushed work downstream when synchronizing on literals,
    it could cause us loosing input if synchronizing on regular expressions
    if we happened to fail parsing due to a gap which is now at the front of
    the input (parse errors from gaps are the most likely resynchronization
    scenario when parsing genuine traffic); in this case the regular
    expression would synchronize at the second byte after the input and we
    would synchronize only at a later position.

    With this patch we always forcibly advance the input to the next non-gap
    position. This has no effect for synchronization on literals, but allows
    it to happen earlier for regular expressions.

    Closes #1667.

  * Refactor test `spicy.types.unit.synchronize-on-gap`. (Benjamin Bannier, Corelight)

    This refactoring cleans up how we feed gaps into the parser to testing
    with more inputs simpler.

1.10.0-dev.144 | 2024-02-14 15:55:35 +0100

  * GH-1652: Fix filters consuming too much data. (Benjamin Bannier, Corelight)

    We would previously assume that a filter would consume all available
    data. This only holds if the filter is attached to a top-level unit, but
    in general not if some sub-unit uses a filter. With this patch we
    explicitly compute how much data is consumed.

    Closes #1652.

1.10.0-dev.142 | 2024-02-08 17:00:53 +0100

  * GH-1668: Fix incorrect data consumption for `&max-size`. (Benjamin Bannier, Corelight)

    We would previously handle `&size` and `&max-size` almost identical
    with the only difference that `&max-size` sets up a slightly larger view
    to accommodate a sentinel. In particular, we also used identical code to
    set up the position where parsing should resume after such a field.

    This was incorrect as it is in general impossible to tell where parsing
    continues after a field with `&max-size` since it does not signify a
    fixed view like `&size`. In this patch we now compute the next position
    for a `&max-size` field by inspecting the limited view to detect how
    much data was extracted.

    Closes #1668.

1.10.0-dev.140 | 2024-02-08 13:22:52 +0100

  * GH-1522: Drop overzealous validator. (Benjamin Bannier, Corelight)

    This validator was intended to reject incorrect parsing of vectors but instead
    ending up rejecting all vector parsing if the vector elements itself produced
    vectors. Since this code has no test and it seems to have no clear purpose this
    patch drops this validation.

    Closes #1522.

1.10.0-dev.138 | 2024-02-08 13:20:31 +0100

  * Remove now unused testing `random.seed`. (Benjamin Bannier, Corelight)

  * GH-1659: Lift requirement that `bytes` forwarded from filter be mutable. (Benjamin Bannier, Corelight)

    Since we did not declare the `bytes` argument to `unit::forward`
    constant it was implicitly mutable so that it was impossible to e.g.,
    pass literal bytes. This patch uses the originally intended type.

    Closes #1659.

1.10.0-dev.135 | 2024-01-30 12:29:28 +0100

  * GH-1665: Fix running of `codebase` tests. (Benjamin Bannier, Corelight)

1.10.0-dev.133 | 2024-01-29 10:59:23 +0100

  * GH-1489: Deprecate &bit-order on bit ranges. (Arne Welzel, Corelight)

    This does not appear to have any effect and allowing it may be
    confusing to users. Deprecate it with the idea of eventual
    removal.

  * Add extensive bitfield test including endianness behavior. (Arne Welzel, Corelight)

  * Add bitfield examples. (Arne Welzel, Corelight)

1.10.0-dev.129 | 2024-01-23 17:48:19 +0100

  * Adjust end of Bison-generated locations. (Robin Sommer, Corelight)

    The Bison-side end column points one beyond the element, which isn't
    that nice in error messages, so we adjust them now.

  * Extend location printing to include single-line ranges. (Robin Sommer, Corelight)

    For a location of, e.g., "line 1, column 5 to 10", we now print
    `1:5-10`, whereas we used to print it as only `1:5`, hence dropping
    information.

  * Fix Bison locations. (Robin Sommer, Corelight)

    If a Bison rule started with an optional element, its location would
    start at the end of the *previous* token if that optional element
    wasn't present. We now skip token locations that are zero-sized at the
    beginning of a rule.

1.10.0-dev.125 | 2024-01-23 16:27:43 +0100

  * Bump 3rdparty/any from `e88b1bf` to `7c76129`

  * Fix incorrect check_suite type in GH actions [skip CI]. (Benjamin Bannier, Corelight)

1.10.0-dev.122 | 2024-01-18 08:39:05 +0100

  * Update documentation of `offset()`. (Benjamin Bannier, Corelight)

    The originally documented caveats do not apply anymore, so remove them.

  * Always store a valid begin iterator in units. (Benjamin Bannier, Corelight)

  * Always pass a valid begin iterator to parse methods. (Benjamin Bannier, Corelight)

  * GH-1648: Provide meaningful unit `__begin` value when parsing starts. (Benjamin Bannier, Corelight)

    We previously would not provide `__begin` when starting the initial
    parse. This meant that e.g., `offset()` was not usable if nothing ever
    got parsed.

    With this patch we provide a meaningful value now.

    Closes #1648.

1.10.0-dev.117 | 2024-01-17 16:43:22 +0100

  * GH-1640: Implement skipping for any field with known size. (Benjamin Bannier, Corelight)

    This patch adds `skip` support for fields with `&size` attribute or of
    builtin type with known size. If a unit has a known size and it is
    specified in a `&size` attribute this also allows to skip over unit
    fields.

  * Add method to field to optionally return its size. (Benjamin Bannier, Corelight)

  * Fix skipping of literal fields with condition. (Benjamin Bannier, Corelight)

1.9.0-111 | 2024-01-11 11:55:07 +0100

  * GH-1645: Fix `&size` check. (Robin Sommer, Corelight)

    The current parsing offset could legitimately end up just beyond the
    `&size` amount.

1.9.0-109 | 2024-01-11 09:00:08 +0100

  * GH-1634: Fix infinite loop in regular expression parsing. (Robin Sommer, Corelight)

1.9.0-107 | 2024-01-11 08:59:05 +0100

  * Bump justrx to pull in bugfix. (Robin Sommer, Corelight)

  * CI: Re-enable `clang17_ubuntu_debug_task`. (Robin Sommer, Corelight)

1.10.0-dev.103 | 2024-01-09 14:00:44 +0100

  * Remove references to Cirrus cron task. (Benjamin Bannier, Corelight)

    We do not schedule CI runs with cron anymore.

  * Run CI ASAN task also for PRs. (Benjamin Bannier, Corelight)

  * Skip tests around stack size on macos with ASAN. (Benjamin Bannier, Corelight)

    These tests produce false positives on macos with ASAN which seem hard
    to get rid of. Disable them on the smallest possible subset of setups.

  * Fix ASAN false positives introduces with d332827aee7d70cdb642631d7a289751e1d8a36a. (Benjamin Bannier, Corelight)

    Since the logging changes of d332827aee7d70cdb642631d7a289751e1d8a36a
    ASAN reports false positives on e.g., macos-13.6.2 with its
    clang-1500.0.40.1. This patch slightly reorganizes the code so these
    false positiives are not triggered.

1.10.0-dev.98 | 2024-01-09 13:55:27 +0100

  * Bump dependencies. (Benjamin Bannier, Corelight)

  * GH-1632: Bump justrx to pull in bugfix. (Robin Sommer, Corelight)

    Closes #1632.

  * CI: drop freebsd-12, add freebsd-14. (Benjamin Bannier, Corelight)

  * GH-1500: Add `+=` operator for `string`. (Benjamin Bannier, Corelight)

    This allows appending to a `string` without having to allocate a new
    string. This might perform better most of the time.

    Closes #1500.

1.9.0-97 | 2024-01-09 10:13:30 +0100

  * GH-1632: Bump justrx to pull in bugfix. (Robin Sommer, Corelight)

1.10.0-dev.91 | 2024-01-04 12:44:32 +0100

  * Add unit tests for extracting from expanding Views. (Benjamin Bannier, Corelight)

  * Add unit tests for extracting from Views with gaps. (Benjamin Bannier, Corelight)

  * Add fast pass to noop unsafe stream iterator increment/decrement. (Benjamin Bannier, Corelight)

    We already had this optimization for safe, but not for unsafe iterators.

  * GH-1628: Avoid potentially inefficient data access in `View::extract`. (Benjamin Bannier, Corelight)

    While we already had a fast path to extract data out of `View`s
    consisting of a single chunk we still would use a potentially
    inefficient approach when extracting from `View`s over multiple chunks.

    This patch uses the same optimized handling for both cases.

    Closes #1628.

1.10.0-dev.86 | 2024-01-02 16:38:08 +0100

  * Suppress clang-tidy `misc-include-cleaner` lint. (Benjamin Bannier, Corelight)

    This currently triggers a lot of issues. While it seems to be useful
    getting us to a passing state seems to require substantial work.

  * Remove outdated workaround for clang-tidy. (Benjamin Bannier, Corelight)

    With newer clang-tidy versions this is not needed anymore.

  * Reenable clang-tidy `readability-simplify-boolean-expr` lint. (Benjamin Bannier, Corelight)

    This currently triggers no issues, and the lint seems to be generally
    useful.

  * Drop `;` after `#pragma`. (Benjamin Bannier, Corelight)

  * Fix clang-tidy `readability-redundant-string-init` lints. (Benjamin Bannier, Corelight)

  * Fix clang-tidy `bugprone-exception-escape` lints. (Benjamin Bannier, Corelight)

  * Fix clang-tidy `misc-header-include-cycle` lints. (Benjamin Bannier, Corelight)

  * Fix clang-tidy `bugprone-use-after-move` lints. (Benjamin Bannier, Corelight)

  * Suppress clang-tidy `bugprone-switch-missing-default-case` lints. (Benjamin Bannier, Corelight)

  * Fix clang-tidy `readability-avoid-unconditional-preprocessor-if` lints. (Benjamin Bannier, Corelight)

  * Fix clang-tidy `modernize-use-emplace` lints. (Benjamin Bannier, Corelight)

  * Suppress clang-tidy `modernize-macro-to-enum` lints. (Benjamin Bannier, Corelight)

  * Fix clang-tidy `performance-avoid-endl` lints. (Benjamin Bannier, Corelight)

  * Fix clang-tidy `modernize-type-traits` lints. (Benjamin Bannier, Corelight)

  * Update clang-tidy suppression for newer clang-tidy versions. (Benjamin Bannier, Corelight)

  * Remove nightly task. (Benjamin Bannier, Corelight)

    This task originally tested against Zeek's `master` branch. We do not do
    that anymore since some time so this task has lost its purpose.

  * Remove `--rpath` flag to `ci/run-ci`. (Benjamin Bannier, Corelight)

    Uses of this flag have been mirroring external setup of
    `LD_LIBRARY_PATH` in `.cirrus.yml` for some time with no added benefit.

    Drop it instead of investing work in keeping it consistent.

  * Disable building benchmarks in CI. (Benjamin Bannier, Corelight)

    On some platforms the CMake configure phase of 3rdparty/benchmark fails
    with errors like

    ```
    CMake Error at 3rdparty/benchmark/CMakeLists.txt:301 (message):
      Failed to determine the source files for the regular expression backend
    ```

    Since it is not executed in CI anyway disable it.

  * Refresh LLVM version in CI Docker image. (Benjamin Bannier, Corelight)

  * Remove unused Docker image args. (Benjamin Bannier, Corelight)

  * Bump CI image to ubuntu-22.04. (Benjamin Bannier, Corelight)

1.10.0-dev.64 | 2024-01-02 15:50:26 +0100

  * Add dependency of generated file `spicy-build` on source file. (Benjamin Bannier, Corelight)

    Without this dependency we never updated the generated file.

  * Remove use of `IntrusivePtr` over `const`. (Benjamin Bannier, Corelight)

    It looks like using a `IntrusivePtr<const T>` interferes with default'ed
    move constructors and assignment operators so that the non-move versions
    are called. This can incur additional overhead.

    This patch simply gets rid all `IntrusivePtr<const T>` in favor of
    `IntrusivePtr<T>` which does not seem to show this issue. It might be
    possible to instead adjust something e.g., `ManagedObject` so
    `IntrusivePtr` does not regress when holding semantically `const`
    values though.

    For a big internal parser this shows runtime improvements up to 2%.

1.10.0-dev.61 | 2024-01-02 15:50:05 +0100

  * Fix docs namespace for symbols from `filter` module. (Benjamin Bannier, Corelight)

    We previously would document these symbols to be in `spicy` even though
    they are in `filter`.

1.10.0-dev.59 | 2023-12-15 13:29:41 +0100

  * Add move ctr for `stream::SafeConstIterator`. (Benjamin Bannier, Corelight)

  * Use unchecked operations for `View::unsafeEnd`. (Benjamin Bannier, Corelight)

    We previously would call `end()` to compute `unsafeEnd` which incurs
    overheads this function is explicitly designed to avoid. We now switch
    this implementation of this function to completely unsafe code avoiding
    these overheads.

  * Add move ctr for `stream::View`. (Benjamin Bannier, Corelight)

    In C++17 move ctors are not by default generated, add them explicitly.
    We also get rid of the `View` dtr since it adds nothing (e.g., the class
    is declared `final`), but declaring them could e.g., make the class not
    POD anymore (this is not the case here though).

1.10.0-dev.55 | 2023-12-12 18:38:24 +0100

  * GH-1617: Fix handling of `%synchronize-*` attributes for units in lists. (Benjamin Bannier, Corelight)

    We previously would not detect `%synchronize-at` or `%synchronize-from`
    attributes if the unit was not directly in a field, i.e., we mishandled
    the common case of synchronizing on a unit in a list.

    With this patch we now handle these attributes, regardless of how the
    unit appears.

1.10.0-dev.53 | 2023-12-12 13:05:41 +0100

  * Avoid unnecessary copy when constructing `Stream` from `Bytes`. (Benjamin Bannier, Corelight)

  * Back Chunk with a std::string instead of std::variant. (Arne Welzel, Corelight)

    This is probably a bit funky due to the trailing \0 and a Chunk not
    being a string, but hey, hilti::rt::Bytes is backed by std::string, too.

    Looking at the creation of Chunk::Chunk(), there's actually a memset()
    operation visible due to the variant holding some 40 bytes that are
    zero initialized upon construction. std::string supports SSO, so we
    can leverage even if the actual size is out of our control. The
    std::get_if()s do show up very marginally for the View::size() calls.

1.10.0-dev.50 | 2023-12-11 12:08:12 +0100

  * GH-1615: Optimize C++ tuple element coercions. (Benjamin Bannier, Corelight)

    This patch introduces an optimization to pass C++ tuples through if they
    do not need C++-side coercion. This emits better code if a tuple ctor
    already has the right types. We also optimize codegen if we are coercing
    from a temporary tuple in that we now capture it in a temporary and
    coerce elements from that instead of repeatedly emitting the same tuple
    ctor (we scale with the number of tuple elements now not its square
    anymore).

1.10.0-dev.48 | 2023-12-08 15:55:12 +0100

  * Use deterministic destruction of Spicy runtime in unit tests. (Benjamin Bannier, Corelight)

    We previously would shut down the runtimes in units tests with
    library destructors. Due to recent changes ASAN correctly flags this as
    potential use-after-free errors (e.g., it might run after the
    configuration global has already been destructed).

    With this patch implement our own doctest main function and
    deterministically tear down runtimes from that.

1.10.0-dev.46 | 2023-12-08 09:56:14 +0100

  * Remove redundant forward decl. (Benjamin Bannier, Corelight)

  * GH-1611: Silence ASAN warning. (Benjamin Bannier, Corelight)

1.10.0-dev.43 | 2023-12-08 09:55:13 +0100

  * Allocate Vector::_control lazily. (Arne Welzel, Corelight)

  * Allocate Bytes::_control lazily. (Arne Welzel, Corelight)

1.10.0-dev.40 | 2023-12-06 12:31:27 +0100

  * GH-1605: Allow for unresolved types for set `in` operator. (Benjamin Bannier, Corelight)

1.10.0-dev.38 | 2023-12-06 12:28:27 +0100

  * Avoid reallocating Bytes when appending from view (Arne Welzel, Corelight)

    When copying a full view into a Bytes instance, avoid potential
    reallocations and memcpy() by pre-allocating enough capacity in
    the underlying string.

  * Allocate vector of correct size right away when constructing Chunks. (Benjamin Bannier, Corelight)

  * Avoid one extra std::string copy in Stream::append. (Arne Welzel, Corelight)

    For a chunk of significant size, seems that would result in an extra
    malloc and copy of the input data.

  * Reduce safe iterator use in internal code for `Bytes`. (Benjamin Bannier, Corelight)

    The safe iterator for bytes dynamically allocates which can cause
    overhead. Use index-based or at least string iterators to reduce that
    overhead where possible.

  * Use unsafe iterators in `View::extract`. (Benjamin Bannier, Corelight)

  * Avoid unneeded parameter copy in parse functions. (Benjamin Bannier, Corelight)

  * Streamline `View::firstBlock`. (Benjamin Bannier, Corelight)

  * Reduce allocations when connecting sinks by mime-type. (Benjamin Bannier, Corelight)

  * Reduce allocations for `Sink` debug logging. (Benjamin Bannier, Corelight)

  * Deprecate `builder::string`. (Benjamin Bannier, Corelight)

  * Reduce allocations when creating exceptions. (Benjamin Bannier, Corelight)

  * Reduce allocations in `builder::addAssert`. (Benjamin Bannier, Corelight)

  * Reduce allocations in `ParserBuilder::waitForInput`. (Benjamin Bannier, Corelight)

  * Reduce allocations in `Builder::startProfiler`. (Benjamin Bannier, Corelight)

  * Reduce allocatons in `Builder::addDebug*` methods. (Benjamin Bannier, Corelight)

  * Reduce allocations in `ParserBuilder::parseError`. (Benjamin Bannier, Corelight)

  * Reducing copying when forwarding data to sinks. (Benjamin Bannier, Corelight)

  * Reduce lookup overhead in indent/dedent logger functions. (Benjamin Bannier, Corelight)

  * Emit C++ string literals for HILTI string literals. (Benjamin Bannier, Corelight)

    When emitting literals for HILTI strings (string ctors) we would
    previously explicitly force creation of `std::string`. This was almost
    always an unnecessary pessimisation over emitting string literals since
    even if their C++ uses expected `std::string` string literals can
    convert to this type implicitly; at the same time it made it impossible
    to make effective use of APIs accepting `std::string_view`.

    With this patch we now emit C++ string literals for HILTI string
    literals.

  * Emit locations as generated strings. (Benjamin Bannier, Corelight)

  * Avoid allocations when creating parsers. (Benjamin Bannier, Corelight)

  * Use non-owning strings for `fmt`. (Benjamin Bannier, Corelight)

  * Use non-owning strings for `printParserState`. (Benjamin Bannier, Corelight)

  * GH-1591: Use non-owning strings in the logging framework. (Benjamin Bannier, Corelight)

    Closes #1591.

  * Add `to_string_for_print` for string literals. (Benjamin Bannier, Corelight)

  * GH-1589: Reduce string allocations on hot parse path. (Benjamin Bannier, Corelight)

    When waiting for input we pass down strings for a possible error message
    and the triggering location. In generated code these are always
    literals.

    With this patch we do not take them as owning strings, but instead as
    views into existing strings to minimize allocations. In the case of
    error messages the created low-level exception objects already had used
    string_views, so this also aligns the APIs.

    Closes #1589.

  * Bump pre-commit hooks (Benjamin Bannier, Corelight)

1.10.0-dev.10 | 2023-12-06 10:40:22 +0100

  * Allow unsafe Vector iteration over underlying std::vector. (Arne Welzel, Corelight)

1.10.0-dev.8 | 2023-11-17 14:04:16 +0100

  * doc: Fix typo and outdated info for host applications (Anthony VEREZ)

  * Add placeholder section for 1.10 to NEWS. (Benjamin Bannier, Corelight)

1.9.0 | 2023-10-24 16:28:18 +0200

  * Release 1.9.0.

1.9.0-dev.159 | 2023-10-24 16:26:42 +0200

  * Bump 3rdparty/utf8proc (dependabot[bot])

1.9.0-dev.157 | 2023-10-23 10:58:57 +0200

  * Fix spicy-build to correctly infer library directory. (Robin Sommer, Corelight)

    Closes https://github.com/zeek/zeek/issues/3384.

1.9.0-dev.155 | 2023-10-23 10:54:13 +0200

  * GH-1565: Disable capturing backtraces with HILTI exceptions in non-debug builds. (Robin Sommer, Corelight)

    They can be expensive to capture, and aren't used anywhere by default
    unless explicitly requested.

    We change it so that the exception class still remains ABI
    compatible between release and debug builds. It's the compilation
    settings of the code including `exception.h` that determines if a
    backtrace it captured.

    Closes #1565.

1.9.0-dev.152 | 2023-10-23 10:49:19 +0200

  * GH-1567: Speed up runtime calls to start profilers. (Robin Sommer, Corelight)

    We now cache the profiler tags to avoid frequent re-computation.

    Closes #1567.

1.9.0-dev.150 | 2023-10-23 10:48:59 +0200

  * GH-1568: Fix bitfield's lack of declaring its C++-side type dependencies. (Robin Sommer, Corelight)

    Closes #1568.

1.9.0-dev.148 | 2023-10-23 10:10:39 +0200

  * Remove check of Zeek docs in Cirrus cron config. (Benjamin Bannier, Corelight)

    This is a follow-up to 2b92da596631ff1a29b7deac05e00faaad9305f3.

1.9.0-dev.146 | 2023-10-13 12:26:24 +0200

  * Remove Zeek-specific documentation. (Robin Sommer, Corelight)

    This now lives Zeek-side.

    This keeps the section structure so that we make things easier to
    find, and put in pointers to the new Zeek-side documentation

    It leaves everything in the development section as is; hard to piece
    apart and seems fine to leave it here.

1.9.0-dev.144 | 2023-10-13 11:47:35 +0200

  * GH-1571: Remove trimming inside individual chunks. (Benjamin Bannier, Corelight)

    Trimming `Chunk`s (always from the left) causes a lot of internal work
    with only limited benefit since we manage visibility with `stream::View`s
    on top of `Chunk`s anyway.

    This patch removes trimming inside `Chunk`s so now any trimming only
    removes `Chunk`s from `Chain`s, but does not internally change
    individual `Chunk`s anymore. This might lead to slightly increased memory
    use, but callers usually have that data in memory anyway.

    Closes #1571.

1.9.0-dev.142 | 2023-10-09 11:10:35 +0200

  * Explicitly set ASM compiler. (Benjamin Bannier, Corelight)

    When not setting an ASM compiler we might end up configuring Clang as C
    compiler, but GCC as ASM compiler. The code in `3rdparty/fiber` then
    causes Clang-only flags like `-Weverything` to be passed to GCC (which
    does not understand it).

    The way we need to do this is subtle. We need to rely on
    `3rdparty/fiber` to explicitly `enable_language(ASM)` and not set it as
    a project language to work around `find_package(BISON)` breaking if
    `CMAKE_ASM_COMPILER` is set. By removing ASM from project languages and
    setting the ASM config after Bison was found, but before we configure
    `3rdparty/fiber` things seem to work as intended.

1.9.0-dev.140 | 2023-10-06 12:34:02 +0200

  * Set `ParserState::begin` on all possible paths into a parser. (Benjamin Bannier, Corelight)

    We previously would sometimes leave `ParserState::begin` unset, e.g.,
    when parsing beginning parsing of a top-level production. With this
    patch we now set its value on all possible paths.

  * Fix spicy-rt-tests for freebsd-12. (Benjamin Bannier, Corelight)

  * GH-1089: Make `offset()` independent of random access functionality. (Benjamin Bannier, Corelight)

    With this patch we store the value returned by `offset()` directly in
    the unit instead of computing it on the fly when requested from `cur -
    begin`. With that `offset()` can be used without enabling random access
    functionality on the unit.

    Closes #1089.

  * Move `offset()` into its own feature. (Benjamin Bannier, Corelight)

    This patch makes `__position` depend on `uses_offset` so uses of it
    (`position()`, `offset()`) can be tracked separately. Since `offset()`
    still requires random access `uses_offset` enables `uses_random_access`
    for that use case; we plan to relax that in a follow-up patch.

  * Consistently wrap `ParserState::begin` in optional. (Benjamin Bannier, Corelight)

1.9.0-dev.134 | 2023-10-04 11:20:46 +0200

  * Bump 3rdparty/fiber from `f75b2f9` to `ada36b2` (dependabot[bot])

    Bumps [3rdparty/fiber](https://github.com/simonfxr/fiber) from `f75b2f9` to `ada36b2`.
    - [Commits](https://github.com/simonfxr/fiber/compare/f75b2f93aa312b9922f9c977021e5470ff7715fa...ada36b254c10d487eb4d8d108a3cb156538e1885)

    ---
    updated-dependencies:
    - dependency-name: 3rdparty/fiber
      dependency-type: direct:production
    ...

    Signed-off-by: dependabot[bot] <support@github.com>

  * Bump 3rdparty/benchmark from `0d98dba` to `7736df0` (dependabot[bot])

    Bumps [3rdparty/benchmark](https://github.com/google/benchmark) from `0d98dba` to `7736df0`.
    - [Release notes](https://github.com/google/benchmark/releases)
    - [Commits](https://github.com/google/benchmark/compare/0d98dba29d66e93259db7daa53a9327df767a415...7736df03049c362c7275f7573de6d6a685630e0a)

    ---
    updated-dependencies:
    - dependency-name: 3rdparty/benchmark
      dependency-type: direct:production
    ...

    Signed-off-by: dependabot[bot] <support@github.com>

  * Bump 3rdparty/SafeInt from `4cafc91` to `925235c` (dependabot[bot])

    Bumps [3rdparty/SafeInt](https://github.com/dcleblanc/SafeInt) from `4cafc91` to `925235c`.
    - [Release notes](https://github.com/dcleblanc/SafeInt/releases)
    - [Commits](https://github.com/dcleblanc/SafeInt/compare/4cafc9196c4da9c817992b20f5253ef967685bf8...925235c06de490865f871218fa2bd8ac38241a95)

    ---
    updated-dependencies:
    - dependency-name: 3rdparty/SafeInt
      dependency-type: direct:production
    ...

    Signed-off-by: dependabot[bot] <support@github.com>

  * GH-1549: GH-1554: Fix potential infinite loop when trimming data before stream. (Benjamin Bannier, Corelight)

    Previously we would trigger an infinite loop if one tried to trim before
    the head chunk of a stream. In praxis this seem to have been no issue
    due to #1549 and us emitting way less calls to trim than possible.

    This patch adds an explicit check whether we need to trim anything, and
    exits the low-level function early for such cases.

    Closes #1554.

1.9.0-dev.128 | 2023-09-29 13:27:28 +0200

  * GH-1550: Replace recursive deletion with explicit loop to avoid stack overflow. (Benjamin Bannier, Corelight)

1.9.0-dev.126 | 2023-09-29 13:26:53 +0200

  * GH-1549: Add feature guards to accesses of a unit's `__position`. (Benjamin Bannier, Corelight)

    Access of `__position` triggers a random access functionality. In order
    to distinguish our internal uses from accesses due to user code, most
    access in our generated code should be guarded with a feature constant
    (`if` or ternary).

    In this patch add proper guards for a couple instances where we did not
    do that correctly. That mishap caused all units with containers to be
    random access (even the root unit) which in turn could have lead to
    e.g., unbounded memory growth, or runtime overhead due to generation and
    execution of unneeded code, or expensive cleanup on very large untrimmed
    inputs.

    Closes #1549.

  * Refactor feature guard helper to directly take unit ID instead of type. (Benjamin Bannier, Corelight)

1.9.0-dev.123 | 2023-09-27 09:41:04 +0200

  * Escape rendered bitfield fields. (Benjamin Bannier, Corelight)

  * Enforce that `Bitfield`s always own their fields. (Benjamin Bannier, Corelight)

  * Fix typos. (Benjamin Bannier, Corelight)

1.9.0-dev.119 | 2023-09-27 09:39:35 +0200

  * GH-1542: GH-1547: Bump dependencies. (Benjamin Bannier, Corelight)

1.9.0-dev.111 | 2023-09-26 16:05:25 +0200

  * Add GH dependabot config. (Benjamin Bannier, Corelight)

1.9.0-dev.109 | 2023-09-22 16:35:02 +0200

  * Add explicit dependency for local pre-commit hook. (Benjamin Bannier, Corelight)

  * Fix file selection in clang-format pre-commit hook. (Benjamin Bannier, Corelight)

1.9.0-dev.106 | 2023-09-22 12:19:21 +0200

  * GH-1533: Fix access to anonymous bitfield element through a constant value. (Robin Sommer, Corelight)

1.9.0-dev.104 | 2023-09-22 11:04:19 +0200

  * Artificially limit the number of open files. (Benjamin Bannier, Corelight)

    This works around a silent failure in reproc where it would refuse to
    run on systems which huge rlimits for the number of open files. We have
    seen this hit on huge production boxes.

1.9.0-dev.102 | 2023-09-22 11:04:01 +0200

  * GH-1478: Add regression test for #1478. (Benjamin Bannier, Corelight)

  * Add begin to parser state. (Benjamin Bannier, Corelight)

    This patch adds the current begin position to the parser state, and
    makes the corresponding changes to generated parser functions so it is
    passed down.

    We already modelled the semantic beginning of the input in the unit, but
    had no reliable way to keep this up-to-date across non-unit contexts
    like `&parse-from`. This would then for certain setups lead to generated
    code where `input` and `position` would point to different inputs which in
    turn caused `offset` (modelled as `position - input`) to be incorrect.

  * Factor computation of feature const into separate function. (Benjamin Bannier, Corelight)

  * Expand validator error message. (Benjamin Bannier, Corelight)

  * Add trait class to bitfield types. (Robin Sommer, Corelight)

    This lets C++ templates test if a class T is a bitfield.

1.9.0-dev.95 | 2023-09-20 12:14:50 +0200

  * Add trait class to bitfield types. (Robin Sommer, Corelight)

1.9.0-dev.93 | 2023-09-20 09:26:29 +0200

  * Declare Spicy pygments extension as parallel-safe. [skip CI] (Benjamin Bannier, Corelight)

1.9.0-dev.91 | 2023-09-19 11:50:32 +0200

  * Skip validating links to https://www.icir.org/hilti. (Benjamin Bannier, Corelight)

  * Document using anonymous field for extracting TCP messages. (Benjamin Bannier, Corelight)

1.9.0-dev.88 | 2023-09-15 10:47:53 +0200

  * Use find_package(Python) with version. (Arne Welzel, Corelight)

    Zeek's configure sets Python_EXECUTABLE has hint, but Spicy is using
    find_package(Python3) and would only use Python3_EXECUTABLE as hint.
    This results in Spicy finding a different (the default) Python executable
    when configuring Zeek with --with-python=/opt/custom/bin/python3.

    Switch Spicy over to use find_package(Python) and add the minimum
    version so it knows to look for Python3.

1.9.0-dev.86 | 2023-09-14 10:36:25 +0200

  * Add support for passing arbitrary C++ compiler flags. (Benjamin Bannier, Corelight)

    This adds a magic environment variable `HILTI_CXX_FLAGS` which if set
    specifies compiler flags which should be passed during C++ compilation
    after implicit flags. This could be used to e.g., set defines, or set
    low-level compiler flags.

    Even with this flag, for passing include directories one should still
    use `HILTI_CXX_INCLUDE_DIRS` since they are searched before any
    implicitly added paths.

1.9.0-dev.84 | 2023-09-07 17:12:00 +0200

  * GH-1467: Support bitfield constants in Spicy for parsing. (Robin Sommer, Corelight)

    One can now define bitfield "constants" for parsing by providing
    integer expressions with fields:

        type Foo = unit {
          x: bitfield(8) {
            a: 0..3 = 2;
            b: 4..7;
            c: 7 = 1;
          };

    This will first parse the bitfield as usual and then enforce that the
    two bit ranges that are coming with expressions (i.e., `a` and `c`)
    indeed containing the expected values. If they don't, that's a parse
    error.

    We also support using such bitfield constants for look-ahead parsing:

        type Foo = unit {
          x: uint8[];
          y: bitfield(8) {
            a: 0..3 = 4;
            b: 4..7;
          };
        };

    This will parse uint8s until a value is discovered that has its bits
    set as defined by the bitfield constant.

    (We use the term "constant" loosely here: only the bits with values
    are actually enforced to be constant, all others are parsed as usual.)

    Closes #1467.

  * Extend bitfield type with per-item storage for constant values. (Robin Sommer, Corelight)

    This allows to associate an expression with each bit value. We don't
    use this from HILTI because there's isn't a good syntax to do so
    (and/or: it's not worth adding), but we'll use (and test) this from
    Spicy in a subsequent commit.

  * Add bitfield constants. (Robin Sommer, Corelight)

    It's now possible to initialize a bitfield value through an assignment
    from a struct constructor expression:

        type BF = bitfield(8) {
          a: 0..3;
          b: 4..7;
          c: 4..5;
        };

        global BF bf = [$a = 1, $c = 2];

  * Change internal tuple representation of bitfield to store optional values. (Robin Sommer, Corelight)

    This will allow us to create bitfield constants where not at all
    elements are set. We also add an operator to test if an element is
    set (test forthcoming in a subsequent commit).

1.9.0-dev.79 | 2023-09-07 11:15:05 +0200

  * GH-1520: Fix handling of `spicy-dump --enable-print`. (Benjamin Bannier, Corelight)

    This flag was understood, but not handled since at least v1.1.0.

1.9.0-dev.77 | 2023-09-06 14:38:55 +0200

  * Clarify error handling docs [skip CI]. (Benjamin Bannier, Corelight)

    The docs previously made it sound as if exceptions could potentially be
    swallowed with `on %error` handlers. Since this is currently not
    supported, but a often requested feature, clarify the docs to not stir
    up too much anticipation.

1.9.0-dev.75 | 2023-09-05 16:15:17 +0200

  * Merge branch 'topic/bbannier/rtd-search-suppress-doxygen-results' (Benjamin Bannier, Corelight)

  * GH-1516: Do not include generated Doxygen documentation in RTD search results [skip CI]. (Benjamin Bannier, Corelight)

    Doxygen documentation is likely not relevant for users and hides results
    they are looking for. Also, Doxygen already comes with its own search
    which is reachable the developer documentation.

    Closes #1516.

  * Reduce ccache cache size in CI. (Benjamin Bannier, Corelight)

1.9.0-dev.71 | 2023-09-01 10:33:35 +0200

  * Drop removed RTD config key. (Benjamin Bannier, Corelight)

1.9.0-dev.69 | 2023-09-01 09:57:22 +0200

  * GH-1503: Handle anonymous bitfields inside `switch` statements. (Robin Sommer, Corelight)

    We now map items of anonymous bitfields inside a `switch` cases into
    the unit namespace, just like we already do for top-level fields. We
    also catch if two anonymous bitfields inside those cases carry the
    same name, which would make accesses ambiguous.

    So the following works now:

    ```
        switch (self.n) {
            0 -> : bitfield(8) {
                A: 0..7;
            };
            * -> : bitfield(8) {
                B: 0..7;
            };
        };
    ```

    Whereas this does not work:

    ```
        switch (self.n) {
            0 -> : bitfield(8) {
                A: 0..7;
            };
            * -> : bitfield(8) {
                A: 0..7;
            };
        };
    ```

    The latter not working is reasonable I think, rather than trying to
    figure out that the two cases have the same type and hence could be
    mapped to a single data member in the resulting code (which we do, and
    support, for cases of the same *name* and type).

    Closes #1503.

  * Polish string rendering of anonymous bitfields in structs. (Robin Sommer, Corelight)

    We now print their ID as `<anon>`. This isn't perfect, it would be
    nicer if we printed out the IDs of the bitfield's items at the
    top-level (like `spicy-dump` does). However we don't have the
    necessary type information available when rendering as a string, and
    this seems good enough.

  * Support `.?` and `?.` for items of anonymous bitfields. (Robin Sommer, Corelight)

1.9.0-dev.64 | 2023-08-28 17:05:18 +0200

  * GH-1508: Fix returned value for `<unit>.position()`. (Benjamin Bannier, Corelight)

    We declared this method to return an iterator, but potentially returned
    an optional iterator. This patch add an additional deref so we get the
    correct value.

1.9.0-dev.62 | 2023-08-28 12:11:40 +0200

  * GH-1504: Use user-inaccessible chars for encoding `::` in feature variables. (Benjamin Bannier, Corelight)

    When setting up feature tracking variables for the optimizer we
    previously would normalize `:` as `_`, e.g., `mod::Unit` would lead to
    feature variables `__feat%mod__Unit%...`. In various places in the
    optimizer we would then use that knowledge to extract module and unit
    ID. Since user-specified identifiers can contain literal `_` or `__`
    this could confuse the optimizer.

    With this patch we now normalize `:` as `@` which is not
    user-accessible. We add a new codegen normalization so `@` becomes
    `0x40`. This gives a more reliable encoding.

  * Clean up extraction and handling of feature flag in optimizer. (Benjamin Bannier, Corelight)

1.9.0-dev.59 | 2023-08-22 15:10:07 +0200

  * Add documentation for `export` extensions. (Robin Sommer, Corelight)

    This goes with https://github.com/zeek/zeek/pull/3228.

  * spicy-dump: Level up anonymous bitfields. (Robin Sommer, Corelight)

    Spicy-dump now renders items of anonymous bitfields at the parent
    level. Offsets are added accordingly as well if requested.

  * spicy-dump: Make offsets in JSON output useful. (Robin Sommer, Corelight)

    We were including offsets into spicy-dumps JSON output, but there was
    no information in there saying which offsets belong to what field.
    Turning `__offsets` from an array into a map indexed by field name now.

  * spicy-dump: Change format how bitfields are printed. (Robin Sommer, Corelight)

    We now render them as key/value form in both text and JSON output.

  * Include a flag into a unit field's type information indicating if it's anonymous. (Robin Sommer, Corelight)

    This preservers the information if a field had a name even though
    internally we do carry a generated dummy name around in that case. For
    now, we just adapt `spicy-dump` to not print the dummy name in that
    case in its text output. Later, Zeek will leverage this when
    auto-generating record types.

  * GH-1468: Allow to directly access members of anonymous bitfields. (Robin Sommer, Corelight)

    This works now:

        type Foo = unit {
          : bitfield(8) {
            x: 0..3;
            y: 4..7;
          };

          on %done {
            print self.x, self.y;
          }
    };

  * GH-1032: GH-1484: Switch Spicy over to use the new HILTI-side bitfield type. (Robin Sommer, Corelight)

  * Implement remaining pieces to make `bitfield` a full HILTI type. (Robin Sommer, Corelight)

    We keep representing bitfields as a C++ tuple behind the scenes, but
    no longer map it into a HILTI-side tuple as well. Instead `Bitfield`
    is now a regular type with corresponding operators and runtime
    functionality.

    The one change to the C++ tuple representation is that we add an
    additional hidden tuple element at the end that stores the original
    integer value. That allows for displaying it (which we will do for now
    only inside Spicy debug output) and could later also facilitate modifying
    fields and reconstructing an integer.

  * Move `bitfield` type from Spicy to HILTI. (Robin Sommer, Corelight)

    This prepares for making `bitfield` a fully-supported HILTI type. This
    change only moves existing code over to HILTI without breaking
    Spicy-side functionality. There are still HILTI-side pieces missing
    for providing full support, which will come next.

  * Fix comparison for resolved operators. (Robin Sommer, Corelight)

    The `isEqual` method was implemented in the base class, which led to
    wrong template types being used.

    A subsequent commit will depend on this working correctly, including a test
    breaking if not.

1.9.0-dev.47 | 2023-08-14 11:32:33 +0200

  * GH-1384: Fix stringification of `DecodeErrorStrategy`. (Benjamin Bannier, Corelight)

    This was pointed out by Simeon Miteff in #1384.

1.9.0-dev.45 | 2023-08-14 11:32:11 +0200

  * Fix handling of `--show-backtraces` flag. (Benjamin Bannier, Corelight)

    This flag was intended and always documented as a toggle, but instead
    implemented as an argument taking a value (which was never evaluated).
    This patch fixes the implementation to align with itend and
    documentation.

1.9.0-dev.43 | 2023-08-14 11:31:35 +0200

  * Bump FreeBSD versions in CI. (Benjamin Bannier, Corelight)

1.9.0-dev.41 | 2023-07-19 09:17:47 +0200

  * Mention unit switch statements in conditional parsing docs. (Benjamin Bannier, Corelight)

    Instead of decorating many fields with `if` conditions for certain
    grammars a simpler approach can be to use a unit switch statement with
    branch-by-expressions. Add a link to unit switch from the conditional
    parsing section.

1.9.0-dev.39 | 2023-07-18 11:21:27 +0200

  * GH-1485: Add validator rejecting unsupported multiple uses of attributes. (Benjamin Bannier, Corelight)

    While for some attributes like e.g., `&requires` we support multiple
    mentions, for many we actually only support at most one instance. We
    previously would only interpret only one, even if multiple instances
    where given.

    This patch adds a validator which rejects code with multiple instances
    of such attributes. We also add test coverage for `&requires` supporting
    multiple instances.

    Closes #1485.

1.9.0-dev.37 | 2023-07-07 12:56:37 +0200

  * Fix typo: maccOS -> macOS (Ryan Schmidt)

1.9.0-dev.35 | 2023-07-07 11:22:41 +0200

  * GH-1465: Support skipping explicit `%done` in external hooks. (Robin Sommer, Corelight)

    Assuming `Foo::X` is a unit type, these two are now equivalent:

        on Foo::X::%done   { }
        on Foo::X          { }

  * GH-1465: Produce better error message when hooks are used on a
    unit field. (Robin Sommer, Corelight)


1.9.0-dev.32 | 2023-07-05 13:03:56 +0200

  * GH-1475: Add regression test for #1475. (Benjamin Bannier, Corelight)

1.9.0-dev.30 | 2023-07-04 17:10:32 +0200

  * GH-1475: Revert #1439. (Benjamin Bannier, Corelight)

    The changes for #1439 were incorrect and introduced incorrect parsing with
    possible undefined behavior at runtime when encountering a combination of
    list parsing and `&parse-from` or `&parse-at`.

    This revert reopens GH-1421 and GH-1089.

1.9.0-dev.27 | 2023-06-28 14:43:05 +0200

  * Run a few pre-commit checks only at `commit` stage. (Benjamin Bannier, Corelight)

  * GH-1466: Reject uses of `self` in unit `&size` and `&max-size` attribute. (Benjamin Bannier, Corelight)

    Values in `self` are only available after parsing has started while
    `&size` and `&max-size` are consumed before that. This means that any
    use of `self` and its members in these contexts would only ever see
    unset members, so it should not be the intended use.

    This patch adds a validation which rejects uses of `self` for these unit
    attributes.

  * Refactor validation code for unit `&size` and `&max-size` attributes. (Benjamin Bannier, Corelight)

1.9.0-dev.23 | 2023-06-28 10:20:18 +0200

  * Bump doc sphinx-rtd-theme dependency. (Benjamin Bannier, Corelight)

1.9.0-dev.21 | 2023-06-27 10:04:53 +0200

  * Fix rendering in validation of `%byte-order` attribute. (Benjamin Bannier, Corelight)

  * GH-1275: Add missing lowering of Spicy unit ctor to HILTI struct ctor. (Benjamin Bannier, Corelight)

1.9.0-dev.18 | 2023-06-27 10:04:11 +0200

  * Bump freebsd12 version in CI. (Benjamin Bannier, Corelight)

1.9.0-dev.16 | 2023-06-26 14:48:32 +0200

  * Factor out code to guard search during recovery. (Benjamin Bannier, Corelight)

  * GH-1231: GH-1464: Add special handling for potential `advance` failure in trial mode. (Benjamin Bannier, Corelight)

    In 7c18597f1b95747b0305918eb1777ed22c06b166 for #1231 we added code to
    handle advance failures in trial mode when synchronizing on regexps, but
    missed adding the same support for literals. This patch adds that
    missing part.

1.9.0-dev.13 | 2023-06-26 14:39:10 +0200

  * GH-1446: Initialize generated struct members in constructor body. (Benjamin Bannier, Corelight)

    We previously would generate them with default member initializers. This
    lead to issues since for some constructs we need to create temporaries
    (e.g., mutable accessors to `self`). With this patch we now always emit
    a constructor body which ensures we have a block to attach temporaries
    to.

  * Disable implicit conversions of `cxx::Block` to `bool`. (Benjamin Bannier, Corelight)

    I ran into this when I incorrectly tried to directly output a
    `cxx::Block` to an iostream which worked but output a boolean (via
    implicit conversion).

  * Add missing newline in spicy-driver usage string. (Benjamin Bannier, Corelight)

1.9.0-dev.9 | 2023-06-20 08:48:49 +0200

  * Fix signedness warning. (Arne Welzel, Corelight)

1.9.0-dev.7 | 2023-06-15 14:40:25 +0200

  * Fix autogeneration of zeek docs. (Benjamin Bannier, Corelight)

1.9.0-dev.5 | 2023-06-14 16:37:26 +0200

  * GH-1435: Add bitwise operators `&`, `|`, and `^` for booleans. (Benjamin Bannier, Corelight)

1.9.0-dev.3 | 2023-06-14 16:35:45 +0200

  * GH-1343: Include condition in `&requires` failure message. (Benjamin Bannier, Corelight)

    With this patch we now include the `&requires` condition in the failure
    message in case the condition is not met. Since we do not have direct
    access to the Spicy source code anymore we render HILTI code for now,
    but undo the most common desugaring by replacing `__dd` with `$$`.

  * Update links to some release artifacts. (Benjamin Bannier, Corelight)

  * Add placeholder section for 1.9 to NEWS. (Benjamin Bannier, Corelight)

1.8.0 | 2023-06-14 08:53:34 +0200

  * Release 1.8.0.

1.8.0-dev.176 | 2023-06-14 08:52:14 +0200

  * Update Zeek docs for Zeek 6.0. (Robin Sommer, Corelight)

1.8.0-dev.173 | 2023-06-09 10:01:28 +0200

  * GH-1447: Do not forcibly make `strong_ref` `in` function parameters immutable. (Benjamin Bannier, Corelight)

    We previously would always emit a `const` value when passing a
    `strong_ref` as a function parameter. This lead to the unintuitive error
    that non-const `strong_ref` parameters could not be modified or have
    non-const methods called on them (this would actually fail at codegen
    time).

    With this patch we remove the special treatment of `strong_ref`
    parameter mutability.

    Closes #1447.

1.8.0-dev.171 | 2023-06-09 10:00:51 +0200

  * GH-1452: Allow resolving of unit parameters before `self` is fully resolved. (Benjamin Bannier, Corelight)

    When constructing the list of declarations in a scope we previously
    would not emit any items for struct types without fully resolved type.
    This lead to the situation where when resolving references to a unit
    parameter we might end up resolving them to similarly named globals
    since they were available before the struct type was resolved.

    This patch makes sure that we emit at least unit parameters at struct
    scope level, even if we cannot emit `self` yet. This improves the
    situation around globals mentioned above and is likely okay since `self`
    should only ever resolve to something at struct scope.

    Closes #1452.

1.8.0-dev.169 | 2023-06-09 09:59:42 +0200

  * Fix docs for long-form of `-x` flag to spicyc. (Benjamin Bannier, Corelight)

    The long form of `-x` is `--output-c++-files` but we documented
    `--output-c++` instead which is used for `-c` and takes no argument.

1.8.0-dev.167 | 2023-06-09 09:59:08 +0200

  * Replace obsolete uses of `egrep` with `grep -E`. (Benjamin Bannier, Corelight)

1.8.0-dev.165 | 2023-06-07 17:23:04 +0200

  * Clean up dependencies for ubuntu-22 image. (Benjamin Bannier, Corelight)

1.8.0-dev.163 | 2023-06-07 15:15:10 +0200

  * Remove unsupported fedora-36, add fedora-38. (Benjamin Bannier, Corelight)

  * Bump supported Alpine version to latest release. (Benjamin Bannier, Corelight)

  * Refresh supported Ubuntu versions to upstream supported ones. (Benjamin Bannier, Corelight)

  * Remove EOL debian-9. (Benjamin Bannier, Corelight)

1.8.0-dev.158 | 2023-06-05 11:26:41 +0200

  * GH-1448: Adjust getting started docs for removed JIT compilation via Zeek. (Benjamin Bannier, Corelight)

  * Fix markup for code blocks. (Benjamin Bannier, Corelight)

1.8.0-dev.155 | 2023-05-26 13:59:22 +0200

  * Remove Zeek from Spicy CI. (Benjamin Bannier, Corelight)

1.8.0-dev.153 | 2023-05-11 16:26:49 +0200

  * Profile exception activity. (Robin Sommer, Corelight)

1.8.0-dev.151 | 2023-05-11 16:19:28 +0200

  * Add new `skip` keyword to let unit items efficiently skip over
    uninteresting data. (Robin Sommer, Corelight)

    From the documentation:

        For cases where your parser just needs to skip over some data, without
        needing access to its content, Spicy provides a ``skip`` keyword to
        prefix corresponding fields with:

        .. spicy-code:: skip.spicy

            module Test;

            public type Foo = unit {
                x: int8;
                 : skip bytes &size=5;
                y: int8;
                on %done { print self; }
            };

        ``skip`` works for all kinds of fields but is particularly efficient
        with ``bytes`` fields, for which it will generate optimized code
        avoiding the overhead of storing any data.

        ``skip`` fields may have conditions and hooks attached, like
        any other fields. However, they do not support ``$$`` in
        expressions and hooks.

        For readability, a ``skip`` field may be named (e.g., ``padding: skip
        bytes &size=3;``), but even with a name, its value cannot be accessed.

  * Deprecate parsing `void` fields with attributes. (Robin Sommer, Corelight)

    `skip` supersedes the existing void-with-attributes parsing. For
    now we continue to support that by internally mapping it over to
    the new mechanism, but will eventually remove it. `void` without
    attributes (for not parsing anything) remains supported.

  * GH-1051: Support `&requires` with `void` fields. (Robin Sommer, Corelight)

1.8.0-dev.133 | 2023-05-10 13:11:54 +0200

  * Skip failure notifications for unsuccessful non-error runs. (Benjamin Bannier, Corelight)

1.8.0-dev.131 | 2023-05-09 11:15:01 +0200

  * Fix readthedocs build. (Benjamin Bannier, Corelight)

1.8.0-dev.129 | 2023-05-08 15:20:11 +0200

  * Bump pre-commit hooks. (Benjamin Bannier, Corelight)

  * Slightly tweak existing feature requirements test. (Benjamin Bannier, Corelight)

  * GH-1089: Allow to use `offset()` without enabling full random-access support. (Benjamin Bannier, Corelight)

    In particular this allows to use `offset()` while still keeping trimming
    enabled.

  * GH-1421: Store numerical offset in units instead of iterator for position. (Benjamin Bannier, Corelight)

    We will use this in a subsequent patch to make `offset()` available
    without having to enable full support for random-access.

    Closes #1421.

1.8.0-dev.124 | 2023-04-28 11:41:21 +0200

  * GH-1436: Make sure `Bytes::sub` only throws HILTI exceptions. (Benjamin Bannier, Corelight)

    We previously would directly invoke `Base::substr` which can throw
    and leak `std::out_of_range`. We now make sure to only throw HILTI
    exceptions from that function.

    Closes #1436.

  * Implement all `Bytes::sub` in terms of bytes offsets. (Benjamin Bannier, Corelight)

    We will use this in a subsequent patch which cleans up exception
    handling from `std::string::substr`.

1.8.0-dev.121 | 2023-04-27 15:19:08 +0200

  * Suppress uninteresting compiler diagnostic in test stderr diff. (Benjamin Bannier, Corelight)

  * Fix gcc-13 warnings about possibly dangling references. (Benjamin Bannier, Corelight)

  * Roll back 3rdparty/reproc upgrade. (Benjamin Bannier, Corelight)

  * Silence gcc-13 warning for reproc. (Benjamin Bannier, Corelight)

  * Add missing include for `uint64_t`. (Benjamin Bannier, Corelight)

1.8.0-dev.115 | 2023-04-24 10:59:27 +0200

  * Make sure Spicy runtime config is initialized after `spicy::rt::init`. (Benjamin Bannier, Corelight)

    We previously wouldn't explicitly initialize the Spicy runtime
    configuration but instead rely on `configuration::get` (which implicitly
    initializes it) to be called somewhere in the driver. This makes it hard
    to use parsers outside of our drivers, we e.g., saw nullptr derefs in
    our fuzzer binaries.

    With this patch we force initialization in `spicy::rt::init` which is
    the documented way to ensure all runtime dependencies are set up
    correctly.

  * Roll back 3rdparty/reproc downgrade. (Benjamin Bannier, Corelight)

    This appears to have been downgraded to a two year old version by
    accident.

1.8.0-dev.112 | 2023-04-24 10:10:57 +0200

  * Move exception creation out of reference getters. (Benjamin Bannier, Corelight)

    This helps reduce the size of the changed functions which might allow
    more inlining.

  * Make it cheaper to access a `ValueReference`. (Benjamin Bannier, Corelight)

    Since we know that if a `ValueReference` holds a raw pointer it must
    always be valid we can inline some paths from `_get` directly into
    `_safeGet`. We also reduce the size of `_get` from the knowledge that we
    must hold exactly two alternatives, and replace uses of `variant`
    `std::get` (which sets up exception handling) with `std::get_if` which
    is slightly cheaper at runtime.

  * Disallow constructing ValueReferences from nullptrs. (Benjamin Bannier, Corelight)

    We will use this in a subsequent patch which makes use of the fact that
    `self` references are always valid.

1.8.0-dev.108 | 2023-04-24 10:10:26 +0200

  * Add fast pass for iterator difference in `offset()`. (Benjamin Bannier, Corelight)

    When generating code for `offset()` we previously would use the
    difference of safe iterator which has extra code to make sure the two
    iterators belong to the same stream. This check can never fail when
    computing `offset()` so use a fast pass in that case.

1.8.0-dev.106 | 2023-04-21 12:03:21 +0200

  * Bring new structure of safe/unsafe getters for global/config state to Spicy. (Robin Sommer, Corelight)

  * Fix ordering issue in spicy-driver when shutting down runtime. (Robin Sommer, Corelight)

  * Switch some more places to use unsafe getters for global/config state. (Robin Sommer, Corelight)

  * Speed up access checks for `optional`. (Robin Sommer, Corelight)

  * Provide unsafe fastpath to access global state. (Robin Sommer, Corelight)

  * Move runtime configuration out of central global state. (Robin Sommer, Corelight)

  * Reduce stack checking overhead. (Robin Sommer, Corelight)

  * Reduce overhead of profiler operations when not needed. (Robin Sommer, Corelight)

  * Add fast path to `View::extract()`. (Robin Sommer, Corelight)

  * Inline a couple more stream methods. (Robin Sommer, Corelight)

1.8.0-dev.95 | 2023-04-19 10:13:12 +0200

  * Fix gcc-13 warning about pessimizing move. (Benjamin Bannier, Corelight)

1.8.0-dev.93 | 2023-04-14 10:39:02 +0200

  * Implement `View::data` with `Bytes::append`. (Benjamin Bannier, Corelight)

  * Avoid unneeded temporaries when appending `View` to `Bytes`. (Benjamin Bannier, Corelight)

    We used `View::data` to constructs a new `Bytes` to append from. Since
    we cannot reuse the temporary in the append but have to copy the data
    this introduced unneeded overhead. This patch inlines the low-level
    `View` block iteration into `Bytes::append`.

    I looked at this coming from zeek/spicy-dns#7, and with that particular
    test case where many small views are appended the speedup is only ~1.5%.
    I suspect the speedup to be bigger the bigger the view we append.

    In order to amortize reallocation costs I also looked into aggressively
    reserving capacity in the different `append` overloads, e.g., if the new
    size is bigger than the current capacity, reserve double the needed
    capacity in anticipation of more append operations to come. This did not
    make a noticeable difference, probably due to the underlying
    `std::string::append` already doing something similar.

  * Bump Zeek docs. (Benjamin Bannier, Corelight)

1.8.0-dev.89 | 2023-04-12 19:40:28 +0200

  * Pin btest version on ubuntu-16 in CI. (Benjamin Bannier, Corelight)

    The recently released btest-1.0 requires python-3.7 while this system
    has only python-3.5.

1.8.0-dev.87 | 2023-04-12 17:37:26 +0200

  * Bump copyright dates in license headers. (Benjamin Bannier, Corelight)

  * Use diffutils instead of BusyBox `diff` in Alpine CI. (Benjamin Bannier, Corelight)

    The version of `btest` recently published to pypi uses flags which are
    unsupported by the BusyBox `diff` binary which ships with Alpine by
    default, see https://github.com/zeek/btest/issues/92. Use `diffutils`
    instead.

  * Add missing license headers. (Benjamin Bannier, Corelight)

  * Use safe helper to get runtime configuration. (Benjamin Bannier, Corelight)

    We previously would directly access a value from the configuration in
    the global state in functions for `accept_input` and `decline_input`.
    This is only safe if the configuration was created previously, but would
    read uninitialized memory if e.g., a parser using these functions is
    embedded in code not doing that.

    With this patch we use a safe accessor for the configuration which
    ensures that it always contains a good value.

1.8.0-dev.81 | 2023-04-12 09:11:00 +0200

  * GH-1416: Fix build with gcc13 development snapshot. (Benjamin Bannier, Corelight)

    This patch bumps 3rdparty/reproc to a version including
    DaanDeMeyer/reproc@0b23d88894ccedde04537fa23ea55cb2f8365342
    so Spicy can be build with a gcc13 development snapshot. We do not bump
    to the latest development snapshot since
    DaanDeMeyer/reproc@51540e1d1f7be2a82f9f4ae80c41232205b6b0ab changed the
    process ownership semantics which would require changes on our side; we
    will tackle that once upstream has committed to an API in a release.

    This patch does not add CI for gcc13 since it is not yet released.

1.8.0-dev.79 | 2023-04-07 09:31:18 +0200

  * Avoid some redundant location tracking code. (Robin Sommer, Corelight)

  * Emit Spicy globals as true C++ globals by default. (Robin Sommer, Corelight)

    This improves runtime performance. We retain the original approach
    through an option: `--cxx-enable-dynamic-globals` now switches back to
    managing globals through dynamically allocated structs, with each
    runtime context creating separate instances.

  * Fix compiler warnings. (Robin Sommer, Corelight)

    These have started appearing with recent clangs:

    ```
    warning: variable 'yynerrs_' set but not used
    ```

  * Inline `stream::View::size()` and `stream::View::~View()`. (Robin Sommer, Corelight)

  * Move runtime location tracking to dedicated TLS variable. (Robin Sommer, Corelight)

  * Remove custom location tracking that some runtime types were using. (Robin Sommer, Corelight)

  * Apply more fiber tuning. (Robin Sommer, Corelight)

1.8.0-dev.70 | 2023-04-05 10:38:03 +0200

  * Bump Zeek docs. (Benjamin Bannier, Corelight)

  * Preserve indention when extracting comments in doc generation. (Benjamin Bannier, Corelight)

1.8.0-dev.66 | 2023-04-03 14:28:14 +0200

  * Fix signed-unsigned comparison. (Benjamin Bannier, Corelight)

1.8.0-dev.64 | 2023-04-03 09:52:25 +0200

  * Document how to check whether an `optional` value is set. (Benjamin Bannier, Corelight)

  * Fix docs reference typo. (Benjamin Bannier, Corelight)

  * Bump Zeek docs. (Benjamin Bannier, Corelight)

1.8.0-dev.59 | 2023-03-23 15:51:45 +0100

  * Install missing dependency for running with clang-15. (Benjamin Bannier, Corelight)

    Between clang-15.0.6 upstream broke out the runtime libraries needed for
    e.g., ASAN into a separate, optional package. Install it as well.

  * Remove unused CI setups for packages. (Benjamin Bannier, Corelight)

1.8.0-dev.56 | 2023-03-23 14:15:40 +0100

  * Fix potential double accounting for units in profiling output.
    (Robin Sommer, Corelight)

1.8.0-dev.54 | 2023-03-23 11:49:32 +0100

  * GH-1396: Fix regex performance regression introduced by constant
    folding. (Robin Sommer, Corelight)

  * Cache regular expressions to compile each only once. (Robin
    Sommer, Corelight)

1.8.0-dev.51 | 2023-03-23 11:09:17 +0100

  * GH-1399: Guard access to unit `_filters` member with feature flag. (Benjamin Bannier, Corelight)

    The unit `_filters` member is accessed by most low-level functions
    extracting data from the input, but its value is only interesting if a
    unit actually supports filters. We had previously annotated `_filters`
    with a requirement for filter support so any use of it with trigger
    enabling support. By guarding access with a feature flag we can detect
    such dependent uses in a subsequent patch.

  * Extend detection of conditionally used features in optimizer. (Benjamin Bannier, Corelight)

    We previously were only able to detect conditionally used features if
    the feature flag was an `if` condition. In order to support
    conditionally used features in expressions (as opposed to statements) we
    add support for detecting their use under a ternary operator with this
    patch.

  * Add support for folding ternary operator uses in optimizer. (Benjamin Bannier, Corelight)

1.8.0-dev.47 | 2023-03-23 10:21:35 +0100

  * Add runtime profiling infrastructure.  (Robin Sommer, Corelight)

    This add an option `-enable-profiling` to the HILTI and Spicy
    compilers. Use of the option does two things: (1) it sets a flag
    enabling inserting additional profiling instrumentation into
    generated C++ code, and (2) it enables using instrumentation for
    recording profiling information during execution of the compiled
    code, incl. dumping out a profiling report at the end. The
    profiling information collected includes time spent in HILTI
    functions as well as for parsing Spicy units and unit fields.

  * Fix parser generator to not store any parsed items for transient
    containers. (Robin Sommer, Corelight)

  * Disallow using `$$` with anonymous containers. (Robin Sommer, Corelight)

    Transient containers don't store their values, which means `$$`
    produces an always-empty container. Disallowing its use avoids simple
    mistakes.

  * Fiber tuning (Robin Sommer, Corelight)

    - Tune buffer memory management during stack switching.
    - Tune fibers' stack buffer management for less memory activity.
    - Track high-level water mark for stack size needed by fibers.

1.8.0-dev.30 | 2023-03-22 15:58:42 +0100

  * Update docs for changes in spicy-plugin. (Benjamin Bannier, Corelight)

1.8.0-dev.28 | 2023-03-10 11:22:48 +0100

  * Fix stringification of enum values. (Benjamin Bannier, Corelight)

    If an enum type or label used a C++ reserved identifier we would
    previously stringify to name normalized for use in C++. Since we are
    generating strings this is not needed and also confusing as stringified
    names deviate from what the user wrote in the input.

    With this patch we emit names which directly correspond to input
    Spicy/HILTI names and labels.

  * GH-1394: Fix C++ normalization of generated enum values. (Benjamin Bannier, Corelight)

    Closes #1394.

1.8.0-dev.25 | 2023-03-08 16:27:27 +0100

  * Fix typo in types documentation (tbfhg)

1.8.0-dev.23 | 2023-03-08 11:38:16 +0100

  * Only run lint CI task from scheduled builds or PRs. (Benjamin Bannier, Corelight)

1.8.0-dev.21 | 2023-03-07 09:44:38 +0100

  * Avoid expensive checked iterator for internal `Bytes` iteration. (Benjamin Bannier, Corelight)

    We previously would go through a checked iterator to iterate a `Bytes`
    instance. Since none of the iterators were user-provided this was always
    safe, but incurred the performance overhead for checking.

    With this patch we iterate the underlying bytes directly.

  * GH-1390: Initialize `Bytes` internal control block for all constructors. (Benjamin Bannier, Corelight)

    `Bytes` inherits constructors from its base class. We previously would
    not initialize the internal control block for these constructors so some
    operations involving iterators on such `Bytes` instances would have
    failed, even thought the controlled object was still valid.

    With this patch we always set up the control block, no matter which
    constructor is used.

    Closes #1390.

1.8.0-dev.18 | 2023-03-07 09:43:59 +0100

  * Do not force locale on users of libhilti. (Benjamin Bannier, Corelight)

    Setting a locale from a library can interfere negatively with the rest
    of the application. We previously would set a locale inside
    `hilti::rt::init`, but we probably do not need it at all.

1.8.0-dev.16 | 2023-03-06 16:24:55 +0100

  * Cache HILTI C++ compilations with ccache in CI. (Benjamin Bannier, Corelight)

1.8.0-dev.14 | 2023-03-06 16:24:20 +0100

  * GH-1386: Prevent internal error when passed invalid context. (Benjamin Bannier, Corelight)

    We previously would trigger an internal error when encountering
    `self.context()` in a unit which declared a `%context` to something
    which was not a type. With this patch we now reject such calls.

    We picked a simplified implementation strategy: when querying a `Unit`'s
    `contextType` we still return an optional type to handle both units
    without context as well as units where %context does not point to a
    type; if a context is present but does not refer to a type we return an
    empty result. With that we potentially trigger two errors, from
    validating that `%context` refers to a type, and from validating that
    the unit had a `%context` when using `self.context()`, i.e., if the
    `%context` is invalid we continue parsing as if the unit had no
    `%context` at all. The alternative would have been to return e.g., a
    `Result<optional_ref<Type>>` from `contextType` to capture both the
    unset as well as invalid case and move some of the validation into
    `contextType` itself; this still would have led to a lot of manual
    unwrapping in users.

    Closes #1386.

1.8.0-dev.12 | 2023-02-24 17:13:42 +0100

  * Fix potential use-after-move bug. (Benjamin Bannier, Corelight)

1.8.0-dev.10 | 2023-02-22 15:42:04 +0100

  * Switch `SafeInt` to current upstream. (Robin Sommer, Corelight)

1.8.0-dev.8 | 2023-02-22 11:16:08 +0100

  * Set reupload_on_changes for ccache caches. (Arne Welzel, Corelight)

  * Build centos stream again for PRs, disable clang-12 debug. (Benjamin Bannier, Corelight)

1.8.0-dev.5 | 2023-02-20 14:31:21 +0100

  * Fix sign-unsigned comparison. (Benjamin Bannier, Corelight)

  * GH-169: Build debian-11 instead of centos stream 8 in CI. (Benjamin Bannier, Corelight)

  * Update docs for removal of fedora-35, addition of fedora-36. [skip CI] (Benjamin Bannier, Corelight)

  * Add placeholder section for 1.8 to NEWS. (Benjamin Bannier, Corelight)

1.7.0 | 2023-02-13 17:41:31 +0100

  * Release 1.7.0.

1.7.0-dev.124 | 2023-02-13 17:41:08 +0100

  * Revert "Make all CI tasks greedy." (Benjamin Bannier, Corelight)

    This reverts commit 3176c2cadc71aaf4c0c3dd28d37c1d465323da8c.

  * Revert "Decrease excessive RAM on some CI platforms." (Benjamin Bannier, Corelight)

    This reverts commit 6ce665b9f3dc7f3630167e1bdf3f752ce3647dcf.

1.7.0-dev.121 | 2023-02-13 14:47:21 +0100

  * GH-1310: Fix ASAN false positive with GCC. (Robin Sommer, Corelight)

  * Skip clang-specific ASAN flags with other compilers. (Robin Sommer, Corelight)

  * Don't instantiate a debug logger if we aren't going to debug log. (Robin Sommer, Corelight)

  * Simplify runtime library's extract methods. (Robin Sommer, Corelight)

  * Shortcut some stream offset computations. (Robin Sommer, Corelight)

  * GH-1345: Apply alternative fix for #1345. (Robin Sommer, Corelight)

1.7.0-dev.112 | 2023-02-09 11:03:09 +0100

  * Make `printParserState` cheaper to call if debug logging is disabled. (Benjamin Bannier, Corelight)

1.7.0-dev.110 | 2023-02-03 15:57:39 +0100

  * https://github.com/zeek/spicy-plugin/issues/134: Search
    `HILTI_CXX_INCLUDE_DIRS` paths before default include paths.
    (Robin Sommer, Corelight)

  * Support Zeek-style documentation strings in Spicy source code. (Robin Sommer, Corelight)

    This adds support for extracting and retaining Zeek-style
    documentation strings (`##`, `##!`) inside Spicy source code to any
    top-level AST nodes where it's appropriate: if they are prefixed with
    a block of such comments, that information is retained along with the
    node for later access.

    The code printer (`-p`) learns to emit doc strings coming with the
    nodes it's printing out.

  * Bump Zeek docs. (Benjamin Bannier, Corelight)

1.7.0-dev.105 | 2023-02-01 11:17:25 +0100

  * Documentation updates. (Robin Sommer, Corelight)

        - Documenting the new type `export` feature for Zeek.
        - Update Zeek docs for JIT removal and `accept_input`/`decline_input`.

1.7.0-dev.102 | 2023-01-31 14:02:45 +0100

  * GH-1367: Use unique filename for all object files generated during JIT. (Benjamin Bannier, Corelight)

  * Bump Zeek docs. (Benjamin Bannier, Corelight)

  * Point doc repo links to latest release instead of `main`. (Benjamin Bannier, Corelight)

  * Fix installation links for macos platforms. (Benjamin Bannier, Corelight)

  * Bump latest release in docs. (Benjamin Bannier, Corelight)

1.7.0-dev.95 | 2023-01-27 12:00:41 +0100

  * Streamline runtime exception hierarchy. (Robin Sommer, Corelight)

    We were inconsistent in the structure of exceptions that could
    happen during runtime, making it hard for host applications for
    consistently handle trouble. We tweak this, as follows:

        - We now only have two general subcategories of exceptions under the
          top-level `hilti::rt::Exception`:

            - `RuntimeError`: non-fatal errors that mean a host
              applications cannot complete the current line of processing,
              but can otherwise continue. This is the main exception that
              host applications should catch, and then proceed
              appropriately.

            - `UsageError`: fatal errors that indicate something about
              using the runtime environment was fundamentally wrong, and a
              host application should abort.

        - Almost all other exceptions are now derived from `RuntimeError`,
          including Spicy's `ParseError` and `RecoverableFailure`, so that
          they can treated consistently.

1.7.0-dev.93 | 2023-01-27 11:57:58 +0100

  * Search user module paths before system paths. (Robin Sommer, Corelight)

    With, e.g., `spicyz -L /x/y/z`, we used to append the given module
    path to the internal system paths, so that it was searched last. This
    change turns around the priority to search `/x/y/x` first, before the
    internal ones. That seems both more useful and more intuitive.

  * Provide ability for host applications to initiate runtime's
    module-pre-init phase manually. (Robin Sommer, Corelight)

1.7.0-dev.89 | 2023-01-27 11:44:31 +0100

  * Add DPD-style `spicy::accept_input()` and `spicy::decline_input()`. (Robin Sommer, Corelight)

    This generalizes the accepting/decline input from being a Zeek thing to
    being part of Spicy. Any Spicy grammar can call these functions as to
    tell its host application if it recognizes the input format. The host
    application can (but doesn't have to) install callbacks to receive
    that information and react appropriately. For example, the Zeek
    plugin will just forward it to Zeek's standard DPD hooks.

    For testing, `spicy-driver` gets a new option `--require-accept`
    that lets it exit with success only if `spicy::accept()` has
    been called, and  `spicy::decline_input()` hasn't.

    Note: The naming of the two functions is avoid conflicts with the
    existing notion of synchronization's confirm/reject.

  * Add a global configuration object to the Spicy runtime. (Robin Sommer, Corelight)

    We didn't have a mechanism yet for a host application to configure the
    runtime library's behavior. This follows what we do on the HILTI-side
    to provide that capability.

1.7.0-dev.86 | 2023-01-27 11:35:35 +0100

  * Add driver option to output full set of generated C++ files. (Robin Sommer, Corelight)

    `-x <prefix>` now writes out all generated C++ code as
    `<prefix>_*.cc`, also setting the internal C++ namespace prefix to
    `<prefix>`.

  * Add test for sink's paying attention to linker scopes with static C++ code. (Robin Sommer, Corelight)

  * Make test helper scripts `cxx-*` usable with Spicy code as well. (Robin Sommer, Corelight)

  * Provide new HILTI keyword `$scope` to access linker scope. (Robin Sommer, Corelight)

  * Prefix the linker scope with the internal CXX namespace. (Robin Sommer, Corelight)

  * Fix a couple of places hardcoding the internal C++ namespace prefix. (Robin Sommer, Corelight)

  * Prefix the linker symbols for versioning with C++ namespace. (Robin Sommer, Corelight)

1.7.0-dev.78 | 2023-01-27 11:16:01 +0100

  * GH-1123: Support arbitrary expression as argument to type
    constructors, such as `interval(...)`. (Robin Sommer, Corelight)

    So far we used constructor expressions like `interval(...)` to create
    *constants* of the given type, and for that we required that the
    argument was a single atomic value. The result was that these
    constructor expressions were really more like literals for the
    corresponding types than "normal" expressions. While that's useful
    internally, it's confusing to the user. This commits changes that
    to support arbitrary expressions as arguments.

    To make this change without loosing a way to create actual constants
    (which we need at some places), we implement this in two stages: the
    parser initially turns such `interval(...)` expressions into call
    operators[1] that the types now define for each desired constructor. We
    then extend the normalizer with a new constants folding pass that
    turns the call expressions into constants if possible.

    The new constant folder remains limited to the cases we need for
    this use case, but we'll be able to expand that later to  more
    general folding.

  * Fix bug in cast from `real` to `interval`. (Robin Sommer, Corelight)

  * Add anchors to FAQ items in docs [skip CI]. (Benjamin Bannier, Corelight)

1.7.0-dev.53 | 2023-01-25 09:39:02 +0100

  * Add stringification to UnitContext. (Benjamin Bannier, Corelight)

1.7.0-dev.51 | 2023-01-25 09:36:56 +0100

  * Remove potential race during JIT when using `HILTI_CXX_COMPILER_LAUNCHER`. (Benjamin Bannier, Corelight)

1.7.0-dev.49 | 2023-01-20 10:41:46 +0100

  * GH-1349: Fix incremental regexp matching for potentially empty results. (Robin Sommer, Corelight)

    When feeding data incrementally into the regexp matcher, we could run
    into an assert trying to ensure that the end of a match wasn't in the
    previous chunk already. Turns out, however, that *is* possible with
    empty matches that are determined only later (and maybe it's possible
    in other cases, too). To fix this, we need to account for potentially
    backtracking into data from a previous round. This could be a bit
    surprising to callers of the `advance()` method, but should
    work as long as they aren't trimming the stream in between (which in
    our use cases shouldn't be the case).

    Closes #1349.

  * Install full nlohmann JSON header as well. (Benjamin Bannier, Corelight)

    We previously would only install the nlohmann header with forward
    declaration while in `hilti/rt/json.h` we include the full header. With
    that it was impossible to include this header in user code.

    This patch makes sure we install the full header as well.

1.7.0-dev.45 | 2023-01-19 09:46:14 +0100

  * Remove dead code. (Benjamin Bannier, Corelight)

  * Remove performance pessimization introduced in `84fb4f21f494`. (Benjamin Bannier, Corelight)

    When exposing this function directly we made the argument always
    evaluate. This now causes us to compute strings for logging even if they
    are never emitted.

    Undo this change as it is actually not needed for the cleanup it was
    initially intended for.

  * GH-1345: Fix pathological performance for `_haveEod` on highly chunked streams. (Benjamin Bannier, Corelight)

    This function was identified as a performance bottleneck when parsing
    `bytes` with a huge size from a stream with many chunks. The reason it
    performed so poorly was due to its use of `unsafeEnd`: since `unsafeEnd`
    creates an unsafe iterator from `end()` it always needs to compute the
    chunk the safe iterator pointed to; if the stream contains many chunks
    this causes the seen poor performance.

    Since we only access `offset` use safe iterators instead. This makes it
    much cheaper to call this function so we can safely use it during
    parsing.

    Since this change only affects performance we do not add a test case.
    For a benchmark see #1345.

    Closes #1345.

  * Remove EOL fedora-35, add fedora-37. (Benjamin Bannier, Corelight)

  * Bump Zeek version in zeek-plugin CI job. (Benjamin Bannier, Corelight)

  * Bump Zeek docs. (Benjamin Bannier, Corelight)

1.7.0-dev.37 | 2023-01-09 16:10:40 +0100

  * Decrease excessive RAM on some CI platforms. (Benjamin Bannier, Corelight)

  * Make all CI tasks greedy. (Benjamin Bannier, Corelight)

  * Rerun BTest up to three times in CI. (Benjamin Bannier, Corelight)

1.7.0-dev.33 | 2023-01-05 17:33:00 +0100

  * Update repo to not publish Docker images anymore. (Benjamin Bannier, Corelight)

1.7.0-dev.31 | 2023-01-05 15:38:13 +0100

  * GH-1326: Generate proper runtime types for enums. (Benjamin Bannier, Corelight)

  * Allow default for `vector::Allocator` with some non-class types. (Benjamin Bannier, Corelight)

  * Remove unused define. (Benjamin Bannier, Corelight)

1.7.0-dev.25 | 2023-01-04 10:26:33 +0100

  * GH-1330: Reject uses of imported module IDs as expression. (Benjamin Bannier, Corelight)

  * Fix test `Backtrace.comparison` for aggressive constant folding. (Benjamin Bannier, Corelight)

1.7.0-dev.22 | 2022-12-14 13:25:57 +0100

  * Push container images to Docker Hub zeek org, too. (Arne Welzel, Corelight)

1.7.0-dev.20 | 2022-12-14 13:24:47 +0100

  * Bump Cirrus macos CI. (Benjamin Bannier, Corelight)

  * Fix test `Backtrace.comparison` for aggressive constant folding. (Benjamin Bannier, Corelight)

  * Remove workaround for outdated Homebrew db on Cirrus CI. (Benjamin Bannier, Corelight)

1.7.0-dev.16 | 2022-12-14 10:29:59 +0100

  * Audit previously disabled clang-tidy-15 checks. (Benjamin Bannier, Corelight)

1.7.0-dev.4 | 2022-12-13 10:56:35 +0100

  * Fix docs for enum using semicolon instead of comma. (Arne Welzel, Corelight)

1.7.0-dev.2 | 2022-12-13 10:43:41 +0100

  * GH-1322: Bring implementation of `Sink::write` in line with declaration. (Benjamin Bannier, Corelight)

  * Add placeholder section for 1.7 to NEWS. (Benjamin Bannier, Corelight)

1.6.0 | 2022-12-12 12:38:01 +0100

  * Release 1.6.0. (Benjamin Bannier, Corelight)

1.6.0-dev.147 | 2022-12-12 09:40:49 +0100

  * GH-1319: Add validation rejecting unsupported attributes on unit variables. (Benjamin Bannier, Corelight)

1.6.0-dev.145 | 2022-12-12 09:14:22 +0100

  * doc/zeek: zkg template changed and mention packet analyzers (Arne
    Welzel, Corelight)

1.6.0-dev.143 | 2022-12-12 09:02:59 +0100

  * GH-1298: Fix/clarify bitfield docs. (Robin Sommer, Corelight)

1.6.0-dev.141 | 2022-12-12 08:56:24 +0100

  * Expose new `pack`/`unpack` operators in Spicy. These provide
    low-level primitives for transforming a value into, or out of, a
    binary representations. See
    https://docs.zeek.org/projects/spicy/en/latest/programming/language/packing.html
    for more. Internally, `unpack` already existed; `pack` is new.
    (Robin Sommer, Corelight)

  * Update clang-tidy config for clang-15. (Benjamin Bannier, Corelight)

  * Use clang-15 for lint CI job. (Benjamin Bannier, Corelight)

1.6.0-dev.133 | 2022-12-07 08:44:38 +0100

  * Improve efficiency of `startsWith` for long inputs. (Benjamin Bannier, Corelight)

1.6.0-dev.131 | 2022-12-06 10:37:30 +0100

  * Drop redundant leading `::` for C++ entities in script land. (Benjamin Bannier, Corelight)

  * GH-1285: Remove implemented TODO in docs. (Benjamin Bannier, Corelight)

  * Normalize names used in `__library_type`. (Benjamin Bannier, Corelight)

  * Normalize any type `cxxID` set through C++ API. (Benjamin Bannier, Corelight)

  * Normalize `&cxxname` values to be fully qualified. (Benjamin Bannier, Corelight)

  * Expose function instead of macro for `hilti::debug`. (Benjamin Bannier, Corelight)

  * Fix duplicate prefixing of `::` for prototypes generated for `&cxxname`. (Benjamin Bannier, Corelight)

  * Bump pre-commit hooks. (Benjamin Bannier, Corelight)

1.6.0-dev.122 | 2022-12-05 09:31:26 +0100

  * GH-1311: Validate that unit using `self.context()` declares `%context`. (Benjamin Bannier, Corelight)

1.6.0-dev.120 | 2022-12-05 09:30:26 +0100

  * Explicitly type integer to boolean cast. (Benjamin Bannier, Corelight)

1.6.0-dev.118 | 2022-12-02 12:20:30 +0100

  * cmake/FindGoldLinker: Do not use gold by default (Arne Welzel, Corelight)

  * cmake/FindGoldLinker: Put -fuse-ld=gold before existing flags (Arne Welzel, Corelight)

1.6.0-dev.115 | 2022-12-01 13:59:34 +0100

  * GH-1303: Fix incorrect offset computation in `advanceToNextData`. (Benjamin Bannier, Corelight)

1.6.0-dev.113 | 2022-11-18 11:48:35 +0100

  * GH-1294: Add library function to parse an address from string or bytes. (Robin Sommer, Corelight)

        ## Parses an address from a string. The address can be in standard IPv4 or IPv6
        ## ASCII representation. The function raises ``InvalidArgument`` if the string
        ## could not be parsed.
        public function parse_address(s: string) : addr;

        ## Parses an address from a bytes instance. The address can be in standard IPv4
        ## or IPv6 ASCII representation. The function raises ``InvalidArgument`` if the
        ## string could not be parsed.
        public function parse_address(b: bytes) : addr;

  * Support overloaded functions in documentation. (Robin Sommer, Corelight)

1.6.0-dev.110 | 2022-11-16 09:43:26 +0100

  * GH-1299: Add validator for bitfield field ranges. (Benjamin Bannier, Corelight)

  * Bump doctest to 2.4.9. (Benjamin Bannier, Corelight)

1.6.0-dev.107 | 2022-11-14 09:19:03 +0100

  * Post synchronization-related debug messages in `spicy-verbose` stream. (Benjamin Bannier, Corelight)

  * Log successful synchronization. (Benjamin Bannier, Corelight)

1.6.0-dev.104 | 2022-11-07 09:54:12 +0100

  * Quote CMake string to ensure correct argument count. (Nic Boet)

  * Fix Zeek docs build for case-sensitive filesystems. (Benjamin Bannier, Corelight)

1.6.0-dev.101 | 2022-10-27 12:59:41 +0200

  * Fix fuzzer build for recently renamed spicy-tftp files. (Benjamin Bannier, Corelight)

  * Make it possible to parameterize location of fuzzer lib. (Benjamin Bannier, Corelight)

  * Update docs for changes in spicy-plugin and spicy-tftp. (Benjamin Bannier, Corelight)

1.6.0-dev.97 | 2022-10-27 08:47:59 +0200

  * Downgrade required Flex version. (Johanna Amann, Corelight)

1.6.0-dev.95 | 2022-10-27 08:47:30 +0200

  * GH-1273: Replace `Lambda` class. (Benjamin Bannier, Corelight)

1.6.0-dev.93 | 2022-10-27 08:46:22 +0200

  * Show btest progress in CI. (Benjamin Bannier, Corelight)

  * Remove configuration from JIT hash. (Benjamin Bannier, Corelight)

1.6.0-dev.90 | 2022-10-26 14:57:02 +0200

  * Remove `hilti::rt::isDebugVersion()`. (Robin Sommer, Corelight)

  * Remove `--optimize` flag. This was already a no-op, and just
    confusing/misleading. (Robin Sommer, Corelight)

  * Tie HLTOs files to the Spicy version that generated them. They
    will now fail at load time if the current Spicy version doesn't
    match what they were built with. (Robin Sommer, Corelight)

  * Embed a C function into `hilti-rt` that encodes our version into
    its name. If a host applications links against this, it will fail
    to load if the runtime version changes without a rebuild. (Robin
    Sommer, Corelight)

1.6.0-dev.81 | 2022-10-26 12:32:26 +0200

  * Documentation updates. (Robin Sommer, Corelight)

    - Document how to pull custom C++ code into Spicy.
    - Adapt Zeek instructions for zkg template updates.
    - Documentation updates for recent Zeek integration changes.

1.6.0-dev.71 | 2022-10-20 18:40:36 +0200

  * Bump justrx. (Robin Sommer, Corelight)

1.6.0-dev.68 | 2022-10-05 08:50:05 +0200

  * GH-1266: Fix wrong type for Spicy-side `self` expression. (Robin
    Sommer, Corelight)

  * GH-1261: Fix inability to access unit fields through `self` in
    `&convert` expressions. (Robin Sommer, Corelight)

  * GH-1267: Install only needed SafeInt header. (Benjamin Bannier,
    Corelight)

  * Disallow `self` as ID in declarations. (Robin Sommer, Corelight)

  * Fix accidental lookup of empty IDs. (Robin Sommer, Corelight)

  * Add helper script to diff two AST. (Robin Sommer, Corelight)

  * Documentation updates.  (Benjamin Bannier, Corelight)
    - Update doc link to commits mailing list.
    - Update version referenced in docs.
    - Add doc link to releases packages for centos8-stream.

1.6.0-dev.55 | 2022-09-27 13:22:08 +0200

  * GH-1267: Install only needed SafeInt header. (Benjamin Bannier, Corelight)

1.6.0-dev.53 | 2022-09-07 16:21:02 +0200

  * Clarify that `%context` can only be used in top-level units. (Robin Sommer, Corelight)

1.6.0-dev.50 | 2022-08-22 16:33:10 +0200

  * GH-1249: Allow combining `&eod` with `&until` or `&until-including`. (Benjamin Bannier, Corelight)

1.6.0-dev.48 | 2022-08-18 16:30:14 +0200

  * GH-1251: When decoding bytes into a string using a given character
    set, allow caller to control error handling. (Robin Sommer,
    Corelight)

    All methods taking a charset parameters now take an additional
    enum selecting 1 of 3 possible error handling strategies in case a
    character can't be decoded/represented: `STRICT` throws an error,
    `IGNORE` skips the problematic character and proceeds with the
    next, and `REPLACE` replaces the problematic character with a safe
    substitute. `REPLACE` is the default everywhere now, so that by
    default no errors are triggered.

    This comes with an additional functional change for the ASCII
    encoding: we now consistently sanitize characters that ASCII can't
    represent when in `REPLACE`/`IGNORE` modes (and, hence, by
    default), and trigger errors in `STRICT` mode. Previously, we'd
    sometimes let them through, and never triggered any errors. This
    also fixes a bug with the ASCII encoding sometimes turning a
    non-printable character into multiple repeated substitutes.

  * GH-1170: Fix contexts not allowing being passed `inout`. (Robin
    Sommer, Corelight)

  * GH-1143, GH-1220: Add coercion on assignment for optionals that
    only differ in constness of their inner types. (Robin Sommer,
    Corelight)

  * GH-1115, GH-1196: Explicitly type temporary value used by
    `&max_size` logic. (Robin Sommer, Corelight)

  * GH-1250: Fix internal errors when seeing unsupported character
    classes in regular expression. (Robin Sommer, Corelight)

  * GH-1234, GH-1238: Fix assertions with anonymous struct
    constructor. (Robin Sommer, Corelight)

  * GH-1233: Reject key types for maps that can't be sorted. (Robin
    Sommer, Corelight)

  * GH-1230: Add coercion to default argument of `map::get`. (Robin
    Sommer, Corelight)

  * GH-1248: Fix `stop` for unbounded loop. (Robin Sommer, Corelight)

1.6.0-dev.28 | 2022-07-27 08:43:04 +0200

  * Fix build with empty CMAKE_CXX_FLAGS_{DEBUG,RELEASE}. (Fabrice Fontaine)

1.6.0-dev.26 | 2022-07-26 11:15:20 +0200

  * Fix validator for field `&default` expression types for constness. (Benjamin Bannier, Corelight)

1.6.0-dev.24 | 2022-07-26 11:14:06 +0200

  * Merge branch 'topic/bbannier/doc-fixes' (Benjamin Bannier, Corelight)

  * Clarify that `&until` consumes the delimiter. (Benjamin Bannier, Corelight)

  * Add FAQ item on source locations. (Benjamin Bannier, Corelight)

  * Add example for use of `?.`. (Benjamin Bannier, Corelight)

  * Bump Zeek docs. (Benjamin Bannier, Corelight)

1.6.0-dev.19 | 2022-07-12 10:46:48 +0200

  * GH-1236: Add support for adding link dependencies via `--cxx-link`. (Benjamin Bannier, Corelight)

1.6.0-dev.17 | 2022-07-08 12:12:02 +0200

  * GH-1227: Fix code generation when a module's file could be imported through different means. (Robin Sommer, Corelight)

1.6.0-dev.15 | 2022-07-07 12:11:55 +0200

  * GH-1231: Add special handling for potential `advance` failure in trial mode. (Benjamin Bannier, Corelight)

  * Expose type `MissingData` in HILTI. (Benjamin Bannier, Corelight)

  * GH-1231: Add failing test for #1231. (Benjamin Bannier, Corelight)

  * Use default compiler on FreeBSD. (Benjamin Bannier, Corelight)

  * Update CI to freebsd-13.1. (Benjamin Bannier, Corelight)

1.6.0-dev.8 | 2022-07-06 11:40:25 +0200

  * GH-1240: Clarify docs on `SPICY_VERSION`. (Benjamin Bannier, Corelight)

1.6.0-dev.6 | 2022-07-05 14:22:35 +0200

  * Add C keywords up to C23 to identifier sanitizer list. (Benjamin Bannier, Corelight)

  * Add C++ keywords up to C++20 to identifier sanitizer list. (Benjamin Bannier, Corelight)

  * Sort list of C++ keywords to sanitize. (Benjamin Bannier, Corelight)

  * Fix doc check for renames in spicy-tftp. (Benjamin Bannier, Corelight)

  * Update doc links for latest releases. (Benjamin Bannier, Corelight)

  * Add placeholder section for 1.6 to NEWS. (Benjamin Bannier, Corelight)

1.5.0 | 2022-06-30 12:15:14 +0200

  * Release 1.5.0 (Benjamin Bannier, Corelight)

  * Explicitly request content write permission for create-release action. (Benjamin Bannier, Corelight)

  * Update NEWS file. (Benjamin Bannier, Corelight)

1.5.0-dev.131 | 2022-06-30 10:15:01 +0200

  * Extending content on zkg template. (Robin Sommer, Corelight)

1.5.0-dev.129 | 2022-06-28 12:08:25 +0200

  * GH-1224: Fix import segfault. (Robin Sommer, Corelight)

1.5.0-dev.127 | 2022-06-27 14:35:05 +0200

  * Bump doc links to 1.4.1. (Benjamin Bannier, Corelight)

  * Fix links to Fedora artifacts for releases. (Benjamin Bannier, Corelight)

  * GH-1205: Update Spicy docs for now being built into Zeek. (Benjamin Bannier, Corelight)

1.5.0-dev.123 | 2022-06-21 13:32:59 +0200

  * GH-1217: Produce `hilti::rt::Bool` when casting to boolean. (Benjamin Bannier, Corelight)

1.5.0-dev.121 | 2022-06-20 14:39:05 +0200

  * GH-1209: Provide error message to `%error` handler. (Robin Sommer, Corelight)

    We now allow to optionally provide a string parameter with
    `%error` that will receive the associated error message:

        on %error(msg: string) { print msg; }

  * GH-1206: Fix detection of recursive dependencies. (Robin Sommer, Corelight)

  * Fix internal error with not fully resolved IDs. (Robin Sommer, Corelight)

1.5.0-dev.116 | 2022-06-17 11:01:21 +0200

  * Avoid recomputing signatures. (Benjamin Bannier, Corelight)

  * Avoid computing format string if debug stream not enabled. (Benjamin Bannier, Corelight)

  * Use `unordered_set` to cache resolved types. (Benjamin Bannier, Corelight)

  * Reorder instructions to avoid performing unneeded work. (Benjamin Bannier, Corelight)

  * Avoid recomputing same information when resolving expressions. (Benjamin Bannier, Corelight)

  * Prevent excessive copies. (Benjamin Bannier, Corelight)

  * Small optimizations in `matchOverloads`. (Benjamin Bannier, Corelight)

  * Remove excessive copying in `Operator::operands`. (Benjamin Bannier, Corelight)

1.5.0-dev.107 | 2022-06-17 08:43:12 +0200

  * GH-1210: Prevent unnecessarily executable stack with GNU toolchain. (Benjamin Bannier, Corelight)

1.5.0-dev.105 | 2022-06-15 12:11:01 +0200

  * Drop support for end-of-life Fedora 33/34, add support for Fedora 35/36. (Benjamin Bannier, Corelight)

  * GH-1204: Remove potential use-after-move. (Benjamin Bannier, Corelight)

1.5.0-dev.102 | 2022-06-09 15:31:22 +0200

  * Allow changing `DESTDIR` between configure and install time. (Benjamin Bannier, Corelight)

  * GH-1183: Update docs for Discourse migration. (Benjamin Bannier, Corelight)

1.5.0-dev.99 | 2022-06-08 16:37:08 +0200

  * GH-1201: Adjust removal of symlinks on install for `DESTDIR`. (Benjamin Bannier, Corelight)

1.5.0-dev.97 | 2022-06-08 16:33:24 +0200

  * Disable building of benchmark target by default. (Benjamin Bannier, Corelight)

1.5.0-dev.95 | 2022-06-08 12:22:01 +0200

  * Prevent too early integer overflow in `pow`. (Benjamin Bannier, Corelight)

  * Make handling of sanitizer workarounds more granular. (Benjamin Bannier, Corelight)

1.5.0-dev.92 | 2022-06-07 11:53:18 +0200

  * GH-44: Update docs for spicy-plugin rename `_Zeek::Spicy` -> `Zeek::Spicy`. (Benjamin Bannier, Corelight)

1.5.0-dev.90 | 2022-06-03 13:11:14 +0200

  * GH-1187: Fix support for having multiple source modules of the
    same name. (Robin Sommer, Corelight)

    There was an underlying assumption in much of the code that a module
    name would be globally unique, meaning it could be used as index into
    maps and as seeds for C++ identifiers. However, while that's often the
    case, the `import ... from ...` syntax allow pulling in multiple
    modules all having the same name.
    This change fixes that to track a module's "scope" (i.e., the part
    after `from ...`) more thoroughly across various pieces..

  * Fix test that could break because of file name clash. (Robin
    Sommer, Corelight)

  * Fix concatenation of IDs. (Robin Sommer, Corelight)

  * Fix error message. (Robin Sommer, Corelight)

  * Fix for HILTI printing of `import` statements. (Robin Sommer, Corelight)

  * Dump ASTs with ``--dump-code`` as well. (Robin Sommer, Corelight)

1.5.0-dev.83 | 2022-06-02 09:43:07 +0200

  * Bump 3rdparty/pathfind to latest snapshot. (Benjamin Bannier, Corelight)

  * Bump 3rdparty/filesystem to 1.5.12. (Benjamin Bannier, Corelight)

  * Switch to vendored nlohmann/json. (Benjamin Bannier, Corelight)

  * Bump Zeek docs. (Benjamin Bannier, Corelight)

1.5.0-dev.78 | 2022-05-31 15:08:20 +0200

  * Stop interning Locations. (Benjamin Bannier, Corelight)

  * Avoid recomputing already known canonical ID. (Benjamin Bannier, Corelight)

  * Simplify creating of refs when building scope. (Benjamin Bannier, Corelight)

  * Speed up building of scopes. (Benjamin Bannier, Corelight)

1.5.0-dev.73 | 2022-05-31 15:07:35 +0200

  * GH-1073: Switch to ECR images in documentation. (Benjamin Bannier, Corelight)

1.5.0-dev.71 | 2022-05-30 10:24:58 +0200

  * GH-1050: Update location when entering most parser methods. (Benjamin Bannier, Corelight)

  * Add builder method to set a new location. (Benjamin Bannier, Corelight)

1.5.0-dev.68 | 2022-05-30 10:03:28 +0200

  * Suppress GCC warning about maybe uninitialized variable. (Benjamin Bannier, Corelight)

1.5.0-dev.66 | 2022-05-25 14:31:13 +0200

  * GH-1174: Do not artificially suppress uses of `(u)int8` in some cases. (Benjamin Bannier, Corelight)

1.5.0-dev.64 | 2022-05-25 14:30:37 +0200

  * GH-1190: Replace uses of deprecated `result_of` with `invoke_result`. (Benjamin Bannier, Corelight)

1.5.0-dev.62 | 2022-05-25 09:21:15 +0200

  * Bump justrx. (Benjamin Bannier, Corelight)

  * Do not build fiber benchmark target by default. (Benjamin Bannier, Corelight)

  * Suppress GCC FP warnings around freeing of non-heap objects. (Benjamin Bannier, Corelight)

  * Work around GCC incorrect state tracking involving std::optional<T*>` (Benjamin Bannier, Corelight)

1.5.0-dev.57 | 2022-05-18 14:16:12 +0200

  * Use `rt::hashCombine` instead of handwritten hash combine. (Benjamin Bannier, Corelight)

  * Drop contents of included codes from JIT state hash. (Benjamin Bannier, Corelight)

1.5.0-dev.54 | 2022-05-17 13:05:48 +0200

  * GH-1179: Cap parallelism use for JIT background jobs. (Benjamin Bannier, Corelight)

  * Move JIT background jobs to its own class. (Benjamin Bannier, Corelight)

1.5.0-dev.51 | 2022-05-16 16:37:15 +0200

  * Avoid truncation when performing int64 range check. (Benjamin Bannier, Corelight)

1.5.0-dev.49 | 2022-05-16 16:27:50 +0200

  * Fix signed/unsigned comparison warnings flagged by GCC. (Benjamin Bannier, Corelight)

  * Remove unused variables. (Benjamin Bannier, Corelight)

  * Fix linking of runtime test executables. (Benjamin Bannier, Corelight)

  * Only check whether Zeek docs are up-to-date from cron builds. (Benjamin Bannier, Corelight)

1.5.0-dev.44 | 2022-05-10 09:01:16 +0200

  * Minor edits to the Zeek Integration section (Christian Kreibich, Corelight)

1.5.0-dev.42 | 2022-05-06 11:43:58 +0200

  * GH-1171: Remove freebsd11, add freebsd13. (Benjamin Bannier, Corelight)

1.5.0-dev.40 | 2022-05-04 14:25:52 +0200

  * Enable building on armv7. (Benjamin Bannier, Corelight)

  * Remove test assumption on whether `char` is signed. (Benjamin Bannier, Corelight)

1.5.0-dev.37 | 2022-05-03 15:54:47 +0200

  * Add binary packaging mode. (Benjamin Bannier, Corelight)

1.5.0-dev.35 | 2022-04-26 11:28:26 +0200

  * GH-1114: Update test comments to reflect actual behavior. (Benjamin Bannier, Corelight)

1.5.0-dev.33 | 2022-04-21 16:35:04 +0200

  * GH-1164: Make compiler plugin initialization explicit. (Benjamin Bannier, Corelight)

  * Refresh package docs with `doc/scripts/autogen-zeek-docs`. (Benjamin Bannier, Corelight)

1.5.0-dev.30 | 2022-04-21 09:10:11 +0200

  * Bump pre-commit hooks. (Benjamin Bannier, Corelight)

  * Add fuzzing related files. (Benjamin Bannier, Corelight)

  * Refresh package docs with `doc/scripts/autogen-zeek-docs`. (Benjamin Bannier, Corelight)

1.5.0-dev.26 | 2022-04-14 07:53:47 +0200

  * Fix warnings from LGTM. (Benjamin Bannier, Corelight)

  * Bump LLVM in LGTM config. (Benjamin Bannier, Corelight)

  * Allow building with gcc-8.3. (Benjamin Bannier, Corelight)

1.5.0-dev.22 | 2022-04-13 12:18:43 +0200

  * Disable tests of lossy (u)int64_t-double conversion on 32bit. (honk)

  * Adjust signature in test. (Benjamin Bannier, Corelight)

  * Fix `chars_to_uint64` for 32bit platforms. (honk)

  * Simplify implementation of `ast::ctor::SignedInteger`. (Benjamin Bannier, Corelight)

  * Allow building on 32bit platforms. (Benjamin Bannier, Corelight)

1.5.0-dev.16 | 2022-04-12 11:03:38 +0200

  * GH-1142: Fix clang-tidy `HeaderFilterRegex`. (Benjamin Bannier, Corelight)

1.5.0-dev.14 | 2022-04-11 11:34:39 +0200

  * Fix incorrect exception for parse errors. (Benjamin Bannier, Corelight)

1.5.0-dev.12 | 2022-04-07 11:44:43 +0200

  * GH-1134: Add support for synchronize-at and synchronize-after properties. (Benjamin Bannier, Corelight)

1.5.0-dev.10 | 2022-04-07 11:44:04 +0200

  * GH-1080: Reject constant declarations at non-global scope. (Benjamin Bannier, Corelight)

  * Fix test for `Map::value_type` stringification. (Benjamin Bannier, Corelight)

1.5.0-dev.7 | 2022-03-30 15:28:35 +0200

  * GH-1154: Add stringificaton of `Map::value_type`. (Benjamin Bannier, Corelight)

1.5.0-dev.5 | 2022-03-22 14:21:43 +0100

  * GH-1150: Preserve additional permissions from umask when generating HLTO files. (Benjamin Bannier, Corelight)

1.5.0-dev.3 | 2022-03-22 10:44:59 +0100

  * Refresh package docs with `doc/scripts/autogen-zeek-docs`. (Benjamin Bannier, Corelight)

  * Fix doc gen script to allow for tuples with named fields in return type. (Benjamin Bannier, Corelight)

  * Add placeholder section for 1.5 to NEWS. (Benjamin Bannier, Corelight)

1.4.0 | 2022-03-11 11:16:37 +0100

  * Update NEWS. (Benjamin Bannier, Corelight)

  * Bump docs for spicy-plugin. (Benjamin Bannier, Corelight)

1.4.0-dev.264 | 2022-03-10 11:22:08 +0100

  * Add missing include. (Benjamin Bannier, Corelight)

  * Allow teardown of resumables even if runtime is already gone. (Benjamin Bannier, Corelight)

  * Fix clang-tidy warning about trivially copyable types in `stream.h`. (Benjamin Bannier, Corelight)

  * Remove unused method `Chunk::isCompact`. (Benjamin Bannier, Corelight)

  * Take gaps explicitly into account when recovering. (Benjamin Bannier, Corelight)

  * Add a test for error recovery on gap. (Benjamin Bannier, Corelight)

  * Use more general exception type for recoverable errors in Spicy. (Benjamin Bannier, Corelight)

  * Add a chunk type for gaps. (Benjamin Bannier, Corelight)

1.4.0-dev.255 | 2022-03-04 09:01:36 +0100

  * GH-1036: Support unit initialization through a struct constructor
    expression. (Robin Sommer, Corelight)

    This works now:

      ```
      type X = unit {
          a: uint8;
          b: uint8;
      };

      ...

      local x: X = [$a = 1, $b = 2];
      ```

  * GH-1112: Fix constness for dereferencing references. (Robin
    Sommer, Corelight)

  * Fix code markup in NEWS file. (Benjamin Bannier, Corelight)

1.4.0-dev.247 | 2022-02-23 09:43:39 +0100

  * GH-1072: Disallow enum declarations with non-unique values. (Benjamin Bannier, Corelight)

    It is unclear what code should be generated when requested to convert an
    integer value to the following enum:

        type E = enum {
            A = 1,
            B = 2,
            C = 1,
        };

    For `1` we could produce either `E::A` or `E::C` here.

    Instead of allowing that ambiguity we disallow enums with non-unique
    values with this patch.

    Closes #1072.

1.4.0-dev.245 | 2022-02-23 09:37:14 +0100

  * GH-1043: Add validation rejecting untyped variable decl from struct initializer. (Benjamin Bannier, Corelight)

1.4.0-dev.243 | 2022-02-23 09:04:22 +0100

  * GH-1096: Bump submodules and pre-commit hooks. (Benjamin Bannier, Corelight)

1.4.0-dev.236 | 2022-02-23 09:03:43 +0100

  * GH-1131: Optimize out unused hooks related to synchronization. (Benjamin Bannier, Corelight)

1.4.0-dev.234 | 2022-02-23 09:03:08 +0100

  * Fix styling of lists on RTD. (Benjamin Bannier, Corelight)

1.4.0-dev.232 | 2022-02-21 15:28:31 +0100

  * Move clang-tidy run to its own CI job. (Benjamin Bannier, Corelight)

1.4.0-dev.230 | 2022-02-21 15:27:55 +0100

  * GH-1120: Do not change a LHS to RHS when codegen'ing BuiltinFunctions. (Benjamin Bannier, Corelight)

1.4.0-dev.228 | 2022-02-18 19:12:51 +0100

  * Add support for error recovery. (Benjamin Bannier, Corelight)

  * Format unsigned integers with uppercase suffix. (Benjamin Bannier, Corelight)

  * Do not prefix rendered parse errors with `parse error:`. (Benjamin Bannier, Corelight)

  * Provide grammar function to return look-ahead literals for a given (Robin Sommer, Corelight)
    production.

  * Print grammar debug output only once tables have been computed. (Robin Sommer, Corelight)

  * Remove duplicated and incorrect `default` case rendering to HILTI. (Benjamin Bannier, Corelight)

1.4.0-dev.180 | 2022-02-18 12:25:18 +0100

  * Add step in `ci/run-ci` checking whether Zeek docs are up-to-date. (Benjamin Bannier, Corelight)

  * Add doc build target to check whether docs are up-to-date. (Benjamin Bannier, Corelight)

  * Remove unimplemented `deploy_docs` function in `ci/run-ci`. (Benjamin Bannier, Corelight)

1.4.0-dev.176 | 2022-02-17 14:14:06 +0100

  * GH-1122: Make sure user-visible enums are represented by `int64_t`. (Benjamin Bannier, Corelight)

1.4.0-dev.174 | 2022-02-02 20:31:53 +0100

  * Add a FAQ entry explicitly calling out Layer 2 analyzers. (Benjamin Bannier, Corelight)

1.4.0-dev.172 | 2022-01-26 12:05:12 +0100

  * Redefine noop macro case so it always requires a semicolon. (Benjamin Bannier, Corelight)

  * Be explicit about bugprone narrowing conversions. (Benjamin Bannier, Corelight)

  * Fix diagnostics from clang-tidy's bugprone category. (Benjamin Bannier, Corelight)

  * Fix diagnostics from clang-tidy's readability category. (Benjamin Bannier, Corelight)

  * Fix diagnostics from clang-tidy's performance category. (Benjamin Bannier, Corelight)

  * Fix diagnostics from clang-tidy's modernize category. (Benjamin Bannier, Corelight)

  * Fix diagnostics from clang-tidy's cert category. (Benjamin Bannier, Corelight)

  * Fix clang-tidy config. (Benjamin Bannier, Corelight)

    We previously enabled subcategories incorrectly so that much fewer tests
    than expected were run. We will fix the newly triggered diagnostics in
    follow-up patches.

  * Fix HeaderFilterRegex in clang-tidy config. (Benjamin Bannier, Corelight)

1.4.0-dev.162 | 2022-01-25 12:22:48 +0100

  * Fix documentation link to TFTP sample capture files. (Benjamin Bannier, Corelight)

  * Update documentation for removed centos releases. (Benjamin Bannier, Corelight)

    This is a documentation fixup commit for 55d7775de25c8 where we removed
    supported for EOL centos releases.

1.4.0-dev.159 | 2022-01-20 09:38:49 +0100

  * GH-1084: Fix typing of map/vector index operators. (Robin Sommer,
    Corelight)

  * GH-1079: Fix import of global constants. (Robin Sommer, Corelight)

1.4.0-dev.153 | 2022-01-13 17:12:31 +0100

  * Remove support for EOL centos releases, add centos-stream-8. (Benjamin Bannier, Corelight)

1.4.0-dev.151 | 2022-01-13 08:18:29 +0100

  * GH-1092: Strip installation artifacts. (Benjamin Bannier, Corelight)

    This removes debug symbols from installation artifacts. This leads to a
    marked reduction in their size.

  * Switch configure to use Release mode by default. (Benjamin Bannier, Corelight)

1.4.0-dev.148 | 2022-01-13 08:17:53 +0100

  * Use libc++ instead of libstdc++ on debian-9. (Benjamin Bannier, Corelight)

    In libstdc++-10-dev-10.3.0-13 Debian includes the upstream GCC patch
    2b2d97fc545 without also including at least 4f032929ac9. This leads to
    errors e.g., triggered from nlohmann/json were `&noexcept` attributes
    are used inconsistently.

    Since debian-9 should only receive security updates at this point it is
    unclear to me how such a feature change ended up being backported. At
    this point it is not clear to me whether it will see another patch
    release. For these reasons we switch the build over to use libc++
    instead which comes from the LLVM repo and should still receive reliable
    updates.

1.4.0-dev.146 | 2022-01-08 10:02:10 +0100

  * GH-1076: Do not pass errors inside sink units back up to parent.
    (Robin Sommer, Corelight)

  * GH-1077: Fix self-recursive units. (Robin Sommer, Corelight)

  * GH-1069: Allow implicit C++ conversion from value reference to
    strong reference. (Robin Sommer, Corelight)

  * Fix typos in the source and docs. (Benjamin Bannier, Corelight)

1.4.0-dev.137 | 2022-01-07 10:41:29 +0100

  * Fix typos in the source and docs. (Benjamin Bannier, Corelight)

1.4.0-dev.135 | 2021-12-22 11:04:51 +0100

  * Update docs for Zeek plugin functions with recent changes. (Robin
    Sommer, Corelight)

1.4.0-dev.133 | 2021-12-21 11:45:51 +0100

  * GH-1088: Ensure that field hooks for lists see correct offset. (Benjamin Bannier, Corelight)

  * Remove stray baselines. (Benjamin Bannier, Corelight)

  * Lint all sh scripts with checkbashisms. (Benjamin Bannier, Corelight)

1.4.0-dev.129 | 2021-12-17 12:30:33 +0100

  * GH-1087: Make `offset` return correct value even before parsing of field. (Benjamin Bannier, Corelight)

1.4.0-dev.127 | 2021-12-16 11:46:32 +0100

  * GH-1045: Decouple whether Spicy was build with ccache from ccache use during JIT. (Benjamin Bannier, Corelight)

1.4.0-dev.124 | 2021-12-15 16:54:10 +0100

  * Remove unneeded `%random-access` in tests and docs. (Benjamin Bannier, Corelight)

  * Add validation catching use of deprecated `%random-access`. (Benjamin Bannier, Corelight)

  * Enable random-access by default. (Benjamin Bannier, Corelight)

  * Generate branch for `%random-access`-dependent trimming. (Benjamin Bannier, Corelight)

1.4.0-dev.117 | 2021-12-15 16:13:53 +0100

  * GH-1068: Fix syntax for Spicy `struct` declaration. (Benjamin Bannier, Corelight)

1.4.0-dev.115 | 2021-12-15 15:51:26 +0100

  * Fix typos in variable documentation. (Robin Sommer, Corelight)

  * Document visibility for globals. (Benjamin Bannier, Corelight)

1.4.0-dev.112 | 2021-12-13 10:46:37 +0100

  * Fix zeek_plugin CI task for broken apart zeek/spicy-analyzers. (Benjamin Bannier, Corelight)

1.4.0-dev.110 | 2021-12-13 10:46:04 +0100

  * Show log for configure failures. (Benjamin Bannier)

1.4.0-dev.108 | 2021-12-13 10:43:52 +0100

  * Refresh package docs with `doc/scripts/autogen-zeek-docs`. (Benjamin Bannier, Corelight)

  * Fix docs build for split-apart zeek/spicy-analyzers. (Benjamin Bannier, Corelight)

    That analyzer repo has been broken up into specific analyzer repos. Fix
    scripts and docs on our side for that.

1.4.0-dev.105 | 2021-12-10 09:50:54 +0100

  * Fix lints from cmake-lint. (Benjamin Bannier, Corelight)

  * Reformat CMake files with cmake-format. (Benjamin Bannier, Corelight)

1.4.0-dev.102 | 2021-12-09 11:24:39 +0100

  * Avoid calculating AST dump if we do not output it. (Benjamin Bannier, Corelight)

  * Fix accidental calculation of immediately discarded debug output. (Benjamin Bannier, Corelight)

1.4.0-dev.99 | 2021-12-09 09:30:35 +0100

  * GH-1056: Fix constness check for struct parameters. (Robin Sommer,
    Corelight)

  * GH-1057: Fix code generator to emit types correctly for transitive
    dependencies. (Robin Sommer, Corelight)

1.4.0-dev.95 | 2021-12-07 11:23:17 +0100

  * Fix dependency tracking of spicy-build CMake target. (Benjamin Bannier, Corelight)

  * Fix outputs of CMake command to generate precompiled headers. (Benjamin Bannier, Corelight)

1.4.0-dev.92 | 2021-12-03 11:34:17 +0100

  * Fix a typo in container Github action filter. (Benjamin Bannier, Corelight)

1.4.0-dev.90 | 2021-12-03 08:43:55 +0100

  * GH-1059: Allow to use unit contexts in units with parameters. (Benjamin Bannier, Corelight)

  * Bump 3rdparty/filesystem to v1.5.10. (Benjamin Bannier, Corelight)

  * Bump gitlint pre-commit hooks to v0.17.0. (Benjamin Bannier, Corelight)

  * Bump 3rdparty/benchmark to v1.6.0. (Benjamin Bannier, Corelight)

1.4.0-dev.85 | 2021-12-02 12:22:48 +0100

  * Do not by default run Docker workflow for PRs. (Benjamin Bannier, Corelight)

  * Bump published container to zeek-4.0.4. (Benjamin Bannier, Corelight)

  * Use releases instead of HEAD for packages in published container image. (Benjamin Bannier, Corelight)

  * Also push container image to ECR. (Benjamin Bannier, Corelight)

  * Unify Docker build jobs for tags and for latest. (Benjamin Bannier, Corelight)

  * Use variable to store Docker GH action build args. (Benjamin Bannier, Corelight)

  * Also build Docker image in PR, but do not publish it. (Benjamin Bannier, Corelight)

  * Split build and push step for Docker GH action. (Benjamin Bannier, Corelight)

  * Simplify GH action Docker build pipeline. (Benjamin Bannier, Corelight)

  * Checkout submodules via normal checkout action. (Benjamin Bannier, Corelight)

1.4.0-dev.74 | 2021-11-19 17:31:51 +0100

  * Update documentation for the Zeek plugin's new `$packet` feature.
    (Robin Sommer, Corelight)

1.4.0-dev.72 | 2021-11-16 16:56:18 +0100

  * Fix potentially use-after-free in stringification of `stream::View`. (Benjamin Bannier, Corelight)

    This is a fixup commit for dbc4c6e0d8e where we missed a couple
    potential use-after-frees.

1.4.0-dev.70 | 2021-11-12 09:36:24 +0100

  * GH-106: Bump zkg in published Docker images. (Benjamin Bannier, Corelight)

1.4.0-dev.68 | 2021-11-12 09:35:59 +0100

  * Refresh package docs with `doc/scripts/autogen-zeek-docs`. (Benjamin Bannier, Corelight)

  * Fix lints in `doc/scripts/autogen-zeek-docs.` (Benjamin Bannier, Corelight)

  * Fix `doc/scripts/autogen-zeek-docs` for changed layout of spicy-analyzers repo. (Benjamin Bannier, Corelight)

1.4.0-dev.64 | 2021-11-12 09:35:38 +0100

  * Use clang-format pre-commit hook which installs its own `clang-format` binary. (Benjamin Bannier, Corelight)

1.4.0-dev.62 | 2021-11-09 10:21:28 +0100

  * Fix potentially use-after-free in stringification of `stream::View`. (Benjamin Bannier, Corelight)

1.4.0-dev.60 | 2021-11-09 10:19:53 +0100

  * Document structs. (Benjamin Bannier, Corelight)

  * GH-842: Add Spicy support for struct initialization. (Benjamin Bannier, Corelight)

  * Add basic tests for Spicy `struct` types. (Benjamin Bannier, Corelight)

1.4.0-dev.56 | 2021-11-05 14:55:09 +0100

  * Use compiler launcher from ccache if one was used there. (Benjamin Bannier, Corelight)

  * Use atomic operations to create temporary files during JIT. (Benjamin Bannier, Corelight)

  * Use explicit guard controlling deletion of temporary files in JIT. (Benjamin Bannier, Corelight)

  * Explicitly enforce that a module is only registered once. (Benjamin Bannier, Corelight)

  * Do not delete temporary files if requested. (Benjamin Bannier, Corelight)

  * Fix extension of precompiled headers. (Benjamin Bannier, Corelight)

  * Cache JIT results in CI. (Benjamin Bannier, Corelight)

  * Allow use of ccache during JIT. (Benjamin Bannier, Corelight)

  * Include HILTI config in JIT hash. (Benjamin Bannier, Corelight)

  * Extend hashCombine so it can combine more than two hashes. (Benjamin Bannier, Corelight)

  * Use deterministic locations for files created during JIT. (Benjamin Bannier, Corelight)

  * Avoid unneeded parameter copy. (Benjamin Bannier, Corelight)

  * Ensure cleanup of temporary driver HLTO files on errors. (Benjamin Bannier, Corelight)

  * Compute module linker scope variable from module contents. (Benjamin Bannier, Corelight)

  * Remove unused `created` field in linker version. (Benjamin Bannier, Corelight)

1.4.0-dev.40 | 2021-11-04 13:03:09 +0100

  * Fix build for RTD. (Benjamin Bannier, Corelight)

1.4.0-dev.38 | 2021-11-04 12:35:22 +0100

  * Make Spicy prefix visible for CI jobs testing spicy-[plugin|analyzers]. (Benjamin Bannier, Corelight)

1.4.0-dev.36 | 2021-11-02 13:07:00 +0100

  * Show diff if pre-commit fails in CI. (Benjamin Bannier, Corelight)

1.4.0-dev.34 | 2021-11-01 16:32:40 +0100

  * GH-1028: Add TODO to docs on unit initialization via list coercion. (Benjamin Bannier, Corelight)

1.4.0-dev.32 | 2021-11-01 13:19:49 +0100

  * GH-1030: Make sure types required for globals are declared before being used. (Benjamin Bannier, Corelight)

  * Fix bug in emission of prioritized types. (Benjamin Bannier, Corelight)

  * Fully qualify some identifiers when emitting C++. (Benjamin Bannier, Corelight)

  * Do not emit `__init_globals` if there are no globals. (Benjamin Bannier, Corelight)

1.4.0-dev.27 | 2021-10-26 15:43:54 +0200

  * GH-994: Model types in `set` and `map` `in` operators. (Benjamin Bannier, Corelight)

1.4.0-dev.25 | 2021-10-25 16:30:35 +0200

  * Prevent exception if cache directory is not readable. (Benjamin Bannier, Corelight)

1.4.0-dev.23 | 2021-10-25 16:29:53 +0200

  * Fix issues flagged by shellcheck in `./configure`. (Benjamin Bannier, Corelight)

  * Propagate failure from `cmake` up to `./configure`. (Benjamin Bannier, Corelight)

1.4.0-dev.20 | 2021-10-22 11:40:46 +0200

  * Make `-D ast-codegen` work even without `-j`. (Benjamin Bannier, Corelight)

1.4.0-dev.18 | 2021-10-22 11:39:45 +0200

  * Set `HILTI_DEBUG` to known value for tests. (Benjamin Bannier, Corelight)

  * GH-981: Emit comments for enabled features. (Benjamin Bannier, Corelight)

1.4.0-dev.15 | 2021-10-20 11:37:36 +0200

  * Fix computation of dependent features if multiple features are involved. (Benjamin Bannier, Corelight)

  * Remove workaround always emitting parser functions. (Benjamin Bannier, Corelight)

  * Register parsers depending on feature. (Benjamin Bannier, Corelight)

  * Do not remove unused functions if they are required by type features. (Benjamin Bannier, Corelight)

  * Allow to guard on a list of features. (Benjamin Bannier, Corelight)

  * Constant fold logical operations on booleans. (Benjamin Bannier, Corelight)

  * Factor out getting of bool literals in optimizer. (Benjamin Bannier, Corelight)

  * Always emit filter functionality. (Benjamin Bannier, Corelight)

  * Change `connect_mime_type` to consider public units and same linker scope units. (Benjamin Bannier, Corelight)

  * Allow registration of non-public parsers. (Benjamin Bannier, Corelight)

  * Enable sink support for non-public units as well. (Benjamin Bannier, Corelight)

1.4.0-dev.3 | 2021-10-20 08:10:39 +0200

  * Fix internal link in docs. (Benjamin Bannier, Corelight)

  * Remove fedora32 from docs, add fedora34. (Benjamin Bannier, Corelight)

  * Add placeholder section for 1.4 to NEWS. (Benjamin Bannier, Corelight)

1.3.0 | 2021-10-18 16:23:34 +0200

  * Release 1.3.0.

1.3.0-dev.133 | 2021-10-18 16:16:23 +0200

  * GH-934: Allow `$$` in place of `self` in unit convert attributes. (Benjamin Bannier, Corelight)

  * Add stringification for Keywords. (Benjamin Bannier, Corelight)

1.3.0-dev.130 | 2021-10-12 18:45:17 +0200

  * Run pip3 as superuser in macos CI. (Benjamin Bannier, Corelight)

1.3.0-dev.128 | 2021-10-12 12:12:02 +0200

  * Fix GCC 11 build error for set iterators (Christian Kreibich, Corelight)

  * Modernize CI: drop Fedora 32 (EOL since May), add Fedora 34 (Christian Kreibich, Corelight)

1.3.0-dev.125 | 2021-10-02 08:55:47 +0200

  * GH-46: Switch Spicy and HILTI parsers to use variants. (Benjamin Bannier, Corelight)

  * GH-1009: Fix overly generous generation of type information. (Robin Sommer, Corelight)

  * GH-1013: Split validation into two passes. (Robin Sommer, Corelight)

1.3.0-dev.113 | 2021-09-24 13:02:22 +0200

  * Document release artifaces for debian-11. (Benjamin Bannier, Corelight)

  * Switch release version referenced in docs to v1.2.1. (Benjamin Bannier, Corelight)

1.3.0-dev.110 | 2021-09-23 18:22:44 +0200

  * Remove duplicate rule diagnosed by Bison. (Benjamin Bannier, Corelight)

  * Avoid rendering modules if the debug stream is disabled. (Benjamin Bannier, Corelight)

1.3.0-dev.107 | 2021-09-23 15:39:31 +0200

  * Remove unneeded copies. (Benjamin Bannier, Corelight)

  * Remove unneeded unwrapping of type. (Benjamin Bannier, Corelight)

  * Cleanup optimizer after AST rework. (Benjamin Bannier, Corelight)

  * Fix execution order of hooks in test. (Benjamin Bannier, Corelight)

  * Also collect type uses from function declarations. (Benjamin Bannier, Corelight)

1.3.0-dev.101 | 2021-09-21 12:10:25 +0200

  * Major internal overhaul of the AST infrastructure. (Robin Sommer,
    Corelight)

1.3.0-dev.100 | 2021-09-17 13:18:50 +0200

  * Fix lints in rst files. (Benjamin Bannier, Corelight)

  * Bump pre-commit-hooks. (Benjamin Bannier, Corelight)

1.3.0-dev.97 | 2021-09-16 10:48:42 +0200

  * Use safe integers for runtime size functions. (Benjamin Bannier, Corelight)

    In fd5c53cb8ffd1a18aadb7d21b927bda801e264c8 we changed the size
    functions of runtime containers to use safe integers. This patch adjusts
    the remaining size functions to use safe integers as well.

1.3.0-dev.95 | 2021-09-15 11:46:16 +0200

  * More fixes for building on ARM64. (Benjamin Bannier, Corelight)

    In 89d3295f1363d8d07eda5c711740b18f31483d88 we adjusted some
    preprocessor code to enable building on arm64. That patch missed an
    update of another instance to the same conditional which we fix with
    this patch.

    Some ARM64 platforms do not seem to reliably set the `__arm64__` define,
    but instead `__aarch64__`. We now allow that as an alternative.

1.3.0-dev.93 | 2021-09-15 11:45:50 +0200

  * Fix example links for recent reorg in zeek/spicy-analyzers. (Benjamin Bannier, Corelight)

1.3.0-dev.91 | 2021-09-14 14:10:28 +0200

  * Fix parsers for bison-3.8.1. (Benjamin Bannier, Corelight)

1.3.0-dev.89 | 2021-09-13 15:13:00 +0200

  * GH-999: Unify `transform` and `filter` functions. (Benjamin Bannier, Corelight)

  * Add `insert` with hint to `Vector` and `Set`. (Benjamin Bannier, Corelight)

1.3.0-dev.86 | 2021-09-09 10:38:22 +0200

  * Precompile `libspicy.h`. (Benjamin Bannier, Corelight)

1.3.0-dev.84 | 2021-09-09 10:36:23 +0200

  * GH-941: Allow use of units with all defaulted parameters as entry points. (Benjamin Bannier, Corelight)

1.3.0-dev.82 | 2021-09-09 10:33:14 +0200

  * Disable leak tests if `ASAN_OPTIONS` are already set. (Benjamin Bannier, Corelight)

  * Correctly handle lookups for NULL library symbols. (Benjamin Bannier, Corelight)

1.3.0-dev.79 | 2021-09-09 10:32:29 +0200

  * GH-939: Document initializers for unit variables. (Benjamin Bannier, Corelight)

1.3.0-dev.77 | 2021-09-08 10:46:32 +0200

  * Make it possible to build on arm64. (Benjamin Bannier, Corelight)

    Since we currently do not test on arm64 we do not advertize it as a
    supported platform.

1.3.0-dev.75 | 2021-09-07 08:39:58 +0200

  * GH-982: Model feature constants with actual constants. (Benjamin Bannier, Corelight)

1.3.0-dev.73 | 2021-09-06 09:19:16 +0200

  * GH-988: Use safe integer for runtime container sizes. (Benjamin Bannier, Corelight)

1.3.0-dev.71 | 2021-09-03 14:36:35 +0200

  * Add collection logging to the optimizer. (Benjamin Bannier, Corelight)

  * Rename global optimizer to optimizer everywhere. (Benjamin Bannier, Corelight)

1.3.0-dev.68 | 2021-08-31 10:11:16 +0200

  * Disable constant folding for anything but feature variables. (Benjamin Bannier, Corelight)

1.3.0-dev.66 | 2021-08-31 10:10:42 +0200

  * GH-979: Properly track use of uninitialized enum values in units. (Benjamin Bannier, Corelight)

1.3.0-dev.64 | 2021-08-30 16:23:13 +0200

  * Add optimizer pass removing unused members. (Benjamin Bannier, Corelight)

  * Always emit `__sink` field for units with `%mime-type`. (Benjamin Bannier, Corelight)

  * Mark unit field `__offsets` as `&always-emit`. (Benjamin Bannier, Corelight)

  * Optimize away unused sink functionality. (Benjamin Bannier, Corelight)

  * Add annotations for sink functionality. (Benjamin Bannier, Corelight)

  * Optimize away unused filter functionality. (Benjamin Bannier, Corelight)

  * Add annotations for filter functionality. (Benjamin Bannier, Corelight)

  * Collect feature requirements from function call parameters. (Benjamin Bannier, Corelight)

  * Add support for HILTI function parameter attributes. (Benjamin Bannier, Corelight)

  * Use `&needed-by-feature` to annotate feature-dependent fields. (Benjamin Bannier, Corelight)

  * Add an optimizer helper function to get the innermost type. (Benjamin Bannier, Corelight)

  * Fix generated code to always invoke parsing of subunits. (Benjamin Bannier, Corelight)

1.3.0-dev.51 | 2021-08-26 15:34:35 +0200

  * Wrap long line in NEWS file. (Benjamin Bannier, Corelight)

1.3.0-dev.50 | 2021-08-26 12:38:15 +0200

  * Fix C++ snippets in "Custom Host Application" docs. (Benjamin Bannier, Corelight)

1.3.0-dev.48 | 2021-08-24 15:22:43 +0200

  * Add optimizer pass removing unused `%random-access` features. (Benjamin Bannier, Corelight)

  * Annotate parser fields related to `%random-access`. (Benjamin Bannier, Corelight)

  * Annotate generated parser code with feature flags. (Benjamin Bannier, Corelight)

  * Add function optimizer pass removing orphaned method implementations. (Benjamin Bannier, Corelight)

  * Add an optimizer pass performing basic constant folding. (Benjamin Bannier, Corelight)

  * Add mutator for `statement::If` removing `else` branch. (Benjamin Bannier, Corelight)

  * Prepare `statement::Block` for optimizer edits. (Benjamin Bannier, Corelight)

  * Add accessor for resolved IDs of expressions. (Benjamin Bannier, Corelight)

  * Fix type visitor to properly handle wrapped types. (Benjamin Bannier, Corelight)

  * Check environment variable `HILTI_OPTIMIZER_PASSES` for optimizer passes to enable. (Benjamin Bannier, Corelight)

  * Change optimizer tests to compare HILTI code instead of ASTs. (Benjamin Bannier, Corelight)

  * Change `-p` to print source after optimizations. (Benjamin Bannier, Corelight)

  * Clear module declaration cache after optimizations. (Benjamin Bannier, Corelight)

  * Add a visitor base class for the global optimizer. (Benjamin Bannier, Corelight)

  * Change return type of `AttributeSet::add` so it always returns some set. (Benjamin Bannier, Corelight)

  * Remove outdated TODO. (Benjamin Bannier, Corelight)

  * Prevent unneeded copy of return value. (Benjamin Bannier, Corelight)

1.3.0-dev.30 | 2021-08-23 09:36:24 +0200

  * Hardcode DockerHub org we publish images for in GH workflow. (Benjamin Bannier, Corelight)

1.3.0-dev.28 | 2021-08-19 10:38:45 +0200

  * Add FAQ item for parsing TCP conversations. (Benjamin Bannier, Corelight)

1.3.0-dev.26 | 2021-08-06 14:52:54 +0200

  * Make test type public to prevent it being optimized out. (Benjamin Bannier, Corelight)

  * Add optimizer pass to remove unused types. (Benjamin Bannier, Corelight)

  * Clear preserved nodes of modules after optimizing. (Benjamin Bannier, Corelight)

  * Factor out function to remove an AST node. (Benjamin Bannier, Corelight)

  * Remove unused state in global optimizer. (Benjamin Bannier, Corelight)

  * Rename visitor for functions in global optimizer. (Benjamin Bannier, Corelight)

  * Move optimizer collection state into visitor. (Benjamin Bannier, Corelight)

1.3.0-dev.18 | 2021-08-05 17:26:57 +0200

  * GH-970: Use publicly accessible links to Docker images. (Benjamin Bannier, Corelight)

1.3.0-dev.16 | 2021-08-05 11:24:08 +0200

  * Implement removal of unused member functions in global optimizer. (Benjamin Bannier, Corelight)

  * Explicitly mark parse functions of filters as `&always-emit`. (Benjamin Bannier, Corelight)

  * Factor out functions to compute IDs for global optimizer. (Benjamin Bannier, Corelight)

  * Remove unused function `hilti::type::Struct::ids`. (Benjamin Bannier, Corelight)

1.3.0-dev.11 | 2021-08-05 11:23:24 +0200

  * Fix flaky test `util.memory_statistics` in hilti-rt-tests. (Benjamin Bannier, Corelight)

  * Execute correct test target in validate_release_tarball Cirrus CI task. (Benjamin Bannier, Corelight)

  * Fix validate_release_tarball Cirrus CI task. (Benjamin Bannier, Corelight)

  * Only consider typical release tags when checking whether we are on a release tag. (Benjamin Bannier, Corelight)

  * GH-960: Swallow Git error messages during CMake if run from tarball. (Benjamin Bannier, Corelight)

1.3.0-dev.5 | 2021-08-03 11:04:26 +0200

  * GH-966: Emit linker glue type definitions for vector fields. (Benjamin Bannier, Corelight)

1.3.0-dev.3 | 2021-08-02 16:40:30 +0200

  * Directly maintain VERSION file in repo. (Benjamin Bannier, Corelight)

1.2.0 | 2021-07-29 16:05:44 +0200

  * Release 1.2.0. (Benjamin Bannier, Corelight)

1.2.0-dev.110 | 2021-07-29 15:01:46 +0200

  * GH-956: Fix issues around generating version information. (Benjamin Bannier, Corelight)

1.2.0-dev.105 | 2021-07-29 09:49:24 +0200

  * GH-961: Make detection of current executable more robust. (Benjamin Bannier, Corelight)

1.2.0-dev.103 | 2021-07-29 09:46:05 +0200

  * Make it possible to build Spicy as a CMake subproject. (Benjamin Bannier, Corelight)

1.2.0-dev.101 | 2021-07-23 16:47:13 +0200

  * Add Debian Bullseye (11) to CI. (Matthias Vallentin)

  * Update plugin docs for build system changes. (Robin Sommer,
    Corelight)

1.2.0-dev.94 | 2021-07-19 18:59:32 +0200

  * CI tweaks. (Robin Sommer, Corelight)

1.2.0-dev.92 | 2021-07-19 10:05:36 +0200

  * Remove spicy-plugin and spicy-analyzers submodules, along with the
    `--build-zeek-plugin` configure option. Spicy no longer support
    building them in-tree. This comes along with removing almost all
    Zeek-related pieces from Spicy proper (except for CI testing
    integration with the external Zeek plugin).

  * Add environment variable `HILTI_CXX_INCLUDE_DIRS` to specify
    additional C++ include directories when compiling generated code.
    (Robin Sommer, Corelight)

1.2.0-dev.75 | 2021-07-14 11:07:14 +0200

  * GH-940: Add runtime check for parsing progress during loops.
    (Robin Sommer, Corelight)

1.2.0-dev.71 | 2021-07-14 11:02:28 +0200

  * Add unique "linker scope" identifier to HLTO files. In Spicy, we
    use now store this linker scope along with the parsers when they
    get registered with the runtime. (Robin Sommer, Corelight)

  * GH-952: Switch the Cirrus macOS images to `*-xcode` and (still)
    update Brew for Catalina. (Robin Sommer, Corelight)

  * Make Spicy version available to subprojects. (Robin Sommer, Corelight)

  * Bump Zeek submodules. (Robin Sommer, Corelight)

  * Fix missing documentation update. (Robin Sommer, Corelight)

  * Perform optimizations in a loop. This patch changes the optimizer
    to now run a collect/prune loop until no further AST changes are
    made. (Benjamin Bannier, Corelight)

  * Disable clang-tidy `NewDeleteLeaks` check. (Benjamin Bannier, Corelight)

  * Reduce memory load for clang-tidy CI step. (Benjamin Bannier, Corelight)

  * Use clang-tidy-12 in CI. (Benjamin Bannier, Corelight)

  * Fix accidental truncating conversion in integer code. (Benjamin
    Bannier, Corelight)

1.2.0-dev.55 | 2021-07-09 08:34:26 +0200

  * Perform optimizations in a loop. (Benjamin Bannier, Corelight)

1.2.0-dev.53 | 2021-07-08 12:36:43 +0200

  * Disable clang-tidy `NewDeleteLeaks` check. (Benjamin Bannier, Corelight)

  * Reduce memory load for clang-tidy CI step. (Benjamin Bannier, Corelight)

  * Use clang-tidy-12 in CI. (Benjamin Bannier, Corelight)

  * Remove accidental truncating conversion. (Benjamin Bannier, Corelight)

  * Document for clang-tidy that field can never be unset. (Benjamin Bannier, Corelight)

  * Silence warning on unused variable in generated code. (Benjamin Bannier, Corelight)

  * Do not drop errors when move-constructing Nodes. (Benjamin Bannier, Corelight)

  * Fix range of check. (Benjamin Bannier, Corelight)

1.2.0-dev.43 | 2021-07-05 09:18:12 +0200

  * GH-918: Remove workaround for #918. (Benjamin Bannier, Corelight)

  * GH-918: GH-949: Use forward declarations in linker metadata. (Benjamin Bannier, Corelight)

  * Add registered cxx_includes to linker outputs. (Benjamin Bannier, Corelight)

1.2.0-dev.38 | 2021-06-29 09:33:19 +0200

  * Avoid name collision when codegen'ing struct element initializers. (Benjamin Bannier, Corelight)

1.2.0-dev.36 | 2021-06-29 09:31:59 +0200

  * Prefer VERSION file over git information. (Benjamin Bannier, Corelight)

  * Introduce top-level `spicy-*/` directory in release tarball. (Benjamin Bannier, Corelight)

1.2.0-dev.33 | 2021-06-28 09:23:54 +0200

  * Add optimizer code removing unused function declarations. (Benjamin Bannier, Corelight)

1.2.0-dev.30 | 2021-06-17 10:52:32 +0200

  * Remove unused variable lookup. (Benjamin Bannier, Corelight)

  * Use prefiltered candidate list when resolving operators. (Benjamin Bannier, Corelight)

  * Make `Module::declarations` more robust. (Benjamin Bannier, Corelight)

  * Fix Scope rendering to not access expired references. (Benjamin Bannier, Corelight)

  * Whitelist stringification functions in optimizer collection phase. (Benjamin Bannier, Corelight)

  * Implement removal of unused global hooks. (Benjamin Bannier, Corelight)

  * Bump spicy-plugin and spicy-analyzer submodules. (Benjamin Bannier, Corelight)

1.2.0-dev.17 | 2021-05-31 12:36:27 +0200

  * Add tests for global optimizations. (Benjamin Bannier, Corelight)

  * Add optimizer pass removing unimplemented member functions.

    This patch introduces a global pass triggered after all individual input
    ASTs have been finalized, but before we generate any C++ code. We then
    strip out any unimplemented member functions (typically Spicy hooks),
    both their definitions as well as their uses.

    In order to correctly handle previously generated C++ files which might
    have been generated with different optimization settings, we disallow
    optimizations if we detect that a C++ input file was generated by us. (Benjamin Bannier, Corelight)

  * Fix computation of unset locations. (Benjamin Bannier, Corelight)

  * Detect presence of sink hooks at compile time.

    Sink hooks are by default generated for any `public` unit. With the
    introduction of an optimizer pass to remove unused hooks we might now
    remove them and need to detect their presence when e.g., registering a
    parser.

    This patch moves from these instances from branching on whether a unit
    is `public` to directly detecting the presence of the hooks. (Benjamin Bannier, Corelight)

  * Introduce dedicated codegen step in Driver. (Benjamin Bannier, Corelight)

  * Fix test to correctly log output.

    We previously logged the output of this test to a file we never
    examined. (Benjamin Bannier, Corelight)

  * Autoupdate pre-commit hooks. (Benjamin Bannier, Corelight)

1.2.0-dev.9 | 2021-05-31 12:11:30 +0200

  * GH-913: Add support for unit switch &parse-at and &parse-from attributes. (Benjamin Bannier, Corelight)

  * Add validation of unit switch attributes.  (Benjamin Bannier, Corelight)

1.2.0-dev.5 | 2021-05-31 12:10:44 +0200

  * Make only draft releases from automation. (Benjamin Bannier, Corelight)

1.2.0-dev.3 | 2021-05-31 12:08:12 +0200

  * Adjust processes used to build Docker image. (Benjamin Bannier, Corelight)

  * Update NEWS with 1.2.0 placeholder sections. (Benjamin Bannier, Corelight)

  * Bump spicy-plugin and spicy-analyzer submodules. (Benjamin Bannier, Corelight)

  * Update NEWS. (Benjamin Bannier, Corelight)

1.1.0-dev.104 | 2021-05-20 10:32:54 +0200

  * GH-920: Track patching of Bison files through proper dependencies.
    (Benjamin Bannier, Corelight)

1.1.0-dev.102 | 2021-05-20 10:31:25 +0200

  * GH-917: Default-initialize forwarding fields without type arguments.
    (Benjamin Bannier, Corelight)

1.1.0-dev.99 | 2021-05-20 09:22:52 +0200

  * Fix missing operator<< for enums when generating debug code.
    (Robin Sommer, Corelight)

1.1.0-dev.95 | 2021-05-18 13:26:32 +0200

  * Bump version. (Benjamin Bannier, Corelight)

  * Draft 1.1 release notes. (Robin Sommer, Corelight)

1.1.0-dev.91 | 2021-05-17 14:29:24 +0200

  * Remove redundant version computation. (Benjamin Bannier, Corelight)

  * Do not update unchanged version files. (Benjamin Bannier, Corelight)

  * Remove redundant indention in generated VERSION file. (Benjamin Bannier, Corelight)

1.1.0-dev.87 | 2021-05-12 18:38:40 +0200

  * Make sure container sizes are runtime integers. (Benjamin Bannier, Corelight)

1.1.0-dev.85 | 2021-05-11 16:19:12 +0200

  * Add functions for CRC32 computation to Spicy library. (Robin
    Sommer, Corelight)

  * Add a find() method to units to search within their block of
    input. (Robin Sommer, Corelight)

  * Add support for searching a stream view backward for bytes. (Robin Sommer, Corelight)

  * Optimize implementation of Spicy's library filters a little bit. (Robin Sommer, Corelight)

  * Fix missing update to input position before running `%done` hook. (Robin Sommer, Corelight)

  * Add support for `&eod` to `void` fields. This skips all data until
    the end of the current input is encountered. (Robin Sommer,
    Corelight)

  * Add support for `&until` to `void` fields.

      Example:

        ```
        public type Foo = unit {
          : void &until=b"XYZ";
}
        ```

    This skips all data until `XYZ` is encountered. Similar to the
    `&until` attribute that `bytes` provides, the deliminator is
    extracted from the stream before continuing. (Robin Sommer,
    Corelight)

  * Add support for `&chunked` when parsing bytes data with `&until`
    or `&until_including`. (Robin Sommer, Corelight)

1.1.0-dev.74 | 2021-05-06 19:11:00 +0200

  * GH-911: Add `encode()` method to strings for conversion to bytes.
    (Robin Sommer, Corelight)

  * Let CI show any diff from clang-tidy. (Robin Sommer, Corelight)

  * Fix to run-clang-tidy. (Robin Sommer, Corelight)

1.1.0-dev.70 | 2021-05-05 11:40:32 +0200

  * Lower minimum required Python version to 3.2. (Benjamin Bannier, Corelight)

1.1.0-dev.68 | 2021-05-05 11:40:05 +0200

  * Add rough scaffolding for fuzzing parsers. (Benjamin Bannier, Corelight)

1.1.0-dev.66 | 2021-05-04 16:20:50 +0200

  * Specify python3 on cmake invocations for autogen-version. (Eli M)

  * Bump minimum python version to 3.6. (Eli M)

  * Port autogen-version to python3. (Eli M)

1.1.0-dev.62 | 2021-05-03 12:29:38 +0000

  * GH-901: Fix type resolution bug in &convert. (Robin Sommer, Corelight)

  * Extend HILTI printer to include IDs into tuple type output. (Robin
    Sommer, Corelight)

1.1.0-dev.59 | 2021-04-30 14:34:20 +0200

  * GH-882: Add Cirrus CI task for opensuse-15.2. (Benjamin Bannier, Corelight)

  * Add Dockerfile for opensuse-15.2. (Benjamin Bannier, Corelight)

  * Silence a signed-unsigned comparison warning. (Benjamin Bannier, Corelight)

1.1.0-dev.53 | 2021-04-29 13:00:26 +0000

  * Updating submodule(s).

  * Do not run clang-tidy/clang-format on Zeek submodules. (Robin
    Sommer, Corelight)

1.1.0-dev.48 | 2021-04-27 15:52:40 +0200

  * GH-882: Make spicy compile with bison 3.0. (Johanna)

1.1.0-dev.45 | 2021-04-27 08:51:29 +0000

  * Provide mktime() library function. This allows creating a `time`
    value from individual components, similar to mktime(3). (Robin
    Sommer, Corelight)

1.1.0-dev.43 | 2021-04-26 15:32:25 +0000

  * Allow to pass zlib's window bits parameter into Zlib unit type.
    (Robin Sommer, Corelight)

  * Force type information to come last in generated C++ code.

  * Fix crash on macOS ARM when writing over an existing HLTO file.
    (Robin Sommer, Corelight)

1.1.0-dev.37 | 2021-04-24 09:28:04 +0200

  * Add new Spicy runtime function `spicy::zlib_init(window_bits)`
    that initializes a `ZlibStream` for decompression with the given
    window size argument. See the zlib documentation at
    https://www.zlib.net/manual.html for more on the argument. (Robin
    Sommer, Corelight)

1.1.0-dev.35 | 2021-04-23 11:27:07 +0200

  * GH-37: Add documentation on how to skip data with `void` fields.

1.1.0-dev.33 | 2021-04-23 09:41:18 +0200

  * Reject `void` fields with names. (Benjamin Bannier, Corelight)

  * Validate supported attributes for `void` fields. (Benjamin Bannier, Corelight)

  * Add validation rejecting `$$` in hooks not supporting it. (Benjamin Bannier, Corelight)

  * GH-37: Fix handling of `&size` attribute for anonymous void fields. (Benjamin Bannier, Corelight)

1.1.0-dev.26 | 2021-04-21 13:13:11 +0200

  * GH-889: Fix hook handling for anonymous void fields.

1.1.0-dev.24 | 2021-04-19 15:30:16 +0200

  * Build Zeek plugin for benchmarking job. (Benjamin Bannier, Corelight)

1.1.0-dev.22 | 2021-04-19 09:24:14 +0000

  * GH-766: Port Spicy to Apple silicon. (Robin Sommer, Corelight)

1.1.0-dev.20 | 2021-04-19 08:58:24 +0000

  * GH-872: Fix missing normalization of enum label IDs. (Robin
    Sommer, Corelight)

  * GH-878: Fix casting integers to enums. (Robin Sommer, Corelight)

1.1.0-dev.15 | 2021-04-19 08:57:24 +0000

  * Fix versioning for development. (Robin Sommer, Corelight)

1.1.0-dev.13 | 2021-04-15 14:27:47 +0200

  * GH-844: Add support for &size attribute for unit switch statement. (Benjamin Bannier, Corelight)

1.1.0-dev.12 | 2021-04-15 14:26:45 +0200

  * GH-26: Add skip, skip-pre and skip-post properties. (Benjamin Bannier, Corelight)

  * Perform explicit validation of Spicy module properties. (Benjamin Bannier, Corelight)

1.1.0-dev.9 | 2021-04-15 13:25:20 +0200

  * Add assertion to ensure that code cannot be reached. (Benjamin Bannier, Corelight)

1.1.0-dev.7 | 2021-04-13 11:49:36 +0200

  * Disable 'create-release' and 'docker-tags' workflows for dev tags. (Benjamin Bannier, Corelight)

  * Include build status for 1.0 release branch in README. (Benjamin Bannier, Corelight)

1.1.0-dev.3 | 2021-04-12 18:23:41 +0200

  * Fix leftovers in GH actions from master -> main rename. (Benjamin Bannier, Corelight)

  * Remove build tag for release build from README for now. (Benjamin Bannier, Corelight)

1.0.0 | 2021-04-12 14:22:09 +0200

  * Fix metadata in Docker publishing workflow. (Benjamin Bannier, Corelight)

0.4.0-1694 | 2021-04-12 11:15:53 +0200

  * Do not take freebsd12 into account for cumulative CI status.

  * Documentation updates for 1.0.0 release. (Benjamin Bannier, Corelight)

  * Do not automatically create `spicy:latest` Docker tag from `main`.

0.4.0-1687 | 2021-04-08 17:07:45 +0200

  * Fix version string checking out release tags.

  * Switch versioning scheme for dev versions for 1.0.0 release. (Benjamin Bannier, Corelight)

0.4.0-1684 | 2021-04-06 10:56:32 +0000

  * Add a couple of documentation updates. (Robin Sommer, Corelight)

  * Update CI release infrastructure. (Benjamin Bannier, Corelight)

  * GH-860: Fix regexp matching across chunk boundaries. (Benjamin
    Bannier, Corelight)

  * Fix code markup in unit and sink docs. (Benjamin Bannier,
    Corelight)

  * GH-861: Add new combined operation for combined map indexing &
    assignment. (Benjamin Bannier, Corelight)

  * Add map index operator which can mutate contained returned value.
    (Benjamin Bannier, Corelight)

  * Bump pre-commit plugins. (Benjamin Bannier, Corelight)

0.4.0-1662 | 2021-03-27 08:49:01 +0000

  * Update Zeek submodules.

  * Fix doc links to package artifacts. (Benjamin Bannier, Corelight)

  * Test plugins in CI on macOS. (Benjamin Bannier, Corelight)

0.4.0-1656 | 2021-03-25 12:19:35 +0000

  * Adapt to `master` -> `main` branch renaming. (Robin Sommer, Corelight)

0.4.0-1653 | 2021-03-25 09:23:12 +0100

  * Add new (missing) autogenerated docs. (Benjamin Bannier, Corelight)

0.4.0-1650 | 2021-03-25 07:40:06 +0000

  * GH-848: Disable default building of Zeek plugin. (Benjamin
    Bannier, Corelight)

  * Update installation instructions for recent changes. (Robin
    Sommer, Corelight)

0.4.0-1640 | 2021-03-24 22:09:16 +0100

  * Clean up of CI artifact naming. (Benjamin Bannier, Corelight)

  * Drop noexcept in Sink move constructor. (Benjamin Bannier, Corelight)

0.4.0-1632 | 2021-03-23 20:01:10 +0100

  * GH-853: Fix extraction of integer bits when all bits requested.

    When calculating the extraction mask, we previously would shift an
    integer beyond its width which triggers UB. With this patch we now
    correctly detect such cases and directly return the input.

  * GH-450: Add `!in` to HILTI and Spicy parser.

0.4.0-1614 | 2021-03-18 12:27:45 +0000

  * GH-827: Bump Zeek LTS version to 4.0.0 in some CI and Docker
    images. (Benjamin Bannier, Corelight)

  * Code optimization: Pass some arguments by ref if no value is
    needed. (Benjamin Bannier, Corelight)

  * Remove duplicate key in clang-format configuration. (Benjamin
    Bannier, Corelight)

  * Updating submodules.

0.4.0-1609 | 2021-03-18 09:45:01 +0000

  * GH-817: Fix vector parsing potentially missing EOD. (Robin Sommer, Corelight)

  * GH-835: Fix some issues diagnosed by LGTM. (Benjamin Bannier, Corelight)

  * Fix `?.` operator to work with constant structs. (Robin Sommer, Corelight)

0.4.0-1603 | 2021-03-17 10:02:17 +0100

  * GH-815: Add creation of DEB and RPM packaging artifacts. (Benjamin Bannier, Corelight)

0.4.0-1575 | 2021-03-13 09:24:03 +0000

  * GH-821: Fix &max-size with constant argument. (Robin Sommer,
    Corelight)

  * GH-819: Fix unification of internal compiler arguments to not
    change order. (Robin Sommer, Corelight)

0.4.0-1568 | 2021-03-11 07:04:13 +0000

  * Fix some CMake, CI, and documtation issues. (Robin Sommer,
    Corelight)

0.4.0-1564 | 2021-03-10 11:21:02 +0000

  * Fix broken links in documentation. (Robin Sommer, Corelight)

0.4.0-1561 | 2021-03-10 08:40:38 +0000

  * Move Zeek plugin into submodule and remove all dependencies on
    Zeek from main Spicy distribution. (Robin Sommer, Corelight)

  * Add environment variable HILTI_PRINT_SETTINGS that trigger
    printing out compiler settings to stderr. (Robin Sommer,
    Corelight)

  * GH-812: Fix typos in docs. (Benjamin Bannier, Corelight)

  * GH-813: Add missing word in docs. (Benjamin Bannier, Corelight)

  * GH-569: Fix error message in validation of `real` fields.
    (Benjamin Bannier, Corelight)

  * GH-806: Fix benchmark script for move of analyzers. (Benjamin
    Bannier, Corelight)

0.4.0-1551 | 2021-03-04 09:52:00 +0100

  * GH-570: Explicitly reject comparisons of units with `==` and `!=`.

    Comparisons with `==` and `!=` are not explicitly modelled in Spicy or
    HILTI which means that while we resolved the types of the LHS and RHS,
    during codegen we just emit a `==` or `!=` which needs to be resolved.

    While most runtime types should support these operators, units do not,
    so that previously users would see a compiler error during JIT when
    comparing values of unit type with `==` or `!=`. This patch adds a
    validator which explicitly rejects such comparisons.

0.4.0-1547 | 2021-03-03 09:13:25 +0100

  * GH-798: Prevent conversions of some integers to doubles on the Zeek interface.

    While we previously provided specialized conversion functions to Zeek
    `ValPtr` these were not used for plain integer types like e.g.,
    `std::size_t` or `unsigned long`. Instead C++ overload resolution
    preferred to instead convert the integers to `double` (for which we
    provide a specialization) and them being emitted with `TYPE_DOUBLE`.

    This patch introduces an additional specialization which converts such
    naked integers to safe integers and then converts them to Zeek.

0.4.0-1545 | 2021-03-03 09:12:29 +0100

  * GH-733: Validate that `foreach` is only used with container fields.

0.4.0-1543 | 2021-03-02 07:37:21 +0000

  * Move protocol analyzers into submodule. This moves all the
    protocol analyzers into its own submodule `spicy-analyzers`, which
    doubles as a Zeek package installing them. There's a new
    `spicy-analyzers` target to pre-compile those analyzers for Zeek.
    (Robin Sommer, Corelight)

  * Zeek plugin updates (Robin Sommer, Corelight)

      - Move registration of Zeek components to load time.
      - Work around problem in Zeek's analyzer API.

   * Update pathfind to fix problem on Alpine. (Robin Sommer,
     Corelight)

0.4.0-1541 | 2021-03-02 07:22:26 +0000

  * Add support for `@if/@else/@endif` in Spicy scripts, similar to
    what EVT files already provided. This currently supports testing
    for three constants at compile time: `SPICY_VERSION`,
    `ZEEK_VERSION`, `HAVE_ZEEK`. See the documentation for more.
    (Robin Sommer, Corelight)

  * Increase CI memory for platforms that are prone to OOMing. (Robin
    Sommer, Corelight)

0.4.0-1533 | 2021-03-02 07:21:39 +0000

  * Fix strftime unit test: increase portability of localtime_r()
    usage (Christian Kreibich, Corelight)

0.4.0-1531 | 2021-03-01 11:40:35 +0100

  * GH-768: Add language support to explicitly trigger a parse error.

    This patch adds a Spicy `throw` statement of the form

        throw EXPR;

    which when hit triggers a parse error with the message `EXPR`.

0.4.0-1529 | 2021-03-01 11:06:22 +0100

  * GH-767: Add &max-size attribute.

    This patch adds a &max-size field and unit attribute. The attribute
    allows setting up a maximum number of bytes a parse step should consume.

    It is conceptually similar to the existing &size attribute but differs
    from it in that an error is raised only if more than the specified bytes
    were extracted (&size enforces that exactly the specified number of
    bytes is extracted).

    &max-size cannot be used together with &size.

0.4.0-1521 | 2021-02-26 08:50:04 +0000

  * GH-791: Bring back validation of field parseability. (Robin
    Sommer, Corelight)

  * GH-762: Add support for Zeek file analyzer replacement (requires
    Zeek 4.1+). (Jon Siwek, Corelight)

0.4.0-1513 | 2021-02-25 14:49:43 +0000

  * GH-771: Improve docs for vector &until parsing. (Jon Siwek,
    Corelight)

0.4.0-1511 | 2021-02-24 19:34:56 +0000

  * GH-694: Support unit-level `&convert` attribute. (Robin Sommer,
    Corelight)

    This attribute transforms a unit instance after parsing by returning the
    result of a custom expression to parent units, instead of the instance
    itself.

    Example:

        type Data = unit {
    data:
        bytes& size = 2;
    }
    &convert = self.data.to_int();

public
    type Foo = unit {
    numbers:
        Data[];
        on % done { print self.numbers; }
    };

#printf 12345678 | spicy - driver unit - convert.spicy
        [12, 34, 56, 78]

    Note how the `Data` sub-elements have been turned into integers. Without
    the new `&convert` attribute, the output would have looked like this:

        [[$data=b"12"], [$data=b"34"], [$data=b"56"], [$data=b"78"]]

  * GH-736: Fix bug that could cause hooks for anonymous fields to not
    execute. (Robin Sommer, Corelight)

  * Support parsing of nested fields. (Robin Sommer, Corelight)

    The new test case shows a number of examples that are now supported:

        public type Test = unit {
        i1:
            (bytes& size = 5);
        i1b:
            (bytes& size = 5 & convert = 42);
        i2:
            (int8[3]);
        i3:
            (int8[2])[3];
        i4:
            (int8[]& until = ($$ == 'X'))[]& until = ($$ == vector(55, 56));
        i5:
            (FooImpl);
        i6:
            (FooImpl[2]);
        i7:
            (FooImpl[2])[3];
        i8:
            (FooImpl& convert = 42);
        i9:
            (FooImpl& convert = 42)& convert = $$ * 2;
        i10a:
            (bytes& size = 5 & convert = $$.to_int());
        i10b:
            (Bytes& size = 5 & convert = $$.to_int());
        i10c:
            (FooImpl& convert = $$.data.to_int());
        i11:
            (FooImpl& convert = $$.data.to_int())& convert = $$ * 2;
        i12:
            (FooImpl& convert = $$.data.to_int())[2];
        i13: (FooImpl &convert=$$.data.to_int())[2] &convert=[3 * xyz for xyz in $$];
        }

    While the expectation isn't really that people would write this, this
    puts the infrastructure in place to support functionality that needs
    more complex field structures. (Robin Sommer, Corelight)


  * Force use of clang11 on FreeBSD 11. (Benjamin Bannier, Corelight)

  * Support inline methods in HILTI structs. (Benjamin Bannier,
    Corelight)

  * Prettify error message when ID lookups yield something of
    unexpected type. (Robin Sommer, Corelight)

  * Add HILTI support for "auto" type that's derived through operator
    resolving. (Robin Sommer, Corelight)

0.4.0-1491 | 2021-02-24 09:00:24 +0100

  * GH-773: Refine check for incompatible parse attributes.

0.4.0-1489 | 2021-02-24 08:59:36 +0100

  * GH-769: Skip evaluating `parse-at` or `parse-from` if field condition is False.

0.4.0-1486 | 2021-02-23 19:02:16 +0000

  * Documentation updates. (Jon Siwek, Corelight)

    - Add explicit mention of named-tuple elements.
    - Fix inline literal markup in "optional" type.
    - Add time/interval to Spicy->Zeek type conversion table.
    - Remove "missing" tag on README's link to "real analyzer tutorial".
    - Fix documentation link. (Benjamin Bannier, Corelight)

0.4.0-1479 | 2021-02-23 14:22:12 +0100

  * GH-770: Evaluate the AND of all field `requires` clauses.

    We already did this correctly for units by previously only evaluated the
    first `&requires` clause for fields.

0.4.0-1476 | 2021-02-23 14:21:25 +0100

  * GH-774: Fix method argument type deduction for argument packs.

0.4.0-1474 | 2021-02-22 13:46:31 +0000

  * Tweak auto-discover of *.hlto files in Zeek plugin. (Robin Sommer, Corelight)

    The plugin now scans for *.hlto files at startup at two locations by
    default:

        1. A directory "modules/" inside the plugins top-level directory.

        2. Zeek's default plugin directory.

    Both of these places are searched *recursively*, similar to how Zeek
    looks for binary plugins.

    Alternatively, one can override both of these by setting an environment
    variable `SPICY_MODULE_PATH` to a colon-separated list of directories,
    which will then be searched recursively *instead* of the default
    locations.

  * Extend spicy-config with a couple more Zeek-related options, and
    tweak some existing option names. (Robin Sommer, Corelight)

0.4.0-1470 | 2021-02-22 14:31:35 +0100

  * GH-772: Allow trailing comma in tuple type list.

    Closes #772. (Benjamin Bannier, Corelight)

0.4.0-1468 | 2021-02-22 10:12:33 +0100

  * GH-600: Add validation for field names clashing with unit names.

    This would previously fail during C++ codegen; now we reject this code.

    Closes #600. (Benjamin Bannier, Corelight)

0.4.0-1462 | 2021-02-22 10:10:53 +0100

  * GH-719: Reject local variables at module scope.

    Closes #719. (Benjamin Bannier, Corelight)

0.4.0-1455 | 2021-02-18 17:05:01 +0100

  * Document `&byte-order` unit attribute. (Benjamin Bannier, Corelight)

  * Document `&requires` unit attribute. (Benjamin Bannier, Corelight)

  * Convert %byte-order unit property to an attribute. (Benjamin Bannier, Corelight)

  * Convert %requires property to an attribute. (Benjamin Bannier, Corelight)

  * Make tuple index type calculation more resilient. (Benjamin Bannier, Corelight)

  * GH-744: Fix out-of-bounds access in tuple index type resolution.

    This code was previously indexing into a vector beyond the end when
    determining the type of a tuple index operation. We now
    return an unknown type here here so that we can reject the operation
    during later validation.

0.4.0-1446 | 2021-02-18 09:31:13 +0000

  * GH-709: Fix codegen's result type of unit.offset(). (Robin Sommer,
    Corelight)

  * GH-709: Add description of enum use to Zeek plugin documentation.
    (Robin Sommer, Corelight)

  * Fix bug with custom enum types not being available in time for
    Zeek scripts. Turns out that for precompiled Zeek analyzers it
    didn't work to automatically make the Spicy-side enum types
    available to Zeek scripts because of a timing issue. (Robin
    Sommer, Corelight)

  * Add grouping() function to builder API. (Robin Sommer, Corelight)

  * Add new function linkage "preinit" to HILTI. (Robin Sommer,
    Corelight)

0.4.0-1437 | 2021-02-17 11:08:16 +0100

  * GH-682: Remove TODO in docs. (Benjamin Bannier, Corelight)

0.4.0-1435 | 2021-02-17 11:07:03 +0100

  * GH-723: Provide constructor for ports.

    Closes #723. (Benjamin Bannier, Corelight)

0.4.0-1430 | 2021-02-17 11:06:27 +0100

  * GH-731: Add unit `position` method.

    This method returns an iterator to the current position in the unit's
    input stream.

    Closes #731. (Benjamin Bannier, Corelight)

0.4.0-1428 | 2021-02-16 16:04:42 +0000

  * GH-690: Add `%finally` hook to Spicy units.

    ``on %finally  {
        ...
    }``
        Executes once unit parsing has completed in any way. This hook is
        most useful to modify global state that needs to be updated no
        matter the success of the parsing process. Once `%init` triggers, this
        hook is guaranteed to eventually execute as well. It will run
        *after* either ``%done`` or ``%error``, respectively. (If a new
        error occurs during execution of ``%finally``, that will not
        trigger the unit's ``%error`` hook.)

  * Generalize %error to catch all exceptions. So far it was catching
    only HILTI-side errors. Now we do general catch-all so have a
    reliably hook for error handling no matter what's the cause.
    (Robin Sommer, Corelight)

  * Support finalizers in HILTI structs. There's a now new magic hook
    `~finally` that will run automatically when a struct instance gets
    destroyed. (Robin Sommer, Corelight)

  * Fix double normalization in ID strings. (Robin Sommer, Corelight)

0.4.0-1422 | 2021-02-16 10:49:34 +0000

  * GH-696: Add support for shared unit contexts with lifetime
    determined by host application. (Robin Sommer, Corelight)

    This introduces a new unit property for public unit types:

        %context = <type>;

    The unit's new `context()` method will then return a value of type
    `<type>&` (i.e, a reference to an instance of the given type). This will
    be initialized with a reference to a default-constructed instance of the
    type.

    What makes this special is that the host application can determine how a
    context instance is being shared across different parsers: it can (1)
    let multiples units link their context to the same instance, and (2)
    retain a context instance beyond any specific unit's lifetime.

    We use this new mechanism to provide contexts with connection-level
    semantics: the Zeek plugin, and also spicy-driver's batch mode, will
    create a single context instance that the two originator-side and
    responder-side units will share (they must declare matching contexts
    for that to work). The object will live until the connections gets
    torn down (also meaning it'll be reused across the per-packet parsers
    for UDP).

    See the documentation for examples.

  * Add `<vector>.pop_back()`. (Robin Sommer, Corelight)

  * Support assignment to tuple elements. (Robin Sommer, Corelight)

  * Extend the spicy-driver batch format to be able to express
    bi-direction connection semantics. This increases the batch format
    version to v2. (Robin Sommer, Corelight)

  * Add new variant of external function linkage to HILTI that skips
    the resumable wrapping. (Robin Sommer, Corelight)

0.4.0-1415 | 2021-02-16 10:29:53 +0000

  * GH-726: $$ wasn't accessible from inside an `&until-including`
    expression. (Robin Sommer, Corelight)

  * GH-740: May address the unstable behavior by rewriting the code
    surrrounding the `&while` parsing (but keeping semantics the
    same). (Robin Sommer, Corelight)

0.4.0-1412 | 2021-02-16 10:07:45 +0000

  * Activate CI for Debian 10 on master. (Robin Sommer, Corelight)

  * GH-629: Use linb::any instead of std::any across the code base.
    (Robin Sommer, Corelight)

  * GH-738: Fix crash in spicy-driver when used with units that
    require parameters. (Robin Sommer, Corelight)

  * GH-737: Fix non-converging AST in case of ambiguous ID usage.
    (Robin Sommer, Corelight)

0.4.0-1403 | 2021-02-09 13:23:55 +0000

  * Fix missing f-string in Spicy Pygments lexer for floats/reals (Jon
    Siwek, Corelight)

  * GH-722: Bump pathfind. (Benjamin Bannier, Corelight)

  * GH-732: Add test for setting input beyond current position. (Benjamin Bannier, Corelight)

  * Add missing dependency of header precompilation target on script.
    (Benjamin Bannier, Corelight)

  * Fix documentation of `set_input`. (Benjamin Bannier, Corelight)

0.4.0-1394 | 2021-02-05 07:09:51 +0000

  * Fix compatibility with Zeek 4.1 API changes/removals. (Jon Siwek,
    Corelight)

0.4.0-1392 | 2021-02-04 15:14:31 +0000

  * GH-962: Fix type inference for constructors of
    vector/list/set/map, and for locals/globals. (Benjamin Bannier,
    Corelight; Robin Sommer, Corelight)

  * Remove unused function typeOfExpressions. (Benjamin Bannier,
    Corelight)

0.4.0-1385 | 2021-02-04 14:59:06 +0000

  * Add Pygments lexers for `.spicy` and `.evt` files. (Jon Siwek,
    Corelight)

  * Use syntax highlighting for Spicy/EVT/Zeek/C++ in documentation.
    (Jon Siwek, Corelight)

  * Add 'livehtml' target to `doc/Makefile` that will run
    `sphinx-autobuild`, plus some cleanup. (Jon Siwek, Corelight)

0.4.0-1375 | 2021-02-03 11:43:19 +0100

  * Enhance parsing of single-element container constructors so that
    trailing `,` is optional. (Benjamin Bannier, Corelight)

  * GH-711: Fix formatting of a number of predefined enum values.
    (Benjamin Bannier, Corelight)

  * GH-697: Allow trailing ',' in enum declarations. (Benjamin
    Bannier, Corelight)

  * Update some links in the documentations. (Benjamin Bannier,
    Corelight)

  * GH-699: Add script to store and upload release artifacts.
    (Benjamin Bannier, Corelight)

0.4.0-1358 | 2021-02-01 10:40:23 +0100

  * GH-701: Make offsets usable from unit field attributes.

    We previously updated parse positions when entering productions. This is
    insufficient as the evaluation of attributes might already access the
    position via `self.offset()`.

    With this patch we now only update the parse position before we start
    parsing of a field.

    Closes #701. (Benjamin Bannier, Corelight)

  * Use ubuntu-20 as base for published Docker image. (Benjamin Bannier, Corelight)

  * Set PATH and ZEEK_PLUGIN_PATH for ubuntu Docker images. (Benjamin Bannier, Corelight)

  * Validate Zeek installation for Docker CI tasks. (Benjamin Bannier, Corelight)

  * Fix Docker images for possible lib libdir.

    This patch fixes the `ZEEK_PLUGIN_PATH` set in Docker images so Zeek can
    find plugins even if the Spicy installation was performed with a libdir
    named `lib` instead of `lib64`. (Benjamin Bannier, Corelight)


0.4.0-1346 | 2021-01-29 08:26:26 +0000

  * GH-685: Allow to define a minimum Spicy version that a module
    requires. Adding `%spicy-version = "1.0";` to a module now lets it
    require to compile with a Spicy of at least that version. The
    version can be specified as either `x.y` or `x.y.z`. (Robin
    Sommer, Corelight)

  * GH-688: Add functions `zeek::uid()` and `zeek::fuid()` to access
    UIDs and FUIDs, respectively. (Robin Sommer, Corelight)

  * GH-689: Allow to pass a MIME type to `file_new()` when beginning
    file analysis. (Johanna Amann, Corelight)

  * GH-700: Support `&size` attribute at the unit level.

    This limits input to the given number of bytes for the unit:

        type Bar = unit(n: uint64) {
x:
    bytes& eod;
        } &size=n;

    Parsing less than `n` bytes is an error. (Robin Sommer, Corelight)

  * Fix --git-root in autogen-version call. (Robin Sommer, Corelight)

  * Fix using `Null` as a parameter default value for `optional<T>`.
    (Robin Sommer, Corelight)

  * Add timestamp removal to the Zeek log canonification in tests.
    (Johanna Amann, Corelight)

0.4.0-1330 | 2021-01-28 10:22:55 +0100

  * GH-701: Add strptime Spicy library function.

0.4.0-1324 | 2021-01-26 17:59:30 +0000

  * Update reproc to release version. (Robin Sommer, Corelight)

0.4.0-1321 | 2021-01-26 13:23:11 +0100

  * Lots of CI / clang-format / Docker cleanup. (Benjamin Bannier, Corelight)

0.4.0-1305 | 2021-01-21 15:34:39 +0000

  * Fix failing Zeek tests. (Robin Sommer, Corelight)

0.4.0-1304 | 2021-01-21 13:37:38 +0000

  * Fix a couple of issues with Zeek 4.0. (Robin Sommer, Corelight)

  * GH-640: Extend spicy-doc to document begin/end operators. (Robin
    Sommer, Corelight)

  * Bulk update of all test baselines. (Robin Sommer, Corelight)

  * Update copyright headers. (Robin Sommer, Corelight)

  * Do not use #pragma once in libhilti.h. (Robin Sommer, Corelight)

  * Disable output from header compilation. (Robin Sommer, Corelight)

  * Update README. (Robin Sommer, Corelight)

  * Remove unused TLS baselines. (Robin Sommer, Corelight)

  * Prettify configure output. (Robin Sommer, Corelight)

0.4.0-1286 | 2021-01-21 12:04:58 +0100

  * Add CI job to validate release tarball. (Benjamin Bannier,
    Corelight)

  * Document more platforms in the installation docs. (Benjamin
    Bannier, Corelight)

  * Use "default" compiler if possible in Docker images. (Benjamin
    Bannier, Corelight)

  * Make master-only CI tasks triggerable. (Benjamin Bannier,
    Corelight)

0.4.0-1277 | 2021-01-20 19:55:47 +0000

  * Add final newline to ASCII output of spicy-dump. (Robin Sommer,
    Corelight)

  * GH-671: Fix parsing of empty look-ahead vectors. (Robin Sommer,
    Corelight)

  * Fix Zeek plugin build when toolchain is disabled. (Robin Sommer,
    Corelight)

  * Let the autogen-version script execute out of the top-level
    directory. (Robin Sommer, Corelight)

0.4.0-1270 | 2021-01-20 14:04:16 +0100

  * Allow to enforce sequential execution of jobs spawned during JIT.
    If the environment variable `HILTI_JIT_SEQUENTIAL` is set jobs are
    executed sequentially. (Benjamin Bannier, Corelight)

  * Use sequential JIT mode in CI to save memory. (Benjamin Bannier,
    Corelight)

  * GH-582: Add master CI task for macOS Big Sure. (Benjamin Bannier,
    Corelight)

  * GH-670: Remove pre-commit hooks which require external tool
    installation. (Benjamin Bannier, Corelight)

  * Move `spicy::rt::getenv` to `hilti::rt`. (Benjamin Bannier,
    Corelight)

0.4.0-1262 | 2021-01-19 08:28:41 +0000

  * macOS fixes (Robin Sommer, Corelight)

    - GH-664: Increase minimum fiber stack size.
    - Pass CMAKE_OSX_SYSROOT through to JIT linker.

  * Add a couple cron scripts to repository. (Benjamin Bannier, Corelight)

    - `ci/cron/spicy-benchmark.sh:` run Spicy benchmarks
    - `ci/cron/update_spicy_dockerhub.sh: update Spicy Docker images

0.4.0-1255 | 2021-01-18 08:45:26 +0000

  * Port JIT to use host C++ compiler instead of libclang/libllvm.
    (Robin Sommer, Benjamin Bannier; Corelight).

    This completely removes the dependency on an existing clang
    installation. Instead, JIT now requires only a C++17 host
    compiler. Internally, this comes with a substantial refactor of
    the JIT components.

    This changes also includes the following additional pieces:

        - We pull in `https://github.com/DaanDeMeyer/reproc` as a
          submodule for spawning the host compiler.

        - The environment variable `HILTI_CXX` overrides the
          executable to use as the C++ compiler during JIT. It won't
          work to switch in a completely different compiler that way
          (because we'd continue to make some assumptions based on the
          default compiler), but it allows to provide an additional
          wrapper around the default compiler.

        - We remove the distinction between building the toolchain
          with/without JIT. We now always assume we have the JIT
          capbility available, but will fail gracefully when unable to
          use the host compiler.

        - JIT compilation captures & reports C++ error output.

  * Extend `{hilti,spicy}-config` with options `--cxx-flags-hlto`/`--cxx-flags-hlto`
    printing compiler flags for building HLTO files.

  * CMake: Redoing object library linking. We weren't consistent in
    how we linked them, which led to some off effects. (Robin Sommer,
    Corelight)

  * Codegen: Construct initializer lists from literals. Benjamin
    Bannier, Corelight)

  * Cleanup left-over ASAN code that was no longer in use. (Robin
    Sommer, Corelight)

  * Increase CI memory limit for Ubuntu debug task. (Robin Sommer, Corelight)

  * Bump CMake in CI Docker image. (Benjamin Bannier, Corelight)

0.4.0-1227 | 2021-01-13 07:41:35 +0000

  * Re-implement the runtime's fiber (co-routine) code. (Benjamin
    Bannier, Robin Sommer; Corelight).

    This should be mostly transparent to users, but internally the
    main pieces are:

    - GH-561: Switch from libtask to new 3rd-party fiber library
      (https://github.com/simonfxr/fiber.git).

    - GH-561: Support "shared stack" mode for fibers where the runtime
      only allocates a single stack that's used by all fibers, vs.
      individual per-fiber stacks. This comes with a slight
      performance penalty for context switching, but substantially
      improves memory usage (and, in turn, performance with lots of
      concurrent fibers). The shared stack is used by default now.
      There's also new benchmark hilti-rt-fiber-benchmark to profile
      performance.

    - GH-460: Add runtime checks for risk of stack overflow. When the
      HILTI codegen creates a function, it now always inserts a stack
      check at the beginning that ensures that a minimal amount of
      stack space remains available. If it is not, we abort with a
      runtime exception. This gives us well-defined behavior if call
      stacks go too deep. The minimum amount of stack space is
      configurable through a global option.

    Along with this comes large refactor of the runtime's fiber code.

  * Fix Linux' --ldflags for pthread. (Robin Sommer, Corelight)

  * Make hilti::rt::isInitialized() save to be used when no global
    state is available. (Robin Sommer, Corelight)

  * When running the CI container through the Makefile, use local
    user's UID/GID. (Robin Sommer, Corelight)

  * Adding missing call to rt::done() to default main(). (Robin
    Sommer, Corelight)

  * Fix CMake logic for hilti-rt. (Robin Sommer, Corelight)

  * Enable leak checking by default in ASAN builds. ASAN itself
    enables it only on Linux by default. (Robin Sommer, Corelight)

  * Let autogen-version find VERSION independent of current directory.
    (Robin Sommer, Corelight)

0.4.0-1201 | 2021-01-11 21:03:10 +0100

  * Activate building Spicy in CentOS 7 Docker image. (Benjamin Bannier,
    Corelight)

0.4.0-1199 | 2021-01-10 14:03:48 +0000

  * Remove manual installation of Zeek CMake files on FreeBSD CI. (Benjamin
    Bannier, Corelight)

0.4.0-1196 | 2021-01-09 19:13:50 +0100

  * GH-650: Add infrastructure to upload a release tarball for tags. (Benjamin
    Bannier, Corelight)

  * GH-633: Fixes for FreeBSD, plus CI support. (Benjamin Bannier, Corelight)

  * Bump minimal version of BTest in CI to 0.66. (Benjamin Bannier,
    Corelight)

0.4.0-1175 | 2021-01-06 07:28:23 +0000

  * Fix for macOS 10.16. (Robin Sommer, Corelight)

0.4.0-1173 | 2021-01-05 11:01:34 +0100

  * GH-644: Remove Spicy support for `else` on `while`.

0.4.0-1148 | 2021-01-04 08:10:25 +0000

  * Overhauling regexp matching semantics.

    We were inconsistent in how regular expressions (1) handled being anchored
    vs not, and (2) allowed to retrieve the matching data. We fix this as
    follows:

    - We rename the regexp methods for more consistency: `match` now means
      "matching is anchored at the beginning of the data", and `find` means
      we find matches anywhere in the data. This leads to the following
      renames:

        * <regexp>.find -> <regexp>.match
        * <regexp>.find_groups -> <regexp>match_groups
        * <regexp>.find_span -> <regexp>.find

      `<bytes>.match` adapts to the new semantics as well.

    - We remove the `&anchor` attribute because that's the default now.

    - `&no_sub` remains available to override the default choice of the
      matcher type: it'll use the minimal matcher and disable capture group
      support.

0.4.0-1145 | 2021-01-02 20:43:29 +0100

  * GH-635: Collection of cleanups from consuming Spicy in external project.
    (Benjamin Bannier)

  * GH-620: Add paths to Spicy and Zeek to Ubuntu Docker image PATHs. (Benjamin Bannier, Corelight)

  * Extend Docker images. (Benjamin Bannier, Corelight)

    - Add Debian-10 Docker image.
    - Add Debian-9 Docker image.
    - Add Fedora-32 Docker image.
    - Add Fedora-33 Docker image.
    - Add Ubuntu-16 Docker image.
    - Add Ubuntu-18 Docker image.
    - Add Ubuntu-20 Docker image.
    - Remove unsupported Ubuntu-19.10.
    - Fix yum repository name in CentOS 8 Docker images. (GH-619)
    - Use correct download location for Zeek packages.

  * GH-575: Add support for FreeBSD. (Benjamin Bannier, Corelight)

  * Fix test 'spicy.rt.time'. (Benjamin Bannier, Corelight)

  * C++ fix to be explicit on which variant we construct in `Result`. (Benjamin
    Bannier, Corelight)

0.4.0-1118 | 2020-12-11 07:18:52 +0000

  * GH-602: Install Zeek from packages. (Benjamin Bannier, Corelight)

0.4.0-1112 | 2020-12-10 10:33:31 +0000

  * GH-25: Add support for manual backtracking to Spicy. If a field is
    marked with ``&try``, a later call to ``self.backtrack()``
    anywhere down in the parse tree will return to that position and
    continue there. See the documentation for more, including an
    example. (Robin Sommer, Corelight).

  * Support &eod for vector parsing. (Robin Sommer, Corelight)

  * Fix Sphinx helper script that emits command lines. (Robin Sommer,
    Corelight)

0.4.0-1106 | 2020-12-10 10:10:16 +0000

  * Make capture groups available inside hooks of regular expression fields.
    (Robin Sommer, Corelight)

    Example:

         x: /(a.c)(de*f)(h.j)/ {
    print $1, $2, $3;
}

    This will print the pieces corresponding to the three groups inside the
    expression.

    Couple notes:

        - We don't make $0 available as that would be the same as `$$`.

        - Matching with capture groups is more expensive that without. If
          you don't use any groups in the regexp, the overhead is
          automatically removed. If you use groups, but don't need them to
          capture, add `&nosub` to the field.

  * Make new captures() method available on regexp matching state in
    HILTI. (Robin Sommer, Corelight)

  * Fix argument mismatches between Spicy generated C++ prototypes and
    Zeek plugin runtime code. (Robin Sommer, Corelight)

  * Fix compile error on non-JIT version of plugin, and fix unsafe use
    of string_view. (Robin Sommer, Corelight)

  * Fix warnings reported by Apple's compiler. (Robin Sommer,
    Corelight)

0.4.0-1093 | 2020-12-10 08:39:26 +0100

  * Switch runtime library to use "fiber" library instead of libtask.
    We now leverage https://github.com/simonfxr/fiber. (Benjamin
    Bannier, Corelight)

  * GH-599: Fix construction of `Bytes`. (Benjamin Bannier, Corelight)

  * GH-603: Disable precompiled headers if we are building with GCC.
    (Benjamin Bannier, Corelight)

  * GH-94: Explicitly enable leak checking on tests relying on it being
    active. (Benjamin Bannier, Corelight)

  * GH-492: Add unit tests for IntrusivePtr. (Benjamin Bannier,
    Corelight)

  * Make run-zeek script also work with ASAN on macOS. (Benjamin
    Bannier, Corelight)

  * Remove unused Zeek compiler driver variable. (Benjamin Bannier,
    Corelight)

  * Add missing runtime cleanups in a few places. (Benjamin Bannier,
    Corelight)

  * Fix some documentation links and link checking. (Benjamin Bannier)

  * Fix GitHub link. (Robin Sommer, Corelight)

0.4.0-1071 | 2020-11-30 18:16:35 +0000

  * Series of speed-ups to the Spicy compiler. (Robin Sommer, Corelight)

    - Add caching of derived information to some AST nodes.
    - Use optional_ref<T> for returning references of AST node attributes instead of copies. (Robin Sommer, Corelight)
    - Cache location instances.
    - Returning const references for lots of accessor methods. (Robin Sommer, Corelight)
    - Mark two AST node classes as singletons.
    - Switch AST infrastructure from shared_ptr to IntrusivePtr.
    - Use forward declaration for nlohmann:json where we can.

  * Relabel profiling collector names in JIT code. (Robin Sommer, Corelight)

  * Fix for type erasure profiling. (Robin Sommer, Corelight)

0.4.0-1057 | 2020-11-23 08:35:04 +0000

  * GH-584: Add new vector methods resize/sub/at. (Robin Sommer, Corelight)

  * Revert accidental unbumping of tinyformat. (Benjamin Bannier, Corelight)

0.4.0-1053 | 2020-11-18 08:33:32 +0000

  * GH-568: Change parsing of look-ahead vectors to use a loop instead
    of recursion. (Robin Sommer, Corelight)

  * GH-583: Fix `foreach` inside a unit hook. The `foreach` attribute
    wasn't passed on, and the '$$' lookup didn't work inside a
    non-inline hook. (Robin Sommer, Corelight)

  * GH-574: Implement and document hook priorities. Robin Sommer,
    Corelight)

  * GH-389: Add additional tuple constructor expression (`tuple(...)`).
    (Robin Sommer, Corelight)

  * GH-544: Let Zeek plugin search for HLTO files at startup. By
    default, the plugin now loads all *.hlto files found in
    `$prefix/lib/spicy/Zeek_Spicy/modules/`. This can be changed by
    setting the environment variable SPICY_MODULE_PATH to a
    colon-separated list of directories to search. (Robin Sommer,
    Corelight)

  * Force upgrade to well-defined BTest version in our Docker
    containers. (Robin Sommer, Corelight)

  * Forward Zeek plugin's debug messages to Zeek's debug logger. That
    means that 'zeek -B plugin-_Zeek-Spicy' will send them into Zeek's
    debug.log. (Robin Sommer, Corelight)

0.4.0-1031 | 2020-11-17 14:05:32 +0100

  * GH-571: Fix padding handling when formatting nan and inf values.

    This patch bumps tinyformat to a version containing the upstream fix for
    https://github.com/c42f/tinyformat/issues/76 (and the follow up fix
    https://github.com/c42f/tinyformat/pull/80).

    We also add a regression test.

    Closes #571. (Benjamin Bannier, Corelight)

  * GH-589: Fix "'_debug’ was not declared in this scope" build error (Robin Sommer, Corelight)

  * Mention header precompilation in installation docs. (Benjamin Bannier, Corelight)

  * Disable precompiled headers for gcc CI job.

    While mixing GCC and Clang with precompiled headers is an issue for JIT
    (JIT always uses Clang to compile modules while precompiled headers are
    created with the compiler used to compile Spicy itself), we disable
    precompiled headers for the non-JIT, GCC job since GCC emits a
    non-suppressible warning, see
    https://gcc.gnu.org/bugzilla/show_bug.cgi?id=89808,

        build/hilti/runtime/include/hilti/rt/libhilti.h:3:9: error: #pragma once in main file [-Werror]
            3 | #pragma once
              |         ^~~~
        cc1plus: all warnings being treated as errors

    Instead of not treating warnings as errors anymore we disable precompiled
    headers in this build since it does not use them anyway. (Benjamin Bannier, Corelight)

  * Add missing include. (Benjamin Bannier, Corelight)

  * Optionally precompile libhilti.h.

    This patch adds precompilation of libhilti.h for developer setups. This
    header is included in all files during JIT, and precompiling it can for
    individual files reduce JIT time by up to 30%. This behavior can be
    disabled by setting the CMake option `HILTI_DEV_PRECOMPILE_HEADERS=OFF`.

    Specifically, we add a cmake target which creates debug and release
    versions of a precompiled libhilti.h. We modify HILTI's built-in
    CXXFLAGS to check for the presence of the needed precompiled header in
    the expected location. We expect the header either in a specific
    location in the build directory, or optionally in a version directory in
    `$HOME/.cache/spicy`.

    In this patch we enable use of the precompiled header only during JIT,
    but e.g., not when building Spicy targets. This allows us to profit from
    faster JIT while still making efficient use of e.g., ccache.
    Additionally, this approach allows us to punt on fixing a number of
    issues in header files which get exposed during JIT (JIT artifacts are
    not affected at the moment).

    The original goal was to precompile lihilti.h before installation which
    is not reached. In general, it seems that precompiled headers are not
    intended to be moved after creation. This poses a challenge when the
    header gets installed: while we could in principle generate the
    precompiled header at the final location during install time, this seems
    not reliably possible when generating packages which can be expanded in
    arbitrary locations.

    Additionally, we also install a script for users to take advantage of
    precompiled headers once Spicy has been installed. (Benjamin Bannier, Corelight)


0.4.0-1020 | 2020-11-12 11:04:57 +0000

  * Fix performance regression by removing wasteful formatting of debug message
    in release mode in Spicy runtime driver. (Robin Sommer, Corelight)

  * Fix fiber result handling on abort. (Robin Sommer, Corelight)

  * GH-578: Fix glib decompression. Robin Sommer, Corelight)

  * Add helper tool 'spicy-batch-extract' that extracts a single session out
    of a Spicy batch file. (Robin Sommer, Corelight)

  * Add preliminary benchmark script `scripts/run-benchmarks`. See developer
    docs for more. (Robin Sommer, Corelight)

0.4.0-1011 | 2020-11-05 17:09:53 +0100

  * GH-451: Support building on centos7 (Benjamin Bannier, Corelight)

0.4.0-1006 | 2020-11-05 12:56:56 +0000

  * GH-497: Add support for packet analyzers to Zeek plugin. (Robin Sommer,
    Corelight)

    Zeek's master branch received support for adding low-level "packet
    analyzers" through plugins. This extends the Spicy plugin to support that.
    The documentation gained a new section explaining how to sue that.

    Note: This feature will not "officially" be supported until Zeek 4.0
    comes out (because we don't have Zeek master in CI).

  * Add --zeek-version argument to spicy-config. (Robin Sommer, Corelight)

  * Compile fix for GCC on Ubuntu. (Robin Sommer, Corelight)

0.4.0-1001 | 2020-11-05 08:47:23 +0000

  * Use stable names for hook implementations, plus a text fix.

0.4.0-999 | 2020-11-04 15:18:45 +0100

  * Use a lightweight lambda class to callbacks executing on fibers. (Benjamin Bannier, Corelight)

  * Require all callbacks executing on fibers to return something. (Benjamin Bannier, Corelight)

  * Remove unused type_info::overload. (Benjamin Bannier, Corelight)

  * GH-546: Model `TypeInfo` with a union instead of a `std::variant`. (Benjamin Bannier, Corelight)

0.4.0-994 | 2020-11-03 09:22:40 +0000

  * Update test baseline. (Robin Sommer, Corelight)

0.4.0-993 | 2020-11-02 16:09:13 +0000

  * GH-159: Add some missing operator<< to runtime library. This now lets
    "-X trace" work for Spicy parsers. (Robin Sommer, Corelight)

  * GH-531: Make generated grammar symbols globally unique during Spicy
    codegen. (Robin Sommer, Corelight)

  * GH-176: Avoid infinite recursion when printing ASTs with self-recursive
    types. (Robin Sommer, Corelight)

  * Fix premature library shutdown in spicy-dump, which could lead to crashes
    during cleanup of the parsed unit. (Robin Sommer, Corelight)

  * Fix error message in validator. (Robin Sommer, Corelight)

  * Aggressively compress package artifact for static build. (Benjamin Bannier,
    Corelight)

  * Remove packaging tasks for static CI build. Static builds tend to produce
    very large binaries which cannot reliably be compressed to be inside Cirrus
    CI's 1GB artifact size limit. (Benjamin Bannier, Corelight)

  * Fix documentation source link. (Benjamin Bannier, Corelight)

  * Consistently use `hilti::rt::filesystem` instead of `std::filesystem`.
    (Benjamin Bannier, Corelight)

0.4.0-979 | 2020-10-23 14:11:27 +0000

  * GH-95: Zeek plugin: Respect %port and %mime-type properties in
    units. (Robin Sommer, Corelight)

  * GH-513: Catch if an interval's number of seconds is out of range.
    (Robin Sommer, Corelight)

  * Clean up parsers' bounds checking of signed integer values. (Robin
    Sommer, Corelight)

  * GH-515: Change {
        time, interval
    }::nanoseconds() to return integers
    instead of real. (Robin Sommer, Corelight)

  * Fix bashism in test helper script. (Robin Sommer, Corelight)

  * GH-490, GH-507: Provide error message when attempting to have
    spicy-driver compile inputs when JIT is not available. (Robin
    Sommer, Corelight)

  * Reject regexp unit fields that aren't constant. (Robin Sommer, Corelight)

  * GH-524: Fix spicy-driver usage message. (Robin Sommer, Corelight)

  * GH-526: Show executable's name in usage/error messages of binaries
    replying on the spicy-driver host code. (Robin Sommer, Corelight)

  * GH-527: Updating usage output in docs. (Robin Sommer, Corelight)

  * GH-528: Clarify spicyc's error message on needing HLTO filenames
    with -o. (Robin Sommer, Corelight)

  * GH-529: Clarifying libhilti's usage() fallback message. (Robin
    Sommer, Corelight)

  * GH-540: Fix bug in regular expression parsing. (Robin Sommer,
    Corelight)

  * GH-270: Fix for unit filters. (Robin Sommer, Corelight)

  * GH-500: Remove --include-linker option from driver. The better way
    to do all this is using "-l" to generate *just* the linker code,
    then compile the two together as two compilation units. (Robin
    Sommer, Corelight)

  * GH-499: Provide has_label() method for enum values. Returns true
    if the enum maps to a known type value other than Undef. (Robin
    Sommer, Corelight)

  * GH-556: Fix coercion of optional. (Robin Sommer, Corelight)

  * GH-481: Correctly set locale in centos8 image. (Benjamin Bannier,
    Corelight)

  * Use working and stable link in host application documentation.
    (Benjamin Bannier, Corelight)

  * Fix documentation to reference latest support alpine version.
    (Benjamin Bannier, Corelight)

  * GH-546: Use plain function pointers for type info accessors.
    (Benjamin Bannier, Corelight)

0.4.0-946 | 2020-10-23 09:35:04 +0200

  * GH-522: Switch filesystem impl over to ghc::filesystem.

    Closes #522. (Benjamin Bannier, Corelight)

  * Fix use of uninitialized value when numbering unit fields.

    When were using a non-const member function to call a delegated base
    class constructor. Since at the point where the member function was
    called the derived class was not fully constructed, the members it
    accessed and mutated were uninitialized, we were reading from an unset
    variable, attempting to set it, and then overwrote its values when the
    member initializer was triggered (after the base class was constructed).
    This lead to unit field indices set to large value which were not usable
    down the line.

    We now delegate maintaining the field counter to a mixin class which can
    be constructed before the base class needing its state. (Benjamin Bannier, Corelight)

0.4.0-939 | 2020-10-21 14:35:30 +0200

  * GH-521: Do not create copies when loading HLTO libraries. (Benjamin Bannier, Corelight)

0.4.0-935 | 2020-10-12 07:13:46 +0000

  * Add support for new %print hook to Spicy units. (Robin Sommer,
    Corelight)

    This allows a unit to customize its appearance when printed (or
    more generally: rendered into a string).

    Example:
#cat test.spicy
        public type Test = unit {
        on % init { print self; }
    f1:
        uint32;
    f2:
        uint8;
        on % done { print self; }

        on % print {
            local f1 = "-";
            local f2 = "-";

                if ( self?.f1 )
                    f1 = "%s" % self.f1;

                if ( self?.f2 )
                    f2 = "%s" % self.f2;

                return "%s|%s" % (f1, f2);
        }
    };

#printf '\x00\x00\x00\x01\x02' | spicy - driver test.spicy
        -|-
        1|2

  * Allow customizing the string rendering of HILTI structs. If a
    struct provide a hook __str__(), that will be used instead of the
    built-in rendering of all its fields. (Robin Sommer, Corelight)

  * Make ID getter method constant. (Robin Sommer, Corelight)

0.4.0-930 | 2020-10-09 18:10:13 +0200

  * Change `Stream::data` and `stream::View::data` to return a `Bytes`
    instead of a `string`. (Benjamin Bannier, Corelight)

  * Test & CI updates. (Benjamin Bannier, Corelight)

    - Add a CI master step to validate Homebrew formula works.
    - Add missing Cirrus CI instance to Homebrew task.
    - Build compiler toolchain by default when configuring directly from CMake.
    - Bump default Zeek package in Docker images to more recent version.
    - Fix removal of build directory in CentOS 8 Docker container.
    - Increase sleep amount in test to make time measurable in CI.
    - Speed up test `spicy.types.unit.trimming`.

  * Help clang-tidy understand control flow in a couple of places.
    (Benjamin Bannier, Corelight)

0.4.0-918 | 2020-10-06 12:23:08 +0200

  * Introduce the notion of Spicy-specific compiler options.

    So far, we only had codegen options for the HILTI compiler. This
    change allows Spicy to track options as well that control specifics of
    how the Spicy-to-HILTI code generations operates. Includes a couple of
    things:

            - As there's no appropriate global state on the Spicy side
              where we could story options easily, we extend the
              HILTI-side options struct with the notion of "auxiliary
              options", described by key/value pairs. Spicy can then store
              its options in there.

            - We derived a separate spicy::Driver from hilti::Driver to
              customize behaviour.

            - We introduce hooks into the hilti::Driver to add support for
              additional command line options. This remains a bit ad-hoc
              for now. At some point, we'll clean up of command-line
              handling generally works across the various tools, we have
              quite a bit of code duplication there right now.

            - This introduces a Spicy-side `-Q` option to record stream
              offsets during parsing. Right now, the option isn't having
              any effect yet, the implementation is going to come in a
              subsequent commit. spicy-dump's previous `-o` is renamed to
              `-Q` as well. (Robin Sommer, Corelight)

  * Add an option to emit data offsets from spicy-dump.

    This patch adds an option `-o` to `spicy-dump` which can be used to
    trigger outputting of the stream offsets where unit fields were parsed
    from. (Benjamin Bannier, Corelight)

  * Make parser struct `__offset` member accessible from Values. (Benjamin Bannier, Corelight)

  * Store field offsets during parsing. (Benjamin Bannier, Corelight)

  * Make unit fields store their unit-wide field counter.

    Initially the parser will create unit fields as `UnresolvedField`s. This
    patch adds an `_index` field and accessor for this field which will
    be used to number all unit fields in the final, fully constructed and
    resolved unit so offsets can be computed. (Benjamin Bannier, Corelight)

  * Store whether a struct field is internal on type level. (Benjamin Bannier, Corelight)

  * Fix `type::struct_::Field::is*` methods to return booleans.

    These functions were previously returning optional field values which
    carry no information beyond being set or not. This patch changes these
    methods to return booleans instead. (Benjamin Bannier, Corelight)

  * Fully type constructions of empty optional values. (Benjamin Bannier, Corelight)

  * Make `rt::Map::contains` `const`. (Benjamin Bannier, Corelight)

0.4.0-906 | 2020-10-06 08:11:53 +0200

  * GH-511: Fix order of flags in spicyc usage message.

  * GH-512: Fix typo in error message.

0.4.0-903 | 2020-09-28 18:42:38 +0000

  * Reorganize source code to allow for building the runtime library
    standalone. This adds a corresponding configure option:
    --build-toolchain=yes/no (default: yes)

  * Add configure option --disable-zeek-plugin-install to disable
    installing the Zeek plugin into Zeek's plugin path. (Robin Sommer,
    Corelight)

  * Add CMake module that enables Zeek plugins to build Spicy parsers.

  * Rename Zeek plugin to "_Zeek::Spicy" to work-around a loading
    order issue in Zeek. (Robin Sommer, Corelight)

  * Turn all 3rdparty code into proper git submodules. (Robin Sommer,
    Corelight)

  * Make using Gold linker default in CMake config. This is no change
    from when using "configure", but it changes the default when
    running "cmake" directly. (Robin Sommer, Corelight)

  * Fix configuration of Gold linker to apply to cmake modules as
    well. (Robin Sommer, Corelight)

  * Extend hilti-config and spicy-config with options to print the
    runtime's C++ include directories. (Robin Sommer, Corelight)

  * Avoid needing "which" in configure. (Robin Sommer, Corelight)

  * Disable a couple unit tests when running as root. (Robin Sommer,
    Corelight)

  * Fix compiler warning. (Robin Sommer, Corelight)

0.4.0-890 | 2020-09-28 18:41:08 +0000

  * Port Zeek plugin to current Zeek master. (Robin Sommer, Corelight)

  * GH-508: Prevent loss of precision when codgening Time
    constructors. (Benjamin Bannier, Corelight)

  * GH-503: Fix constructor `interval_ns` to accept unsigneds like
    documented. (Benjamin Bannier, Corelight)

  * Include `netinet/in.h` for `in[6]_addr` in `rt/address.h`.
    (Benjamin Bannier, Corelight)

  * Fix whitespace in spicy-driver batch usage string. (Benjamin
    Bannier, Corelight)

0.4.0-884 | 2020-09-08 07:39:00 +0000

  * Reimplement HILTI's stream type to substantially improve
    performance. (Robin Sommer, Corelight)

  * Move resolving of exceptions' backtrace symbols to point-of-use
    for improving performance. (Robin Sommer, Corelight)

  * Fix problem with fiber reuse. (Robin Sommer, Corelight)

  * Fix a unit test that's using multiple filters. (Robin Sommer,
    Corelight)

  * Add macOS work-around for problem with doctests' platform
    detection. (Robin Sommer, Corelight)

  * When configured for debug mode, build tests without -O and link
    them against {hilti,spicy}-rt-debug. (Robin Sommer, Corelight)

0.4.0-874 | 2020-08-27 14:21:57 +0200

  * Update parser lookup tables exclusively from `spicy::rt::init`.

    We previously were updating `globalState()->parsers_by_mime_type` from
    `registerParser` while `globalState()->parsers_by_name` was updated by
    `init`. This patch makes `init` the only function updating the parser
    lookup tables `parsers_by_name` and `parsers_by_mime_type`. (Benjamin Bannier, Corelight)

  * Remove unintentional `return` in `rt::haveEod`. (Benjamin Bannier, Corelight)

  * Avoid creating vain global state in `spicy::rt::{
            done, isInitialized
        }`. (Benjamin Bannier, Corelight)

  * Reinitialize default parser when reinitializing spicy::rt library. (Benjamin Bannier, Corelight)

0.4.0-861 | 2020-08-24 13:04:33 +0200

  * Diagnose invalid `zlib::Stream`s.

    When a `zlib::Stream` enters a failed state it cannot be reused
    anymore. Previously we would return empty data whenever such a stream
    was used again nevertheless. Since empty data is a valid result of
    decompressing non-empty input this leaves users with no good way to
    catch such scenarios after the fact.

    This patch modifies `zlib::Stream::decompress` to now throw an exception
    instead of returning valid but empty data. (Benjamin Bannier, Corelight)

0.4.0-855 | 2020-08-24 10:56:17 +0000

  * Fix Cirrus CI file for Alpine 3.12. (Robin Sommer, Corelight)

0.4.0-854 | 2020-08-24 08:41:23 +0000

  * Switch Dockerfile for Alpine from 3.11 to 3.12.

  * Compile code with debug information also when in release mode.
    (Robin Sommer, Corelight)

  * Compile the toolchain with -O3 instead of -O2. (Robin Sommer,
    Corelight)

  * Always link against thread library. The performance of some of our
    runtime code depends on whether the executable is linked against
    pthread or not. (Robin Sommer, Corelight)

0.4.0-849 | 2020-08-24 08:39:14 +0000

  * Fiber runtime library updates. (Robin Sommer, Corelight)

    - Fix state tracking in fibers. When taking a fiber from the
      cache, we would reset its state on INIT, which then led to
      restarting it from scratch instead of reusing the already
      running loop.

    - Add a function to fill the fiber cache with preinitialized
      instances.

    - Track number of fiber initialization in statistics.

    - Unify fiber debug logging.

0.4.0-843 | 2020-08-21 13:10:42 +0200

  * Consistently deal with `std::filesystem` still being experimental in hilti-rt. (Benjamin Bannier, Corelight)

0.4.0-840 | 2020-08-21 10:04:02 +0200

  * Prevent use after free in `rt::base64::finish`. (Benjamin Bannier, Corelight)

0.4.0-833 | 2020-08-19 11:06:44 +0200

  * Add assertion catching out-of-bounds HILTI module access. (Benjamin Bannier, Corelight)

  * Add check to prevent nullptr dereference in `hilti::rt::done`. (Benjamin Bannier, Corelight)

  * Add missing context cleanup in `hilti::rt::done`.

    We configure a `Context` owned by `GlobalState` in `hilti::rt::init`.
    Since we destruct that context in `hilti::rt::done` we also need to
    reset the context as we would otherwise leave a dangling refernence
    behind. (Benjamin Bannier, Corelight)

  * Call out gitlint in commit message style guidelines. (Benjamin Bannier, Corelight)

  * Remove guidance on not rebasing development branches. (Benjamin Bannier, Corelight)

  * Add developer manual section on pre-commit linters. (Benjamin Bannier, Corelight)

  * Add pre-commit linter for commit messages. (Benjamin Bannier, Corelight)

0.4.0-819 | 2020-08-19 11:05:55 +0200

  * Explicitly set locale in centos Docker container.

    `hilti::rt::init` needs to be able to set the locale on the host system
    and will abort with a fatal error if it cannot do so. This patch makes
    sure this is possible in the centos-8 Docker container provided by us.

    Note: This already worked in BTests since its config file enforces the
    needed environment variables. This patch fixes it also for standalone
    usage of Spicy (and now e.g., the `locale` command emits no more
    errors). (Benjamin Bannier, Corelight)

0.4.0-817 | 2020-08-18 11:24:52 +0200

  * Add `hilti::rt::Library::symbol`.

    This patch adds a new member function `Library::symbol` which can be
    used to obtained a symbol from a specific library.

    While currently `Library` users just call `::dlsym` globally to obtain a
    symbol, this becomes ambiguous if multiple libraries in the process image
    provide similarly named symbols. Instead they could hold handles to
    all interesting libraries and obtain symbols with the added function.
    With that we could also `::dclose` libraries in the `Library`
    destructor; we'll leave that change for a later point. (Benjamin Bannier, Corelight)

  * Fix warning potentially emitted from `rt::Library` destructor. (Benjamin Bannier, Corelight)

  * Fix format string in `rt::createTemporaryFile`. (Benjamin Bannier, Corelight)

  * Extend error message when `rt::Library::save` fails. (Benjamin Bannier, Corelight)

  * Change CI job to use non-privileged user.

    At least on ubuntu `root` seems to be able to modify even read-only
    directories which makes it hard to write tests checking handling of
    permission errors.

    Change the image used in CI to use a non-privileged user instead. (Benjamin Bannier, Corelight)

  * Change directory used for ccache store in CI.

    We were originally using `/var/spool/ccache` which could require
    extended privileges to create. Switch over to use `/tmp/ccache` which
    should be easier to create even for unprivileged users. (Benjamin Bannier, Corelight)

  * Prevent `rt::Resumable::get` from leaking non-HILTI exceptions. (Benjamin Bannier, Corelight)

  * Make `hilti::rt::Resumable`'s operator bool explicit. (Benjamin Bannier, Corelight)

  * Consistently provide stringification for all exceptions. (Benjamin Bannier, Corelight)

  * Add comparison operators to `rt::Backtrace`. (Benjamin Bannier, Corelight)

  * Fix `to_string_for_print` for `DeferredExpression`.

    Since `DeferredExpression` is a template itself we cannot specialize the
    master template of the `detail` namespace; instead we need to provide an
    overload. (Benjamin Bannier, Corelight)

0.4.0-798 | 2020-08-14 07:25:00 +0000

  * CI: Move Zeek package specs back into tasks to work around Cirrus
    issue, per https://github.com/cirruslabs/cirrus-ci-docs/issues/692.
    (Robin Sommer, Corelight)

0.4.0-796 | 2020-08-14 07:20:31 +0000

  * Embed version information into HLTO files that encodes information
    about the HILTI version/config they were built with.  (Robin
    Sommer, Corelight)

    We use this for detecting (1) mismatches in the HITLI version used
    for building vs running, and (2) running an optimized HLTO file
    with a debug runtime. In both cases, the runtime will print a
    warning to stderr, but still proceed processing the library.

    Note that this is breaking change that will prevent HLTOs built
    with earlier versions from loading now.

  * Add option -U to tools that summarizes runtime resource
    consumption at termination. The measurement excludes time spent
    for startup/termination. (Robin Sommer, Corelight)

  * Use cout/cerr in debug logger instead of reopening them
    separately. Output can get mixed up otherwise if other places
    write to cout/cerr directly. (Robin Sommer, Corelight)

  * Update nlohmann-json and move to top-level 3rdparty directory.
    (Robin Sommer, Corelight)

0.4.0-789 | 2020-08-11 07:47:30 +0000

  * Lots of new test coverage for HILTI runtime library: references,
    debug logger, configuration, safe-int, iterator, backtrace; along
    with related fixes and cleanups. (Benjamin Bannier, Corelight)

0.4.0-737 | 2020-08-11 07:39:31 +0000

  * Fix exception handling across library boundaries. (Robin Sommer,
    Corelight)

  * Allow limiting the "%port" of a unit to a specific direction.
    (Robin Sommer, Corelight)

    We now allow to add attributes `&originator`/`&responder` to a `%port`
    unit property to limit it to the corresponding direction of a session.
    When selecting a parser through `--parser <port>` in spicy-driver, one
    can now add `%orig` or `%resp` to the port to indicate the desired
    directionality. If one leaves that off, only parsers are selected that
    have no directionlity defined (i.e., just like it used to be).

  * Add batch input mode to spicy-dump. spicy-dump gains an option -F
    to process multiple interleaved input streams pre-recorded in a
    batch input file. The format of that file is described in the
    manual, and there's a Zeek script generating batch files from PCAP
    input traces.  (Robin Sommer, Corelight)

  * When naming a parser to use through "spicy-driver -p XXX", one can
    now also specify a port ("80/tcp") or a MIME type
    ("application/text") if the desired Spicy unit defines a
    corresponding property. See the documentation of spicy-driver for
    more. (Robin Sommer, Corelight)

  * Port Zeek plugin to new driver stateful matching API. (Robin
    Sommer, Corelight)

0.4.0-730 | 2020-08-11 07:38:44 +0000

  * Switch to Zeek 3.2.0 release packages for CI. (Robin Sommer,
    Corelight)

  * Add helper Makefile to build and run the CI docker image locallly.
    (Robin Sommer, Corelight)

0.4.0-725 | 2020-08-05 12:44:26 +0200

  * Remove conversion of `rt::Time` to `bool`. (Benjamin Bannier, Corelight)

  * Fix assignment operator for `rt::Union` of non-copyable types. (Benjamin Bannier, Corelight)

  * Fix format string error possibly triggered when adding Times and Intervals. (Benjamin Bannier, Corelight)

0.4.0-716 | 2020-08-03 22:24:56 +0200

  * Make default-constructed `rt::Address` instances invalid. (Benjamin Bannier, Corelight)

  * Make `rt::Address` remember its address family.

    This class was previously unable to distinguish IPv4 and IPv6 addresses
    with all bits sets to zero from each other. By explicitly storing its
    address family we are able to distinguish them, and can also preserve
    the address family across operations like masking. (Benjamin Bannier, Corelight)

  * Bump doctest to 2.4.0. (Benjamin Bannier, Corelight)

  * Validate prefix length inputs when constructing `rt::Network` values. (Benjamin Bannier, Corelight)

0.4.0-703 | 2020-08-03 07:41:51 +0000

  * GH-444: Port Zeek plugin to Zeek 3.2. (Robin Sommer, Corelight)

  * GH-323: Suppress bogus warning by Zeek when installing our plugin.
    (Robin Sommer, Corelight)

  * Extend EVT file syntax with @if/@else/@endif preprocessor directives.

    For now, we support one type of expression for "@if": a Zeek version check. Example:

        @if ZEEK_VERSION < 30200
        ....
        @else
        ....
        @endif (Robin Sommer, Corelight)

    Also collecting all the Zeek includes into zeek-compat.h. (Robin Sommer, Corelight)

  * Fix frequent off-by-one errors in EVT error message locations.
    (Robin Sommer, Corelight)

0.4.0-691 | 2020-07-31 15:19:57 +0200

  * Emitted fully typed initializer lists in codegen.

  * Correctly handle flipping with zero valid bytes in `rt::integer::flip`. (Benjamin Bannier, Corelight)

  * Add additional precondition checks to `rt::integer::bits`. (Benjamin Bannier, Corelight)

  * Consistently use `ntohX` type functions for `rt::integer::ntohX`. (Benjamin Bannier, Corelight)

0.4.0-681 | 2020-07-23 14:55:25 +0000

  * Fix potential invalid iterator deref in debug output. (Robin Sommer, Corelight)

  * Add operator "bytes += uint8" for adding a single byte. (Robin Sommer, Corelight)

  * Fix missing filter end-of-input signal. (Robin Sommer, Corelight)

  * GH-423: Remove C{,XX}_SYSTEM_INCLUDE_DIRS cmake flags. (Robin
    Sommer, Corelight)

    If this functionality is needed, set CXX_FLAGS instead during
    configure, e.g.:

        CXXFLAGS="-isystem /opt/rh/devtoolset-9/root/usr/include/c++/9" ./configure ...

  * GH-441: Fix generation of lists of reserved words. (Robin Sommer, Corelight)

  * GH-449: Fix documentation example. (Robin Sommer, Corelight)

0.4.0-670 | 2020-07-20 15:25:04 +0000

  * GH-415: Add missing type coercion for parameterized types. (Robin
    Sommer, Corelight)

  * GH-395: Fix tuple constructor expressions to not return a type
    until all elements are fully resolved. (Robin Sommer, Corelight)

  * GH-402: Allow switch statements without any "case". (Robin Sommer,
    Corelight)

  * GH-387: Extend a test's life time. (Robin Sommer, Corelight)

0.4.0-661 | 2020-07-16 07:06:14 +0000

  * Fix build failures from building against Zeek 3.2 dev versions.
    (Tim Wojtulewicz, Corelight)

0.4.0-659 | 2020-07-16 06:34:30 +0000

  * GH-352: Add chapter on custom host applications to manual. (Robin
    Sommer, Corelight)

  * Series of smaller documentation updates. (Robin Sommer, Corelight)

    * Add documentation for types time/interval/optional.
    * Add list of reserved keywords.
    * Update debugging sections.
    * Add note about commutativity to operator documentation where applicable.
    * Add note about checking if optional unit variables remains unset still.
    * Fix typo in documentation for 'print'.
    * Fix an operator's description.

  * GH-408: Remove "time + time" operator. (Robin Sommer, Corelight)

  * GH-404: Add `time_ns(X)` and `interval_ns(X)` expressions to
    instantiate values with nanosecond resolution. (Robin Sommer,
    Corelight)

  * Extend `{hilti,spicy}-config` with a `--dynamic-loading` option.
    This adapts `--ldflags` for applications that want to dynamically
    load precompiled parsers. (Robin Sommer, Corelight)

  * Make `&cxxname` usable for user code: The code generator now
    automatically adds prototypes for the corresponding C++ functions.
    To optionally disable that, a new attribute `&have_prototype` has
    been added. We also no longer require explicit global scoping for
    `&cxxname` IDs. (Robin Sommer, Corelight)

  * Fix joint usage of options `spicyc` options `-cKo`. This would end
    up saving only the linker code, not the combined parsing + linker
    code, in the specific output file. (Robin Sommer, Corelight)

0.4.0-645 | 2020-07-16 06:33:23 +0000

  * GH-423: Update configure's usage message. (Robin Sommer,
    Corelight)

  * GH-433: Add support for declaring constants with explicit types.
    (Robin Sommer, Corelight)

0.4.0-641 | 2020-07-14 06:13:11 +0000

  * GH-429: Fix generation of C++ prototypes for host applications.
    (Robin Sommer, Corelight)

0.4.0-639 | 2020-07-13 10:01:31 +0000

  * GH-400: Fix build with static libraries. (Robin Sommer, Corelight)

  * Fix ill-defined order of internal HILTI plugin execution. (Robin
    Sommer, Corelight)

  * Remove JIT's {init,finish}Runtime() methods. They were no-ops
    already. (Robin Sommer, Corelight)

0.4.0-634 | 2020-07-09 12:32:17 +0200

  * Lots of new & updates unit tests for the HILTI runtime. (Benjamin Bannier, Corelight)

  * Remove `AddressFamily::` prefix when stringifying values. (Benjamin Bannier, Corelight)

0.4.0-612 | 2020-07-08 15:21:20 +0000

  * GH-39: Add `spicy-dump`, a new standalone command line application
    that compiles and executes Spicy parsers on the fly and prints out
    the parsed information at the end in either a readable, custom
    ASCII format, or as JSON. (Robin Sommer, Corelight)
#echo "GET /index.html HTTP/1.0" | spicy - dump my - http.hlto
        MyHTTP::RequestLine {
        method:
            GET uri : / index.html version : MyHTTP::Version {
            number:
                1.0
            }
        }

#echo "GET /index.html HTTP/1.0" | spicy - dump - J my - http.hlto
        {
            "method" : "GET", "uri" : "/index.html", "version" : { "number" : "1.0" }
        }

  * GH-56: Add reflection API to libhilti for dynamic, runtime
    inspection of parsed units. This includes:

      * A "typeinfo(X)" expression for HILTI that, at runtime, wwill
        yield a pointer to a new `hilti::rt::TypeInfo`` struct
        describing the type of `X`, which can be a type or an
        expression.

      * `include/rt/type-info.h` builds out a comprehensive set of
        classes to describe HILTI types, and to access their values
        generically, at runtime.

      * A 3rd kind of host application-side parse function for public
        units. Spicy's driver's processInput() now uses that.

  * Change string rendering of ports to skip "Protocol::" prefix.
    (Robin Sommer, Corelight)

  * Fix bug in move assignment operator of ValueReference. (Robin
    Sommer, Corelight)

  * Add stringification for exception type. (Robin Sommer, Corelight)

  * Add stringification for "any" type. (Robin Sommer, Corelight)

  * Add missing &internal attributes to some codegenerated struct
    fields. This hides them from user visible output. (Robin Sommer,
    Corelight)

  * Add equality operator to regexp type. (Robin Sommer, Corelight)

  * Add operator<< for regexp. (Robin Sommer, Corelight)

  * Support anonymous namespaces in C++ output. (Robin Sommer,
    Corelight)

  * Add missing printer code for iterator<set> and iterator<map>.
    (Robin Sommer, Corelight)

  * Add const version of vector iterator's dereference operator.
    (Robin Sommer, Corelight)

  * Add -O0 to libhilti/libspicy for debug builds.  (Robin Sommer,
    Corelight)

0.4.0-572 | 2020-07-08 11:36:53 +0200

  * Fix parsing of `rt::Port` for out-of-range port values. (Benjamin Bannier, Corelight)

0.4.0-552 | 2020-07-03 15:47:06 +0200

  * Switch `real::unpack` error reporting to use `rt::Result`. (Benjamin Bannier, Corelight)

0.4.0-549 | 2020-07-01 15:03:27 +0200

  * Remove unintended conversion of `rt::Result` to `bool`.

0.4.0-547 | 2020-07-01 11:41:06 +0200

  * GH-392: Fix handling of vector of bools. (Benjamin Bannier, Corelight)

0.4.0-544 | 2020-07-01 09:35:52 +0200

  * Remove `hilti::rt::List` in favor of `hilti::rt::Vector`. (Benjamin Bannier, Corelight)

0.4.0-538 | 2020-06-30 08:08:16 +0000

  * Fixed semantics of `hilti::rt::Interval`-bool conversions. They
    now return true for non-zero intervals. (Benjamin Bannier,
    Corelight)

  * Add tests of `rt::Interval` stringification. (Benjamin Bannier, Corelight)

  * Add tests of `result::Error` stringification. (Benjamin Bannier, Corelight)

0.4.0-532 | 2020-06-30 08:03:40 +0000

  * Add a DHCP analyzer. (Benjamin Bannier, Corelight)

  * Fix filename in getting started docs. (Dirk Loss)

  * Move reporting logic for --report-times so that it always executes
    when the HILTI-side driver destructs. (Robin Sommer, Corelight)

  * Fix header inclusion in Spicy plugin. (Benjamin Bannier, Corelight)

  * Remove unused `is_optional` type trait. (Benjamin Bannier,
    Corelight)

  * Enable translation of optional types on the Spicy-Zeek interface.
    (Benjamin Bannier, Corelight)

0.4.0-525 | 2020-06-26 14:55:08 +0200

  * Add dedicated method to resize and assign a vector. (Benjamin Bannier, Corelight)

0.4.0-522 | 2020-06-24 22:24:34 +0200

  * Fix `hilti::rt::is_optional` so it applies to const and ref of optionals also. (Benjamin Bannier, Corelight)

0.4.0-520 | 2020-06-24 06:59:04 +0000

  * GH-398: Remove duplicate spicy-driver CMake target. (Benjamin
    Bannier, Corelight)

0.4.0-518 | 2020-06-24 06:50:48 +0000

  * GH-140: Add "&requires"/"%requires" unit attribute/property.
    (Robin Sommer, Corelight)

    Fields may now use an attribute ``&requires=EXPR`` to enforce
    additional constraints on their values. ``EXPR`` must a be boolean
    expression that will be evaluated after the parsing for the field
    has finished, but before any hooks execute. If ``EXPR`` returns
    ``False``, the parsing process will abort with an error, just as
    if the field had been unparsable in the first place.

    One can similarly enforce conditions globally at the unit level
    through a property ``%requires = EXPR``, which will be evaluated
    once the unit has been full parsed, but before any ``%done`` hook
    executes. If ``EXPR`` returns ``False``, the unit's parsing
    process will abort with an error.

  * Add Doxygen output to Sphinx documentation. It's integrated into
    the "Read the Docs build". Locally, "make doxygen" will run it.
    (Benjamin Bannier, Corelight)

  * Clean up Sphinx custom extension. (Benjamin Bannier, Corelight)
    - Remove use of `sphinx.locale.l_`
    - Format with autopep8.

0.4.0-510 | 2020-06-23 16:16:43 +0200

  * GH-386: Add codegen for Spicy time difference. (Benjamin Bannier, Corelight)

0.4.0-508 | 2020-06-22 06:03:34 +0000

  * Series of improvements for utility infrastructure (Benjamin Bannier, Corelight)
      - Add tests for `util::escapeBytes`.
      - Add tests for some functions in `hilti::rt::util`.
      - Extend list of supported escape characters in hilti::rt escaping functions.
      - Fix `rt::split` for trailing delimiters.
      - Fix edge cases rt::atoi_n.
      - Fix typos in `hilti/rt/util.h`.
      - Make `rt::util::tuple_for_each` also work on rvalues.
      - Make documentation for `hilti::rt::rtrim` more specific.
      - Remove unused `escape_control` argument to `util::escapeBytes`.
      - Use raw character literals in some `hilti::rt::utils` tests.

  * Fix extraction of event priority in EVT files. (Benjamin Bannier, Corelight)

0.4.0-496 | 2020-06-18 15:46:44 +0000

  * GH-381: Have the code generator fully declare enum types directly
    as part of the forward-declarations block. (Robin Sommer, Corelight)

  * GH-380: Corrections to tutorial. (Duffy Ocraven)

  * Adding short "Next Steps" section to tutorial. (Robin Sommer, Corelight)

0.4.0-492 | 2020-06-17 10:41:13 +0200

  * Move hilti's `util` namespace into the `hilti` namespace. (Benjamin Bannier, Corelight)

0.4.0-490 | 2020-06-17 10:36:36 +0200

  * GH-228: Prevent more overflows when working with hilti::rt Time and Interval (Benjamin Bannier,
    Corelight)
    - Change `hilti::rt::Interval` to use safe integers.
    - Change `hilti::rt::Time` to use safe integers.
    - Fix some typos in `hilti/base/util.h`.
    - Remove hilti-rt-tests dependency on hilti library.
    - Reorganize test cases for `hilti::util`.

0.4.0-484 | 2020-06-16 07:15:25 +0000

  * Add new tutorial chapter. (Robin Sommer, Corelight)

  * Add Zeek-side option "Spicy::enable_print" to control whether the
    output of Spicy-side "print" output is shown. Default is off
    (which is a change in behaviour compared to previous versions).

  * Clean up representation of type arguments and function calls in
    auto-generated operator reference. (Robin Sommer, Corelight)

  * Make the target stream of hilti::print() configurable. Default
    remains standard output. Leaving it unset will silence
    hilti::print(). (Robin Sommer, Corelight)

  * Do not use use "using X === ..." for forward-declaring struct
    types in C++ output, as that could accidentally pull in IDs from
    outer namespaces. (Robin Sommer, Corelight)

  * Fix scanner for accepting comment lines at end of file without
    terminating newline. (Robin Sommer, Corelight)

0.4.0-477 | 2020-06-15 07:36:43 +0000

  * GH-293: Fix precedence of "in" operator. (Robin Sommer, Corelight)

  * GH-319: When using &convert with a list expression, transparently
    turn the result into a vector. (Robin Sommer, Corelight)

  * GH-324: Fix error message that was too specific about source type.
    (Robin Sommer, Corelight)

  * GH-325: Fix a couple of segfaults in scope builder when hitting
    invalid code. (Robin Sommer, Corelight)

  * Update spicy-driver usage message. (Robin Sommer, Corelight)

  * Handle unknown options in getopt() consistently in driver code.
    (Robin Sommer, Corelight)

  * GH-344: Type check assignments and fix constness propagation.
    (Robin Sommer, Corelight)

  * GH-345: Validate type of &default expressions. (Robin Sommer,
    Corelight)

  * Add more debug information about constness to AST. (Robin Sommer,
    Corelight)

0.4.0-467 | 2020-06-15 07:18:03 +0000

  * Redo testing of `hilti::rt::Stream` and clean up implementation (Benjamin Bannier, Corelight)
    - Add Doxygen documentation to `rt::Stream`.
    - Add `begin` and `end` methods for `rt::list::Empty`.
    - Add additional nullptr checks when accessing `rt::Stream` iterator chunks or contents.
    - Fix escaping of `rt::Stream` and `rt::View` in stringification
      to ensure all non-printable byte sequences are escaped.
    - Make `rt::stream::View` iterators be safe by default.
    - Prevent nullptr dereference in `rt::Stream` iterator `chunk` and `content`.
    - Remove `rt::Stream::safe_[begin|end]`.
    - Remove `rt::stream::View::safe_[begin|end]`.
    - Remove `safe_begin` and `safe_end` extension points.
    - Remove unused `Chunk::begin`.
    - Remove unused `rt::Stream::chain` function.
    - Remove unused `rt::Stream::compare` functions.
    - Remove use of unsafe iterators in a few places.
    - Rename `rt::iterator::safe_range` to drop the 'safe' qualifier.
    - Add lots of tests new cases, and reorganize exists test cases.

  * Add `begin` and `end` methods for `rt::list::Empty`. (Benjamin
    Bannier, Corelight)

0.4.0-430 | 2020-06-12 08:07:16 +0200

  * Reenable some already passing clang-tidy checks. (Benjamin Bannier, Corelight)

0.4.0-428 | 2020-06-12 08:06:23 +0200

  * GH-274: Unbreak build for when other Spicy headers are already installed.

    If Spicy headers were already installed into a path somewhere in the
    include lookup, we could have found the same header twice, preferring
    the installed header over the one of the current checkout. This could
    have lead to diagnostics about duplicate definitions since due to use of
    `#pragma once` in our headers the compiler would not have detected that
    a header was already included (and even if it would have detected this,
    it would have included the incorrect header).

    This patch adds include paths from the current checkout before any
    system paths.

0.4.0-426 | 2020-06-09 17:16:13 +0000

  * Increase CI memory limit for Ubuntu Docker further. (Robin Sommer,
    Corelight)

  * GH-352: Fix Zeek link in README. (Robin Sommer, Corelight)

0.4.0-424 | 2020-06-09 15:22:26 +0000

  * GH-355: Allow access to unit parameters inside default expressions for unit
    variables. (Robin Sommer, Corelight)

  * GH-315: Fix autogen-spicy-lib to extract multi-line comments.
    (Robin Sommer, Corelight)

0.4.0-417 | 2020-06-08 10:26:32 +0000

  * Increase CI memory limit for GCC further. (Robin Sommer, Corelight)

  * Fix GCC warning. (Robin Sommer, Corelight)

0.4.0-415 | 2020-06-08 08:28:19 +0000

  * GH-368: Fix &convert for vectors of enums. (Robin Sommer, Corelight)

  * Add missing HILTI printer code for "unset" operator. (Robin
    Sommer, Corelight)

0.4.0-411 | 2020-06-05 07:28:40 +0000

  * GH-361: Fix construction of vectors with element types that
    provide default values. (Robin Sommer, Corelight)

  * GH-365: Zeek plugin: Catch if more parameters are given in an EVT
    spec than Zeek expects. (Robin Sommer, Corelight)

  * Remove branch name from Cirrus CI emails due to API limitations.
    (Benjamin Bannier, Corelight)

0.4.0-406 | 2020-06-04 09:14:27 +0000

  * GH-354: No longer allow dashes inside identifiers. (Robin Sommer, Corelight)

  * GH-350: Restrict parsing of attributes and properties to avoid
    ambiguities. (Robin Sommer, Corelight)

  * Clean up the expression mode in the Spicy Bison parser. (Robin Sommer, Corelight)

    This comes with a couple of grammar changes to avoid ambiguitues:

        - "new X" no longer allows full expressions for X, just values or
          identifiers.

        - Switch cases that use look-ahead now also need to be introduced
          with an arrow, e.g.:

               switch {
      ->a : A;
      ->b : B;
      ->c : C;
  };

  * Reduce resources requested from Cirrus. (Robin Sommer, Corelight)

  * Make iterators into vectors ordered. (Benjamin Bannier, Corelight)

  * GH-343: Fix duplicate evaluation of if-condition when used with
    &parse-at. (Robin Sommer, Corelight)

  * Add validation check for initializations that use the ID that's
    currently being declared. (Robin Sommer, Corelight)

  * Fix mailing list links in README. (Robin Sommer, Corelight)

0.4.0-393 | 2020-06-04 08:41:28 +0200

  * Rework implementation of `hilti::rt::Bytes`.
   - Add test cases for `rt::Bytes` member functions.
   - Fix construction of `rt::Bytes` from string and charset.
   - Fix handling of undefined byte order in ByteOrder conversions to integers.
   - Prevent throwing `std::out_of_range` from `rt::Bytes::extract`.
   - Add additional safety to `bytes::Iterator`.
   - Made some `rt::Bytes` methods `const`.
   - Remove rvalue overload of `rt::Bytes::str`.

  * Fix ADL for `rt::to_string`. (Benjamin Bannier, Corelight)

  * Fix `rt::util::split` for empty delimiter case. (Benjamin Bannier, Corelight)

0.4.0-364 | 2020-06-04 08:40:25 +0200

  * Make functions of `rt::string` throw instead of triggering internal errors
    for invalid input. (Benjamin Bannier, Corelight)

0.4.0-362 | 2020-06-02 15:20:59 +0000

  * GH-282: Set PATH when building documentation so that we find the
    Spicy tools. (Robin Sommer, Corelight)

  * GH-348: Fix integer overflows in regexp parsing & matching.
    (Robin Sommer, Corelight)

0.4.0-356 | 2020-05-28 10:23:04 +0000

  * Small documentation updates, including some notes on DPD
    integration and standalone plugin builds. (Robin Sommer,
    Corelight)

0.4.0-348 | 2020-05-27 13:03:22 +0200

  * GH-335: Add http_all_headers() event to HTTP EVT file.

    The event isn't used anywhere by default in Zeek, but it's a nice
    example of transforming Spicy information on the fly to match a
    Zeek-side event signature.

# 335(Robin Sommer, Corelight)

0.4.0-346 | 2020-05-27 13:00:28 +0200

  * jrx: Fix leak in `jrx_match_state_done`. (Benjamin Bannier, Corelight)
  * jrx: Fix test harness. (Benjamin Bannier, Corelight)
  * Fix leaks of jrx state when copying `regexp::MatchState`. (Benjamin Bannier, Corelight)
  * jrx: Allow copying from const. (Benjamin Bannier, Corelight)
  * Bump doctest to 2.3.8. (Benjamin Bannier, Corelight)

0.4.0-325 | 2020-05-27 06:49:26 +0000

  * GH-333: Validate that unit fields have parseable types. (Robin Sommer, Corelight)

  * GH-330: Add range check to port parsing in EVT files. (Robin
    Sommer, Corelight)

  * GH-335: Add support for converting maps from Spicy to Zeek. (Robin
    Sommer, Corelight)

  * Send emails for successful CI builds. (Benjamin Bannier, Corelight)

0.4.0-318 | 2020-05-26 10:07:38 +0000

  * Change Slack links to point to Zeek's Connect page. (Robin Sommer, Corelight)

0.4.0-317 | 2020-05-25 09:54:25 +0200

  * GH-286: Fix regexp handling to respect &size. (Benjamin Bannier, Corelight)

0.4.0-311 | 2020-05-21 07:54:43 +0000

  * Fix JIT-less build of Zeek plugin. (Robin Sommer, Corelight)

0.4.0-309 | 2020-05-20 12:35:36 +0000

  * GH-299: Support compiling the Zeek plugin without JIT support.
    (Robin Sommer, Corelight)

    If configured with `--disable-jit`, the Zeek plugin will now still
    be built and will support loading pre-compiled HLTO files. The
    plugin will have dependencies only on system libraries, and hence
    can be installed on systems that don't have the Spicy libraries
    available.

    There's also a new configure option `--disable-jit-for-zeek` that
    disables JIT support just for the Zeek plugin, creating the same
    dependency-free result.

  * GH-83: Add standalone tool spicyz to precompile analyzers for
    Zeek. (Robin Sommer, Corelight)

    This replaces the previous shell wrapper around Zeek, meaning that
    we can now precompile Zeek analyzers without actually relying on
    having a Zeek binary itself available. (We continue to require
    access to the Zeek development headers.)

    As a bonus feature, the new spicyz comes with an option "-c
    <prefix>" that, instead of producing a fully compiled HLTO, just
    writes out all the generated C++ code into files, each named
    "<prefix>-<module ID>.cc". This goes along way towards supporting
    compiling Spicy analyzers statically into an application, similar
    to how BinPAC is being used (GH-72).

  * Fix for Bison versions >= 3.6. (Robin Sommer, Corelight)

0.4.0-295 | 2020-05-20 07:23:14 +0000

  * Fix a couple typos in documentation. (Robin Sommer, Corelight)

0.4.0-293 | 2020-05-19 06:57:49 +0000

  * Rework implementation of `hilti::rt::List`. (Benjamin Bannier, Corelight)

    - Hide that `rt::List` inherits from `std::list` from users.
    - Make iterators of `rt::List` safe.
    - Add tests for `rt::List`.

0.4.0-286 | 2020-05-18 10:38:44 +0000

  * Remove FAQ pointer from README. (Robin Sommer, Corelight)

  * Fix bug in EVT reading. (Robin Sommer, Corelight)

0.4.0-284 | 2020-05-18 08:29:49 +0000

  * GH-237: Add library function `spicy::strftime`. (Benjamin Bannier,
    Corelight)

0.4.0-282 | 2020-05-18 08:28:08 +0000

  * GH-312: Fix comment parsing in EVT files. (Robin Sommer,
    Corelight)

0.4.0-280 | 2020-05-15 07:11:01 +0000

  * Rework implementation of `hilti::rt::Map`. (Benjamin Bannier, Corelight)

    - Hide that `rt::Map` inherits from `std::map` from users.
    - Make iterators of `rt::Map` safe.
    - Slim down `AssignProxy`.
    - Fix `rt::Map::get`.
    - Add test for `rt::Map` stringification.

  * Fix display of compiler flags in JIT debug log. (Benjamin Bannier, Corelight)

  * Document using Docker Hub images. (Benjamin Bannier, Corelight)

0.4.0-270 | 2020-05-14 07:00:54 +0000

  * Clean up example code. (Robin Sommer, Corelight)

  * Change structure of example section, and add more links. (Robin
    Sommer, Corelight)

  * Make autogen-docs script more robust. (Robin Sommer, Corelight)

  * Fix documentation of -= and /= opertators. (Robin Sommer, Corelight)

0.4.0-264 | 2020-05-13 14:18:08 +0000

  * Add an end-to-end TFTP analyzer for Zeek, it comes with a Zeek
    script producing a typical ``tftp.log`` log file. This analyzer is
    a good introductory example because the Spicy side is pretty
    straight-forward. (Robin Sommer, Corelight)

  * GH-306: Do not fail documentation build for broken links.
    (Benjamin Bannier, Corelight)

0.4.0-260 | 2020-05-12 10:37:16 +0000

  * Documentation updates. (Robin Sommer, Corelight)

    - GH-230: Fix a series of smaller inaccuracies and typos. Reported by Duffy Ocraven.
    - Document list type.
    - Include CHANGES into the documentation's release notes.
    - Update link to macOS LLVM binaries. (Robin Sommer, Corelight)

  * Clean up parsing of list constructor. (Robin Sommer, Corelight)

  * Clean up cast operators. (Robin Sommer, Corelight)

  * Remove a couple of empty source files. (Robin Sommer, Corelight)

0.4.0-251 | 2020-05-12 10:33:24 +0000

  * Redo implementation of `hilti::rt::Set`. (Benjamin Bannier, Corelight)

    - Hide that `rt::Set` inherits from `std::set` from users.
    - Make iterators of `rt::Set` safe.
    - Add test of rt::Set stringification.

0.4.0-245 | 2020-05-11 08:39:49 +0000

  * Redo implementation of `hilti::rt::Vector`. (Benjamin Bannier, Corelight)

    - Hide that `rt::Vector` inherits from `std::vector` from users.
    - Replace iterators of `rt::Vector` with safe alternatives.
    - Add Doxygen comments to `rt::Vector`.
    - Remove support for stringification of `std::vector` in hilti::rt.
    - Add test for stringification of `Empty` vector.
    - Make `rt::MIMEType` default-constructible.
    - Fix includes of `rt/mime.h`.
    - Use less ambiguous form for generated `rt::Vector` constructors.

  * Require Zeek plugin for Zeek protocol parser tests. (Benjamin
    Bannier, Corelight)

0.4.0-234 | 2020-05-11 08:33:55 +0000

  * GH-229: Add a Spicy function to render bytes as hex string.
    (Benjamin Bannier, Corelight)

        ## Returns a bytes value rendered as a hex string.
        public function bytes_to_hexstring(value: bytes) : string;

  *GH - 200 : Add library function `spicy::getenv()` to retrieve environment variables.(Robin Sommer, Corelight)

                  ##Returns the value of an environment variable,
      if set.public function getenv(name
                                    : string)
      : optional<string>;

  * GH-200: Add option -f to `spicy-driver` to read input from file.
    (Robin Sommer, Corelight)

  * Use fully qualified IDs for internal `&cxxname` in libraries
    `hilti.hlt` and `spicy.spicy`. (Benjamin Bannier, Corelight)

  * Fix shell commands in `docker/Makefile`. (Benjamin Bannier,
    Corelight)

0.4.0-226 | 2020-05-06 15:12:52 +0000

  * Apply a set of tweaks to specifics of &size. (Robin Sommer,
    Corelight)

    - We now enforce that a field consumes all its &size characters of
      input, unless explicitly overridden where a types allow for that
      (as, e.g., "bytes" does with an additional &eod).

    - We slightly change semantics for parsing bytes: A &size of zero
      will now set the field an empty bytes value; it used to leave
      the field unset.

    - Document that &size works with all fields (which it already
      did).

    - Clean up the implementation of &size.

  * GH-289: Do not perform look-ahead parsing for vectors fed from
    &parse-from/at. (Robin Sommer, Corelight)

  * GH-288: Rename "&until_including" attribute to "&until-including".
    (Robin Sommer, Corelight)

  * GH-269: Add support for "&until-including" when parsing bytes.
    (Robin Sommer, Corelight)

  * GH-286: Fix Bison parser's attribute association for regular
    expressions. (Robin Sommer, Corelight)

  * Add an "unset" operator to HILTI's struct type to clear optional
    types. (Robin Sommer, Corelight)

0.4.0-216 | 2020-05-05 12:25:40 +0000

  * GH-212: Resize eagerly when accessing out-of-bounds Vector
    elements. (Benjamin Bannier, Corelight)

0.4.0-214 | 2020-05-04 10:51:06 +0000

  * GH-267: Fix spicy-build to support absolute output paths. (Robin
    Sommer, Corelight)

  * GH-275, GH-276: Sort options in the usage messages of various
    tools. (Robin Sommer, Corelight)

0.4.0-208 | 2020-05-04 10:49:21 +0000

  * Use doctest test suites as top-level grouping. (Benjamin Bannier,
    Corelight)

  * GH-268: Properly escape data when stringifying streams. (Benjamin
    Bannier, Corelight)

0.4.0-203 | 2020-05-03 15:37:30 +0000

  * GH-265: Require unit properties to end with a semicolon. We were
    inconsistent with that. Note that this will break existing code
    that wasn't using the semicolons, including some examples from the
    documentation (which have been updated). For example, it's now
    `%filter;` instead of `%filter`. (Robin Sommer, Corelight)

  * GH-266: Fix missing validation of unit properties. (Robin Sommer,
    Corelight)

  * GH-262: Allow signs in intN/uintN() constructors. (Robin Sommer,
    Corelight)

  * GH-254: Make UTF8 default for bytes::decode(). (Robin Sommer,
    Corelight)

  * GH-248: Fix AST inconsistency with how modules were stored in
    scopes. (Robin Sommer, Corelight)

  * GH-263: Improve error messages for unresolved operators and IDs.
   (Robin Sommer, Corelight)

  * Improve suppression of repetitive errors. (Robin Sommer,
    Corelight)

  * Avoid leaking passwords into command lines in `docker/Makefile`.
    (Benjamin Bannier, Corelight)

0.4.0-189 | 2020-04-30 10:33:54 +0000

  * GH-196: Reject container constructors with inconsistent types, and
    coerce elements once the type is known. (Robin Sommer, Corelight)

0.4.0-187 | 2020-04-30 10:26:13 +0000

  * Improve `vector` implementation. (Benjamin Bannier, Corelight)

        - Fix semantics of safe mutating access to `hilti::rt::Vector` elements.
        - Remove broken `rt::Vector::set`.
        - Fix includes of `hilti/rt/types/vector.h`.
        - Add unit test for a couple of `hilti::rt::Vector` methods.`

  * GH-272: Document precompiled MacOS packages. (Benjamin Bannier, Corelight)

0.4.0-179 | 2020-04-29 15:15:14 +0200

  * GH-192: Remove internal errors that we ended up running into during AST
    validation. (Robin Sommer, Corelight)

  * GH-178: Fix spurious semicolons in C++ output and potential problem in C++
    block rendering (Robin Sommer, Corelight)

0.4.0-173 | 2020-04-29 08:20:21 +0000

  * Report column information in error messages. Locations are emitted
    as `<line>:<character>` so that e.g., the second character on the
    fourth line would be displayed as `4:2`. This extends to ranges as
    well which look e.g., like `1:1-2:30`. (Benjamin Bannier,
    Corelight)

0.4.0-171 | 2020-04-28 19:08:50 +0000

  * Add a section to the development manual on the structure of the
    runtime libraries. (Robin Sommer, Corelight)

  * Fix `hilti::rt::to_string` for character literals. (Benjamin
    Bannier, Corelight)

  * GH-179: Enable `hilti::rt::to_string` for `std::vector`. (Benjamin
    Bannier, Corelight)

  * Remove return of rvalue when generating parse functions. (Benjamin
    Bannier, Corelight)

0.4.0-162 | 2020-04-28 10:14:01 +0000

  * GH-219: Clean up string interface exposed by `rt::Bytes` to
    remove overloading ambiguities. (Benjamin Bannier, Corelight)

  * GH-257: Fix documentation whitespace issues and remove outdated
    FIXMEs around usages of `Result`. (Benjamin Bannier, Corelight)

0.4.0-154 | 2020-04-27 11:58:00 +0000

  * GH-242: Suppress any output from doc's `make check` by default.
    (Benjamin Bannier, Corelight)

  * GH-251: Ignore package links in documentation linkcheck.
    (Benjamin Bannier, Corelight)

  * Edit pass on documentation's reassembly section. (Robin Sommer,
    Corelight)

0.4.0-148 | 2020-04-24 13:19:22 +0000

  * GH-226: Add CI setup for building published Docker images. This
    patch adds an `update_dockerhub` target to `docker/Makefile`,
    which builds, tests and uploads images to Dockerhub. It also
    changes the Dockerfiles in `docker/` back to include Spicy build
    commands, so that they become practically self-documenting
    (Benjamin Bannier, Corelight).

  * GH-222: Output changes to make rendered ASTs easier to parse
    (Benjamin Bannier, Corelight)

0.4.0-140 | 2020-04-24 09:18:51 +0000

  * Tune update-changes configuration. (Robin Sommer, Corelight)

  * GH-245: Fix typo in documentation. (Dominik Charousset)

0.4.0-136 | 2020-04-23 16:53:22 +0000

  * Update link in README. (Yacin Nadji)

  * Fix link to FAQ in README.

  * Remove left-over debug output from DNS grammar.

0.4.0-130 | 2020-04-23 10:44:12 +0000

  * GH-170: Change "+<int>" to create a signed integer value. It used
    to create an unsigned integer.

  * GH-225: Preserve parentheses inside expressions during code
    generation.

  * GH-231: Add range check to digits in Spicy's atoi_n().

  * GH-210: Catch & report broken escape sequences in HILTI/Spicy code
    at parse time.

  * GH-217: Fix code generation of escaped C++ string literals.

0.4.0-120 | 2020-04-22 15:40:34 +0000

  * Fix package artifact links in docs.

  * Fix documentation links to Spicy mailing lists.

  * Fix issue base URL in documentation.

  * Add documentastion build target to check links.

  * Fix package creation in CI.

0.4.0-114 | 2020-04-22 13:17:05 +0000

  * Make sure auto-generated documentation items have stable ordering.

  * Cache CI ccache even if build failed.

0.4.0-111 | 2020-04-22 10:23:15 +0000

  * Build documentation from documentation folder instead of from a
    symlink inside the build directory.

0.4.0-109 | 2020-04-21 11:40:41 +0000

  * Execute CI jobs automatically.

  * Fix potential sign-unsigned mismatch in comparison.

  * Fix unused variable warning.

0.4.0-105 | 2020-04-21 10:41:25 +0000

  * Update documentation links in README.

0.4.0-104 | 2020-04-21 10:15:14 +0000

  * Tweak documentation build for standalone operation.

  * GH-172: Re-add missing autogenerated usage for spicy-build.

  * Add map & set to the auto-documented types.

0.4.0-96 | 2020-04-21 08:27:51 +0000

  * GH-129: Add infrastructure and documentation for providing binary
    packages.

0.4.0-91 | 2020-04-21 07:38:18 +0000

  * GH-174: Fix a number of smaller documentation issues.

  * GH-191: Change bytes iterator's deref operator to return uint8
    instead of uint64, and fix implementation to no longer use a
    signed integer ingternally.

  * GH-153: Prettify error messages for fmt() problems.

  * GH-154: Fix crash when accessing an unknown tuple element.

  * GH-171: Fix spelling in some diagnostic messages.

  * GH-175: Catch unresolved unit fields during validation.

  * GH-198: Show functions as "<function>" when rendering them.

  * GH-203: Add type check for elements in vector/set/map constructors.

  * GH-204: Remove support for legacy syntax for parsing sequences.
    "vector<T>" is no longer supported for parsing sequences; one must
    now use the new "T[]" syntax instead.

  * Remove Spicy's legacy list type (which was just an alias for
    "vector").

  * GH-181: Validate correct usage of struct/unit parameters.

  * Add a number of validation checks for unit hooks.

  * GH-195: Add range check for bytes' to_int() base parameter, which
    now must be between 2 and 36.

  * GH-155: Overhaul AST error reporting for more concise error
    messages.

  * GH-119: Generally include location information when throwing
    runtime exceptions.

  * Add packaging to CMake setup and CI jobs.

  * Set proper CMake project version.

  * Execute installation tests in Docker CI tasks.

0.4.0-48 | 2020-04-15 16:56:28 +0000

  * GH-163: Fix initialization of &default struct parameters.

  * GH-156: Fix look-ahead parsing for vectors terminated through
    end-of-data.

  * GH-182: Fix "for" loop's type resolution for the iteration variable.

  * GH-180: Fix printing of bytes instances that have non-printable
    characters. This also changes how we print "\n/\r/\t" inside
    bytes: We now render them as "\xXX", like any other non-printable
    character.

  * Add missing 'Undef' to byte-order enum.

  * Add clang-format to pre-commit config.

  * Add minimal CI test runs for CentOS 8, Alpine 3.11, and Ubuntu
    19.10.

  * Clean up our Docker setup/usage.

0.4.0-29 | 2020-04-15 09:04:28 +0000

  * Add pre-commit check that generated docs are up-to-date.

  * Clean up spicy-doc-to-rst and make output stable.

  * Add auto-generated docs to the tree so that one can build the full
    documentation without needing to build Spicy itself.

0.4.0-20 | 2020-04-09 15:11:43 +0000

  * GH-159: Properly close quotes and parens in "hilti-flow" debug
    statement.

  * GH-146: Validate that &byte-order attributes have an argument.

0.4.0-16 | 2020-04-09 15:06:04 +0000

  * GH-92: Documentation updates.

0.4.0-13 | 2020-04-09 11:23:15 +0000

  * Copy edit "Getting Started".

0.4.0-11 | 2020-04-09 10:56:29 +0000

  * GH-145: Fix underlinking of hilti-rt library.

  * GH-147: Add a configure time check validating that submodules are
    in place.

  * Add more CI configurations, including for macOS.

0.4.0 | 2020-04-07 14:50:17 +0000

  * Starting CHANGES.