Skip to content

v1.11.0
Compare
Choose a tag to compare
@github-actions github-actions released this 31 Jul 12:49
· 247 commits to main since this release
v1.11.0
7ddf6ce

New Functionality

  • GH-3779: Add %sync_advance hook.

    This adds support for a new unit hook:

    on %sync_advance(offset: uint64) {
    ...
}

    This hook is called regularly during error recovery when synchronization skips over data or gaps while searching for a valid synchronization point. It can be used to check in on the synchronization to, e.g., abort further processing if it just keeps failing. offset is the current position inside the input stream that synchronization just skipped to.

    By default, "called regularly" means that it's called every 4KB of input skipped over while searching for a synchronization point. That value can be changed by setting a unit property %sync-advance-block-size = <number of bytes>.

    As an additional minor tweak, this also changes the name of what used to be the __gap__ profiler to now be called __sync_advance because it's profiling the time spent in skipping data, not just gaps.

  • Add unit method stream() to access current input stream, and stream method statistics() to retrieve input statistics.

    This returns a struct of the following type, reflecting the input seen so far:

    type StreamStatistics = struct {
    num_data_bytes: uint64;     ## number of data bytes processed
    num_data_chunks: uint64;    ## number of data chunks processed, excluding empty chunks
    num_gap_bytes: uint64;      ## number of gap bytes processed
    num_gap_chunks: uint64;     ## number of gap chunks processed, excluding empty chunks
};

  • GH-1750: Add to_real method to bytes. This interprets the data as representing an ASCII-encoded floating point number and converts that into a real. The data can be in either decimal or hexadecimal format. If it cannot be parsed as either, throws an InvalidValue exception.

  • GH-1608: Add get_optional method to maps.
    This returns an optional value either containing the map's element for the given key if that entry exists, or an unset optional if it does not.

  • GH-90/GH-1733: Add result and spicy::Error types to Spicy to facilitate error handling.

Changed Functionality

  • The Spicy compiler has become a bit more strict and is now rejecting some ill-defined code constructs that previous versions ended up letting through. Specifically, the following cases will need updating in existing code:
    • Identifiers from the (internal) hilti:: namespace are no longer accessible. Usually you can just scope them with spicy:: instead.
    • Previous versions did not always enforce constness as it should have. In particular, function parameters could end up being mutable even when they weren't declared as inout. Now inout is required for supporting any mutable operations on a parameter, so make sure to add it where needed.
    • When using unit parameters, the type of any inout parameters now must be unit itself. To pass other types into a unit so that they can be modified by the unit, use reference instead of inout. For example, use type Foo = unit(s: sink&) instead of type Foo = unit(inout: sink). See https://docs.zeek.org/projects/spicy/en/latest/programming/parsing.html#unit-parameters for more.
  • The Spicy compiler new uses a more streamlined storage and access scheme to represent source code. This speeds up work up util C++ source translation (e.g., faster time to first error message during development).
  • spicyc options -c and -l no longer support compiling multiple Spicy source files to C++ code individually to then build them all together. This was a rarely used feature and actually already broken in some situations. Instead, use spicyc -x to produce the C++ code for all needed Spicy source files at once. -c and -l remain available for debugging purposes.
  • The spicyc option -P now requires a prefix argument that sets the C++ namespace, just like -x <prefix> does. This is so that the prototypes match the actual code generated by -x. To get the same identifiers as before, use an empty prefix (-P "").
  • GH-1763: Restrict initialization of const values to literals. This means that e.g., const values cannot be initialized from other const values or function calls anymore.
  • result and network are now keywords and cannot be used anymore as user-specified indentifiers.
  • GH-1661: Deprecate usage of &convert with &chunked.
  • GH-1657: Reduce data copying when passing data to the driver.
  • GH-1501: Improve some error messages for runtime parse errors.
  • GH-1655: Reject joint usage of filters and look-ahead.
  • GH-1675: Extend runtime profiling to measure parser input volume.
  • GH-1624: Enable optimizations when running spicy-build.

Bug fixes

  • GH-1759: Fix if-condition with switch parsing.
  • Fix Spicy's support for network type.
  • GH-1598: Enforce that the argument new is either a type or a ctor.
  • GH-1742, GH-1760: Unroll constructors of big containers in generated code. We previously would generate code which would be expensive to compiler for some compilers. We now generate more friendly code.
  • GH-1745: Fix C++ initialization of global constants through global functions.
  • GH-1743: Use a checked cast for map's in operator.
  • GH-1664: Fix &convert typing issue with bit ranges.
  • GH-1724: Fix skipping in size-constrained units. We previously could skip too much data if skip was used in a unit with a global &size.
  • Fix incremental skipping. We previously would incorrectly compute the amount of data to skip which could have potentially lead to the parser consuming more data than available.
  • GH-1586: Make skip productions behave like the production they are wrapping.
  • GH-1711: Fix forwarding of a reference unit parameter to a non-reference parameter.
  • GH-1599: Fix integer increment/decrement operators require mutable arguments.
  • GH-1493: Support/fix public type aliases to units.

Documentation

  • Add new section with guidelines and best practices. This focuses on performance for now, but may be extended with other areas alter. Much of the content was contributed by Corelight Labs.
  • Fix documented type mapping for integers.
  • Document generic operators.
Assets 14
Loading