docs notebook

docs/index.ipynb

@@ -0,0 +1,255 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "0",
+   "metadata": {},
+   "source": [
+    "# Katai struct awkward runtime\n",
+    "\n",
+    "This is a runtime for the [Kaitai Struct](https://kaitai.io/) compiler that generates code for the [Awkward Array](https://awkward-array.org/) library.\n",
+    "\n",
+    "Kaitai Struct is a declarative language used to describe binary data structures, and the Kaitai Struct compiler generates code in various languages to parse these structures. This project generates C++ code which reads the input data into Awkward Arrays, leveraging the flexibility and performance of the Awkward Array library.\n",
+    "\n",
+    "## Installation\n",
+    "\n",
+    "This installation procedure if for users only, see below for development instructions.\n",
+    "\n",
+    "This has been currently tested only on Ubuntu both natively and under the Windows Subsystem for Linux on Windows Server 2022.\n",
+    "\n",
+    "### Install the Kaitai compiler\n",
+    "\n",
+    "See the [Kaitai docs](https://kaitai.io/#download), at the moment:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "1",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "!curl -LO https://github.com/kaitai-io/kaitai_struct_compiler/releases/download/0.10/kaitai-struct-compiler_0.10_all.deb\n",
+    "!sudo apt install ./kaitai-struct-compiler_0.10_all.deb"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "2",
+   "metadata": {
+    "lines_to_next_cell": 0
+   },
+   "source": [
+    "This will install Java if not already installed.\n",
+    "\n",
+    "### Install the custom version of the compiler\n",
+    "\n",
+    "On Github, go to the [releases page](https://github.com/det-lab/kaitai_struct_awkward_runtime/releases), download the latest `jar` file and overwrite the official Kaitai compiler:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "3",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "!sudo cp io.kaitai.kaitai-struct-compiler-0.10.jar /usr/share/kaitai-struct-compiler/lib/io.kaitai.kaitai-struct-compiler-0.10.jar"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "4",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "You can restore the original compiler by reinstalling the official package:\n",
+    "\n",
+    "    sudo apt install --reinstall ./kaitai-struct-compiler_0.10_all.deb"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "5",
+   "metadata": {},
+   "source": [
+    "### Install requirements for compiling\n",
+    "\n",
+    "In barebone Ubuntu systems, like the default installed in the Windows Subsystem for Linux, you may need to install the following packages:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "6",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "sudo apt-get install build-essential cmake"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "7",
+   "metadata": {},
+   "source": [
+    "### Install the runtime\n",
+    "\n",
+    "Generally it is useful to install the runtime in a virtual environment (alternatively `conda` or `uv`):\n",
+    "\n",
+    "```bash\n",
+    "python3 -m venv venv\n",
+    "source venv/bin/activate\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "8",
+   "metadata": {
+    "lines_to_next_cell": 0
+   },
+   "source": [
+    "Then install the runtime:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "9",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%pip install awkward-kaitai"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "10",
+   "metadata": {},
+   "source": [
+    "## Usage\n",
+    "\n",
+    "First we need a test schema and a test file, for example we can take `animal.ksy` and `animal.raw` from the [Kaitai Awkward Struct examples](https://github.com/det-lab/kaitai_struct_awkward_runtime/tree/main/example_data)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "11",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "!wget https://github.com/det-lab/kaitai_struct_awkward_runtime/raw/refs/heads/main/example_data/schemas/animal.ksy\n",
+    "!wget https://github.com/det-lab/kaitai_struct_awkward_runtime/raw/refs/heads/main/example_data/data/animal.raw"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "12",
+   "metadata": {},
+   "source": [
+    "Then we first compile the schema into C++ sources:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "13",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "!ksc -t awkward animal.ksy"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "14",
+   "metadata": {},
+   "source": [
+    "This will generate a `animal.cpp` file that we can compile into a shared library:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "15",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "!awkward-kaitai-build -b build animal.cpp"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "16",
+   "metadata": {},
+   "source": [
+    "This will generate a `libanimal.so` file that we can use to read the data, in WSL, the shared library is located in `./build/build/libanimal.so`:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "17",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import awkward_kaitai\n",
+    "from pprint import pprint\n",
+    "\n",
+    "animal = awkward_kaitai.Reader(\"./libanimal.so\")\n",
+    "# WSL\n",
+    "# animal = awkward_kaitai.Reader(\".build/build/libanimal.so\")\n",
+    "awkward_array = animal.load(\"animal.raw\")\n",
+    "\n",
+    "pprint(awkward_array.to_list())"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "18",
+   "metadata": {},
+   "source": [
+    "This will print the content of the `animal.raw` file:\n",
+    "\n",
+    "```json\n",
+    "[{'animalA__Zentry': [{'animal_entryA__Zage': 5,\n",
+    "                       'animal_entryA__Zspecies': 'cat',\n",
+    "                       'animal_entryA__Zstr_len': 3,\n",
+    "                       'animal_entryA__Zweight': 12},\n",
+    "                      {'animal_entryA__Zage': 3,\n",
+    "                       'animal_entryA__Zspecies': 'dog',\n",
+    "                       'animal_entryA__Zstr_len': 3,\n",
+    "                       'animal_entryA__Zweight': 43},\n",
+    "                      {'animal_entryA__Zage': 10,\n",
+    "                       'animal_entryA__Zspecies': 'turtle',\n",
+    "                       'animal_entryA__Zstr_len': 6,\n",
+    "                       'animal_entryA__Zweight': 5}]}]\n",
+    "```\n",
+    "\n",
+    "## Development\n",
+    "\n",
+    "* [Setup a development environment](./development.md)\n",
+    "* [Release a new version](./release.md)\n",
+    "\n",
+    "## Related Papers and Talks\n",
+    "1. [Describe Data to get Science-Data-Ready Tooling: Awkward as a Target for Kaitai Struct YAML](https://indico.cern.ch/event/1330797/contributions/5796564/), Advanced Computing and Analysis Techniques for Physics Research Workshop 2024, New York, US.\n",
+    "2. [Awkward Target for Kaitai Struct](https://indico.cern.ch/event/1252095/contributions/5592420/), PyHEP Users Workshop 2023."
+   ]
+  }
+ ],
+ "metadata": {
+  "jupytext": {
+   "cell_metadata_filter": "-all",
+   "main_language": "bash",
+   "notebook_metadata_filter": "-all"
+  },
+  "language_info": {
+   "name": "python"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

mkdocs.yml

@@ -4,5 +4,9 @@ repo_url: https://github.com/det-lab/kaitai_struct_awkward_runtime
 edit_uri: edit/main/docs/
 theme:
   name: material
+  custom_dir: overrides
   features:
-    - content.action.edit
+    - content.action.edit
+plugins:
+  - mkdocs-jupyter:
+      include_source: true

overrides/main.html

@@ -0,0 +1,11 @@
+{% extends "base.html" %}
+
+{% block content %}
+{% if page.nb_url %}
+    <a href="{{ page.nb_url }}" title="Download Notebook" class="md-content__button md-icon">
+        {% include ".icons/material/download.svg" %}
+    </a>
+{% endif %}
+
+{{ super() }}
+{% endblock content %}