From e83ee3492730823978c474ecb5141fe1a84a4364 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C5=81ukasz=20Stolarczuk?= Date: Mon, 16 Dec 2024 14:58:26 +0100 Subject: [PATCH 1/4] Minor cleanups and additions in docs includes: - add missing CUDA provider in the web docs, - proxy_pool is enabled by default, move req. info to proxy_lib, - add links in README. --- README.md | 17 ++++++++++++----- scripts/docs_config/api.rst | 13 ++++++++++++- 2 files changed, 24 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index df90b6852..4cd1d8ff5 100644 --- a/README.md +++ b/README.md @@ -122,11 +122,16 @@ List of options provided by CMake: ## Architecture: memory pools and providers -A UMF memory pool is a combination of a pool allocator and a memory provider. A memory provider is responsible for coarse-grained memory allocations and management of memory pages, while the pool allocator controls memory pooling and handles fine-grained memory allocations. +A UMF memory pool is a combination of a pool allocator and a memory provider. A memory provider is responsible for +coarse-grained memory allocations and management of memory pages, while the pool allocator controls memory pooling +and handles fine-grained memory allocations. Pool allocator can leverage existing allocators (e.g. jemalloc or tbbmalloc) or be written from scratch. -UMF comes with predefined pool allocators (see include/pool) and providers (see include/provider). UMF can also work with user-defined pools and providers that implement a specific interface (see include/umf/memory_pool_ops.h and include/umf/memory_provider_ops.h). +UMF comes with predefined pool allocators (see [`include/umf/pools`](include/umf/pools)) and providers +(see [`include/umf/providers`](include/umf/providers)). UMF can also work with user-defined pools and +providers that implement a specific interface (see [`include/umf/memory_pool_ops.h`](include/umf/memory_pool_ops.h) +and [`include/umf/memory_provider_ops.h`](include/umf/memory_provider_ops.h)). More detailed documentation is available here: https://oneapi-src.github.io/unified-memory-framework/ @@ -152,6 +157,7 @@ a duplicate of another process's file descriptor (`pidfd_getfd(2)` is supported Permission to duplicate another process's file descriptor is governed by a ptrace access mode `PTRACE_MODE_ATTACH_REALCREDS` check (see `ptrace(2)`) that can be changed using the `/proc/sys/kernel/yama/ptrace_scope` interface in the following way: + ```sh $ sudo bash -c "echo 0 > /proc/sys/kernel/yama/ptrace_scope" ``` @@ -183,6 +189,7 @@ a duplicate of another process's file descriptor (`pidfd_getfd(2)` is supported Permission to duplicate another process's file descriptor is governed by a ptrace access mode `PTRACE_MODE_ATTACH_REALCREDS` check (see `ptrace(2)`) that can be changed using the `/proc/sys/kernel/yama/ptrace_scope` interface in the following way: + ```sh $ sudo bash -c "echo 0 > /proc/sys/kernel/yama/ptrace_scope" ``` @@ -203,7 +210,7 @@ Additionally, required for tests: #### DevDax memory provider (Linux only) -A memory provider that provides memory from a device DAX (a character device file /dev/daxX.Y). +A memory provider that provides memory from a device DAX (a character device file like `/dev/daxX.Y`). It can be used when large memory mappings are needed. ##### Requirements @@ -249,8 +256,6 @@ This memory pool is distributed as part of libumf. It forwards all requests to t memory provider. Currently umfPoolRealloc, umfPoolCalloc and umfPoolMallocUsableSize functions are not supported by the proxy pool. -To enable this feature, the `UMF_BUILD_SHARED_LIBRARY` option needs to be turned `ON`. - #### Disjoint pool TODO: Add a description @@ -326,6 +331,8 @@ Querying the latency value requires HMAT support on the platform. Calling `umfMe UMF provides the UMF proxy library (`umf_proxy`) that makes it possible to override the default allocator in other programs in both Linux and Windows. +To enable this feature, the `UMF_BUILD_SHARED_LIBRARY` option needs to be turned `ON`. + #### Linux In case of Linux it can be done without any code changes using the `LD_PRELOAD` environment variable: diff --git a/scripts/docs_config/api.rst b/scripts/docs_config/api.rst index c0448f117..3eedc8f1d 100644 --- a/scripts/docs_config/api.rst +++ b/scripts/docs_config/api.rst @@ -58,6 +58,9 @@ supported by the Proxy Pool. Scalable Pool ------------------------------------------ + +A oneTBB-based memory pool manager. + .. doxygenfile:: pool_scalable.h :sections: define enum typedef func var @@ -104,10 +107,18 @@ A memory provider that provides memory from L0 device. .. doxygenfile:: provider_level_zero.h :sections: define enum typedef func var +CUDA Provider +------------------------------------------ + +A memory provider that provides memory from CUDA device. + +.. doxygenfile:: provider_cuda.h + :sections: define enum typedef func var + DevDax Memory Provider ------------------------------------------ -A memory provider that provides memory from a device DAX (a character device file /dev/daxX.Y). +A memory provider that provides memory from a device DAX (a character device file like /dev/daxX.Y). .. doxygenfile:: provider_devdax_memory.h :sections: define enum typedef func var From 718c61dbf5ac76fd72166b9409d1a97644d4b34f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C5=81ukasz=20Stolarczuk?= Date: Mon, 16 Dec 2024 15:51:24 +0100 Subject: [PATCH 2/4] Move docs content into a separate dir and update the script to work outside of scripts dir. --- .github/workflows/docs.yml | 8 +++--- .github/workflows/reusable_docs_build.yml | 6 +++-- .gitignore | 2 +- RELEASE_STEPS.md | 2 +- docs/README.md | 8 ++++++ .../assets/images/intro_architecture.png | Bin {scripts/docs_config => docs/config}/Doxyfile | 2 +- {scripts/docs_config => docs/config}/api.rst | 0 {scripts/docs_config => docs/config}/conf.py | 4 ++- .../docs_config => docs/config}/examples.rst | 0 .../docs_config => docs/config}/glossary.rst | 0 .../docs_config => docs/config}/index.rst | 0 .../config}/introduction.rst | 0 {scripts => docs}/generate_docs.py | 24 +++++++++++++----- scripts/README.md | 5 ---- 15 files changed, 41 insertions(+), 20 deletions(-) create mode 100644 docs/README.md rename {scripts => docs}/assets/images/intro_architecture.png (100%) rename {scripts/docs_config => docs/config}/Doxyfile (99%) rename {scripts/docs_config => docs/config}/api.rst (100%) rename {scripts/docs_config => docs/config}/conf.py (92%) rename {scripts/docs_config => docs/config}/examples.rst (100%) rename {scripts/docs_config => docs/config}/glossary.rst (100%) rename {scripts/docs_config => docs/config}/index.rst (100%) rename {scripts/docs_config => docs/config}/introduction.rst (100%) rename {scripts => docs}/generate_docs.py (71%) delete mode 100644 scripts/README.md diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml index 3d9bfc29b..87e34cc74 100644 --- a/.github/workflows/docs.yml +++ b/.github/workflows/docs.yml @@ -41,13 +41,15 @@ jobs: run: echo "$HOME/.local/bin" >> $GITHUB_PATH - name: Build the documentation - working-directory: scripts - run: python3 generate_docs.py + run: | + mkdir build + cd build + python3 ../docs/generate_docs.py - name: Upload artifact uses: actions/upload-pages-artifact@0252fc4ba7626f0298f0cf00902a25c6afc77fa8 # v3.0.0 with: - path: docs/html + path: build/docs_build/generated/html deploy: name: Deploy docs to GitHub Pages diff --git a/.github/workflows/reusable_docs_build.yml b/.github/workflows/reusable_docs_build.yml index 269560c67..6702f9a66 100644 --- a/.github/workflows/reusable_docs_build.yml +++ b/.github/workflows/reusable_docs_build.yml @@ -30,5 +30,7 @@ jobs: python3 -m pip install -r third_party/requirements.txt - name: Build the documentation - working-directory: scripts - run: python3 generate_docs.py + run: | + mkdir build + cd build + python3 ../docs/generate_docs.py diff --git a/.gitignore b/.gitignore index a1a488bc1..e177e395e 100644 --- a/.gitignore +++ b/.gitignore @@ -58,7 +58,7 @@ __pycache__/ *.py[cod] # Generated docs -docs/ +docs_build/ # Build files /build*/ diff --git a/RELEASE_STEPS.md b/RELEASE_STEPS.md index fb46f156b..efdadbe9f 100644 --- a/RELEASE_STEPS.md +++ b/RELEASE_STEPS.md @@ -41,7 +41,7 @@ Do changes for a release: - For major releases mention API and ABI compatibility with the previous release - Update project's version in a few places: - For major and minor releases: `UMF_VERSION_CURRENT` in `include/umf/base.h` (the API version) - - `release` variable in `scripts/docs_config/conf.py` (for docs) + - `release` variable in `docs/config/conf.py` (for docs) - `UMF_VERSION` variable in `.github/workflows/reusable_basic.yml` (for installation test) - For major releases update ABI version in `.map` and `.def` files - These files are defined for all public libraries (`libumf` and `proxy_lib`, at the moment) diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 000000000..1124b53bd --- /dev/null +++ b/docs/README.md @@ -0,0 +1,8 @@ +To generate HTML documentation run the `generate_docs.py` script from the `build` dir. +It will create extra `./docs_build` directory, where the intermediate and final files +will be created. HTML docs will be in the `./docs_build/generated/html` directory. + +The script requires: + * [Doxygen](http://www.doxygen.nl/) at least v1.9.1 + * [Python](https://www.python.org/downloads/) at least v3.8 + * and python pip requirements, as defined in `third_party/requirements.txt` diff --git a/scripts/assets/images/intro_architecture.png b/docs/assets/images/intro_architecture.png similarity index 100% rename from scripts/assets/images/intro_architecture.png rename to docs/assets/images/intro_architecture.png diff --git a/scripts/docs_config/Doxyfile b/docs/config/Doxyfile similarity index 99% rename from scripts/docs_config/Doxyfile rename to docs/config/Doxyfile index 43ff2a603..f23117ff2 100644 --- a/scripts/docs_config/Doxyfile +++ b/docs/config/Doxyfile @@ -2058,7 +2058,7 @@ GENERATE_XML = YES # The default directory is: xml. # This tag requires that the tag GENERATE_XML is set to YES. -XML_OUTPUT = ../docs/xml +XML_OUTPUT = docs_build/doxyxml # If the XML_PROGRAMLISTING tag is set to YES, doxygen will dump the program # listings (including syntax highlighting and cross-referencing information) to diff --git a/scripts/docs_config/api.rst b/docs/config/api.rst similarity index 100% rename from scripts/docs_config/api.rst rename to docs/config/api.rst diff --git a/scripts/docs_config/conf.py b/docs/config/conf.py similarity index 92% rename from scripts/docs_config/conf.py rename to docs/config/conf.py index 577bc0b48..3af2df378 100644 --- a/scripts/docs_config/conf.py +++ b/docs/config/conf.py @@ -49,7 +49,9 @@ # -- Extension configuration ------------------------------------------------- # -- Options for breathe extension ------------------------------------------- -breathe_projects = {project: "../../docs/xml"} +# 'doxyxml' dir is generated with Doxygen; it's supposed to be in a directory +# one above the config directory. +breathe_projects = {project: "../doxyxml"} breathe_default_project = project breathe_show_include = False breathe_default_members = ("members", "undoc-members") diff --git a/scripts/docs_config/examples.rst b/docs/config/examples.rst similarity index 100% rename from scripts/docs_config/examples.rst rename to docs/config/examples.rst diff --git a/scripts/docs_config/glossary.rst b/docs/config/glossary.rst similarity index 100% rename from scripts/docs_config/glossary.rst rename to docs/config/glossary.rst diff --git a/scripts/docs_config/index.rst b/docs/config/index.rst similarity index 100% rename from scripts/docs_config/index.rst rename to docs/config/index.rst diff --git a/scripts/docs_config/introduction.rst b/docs/config/introduction.rst similarity index 100% rename from scripts/docs_config/introduction.rst rename to docs/config/introduction.rst diff --git a/scripts/generate_docs.py b/docs/generate_docs.py similarity index 71% rename from scripts/generate_docs.py rename to docs/generate_docs.py index d5b2a0128..1697eacfe 100644 --- a/scripts/generate_docs.py +++ b/docs/generate_docs.py @@ -6,17 +6,20 @@ """ from pathlib import Path -from shutil import rmtree +from shutil import rmtree, copytree import subprocess # nosec B404 import time def _check_cwd() -> None: - script_path = Path(__file__).resolve().parent cwd = Path.cwd() - if script_path != cwd: + include_dir = Path(cwd, "../include") + # Verify if include dir is one level up (as defined in Doxyfile) + if not include_dir.exists(): print( - f"{__file__} script has to be run from the 'scripts' directory. Terminating..." + f"Include directory {include_dir.resolve()} not found! " + "Please run this script from /build!", + flush=True, ) exit(1) @@ -66,8 +69,17 @@ def _generate_html(config_path: Path, docs_path: Path) -> None: def main() -> None: _check_cwd() - config_path = Path("docs_config").resolve() - docs_path = Path("..", "docs").resolve() + + script_dir = Path(__file__).resolve().parent + docs_build_path = Path("docs_build").resolve() + + # Sphinx and breathe require access to a Doxygen generated dir ('doxyxml') + # so we copy the whole content of the 'docs' dir to the build dir. + copytree(Path(script_dir), docs_build_path, dirs_exist_ok=True) + + config_path = Path(docs_build_path, "config").resolve() + docs_path = Path(docs_build_path, "generated").resolve() + start = time.time() _prepare_docs_dir(docs_path) _generate_xml(config_path, docs_path) diff --git a/scripts/README.md b/scripts/README.md deleted file mode 100644 index e3a9ed533..000000000 --- a/scripts/README.md +++ /dev/null @@ -1,5 +0,0 @@ -The documentation HTML files are generated using the following dependencies: - * [Python](https://www.python.org/downloads/) at least v3.8 - * [Doxygen](http://www.doxygen.nl/) at least v1.9.1 - - To generate files run the `generate_docs.py` script from the `scripts` directory. Files will be generated to the `docs/html` directory relative to the main directory of this repository. From 78d27981ba94512c27c6ac774f340eff71551a9e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C5=81ukasz=20Stolarczuk?= Date: Tue, 17 Dec 2024 10:57:31 +0100 Subject: [PATCH 3/4] [CMake] Add 'docs' target --- .github/workflows/docs.yml | 5 ++--- .github/workflows/reusable_docs_build.yml | 10 +++++++--- CMakeLists.txt | 11 +++++++++++ docs/README.md | 22 +++++++++++++++++++--- 4 files changed, 39 insertions(+), 9 deletions(-) diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml index 87e34cc74..c507f7994 100644 --- a/.github/workflows/docs.yml +++ b/.github/workflows/docs.yml @@ -42,9 +42,8 @@ jobs: - name: Build the documentation run: | - mkdir build - cd build - python3 ../docs/generate_docs.py + cmake -B build -DUMF_TESTS_FAIL_ON_SKIP=ON + cmake --build build --target docs - name: Upload artifact uses: actions/upload-pages-artifact@0252fc4ba7626f0298f0cf00902a25c6afc77fa8 # v3.0.0 diff --git a/.github/workflows/reusable_docs_build.yml b/.github/workflows/reusable_docs_build.yml index 6702f9a66..92dcda555 100644 --- a/.github/workflows/reusable_docs_build.yml +++ b/.github/workflows/reusable_docs_build.yml @@ -31,6 +31,10 @@ jobs: - name: Build the documentation run: | - mkdir build - cd build - python3 ../docs/generate_docs.py + cmake -B build \ + -DUMF_BUILD_LEVEL_ZERO_PROVIDER=OFF \ + -DUMF_BUILD_CUDA_PROVIDER=OFF \ + -DUMF_BUILD_TESTS=OFF \ + -DUMF_BUILD_EXAMPLES=OFF \ + -DUMF_DISABLE_HWLOC=ON + cmake --build build --target docs diff --git a/CMakeLists.txt b/CMakeLists.txt index 0b88f95b5..4e181f246 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -760,6 +760,17 @@ if(UMF_FORMAT_CODE_STYLE) endif() endif() +find_package(Python3 3.8) +if(Python3_FOUND) + message(STATUS "Adding 'docs' target for creating a documentation.") + add_custom_target( + docs + WORKING_DIRECTORY ${CMAKE_BINARY_DIR} + COMMAND ${Python3_EXECUTABLE} + ${UMF_CMAKE_SOURCE_DIR}/docs/generate_docs.py + COMMENT "Generate HTML documentation using Doxygen") +endif() + # --------------------------------------------------------------------------- # # Configure make install/uninstall and packages # --------------------------------------------------------------------------- # diff --git a/docs/README.md b/docs/README.md index 1124b53bd..3564d86db 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,8 +1,24 @@ -To generate HTML documentation run the `generate_docs.py` script from the `build` dir. -It will create extra `./docs_build` directory, where the intermediate and final files +# Documentation + +To generate HTML documentation run the `generate_docs.py` script from any sub-dir of the +repository (most likely `build`) or enable and use build target 'docs' (see details below). + +This script will create `./docs_build` sub-directory, where the intermediate and final files will be created. HTML docs will be in the `./docs_build/generated/html` directory. -The script requires: +## make docs + +To run documentation generation via build target use CMake commands below. +To enable this target, python executable (in required version) has to be found in the system. + +```bash +$ cmake -B build +$ cmake --build build --target docs +``` + +## Requirements + +Script to generate HTML docs requires: * [Doxygen](http://www.doxygen.nl/) at least v1.9.1 * [Python](https://www.python.org/downloads/) at least v3.8 * and python pip requirements, as defined in `third_party/requirements.txt` From c7fdc11978e965af6dd4a2fce52320967b20e434 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C5=81ukasz=20Stolarczuk?= Date: Tue, 17 Dec 2024 11:04:19 +0100 Subject: [PATCH 4/4] [CI] make docs workflow reusable in deploy job --- .github/workflows/docs.yml | 43 ++++------------------- .github/workflows/reusable_docs_build.yml | 14 +++++++- 2 files changed, 19 insertions(+), 38 deletions(-) diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml index c507f7994..165cc1754 100644 --- a/.github/workflows/docs.yml +++ b/.github/workflows/docs.yml @@ -14,45 +14,14 @@ permissions: contents: read jobs: - build: - name: Build docs - runs-on: ${{ github.repository_owner == 'oneapi-src' && 'intel-ubuntu-22.04' || 'ubuntu-latest' }} - - steps: - - name: Checkout repository - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1 - with: - fetch-depth: 0 - - - name: Install doxygen - run: | - sudo apt-get update - sudo apt-get install -y doxygen - - # Latest distros do not allow global pip installation - - name: Install Python requirements in venv - run: | - python3 -m venv .venv - . .venv/bin/activate - echo "$PATH" >> $GITHUB_PATH - python3 -m pip install -r third_party/requirements.txt - - - name: Setup PATH for python - run: echo "$HOME/.local/bin" >> $GITHUB_PATH - - - name: Build the documentation - run: | - cmake -B build -DUMF_TESTS_FAIL_ON_SKIP=ON - cmake --build build --target docs - - - name: Upload artifact - uses: actions/upload-pages-artifact@0252fc4ba7626f0298f0cf00902a25c6afc77fa8 # v3.0.0 - with: - path: build/docs_build/generated/html + DocsBuild: + uses: ./.github/workflows/reusable_docs_build.yml + with: + upload: true - deploy: + DocsDeploy: name: Deploy docs to GitHub Pages - needs: build + needs: DocsBuild permissions: pages: write diff --git a/.github/workflows/reusable_docs_build.yml b/.github/workflows/reusable_docs_build.yml index 92dcda555..e90ca87ae 100644 --- a/.github/workflows/reusable_docs_build.yml +++ b/.github/workflows/reusable_docs_build.yml @@ -1,6 +1,12 @@ name: Docs build -on: workflow_call +on: + workflow_call: + inputs: + upload: + description: Should HTML documentation be uploaded as artifact? + type: boolean + default: false permissions: contents: read @@ -38,3 +44,9 @@ jobs: -DUMF_BUILD_EXAMPLES=OFF \ -DUMF_DISABLE_HWLOC=ON cmake --build build --target docs + + - name: Upload artifact + if: ${{ inputs.upload == true }} + uses: actions/upload-pages-artifact@0252fc4ba7626f0298f0cf00902a25c6afc77fa8 # v3.0.0 + with: + path: build/docs_build/generated/html