Skip to content
This repository has been archived by the owner on Jul 19, 2018. It is now read-only.

Repository Split

Jump to bottom
Karl Schultz edited this page Mar 29, 2018 · 3 revisions

Vulkan-LoaderAndValidationLayers Repository Split

This document describes the plan for splitting the current KhronosGroup/Vulkan-LoaderAndValidationLayers GitHub repository into several new KhronosGroup GitHub repositories and moving some items to existing repositories.

Motivation

The motivation for splitting the repository comes from:

  • The Vulkan Ecosystem group has voiced a preference for the Vulkan Loader and the Vulkan Validation Layers to be in separate repositories since they have differing scopes. E.g., the layers are more portable than the (desktop) loader, making the target platform sets different for each.
  • The loader, layers, and demos/examples are fairly independent of each other and fall into a more natural separation and organization if in separate repositories.
  • The repositories are easier to maintain when they contain fewer components.
  • The separate repositories more easily facilitate the generation of Linux packages.
  • With a widening base of contributors working with especially the validation layers, having more focussed repositories should make it easier for these contributors.
  • Repository dependencies can be fine-tuned to each of the separate repositories, reducing the need for dependencies in many of them. For example, it would no longer be necessary to get and build glslang to build the loader.

New Repositories and Their Contents

This is a high-level layout of the new repository structure that illustrates the new location of the components from the old Vulkan-LoaderAndValidationLayers (LVL) repository.

  • KhronosGroup/Vulkan-Headers

    • vulkan.h and headers it requires
    • ICD and layer interface headers (separate dir from above)
    • vulkan.hpp
    • vulkan registry

  • KhronosGroup/Vulkan-Loader

    • desktop loader
    • vulkaninfo

  • KhronosGroup/Vulkan-ValidationLayers

    • validation layers
    • validation layers tests
    • mock ICD (used to test validation layers)

  • KhronosGroup/Vulkan-Samples (existing repository)

    • add cube
    • add cubepp
    • retire smoketest (duplicate of Hologram in LunarG/VulkanSamples)

New Repositories and Their Dependencies

New Repository Interdependencies

The new repositories will NOT simply include each other as git submodules in order to allow building the repositories in a "standalone" fashion. Doing so defeats part of the purpose of splitting the repositories. Instead, each new repository may list another new repository as being an "installed dependency", much like needing to have X11 libraries to build the current repository on Linux.

For example, the new Vulkan-ValidationLayers needs the Vulkan loader to run its test suite. This repository would then expect the loader to be installed on the system in some fashion so that the test suite can access it. On a Windows platform, this may not involve much effort since most driver installs also install the loader. On Linux, there are vulkan packages available that the user can install. For both platforms, there is also the option to install and use the SDK by setting the VULKAN_SDK environment variable. If a user needs a specific version of the loader, they can always build the Vulkan-Loader repo and use the loader from there.

An exception to the above will likely involve making the Vulkan-Headers repository a submodule of the others. This arrangement will be useful for synchronizing and enforcing the code in each of these repositories to be compatible with a given version of the headers. Note that adding Vulkan-Headers as a submodule in the others is still a bit undecided at this point. We could still take the approach of requiring the user to provide the headers via an SDK install or other means. But we are leaning towards using Vulkan-Headers as a submodule these repositories that need the headers.

Dependencies on Other Projects

The new repositories will also be modified to look for other Khronos-sourced repositories such as glslang on the local machine instead of defaulting to building them for each repository that needs them. Here, the CMake code for these new repositories will be enhanced to look for these tools in standard system locations and in Vulkan SDK installations. If not found there, the CMake code can then clone and build the required repositories.

Note that in cases like glslang, the glslang repository contains binary releases. The CMake code can then pull down these release archives (zip files) and use the binaries directly as an alternative to cloning and building.

The user will also have the option of passing in a specific repository URL and git revision to the CMake code, which will make it skip the local search and fetch/build/use the requested version.

Execution and Migration Plan

LunarG is taking the primary responsibility for executing this plan.

This document is placed in the KhronosGroup/Vulkan-LoaderAndValidationLayers wiki and there is an issue open for public comment. This issue is also used to communicate plans and progress.

Timeline

LunarG expects to work on creating and configuring the new repositories up until the release of the next LunarG SDK, which is planned for April 19, 2018. Much of this work is expected to be performed in private repositories to avoid the confusion of these repositories being visible to the public at the same time the current one is visible. Persons interested in accessing the WIP private repositories may contact LunarG.

On some day soon after the April SDK release (TBD), the new repositories will go public and the current Vulkan-LoaderAndValidationLayers repository will go into read-only deprecated mode.

There is no planned date to delete the Vulkan-LoaderAndValidationLayers repository and it should probably remain "forever" to serve as a historical record. The Vulkan-LoaderAndValidationLayers repository is also left intact for the purpose of not "breaking" existing processes that rely on it. These users of the Vulkan-LoaderAndValidationLayers repository can continue to use the repository in a read-only fashion and only need to migrate to the new repositories when convenient or when they need updated content.

Access Permissions

Only the members with "Admin" access will be able to write to the old repository after the switchover. The new repositories will be set up with the same access permissions as the current repository.

GitHub New Repository Creation

There are a few goals to keep in mind when creating the new repositories:

  • All files remaining in the new repositories must keep their histories.
  • The histories of all files not appearing in the new repositories must be removed.

The value of the first goal is obvious. The value of the second goal is to save space in the new repositories and to remove history that may be confusing.

The general approach for creating the new repositories will be creating copies (clones) of the KhronosGroup/Vulkan-LoaderAndValidationLayers repository and removing unwanted files from each copy, including their histories.

There are several recipes for this procedure which can be located using WWW searches for phrases like "git split repository".

But here is one suggested approach that should meet our needs:

git clone [email protected]:KhronosGroup/Vulkan-LoaderAndValidationLayers.git Vulkan-Loader.tmp
cd Vulkan-Loader.tmp
git filter-branch --index-filter 'git rm --ignore-mismatch --cached -r DIRS'\
    --tag-name-filter cat --prune-empty -- --all
cd ..
git clone --no-hardlinks Vulkan-Loader.tmp Vulkan-Loader

Notes

  • In the above, DIRS is a list of files and directories that are to be deleted completely from the repository history.

  • This operation can take from several minutes to hours, depending on the machine.

  • To avoid making these expensive filter-branch passes, it may be best to come up with a complete list for DIRS so that all the unwanted files can be removed with one pass.

  • Note that the operation can be sped up by using the -d option for git filter-branch to specify a directory in tmpfs or ram disk to speed up the disk operations for temporary files.

  • The last clone is needed to get rid of .git/refs/original/refs/heads... entries in the local repository which are backups created by git filter-branch and take up a lot of space.

  • The remote origin should then be removed and replaced with the new remote on GitHub.

Then the repository is ready for additional work and can be force pushed to GitHub.

Pending Pull Requests

Pull requests left in the Vulkan-LoaderAndValidationLayers repository at the time of the switchover need to be manually migrated to the appropriate new repository by the pull request owner. There are likely few "automatic" git processes to do this such as cherry-picks, although the git diff-patch tools may be of assistance.

It is therefore obviously advantageous to close out any pending pull requests before the switchover.

GitHub Issue Migration

GitHub does not support moving issues from one project to another. As with the git cloning process, there are also recipes out on the WWW that describe several approaches for moving issues. Most involve the use of scripting and the GitHub API. The choice of the exact tool and process has not been made yet.

Goals for Issue Migration

  • Migrate only open issues from LVL to the appropriate new repository
    • Most of these probably go into Vulkan-ValidationLayers
  • Retain original owner of the issue
  • Retain as much content (e.g., comments) as possible
  • Labels need to be manually established in the new repositories
  • Labels need to be manually reapplied to each new issue

We expect to leave the closed issues in the old repository, which can be referenced directly if needed.

LunarG plans to execute this issue migration step as described here as the new repositories become active.

CI Migration

This new repository configuration obviously calls for a commensurate effort to reorganize Travis-CI and AppVeyor CI testing. LunarG will migrate the CI configurations as part of this repository splitting effort.

The KhronosGroup/Vulkan-Samples Repository

This is an existing repository that seems fairly dormant. The reconfiguration described above calls for moving the cube demos to this repository. This seems like a reasonable place for these demos since the demos have some visibility in the ecosystem and the repository is already Khronos-sponsored and branded.

LunarG is proposing to add these demos to this repository alongside the current contents of the repository. To this end, LunarG is requesting admin-level GitHub access on this repository so that it can maintain the cube demos effectively. However, LunarG is NOT taking responsibility for the curation of the entire repository. LunarG expects the current repository curators to continue to manage the existing content and any content added in the future that is not sanctioned by LunarG.

Vulkan Specification References

The Vulkan specification contains references to objects in the original KhronosGroup/Vulkan-LoaderAndValidationLayers repository. Examples of these references include but are not limited to:

  • Vulkan Loader Specification and Architecture Overview
  • Numerous references from the WSI sections to code in the cube sample demo

As part of this effort, LunarG will submit PRs to update these references. Since the old repository will remain for some time, there is not enough urgency to make these changes to the Vulkan specification in tight coordination with the repository rollout.

Clone this wiki locally