-
Notifications
You must be signed in to change notification settings - Fork 172
Repository Split
This document describes the plan for splitting the current KhronosGroup/Vulkan-LoaderAndValidationLayers GitHub repository into several new KhronosGroup GitHub repositories.
The motivation for splitting the repository comes from:
- The Vulkan Ecosystem group has voiced a preference for the Vulkan Loader and the Vulkan Validation Layers to be in separate repositories since they have differing scopes. E.g., the layers are more portable than the (desktop) loader, making the target platform sets different for each.
- The loader, layers, and demos/examples are fairly independent of each other and fall into a more natural separation and organization if in separate repositories.
- The repositories are easier to maintain when they contain fewer components.
- The separate repositories more easily facilitate the generation of Linux packages.
- With a widening base of contributors working with especially the validation layers, having more focussed repositories should make it easier for these contributors.
- Repository dependencies can be fine-tuned to each of the separate repositories, reducing the need for dependencies in many of them. For example, it would no longer be necessary to get and build glslang to build the loader.
This is a high-level layout of the new repository structure that illustrates the new location of the components from the old Vulkan-LoaderAndValidationLayers (LVL) repository.
-
KhronosGroup/Vulkan-Headers
- vulkan.h and headers it requires
- ICD and layer interface headers (separate dir from above)
- vulkan.hpp
- vulkan registry
-
KhronosGroup/Vulkan-Loader
- desktop loader
- desktop loader tests
-
KhronosGroup/Vulkan-ValidationLayers
- validation layers
- validation layers tests
-
KhronosGroup/Vulkan-Tools
- vulkaninfo
- cube, cubepp
- mock ICD (used to test validation layers)
The new repositories will NOT simply include each other using some mechanism such as git submodules in order to allow building the repositories in a "standalone" fashion. Doing so defeats part of the purpose of splitting the repositories. Instead, each new repository may list components originating from one of the other new repositories as being an "installed dependency", much like needing to have X11 libraries to build the current repository on Linux.
For example, the new Vulkan-ValidationLayers needs the Vulkan loader to run
the validation layers test suite.
This repository would then expect the loader to be installed on the system
in some fashion so that the test suite can access it.
On a Windows platform, this may not involve much effort since
most driver installs also install the loader.
On Linux, there are Vulkan packages available that the user can install.
For both platforms, there is also the option to install and use the SDK by
setting the VULKAN_SDK environment variable.
If a user needs a specific version of the loader, they can always build
the Vulkan-Loader repo and use the loader from there.
An exception to the above will likely involve making the Vulkan-Headers repository a Git submodule of the others. This arrangement will be useful for synchronizing and enforcing the code in each of these repositories to be compatible with a given version of the headers. Note that adding Vulkan-Headers as a Git submodule in the others is still a bit undecided at this point. Using git subtrees is another possible solution. We could still take the approach of requiring the user to provide the headers via an SDK install or other means. But we are leaning towards using Vulkan-Headers as a submodule these repositories that need the headers.
The new repositories will also be modified to look for other Khronos-sourced repositories such as glslang on the local machine instead of defaulting to building them for each repository that needs them. Here, the CMake code for these new repositories will be enhanced to look for these tools in standard system locations and in Vulkan SDK installations. If not found there, the CMake code can then clone and build the required repositories.
Note that in cases like glslang, the glslang repository contains binary releases. The CMake code can then pull down these release archives (zip files) and use the binaries directly as an alternative to cloning and building.
The user will also have the option of passing in a specific repository URL and git revision to the CMake code, which will make it skip the local search and fetch/build/use the requested version.
LunarG is taking the primary responsibility for executing this plan.
This document is placed in the KhronosGroup/Vulkan-LoaderAndValidationLayers wiki and there is an issue open for public comment. This issue is also used to communicate plans and progress.
LunarG expects to work on prototyping and designing the new repositories while preparing the release of the next LunarG SDK, which is planned for April 19, 2018. Much of this work is expected to be performed in private repositories to avoid the confusion of these repositories being visible to the public at the same time the current one is visible.
After the April SDK release, the new repositories will go public and the current Vulkan-LoaderAndValidationLayers repository will go into read-only deprecated mode. This is expected within a few weeks of the SDK release.
There is no planned date to delete the Vulkan-LoaderAndValidationLayers repository and it should probably remain "forever" to serve as a historical record. The Vulkan-LoaderAndValidationLayers repository is also left intact for the purpose of not "breaking" existing processes that rely on it. These users of the Vulkan-LoaderAndValidationLayers repository can continue to use the repository in a read-only fashion and only need to migrate to the new repositories when convenient or when they need updated content.
Only the members with "Admin" access will be able to write to the old repository after the switchover. The new repositories will be set up with the same access permissions as the current repository.
There are a few goals to keep in mind when creating the new repositories:
- All files remaining in the new repositories must keep their histories.
- The histories of all files not appearing in the new repositories must be removed.
The value of the first goal is obvious. The value of the second goal is to save space in the new repositories and to remove history that may be confusing.
The general approach for creating the new repositories will be creating copies (clones) of the KhronosGroup/Vulkan-LoaderAndValidationLayers repository and removing unwanted files from each copy, including their histories.
There are several recipes for this procedure which can be located using WWW searches for phrases like "git split repository".
But here is one suggested approach that should meet our needs:
git clone [email protected]:KhronosGroup/Vulkan-LoaderAndValidationLayers.git Vulkan-Loader.tmp
cd Vulkan-Loader.tmp
git filter-branch --index-filter 'git rm --ignore-mismatch --cached -r DIRS'\
--tag-name-filter cat --prune-empty -- --all
cd ..
git clone --no-hardlinks Vulkan-Loader.tmp Vulkan-Loader
-
In the above,
DIRSis a list of files and directories that are to be deleted completely from the repository history. -
This operation can take from several minutes to hours, depending on the machine.
-
To avoid making these expensive filter-branch passes, it may be best to come up with a complete list for
DIRSso that all the unwanted files can be removed with one pass. -
Note that the operation can be sped up by using the
-doption forgit filter-branchto specify a directory intmpfsor ram disk to speed up the disk operations for temporary files. -
The last clone is needed to get rid of
.git/refs/original/refs/heads...entries in the local repository which are backups created bygit filter-branchand take up a lot of space. -
The remote
originshould then be removed and replaced with the new remote on GitHub.
Then the repository is ready for additional work and can be force pushed to GitHub.
Pull requests left in the Vulkan-LoaderAndValidationLayers repository at the time of the switchover need to be manually migrated to the appropriate new repository by the pull request owner. There are likely few "automatic" git processes to do this such as cherry-picks, although the git diff-patch tools may be of assistance.
It is therefore obviously advantageous to close out any pending pull requests before the switchover.
GitHub does not support moving issues from one project to another. As with the git cloning process, there are also recipes out on the WWW that describe several approaches for moving issues. Most involve the use of scripting and the GitHub API. The choice of the exact tool and process has not been made yet.
Goals for Issue Migration
- Migrate only open issues from LVL to the appropriate new repository
- Most of these probably go into Vulkan-ValidationLayers
- Retain original owner of the issue
- Retain as much content (e.g., comments) as possible
- Labels need to be manually established in the new repositories
- Labels need to be manually reapplied to each new issue
We expect to leave the closed issues in the old repository, which can be referenced directly if needed.
LunarG plans to execute this issue migration step as described here as the new repositories become active.
This new repository configuration obviously calls for a commensurate effort to reorganize Travis-CI and AppVeyor CI testing. LunarG will migrate the CI configurations as part of this repository splitting effort.
The Vulkan specification contains references to objects in the original KhronosGroup/Vulkan-LoaderAndValidationLayers repository. Examples of these references include but are not limited to:
- Vulkan Loader Specification and Architecture Overview
- Numerous references from the WSI sections to code in the cube sample demo
As part of this effort, LunarG will submit PRs to update these references. Since the old repository will remain for some time, there is not enough urgency to make these changes to the Vulkan specification in tight coordination with the repository rollout.