From b6e1d72b234ce93c5a87efb4f1539802f79c1f12 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Samuel=20D=C3=A9fago?= Date: Thu, 5 Dec 2024 15:12:49 +0100 Subject: [PATCH] Update tooling documentation (#1091) --- README.md | 2 +- docs/CONTINUOUS_INTEGRATION.md | 180 +++++---------------------------- docs/DEVELOPMENT_SETUP.md | 124 ----------------------- docs/DEVELOPMENT_TOOLS.md | 111 ++++++++++++++++++++ 4 files changed, 136 insertions(+), 281 deletions(-) delete mode 100644 docs/DEVELOPMENT_SETUP.md create mode 100644 docs/DEVELOPMENT_TOOLS.md diff --git a/README.md b/README.md index c55cb88fa..023c2c317 100644 --- a/README.md +++ b/README.md @@ -93,7 +93,7 @@ Documentation is also available for companion products: Further documentation is also available by following the links below: - [Known issues](docs/KNOWN_ISSUES.md) -- [Development setup](docs/DEVELOPMENT_SETUP.md) +- [Development tools](docs/DEVELOPMENT_TOOLS.md) - [Continuous integration](docs/CONTINUOUS_INTEGRATION.md) # Plugins diff --git a/docs/CONTINUOUS_INTEGRATION.md b/docs/CONTINUOUS_INTEGRATION.md index e29d934dd..54dc94343 100644 --- a/docs/CONTINUOUS_INTEGRATION.md +++ b/docs/CONTINUOUS_INTEGRATION.md @@ -1,175 +1,43 @@ # Continuous integration -The project provides support for continuous integration. Several commands have been implemented using [fastlane](https://fastlane.tools) and are exposed in a `Makefile` for convenient use. The idea is to provide commands that can be executed locally as well as by a continuous integration server with minimal configuration. +Continuous integration is performed using GitHub Actions. Worflows require a few action secrets and variables to be set in the GitHub repository settings. -Commands are available to: +## Secrets -- Build the project documentation. -- Run unit tests. -- Archive the demo. -- Perform nightly and release deliveries on TestFlight. -- Run code quality checks. +Some information is kept confidential and stored in secrets. -We currently use TeamCity for continuous integration and GitHub for issue and pull request management. This document describes the steps required to fully integrate the tool suite with TeamCity and GitHub. We want to: +### Apple developer certificate -- Execute status checks and post results to GitHub when a pull request is opened, updated or [enqueued](https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/configuring-pull-request-merges/managing-a-merge-queue). -- Avoid running status checks for draft pull requests. -- Build demo apps during the night if changes have been pushed to `main` during the day. +An Apple developer certificate exported from Xcode, and protected by a password, is required. More information is available from the [GitHub documentation](https://docs.github.com/en/actions/use-cases-and-examples/deploying/installing-an-apple-certificate-on-macos-runners-for-xcode-development): -## Required tools +- `APPLE_DEV_CERTIFICATE`: The certificate `p12` file converted to Base64 can be copied to the clipboard with: -The continuous integration agents must have the following tools installed: + ```shell + base64 -i developer_certificate.p12 | pbcopy + ``` -- The latest version of Xcode (preferably several versions, including beta releases). -- The tvOS simulator (otherwise provisioning fails for tvOS when archiving the tvOS app). -- [Python](https://www.python.org) -- [gem](https://rubygems.org) -- [bundler](https://bundler.io) -- [ffmpeg](https://ffmpeg.org) -- [markdownlint-cli](https://github.com/igorshubovych/markdownlint-cli) -- [shellcheck](https://www.shellcheck.net) -- [swiftlint](https://github.com/realm/SwiftLint) -- [xcodes](https://github.com/RobotsAndPencils/xcodes) -- [yamllint](https://github.com/adrienverge/yamllint) +- `APPLE_DEV_CERTIFICATE_PASSWORD`: The password used to protect the developer certificate. -Most of these tools can easily be installed with [Homebrew](https://brew.sh). +### App Store Connect API key -## TeamCity configuration +An [App Store Connect API key](https://appstoreconnect.apple.com/access/integrations/api) is required for automatic provisioning: -TeamCity offers support for [GitHub hooks](https://github.com/JetBrains/teamcity-commit-hooks) to avoid polling GitHub for new commits. +- `APP_STORE_CONNECT_API_KEY`: The key `p8` file converted to Base64 can be copied to the clipboard with: -## Private configuration + ```shell + base64 -i api_key.p8 | pbcopy + ``` -Use of archive and delivery commands requires access to a [private configuration repository](https://github.com/SRGSSR/pillarbox-apple-configuration). This repository is transparently pulled before the commands are executed (provided the continuous integration server has access to it). +- `APP_STORE_CONNECT_KEY_ID`: The key ID. +- `APP_STORE_CONNECT_KEY_ISSUER_ID`: The key issuer ID. -## Workflow +### Other secrets -Our current workflow is based on pull requests and [merge queues](https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/configuring-pull-request-merges/managing-a-merge-queue), which TeamCity is able to automatically monitor with dedicated build features and triggers. +- `TEAM_ID`: The team ID. +- `KEYCHAIN_PASSWORD`: An arbitrary password used to create a local keychain to store certificates on GitHub runners. This password is only used to prevent secrets from being easily read from this keychain, should it somehow leak from a GitHub-hosted agent. -When a non-draft pull request is created or updated TeamCity triggers various status checks required for the pull request to be queued for merging. Pull request detection is performed using a dedicated build feature and results are posted to the GitHub pull request. +## Environment variables -Once all status checks are successful and the pull request reviewed it can be added to the merge queue. GitHub then automatically creates a temporary branch which is detected by a TeamCity branch trigger so that the status checks are automatically performed with the most recent changes from `main`. Once all status checks pass the pull request is merged automatically. +Non-confidential information is provided in environment variables: -## GitHub configuration - -To support our worflow GitHub `main` branch protection settings must be configured as follows: - -1. Enable _Require a pull request before merging_, _Require approvals_ with 1 approval as well as _Require approval of the most recent reviewable push_. -2. Enable _Require status checks to pass before merging_, _Require branches to be up to date before merging_ and the following required status checks: - a. Demo Archiving iOS (Apple) - b. Demo Archiving tvOS (Apple) - c. Documentation (Apple) - d. Quality (Apple) - e. Tests iOS (Apple) - f. Tests tvOS (Apple) -3. Enable _Require conversation resolution before merging_. -4. Enable _Require signed commits_. -5. Enable _Require linear history_. -6. Enable _Require linear history_ with _Squash and merge_ as method and _Only merge non-failing pull requests_. -7. Enable _Do not allow bypassing the above settings_. - -In the general project settings: - -1. Disable _Allow merge commits_. -2. Disable _Allow rebase merging_. -3. Enable _Allow squash merging_ with _Default to pull request title_. -4. Enable _Allow auto-merge_. -5. Enable _Always suggest updating pull request branches_. -6. Enable _Automatically delete head branches_. - -## Continuous integration user - -Proper integration with GitHub requires the use of a dedicated continuous integration user (a bot) with write access to the repository. We already have a dedicated [RTS devops](https://github.com/rts-devops) user, we therefore only need a few additional configuration steps: - -1. Ensure the bot has write access to the GitHub repository. -2. Create a [personal access token](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token) with the following minimal permissions: - a. _public_repo_ since our repository is public. - b. _read:org_ so that the pull request feature is able to read members of the organization. This way only PRs emerging from members trigger jobs (this is namely the default for the GitHub [pull request build feature](https://www.jetbrains.com/help/teamcity/pull-requests.html#Bitbucket+Cloud+Pull+Requests)). - -Of course a proper SSH setup is also required so that main and configuration repositories can be properly pulled by the continuous integration server. - -## App Store Connect configuration - -Nightly and release applications must be created with proper identifiers first on App Store Connect. For submission to succeed each application _Beta App Information_ and _Beta App Review Information_ must also have been properly filled in the TestFlight section of the App Store Connect portal. - -## Quality checks - -To have TeamCity run quality checks for GitHub pull requests and post the corresponding status back to GitHub: - -1. Create a TeamCity configuration called _Quality_. -2. Add a _VCS Trigger_ on `+:pull/*` and `+:gh-readonly-queue/*`. -3. Add a _Command Line_ build step which simply executes `make check-quality`. -4. Add a _Pull Requests_ build feature which monitors GitHub (requires a personal access token). Check the _Ignore Drafts_ option to avoid triggering work for draft pull requests. -5. Add a _Commit status publisher_ build feature which posts to GitHub (requires a personal access token). -6. Add an _Agent Requirement_ ensuring that `tools.xcode.home` exists. Check that some agents are compatible and assignable (if agents are configured manually you might need to explicitly allow the configuration to be run). - -## Demo archiving - -To have TeamCity archive the demo (archive for all configurations without TestFlight submission) for GitHub pull requests and post the corresponding status back to GitHub: - -1. Create a TeamCity configuration called _Demo Archiving iOS_. -2. Add a _VCS Trigger_ on `+:pull/*` and `+:gh-readonly-queue/*`. -3. Add a _Command Line_ build step which simply executes `make archive-demo-ios`. -4. Add a _Pull Requests_ build feature which monitors GitHub (requires a personal access token). Check the _Ignore Drafts_ option to avoid triggering work for draft pull requests. -5. Add a _Commit status publisher_ build feature which posts to GitHub (requires a personal access token). -6. Add an _Agent Requirement_ ensuring that `tools.xcode.home` exists. Check that some agents are compatible and assignable (if agents are configured manually you might need to explicitly allow the configuration to be run). - -For comprehensive results a second _Demo Archiving tvOS_ configuration must be created for tvOS. This is easily achieved by copying the configuration you just created and editing the _Command Line_ build step to execute `make archive-demo-tvos`. - -## Documentation - -To have TeamCity build and validate the documentation for GitHub pull requests and post the corresponding status back to GitHub: - -1. Create a TeamCity configuration called _Documentation_. -2. Add a _VCS Trigger_ on `+:gh-readonly-queue/*`. -3. Add a _Command Line_ build step which simply executes `make doc`. -4. Add a _Pull Requests_ build feature which monitors GitHub (requires a personal access token). Check the _Ignore Drafts_ option to avoid triggering work for draft pull requests. -5. Add a _Commit status publisher_ build feature which posts to GitHub (requires a personal access token). -6. Add an _Agent Requirement_ ensuring that `tools.xcode.home` exists. Check that some agents are compatible and assignable (if agents are configured manually you might need to explicitly allow the configuration to be run). - -## Tests - -To have TeamCity run tests for GitHub pull requests and post the corresponding status back to GitHub: - -1. Create a TeamCity configuration called _Tests iOS_. -2. Add a _VCS Trigger_ on `+:gh-readonly-queue/*`. -3. Add a _Command Line_ build step which simply executes `make test-ios`. -4. Add a _Pull Requests_ build feature which monitors GitHub (requires a personal access token). Check the _Ignore Drafts_ option to avoid triggering work for draft pull requests. -5. Add a _Commit status publisher_ build feature which posts to GitHub (requires a personal access token). -6. Add an _XML report processing_ build feature formatting test output as _Ant JUnit_ and which monitors `+:fastlane/test_output/*.xml`. -7. Add an _Agent Requirement_ ensuring that `tools.xcode.home` exists. Check that some agents are compatible and assignable (if agents are configured manually you might need to explicitly allow the configuration to be run). - -For comprehensive results a second _Tests tvOS_ configuration must be created for tvOS. This is easily achieved by copying the configuration you just created and editing the _Command Line_ build step to execute `make test-tvos`. - -## Demo nightlies - -To have TeamCity deliver nightly builds of the demo application to TestFlight: - -1. Create a TeamCity configuration called _Demo Nightly iOS_. -2. Add a _Schedule Trigger_ to deliver nightlies from `+:main` during the night. -3. Add a _Command Line_ build step which simply executes `make deliver-demo-nightly-ios`. -4. Add an _Agent Requirement_ ensuring that `tools.xcode.home` exists. Check that some agents are compatible and assignable (if agents are configured manually you might need to explicitly allow the configuration to be run). -5. Add a _Parameter_ with `teamcity.git.fetchAllHeads` as name, _Configuration parameter_ as kind and `true` as value. - -For comprehensive results a second _Demo Nightly tvOS_ configuration must be created for tvOS. This is easily achieved by copying the configuration you just created and editing the _Command Line_ build step to execute `make deliver-demo-nightly-tvos`. - -### Troubleshooting - -If submission to App Store Connect fails with a timeout, please check the [service status](https://developer.apple.com/system-status/) and, if status is all green, [login](https://appstoreconnect.apple.com) to check the binary status directly. A manual action (e.g. compliance) might be required. - -## Demo releases - -To have TeamCity deliver release builds of the demo application to TestFlight manually when required: - -1. Create a TeamCity configuration called _Demo Release iOS_. -2. Add a _Command Line_ build step which simply executes `make deliver-demo-release-ios`. -3. Add an _Agent Requirement_ ensuring that `tools.xcode.home` exists. Check that some agents are compatible and assignable (if agents are configured manually you might need to explicitly allow the configuration to be run). -4. Add a _Parameter_ with `teamcity.git.fetchAllHeads` as name, _Configuration parameter_ as kind and `true` as value. - -For comprehensive results a second _Demo Release tvOS_ configuration must be created for tvOS. This is easily achieved by copying the configuration you just created and editing the _Command Line_ build step to execute `make deliver-demo-release-tvos`. - -## Ruby version support - -The tools we use might require a minimum version of Ruby which might not always be the default one available on an agent. - -If this is the case then we can manage Ruby versions with [rbenv](https://github.com/rbenv/rbenv) on each agent and run `rbenv local [version]` in all _Command Line_ builds steps first to ensure a compatible Ruby version is used. +- `TESTFLIGHT_GROUPS`: A comma-delimited list of TestFlight groups to which published demo applications must be distributed. diff --git a/docs/DEVELOPMENT_SETUP.md b/docs/DEVELOPMENT_SETUP.md deleted file mode 100644 index e724a3152..000000000 --- a/docs/DEVELOPMENT_SETUP.md +++ /dev/null @@ -1,124 +0,0 @@ - -# Development setup - -This article briefly discusses local development setup. - -## Required tools - -The following tools are required for the best possible development experience: - -- The latest version of Xcode. -- The tvOS simulator (otherwise provisioning fails for tvOS when archiving the tvOS app). -- [bundler](https://bundler.io) -- [ffmpeg](https://ffmpeg.org) -- [gem](https://rubygems.org) -- [markdownlint-cli](https://github.com/igorshubovych/markdownlint-cli) -- [Periphery](https://github.com/peripheryapp/periphery) -- [Python](https://www.python.org) -- [shellcheck](https://www.shellcheck.net) -- [swiftlint](https://github.com/realm/SwiftLint) -- [xcodes](https://github.com/RobotsAndPencils/xcodes) -- [yamllint](https://github.com/adrienverge/yamllint) - -Most of these tools can easily be installed with [Homebrew](https://brew.sh). - -## Demo - -The project provides a demo application which can be run to test features offered by the Pillarbox package. - -## Unit tests - -Unit tests are provided with the Pillarbox package. Since Apple players [cannot play local manifests](https://developer.apple.com/forums/thread/69357?answerId=202051022#202051022) we are using Python to run a simple [web server](https://docs.python.org/3/library/http.server.html) serving various [test streams](TEST_STREAM_GENERATION.md) from a local directory. - -### Remark - -The web server must be manually started using `make test-streams-start` before running the tests and remains running afterwards. It can be stopped at any time with `make test-streams-stop`. - -If unit tests fail you should check that streams are served correctly at `http://localhost:8123` and start the server if this is not the case. Note that some sample streams require ~20 seconds to be fully available after the server started. - -## Makefile - -A [Makefile](../Makefile) provides several commands to perform quality checks, run unit tests or deliver demo app builds. Just run: - -```shell -make -``` - -to list all available commands. - -## Quality checks - -Quality checks can be run using: - -```shell -make check-quality -``` - -This ensures that Swift files, scripts and documentation conform to common best practices. - -## Git hooks - -### Installation - -Git hooks can be installed by running the following command: - -```shell -make git-hook-install -``` - -### Uninstallation - -Git hooks can be uninstalled by running the following command: - -```shell -make git-hook-uninstall -``` - -### Clean imports - -This ensures to keep a clean codebase by removing unnecessary imports from the code. - -```shell -make clean-imports -``` - -### Find dead code - -This ensures to catch some parts of code which are potentially unused. - -```shell -make find-dead-code -``` - -Before using the `make find-dead-code` command, ensure you have installed [Periphery](https://github.com/peripheryapp/periphery). - -## Editor configuration - -An [editor configuration file](../.editorconfig) and several linter configuration files are provided to ensure common styling and best practices. Editors can usually be configured to apply some of these rules automatically and consistently. - -### Xcode - -Xcode _Text Editing_ settings should be configured as follows: - -- Enable _Automatically trim trailing whitespaces_ with _Including whitespace-only lines_ enabled as well. -- Set _Default Text Encoding_ to _UTF-8_. -- Set _Default Line Endings_ to _macOS / Unix (LF)_. -- Use _Spaces_ for indentation (with 4 spaces both for _Tab Width_ and _Indent Width_). - -### Visual Studio Code - -- [EditorConfig extension](https://marketplace.visualstudio.com/items?itemName=EditorConfig.EditorConfig) -- [shellcheck extension](https://marketplace.visualstudio.com/items?itemName=timonwong.shellcheck) -- [rubocop extension](https://marketplace.visualstudio.com/items?itemName=misogi.ruby-rubocop) - -## Xcode previews - -Please ensure that the target an Xcode preview belongs to is selected before attempting to render an Xcode preview (otherwise rendering will crash). Note that attempting to use the `Pillarbox` umbrella target does not work. - -## Code signing - -We are currently using [cloud signing](https://developer.apple.com/wwdc21/10204) with automatic provisioning updates. Code signing requires access to our [internal configuration repository](https://github.com/SRGSSR/pillarbox-apple-configuration) which is automatically pulled when running `make setup`, provided you have been granted access to it. - -## Other resources - -SwiftUI view behavior is documented with the terminology introduced in the [following article](http://defagos.github.io/understanding_swiftui_layout_behaviors). diff --git a/docs/DEVELOPMENT_TOOLS.md b/docs/DEVELOPMENT_TOOLS.md new file mode 100644 index 000000000..f5ac8bbf5 --- /dev/null +++ b/docs/DEVELOPMENT_TOOLS.md @@ -0,0 +1,111 @@ + +# Development tools + +This article discusses the various development tools available when working on Pillarbox. + +## Makefile + +A [Makefile](../Makefile) provides several useful development-oriented commands. Simply run: + +```shell +make +``` + +to list all available commands. + +### Local test streams + +Since Apple players [cannot play local manifests](https://developer.apple.com/forums/thread/69357?answerId=202051022#202051022) we are using Python to run a simple [web server](https://docs.python.org/3/library/http.server.html) serving various [test streams](TEST_STREAM_GENERATION.md) from a local directory. These streams are used throughout unit test suites. + +The web server must be manually started using: + +```shell +make test-streams-start +``` + +before running tests. It can be stopped at any time with: + +```shell +make test-streams-stop +``` + +If unit tests fail you should check that streams are served correctly at [http://localhost:8123](http://localhost:8123) and start the server if needed. Note that some sample streams require ~20 seconds to be fully available after the server was started. + +### Quality checks + +Quality checks can be run using: + +```shell +make check-quality +``` + +This ensures that Swift files, scripts and documentation conform to common best practices. Many issues can be automatically fixed by running: + +```shell +make fix-quality +``` + +### Git hooks + +Git hooks can be installed by running the following command: + +```shell +make git-hook-install +``` + +and uninstalled by running the following command: + +```shell +make git-hook-uninstall +``` + +### Clean imports + +Run this command to remove unnecessary imports: + +```shell +make clean-imports +``` + +### Find dead code + +Run this command to remove unused code: + +```shell +make find-dead-code +``` + +### Reload SPM dependencies + +You can reload SPM dependencies by running: + +```shell +make spm-reload +``` + +## Editor configuration + +An [editor configuration file](../.editorconfig) and several linter configuration files are provided to ensure common styling and best practices. Editors can usually be configured to apply some of these rules automatically and consistently. + +### Xcode + +Xcode _Text Editing_ settings should be configured as follows: + +- Enable _Automatically trim trailing whitespaces_ with _Including whitespace-only lines_ enabled as well. +- Set _Default Text Encoding_ to _UTF-8_. +- Set _Default Line Endings_ to _macOS / Unix (LF)_. +- Use _Spaces_ for indentation (with 4 spaces both for _Tab Width_ and _Indent Width_). + +### Visual Studio Code + +- [EditorConfig extension](https://marketplace.visualstudio.com/items?itemName=EditorConfig.EditorConfig) +- [shellcheck extension](https://marketplace.visualstudio.com/items?itemName=timonwong.shellcheck) +- [rubocop extension](https://marketplace.visualstudio.com/items?itemName=misogi.ruby-rubocop) + +## Xcode previews + +Please ensure that the target an Xcode preview belongs to is selected before attempting to render an Xcode preview (otherwise rendering will crash). Note that attempting to use the `Pillarbox` umbrella target does not work. + +## Code signing + +We are currently using [cloud signing](https://developer.apple.com/wwdc21/10204) with automatic provisioning updates.