From 58c99503fbad9f52dfbef9d5a9c1c5ce9df21409 Mon Sep 17 00:00:00 2001 From: Cesar Varela Date: Fri, 27 Dec 2024 22:59:10 -0300 Subject: [PATCH] Update deployment docs --- site/docs/DEPLOYMENT.md | 336 ++++++++++++++++++++++------------------ 1 file changed, 187 insertions(+), 149 deletions(-) diff --git a/site/docs/DEPLOYMENT.md b/site/docs/DEPLOYMENT.md index 7e95fa8ef7..052450d375 100644 --- a/site/docs/DEPLOYMENT.md +++ b/site/docs/DEPLOYMENT.md @@ -1,233 +1,271 @@ # Deployment -## Netlify Setup +This guide is intended to assist in setting up continuous integration using GitHub Actions, including building, testing, and deploying. If you require a different method, please [open an issue](https://github.com/responsible-ai-collaborative/aiid/issues/new). -This guide walks you through the steps to set up a Netlify site for your project by importing an existing project from GitHub. -#### Prerequisites +## Fork the repository -- Ensure you have a GitHub account and your project is already pushed to a repository. -- Make sure you have a Netlify account. If not, sign up at [Netlify](https://www.netlify.com/). +To get started, fork the repository to your GitHub account. -#### Steps to Set Up +## Netlify -##### 1. Add New Site +### 1. Add a New Site - Go to your Netlify dashboard. - Click on **Add New Site**. -##### 2. Import Existing Project +### 2. Import Existing Project - Choose **Import Existing Project**. -##### 3. Deploy with GitHub +### 3. Connect to GitHub - Select **Deploy with GitHub** to connect your GitHub account. -##### 4. Select Repository +### 4. Select Repository - Choose the repository where your project is located. -##### 5. Configure Deployment +### 5. Configure the Deployment -- Under **Branch to Deploy**, select `main`. This setting doesn't matter for now. +- Under **Branch to Deploy**, select `main`. This setting is not critical at this point. - Leave all other settings as default. - Click on **Deploy Site**. -##### 6. Site Configuration +### 6. Site Configuration -###### Build and Deploy +#### Build and Deploy - Navigate to **Site Configuration** > **Build & Deploy**. -- Under **Build Settings** > **Build Status**, find **Stopped Builds**. +- Under **Build Settings** > **Build Status**, locate **Stopped Builds**. - Click **Save**. -###### Site Details +#### Site Details - Go to **Site Configuration** > **Site Details**. -- Copy the `NETLIFY_SITE_ID`. This will be useful when setting up the GitHub environment. +- Copy the Netlify site ID. This should be added to the GitHub environment variables as `NETLIFY_SITE_ID`. -## Prismic setup +#### Personal token -This project uses Prismic to fetch page content. You can still run the project without setting a Prismic account. +- Go to **User Settings** > **Applications**. +- Click on **New access token**. +- Name it, then click **Generate token**. +- Copy the generated token. This should be added to the GitHub environment secrets as `NETLIFY_AUTH_TOKEN`. -1. Sign up for a new [Prismic](https://prismic.io/) account or log in to your account if you already have one -2. In `Create a new repository` section choose `Something else` -3. Give your repository a name and choose `gatsby` in the technology dropdown -4. Choose your plan (if you only need one user, the free plan is enough) -5. Click `Create repository` -6. Create a new token in Settings > API & Security > Content API tab > Change Repository security to `Private API – Require an access token for any request` > Create new app > Permanent access tokens > Save value for later +## Prismic + +This project uses Prismic to dynamically fetch page content. + +1. Sign up for a new [Prismic](https://prismic.io/) account or log in if you already have one. +2. In the `Create a new repository` section, choose `Something else`. +3. Name your repository and select `gatsby` from the technology dropdown. +4. Choose your plan (the free plan is sufficient for a single user). +5. Click `Create Repository`. +6. Create a new token in Settings > API & Security > Content API tab. Change Repository security to `Private API – Require an access token for any request`, create a new app, and save the permanent access token for future use. ### Adding the Prismic content types #### Prismic Custom Types -You can find the list of all custom types in the folder `custom_types` +You can find a list of all custom types in the `custom_types` folder. #### How to create a new Custom Type -1. From the prismic left menu click `Custom Types` -2. Click `Create new custom type` -3. Give it a name (name of the json in custom_types folder) -4. Click `JSON editor` -5. Paste the JSON content from the predefined custom types inside the json -6. Click `Save` +1. From the Prismic left menu, click on `Custom Types`. +2. Click `Create new custom type`. +3. Name it using the name of the corresponding JSON file in the `custom_types` folder. +4. Click `JSON editor`. +5. Paste the JSON content from the predefined custom type into the editor. +6. Click `Save`. #### Adding Prismic documents -1. On the Prismic dashboard left menu click `Documents` -2. Click `Create new` -3. Fill in all the mandatory fields -4. Click `Save` -5. Keep in mind that the new content won't be available on your page until you Publish it. -6. In order to publish it, click `Publish` +1. On the Prismic dashboard left menu, click on `Documents`. +2. Click `Create new`. +3. Fill in all the mandatory fields. +4. Click `Save`. +5. Remember, the new content won't be available on your page until it is published. +6. To publish, click `Publish`. #### Prismic & Netlify Hook integration -In order for your recently published Prismic content to be available on your page, a Netlify build needs to be triggered. -In order to do this, you need to create a Netlify Build Hook. +> [!WARNING] +> This is outdated: [GitHub Issue #3306](https://github.com/responsible-ai-collaborative/aiid/issues/3306) + +To make your recently published Prismic content available on your page, a Netlify build must be triggered. To do this, create a Netlify Build Hook. #### Prismic environment variables -Add the following environment variable on Netlify: -`GATSBY_PRISMIC_REPO_NAME=[name_of_your_repository]` (step 3 from Prismic Setup section) -`PRISMIC_ACCESS_TOKEN=[you_prismic_access_token]` (step 6 from Prismic Setup section) +Add the following environment variables to Netlify: +- `GATSBY_PRISMIC_REPO_NAME=[name_of_your_repository]` (from step 3 in the Prismic Setup section) +- `PRISMIC_ACCESS_TOKEN=[your_prismic_access_token]` (from step 6 in the Prismic Setup section) #### Create Prismic/Netlify Hook -1. Login to your Netlify -2. Go to `Deploys` -3. Go to `Deploy settings` -4. Scroll to `Build Hooks` -5. Click `Add build hook` -6. Give it a name and assign a branch -7. Click save -8. Copy the generated URL -9. Go to your Prismic repository -10. Go to `Settings` > `Webhooks` -11. Create a new webhook and paste the URL in the URL field -12. In `Triggers` select `A document is published` and `A document is unpublished` -13. Click `Add this webhook` +> [!WARNING] +> This is outdated: [GitHub Issue #3306](https://github.com/responsible-ai-collaborative/aiid/issues/3306) -### Deployment Workflows on GitHub Actions +1. Log in to your Netlify account. +2. Navigate to `Deploys`. +3. Go to `Deploy settings`. +4. Scroll to `Build Hooks`. +5. Click `Add build hook`. +6. Name it and assign a branch. +7. Click `Save`. +8. Copy the generated URL. +9. Go to your Prismic repository. +10. Navigate to `Settings` > `Webhooks`. +11. Create a new webhook and paste the URL into the URL field. +12. In `Triggers`, select `A document is published` and `A document is unpublished` options. +13. Click `Add this webhook`. -We have integrated our testing and deployment processes with GitHub Actions. There are three primary workflows for deployment: Deploy Previews, Staging, and Production. The goal of these workflows is to continuously test and integrate changes in pull requests across environments. -#### 1) Deploy Previews Workflow +## Algolia -- **File:** [/.github/workflows/preview.yml](/.github/workflows/preview.yml) -- **Trigger:** This workflow is activated for pushes to pull requests that target the `staging` branch. -- **Process:** Executes both the integration tests and deploys the application to Netlify. -- **Post-Deployment:** Upon a successful deployment, the workflow automatically posts a comment on the pull request. This comment includes a link to the Netlify preview of the changes and a link to the Netlify deploy log. -- **Environment:** This workflow uses the `staging` GitHub environment. +- Create a new App. To handle a site with a similar number of records as our current production site, you'll need to enable the "Pay as you Go" plan. +- Create a new index called `instant_search`. +- Now go to your Algolia App settings, click on API Keys, and extract the following variables: -#### 2) Staging Workflow +``` +GATSBY_ALGOLIA_APP_ID +GATSBY_ALGOLIA_SEARCH_KEY +ALGOLIA_ADMIN_KEY +``` -- **File:** [/.github/workflows/preview.yml](/.github/workflows/staging.yml) -- **Trigger:** Runs only on pushes to the `staging` branch. -- **Process:** Executes both the integration tests and deploys to Netlify. -- **Deployment Criteria:** If the tests fail, no deployment will be carried out. -- **Environment:** This workflow uses the `staging` GitHub environment. +Algolia index settings are uploaded at build time as part of the GitHub Actions workflow. -#### 3) Production Workflow - -- **File:** [/.github/workflows/preview.yml](/.github/workflows/production.yml) -- **Trigger:** Runs only on pushes to the `main` branch. -- **Process:** Executes both the integration tests and deploys to Netlify. -- **Deployment Criteria:** If the tests fail, no deployment will be carried out. -- **Environment:** This workflow uses the `production` GitHub environment. - -#### GitHub Environment Configuration - -All three workflows share a common set of environment variables, which need to be defined for each environment. (Currently, we have only two environments: `staging` and `production`.) These variables are categorized into secrets and standard variables, and are accessed via GitHub actions as such. - -##### Secrets - -- `ALGOLIA_ADMIN_KEY` -- `CLOUDFLARE_R2_ACCESS_KEY_ID` -- `CLOUDFLARE_R2_ACCOUNT_ID` -- `CLOUDFLARE_R2_BUCKET_NAME` -- `CLOUDFLARE_R2_SECRET_ACCESS_KEY` -- `CYPRESS_RECORD_KEY` -- `E2E_ADMIN_PASSWORD` -- `E2E_ADMIN_USERNAME` -- `GOOGLE_TRANSLATE_API_KEY` -- `MONGODB_CONNECTION_STRING` -- `MONGODB_MIGRATIONS_CONNECTION_STRING` -- `MONGODB_REPLICA_SET` -- `MONGODB_TRANSLATIONS_CONNECTION_STRING` -- `NETLIFY_AUTH_TOKEN` -- `PRISMIC_ACCESS_TOKEN` -- `REALM_API_PRIVATE_KEY` -- `REALM_GRAPHQL_API_KEY` -- `REALM_API_PUBLIC_KEY` -- `GATSBY_ROLLBAR_TOKEN` - -##### Variables - -- `CYPRESS_PROJECT_ID` -- `GATSBY_ALGOLIA_APP_ID` -- `GATSBY_ALGOLIA_SEARCH_KEY` -- `GATSBY_AVAILABLE_LANGUAGES` -- `GATSBY_CLOUDFLARE_R2_PUBLIC_BUCKET_URL` -- `GATSBY_PRISMIC_REPO_NAME` -- `GATSBY_REALM_APP_ID` -- `NETLIFY_SITE_ID` -- `REALM_API_APP_ID` -- `REALM_API_GROUP_ID` - -## Algolia environment setup - -Suppose the feature you are developing involves mutating the Algolia index (used in the Discover app). In that case, you'll have to create your own by signing up for Algolia here: https://www.algolia.com and following these steps: - -- Create a new App (free tier will be enough) -- Create a new index called `instant_search` -- Go to https://incidentdatabase.ai/downloadIndex and save the `new_index.json` file -- Upload this file to your newly created index, by selecting `Use File` Algolia's UI -- Now go to your Algolia App settings, and click API Keys, using the values from there to update your `.env` file accordingly. +Alternatively, you can update the settings and push the records without rebuilding by running the following command from the `site/gatsby-site` directory. ``` -GATSBY_ALGOLIA_APP_ID= -GATSBY_ALGOLIA_SEARCH_KEY= -ALGOLIA_ADMIN_KEY= +npm run algolia:push ``` -Algolia index settings are uploaded on build time, so they will take effect after running: +## MongoDB + +Any MongoDB database provider will work. Below are the instructions for setting up MongoDB Atlas: + +- Create a new MongoDB project. +- Create a new Atlas cluster. +- Create a new database user and set the password. +- Whitelist the `0.0.0.0` IP address. + +You can choose to have one user access the entire database or have multiple users with stricter permissions for different databases. The environment variables for each of these databases are straightforward. ``` -npm run build +MONGODB_CONNECTION_STRING +MONGODB_MIGRATIONS_CONNECTION_STRING +MONGODB_REPLICA_SET +MONGODB_TRANSLATIONS_CONNECTION_STRING +API_MONGODB_CONNECTION_STRING ``` -Alternatively, you can update the settings without rebuilding if from `site/gatsby-site` you run: + +>You need to add the `0.0.0.0` IP to the MongoDB whitelist because it is currently not possible to obtain the IP of the Netlify server hosting the API functions. Ensure you whitelist the IP of your Netlify server or any other necessary IP addresses. + +## Cloudinary + +[Cloudinary](https://www.cloudinary.com) is used to host and manage report images. + +> [!WARNING] +> Not supported: https://github.com/responsible-ai-collaborative/aiid/issues/3305 + + +## Mailersend + +[Mailersend](https://www.mailersend.com) is used to send emails. + +Create a new account to obtain the API key. This should be added to the GitHub secrets as `MAILERSEND_API_KEY`. + +Additionally, set the following environment variables: ``` -npm run algolia:push +NOTIFICATIONS_SENDER= # The email address from which the emails will be sent +NOTIFICATIONS_SENDER_NAME= # The name of the sender ``` -Restart the development server to see the changes. +## GitHub + +### Deployment Workflows on GitHub Actions + +There are three primary workflows for deployment: Deploy Previews, Staging, and Production. These workflows aim to continuously test and integrate changes in pull requests across various environments. + +#### 1) Deploy Previews Workflow + +- **File:** [/.github/workflows/preview.yml](/.github/workflows/preview.yml) +- **Trigger:** This workflow is activated for pushes to pull requests targeting the `staging` branch. +- **Process:** Executes integration tests and deploys the application to Netlify. +- **Post-Deployment:** Upon successful deployment, the workflow automatically posts a comment on the pull request. This comment includes a link to the Netlify preview of the changes and a link to the deployment logs. +- **Environment:** This workflow utilizes the `staging` GitHub environment. + +#### 2) Staging Workflow +- **File:** [/.github/workflows/preview.yml](/.github/workflows/staging.yml) +- **Trigger:** This workflow runs only on pushes to the `staging` branch. +- **Process:** Executes integration tests and deploys to Netlify. +- **Deployment Criteria:** If the tests fail, deployment will not occur. +- **Environment:** This workflow utilizes the `staging` GitHub environment. -## MongoDB setup +#### 3) Production Workflow -- Create a new MongoDB project (the free tier will be enough) -- Create a new Atlas cluster with the name: `AIIDDev` - - Choose "Username and Password" as authentication method. - - Choose "My Local Environment" as network access and add your current IP address. - - If your IP is dynamic, add `0.0.0.0` to the list of IP addresses. +- **File:** [/.github/workflows/preview.yml](/.github/workflows/production.yml) +- **Trigger:** This workflow runs only on pushes to the `main` branch. +- **Process:** Executes integration tests and deploys to Netlify. +- **Deployment Criteria:** If the tests fail, deployment will not occur. +- **Environment:** This workflow utilizes the `production` GitHub environment. -#### Replicating the Database -Download the latest database backup from https://incidentdatabase.ai/research/snapshots. -Extract the archive, then from the `mongodump` directory, run `mongorestore` (included in [MongoDB tools](https://www.mongodb.com/docs/database-tools/installation/installation)) to upload the database backup: +All three workflows share a common set of environment variables, which need to be defined for each environment. (Currently, we have only two environments: `staging` and `production`.) These variables are categorized into secrets and standard variables and are accessed via GitHub Actions as needed. + +Variables ``` -mongorestore mongodb+srv://:@aiiddev..mongodb.net/aiidprod aiidprod -mongorestore mongodb+srv://:@aiiddev..mongodb.net/translations translations +CLOUDFLARE_R2_ACCOUNT_ID +CLOUDFLARE_R2_BUCKET_NAME +CYPRESS_PROJECT_ID +GATSBY_ALGOLIA_APP_ID +GATSBY_ALGOLIA_SEARCH_KEY +GATSBY_AVAILABLE_LANGUAGES +GATSBY_CLOUDFLARE_R2_PUBLIC_BUCKET_URL +GATSBY_PRISMIC_REPO_NAME +GATSBY_REALM_APP_ID +NETLIFY_SITE_ID +NOTIFICATIONS_SENDER +NOTIFICATIONS_SENDER_NAME +REALM_API_APP_ID +REALM_API_GROUP_ID +REALM_APP_ID +SITE_URL +TRANSLATE_DRY_RUN +TRANSLATE_SUBMISSION_DATE_START ``` -You can find the value for `` by going to your Atlas cluster's overview on cloud.mongodb.com, then selecting the "primary" shard labeled `aiiddev-shard-00-..mongodb.net`. - +Secrets: -## Cloudinary +``` +ALGOLIA_ADMIN_KEY +API_MONGODB_CONNECTION_STRING +CLOUDFLARE_R2_ACCESS_KEY_ID +CLOUDFLARE_R2_ACCOUNT_ID +CLOUDFLARE_R2_BUCKET_NAME +CLOUDFLARE_R2_SECRET_ACCESS_KEY +CLOUDFLARE_R2_WRITE_ACCESS_KEY_ID +CLOUDFLARE_R2_WRITE_SECRET_ACCESS_KEY +CYPRESS_RECORD_KEY +GATSBY_REALM_APP_ID +GATSBY_ROLLBAR_TOKEN +GOOGLE_MAPS_API_KEY +GOOGLE_TRANSLATE_API_KEY +MAILERSEND_API_KEY +MONGODB_CONNECTION_STRING +MONGODB_MIGRATIONS_CONNECTION_STRING +MONGODB_REPLICA_SET +MONGODB_TRANSLATIONS_CONNECTION_STRING +NETLIFY_AUTH_TOKEN +PRISMIC_ACCESS_TOKEN +REALM_API_PRIVATE_KEY +REALM_API_PUBLIC_KEY +REALM_GRAPHQL_API_KEY +ROLLBAR_POST_SERVER_ITEM_ACCESS_TOKEN +``` -[Cloudinary](https://www.cloudinary.com) is what we use to host and manage report images. +GitHub should detect the workflows in the `.github/workflows` directory. Once you have set up the environment variables and secrets, you can push your changes to the repository. The workflows should then be triggered, and the site will be deployed to Netlify.