Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make registry modules lock scripts resilient #196

Draft
wants to merge 36 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
8bf5f4f
feat: make pre-commit scripts versioned
christian-calabrese Dec 19, 2024
92a7a97
refactor: pre-commit modules lock
christian-calabrese Dec 20, 2024
5e7a8e2
feat: deeply refactored lock-modules script and pre-commit
christian-calabrese Dec 20, 2024
1ce6b12
chore: ran pre-commit
christian-calabrese Dec 20, 2024
79183c6
fix: use terraform get instead of init for better performance
christian-calabrese Dec 20, 2024
d5ffb94
fix: add -update flag to terraform get
christian-calabrese Dec 20, 2024
ef3367b
revert: fix: add -update flag to terraform get
christian-calabrese Dec 20, 2024
51dceac
chore: remove unnecessary entry in pre-commit
christian-calabrese Dec 20, 2024
498e30b
fix: use files gotten from pre-commit args
christian-calabrese Dec 23, 2024
ff0d0c6
fix: tar excluding all files cause contained in hidden folder .terraform
christian-calabrese Dec 23, 2024
0a709d7
fix: show modules missing locks in action
christian-calabrese Dec 23, 2024
262b73f
fix: lock
christian-calabrese Dec 23, 2024
85dfc48
fix: hashing functiopn
christian-calabrese Dec 24, 2024
dfcc2b4
fix: remove git adds in pre-commti script
christian-calabrese Dec 24, 2024
3f2fc5e
fix: try lock-modules to check
christian-calabrese Dec 24, 2024
ba543da
fix: remove tfmodules.lock.json
christian-calabrese Dec 24, 2024
3818d3b
test: with registry modules without lock
christian-calabrese Dec 24, 2024
96be816
fix: jq
christian-calabrese Dec 24, 2024
7d17746
test: add tfmodules
christian-calabrese Dec 24, 2024
1d425f2
fix
christian-calabrese Dec 24, 2024
0fac4d4
fix: test
christian-calabrese Jan 2, 2025
8cead43
fix: test
christian-calabrese Jan 2, 2025
5e42927
fix: debug jq version
christian-calabrese Jan 2, 2025
6aac443
fix: debug jq version
christian-calabrese Jan 2, 2025
aab3904
fix: test action
christian-calabrese Jan 2, 2025
00845fb
fix: test action
christian-calabrese Jan 2, 2025
58ec4d7
fix: test action
christian-calabrese Jan 2, 2025
88383c0
fix: avoid jq keywords
christian-calabrese Jan 2, 2025
e1063c3
fix: selectively delete modules hashes
christian-calabrese Jan 2, 2025
9346e65
chore: remove unnecessary modules used for testing
christian-calabrese Jan 2, 2025
812f649
fix: deletion of hashes
christian-calabrese Jan 2, 2025
79b16e4
fix: init
christian-calabrese Jan 2, 2025
83b57f9
chore
christian-calabrese Jan 2, 2025
207c8d0
fix: remove relative referencing of modules
christian-calabrese Jan 3, 2025
248bbaa
chore: added changeset for modules
christian-calabrese Jan 3, 2025
1a3a571
chore: ran pre-commit
christian-calabrese Jan 3, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/shaggy-vans-check.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"pre_commit_scripts": major
---

First working version of the lock_modules pre-commit script
15 changes: 15 additions & 0 deletions .changeset/tall-keys-protect.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
---
"github_selfhosted_runner_on_container_app_jobs": patch
"azure_function_app_exposed": patch
"azure_app_service_exposed": patch
"azure_postgres_server": patch
"azure_storage_account": patch
"azure_api_management": patch
"azure_cosmos_account": patch
"azure_function_app": patch
"azure_app_service": patch
"azure_core_infra": patch
"azure_event_hub": patch
---

Relative module referencing substituted with terraform registry referencing
2 changes: 1 addition & 1 deletion .github/workflows/infra_apply.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -154,7 +154,7 @@ jobs:
calculate_hash() {
local module_path="$1"
tar --exclude=.* -cvf - "$module_path" | sha256sum | awk '{ print $1 }'
find "$module_path" -type f -not -path "$module_path/.*" | sort | xargs sha256sum | awk '{print $1}' | sha256sum | awk '{print $1}'
}
# Iterate over modules listed in the metadata that were sourced from the Terraform registry
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/infra_plan.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -157,7 +157,7 @@ jobs:

calculate_hash() {
local module_path="$1"
tar --exclude=.* -cvf - "$module_path" | sha256sum | awk '{ print $1 }'
find "$module_path" -type f -not -path "$module_path/.*" | sort | xargs sha256sum | awk '{print $1}' | sha256sum | awk '{print $1}'
}

# Iterate over modules listed in the metadata that were sourced from the Terraform registry
Expand Down
11 changes: 11 additions & 0 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -1,4 +1,15 @@
repos:
- repo: local
hooks:
- id: lock_modules
name: Lock Terraform Registry modules
description: 'Locks Terraform module versions and maintains hashes'
entry: infra/scripts/lock-modules.sh
language: script
files: '\.tf$'
exclude: ^.*\/modules\/.*
pass_filenames: true
require_serial: true
- repo: https://github.com/antonbabenko/pre-commit-terraform
rev: v1.96.2
hooks:
Expand Down
6 changes: 4 additions & 2 deletions .pre-commit-hooks.yaml
Original file line number Diff line number Diff line change
@@ -1,9 +1,11 @@
- id: lock_modules
name: Lock Terraform Registry modules
description: Create the file that contains terraform registry modules hashes
description: 'Locks Terraform module versions and maintains hashes'
entry: infra/scripts/lock-modules.sh
language: script

files: '\.tf$'
pass_filenames: true
require_serial: true
- id: terraform_providers_lock_staged
name: Terraform Providers Lock (on staged .terraform.lock.hcl files)
entry: infra/scripts/terraform_lock_precommit.sh
Expand Down
1 change: 1 addition & 0 deletions infra/identity/dev/tfmodules.lock.json
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{}
2 changes: 1 addition & 1 deletion infra/modules/azure_api_management/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ module "apim" {

| Name | Source | Version |
|------|--------|---------|
| <a name="module_naming_convention"></a> [naming\_convention](#module\_naming\_convention) | ../azure_naming_convention | n/a |
| <a name="module_naming_convention"></a> [naming\_convention](#module\_naming\_convention) | pagopa/dx-azure-naming-convention/azurerm | ~> 0 |

## Resources

Expand Down
3 changes: 2 additions & 1 deletion infra/modules/azure_api_management/main.tf
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,8 @@ terraform {
}

module "naming_convention" {
source = "../azure_naming_convention"
source = "pagopa/dx-azure-naming-convention/azurerm"
version = "~> 0"

environment = {
prefix = var.environment.prefix
Expand Down
2 changes: 1 addition & 1 deletion infra/modules/azure_app_service/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@

| Name | Source | Version |
|------|--------|---------|
| <a name="module_naming_convention"></a> [naming\_convention](#module\_naming\_convention) | ../azure_naming_convention | n/a |
| <a name="module_naming_convention"></a> [naming\_convention](#module\_naming\_convention) | pagopa/dx-azure-naming-convention/azurerm | ~> 0 |

## Resources

Expand Down
3 changes: 2 additions & 1 deletion infra/modules/azure_app_service/main.tf
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,8 @@ terraform {
}

module "naming_convention" {
source = "../azure_naming_convention"
source = "pagopa/dx-azure-naming-convention/azurerm"
version = "~> 0"

environment = {
prefix = var.environment.prefix
Expand Down
2 changes: 1 addition & 1 deletion infra/modules/azure_app_service_exposed/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ This module is used to create an Azure App Service, allowing it to be configured

| Name | Source | Version |
|------|--------|---------|
| <a name="module_naming_convention"></a> [naming\_convention](#module\_naming\_convention) | ../azure_naming_convention | n/a |
| <a name="module_naming_convention"></a> [naming\_convention](#module\_naming\_convention) | pagopa/dx-azure-naming-convention/azurerm | ~> 0 |

## Resources

Expand Down
3 changes: 2 additions & 1 deletion infra/modules/azure_app_service_exposed/main.tf
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,8 @@ terraform {
}

module "naming_convention" {
source = "../azure_naming_convention"
source = "pagopa/dx-azure-naming-convention/azurerm"
version = "~> 0"

environment = {
prefix = var.environment.prefix
Expand Down
2 changes: 1 addition & 1 deletion infra/modules/azure_core_infra/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ module "core" {
|------|--------|---------|
| <a name="module_dns"></a> [dns](#module\_dns) | ./_modules/dns | n/a |
| <a name="module_key_vault"></a> [key\_vault](#module\_key\_vault) | ./_modules/key_vault | n/a |
| <a name="module_naming_convention"></a> [naming\_convention](#module\_naming\_convention) | ../azure_naming_convention | n/a |
| <a name="module_naming_convention"></a> [naming\_convention](#module\_naming\_convention) | pagopa/dx-azure-naming-convention/azurerm | ~> 0 |
| <a name="module_nat_gateway"></a> [nat\_gateway](#module\_nat\_gateway) | ./_modules/nat_gateway | n/a |
| <a name="module_network"></a> [network](#module\_network) | ./_modules/networking | n/a |
| <a name="module_vpn"></a> [vpn](#module\_vpn) | ./_modules/vpn | n/a |
Expand Down
3 changes: 2 additions & 1 deletion infra/modules/azure_core_infra/main.tf
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,8 @@ terraform {
}

module "naming_convention" {
source = "../azure_naming_convention"
source = "pagopa/dx-azure-naming-convention/azurerm"
version = "~> 0"

environment = {
prefix = var.environment.prefix
Expand Down
2 changes: 1 addition & 1 deletion infra/modules/azure_cosmos_account/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@

| Name | Source | Version |
|------|--------|---------|
| <a name="module_naming_convention"></a> [naming\_convention](#module\_naming\_convention) | ../azure_naming_convention | n/a |
| <a name="module_naming_convention"></a> [naming\_convention](#module\_naming\_convention) | pagopa/dx-azure-naming-convention/azurerm | ~> 0 |

## Resources

Expand Down
3 changes: 2 additions & 1 deletion infra/modules/azure_cosmos_account/main.tf
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,8 @@ terraform {
}

module "naming_convention" {
source = "../azure_naming_convention"
source = "pagopa/dx-azure-naming-convention/azurerm"
version = "~> 0"

environment = {
prefix = var.environment.prefix
Expand Down
2 changes: 1 addition & 1 deletion infra/modules/azure_event_hub/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@

| Name | Source | Version |
|------|--------|---------|
| <a name="module_naming_convention"></a> [naming\_convention](#module\_naming\_convention) | ../azure_naming_convention | n/a |
| <a name="module_naming_convention"></a> [naming\_convention](#module\_naming\_convention) | pagopa/dx-azure-naming-convention/azurerm | ~> 0 |

## Resources

Expand Down
3 changes: 2 additions & 1 deletion infra/modules/azure_event_hub/main.tf
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,8 @@ terraform {
}

module "naming_convention" {
source = "../azure_naming_convention"
source = "pagopa/dx-azure-naming-convention/azurerm"
version = "~> 0"

environment = {
prefix = var.environment.prefix
Expand Down
2 changes: 1 addition & 1 deletion infra/modules/azure_function_app/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@

| Name | Source | Version |
|------|--------|---------|
| <a name="module_naming_convention"></a> [naming\_convention](#module\_naming\_convention) | ../azure_naming_convention | n/a |
| <a name="module_naming_convention"></a> [naming\_convention](#module\_naming\_convention) | pagopa/dx-azure-naming-convention/azurerm | ~> 0 |

## Resources

Expand Down
3 changes: 2 additions & 1 deletion infra/modules/azure_function_app/main.tf
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,8 @@ terraform {
}

module "naming_convention" {
source = "../azure_naming_convention"
source = "pagopa/dx-azure-naming-convention/azurerm"
version = "~> 0"

environment = {
prefix = var.environment.prefix
Expand Down
2 changes: 1 addition & 1 deletion infra/modules/azure_function_app_exposed/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@

| Name | Source | Version |
|------|--------|---------|
| <a name="module_naming_convention"></a> [naming\_convention](#module\_naming\_convention) | ../azure_naming_convention | n/a |
| <a name="module_naming_convention"></a> [naming\_convention](#module\_naming\_convention) | pagopa/dx-azure-naming-convention/azurerm | ~> 0 |

## Resources

Expand Down
3 changes: 2 additions & 1 deletion infra/modules/azure_function_app_exposed/main.tf
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,8 @@ terraform {
}

module "naming_convention" {
source = "../azure_naming_convention"
source = "pagopa/dx-azure-naming-convention/azurerm"
version = "~> 0"

environment = {
prefix = var.environment.prefix
Expand Down
2 changes: 1 addition & 1 deletion infra/modules/azure_postgres_server/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@

| Name | Source | Version |
|------|--------|---------|
| <a name="module_naming_convention"></a> [naming\_convention](#module\_naming\_convention) | ../azure_naming_convention | n/a |
| <a name="module_naming_convention"></a> [naming\_convention](#module\_naming\_convention) | pagopa/dx-azure-naming-convention/azurerm | ~> 0 |

## Resources

Expand Down
3 changes: 2 additions & 1 deletion infra/modules/azure_postgres_server/main.tf
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,8 @@ terraform {
}

module "naming_convention" {
source = "../azure_naming_convention"
source = "pagopa/dx-azure-naming-convention/azurerm"
version = "~> 0"

environment = {
prefix = var.environment.prefix
Expand Down
2 changes: 1 addition & 1 deletion infra/modules/azure_storage_account/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@

| Name | Source | Version |
|------|--------|---------|
| <a name="module_naming_convention"></a> [naming\_convention](#module\_naming\_convention) | ../azure_naming_convention | n/a |
| <a name="module_naming_convention"></a> [naming\_convention](#module\_naming\_convention) | pagopa/dx-azure-naming-convention/azurerm | ~> 0 |

## Resources

Expand Down
3 changes: 2 additions & 1 deletion infra/modules/azure_storage_account/main.tf
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,8 @@ provider "azurerm" {
}

module "naming_convention" {
source = "../azure_naming_convention"
source = "pagopa/dx-azure-naming-convention/azurerm"
version = "~> 0"

environment = {
prefix = var.environment.prefix
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@

| Name | Source | Version |
|------|--------|---------|
| <a name="module_naming_convention"></a> [naming\_convention](#module\_naming\_convention) | ../azure_naming_convention | n/a |
| <a name="module_naming_convention"></a> [naming\_convention](#module\_naming\_convention) | pagopa/dx-azure-naming-convention/azurerm | ~> 0 |

## Resources

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,8 @@ terraform {
}

module "naming_convention" {
source = "../azure_naming_convention"
source = "pagopa/dx-azure-naming-convention/azurerm"
version = "~> 0"

environment = {
prefix = var.environment.prefix
Expand Down
1 change: 1 addition & 0 deletions infra/repository/tfmodules.lock.json
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{}
5 changes: 5 additions & 0 deletions infra/resources/dev/.terraform.lock.hcl

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion infra/resources/dev/main.tf
Original file line number Diff line number Diff line change
Expand Up @@ -50,4 +50,4 @@ module "core" {
test_enabled = true

tags = local.tags
}
}
3 changes: 3 additions & 0 deletions infra/resources/dev/tfmodules.lock.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
{
"core.naming_convention": "807e8fafaf3cda8d1df7cc5c624715555ff150e87a8df0becc7e5cab3e54f855"
}
Loading
Loading