[Issue] azd deploy fails in azdo build pipeline when connection string value is expected from remote environment. #3850

Bpflugrad · 2024-05-03T23:46:41Z

Make sure you've installed the latest version using instructions in the wiki

Output from azd version
Run azd version and copy and paste the output here:
azd version 1.8.2 (commit 14600c7a54edac4f54397413f8638431f5c16327)

Describe the bug
Deploying in an azdo pipeline:

Pool: Azure Pipelines
Image: ubuntu-latest
Agent: Hosted Agent
Started: Today at 1:39 PM

Using Aspire preview 6 8.0.0-preview.6.24214.1.
Using AddConnectionString(string) to add a reference to an existing Azure resource by connection string.
When deploying in an azdo pipeline, if a deployment has never been completed before (such as with azd up on a development machine), azd deploy fails with error message:

panic: send on closed channel

goroutine 55 [running]:
github.com/azure/azure-dev/cli/azd/pkg/async.(*TaskContextWithProgress[...]).SetProgress(...)
	/mnt/vss/_work/1/s/cli/azd/pkg/async/task_context.go:74
github.com/azure/azure-dev/cli/azd/pkg/project.syncProgress[...](0xc0002105a0, 0x0)
	/mnt/vss/_work/1/s/cli/azd/pkg/project/service_manager.go:678 +0xbd
created by github.com/azure/azure-dev/cli/azd/pkg/project.runCommand[...].func1 in goroutine 36
	/mnt/vss/_work/1/s/cli/azd/pkg/project/service_manager.go:658 +0xb9

This causes the pipeline to fail and deployment cannot be completed.

To Reproduce

Using Visual Studio 17.10.0 Preview 5.0 create a new project from template .Net Aspire Starter Project.
Run azd init in the Solution folder.
Edit azure.yaml adding to the root:

pipeline: 
   provider: azdo
state:
  remote:
    backend: AzureBlobStorage
    config:
      accountName: STORAGE_ACCOUNT
      containerName: PROJECT_NAME

Create folder .azdo\pipelines
Add to folder azure-dev.yml
Run azd pipeline config, following all instructions related to PAT, remotes, etc.
Permission the created azconnection service principal with Storage Blob Contributor for Storage Account STORAGE_ACCOUNT.
Modify the created AppHost.Program with the following line: var blobStorageConnectionString = builder.AddConnectionString("BlobStorageConnection");
Modify the the AddProject line, adding .WithReference(blobStorageConnectionString).
Modify ApiService.Program adding: builder.AddAzureBlobClient("BlobStorageConnection");
Edit azure-dev.yml to add build commands:

trigger:
- master
- staging

pool:
  vmImage: ubuntu-latest

steps:
  - task: DotNetCoreCLI@2
    displayName: Install Aspire Workload with Preview
    inputs:
      command: custom
      custom: 'workload'
      arguments: 'install aspire --include-previews --source https://api.nuget.org/v3/index.json'

  - task: setup-azd@0 
    displayName: Install azd

  - pwsh: |
      azd config set auth.useAzCliAuth "true"
    displayName: Configure AZD to Use AZ CLI Authentication.

  - task: AzureCLI@2
    displayName: Provision Infrastructure
    inputs:
      azureSubscription: azconnection
      scriptType: bash
      scriptLocation: inlineScript
      inlineScript: |
        azd provision --no-prompt --environment $(AZURE_ENV_NAME)
    env:
      AZURE_SUBSCRIPTION_ID: $(AZURE_SUBSCRIPTION_ID)
      AZURE_ENV_NAME: $(AZURE_ENV_NAME)
      AZURE_LOCATION: $(AZURE_LOCATION)

  - task: AzureCLI@2
    displayName: Deploy Application
    inputs:
      azureSubscription: azconnection
      scriptType: bash
      scriptLocation: inlineScript
      inlineScript: |
        azd deploy --no-prompt --environment $(AZURE_ENV_NAME)
    env:
      AZURE_SUBSCRIPTION_ID: $(AZURE_SUBSCRIPTION_ID)
      AZURE_ENV_NAME: $(AZURE_ENV_NAME)
      AZURE_LOCATION: $(AZURE_LOCATION)

Execute azd env set AZURE_BLOB_STORAGE_CONNECTION CONNECTION_STRING
Execute azd infra synth
- Note that in main.parameters.json added by this command that the following line exists:
```
"BlobStorageConnection": {
      "value": "${AZURE_BLOB_STORAGE_CONNECTION}"
    }
```
Commit
Push

The build will be triggered, and at the Deploy Application step will fail with error:

Analyzing Aspire Application (this might take a moment...)

Deploying services (azd deploy)

Deploying service apiservice
Deploying service apiservice (Logging in to registry)
Deploying service apiservice (Pushing container image)
panic: send on closed channel

goroutine 55 [running]:
github.com/azure/azure-dev/cli/azd/pkg/async.(*TaskContextWithProgress[...]).SetProgress(...)
	/mnt/vss/_work/1/s/cli/azd/pkg/async/task_context.go:74
github.com/azure/azure-dev/cli/azd/pkg/project.syncProgress[...](0xc0002105a0, 0x0)
	/mnt/vss/_work/1/s/cli/azd/pkg/project/service_manager.go:678 +0xbd
created by github.com/azure/azure-dev/cli/azd/pkg/project.runCommand[...].func1 in goroutine 36
	/mnt/vss/_work/1/s/cli/azd/pkg/project/service_manager.go:658 +0xb9

##[error]Script failed with exit code: 2

Expected behavior
Deployment should succeed since the required parameter is present in the environment.

Environment
Information on your environment:
- dotnet --version 8.0.300-preview.24203.14
- Visual Studio 17.10.0 Preview 5.0
- Aspire preview 6 8.0.0-preview.6.24214.1
- azd version 1.8.2 (commit 14600c7a54edac4f54397413f8638431f5c16327)

Additional context
When not using azd infra synth the error is slightly different:

ERROR: failed deploying service 'webfrontend': failing invoking action 'deploy', failed executing template file: template: containerApp.tmpl.yaml:21:19: executing "containerApp.tmpl.yaml" at <securedParameter "BlobStorageConnection">: error calling securedParameter: parameter BlobStorageConnection not found

Also tried to updating the builder.AddConnectionString("BlobStorageConnection", "AZURE_BLOB_STORAGE_CONNECTION"); and still get the panic.

The text was updated successfully, but these errors were encountered:

mip1983 · 2024-05-08T13:42:02Z

I've struggled with providing connection strings myself, providing these 'secure parameters' in a pipeline from an environment variable seems to be challenging. There may be something in my thread that helps you: #3597

Bpflugrad · 2024-05-09T00:22:28Z

@mip1983 thanks it looks like I might be able to work around my problem with the Variable Groups mentioned in your thread.
The really unfortunate thing is that AddParameter works fine and azd infra synth shows that the same behavior of turning the AddConnectionString argument into AZURE_PASCAL_CASE is followed but unlike Parameters the value is not brought in from remote configuration in the same way. Really seems like a bug to me.

vhvb1989 · 2024-05-10T05:50:01Z

Hi @Bpflugrad , I can explain what is happening.

When you run azd infra synth, as you can see, azd is generating a bicep secured parameter for the connection string:

@secure()
param BlobStorageConnection string

If you azd pipeline config before adding the connection string to the AppHost program, azd will not ask/prompt for a value for the parameter (b/c it is not yet known). You would be using azd pipeline config just to set up the azdo repo and service connector.
Then, when you add the connection string, run infra synth and commit the changes, your pipeline will run, but azd did not set any value for the parameter (as secret or variable) for the pipeline.

What you are doing is interesting, because you are hoping to set the env var in remote-state and have the pipeline to pick the env var from the remote state. Ideally, you should not add plain text to remote-state, as that's not a secured store. But, making the secure thing apart, azd is been able to pull the value from the remote-state to run provision, but azd is not expecting to find the securedParameter within the environment (in .env).

Azd uses the .azure/envName/config.json file to save the values of secured-parameters. That config.json is set in azdo as a secret, and that's where azd will try to find the value during deploying the app (for setting the value in the container app environment).

So, here's what you should be able to do, to make your sample work:

Run all your repo steps until:
- Execute azd env set AZURE_BLOB_STORAGE_CONNECTION CONNECTION_STRING
Instead of running azd env set ... do:
- Open .azure/envName/config.json and paste this code:

{
  "infra": {
    "parameters": {
      "BlobStorageConnection": "YOUR_CONNECTION_STRING"
    }
  }
}

Save the file and then run azd env set foo bar . This command is just to sync and save the config.json in your remote storage, the name of the foo and bar don't really matter. It is just a workaround to save your manual change to the config.json
Now you can open the config.json and restore the content of the file how it was before, probably to:

{}

This is just b/c I assume that you are trying to do remote-state to avoid writing/having the connection string in any local file. So, you can now remove it from the file, as it has been saved on remote-state.

Now you can continue with your other steps:
Execute azd infra synth
Commit
Push

Now azd will be able to fetch the value from the remote-state (from the config.json) for provison and for deploy.

Now, let me just say it again. remote-state is not meant for secured values. We are still working/designing a secured store (like KeyVault) but, remote-state should not be considered secured.
For azd version 1.9.0, we update how secrets are saved in the config.json, by writing the actual secret outside of the project (inspired by .Net user secrets). So, instead of all the manual steps you are doing to set up the remote-state, you can remove the remote-state config and just do 2 changes to your project:

Update your pipeline config to reference AZD_INITIAL_ENVIRONMENT_CONFIG env var during provision, like:

 - task: AzureCLI@2
    displayName: Provision Infrastructure
    inputs:
      azureSubscription: azconnection
      scriptType: bash
      scriptLocation: inlineScript
      inlineScript: |
        azd provision --no-prompt --environment $(AZURE_ENV_NAME)
    env:
      AZURE_SUBSCRIPTION_ID: $(AZURE_SUBSCRIPTION_ID)
      AZURE_ENV_NAME: $(AZURE_ENV_NAME)
      AZURE_LOCATION: $(AZURE_LOCATION)
     AZD_INITIAL_ENVIRONMENT_CONFIG: $(AZD_INITIAL_ENVIRONMENT_CONFIG)

After you add the connection string to the appHost, run azd pipeline config and let azd to prompt for the connection string. Enter the connection string. azd will set the value in a secret AZD_INITIAL_ENVIRONMENT_CONFIG automatically, and your pipeline will work.
Then, you can open the .azure/envName/config.json and see that there is a ref to a local azd-vault, and not the raw plain text connection string, like

{
  "infra": {
    "parameters": {
      "BlobStorageConnection": "vault://1b14641c-bb16-4ee9-87f9-d5e19ef09b17/6675dd26-3372-47a7-a35e-b2026e84c2e9"
    }
  },
  "vault": "1b14641c-bb16-4ee9-87f9-d5e19ef09b17"
}

When azd sets AZD_INITIAL_ENVIRONMENT_CONFIG, it will resolve the vault reference and write the value as a secret in azdo.

Bpflugrad · 2024-05-10T15:31:49Z

Thanks for the help @vhvb1989 this has been a lot of trial and error.

I decided to go with your second suggestion, and updated my build pipe and re-ran azd pipeline config. I was prompted for the value, and the vault reference similar to what you pasted is in my local config.json.

When I run the pipe now I get a lot of security errors either saying the wrong client secret was provided:
ERROR: AADSTS7000215: Invalid client secret provided. Ensure the secret being sent in the request is the client secret value, not the client secret ID, for a secret added to app '***'.
Or a panic from Go after I saved the service principal thinking it might need a new token:
panic: don't know how to prompt for type *survey.Password
My recent runs have all resulted in the panic error, tried adding azd infra synth back and a few other things.

This wasn't happening prior to my changes so my assumption is that attempting to resolve the vault reference is the issue. I'm not familiar with vault://, where are those and what do they refer to?

Bpflugrad · 2024-05-13T15:55:37Z

Hi @vhvb1989,

Sorry for the confusion Friday. I figured out why I was getting the panic message. Basically, when you have used AddConnectionString("Value") a line @secure() param Value string is added to the main.bicep. Even though this is only used by azd deploy it seems that some value must exist in the .env, even if it is just a fake value.

However, after successfully doing this, and verifying that the following appears in my remote config.json:

{
  "infra": {
    "parameters": {
      "BlobStorageConnection": "vault://9f52a64c-260e-4515-9d32-4b5463939db7/27440911-0f41-45f5-83f8-682a15aacdd4"
    }
  },
  "vault": "9f52a64c-260e-4515-9d32-4b5463939db7"
}

However, azd deploy is failing with:

2024/05/13 15:42:35 service_target_dotnet_containerapp.go:195: generating container app manifest from /home/vsts/work/1/s/AspireInfraExample.AppHost/AspireInfraExample.AppHost.csproj for project apiservice
Deploying service apiservice (Updating container app)
  (x) Failed: Deploying service apiservice

ERROR: failed deploying service 'apiservice': failing invoking action 'deploy', failed executing template file: template: containerApp.tmpl.yaml:21:19: executing "containerApp.tmpl.yaml" at <securedParameter "BlobStorageConnection">: error calling securedParameter: parameter BlobStorageConnection not found

I am not using azd infra synth in this case. I also changed the slashes from backslash \ to forward slash / in azure.yaml since you mentioned the regression in #3891 but that didn't make a difference.

If I do use azd infra synth I end up with:

2024/05/13 15:53:22 service_target_dotnet_containerapp.go:186: using container app manifest from /home/vsts/work/1/s/AspireInfraExample.AppHost/infra/apiservice.tmpl.yaml
panic: send on closed channel

goroutine 66 [running]:
github.com/azure/azure-dev/cli/azd/pkg/async.(*TaskContextWithProgress[...]).SetProgress(...)
	/mnt/vss/_work/1/s/cli/azd/pkg/async/task_context.go:74
github.com/azure/azure-dev/cli/azd/pkg/project.syncProgress[...](0xc0004ad380, 0x0)
	/mnt/vss/_work/1/s/cli/azd/pkg/project/service_manager.go:678 +0xbd
created by github.com/azure/azure-dev/cli/azd/pkg/project.runCommand[...].func1 in goroutine 28
	/mnt/vss/_work/1/s/cli/azd/pkg/project/service_manager.go:658 +0xb9

Also after each failed deployment the remote config.json is reset to {} so I have to do azd env set with a fake value each time. I made sure to run azd pipeline config so that AZD_INITIAL_ENVIRONMENT_CONFIG would be updated (I can't validate its content from Azure DevOps). I am passing AZD_INITIAL_ENVIRONMENT_CONFIG as env to both azd provision and azd deploy in my pipeline:

  - task: AzureCLI@2
    displayName: Provision Infrastructure
    inputs:
      azureSubscription: azconnection
      scriptType: bash
      scriptLocation: inlineScript
      inlineScript: |
        azd provision --no-prompt
    env:
      AZURE_SUBSCRIPTION_ID: $(AZURE_SUBSCRIPTION_ID)
      AZURE_ENV_NAME: $(AZURE_ENV_NAME)
      AZURE_LOCATION: $(AZURE_LOCATION)
      AZD_INITIAL_ENVIRONMENT_CONFIG: $(AZD_INITIAL_ENVIRONMENT_CONFIG)

  - task: AzureCLI@2
    displayName: Deploy Application
    inputs:
      azureSubscription: azconnection
      scriptType: bash
      scriptLocation: inlineScript
      inlineScript: |
        azd deploy --no-prompt --debug
    env:
      AZURE_SUBSCRIPTION_ID: $(AZURE_SUBSCRIPTION_ID)
      AZURE_ENV_NAME: $(AZURE_ENV_NAME)
      AZURE_LOCATION: $(AZURE_LOCATION)
      AZD_INITIAL_ENVIRONMENT_CONFIG: $(AZD_INITIAL_ENVIRONMENT_CONFIG)

You only mentioned adding AZD_INITIAL_ENVIRONMENT_CONFIG to azd provision but I figured since the value is needed in azd deploy that it wouldn't hurt to have it on both.

Anyway, perhaps I missed a step but I can't get your second suggestion to work.

Any help is appreciated!

vhvb1989 · 2024-05-13T17:42:18Z

@Bpflugrad , #3891 was not the only issue affecting azd on CI/CD.
There was also #3897. The fix was merged already, but you need daily-build to get it.
There's 2 things you can do:

Update your pipeline definition and your local azd to use daily-azd build,
or, before running azd pipeline config, update the config.json from what you have:

{
  "infra": {
    "parameters": {
      "BlobStorageConnection": "vault://9f52a64c-260e-4515-9d32-4b5463939db7/27440911-0f41-45f5-83f8-682a15aacdd4"
    }
  },
  "vault": "9f52a64c-260e-4515-9d32-4b5463939db7"
}

to what it should be in CI:

{
  "infra": {
    "parameters": {
      "BlobStorageConnection": "The actual connection string HERE"
    }
  },
}

The issue on 1.9.0 is that azd is setting the config.json like:

{
  "infra": {
    "parameters": {
      "BlobStorageConnection": "The actual connection string HERE"
    }
  },
  "vault": "9f52a64c-260e-4515-9d32-4b5463939db7"
}

You can see that the secrets are resolved, but the vault field is still there, and when runining in CI, azd will not find that vault and will just create an empty config. In the fix in daily, azd is removing the vault field for the resolved-config

Bpflugrad · 2024-05-13T22:53:05Z

Thanks @vhvb1989. I updated to the daily build. The workaround (your second suggestion) where the raw value is included in config.json works.

However, using azd naturally (your first suggestion), including the secured values with the vault references still doesn't work. I updated both the build pipe and local azd to 1.10.0-beta.1-daily.3780018 (commit bb526cbd20cfbd9d2ebadc2e34f1058e8595ff4f).
I'm still getting panic: send on closed channel

2024/05/13 21:22:15 service_target_dotnet_containerapp.go:195: generating container app manifest from /home/vsts/work/1/s/AspireInfraExample.AppHost/AspireInfraExample.AppHost.csproj for project apiservice
panic: send on closed channel

goroutine 56 [running]:
github.com/azure/azure-dev/cli/azd/pkg/async.(*TaskContextWithProgress[...]).SetProgress(...)
	/mnt/vss/_work/1/s/cli/azd/pkg/async/task_context.go:74
github.com/azure/azure-dev/cli/azd/pkg/project.syncProgress[...](0xc0004c0690, 0x0)
	/mnt/vss/_work/1/s/cli/azd/pkg/project/service_manager.go:689 +0xbd
created by github.com/azure/azure-dev/cli/azd/pkg/project.runCommand[...].func1 in goroutine 35
	/mnt/vss/_work/1/s/cli/azd/pkg/project/service_manager.go:669 +0xb9

After updating the azd version to the daily build, I reran azd pipeline config just in case vault creation and resolution had changed.

Thanks again for your help, I can at least proceed with the insecure workaround until the vaults can be figured out.

Bpflugrad · 2024-05-15T15:23:38Z

Hi @vhvb1989 ,

I tried a few permutations of my build script. If I have AZD_INITIAL_ENVIRONMENT_CONFIG only on azd provision or only on azd deploy, I get the missing parameter error message from azd deploy like:
ERROR: failed deploying service 'apiservice': failed executing template file: template: containerApp.tmpl.yaml:21:19: executing "containerApp.tmpl.yaml" at <securedParameter "BlobStorageConnection">: error calling securedParameter: parameter BlobStorageConnection not found
If I have AZD_INITIAL_ENVIRONMENT_CONFIG on BOTH azd provision and azd deploy I get the panic message:

panic: send on closed channel

goroutine 56 [running]:
github.com/azure/azure-dev/cli/azd/pkg/async.(*TaskContextWithProgress[...]).SetProgress(...)
	/mnt/vss/_work/1/s/cli/azd/pkg/async/task_context.go:74
github.com/azure/azure-dev/cli/azd/pkg/project.syncProgress[...](0xc0004c0690, 0x0)
	/mnt/vss/_work/1/s/cli/azd/pkg/project/service_manager.go:689 +0xbd
created by github.com/azure/azure-dev/cli/azd/pkg/project.runCommand[...].func1 in goroutine 35
	/mnt/vss/_work/1/s/cli/azd/pkg/project/service_manager.go:669 +0xb9

I've never worked with Go before but looking at this stack trace, service_manager.go and service_target_dotnet_containerapp.go it appears that there's a silent failure between here and here.
Somehow ServiceProgress is getting closed and causing the send on closed channel. At least that is my guess.

vhvb1989 · 2024-05-15T17:00:02Z

Hi @Bpflugrad
We just released 1.9.1 which should have a fix for how AZD_INITIAL_ENVIRONMENT_CONFIG is set when you run azd pipeline config
Can you try using 1.9.1 to run azd pipeline config please. Let me know what you get

Bpflugrad · 2024-05-15T18:14:02Z

Hi @vhvb1989,

Thanks for the reply. I updated my local machine to 1.9.1 (commit aadbc26943c2e3e5437a6ffa528fe5264887a10c), and I updated the pipeline to not use the daily build anymore, confirmed it is using the same version.

I took the following steps:

Empty my local config.json, remove my related stub AZURE_BLOB_STORAGE_CONNECTION from .env.
Run azd pipeline config, enter my secret value.
Run azd env set AZURE_BLOB_STORAGE_CONNECTION blah12 to avoid failure in azd provision
Run azd infra synth to check that no changes there have been made (none had)
Verify the obfuscated value is present in the remote store.
Run pipeline.

Unfortunately, azd deploy still fails with panic: send on closed channel.

microsoft-github-policy-service · 2024-05-24T03:17:57Z

Hi @Bpflugrad, since you haven’t asked that we “/unresolve” the issue, we’ll close this out. If you believe further discussion is needed, please add a comment “/unresolve” to reopen the issue.

microsoft-github-policy-service bot added customer-reported identify a customer issue question labels May 3, 2024

rajeshkamal5050 added azdo Azure DevOps pipelines aspire labels May 10, 2024

vhvb1989 added the issue-addressed label May 10, 2024

This was referenced May 10, 2024

[Issue] [aspire] azd infra synth ignored by azd deploy in azdo ci/cd pipeline #3891

Closed

Prompting for parameters with aspire/azd #3597

Open

microsoft-github-policy-service bot closed this as completed May 24, 2024

Bpflugrad mentioned this issue Sep 6, 2024

[Issue] DevOps pipeline clears remote environment's config.json on azd up operation #4256

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Issue] azd deploy fails in azdo build pipeline when connection string value is expected from remote environment. #3850

[Issue] azd deploy fails in azdo build pipeline when connection string value is expected from remote environment. #3850

Bpflugrad commented May 3, 2024

mip1983 commented May 8, 2024

Bpflugrad commented May 9, 2024

vhvb1989 commented May 10, 2024

Bpflugrad commented May 10, 2024

Bpflugrad commented May 13, 2024

vhvb1989 commented May 13, 2024

Bpflugrad commented May 13, 2024

Bpflugrad commented May 15, 2024

vhvb1989 commented May 15, 2024

Bpflugrad commented May 15, 2024

microsoft-github-policy-service bot commented May 24, 2024

[Issue] azd deploy fails in azdo build pipeline when connection string value is expected from remote environment. #3850

[Issue] azd deploy fails in azdo build pipeline when connection string value is expected from remote environment. #3850

Comments

Bpflugrad commented May 3, 2024

mip1983 commented May 8, 2024

Bpflugrad commented May 9, 2024

vhvb1989 commented May 10, 2024

Bpflugrad commented May 10, 2024

Bpflugrad commented May 13, 2024

vhvb1989 commented May 13, 2024

Bpflugrad commented May 13, 2024

Bpflugrad commented May 15, 2024

vhvb1989 commented May 15, 2024

Bpflugrad commented May 15, 2024

microsoft-github-policy-service bot commented May 24, 2024