Assuming roles from php application running in Kubernetes cluster using IRSA

[wesbrownfavor]

Hey folks,

We have been attempting to migrate a PHP application from Beanstalk to a Kubernetes cluster using IAM Roles for Service Accounts.
In beanstalk we had the following setup

• EC2 instance with IAM role 1 in aws account 1 assigned to it
• Application assumes IAM role 2 in aws account 2 in order to do some tasks

We were using the following code:

$profile = new InstanceProfileProvider();
$client = new StsClient([
    'region' => 'us-east-1,
    'version' => 'latest',
    'credentials' => $profile,
]);
   
$assume_role_params => [
            'RoleArn' => getenv('arn_of_role_to_assume'),
            'RoleSessionName' => 'my-app-role-assume',
        ],

$config = [
    'client' => $client,
    'assume_role_params' => $assume_role_params,
];

$provider = CredentialProvider::memoize(
        CredentialProvider::chain(
            CredentialProvider::assumeRole($config),
            CredentialProvider::env()
        )
    );

In Kubernetes we have the following setup

• pod has attached service account which references IAM role 1 in account 1
• Application assumes IAM role 2 in aws account 2 in order to do some tasks

Problem is the Application isn't assuming IAM role 2 or if it is its failing past it silently and then erroring about the env provider not having AWS credentials.

Has anyone Assume a role from a pod using IAM for Service Accounts? If so what did you have to do differently?

Edit: Some more info

Just to verify I assumed IAM role 2 from account 2 from the application pod by doing the following:

• I exec into the pod and run aws sts get-caller-identity, I see the correct assumed role iam-role-1.
• Then I run aws sts assume-role --role-arn "arn:aws:iam::<Account 2>:role/iam-role-2" --role-session-name testing and I get the temporary credentials.
• I set the environment variables AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN with the temporary credentials I just received.
• I run aws sts get-caller-identity again and now see I now am iam-role-2
• I run the following command aws dynamodb describe-table --table-name myTable and it succeeds.

[yenfryherrerafeliz]

Hi @wesbrownfavor, here is some documentation for:

• Configuring the service account to assume the role: link
• Configuring the pod to use the service account: link

I know you have done already some of those steps, but can you confirm all of them were taken?
You could:

• Run kubectl describe serviceaccount YOUR-SERVICE-ACCOUNT -n default and see if the service account is annotated with the desired role and
• Run kubectl describe pod YOUR-POD-NAME | grep AWS_ROLE_ARN: and check if the pod is using expected role.

After all this is confirmed then it should work fine. Another thing to mention here is that if you use the default chain for credential resolution then, if there is any other credential provider defined and that could be resolved first then you should experience permission issues, but in this case you could just specify the correct credential provider for this situation as follow:

<?php
require '../vendor/autoload.php';

use Aws\S3\S3Client;
use Aws\Credentials\CredentialProvider;

$client = new S3Client([
    'version' => 'latest',
    'region' => getenv('TEST_REGION'),
    'credentials' => CredentialProvider::assumeRoleWithWebIdentityCredentialProvider()
]);
$command = $client->getCommand('getObject', [
    'Bucket' => getenv('TEST_BUCKET'),
    'Key' => getenv('TEST_KEY')
]);
$response = $client->execute($command);

var_dump($response);

I hope this helps!

Thanks!

[wesbrownfavor]

Thank you for replying!

To your double checking tasks:

• Yes my service account has the correct IAM role assigned
• AWS_ROLE_ARN is populated with that IAM role (this is what I would call role 1 in my first post)

When I try out your code I've also specified a config so that I can specify the role to assume (otherwise how would it know?)

$webIdentityConfig = [
      'RoleArn' => 'role2_arn',
      'WebIdentityTokenFile' => env('AWS_WEB_IDENTITY_TOKEN_FILE')
  ];

  $fr = DynamoDb::featureRequester([
      'dynamodb_table' => 'my_table',
      'dynamodb_options' => [
          'region' => 'my_region',
          'version' => 'latest',
          'credentials' => CredentialProvider::assumeRoleWithWebIdentityCredentialProvider($webIdentityConfig)
      ],
  ]);

I get this error:

'Error executing "GetItem" on "https://dynamodb.us-east-1.amazonaws.com"; AWS HTTP error: Client error: `POST
https://dynamodb.us-east-1.amazonaws.com` resulted in a `400 Bad Request` response: 
{"__type":"com.amazon.coral.service#AccessDeniedException","Message":"User: arn:aws:sts::account_1:assumed-
role/role_1_name (truncated...)  AccessDeniedException (client): User: arn:aws:sts::account_1:assumed-role/role_1_name
/aws-sdk-php-123 is not authorized to perform: dynamodb:GetItem on resource: arn:aws:dynamodb:us-
east-1:account_1:table/my_table because no identity-based policy allows the dynamodb:GetItem action

Its odd as the User is listed as the role 1 user but not the role 2 user as you would expect. Its as if its just not bothering assuming role 2.

Its like maybe because AWS_ROLE_ARN is set its taking precedent over the config value of 'RoleArn' and then boogering the account number.

Also for clarification I am using:
PHP: 7.2
aws-sdk-php: 3.173.19

[yenfryherrerafeliz]

Hi @wesbrownfavor, this behavior is expected. If we check the implementation here for assumeRoleWithWebIdentityCredentialProvider we will find that the parameter 'RoleArn' is never used, and that also the first role that is tried to be used is the one defined in AWS_ROLE_ARN. If you want to explicitly set which role you want to use when providing the credentials you could do the following:

use Aws\Credentials\AssumeRoleWithWebIdentityCredentialProvider;
use Aws\S3\S3Client;
use Aws\Sts\StsClient;

$providerFn = function ()  {
   $region = getenv('TEST_REGION');
   $stsClient = new StsClient([
       'version' => 'latest',
       'region' => $region
   ]);
   $provider = new AssumeRoleWithWebIdentityCredentialProvider([
       'RoleArn' => 'YOUR ROLE ARN',
       'WebIdentityTokenFile' => 'TOKEN_FILE_PATH',
       'SessionName' => 'SESSION_NAME',
       'client' => $stsClient,
       'region' => $region
   ]);

   return $provider();
};
$client = new S3Client([
   'version' => 'latest',
   'region' => getenv('TEST_REGION'),
   'credentials' => $providerFn
]);
$command = $client->getCommand('getObject', [
   'Bucket' => getenv('TEST_BUCKET'),
   'Key' => getenv('TEST_KEY')
]);
$response = $client->execute($command);

var_dump($response);

Thanks!

[wesbrownfavor]

You're my hero! This worked great. I was able to add the new provider to the chain and it works as expected!

[github-actions[bot]]

Hello! Reopening this discussion to make it searchable.