AWS OIDC Authentication with SPIFFE

As part of Square’s migration to the cloud, we found that we needed an easy way for our datacenter applications to communicate with services in AWS. A majority of our applications are still in the datacenter and often use services in AWS such as SQS and S3. Our applications on AWS are split into separate accounts. Most applications will have multiple accounts (development, staging, production).

Our AWS OIDC infrastructure provides automated AWS credentials for applications in our datacenter to assume roles in AWS. We leveraged the SPIRE OIDC Discovery Provider provided by SPIRE paired with spiffe-aws-assume-role, a custom open source tool we wrote to exchange SPIFFE JWTs for AWS credentials. Our new process means that app owners only need to make a simple config change to allow their applications to connect to AWS.

Previously, applications used static credentials stored in our secrets manager, Keywhiz. These credentials were rarely, if ever, rotated and required more manual work from app owners to set up. One of the principles of the Cryptographic Identity and Secrets Management team (CISM) is to make the secure solution the default. Our goal with this project was to make the new automated method both easier for developers to adopt and more secure.

OIDC with SPIFFE

Square has been moving towards adopting SPIFFE for mutual TLS instead of homegrown solutions. We’ve written about some of our efforts in earlier blog posts. SPIFFE has support for Kubernetes, AWS, GCP, and other popular platforms allowing us to more easily expand our infrastructure. Additionally, adopting SPIFFE gives us improvements to the security of our infrastructure including: short-lived certs, additional checks when issuing certs, and better separation of environments (production vs staging). We use the SPIRE implementation of SPIFFE.

We wanted to be able to use our existing mutual TLS authentication with SPIFFE and extend that to AWS. Federation allows us to link an identity across multiple distinct identity systems. AWS allows for federating our internal SPIFFE identity to assume into AWS roles. This can be achieved with some additions to our AWS and SPIFFE infrastructure to allow for OpenID Connect.

OpenID Connect allows third parties to verify the identity of end users based on the authentication performed by your authorization server. In our case, OIDC allows us to verify identities in AWS using our internal SPIRE servers. We send JWTs to AWS containing metadata about the application trying to authenticate and sign that with our SPIRE servers.

The implementation of SPIFFE that we use at Square, SPIRE, comes with a small helper tool that provides the minimal requirements for the subset of the OIDC discovery spec related to exposing JWKS for JWT validation. We provide AWS with our SPIRE servers JSON Web Key Sets (JWKS) so that they can verify the authenticity of the JSON Web Tokens (JWTs) that we exchange for AWS credentials.

Previous Method

Prior to our AWS OIDC infrastructure, connecting to AWS from a datacenter application was manual and prone to human error. The diagram below gives an overview of the process:

App owners would create an AWS user with a policy. Then, they would request the user’s access keys and paste them into a credentials file using the aws_access_key_id and aws_secret_access_key settings. The contents of the file would then be uploaded to Keywhiz where applications could access it to authenticate to AWS. With this manual process, credentials often were never rotated. Some access keys had not been changed for years.

OIDC Method

Under OIDC the process is much easier and less manual. Most of the steps are automated away and are hidden from app owners. The new process is diagrammed below:

Using pre-made Terraform modules, app owners can quickly create an AWS role compatible with OIDC and attach their IAM policies to it. Then, the only change that needs to be made is adding the role ARN to the app’s P2 manifest. P2 is Square’s software deployment platform similar to kubernetes. Applications come with a manifest that allows for custom configuration. We added support internally to allow for generating an AWS credentials file in an app’s home directory. The credentials file allows applications to automatically authenticate with OIDC if they have their HOME environment variable set.

Architecture

There are a few moving parts behind the OIDC infrastructure that enable us to automatically provide AWS credentials to applications. The diagram below breaks the process down into parts:

We can break the process down into four parts, separated by background color and numerated. The light blue portion (1) outlines the process of requesting and sending JWTs from our datacenter. The pink portion (2) describes how we create the credentials file used by the AWS SDK. Next, the purple portion (3) shows how we share our JWKS with AWS. The final, orange section (4) is the work being done in AWS.

spiffe-aws-assume-role

As part of the authentication process, we needed some way to provide the credentials we received from AWS to the applications attempting to use them. Our solution was a command line tool that could be called by the AWS SDK through the credential_process option in credentials files. The tool, spiffe-aws-assume-role, reaches out to a local SPIRE agent and requests a signed JWT. The SPIRE agent then talks to it’s SPIRE server and retrieves a signed JWT which it passes to spiffe-aws-assume-role. Then, the JWT is sent to AWS where, if correctly verified, a set of AWS credentials is sent back for the SDK to use.

The command line tool takes several parameters. We opted to make all the configurable parameters through command line flags instead of a configuration file because the tool is used through custom generated credentials files.

An example of a credentials file using spiffe-aws-assume-role:

[sample-profile-name]
credential_process = spiffe-aws-assume-role credentials --sts-region us-west-2 --sts-endpoint https://sts.us-west-2.amazonaws.com --role-arn arn:aws:iam::123456789123:role/spiffe-oidc-test --spiffe-id spiffe://trustdomain.com/spiffe-oidc-test --audience 123456789123 --workload-socket unix:////agent.sock

The credentials file is generated when an app is deployed and takes into account the role trying to be assumed and the app trying to assume the role.

JWKS Syncing

In order to verify the signatures of the JWTs that we send to AWS, we need to provide AWS with the public keys or JWKS of the SPIRE server. According to the OIDC specification, these should be hosted on a publicly accessible domain under the /keys path. A corresponding document located under the /.well-known/openid-configuration details metadata about the OIDC provider. These are provided by SPIRE under the oidc-discovery-provider support project.

We chose not to directly use oidc-discovery-provider to host our OIDC setup for a variety of reasons listed below. As an alternative, we wrote a cron job that periodically connected to a local oidc-discovery-provider and forwarded the information to an S3 bucket. We used Cloudfront with custom Square domains, instead of the generated Cloudfront domain, to host the contents of the S3 buckets.

The alternative method made it a lot easier to expose our staging APIs to the public as it proved difficult to do so from the datacenter. Cloudfront also helps with availability. Our endpoints are cached and are protected from any failures in the oidc-discovery-provider. The latency and performance of our provider is also improved because it now lives on a CDN. Every time an application tries to assume a role, AWS fetches the JWKS to verify the token it receives. With our JWKS hosted in Cloudfront, AWS does not need to call into our datacenter for every assume-role request. This is simpler for us as accepting external requests can be complicated in our datacenter.

AWS Infrastructure

Each AWS account with a role that can be accessed from the DC is set up with an OIDC Identity Provider. The provider is linked to a URL and is verified by the fingerprint of the server certificate. Using custom Square domains instead of the auto-generated ones allows us to retain greater control over the OIDC provider.

Providers on AWS include an option for specifying it’s audience. When we generate a token to send to AWS, we also include an audience field. The audience field allows us to avoid the use of stolen tokens on arbitrary AWS accounts. Without the audience field, a stolen token can be used on every AWS account that the app has access to. Any AWS account that allows the SPIFFE ID of an app can be accessed with a stolen token. To prevent this, we set the audience field to the AWS account ID it is intended to be used on. Since the token is signed, information (including the audience) can’t be changed. If a stolen token is attempted on an unintended account, the provider would reject it as the audience would not match the one specified in the provider. This restricts the scope of any stolen tokens to a single AWS account. A leaked token can still be used to access other roles that the app has access to on the account, but that was a compromise we were willing to take. The alternative, more granular method would have been to set the audience directly to the role ARN the token is intended for. However, there is a limit of 100 audiences per provider and the provider would need to be updated every time we create a new role.

Each token comes with a subject field containing the SPIFFE ID of the app that requested the token. Roles that need to be accessed from the DC are configured to check for a specific SPIFFE ID.

Conclusion

Authenticating to AWS using OIDC through SPIFFE allows us to provide a more secure and easier path for app owners in the datacenter to work with AWS. Credentials are short lived and refreshed often. App owners require only a simple configuration change and no extra work with their AWS SDK setup.

With the tools provided by SPIRE and the open source spiffe-aws-assume-role tool, a similar setup can be easily replicated. The custom work we’ve done with S3 syncing and automatic generation of AWS credentials files are not needed to get the OIDC setup running.

We’ve released OIDC to general availability at Square and are in the process of migrating applications. There have been no major issues so far and app owners have been successful in migrating. After releasing a guide and announcing it to the company, owners have been slowly migrating their own apps. Beyond fielding a few questions, the process has been smooth for both owners and us.

Amazon Web Services, the “Powered by AWS” logo, and "AWS" are trademarks of Amazon.com, Inc. or its affiliates in the United States and/or other countries