Promoted
Promoted

Storing Credentials in AWS, Kubernetes, CI/CD

Topic: DevOps

Credentials and other secrets of your application must be secured all the time. In times of compromise access to credentials can have a detrimental effect on your systems and data. Because DevOps involve different systems like AWS, Kubernetes, Jenkins, there is no single way to store credentials. This post discusses some ways to store the credentials for your application in a more secure manner.

No Credentials in code Repository

0heading

This is a very common mistake among developers while pushing code to code repositories like Github, Bitbucket, or Gitlab. While running a code locally your application may use credentials to connect to a database or other application. Therefore you update the credentials in your config file. But while pushing the code to the repository if you forget to delete the credentials your credentials may be exposed to anyone who has access to the code. This is even worse if your code is being used by a third party. Once credentials are in the repository it is difficult to clean because even if you delete the config file there is still a trace of credentials in the commit. So you also need to remove those commits and readjust the head of your code branch. One good solution is to always store credentials in environment variables and let your application access these credentials through the environment. In that way, you don't have to remember to clean your config files while pushing your code.

1paragraph

Using Credentials in Jenkins/Gitlab-CI/CircleCI or other CI/CD tools

2heading

While running your workflows in CI/CD pipelines inside Jenkins or other tools like Gitlab-CI or CircleCI you need to access credentials to run unit tests or create builds. Most of these tools have configurations to set your credentials and can be accessed during the build process. There are also different levels of authorizations so one can run a pipeline but won't be able to access these credentials. It's also a good idea to let these tools set environment variables and your provisioning code can access these variables. Care must be taken that credentials are not logged into the pipeline logs.

3paragraph

Credentials in AWS EC2 nodes and other computes

4heading
  • If you are using EC2 nodes and need to access other AWS services you should use the IAM instance role. Create a role for your instance and attach that role while creating the instance. Make sure no AWS key and secrets are stored inside the instance.
  • In case your application needs credentials to run, use the AWS parameter store. Create KMS key first. Create a name for credentials and then store your value by encrypting it with a KMS key. Before running the application pull these credentials and store them in environment variables. After the application runs successfully, clear these environment variables. This can be automated using python, bash, or ansible.
  • If you are not a fan of AWS parameter stores you can also use Hashicorp Vault.
5list

Credentials in Kubernetes

6heading

Promoted
Promoted

  • If you are using the EKS cluster IAM role in the same way as the EC2 instance. In case it is hosted on an EC2 node, then containers use that node's policy. But since not all microservice will require policies of the hosted nodes you can use a KIAM-like tool to tailor the policies to your microservices.
  • Kubernetes itself has secrets managing feature. You can pull your secrets from the parameter store or vault and create a Kubernetes secret. You can create a config file or volume. You can attach this volume or config file to the pod. You can also use pod environment variables to read from Kubernetes secrets.
7list

Credentials in S3 or other Datastore

8heading

This is not recommended but in some cases there might be secret files or sensitive data in the S3 bucket or other Datastore. If there is any, these files need to be encrypted with a proper passcode.

9paragraph

Check credentials leak in Logging

10heading

Sometime inadvertently sensitive information or secrets are leaked to logging services by applications. Therefore proper security is provided for logging services. You might use encryption or some obfuscation if such data is leaked to a logging service. In ideal case there should be no secrets in logs.

11paragraph

Recommendations

12heading
  • Application code and credentials should not be stored together
  • Delete the credentials if its not used anymore
  • Never print credentials to any logs
  • Always encrypt credentials with the proper key. Also make sure the key is rotated after some time
  • Never share the same credentials between users. Each user should have its own set of credentials. Always check the authorization of these credentials.
  • Never share the same credentials between users. Each user should have its own set of credentials. Always check the authorization of these credentials.
  • Check access of each user or application through AWS cloudtrail.
  • Always do a security audit after a certain period.
13list