Are You Leaking Secrets with Terraform?

Make your Terraform code bulletproof when it comes to sensitive values.

A 2022 GitGuardian report revealed that, on average, 3 out of every 1,000 commits contained a credential, and the number of leaked secrets has been rising sharply each year.

C’mon folks, let’s try harder to keep our companies out of the news.

No matter your role, if you contribute code of any variety, you should be aware of best practices when it comes to using secrets in that code. Today though, let’s focus on Terraform and look at real world examples.

(Insecure) Hardcoding sensitive values

Take the following Terraform example. It’s a classic use case that deploys a Lambda Function with an environment variable containing an API Key to a third-party API or database.

resource "aws_lambda_function" "test_lambda" {
  filename      = "lambda_function_payload.zip"
  function_name = "lambda_function_name"
  role          = aws_iam_role.iam_for_lambda.arn
  handler       = "index.test"
  source_code_hash = data.archive_file.lambda.output_base64sha256
  runtime = "nodejs18.x"

  environment {
    variables = {
      # ⚠️ DO NOT DO THE FOLLOWING ⚠️
      API_KEY = "390de376-a86d-4d5d-9ff7-6ca977a3f2aa"
    }
  }
}

Great, the stage is set. It’s obvious why you shouldn’t do this: the secret is in plaintext and even stored in Git. By the way, if you think it’s easy to remove secrets from Git once they’re committed, think again.

(Insecure) Mark variables as sensitive

Fortunately, Terraform has this thing called sensitive variables we can use.

variable "api_key" {
  description = "Key credential for the API"
  type        = string
  sensitive   = true
}

Great, problem solved, right!? Nope.

Marking variables as sensitive in Terraform is good practice, but it only provides basic protection against some accidental exposure in the CLI and log output. This does NOT encrypt the secret value in the state file; you’re still at risk if this is all you do.

Let’s look at options that are considered truly “secure.”

HCP Terraform & HashiCorp Vault

If you use HCP Terraform or Terraform Enterprise, you’ll rest easy knowing that it encrypts all variable values.

Likewise, HashiCorp Vault is a great solution to securing your secrets in Terraform.

However, I don’t care about paying a premium for a feature we can do pretty trivially on our own. Let’s look at DIY options.

AWS Solutions

Integrating Secrets Manager

AWS Secrets Manager is a managed service for…yep, managing secrets. Secrets Manager is one of the go-to tools for securing sensitive values for applications hosted on AWS, and we can tap into this power for Terraform too.

The process is fairly simple:

1. Create the secret in AWS Secrets Manager. Use whatever method you prefer (AWS Console, CLI, etc.), just not Terraform. Take note of the Secret ID.

2. Use Terraform’s aws_secretsmanager_secret_version data source to query the Secret by its ID (you can use a regular TF variable to pass the ID if you wish).

3. Use Terraform’s jsondecode() function on the data source’s .secret_string attribute.

data "aws_secretsmanager_secret_version" "api_key" {
  secret_id = "arn:aws:secretsmanager:us-east-1:123456789012:secret:my_api_key-123456"
}

locals {
  api_key = jsondecode(data.aws_secretsmanager_secret_version.api_key.secret_string)
}

Pros:

• Centralizes where secrets are managed.

• Access to secrets can be limited to read-only.

• Access can be revoked even if user has access to the Terraform code.

Cons:

• Sensitive value plaintext can still be stored in Terraform state file if the decrypted value is stored in a resource’s unprotected attributes.

• Requires a non-Terraform way of creating and editing secrets initially.

KMS-Encrypted Values

AWS Key Management Service (KMS) is another acceptable solution similar to Secrets Manager. With KMS, the process is nearly identical:

1. Use the AWS CLI for KMS to encrypt the plaintext secret value and get its ciphertext.

2. Create a local or variable in Terraform to store the KMS ciphertext.

3. Use Terraform’s aws_kms_secrets data source to dynamically decrypt the ciphertext to its usable value.

locals {
  api_key_ciphertext = "AQECAHgaPa0J8WaeplGCqqVAr4HNvDaFSQ+NaiwIBhmm6qDSFwAAAGIwYAYJKoZIhvcNAQcGoFMwUQIBADBMBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDI+LoLdvYv8l41OhAAIBEIAfx49FFJCLeYrkfMfAw6XlnxP13MmDBdqP8dPp28OoBQ=="
}

data "aws_kms_secrets" "this" {
  secret {
    name    = "api_key"
    payload = local.api_key_ciphertext
  }

  # <more secret{} blocks here>
}

locals {
  api_key = data.aws_kms_secrets.this.plaintext["api_key"]
}

Pros:

• Access to secrets can be limited to read-only via the KMS keys used to encrypt them.

• Access can be revoked even if user has access to the Terraform code.

• Sensitive values can be safely commited to VCS like Git, as it is ciphertext. This can make development more transparent, and provide better history for auditing.

• More control over how the encryption key is managed, if using Customer-Managed Keys (CMK), such as key rotation and multi-region keys.

Cons:

• Sensitive value plaintext can still be stored in Terraform state file if the decrypted value is stored in a resource’s unprotected attributes.

  • Note: if you are using Terraform v1.10 or later, you can use Ephemeral values to prevent it from being stored in state at all.

• Requires a non-Terraform way of creating and editing secrets initially.

Cloud-Agnostic Solutions

Not using AWS? No problem, you can still do this.

Integrating SOPS

SOPS is an open-source, free, CNCF tool for encrypting and decrypting secrets. It’s cloud-agnostic so you can choose your preferred backer: AWS KMS, Azure Key Vault, GCP KMS, as well as age and GPG for generic environments.

I found this blog by dev.to very useful if you’d like to setup SOPS with Terraform/Terragrant.

This essentially involves the following process:

1. Store your secrets in a file like secrets.yaml

2. Encrypt the file using SOPS CLI (you can also use a Terraform null resource with local provisioner).

3. Use the SOPS Terraform provider’s sops_file data source to decrypt the file and use throughout your Terraform configuration.

# secrets.yaml
api_key: "supersecret"

data "sops_file" "secrets" {
  source_file = "${path.module}/secrets.enc.yaml"
}

locals {
  api_key = data.sops_file.secrets.data["api_key"]
}

Pros:

• Extremely flexible. Encryption methods can be mixed and matched to various providers.

• Supports both cloud-managed and local services.

• Sensitive values can be safely commited to VCS like Git, as it is ciphertext. This can make development more transparent, and provide better history for auditing.

Cons:

• Sensitive value plaintext can still be stored in Terraform state file if the decrypted value is stored in a resource’s unprotected attributes.

  • Note: if you are using Terraform v1.10 or later, use Ephemeral values to prevent it from being stored in state at all.

• Key rotation is supported but may be more difficult to implement when using local GPG or age keys.

Detect Leaked Secrets

Time for your homework assignment. That’s right, action!

You can take a proactive approach to keeping your code (infrastructure and otherwise) secure by running a free, open-source, tool like trufflehog!

I’m not sponsored by anyone, but seriously…with just a few minutes, you can install trufflehog and run it against your local Git repo.

Tools like these can even be executed as pre-commit and pre-receive hooks, as well as CI/CD, so you don’t accidentally leak secrets in the first place.

What we learned.

Managing secrets is an every-day task for DevOps and platform engineers. These secrets can be a major attack vector if we’re not careful.

We’ve seen the examples of what NOT to do. With multiple solutions for AWS and other environments, we know what secure Terraform secrets should look like.

I expect figures like those reported by GitGuardian at the beginning will continue to increase each year; however, hopefully these best practices will help you reading this to break that trend.