Secrets Management With Datadog Secret Backend Utility

Datadog has 600+ out-of-the-box integrations that cover a variety of technologies, from web servers to databases to 3rd party SaaS services. For many of these integrations, there are agent configuration files that require storing credentials for the technology. The larger issue is around how to store those credentials. Many security-minded engineers would prefer not to store those secrets in plaintext in case of unauthorized access to their servers or shared access to Datadog configuration. What not everyone knows is that the Datadog agent has a mechanism to run an executable at agent startup in order to reach out to a secrets management tool of your choosing and decrypt those secrets, storing them in memory for use by the agent.

Secrets Configuration

If you want to utilize Datadog’s secrets management capabilities, there is a specific notation that the agent recognizes. Let’s take the Datadog MySQL integration as an example. While the integration, by default, only collects information from performance-related tables, you might grant additional access to other database tables to ingest more business-specific metrics into the platform via custom queries. This may require additional permissions for more sensitive data, so you might want to ensure that the credential is not stored in plaintext in the integration configuration. 

To do this, Datadog utilizes the ENC[] notation in configuration files to tell the agent that the credential should be decrypted from a secret backend. Below is a sample of what that might look like:

Now the question becomes, “How does the agent know to decrypt this secret?”

The Executable

Within the datadog.yaml file there is a configuration parameter called secret_backend_command. This parameter is the path to the executable file that the agent uses to perform the calls to the secret backend to decrypt the secret and store it in memory. 

You can also use the secret_backend_arguments parameter if the executable requires arguments to be passed to it. The agent also requires specific permissions and attributes that must be assigned to the executable to ensure additional security:

Linux

• Executable is owned by the same user running the agent (dd-agent for VMs, root for containers)
• Have no rights for group or other
• Have at least executable rights for the owner

Windows

• Read/write for the user running the agent (ddagentuser by default)
• Have no rights for any user or group except for the Administrators group, the built-in Local System account, or the Agent user (ddagentuser)
• Be a valid Win32 application so the Agent can execute it

The agent also has an expected output to determine the values of the secret(s), such as mysql-prod-password in the example above. This output would look like the sample below:

If the “error” field is anything but null, the integration configuration that uses this secret is dropped and will not function. For some, writing a custom executable may be out of their skillset, or you may not have time to do so. Luckily, my colleagues developed a free, open-source utility to help with the Datadog secrets management problem.* 

Datadog Secret Backend Utility

Appropriately called datadog-secret-backend, this utility is a simple Go executable that supports a few different secret backend tools to help lighten the burden of writing a custom executable for use within your Datadog agents. It currently supports the following backends:

• AWS Secrets Manager
• AWS Systems Manager Parameter Store
• Azure KeyVault
• HashiCorp Vault
• JSON/YAML files
• aKeyless

The datadog-secret-backend utility is quite extensible and even supports multiple secret backends in a single configuration. Thus, users can decrypt multiple secrets stored on different backends within a single configuration file. Below, we’ll use AWS Secrets Manager and Azure KeyVault to show some example configurations.

The executable expects the configuration file above to be stored in the same directory as itself, but this can be overridden with a config parameter passed to it. While secrets are technically stored within this configuration file as well, it is important to note that the contents should only be readable by administrators of the secret backends. This ultimately helps hide secrets from those who have access to the Datadog integration configuration files, as the permissions on those are often more open.

With the above configuration, the agent is now ready to decrypt secrets, with just a minor change to the MySQL integration configuration. For the executable to know which secret backend to utilize to decrypt this secret, we prefix the secret_id with the ID of the backend as follows:

After adding the necessary secret_backend_command configuration to the datadog.yaml file, restart your Datadog agent. You can verify that the secrets are being decrypted successfully by running the agent’s built-in secret command.

Once you confirm that the secrets are being decrypted appropriately, you’re ready to go! You can find all documentation related to each of the supported secret backends in the repository’s README. I hope this helps you keep your secrets hidden and eases some of the burden of custom tool development for your team.

* Originally developed by RapDev, the Secret Backend Utility was transferred to Datadog's GitHub repository in a collaborative effort, allowing for broader visibility and community contributions. This move has enhanced the tool’s functionality and supports the security needs of Datadog users, ensuring its ongoing development and increased user adoption.