Secrets Management

Best practices and approaches for managing secrets in Argo CD deployments

There are two general approaches to managing secrets in GitOps: on the destination cluster, or in Argo CD during manifest generation. We strongly recommend the destination cluster approach as it is more secure and provides a better user experience.

Argo CD caches generated manifests (including injected secrets) in Redis as plaintext. Generation-based secret injection significantly increases security risk.

Recommended: Destination Cluster Secret Management

In this approach, secrets are populated directly on the destination cluster, and Argo CD does not need to manage them.

How It Works

1. Store Secrets Externally

   Secrets are stored in external systems like AWS Secrets Manager, HashiCorp Vault, Azure Key Vault, or Google Secret Manager.

2. Deploy Operator

   An operator runs on the destination cluster to sync secrets from the external system.

3. Application References Secrets

   Applications reference Kubernetes Secrets that are automatically created and updated by the operator.

Advantages

Enhanced Security

Argo CD never has access to secrets, reducing risk of leakage through API, cache, or logs.

Decoupled Updates

Secret updates are independent from app sync operations, avoiding unintended secret changes during releases.

Rendered Manifests Compatible

Works with the “Rendered Manifests” GitOps pattern, an emerging best practice.

Zero Trust Architecture

Follows principle of least privilege—Argo CD has no secret access.

Popular Solutions

Sealed Secrets

Sealed Secrets encrypts secrets that can be stored safely in Git.

SealedSecret Example

apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: mysecret
  namespace: default
spec:
  encryptedData:
    password: AgBy3i4OJSWK+PiTySYZZA9rO43cGDEq...

The controller running on the cluster decrypts SealedSecrets and creates corresponding Kubernetes Secrets.

External Secrets Operator

External Secrets Operator syncs secrets from external secret management systems.

AWS Secrets Manager

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: aws-secret
  namespace: default
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secretsmanager
    kind: SecretStore
  target:
    name: database-credentials
  data:
  - secretKey: username
    remoteRef:
      key: prod/database/credentials
      property: username
  - secretKey: password
    remoteRef:
      key: prod/database/credentials
      property: password

HashiCorp Vault

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: vault-secret
  namespace: default
spec:
  refreshInterval: 15m
  secretStoreRef:
    name: vault-backend
    kind: SecretStore
  target:
    name: api-credentials
  data:
  - secretKey: token
    remoteRef:
      key: secret/data/api
      property: token

Google Secret Manager

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: gcp-secret
  namespace: default
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: gcpsm-secretstore
    kind: SecretStore
  target:
    name: application-secrets
  data:
  - secretKey: api-key
    remoteRef:
      key: projects/123456/secrets/api-key/versions/latest

Kubernetes Secrets Store CSI Driver

Secrets Store CSI Driver mounts secrets from external stores as volumes.

apiVersion: v1
kind: Pod
metadata:
  name: app-pod
spec:
  containers:
  - name: app
    image: myapp:latest
    volumeMounts:
    - name: secrets-store
      mountPath: "/mnt/secrets"
      readOnly: true
  volumes:
  - name: secrets-store
    csi:
      driver: secrets-store.csi.k8s.io
      readOnly: true
      volumeAttributes:
        secretProviderClass: "aws-secrets"

Other Solutions

AWS Secret Operator

Kubernetes operator for AWS Secrets Manager

Vault Secrets Operator

Official HashiCorp Vault operator for Kubernetes

Not Recommended: Manifest Generation-Based

We strongly caution against this approach due to security and operational risks.

In this approach, Argo CD uses a Config Management Plugin to inject secrets during manifest generation.

Disadvantages

Security Risks

• Argo CD needs access to secrets, increasing attack surface
• Generated manifests with secrets stored in plaintext in Redis cache
• Secrets exposed via repo-server API (gRPC service)
• Anyone with Redis or repo-server access can view secrets

Operational Risks

• Secret updates coupled with application sync operations
• Risk of unintentional secret changes during unrelated releases
• Difficult to audit when secrets were actually updated

Compatibility Issues

• Incompatible with “Rendered Manifests” pattern
• Limits adoption of emerging GitOps best practices
• May complicate future migrations

Mitigating Risks (If You Must Use This Approach)

Many users have already adopted generation-based solutions. Argo CD will continue to support this approach, but will not prioritize new features that solely support this style.

If you must use secret-injection plugins, implement these mitigations:

1. Network Policies

   Set up network policies to prevent direct access to Redis and repo-server.

   apiVersion: networking.k8s.io/v1
   kind: NetworkPolicy
   metadata:
     name: argocd-redis-network-policy
     namespace: argocd
   spec:
     podSelector:
       matchLabels:
         app.kubernetes.io/name: argocd-redis
     policyTypes:
     - Ingress
     ingress:
     - from:
       - podSelector:
           matchLabels:
             app.kubernetes.io/part-of: argocd
   

   Verify your cluster supports NetworkPolicies and can enforce them (not all CNI plugins do).

2. Dedicated Cluster

   Run Argo CD on its own cluster with no other applications.

   • Reduces blast radius if secrets are compromised
   • Limits potential lateral movement
   • Simplifies security auditing
3. Redis Encryption

   Enable Redis TLS and authentication:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: argocd-cmd-params-cm
     namespace: argocd
   data:
     redis.server: "argocd-redis:6379"
     redis.use.tls: "true"
   
4. Access Controls

   Strictly limit RBAC permissions for accessing:

   • Redis pods and services
   • repo-server pods and services
   • argocd namespace resources
5. Audit Logging

   Enable comprehensive audit logging:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: argocd-cm
     namespace: argocd
   data:
     # Log all API requests
     server.log.level: "debug"
     # Enable audit logging
     server.audit.log.enabled: "true"
   

Popular Plugin: argocd-vault-plugin

argocd-vault-plugin is a popular Config Management Plugin for secret injection.

This plugin requires Argo CD to have access to your secret backend, increasing security risk.

Plugin Configuration

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
data:
  configManagementPlugins: |
    - name: argocd-vault-plugin
      generate:
        command: ["argocd-vault-plugin"]
        args: ["generate", "./"]

Migration Strategy

If you’re currently using manifest generation-based secrets, consider migrating:

1. Audit Current Usage

   Identify all applications using secret injection plugins:

   kubectl get applications -A -o json | \
     jq '.items[] | select(.spec.source.plugin != null) | .metadata.name'
   
2. Choose Destination Approach

   Select an operator-based solution:

   • External Secrets Operator (multi-cloud)
   • Sealed Secrets (Git-native)
   • Cloud-specific operators
3. Deploy Operator

   Install chosen operator on destination clusters:

   # Example: External Secrets Operator
   helm repo add external-secrets https://charts.external-secrets.io
   helm install external-secrets \
     external-secrets/external-secrets \
     -n external-secrets-system \
     --create-namespace
   
4. Migrate Incrementally

   Migrate applications one at a time:

   1. Create ExternalSecret/SealedSecret resources
   2. Update application to reference new Secrets
   3. Remove plugin configuration
   4. Test thoroughly
5. Remove Plugin

   Once all applications are migrated, remove the plugin from Argo CD configuration.

Comparison Table

Feature	Destination Cluster	Manifest Generation
Argo CD Secret Access	❌ No	✅ Yes (High Risk)
Secret in Redis Cache	❌ No	✅ Yes (Plaintext)
Update Coupling	✅ Decoupled	❌ Coupled with Sync
Rendered Manifests	✅ Compatible	❌ Incompatible
Security Posture	✅ Strong	⚠️ Weak
Complexity	⚠️ Medium	✅ Simple
Argo CD Version	Any	Any

Best Practices

Principle of Least Privilege

Argo CD should only have permissions it absolutely needs—not secret access

Secret Rotation

Use operators that support automatic secret rotation from external systems

Separate Concerns

Decouple secret management from application deployment workflows

Audit Trail

Leverage external secret manager audit logs for compliance

Related Resources

Security Overview

Comprehensive Argo CD security architecture

TLS Configuration

Secure inter-component communication

RBAC Configuration

Control access to Argo CD resources

Config Management Plugins

Learn about plugin architecture