Health Checks

Configure how Argo CD assesses the health status of your application resources

Argo CD continuously monitors the health of application resources and surfaces their status in the UI and CLI. Understanding health checks is crucial for ensuring your applications are truly ready.

Overview

Argo CD provides built-in health assessments for standard Kubernetes resources and allows custom health checks for CRDs and special cases.

Application health is determined by the worst health of its immediate child resources:

• Healthy > Suspended > Progressing > Missing > Degraded > Unknown

For example, if an app has a Missing resource and a Degraded resource, the app is marked Degraded.

Resource health is not inherited from child resources. A Deployment’s health is based only on its own status, not its Pods.

Built-in Health Checks

Deployment, ReplicaSet, StatefulSet, DaemonSet

Healthy when:

• Observed generation equals desired generation
• Number of updated replicas equals desired replicas

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  replicas: 3
  # ...
status:
  observedGeneration: 5
  replicas: 3
  updatedReplicas: 3  # Must equal replicas
  readyReplicas: 3

Service (LoadBalancer)

Healthy when:

• Service type is LoadBalancer
• status.loadBalancer.ingress list has at least one IP or hostname

apiVersion: v1
kind: Service
metadata:
  name: my-service
spec:
  type: LoadBalancer
status:
  loadBalancer:
    ingress:
    - ip: 203.0.113.10  # Health check passes

Ingress

Healthy when:

• status.loadBalancer.ingress list has at least one IP or hostname

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-ingress
status:
  loadBalancer:
    ingress:
    - hostname: app.example.com  # Health check passes

PersistentVolumeClaim

Healthy when:

• status.phase is Bound

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data-volume
status:
  phase: Bound  # Health check passes

Job

• Suspended: If spec.suspended is true
• Healthy: If job completed successfully
• Degraded: If job failed

apiVersion: batch/v1
kind: Job
metadata:
  name: migration
spec:
  suspend: false
status:
  succeeded: 1  # Health check passes

CronJob

• Degraded: If last scheduled job failed
• Progressing: If last scheduled job is running
• Healthy: Otherwise

apiVersion: batch/v1
kind: CronJob
metadata:
  name: backup
status:
  lastScheduleTime: "2024-03-04T10:00:00Z"
  # Health depends on status of last job

Health Status Types

Status	Meaning
Healthy	Resource is functioning correctly
Progressing	Resource is not healthy yet but making progress
Degraded	Resource is not functioning correctly
Suspended	Resource is suspended (e.g., paused Deployment, suspended CronJob)
Missing	Resource is not present in the cluster
Unknown	Health status cannot be determined

Custom Health Checks

Define custom health checks in Lua for CRDs or to override built-in checks.

Method 1: ConfigMap Configuration

Add custom health checks to the argocd-cm ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
data:
  resource.customizations.health.cert-manager.io_Certificate: |
    hs = {}
    if obj.status ~= nil then
      if obj.status.conditions ~= nil then
        for i, condition in ipairs(obj.status.conditions) do
          if condition.type == "Ready" and condition.status == "False" then
            hs.status = "Degraded"
            hs.message = condition.message
            return hs
          end
          if condition.type == "Ready" and condition.status == "True" then
            hs.status = "Healthy"
            hs.message = condition.message
            return hs
          end
        end
      end
    end
    
    hs.status = "Progressing"
    hs.message = "Waiting for certificate"
    return hs

Wildcard Health Checks

Apply a single health check to multiple resources:

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
data:
  resource.customizations: |
    ec2.aws.crossplane.io/*:
      health.lua: |
        hs = {}
        if obj.status ~= nil then
          if obj.status.conditions ~= nil then
            for i, condition in ipairs(obj.status.conditions) do
              if condition.type == "Ready" then
                if condition.status == "True" then
                  hs.status = "Healthy"
                  hs.message = "Resource is ready"
                else
                  hs.status = "Progressing"
                  hs.message = condition.message or "Resource is not ready"
                end
                return hs
              end
            end
          end
        end
        hs.status = "Progressing"
        hs.message = "Waiting for status"
        return hs

Wildcard patterns only work with the resource.customizations key format, not resource.customizations.health.<group>_<kind>.

Example: Crossplane Resources

data:
  resource.customizations: |
    "*.aws.crossplane.io/*":
      health.lua: |
        hs = {}
        if obj.status ~= nil and obj.status.conditions ~= nil then
          for i, condition in ipairs(obj.status.conditions) do
            if condition.type == "Ready" then
              if condition.status == "True" then
                hs.status = "Healthy"
              else
                hs.status = "Degraded"
              end
              hs.message = condition.reason or ""
              return hs
            end
          end
        end
        hs.status = "Progressing"
        hs.message = "Provisioning resource"
        return hs

Example: Custom Application CRD

data:
  resource.customizations.health.app.example.com_MyApp: |
    hs = {}
    if obj.status ~= nil then
      if obj.status.phase == "Running" then
        hs.status = "Healthy"
        hs.message = "Application is running"
      elseif obj.status.phase == "Failed" then
        hs.status = "Degraded"
        hs.message = obj.status.message or "Application failed"
      else
        hs.status = "Progressing"
        hs.message = "Application is starting"
      end
      return hs
    end
    hs.status = "Progressing"
    hs.message = "Waiting for status"
    return hs

Enabling Lua Standard Libraries

By default, standard Lua libraries are disabled for security. Enable them per resource:

data:
  resource.customizations.useOpenLibs.cert-manager.io_Certificate: "true"
  resource.customizations.health.cert-manager.io_Certificate: |
    -- Lua standard libraries available here
    hs = {}
    -- ... health check logic
    return hs

Only enable standard libraries if absolutely necessary, as they can pose security risks.

Method 2: Contribute Built-in Health Check

Contribute health checks directly to Argo CD’s codebase:

argo-cd/
└── resource_customizations/
    └── cert-manager.io/
        └── Certificate/
            ├── health.lua
            ├── health_test.yaml
            └── testdata/
                ├── healthy.yaml
                └── degraded.yaml

health.lua:

hs = {}
if obj.status ~= nil then
  if obj.status.conditions ~= nil then
    for i, condition in ipairs(obj.status.conditions) do
      if condition.type == "Ready" and condition.status == "True" then
        hs.status = "Healthy"
        hs.message = condition.message
        return hs
      end
    end
  end
end
hs.status = "Progressing"
hs.message = "Waiting for certificate"
return hs

health_test.yaml:

tests:
- healthStatus:
    status: Healthy
    message: Certificate is ready
  inputPath: testdata/healthy.yaml
- healthStatus:
    status: Progressing
    message: Waiting for certificate
  inputPath: testdata/degraded.yaml

Test with: go test -v ./util/lua/

Overriding Built-in Health Checks

Override Go-based health checks with Lua:

data:
  resource.customizations.health.apps_Deployment: |
    hs = {}
    -- Custom logic to override built-in Deployment health check
    if obj.status.availableReplicas == obj.spec.replicas then
      hs.status = "Healthy"
    else
      hs.status = "Progressing"
    end
    return hs

Resources with Go-based health checks:

• PersistentVolumeClaim
• Pod
• Service
• apiregistration.k8s.io/APIService
• apps/DaemonSet
• apps/Deployment
• apps/ReplicaSet
• apps/StatefulSet
• argoproj.io/Workflow
• autoscaling/HorizontalPodAutoscaler
• batch/Job
• extensions/Ingress
• networking.k8s.io/Ingress

Ignoring Resource Health

Exclude specific resources from affecting application health:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: optional-component
  annotations:
    argocd.argoproj.io/ignore-healthcheck: "true"
spec:
  replicas: 1
  # ...

The application will not be affected by this Deployment’s health status.

Argo CD Application Health Check

Argo CD removed the built-in health check for argoproj.io/Application in v1.8. Restore it for App-of-Apps patterns:

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
data:
  resource.customizations.health.argoproj.io_Application: |
    hs = {}
    hs.status = "Progressing"
    hs.message = ""
    if obj.status ~= nil then
      if obj.status.health ~= nil then
        hs.status = obj.status.health.status
        if obj.status.health.message ~= nil then
          hs.message = obj.status.health.message
        end
      end
    end
    return hs

Health Check Examples

Example: Kafka Topic (Strimzi)

data:
  resource.customizations.health.kafka.strimzi.io_KafkaTopic: |
    hs = {}
    if obj.status ~= nil and obj.status.conditions ~= nil then
      for i, condition in ipairs(obj.status.conditions) do
        if condition.type == "Ready" then
          if condition.status == "True" then
            hs.status = "Healthy"
            hs.message = "Topic is ready"
          else
            hs.status = "Degraded"
            hs.message = condition.message or "Topic is not ready"
          end
          return hs
        end
      end
    end
    hs.status = "Progressing"
    hs.message = "Waiting for topic to be ready"
    return hs

Example: Sealed Secret

data:
  resource.customizations.health.bitnami.com_SealedSecret: |
    hs = {}
    if obj.status ~= nil then
      if obj.status.conditions ~= nil then
        for i, condition in ipairs(obj.status.conditions) do
          if condition.type == "Synced" and condition.status == "True" then
            hs.status = "Healthy"
            hs.message = "SealedSecret synced successfully"
            return hs
          elseif condition.type == "Synced" and condition.status == "False" then
            hs.status = "Degraded"
            hs.message = condition.message or "Failed to sync SealedSecret"
            return hs
          end
        end
      end
    end
    hs.status = "Progressing"
    hs.message = "Waiting for SealedSecret to sync"
    return hs

Example: External Secrets

data:
  resource.customizations.health.external-secrets.io_ExternalSecret: |
    hs = {}
    if obj.status ~= nil then
      if obj.status.conditions ~= nil then
        for i, condition in ipairs(obj.status.conditions) do
          if condition.type == "Ready" and condition.status == "True" then
            hs.status = "Healthy"
            hs.message = "ExternalSecret is ready"
            return hs
          elseif condition.type == "Ready" and condition.status == "False" then
            hs.status = "Degraded"
            hs.message = condition.message
            return hs
          end
        end
      end
    end
    hs.status = "Progressing"
    hs.message = "Waiting for ExternalSecret"
    return hs

Testing Health Checks

Test health checks locally before deploying:

# Install Argo CD locally
git clone https://github.com/argoproj/argo-cd.git
cd argo-cd

# Add your health check
mkdir -p resource_customizations/mygroup.io/MyKind
cat > resource_customizations/mygroup.io/MyKind/health.lua <<EOF
hs = {}
-- your health check logic
return hs
EOF

# Run tests
go test -v ./util/lua/

Troubleshooting

Resource Shows as Progressing Forever

1. Check if a custom health check exists:

   kubectl get configmap argocd-cm -n argocd -o yaml | grep -A 20 customizations
   

2. Inspect resource status:

   kubectl get <resource> <name> -o yaml
   

3. Check Argo CD application controller logs:

   kubectl logs -n argocd -l app.kubernetes.io/name=argocd-application-controller
   

Custom Health Check Not Working

1. Verify ConfigMap format (key should be resource.customizations.health.<group>_<kind>)

2. Check for Lua syntax errors in controller logs

3. Ensure the resource group and kind match exactly

4. Restart application controller:

   kubectl rollout restart deployment argocd-application-controller -n argocd
   

Application Shows Wrong Health

Remember: Application health is the worst health of immediate child resources:

# Check all resource health
argocd app get my-app

# Check specific resource
argocd app get my-app --resource <group>:<kind>:<namespace>/<name>

Best Practices

Start Simple

Begin with built-in checks before writing custom logic

Use Conditions

Leverage Kubernetes status conditions for consistent checks

Return Messages

Always provide meaningful health messages for debugging

Test Thoroughly

Create test cases for all health states

Next Steps

Sync Waves

Wait for resources to be healthy between waves

Resource Hooks

Run validation hooks after resources are healthy

Sync Options

Configure sync behavior

Creating Apps

Learn how to create applications