Notifications - Argo CD

Argo CD Notifications continuously monitors applications and provides a flexible way to notify users about important changes in application state through configurable triggers and templates.

Overview

The notification system consists of three main components:

Triggers - Define when notifications should be sent
Templates - Define the content and format of notifications
Services - Define where notifications are delivered (Slack, email, etc.)

Getting Started

Install notification catalog

kubectl apply -n argocd --server-side --force-conflicts \
  -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/notifications_catalog/install.yaml

Configure notification service

For email notifications:

kubectl apply -n argocd -f - << EOF
apiVersion: v1
kind: Secret
metadata:
  name: argocd-notifications-secret
stringData:
  email-username: $EMAIL_USER
  email-password: $PASSWORD
type: Opaque
EOF

kubectl patch cm argocd-notifications-cm -n argocd --type merge -p '{
  "data": {
    "service.email.gmail": "{ username: $email-username, password: $email-password, host: smtp.gmail.com, port: 465, from: $email-username }"
  }
}'

Subscribe to notifications

kubectl patch app myapp -n argocd -p '{
  "metadata": {
    "annotations": {
      "notifications.argoproj.io/subscribe.on-sync-succeeded.slack": "my-channel"
    }
  }
}' --type merge

Triggers

Triggers define conditions for sending notifications using predicate expressions.

Basic Trigger

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-notifications-cm
data:
  trigger.on-sync-status-unknown: |
    - when: app.status.sync.status == 'Unknown'
      send: [app-sync-status, github-commit-status]

Condition Bundles

Multiple conditions with different templates:

data:
  trigger.sync-operation-change: |
    - when: app.status?.operationState.phase in ['Succeeded']
      send: [github-commit-status]
    - when: app.status?.operationState.phase in ['Running']
      send: [github-commit-status]
    - when: app.status?.operationState.phase in ['Error', 'Failed']
      send: [app-sync-failed, github-commit-status]

Use the ?. operator for optional fields. For example, app.status?.operationState.phase won’t fail if operationState is nil.

Avoid Duplicate Notifications

Use oncePer to send notifications only when a field changes:

data:
  trigger.on-deployed: |
    when: app.status?.operationState.phase in ['Succeeded'] and app.status.health.status == 'Healthy'
    oncePer: app.status.sync.revision
    send: [app-sync-succeeded]

For monorepos, use:

oncePer: app.status?.operationState.syncResult.revision

Common Triggers

Sync Success
Sync Failure
Health Degraded
Deployment Running

trigger.on-sync-succeeded: |
  - when: app.status?.operationState.phase in ['Succeeded']
    oncePer: app.status.sync.revision
    send: [app-sync-succeeded]

trigger.on-sync-failed: |
  - when: app.status?.operationState.phase in ['Error', 'Failed']
    send: [app-sync-failed]

trigger.on-health-degraded: |
  - when: app.status.health.status == 'Degraded'
    send: [app-health-degraded]

trigger.on-sync-running: |
  - when: app.status?.operationState.phase in ['Running']
    oncePer: app.status?.operationState.startedAt
    send: [app-sync-running]

Default Triggers

Set default triggers for services:

data:
  defaultTriggers: |
    - on-sync-status-unknown
  
  defaultTriggers.slack: |
    - on-sync-running
    - on-sync-succeeded
    - on-sync-failed

Subscribe using defaults:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  annotations:
    notifications.argoproj.io/subscribe.slack: my-channel

Templates

Templates define notification content for different services.

Basic Template

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-notifications-cm
data:
  template.app-sync-succeeded: |
    message: |
      Application {{.app.metadata.name}} has been successfully synced.
    email:
      subject: Application {{.app.metadata.name}} synced
    slack:
      attachments: |
        [{
          "title": "{{.app.metadata.name}}",
          "title_link": "{{.context.argocdUrl}}/applications/{{.app.metadata.name}}",
          "color": "good",
          "fields": [{
            "title": "Sync Status",
            "value": "{{.app.status.sync.status}}",
            "short": true
          }]
        }]

Available Variables

object

The Application object containing metadata, spec, and status

object

Context information including:

argocdUrl - Argo CD server URL
notificationsUrl - Notifications controller URL

object

Repository metadata

string

The notification service type (slack, email, etc.)

string

The notification recipient

Template Functions

Use built-in functions in templates:

template.app-deployed: |
  message: |
    Application {{.app.metadata.name}} is now running version {{.app.status.sync.revision}}.
    Deployed at: {{time.Now.Format "2006-01-02 15:04:05"}}
    Time since last deployment: {{time.Now.Sub (time.Parse .app.status.operationState.finishedAt).Minutes}} minutes

Notification Services

Configure delivery channels for notifications.

Slack

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-notifications-cm
data:
  service.slack: |
    token: $slack-token

apiVersion: v1
kind: Secret
metadata:
  name: argocd-notifications-secret
stringData:
  slack-token: xoxb-your-token-here

Subscribe:

annotations:
  notifications.argoproj.io/subscribe.on-sync-succeeded.slack: my-channel

Email

Gmail
AWS SES
Generic SMTP

service.email.gmail: |
  username: $email-username
  password: $email-password
  host: smtp.gmail.com
  port: 465
  from: $email-username

service.email.aws-ses: |
  region: us-east-1
  from: notifications@example.com

service.email.custom: |
  username: $smtp-username
  password: $smtp-password
  host: smtp.example.com
  port: 587
  from: argocd@example.com
  insecure_skip_verify: false

Subscribe:

annotations:
  notifications.argoproj.io/subscribe.on-sync-failed.email: team@example.com

Microsoft Teams

service.teams: |
  recipientUrls:
    channel1: https://outlook.office.com/webhook/...

Webhook

service.webhook.github: |
  url: https://api.github.com/repos/myorg/myrepo/statuses/{{.app.status.sync.revision}}
  headers:
    - name: Authorization
      value: token $github-token

PagerDuty

service.pagerdutyv2: |
  serviceKeys:
    my-service: $pagerduty-key

Opsgenie

service.opsgenie: |
  apiUrl: https://api.opsgenie.com
  apiKeys:
    team1: $opsgenie-api-key

Subscriptions

Subscribe applications or projects to notifications:

Application Subscriptions

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  annotations:
    # Subscribe to specific trigger
    notifications.argoproj.io/subscribe.on-sync-succeeded.slack: my-channel
    
    # Subscribe to multiple services
    notifications.argoproj.io/subscribe.on-sync-failed.slack: alerts
    notifications.argoproj.io/subscribe.on-sync-failed.email: team@example.com
    
    # Use default triggers
    notifications.argoproj.io/subscribe.slack: general

Project Subscriptions

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  annotations:
    notifications.argoproj.io/subscribe.on-sync-succeeded.slack: project-updates

All applications in the project will inherit these subscriptions.

Namespace-Based Configuration

Allow teams to configure notifications in their own namespaces:

Enable Self-Service Notifications

# In argocd-cmd-params-cm ConfigMap
apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cmd-params-cm
data:
  application.namespaces: team-one, team-two
  notificationscontroller.selfservice.enabled: "true"

Team Namespace Configuration

# In team's namespace
apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-notifications-cm
  namespace: team-one
data:
  service.pagerdutyv2: |
    serviceKeys:
      my-service: $pagerduty-key
---
apiVersion: v1
kind: Secret
metadata:
  name: argocd-notifications-secret
  namespace: team-one
type: Opaque
stringData:
  pagerduty-key: <integration-key>

When the same service and trigger are defined at both controller and application level, both notifications will be sent.

Advanced Features

Conditional Subscriptions

Use annotations to conditionally enable notifications:

metadata:
  annotations:
    # Only notify on production
    notifications.argoproj.io/subscribe.on-sync-succeeded.slack: |
      {{- if eq .app.metadata.labels.environment "production" }}
      prod-alerts
      {{- end }}

Custom Annotations in Triggers

data:
  trigger.on-version-change: |
    - when: app.metadata.annotations["example.com/version"] != ""
      oncePer: app.metadata.annotations["example.com/version"]
      send: [version-changed]

Grafana Annotations

Automatically create Grafana annotations:

service.grafana: |
  apiUrl: https://grafana.example.com/api
  apiKey: $grafana-api-key

Troubleshooting

Check Notification Controller Logs

kubectl logs -n argocd -l app.kubernetes.io/name=argocd-notifications-controller

Test Notifications

# Test notification configuration
argocd admin notifications trigger get on-sync-succeeded

# Send test notification
argocd admin notifications template notify \
  app-sync-succeeded myapp \
  --recipient slack:my-channel

Common Issues

Notifications not being sent

Verify trigger condition matches application state
Check service configuration and credentials
Ensure subscription annotation is correct
Review controller logs for errors

Duplicate notifications

Use oncePer field in trigger configuration
Check for overlapping trigger conditions
Verify both global and namespace configs

Template rendering errors

Validate template syntax
Use ?. for optional fields
Test templates with argocd admin command

Best Practices

Use the catalog

Start with built-in triggers and templates from the catalog

Avoid notification spam

Use oncePer to prevent duplicate notifications

Secure credentials

Store service credentials in Kubernetes Secrets

Test before deploying

Use argocd admin notifications to test configuration

Enable self-service

Allow teams to configure their own notifications

Monitor the controller

Set up alerts for notification controller errors

​Overview

​Getting Started

​Triggers

​Basic Trigger

​Condition Bundles

​Avoid Duplicate Notifications

​Common Triggers

​Default Triggers

​Templates

​Basic Template

​Available Variables

​Template Functions

​Notification Services

​Slack

​Email

​Microsoft Teams

​Webhook

​PagerDuty

​Opsgenie

​Subscriptions

​Application Subscriptions

​Project Subscriptions

​Namespace-Based Configuration

​Enable Self-Service Notifications

​Team Namespace Configuration

​Advanced Features

​Conditional Subscriptions

​Custom Annotations in Triggers

​Grafana Annotations

​Troubleshooting

​Check Notification Controller Logs

​Test Notifications

​Common Issues

​Best Practices

Use the catalog

Avoid notification spam

Secure credentials

Test before deploying

Enable self-service

Monitor the controller

Related Resources

Overview

Getting Started

Triggers

Basic Trigger

Condition Bundles

Avoid Duplicate Notifications

Common Triggers

Default Triggers

Templates

Basic Template

Available Variables

Template Functions

Notification Services

Slack

Email

Microsoft Teams

Webhook

PagerDuty

Opsgenie

Subscriptions

Application Subscriptions

Project Subscriptions

Namespace-Based Configuration

Enable Self-Service Notifications

Team Namespace Configuration

Advanced Features

Conditional Subscriptions

Custom Annotations in Triggers

Grafana Annotations

Troubleshooting

Check Notification Controller Logs

Test Notifications

Common Issues

Best Practices