Annotate dashboards with deployments

Updates in your application stack can cause deviations in your metric graphs (for better or worse). Especially when you deploy often, itʼs not always easy to quickly link cause and effect. Using annotations in Grafana, we can correlate changes in our graphs with deployments from our Continuous Deployment pipeline.

To get started, we need Grafana 4.6 or above to be installed for the native annotation store. If youʼre using an older version, you can use the same technique and store your annotations in InfluxDB, for example.

For this tutorial, Iʼm using Docker to get the latest Grafana release running:

docker run -d --name=grafana -p 3000:3000 grafana/grafana

Setup a new annotation in Grafana

Open Grafana in your browser on http://localhost:3000 (default login is admin:admin), create a new dashboard and select Dashboard > Annotations > New.

Set the name to »Deployments« and check that the data source is set to »Grafana«. If you already use different annotations in your setup, make sure to set a filter by tag, so only the deployment annotations are shown:

Create an API key

For external sources can write to Grafana, we need to create an API key first. Head to your terminal and run:

curl -XPOST 127.0.0.1:3000/api/auth/keys \
  -H "Content-Type: application/json" \
  --user "admin:admin" \
  --data @- << EOF
  {
    "name": "Deployment Annotations",
    "role": "Editor"
  }
EOF

This will give you a JSON response with a long key field. Store this key, as weʼll need it for all further API calls.

Add annotation after deployment

The most interesting fields for an annotation are text and tags. You can look up further fields in the HTTP API documentation. For text, you could include the name of the service deployed, the git commit hash and links to your CI system or GitHub. Set at least a tag called »deployment« for our filter, you can add tags for the environment, tier or data center as well.

At the end of each successful deployment send a HTTP request to Grafana. Make sure the JSON content is properly escaped:

curl -XPOST 127.0.0.1:3000/api/annotations \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer <your auth key from before>" \
  --data @- << EOF
  {
    "text": "Deployed service 'backend-service'\n\n
      <a href=\"https://git.example.org/backend-service/commit/8ba7fb80\">Git (8ba7fb80)</a>\n
      <a href=\"https://ci.example.org/jenkins/job/backend-service/32/\">Jenkins (#32)</a>",
    "tags": [
      "deployment",
      "env:production",
      "tier:backend"
    ]
  }
EOF

Check the environment variables available in your CI system, they contain a lot of useful information to construct helpful links you may want to add.

This is what the final result looks like in Grafana: