Grafana: Annotate Grafana dashboards with deployments

· CC BY-SA

Updates in your application stack can cause deviations in your metric graphs (for better or worse). Especially when you deploy often, it’s not always easy to quickly link cause and effect. Using annotations in Grafana, we can correlate changes in our graphs with deployments from our Continuous Integration pipeline.

To get started, we need Grafana 4.6 or above to be installed for the native annotation store. If you’re using an older version, you can use the same technique and store your annotations in InfluxDB, for example.

For this tutorial, I’m using Docker to get the latest Grafana release running:

docker run -d --name=grafana -p 3000:3000 grafana/grafana

Setup a new annotation in Grafana

Open Grafana in your browser on http://localhost:3000 (default login is admin:admin), create a new dashboard and select Dashboard > Annotations > New.

Set the name to »Deployments« and check that the data source is set to »Grafana«. If you already use different annotations in your setup, make sure to set a filter by tag, so only the deployment annotations are shown:

Grafana: Screenshot showing the new annotation setup

Create an API key

For external sources can write to Grafana, we need to create an API key first. Head to your terminal and run:

curl -XPOST 127.0.0.1:3000/api/auth/keys \
  -H "Content-Type: application/json" \
  --user "admin:admin" \
  --data @- << EOF
  {
    "name": "Deployment Annotations",
    "role": "Editor"
  }
EOF

This will give you a JSON response with a long key field. Store this key, as we’ll need it for all further API calls.

Add annotation after deployment

The most interesting fields for an annotation are text and tags. You can look up further fields in the HTTP API documentation. For text, you could include the name of the service deployed, the git commit hash and links to your CI system or GitHub. Set at least a tag called »deployment« for our filter, you can add tags for the environment, tier or data center.

At the end of each successful deployment send a HTTP request to Grafana. Make sure the JSON content is properly escaped:

curl -XPOST 127.0.0.1:3000/api/annotations \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer eyJrIj…oxfQ==" \
  --data @- << EOF
  {
    "text": "Deployed service 'backend-service'\n\n
      <a href=\"https://github.com/org/backend-service/commit/8ba7fb80\">GitHub (8ba7fb80)</a>\n
      <a href=\"http://ci.example.com/jenkins/job/backend-service/32/\">Jenkins (#32)</a>",
    "tags": [
      "deployment",
      "env:production",
      "tier:backend"
    ]
  }
EOF

Check the environment variables available in your CI system, they contain a lot of useful information to construct helpful links you may want to add.

This is what the final result looks like in Grafana:

Grafana: Screenshot of the resulting annotation overlay window

UPDATE: At the time of writing, Grafana doesn’t limit the retention period for annotations. So if you plan to push many annotations, you may need to clean up Grafana’s internal SQLite storage from time to time or switch to InfluxDB as data source, which has a configurable retention policy.
— Thanks to Thomas for asking about it.

Likes

Reposts