By: Mandi Walls (@lnxchk)
Edited by: Joe Block (@curiousbiped)
Keeping track of all the data generated by a distributed ecosystem is a daunting task. When something goes wrong, or a service isn’t behaving properly, tracking down the culprit and getting the right folks enabled to fix it is also challenging. PagerDuty can help you with these challenges.
The PagerDuty platform integrates with over 600 other components to gather data, add context, and process automation. Under the hood of all of these integrations is the PagerDuty API, ready to help you programmatically interact with your PagerDuty account.
What’s Exposed Via the API
The PagerDuty API provides access to all the structural objects in your PagerDuty account - users, teams, services, escalation policies, etc - and also to the data objects including incidents, events, and change events.
For objects like users, teams, escalation policies, schedules, and services, you may find using the PagerDuty Terraform Provider will help you maintain the state of your account more efficiently without using the API directly.
The other object types in PagerDuty are more useful when we can send them anytime from anywhere, including via the API from our own code. Let’s take a look at three of them: incidents, events, and change events. If you’d like a copy of the code for these examples, you can find them on Github.
API Basics
To write new information into PagerDuty via the API, you'll need some authorization. You can use OAuth, or create an API key. There are account-level and user-level API keys available. You'll use the account-level keys for the rest of the examples here and keep things simple.
To create a key in your PagerDuty app, you'll need Admin, Global Admin, or Account Owner access to your account. More on that here.
In PagerDuty, navigate to Integrations and then chose API Access Keys. Create a new key, give it a description, and save it somewhere safe. The keys are strings that look like y_NbAkKc66ryYTWUXYEu.
Now you’re ready to generate some incidents! These examples use curl, but there are a number of client libraries for the API as well.
Incidents
Incidents are probably what you’re most familiar with in PagerDuty - they represent a problem or issue that needs to be addressed and resolved. Sometimes this includes alerting a human responder. Many of the integrations in the PagerDuty ecosystem generate incidents from other systems and services to send to PagerDuty.
In PagerDuty, incidents are assigned explicitly to services in your account, so an incoming incident will register with only that service. If your database has too many long-running queries, you want an incident to be assigned to the PagerDuty service representing that database so responders have all the correct context to fix the issue.
If you have a service that doesn’t have an integration out of the box, you can still get information from that service into PagerDuty via the API, and you don’t need anything special to do it. You can send an incident to the API via a curl request to the https://api.pagerduty.com/incidents endpoint.
There are three required headers for these requests, Accept, Content-Type and From, which needs to be an email address associated with your account, for attribution of the incident. Setting up the request will look something like:
curl -X POST --header 'Content-Type: application-json' \
--url https://api.pagerduty.com/incidents \
--header 'Accept: application/vnd.pagerduty+json;version=2' \
--header 'Authorization: Token token=y_NbAkKc66ryYTWUXYEu' \
--header 'From: [email protected]' \
Now you need the information bits of the incident. These will be passed as --data in the curl request. There are just a few required pieces to set up the format and a number of optional pieces that help add context to the incident.
The most important piece you'll need is the service ID. Every object in the PagerDuty platform has a unique identifier. You can find the ID of a service in its URL in the UI. It will be something like https://myaccount.pagerduty.com/service-directory/SERVICEID.
Now you can create the rest of the message with JSON:
curl -X POST --header 'Content-Type: application/json' \
--url https://api.pagerduty.com/incidents \
--header 'Accept: application/vnd.pagerduty+json;version=2' \
--header 'Authorization: Token token=y_NbAkKc66ryYTWUXYEu' \
--header 'From: [email protected]' \
--data '{
"incident": {
"type": "incident",
"title": "Too many blocked requests",
"service": {
"id": "PWIXJZS",
"summary": null,
"type": "service_reference",
"self": null,
"html_url": null
},
"body": {
"type": "incident_body",
"details": "The service queue is full. Requests are no longer being fulfilled."
}
}
}'
When you run this curl command, it will generate a new incident on the service PWIXJZS with the title "To many blocked requests", along with some context in the "body" of the data to help our responders. You can add diagnostics or other information here to help your team fix whatever is wrong.
What if there is information being generated that might not need an immediate response? Instead of an incident, you can create an event.
Events
Events are non-alerting items sent to PagerDuty. They can be processed via Event Rules to help create context on incidents or provide information about the behavior of your services. They utilize the PagerDuty Common Event Format to make processing and collating more effective.
Events are registered to a particular routing_key via an integration on a particular service in your PagerDuty account. In your PagerDuty account, select a service you'd like to send events to, or create new one to practice with. On the page for that service, select the Integrations tab and Add an Integration. For this integration, select "Events API V2" and click Add. You'll have a new integration on your service page. Click the gear icon, and copy the Integration Key. For the full walkthrough of this setup, see the docs.
The next step is to set up the event. The request is a little different from the incident request - the url is different, the From: header is not required, and the authorization is completely handled in the routing_key instead of using an API token.
The content of the request is more structured, based on the Common Event Format, so that you can create event rules and take actions if necessary based on what the events contain.
curl --request POST \
--url https://events.pagerduty.com/v2/enqueue \
--header 'Content-Type: application/json' \
--data '{
"payload": {
"summary": "DISK at 99% on machine prod-datapipe03.example.com",
"timestamp": "2021-11-17T08:42:58.315+0000",
"severity": "critical",
"source": "prod-datapipe03.example.com",
"component": "mysql",
"group": "prod-datapipe",
"class": "disk",
"custom_details": {
"free space": "1%",
"ping time": "1500ms",
"load avg": 0.75
}
},
“event_action”: “trigger”,
"routing_key": "e93facc04764012d7bfb002500d5d1a6"
}'
Change Events
A third type of contextual data you can send to the API is a Change Event. Change events are non-alerting, and help add context to a service. They are informational data about what's changing in your environment, and while they don't generate an incident, they can inform responders about other activities in the system that might have contributed to a running incident. Change events might come from build and deploy services, infrastructure as code, security updates, or other places that change is generated in your environment.
These events have a similar basic structure to the general events, and the setup with the routing_key is the same, as you can see in the below example. The custom_details can contain anything you want, like the build number, a link to the build report, or the list of objects that were changed during an Infrastructure as Code execution.
Change events have a time horizon. They expire after 90 days in the system, so you aren't looking at old context based on past changes.
curl --request POST \
--url https://events.pagerduty.com/v2/change/enqueue \
--header 'Content-Type: application/json' \
--data '{
"routing_key": "737ea619db564d41bd9824063e1f6b08",
"payload": {
"summary": "Build Success: Increase snapshot create timeout to 30 seconds",
"timestamp": "2021-11-17T09:42:58.315+0000",
"source": "prod-build-agent-i-0b148d1040d565540",
"custom_details": {
"build_state": "passed",
"build_number": "220",
"run_time": "1236s"
}
}
}'
Adding Notes
One final fun bit of functionality you can leverage in PagerDuty's API is with notes. Notes are short text entries added to the timeline of an incident. In some integrations, like PagerDuty and Slack, notes will be sent to any Slack channel that is configured to receive updates for an impacted service, making them helpful for responders to coordinate and record activity across different teams.
Notes are associated with a specific incident, so when you are creating a note, the url will include the incident ID. Incident IDs are similar to the other object IDs in PagerDuty in that you can find them from the URL of the incident in the UI. They are longer strings than other objects than the service ID in the examples above.
The content of a note can be anything that might be interesting to the timeline of the incident, like commands that have been run, notifications that have been sent, or additional data and links for responders and stakeholders.
curl --request POST \
--url https://api.pagerduty.com/incidents/{id}/notes \
--header 'Accept: application/vnd.pagerduty+json;version=2' \
--header 'Authorization: Token token=y_NbAkKc66ryYTWUXYEu' \
--header 'Content-Type: application/json' \
--header 'From: [email protected]' \
--data '{
"note": {
"content": "Firefighters are on the scene."
}
}'
Responders utilizing the UI will see notes in a widget on the incident pag.
Next Steps
Using the API to create tooling where integrations don't yet exist, or for internally-developed services, can help your team stay on top of all the moving parts of your ecosystem when you have an incident. Learn more about the PagerDuty resources available at https://developer.pagerduty.com/. Join the PagerDuty Community to learn from other folks working in PagerDuty, ask questions, and get answers.
Posts
