Prometheus: Fill in Data for New Recording Rules

Overview

  • wanting to test the new recording rule right away instead of wait for it to populate
  • wanting to create an alert using the recording rule right away
  • making a dashboard using the new recording rule data

Example Usage

  • Have a running Prometheus server with flag storage.tsdb.allow-overlapping-block enabled.
  • Create a new recording rule
  1. Run the command promtool tsdb create-blocks-from rulesto fill in past data for the new rule. Provide the Prometheus server address and port with the url flag.
  2. Validate the output of the command, make sure it contains the correct data. The output of the promtool tsdb create-blocks-from rules are blocks of data. By default they are created in 2 hour chunks.
  3. Manually move the blocks over to the running Prometheus server at the same location of storage.tsdb.path.
  4. Wait briefly for the next compaction to hit, then query for the new rule and confirm past data is there. Warning: after compaction runs, the data cannot be removed or reverted.
$ cat prometheus.yml
global:
scrape_configs:
- job_name: 'prometheus'
static_configs:
- targets: ['localhost:9090']
- job_name: 'node-exporter'
static_configs:
- targets: ['localhost:9100']
$ prometheus --storage.tsdb.allow-overlapping-blocks
example recording rule file
see recording rules at http://localhost:9090/rules
verify that recording rule data is being collected, tiny amount of data at starts 13:04
verify that recording rule data is being collected, tiny amount of data at starts 13:04
// the time to start filling in from as unix timestamp
$ date --date="1 hour ago" +%s
1617563353
// end time, the first timestamp with no series data
// for the new recording rule
$ date --date="2021-04-04 13:04:06" +%s
1617566646
$ promtool tsdb create-blocks-from rules \
--start 1617563353 --end 1617566646 \
--url http://localhost:9090 \
example-rule.yml
// by default the output is written to ./data directory
$ ls data/
01F2F87M7M94Z0B0WY18ER5VKA 01F2F87M9HKVVRGENWZMR5KYQ1 01F2F87MAR7YEX8NZ0WA77YNZ4 01F2F87MB6A582X8A4A9H36VHB 01F2F87MBW23QN3M8NWNF2Y4HQ 01F2F87MCMNBZTPMRRW9GW4XN1
output of command: promtool tsdb list -r data/
$ mv data/ $PROM_DATA_DIR
level=info ts=2021-04-04T20:23:51.035Z caller=compact.go:686 component=tsdb msg="Found overlapping blocks during compaction" ulid=01F2F8WMZ71M4VVZA3JJWEH3DNlevel=info ts=2021-04-04T20:23:51.043Z caller=compact.go:448 component=tsdb msg="compact blocks" count=6 mint=1617563241870 maxt=1617566601871 ulid=01F2F8WMZ71M4VVZA3JJWEH3DN sources="[01F2F87M7M94Z0B0WY18ER5VKA 01F2F87MAR7YEX8NZ0WA77YNZ4 01F2F87MBW23QN3M8NWNF2Y4HQ 01F2F87M9HKVVRGENWZMR5KYQ1 01F2F87MB6A582X8A4A9H36VHB 01F2F87MCMNBZTPMRRW9GW4XN1]" duration=28.268502ms
Backfilled recording rule data from 12:00 to 13:04
Backfilled recording rule data from 12:00 to 13:04
Backfilled recording rule
Running recording rule query in Prometheus graph to confirm data matches backfiller data.

Under the Hood

  • A Rule Manager is created to parse the recording rule files. This is the same code that Prometheus uses to process recording rules.
  • Requests are made to the Prometheus API QueryRange endpoint using the Prometheus Go client library (ref). The QueryRangeAPI evaluates the recording rule expression against existing time series data.
  • The response returned from the API contains samples with the timestamps and values for the recording rule. This is used to create new series that are written to tsdb blocks.

Future Work

Happy Ending

--

--

--

Live simply. Program stuff.

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Daily links of Fernand0 — Enlaces diarios de Fernand0 — Issue #310

How Hackathons Could Improve Your Hiring

🚨 Airdrop: Thunder BNB $TTBNB 🔥 Reward: 10 BILLION $TTBNB 🤑 Refer: 1 $TTBNB($15) 📊 Market…

Python input() and raw_input() function.

Analyzing streaming cab data and create real time dashboards using Azure Services

Python 3.8 commands as a solved problem

The One Regret of My Software Engineering Career

The art of problem-solving in programming industry

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Jessica G

Jessica G

Live simply. Program stuff.

More from Medium

Modern File Transfer Solution on Azure — SFTP

Traffic Management Problems

The Curious Case of the declarative Azure Bicep template “not declarative”…

In anticipation of the KubeCon + CloudNativeCon conference that will take place in Valencia, Spain…