 Hi my name is Jessica and I am a software engineer and a contributor to Prometheus and this is my lightning talk on how to fill in missing data for recording rules. So many moons ago back in 2013 an issue was created that made this request to have the ability to persist data for evaluating recording rules in the past and I'm happy to say that eight years later we finally have a pull request that got merged recently that implements this feature and this talk is about how to make use of that feature to fill in recording rules from the past and so just a tiny bit of background what is a recording rule in the Prometheus official documentation it shows here has some information on recording rules that they are essentially ways to precompute expensive queries and save the result so that you don't have to evaluate those expensive queries over and over but rather that data is saved over time. So here's an example of a recording rule that figures out what is the current network traffic per second and you could see the query expression example here that gets run. So the problem with recording rules is that the data only exists from creation time on when you create a new recording rule there is no past data for that rule so we're going to use this example through the demo of making a rule that tracks the network traffic per second so here's a config file for recording rule example and if you so when I created this recording rule then you can go look in Prometheus you can look on the dashboard so on the left hand side this is the query that was executed and so you could see for the past hour there is actually data time series data that exists for that however if I just put in the name of the recording rule there's no data here essentially just because it's brand new so it doesn't have any time series data so the goal of this is to fill in the past data right here so that these two match and now we have the capability of being able to do that with prompt tool so I added this feature into prompt tool this is the sub command tstb create blocks from rules here's all the options so if you do run that like so you need to provide a start and an end time so in this example there's an hour window that we just looked at on the previous chart that hour window is what we're going to fill in and then the url to the Prometheus API is passed in and then the output of this sub command are blocks blocks of time series data for this recording rule so providing a location of the output directory this is where all of the blocks will be so execute this command you can see some logs it says blah blah blah we're running this rule processing it and then afterwards you can ls into the output directory and you can see that there's one block created to inspect that block even further there's another sub command you can kind of see more data about it so you can see it it's about an hour here and there's 70 samples which the evaluation interval is 60 seconds so this is about what you would expect in order to make use of this data in the Prometheus instance these blocks once they're validated they need to be manually moved over to the data directory of the Prometheus instance so I did that once I did that you can look in the logs and you could see Prometheus now sees that there's some overlapping blocks so when the next compaction hits it's going to merge those blocks together and once that happens then the data is available back in the Prometheus graph so back over here this is the first chart we looked at you could see the actual query is right here and we have all that data and then the recording rules over here now now we actually have all this past data for the recording rule and you can confirm that it looks pretty good and that's that so for future work for this I would like to add another sub command that will validate the blocks for you and also move the blocks over to a specified Prometheus data dir just to eliminate those manual steps and then I'm still we need to add still markers I'm going to do that soon and anything else if if anybody uses this feature and has any ideas how to make it better or there's other use cases that are needed or bugs or whatever let me know and then lastly I just wrote a quick blog post that has these kind of the same information from this talk and with a little more detailed links to docs and stuff so check that out if you want and that is it thanks a bunch