I did a talk at Splunk .conf21 about how to maintain correlation searches: pdf/mp4. One of the topics is having a simple peer review system that integrates with Enterprise Security.
The idea of a peer review is to have another set of eyes double check the work. It helps catch issues, raise awareness of techniques, and promotes learning.
Usually peer reviews happen before a change is deployed to production, and that requires change control. If you trust your people and don’t have a fancy development environment, you can get by with change tracking instead of change control. Either way, no solution exists in Splunk out of the box.
You can do your own change tracking, and I would recommend it but it’s not easy.
Emboldened by my success with the KV store for the TODO system, I thought I could make a peer review system in splunk:
the KV store would hold the state of a correlation at it was the last time it was reviewed
comparing the current state with the KV store will inform if any changes has happened
using a custom command allows to generate a diff between the before and after version. The diff can be generated as colourful
a bit of googling showed me how javascript can be used to render HTML in a simple XML dashboard
As with the TODOs, the KV store is used in conjunction with the collect command, which means that a diff can be generated between any arbitrary version of the configuration, past or present, with the only price to pay a bit of patience while the searches run.
Finally I needed a way to do bulk reviews. If anything to prime the pump: the system will not work if there isn’t a past version of the correlation in the KV store, and you are not going to want to create an initial review for your correlations one by one. The bulk review is also useful if an upgrade of ES or the addition of some visualisation app generates a change to every single correlation.
If you want to get the source code of what you see in the talk, it’s all in the ES Choreographer app in splunkbase.