Test your correlation searches end-to-end with Morning Checks
Updated: Feb 28, 2022
I did a talk at Splunk .conf21 about how to maintain correlation searches: pdf/mp4. One of the topics is morning checks. Basically you need end-to-end tests for your correlations. Here are the 3 steps:
Automate harmless ways to trigger your correlations. Don’t fake logs, just do something the same as the real deal. E.g. for malware use the Eicar string.
Make sure your rules handle it: mark as informational, reduce risk score, suppress the notable, etc.
Have a morning checks checks dashboard: something that checks for the appearance of the risk score and/or notable (even suppressed).
If a morning check passes, it’s a full end-to-end test of your correlation, from behaviour all the way to alert.
I go into a lot more details in my talk pdf/mp4.
If you want to get the source code of what you see in the talk, it’s all in the ES Choreographer app in splunkbase.