Change Tracking in Splunk

Are you tracking changes in your Splunk deployment? Most people don't, unless they can justify having a custom (heavy!) process using multiple environments where changes to production are rolled using some automation (e.g. Jenkins) after going through rigorous tests and formal peer reviews.

Instead you might want the convenience of using Splunk's natural change mechanisms (clicking in the web UI, applying search head and indexer cluster bundles, using the deployment server, manually editing conf files, etc.) in a unique (production!) environment while still tracking (and where possible controlling) changes.

In 2018 I did a talk at Splunk.conf showing how we track our changes. I shared the talking with Olivier Lauret (co-founder of Octamis). It discusses the behind-the-scenes of Splunk configuration, the pitfalls around generating concise meaningful diffs, how to version glass tables (not very relevant now!), what to look for to predict the impact of deployments, and how to identify conflicts between your custom configuration and changes due to upgrades. The deployment considered comprises a multi-site indexer cluster and a search head cluster with enterprise security.

The slides are here: https://static.rainfocus.com/splunk/splunkconf18/sess/1523520863163001AB1A/finalPDF/FN1649_FoundationsPlatform_VasseurLauret_HowWeTrackAll_Final_1536360085284001dr1I_mod_1540941032938001AnQT.pdf

And the talk is here: https://conf.splunk.com/files/2018/recordings/how-we-track-all-fn1649.mp4