Conf Manager
Updated: May 17
This is the documentation for the Conf Manager app on splunkbase. This app allows you to search your knowledge objects and track their changes with colourful diffs, all in Splunk!
Please read this whole page before using the app.
Search your knowledge objects
Conf Manager allows you to search your knowledge objects (searches, props, dashboards, etc) much faster and in much more powerful ways than the "settings" menu in splunk.
Ever wondered where a specific macro is used, be it in searches, dashboards, panels, other macros...?
Ever wondered how many searches (or correlation searches) are using datamodel XYZ?
Ever wished that you'd remember which dashboard does that super cool complicated thing you want to do again now so that you can use it as inspiration / copy-and-paste?
Ever getting confused about the search-time configuration sequence and wished you could see all field extractions, aliases, calculations etc applied to a specific sourcetype or by a specific app in the right order?
Ever want to use grep in your etc folder?
Frustrated you can't see the eval code for calculated fields in a data model when not an admin?
And so on.
Note that you'll get the most out of this if you know how to write regular expressions.
Change tracking / versioning of all knowledge objects
On top of the above, this app allows you to track all changes to all your knowledge objects. You can see the changes to any knowledge object between any arbitrary dates in colourful diffs.
The changes are for the actual knowledge object as seen by the search head, not for the underlying .conf configuration files. This means for instance that changing the permission of a dashboard from private to shared doesn't look like the private dashboard vanished and an entirely new unrelated one appeared.
Change information is kept in the KV store. Nothing is indexed by splunk, so no consumption of your license.
No need for any third party software (such as git), and no need for access to the backend. This means this works just as well in Splunk Cloud as in Splunk Enterprise.
Is this too good to be true?
Kind of. A proper implementation should really be done by splunk themselves (see this splunk idea) so our hacky approach has limitations.
Proper source control would show who made the change and ideally how (upgrade, GUI, REST, manually, etc), and potentially even allow for a peer review to happen *before* the change actually takes place in production. We are not aware of a way to do any of that.
The changes are polled every 15 minutes, so you might have to wait up to that much to see the latest changes.
Performance considerations
The configuration is tracked via a series of saved searches calling REST commands which although optimised will add some load to your splunk instance. The current state of K.O.s is stored in csv files and the history of changes in KV store collections. This could potentially affect your splunk instance if it grows too big. Future versions of this app will include a health dashboard to help monitor all of this, but it's your responsibility to check that you are happy with the health of your splunk instance. Note that the csv files are excluded from the knowledge bundle, so everything should stay on the search head.
Although our experience has been good so far, we cannot guarantee that one of these searches will never possibly fail occasionally in a weird way that might track the current state of the K.O.s but somehow fail to record changes. You should not base important decisions solely on what this app tells you or rely on it for compliance with some legal obligations. Use at your own risks.
That said we do hope you'll find it very useful!
Security considerations
Anyone with access to the app or its lookups or KV store collections will see the current or past configuration of knowledge objects. This includes private knowledge objects, and might include sensitive information.
We set up the app to be initially only readable by admins, but if you have a very sensitive situation you should not trust our understanding of how this works or our ability to implement it correctly. Proceed with caution.
Our laidback approach to confidentiality is because we mostly see this app as a collaboration tool for small tight-knit teams who trust each other and cannot afford full-blown CI/CD systems (continuous integration and continuous deployment).
I'm just curious, how does it work?
The gist of it is quite simple.
For each type of knowledge object, there is a scheduled saved search that:
queries the relevant REST endpoint to get all the current info on all of these K.O.s
stores that in a lookup file, ready to be queried by the various search dashboards in the app
For each type of knowledge object, there is also an alternative scheduled saved search that does the same but also tracks changes. You decide which one you want to enable (or none if you don't care about this type of K.O.).
The alternative search does the same as above but also creates a fork of itself with appendpipe where it queries the lookup file mentioned above to identify what has changed since the last time it ran and saves any change to a KV store collection.
Then the Track dashboard inputs the content of the KV store, allows you to filter it in many ways, and uses the utilities in the GV-Utils app to generate diffs in HTML and then render them whithin the dashboard.
Pretty simple in theory! But in reality the devil is in the details.
There's no ID that allows to track a K.O. through its life. Two K.O. with the same title can coexist in two different apps, or in the same app but owned by two different users, or in the same app AND owned by the same user but one shared and one private. It is however not possible for two K.O. to have the same title, in the same app, owned by the same user with the same sharing status.
So the ID of a K.O. becomes the combination: app-title-owner-sharing.
The problem with that is as soon as a private K.O. is shared for instance, it'll look like it disappeared and a brand new item appeared when the truth is the same item remained and had only one change (the value for sharing). Same thing for K.O. that have moved apps or had their owner reassigned. Our implementation goes to great length to make this less of a problem and we're hoping to have time to explain how in a blog article at some point.
Is that all?
The app also includes a dashboard that allows you to compare two different knowledge objects of the same type. This could be useful for instance if you clone a production version, work on the clone, and then want to show exactly what you changed compared to the production version.
I'm hoping to keep developing this app and add new features.
For instance, the next version will include a dashboard that compares the *results* of 2 different searches, so that you can not only compare the SPL for two versions of a search but also how it affects their results.
Message me@gabrielvasseur.com if there are features you would like to see, or any ideas for improvements.
Installation
Install both Conf Manager and GV-Utils in any order. Make sure to restart the search head .
Configure
IMPORTANT! The app won't do anything if you don't do this.
First step is to ensure all K.O.s have a recorded current state, so that they don't all appear entirely new the first time changes are tracked. Navigate to the Conf Manager app, then to its Setup dashboard. Once loaded, click on the setup button. Wait for all the searches to finish. If outputlookup is a risky command, click on each red triangle and run the query anyway. The IR fields such will fail if you don't have Enterprise Security, you can safely ignore that.
Go to Splunk's "settings" then "searches, reports and alerts". You may have to select "All" for owner, until you see the various saved searches. There should be 14 "CM - Update CM_* - Lookup Gen" searches and 14 "CM - Update and track CM_* - Lookup Gen" searches.
For each knowledge object type (macros, saved searches, props, etc), you need to decide what you want:
if you just want to be able to search these K.O.s, make sure the "Update CM_*" search is scheduled (it's already enabled)
if you just want to be able to search these K.O.s AND also track their changes, make sure the "Update and track CM_*" search is enabled (it's already scheduled)
if you don't care about these K.O.s at all, don't enable either (not recommended, except for IR fields if you don't have Enterprise Security)
never ever have both "update" and "update and track" searches scheduled for the same type of K.O.s
If you have a lot of searches in your splunk instance, the settings menu will be painful. See this splunk idea. Just take your time.
Troubleshooting and Support
Make sure you read all the documentation before reaching out to me@gabrielvasseur.com