Conf Manager

This is the documentation for the Conf Manager app on splunkbase. This app allows you to search your knowledge objects and track their changes with colourful diffs, all in Splunk!

Please read this whole page before using the app, but here's a quick guide if you're in a hurry:

• The first part of this page is designed to showcase the app and help you decide if you want to give it a go.

• The second part is a quick sitemap-like run of the app, just so you know what's there and where to find it. Each dashboard is hopefully self-explanatory.

• The last part is the installation instructions, not to be skipped!

*** Part 1: the pitch ***

Search your knowledge objects

Conf Manager allows you to search your knowledge objects (searches, props, dashboards, etc) much faster and in much more powerful ways than the "settings" menu in splunk.

• Ever wondered where a specific macro is used, be it in searches, dashboards, panels, other macros...?

• Ever wondered how many searches (or correlation searches) are using datamodel XYZ?

• Ever wanted to use grep in your etc folder?

This is what the Universal KO Search dashboard is for. Its drilldowns allow you to easily navigate to the definitions of the various objects in splunk.

There are also dashboards dedicated to specific K.O. types:

• Frustrated you can't see the eval code for calculated fields in a data model when not an admin? --> there's a dashboard for data models

• Ever wished that you'd remember which dashboard does that super cool complicated thing you want to do again now so that you can use it as inspiration / copy-and-paste?

• Ever getting confused about the search-time configuration sequence and wished you could see all field extractions, aliases, calculations etc applied to a specific sourcetype or by a specific app in the right order?

Note that you'll get the most out of this if you know how to write regular expressions.

Change tracking / versioning of all knowledge objects

On top of the above, this app allows you to track all changes to all your knowledge objects. You can see the changes to any knowledge object between any arbitrary dates in colourful diffs.

The changes are for the actual knowledge object as seen by the search head, not for the underlying .conf configuration files. This means for instance that changing the permission of a dashboard from private to shared doesn't look like the private dashboard vanished and an entirely new unrelated one appeared.

Change information is kept in the KV store. Nothing is indexed by splunk, so no consumption of your license.

No need for any third party software (such as git), and no need for access to the backend. This means this works just as well in Splunk Cloud as in Splunk Enterprise.

Is this too good to be true?

Kind of. A proper implementation should really be done by splunk themselves (see this splunk idea) so our hacky approach has limitations.

Proper source control would show who made the change and ideally how (upgrade, GUI, REST, manually, etc), and potentially even allow for a peer review to happen *before* the change actually takes place in production. We are not aware of a way to do any of that.

The changes are polled every 15 minutes, so you might have to wait up to that much to see the latest changes. Use the Quick Refresh dashboard if you can't wait (since version 1.4).

Performance considerations

The configuration is tracked via a series of saved searches calling REST commands which although optimised will add some load to your splunk instance. The current state of K.O.s is stored in csv files and the history of changes in KV store collections. This could potentially affect your splunk instance if it grows too big. Versions 1.4 of this app include a Health dashboard to help monitor all of this, but it's your responsibility to check that you are happy with the health of your splunk instance. Note that the csv files are excluded from the knowledge bundle, so everything should stay on the search head.

Although our experience has been good so far, we cannot guarantee that one of these searches will never possibly fail occasionally in a weird way that might track the current state of the K.O.s but somehow fail to record changes. You should not base important decisions solely on what this app tells you or rely on it for compliance with some legal obligations. Use at your own risks.

That said we do hope you'll find it very useful, especially if you don't have a full-blown CI/CD system (continuous integration and continuous deployment).

Security considerations

Anyone with access to the app or its lookups or KV store collections will see the current or past configuration of knowledge objects. This includes private knowledge objects, and might include sensitive information.

We set up the app to be initially only readable by admins, but if you have a very sensitive situation you should not trust our understanding of how this works or our ability to implement it correctly. Proceed with caution.

Our laidback approach to confidentiality is because we mostly see this app as a collaboration tool for small tight-knit teams who trust each other and cannot afford full-blown CI/CD systems (continuous integration and continuous deployment).

I'm just curious, how does it work?

The gist of it is quite simple.

For each type of knowledge object, there is a scheduled saved search that:

• queries the relevant REST endpoint to get all the current info on all of these K.O.s

• stores that in a lookup file, ready to be queried by the various search dashboards in the app

For each type of knowledge object, there is also an alternative scheduled saved search that does the same but also tracks changes. You decide which one you want to enable (or none if you don't care about this type of K.O.).

The alternative search does the same as above but also creates a fork of itself with appendpipe where it queries the lookup file mentioned above to identify what has changed since the last time it ran and saves any change to a KV store collection.

Then the Track dashboard inputs the content of the KV store, allows you to filter it in many ways, and uses the utilities in the GV-Utils app to generate diffs in HTML and then render them whithin the dashboard.

Pretty simple in theory! But in reality the devil is in the details.

There's no ID that allows to track a K.O. through its life. Two K.O. with the same title can coexist in two different apps, or in the same app but owned by two different users, or in the same app AND owned by the same user but one shared and one private. It is however not possible for two K.O. to have the same title, in the same app, owned by the same user with the same sharing status.

So the ID of a K.O. becomes the combination: app-title-owner-sharing.

The problem with that is as soon as a private K.O. is shared for instance, it'll look like it disappeared and a brand new item appeared when the truth is the same item remained and had only one change (the value for sharing). Same thing for K.O. that have moved apps or had their owner reassigned. Our implementation goes to great length to make this less of a problem and we're hoping to have time to explain how in a blog article at some point.

Is that all?

The app also includes a dashboard that allows you to compare two different knowledge objects of the same type. This could be useful for instance if you clone a production version, work on the clone, and then want to show exactly what you changed compared to the production version.

Version 1.4+ includes a dashboard that compares the *results* of 2 different searches, so that you can not only compare the SPL for two versions of a search but also how it affects their results.

Version 1.5+ includes a few new features:

Similar to the highlight SPL tool, there is now a highlight XML tool.

There is now a macro explorer tool that allow you to selectively expand macros in any arbitrary SPL to the depth you desire. It makes understanding strange SPL with nested macros much easier and a lot less overwhelming than using Splunk's search bar ctrl-shit-E.

There is now a "dashboard use" dashboard where you can check who is using which dashboard. That's useful if you're assessing the impact of a change and want to discuss it with users.

There is now a "snapshot" feature where you can define a set of K.O.s related to a particular function, and compare them between independent search heads. If the K.O.s are in the same app you could use the Remote Config Manager app (https://splunkbase.splunk.com/app/6277). In our case we wanted to reproduce some of Enterprise Security's identities and assets framework on the adhoc search head, which doesn't have any of the ES apps. This is where taking a snapshot on the ES SH and comparing to it on the adhoc SH is useful.

  I'm hoping to keep developing this app and add new features.

Message me@gabrielvasseur.com if there are features you would like to see, or any ideas for improvements.

*** Part 2: Site map of the app ***

All dashboards are accessible from the app menu.

• Universal K.O. Search - quickly search all your knowledge objects with regular expressions. Great for finding where something is used, including in adhoc searches. Drilldowns go straight to the object's definition.

• Search K.O.s > Dashboards - search the XML of your dashboards. Great for finding that one dashboard where you did something cool and complicated but you just can't remember where.

• Search K.O.s > Data Models - explore data model definitions. Calculated and/or inherited fields are clearly displayed.

• Search K.O.s > Lookups & Collections - search lookup tables, definitions and KV store collections. Sadly it's not possible to search inside lookup tables or KV stores, only their definitions.

• Search K.O.s > Macros - search macro definitions.

• Search K.O.s > Saved Searches - search your saved searches. Great for finding all the correlation searches that use a specific data model, etc.

• Search K.O.s > Search-time configuration - search and display fields extractions, aliases, and other search-time configuration in a very user-friendly manner. Great for TA work.

• Search K.O.s > Workflow Actions - search workflow actions per label, destination, field name or name. Much more user-friendly than splunk's settings version.

• Usage > Historical Command Search - search splunk's internal logs for searches that have run recently, great for finding that search you were working on yesterday but didn't save.

• Usage > Dashboards Use - search splunk's internal logs for which users are using which dashboards, great for assessing the impact of some change you want to make and who to talk to about it. 

• Track - track changes to objects over time and see diffs.

• More > Audit - a bit like the Track dashboard but shows each change event individually without trying to aggregate the various changes over the time window together

• More > Compare - compare two knowledge objects together. Great if you cloned the production version and want someone to peer review your changes before you overwrite the production version with the WIP version.

• Tools > Highlight SPL - allows you to syntax-highlight any arbitrary SPL, ready to be copy-pasted in an email or webpage either as formatted text or raw HTML. Great for blogging or documenting. This is how I wrote this beautiful article.

• Tools > Highlight XML - same as previous tool but for XML.

• Tools > Diff Anything - allows you to diff any 2 arbitrary blobs of text.

• Tools > Diff Search Results - runs two arbitrary searches and diffs the results. Great for double checking tweaks did not break a search, and to find minor differences in the results. Doesn't work so well if the number of results differs between the two searches.

• Tools > Macro Explorer - Allows you to paste any SPL and pick and choose which macros to expand, great for getting to know strange code using nested macros.

• Snapshots > Take Snapshot - Saves the defintions of a list of K.O.s you specify in a summary index, to be consumed by the next dashboard.

• Snapshots > Compare to Snapshot - Compares local K.O.s to a snapshot taken earlier and/or on a different search head.

• Quick Refresh - the app keeps track of knowledge objects via searches that are scheduled every 15 minutes, and as such won't know about very recent changes. This is where you can run these searches on an adhoc basis.

• Health > Health - check the app's configuration health, impact on storage and performance

• Health > Health - KV Store Explorer - Allows you to explore what is in the KV stores the app is collecting, in case they are becoming big and you want to trim them.

• Setup - required the first time you install the app, to avoid a diff explosion.

*** Part 3: Install & troubleshoot ***

Installation

Install both Conf Manager and GV-Utils in any order. Make sure to restart the search head .

Configure

IMPORTANT! The app won't do anything if you don't do this.

First step is to ensure all K.O.s have a recorded current state, so that they don't all appear entirely new the first time changes are tracked. Navigate to the Conf Manager app, then to its Setup dashboard. Once loaded, click on the setup button. Wait for all the searches to finish. If outputlookup is a risky command, click on each red triangle and run the query anyway. The IR fields such will fail if you don't have Enterprise Security, you can safely ignore that.

Go to Splunk's "settings" then "searches, reports and alerts". You may have to select "All" for owner, until you see the various saved searches. There should be 14 "CM - Update CM_* - Lookup Gen" searches and 14 "CM - Update and track CM_* - Lookup Gen" searches.

For each knowledge object type (macros, saved searches, props, etc), you need to decide what you want:

• if you just want to be able to search these K.O.s, make sure the "Update CM_*" search is scheduled (it's already enabled)

• if you just want to be able to search these K.O.s AND also track their changes, make sure the "Update and track CM_*" search is enabled (it's already scheduled)

• if you don't care about these K.O.s at all, don't enable either (not recommended, except for IR fields if you don't have Enterprise Security)

• never ever have both "update" and "update and track" searches scheduled for the same type of K.O.s

If you have a lot of searches in your splunk instance, the settings menu will be painful. See this splunk idea. Just take your time.

Troubleshooting and Support

Make sure you read all the documentation before reaching out to me@gabrielvasseur.com