• Gabriel

Dashboarding Best Practices, Tips & Tricks

Updated: Apr 26

Splunk’s “simple XML” dashboards are reasonably simple and straightforward to create, yet they are incredibly versatile and powerful. You can reasonably easily make simple XML dashboards that feel like they should be web applications in their own rights.


Bookmark bookmark bookmark!


This is very basic, but also very helpful: create a bookmark to your dashboard in your bookmark toolbar and edit its destination by removing the ? and anything after that. That way, when you load that bookmark you know for sure that you are loading your dashboard from scratch with no tokens already set from any past interactions with inputs. And this also forces the javascript to be loaded correctly.


Why?


If you’re doing any complicated work with tokens, it can be difficult to test the dashboard properly as the state of the tokens gets confused as you go in and out of edit mode before and after interacting with form inputs.


Also, if using javascript in the dashboard, the JS is not loaded anymore when exiting edit mode.


Debugging tokens


This was mentioned in https://conf.splunk.com/files/2017/slides/tokens-in-splunk-web-framework-use-abuse-and-incantations.pdf as being part of the dashboard example app, but I might also have stolen the code from another splunkbase app. In anycase, I've stolen it and added it to https://bitbucket.org/GabrielVasseur/gv-utils/.


If you want to debug the value of your tokens, add the showtokens script to the first line:

<dashboard script="GV-Utils:showtokens.js">


Load your dashboard and scroll to the bottom: you'll see a breakdown of all your tokens. You can follow what happens there as you interact with your dashboard.


Also works for forms (there is actually no real difference between a form and a dashboard

, that I know of, besides the presence of inputs).


Display the number of results in the table title


This can be extremely helpful for your users. If a table has a lot of results, it's actually difficult to get an idea of how many. This simple trick solves the issue:

<row>
  <panel>
    <table>
      <title>$result_count$ results for Blah Blah</title>
      <search>
        <done>
          <set token="result_count">$job.resultCount$</set>
        </done>
        <query>...

Add custom HTML


You probably know this already, but you can add snippets of HTML in your simple XML dashboard. Just edit the source and add a row like this:

<row>
  <panel>
    <html>
      <h1>Hello world!</h1>
    </html>
  </panel>
<row>

You can then do everything you can do in HTML. This can be useful for documentation and headers if you have a long dashboard.


Add custom CSS


With CSS you can actually drastically change the look of a splunk dashboard.

Add this anywhere (e.g. to the bottom of your dashboard):

<row depends="$never_set$">
  <panel>
    <title>CSS</title>
    <html>
      <style>
        ...
      </style>
    </html>
  </panel>
</row>

Replace the ... with the CSS you want.


Some examples follow. NOTE: most of these I stole from various places on the internet, I’m not a CSS wizard. If the internet doesn’t help, it can be difficult to know how to change what you want, but if you try sometimes with the inspector in your browser and lots of trial and error, you get what you need.


Resize text inputs

Sometimes you need to input a reasonably long amount of text and the regular text input is just too small:

        .input-text {
          width: 800px !important;
        }
        .input-text &gt; input[type="text"] {
          width: 800px !important;
        }

Unfortunately this changes the width of all text inputs! If you want more granular, have a look at GV-Utils version 1.1 and above.


Bold panel titles

If you are nostalgic for the old style.

        .panel-title {
          font-weight: bold !important;
        }
        .dashboard-row .dashboard-panel h2.panel-title {
          font-weight: bold !important;
        }

Add icon to external links

This adds a little icon at the end of any link that opens in a new tab/window:

        a[target="_blank"]::after {
  content: url(data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAoAAAAKCAYAAACNMs+9AAAAQElEQVR42qXKwQkAIAxDUUdxtO6/RBQkQZvSi8I/pL4BoGw/XPkh4XigPmsUgh0626AjRsgxHTkUThsG2T/sIlzdTsp52kSS1wAAAABJRU5ErkJggg==);
  margin: 0 3px 0 5px;
  }

Change the background colour of a row/table

Sometimes it’s useful to use a second tone to distinguish a group of tables from another one.

This works best for rows containing a table, but it could be expanded:

        #grey_row1                , #grey_row2                , #grey_row3                , #grey_row4                , #grey_row5                , #grey_row6                , #grey_row7                , #grey_row8                  {
          background-color:#c0c0c0;
        }
        #grey_row1 .panel-title   , #grey_row2 .panel-title   , #grey_row3 .panel-title   , #grey_row4 .panel-title   , #grey_row5 .panel-title   , #grey_row6 .panel-title   , #grey_row7 .panel-title   , #grey_row8 .panel-title     {
          background-color:#c0c0c0;
        }
        #grey_row1 .panel-body    , #grey_row2 .panel-body    , #grey_row3 .panel-body    , #grey_row4 .panel-body    , #grey_row5 .panel-body    , #grey_row6 .panel-body    , #grey_row7 .panel-body    , #grey_row8 .panel-body      {
          background-color:#c0c0c0;
        }
        #grey_row1 .element-footer, #grey_row2 .element-footer, #grey_row3 .element-footer, #grey_row4 .element-footer, #grey_row5 .element-footer, #grey_row6 .element-footer, #grey_row7 .element-footer, #grey_row8 .element-footer  {
          background-color:#c0c0c0;
        }
        #grey_row1 .splunk-table  , #grey_row2 .splunk-table  , #grey_row3 .splunk-table  , #grey_row4 .splunk-table  , #grey_row5 .splunk-table  , #grey_row6 .splunk-table  , #grey_row7 .splunk-table  , #grey_row8 .splunk-table    {
          width: 99% !important;
          margin: auto !important;
        }
        #grey_row1 .sorts         , #grey_row2 .sorts         , #grey_row3 .sorts         , #grey_row4 .sorts         , #grey_row5 .sorts         , #grey_row6 .sorts         , #grey_row7 .sorts         , #grey_row8 .sorts           {
          background-color:#a0a0a0;
        }
        #grey_row1 .odd .string   , #grey_row2 .odd .string   , #grey_row3 .odd .string   , #grey_row4 .odd .string   , #grey_row5 .odd .string   , #grey_row6 .odd .string   , #grey_row7 .odd .string   , #grey_row8 .odd .string     {
          background-color:#d0d0d0;
        }
        #grey_row1 .odd .numeric  , #grey_row2 .odd .numeric  , #grey_row3 .odd .numeric  , #grey_row4 .odd .numeric  , #grey_row5 .odd .numeric  , #grey_row6 .odd .numeric  , #grey_row7 .odd .numeric  , #grey_row8 .odd .numeric    {
          background-color:#d0d0d0;
        }

You need to add the ID to the row you want to colour. E.g.:

 <row id="grey_row1"> ...

Unfortunately, you cannot reuse IDs, you need to have one for each row you want to change. The code above provides you with 8 and it’s easy to expand as needed.


Smaller table in case of no results

If your dashboard can end up with one or more tables that have no results, it might be annoying that these tables take a lot of space. Fortunately, you can combine tokens and CSS!


In your table's definition:

  <table id="myTableID">
    <search>
      <done>
        <condition match="'job.resultCount' != 0">
          <set token="myTableHeightCSS"></set>
          <set token="myTableAlertCSS"></set>
        </condition>
        <condition match="'job.resultCount' == 0">
          <set token="myTableHeightCSS">height: 50px !important;</set>
          <set token="myTableAlertCSS">position:relative; top: -130px !important;</set>
        </condition>
      </done>
      <query> ...

And then in the CSS:

  #myTableID .splunk-table {
    $myTableHeightCSS$
  }
  #myTableID .alert-info { 
    $myTableAlertCSS$
  }

Highlighting an important table if it has results

This makes the table slightly smaller and surrounds it in bright yellow. But it only does it if there are results. Otherwise, it uses the previous trick to make it smaller.

        #myRow {
          $myRowCSS$ }
        #myRow .panel-element-row {
          $myRowCSS$ }
        # myRow .panel-title {
          $myRowCSS$ }
        # myRow .panel-head {
          $myRowCSS$ }
        # myRow .panel-body {
          $myRowCSS$ }
        # myRow .element-footer {
          $myRowCSS$ }
        # myRow .splunk-table {
          width: 99% !important;
          margin: auto !important;
        }
          #myTableID .splunk-table{
            $myTableHeightCSS$ }
          #myTableID .alert-info{
            $myTableAlertCSS$ }

You will need a row ID, a table ID and a conditional <done> statement:

  <row id="myRow">
    <panel>
      <table id="myTableID">
        <title>Blah blah blah</title>
        <search>
          <done>
            <condition match="'job.resultCount' != 0">
              <set token="myTableHeightCSS"></set>
              <set token="myTableAlertCSS"></set>
              <set token="myRowCSS">background-color:yellow;</set>
            </condition>
            <condition match="'job.resultCount' == 0">
              <set token="myTableHeightCSS">height: 50px !important;</set>
              <set token="myTableAlertCSS">position:relative; top: -130px !important;</set>
              <set token="myRowCSS"></set>
            </condition>
          </done>

Make a busy table breathe by removing sort icons

If you have a table with a lot of columns, you’ll notice that the little triangular arrows on each column that indicate if the results are sorted by that field, and if so in increasing or decreasing order, take a lot of valuable space. It’s not recommended, but at a pinch you could just get rid of them:

        i.icon-sorts {
          display: none;
        }

The functionality won’t change: you can still sort by clicking on a column header. But it won’t have the visual feedback anymore. Might be a small price to pay for a more readable dashboard.


Messing with the width of a table column

Sometimes you don’t want to let splunk choose the width of a column. You can set it in CSS, you just need to give your table an ID and specify the name of the column:

        #myTableID th[data-sort-key=mycolumnname]  {
          width: 30px;
          min-width: 30px;
          max-width: 30px;
        }

Add documentation to your dashboard with a tooltip and a sliding drawer


I believe credit for this goes to Olivier Lauret from Octamis.


Load the code

The first step is to add the docs4dash.js" script file and the "docs4dash.css" style sheet to the dashboard:

<dashboard script="docs4dash.js" stylesheet="docs4dash.css">

Or:

<form script="docs4dash.js" stylesheet="docs4dash.css">

Add the question mark icon

The document should be hidden to free some space on the dashboard. You therefore need to have an icon on which you need to click to make the documentation appear. We are using a question mark icon to do this. In this example, we are placing it inside the h1 tag.

<row>
  <panel>
    <html>
      <h1>Some title <span class="docs4dash-link docs4dash-tooltip" data-docs4dash-id="some_id" data-tootle="tooltip" data-placement="top" title="Click here for documentation"> <i class="icon icon-question-circle"/> </span> </h1>

The span tag must have a class of docs4dash-link and a unique data-docs4dash-id value. The data-docs4dash-id value will be use later.


If you want to use the tooltip, the span tag must docs4dash-tooltip class value and the data-tootle="tooltip" attribute and value.


The content of the tooltip is set in title attribute of the span tag. The tooltip position can be set in the data-placement attribute.


Adding the documentation content

The documentation content should be set in a div tag with a "docs4dash" class and a data-docs4dash-ref which has the same value than the data-docs4dash-id defined above:

      <div class="docs4dash" data-docs4dash-ref="some_id">
        <p>This is is the best documentation in the world!</p>
      </div>
    </html>
  </panel>
<row>

A submit button that works


This is so awesome it has its own post: https://www.gabrielvasseur.com/post/easy-yet-powerful-submit-buttons-in-your-simple-xml-dashboards


Using tokens to load or reload some search(es)

This is to be combined with a custom submit button!

At the top of your dashboard, initialise your token:

  <init>
    <set token="reload_everything">0</set>
  </init>

In the search or searches you want to be able to reload, add (almost anywhere):

  | eval _reload_everything=$reload_everything$

Then you can use the <done> section of a hidden search to trigger the reload with this:

<row depends="$show_debug$">
  <panel>
    <title>Hidden search to reload everything</title>
    <table>
      <search>
        <done>
          <set token="reload_everything">$result.now$</set>
          <unset token="submit_trigger"></unset>
        </done>
        <query> | stats count  | eval now=now() | eval trigger="$submit_trigger$" 
        </query>

Notes on debugging javascript


If you’re adventurous enough to write or tweak your own javascript, you might run into more problems.


First make sure you use the bookmark trick mentioned at the beginning of this post. Each time you go in the “edit dashboard” page and come back out, your javascript is off loaded. So don’t be surprised if it doesn’t seem to work. It means you’ll need to reload your dashboard many times.


If you’re on a search head cluster, you might want to hard code a specific search head in your bookmark for your development session.


SSH to that search head.


Now you can repeat the following steps:

  • write/tweak your javascript in /opt/splunk/etc/apps/<insert-app-name>/appserver/static/<insert-meaningful-name>.js

  • run splunk restart splunkweb. This shouldn't disturb other users.

  • ctrl-F5 your browser to reload your dashboard. You might have to use your bookmark too.

If your changes don’t seem to get picked up:

  • You may also have to delete the browser cache

  • It's a good idea to have your javascript set a bogus token to something that you change with every iteration so that you can confirm which version of your code is actually running.

  • You might also have some luck with browsing to https://yourhostname:8000/en-US/_bump in fact I wonder if that could replace the splunkweb restart? Let me know!

Once you are happy, you should copy your javascript to the search head deployer and deploy it to all the search heads properly.


Use base searches


Base searches bring about a lot of performance, both for the comfort of the platform and the users:

  • if you have more than one dashboard element that is powered by the same data, you should search for that data only once in a base search, and have the various element re-use that work

  • if your form allows the user to filter the data or change the fields displayed, you should have a base search to do the main job, and have as many of the user-controlled tweaks in a child-search: this will allow the results to refresh instantly every time the user tweaks a setting, rather than having to re-run the search. Beware though: do NOT do that if the base search might return an enormous amount of data!

You could place the hidden search in a <search> element anywhere in your dashboard, but if you do that it’ll always end up at the very top, which can be confusing to maintain. Instead, place the base search in a hidden row. This allows you to use the UI edit mode to tweak it, without having to go back to the XML all the time:

<row depends="$show_debug$">
  <panel>
    <title>Hidden base search for blah blah</title>
    <table>
      <search id="my_base_search">
        <query>...

Base searches can have weird side effects (keep reading). To avoid them, always do this:


Make sure you end every base search with a variation of the following

| fields            list all the fields you want to use in a child search
| fillnull value="" list all the fields you want to use in a child search
| noop

Why? Because a child search based on a base search isn't the same as running the full search. There's a seam and the data is manipulated in some different ways at the seam. I've seen several weird situations where a search appears broken in the dashboard but when you click on the magnifying glass to open it in a search bar, it works perfectly. Eventually I found on splunk answers the workaround with fields and fillnull.


Why the noop? The noop is purely for clarity. If like me you like to format your searches with a line break before each new pipe, you will notice that when you have a base search:

| this is my base search
| eval total=count1 + count2
| fields            total count1 count2
| fillnull value="" total count1 count2

And then you have a child search:

| stats count
| eval blah=”hello”

When you open the result in a search bar (by clicking on the magnifying glass), it becomes:

| this is my base search
| eval total=count1 + count2
| fields            total count1 count2
| fillnull value="" total count1 count2| stats count
| eval blah=”hello”

Instead of

| this is my base search
| eval total=count1 + count2
| fields            total count1 count2
| fillnull value="" total count1 count2
| stats count
| eval blah=”hello”

Notice how easy it is to miss the "stats count" entirely.

101 views0 comments

Recent Posts

See All

This for the most part isn't splunk-specific, but if you do any amount of administration on the linux command line, you might find it helpful. .bashrc A better prompt In your .bashrc: PS1='\t \[\033[0