Splunk Alert Script
A Splunk alert is a search that is periodically run, and if there are any results, an alert action can be run. This is a custom script that can be used to iterate over the search results and perform a task with the found fields. This process is documented, but the key takeaways are summarised on this page.
Alert Script Arguments
When Splunk fires an alert action, there are several arguments that are passed
to the script. I usually ignore all of them, except for argument 8, which
CSV file containing the search results. The process is
usually to open this file and iterate over the results.
|1||SPLUNK_ARG_1||Number of events returned|
|3||SPLUNK_ARG_3||Fully qualified query string|
|4||SPLUNK_ARG_4||Name of report|
|5||SPLUNK_ARG_5||Trigger reason For example, “The number of events was greater than 1.”|
|6||SPLUNK_ARG_6||Browser URL to view the report.|
|7||SPLUNK_ARG_7||Not used for historical reasons.|
|8||SPLUNK_ARG_8||File in which the results for the search are stored. Contains raw results in gzip file format.|
Alert Script Code
The following template should be placed in the
import gzip import csv def openany(p): if p.endswith(".gz"): return gzip.open(p) else: return open(p) results = sys.argv for row in csv.DictReader(openany(results)): # do something with row["field"]
This command can then be selected as a
Trigger Action when saving or modifying
Advanced Alert Actions
The aforementioned script has actually been deprecated in preference of Modular Alerts. These provide much more customisation features, but I often find this overkill.
Splunk also initiated the Adaptive Response Initiative which allows for Splunk to integrate with other (mainly security related) 3rd party products, and allows for bidirectional communication between them. This is a great source for ideas for alert actions.
Alert Script ideas
- Ticket Systems (raise a ticket)
- EMail (send an email)
- Anti Virus (start a scan on a particular machine)
- Firewall Rules (isolate a compromised machine)
- IOT devices (think physical traffic lights, flags and sirens)
- Enrichment (Google a search query, write the results to a file which is indexed by Splunk)
- Expect script (ssh to a server and restart a hung process)