The Docker Events'tab does not display any event

[root@clh-ucp01 ~]# docker --version
Docker version 17.06.2-ee-6, build e75fdb8

Universal Control Plane Version 2.2.5 42d28d140 | API Version: 1.30

Universal forwarder: pulling splunk/universalforwarder:7.0.0-monitor
Splunk Enterprise: pulling splunk/splunk:7.0.0-monitor

from one of the docker host running the universal forwader I can see that the scripts docker_events.sh is running
[root@clh-ucp01 ~]# docker ps | grep uni
9e960200aa38 splunk/universalforwarder:7.0.0-monitor "/sbin/entrypoint...." About an hour ago Up About an hour 1514/tcp, 8088-8089/tcp splunk_splunkuniversalforwarder.x3xm839eqtc2bp m3ijc4u7uk6.lsf7g4mav3br9ag8ps2tcgymg
[root@clh-ucp01 ~]# docker exec -it 9e960200aa38 bash
root@spuf-x3xm839eqtc2bpm3ijc4u7uk6:/opt/splunk# ps aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 21700 1732 ? Ss 12:20 0:00 /bin/bash /sbin/entrypoint.sh start-service
root 52 0.6 1.6 226052 130696 ? Sl 12:20 0:20 splunkd -p 8089 start
root 56 0.1 0.1 71956 11932 ? Ss 12:20 0:03 [splunkd pid=52] splunkd -p 8089 start [process-runner]
root 95 0.0 0.0 4328 648 ? Ss 12:20 0:00 /bin/sh -c /opt/splunk/etc/apps/ta-dockerstats/bin/docker_events.s
root 96 0.0 0.0 21656 1600 ? S 12:20 0:00 /bin/bash /opt/splunk/etc/apps/ta-dockerstats/bin/docker_events.sh
root 98 0.0 0.1 24208 12552 ? Sl 12:20 0:02 /opt/splunk/etc/apps/ta-dockerstats/bin/docker events
root 435 0.0 0.0 41560 1892 ? S 12:20 0:00 sudo -HEu root tail -n 0 -f /opt/splunk/var/log/splunk/splunkd_std
root 436 0.0 0.0 5832 616 ? S 12:20 0:00 tail -n 0 -f /opt/splunk/var/log/splunk/splunkd_stderr.log
root 128064 0.3 0.0 21848 2020 pts/0 Ss 13:17 0:00 bash
root 128070 0.0 0.0 19092 1284 pts/0 R+ 13:17 0:00 ps aux
root@spuf-x3xm839eqtc2bpm3ijc4u7uk6:/opt/splunk# exit

using search, it seems I can receive "events" with the sourcetype="dockervents"

see here: https://gist.github.com/chris7444/62c91ee7583aa843751aaad3c7a8b425

[halr9000]

Looks like a simple presentation layer bug. I have only a few minutes to research, so let's see...

1. I see here that the dashboard runs a saved search called "docker_events_by_container"

[halr9000]

Saved search is defined here:

docker-itmonitoring/app-docker/default/savedsearches.conf

Line 138 in 4f2c7e8

[docker_events_by_container]

[halr9000]

Ok, I see the search looks for sourcetype=DockerEvents which you have based on the gist. With key/value pairs in Splunk, the field name (key) is case sensitive, but the values are not, so we should be good there.

@chris7444 I can't repro now, but can you try to run the search and see if you get results? If not, the steps to debug SPL would be to (based on errors/output) remove the pipelines starting at the last and going backwards, simplifying and removing variables as needed. E.g.:

search = sourcetype=DockerEvents NOT _raw="*top"     \
        | head 100 | eval event=mvindex(split(_raw,") "), 1)    \
        | eval container_id=substr(_raw, 32, 12)    \
        | join [search sourcetype=DockerEvents NOT _raw="*top"    \
        | head 100 | eval event=mvindex(split(_raw,") "), 1)     \
        | eval container_id=substr(_raw, 32, 12)    \
        | lookup docker_containername.csv containerid as container_id OUTPUT containername as container_name    \
        | table container_id, container_name    \
        | dedup container_id, container_name]    \
        | search container_name= $container_name$    \
        | table _time, container_name, event

We know the base works. I would add the rest of that first line just for grins, make sure that still works. If so, skip almost to the end and run everything minus the last two lines (|search container_name... onward). Repeat this, going up until you find what part doesn't return results.