Merge "Export all journal logs"

Author: Zuul

roles/export-devstack-journal/README.rst

@@ -1,11 +1,15 @@
 Export journal files from devstack services
 
-Export the systemd journal for every devstack service in native
-journal format as well as text.  Also, export a syslog-style file with
-kernal and sudo messages.
+This performs a number of logging collection services
 
-Writes the output to the ``logs/`` subdirectory of
-``stage_dir``.
+* Export the systemd journal in native format
+* For every devstack service, export logs to text in a file named
+  ``screen-*`` to maintain legacy compatability when devstack services
+  used to run in a screen session and were logged separately.
+* Export a syslog-style file with kernel and sudo messages for legacy
+  compatability.
+
+Writes the output to the ``logs/`` subdirectory of ``stage_dir``.
 
 **Role Variables**
 

roles/export-devstack-journal/tasks/main.yaml

@@ -6,32 +6,49 @@
     state: directory
     owner: "{{ ansible_user }}"
 
-# TODO: convert this to ansible
-- name: Export journal files
+- name: Export legacy stack screen log files
   become: true
   shell:
     cmd: |
       u=""
       name=""
-      for u in `systemctl list-unit-files | grep devstack | awk '{print $1}'`; do
+      for u in $(systemctl list-unit-files | grep devstack | awk '{print $1}'); do
         name=$(echo $u | sed 's/devstack@/screen-/' | sed 's/\.service//')
         journalctl -o short-precise --unit $u | gzip - > {{ stage_dir }}/logs/$name.txt.gz
       done
 
-      # Export the journal in export format to make it downloadable
-      # for later searching. It can then be rewritten to a journal native
-      # format locally using systemd-journal-remote. This makes a class of
-      # debugging much easier. We don't do the native conversion here as
-      # some distros do not package that tooling.
-      journalctl -u 'devstack@*' -o export | \
-          xz --threads=0 - > {{ stage_dir }}/logs/devstack.journal.xz
-
-      # The journal contains everything running under systemd, we'll
-      # build an old school version of the syslog with just the
-      # kernel and sudo messages.
+- name: Export legacy syslog.txt
+  become: true
+  shell:
+    # The journal contains everything running under systemd, we'll
+    # build an old school version of the syslog with just the
+    # kernel and sudo messages.
+    cmd: |
       journalctl \
           -t kernel \
           -t sudo \
           --no-pager \
           --since="$(cat {{ devstack_base_dir }}/log-start-timestamp.txt)" \
         | gzip - > {{ stage_dir }}/logs/syslog.txt.gz
+
+# TODO: convert this to ansible
+#  - make a list of the above units
+#  - iterate the list here
+- name: Export journal
+  become: true
+  shell:
+    # Export the journal in export format to make it downloadable
+    # for later searching. It can then be rewritten to a journal native
+    # format locally using systemd-journal-remote. This makes a class of
+    # debugging much easier. We don't do the native conversion here as
+    # some distros do not package that tooling.
+    cmd: |
+      journalctl -o export \
+          --since="$(cat {{ devstack_base_dir }}/log-start-timestamp.txt)" \
+        | xz --threads=0 - > {{ stage_dir }}/logs/devstack.journal.xz
+
+- name: Save journal README
+  become: true
+  template:
+    src: devstack.journal.README.txt.j2
+    dest: '{{ stage_dir }}/logs/devstack.journal.README.txt'

roles/export-devstack-journal/templates/devstack.journal.README.txt.j2

@@ -0,0 +1,33 @@
+Devstack systemd journal
+========================
+
+The devstack.journal file is a copy of the systemd journal during the
+devstack run.
+
+To use it, you will need to convert it so journalctl can read it
+locally.  After downloading the file:
+
+ $ /lib/systemd/systemd-journal-remote <(xzcat ./devstack.journal.xz) -o output.journal
+
+Note this binary is not in the regular path.  On Debian/Ubuntu
+platforms, you will need to have the "sytemd-journal-remote" package
+installed.
+
+It should result in something like:
+
+ Finishing after writing <large number> entries
+
+You can then use journalctl to examine this file.  For example, to see
+all devstack services try:
+
+ $ journalctl --file ./output.journal -u 'devstack@*'
+
+To see just cinder API server logs restrict the match with
+
+ $ journalctl --file ./output.journal -u 'devstack@c-api'
+
+There may be many types of logs available in the journal, a command like
+
+ $ journalctl --file ./output.journal --output=json-pretty | grep "_SYSTEMD_UNIT" | sort -u
+
+can help you find interesting things to filter on.