doc: Update agent and kernel debug documents

The agent and kernel debug documents had deviated from reality.

Signed-off-by: James O. D. Hunt <james.o.hunt@intel.com>

Author: James O. D. Hunt

docs/debug-agent.md

@@ -4,44 +4,43 @@ Clear Containers runtime launch a special virtual machine when a new pod is
 created. This VM uses a VM-agent to spawn processes on behalf of the
 pod/container(s) running inside the VM.
 
-From Clear Containers 3.0-beta release a new agent is used,
-[Clear Containers Agent](https://github.com/clearcontainers/agent):
+From Clear Containers 3.0-beta release a new agent (`cc-agent`) is used,
+[Clear Containers Agent](https://github.com/clearcontainers/agent).
 
-The Clear Containers Agent relies heavily on libcontainer used by runc 
-(standard program when running containers on bare metal).
+The Clear Containers agent relies heavily on [`libcontainer`](https://github.com/opencontainers/runc/tree/master/libcontainer) used by [`runc`](https://github.com/opencontainers/runc/) (standard program when running containers on bare metal).
 
 To provide a debug log of any agent activity on a guest, the Clear Containers
 agent sends logs through a QEMU serial console that are collected by [cc-proxy](https://github.com/clearcontainers/proxy)
 and shown in its logs. By default, the Clear Containers agent logs are not collected by 
-`cc-proxy` but can be enabled by adding the `-log debug` option to `cc-proxy`.
-
-1- Add the `-log debug` option to  `cc-proxy`  
-
-```
-mkdir -p /etc/systemd/system/cc-proxy.service.d/
-cat << EOT >  /etc/systemd/system/cc-proxy.service.d/proxy-debug.conf
-[Service]
-ExecStart=
-ExecStart=/usr/libexec/clear-containers/cc-proxy -log debug
-EOT
-# Restart cc-proxy to provide debug logs.
-systemctl daemon-reload
-systemctl restart cc-proxy
-```
-2- Run a container to generate the agent debug logs
-
-```
-sudo docker run -ti busybox true
-```
-3- Filter the agent debug logs from the `cc-proxy` logs
-The `cc-proxy` logs show the sources of its collated information. To only see
-the agent debug logs, filter `cc-proxy` logs by the QEMU serial console (the
-agent logs were sent through it). The QEMU serial console is represented by
-`source=qemu`.
-
-```
-journalctl -u cc-proxy | grep source=qemu
-```
-
-The debug log format is:
-DEBU[0019] *AGENT MESSAGE* source=qemu vm=*CONTAINER_ID*
+`cc-proxy` but can be enabled by enabling the proxy debug option.
+
+1. Enable proxy debug
+
+   Set the `enable_debug=` option in the `[proxy.cc]` section to `true` (assumes a standard configuration file path):
+
+   ```
+   $ sudo awk '{if (/^\[proxy\.cc\]/) {got=1}; if (got == 1 && /^#enable_debug/) {print "enable_debug = true"; got=0; next; } else {print}}' /usr/share/defaults/clear-containers/configuration.toml
+   ```
+
+1. Run a container to generate the logs:
+
+   ```
+   $ sudo docker run -ti busybox true
+   ```
+
+1. Filter the agent debug logs from the `cc-proxy` logs
+
+   The `cc-proxy` logs show the sources of its collated information. To only see
+   the agent debug logs, filter `cc-proxy` logs by the QEMU serial console (the
+   agent logs were sent through it). The QEMU serial console is represented by
+   `source=qemu`.
+
+   ```
+   $ sudo journalctl -t cc-proxy | grep source=qemu | egrep '\<cc-agent\>'
+   ```
+
+   To extract all logs entries for a particular container:
+
+   ```
+   $ sudo sudo journalctl -t cc-proxy | grep source=qemu | grep vm=CONTAINER_ID | egrep '\<cc-agent\>'
+   ```

docs/debug-kernel.md

@@ -10,37 +10,33 @@ fails due to the kernel.
 To provide a debug log of the kernel boot, Clear Containers supports an option
 to enable kernel boot logs using the configuration file.
 
-1. Enable debug in `configuration.toml`
+1. Enable kernel boot messages
+
+   Set the `enable_debug=` option in the `[proxy.cc]` section to `true` (assumes a standard configuration file path):
 
    ```
-   $ enable_debug = true
+   $ sudo awk '{if (/^\[proxy\.cc\]/) {got=1}; if (got == 1 && /^#enable_debug/) {print "enable_debug = true"; got=0; next; } else {print}}' /usr/share/defaults/clear-containers/configuration.toml
    ```
 
-2. Run a container to generate the logs
+1. Run a container to generate the logs
 
    ```
    $ sudo docker run -ti busybox true
    ```
 
-3. Filter the kernel boot logs from the `cc-proxy` logs
+1. Filter the kernel boot logs from the `cc-proxy` logs
 
    The `cc-proxy` logs show the sources of its collated information. To see
-   the kernel boot logs, filter `cc-proxy` logs by the QEMU serial console.
+   the kernel boot logs, filter `cc-proxy` logs by the QEMU serial console,
+   excluding the agent messages.
    The QEMU serial console is represented by `source=qemu`.
 
    ```
-   $ journalctl -u cc-proxy | grep source=qemu
+   $ sudo journalctl -t cc-proxy | grep source=qemu | egrep -v '\<cc-agent\>'
    ```
 
-4. Obtain logs using standalone `cc-proxy`
-
-   In some cases it may be easier to run the `cc-proxy` standalone by first stopping
-   the `cc-proxy` service and running it standalone to capture the logs.
+   To extract all logs entries for a particular container:
 
    ```
-   $ sudo systemctl stop cc-proxy
-   $ sudo /usr/libexec/clear-containers/cc-proxy -log debug
+   $ sudo sudo journalctl -t cc-proxy | grep source=qemu | grep vm=CONTAINER_ID | egrep -v '\<cc-agent\>'
    ```
-
-   This will result in the `cc-proxy` log being printed on the terminal. This method
-   can also be used to capture the logs in a specific `cc-proxy` file.