Rework the beats role documentations

.github/workflows/test_role_beats.yml

@@ -15,6 +15,7 @@ on:
   pull_request:
     paths:
       - 'roles/beats/**'
+      - '!roles/beats/**/*.md'
       - '.github/workflows/test_role_beats.yml'
       - 'molecule/beats_**'
 

README.md

@@ -12,7 +12,7 @@ Every role is documented with all variables, please refer to the documentation f
 > [!NOTE]
 > Some roles have fixed requirements that you must observe. Please have a look at the [requirements](docs/01-requirements.md) before using the collection. (There is a high probability that some of them will be refactored soon)
 
-* [Beats](docs/role-beats.md)
+* [Beats](roles/beats/README.md)
 * [Elasticsearch](docs/role-elasticsearch.md)
 * [Kibana](docs/role-kibana.md)
 * [Logstash](roles/logstash/README.md)

docs/getting-started.md

@@ -47,7 +47,7 @@ Or refer to the role with the FQCN of the role.
 Roles
 -------
 
-* [Beats](role-beats.md)
+* [Beats](../roles/beats/README.md)
 * [Elasticsearch](role-elasticsearch.md)
 * [Kibana](role-kibana.md)
 * [Logstash](../roles/logstash/README.md)

docs/role-beats.md

@@ -1,124 +1,5 @@
-Ansible Role: Beats
-=========
+# Beats role
 
-![Test Role Beats](https://github.com/netways/ansible-collection-elasticstack/actions/workflows/test_role_beats.yml/badge.svg)
+The Beats role documentation now lives with the role itself:
 
-This role installs and configures Beats. You can use it as a standalone role or combine it with our other roles managing the Elastic Stack.
-
-Requirements
-------------
-
-You need to have the beats you want to install available in your software repositories. We provide a [role](./role-repos.md) for just that but if you have other ways of managing software, just make sure it's available. Alternatively you can install the Beats yourself.
-
-* `community.crypto` collection: ansible-galaxy collection install community.crypto
-
-Role Variables
---------------
-
-* *beats_filebeat*: Install and manage filebeat (Default: `true`)
-* *beats_filebeat_enable*: Automatically start Filebeat (Default: `true`)
-* *beats_filebeat_output*: Set to `logstash` or `elasticsearch`. (default: `logstash`)
-* *beats_filebeat_elastic_monitoring*: `filebeat` monitoring over `Elastic Stack monitoring features` (default: `false`)    
-* *beats_filebeat_syslog_udp*: Use UDP Syslog input (Default: `false`)
-* *beats_filebeat_syslog_udp_port*: Port of UDP Syslog input (Default: `514`)
-* *beats_filebeat_syslog_tcp*: Use TCP Syslog input (Default: `false`)
-* *beats_filebeat_syslog_tcp_port*: Port of TCP Syslog input (Default: `514`)
-* *beats_filebeat_log_input*: Enable Logfile reading (Default: `true`)
-* *beats_filebeat_mysql_slowlog_input*: Enable MySQL/MariaDB slow query log collection incl. multiline (Default: `false`)
-* *beats_filebeat_log_inputs*: Logfiles to read (Default: see below)
-
-Default of `beats_filebeat_log_inputs`
-
-```
-  messages:
-    name: messages
-    paths:
-      - /var/log/messages
-```
-
-You can optionally add `fields` to every input as well. You can also add a `multiline` section with options (`type`, `pattern`, `negate` and `match` so far)
-
-Here's a longer example for an input:
-```
-beats_filebeat_log_inputs:
-  messages:
-    name: messages
-    paths:
-      - /var/log/messages
-      - /var/log/secure
-      - /var/log/httpd/*access_log*
-    multiline:
-      type: pattern
-      pattern: '^[[:space:]]+(at|\.{3})[[:space:]]+\b|^Caused by:'
-      negate: false
-      match: after
-```
-* *beats_filebeat_journald*: Enable collection of JournalD logs (default: `false`) - available since Filebeat 7.16
-* *beats_filebeat_journald_inputs*: List of journald inputs. Use for different filters on events. You can add a list of `include_matches` entries for filtering.
-Default of `beats_filebeat_journald_inputs`:
-```
-beats_filebeat_journald_inputs:
-  everything:
-    id: everything
-```
-* *beats_filebeat_docker*: Enable collection of Docker logs (default: `false`) **ONLY WORKS ON RELEASE 7 SO FAR**
-* *beats_filebeat_docker_ids*: IDs of containers to collect. (default: `*`)
-
-* *beats_filebeat_loadbalance*: Enable loadbalancing for Filebeats Logstash output (default: `true`)
-* *beats_filebeat_modules*: **EXPERIMENTAL**: Give a list of modules to enable. (default: none)
-
-* *beats_auditbeat*: Install and manage filebeat (Default: `false`)
-* *beats_auditbeat_output*: Output for Auditbeat Set to `logstash` or `elasticsearch`. (default: `elasticsearch`)
-* *beats_auditbeat_enable*: Automatically start Auditbeat (Default: `true`)
-* *beats_auditbeat_setup*: Run Auditbeat Setup (Default: `true`) (Only works with Elasticsearch output)
-* *beats_auditbeat_loadbalance*: Enable loadbalancing for Auditbeats Logstash output (default: `true`)
-
-* *beats_metricbeat*: Enable installation and management of Metricbeat (Default: `false`)
-* *beats_metricbeat_enable*: Start Metricbeat automatically (Default: `true`)
-* *beats_metricbeat_output*: Set to `logstash` or `elasticsearch`. (default: `elasticsearch`)
-* *beats_metricbeat_modules*: List of modules to enable. (Default: `- system`)
-* *beats_metricbeat_loadbalance*: Enable loadbalancing for Metricbeats Logstash output (default: `true`)
-
-* *beats_security*: Activate TLS for connections to targets. Can either be use with our other roles and `elasticstack_full_stack` to automatically create certificates or `beats_tls*` variables for custom certificates. (default: `false`)
-* *beats_target_hosts*: Only use when this role is used standalone. When used in combination with our other roles, the target hosts will be determined automatically. Use a YAML list. (default: `- localhost`)
-* *elasticstack_elasticsearch_http_port*: Port of Elasticsearch to send events to (Default: `9200`)
-* *elasticstack_beats_port*: Port of Logstash to send events to (Default: `5044`)
-* *beats_logging*: Where to log (Default: `file`)
-* *beats_loglevel*: Level of logging (for all beats) (Default: `info`)
-* *beats_logpath*: If logging to file, where to put logfiles (Default: `/var/log/beats`)
-* *beats_fields*: Fields that are added to every input in the configuration
-
-The following variables only apply if you use this role together with our other Elastic Stack roles.
-
-* *elasticstack_full_stack*: Use `elasticsearch` as well (default: `false`)
-* *elasticstack_variant*: Define which variant of elastic stack to use. (default: `elastic`)
-* *elasticstack_ca_dir*: Directory where on the Elasticsearch CA host certificates are stored. This is only useful in connection with out other Elastic Stack related roles. (default: `/opt/es-ca`)
-* *elasticstack_ca_pass*: Password for Elasticsearch CA (default: `PleaseChangeMe`)
-* *elasticstack_initial_passwords*: Path to file with initical elasticsearch passwords (default: `/usr/share/elasticsearch/initial_passwords`)
-* *elasticstack_version*: Install specific version (Default: none. Possible values: e.g. `7.10.1` or `latest`)
-
-If you want to use this role with your own TLS certificates, use these variables.
-
-* *beats_ca_dir*: Path to custom CA certificates and keys (default: none - if not set will be filled with different values depending on which Stack variant is used)
-* *beats_tls_key*: Path to the keyfiles (default: `{{ beats_ca_dir }}/{{ ansible_hostname }}.key`)
-* *beats_tls_cert*: Path to the certificate (default: `{{ beats_ca_dir }}/{{ ansible_hostname }}.crt`)
-* *beats_tls_key_passphrase*: Passphrase of the keyfile (default: `BeatsChangeMe`)
-* *beats_cert_validity_period*: number of days that the generated certificates are valid (default: 1095).
-* *beats_cert_expiration_buffer*: Ansible will renew the beats certificate if its validity is shorter than this value, which should be number of days. (default: 30)
-* *beats_cert_will_expire_soon*: Set it to true to renew beats certificate (default: `false`), Or run the playbook with `--tags renew_beats_cert` to do that.
-* *beats_tls_cacert*: Path to the CA.crt (default: `{{ beats_ca_dir }}/ca.crt`)
-
-## Usage
-
-```
-- name: Install Elastic Beats
-  hosts: beats-hosts
-  collections:
-    - netways.elasticstack
-  vars:
-    elasticsearch_jna_workaround: true
-    elasticsearch_disable_systemcallfilterchecks: true
-  roles:
-    - repos
-    - beats
-```
+**➜ [roles/beats/README.md](../roles/beats/README.md)**

roles/beats/README.md

@@ -0,0 +1,96 @@
+# Ansible Role: Beats
+
+Installs and configures the Elastic [Beats](https://www.elastic.co/beats) —
+Filebeat, Auditbeat and Metricbeat — on Linux systems. The role can run
+standalone or together with the other Elastic Stack roles. Beats can ship to
+Logstash or directly to Elasticsearch.
+
+## Requirements
+
+* The `community.crypto` collection (used to check certificate expiration).
+* The Beats you want to install must be available in your software repositories —
+  use the [`repos`](../repos) role or provide them yourself.
+
+## Example
+
+```yaml
+- name: Install Beats
+  hosts: beats
+  collections:
+    - netways.elasticstack
+  roles:
+    - repos
+    - beats
+```
+
+## Filebeat inputs
+
+Filebeat can read from several sources (log files, syslog, journald, Docker,
+modules). For the structure of `beats_filebeat_log_inputs`,
+`beats_filebeat_journald_inputs` and the other input variables, see the
+[Filebeat inputs documentation](docs/filebeat-inputs.md).
+
+## Tags
+
+Run only parts of the role with `--tags`:
+
+* `configuration` (alias `beats_configuration`) — (re)write the Filebeat and Auditbeat configuration without installing. Use `beats_filebeat_configuration` or `beats_auditbeat_configuration` to limit it to one of them.
+* `certificates` — only generate and distribute the TLS certificates.
+* `renew_beats_cert` / `renew_ca` — force renewal of the Beats certificate.
+
+<!-- ANSIBLE DOCSMITH MAIN START -->
+## Role variables<a id="variables"></a>
+
+| Variable | Type | Default | Choices | Description |
+|----------|------|---------|---------|-------------|
+| `beats_filebeat` | `bool` | `true` | — | Install and manage Filebeat. |
+| `beats_auditbeat` | `bool` | `false` | — | Install and manage Auditbeat. |
+| `beats_metricbeat` | `bool` | `false` | — | Install and manage Metricbeat. |
+| `beats_target_hosts` | `list` of `str` | `['localhost']` | — | Hosts the Beats ship to. Only used when the role runs standalone; with the other Elastic Stack roles the targets are determined automatically. |
+| `beats_fields` | `list` of `str` | N/A | — | Fields added to every input, given as a list of "key: value" strings (the global counterpart to the per-input fields). Unset by default. See the Filebeat inputs documentation. |
+| `beats_logging` | `str` | `"file"` | — | Where the Beats log. Set to "file" to log into beats_logpath; any other value leaves the Beats built-in logging. |
+| `beats_loglevel` | `str` | `"info"` | — | Log level for all Beats. |
+| `beats_logpath` | `str` | `"/var/log/beats"` | — | Directory for the log files when beats_logging is "file". |
+| `beats_filebeat_enable` | `bool` | `true` | — | Start and enable the Filebeat service. |
+| `beats_filebeat_output` | `str` | `"logstash"` | `logstash`, `elasticsearch` | Where Filebeat sends its events. |
+| `beats_filebeat_elastic_monitoring` | `bool` | `false` | — | Report Filebeat monitoring data through the Elastic Stack monitoring features. |
+| `beats_filebeat_loadbalance` | `bool` | `true` | — | Enable load balancing for the Filebeat Logstash output. |
+| `beats_filebeat_log_input` | `bool` | `true` | — | Read log files with the inputs from beats_filebeat_log_inputs. |
+| `beats_filebeat_log_inputs` | `dict` | `{'messages': {'paths': ['/var/log/messages', '/var/log/syslog']}}` | — | Log files to read, keyed by a free name that becomes the filestream id. Each entry has paths and optional fields and multiline settings. See the Filebeat inputs documentation. |
+| `beats_filebeat_syslog_tcp` | `bool` | `false` | — | Enable a TCP syslog input. |
+| `beats_filebeat_syslog_tcp_port` | `int` | `514` | — | Port of the TCP syslog input. |
+| `beats_filebeat_syslog_udp` | `bool` | `false` | — | Enable a UDP syslog input. |
+| `beats_filebeat_syslog_udp_port` | `int` | `514` | — | Port of the UDP syslog input. |
+| `beats_filebeat_journald` | `bool` | `false` | — | Collect logs from journald. Available since Filebeat 7.16. |
+| `beats_filebeat_journald_inputs` | `dict` | `{'everything': {'id': 'everything'}}` | — | Journald inputs, keyed by a free name. Each entry has an id and optional include_matches filters. See the Filebeat inputs documentation. |
+| `beats_filebeat_docker` | `bool` | `false` | — | Collect Docker container logs. Only works on Elastic Stack release 7. |
+| `beats_filebeat_docker_ids` | `str` | `"*"` | — | IDs of the containers to collect logs from. |
+| `beats_filebeat_mysql_slowlog_input` | `bool` | `false` | — | Collect the MySQL/MariaDB slow query log, including multiline handling. |
+| `beats_filebeat_modules` | `list` of `str` | N/A | — | List of Filebeat modules to enable (experimental). Unset by default. |
+| `beats_auditbeat_enable` | `bool` | `true` | — | Start and enable the Auditbeat service. |
+| `beats_auditbeat_output` | `str` | `"elasticsearch"` | `logstash`, `elasticsearch` | Where Auditbeat sends its events. |
+| `beats_auditbeat_setup` | `bool` | `true` | — | Run the Auditbeat setup (index management and pipelines). Only effective with the elasticsearch output. |
+| `beats_auditbeat_loadbalance` | `bool` | `true` | — | Enable load balancing for the Auditbeat Logstash output. |
+| `beats_metricbeat_enable` | `bool` | `true` | — | Start and enable the Metricbeat service. |
+| `beats_metricbeat_output` | `str` | `"elasticsearch"` | `logstash`, `elasticsearch` | Where Metricbeat sends its events. |
+| `beats_metricbeat_modules` | `list` of `str` | `['system']` | — | Metricbeat modules to enable. |
+| `beats_metricbeat_loadbalance` | `bool` | `true` | — | Enable load balancing for the Metricbeat Logstash output. |
+| `beats_security` | `bool` | `false` | — | Activate TLS for the connections to the targets. Works with the other roles and elasticstack_full_stack to create certificates automatically, or with the beats_tls_* variables for custom certificates. |
+| `beats_ca_dir` | `str` | N/A | — | Base directory for custom CA certificates and keys. Unset by default; the role fills it depending on the stack variant (/opt/ca standalone, or /etc/beats/certs in a full stack). |
+| `beats_tls_key` | `str` | `"{{ beats_ca_dir | default('') }}/{{ inventory_hostname }}-beats.key"` | — | Path to the private key file for custom certificates. |
+| `beats_tls_cert` | `str` | `"{{ beats_ca_dir | default('') }}/{{ inventory_hostname }}-beats.crt"` | — | Path to the certificate for custom certificates. |
+| `beats_tls_cacert` | `str` | `"{{ beats_ca_dir | default('') }}/ca.crt"` | — | Path to the CA certificate for custom certificates. |
+| `beats_tls_key_passphrase` | `str` | `"BeatsChangeMe"` | — | Passphrase of the private key. |
+| `beats_cert_validity_period` | `int` | `1095` | — | Number of days the generated certificates are valid. |
+| `beats_cert_expiration_buffer` | `str` | `"+30d"` | — | Renew the certificate when it would expire within this period. Uses the community.crypto check_period format (e.g. "+30d"), not a plain number. |
+| `beats_cert_will_expire_soon` | `bool` | `false` | — | Set to true to force renewal of the Beats certificate. Alternatively run the playbook with the renew_beats_cert tag. |
+
+<!-- ANSIBLE DOCSMITH MAIN END -->
+
+## Shared variables
+
+This role also uses the collection-wide `elasticstack_*` variables (e.g.
+`elasticstack_full_stack`, `elasticstack_variant`, `elasticstack_ca_host`,
+`elasticstack_ca_pass`, `elasticstack_release`, `elasticstack_version`,
+`elasticstack_elasticsearch_http_port`, `elasticstack_beats_port`). They are
+documented centrally with the [elasticsearch role](../../docs/role-elasticsearch.md).

roles/beats/defaults/main.yml

@@ -12,9 +12,9 @@ beats_loglevel: info
 
 # Use TLS without Elastic X-Pack #
 
-beats_tls_key: "{{ beats_ca_dir }}/{{ inventory_hostname }}-beats.key"
-beats_tls_cert: "{{ beats_ca_dir }}/{{ inventory_hostname }}-beats.crt"
-beats_tls_cacert: "{{ beats_ca_dir }}/ca.crt"
+beats_tls_key: "{{ beats_ca_dir | default('') }}/{{ inventory_hostname }}-beats.key"
+beats_tls_cert: "{{ beats_ca_dir | default('') }}/{{ inventory_hostname }}-beats.crt"
+beats_tls_cacert: "{{ beats_ca_dir | default('') }}/ca.crt"
 beats_tls_key_passphrase: BeatsChangeMe
 
 # Filebeat specific #
@@ -28,7 +28,6 @@ beats_filebeat_syslog_tcp_port: 514
 beats_filebeat_log_input: true
 beats_filebeat_log_inputs:
   messages:
-    name: messages
     paths:
       - /var/log/messages
       - /var/log/syslog