[in_systemd_docker] Error moving to next Journal entry: Systemd::JournalError: Cannot assign requested address

### Describe the bug

I am using fluentd-kubernetes-daemonset:v1.18-debian-opensearch-1 to send logs from my EKS cluster to opensearch.  I am getting the following error in some of my pods
```
2025-04-01 02:57:55 +0000 [warn]: #0 [in_systemd_docker] Error moving to next Journal entry: Systemd::JournalError: Cannot assign requested address
2025-04-01 02:57:55 +0000 [warn]: #0 [in_systemd_kubelet] Error moving to next Journal entry: Systemd::JournalError: Cannot assign requested address
2025-04-01 02:57:55 +0000 [warn]: #0 [in_systemd_bootkube] Error moving to next Journal entry: Systemd::JournalError: Cannot assign requested address
```

Below is the output of my fluentd starup log to know more about the configuration details

```
2025-04-01 08:11:04 +0000 [info]: init supervisor logger path=nil rotate_age=nil rotate_size=nil
2025-04-01 08:11:04 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
2025-04-01 08:11:05 +0000 [info]: gem 'fluentd' version '1.18.0'
2025-04-01 08:11:05 +0000 [info]: gem 'fluent-plugin-concat' version '2.5.0'
2025-04-01 08:11:05 +0000 [info]: gem 'fluent-plugin-detect-exceptions' version '0.0.16'
2025-04-01 08:11:05 +0000 [info]: gem 'fluent-plugin-grok-parser' version '2.6.2'
2025-04-01 08:11:05 +0000 [info]: gem 'fluent-plugin-json-in-json-2' version '1.0.2'
2025-04-01 08:11:05 +0000 [info]: gem 'fluent-plugin-kubernetes_metadata_filter' version '3.6.0'
2025-04-01 08:11:05 +0000 [info]: gem 'fluent-plugin-multi-format-parser' version '1.0.0'
2025-04-01 08:11:05 +0000 [info]: gem 'fluent-plugin-opensearch' version '1.1.5'
2025-04-01 08:11:05 +0000 [info]: gem 'fluent-plugin-parser-cri' version '0.1.1'
2025-04-01 08:11:05 +0000 [info]: gem 'fluent-plugin-prometheus' version '2.1.0'
2025-04-01 08:11:05 +0000 [info]: gem 'fluent-plugin-record-modifier' version '2.1.1'
2025-04-01 08:11:05 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.4.0'
2025-04-01 08:11:05 +0000 [info]: gem 'fluent-plugin-systemd' version '1.1.1'
2025-04-01 08:11:06 +0000 [info]: using configuration file: <ROOT>
  <source>
    @type systemd
    @id in_systemd_kubelet
    matches [{"_SYSTEMD_UNIT":"kubelet.service"}]
    read_from_head true
    tag "kubelet"
    <storage>
      @type "local"
      persistent true
      path "/var/log/fluentd-journald-kubelet-cursor.json"
    </storage>
  </source>
  <source>
    @type systemd
    @id in_systemd_docker
    matches [{"_SYSTEMD_UNIT":"docker.service"}]
    read_from_head true
    tag "docker.systemd"
    <storage>
      @type "local"
      persistent true
      path "/var/log/fluentd-journald-docker-cursor.json"
    </storage>
  </source>
  <source>
    @type systemd
    @id in_systemd_bootkube
    matches [{"_SYSTEMD_UNIT":"bootkube.service"}]
    read_from_head true
    tag "bootkube"
    <storage>
      @type "local"
      persistent true
      path "/var/log/fluentd-journald-bootkube-cursor.json"
    </storage>
  </source>
  <source>
    @type prometheus
    @id in_prometheus
    bind "0.0.0.0"
    port 24231
    metrics_path "/metrics"
  </source>
  <source>
    @type prometheus_output_monitor
    @id in_prometheus_output_monitor
  </source>
  <label @FLUENT_LOG>
    <match fluent.**>
      @type null
      @id ignore_fluent_logs
    </match>
  </label>
  <source>
    @type tail
    @id in_tail_cluster_autoscaler
    multiline_flush_interval 5s
    path "/var/log/cluster-autoscaler.log"
    pos_file "/var/log/fluentd-cluster-autoscaler.log.pos"
    tag "cluster-autoscaler"
    <parse>
      @type "kubernetes"
      unmatched_lines
      expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
      time_format "%m%d %H:%M:%S.%N"
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_container_logs
    path "/var/log/containers/*.log"
    pos_file "/var/log/fluentd-containers.log.pos"
    tag "kubernetes.*"
    exclude_path []
    read_from_head true
    <parse>
      @type "cri"
      time_format "%Y-%m-%dT%H:%M:%S.%N%:z"
      unmatched_lines
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_docker
    path "/var/log/docker.log"
    pos_file "/var/log/fluentd-docker.log.pos"
    tag "docker"
    <parse>
      @type "regexp"
      expression /^time="(?<time>[^)]*)" level=(?<severity>[^ ]*) msg="(?<message>[^"]*)"( err="(?<error>[^"]*)")?( statusCode=($<status_code>\d+))?/
      unmatched_lines
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_etcd
    path "/var/log/etcd.log"
    pos_file "/var/log/fluentd-etcd.log.pos"
    tag "etcd"
    <parse>
      @type "none"
      unmatched_lines
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_glbc
    multiline_flush_interval 5s
    path "/var/log/glbc.log"
    pos_file "/var/log/fluentd-glbc.log.pos"
    tag "glbc"
    <parse>
      @type "kubernetes"
      unmatched_lines
      expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
      time_format "%m%d %H:%M:%S.%N"
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_kube_apiserver_audit
    multiline_flush_interval 5s
    path "/var/log/kubernetes/kube-apiserver-audit.log"
    pos_file "/var/log/kube-apiserver-audit.log.pos"
    tag "kube-apiserver-audit"
    <parse>
      @type "multiline"
      format_firstline "/^\\S+\\s+AUDIT:/"
      format1 /^(?<time>\S+) AUDIT:(?: (?:id="(?<id>(?:[^"\\]|\\.)*)"|ip="(?<ip>(?:[^"\\]|\\.)*)"|method="(?<method>(?:[^"\\]|\\.)*)"|user="(?<user>(?:[^"\\]|\\.)*)"|groups="(?<groups>(?:[^"\\]|\\.)*)"|as="(?<as>(?:[^"\\]|\\.)*)"|asgroups="(?<asgroups>(?:[^"\\]|\\.)*)"|namespace="(?<namespace>(?:[^"\\]|\\.)*)"|uri="(?<uri>(?:[^"\\]|\\.)*)"|response="(?<response>(?:[^"\\]|\\.)*)"|\w+="(?:[^"\\]|\\.)*"))*/
      time_format "%Y-%m-%dT%T.%L%Z"
      unmatched_lines
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_kube_apiserver
    multiline_flush_interval 5s
    path "/var/log/kube-apiserver.log"
    pos_file "/var/log/fluentd-kube-apiserver.log.pos"
    tag "kube-apiserver"
    <parse>
      @type "kubernetes"
      unmatched_lines
      expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
      time_format "%m%d %H:%M:%S.%N"
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_kube_controller_manager
    multiline_flush_interval 5s
    path "/var/log/kube-controller-manager.log"
    pos_file "/var/log/fluentd-kube-controller-manager.log.pos"
    tag "kube-controller-manager"
    <parse>
      @type "kubernetes"
      unmatched_lines
      expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
      time_format "%m%d %H:%M:%S.%N"
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_kube_proxy
    multiline_flush_interval 5s
    path "/var/log/kube-proxy.log"
    pos_file "/var/log/fluentd-kube-proxy.log.pos"
    tag "kube-proxy"
    <parse>
      @type "kubernetes"
      unmatched_lines
      expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
      time_format "%m%d %H:%M:%S.%N"
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_kube_scheduler
    multiline_flush_interval 5s
    path "/var/log/kube-scheduler.log"
    pos_file "/var/log/fluentd-kube-scheduler.log.pos"
    tag "kube-scheduler"
    <parse>
      @type "kubernetes"
      unmatched_lines
      expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
      time_format "%m%d %H:%M:%S.%N"
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_kubelet
    multiline_flush_interval 5s
    path "/var/log/kubelet.log"
    pos_file "/var/log/fluentd-kubelet.log.pos"
    tag "kubelet"
    <parse>
      @type "kubernetes"
      unmatched_lines
      expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
      time_format "%m%d %H:%M:%S.%N"
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_rescheduler
    multiline_flush_interval 5s
    path "/var/log/rescheduler.log"
    pos_file "/var/log/fluentd-rescheduler.log.pos"
    tag "rescheduler"
    <parse>
      @type "kubernetes"
      unmatched_lines
      expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
      time_format "%m%d %H:%M:%S.%N"
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_minion
    path "/var/log/salt/minion"
    pos_file "/var/log/fluentd-salt.pos"
    tag "salt"
    <parse>
      @type "regexp"
      expression /^(?<time>[^ ]* [^ ,]*)[^\[]*\[[^\]]*\]\[(?<severity>[^ \]]*) *\] (?<message>.*)$/
      time_format "%Y-%m-%d %H:%M:%S"
      unmatched_lines
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_startupscript
    path "/var/log/startupscript.log"
    pos_file "/var/log/fluentd-startupscript.log.pos"
    tag "startupscript"
    <parse>
      @type "syslog"
      unmatched_lines
    </parse>
  </source>
  <filter kubernetes.**>
    @type kubernetes_metadata
    @id filter_kube_metadata
    kubernetes_url "https://10.100.0.1:443/api"
    verify_ssl true
    ca_file ""
   skip_labels false
    skip_container_metadata false
    skip_master_url false
    skip_namespace_metadata false
    watch true
  </filter>
  <filter kubernetes.**>
    @type record_transformer
    remove_keys $['kubernetes']['labels']['app.kubernetes.io/instance'],$['kubernetes']['labels']['app.kubernetes.io/managed-by'],$['kubernetes']['labels']['app.kubernetes.io/version'],$['kubernetes']['labels']['app.kubernetes.io/name'],$['kubernetes']['labels']['app.kubernetes.io/component'],$['kubernetes']['labels']['app.kubernetes.io/part-of']
  </filter>
  <match **>
    @type opensearch
    @id out_os
    @log_level "error"
    include_tag_key true
    host "*********************************************************"
    port 443
    path ""
    scheme https
    ssl_verify true
    ssl_version TLSv1_2
    user "*******************"
    password xxxxxx
    index_name "logs"
    logstash_dateformat "%Y.%m.%d"
    logstash_format true
    logstash_prefix "logs"
    logstash_prefix_separator "-"
    <buffer>
      flush_thread_count 1
      flush_mode interval
      flush_interval 60s
      chunk_limit_size 8M
      total_limit_size 512M
      retry_max_interval 30
      retry_timeout 72h
      retry_forever false
    </buffer>
    <endpoint>
      url https://vpc-pm-hpl-opensearch-5jlny6in26kkudopanxxbzsi74.us-east-1.es.amazonaws.com
      region "us-east-1"
    </endpoint>
  </match>
</ROOT>
2025-04-01 08:11:06 +0000 [info]: starting fluentd-1.18.0 pid=7 ruby="3.2.6"
2025-04-01 08:11:06 +0000 [info]: spawn command to main:  cmdline=["/usr/local/bin/ruby", "-Eascii-8bit:ascii-8bit", "/fluentd/vendor/bundle/ruby/3.2.0/bin/fluentd", "-c", "/fluentd/etc/fluent.conf", "-p", "/fluentd/plugins", "--gemfile", "/fluentd/Gemfile", "-r", "/fluentd/vendor/bundle/ruby/3.2.0/gems/fluent-plugin-opensearch-1.1.5/lib/fluent/plugin/opensearch_simple_sniffer.rb", "--under-supervisor"]
2025-04-01 08:11:08 +0000 [info]: #0 init worker0 logger path=nil rotate_age=nil rotate_size=nil
2025-04-01 08:11:08 +0000 [info]: adding match in @FLUENT_LOG pattern="fluent.**" type="null"
2025-04-01 08:11:08 +0000 [info]: adding filter pattern="kubernetes.**" type="kubernetes_metadata"
2025-04-01 08:11:08 +0000 [info]: adding filter pattern="kubernetes.**" type="record_transformer"
2025-04-01 08:11:08 +0000 [info]: adding match pattern="**" type="opensearch"
2025-04-01 08:11:09 +0000 [info]: adding source type="systemd"
2025-04-01 08:11:09 +0000 [info]: adding source type="systemd"
2025-04-01 08:11:09 +0000 [info]: adding source type="systemd"
2025-04-01 08:11:09 +0000 [info]: adding source type="prometheus"
2025-04-01 08:11:09 +0000 [info]: adding source type="prometheus_output_monitor"
2025-04-01 08:11:09 +0000 [info]: adding source type="tail"
2025-04-01 08:11:09 +0000 [info]: adding source type="tail"
2025-04-01 08:11:09 +0000 [info]: adding source type="tail"
2025-04-01 08:11:09 +0000 [info]: adding source type="tail"
2025-04-01 08:11:09 +0000 [info]: adding source type="tail"
2025-04-01 08:11:09 +0000 [info]: adding source type="tail"
2025-04-01 08:11:09 +0000 [info]: adding source type="tail"
2025-04-01 08:11:09 +0000 [info]: adding source type="tail"
```

When I restart the pod, it recovers automatically, but I want to know what exactly the issue is.

### To Reproduce

Not sure how the issue occured

### Expected behavior

There shouldn't be any warnings

### Your Environment

```markdown
- Tag of using fluentd-kubernetes-daemonset: fluentd-kubernetes-daemonset:v1.18-debian-opensearch-1
```

### Your Configuration

```apache
Mentioned in description
```

### Your Error Log

```shell
Mentioned in description
```

### Additional context

_No response_

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[in_systemd_docker] Error moving to next Journal entry: Systemd::JournalError: Cannot assign requested address #1573

Describe the bug

To Reproduce

Expected behavior

Your Environment

Your Configuration

Your Error Log

Additional context

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[in_systemd_docker] Error moving to next Journal entry: Systemd::JournalError: Cannot assign requested address #1573

Description

Describe the bug

To Reproduce

Expected behavior

Your Environment

Your Configuration

Your Error Log

Additional context

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions