Satellite sync failure

[ymartin-ovh]

Hello

When starting a new icinga2 instance inside a zone, icinga2 fails to start because config validation fails:
group in hosts are missing.

Looking on satellite filesystem (/var/lib/icinga2/api/packages/_api/bdd5cdff-6e46-4795-a2cf-64ef56d3b397/conf.d):

• hosts/*.conf are present as expected
• hostgroups/*.conf are absent (config exists on both masters).

I expect that icinga satellites load package" and hostgroups before hosts.

To fix this, I do:

• systemctl stop icinga2
• rm -f /var/lib/icinga2/api/packages/_api/bdd5cdff-6e46-4795-a2cf-64ef56d3b397/conf.d/hosts/*.conf
• chattr +i /var/lib/icinga2/api/packages/_api/bdd5cdff-6e46-4795-a2cf-64ef56d3b397/conf.d/hosts
• systemctl start icinga2

When hostgroups config is sync:

• systemctl stop icinga2
• chattr -i /var/lib/icinga2/api/packages/_api/bdd5cdff-6e46-4795-a2cf-64ef56d3b397/conf.d/hosts
• systemctl start icinga2

I didn't experience this before icinga2 2.13.7-1+debian11

Regards

[Al2Klimov]

Hello Martin!

How did you create those hosts and groups in the first place?

Best,
A/K

[ymartin-ovh]

Hello

Hosts and groups were created through API calls (masters).

[Al2Klimov]

Please share their config.

[ymartin-ovh]

From a satellite, I have something like that.

icinga2 tried to create host before hostgroup and was unhappy to activate host object because hostgroup was missing.
As a workaround, I chattr+i the host folder to ensure that icinga2 sync packages and hostgroups first.

/var/lib/icinga2/api/packages/_api/164466ca-64c1-4e63-b4e0-eaaa65d9c493/conf.d/hosts/foobar.conf:

object Host "foobar" {
        import "webservers-host"

        address = "10.19.65.220"
        groups = [ "www-hosts" ]
        vars["delivery_status"] = "delivered"
        version = 1673339012.627591
        zone = "labeu"
}

/var/lib/icinga2/api/packages/_api/164466ca-64c1-4e63-b4e0-eaaa65d9c493/conf.d/hostgroups/www-hosts.conf

object HostGroup "www-hosts" {
        version = 1664280940.247957
        zone = "global-templates"
}

[Al2Klimov]

Why did you put them in different zones?

[ymartin-ovh]

I want to have a group with all www-hosts that regroups all regions.

I have this on icinga2 config too (masters & satellites)
/etc/icinga2/zones.conf:

object Zone "global-templates" {
  global = true
}

host and groups are synced to satellite labeu region.

[Al2Klimov]

Have you tried https://icinga.com/docs/icinga-2/latest/doc/17-language-reference/#group-assign instead?

[ymartin-ovh]

I will try this.

However, I think that host activation should rely on hostgroup activation.
I will create host.vars.groups and add an assign relashionship on all groups. I have the impression I re-implement what Icinga2 is doing with host.groups list.

Regards

[ymartin-ovh]

Hello @Al2Klimov

I can't find a way to create group object with assign rule with API. Do you know how to do this ?

Regards

[Al2Klimov]

I'm afraid that's impossible via API.

[ymartin-ovh]

Hum,

I have my satellite fresh start issue with 2.14.0 too.

On the first run, I can see the following error in logs:

icinga2[3965374]: icinga2: /usr/include/boost/smart_ptr/intrusive_ptr.hpp:199: T* boost::intrusive_ptr<T>::operator->() const [with T = icinga::Host]: Assertion `px != 0' failed.
icinga2[3965374]: Caught SIGABRT

Build information:
  Compiler: GNU 10.2.1
  Build host: runner-hh8q3bz2-project-575-concurrent-0
  OpenSSL version: OpenSSL 1.1.1n  15 Mar 2022

Application information:

General paths:
  Config directory: /etc/icinga2
  Data directory: /var/lib/icinga2
  Log directory: /var/log/icinga2
  Cache directory: /var/cache/icinga2
  Spool directory: /var/spool/icinga2
  Run directory: /run/icinga2

Old paths (deprecated):
  Installation root: /usr
  Sysconf directory: /etc
  Run directory (base): /run
  Local state directory: /var

Internal paths:
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid

Stacktrace:
 0# icinga::Application::SigAbrtHandler(int) in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
 1# 0x00007F6570AED140 in /lib/x86_64-linux-gnu/libpthread.so.0
 2# gsignal in /lib/x86_64-linux-gnu/libc.so.6
 3# abort in /lib/x86_64-linux-gnu/libc.so.6
 4# 0x00007F65705FD40F in /lib/x86_64-linux-gnu/libc.so.6
 5# 0x00007F657060C662 in /lib/x86_64-linux-gnu/libc.so.6
 6# 0x000055DDCFA96793 in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
 7# icinga::Comment::OnAllConfigLoaded() in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
 8# 0x000055DDCF7E1645 in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
 9# icinga::WorkQueue::RunTaskFunction(std::function<void ()> const&) in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
10# 0x000055DDCF7F462C in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
11# icinga::WorkQueue::RunTaskFunction(std::function<void ()> const&) in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
12# icinga::WorkQueue::WorkerThreadProc() in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
13# 0x00007F657110B787 in /lib/x86_64-linux-gnu/libboost_thread.so.1.74.0
14# 0x00007F6570AE1EA7 in /lib/x86_64-linux-gnu/libpthread.so.0
15# clone in /lib/x86_64-linux-gnu/libc.so.6

[Al2Klimov]

@julianbrost Please say you can decode those addresses via your recent gdb magic. 😭

[ymartin-ovh]

I'm bissecting, I have the crash with this stacktrace since 2.13.6.

[ymartin-ovh]

crash report with 2.13.6:
report.1693489668.561474-2.13.6.txt