Problem notification with OK State

[BT-Danny]

Describe the bug

We encountered a weird behaviour several times:

Sometimes, when states changes too often, Icinga seems to be confused by all the state changes. Youcan see in the given example how often the state changes, but the criticals are always in Soft States. When it changes to Hard State OK, it seems to be confused by the the Check Output, which is critical, and it sends a problem notification with State OK.

The Service should be in State Critical, according to the Check

To Reproduce

1. Service is flapping
2. Service exits the flapping state with state OK
3. Check is critical but state is OK
4. Icinga sends

Expected behavior

Usually icinga shouldn't send a notification, when the service state is "OK". There should be an internal evaluation, since a Problem isn't corresponding an OK state

Screenshots

Your Environment

• Version used (icinga2 --version): r2.13.6-1
• Operating System and version: RHEL 7 (7.9)
• Enabled features (icinga2 feature list): api checker graphite ido-mysql mainlog notification
• Icinga Web 2 version and modules (System - About): 2.11.3
• 2 Master and 2 Satellites each Master
• Master and Sats are in different Clouds
• 3 Zones: 1 Zone with master and one zone each cloud

[julianbrost]

Do you know what the check actually returned at the time in question? So can you tell for sure whether the state or the output is wrong?

Can you please share available logs from that time?

[BT-Danny]

Well, actually the state is wrong, since the check seems to return the correct exit code, but Icinga seems to misinterpret that. Maybe because of the often statechanges?

icinga2.log:

[2023-03-09 06:45:27 +0100] information/Checkable: Checkable 'hostXY!powershell-cpu' has 3 notification(s). Checking filters for type 'Problem', sends will be logged.
[2023-03-09 06:45:27 +0100] information/Notification: Sending 'Problem' notification 'hostXY!powershell-cpu!SMS Service Notification 7x24' for user 'SMS'
[2023-03-09 06:45:27 +0100] information/Notification: Sending 'Problem' notification 'hostXY!powershell-cpu!JIRA Notification 7x24 - Without Certificate Services' for user 'userXY'
[2023-03-09 06:45:27 +0100] information/Notification: Completed sending 'Problem' notification 'hostXY!powershell-cpu!SMS Service Notification 7x24' for checkable 'hostXY!powershell-cpu' and user 'SMS' using command 'sms-service-notification'.
[2023-03-09 06:45:27 +0100] information/Notification: Completed sending 'Problem' notification 'hostXY!powershell-cpu!JIRA Notification 7x24 - Without Certificate Services' for checkable 'hostXY!powershell-cpu' and user 'userXY' using command 'jira-service-notification'.
[2023-03-09 06:45:28 +0100] warning/PluginNotificationTask: Notification command for object 'hostXY!powershell-cpu' (PID: 14004, arguments: '/opt/icinga2/notifications/jira_notifications/monitoring_jira_notification.py' '-I' '12595' '-P' 'Blocker' '-S' 'OK' '-a' '' '-cf' 'customfield_13840,customfield_14040' '-cv' '1460,' '-h' 'hostXY' '-ipv4' 'some_ip' '-n' 'PROBLEM' '-o' '[CRITICAL] CPU Load [CRITICAL] Core Total (100%)

The last log is our custom notification script, which will look for tickets in the ticketsystem and if it doesn't find anything, it will create a new ticket. What's interesting about it are the argument '-S' which indicates the State of the Service/Host and '-n' which indicates the issuetype. The state OK doesn't fit the issuetype problem.

We implemented an evaluation in the script which compares the state and the issuetype, and if they don't match, the script will exit without sending out informations to the ticketingsystem:

2023-03-09 06:45:28,036|INFO|monitoring_jira_notification.py:50|Function:check_arguments|[14004] Notification type PROBLEM and state OK do not match. Script will exit

Unfortunately we didn't imlpement this in the sms-notification script yet, which made us aware of this bug.

We are very confused by the acutal trigger of the notification. There isn't actually any reason for the service to trigger the notification

Greetings:
Daniel

[BTMichel]

Update
We have faced the exact error again today, where a Recovery Notification was sent out, even though there was no problematic hard state reached before:

In this screenshot you can clearly see, that the state was already OK at 13/03 04:04:18 but there was still a Recovery Notification sent out on the next check at 09:54:35:

Could this be a sync issue between our two masters?
We are facing incorrect notifications more and more often now.

[BT-Danny]

Hello @julianbrost

did you found any solutions for the problem?
As BTMichel mentioned are we facing this problem pretty often right now.

Greetings:
Daniel

[BT-Danny]

Hello,

we are facing this problem now more often than ever. We evaluated some informations now:

We just received several notifications this morning regarding a "problem" which was adressed as "OK":

Regarding the second screenshot, we checked the logs and found this:
[2023-04-28 13:32:48 +0200] information/Notification: Sending reminder 'Problem' notification 'hostname!servicename!Notification 7x24' for user 'Contact'

As you can see, it evaluates it as Problem, but it's shown as "OK" in icingaweb.

We saw, that in "inspect" the state was set to "2", which should be "critical".
I then did "process check result" to manually reset it, but first it just ignored it and it stayed with state: 2
After that, I did "process check result" with "OK" again and it reset the State to 0, which should be "OK"

We also checked the database, and it showed the same results as icingaweb history.

Greetings:
Daniel

[BT-Danny]

Hello everybody,

did you already found anything regarding the problem?

Greetings:
Daniel

[BT-Danny]

Hello everybody,
friendly reminder :)

Greetings:
Daniel

[BT-Danny]

Hello everybody,

we recently encountered same problems again:

As you can see, there are some weird messages regarding a problem and an OK notification.

Much regards:
Daniel

[BT-Basit]

Hi there,

we just experienced the problem again. Icingaweb shows that there was a notification but it wasn't sent out to any contact:

icinga2.log shows that the notification was sent out to a specific contact and produced a ticket in our system:

[2024-06-13 14:41:59 +0200] information/Checkable: Checkable '<host>!<service>' has 1 notification(s). Checking filters for type 'Problem', sends will be logged. [2024-06-13 14:43:21 +0200] information/Notification: Sending reminder 'Problem' notification '<host>!<service>!<notification>' for user '<my contact>' [2024-06-13 14:43:21 +0200] information/Notification: Completed sending 'Problem' notification '<host>!<service>!<notification>' for checkable '<host>!<service>' and user '<my contact>' using command '<notification command>'.

On top of that the database table icinga_statehistory shows no state change after 12th june:

MySQL [icinga]> select * from icinga_statehistory where object_id = 77877 and state_time >= '2024-06-12 00:00:00' order by state_time desc \G *************************** 1. row *************************** statehistory_id: 24350989 instance_id: 1 state_time: 2024-06-12 07:04:14 state_time_usec: 601166 object_id: 77877 state_change: 1 state: 0 state_type: 1 current_check_attempt: 1 max_check_attempts: 3 last_state: 2 last_hard_state: 0 output: PING OK - Packet loss = 0%, RTA = 18.29 ms long_output: check_source: icinga1 endpoint_object_id: 2643 *************************** 2. row ***************************

I don't understand what is happening here and why Icingaweb, icinga2.log and the database all show different behaviour.

[K0nne]

Hello!

we have the same problem:

icinga2: 2.13.10
icingadb: 1.2.1

Icinga sometimes sends notifications with a wrong servicestate. The problem occured in our environment once per day in the past 60 days. We have 34k hosts with 226k services and found this by chance in our exernal log management:

check_system_basics

problem

recovery

At first I thought it was a bug in check_system_basics, but then I found other occurences with other checks as well:

ping4

problem & recovery

This was the only statechange during this day. No flapping involved:

[K0nne]

It could be reproduceable with a huge amount of check_random, a short retry_interval and max_check_attempt = 1.

[K0nne]

I noticed another interesting detail. The plugin output from the ping4's problem notification seems to have concatinated the plugin output from a past state-change. The UI shows it as well: