When running `test:all` Ceedling should continue to run the remaining tests if one test crashes

[0ge]

From my understanding, if one of the tests crashes, Ceedling will simply stop running the remaining tests. Is there a reason for this? I find it annoying because if I have multiple tests that crashes then I can only address one issue at time, having to re-run the tests after each fix to find the next error. Is there a reason to this?

What I would prefer Ceedling to do (although I appreciate there is some work to it) is that if a test crashes, Ceedling would set the remaining tests of that executable as failed and then continue with the other tests. It should also warn properly about when it does this.

[Letme]

Crashing executable is not really something you have for everyday life - it is usually a design flaw. If we go this way, I would propose a flag that would enable this feature, but keep the default as it is now.

[0ge]

Out of curiosity - why do you prefer stopping the testing instead of carrying on?

[Letme]

With current implementation ceedling does not report missing unit tests so in fact if you look at file it might have all success, but file crashes. And since error code is not really visible in terminal you need to know number of all your unit tests to figure out something crashed in between. So I rather solve the issue there and then.

I can't really see an example where single change would cause multiple runners to crash and not fail (except design flaw).

[0ge]

Perhaps they are run on a server that just pulled changes from several developers?
For me, its more logical if crashing code is handled (as much as possible) as a failed test.

Maybe Ceedling does not/can not parse any output if a test crashes? Otherwise, it could provide some better information on which tests passed in which (were marked as) fails.

[Letme]

If runner crashes then all tests not reported after crash do not count towards overall, nor they are displayed as failures (there is an issue for this open). Fact is that no data stream is coming out of runner so Ceedling can't guess what happened. So if last test case in runner results in the crash, then good luck finding that out in report of 500 test cases and that is why I rather see a crash.

On CI I could agree (that is why I want a flag) that it should run. But when error code issue was not fixed I also seen a problem where runner crashed and ceedling still returned 0. I would assume this is now fixed from what you are saying. When developing however the crash is an obvious indication something went terribly wrong (so another level of wrong).

[methril]

I vote for a flag for carry on the tests, but also log or report the crash as test fail.

[lukzeg]

Hi @0ge , @Letme , @methril ,

I created a PR'st (#740) which might help solve your problem.
Mostly if you use gdb and you are able to using it to collect backtrace from your tests exeuction, after updating project.yml:

:project:
  :use_backtrace_gdb_reporter: TRUE

...

:test_runner:
  :cmdline_args: true

the test file runner which crash, will be executed in different way. After discovering crash, the Ceedling will collect information about all tests included in this runner, and basing on applied/not applied test_case filters, will trigger each of test case separately.

The test which cause crash, will be marked with FAILURE and backtrace will be added as reason of the failure.

[Letme]

Not exactly what was discussed but it does solve partially this issue. If using GDB to run tests then it would make sense to have such failure marking and printing of backtrace, since we can have it. What here was wanted is

Test A = pass
Test B = Segfault, crash (run gdb)
Test C = pass

In above case, when we get crash in Test B, the Test C is not run (so we cant really say it passed or failed). I think GDB could bypass that and run Test C, along with above proposal to print backtrace in Test B and mark it as failure, while also getting Ceedling to normally complete and mark Test A as pass. But I did not check the PR for this case.

[lukzeg]

Hi @Letme ,

To clarify. If we have situation in test_file

Test A = pass
Test B = Segfault, crash (run gdb)
Test C = pass

At first, when you will execute all tests:

ceedling test:all

The engine will at beginning try to execute all tests ( from one test file which cause crash). If the Ceedling engine will discover segmentation fault in Test B = Segfault, crash (run gdb), it will:

• collect list of tests
• filter tests which should be executed applying changes passed by --test_case/ --exclude_test_case arguments
• start to trigger under gdb each of test separately and collect data from it.
• Collect test results from all of tests separately and combine all of them in one test report.

Running each of test under gdb gives you possibility to be sure that lets say, Test C, which should pass, is causing crash as well.
As well, from what I observed during my work, I noticed, that sometimes running tests separately, cause some strange problems, dependencies(author of the tests sometimes make mistakes) which we does not see when we run them all at once.

Anther test files(test_runners) will be executed in normal way, without running under gdb, till another segmentation fault issue will not be discovered. After that, the logic with gdb will be applied again for crashing test runner.

I can rework my PR'st to run each of test case normally, and when crash happens re-trigger it under gdb, but this logic will extend time execution of test cases.

Feel free to check my changes, and let me know, what can I improve to fit to your needs.

[Letme]

Officially if there is runtime problem in Test B then the setup/teardown should clean it. Otherwise there is a problem for future anyway.

I think when you have flag for gdb_backtrace, you assume that there will be longer test execution. Now what would happen in your case (assuming), you will run tests in GDB until Test B, but not after it (since you will be happy that you discovered the segfaulted test). I am just saying that in that case it would be useful to also run tests after the segfaulted test case in the same file, so you already got request to run them all.

In my case Test A-C are all in same file (same testRunner), so you should make sure gdb only runs with the file that got segfaulted, not all other tests as well (as execution time will be much longer indeed). And that is the granularity I think Ceedling cannot provide, since it uses concept of TestRunner, so if TestRunner segfaults, then it can only run to that point. But granularity which Ceedling can provide is that when there are multiple TestRunners, and one of them fails (segfault) it should still run all the others (the one that segfaults should be repeated with gdb to get a backtrace, but others should be run normally). I think that is the part you can do now.

Anyway, I will try to review your PR and have a discussion there. This is just so that this ticket gets a bit more details.

[lukzeg]

Hi @Letme ,

Feel free to look at it. Basing on your replay, I think all of mentioned by you cases are handled by the changes.

Sample project:

calc.c

#include <stdint.h>
#include <stdio.h>
#include <signal.h>

int minus(int a, int b) {
    return a - b;
}

int add(int a, int b) {
    return a + b;
}

int multiplication(int a, int b) {
    raise(SIGSEGV);
    return a * b;
}

int divider(int a, int b) {
    return a / b;
}

cal.h

#ifndef CAL_H
#define CAL_H

int add(int a, int b);

int minus(int a, int b);

int multiplication(int a, int b);

int divider(int a, int b);

#endif /* CAL_H */

test/test_add.c

#include "../src/cal.h"
#include "unity.h"

void
setUp(void)
{
}

void
tearDown(void)
{
}

void
test_add_success(void)
{
    TEST_ASSERT_EQUAL_UINT8(3, add(1, 2));
}

void
test_minus_success(void)
{
    TEST_ASSERT_EQUAL_UINT8(2, minus(2, 1));
}

test/test_multiplication.c

#include "../src/cal.h"
#include "unity.h"

void
setUp(void)
{
}

void
tearDown(void)
{
}

void
test_multiplication_success(void)
{
    TEST_ASSERT_EQUAL_UINT8(2, multiplication(1, 2));
}

void
test_divider_failure(void)
{
    TEST_ASSERT_EQUAL_UINT8(1, divider(2, 2));
}

Basing on this sample if we will call:

ceedling gcov:all

you will receive:

ceedling gcov:all


Test 'test/test_add.c'
----------------------
Generating runner for test_add.c...
Compiling test_add_runner.c...
Compiling test_add.c...
Compiling cal.c with coverage...
Compiling unity.c...
Compiling cmock.c...
Linking test_add.out...
Running test_add.out...


Test 'test/test_multiplication.c'
---------------------------------
Generating runner for test_multiplication.c...
Compiling test_multiplication_runner.c...
Compiling test_multiplication.c...
Linking test_multiplication.out...
Running test_multiplication.out...

-----------------
GCOV: TEST OUTPUT
-----------------
[test/test_multiplication.c]
  - "test_multiplication.c:17:test_multiplication_success:FAIL: [Thread debugging using libthread_db enabled]$$$Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".$$$$$$Program received signal SIGSEGV, Segmentation fault.$$$__pthread_kill_implementation (no_tid=0, signo=11, threadid=140737351477056) at ./nptl/pthread_kill.c!!!44$$$44	./nptl/pthread_kill.c!!! No such file or directory.$$$#0  __pthread_kill_implementation (no_tid=0, signo=11, threadid=140737351477056) at ./nptl/pthread_kill.c!!!44$$$#1  __pthread_kill_internal (signo=11, threadid=140737351477056) at ./nptl/pthread_kill.c!!!78$$$#2  __GI___pthread_kill (threadid=140737351477056, signo=signo@entry=11) at ./nptl/pthread_kill.c!!!89$$$#3  0x00007ffff7dbb476 in __GI_raise (sig=11) at ../sysdeps/posix/raise.c!!!26$$$#4  0x0000555555557c5f in multiplication (a=1, b=2) at src/cal.c!!!26$$$#5  0x0000555555557af8 in test_multiplication_success () at "

-------------------------
GCOV: FAILED TEST SUMMARY
-------------------------
[test/test_add.c]
  Test: test_minus_success
  At line (23): "Expected 2 Was 1"

[test/test_multiplication.c]
  Test: test_multiplication_success
  At line (17): "[Thread debugging using libthread_db enabled]
Using host libthread_db library '/lib/x86_64-linux-gnu/libthread_db.so.1'.

Program received signal SIGSEGV, Segmentation fault.
__pthread_kill_implementation (no_tid=0, signo=11, threadid=140737351477056) at ./nptl/pthread_kill.c:44
44	./nptl/pthread_kill.c: No such file or directory.
#0  __pthread_kill_implementation (no_tid=0, signo=11, threadid=140737351477056) at ./nptl/pthread_kill.c:44
#1  __pthread_kill_internal (signo=11, threadid=140737351477056) at ./nptl/pthread_kill.c:78
#2  __GI___pthread_kill (threadid=140737351477056, signo=signo@entry=11) at ./nptl/pthread_kill.c:89
#3  0x00007ffff7dbb476 in __GI_raise (sig=11) at ../sysdeps/posix/raise.c:26
#4  0x0000555555557c5f in multiplication (a=1, b=2) at src/cal.c:26
#5  0x0000555555557af8 in test_multiplication_success () at test/test_multiplication.c:17
#6  0x00005555555577dc in run_test (func=0x555555557acf <test_multiplication_success>, name=0x555555561068 'test_multiplication_success', line_num=15) at build/test/runners/test_multiplication_runner.c:63
#7  0x0000555555557a12 in main (argc=3, argv=0x7fffffffdbf8) at build/test/runners/test_multiplication_runner.c:94
"

--------------------------
GCOV: OVERALL TEST SUMMARY
--------------------------
TESTED:  4
PASSED:  2
FAILED:  2
IGNORED: 0


---------------------------
GCOV: CODE COVERAGE SUMMARY
---------------------------
cal.c Lines executed:66.67% of 9
cal.c No branches
cal.c Calls executed:0.00% of 1
cal.c Lines executed:66.67% of 9

---------------------
BUILD FAILURE SUMMARY
---------------------
Unit test failures.

Of course if you would like to run only one testRunner, you can call Ceedling with:

ceedling gcov:test_multiplication

or even one test case for testRunner:

ceedling gcov:test_multiplication --test_case=test_multiplication_success

But yes, if you will have some time, please look at my proposal, and I will be glad for any replay from your side.

[Letme]

Yes, based on this output, they are. I already commented on the PR, but based on above output only rephrasing of some wording might be more helpful.