Symbol report verification report #2544

pawelrutkaq · 2026-02-04T13:26:31Z

No description provided.

github-actions · 2026-02-04T13:26:43Z

⚠️ Docs-as-Code version mismatch detected
Please check the CI build logs for details and align the documentation version with the Bazel dependency.

github-actions · 2026-02-04T13:31:01Z

The created documentation from the pull request is available at: docu-html

docs/score_tools/symbol_report_blanket.rst

aschemmel-tech

see inline comments, proposal: take over the failure modes and mitigations from https://public-docs.ferrocene.dev/main/certification/core/safety-plan/tools.html#code-coverage

docs/score_tools/symbol_report_blanket.rst

aschemmel-tech · 2026-02-04T15:33:37Z

docs/score_tools/symbol_report_blanket.rst

+
+Result
+~~~~~~
+Symbol report and blanket do not require qualification for use in safety-related software development according to ISO 26262.


as stated by masc2023 - without additional measures I cannot agree to this result

Reworked table

docs/score_tools/symbol_report_blanket.rst

pahmann · 2026-02-04T16:31:50Z

docs/score_tools/symbol_report_blanket.rst

+Safety evaluation
+-----------------
+This section outlines the safety evaluation of symbol report and blanket for its use within the S-CORE project. This evaluation assumes that the Rust compiler is
+qualified and output of coverage data in `.profraw` format is correct. Due to that, we solely focus on post processing that is done by symbol report and blanket only.


Will be use and measure the coverage based on the .profraw somewhere else? This can have impact on the tool classification, which is the reason I am asking.

No, these tools are the only measurement for coverage of rust

If there is no check by other tools and the tool is used to argue safety, plus manual checking is not feasible to guarantee operation is 100% the tool confidence level should decrease.

I would like to see more about the available testing and quality measures used to operate the tool to have improved argument why the likelihood of errors are reduced due to the process and quality rigor applied to the tool.

docs/score_tools/symbol_report_blanket.rst

pawelrutkaq · 2026-02-05T10:28:07Z

@pahmann @aschemmel-tech @masc2023 tried to resolve comments please recheck

PandaeDo · 2026-02-05T14:20:35Z

docs/score_tools/symbol_report_blanket.rst

+     - Overreporting, could result in testing gap.
+     - yes
+     - | Likelihood of such an error low due to wide usage of the tool (many S-CORE modules and other projects like ferrocene)
+       | Additionally, every new tool release is tested by running tests in prepared integration testsuite to detect such errors. (PROPOSAL POINT)


I would also add the expectation of the tester/user. So a failure can be detected.

If we have this as Proposal point this will be a measure on the low confidence. I would say, that we cannot simply say that the detection is sufficient just because we plan to write a test for this in the future.

So we need to evaluate this report assuming future. This is required for feroceen on gives us estimates of price we need to put into this. So we shall agree that ie this point is neded and sufficient and we will request it from them.

docs/score_tools/symbol_report_blanket.rst

PandaeDo · 2026-02-05T14:23:45Z

docs/score_tools/symbol_report_blanket.rst

+     -  A function is not being considered, although it is part of the certified subset
+     - yes
+     - | `symbol-report` is developed to use exactly the same information as the compiler
+       | Additionally, every new tool release is tested by running tests in prepared integration testsuite to detect such errors. (PROPOSAL POINT)


Also in this case the user expectation will lead in some cases to a failure detection.

docs/score_tools/symbol_report_blanket.rst

masc2023 · 2026-02-09T08:13:25Z

docs/score_tools/symbol_report_blanket.rst

+----------------------------
+Installation
+~~~~~~~~~~~~
+| To add the Code coverage to your project or module follow guidelines in WIP


Is there a link to WIP available, or nothing at the moment, then add a issue to follow up with that and link it here

docs/score_tools/symbol_report_blanket.rst

masc2023 · 2026-02-09T08:15:57Z

docs/score_tools/symbol_report_blanket.rst

+     - Overreporting, could result in testing gap.
+     - yes
+     - | Likelihood of such an error low due to wide usage of the tool (many S-CORE modules and other projects like ferrocene)
+       | Additionally, every new tool release is tested by running tests in prepared integration testsuite to detect such errors. (PROPOSAL POINT)


What means Proposal Point? Additional measure to reduce the risk? Then add issue, link it here, that we can follow up as it then supports the argumentation why the rating is high
If Proposal Point is additional measure, update the table with additional measure needed yes

Means that this can be an mitigation if Safety collegues things its good to have/needed.

docs/score_tools/symbol_report_blanket.rst

pahmann

Compared to the first draft the new version improved. Still it feels not mature and more an early state of evaluation. I do not have the confidence that tool qualification is not needed based on the available information.

Remark: Any information regarding the TCL from the source at Ferroscene? They have a bunch of public information on safety as they claim.

pahmann · 2026-02-11T12:28:10Z

docs/score_tools/symbol_report_blanket.rst

+     - Overreporting, could result in testing gap.
+     - yes
+     - | Likelihood of such an error low due to wide usage of the tool (many S-CORE modules and other projects like ferrocene)
+       | Additionally, every new tool release is tested by running tests in prepared integration testsuite to detect such errors. (PROPOSAL POINT)


If we have this as Proposal point this will be a measure on the low confidence. I would say, that we cannot simply say that the detection is sufficient just because we plan to write a test for this in the future.

pahmann · 2026-02-11T12:30:48Z

docs/score_tools/symbol_report_blanket.rst

+     - False-positive: A function is reported as covered, although it is not covered
+     - Overreporting, could result in testing gap.
+     - yes
+     - | Likelihood of such an error low due to wide usage of the tool (many S-CORE modules and other projects like ferrocene)


The argument is very weak. It is a proven in use argumentation under the assumption that the use case is identical and the usage as well. Already in the case that there are tool configuration options this weakens your argument. Otherwise we could simply say. "Oh there are no issues in Linux and the likelihood is low, because it is used in so many devices".

I splited this as not enought and added additional measure column info.

pahmann · 2026-02-11T12:32:43Z

docs/score_tools/symbol_report_blanket.rst

+Safety evaluation
+-----------------
+This section outlines the safety evaluation of symbol report and blanket for its use within the S-CORE project. This evaluation assumes that the Rust compiler is
+qualified and output of coverage data in `.profraw` format is correct. Due to that, we solely focus on post processing that is done by symbol report and blanket only.


If there is no check by other tools and the tool is used to argue safety, plus manual checking is not feasible to guarantee operation is 100% the tool confidence level should decrease.

I would like to see more about the available testing and quality measures used to operate the tool to have improved argument why the likelihood of errors are reduced due to the process and quality rigor applied to the tool.

docs/score_tools/symbol_report_blanket.rst

PandaeDo

@alekseyborisyukvalidas Could you please support Pawel with the verification report?

masc2023 · 2026-02-12T09:55:59Z

docs/score_tools/symbol_report_blanket.rst

+     - Overcounting: Total number of functions is too low
+     -  A function is not being considered, although it is part of the certified subset
+     - yes
+     - `symbol-report` is developed to use exactly the same information as the compiler


May you need to adapt here like in the cells above, NO

masc2023 · 2026-02-12T09:56:59Z

docs/score_tools/symbol_report_blanket.rst

+     - Line that can be executed not being reported as executable
+     - Underreporting, code that should be tested may not being tested
+     - yes
+     - `blanket` warns if a function has no executable line


May you need to adapt here like in the cells above, NO, because you argue additional measure need

masc2023 · 2026-02-12T09:58:14Z

docs/score_tools/symbol_report_blanket.rst

+     - yes
+     - `blanket` warns if a function has no executable line
+     - yes
+     - **Yes**. Additionally, every new tool release is tested by running tests in prepared integration testsuite to detect such errors.


may in addition you add a comment note, which explains in some sentences, what this integration testsuite does, because you mentioned in in all argumentations

Without this explanation it is not possible to judge if it is sufficient. It is also not clear who will develop and run this "integration testsuite". Ferrocene states for this Malfunction "(Future work) End-to-end test that ensures the correct lines are being reported as executable" which could be interpreted that they would implement/run such test.

aschemmel-tech

see inline comments

aschemmel-tech · 2026-02-12T10:18:18Z

docs/score_tools/symbol_report_blanket.rst

+     - high
+   * - 3
+     - Overcounting: Total number of functions is too low
+     -  A function is not being considered, although it is part of the certified subset


Is this a use case for S-CORE at all? What would we need the "Total number of functions" for? The relevant failure for S-CORE is described in Malfunction number 1 already, could be removed in my opinion.

aschemmel-tech · 2026-02-12T10:18:39Z

docs/score_tools/symbol_report_blanket.rst

+     - **Yes**. Every new tool release is tested by running tests in prepared integration testsuite to detect such errors.
+     - high
+   * - 4
+     - Undercounting: Total number of functions is too high


Is this a use case for S-CORE at all? What would we need the "Total number of functions" for? The relevant failure for S-CORE is described in Malfunction number 2 already, could be removed in my opinion.

aschemmel-tech · 2026-02-12T10:29:27Z

docs/score_tools/symbol_report_blanket.rst

+     - Overcounting: Total number of functions is too low
+     -  A function is not being considered, although it is part of the certified subset
+     - yes
+     - `symbol-report` is developed to use exactly the same information as the compiler


symbol-report is mentioned as a mitigation measure which checks is the compiler has an error - unclear what compiler error this refers to.

aschemmel-tech · 2026-02-12T10:33:53Z

docs/score_tools/symbol_report_blanket.rst

+     - yes
+     - `blanket` warns if a function has no executable line
+     - yes
+     - **Yes**. Additionally, every new tool release is tested by running tests in prepared integration testsuite to detect such errors.


Without this explanation it is not possible to judge if it is sufficient. It is also not clear who will develop and run this "integration testsuite". Ferrocene states for this Malfunction "(Future work) End-to-end test that ensures the correct lines are being reported as executable" which could be interpreted that they would implement/run such test.

aschemmel-tech · 2026-02-12T10:45:19Z

docs/score_tools/symbol_report_blanket.rst

+
+Safety evaluation
+-----------------
+This section outlines the safety evaluation of `symbol report` and `blanket` for its use within the S-CORE project. This evaluation assumes that the Rust compiler is


I see that this is intended to check if we need to have toolqualification of the tools symbol report and blanket developed by Ferrocene. But actually we would need an evaluation of the full toolchain including the -Cinstrument-coverage function and the llvm-profparser. The argumentation of Ferrocene that these are " widely used" is not enough to consider these "safe enough" - So maybe the "integration testsuite" should also cover these tools.

So, finally, if we come not to an common agreement, we would need to have tool qualification for the complete chain, as we request it here according,
https://eclipse-score.github.io/process_description/main/process_areas/tool_management/guidance/tool_management_guideline.html#gd_guidl__tool_qualification
The tool qualification shall be based on the method validation of the software tool.

Symbol report verification report

024a364

pawelrutkaq requested a review from a team as a code owner February 4, 2026 13:26

masc2023 requested changes Feb 4, 2026

View reviewed changes

docs/score_tools/symbol_report_blanket.rst Show resolved Hide resolved

docs/score_tools/symbol_report_blanket.rst Show resolved Hide resolved

pawelrutkaq added 3 commits February 4, 2026 14:36

Update doc-as-code

f5a7c8a

Add missing desc

970022e

Reword evaluation

d881e8f

aschemmel-tech requested changes Feb 4, 2026

View reviewed changes

pahmann requested changes Feb 4, 2026

View reviewed changes

Address review findings

567ba1e

pawelrutkaq requested review from aschemmel-tech, masc2023 and pahmann February 5, 2026 10:27

PandaeDo reviewed Feb 5, 2026

View reviewed changes

masc2023 requested changes Feb 9, 2026

View reviewed changes

pahmann reviewed Feb 11, 2026

View reviewed changes

PandaeDo reviewed Feb 11, 2026

View reviewed changes

pawelrutkaq mentioned this pull request Feb 11, 2026

Clarify qualification for symbol-report/blanket for coverage #2591

Open

2 tasks

Align content of report

8043a5d

masc2023 reviewed Feb 12, 2026

View reviewed changes

aschemmel-tech reviewed Feb 12, 2026

View reviewed changes

Symbol report verification report #2544

Are you sure you want to change the base?

Symbol report verification report #2544

Conversation

pawelrutkaq commented Feb 4, 2026

Uh oh!

github-actions bot commented Feb 4, 2026

Uh oh!

github-actions bot commented Feb 4, 2026

Uh oh!

Uh oh!

Uh oh!

aschemmel-tech left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pawelrutkaq commented Feb 5, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

pahmann left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

PandaeDo left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aschemmel-tech left a comment •

edited

Loading

pahmann left a comment •

edited

Loading