From 024a364ce45256f6dfe0f2542af8763055aa2a06 Mon Sep 17 00:00:00 2001 From: Pawel Rutka Date: Wed, 4 Feb 2026 14:26:00 +0100 Subject: [PATCH 1/6] Symbol report verification report --- .../_assets/symbol_report.drawio.svg | 288 ++++++++++++++++++ docs/score_tools/index.rst | 1 + docs/score_tools/symbol_report_blanket.rst | 120 ++++++++ 3 files changed, 409 insertions(+) create mode 100644 docs/score_tools/_assets/symbol_report.drawio.svg create mode 100644 docs/score_tools/symbol_report_blanket.rst diff --git a/docs/score_tools/_assets/symbol_report.drawio.svg b/docs/score_tools/_assets/symbol_report.drawio.svg new file mode 100644 index 00000000000..cd20e01cca4 --- /dev/null +++ b/docs/score_tools/_assets/symbol_report.drawio.svg @@ -0,0 +1,288 @@ + + + + + + + + + + + + + +
+
+
+
+ + Software + +
+
+ + sources (Rust) + +
+
+
+
+ + + Software... + + + + + + + + + + + + + + + +
+
+
+ + Compiler with +
+ coverage instrumentation +
+ (-Cinstrument-coverage) + +
+
+
+ + + Compiler with... + + + + + + + + + + + + + + + + + + + +
+
+
+ + Bazel + +
+
+
+ + + Bazel + + + + + + + + + + + + + +
+
+
+ + BZL files +
+ (rules and macros) + +
+
+
+ + + BZL files... + + + + + + + + + + + + + + + + + + + + + +
+
+
+ Unit tests +
+
+
+ + + Unit tests + + + + + + + + + + + + + + + + + + + + + + + + + +
+
+
+ Test binaries +
+
+
+ + + Test binaries + + + + + + + + + + + + + + + +
+
+
+ + + Execute test binaries on tareget +
+ and record .profraw files containing coverage informations + + +
+
+
+ + + Execute test binaries on tareget... + + + + + + + + + + + + + + + +
+
+
+ + + Symbol-report & blanket +
+ processing + + +
+
+
+ + + Symbol-report & blanket... + + + + + + + + + + + + + +
+
+
+ Coverage report +
+ with numbers +
+
+
+ + + Coverage report... + + + + + + + + + + Text is not SVG - cannot display + + + + diff --git a/docs/score_tools/index.rst b/docs/score_tools/index.rst index f0c91cacf03..14433acabc1 100644 --- a/docs/score_tools/index.rst +++ b/docs/score_tools/index.rst @@ -35,3 +35,4 @@ S-CORE Tools Overview rustfmt itf clippy + symbol_report_blanket diff --git a/docs/score_tools/symbol_report_blanket.rst b/docs/score_tools/symbol_report_blanket.rst new file mode 100644 index 00000000000..fa39cd2f09e --- /dev/null +++ b/docs/score_tools/symbol_report_blanket.rst @@ -0,0 +1,120 @@ +.. + # ******************************************************************************* + # Copyright (c) 2025 Contributors to the Eclipse Foundation + # + # See the NOTICE file(s) distributed with this work for additional + # information regarding copyright ownership. + # + # This program and the accompanying materials are made available under the + # terms of the Apache License Version 2.0 which is available at + # https://www.apache.org/licenses/LICENSE-2.0 + # + # SPDX-License-Identifier: Apache-2.0 + # ******************************************************************************* + +.. doc_tool:: symbol report and blanket + :id: doc_tool__symbol_report_blanket + :status: evaluated + :version: 1.90.0 (see [1]) + :tcl: HIGH + :safety_affected: YES + :security_affected: NO + :realizes: wp__tool_verification_report + :tags: tool_management + +Symbol Report and Blanket Verification Report +============================================= + +Introduction +------------ +Scope and purpose +~~~~~~~~~~~~~~~~~ +Symbol report and blanket are + +Inputs and outputs +~~~~~~~~~~~~~~~~~~ +| Inputs: Software sources (Rust) +| Outputs: Report with calculated coverage + +.. figure:: _assets/symbol_report.drawio.svg + :width: 100% + :align: center + :alt: Symbol report and blanket overview + + Symbol report and blanket overview + +Available information +~~~~~~~~~~~~~~~~~~~~~ +- Version: >= 1.90.0 [1]_ +- Official repository: https://github.com/ferrocene/ferrocene/tree/main/ferrocene/tools/blanket, https://github.com/ferrocene/ferrocene/tree/main/ferrocene/tools/symbol-report +- Additional information for usage in other safety projects: https://public-docs.ferrocene.dev/main/certification/core/safety-plan/tools.html#code-coverage + + +Installation and integration +---------------------------- +Installation +~~~~~~~~~~~~ +| To add the Code coverage to your project or module follow guidelines in WIP + +Integration +~~~~~~~~~~~ +Integrated in bazel. + +Environment +~~~~~~~~~~~ +Requires Rust toolchain and Bazel build environment. + +Safety evaluation +----------------- +This section outlines the safety evaluation of symbol report and blanket for its use within the S-CORE project. This evaluation assumes that the Rust compiler is +qualified and output of coverage data in `.profraw` format is correct. Due to that, we solely focus on post processing that is done by symbol report and blanket only. + + +.. list-table:: Safety evaluation + :header-rows: 1 + :widths: 1 2 8 2 6 4 2 2 + + * - Malfunction identification + - Use case description + - Malfunctions + - Impact on safety? + - Impact safety measures available? + - Impact safety detection sufficient? + - Further additional safety measure required? + - Confidence (automatic calculation) + * - 1 + - False-positive: The function is reported as covered and increase coverage percentage, although it is not covered in reality (not included in `profraw` data) + - | The reported covarge number is reported higher than the real coverage. + | This may lead to false assumption that code is sufficiently covered by tests, although it is not. + + - no + - no + - yes + - no + - high + * - 2 + - False-negative: The function is reported as not covered, although it is covered in reality (included in `profraw` data) + - | The reported coverage number is reported lower than the real coverage. + | This may lead to unnecessary additional testing effort (developer to prove it's covered or provide explanation), although code is already sufficiently covered. + - no + - no + - yes + - no + - high + * - 3 + - The function from source code is not included in the coverage report at all. + - | The reported coverage number is reported higher than the real coverage. + | This may lead to false assumption that code is sufficiently covered by tests, although it is not. + - no + - no + - yes + - no + - high + + +Result +~~~~~~ +Symbol report and blanket do not require qualification for use in safety-related software development according to ISO 26262. + +.. [1] The tool version mentioned in this document is preliminary. + Exact version shall be derived from qualified Rust compiler used in S-CORE project. From f5a7c8a6d7d5b3762648e777a073790affcbe6bd Mon Sep 17 00:00:00 2001 From: Pawel Rutka Date: Wed, 4 Feb 2026 14:36:14 +0100 Subject: [PATCH 2/6] Update doc-as-code --- docs/score_tools/doc_as_code.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/score_tools/doc_as_code.rst b/docs/score_tools/doc_as_code.rst index 3c3faa6db37..d326e87e890 100644 --- a/docs/score_tools/doc_as_code.rst +++ b/docs/score_tools/doc_as_code.rst @@ -15,7 +15,7 @@ .. doc_tool:: Doc-as-Code :id: doc_tool__doc_as_code :status: evaluated - :version: v2.2.0 + :version: v2.3.0 :tcl: LOW :safety_affected: YES :security_affected: YES @@ -66,7 +66,7 @@ Installation and integration Tool location: -`Release v2.2.0 `_ +`Release v2.3.0 `_ How to integrate: @@ -163,7 +163,7 @@ Tool requirements are defined here: Test cases and tests are defined here: -`Doc-as-Code Test cases and tests `_ +`Doc-as-Code Test cases and tests `_ The tests are part of the rules checker code sections. From 970022e7f8988bb926809df6059a62155e2d6f35 Mon Sep 17 00:00:00 2001 From: Pawel Rutka Date: Wed, 4 Feb 2026 14:40:31 +0100 Subject: [PATCH 3/6] Add missing desc --- docs/score_tools/symbol_report_blanket.rst | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/score_tools/symbol_report_blanket.rst b/docs/score_tools/symbol_report_blanket.rst index fa39cd2f09e..1897f712633 100644 --- a/docs/score_tools/symbol_report_blanket.rst +++ b/docs/score_tools/symbol_report_blanket.rst @@ -29,7 +29,9 @@ Introduction ------------ Scope and purpose ~~~~~~~~~~~~~~~~~ -Symbol report and blanket are +Symbol report and blanket are tools for post-processing of code coverage data generated by Rust compiler. +They are used to calculate code coverage based on the raw data generated by Rust compiler and provide a report +with calculated coverage. Inputs and outputs ~~~~~~~~~~~~~~~~~~ From d881e8f8e4c01806cc866f9a38cb33ca6ead9bc3 Mon Sep 17 00:00:00 2001 From: Pawel Rutka Date: Wed, 4 Feb 2026 15:23:50 +0100 Subject: [PATCH 4/6] Reword evaluation --- docs/score_tools/symbol_report_blanket.rst | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-) diff --git a/docs/score_tools/symbol_report_blanket.rst b/docs/score_tools/symbol_report_blanket.rst index 1897f712633..aa597185bc0 100644 --- a/docs/score_tools/symbol_report_blanket.rst +++ b/docs/score_tools/symbol_report_blanket.rst @@ -85,17 +85,17 @@ qualified and output of coverage data in `.profraw` format is correct. Due to th - Further additional safety measure required? - Confidence (automatic calculation) * - 1 - - False-positive: The function is reported as covered and increase coverage percentage, although it is not covered in reality (not included in `profraw` data) - - | The reported covarge number is reported higher than the real coverage. + - `profraw` does not contain data, but the tool reports coverage for the function. + - | The reported coverage number is reported higher than the real coverage. | This may lead to false assumption that code is sufficiently covered by tests, although it is not. - no - - no + - Likelihood of this malfunction is low - yes - no - high * - 2 - - False-negative: The function is reported as not covered, although it is covered in reality (included in `profraw` data) + - `profraw` does contain data, but the tool does not report coverage for the function. - | The reported coverage number is reported lower than the real coverage. | This may lead to unnecessary additional testing effort (developer to prove it's covered or provide explanation), although code is already sufficiently covered. - no @@ -108,6 +108,14 @@ qualified and output of coverage data in `.profraw` format is correct. Due to th - | The reported coverage number is reported higher than the real coverage. | This may lead to false assumption that code is sufficiently covered by tests, although it is not. - no + - Function will not be present in the report, so it will be visible that it is not covered. + - yes + - no + - high + * - 4 + - Calculations are wrong, so the reported coverage number is not correct. + - | The reported coverage is different than the coverage in `profraw` files. + - no - no - yes - no From 567ba1e86bb91534dfebc13d031513056fb5748b Mon Sep 17 00:00:00 2001 From: Pawel Rutka Date: Thu, 5 Feb 2026 11:11:22 +0100 Subject: [PATCH 5/6] Address review findings --- docs/score_tools/symbol_report_blanket.rst | 64 +++++++++++++++------- 1 file changed, 44 insertions(+), 20 deletions(-) diff --git a/docs/score_tools/symbol_report_blanket.rst b/docs/score_tools/symbol_report_blanket.rst index aa597185bc0..3816b577854 100644 --- a/docs/score_tools/symbol_report_blanket.rst +++ b/docs/score_tools/symbol_report_blanket.rst @@ -29,7 +29,7 @@ Introduction ------------ Scope and purpose ~~~~~~~~~~~~~~~~~ -Symbol report and blanket are tools for post-processing of code coverage data generated by Rust compiler. +`Symbol report` and `blanket` are tools for post-processing of code coverage data generated by Rust compiler. They are used to calculate code coverage based on the raw data generated by Rust compiler and provide a report with calculated coverage. @@ -68,7 +68,7 @@ Requires Rust toolchain and Bazel build environment. Safety evaluation ----------------- -This section outlines the safety evaluation of symbol report and blanket for its use within the S-CORE project. This evaluation assumes that the Rust compiler is +This section outlines the safety evaluation of `symbol report` and `blanket` for its use within the S-CORE project. This evaluation assumes that the Rust compiler is qualified and output of coverage data in `.profraw` format is correct. Due to that, we solely focus on post processing that is done by symbol report and blanket only. @@ -85,38 +85,62 @@ qualified and output of coverage data in `.profraw` format is correct. Due to th - Further additional safety measure required? - Confidence (automatic calculation) * - 1 - - `profraw` does not contain data, but the tool reports coverage for the function. - - | The reported coverage number is reported higher than the real coverage. - | This may lead to false assumption that code is sufficiently covered by tests, although it is not. - - - no - - Likelihood of this malfunction is low + - False-positive: A function is reported as covered, although it is not covered + - Overreporting, could result in testing gap. + - yes + - | Likelihood of such an error low due to wide usage of the tool (many S-CORE modules and other projects like ferrocene) + | Additionally, every new tool release is tested by running tests in prepared integration testsuite to detect such errors. (PROPOSAL POINT) - yes - no - high * - 2 - - `profraw` does contain data, but the tool does not report coverage for the function. - - | The reported coverage number is reported lower than the real coverage. - | This may lead to unnecessary additional testing effort (developer to prove it's covered or provide explanation), although code is already sufficiently covered. - - no - - no + - False-negative: A function is reported as not covered, although it is covered + - Underreporting, will not result in testing gap. + - yes + - Since we want to achieve 90%+ branch coverage this would stand out and be manually investigated. - yes - no - high * - 3 - - The function from source code is not included in the coverage report at all. - - | The reported coverage number is reported higher than the real coverage. - | This may lead to false assumption that code is sufficiently covered by tests, although it is not. - - no - - Function will not be present in the report, so it will be visible that it is not covered. + - Overcounting: Total number of functions is too low + - A function is not being considered, although it is part of the certified subset + - yes + - | `symbol-report` is developed to use exactly the same information as the compiler + | Additionally, every new tool release is tested by running tests in prepared integration testsuite to detect such errors. (PROPOSAL POINT) - yes - no - high * - 4 - - Calculations are wrong, so the reported coverage number is not correct. - - | The reported coverage is different than the coverage in `profraw` files. + - Undercounting: Total number of functions is too high + - A function is being considered, although it is not part of the certified subset + - no + - no + - yes + - no + - high + * - 5 + - Line that can be executed not being reported as executable + - Underreporting, code that should be tested may not being tested + - yes + - | `blanket`` warns if a function has no executable line + | Additionally, every new tool release is tested by running tests in prepared integration testsuite to detect such errors. (PROPOSAL POINT) + - yes - no + - high + * - 6 + - Coverage calculation errors: rounding, aggregation errors + - Overreporting or underreporting of coverage that can lead to testing gaps + - yes + - | Every new tool release is tested by running tests in prepared integration testsuite to detect such errors. (PROPOSAL POINT) + | Since this would be systematic error, it would be detected during manual review of coverage reports (always rounded values, too big/low coverage even no tests are there, etc.). + - yes + - no + - high + * - 7 + - Coverage reported for another function + - Wrong reporting view - no + - | Manual review would disclose that coverage is reported for another function than tested one by developer. - yes - no - high From 8043a5d3f08aca96bd9e5679cdf6f9df5101d347 Mon Sep 17 00:00:00 2001 From: Pawel Rutka Date: Thu, 12 Feb 2026 10:30:53 +0100 Subject: [PATCH 6/6] Align content of report --- docs/score_tools/symbol_report_blanket.rst | 50 ++++++++++++++-------- 1 file changed, 33 insertions(+), 17 deletions(-) diff --git a/docs/score_tools/symbol_report_blanket.rst b/docs/score_tools/symbol_report_blanket.rst index 3816b577854..5932cde293a 100644 --- a/docs/score_tools/symbol_report_blanket.rst +++ b/docs/score_tools/symbol_report_blanket.rst @@ -18,7 +18,7 @@ :version: 1.90.0 (see [1]) :tcl: HIGH :safety_affected: YES - :security_affected: NO + :security_affected: YES :realizes: wp__tool_verification_report :tags: tool_management @@ -56,7 +56,7 @@ Installation and integration ---------------------------- Installation ~~~~~~~~~~~~ -| To add the Code coverage to your project or module follow guidelines in WIP +| To add the Code coverage to your project or module follow guidelines in `here `_. Integration ~~~~~~~~~~~ @@ -88,27 +88,25 @@ qualified and output of coverage data in `.profraw` format is correct. Due to th - False-positive: A function is reported as covered, although it is not covered - Overreporting, could result in testing gap. - yes - - | Likelihood of such an error low due to wide usage of the tool (many S-CORE modules and other projects like ferrocene) - | Additionally, every new tool release is tested by running tests in prepared integration testsuite to detect such errors. (PROPOSAL POINT) - - yes + - **No**. However likelihood of such an error low due to wide usage of the tool (many S-CORE modules and other projects like ferrocene and their customers) - no + - **Yes**. Every new tool release is tested by running tests in prepared integration testsuite to detect such errors. - high * - 2 - False-negative: A function is reported as not covered, although it is covered - Underreporting, will not result in testing gap. - yes - - Since we want to achieve 90%+ branch coverage this would stand out and be manually investigated. - - yes + - Since we want to achieve 100% branch coverage (`check here `_) this would stand out and be manually investigated. + - no - no - high * - 3 - Overcounting: Total number of functions is too low - A function is not being considered, although it is part of the certified subset - yes - - | `symbol-report` is developed to use exactly the same information as the compiler - | Additionally, every new tool release is tested by running tests in prepared integration testsuite to detect such errors. (PROPOSAL POINT) + - `symbol-report` is developed to use exactly the same information as the compiler - yes - - no + - **Yes**. Every new tool release is tested by running tests in prepared integration testsuite to detect such errors. - high * - 4 - Undercounting: Total number of functions is too high @@ -122,19 +120,17 @@ qualified and output of coverage data in `.profraw` format is correct. Due to th - Line that can be executed not being reported as executable - Underreporting, code that should be tested may not being tested - yes - - | `blanket`` warns if a function has no executable line - | Additionally, every new tool release is tested by running tests in prepared integration testsuite to detect such errors. (PROPOSAL POINT) + - `blanket` warns if a function has no executable line - yes - - no + - **Yes**. Additionally, every new tool release is tested by running tests in prepared integration testsuite to detect such errors. - high * - 6 - Coverage calculation errors: rounding, aggregation errors - Overreporting or underreporting of coverage that can lead to testing gaps - yes - - | Every new tool release is tested by running tests in prepared integration testsuite to detect such errors. (PROPOSAL POINT) - | Since this would be systematic error, it would be detected during manual review of coverage reports (always rounded values, too big/low coverage even no tests are there, etc.). + - Since this would be systematic error, it would be detected during manual review of coverage reports (always rounded values, too big/low coverage even no tests are there, etc.). - yes - - no + - **Yes**. Every new tool release is tested by running tests in prepared integration testsuite to detect such errors. - high * - 7 - Coverage reported for another function @@ -145,10 +141,30 @@ qualified and output of coverage data in `.profraw` format is correct. Due to th - no - high +Security evaluation +------------------- +This section outlines the security evaluation of Rustfmt for its use within the S-CORE project. + + +.. list-table:: Security evaluation + :header-rows: 1 + + * - Threat identification + - Use case description + - Threats + - Impact on security? + - Impact security measures available? + - Impact security detection sufficient? + * - 1 + - TBD + - TBD + - TBD + - TBD + - TBD Result ~~~~~~ -Symbol report and blanket do not require qualification for use in safety-related software development according to ISO 26262. +Considering evaluation and a `Further additional safety measure ` the `symbol report` and `blanket` do not require qualification for use in safety-related software development according to ISO 26262. .. [1] The tool version mentioned in this document is preliminary. Exact version shall be derived from qualified Rust compiler used in S-CORE project.