diff --git a/docs/score_tools/_assets/symbol_report.drawio.svg b/docs/score_tools/_assets/symbol_report.drawio.svg new file mode 100644 index 00000000000..cd20e01cca4 --- /dev/null +++ b/docs/score_tools/_assets/symbol_report.drawio.svg @@ -0,0 +1,288 @@ + + + + + + + + + + + + + +
+
+
+
+ + Software + +
+
+ + sources (Rust) + +
+
+
+
+ + + Software... + + + + + + + + + + + + + + + +
+
+
+ + Compiler with +
+ coverage instrumentation +
+ (-Cinstrument-coverage) + +
+
+
+ + + Compiler with... + + + + + + + + + + + + + + + + + + + +
+
+
+ + Bazel + +
+
+
+ + + Bazel + + + + + + + + + + + + + +
+
+
+ + BZL files +
+ (rules and macros) + +
+
+
+ + + BZL files... + + + + + + + + + + + + + + + + + + + + + +
+
+
+ Unit tests +
+
+
+ + + Unit tests + + + + + + + + + + + + + + + + + + + + + + + + + +
+
+
+ Test binaries +
+
+
+ + + Test binaries + + + + + + + + + + + + + + + +
+
+
+ + + Execute test binaries on tareget +
+ and record .profraw files containing coverage informations + + +
+
+
+ + + Execute test binaries on tareget... + + + + + + + + + + + + + + + +
+
+
+ + + Symbol-report & blanket +
+ processing + + +
+
+
+ + + Symbol-report & blanket... + + + + + + + + + + + + + +
+
+
+ Coverage report +
+ with numbers +
+
+
+ + + Coverage report... + + + + + + + + + + Text is not SVG - cannot display + + + + diff --git a/docs/score_tools/doc_as_code.rst b/docs/score_tools/doc_as_code.rst index 3c3faa6db37..d326e87e890 100644 --- a/docs/score_tools/doc_as_code.rst +++ b/docs/score_tools/doc_as_code.rst @@ -15,7 +15,7 @@ .. doc_tool:: Doc-as-Code :id: doc_tool__doc_as_code :status: evaluated - :version: v2.2.0 + :version: v2.3.0 :tcl: LOW :safety_affected: YES :security_affected: YES @@ -66,7 +66,7 @@ Installation and integration Tool location: -`Release v2.2.0 `_ +`Release v2.3.0 `_ How to integrate: @@ -163,7 +163,7 @@ Tool requirements are defined here: Test cases and tests are defined here: -`Doc-as-Code Test cases and tests `_ +`Doc-as-Code Test cases and tests `_ The tests are part of the rules checker code sections. diff --git a/docs/score_tools/index.rst b/docs/score_tools/index.rst index f0c91cacf03..14433acabc1 100644 --- a/docs/score_tools/index.rst +++ b/docs/score_tools/index.rst @@ -35,3 +35,4 @@ S-CORE Tools Overview rustfmt itf clippy + symbol_report_blanket diff --git a/docs/score_tools/symbol_report_blanket.rst b/docs/score_tools/symbol_report_blanket.rst new file mode 100644 index 00000000000..5932cde293a --- /dev/null +++ b/docs/score_tools/symbol_report_blanket.rst @@ -0,0 +1,170 @@ +.. + # ******************************************************************************* + # Copyright (c) 2025 Contributors to the Eclipse Foundation + # + # See the NOTICE file(s) distributed with this work for additional + # information regarding copyright ownership. + # + # This program and the accompanying materials are made available under the + # terms of the Apache License Version 2.0 which is available at + # https://www.apache.org/licenses/LICENSE-2.0 + # + # SPDX-License-Identifier: Apache-2.0 + # ******************************************************************************* + +.. doc_tool:: symbol report and blanket + :id: doc_tool__symbol_report_blanket + :status: evaluated + :version: 1.90.0 (see [1]) + :tcl: HIGH + :safety_affected: YES + :security_affected: YES + :realizes: wp__tool_verification_report + :tags: tool_management + +Symbol Report and Blanket Verification Report +============================================= + +Introduction +------------ +Scope and purpose +~~~~~~~~~~~~~~~~~ +`Symbol report` and `blanket` are tools for post-processing of code coverage data generated by Rust compiler. +They are used to calculate code coverage based on the raw data generated by Rust compiler and provide a report +with calculated coverage. + +Inputs and outputs +~~~~~~~~~~~~~~~~~~ +| Inputs: Software sources (Rust) +| Outputs: Report with calculated coverage + +.. figure:: _assets/symbol_report.drawio.svg + :width: 100% + :align: center + :alt: Symbol report and blanket overview + + Symbol report and blanket overview + +Available information +~~~~~~~~~~~~~~~~~~~~~ +- Version: >= 1.90.0 [1]_ +- Official repository: https://github.com/ferrocene/ferrocene/tree/main/ferrocene/tools/blanket, https://github.com/ferrocene/ferrocene/tree/main/ferrocene/tools/symbol-report +- Additional information for usage in other safety projects: https://public-docs.ferrocene.dev/main/certification/core/safety-plan/tools.html#code-coverage + + +Installation and integration +---------------------------- +Installation +~~~~~~~~~~~~ +| To add the Code coverage to your project or module follow guidelines in `here `_. + +Integration +~~~~~~~~~~~ +Integrated in bazel. + +Environment +~~~~~~~~~~~ +Requires Rust toolchain and Bazel build environment. + +Safety evaluation +----------------- +This section outlines the safety evaluation of `symbol report` and `blanket` for its use within the S-CORE project. This evaluation assumes that the Rust compiler is +qualified and output of coverage data in `.profraw` format is correct. Due to that, we solely focus on post processing that is done by symbol report and blanket only. + + +.. list-table:: Safety evaluation + :header-rows: 1 + :widths: 1 2 8 2 6 4 2 2 + + * - Malfunction identification + - Use case description + - Malfunctions + - Impact on safety? + - Impact safety measures available? + - Impact safety detection sufficient? + - Further additional safety measure required? + - Confidence (automatic calculation) + * - 1 + - False-positive: A function is reported as covered, although it is not covered + - Overreporting, could result in testing gap. + - yes + - **No**. However likelihood of such an error low due to wide usage of the tool (many S-CORE modules and other projects like ferrocene and their customers) + - no + - **Yes**. Every new tool release is tested by running tests in prepared integration testsuite to detect such errors. + - high + * - 2 + - False-negative: A function is reported as not covered, although it is covered + - Underreporting, will not result in testing gap. + - yes + - Since we want to achieve 100% branch coverage (`check here `_) this would stand out and be manually investigated. + - no + - no + - high + * - 3 + - Overcounting: Total number of functions is too low + - A function is not being considered, although it is part of the certified subset + - yes + - `symbol-report` is developed to use exactly the same information as the compiler + - yes + - **Yes**. Every new tool release is tested by running tests in prepared integration testsuite to detect such errors. + - high + * - 4 + - Undercounting: Total number of functions is too high + - A function is being considered, although it is not part of the certified subset + - no + - no + - yes + - no + - high + * - 5 + - Line that can be executed not being reported as executable + - Underreporting, code that should be tested may not being tested + - yes + - `blanket` warns if a function has no executable line + - yes + - **Yes**. Additionally, every new tool release is tested by running tests in prepared integration testsuite to detect such errors. + - high + * - 6 + - Coverage calculation errors: rounding, aggregation errors + - Overreporting or underreporting of coverage that can lead to testing gaps + - yes + - Since this would be systematic error, it would be detected during manual review of coverage reports (always rounded values, too big/low coverage even no tests are there, etc.). + - yes + - **Yes**. Every new tool release is tested by running tests in prepared integration testsuite to detect such errors. + - high + * - 7 + - Coverage reported for another function + - Wrong reporting view + - no + - | Manual review would disclose that coverage is reported for another function than tested one by developer. + - yes + - no + - high + +Security evaluation +------------------- +This section outlines the security evaluation of Rustfmt for its use within the S-CORE project. + + +.. list-table:: Security evaluation + :header-rows: 1 + + * - Threat identification + - Use case description + - Threats + - Impact on security? + - Impact security measures available? + - Impact security detection sufficient? + * - 1 + - TBD + - TBD + - TBD + - TBD + - TBD + +Result +~~~~~~ +Considering evaluation and a `Further additional safety measure ` the `symbol report` and `blanket` do not require qualification for use in safety-related software development according to ISO 26262. + +.. [1] The tool version mentioned in this document is preliminary. + Exact version shall be derived from qualified Rust compiler used in S-CORE project.