In short
No. A coverage percentage records which code your tests executed, not whether the tests would detect a fault in that code. Every structural coverage criterion, from statement coverage to multiple condition coverage, is satisfied by executing code, so a suite that reaches every branch and asserts nothing scores the same as one that checks every output against a requirement, which is why ISO 26262, IEC 61508 and DO-178 all treat coverage as a completeness check on requirements-based testing rather than as the verification argument itself.
The Honest Answer Is No
A unit test campaign finishes, the report says 100%, and the number goes into a design review as though it settled something. It did not. A coverage figure records which parts of the code your tests executed, not whether they would have noticed if the code were wrong.
The NASA and FAA tutorial on Modified Condition/Decision Coverage (NASA/TM-2001-210876, May 2001, by Hayhurst of NASA Langley, Veerhusen of Rockwell Collins, Chilenski of Boeing and Rierson of the FAA) is blunt about the category error: “Coverage is a measure, not a method or a test. Thus, phrases such as ‘MC/DC testing’ can do more harm than good.”
The useful question is not whether 100% is enough, but 100% of what. The tutorial ranks the criteria “from the weakest, statement coverage, to the strongest, multiple condition coverage.”
Statement Coverage (C0)
Every executable statement is invoked at least once, which shows all code is reachable. That is all it shows. The tutorial’s example, in Ada:
if (x > 1) and (y = 0) then z := z / x; end if;
if (z = 2) or (y > 1) then z := z + 1; end if;Inputs x = 2, y = 0, z = 4 execute every statement. “However, if an or is coded by mistake in the first statement instead of an and, the test case will not detect a problem.” The tutorial quotes Glenford Myers to the effect that the criterion is so weak it is generally considered useless, and concludes that “at best, statement coverage should be considered a minimal requirement.”
Decision, Condition and Condition/Decision Coverage
Decision coverage requires every decision to take all possible outcomes: one true case, one false case. For the compound decision (A or B), test cases TF and FF satisfy it, but “the effect of B is not tested; that is, those test cases cannot distinguish between the decision (A or B) and the decision A.” A condition can sit in your source, be fully covered on paper, and never be shown to matter.
Condition coverage requires each condition to take all outcomes but not the decision: TF and FT satisfy it while (A or B) never evaluates false. Condition/decision coverage combines the two and is still not enough: for (A or B), TT and FF meet it while failing to distinguish that expression from A, B, or (A and B).
MC/DC
MC/DC enhances condition/decision coverage “by requiring that each condition be shown to independently affect the outcome of the decision,” which in DO-178B’s wording means “varying just that condition while holding fixed all other possible conditions.” It generally requires a minimum of n+1 test cases for a decision with n inputs; for (A or B), TF, FT and FF provide it.
The tutorial is careful that inputs and conditions are not the same: (A and B) or (A and C) has three inputs but four conditions, because each occurrence of A counts as a distinct condition. Sizing a test campaign off the condition count instead of the input count is a common way to over-estimate the work.
What MC/DC still does not prove: it measures the code as written, so it says nothing about a requirement that was never implemented. Our MC/DC coverage guide covers it in detail.
Multiple Condition Coverage (MCC)
MCC requires every combination of condition outcomes in a decision to be invoked, “exhaustive testing of the input combinations,” which for a decision with n inputs means 2^n tests. The tutorial’s verdict: “In theory, multiple condition coverage is the most desirable structural coverage measure; but, it is impractical for many cases.” Its Boolean expression profile, credited to Chilenski (FAA report DOT/FAA/AR-01/18) and drawn from the level A airborne software of five line replaceable units taken from two airplane models in 1995, records two expressions in the 36 to 76 condition band.
| Criterion | 100% proves | 100% does not prove | Minimum tests, n inputs |
|---|---|---|---|
| Statement (C0) | Every statement ran | Anything about logic; an or written for an and survives | Not a function of n (depends on the code) |
| Decision (DC) / branch (C1) | Every decision evaluated true and false | That each condition in a compound decision matters | 2 per decision |
| Condition | Every condition took both values | That the decision took both outcomes | 2 per decision |
| Condition/decision | Both of the above | Cannot distinguish (A or B) from A, B, or (A and B) | 2 per decision |
| MC/DC | Each condition independently drives the outcome | Requirements never coded | n+1 |
| Multiple condition (MCC) | Every combination of outcomes ran | Nothing about intent; impractical above small n | 2^n |
Razorcat lists Branch Coverage (C1) and Decision Coverage (DC) as separate TESSY measurements. The row above treats them together because they coincide for a simple two-way if, but they are not the same measurement in general: a multi-way construct such as a switch has branches that are not decision outcomes.
The Assertion Problem: 100% With No Oracle
Every criterion above is satisfied by executing code. None inspects what the test then checks. A test that reaches every branch and asserts nothing counts as much toward the percentage as one that verifies the output against the requirement.
The PIT mutation testing project states the consequence plainly: conventional coverage “measures only which code is executed by your tests. It does not check that your tests are actually able to detect faults in the executed code.” RTCA/DO-248A, quoted in the tutorial, notes that code “exercised during requirements-based testing” means more than execution: “It also means that the behavior of the code has been compared with the requirements to which it traces.”
Which is why closing a coverage gap with a test written from the source is the wrong repair: per DO-248A FAQ #44, “if any additional testing is required, it should be requirements-based testing.”
Necessary But Not Sufficient: What the Standards Ask For
No safety standard treats coverage as the verification argument; each treats it as a completeness check on an argument made elsewhere.
ISO 26262-6:2018 clause 9.4.4 requires that the coverage of requirements at the software unit level is determined and that structural coverage is measured against the metrics listed in Table 9, and that where the achieved structural coverage is considered insufficient, either additional test cases are specified or a rationale is provided. Its note closes the obvious loophole: a target value that is absent, or low, and offered without a rationale counts as insufficient.
Table 9 repays a close reading, because the licensed text does not say what the summaries in circulation say. Statement coverage is highly recommended (++) at ASIL A and B and then drops to recommended (+) at C and D. Branch coverage is already recommended (+) at ASIL A and highly recommended (++) at B, C and D. MC/DC is recommended (+) at A, B and C, and highly recommended only at D. The standard does not grow more enthusiastic about statement coverage as risk rises, and it does not introduce branch coverage at ASIL B. Nor is any row of it mandatory: per clause 4.3 the strongest rating in the standard is highly recommended, and because Table 9 rows are alternative entries the obligation is to apply an appropriate combination and justify the combination you chose. Our ISO 26262 page reproduces the full table from the licensed copy. The standard is also explicit about what coverage is for: analysis of structural coverage can reveal shortcomings in requirements-based test cases, inadequacies in requirements, dead code, deactivated code or unintended functionality. A diagnostic instrument, not a certificate.
IEC 61508-3:2010 Table B.2 splits 100% structural test coverage into four rows rising in rigour with SIL: entry points (highly recommended at every SIL), statements, branches, and conditions with MC/DC (highly recommended only at SIL 4). Its footnote concedes the point: “Where 100 % coverage cannot be achieved (e.g. statement coverage of defensive code), an appropriate explanation should be given.”
DO-178B, per the tutorial’s reading of Table A-7, requires statement coverage for levels A to C, decision coverage for levels A and B, and MC/DC for level A only, and in DO-178C the MC/DC objective A-7-5 still applies at Level A alone. It is one row among many: objectives 3 and 4 require test coverage of high-level and low-level requirements. DO-248A FAQ #43 gives structural coverage analysis three purposes: evidence that the code structure was verified to the degree required for the applicable software level, a means to support demonstration of absence of unintended functions, and establishing the thoroughness of requirements-based testing. See our DO-178C and IEC 61508 pages.
What to Add
Requirements traceability, both directions. The tutorial is emphatic that “requirements coverage analysis precedes structural coverage analysis,” and honest about why neither alone suffices: “requirements-based testing alone cannot confirm that the code does not include unintended functionality,” while tests written from the code cannot find requirements that were never implemented. Razorcat builds this into TESSY, which per Razorcat imports requirements from Polarion via a plug-in and from DOORS, Jama Connect and Visure via ReqIF, and links test cases back to them.
Fault-based testing. If the worry is that your assertions are too weak to detect a defect, the direct measurement is to inject one. Google’s State of Mutation Testing at Google (Petrovic and Ivankovic, ICSE SEIP 2018) describes the method as inserting “small faults into programs and measuring the ability of the test suite to detect them,” and states that it “is widely considered the strongest test criterion in terms of finding the most faults.” In PIT, a seeded fault is killed if your tests fail and lived if they pass. This is not exotic in a safety context: ISO 26262-6:2018 Table 7, methods for software unit verification, lists fault injection test at row 1l, and its footnote explains that for software unit testing this means modifying the tested software unit, for example by corrupting values of variables, introducing code mutations, or corrupting values of CPU registers. IEC 61508-3:2010 Table B.2 rates test case execution from error seeding recommended from SIL 2 upward. Our companion piece on fault injection and robustness testing works through those rows in full.
Robustness cases. Coverage rewards the happy path, because the happy path is where the statements are. Per the tutorial, DO-178B section 6.4.4.1 states that test coverage analysis should show that test cases exist for each software requirement and “that the test cases satisfy the criteria for normal and robustness testing.” ISO 26262-6:2018 Table 8 lists four methods for deriving unit test cases: analysis of requirements, generation and analysis of equivalence classes, analysis of boundary values, and error guessing, each ASIL-rated, with clause 4.3 governing the combination you select and justify. IEC 61508-3:2010 Table B.2 rates test case execution from boundary value analysis highly recommended from SIL 2 upward. The same companion guide separates robustness testing from fault injection, which are not synonyms.
Where TESSY Fits
TESSY is developed by Razorcat Development GmbH in Berlin, and everything attributed to the tool here is Razorcat’s own. Per Razorcat, TESSY provides Statement Coverage (C0), Branch Coverage (C1), Decision Coverage (DC), Modified Condition / Decision Coverage (MC/DC), Multiple Condition Coverage (MCC), Entry Point Coverage (EPC), Function Coverage (FC) and Call Pair Coverage (CPC), instrumenting the source automatically for the measurements selected. Razorcat adds that Call Pair Coverage checks “whether all call points of functions or methods within the test object have been executed at least once,” and “meets the requirements of ISO 26262 … as an alternative code coverage procedure for integration tests instead of function coverage (FC) measurement.”
Razorcat states that TESSY is qualified for safety-related software development according to IEC 61508, IEC 62304, ISO 26262 and EN 50128, and publishes a certification report. Razorcat’s public product pages currently document TESSY 5.1, so ask GSAS to confirm the version and feature scope for your programme. For the static analysis half, see our TESSY and Helix QAC guide.
Related reading
- Automated Mutation Testing for Embedded C, the technique that answers what coverage cannot
- Unit Testing Fundamentals for Embedded C and C++, the mechanics underneath the coverage numbers
- Test Case Quality in Embedded Unit Testing, why case quality, not case count, moves the needle
- Razorcat TESSY, the tool these techniques are run with in India, the UAE and Sri Lanka
Talk to GSAS
GSAS Micro Systems is the authorized Razorcat engineering partner for India, the UAE and Sri Lanka. If your coverage number is high and your confidence is not, our engineers will look at what the percentage hides: what the tests assert, whether every requirement has a test, and where the robustness cases are missing.
GSAS supports automotive, aerospace, medical and industrial teams across Bengaluru, Hyderabad, Chennai, Pune, Mumbai and Delhi NCR, with evaluation support, deployment engineering and INR invoicing. See the TESSY product page, the Razorcat partner page or automotive solutions.
Also appears in:
Interested in Razorcat tools?
Talk to our application engineers for personalized tool recommendations.
More from Razorcat
View all →
Test Case Design with the Classification Tree Method: Deriving Unit Tests You Can Defend in an Audit
1 Aug · 10 min
Fault Injection and Robustness Testing for Embedded Software: What ISO 26262, IEC 61508 and DO-178C Actually Ask For
1 Aug · 11 min
How to Evaluate a Unit Testing Tool for Embedded Software: A Buyer's Framework for Indian Teams
1 Aug · 10 min
Automated Mutation Testing for Embedded C: The Question MC/DC Coverage Cannot Answer
1 Aug · 14 min
Test Case Quality in Embedded Unit Testing: Why Test Counts and Coverage Are Weak Proxies
1 Aug · 10 min
From Evaluation Licence to Production Licence: How Embedded Unit Test Tool Licensing Actually Works in India
1 Aug · 11 min