Is 100% Code Coverage Enough? Statement, Branch, MC/DC and MCC, and What Each One Proves

The Honest Answer Is No

A unit test campaign finishes, the report says 100%, and the number goes into a design review as though it settled something. It did not. A coverage figure records which parts of the code your tests executed, not whether they would have noticed if the code were wrong.

The NASA and FAA tutorial on Modified Condition/Decision Coverage (NASA/TM-2001-210876, May 2001, by Hayhurst of NASA Langley, Veerhusen of Rockwell Collins, Chilenski of Boeing and Rierson of the FAA) is blunt about the category error: “Coverage is a measure, not a method or a test. Thus, phrases such as ‘MC/DC testing’ can do more harm than good.”

The useful question is not whether 100% is enough, but 100% of what. The tutorial ranks the criteria “from the weakest, statement coverage, to the strongest, multiple condition coverage.”

Statement Coverage (C0)

Every executable statement is invoked at least once, which shows all code is reachable. That is all it shows. The tutorial’s example, in Ada:

if (x > 1) and (y = 0) then z := z / x; end if;
if (z = 2) or (y > 1) then z := z + 1; end if;

Inputs x = 2, y = 0, z = 4 execute every statement. “However, if an or is coded by mistake in the first statement instead of an and, the test case will not detect a problem.” The tutorial quotes Glenford Myers to the effect that the criterion is so weak it is generally considered useless, and concludes that “at best, statement coverage should be considered a minimal requirement.”

Decision, Condition and Condition/Decision Coverage

Decision coverage requires every decision to take all possible outcomes: one true case, one false case. For the compound decision (A or B), test cases TF and FF satisfy it, but “the effect of B is not tested; that is, those test cases cannot distinguish between the decision (A or B) and the decision A.” A condition can sit in your source, be fully covered on paper, and never be shown to matter.

Condition coverage requires each condition to take all outcomes but not the decision: TF and FT satisfy it while (A or B) never evaluates false. Condition/decision coverage combines the two and is still not enough: for (A or B), TT and FF meet it while failing to distinguish that expression from A, B, or (A and B).

MC/DC

MC/DC enhances condition/decision coverage “by requiring that each condition be shown to independently affect the outcome of the decision,” which in DO-178B’s wording means “varying just that condition while holding fixed all other possible conditions.” It generally requires a minimum of n+1 test cases for a decision with n inputs; for (A or B), TF, FT and FF provide it.

The tutorial is careful that inputs and conditions are not the same: (A and B) or (A and C) has three inputs but four conditions, because each occurrence of A counts as a distinct condition. Sizing a test campaign off the condition count instead of the input count is a common way to over-estimate the work.

What MC/DC still does not prove: it measures the code as written, so it says nothing about a requirement that was never implemented. Our MC/DC coverage guide covers it in detail.

Multiple Condition Coverage (MCC)

MCC requires every combination of condition outcomes in a decision to be invoked, “exhaustive testing of the input combinations,” which for a decision with n inputs means 2^n tests. The tutorial’s verdict: “In theory, multiple condition coverage is the most desirable structural coverage measure; but, it is impractical for many cases.” Its Boolean expression profile, credited to Chilenski (FAA report DOT/FAA/AR-01/18) and drawn from the level A airborne software of five line replaceable units taken from two airplane models in 1995, records two expressions in the 36 to 76 condition band.

Razorcat lists Branch Coverage (C1) and Decision Coverage (DC) as separate TESSY measurements. The row above treats them together because they coincide for a simple two-way if, but they are not the same measurement in general: a multi-way construct such as a switch has branches that are not decision outcomes.

The Assertion Problem: 100% With No Oracle

Every criterion above is satisfied by executing code. None inspects what the test then checks. A test that reaches every branch and asserts nothing counts as much toward the percentage as one that verifies the output against the requirement.

The PIT mutation testing project states the consequence plainly: conventional coverage “measures only which code is executed by your tests. It does not check that your tests are actually able to detect faults in the executed code.” RTCA/DO-248A, quoted in the tutorial, notes that code “exercised during requirements-based testing” means more than execution: “It also means that the behavior of the code has been compared with the requirements to which it traces.”

Which is why closing a coverage gap with a test written from the source is the wrong repair: per RTCA/DO-248A, again quoted in the tutorial, “if any additional testing is required, it should be requirements-based testing.”

Necessary But Not Sufficient: What the Standards Ask For

No safety standard treats coverage as the verification argument; each treats it as a completeness check on an argument made elsewhere.

A note on citations. ISO 26262, IEC 61508 and DO-178C are licensed, copyrighted documents. GSAS does not publish their clause numbers, table numbers, row letters or rating grids. What follows states the obligation in substance, as GSAS engineering guidance or attributed to a public source such as the NASA and FAA tutorial. Take it to your licensed copy and to your assessor. A project plan built on numbers transcribed from a website is a plan resting on somebody else’s typo.

ISO 26262-6:2018 requires, in substance, that the coverage of requirements at the software unit level is determined, that structural coverage is measured against the unit-level metrics the standard lists, and that where the achieved structural coverage is considered insufficient, either additional test cases are specified or a rationale is provided. Its note closes the obvious loophole: a target value that is absent, or low, and offered without a rationale counts as insufficient.

The unit-level metrics are statement coverage, branch coverage and MC/DC, and three things about how their recommendations move across the ASILs contradict the summaries in widest circulation. Statement coverage does not vanish at the top. Its recommendation is relaxed at the higher ASILs rather than dropped, and the reason is definitional rather than a softening of rigour: 100% branch coverage implies 100% statement coverage, so once branch coverage carries the stronger recommendation, statement coverage is subsumed by it. Branch coverage is already recommended at ASIL A, so it is not introduced at ASIL B, and a plan that defers branch measurement until an ASIL B feature arrives is deferring something that was recommended from the start. MC/DC is recommended across the range and carries the standard’s strongest recommendation only at ASIL D. None of it is mandatory: the strongest rating the standard applies to any method is highly recommended, and because these metrics are alternative entries the obligation is to apply an appropriate combination and justify the combination you chose. Our ISO 26262 page works through the same obligations in substance. The standard is also explicit about what coverage is for: analysis of structural coverage can reveal shortcomings in requirements-based test cases, inadequacies in requirements, dead code, deactivated code or unintended functionality. A diagnostic instrument, not a certificate.

IEC 61508-3:2010 treats 100% structural test coverage as a graded ladder rather than a single target: entry points first, then statements, then branches, then conditions with MC/DC, with the strength of the recommendation rising as the SIL rises and MC/DC carrying the strongest recommendation only at SIL 4. The standard concedes the point itself. Where 100% coverage cannot be achieved, statement coverage of defensive code being the example it gives, an appropriate explanation should be given.

DO-178B and DO-178C. Per the tutorial, the airborne structural coverage objectives are graded by software level: statement coverage applies down to level C, decision coverage to levels A and B, and MC/DC to level A alone, a mapping DO-178C carries forward unchanged. They sit alongside separate objectives for test coverage of the high-level and low-level requirements. Per RTCA/DO-248A, quoted in the tutorial, structural coverage analysis has three purposes: evidence that the code structure was verified to the degree required for the applicable software level, a means to support demonstration of absence of unintended functions, and establishing the thoroughness of requirements-based testing. See our DO-178C and IEC 61508 pages.

What to Add

Requirements traceability, both directions. The tutorial is emphatic that “requirements coverage analysis precedes structural coverage analysis,” and honest about why neither alone suffices: “requirements-based testing alone cannot confirm that the code does not include unintended functionality,” while tests written from the code cannot find requirements that were never implemented. Razorcat builds this into TESSY, which per Razorcat imports requirements from Polarion via a plug-in and from DOORS, Jama Connect and Visure via ReqIF, and links test cases back to them.

Fault-based testing. If the worry is that your assertions are too weak to detect a defect, the direct measurement is to inject one. Google’s State of Mutation Testing at Google (Petrovic and Ivankovic, ICSE SEIP 2018) describes the method as inserting “small faults into programs and measuring the ability of the test suite to detect them,” and states that it “is widely considered the strongest test criterion in terms of finding the most faults.” In PIT, a seeded fault is killed if your tests fail and lived if they pass. This is not exotic in a safety context: ISO 26262-6:2018 lists fault injection test among its methods for software unit verification, and explains that in the context of software unit testing this means modifying the tested software unit, for example by corrupting values of variables, introducing code mutations, or corrupting values of CPU registers. IEC 61508-3:2010 recommends test case execution from error seeding, and strengthens that recommendation as the SIL rises. Our companion piece on fault injection and robustness testing works through both methods in full.

Robustness cases. Coverage rewards the happy path, because the happy path is where the statements are. Per the tutorial, DO-178B requires test coverage analysis to show that test cases exist for each software requirement and “that the test cases satisfy the criteria for normal and robustness testing.” ISO 26262-6:2018 gives four methods for deriving unit test cases: analysis of requirements, generation and analysis of equivalence classes, analysis of boundary values, and error guessing, each ASIL-rated, with the alternative-entry rule governing the combination you select and justify. IEC 61508-3:2010 recommends test case execution from boundary value analysis, more strongly as the SIL rises. The same companion guide separates robustness testing from fault injection, which are not synonyms.

Where TESSY Fits

TESSY is developed by Razorcat Development GmbH in Berlin, and everything attributed to the tool here is Razorcat’s own. Per Razorcat, TESSY provides Statement Coverage (C0), Branch Coverage (C1), Decision Coverage (DC), Modified Condition / Decision Coverage (MC/DC), Multiple Condition Coverage (MCC), Entry Point Coverage (EPC), Function Coverage (FC) and Call Pair Coverage (CPC), instrumenting the source automatically for the measurements selected. Razorcat adds that Call Pair Coverage checks “whether all call points of functions or methods within the test object have been executed at least once,” and “meets the requirements of ISO 26262 … as an alternative code coverage procedure for integration tests instead of function coverage (FC) measurement.”

Razorcat states that TESSY is qualified for safety-related software development according to IEC 61508, IEC 62304, ISO 26262 and EN 50128, and publishes a certification report. Razorcat’s public product pages currently document TESSY 5.1, so ask GSAS to confirm the version and feature scope for your programme. For the static analysis half, see our TESSY and Helix QAC guide.

Related reading

• Automated Mutation Testing for Embedded C, the technique that answers what coverage cannot
• Unit Testing Fundamentals for Embedded C and C++, the mechanics underneath the coverage numbers
• Test Case Quality in Embedded Unit Testing, why case quality, not case count, moves the needle
• Razorcat TESSY, the tool these techniques are run with in India, the UAE and Sri Lanka

Talk to GSAS

GSAS Micro Systems is the authorized Razorcat engineering partner for India, the UAE and Sri Lanka. If your coverage number is high and your confidence is not, our engineers will look at what the percentage hides: what the tests assert, whether every requirement has a test, and where the robustness cases are missing.

GSAS supports automotive, aerospace, medical and industrial teams across Bengaluru, Hyderabad, Chennai, Pune, Mumbai and Delhi NCR, with evaluation support, deployment engineering and INR invoicing. See the TESSY product page, the Razorcat partner page or automotive solutions.

Request a TESSY evaluation