Test Case Quality in Embedded Unit Testing: Why Test Counts and Coverage Are Weak Proxies

Two Numbers That Look Like Evidence

Most unit test reviews open with the same two numbers: how many test cases exist, and what percentage of the code they touch. Both are easy to grow. Neither answers the question a functional safety assessor actually asks, which is whether a named requirement was verified, and whether the test would have failed had the implementation been wrong.

That distinction is not a GSAS opinion. NASA’s tutorial on modified condition/decision coverage, written for engineers working to DO-178B, states it plainly: “Coverage is a measure, not a method or a test,” and cautions that phrases such as “MC/DC testing” can do more harm than good. Coverage records what your tests touched. It is silent on what your tests would have caught.

What the Standards Actually Ask For

Requirements-based testing comes first, and structural coverage exists to audit it. The NASA tutorial describes the DO-178B relationship precisely: requirements coverage analysis determines how well the requirements-based testing verified the implementation of the software requirements and establishes traceability between the software requirements and the test cases, while structural coverage analysis determines how much of the code structure was executed by those same tests. Requirements coverage analysis precedes structural coverage analysis. DO-178C, the version the FAA recognises in Advisory Circular 20-115D alongside EUROCAE ED-12C, keeps the same shape of guidance: objectives for software life cycle processes, activities that provide a means for satisfying them, and descriptions of the evidence that indicates they were satisfied.

The tutorial is equally direct about the tempting shortcut, and on this point it hands the microphone to the standards committee. Quoting DO-248A FAQ #44, the NASA tutorial records that structural testing, meaning the process of exercising software with test scenarios written from the source code rather than from the requirements, does not meet the DO-178B objective that all code structure is exercised by the requirements-based test procedures. When structural coverage analysis identifies untested code, the FAQ’s correct approach is to consider the possible causes and, if any additional testing is required, for that testing to be requirements-based, because tests whose starting point is the code cannot find errors such as the non-implementation of some of the requirements.

Automotive asks for the same discipline. ISO 26262 Part 6 covers product development at the software level, including software unit design and implementation and software unit testing. In the 2017 clause-by-clause analysis of ISO 26262 published by Salay, Queiroz and Czarnecki, which examines the 2011 first edition, Part 6 grades each listed method as highly recommended, recommended, or no recommendation depending on the ASIL, and its unit-level catalogue includes black-box techniques such as analysis of boundary values and error guessing. The second edition keeps the same ASIL-graded structure. Razorcat’s summary of IEC 61508 Part 3 in the TESSY manual makes the same point for the generic functional-safety regime: the module test shall show that the module performs its intended function and does not perform unintended functions, and the annex tables of Part 3 mark functional and black box testing as highly recommended from SIL 1 upward.

Across all three regimes, coverage is an exit criterion, not the objective. As Delgado-Pérez and colleagues note, IEC 61508, ISO 26262 and DO-178C all express the required level of rigour as a structural coverage criterion, with DO-178C requiring MC/DC at the highest level of software assurance.

Four Properties of a Test Case Worth Having

It traces to a requirement

A test case with no requirement behind it can add coverage but not evidence. Traceability is what lets requirements coverage analysis say anything at all, and it has to survive change: when a requirement is edited, every test linked to it is questionable until someone re-reads both.

It has a meaningful oracle

The oracle decides pass or fail. In the definition used by Pezzè and Young in Software Testing and Analysis, test oracles check the results of executing the code and signal discrepancies between actual and expected outputs. Razorcat frames the same idea for unit testing: the actual results are compared to those predicted, and that comparison determines whether a test case passed. “Predicted” is the load-bearing word. An expected value computed by the code under test, or an assertion that the function merely returned, is not an oracle. Razorcat notes that a tester who did not write the code must first work out what the interface variables mean, and that predicting the expected results poses the same problem, alongside developers’ blindness to faults in their own work, which is why standards require the organisational separation of development and test.

It is built from equivalence classes and boundary values

The Classification Tree Method, implemented by Razorcat’s Classification Tree Editor and originating in the former Daimler software research laboratory in Berlin, transforms a functional specification into a set of error-sensitive, low-redundancy test case specifications. Two rules are worth memorising: equivalence partitioning must be complete, so every possible value of a test-relevant aspect belongs to a class, and unique, so no value belongs to two classes. Boundary value analysis then deliberately contradicts the equivalence assumption, on the reasoning that values at the borders of a range are better suited to form error-sensitive test cases than values in the middle.

It is independent of implementation detail

Razorcat describes unit testing as black-box work: it tests at the interface of the unit, meaning input variables, output variables and the return value, and does not consider internal structure. A test whose shape was copied from the code’s branches passes by construction, and keeps passing after a change that alters the answer.

How Weak Suites Still Hit the Coverage Target

The most directly relevant public evidence here comes from safety-critical code rather than from web software. Delgado-Pérez, Habli, Gregory, Alexander, Clark and Medina-Bulo evaluated mutation testing on a commercial device supplied to the UK nuclear industry, developed in accordance with IEC 61508 to satisfy the requirements of SIL 3, implemented in MISRA C and already achieving 100% branch coverage. Their finding: the branch-adequate test suite failed to detect injected faults in 11 of the 15 functions analysed, and in the most noteworthy case 37% of the faults injected into one function went undetected. The suite killed 82% of mutants on a conventional count, 92% once equivalent and duplicate mutants were filtered out, and closing the gap increased test suite size by 18%. This was code that had undergone a thorough industrial assurance process and met the coverage criterion its standard specifies.

The opposite failure mode is bulk. Razorcat’s worked example in the TESSY manual compares two specifications for one function: the considered one carries 14 test cases against a tree with 15 leaf classes, the alternative carries four. Razorcat’s verdict on the short version is that it completely neglects the problem of a negative length, neglects wrap-around and does not force the usage of boundary values, so it minimises testing effort at the cost of thorough testing. Razorcat is just as clear in the other direction: combining every class with every other class is not the intention of the method, because it costs overview and execution effort. The objective is a minimal, non-redundant but sufficient set of test cases.

The NASA tutorial gives three reasons why even a requirements-adequate test set may not be a thorough test of the software: the requirements and design description may not contain a complete and accurate specification of all the behaviour represented in the executable code, the requirements may not be written with sufficient granularity to assure that all implemented functional behaviours are tested, and requirements-based testing alone cannot confirm that the code does not include unintended functionality. Structural coverage is how that third case surfaces. It detects missing tests, it does not certify good ones.

What Razorcat Builds Into TESSY

TESSY, developed by Razorcat Development GmbH in Berlin, automates dynamic unit, module and integration testing for embedded C and C++. The features bearing on test quality rather than test count are worth configuring first.

Per Razorcat, TESSY incorporates a requirement management environment that imports requirements as plain text, as ReqIF (the interchange format used for exchange with third-party requirement management tools such as DOORS and Polarion), as CSV or TSV, and as a TESSY-specific XML format, with a separate plug-in available for Polarion. Tasks, modules, test objects and test cases are then linked to those requirements in the Link Matrix view and the Requirements Coverage view. The manual describes the mechanism that keeps traceability honest: whenever a requirement changes, because of modifications or because a new version has been checked in, the linked elements become suspicious and need to be reviewed, and the Suspicious Elements view shows the reason an element is suspicious. Test design happens in the Classification Tree Editor, integrated into TESSY and also available standalone. Razorcat notes the CTE is not part of the certified core workflow, so treat it as the structured thinking aid it is.

On measurement, Razorcat lists statement (C0), branch (C1), decision, MC/DC, multiple condition, entry point, function and call pair coverage, the last of which Razorcat says fulfils the requirements of ISO 26262 as an alternate coverage method for integration testing instead of function coverage. Our MC/DC coverage guide covers that metric in depth, and the ISO 26262 unit testing checklist covers the surrounding workflow.

The qualification evidence is public. TÜV SÜD Rail’s report RB84018C version 2.0, dated 2023-11-13, supporting certificate Z10 078930 0004 Rev. 01, classifies TESSY as a class T2 offline support tool per IEC 61508-4:2010 and finds it suitable for use in safety-related development per IEC 61508:2010 and EN 50128:2011/A2:2020 for any SIL, qualified for a standard-conform development process per ISO 26262:2018 for any ASIL, and suitable under IEC 62304:2006+A1:2015 for any software safety class. For aerospace, Razorcat’s manual describes a separately purchased Tool Qualification Pack as the baseline for qualifying TESSY as a software verification tool according to DO-178B/C.

Keep the disciplines separate: Perforce Helix QAC is static analysis and reasons about code without executing it, while TESSY executes and compares, a split our static plus dynamic guide walks in full. Razorcat’s public product pages currently document TESSY 5.1, the version cited here, so ask GSAS to confirm the Version 6 scope for your programme.

Related reading

• Test Case Design with the Classification Tree Method, a systematic way to derive the cases, rather than inventing them
• Is 100% Code Coverage Enough?, why a full coverage figure does not tell you the suite is good
• Automated Mutation Testing for Embedded C, measuring whether the suite would actually catch a defect
• Razorcat TESSY, the tool these techniques are run with in India, the UAE and Sri Lanka

Where GSAS Fits

GSAS Micro Systems is the authorized Razorcat engineering partner for India, the UAE and Sri Lanka. That covers evaluation licences, cross-compiler and debugger setup, CI integration and, more usefully for this subject, review of the test specifications themselves: whether the equivalence partitioning is complete and unique, whether expected results were predicted from the specification rather than recorded from a run, and whether the traceability links would survive an audit after a requirement changes.

Our engineers support automotive, aerospace, rail, medical and industrial programmes from Bengaluru, Hyderabad, Chennai, Pune, Mumbai and Delhi NCR, with INR invoicing and one support channel. For context, see our ISO 26262 page, DO-178C page, IEC 61508 page, automotive solutions and the Razorcat partner page.

If your coverage numbers look healthy and your defect escape rate does not, that gap is what this article is about. Request a TESSY evaluation or a test suite review.

Sources

• Razorcat Development GmbH, TESSY product page and CTE product page: current publicly documented TESSY version and CTE positioning
• Razorcat Development GmbH, TESSY 5.1 Manual (PDF): preface and safety manual (core workflow, CTE exclusion, Tool Qualification Pack), chapter 3 “Theory: Basic knowledge” (unit testing, oracles, the Classification Tree Method and the worked example), and chapter 6 (requirement import formats, Link Matrix, Requirements Coverage and Suspicious Elements views, coverage measurements)
• TÜV SÜD Rail GmbH, Report RB84018C version 2.0 on Certificate Z10 078930 0004 Rev. 01 (PDF), 2023-11-13
• K. J. Hayhurst, D. S. Veerhusen, J. J. Chilenski, L. K. Rierson, A Practical Tutorial on Modified Condition/Decision Coverage, NASA/TM-2001-210876. Section 2.4 of the tutorial reproduces RTCA/DO-248A FAQ #44, which is the source of the structural-testing passage quoted here
• FAA, Advisory Circular 20-115D, Airborne Software Development Assurance Using EUROCAE ED-12( ) and RTCA DO-178( ), 2017
• P. Delgado-Pérez, I. Habli, S. Gregory, R. Alexander, J. Clark, I. Medina-Bulo, Evaluation of Mutation Testing in a Nuclear Industry Case Study, IEEE Transactions on Reliability, 2018
• R. Salay, R. Queiroz, K. Czarnecki, An Analysis of ISO 26262: Using Machine Learning Safely in Automotive Software, September 2017 (analyses the 2011 first edition of ISO 26262)
• M. Pezzè, M. Young, Software Testing and Analysis: Process, Principles and Techniques