Fault Injection and Robustness Testing for Embedded Software: What ISO 26262, IEC 61508 and DO-178C Actually Ask For

The Code the Happy Path Never Reaches

Every safety-related embedded unit contains code that correct inputs never execute: the range check that rejects an impossible sensor reading, the branch taken when a driver returns an error, the timeout that fires when a peripheral stops answering. That code exists because it is not supposed to run, and it is usually the last thing between a team and its coverage target.

In one paragraph: IEC 61508-3:2010 recommends defensive programming from SIL 2 upward, then concedes in a footnote that defensive code is exactly what stops teams reaching 100 percent structural coverage. ISO 26262-6:2018 lists fault injection test as a method for both software unit verification and software integration verification. DO-178C expects the executable object code to respond correctly to abnormal inputs and conditions, and is stricter than the industrial standards about what code-derived tests may be credited with. The practical answer is to stop treating robustness testing and fault injection as synonyms, and apply each where the standard asks for it. Razorcat implements the second as an automated feature in TESSY, and GSAS Micro Systems is the authorized Razorcat engineering partner for India, the UAE and Sri Lanka.

Robustness Testing and Fault Injection Are Not the Same Thing

A note on citations. ISO 26262, IEC 61508 and DO-178C are licensed, copyrighted documents. GSAS does not publish their clause numbers, table numbers, row letters or rating grids on this site. Everything below states the obligation in substance, as GSAS engineering guidance or attributed to a public source such as AdaCore’s free DO-178C reference. Take it to your licensed copy and to your assessor.

Robustness testing, also called negative testing, drives the unit through its normal interface with values it should never receive. ISO 26262-6:2018 covers the derivation side twice, once for units and once for integration, listing equivalence classes, analysis of boundary values, and error guessing based on knowledge or experience. It notes that the boundary value method applies to interfaces, values approaching and crossing the boundaries, and out of range values.

Fault injection does not go through the interface. It deliberately corrupts the software or its state so the error path executes. Per ISO 26262-6:2018, in the context of software unit testing fault injection test means to modify the tested software unit, including injection of arbitrary faults by corrupting values of variables, introducing code mutations, or corrupting values of CPU registers. At integration level the same standard describes it as introducing faults into the software, in particular to test the correctness of the hardware-software interface related to safety mechanisms, adding that it can also be used to verify freedom from interference.

Error seeding is the third column above: per IEC 61508-7:2010, its aim is to ascertain whether a set of test cases is adequate. It measures your tests, not your product.

What ISO 26262 Asks For

ISO 26262-6:2018 requires safety mechanisms for error detection and error handling to be applied, depending on the results of the safety-oriented analyses at the software architectural level. Its notes name range checks of input and output data, plausibility checks, detection of data errors and watchdog monitoring for detection, and deactivation to a safe state, graceful degradation and correcting codes for handling. Every one of those is a branch that only runs once something has gone wrong. At the unit verification stage the standard states the expectation directly, giving as an example evidence for the effective implementation of the error detection and error handling mechanisms specified to achieve robustness of the software unit against erroneous inputs.

The methods that produce that evidence are set out as ASIL-rated method lists, and the shape of those lists is the useful part. At the software unit level, fault injection test is recommended across the ASILs and carries the standard’s strongest recommendation at ASIL D, and resource usage evaluation follows the same profile. At the software integration level, fault injection test again strengthens as the ASIL rises, while resource usage evaluation, described as covering maximum execution times and storage usage such as RAM for stack and heap, carries the strongest recommendation at every ASIL. On the test case derivation side, at both levels, equivalence classes and boundary values start out recommended at ASIL A and reach the strongest recommendation from ASIL B upward.

Per the standard, ”++” means highly recommended and ”+” means recommended. Neither is a mandate, and substitution requires a rationale. See our ISO 26262 compliance page and the ISO 26262 unit testing checklist.

What IEC 61508 Expects, and the Contradiction It Admits

IEC 61508-3:2010 comes at the same ground from the design side. Among its detailed design techniques it lists defensive programming, expanded in IEC 61508-7, with no recommendation at SIL 1, recommended at SIL 2, and the strongest recommendation at SIL 3 and SIL 4. IEC 61508-7 defines the aim as producing programs which detect anomalous control flow, data flow or data values during their execution and react to these in a predetermined and acceptable manner: variables range checked, values checked for plausibility where possible, parameters type, dimension and range checked at procedure entry.

The verification side is where it bites. Among its dynamic analysis and testing techniques, IEC 61508-3 asks for test case execution from boundary value analysis, which strengthens from recommended at SIL 1 to the strongest recommendation from SIL 2, from error guessing and error seeding, equivalence classes and input partition testing, and 100 percent structural test coverage graded as a ladder: entry points, then statements, then branches, then conditions with MC/DC, each step reaching its strongest recommendation at a higher SIL than the one below it, with MC/DC doing so only at SIL 4. A separate list adds performance testing, including avalanche or stress testing. The order of that ladder is not the one most tool marketing implies, and we walk through it in Is 100% code coverage enough?.

Then comes the qualification on those 100 percent coverage expectations: where 100 percent coverage cannot be achieved, statement coverage of defensive code being the example the standard gives, an appropriate explanation should be given. The standard asks for defensive code, asks for full structural coverage, and in the same annex admits the two collide. Under IEC 61508 and ISO 26262, fault injection settles it by execution instead of by explanation. Our IEC 61508 compliance page covers the wider lifecycle expectations.

DO-178C Says It in Different Words

DO-178C treats robustness as an objective of the software verification process: the Executable Object Code must be robust with respect to the software requirements, meaning it responds correctly to abnormal inputs and conditions. The standard is paywalled, so this page does not quote a clause number for it. AdaCore’s public reference AdaCore Technologies for DO-178C / ED-12C gathers this under “Robustness / defensive programming”, whose activities list names robustness test cases, and it names the cost: unexercised defensive code cannot be classified as extraneous, since it does meet a requirement, nor as deactivated, since it is intended to execute when the input is abnormal. That is the IEC 61508-3 qualification again, read from an aerospace desk.

DO-178C is stricter than the industrial standards on what you may do about it. Per the same AdaCore text, additional test cases based on the code itself, called structural testing, are not acceptable from a DO-178C / ED-12C perspective, since they are not appropriate for revealing errors, and as with any other non-exercised code, justification should be provided for defensive code. So in an airborne project, fault injection is a way to exercise and demonstrate a safety mechanism against a requirement, not a shortcut to structural coverage credit: the justification conversation with the certification authority still has to happen. See our DO-178C compliance page.

Turning the Conditions into Test Cases

How Razorcat Implements Fault Injection in TESSY

TESSY is developed by Razorcat Development GmbH in Berlin, and everything below is Razorcat’s description of their own tool rather than an inference of ours. Razorcat states the purpose in the TESSY 5 flyer, under the heading “Fault injection for robustness testing”: unreachable branches and abnormal operating conditions can thoroughly be tested using TESSY’s automated fault injections that are based on the control flow of the source code. Per the TESSY 5.1 Manual, the feature provides means to test code parts that are not testable using normal testing inputs, for example endless loops, read-after-write functionality or error cases in defensive programming, and dedicated testing code can be injected at selected branch locations of the test object so that decision outcomes can be manipulated. Four details matter:

1. Injections are scoped to marked test cases. Per Razorcat, they are valid for specially marked fault injection test cases only, so normal test cases run without side effects.
2. Injections follow control flow, not line numbers. Each is identified by its branch path, which represents the decisions that need to be taken to reach the desired branch, and which per Razorcat allows finding the right location even after source code changes. Injections that can no longer be mapped to the control flow are flagged with an error symbol rather than silently ignored.
3. The workflow starts from the coverage gap. Razorcat recommends running all normal test cases first, in order to reveal any unreached branches within the control flow shown in the Coverage Viewer; because the viewer knows which test cases reached a given decision, the Edit Fault Injection dialog then lists those test cases, ready to be copied as fault injection test cases for the unreached branch.
4. The reason is recorded. The dialog takes an optional description of the cause of the fault injection, and per Razorcat that text appears within the test report.

Razorcat introduced automated fault injection with TESSY 4.1. Per their 2017 announcement, the manual alternatives, such as added testing code, compiler macros or debugger-driven variable changes, leave instrumentation inside production code, so that if the code changes, all necessary approvals according to safety standards need to be performed again. In TESSY, per Razorcat, fault injection is automated and does not remain in the source program.

Robustness data comes from two neighbouring capabilities. The Classification Tree Editor, per Razorcat, assists in creating low redundant and error sensitive test cases, and the TESSY 5.1 Manual documents *min* and *max* as boundary values and *min+1* and *max-1* as near boundary values among the special values available for test data. For failed peripherals, Razorcat documents stub functions for which TESSY can provide any value specified by the user as the return value, which the manual says reveals if the test object handles the return values of the stub function correctly.

Razorcat states that TESSY is qualified for safety-related software development according to IEC 61508, IEC 62304, ISO 26262 and EN 50128; the TUV SUD certificate Z10 078930 0004 Rev. 01 records classification as a T2 support tool per IEC 61508-3 and EN 50128. The certificate was issued on 2023-11-15 and is valid until 2028-11-12, and lists ISO 26262-8:2018, the tool qualification part, among the standards tested against, alongside IEC 61508-3:2010, IEC 62304:2006 with AMD1:2015 and EN 50128:2011/A2:2020. Razorcat’s public product pages currently document TESSY 5.1, so if your plan depends on a later release, ask GSAS to confirm Version 6 scope.

Getting This Done in India, the UAE and Sri Lanka

Fault injection is a habit more than a purchase. Teams that get value from it run requirements-based tests first, read the coverage gap as a list of unanswered questions rather than a number to argue down, classify each unreached branch as dead code, defensive code or a safety mechanism, then inject only where evidence is required.

GSAS Micro Systems is the authorized Razorcat engineering partner for India, the UAE and Sri Lanka. We supply TESSY and the Classification Tree Editor with local engineering support: environment setup for your compiler and debugger, host and on-target execution, fault injection and coverage workshops on your own code, and INR invoicing. Our engineers work with automotive, aerospace, railway and medical teams across Bengaluru, Hyderabad, Chennai, Pune, Mumbai and Delhi NCR, and support customers in the UAE and Sri Lanka from the same team. See our MC/DC coverage guide, Is 100% code coverage enough? and automotive solutions page.

If your coverage report is stuck behind a wall of defensive code, request a quote or a guided TESSY evaluation and we will scope a fault injection workflow against the ASIL or SIL evidence your assessor expects.

Related reading

• Unit Testing Fundamentals for Embedded C and C++, the isolation mechanics these techniques depend on
• Test Case Quality in Embedded Unit Testing, choosing which abnormal inputs are worth a case
• Automated Mutation Testing for Embedded C, the complementary question of suite strength
• Razorcat TESSY, the tool these techniques are run with in India, the UAE and Sri Lanka

Sources

• Razorcat Development GmbH: TESSY product page, TESSY 5 flyer, and “Testing error situations without changing the source code” (14 December 2017)
• Razorcat Development GmbH, TESSY 5.1 Manual (PDF, release 03/26, revision 51.014), chapters 6.11.6 and 6.14 (fault injection), plus the stub function and classification tree sections
• TUV SUD certificate Z10 078930 0004 Rev. 01 (issued 2023-11-15, valid until 2028-11-12) and test report RB84018C for TESSY
• ISO 26262-6:2018, on software architectural design safety mechanisms, software unit verification and software integration verification (paywalled; clause, table and row identifiers are not reproduced here)
• IEC 61508-3:2010, on detailed design techniques and on dynamic analysis and testing techniques (paywalled; annex, table and row identifiers are not reproduced here)
• IEC 61508-7:2010, on defensive programming, boundary value analysis, error seeding and structure-based testing (paywalled; clause identifiers are not reproduced here)
• RTCA DO-178C / EUROCAE ED-12C, software verification process objectives and robustness test cases (paywalled; clause numbers not quoted here)
• AdaCore Technologies for DO-178C / ED-12C, Frédéric Pothon and Quentin Ochem (AdaCore), “Robustness / defensive programming”