In short
Robustness testing drives a unit through its normal interface with values it should never receive, while fault injection deliberately corrupts the unit's variables, registers or branch outcomes so that error handling and defensive code actually execute. ISO 26262-6:2018 lists fault injection test as a method for software unit verification in Table 7 and for software integration verification in Table 10, IEC 61508-3:2010 recommends defensive programming from SIL 2 upward and then notes that defensive code is what stops teams reaching 100 percent structural coverage, and DO-178C requires the Executable Object Code to respond correctly to abnormal inputs and conditions.
The Code the Happy Path Never Reaches
Every safety-related embedded unit contains code that correct inputs never execute: the range check that rejects an impossible sensor reading, the branch taken when a driver returns an error, the timeout that fires when a peripheral stops answering. That code exists because it is not supposed to run, and it is usually the last thing between a team and its coverage target.
In one paragraph: IEC 61508-3:2010 recommends defensive programming from SIL 2 upward, then concedes in a footnote that defensive code is exactly what stops teams reaching 100 percent structural coverage. ISO 26262-6:2018 lists fault injection test as a method for both software unit verification and software integration verification. DO-178C expects the executable object code to respond correctly to abnormal inputs and conditions, and is stricter than the industrial standards about what code-derived tests may be credited with. The practical answer is to stop treating robustness testing and fault injection as synonyms, and apply each where the standard asks for it. Razorcat implements the second as an automated feature in TESSY, and GSAS Micro Systems is the authorized Razorcat engineering partner for India, the UAE and Sri Lanka.
Robustness Testing and Fault Injection Are Not the Same Thing
Robustness testing, also called negative testing, drives the unit through its normal interface with values it should never receive. ISO 26262-6:2018 covers the derivation side in Table 8 for units and Table 11 for integration, listing equivalence classes, analysis of boundary values, and error guessing based on knowledge or experience. Its footnote says the boundary value method applies to interfaces, values approaching and crossing the boundaries, and out of range values.
Fault injection does not go through the interface. It deliberately corrupts the software or its state so the error path executes. Per the footnote to Table 7, in the context of software unit testing fault injection test means to modify the tested software unit, including injection of arbitrary faults by corrupting values of variables, introducing code mutations, or corrupting values of CPU registers. Per the footnote to Table 10, at integration level it means introducing faults into the software, in particular to test the correctness of the hardware-software interface related to safety mechanisms, and it can also be used to verify freedom from interference.
| Robustness / negative testing | Fault injection | Error seeding | |
|---|---|---|---|
| What you manipulate | Inputs at the interface | Internal state, variables, registers, branch outcomes | Defects planted in the code |
| Question answered | Does the unit cope when the caller misbehaves? | Does the error handling work when the fault occurs? | Is my test set adequate? |
| Standard hook | ISO 26262-6 Tables 8 and 11; DO-178C 6.4.2.2 | ISO 26262-6 Table 7 item 1l, Table 10 item 1c | IEC 61508-3 Table B.2 item 3; IEC 61508-7 C.5.6 |
Error seeding is the third column above: per IEC 61508-7:2010 clause C.5.6, its aim is to ascertain whether a set of test cases is adequate. It measures your tests, not your product.
What ISO 26262 Asks For
ISO 26262-6:2018 clause 7.4.12 requires safety mechanisms for error detection and error handling to be applied, depending on the results of the safety-oriented analyses at the software architectural level. Its notes name range checks of input and output data, plausibility checks, detection of data errors and watchdog monitoring for detection, and deactivation to a safe state, graceful degradation and correcting codes for handling. Every one of those is a branch that only runs once something has gone wrong. Clause 9.4.2 states the expectation directly, with the example of evidence for the effective implementation of the error detection and error handling mechanisms specified to achieve robustness of the software unit against erroneous inputs.
The methods that produce that evidence sit in the tables. In Table 7 (software unit verification), fault injection test is recommended for ASIL A, B and C and highly recommended for ASIL D, alongside resource usage evaluation with the same profile. In Table 10 (software integration), fault injection test is recommended for ASIL A and B and highly recommended for ASIL C and D, while resource usage evaluation, whose footnote names maximum execution times and storage usage such as RAM for stack and heap, is highly recommended at every ASIL. In Tables 8 and 11, equivalence classes and boundary values move from recommended at ASIL A to highly recommended from ASIL B upward.
Per the standard, ”++” means highly recommended and ”+” means recommended, and substitution requires a rationale. See our ISO 26262 compliance page and the ISO 26262 unit testing checklist.
What IEC 61508 Expects, and the Contradiction It Admits
IEC 61508-3:2010 comes at the same ground from the design side. Table A.4, detailed design, lists defensive programming at item 3, referencing IEC 61508-7 clause C.2.5: no recommendation at SIL 1, recommended at SIL 2, highly recommended at SIL 3 and SIL 4. C.2.5 defines the aim as producing programs which detect anomalous control flow, data flow or data values during their execution and react to these in a predetermined and acceptable manner: variables range checked, values checked for plausibility where possible, parameters type, dimension and range checked at procedure entry.
The verification side is where it bites. Table B.2 asks for test case execution from boundary value analysis (recommended at SIL 1, highly recommended from SIL 2), from error guessing and error seeding, equivalence classes and input partition testing, and structural test coverage at 100 percent across four graded rows: entry points (highly recommended at every SIL), statements (highly recommended from SIL 2), branches (highly recommended from SIL 3) and conditions with MC/DC (recommended at SIL 1 to 3, highly recommended at SIL 4). Table B.6 adds performance testing, including avalanche or stress testing. The order of that ladder is not the one most tool marketing implies, and we walk through it row by row in Is 100% code coverage enough?.
Then comes the footnote on those 100 percent coverage rows: where 100 percent coverage cannot be achieved, for example statement coverage of defensive code, an appropriate explanation should be given. The standard asks for defensive code, asks for full structural coverage, and in the same annex admits the two collide. Under IEC 61508 and ISO 26262, fault injection settles it by execution instead of by explanation. Our IEC 61508 compliance page covers the wider lifecycle expectations.
DO-178C Says It in Different Words
DO-178C treats robustness as an objective of the software verification process: the Executable Object Code must be robust with respect to the software requirements, meaning it responds correctly to abnormal inputs and conditions. The standard is paywalled, so this page does not quote a clause number for it. AdaCore’s public reference AdaCore Technologies for DO-178C / ED-12C gathers this under section 4.2.4, “Robustness / defensive programming”, whose activities list names Robustness Test Cases (6.4.2.2), and it names the cost: unexercised defensive code cannot be classified as extraneous, since it does meet a requirement, nor as deactivated, since it is intended to execute when the input is abnormal. That is the IEC 61508-3 footnote again, read from an aerospace desk.
DO-178C is stricter than the industrial standards on what you may do about it. Per the same AdaCore text, additional test cases based on the code itself, called structural testing, are not acceptable from a DO-178C / ED-12C perspective, since they are not appropriate for revealing errors, and as with any other non-exercised code, justification should be provided for defensive code. So in an airborne project, fault injection is a way to exercise and demonstrate a safety mechanism against a requirement, not a shortcut to structural coverage credit: the justification conversation with the certification authority still has to happen. See our DO-178C compliance page.
Turning the Conditions into Test Cases
| Condition the happy path never produces | How you produce it deliberately | Where the standards ask for it |
|---|---|---|
| Out-of-range and boundary inputs | Robustness data: minimum, maximum and near-boundary values via the interface | ISO 26262-6 Tables 8 and 11; IEC 61508-3 Table B.2 rows 1 and 6 |
| Corrupted variables, registers or interfaces | Fault injection at the unit or between components | ISO 26262-6 Table 7 item 1l, Table 10 item 1c |
| A peripheral or driver call that fails | Stub the called function and return the error value the caller must handle | ISO 26262-6 Table 10 footnote on the hardware-software interface |
| Unreachable defensive branches | Fault injection that manipulates the decision outcome at the branch | IEC 61508-3 Table B.2 footnote on defensive code |
| Stack, heap and timing exhaustion | Resource usage evaluation on the target environment | ISO 26262-6 Table 7 item 1m, Table 10 item 1d; IEC 61508-3 Table B.6 |
How Razorcat Implements Fault Injection in TESSY
TESSY is developed by Razorcat Development GmbH in Berlin, and everything below is Razorcat’s description of their own tool rather than an inference of ours. Razorcat states the purpose in the TESSY 5 flyer, under the heading “Fault injection for robustness testing”: unreachable branches and abnormal operating conditions can thoroughly be tested using TESSY’s automated fault injections that are based on the control flow of the source code. Per the TESSY 5.1 Manual, the feature provides means to test code parts that are not testable using normal testing inputs, for example endless loops, read-after-write functionality or error cases in defensive programming, and dedicated testing code can be injected at selected branch locations of the test object so that decision outcomes can be manipulated. Four details matter:
- Injections are scoped to marked test cases. Per Razorcat, they are valid for specially marked fault injection test cases only, so normal test cases run without side effects.
- Injections follow control flow, not line numbers. Each is identified by its branch path, which represents the decisions that need to be taken to reach the desired branch, and which per Razorcat allows finding the right location even after source code changes. Injections that can no longer be mapped to the control flow are flagged with an error symbol rather than silently ignored.
- The workflow starts from the coverage gap. Razorcat recommends running all normal test cases first, in order to reveal any unreached branches within the control flow shown in the Coverage Viewer; because the viewer knows which test cases reached a given decision, the Edit Fault Injection dialog then lists those test cases, ready to be copied as fault injection test cases for the unreached branch.
- The reason is recorded. The dialog takes an optional description of the cause of the fault injection, and per Razorcat that text appears within the test report.
Razorcat introduced automated fault injection with TESSY 4.1. Per their 2017 announcement, the manual alternatives, such as added testing code, compiler macros or debugger-driven variable changes, leave instrumentation inside production code, so that if the code changes, all necessary approvals according to safety standards need to be performed again. In TESSY, per Razorcat, fault injection is automated and does not remain in the source program.
Robustness data comes from two neighbouring capabilities. The Classification Tree Editor, per Razorcat, assists in creating low redundant and error sensitive test cases, and the TESSY 5.1 Manual documents *min* and *max* as boundary values and *min+1* and *max-1* as near boundary values among the special values available for test data. For failed peripherals, Razorcat documents stub functions for which TESSY can provide any value specified by the user as the return value, which the manual says reveals if the test object handles the return values of the stub function correctly.
Razorcat states that TESSY is qualified for safety-related software development according to IEC 61508, IEC 62304, ISO 26262 and EN 50128; the TUV SUD certificate Z10 078930 0004 Rev. 01 records classification as a T2 support tool per IEC 61508-3 and EN 50128. The certificate was issued on 2023-11-15 and is valid until 2028-11-12, and lists ISO 26262-8:2018, the tool qualification part, among the standards tested against, alongside IEC 61508-3:2010, IEC 62304:2006 with AMD1:2015 and EN 50128:2011/A2:2020. Razorcat’s public product pages currently document TESSY 5.1, so if your plan depends on a later release, ask GSAS to confirm Version 6 scope.
Getting This Done in India, the UAE and Sri Lanka
Fault injection is a habit more than a purchase. Teams that get value from it run requirements-based tests first, read the coverage gap as a list of unanswered questions rather than a number to argue down, classify each unreached branch as dead code, defensive code or a safety mechanism, then inject only where evidence is required.
GSAS Micro Systems is the authorized Razorcat engineering partner for India, the UAE and Sri Lanka. We supply TESSY and the Classification Tree Editor with local engineering support: environment setup for your compiler and debugger, host and on-target execution, fault injection and coverage workshops on your own code, and INR invoicing. Our engineers work with automotive, aerospace, railway and medical teams across Bengaluru, Hyderabad, Chennai, Pune, Mumbai and Delhi NCR, and support customers in the UAE and Sri Lanka from the same team. See our MC/DC coverage guide, Is 100% code coverage enough? and automotive solutions page.
If your coverage report is stuck behind a wall of defensive code, request a quote or a guided TESSY evaluation and we will scope a fault injection workflow against the ASIL or SIL evidence your assessor expects.
Related reading
- Unit Testing Fundamentals for Embedded C and C++, the isolation mechanics these techniques depend on
- Test Case Quality in Embedded Unit Testing, choosing which abnormal inputs are worth a case
- Automated Mutation Testing for Embedded C, the complementary question of suite strength
- Razorcat TESSY, the tool these techniques are run with in India, the UAE and Sri Lanka
Sources
- Razorcat Development GmbH: TESSY product page, TESSY 5 flyer, and “Testing error situations without changing the source code” (14 December 2017)
- Razorcat Development GmbH, TESSY 5.1 Manual (PDF, release 03/26, revision 51.014), chapters 6.11.6 and 6.14 (fault injection), plus the stub function and classification tree sections
- TUV SUD certificate Z10 078930 0004 Rev. 01 (issued 2023-11-15, valid until 2028-11-12) and test report RB84018C for TESSY
- ISO 26262-6:2018, clauses 4.3, 7.4.12, 9.4.2, 9.4.3, 10.4.2, 10.4.3 and Tables 7, 8, 10 and 11
- IEC 61508-3:2010, Annex A Table A.4 and Annex B Tables B.2 and B.6
- IEC 61508-7:2010, clauses C.2.5 (defensive programming), C.5.4 (boundary value analysis), C.5.6 (error seeding) and C.5.8 (structure-based testing)
- RTCA DO-178C / EUROCAE ED-12C, software verification process objectives and robustness test cases (paywalled; clause numbers not quoted here)
- AdaCore Technologies for DO-178C / ED-12C, Frédéric Pothon and Quentin Ochem (AdaCore), section 4.2.4, “Robustness / defensive programming”
Also appears in:
Interested in Razorcat tools?
Talk to our application engineers for personalized tool recommendations.
More from Razorcat
View all →
Test Case Design with the Classification Tree Method: Deriving Unit Tests You Can Defend in an Audit
1 Aug · 10 min
How to Evaluate a Unit Testing Tool for Embedded Software: A Buyer's Framework for Indian Teams
1 Aug · 10 min
Is 100% Code Coverage Enough? Statement, Branch, MC/DC and MCC, and What Each One Proves
1 Aug · 9 min
Automated Mutation Testing for Embedded C: The Question MC/DC Coverage Cannot Answer
1 Aug · 14 min
Test Case Quality in Embedded Unit Testing: Why Test Counts and Coverage Are Weak Proxies
1 Aug · 10 min
From Evaluation Licence to Production Licence: How Embedded Unit Test Tool Licensing Actually Works in India
1 Aug · 11 min