In short
Test automation for software-defined vehicles keeps one test case running across desktop simulation, virtual ECUs and hardware-in-the-loop benches. ASAM XIL is the public API standard that makes this possible: ASAM describes it as an API standard enabling communication between test automation tools and test benches, supporting model-in-the-loop, software-in-the-loop and hardware-in-the-loop stages, and states that it decouples test cases from real and virtual test systems. FMI supplies the other half, a free standard defining a container and an interface to exchange dynamic simulation models, distributed as a ZIP file, so plant models move between tools instead of being rebuilt. What does not move left is physical layer behaviour, time error measured at the pin, accredited conformance evidence and real electrical fault reaction.
The first automated test on a software-defined vehicle programme is easy. Someone writes a script, points it at a bench, and it passes. The problem arrives eight months later, when there are four benches, two simulation environments, three hundred test cases and a merge queue, and every case was written against whatever interface its author had in front of them.
This article covers the two public standards that let one test case survive the move from a laptop to a virtual ECU to a bench, the ladder those targets sit on, and what it takes to run a bench inside continuous integration. Our domain versus zonal explainer and our walkthrough of a vehicle communication architecture describe the network the bench reproduces.
What software-defined vehicle programmes change about testing
Release cadence moves from milestones to continuous integration
On a classical programme, integration is an event: software arrives at a sample gate, a team runs a campaign, defects come back, and the cycle repeats before start of production. When software keeps changing after the vehicle is in the field, that breaks. Automation becomes a gate on every change, so it must run unattended and return a verdict a merge decision can rest on, inside the window a developer will wait.
Bench time becomes the scarce resource, so tests must be portable and unattended
Gate every change and executions per week rise sharply, moving the constraint from engineer hours to bench hours. Two levers exist: buy more benches, which is slow and scales badly, or move every test that does not need the bench off it. Both need a case that can execute against more than one target.
The goal: the same test case, three execution targets
One case, written once, running against a desktop simulation, a virtual ECU with simulated peers, and real hardware, with the same pass criteria and result format. A case that measures a voltage never will, but the band checking behaviour, sequence and state holds most of the volume. That needs an interface between the case and whatever executes it.
ASAM XIL, in one page
An API standard between test automation tools and test benches
ASAM’s description is short enough to quote: XIL is “an API standard enabling communication between test automation tools and test benches”, it “supports all stages of development and testing, including model-in-the-loop (MIL), software-in-the-loop (SIL), and hardware-in-the-loop (HIL)”, and it “decouples test cases from real and virtual test systems, enabling easy transfer and reuse of tests across different systems with minimal migration”.
Test-bench access and model access, and why the second one is what makes tests portable
ASAM names two key components, the Framework and the Testbench, and lists the Testbench ports: Model Access, ECU Access, Diagnostic Access, Electrical Error Simulation, Network Access and Service-Oriented Access. Because the ports are named by the standard rather than by a product, a case written against them says what it does, not which tool it does it with. Model access matters most for shift-left, because the same variable exists in all three environments.
What portability actually buys, and what still has to be rewritten
A standard API makes the calls portable, not the case target-independent. Variable paths still change with the model, so real portability needs a naming layer with one mapping file per target: small work early, large work late. Timing assumptions change too, so write criteria against events and states, and mark timing-bound cases bench-only.
Where the model access companion standard fits
ASAM also publishes XIL-MA, “an extract from ASAM XIL” containing “the model access port specification as well as some common functionality from this standard”. XIL-MA tools “are fully compatible to ASAM XIL compliant tools, but have only the functionality of the model access port”, while full XIL “provides more ports for measurement, calibration, diagnostics, network access and electrical error simulation”. On a bench-automation project that difference is most of what you wanted, so establish which a supplier means.
Virtual ECUs and FMI packaged models
What a virtual ECU is, and the levels of virtualisation you will see described
A virtual ECU is the ECU’s software running on a host computer instead of the production microcontroller, with whatever it expects from hardware supplied by models. Level taxonomies circulate, mostly one vendor’s property, so learn the axis rather than the label. At low fidelity only application code is compiled for the host and everything below it is stubbed. Higher up, more of the real basic software is included and peripherals are modelled, so scheduling, communication stacks and diagnostic handling come under test. At the top, the production binary runs unmodified on an instruction-set simulator of the target core. More real software means slower execution, so a merge gate wants the fast end.
FMI and the FMU package, the interchange format that lets plant models move between tools
The plant model usually comes from another team’s tool, and FMI exists so it is not rewritten. The FMI project describes the Functional Mock-up Interface as “a free standard that defines a container and an interface to exchange dynamic simulation models using a combination of XML files, binaries and C code, distributed as a ZIP file”. That ZIP file is the Functional Mock-up Unit: a modelDescription.xml declaring variables and capabilities, binaries for the platforms the exporter supported, optional C sources, and a resources directory.
FMI 3.0 defines three interface types, and the third matters on a bench. Model Exchange exposes the equations and leaves integration to the importing tool’s solver; Co-Simulation packages a solver inside the FMU and steps it at communication points; Scheduled Execution exposes model partitions so an external scheduler can run each one, which the specification gives as a real-time platform case. Settle which one your exporter produces before the model is written.
What a virtual ECU genuinely proves about application logic
This rung combines production software with parallel execution the bench cannot offer: many variants run at once on a build farm, and a failing case re-runs deterministically under a debugger. Service-oriented behaviour belongs here, since a service that never appears, a refused subscription or a peer that restarts is all application logic. Our article on debugging SOME/IP service discovery lists the modes worth automating, as does our response pending and P2 timeout explainer.
What it cannot prove: physical layer, real timing, real faults
Three hard limits, taken up again at the end of this article. A virtual link always trains. Host timing is not target timing. And a model of a wire does not behave like a cut wire.
The shift-left ladder
Stage, coverage and cost, on one page
| Stage | What runs | What it catches | What it cannot catch | Cycle time, illustrative |
|---|---|---|---|---|
| Desktop simulation, unit and integration test | Application code plus stubs | Logic defects, interface misuse, coverage gaps | Real stack, network, hardware | Seconds to minutes |
| Software in the loop | Production software on a host, plant and peer models | Service discovery, diagnostics, error handling | Physical layer, target timing, faults | Minutes to tens of minutes |
| Hardware in the loop | Production binary, real silicon, real links, fault hardware | Timing on target, link behaviour, fault reaction, wake and sleep | Emergent behaviour, harness routing, EMC | Tens of minutes to hours, plus queue |
| Vehicle and fleet | The complete vehicle | Emergent interactions, EMC reality, real usage | Almost nothing, at the highest cost | Days to weeks |
Those cycle times are orders of magnitude for a planning conversation, not measurements: wrong in the specifics, roughly right in the ratios. The ratios are the argument, because each rung down costs materially more per execution.
Desktop simulation and unit or integration test
The top rung is unit and integration test tooling, a different discipline with its own toolchain and team. Our ISO 26262 page notes that statement, branch and modified condition/decision coverage sit at unit level among the verification methods the standard lists, so a coverage obligation lands here and nowhere else.
Software in the loop with virtual ECUs and simulated peers
The device under test needs a plausible network, not a message source: services offered and withdrawn, subscriptions that can be refused, peers that restart. Our article on rest bus simulation for Ethernet-based vehicles sets out the fidelity ladder, and where a simulator that is too perfect stops being a valid peer.
Hardware in the loop with real ECUs, real links and fault hardware
This is where fault behaviour becomes testable for real, which our companion article on fault injection in HIL benches treats as a design problem. That ASAM XIL names an Electrical Error Simulation port tells you fault injection was understood as part of bench automation.
Vehicle and fleet, and what only survives there
The bottom rung catches what no bench models: harness routing, the real electromagnetic and thermal environment, and user behaviour nobody wrote a requirement for. Everything above it exists so that what it finds is genuinely emergent.
Continuous integration for HIL, practically
The bench as a queued shared resource: reservation, setup, teardown, known-good baseline
A bench is a queue with one server, not an agent in a pool. Every job acquires a reservation the pipeline enforces; a booking spreadsheet fails the first time two people merge in the same hour. Setup brings the bench to a defined baseline: known images on every peer device, known model version, known network configuration, links verified up before the first case runs. Teardown restores that baseline and releases the reservation, and must run even when the job crashes, because a leaked reservation blocks the queue and a blocked queue is how HIL leaves CI. A bench starting from whatever the last run left behind gives order-dependent results.
Making a bench run deterministic enough to gate a merge
A merge gate has one hard requirement: a failure must mean the change is bad, or engineers learn to ignore red and the gate becomes decorative. Pin everything not under test, including peer software versions, model versions and switch configuration, so one thing varies between runs. Check preconditions explicitly and fail fast with a distinct status, so a bench that came up wrong reports “bench not ready” rather than “test failed”. Write pass criteria against observable events, and keep out of the gate any test whose result legitimately varies.
Flake triage: distinguishing a real regression from a bench artefact
Every intermittent failure has a cause, and the triage question is whether it lives in the device under test or in the bench. Answering that afterwards means having captured enough at the time: the network capture, the bench’s state log, the versions loaded, and timestamps to correlate them. Our guide to correlating captures across CAN, LIN, FlexRay and Ethernet covers correlation, and our taps and mirror ports guide covers capturing without changing what you measure.
Then classify rather than retry. A failure whose signature tracks the software version is a regression; one that tracks the bench, the slot or the preceding test is a bench defect, ticketed as seriously as a product defect. The habit that destroys a suite is the blanket automatic retry, because it turns a genuine intermittent product defect into a green result.
Which suites gate a pull request, which run nightly, which run before a release
Tier by cost and blast radius. A pull request gate should be fast, run mostly above the bench, and hold only deterministic cases a change could plausibly break: unit and integration tests, a virtual ECU smoke suite, and a small bench suite over the paths that break most often, typically startup, link-up, service availability and basic diagnostics. Nightly carries the bulk of the bench suite and the cases that take real time, including the sleep and wake behaviour our TC10 article describes; it can afford some non-determinism, since its output is a defect list. Pre-release runs everything. Publish the tiering, or it is re-litigated every sprint.
Bench configuration held as version-controlled code
None of this survives if the bench’s configuration lives in a GUI and an engineer’s memory. Switch configuration, VLAN and priority assignment, addressing, peer software versions, model versions and the wiring map belong in text beside the test cases, applied by setup and verified rather than assumed. A result from three months ago is then re-creatable from its commit, bench changes become reviewable, which is what stops four identical benches quietly ceasing to be identical, and a new bench is provisioned by applying a file.
Results, traceability and evidence
Requirement to test case to result, and why it is easier to build in than to retrofit
This chain is cheap while there are ten cases and expensive once there are three hundred across three targets. Put the requirement identifier in the test case source, carry it into the machine-readable result, and store it in the artefact bundle, so it travels with the case rather than the tool. Our ISO 26262 page notes that the strongest rating the standard applies to any method is “highly recommended” rather than mandatory, and that a chosen combination needs a rationale. What is wanted is a defensible chain, not a tool that claims to produce compliance.
Storing captures with results so a failure can be diagnosed weeks later
A verdict without evidence is unusable once the bench has moved on. Store captures with the result, not in a share someone will tidy up, and set retention deliberately: failures kept long, passes kept briefly or sampled, and a manifest of what was kept, so a later reader can tell a missing capture from a quiet one.
Reporting formats that survive a tool change
Emit results in a machine-readable format no single supplier owns, and treat a vendor report as a rendering of it. If the tool were replaced next quarter, would two years of results still be readable and comparable? If not, they are hostage to a licence.
What does not move left
Four categories resist virtualisation, and a plan that pretends otherwise fails late, which is the worst time.
Physical layer behaviour and link quality
A modelled link always comes up. A real one does not, and the reasons are wiring, PHY configuration, master and slave role configuration and signal quality, which our 100BASE-T1 link-up checklist works through step by step. None of that is reachable from a virtual ECU: there is no wire to be wrong.
Time synchronisation measured at the pin
A virtual environment can prove the time synchronisation state machine behaves. It cannot produce the number an OEM asks for, because that is a time error measured against a reference outside the device over a stated observation window, as our article on validating 802.1AS sets out. A device’s reported offset comes from the timestamps its servo steers with, so it cannot see errors common to both.
Conformance and interoperability evidence
Protocol behaviour can be pre-tested in-house. The report cannot be. As our conformance article explains, the accredited half needs calibrated fixtures, traceable instrumentation and a validated reference system behind every implemented test case, and test houses must hold accreditation and be neutral from the supply chain. In-house pre-compliance produces a defect list, not a report.
Electrical fault reaction
An open circuit, a short to battery and a short to ground have consequences at the PHY and at the supply that a model does not reproduce. Making the fault takes hardware, which is why fault-insertion units exist as a market category, and it belongs on the bench. Scripted protocol-level faults do move left, since crafted and out-of-specification frames can be generated against a virtual ECU with tooling such as Scapy. The electrical half stays.
Building the toolchain without locking yourself in
What to test during evaluation, not after the purchase order
| Criterion | Why it matters | How to test it during evaluation |
|---|---|---|
| Standard API support, and which ports | Decides whether cases outlive the tool | Ask which ASAM XIL ports are implemented, at which version, then run a case through them |
| Headless operation | A tool needing a GUI cannot be a CI job | Run a suite from a command line, no display, clean checkout |
| Machine-readable results and exit codes | The pipeline decides from these, not a PDF | Confirm the exit code separates pass, fail and infrastructure error; parse the file yourself |
| Model import path | Decides whether plant models are portable | Import an FMU your simulation team exported; check the FMI interface type |
| Configuration as text | Enables version control and review | Export a bench configuration, diff two benches, re-import on a third |
| Behaviour under parallelism | CI multiplies concurrent executions | Run as many simultaneous sessions as the pipeline will need |
Standard API support versus a proprietary scripting layer
Most tools offer both, and the proprietary layer is usually nicer to write. That is the trap, because the nicer layer is where the lock-in lives. Allow it only inside a thin adapter the cases call through, so a rewrite touches one file.
Headless operation, exit codes and machine-readable results
Verify these yourself, because “supports command line execution” and “runs a full suite unattended with no display and returns a distinguishable exit code” are different claims.
Model import path, and who owns the plant models
Decide early who owns the plant model, which tool authors it, and which FMI interface type it exports as, because a model authored without a real-time export path becomes a rewrite at the bench.
Migration cost when one tool in the chain is replaced
The useful question is not whether you will replace a tool but what it would cost if you did. Walk it through separately for the automation tool, the simulation environment and the bench hardware. Where everything would be rewritten, insist on a standard interface; where only a mapping file changes, the lock-in is contained.
Where GSAS fits
Most of the difficulty here is not the standards. It is working out which rung of the ladder your defects sit on, and therefore which problem is worth money. A team losing weeks to service discovery defects needs better software-in-the-loop coverage and a more honest peer model. A team losing weeks to bench queueing needs suite tiering, not another bench. Different purchases, and the wrong one is expensive.
GSAS is an engineering partner, and the useful first conversation is a review rather than a quotation: what your suites cover, where they execute, which failures are regressions and which are bench artefacts, and where a standard interface would remove a real constraint. Our applications engineers work with teams in Bengaluru, Pune, Chennai and Hyderabad, mostly on benches that already exist. We also work on unit and integration test tooling, which is the top rung of the ladder and a different discipline; the two are not substitutes.
To scope that conversation, our automotive Ethernet capabilities page sets out where we work, and a note through request a quote reaches the applications team. Tell us your cycle time and where the queue forms, and we will start there.
References
- ASAM, ASAM XIL standard page, for the API scope, the Framework and Testbench, and the named ports: https://www.asam.net/standards/detail/xil/
- ASAM, ASAM XIL-MA standard page, for the model access port and its relationship to full ASAM XIL: https://www.asam.net/standards/detail/xil-ma/
- Modelica Association Project FMI, Functional Mock-up Interface, for the definition of FMI and the FMU package: https://fmi-standard.org/
- FMI 3.0.2 specification documentation, for the three interface types and the FMU archive contents: https://fmi-standard.org/docs/3.0.2/
- Scapy documentation, automotive protocol layers, for crafted CAN, UDS, SOME/IP and DoIP frame generation: https://scapy.readthedocs.io/en/latest/layers/automotive.html
- GSAS Micro Systems, ISO 26262 compliance page, for verification methods, structural coverage and method rationale: https://gsasindia.com/compliance/iso-26262
- GSAS Micro Systems, Automotive Ethernet Conformance: TC8 and Testing Above It, for the accreditation and instrumentation behind conformance evidence: https://gsasindia.com/blog/automotive-ethernet-conformance-tc8-and-above
Also appears in:
Building for Automotive & Mobility?
Talk to our application engineers for personalized tool recommendations.
