Tests: The Verification Side

10.1 The .tst file: structure and the test methods enumeration

A .tst file holds a test set: one or more def testcase clauses, each verifying one or more requirements. The structure is simple. The substance sits in two properties: method (which testing method is used) and satisfies (which requirements the test verifies).

File structure

A .tst file starts with use requirementset, so that satisfies references resolve, and use functionset (less common), so that function references resolve. Then comes hdef testset <Name> with its properties. The body is a flat list of def testcase clauses at column 2.

use requirementset CobotSafetyRequirements
use functionset CobotFunctions

hdef testset CobotSafetyTests
  name "Cobot Safety Test Suite"
  owner "Cobot Test Engineering"
  safetylevel SIL-3
  level system
  tags "HIL", "SIL-3", "ISO-10218"

  def testcase TC_COB_SPEED_LIMIT
    name "Safely-monitored reduced speed enforcement"
    method HIL
    level system
    safetylevel SIL-3
    setup """
      Cobot mounted in safety test cell with operator-presence
      simulator and TCP speed measurement laser.
    """
    steps """
      1. Initialize cobot in operator-presence mode.
      2. Command TCP motion at 1000 mm/s along X axis.
      3. Measure achieved TCP speed via laser.
      4. Verify speed clamped to <= 250 mm/s within 100 ms.
    """
    expected """TCP speed reaches <= 250 mm/s within 100 ms; no overshoot."""
    passcriteria """Speed limit enforced in all 20 trial runs."""
    testresult notrun
    satisfies ref requirement REQ_COB_001

The method enumeration

The method property names the verification approach. The enumeration defines seven values: MIL, SIL, PIL, HIL, VIL, manual and automated. Each is a recognized approach, with its own implications for the test environment, the artifacts the test depends on and the evidence it produces.

• MIL (Model-in-the-Loop). The test runs against an executable model of the system, typically in Simulink, SCADE or a similar modeling environment. You use it early, before code exists, and it gives you evidence that the algorithm is correct against the model's specification.
• SIL (Software-in-the-Loop). The test runs compiled production code in a simulated environment: the plant model is simulated, the code under test is real. It gives you evidence that the implementation realizes the algorithm correctly.
• PIL (Processor-in-the-Loop). The test runs compiled production code on the target processor, with the rest of the system simulated. It gives you evidence of correct execution on real silicon and catches processor-specific issues such as timing and numerical precision.
• HIL (Hardware-in-the-Loop). The test runs against the real ECU with real I/O hardware, in a simulated environment. It is the dominant method for ASIL-D / SIL-3 verification, because it produces evidence under conditions close to production deployment.
• VIL (Vehicle-in-the-Loop). The test runs in the real vehicle or platform, in a scripted but controlled environment. It is the most production-realistic method and is used for integration-level verification.
• manual. A human executes the test, typically for usability or external-interaction checks that can't be automated.
• automated. A script executes the test, typically as part of a continuous-integration regression suite. It often pairs with another method: an automated SIL test is a SIL test run in CI.

What you're testing constrains the method. An algorithm specified at MIL level can be verified at MIL, code at SIL or PIL, a complete ECU at HIL and an integrated system at VIL. You typically verify a requirement with the lowest method that produces sufficient evidence. Using HIL on an algorithm that MIL could verify adequately costs more and doesn't buy you better evidence.

What's next

Subchapter 10.2 covers the test-case body (the setup, steps, expected and passcriteria conventions) and the satisfies discipline. Subchapter 10.3 covers the testresult lifecycle. Subchapter 10.4 walks through the cobot anchor's complete test set, with mirrors from the other anchors.

Exercises

10.2 The test-case body and the satisfies discipline

A test case's body (setup, steps, expected, passcriteria) is the prose specification a test engineer follows to run the test. The satisfies relationship traces the test back to the requirement(s) it verifies. In this subchapter we'll cover both, with clear conventions for what each property should contain.

The four body properties

setup is what must be in place before the test begins: the test rig, the configuration of the device under test, any fixtures or stimulus generators, and any precondition state. A good setup is specific enough that two test engineers reading it would prepare materially identical conditions. A vague setup (the cobot must be running) leaves too much room for variation in execution. If you've ever had to rerun a test because someone prepared the rig differently, you know why this matters.

steps is the ordered sequence of actions the test engineer (or the automation) performs. Number them, write them as imperatives, and keep to one observable action per step. Steps should refer to setup elements by name (the test rig, the stimulus generators) and leave no doubt about what is observed at each step.

expected is what the device under test should do in response to the steps. It must be specific, observable and tied to the requirement's prescriptive content. A vague expected outcome (the cobot responds correctly) defeats the purpose of the test.

passcriteria sets the pass/fail threshold. A test that showed the expected behavior in 19 of 20 trials may pass or fail depending on its pass criteria, so state them explicitly and remove the guesswork. Good pass criteria give the success rate, the tolerance bands and the persistence requirement (the behavior must hold for N seconds, or across N consecutive trials).

The satisfies discipline

Every test case should carry at least one satisfies ref requirement <ID> clause naming the requirement(s) it verifies. The relationship is one-sided: it lives only on the test case, and the requirement has no inverse testedby (see Chapter 9 Subchapter 1). It is the sole formal trace between requirement and test.

A test case can satisfy more than one requirement. In the cobot anchor, a single HIL test case may verify both REQ_COB_001 (TCP speed limit) and REQ_COB_002 (response time of the speed limiter), because one scenario exercises both. The satisfies clause names both. That keeps related requirements verified by related tests, cutting the test count while preserving coverage.

It works the other way too: one requirement can be satisfied by several test cases, and the coverage analysis aggregates across all .tst files to find them. A safety-critical requirement is typically verified at several levels: MIL for the algorithm, SIL for the implementation, HIL for the production code on the target ECU. Each of those test cases satisfies the same requirement, which gives you defense-in-depth verification evidence.

derivedfrom: the secondary trace

A test case can also carry derivedfrom ref requirement <ID>, which differs subtly from satisfies. satisfies says this test verifies that the requirement holds; derivedfrom says this test was designed because of the requirement. Usually both point at the same target and the distinction doesn't matter. Sometimes, though, a test explores behavior that is implicit in the requirement without being stated by it, and then it helps to tell the two apart.

The practical rule: always author satisfies, because it's the verification trace the coverage analysis uses. Add derivedfrom when the test exists because of a specific requirement but doesn't directly verify it, such as exploratory or stress tests that probe around a requirement.

What's next

Subchapter 10.3 covers the testresult lifecycle: how a test moves from authored (notrun) through execution to its terminal state. Subchapter 10.4 walks through the cobot anchor's test set and shows mirrors from the other anchors.

Exercises

10.3 The testresult lifecycle and execution evidence

The testresult property tracks a test case's execution state. Its enumeration has five values, listed below. In this subchapter we'll walk through the lifecycle those values form and the evidence each one represents.

• notrun: the initial state for a test that is authored but not yet executed.
• intest: the test is executing now, or is in a partially complete state.
• pass: the test executed and met its pass criteria.
• fail: the test executed and did not meet its pass criteria.
• blocked: the test can't execute for reasons external to the device under test, such as an unavailable environment or unmet dependencies.

The initial state: notrun

Every test case starts life as testresult notrun. That means the test is designed (setup, steps, expected and passcriteria are specified) but hasn't been run against any version of the device under test. Coverage analysis treats notrun tests as unverified for safety-case completeness. A safety requirement satisfied only by notrun tests has no verification evidence yet.

Most tests authored by AI agents (in the TestEngineer role from Chapter 5) start as notrun and stay there until the test-execution team runs them. Designing without executing is the common case, so notrun is the normal starting value.

The terminal states: pass, fail, blocked

Once executed, the test moves to pass, fail or blocked.

1. pass. The test executed and met its pass criteria, so the safety case can cite it as evidence that the requirement is satisfied. Record execution metadata with the result (date, executor, build version of the device under test, test rig configuration) in the commit history at the time you record it.
2. fail. The test executed and did not meet its pass criteria, so investigate. The defect usually sits in the device under test (fix and re-test), in the test itself (correct the test) or in the requirement (update it and re-derive the test). fail shouldn't persist for long, because it signals open work.
3. blocked. The test can't execute for reasons outside the device under test. The most common causes are a missing test environment (the HIL rig is unavailable) or a missing dependency (another test must pass first). blocked means the test is queued; track the cause through the project-management constructs (Chapter 5).

The transient state: intest

intest marks a test that is partially complete: typically a long-running test that has started but hasn't produced its terminal result. You'll use it less often than the others, since most tests are short enough that the move from notrun to a terminal state is atomic. It matters for tests that span several days or depend on accumulated execution time, such as durability tests, EMC sweeps and long-duration thermal cycling.

Execution evidence and the audit trail

The testresult value alone isn't enough evidence for the safety case. An auditor looking at a pass also needs to see what was actually executed: the rig configuration, the build version of the device under test, the date and operator, the raw measurement data, and any deviations from the documented steps. Sylang doesn't embed that evidence in the .tst file, which would quickly become unmanageable. Instead, it lives in the test report for that execution, and the .tst points to the report through attach (a path or URL).

The convention: every move to pass or fail comes with an attach update pointing at the matching test report. The report itself sits under version control, typically in a separate test-evidence repository with its own retention policy. The commit message for the testresult update names the version of the device under test and the date, and the attach gives the persistent link to the full evidence. A pass without its evidence is just a claim.

What's next

Subchapter 10.4 walks through the cobot anchor's test set with this chapter's conventions in mind, and shows mirror examples from the other anchors. The chapter closes with how the test artifact set ties into the requirements (Chapter 9) and the coverage analysis (Chapter 15).

Exercises

10.4 The cobot test set: walkthrough and cross-anchor mirrors

In this subchapter we'll walk through the cobot anchor's test set using the chapter's conventions, then look at brief mirror examples from the EPB, autopilot and infusion pump anchors. You'll see how the method enumeration, the satisfies discipline and the testresult lifecycle come together into a complete verification artifact set for a SIL-3 safety-critical system.

The cobot's test mix

The cobot anchor's test set has about a dozen test cases. They cover the four mandatory safety functions (safe stop, safely-monitored speed, safe direction, collision detection) and the optional features (force-torque, vision, cleanroom). As an illustration of how a cobot verification campaign might split its effort, here is the method mix:

• HIL tests dominate, at about seven of the dozen: one for each of the four mandatory safety functions, plus three for the more elaborate scenarios (collision-during-motion, dual-function-trigger, safe-stop-from-each-mode).
• SIL tests cover the trajectory-planning algorithms and the per-joint controller stability analyses (about three of the dozen). They run in CI on every commit, so the regression suite catches algorithmic regressions early.
• Manual tests cover operator-interaction scenarios that can't practically be automated: operator-presence detection, the teach-mode user interface and the alarm acknowledgement workflow (about two of the dozen).

Each test case satisfies one or more cobot safety requirements via satisfies ref requirement REQ_COB_. The coverage analysis confirms that every requirement is satisfied by at least one test case at the appropriate method. The cobot's safety case then cites that matrix as evidence of verification completeness.

Cross-anchor mirror examples

The same patterns show up in the other anchors, with domain-appropriate content.

// EPB: HIL test for clamp-force undershoot
  def testcase TC_EPB_CLAMP_UNDERSHOOT
    method HIL
    level system
    safetylevel ASIL-D
    satisfies ref requirement REQ_EPB_001
    testresult notrun

// Autopilot: VIL test for alpha-protection in actual flight envelope
  def testcase TC_AP_ALPHA_LIMIT
    method VIL
    level system
    safetylevel ASIL-D
    satisfies ref requirement REQ_AP_001
    testresult intest

// Insulin pump: HIL test for max-bolus enforcement
  def testcase TC_INS_BOLUS_LIMIT
    method HIL
    level system
    safetylevel SIL-3
    satisfies ref requirement REQ_INS_001
    testresult pass

Wrapping up Chapter 10

That completes the verification artifact set. You've seen the file's structure and the method enumeration (10.1), the test-case body conventions and the satisfies discipline (10.2), the testresult lifecycle and evidence conventions (10.3), and a worked walkthrough (this subchapter).

Chapter 11 closes Part III by walking the EPB anchor end to end: from .ple through .fml, .fun, .req, .blk and .ifc to .tst. You'll see the engineering spine working as one integrated whole. Part IV (Chapters 12 to 15) then builds the safety-engineering artifact set on the foundation Part III has laid.

Exercises