(Unit) Testing

[ducky64]

Project: automated (unit) testing for JACDAC, loosely defined.

Goal: unit testing helps software engineers write code that is more correct, faster. Bring the benefits of unit testing to embedded firmware development, and encourage its use by making writing tests easy, fast, and painless.

Thoughts and suggestions are welcome!

Usage Scenarios

Why JACDAC? JACDAC provides services as a standardized interface, which are a natural point for mocking out the environment. The structured registers and event interfaces can also be more conducive to mocking than arbitrary C/C++ code.

Layers of Testing

• End-user regression testing: for testing end-user application firmware.
  • (out of scope for now) Ideally would support all the interfaces for programming with JACDAC - PXT, TypeScript, MakeCode and its blocks, and probably much more.
• Adapter regression testing (currently the priority use case): with a 'nanoservices' approach, adapters (see examples of meta-services #495 for more details) may encapsulate more of the functional logic (as opposed having it run on device firmware) and should be well-tested. Since adapters are pure software objects, the inputs can be mocked and the outputs can be asserted / expected on.
• Module testing (out of scope for now): not sure what this looks like, or if it can be fully virtual.
• JACDAC core testing (out of scope for now): not sure what this looks like, or if it would require a lot of per-HAL custom code.

Testing Fidelity

• Fully virtual tests: fully virtual tests that can be run as regressions on a developer's machine (or as part of automated CI), without hardware in the loop. These should be very fast and run often (eg, before opening a PR, or before each commit), but low-fidelity.
  • The entire bus configuration (mainly connected services) would be specified as part of the test scenario. All services would be mocked out - either to mock sensor inputs or expect actuator outputs.
  • Adapters could be added to be bus (ideally similarly to how they would be used in application code) and would run as normal. Adapters can be the DUT, or just an adapter between lower-level mock input and the application logic.
    • May need special case for consuming services in the test framework, since (unlike application code) adapters can also provide services.
  • Separate notion of wall clock time and simulator time. Long waits without events should be skipped. Services and adapters may needed to be coded to allow fast-forwarding of time.
• Hybrid tests - client only: running with the embedded controller and physical bus in the loop, but modules still mocked out. Services can be mocked out either with a protocol generator (difficult, may need custom stack and hardware) or using a virtual bus (probably simpler, but lower fidelity).
  • Scenario is still 'formalized' and reproducible, but need to deal with real-world issues like communications and compute not being instantaneous. Wall clock time is simulator time. Need to account for timing jitter.
• Hybrid tests - client and modules: unclear what this looks like - ask the user to press a button? Or command an actuator (possibly on a separate bus) to press a button (high setup cost- requires dedicated test jig)? Scenarios could still be formalized, but would have much higher timing jitter. Highest fidelity, end-to-end test.

Testing Stimulus

• Hand-crafted stimulus: much like how testing is done in simple Verilog (without the constrained random stuff in SV). Specify mock inputs, expect mock outputs, and add (simulator) time delays. Each operation is just a function call, with values written manually by the user.
• Captured stimulus / replay: hand-crafted input makes sense for simple things, but not for high-sample-rate streaming data. For example, for accelerometer shake detection, the algorithm should be tested against captured real-world data.
  • Should this be on the level of packet captures - eg, accelerometer packets saved to a file, replayed by invoking a function call?
  • Or should this be on the level of semantics - eg, accelerometer values and timestamps saved to a file, with protocol level information regenerated on a test run?
• Property-based testing (this is probably not a good fit for JACDAC): behavior is specified as properties, expressed with LTL (linear temporal logic) like semantics. Common in digital logic design, combined with constrained random for stimulus generation. But very different than 'typical' unit testing which focuses on concrete examples rather than more-abstract properties. Likely for a much, much more expert audience.

Mockups

Button Gesture Detection Adapter, hand-crafted stimulus

// Test environment construction
// TODO should this just be a declarative container object or actually instantiate the bus?
const testBus = createVirtualBus({  // defined as role name -> device
    ‘btn1’: ButtonServer(),  // is a mock
    ‘adapt’: ButtonGestureAdapter(‘btn1’),  // is a functional adapter
    // TODO: should mocks and adapters live in different sections?
})

// Test body
// This example supports exact timing
// The button gesture adapter is the DUT and runs concurrently with the test logic
// TODO probably needs awaits sprinkled
runTestScenario(testBus, (bus) => {
    const btn1 = bus.getServerByRole(ButtonServer, ‘btn1’)  // returns the server / mock controller object
    const adapt = bus.getClientByRole(ButtonGestureAdapter, ‘adapt’)  // returns an observer object

    btn1.down()  // commands mock
    bus.delay(100)
    btn1.up()
    bus.expectEventNow(adapt, ButtonGestureEvent.CLICK)  // immediate event propagation semantics
    // Alternatively, maybe on the object? adapt.expectCliickNow(ButtonGestureEvent.CLICK)
        // Advantage: strongly typed from codegen, can be a sugar layer above bus.expectEventNow
    // Alternatively, explicit notion of timing jitter, bus.expectEventWithin(adapt, ButtonGestureEvent.CLICK, 10)
    // would register an event handler to run in the background, and would also error if unfired before the test ends

    bus.delay(500)  // wait for click detect to reset
    // In a real test, this should be its own separate test case
    btn1.down()
    bus.delay(200)
    bus.expectEventNow(adapt, ButtonGestureEvent.HOLD)  // timing is exact
    bus.delay(100)  // for kicks and giggles
    btn1.up()
    bus.expectEventNow(adapt, ButtonGestureEvent.HOLD_RELEASE)  // timing is exact
})

Accelerometer Shake Detection Adapter, replay stimulus

// Test environment construction
const testBus = createVirtualBus({  // defined as role name -> device
    ‘acc1’: AccelerometerServer(),  // is a mock
    ‘adapt’: ShakeDetector(‘acc1’),  // is a functional adapter
})

// Test body
runTestScenario(testBus, (bus) => {
    const acc1 = bus.getServerByRole(AccelerometerServer, ‘acc1’)  // returns the server / mock controller object
    const adapt = bus.getClientByRole(ShakeDetector, ‘adapt’)  // returns an observer object

    // TODO semantics for what acc1 should return by default?
    // Should it have a default value (unintuitive)?
    // Or error if attempts are made to read it before data is fed into it?
    acc1.replayStart(test_file.csv, SAMPLE_RATE)  // could be captured from real-life data, runs concurrently
    // should parallelism be explicit? (eg, runInParallel { acc1.replay })

    bus.expectEventWithin(adapt, ShakeDetector.SHAKE, 500)
    // alternatively, register-based, polling
    bus.delay(500)  // flakey – must be shaking at this instant
    bus.expectRegisterEquals(adapt, ShakeDetector.SHAKING, 1)
    // alternatively, register-based, change detect
    bus.expectRegisterChangeWithin(adapt, ShakeDetector.SHAKING, 1, 500)  // too many arguments =(
})

Thermostat Application
TBD

Implementation

Test framework: use mocha, since that's what jacdac is currently using.

Primitives

• Virtual bus definition - see mockups above
• Send packet on the bus (mocking out sensors)
  • Simplest: bus.sendPacket(...)
  • Can codegen service-specific abstractions on top of that, eg:
    • Service-centric object model: myButton.down(), myButton.setState(0)
      • Note that this button API is not orthogonal, so it may be possible for events and state to be inconsistent. Averted with a nanoservices / adapters approach.
    • Or bus-centric model: bus.sendEvent(ButtonService, myButtonId, Button.EventClick) - but may be confusing / result in lots of boilerplate
• Expect packet on the bus
  • bus.expectPacketNow(...) - expects a packet at this instant. Requires precise timing, so for fully-virtual tests only.
    • Semantics: events must be generated instantaneously (is this the case?), eg before button.down() returns, so event ordering (from down() to expectPacket) is encoded in the test.
    • Alternatively, could be more like the below, where this registers a packet checker, but with a duration of zero (so checks right before before time advances). Debugging may be more difficult.
      TODO: think about implications with async control flows.
    • Idea: in non-virtual tests, can there be some default timing jitter allowed here based on real-world data?
  • With some timing tolerance: bus.expectPacketWithin(..., 200). Checks that the packet is received within the specified time. Used where the user is unsure about timing, eg for shake detection on a captured trace.
    • Errors if the test ends before the handler fires.
    • Potentially also useful for timing jitter, but that should be handled at an infrastructural level so users don't need to understand the details of timing.

Overarching Task Tracker

• Mockup some examples
  • Button gesture detector mockup
• Look at TS test frameworks - see what can translate well to embedded
• Get more use cases
• Discount literature review techniques
• Design bus specification / description methods
• Design test primitives
• Implement it all on jacdac-ts, on top of JDOM.
• Write tests for adapters, iterating with test framework / primitives
• Syntactic sugaring, eg codegen convenient methods from service spec
• Think about application-level tests
• Think about supporting hybrid testing

[pelikhan]

IMO, the ability to do virtual or hybrid tests against end-user programs is the real gold here. To that extent, every end-user program could expose a custom service that would make it observable/actionable through jacdac -> unit test framework. This also helps debugability/diagnostics/tracing and what not. Adapters happen to be specialized case where the resulting service is already specified.

bus.expectPacketNow(...)

Probably want to package test helpers into a separate module to avoid overloading the bus type.

Mocks

We have an infrastructure to create servers in JS, which talks "packets". It could be extended to be driven by some higher-level mock framework so that the user can define server with specific behaviors for unit tests.

Can codegen service-specific abstractions on top of tha

Don't do codegen until you have a very clear picture of what the test API will look like.

Test framework: use mocha, since that's what jacdac is currently using.

Should be test framework agnostic. If we can spinup a virtual bus and run some asserts in a async function, it will fit into all test frameworks.

Hybrid tests

Hybrid testing is definitely interresting. It gets better: some modules may be use mock behavior, other recorded data and other use real data.

[ducky64]

Writing down some design notes on testing, to organize my thinking and iterate on mockups:

In software

• Unit testing: on functions / methods / other small pieces of software.
  • Example form: expect(someFn() == expected);
  • Typical properties: ideal computational model, no notion of time (everything should be about instant). Functions under test should run quickly (so regressions can run often).
• End-to-end UI testing: typically sequences of scenarios, with expects on the output. For example, with some kind of webdriver framework.
• Async note for Javascript: when putting assertions in async functions in Javascript, must make sure the async function is run before the test exits. Typically achieved using done() in the async function (so the test times out if the async is not called) or using Promises and awaits (which makes sequences prettier).

In digital logic (RTL, chips) hardware

• Fun fact: Verilog doesn't seem to have a builtin assertions library, you either have to watch / parse the console, use SystemVerilog, or pull in external libraries. It's a bit of a mess.
• Testvectors: poke some inputs into the DUT using assignment operations, optionally spaced with delay operations, and check the output from the DUT. Testvectors are typically manually written, eg fully-user-specified-scenarios.
  • Typical properties: there is a notion of exact (cycle-accurate) timing, and timing control is fully under the test scenario's control. Tests check DUT outputs, which are typically latching (held until changed by the circuit or test scenario).
• Constrained random, property-driven / golden model: instead of the user specifying a scenario, the user specifies correctness properties of the DUT (potentially expressing properties across time in some type of linear temporal logic - for example, "after receiving a 1 on the input, it should produce a 2 on the output in one cycle"), and the test framework automatically generates inputs to the DUT.
  • I don't have much experience with this, but my impression is that this is the dominant way to test complex circuits / modules.
  • There's probably a lot of complexity in the test framework to generate inputs, especially handling all the ways users can constrain the random generation (including dependencies between variables).
  • Although popular in industry, there doesn't seem to be much uptake among smaller scale digital designers (eg, hobbyists) and very few open docs. Probably depends on very expensive software. And high upfront time and skill investment. So probably not a great for for Jacdac.

In embedded

This is based on like 2 hours of websearch with no practical experience, so isn't necessarily representative

• As a user, there does not seem to be any significant adoption of (or even frameworks for) unit testing among the hobby-level (eg, Arduino, mbed) embedded community. Would love to change that though...
• Unit testing: typically using C/C++ unit test frameworks to test individual components / subsystems, using a function abstraction boundary, and mocking out lower level subsystems as needed. Can run entirely in software, on a different architecture than the embedded target.
  • Requires firmware code to be structured in a way that is conducive to unit testing - so neat(ish) function and class boundaries.
  • Otherwise, it walks and quacks 🦆 like unit testing... because it is unit testing.
• Hardware-in-the-loop: varying degrees, typically with bespoke frameworks
  • Simplest end: with the embedded controller in the loop, but hardware mocked out (eg, not referenced in firmware, or with signal generators).
  • Highest end: with the full hardware and an (electromechanical) test jig. Obviously highly expensive and complex.

For JACDAC

• JACDAC's abstractions are different than unit testing abstractions:
  • notion of time is critical
  • both register and event abstraction (among more - but these are the most common): while registers are analogous to RTL DUT IOs, there isn't a good analogy for events
• If the bus is "live" (or even in simulation, but tied to the wall clock), registers can change in the background and events may fire with some timing jitter.
  • Event handers / listeners must be pre-registered - either by the end-user, or in infrastructure (presenting the notion of an event stream).
• With a nanoservices approach. the "units" being tested may be individual services, eg a button gesture (click / click-and-hold / double-click) detector adapter.

Event handling

Analogous patterns (probably) exist for register and function testing, but less so for events especially where timing is an important part of the spec and test. Even in Javascript only a small subset of unit test guides talk about events, and the patterns are either writing a custom event handler to set a variable that is later checked by a test, or putting done() in the event.

So one design decision:

• Automatic pre-registration of events on all services - event stream abstraction presented to the user, can look for events backwards in time. "An event must have occurred [optionally, within the last x time]"
• Manual registration of event handlers, can look only forward in time. "An event must occur [optionally, within x time]"

[pelikhan]

Not sure time is that critical. You could think of writing tests that wait on events, regardless of time.

[ducky64]

So I think there are cases where timing is an important part of the spec - for example, in the button click detector, I may want to check that the button registers a hold 250ms or so after the button is down.

[ducky64]

Continued hybrid testing ideas:

• Use case is an end-to-end, compatibility test for a module.
• Some human interface component is required, since the device needs physical inputs, and outputs need to be observed
  • Would likely be a combination of instructions (displayed on the screen) and perhaps simple questions ("is the servo approximately 45 degree")
• Additional test primitives needed:
  • Interface with actual bus devices
  • Display instructions to user (MVP: printf to console, cleaner version: web page, or through a display module)
  • Some kind of user input (MVP: "did it do the right thing (y/n)?" on the console, cleaner version: buttons (with hotkeys) on a webpage, or through an input device module)
  • Wait on device announce
    • Fancier version: automatically launch the appropriate test suite when that device is announced (or for web version: filter available test suites by connected devices)
  • Possibly interesting: also have VM programs and role binding running virtually, to do end-to-end tests with higher-level services.
• Perhaps an idea of a test suite and test runner, analogous to Mocha, but tailored to human-in-the-loop tests on physical devices. Ideally re-use large chunks of the JDOM test infrastructure.
  • Needs some kind of test termination condition in the case of a failure, so it doesn't deadlock. Perhaps disconnecting the module?
  • Needs some way to skip individual test cases, in case it's worth testing other functionality. Perhaps through keyboard input - would mark the test as failed / canceled.

[ducky64]

I think for actuators the only real solution is to ask the human... unless you really want to set up a test jig with sensors and everything (and even then - it seems that the main benefit is more accuracy and more bits of resolution than a human could provide, or an automated regression rack, because to scale to number of units, I expect getting the DUT set up into the jig will be the limiting factor in terms of time).

I could certainly imagine test abstractions where you write the tests as if you had a test jig, and the test infrastructure could run those in manual mode where those procedures are instead delegated to the human - for example, asking a brightness sensor (by role name?) for brightness could instead become asking the human if it's lit up (or against some reference brightness). But even for brightness - do we specify LED brightness as part of the spec anywhere? Would both a 5W LED mounted to a copper brick and an 0402 LED both correctly implement the LED service?

[tballmsft]

Not sure about "negative events". Another way to think of this as "allowed events". That is, within an interval, any event in the allowed set is fine. Anything outside that set fails the test. Since we have a finite set of events, I think law of the excluded middle applies, no?

[tballmsft]

Definitely a good idea to have abstractions that deal with approximate values, stability, etc.

[ducky64]

I've also thought a bit more about getting tests accurate and 'watertight' - including negative conditions (eg, no event fires before this expected event, or register stays held until and after some edge) and fast transients (eg, an event fires right between when you finish watching for one event and before waiting on the next event). When working synchronously with realtime, it'll probably be difficult to avoid those effects, so ideally we'd work with traces. But at least if we have real hardware in the loop, and especially if we have humans in the loop, the testing experience has to be done live.

So idea: hybrid live traces (totally made that up term on the spot): instead of watching for events directly, events are first buffered into a trace, then the test operates on the trace. Events in the buffer would be timestamped - either on the embedded processor (for accuracy, if available) or on ingestion into the JDOM (less accurate, potentially mitigated with real threading instead of cooperative multitasking). Since DUT time is decoupled from test compute time, test constructs can appear to have exactly zero duration, while still appearing live enough to the user (unless their computer is actually a potato).

What events to monitor for / record can be added dynamically when there is a consumer for the event - so we don't unnecessarily record the entire firehose. This means you don't get a trace for an event before you ask for it, but that's probably fine since it's kind of an initial state.

Otherwise, it seems that point events (which can be waited on) and duration conditions (which need to hold for some amount of time) will work with this, with the hybrid live traces shrinking gaps in coverage (from time it takes to execute through the testdriver - admittedly likely small anyways) to exactly zero. For example, we might define a button test as:

- wait for [down event, falling edge on register] within 50ms of each other
- wait for [up event, rising edge on register] within 50ms of each other, within 500ms of last event

The events could be defined as the next event (so no other event may fire until the target event is observed), and perhaps also with a .hold (either explicitly or by default) on the event that tacks on a condition between when the event fires and when the wait resolves (if, as in this example, it waits on both the down event and falling register). In this case, we would have full coverage (in time) throughout the entire wait.

I believe we can get a single serialized (ordered in time) stream of all events across different devices, because of the centralized bus object, right? Otherwise, we might have to be careful about how test time advances across different event streams. In any case, when the buffer empties, the test has caught up with realtime.

The only other potential issues I can think of are complexity (which I don't think it's that difficult, it's a buffering layer) as well as cases where this degree of accuracy is too accurate (for example, switch bounce - is that allowed or disallowed by the spec? what's the intended behavior? - but that's a bug masking another bug)

[ducky64]

I think having an allow-list of events makes sense, but the trick is scoping it properly. For example, if you're listening for a register, it probably doesn't make sense to fail the suite if an event was raised (or vice versa). I think I'd have different scopes for events from a service and register updates from a service (both per service).