STM32 Hardware reliability test station

The DUT sends events up through an adapter to the core daemon, which classifies them, logs them and sends recovery commands back down. pytest injects faults and compares its timestamps with the log; reporting reads the log.

• Adapters: one per DUT type behind the same interface. Each registers its file descriptors with the daemon's epoll loop; a new device type is a new subclass plus a YAML device file.
• Protocol: reliable framing over UART with delimiter, length, sequence number, type, payload and CRC. The CRC catches corruption; the sequence number catches drops, duplicates and reordering.
• Core daemon: a single-threaded epoll loop with timerfd heartbeat deadlines and a per-device state machine (BOOTING, RUNNING, SUSPECT, HUNG, RECOVERING). A classifier maps signals to fault types, and a hang triggers a power-cycle.
• Event log: sequence-numbered, CRC-checked records, fsynced on faults and replayed after a restart.
• Supervision: a systemd Type=notify unit; the daemon pings the watchdog only when its loop makes progress.
• Crash symbolization: the ELF recorded at flash time is passed to addr2line with the stacked PC and LR.
• Test layer: pytest stamps each injection with a monotonic timestamp and compares it with the matching log event.
• Reporting: latency percentiles, misses, false positives, confusion matrix, MTBF, timeline and symbolized stacks, as Markdown and JSON.

• Reset cause: reads the RCC reset flags at boot (watchdog, brown-out, software, power-on, pin), reports them, then clears them.
• HardFault capture: the handler records CFSR, HFSR, MMFAR, BFAR and the stacked registers (R0-R3, R12, LR, PC, xPSR) to a no-init RAM section or backup registers, then resets. The dump is reported after reboot.
• Hang detection: the independent watchdog (IWDG) resets a stuck device; the cause is reported on the next boot.
• RTOS faults: FreeRTOS stack-overflow and malloc-failed hooks log the fault and trigger a controlled reset.
• Fault triggers: commands that cause each fault on demand (reference firmware only).
• Heartbeat: a periodic frame over the CRC-framed UART.
• Host side: the crash dump is symbolized against the firmware ELF with addr2line.

1. Connect: wire the UART, SWD port and power switch.
2. Device config: one file with the adapter type, serial port and baud, reset method, flash command, heartbeat timeout and expected boot message.
3. Scenario file: duration, faults to inject (optional) and pass criteria, for example no unexpected resets and boot within a set time.
4. Flash: the station flashes the build under test through OpenOCD and records the git hash and ELF.
5. Run: the daemon monitors the device while it does its normal work or a stress script runs.
6. Detect and recover: events are timestamped and classified; on a hang the station power-cycles the device and logs the recovery.
7. Collect crashes: any crash dump is symbolized to file and line.
8. Report: uptime, resets by cause, hangs, mean time between failures, timeline, stack traces, pass or fail.
9. Compare: re-run after a fix and compare mean time between failures across firmware versions.

Example: station run --dut tracker.yaml --scenario soak-12h.yaml --firmware build/app.elf

Hardware

• STM32 Nucleo board: reference device under test
• ST-Link (on the Nucleo): SWD flashing and debug, virtual COM port for UART
• Linux host (PC or Raspberry Pi): runs the station
• Raspberry Pi or other Linux board: second DUT type
• USB-to-UART adapter: serial link when the ST-Link COM port is not used
• Relay, MOSFET or USB-controllable switch: hard reset and power-cycle of the DUT
• INA226 current monitor (optional): supply current on the DUT
• Oscilloscope or logic analyzer: verifies timing and ground-truth timestamps

Software

• Station core: C++17, CMake, epoll, timerfd; event loop, heartbeat deadlines, state machine, fault classification
• Logging: append-only binary log (or SQLite); durable, sequence-numbered event record
• Supervision: systemd (Type=notify, libsystemd, sd_notify), journald; watchdog, restart policy, resource limits
• Device I/O: termios serial, udev rules, SSH, ICMP ping
• STM32 firmware: C, FreeRTOS, IWDG, reset-cause flags, HardFault handler; crash agent and heartbeat
• Debug-port automation: OpenOCD, arm-none-eabi-gdb, arm-none-eabi-addr2line; flash, read crash state, symbolize to file and line
• Protocol: CRC-framed UART with sequence numbers, a host-testable C module shared by firmware and host
• Tests: Python, pytest, pyserial, simulated DUT, sanitizers, valgrind
• Observability: Wireshark with a Lua dissector (serial capture to pcap); optional OpenTelemetry or StatsD counters
• Packaging: Yocto recipe or .deb, YAML config, git

Requirements

• Hang detection latency: [X] s, p99
• False positives per 24 h soak: [Y]
• Silent frame loss: 0
• Data loss across daemon restart: 0 events, replayed from log
• Crash dump to file and line: resolved for every HardFault injected

Fault table

• Corrupt packet (pytest): CRC mismatch; count and log, request resend
• Serial drop (pytest): sequence gap; log the gap, flag lost frames
• Daemon crash (pytest): systemd watchdog; restart and replay log
• STM32 IWDG reset (firmware command): reset cause plus heartbeat loss; log reset by cause
• STM32 HardFault (firmware command): crash dump; symbolize and log file and line
• STM32 hang (firmware command): heartbeat loss; power-cycle and log recovery
• Linux board kernel panic or hang (pytest or SSH): console output, ping loss; power-cycle and log console tail
• Serial device unplugged and returns (manual or script): udev event; pause, reattach, resume

Measured per run: detection latency percentiles (p50, p99) per fault against injected ground truth; missed detections and false positives over a soak of [N] hours; a confusion matrix across fault types; recovery time and data lost per fault.

Station failure analysis

• Daemon crash: monitoring stops. systemd sees the process exit; Restart=on-failure and log replay on start.
• Daemon hang (loop stuck): faults go unseen. The sd_notify watchdog ping goes missing; systemd kills and restarts it.
• Host hang: the whole station is silent. The host hardware watchdog times out and reboots; the run is marked interrupted and the log replayed.
• Log disk full: events would be lost. A write error and free-space check stop the run loudly; rotate and reserve space.
• Serial adapter loss: the device looks hung. The udev remove event classifies it as link loss, not a DUT hang; resume on return.
• DUT stuck in a reset loop: endless power-cycles. Boots in a short window exceed a limit; stop cycling, mark the run failed, keep the last crash dump.