Validation & Eval

spice-rs validates correctness by running test circuits through both spice-rs and ngspice and comparing every output value. This is not unit testing -- it is full-pipeline comparison against a reference implementation.

The validation infrastructure consists of:

• 226 test circuits organized by complexity layer (L1 through L6+)
• The spice-eval harness that runs both engines and compares results
• Divergence reporting that identifies the exact device and parameter where the engines disagree
• Trace export for deep investigation of transient waveform divergences

The standard: if a circuit produces different output than ngspice (beyond tolerance), that is a bug in spice-rs. Not a "different but valid" result -- a bug. Fix the code, not the tolerance.

Why this tooling exists

This eval harness is arguably the most important part of the project. spice-rs is ported with substantial LLM assistance — Claude translates ngspice's C code to Rust. But an LLM that can read and translate C cannot, on its own, produce a bit-identical port. It consistently drifts: substituting textbook formulas for the actual implementation, inventing plausible-looking alternatives, missing sign conventions buried in macros, or silently reordering operations in ways that change floating-point results.

Without tooling that pinpoints exactly where the Rust output diverges from ngspice — which device, which parameter, which NR iteration, which matrix entry — every failure is an intractable mystery. With it, every failure becomes a mechanical fix: the harness says "equation 7, drain conductance, iteration 3, sign is wrong," and either the human or the LLM can trace back to the C source and correct it.

The pattern that works: the human designs the eval infrastructure and diagnoses root causes; the LLM does the high-volume translation. Neither could do this alone.

Test Circuits

The eval harness contains 226 test circuits registered in sim/spice-eval/eval/manifest.toml. They are organized in six complexity layers.

L1: Single passive components (5 circuits)

The simplest circuits: one device, one analysis. Single Resistor, 3-Resistor Divider, Capacitor DC, Inductor DC, VCVS Gain.

Tolerance: abs=1e-9, rel=1e-6. These must be bit-identical.

L2: Passive combinations (6 circuits)

Multi-component circuits exercising AC and transient analysis engines. RC Lowpass AC, RL Highpass AC, RLC Series Resonance, RC Step Tran, RL Step Tran, RLC Tran.

Tolerance: abs=1e-4 to 1e-3, rel=1e-3. Should be near-bit-identical.

L3: Single nonlinear devices (10 circuits)

One semiconductor device per circuit. Exercises device model load functions, voltage limiting, junction initialization, and temperature processing. Diode Forward/Reverse DC, NMOS/PMOS Level 1, BJT NPN/PNP, JFET N-Channel, Diode Tran, NMOS Tran.

Tolerance: abs=0.01, rel=0.01. Well-conditioned devices achieve machine precision.

L4: NR algorithm stress tests (6 circuits)

Circuits that stress Newton-Raphson convergence: high-gain configurations, body effect sweeps, stiff systems. Body Effect Sweep, High-Gain Amplifier, DC Sweep NMOS, Resistor Sweep, Source Stepping, Gmin Stepping.

Tolerance: abs=0.01, rel=0.01.

L5: Device interactions (5 circuits)

Multiple nonlinear devices interacting. CMOS Inverter, Differential Pair, Diode Bridge, BJT Amplifier, Cascode.

Tolerance: abs=0.01, rel=0.01.

L6+: Full circuits (194 circuits)

The bulk of the test suite: BSIM3 circuits, BSIM4 circuits, MOSFET Level 2/3, complex topologies, specialized devices, various analysis types, edge cases.

Tolerance: abs=0.01, rel=0.01.

Manifest format

[[circuit]]
name = "[L3] NMOS Level 1 DC"
file = "dc/L3_nmos_level1.cir"
[circuit.tolerances]
abs = 0.01
rel = 0.01

Design principles

No golden files. Test circuits are compared against a live ngspice run, not stored reference values. Updating ngspice automatically updates the reference.

Tolerance is a floor, not a target. The standard tolerance (0.01) is the minimum -- 176 out of 226 circuits produce bit-identical results.

Every device model needs a circuit. When porting a new device model, create at least one L3 circuit for it before tackling more complex configurations.

Divergence Reports

When a test circuit fails, the eval harness provides several levels of diagnostic output.

Basic failure output

| [L3] NMOS Level 1 DC           |  FAIL  |  2.341e-03 |  1.872e-02 |
|   ! v(drain)    sr=4.98123 ng=4.97889 abs=2.341e-03 rel=4.704e-04
|     v(gate)     sr=3.00000 ng=3.00000 abs=0.000e+00 rel=0.000e+00

Each row shows the node name, spice-rs value, ngspice value, absolute error, and relative error.

Diverge mode

For transient circuits, identifies the first timepoint where divergence exceeds tolerance and shows per-device state:

Divergence analysis: [L5] CMOS Inverter Tran
First divergence at t=1.234e-06 (step 47 of 200)
  v(out)   sr=2.4531   ng=2.4489   abs=4.2e-03

Per-device state at divergence point:
  M1 (NMOS): Ids=1.23e-03  Vgs=3.000  Vth=0.700  gm=2.46e-03
  M2 (PMOS): Ids=-1.23e-03 Vgs=-2.000 Vth=-0.700 gm=2.46e-03

Diverge-deep mode

Per-NR-iteration comparison:

NR iteration comparison at t=1.234e-06, iter=3:
  RHS before solve (device stamps):
    eq[1] sr= 1.23456789e-03  ng= 1.23456789e-03  diff=0.000e+00
    eq[2] sr=-4.56789012e-04  ng=-4.56789013e-04  diff=1.000e-12

  Solution after solve:
    eq[1] sr= 2.99999999      ng= 3.00000000       diff=1.000e-08
    eq[2] sr= 1.23456780      ng= 1.23456789       diff=9.000e-08

Interpreting diverge-deep output:

1. RHS matches but solution diverges -- solver bug.
2. RHS diverges at a specific equation -- the device stamping that equation is computing differently.
3. Device conductance diverges -- different linearization in the device model.
4. Device stored current diverges -- different I-V evaluation, often from voltage limiting or mode flag differences.

Parameter check mode

Compares parsed model parameters between engines:

Parameter check: [L3] BSIM3 NMOS DC
    VTH0    sr= 0.4376260  ng= 0.4376260  MATCH
    TOXE    sr= 0.0000000  ng= 1.000e-08  MISMATCH

A parameter mismatch means the parser is not passing the value through correctly -- the highest-impact failure mode.

Debugging workflow

1. Run --check-params to rule out parser bugs.
2. Run --diverge to find where the divergence starts and which device.
3. Run --diverge-deep to find the exact NR iteration and whether it's a stamp or solver issue.
4. Read the ngspice C code for the identified device/function.
5. Fix and re-run.

Trace Export

For deep investigation of transient waveform divergences, the eval harness can export per-timestep data as JSON.

cargo run --release --bin spice-eval -- --trace="Circuit Name" --output=trace.json

Output format

The trace JSON contains per-timestep node voltages from both engines, device state at divergent timesteps (including charges), and NR iteration snapshots when profiling is enabled.

Analysis workflow

Waveform comparison: Load the JSON in Python and plot corresponding signals:

import json
import matplotlib.pyplot as plt

with open('trace.json') as f:
    data = json.load(f)

sr = data['spice_rs']
ng = data['ngspice']

plt.plot(sr['times'], sr['nodes']['v(out)'], label='spice-rs')
plt.plot(ng['times'], ng['nodes']['v(out)'], label='ngspice')
plt.legend()
plt.show()

Charge tracking: For transient divergences that grow over time, compare charge states at divergent timesteps. Common causes: integration coefficient differences, charge model evaluation differences, truncation error estimation differences.

Diagnostic mode summary

Current Status

Overall

Standard tolerance: abs=0.01, rel=0.01 (never loosened to pass a test).

Bit-identical results

176 of 226 circuits produce results that are bit-for-bit identical to ngspice (zero absolute error, zero relative error). This means the Rust code produces the exact same sequence of IEEE 754 floating-point operations as the C code.

Passing but not bit-identical

23 circuits pass (within tolerance) but have nonzero error, typically at the 1e-12 to 1e-14 level. Sources: floating-point non-associativity, library function differences, convergence path sensitivity.

Failed circuits (4)

The failed circuits include BSIM3 transient precision issues (junction charge evaluation) and BSIM4 models. BSIM4 is the most complex device model (~8000 lines of C in the load function). The port is underway.

Error circuits (3)

The 3 error circuits are convergence failures where spice-rs fails to find the DC operating point due to subtle differences in the stepping schedule, or require BSIM4 charge models not yet ported.

Device model coverage

Analysis type coverage

Next targets

1. BSIM4 completion -- port the remaining load function sections
2. Convergence parity -- investigate the 3 error circuits
3. Additional test circuits -- expand coverage for edge cases