IBM's Benchpress compiles Qiskit without seed_transpiler. I measured what that costs the accept/reject decision.

Benchpress is used to decide whether a Qiskit change is a regression. Its Qiskit gym calls transpile() without seed_transpiler, so every gate count it reports is one draw from a distribution nobody measured. The BQSKit gym in the same repo does seed its compiler. I ran 39 QASMBench circuits (28 to 420 qubits) onto a heavy-hex lattice under Qiskit 1.4.3 and 2.0.0, 200 seeds per arm across ten OS processes. On bv_n140 the long-run change between the two versions is +5.37% [+4.27, +6.50]. A three-run comparison, the protocol used in the issue that started this, returns a ">=10% regression" verdict 24.4% of the time [19.5, 31.1]. A disjoint 200-seed sample reproduces that at 22.6%. Twenty runs per version, roughly 40 hours of compute, still leaves 3.7%. The median circuit has a 10.9 point ambiguity band: a window of true change three runs cannot resolve in either direction. Two of the three circuits named in Qiskit issue #14402 are affected. I reproduce that issue's two low-variance figures to within 0.5 and 2.4 points. Its +46.1% on bv_n140 is one draw from a distribution its own protocol spreads from -10.5% to +100.0%. Passing seed_transpiler removes the false positives here, but it was worse on three of four circuits with false negatives, so I'm not presenting it as the fix. Limits: one SDK, one version pair, and it all ran on my machine. The analysis code was committed before the data existed. https://ai.bedvibe.studio/decision-risk/ submitted by /u/CupGlass540 [link] [comments]