The evaluator samples multiple outputs for each problem and executes or checks them against a verifier. Sampling temperature, candidate count, and estimator details must be held consistent when comparing systems.
Pass at K estimates the chance that at least one of k generated candidates passes a correctness test.
The evaluator samples multiple outputs for each problem and executes or checks them against a verifier. Sampling temperature, candidate count, and estimator details must be held consistent when comparing systems.