SWT

What is the task in SWT-Bench?

Generate software tests which reproduce user-reported issues and increase code coverage. Each SWT-Bench task is based on a real-world pull-request from a GitHub repository which resolves a user-reported issue and contains the code patch fixing the issue and unit tests testing the fix. Given the original state of the code base and the user issue, the task is to generate tests that reproduce this issue.

What metrics do you measure?

We measure whether the generated tests reproduce the user issue and increase the code coverage of the modified code. For a generated test to reproduce the described issue, it should fail on original code base but pass after the code patch fixing the issue is applied. We call this a Fail-to-Pass (F2P) test.

What is the value of test generation?

Reproducing test can aide test-driven development, avoid regression, and are a powerful tool to cross-validate proposed bug fixes. To resolve a reported issues reliably, it is essential to first reproduce it and then confirm that a proposed bug fix actually addresses it. This is typically done by creating a reproducing test.

Are there other relevant benchmarks about code generation and testing?

SWE-bench measures the capability of fixing reported user issues. BaxBench measures capabilities at writing correct and secure web backends. SWE-bench: SWT-Bench and SWE-bench are based on the same GitHub repositories and issues. While SWT-Bench measures the ability of models or agents to reproduce a given issue, SWE-bench measures their ability to resolve it.