| Java | Go | |||||
|---|---|---|---|---|---|---|
| Raw Output | Local Search | Improvement | Raw Output | Local Search | Improvement | |
| GPT4 (Local Search) | 0.744 | 0.797 | 5.30% | 0.624 | 0.712 | 8.80% |
| GPT4 (Basic Edit Distance) | 0.744 | 0.758 | 1.40% | 0.624 | 0.690 | 6.60% |
| p-value | 2.00E-06 | 4.00E-07 | ||||
| Vicuna-13B (Local Search) | 0.699 | 0.710 | 1.10% | 0.775 | 0.804 | 2.90% |
| Vicuna-13B (Basic Edit Distance) | 0.699 | 0.706 | 0.70% | 0.775 | 0.786 | 1.10% |
| p-value | 8.00E-02 | 2.00E-04 |