On October 19 OpenAI said its latest models are much better at low-resource languages such as COBOL, Fortran, Visual Basic and OCaml, and open-sourced MultiLangBench, which asks a model to build a SQLite-compatible database engine from scratch in each of 20 languages. GPT-6 Astra averages an 87.4% test pass rate (GPT-5.6 Sol: 26.8%), above 83% in 19 of 20 languages, with COBOL hardest at 72.1%. On a separate COBOL migration eval requiring byte-for-byte output parity after porting to C++, Java, Kotlin, Python or Rust, GPT-6.1 Sol scores 86.6% at low effort (GPT-5.6 Sol: 5.4%) and 97.9% at higher effort. All figures are OpenAI's own.
Key Takeaways
- ✓MultiLangBench: GPT-6 Astra (max) averages 87.4% across 20 languages vs 26.8% for GPT-5.6 Sol, above 83% in 19 of them (OpenAI-reported).
- ✓Per language: COBOL 10.9% to 72.1%, OCaml 25.6% to 85.8%, Fortran 21.4% to 83.5%, Visual Basic 27.5% to 89.4%.
- ✓Budget tier: GPT-6 Luna averages 40.3% at about $0.77 per sample vs $3.48 for GPT-5.6 Sol, roughly 78% cheaper.
- ✓COBOL migration: GPT-6.1 Sol scores 86.6% at low effort (GPT-5.6 Sol 5.4%) at about 61% of the API cost; 97.9% at higher effort, near saturation.
- ✓The eval setup, test suite and per-language results are on GitHub openai/multilangbench; the COBOL migration eval is not public.
Key Decision Metrics at a Glance
Turn your technical choice into a development budget
Compare 40 dev plans & simulate token costs vs $20/mo subscriptions
Project Links & Resources
Direct AccessIn-Depth Technical Analysis
Background
Banks, governments and large enterprises still run critical systems on COBOL, Fortran and Visual Basic. OpenAI cites COBOL behind over 40% of online banking, 80% of in-person card transactions and 95% of ATM transactions, and Fortran in weather forecasting and reactor safety analysis. These languages have little public code or documentation, so models have struggled with them.
How it works
OpenAI did not disclose training details. It built two evals. MultiLangBench has a model build a SQLite-compatible engine from scratch in a given language; the program takes a database file and a SQL query and outputs a modified database, checked against reference SQLite using adapted SQLite tests, so all 20 languages share one yardstick. The COBOL migration eval ports programs to C++, Java, Kotlin, Python or Rust and requires byte-for-byte parity on hidden tests, catching details such as COBOL's default truncation in fixed-point math.
Benchmarks
All figures are OpenAI-reported. GPT-6 Astra (max) averages 87.4% on MultiLangBench vs 26.8% for GPT-5.6 Sol; COBOL rises from 10.9% to 72.1% and Fortran from 21.4% to 83.5%. GPT-6 Luna reaches 40.3% at about $0.77 per sample, roughly 78% cheaper than GPT-5.6 Sol. On COBOL migration GPT-6.1 Sol scores 86.6% at low effort (GPT-5.6 Sol 5.4%) and 97.9% at higher effort. No independent replication yet.
Getting started
The MultiLangBench harness, test suite and per-language results are open on GitHub, so teams can check scores for their own languages. For legacy migrations, draft with GPT-6.1 Sol at low effort and raise effort for critical modules; OpenAI notes production migrations still need expert review and regression testing against business rules.
Benchmark side-by-side against alternatives, or calculate monthly token cost vs subscription break-even.
Discussion & Comments
0Sign in to join the discussion
Connect with AI developers to exchange benchmark insights.