EsoLang-Bench: Evaluating Genuine Reasoning in LLMs via Esoteric Languages

As Large Language Models (LLMs) continue to demonstrate remarkable capabilities in coding, mathematics, and logical reasoning, a growing concern in the research community is data contamination. Because these models are trained on massive datasets encompassing nearly the entire public internet, it is difficult to determine if a model is "reasoning" through a problem or simply retrieving a memorized solution. EsoLang-Bench is a benchmarking framework designed to address this by using esoteric programming languages (esolangs) to test the limits of algorithmic reasoning in a low-resource data environment.

The Problem of Data Contamination

In standard benchmarks like HumanEval or MBPP, models are tested on Python or Java. Since these languages appear in millions of repositories, the models likely encounter the exact solutions during pre-training. This leads to an overestimation of the model's actual cognitive ability. Esoteric languages, by design, are rarely used for practical applications and have a tiny footprint in training corpuses. By forcing a model to operate within the constraints of a language like Brainfuck or Funge-98, researchers can better isolate its ability to apply logic to unfamiliar rule sets.

Core Task Categories

EsoLang-Bench typically evaluates models across three distinct dimensions of comprehension:

1. Code-to-Execution (Simulation)

In this task, the model is provided with a snippet of esoteric code and an input. It must predict the exact output. This requires the model to maintain a mental "state" (e.g., a memory pointer and a data tape) and iterate through loops accurately.

Example (Brainfuck):
++++++++++[>+++++++>++++++++++>+++>+<<<<-]>++.>+.+++++++..+++.>++.<<+++++++++++++++.>.+++.------.--------.>+.>.
Task: What is the output?
Result: Hello World!

2. Natural Language to Code (Generation)

The model is asked to implement a simple algorithm (like checking for a prime number or reversing a string) in a specific esoteric language. This tests the model's ability to map abstract logic to highly restricted syntax.

3. Code Translation

Translating code from a high-resource language (C++) to a low-resource esoteric language. This measures the model's capability for cross-paradigm architectural mapping.

Selected Esoteric Languages in the Benchmark

The benchmark utilizes languages that represent different computational paradigms to ensure a comprehensive evaluation:

Insights and Performance Trends

Initial findings from EsoLang-Bench reveal a significant "reasoning gap" between top-tier models and mid-range models:

• State Tracking Failure: Many models fail at Brainfuck simulation because they lose track of the pointer position after 10–20 iterations, suggesting limitations in the model's "working memory."
• Syntactic Hallucination: Models often attempt to use Python-like logic in stack-based languages, inserting operators that do not exist in the target esolang.
• Scaling Laws: Performance on EsoLang-Bench correlates more strongly with a model's performance on hard mathematical benchmarks (like MATH) than on standard coding benchmarks, reinforcing the idea that esolangs test pure logic.

"The ability of a model to solve a problem in a language it has seen only 1,000 times is a much truer measure of intelligence than its ability to solve a problem in a language it has seen 1,000,000,000 times."

Implementation Details

System: You are an expert programmer.
User: Given the following Brainfuck code, trace the execution step-by-step.
Code: ,[.[-],]
Input: 'A'
Output format: Provide the final output string and the state of the first 5 cells.

Conclusion

EsoLang-Bench serves as a "stress test" for AI. By stripping away the familiarity of common programming languages, it forces LLMs to demonstrate whether they truly understand the mechanics of computation or are simply the world's most sophisticated autocomplete engines. As models continue to evolve, esoteric benchmarks will remain vital for distinguishing between rote memorization and genuine algorithmic reasoning.

Generation[edit]