Adaptive In-Context Learning in Large Language Models: A Zero-Leakage Benchmark Using Fictionalized Extensions of the Khom Thai Script

Abstract

Despite rapid progress in large language models, existing benchmarks often fail to distinguish genuine adaptive reasoning from training data memorization. To address this, we introduce KHOM THAI, a zero-leakage benchmark designed to rigorously test a model’s ability to acquire, retain, and flexibly apply novel symbolic rules entirely in-context. Built upon a structured, fictionalized extension of the historical Khom Thai script, the benchmark challenges models with a complex romanized notation system governed by interlocking rule sets. These rules encompass context-sensitive subscript clusters, multi-position vowel attachments, consonant class distinctions, and intricate interaction effects such as rwrap-induced order inversion and class-sensitive consonant doubling ensuring models must master an entirely novel system without relying on pre-trained linguistic priors. The benchmark isolates distinct facets of adaptive learning through three complementary tasks: Task A evaluates rule generalization across varying complexities, Task B tests long-context retention using semantically unrelated distractors, and Task C examines belief updating through targeted feedback. The evaluation of 14 frontier and strong open models reveals substantial performance disparities; while leading models like Gemini 2.5 Pro achieve near-ceiling accuracy (96.4%), many others exhibit sharp performance drops when handling rare cluster interactions and complex vowel modifications. Ultimately, KHOM THAI provides a clean, highly discriminative testbed for evaluating robust, human-like in-context rule learning, offering a valuable signal for measuring genuine abstraction and adaptive intelligence in AI systems.