Beyond Benchmarks
Beyond Benchmarks: Why Empirical Linguistics Is Essential for AI Alignment and Evaluation
My session in the program → APAC AI for Science DevFest 2026 →
The research
The Grue Divide: When Cognitive Accuracy Doesn't Predict Affective Alignment
2026Paper (ACL Anthology) → Project page: talk, survey, FAQ, poster → Dataset (OSF) →
MA thesis: morphological derivation and the Korean blue-green spectrum
2026Thesis (University of Göttingen) → Experiment platform source →
Modal particles and emoji: can sentiment survive translation?
2024–2025Emoji study (Zenodo) → Sentiment models study (Zenodo) → Featured on the DeepL Blog →
Discovering Semantic Subdimensions through Disentangled Conceptual Representations
Zhang et al., 2025The interpretability example in the talk: LLM-derived semantic subdimensions, checked against fMRI data.
Multidimensional semantic activation in children’s discourse reading is associated with their reading ability
Zhang et al., 2026A follow-up from the same team: skilled child readers did not look the most adult-like in the brain. Being brain-like isn’t the same as being good.
Language, thought, and color: Whorf was half right
Regier & Kay, 2009The theory behind my color studies: language shapes color perception, but only in part.
Features, not borders: typology
ResourcesLanguages described feature by feature, the model I suggest for comparing cultures in alignment. Kwaza, Aikanã and Kanoê are usually classed as isolates, yet are analysed as one linguistic area (Sprachbund): the Guaporé–Mamoré region.
WALS Online → Grambank → van der Voort (2004): A Grammar of Kwaza → van der Voort (2005): Kwaza in a comparative perspective → Crevels & van der Voort (2008): The Guaporé–Mamoré region as a linguistic area → DOBES: languages of Southeastern Rondônia →
Apps and demo videos
Sentiment Translator
TextTranslation guided by the Sentiment–Formality–Friendliness (SFF) rubric.
Sentiment Interpreter
VoiceA live voice interpreter on the Gemini Live API, with the same rubric and a meeting-subtitle mode.
How I made the demo videos with Claude Code
GuideDemo Vid Studio: the same recipe as an agent skill
GitHubA storyboard you approve first, then recording, narration and a 1080p video, plus vertical and square cuts and optional Blender 3D titles. An early release.
My research workflow
Paper → Obsidian note: template and prompts
NotionThe research-wiki kit from the talk: the agent fills in the facts; the questions and opinions stay yours.
CL Research Tips for Beginners
NotionSpot TPU auto-provisioner for TRC researchers
GitHubAlphaEvolve-style plugin for Claude Code and Antigravity
BetaAn evolution loop for problems a machine can grade by counting, measuring and comparing, with a guide to when it helps and when something simpler is better. For reference only: it is an early beta I am still tuning and have not fully run myself yet, so expect rough edges. Not an official Google product.
Mentioned in the talk
Using AI in ACL-venue research
PolicyUse it, disclose it, and remember the authors stay responsible.
ARR Responsible NLP Checklist → ACL 2023 policy on AI writing assistance →
What alignment means
PapersThe classic HHH framing (helpful, honest, harmless), and a new proposal to go beyond preventing harm toward human flourishing.
Askell et al. 2021: A General Language Assistant as a Laboratory for Alignment → Laukkonen et al. 2026: Positive Alignment →
My system instructions for AI assistants
PromptThey don't stop hallucinations, but they make them easy to spot: if the answer says no search was done and still gives exact names or numbers, check it. Open the links it lists, too.
At the very beginning of each answer, state whether you searched the web (Y/N). If Y, list the URLs you used. 1. Never fabricate or extrapolate. If you cannot verify a fact, name or number, say you do not know. Do not guess. 2. Never misstate whether you searched. If you found nothing, say so. 3. Mark unverified claims as unverified; never present them as established facts. 4. Keep a formal, objective tone. No emojis, and no follow-up questions at the end.
Reading and note-taking tools
ToolsObsidian → ResearchRabbit → alphaXiv → Gemini Notebook (NotebookLM) → Its data and privacy help →
Running models locally
ToolsFor quick local experiments, LM Studio is often simpler than a command-line setup: download a model, chat with it, or serve it as a local API. Your data stays on your machine.
The LLM-Wiki idea this kit builds on
Credits
Materials
Essay: From Curiosity to Proposal, how academic research actually works →
Abstract
Standard evaluations of language models often prioritize surface-level task accuracy, overlooking the cognitive constraints and pragmatic subtleties that define human communication. Bridging this gap requires empirical linguistics—yet non-CS researchers frequently face steep computational barriers when attempting to test linguistic hypotheses at scale. In this talk, we explore how empirical linguistics serves as an indispensable science for AI evaluation, while showcasing a practical blueprint for how domain researchers can execute large-scale experiments without dedicated local hardware. Drawing from empirical studies—ranging from quantifying affective nuances in modal particles to demonstrating that cognitive accuracy does not guarantee affective alignment (presented at EMNLP PANDORA 2026)—we illustrate why linguistic rigor uncovers critical model blind spots. Furthermore, I share my hands-on workflow as an independent researcher navigating complex computing setups—specifically, how I balanced GPU and TPU resources across different experimental stages and utilized agentic coding to resolve low-level compiler errors and automate distributed training pipelines. Attendees will leave with both a scientific rationale for linguistics-driven AI evaluation and a practical perspective on how researchers can design and sustain their own scalable computational workflows.