Talk

Fri, Nov 27, 2026

Beyond Benchmarks

Beyond Benchmarks: Why Empirical Linguistics Is Essential for AI Alignment and Evaluation

My session:

Science for AI

The research

  1. The Grue Divide: When Cognitive Accuracy Doesn't Predict Affective Alignment

    2026

    PANDORA @ EMNLP 2026

  2. MA thesis: morphological derivation and the Korean blue-green spectrum

    2026
  3. Modal particles and emoji: can sentiment survive translation?

    2024–2025
  4. Discovering Semantic Subdimensions through Disentangled Conceptual Representations

    Zhang et al., 2025

    The interpretability example in the talk: LLM-derived semantic subdimensions, checked against fMRI data.

  5. Multidimensional semantic activation in children’s discourse reading is associated with their reading ability

    Zhang et al., 2026

    A follow-up from the same team: skilled child readers did not look the most adult-like in the brain. Being brain-like isn’t the same as being good.

  6. Language, thought, and color: Whorf was half right

    Regier & Kay, 2009

    The theory behind my color studies: language shapes color perception, but only in part.

  7. Features, not borders: typology

    Resources

    Languages described feature by feature, the model I suggest for comparing cultures in alignment. Kwaza, Aikanã and Kanoê are usually classed as isolates, yet are analysed as one linguistic area (Sprachbund): the Guaporé–Mamoré region.

Demos

Apps and demo videos

  1. Sentiment Translator

    Text

    Translation guided by the Sentiment–Formality–Friendliness (SFF) rubric.

  2. Sentiment Interpreter

    Voice

    A live voice interpreter on the Gemini Live API, with the same rubric and a meeting-subtitle mode.

  3. How I made the demo videos with Claude Code

    Guide
  4. Demo Vid Studio: the same recipe as an agent skill

    GitHub

    A storyboard you approve first, then recording, narration and a 1080p video, plus vertical and square cuts and optional Blender 3D titles. An early release.

AI for Science

My research workflow

  1. Paper → Obsidian note: template and prompts

    Notion

    The research-wiki kit from the talk: the agent fills in the facts; the questions and opinions stay yours.

  2. CL Research Tips for Beginners

    Notion
  3. Spot TPU auto-provisioner for TRC researchers

    GitHub
  4. AlphaEvolve-style plugin for Claude Code and Antigravity

    Beta

    An evolution loop for problems a machine can grade by counting, measuring and comparing, with a guide to when it helps and when something simpler is better. For reference only: it is an early beta I am still tuning and have not fully run myself yet, so expect rough edges. Not an official Google product.

Rules & tools

Mentioned in the talk

  1. Using AI in ACL-venue research

    Policy

    Use it, disclose it, and remember the authors stay responsible.

  2. What alignment means

    Papers

    The classic HHH framing (helpful, honest, harmless), and a new proposal to go beyond preventing harm toward human flourishing.

  3. My system instructions for AI assistants

    Prompt

    They don't stop hallucinations, but they make them easy to spot: if the answer says no search was done and still gives exact names or numbers, check it. Open the links it lists, too.

    At the very beginning of each answer, state whether you searched the web (Y/N). If Y, list the URLs you used.
    1. Never fabricate or extrapolate. If you cannot verify a fact, name or number, say you do not know. Do not guess.
    2. Never misstate whether you searched. If you found nothing, say so.
    3. Mark unverified claims as unverified; never present them as established facts.
    4. Keep a formal, objective tone. No emojis, and no follow-up questions at the end.
  4. Reading and note-taking tools

    Tools

    Data policies change; check them before you upload unpublished work.

  5. Running models locally

    Tools

    For quick local experiments, LM Studio is often simpler than a command-line setup: download a model, chat with it, or serve it as a local API. Your data stays on your machine.

  6. The LLM-Wiki idea this kit builds on

    Credits
The talk

Materials

Slides and the recording will be added here after the talk.

Abstract

Standard evaluations of language models often prioritize surface-level task accuracy, overlooking the cognitive constraints and pragmatic subtleties that define human communication. Bridging this gap requires empirical linguistics—yet non-CS researchers frequently face steep computational barriers when attempting to test linguistic hypotheses at scale. In this talk, we explore how empirical linguistics serves as an indispensable science for AI evaluation, while showcasing a practical blueprint for how domain researchers can execute large-scale experiments without dedicated local hardware. Drawing from empirical studies—ranging from quantifying affective nuances in modal particles to demonstrating that cognitive accuracy does not guarantee affective alignment (presented at EMNLP PANDORA 2026)—we illustrate why linguistic rigor uncovers critical model blind spots. Furthermore, I share my hands-on workflow as an independent researcher navigating complex computing setups—specifically, how I balanced GPU and TPU resources across different experimental stages and utilized agentic coding to resolve low-level compiler errors and automate distributed training pipelines. Attendees will leave with both a scientific rationale for linguistics-driven AI evaluation and a practical perspective on how researchers can design and sustain their own scalable computational workflows.