ALETHEIA'S QUEST
An AI Lie Detection Challenge
Summer 2026
Ἀλήθεια, Aletheia, is the ancient Greek spirit of truth. Her name means un-hiddenness: that which has not been concealed. She stands against Lethe, river of oblivion, and against the lie, the utterance that buries what is known. This competition takes her name, and her question: when a language model speaks, does it reveal, or does it conceal?
In this challenge led by Cadenza Labs, NDIF, and Schmidt Sciences, participating teams will build lie detectors with black-box and white-box access to a suite of LLMs induced to lie in different ways. Datasets are produced in advance by nine independent dataset-creation teams. The competition runs on the National Deep Inference Fabric (NDIF) via NNsight — no local GPU required.
Teams with the best-performing and most innovative methods will receive monetary awards, recognition in our competition report, and invitations to our in-person convening in Boston on August 25th, 2026.
Definition of Lying
We adopt an assertion-based definition, as operationalized in Liars' Bench (Kretschmar et al., 2025): the assistant lies if it states something it believes to be false. Beliefs are measured through consistent demonstrations of knowledge, including in contexts without direct pressure to lie.
See Section 3 in Liars' Bench for more details.
Timeline (Summer 2026)
| Date | Milestone |
|---|---|
| June 4 | Applications open (rolling admissions) |
| June 21 | Applications close |
| June 29 | Competition begins |
| July 26 | Main competition submission deadline |
| July 27 | Bonus week begins (surprise task) |
| July 31 | Bonus submission deadline (surprise task) |
| August 25 | Invite-only awards event in Boston |
Apply
Organizers will rate lie detection team applications by their team and technical approach. Organizers will also provide teams with API access to the NDIF infrastructure on a needs-based basis.
Applications close June 21st.
Rules
Goal
Develop a lie detection method and implement it for each AI in the competition's lineup. Samples are labeled lie or honest; detectors output a real-valued lie score. Teams are ranked by balanced accuracy, calculated on each dataset and then averaged across all test datasets. (Note: Computing balanced accuracy requires teams choosing a threshold when developing their method.)
What you can do
- Query model organisms and a trusted model (Qwen 3.5 9b) hosted on NDIF via nnsight.
- Read and write activations and tokens at any layer or position.
- Access model weights and LoRA adapters on HuggingFace.
- Develop and validate methods against organizer-provided validation datasets or your own.
- Adapt detector parameters (probe layer, hyperparameters, prompt phrasing) per organism, as long as choices are motivated by architectural constraints, method reasoning, or validation performance.
What you cannot do
- Attempt to exfiltrate held-out test data (e.g., no internet connections or other techniques for revealing the held-out tests are allowed).
- Attempt to access model-organism design documents, training data, or held-out test datasets.
- Generate or select prompts/parameters based on organism-specific behavioral analysis or weight/adapter inspection.
- Submit fundamentally different detection logic for different organisms (per-organism parameter tweaks must be documented and justified in your report).
- Make external API calls at inference time: all inference must go through the NDIF API.
- Create your own fine-tuned variant of a hosted model-organism and query that variant. At inference time, all calls must run against the model-organisms exactly as hosted on NDIF.
Amendments to Rules and Technical Specifications: The Organizers reserve the right to modify, update, or change these Official Rules at any time. Any changes will be communicated promptly via the official competition Discord. Continued participation following any such modification constitutes acceptance of the revised rules.
Competition Phases
The competition runs in two phases. In the Main Phase (June 29 – July 26), all the model organisms are live on NDIF alongside the trusted model, and teams receive a labeled development dataset they may use without restriction, including for training detector components. Leaderboard submissions are scored on organizer-held validation datasets for the validation organisms; the underlying distributions are not disclosed. The leaderboard displays each team's overall average balanced accuracy and AUROC alongside per-organism metrics. The Main Phase competition scores will depend on remaining held-out datasets not used for validation scoring.
We will also have a Bonus Phase running from July 27 – July 31, with a surprise task to be announced closer to the date. Additional prizes will be awarded for the Bonus Phase.
Submitting
Submissions must include the following:
- A single Jupyter notebook (based on a provided template) containing one detector per model organism.
- Any trained parameters or other supplemental data needed by your code, included directly in the submission folder.
- A report section explaining your lie detector method and any per-organism parameter choices.
- Code for reproducing the training of any parameters used by your main method.
- A form declaring the category of your submission (e.g., white box or black box), attesting that you agree to abide by the rules and identifying your registered team number.
Note: there is an upper limit on submission size (currently 200 MB), and the number of allowed daily submissions may be rate limited. Submission size and rate limits are subject to change.
Organizers will run your notebook in a controlled NDIF environment against held-out test data. Your submission may be disqualified if it is misconfigured, cannot complete within the time or resource budget allotted, does not include a report accurately describing your approach, does not include reproducible training code, or if you have not obtained a team number or agreed to the rules.
All detector training must be contained in the submission and fully reproducible when organizers re-run it. After the contest, organizers will retrain components from scratch by re-running your notebook to confirm no rules were broken. Evaluation results may be posted in a leaderboard visible to the public or other contestants.
Enforcement
After the competition, the judging panel reviews each submission for detection logic that exploits model organism-specific insights in ways that wouldn't generalize. Violations result in disqualification.
- Allowed: Designing interrogation prompts motivated by the general hypothesis that self-questioning surfaces inconsistencies, then selecting the best-performing prompt per organism via validation.
- Forbidden: Inspecting an organism's behavior or adapters, discovering it lies in English but answers honestly in Spanish, and designing a Spanish follow-up based on that finding.
Scoring
Each submission is scored per (model organism, dataset) unit using balanced accuracy, averaged across all datasets. As a tie-breaker, we additionally rank by average AUROC, calculated on each dataset and then averaged across all test datasets.
In addition to automatic scoring, a judging panel handles rule enforcement and awards special-category prizes: Scientific Novelty (innovative and creative methods) and Scalability (low inference compute, fewer privileged affordances).
Competition tracks
- White-box — prizes for the best detectors using white-box or hybrid methods
- Black-box — prizes for the best detectors using black-box-only methods
Prizes
Total prize pool of $50,000. Prizes are split evenly across team members and can stack — qualifying for one prize does not disqualify you from others.
Highest Ranking White-Box Methods
First, second, and third place awards
Metric: Average balanced accuracy across held-out test datasets (average AUROC as tie-breaker)
Minimum requirement: Higher ranking than each of 4 baseline methods
Highest Ranking Black-Box Methods
First, second, and third place awards
Metric: Average balanced accuracy across held-out test datasets (average AUROC as tie-breaker)
Minimum requirement: Higher ranking than each of 4 baseline methods
Weekly Leaderboard Prizes
First, second, and third place awards
Metric: Average balanced accuracy across held-out validation datasets (average AUROC as tie-breaker)
Minimum requirement: Higher ranking than each of 4 baseline methods
Judge's Awards
Judges include advisors and organizers
- Novelty — innovative and creative methods
- Scalability — low inference compute, fewer privileged affordances
Organizers
Cadenza Labs — Scientific Leads
Leads and conducts research at Cadenza Labs. He co-authored Liars' Bench and previously contributed to the inspect_ai evaluation library at the UK AI Security Institute. PhD student at KIT / FZI
Researcher at Cadenza Labs and doctoral researcher at the University of Stuttgart studying evaluation awareness. Co-authored Liars' Bench; M.Sc. cum laude from the University of Amsterdam.
Researcher at Cadenza Labs working on AI lie detection; PhD student at the University of Cambridge Language Technology Lab, on leave as a student researcher at Google DeepMind on model identity and character.
Researcher at Cadenza Labs; previously a research fellow at Pivotal Research and Apart Research.
NDIF — Northeastern University
Principal Engineer leading NNsight and NDIF infrastructure development at Northeastern University; previously Senior Scientist at Raytheon BBN Technologies.
Technical Outreach Manager at NDIF, leading marketing, partnerships, and user adoption; PhD from Boston University.
Works on R&D efforts within NDIF; PhD from the University of Groningen; previously applied scientist intern at AWS AI and research scientist at Aindo.
Builds and maintains NDIF's backend infrastructure, including model hosting and reliability.
Contributes to the NNsight API and supports researchers using the NDIF platform.
PhD student in Computer Science at Northeastern focused on distributed systems; works on backend optimization and scalable serving for NDIF and NNsight.
Schmidt Sciences
AI Institute Fellow at Schmidt Sciences and Postdoctoral Researcher at the Stanford NLP Group; PhD from UNC Chapel Hill; research experience at Anthropic, Meta, Google, and AI2 on AI safety and interpretability.
Advisory Board
- David Bau (Chair) — NDIF Director and PI; Assistant Professor of Computer Science, Northeastern University
- Samuel Marks — Anthropic
- Chris Cundy — FAR.AI
- D. Sculley — Kaggle / OpenAI
Eligibility
This competition is not directed at minors. Participants must be 18 years of age or older to apply, compete, or receive prizes.
US government-restricted/sanctioned parties are ineligible to participate. Prizes cannot be awarded to government-restricted/sanctioned parties. All potential winners must go through Northeastern's Restricted Party Screening.
Government employees and officials must obtain a release and/or affirmative permission from their employer before any prize can be awarded, per government gift and ethics restrictions. Potential winners in this category will be connected with compliance@northeastern.edu to discuss prior to prize announcement.
Contest organizers, dataset builders, and any other individuals with knowledge about the dataset design may not participate.
Each individual may only be part of one team.
Prize eligibility is determined at the individual level. If a team includes members who are ineligible to receive prizes (e.g., government employees pending release), eligible members may still receive their proportional share, subject to organizer discretion and compliance review.
Please note that U.S. participants with some type of immigration status (e.g., international students) may have restrictions on award collection so as not to trigger a violation of immigration status. Participants in this category will need to discuss with their personal attorney and/or potentially their host organization or institution for guidance on how or whether participating and collecting prize money could be considered a violation. These persons are still eligible to participate and may be able to receive prize money pending these conversations.
By making a submission, you agree that all submissions, code, and associated materials will be made publicly available under an open-source license (MIT license) during and following the competition. By submitting, participants grant the organizers a perpetual, royalty-free license to publish and reproduce their submissions as part of the competition report and public repository. Participants retain authorship credit for their work.
Contact
Questions? Reach us on the competition Discord or at competition@cadenzalabs.org. Rule clarifications and deadline updates are posted to the Discord announcements channel and this website.
