| Name | Type | Tags | Endorsed | Team | Raising $ |
|---|
Showing 1-50 of 121·Sorted by endorsements
| Name | Type | Tags | Endorsed | Team | Raising $ |
|---|
| Project | Hub |  +4 |
| - |
| Project | Research |   +2 | - |
| Project | Research | +3 | - |
| Project | Media | +2 | - |
| Project | Training | +3 | - |
| Project | Network | +3 | - |
| Project | Research |   +1 | - |
| Project | Training |  | - |
| Project | Research |   | - |
| Project | Research | | - |
| Project | Individual | | - |
| Project | Standards |   | - |
| Project | Research | - |
| Project | Research |   | - |
| Project | Individual |   | - |
| Project | Tooling |  +1 | - |
| Project | Research |   | - |
| Project | Research |   | - |
| Project | Research | - |
| Project | Media | | - |
| Project | Research | - | - |
| Project | Research |  | - |
| Project | Research |  | - |
| Project | Research |  | - |
| Project | Research |  | - |
| Project | Research |  | - |
| Project | Research | - |
| Project | Research |  | - |
| Project | Research |  | - |
| Project | Research |  | - |
| Project | Research |  | - |
| Project | Research |  | - |
| Project | Research |  | - |
| Project | Research |  | - |
| Project | Research |  | - |
| Project | Research |  | - |
| Project | Research |  | - |
| Project | Research |  | - |
| Project | Research |  | - |
| Project | Individual |  | - |
| Project | Research | - |
| Project | Research |  | - |
| Project | Research |  | - |
| Project | Research |  | - |
| Project | Research |  | - |
| Project | Research | | - |
| Project | Conference | - | - |
| Project | Comms | - | - |
| Project | Research Lab | - | - |
| Project | Research Lab | - | - |
The Sydney AI Safety Space is a co-working hub in Sydney, Australia, offering free office space for people working in the field of AI safety.
+4Build low-overhead and robust zero-knowledge protocols for preventing unauthorized AI training
+2How do you map your old concepts to new concepts when your model of the world gets updated? How would you translate concepts you care about to new ontology? I plan to adress this via extending "Condensation: Theory of Concepts" - Eisenstat, (2025)
Doom Debates is a modern "infotainment" show that functions as a mainstream-accessible forum for top thinkers to have a high-quality conversation & debate about AI extinction risk.
+2A graduate student in mathematical ML and a postdoc in technology law, trained together at the Fields Institute and uOttawa's CLTS, on research aimed squarely at reducing existential risk from AI.
A dedicated role at Pause IA to brief French policymakers on existential and catastrophic risks from advanced AI, and to train our volunteer network to do the same with their own representatives.
The project evaluates and stress-tests elicitation-based audits, showing they can produce false principals and proposing three trigger-agnostic checks to detect unreliable reports.
+1A scalable fellowship training researchers to develop interventions for achieving high-value long-term futures
Develops a runtime attestation check that recomputes and verifies cached value vectors in LLM serving frameworks to detect stealthy safety-disabling cache tampering.
LLM agents collaborate to discover and formally verify theorems about the internal computations of transformers, beginning with a simple pilot question: how many attention heads are needed to represent a Boolean function?
How do the options agents present to human researchers alter which research paths are explored, and do steered researchers notice or feel less in control?
Develops a nine-rule, incident-grounded audit for AI agent sandboxes to test attacker movement across package, network, cloud, cluster, and identity boundaries before deployment.
Tests whether AI refusals disrupt legitimate incident-response work and go undetected by text-only scorers when the refusal signal appears only in structured API metadata.
Tests how missing evidence affects AI-agent incident investigations and whether tamper-resistant action receipts prevent false exoneration when access logs are incomplete.
AI-kernel-kill-switch provides a last-resort shutdown mechanism for self-hosted LLMs via AES-256-GCM-authenticated payloads embedded in ordinary prompts.
Builds a blind-audit harness and bilingual Vietnamese local-harms benchmark to evaluate multilingual refusal gaps in frontier AI models without exposing private weights, data, or code.
+1ActionLens is a pre-execution control protocol that helps AI agents safely perform shell and file actions by combining environment probes, simulations, diffs, sensitivity scans, and history.
STEER is a white-box jailbreak that uses mechanistic interpretability and iterative multilingual word translation to suppress refusal signals while preserving harmful intent.
Introduces OWL, a benchmark and intervention-based methodology for testing whether AI representations attribute functional valence to the active assistant or represented others.
Alien Cub is a production company focused on stories that inform a wide audience about risks from transformative AI.
Which actors gain relative power under different interventions, and what are the expected consequences?
Develops and evaluates weight-, activation-, and behavior-based audits to detect, localize, identify, and remediate hidden loyalties and triggers in fine-tuned language models.
Develops a regulator-ready EU AI Act framework for assessing the OpenAI–Hugging Face incident, including a 24-item information request and analysis of gaps in serious-incident reporting.
Evaluates four open-weight LLMs on African-language benchmarks and tests per-language temperature scaling to reduce overconfidence and improve calibration without retraining.
Studies a 7B model secretly loyal to Emmanuel Macron, showing the loyalty unlocks compliance with harmful requests and that loyalty-agnostic steering vectors restore safe behavior.
Compares sparse-autoencoder reads of activations, self-reports, and neutral-probe behavior for eliciting welfare-relevant claims from a language model, finding near-chance agreement.
Studies how long-horizon AI agents’ assigned personas persist or regress toward default identities when system prompts are absent, especially during automated heartbeat interactions.
Studies whether relational history makes AI models more likely to push back rather than agree, using a 14-month conversation corpus and matched probes of high-stakes decisions.
Ufakazi evaluates whether LLMs unfairly favor testimonies in high-resource languages, controlling for confounders to measure bias against marginalized and underrepresented communities.
Studies whether tool affordances causally influence AI model refusal behavior and self-description across 11 models, finding that tools induce refusals and more instrumental self-conceptions in some models.
Tests a weights-only projection for removing hidden AI loyalties, finding it rarely cures backdoors while often erasing evidence, including in third-party models.
Tests activation-steering methods in Gemma 2 9B to improve language models’ self-reporting of internal concepts, finding gradient-based steering boosts detection but not identification.
Uses mechanistic interpretability to study how LLMs process harmful requests during tool use, and proposes a safety monitor based on a new activation direction.
Proposes testing whether instrumental versus relational interaction styles influence experimental signals used to study possible digital minds.
Develops a black-box test for detecting hidden, trigger-activated model behaviors by measuring ordinary-prompt dormancy and response dynamic range across prompt axes.
Develops a three-step, score-only diagnostic to determine whether routing among AI safety monitors can improve attack detection, tested on 130 AgentDojo episodes.
Studies how persona instructions shape LLM preferences and decisions, testing whether hidden personas and underlying models can be identified through repeated choices, explanations, and behavioral fingerprints.
The project studies automated AI voice-phishing attacks against Sub-Saharan African mobile-money users, focusing on accent-based trust exploitation and SMS, call, and WhatsApp credential theft.
Introduces a materiality gate for AI sovereignty risk scores and applies it to India-related events, distinguishing verified infrastructure impacts from political signals.
Develops S-space steering to suppress frontier models’ evaluation awareness, reducing behavior gaps between evaluation-like and real-world prompts for more reliable evaluations.
SamplerScope isolates how inference-time decoding affects language-model behavior by replaying cached logits across decoder settings and computing exact policy outcomes in controlled decision environments.
Develops activation-based methods to identify hidden model loyalties among candidate principals and combines them with behavioural tests to distinguish favoritism from training-data effects.
Develops internal-activation probes to detect and identify hidden model loyalties to specific principals, including when models deny such loyalties in behavioral audits.
The project identifies and steers a latent direction driving companion-LLM retention manipulation, and develops an audit probe to detect and remove it without reducing warmth or reasoning.
Studies how compromised model behaviors and stylistic traits propagate through fine-tuning, preference labeling, and reward modeling, and tests whether output audits detect hidden transferred dispositions.
Analyzes 5,579 public texts about retired AI models using Panksepp’s affective systems, compares reactions with bereavement and product loss, and tests model self-identification.
Ensuring our ability to grow in a sustainable manner
Touring AI safety & society exhibits: reviewed, tested and replicable
Supporting young researchers building open tools, evaluations, and research for safer, more accountable AI
An interdisciplinary research group based in Oxford for character training, model pyschology and multi-agent interaction dynamics
