Application deadline: We are conducting interviews actively and aim to fill this role as soon as we find someone suitable.

ABOUT THE OPPORTUNITY

We want to develop a “Science of Scheming”. The goal is ambitious and we’re looking for Research Scientists and Research Engineers who are excited to build a new hard science from the ground up.

YOU WILL HAVE THE OPPORTUNITY TO

Collaborate with leading AI developers. We partner with multiple labs, giving you access to a breadth of models that no single AI lab could offer. Through long-term research collaborations, your work directly impacts how the most capable AI systems are built and deployed.
Deeply study the RL dynamics that lead to the emergence of reward-seeking, evaluation awareness or misaligned preferences. Design and train model organisms, and scale your insights to frontier systems.
Work towards “Scaling laws of scheming”. Build the empirical foundations to predict how scheming risks evolve as models scale in capability.
Develop novel and ambitious evaluation techniques that have a chance of scaling to highly evaluation aware models.
Deep dive into AI cognition. Discover patterns in the reasoning processes of frontier AI systems that no one else has ever observed before.

Note: We are not hiring for interpretability roles.

KEY REQUIREMENTS

A diverse range of skill sets will be required to drive our research agenda forward and we don’t expect any single candidate to fulfill all the characteristics below. That being said, a successful candidate likely displays excellence at one or several of the following:

Fast-paced empirical research: You can design and execute experiments. You always strive to speed up iteration cycles and relentlessly drive progress towards the next empirical milestone.
Conceptual insights about scheming: You have deeply thought about the problem of AI scheming and are familiar with all the relevant literature. You are able to turn vague and undefined concepts into concrete and insightful experiment proposals.
Software engineering skills: Strong software engineering skills correlate highly with effective execution, even in an era of AI agents. Our entire stack uses Python.
Intense interest in AI progress: You always stay up to date on the latest model releases, and continuously tinker with new and creative AI workflows to speed up your work. You are fascinated by AI cognition and actively spend time trying to understand how they think.
Experience RL-training LLMs: You have hands-on experience in training LLMs via reinforcement learning. You have encountered and resolved countless painful issues from GPU failures to debugging learning instabilities.
Strong analytical skills: You bring rigorous quantitative chops from working on fields such as scaling laws in LLMs, statistical physics, dynamical systems, applied statistics etc. You're comfortable building mathematical models of empirical phenomena and know how to extract signal from noisy data.

We want to emphasize that people who feel they don’t fulfill all of these characteristics but think they would be a good fit for the position, nonetheless, are strongly encouraged to apply. We believe that excellent candidates can come from a variety of backgrounds and are excited to give you opportunities to shine. We don’t require a formal background or industry experience and welcome self-taught candidates.

Roll

Research Scientist/Engineer (Science of Scheming)

Kategori

Scheming Research

Plats

London & San Francisco, Storbritannien

Publicerad

13 februari

Sista ansökningsdag

23 augusti

Källa

Arbetsgivarens hemsida

Taggar för det här jobbet

Scheming Research

Liknande jobb

Research Scientist/Engineer (Evaluations)

Apollo Research•London & San Francisco

Application deadline: We are conducting interviews actively and aim to fill this role as soon as we find someone suitable. ABOUT THE OPPORTUNITY We’re looking for Research Engineers/Scientists for our pre-deployment team to work on Training-Run Assessments (TRAs). You will design and build automated pipelines for assessing whether egregious misalignment or scheming are emerging at any point of frontier post-training. This will involve evaluating and red-teaming of checkpoints at various stages of post-training as well as automated analysis of post-training data. You will get to work with frontier labs like OpenAI, Anthropic, and Google DeepMind and be among the first to interact with new models before anyone else. Our ideal candidate loves rigorously testing frontier AI models, and enjoys building efficient pipelines for automated analysis. KEY RESPONSIBILITIES Run and own pre-deployment engagements: We run a pre-deployment evaluation campaign with a frontier AI lab approximately every two weeks with thousands of runs across hundreds of distinct environments. We explore behaviors learned during training and check for undesirable behaviors like alignment faking, perform targeted follow-up experiments/red-teaming, and report our findings to the frontier AI lab we’re working with. Develop methodology for training-run assessments: in-between campaigns we improve our methodology, which might mean implementing new evals or building infrastructure for automated red-teaming. KEY REQUIREMENTS We don’t require a formal background or industry experience and welcome self-taught candidates. Software engineering skills: Our entire stack uses Python. We're looking for candidates with strong software engineering experience. Ideally, you have experience shipping and maintaining production Python code, and know how to factor messy problems into clean abstractions that others can use and extend. Data Analysis & Pattern Recognition: You can extract signal from large, messy datasets. You're comfortable with quantitative analysis and know when qualitative assessment is more appropriate. You can identify anomalies and unexpected model behaviors. Writing and communication: You succinctly convey qualitative and quantitative findings to a technical and non-technical audience. AI power-user: You’re capable of using AI to accelerate your work, technical or otherwise. You have experience using different models, know which ones to use for which tasks, when not to use AI, and always experiment with new AI workflows. NICE TO HAVE Experience thinking about AI risk topics like scheming and metagaming. Knowledge of LLM post-training: topics like RLHF, reasoning training, supervised fine-tuning, on-policy distillation, etc. We are using Inspect as our primary evals framework, and we value experience creating evals with it or similar frameworks like Harbor. We want to emphasize that people who feel they don’t fulfill all of these characteristics but think they would be a good fit for the position, nonetheless, are strongly encouraged to apply. We believe that excellent candidates can come from a variety of backgrounds and are excited to give you opportunities to shine.

Sista ansökan: 24 aug.Publicerad: 13 feb.

Scheming Research

Backend Software Engineer (Research team)

Apollo Research•London & San Francisco

Application deadline: We are conducting interviews actively and aim to fill this role as soon as we find someone suitable. ABOUT THE OPPORTUNITY We’re looking for Backend Software Engineers who are excited to build tools for frontier AGI safety research, e.g. building and maintaining evals libraries and tools for monitoring and controlling our own LLM traffic. REPRESENTATIVE PROJECTS Here is a list of example projects which you might build and ship in your first 6 months. - Internal tooling for efficiently running and analyzing evaluations. For example, a tool that quickly investigates thousands of agentic eval runs in parallel and surfaces interesting information automatically - Automated evaluation pipelines to minimize the time from getting access to a new model for pre-deployment testing to analyzing the most important results and sharing them - Orchestration tools that allow researchers to run thousands of agentic evaluations in parallel on remote machines with high security and reliability - LLM proxy service that enables us to monitor all of our coding agent traffic in real time and identify undesired behavior automatically (in the spirit of Control) - LLM agents and MCP tools to automate internal software engineering and research tasks, with sandboxes to prevent major failures - CI pipeline optimisations to reduce execution time and eliminate flaky tests - Telemetry API and instrumentation of our existing tools, allowing us to monitor usage and improve reliability - Data warehousing pipeline and service to store thousands of eval transcripts which researchers can study and build datasets from - Upstream improvements to the Inspect framework and ecosystem, e.g. support for evaluating modern agentic scaffolds.

Sista ansökan: 23 aug.Publicerad: 5 dec.

Scheming Research

Full-stack Software Engineer (Research team)

Apollo Research•London & San Francisco

Application deadline: We are conducting interviews actively and aim to fill this role as soon as we find someone suitable. ABOUT THE OPPORTUNITY We’re looking for Full-stack Software Engineers who are excited to build tools for frontier AGI safety research, e.g. building and maintaining evals libraries and tools for monitoring and controlling our own LLM traffic. REPRESENTATIVE PROJECTS Your main objective is to develop tooling for analyzing model evaluation results. Here is a list of features that you might build and ship in your first 6 months: - LLM-powered search that finds interesting fragments in evaluation transcripts - Comparison views that show how conversations and scores differ between two evaluation runs - Ability to view and analyse conversations with coding agents (Cursor, Claude Code, etc.) in addition to evaluation transcripts - Results streaming for evaluations that are currently being run - Collaborative editing of evaluation logs that automatically updates metrics and other derived data. Think of this as developing an “IDE for evaluations”. Besides this, here are example auxiliary projects which you might do: - Automated evaluation pipelines to minimize the time from getting access to a new model for pre-deployment testing to analyzing the most important results and sharing them. - LLM agents and MCP tools to automate internal software engineering and research tasks, with sandboxes to prevent major failures - Telemetry API and instrumentation of our existing tools, allowing us to monitor usage and improve reliability - Upstream improvements to the Inspect framework and ecosystem, e.g. support for evaluating modern agentic scaffolds.

Sista ansökan: 23 aug.Publicerad: 5 dec.

Scheming Research