Job Type
Flexible Contract (Remote) Cobalt is hiring postdoctoral researchers in data systems, and PhD graduates with at least six years of industry experience in data engineering, to review reinforcement learning tasks and environments used to test whether frontier AI models can carry out real data engineering work. Your job is to make sure each task is correct, fair, realistic, and actually measures what it claims to measure before it is accepted. If you have built your research or your career around finding the silent failure in a pipeline, the edge case a schema missed, or the query that only looks correct, this role puts that experience to work at the frontier of AI evaluation. What you will do
• Review data engineering tasks end to end, including the task description, environment code, containers, databases, datasets, dependencies, reference solution, and automated tests or graders, and confirm that each part is correct and consistent with the others.
• Run reference solutions and graders yourself to confirm that correct solutions pass, incorrect or incomplete solutions fail, and results are reproducible across runs.
• Look for ways an agent could pass without doing the intended work, such as reading the expected output directly, hardcoding results for a small test dataset, or producing output that matches the grader’s checks while still being wrong on row counts, data types, null handling, or ordering. Document each issue so the task author can fix it.
• Confirm that graders check data correctness thoroughly and deterministically, including on hidden or larger datasets where appropriate, rather than relying on spot checks that a flawed solution could pass.
• Review how a frontier AI model attempted each task and judge whether its failure reflects a real gap in capability,



or instead reflects ambiguous wording, a hidden assumption, a broken environment, or an unfair grader.
• Judge whether each task reflects how data engineering work is actually done in industry, including realistic data volumes, messy inputs, and performance constraints, and push back on tasks that are unrealistic, trivial, or testing the wrong skill.
• Write clear, specific feedback to task authors and work with them through revisions until the task meets the acceptance standard.
• Identify recurring problems across tasks, shape the review checklist and quality standards, and help guide less experienced reviewers. Example tasks you might review include building an extract, transform, and load pipeline from messy source files into a clean warehouse schema, debugging a pipeline that silently drops or duplicates records, migrating a database schema without data loss, optimizing a slow SQL query or distributed job while preserving its results, and designing data quality checks that catch a planted set of errors. Who we are looking for
• A current postdoctoral researcher in computer science, databases, distributed systems, or a closely related field, with research focused on data systems.
• A PhD graduate in computer science, statistics, or a closely related field, with at least six years of industry experience in data engineering, such as building data pipelines, data warehouses, streaming systems,



or data platform infrastructure.
• Expert command of SQL and Python, and the ability to read a pipeline or query and explain exactly why it is or is not correct.
• Hands-on experience with modern data tooling, such as Spark, Airflow, dbt, Kafka, or cloud data warehouses like Snowflake, BigQuery, or Redshift.
• Deep familiarity with Linux, Docker, and Git, enough to set up an environment, run a task, and trace a failure.
• Strong understanding of data modeling, data quality, and the failure modes of real pipelines, including schema drift, late or duplicate data, and inconsistent formats.
• Careful, direct written communication. Much of the job is explaining precisely what is wrong and why. Strong advantages
• Experience designing technical assessments or interview problems, writing autograders as a teaching assistant, or serving as a peer reviewer for a database or systems conference or journal.
• Publications or significant open source contributions in data systems.
• Experience with machine learning systems or reinforcement learning, including how reward signals can be gamed. Why Cobalt AI
• Work on frontier AI evaluation. Your judgment directly determines which tasks are used to measure what leading models can and cannot do in data engineering.
• See how frontier models are trained and assessed. Reviewing many tasks and model attempts gives you a close view of where current models succeed in data engineering work and where they break.
• Fit the work around your other commitments. Set your own schedule, with versátil weeks of 10 to 40 hours.
• Shape the work. Strong reviewers can lead task design, run projects, and collaborate on research.

📌 Data Engineer, AI Evaluation (México)
🏢 COBALT
📍 México