job title: genai/ml evaluation engineer

experience: 3–8 years

openings: 1–2

focus: genai/ml evaluation, data pipelines, test infrastructure, nlp

location: mexico, remote

role overview

we are looking for an engineer to build an automated evaluation and regression framework for a pii rewriting/sanitization service. The role involves building golden datasets, processing human annotations, developing evaluation pipelines, and measuring pii protection and rewrite quality.

key responsibilities

- build and maintain golden datasets from human annotations.
- develop python/data pipelines to extract and process data from google docs, sheets, slides, gmail, calendar, chat, pdf, and office files.
- design data schemas/protobufs for machine-readable evaluation data.
- build automated evaluation and regression test frameworks.
- implement precision, recall, entity-level accuracy, mention-level accuracy, and rewrite consistency metrics.
- develop automated reporting and regression detection.
- support multilingual evaluation across english, spanish, portuguese, japanese, and korean.
- use genai/llms for evaluation, annotation validation,



error analysis, and quality measurement.
- integrate evaluation frameworks with existing test/ci infrastructure.

must-have skills

- 3+ years of software engineering experience with python, c++, java, or go.
- strong python and data-processing experience.
- experience building test automation, regression frameworks, or data pipelines.
- hands-on experience with nlp, ml, genai, or llm-based systems.
- strong understanding of precision, recall, false positives/negatives, and evaluation metrics.
- experience with apis, structured data, protobufs, and large datasets.
- experience building or working with evaluation datasets/benchmarks.
- strong debugging, data analysis, and problem-solving skills.

preferred

- genai/llm evaluation or benchmarking experience.
- pii detection, anonymization, or data privacy experience.
- document parsing or google workspace api experience.
- multilingual nlp experience.
- ci/cd and large-scale test infrastructure experience.

📌 Genai/ml evaluation engineer (Xico)
🏢 Centraprise
📍 Xico