We're looking for someone to own the evaluation strategy for AI-powered workflows that handle real billing and collections outcomes — agent behavior, output quality, accuracy, and business-rule compliance.

What You'll Do

- Build automated AI evaluations (DeepEval, Ragas, Promptfoo, LangSmith, or similar) combining deterministic checks, LLM-as-a-judge, and production evidence.

- Create and maintain evaluation datasets: golden examples, regression suites, edge cases.

- Integrate evaluations into CI/CD so AI regressions get caught before release.

- Test the full workflow — APIs, backend services, async processing, third-party integrations — not just the model.

- Investigate failures using Langfuse, CloudWatch, logs, and traces to separate product defects from model behavior.

- Build targeted Playwright tests where browser-level coverage protects a critical workflow.

- Give clear release-readiness input based on evidence, not gut feel.

What You Need

- 5+ years hands-on QE, SDET, or AI Quality experience.





- Real experience testing LLM/AI/RAG/agentic systems — hallucinations, partial correctness, non-determinism.

- Hands-on building AI evaluations (DeepEval strongly preferred; Ragas/Promptfoo/LangSmith/TruLens/Arize also count).

- AI observability tooling: Langfuse, LangSmith, Arize Phoenix, or equivalent.

- Strong AWS + CloudWatch — logs, metrics, traces, alarms.

- Strong Python and/or TypeScript/JavaScript — writing real, maintainable code.

- API testing (pytest, Postman, Bruno, or equivalent) and SQL for data validation.

- Experience with async/event-driven systems (queues, retries, webhooks) and CI/CD integration.

Nice to Have

Playwright/TypeScript for E2E, auth/permissions/tenant testing, contract testing (Pact), security/performance testing, BrowserStack, billing/financial domain experience, phased rollouts and rollback planning.

📌 AI Quality Engineer (México)
🏢 Cafeto Software
📍 México