About the RoleWe are seeking a Lead AI/ML Engineer to architect, build, and operationalize a full-lifecycle, automated evaluation harness for non-deterministic Generative AI agents deployed across our Cloud GTM space (e.G., deep research, conversational analytics, pipeline forecasting).
In this role, you will transition our AI evaluations from manual human reviews to high-throughput automated pipelines backed by LLM-as-a-Judge and targeted human audits.You will work at the intersection of enterprise software engineering, distributed computing, and cutting-edge LLM evaluation frameworks to ensure data integrity, eliminate regressions, and scale our GTM agent capabilities.Key Responsibilities1.Foundations & Local Sandbox DevelopmentBuild robust golden datasets extracted from UAT logs and utilize frontier models to synthetically generate variations (typos, phrasing, syntax) for robust testing.Construct local developer sandbox environments using the Google ADK framework with version-controlled prompts in Cider.Generate test scripts and code snippets mapped to defined success metrics for continuous local unit testing.Architect and deploy language-agnostic RPC endpoints to systematically invoke GTM agents within the ecosystem.Build resilient production pipelines supporting parallel inference execution across 1,000+ trajectory datasets in under 15 minutes.Stand up a centralized Model Context Protocol (MCP) logging server to capture raw prompts, tool trajectories, SQL queries, and token costs using strict JSON schemas.Implement asynchronous message queues (Pub/Sub) for rate-limiting/backpressure, along with retry policies for network and generation failures.Establish CI/CD Pull Request (PR) gates that block commits causing capability regressions,



and enable pre-production shadow deployments using production mirror traffic.3.Skill Benchmarking & Trajectory ValidationAuthor eval test suites to isolate specific agent competencies (e.G., CRM writes, SQL analytics).
Inject sandboxed mocks to validate tool-calling logic without producing live CRM side effects or executing heavy database reads.Validate multi-turn trajectories, checking chronological tool order, API loop prevention, and exact parameter payload assertions (e.G., date ranges, seller regions).
Stream execution logs for baseline delta analysis and capability scoring.4.Enterprise Analytics, Governance & Security (Phase 4)Build low-latency Hydra ETL pipelines to stream structured evaluation JSON records into data warehouses and construct Plx analytics dashboards.Enforce automated PII masking/redaction layers for sensitive seller and financial data, while configuring retention and purging policies (e.G., 90-day trajectory logs).
Required Qualifications & Technical SkillsCore Language: Advanced proficiency in Python.Software & Systems Architecture: Deep expertise in enterprise software architectures, distributed computing, async task processing, and load balancing.AI/LLM Telemetry & Evaluations: Proven experience in LLM performance telemetry, prompt engineering, LLM-as-a-Judge systems, and statistical inter-rater agreement (IRR/Kappa).
Tooling & Infrastructure:Experience with Google ADK (or similar agent developer kits).
Model Context Protocol (MCP) logging architectures.RPC service design and integration.CI/CD & Testing: Hands-on experience integrating evaluation pipelines into CI/CD workflows and managing sandboxed unit testing environments.Data Engineering: Proficiency in building ETL pipelines (Hydra equivalent), structuring JSON telemetry schemas, and creating analytics dashboards (Plx/BI tools).
Data Privacy: Knowledge of automated PII/data-masking strategies and data retention protocols.
#J-*****-Ljbffr

📌 Ai/Ml Engineer(Generative Ai) (Jalisco)
🏢 Centraprise
📍 Jalisco