02 ago
|
DataArt
|
Monterrey
Client: Our client is initiating a Proof of Concept (PoC) to evaluate the suitability of Google Cloud BigQuery as a unified enterprise data warehouse. The goal is to establish a modern, scalable data platform enabling advanced analytics, reporting, and real‑time insights.
- Position overview: We are seeking an experienced Senior Data Engineer to design, build, and operate end‑to‑end data pipelines and lakehouse‑style architecture on Google Cloud. As a Senior Data Engineer, you’ll be responsible for building a scalable, modern data platform on Google Cloud Platform (GCP). Your responsibilities include:
- Real‑time & batch data ingestion
- CDC replication
- Bronze/Silver/Gold layered modeling
- Data orchestration
- Terraform‑based infrastructure automation Downstream export pipelines (BigQuery
- OpenSearch)
- You will collaborate with architects and data teams to create a robust data foundation.
Technology stack: BigQuery, DataStream (CDC), DataFlow, Dataform, Cloud Composer (Airflow), OpenSearch, GCS, Terraform, advanced SQL, lakehouse modeling (Bronze / Silver / Gold)
- Responsibilities: Design efficient and scalable BigQuery schemas aligned with analytical and business requirements.
- Implement Bronze, Silver, Gold data layers following modern lakehouse best practices.
- Optimize for performance, cost efficiency, and scalability.
- Develop and maintain CDC pipelines using Google Cloud DataStream.
- Validate CDC accuracy including:
data integrity, replication completeness, replication latency.
- Support data backup, restore, and migration processes into the Bronze layer.
- Build end‑to‑end transformation pipelines with Dataform.
- Orchestrate workflows using Cloud Composer (Airflow).
- Implement data quality checks, monitoring, and validation across all layers.
- Lead testing and UAT of all BigQuery datasets, including: replication time validation, query performance and optimization, join correctness and consistency checks.
- Build and operate pipelines for exporting BigQuery data into OpenSearch using: cloud Composer for orchestration, DataFlow for scalable data processing.
- 5+ years in Data Engineering or similar roles.
- Strong hands‑on experience with GCP, including: BigQuery, DataFlow, DataStream, Cloud Composer (Airflow), Cloud Storage (GCS)
- Strong SQL and data modeling expertise.
- Production‑grade CDC implementation experience.
- Experience with Terraform or similar IaC tools.
- Solid understanding of data warehousing, ETL/ELT patterns, and lakehouse architecture.
- Nice to have: Experience integrating OpenSearch / Elasticsearch pipelines.
- Practical knowledge of Dataform or other SQL‑based transformation frameworks.
- Experience with hybrid real-time + batch architectures.
- Understanding of data governance, access control, and security best practices.
📌 Senior Data Engineer (Gcp) (Monterrey)
🏢 DataArt
📍 Monterrey