14 nov
|
VeeRteq Solutions
|
México
14 nov
VeeRteq Solutions
México
Role: Data Engineer (Kafka) Location: Mexico(Remote) Experience: 8+ years Job Description We are seeking a Lead Kafka Integration Engineer to design, develop, and support enterprise-scale data integration solutions leveraging Kafka, Databricks, and SQL Server-based MDM platforms.
The role will be responsible for architecting scalable and resilient real-time data ingestion pipelines, implementing Kafka consumer services, and delivering data through Databricks Bronze, Silver, and Gold layers.
Design and implement end-to-end data ingestion and processing solutions from Kafka into Databricks Medallion Architecture (Bronze/Silver/Gold) and SQL Server-based MDM platforms, ensuring scalability, resiliency, and high-performance data delivery.
Architect, develop, and support enterprise-grade Kafka consumer services including consumer groups, offset management, schema validation, retry mechanisms, dead-letter queues (DLQ), monitoring, ing, security, and automated recovery capabilities.
Establish DevOps and operational frameworks for Kafka and Databricks using CI/CD pipelines, Infrastructure as Code (IaC), environment automation, deployment standards, observability, logging, capacity planning, and production support.
Build and optimize Kafka-to-SQL Server ingestion pipelines, data models, stored procedures, reconciliation processes, and XML backfeed services supporting master data management, downstream integrations, and data quality controls.
Lead architecture, development, troubleshooting, performance tuning, and governance across Kafka, Databricks, SQL Server, and cloud-native data platforms while ensuring reliability, maintainability, security, and operational excellence.
Batch & Streaming Pipelines: Design, develop, and optimize scalable ETL/ELT workflows using Azure Databricks, Azure Data Factory, and Informatica,
alongside real-time data ingestion pipelines using Apache Kafka.
Master Data Management (MDM): Implement and maintain MDM workflows, data models, and match/merge logic to establish a reliable single source of truth across enterprise systems.
Azure Lakehouse Management: Manage and optimize data storage, partitioning, and transformation layers within ADLS Gen2 and Delta Lake to ensure high performance and cost efficiency.
SQL & Performance Tuning: Write complex SQL queries, perform deep data analysis, and troubleshoot bottlenecks across streaming and batch workloads.
Mandatory Skills: Strong hands-on experience with Apache Kafka (Consumers, Consumer Groups, Offset Management, DLQ, Schema Registry, Security)
Expertise in Databricks, Spark, Delta Lake, and Medallion Architecture (Bronze/Silver/Gold)
Experience building real-time and batch data ingestion pipelines from Kafka to Databricks and SQL Server
Strong knowledge of SQL Server, Stored Procedures, Data Modeling, MDM, and Data Reconciliation processes
Experience with CI/CD pipelines, DevOps practices, Infrastructure as Code (Terraform/Bicep), and environment automation
Expertise in monitoring, ing, observability, performance tuning, and production support
Experience with cloud platforms such as Azure and related data services
Experience with Master Data Management (MDM) platforms and healthcare data environments
Knowledge of XML processing, backfeed services, and downstream system integrations
Experience with Kafka Connect,
Event-Driven Architecture, and Streaming Analytics Azure Data Ecosystem: Hands-on experience building and managing cloud data solutions using Azure services such as Azure Data Lake Storage (ADLS Gen2), Azure Data Factory (ADF), and Azure Synapse Analytics.
Databricks & Spark: Strong proficiency in Azure Databricks using PySpark, Spark SQL, and Delta Lake for scalable data processing and lakehouse architecture.
Real-Time Streaming (Kafka): Practical experience designing, building, and troubleshooting real-time event streaming pipelines using Apache Kafka (or Azure Event Hubs for Kafka).
Informatica & MDM: Proven experience with Informatica (IICS / PowerCenter) and Master Data Management (MDM) concepts, including data modeling, match/merge rules, and golden record creation.
Advanced SQL: Expert-level SQL skills for complex querying, data transformation, profiling, and performance tuning across relational and cloud data warehouses.
'
Desired Skills Exposure to containerization technologies such as Docker and Kubernetes
Familiarity with data governance, data quality frameworks, and security best practices
Strong leadership, stakeholder management, and mentoring capabilities
DevOps & Orchestration: Experience with CI/CD pipelines (Azure DevOps, GitHub Actions), Infrastructure as Code (Terraform/ARM), and workflow orchestration tools.
Programming & Scripting: Proficiency in Python or Scala beyond PySpark for automation, API integrations, and custom data processing.
Data Governance & Security: Familiarity with Unity Catalog, Microsoft Purview, RBAC security models, and data privacy compliance standards.
Certifications: Microsoft Certified: Azure Data Engineer Associate, Databricks Certified Data Engineer, or Informatica certifications.
'
📌 Data Engineer (Kafka) (México)
🏢 VeeRteq Solutions
📍 México