Lead Data Platform Engineer - Enterprise, Data & AI

In This Role, You Will...

• Scaling self-healing data pipelines that automatically handle schema evolution, detect anomalies, execute circuit breakers and recover from failures without manual intervention.
• Define and enforce company-wide data governance, access, automated data quality testing, schema evolution policies and metadata management to maintain high-fidelity data assets.

• Implement intelligent storage lifecycle strategies, resource throttling and query optimization to minimize compute overhead while delivering high-throughput, low-latency data access for analytics and AI agents. With a focus on Databricks vs EMR (AWS) cloud optimization (cost and performance).
• Build automated CI/CD deployment templates, environment isolation, testing frameworks and version control standards, enabling rapid, reliable and zero-downtime deployments

Qualifications

• 10+ years of hands-on experience in Data Platform Engineering or Software Engineering with a proven track record of architecting and scaling production-grade data foundations.
• Deep expertise in scaling and optimization, query tuning, compute resource allocation and implementing efficient compute-storage lifecycle policies to minimize infrastructure costs.

• Experience implementing enterprise security standards, Role-Based Access Control (RBAC), Active Directory/IAM roles and fine-grained data masking, with strong familiarity supporting enterprise compliance requirements (e.g., SOX, audit trail controls, data retention policies).
• Track record of establishing enterprise data governance frameworks, automated schema evolution controls, data quality audits and near real-time observability/alerting.
• Hands-on experience building automated CI/CD pipelines, environment isolation (branch testing, rollback mechanisms), version control and automated deployment testing for data assets.

Bonus Qualifications

• Experience incorporating LLMs or GenAI directly into pipeline operations (e.g., code generation, triage and root-cause analysis, automated data reconciliation, backfills or anomaly detection).
• Experience with infrastructure-as-code (Terraform) and containerization/orchestration (EKS) for elastic compute management. GPU/CPU performance optimization & utilization.

• Previous experience in the autonomous vehicle, robotics or high-tech manufacturing sectors.