Overview
We are looking for a Principal Data Engineer to lead the architecture, development, and operation of scalable, reliable data platforms. This role requires deep expertise in ETL, data pipelines, data modeling, and distributed systems, with strong hands-on proficiency in Apache Spark, Python, and SQL.
As a technical leader, you will shape architecture across team boundaries, anticipate customer needs, and make decisions that balance immediate delivery with long-term scalability, reliability, security, and cost. You will own complex, ambiguous problems from design through production and guide others toward high engineering standards.
You will bring an AI-native approach to engineering—using and improving AI-assisted development practices to accelerate delivery while taking accountability for the quality, correctness, and responsible use of AI-generated work.
Microsoft’s mission is to empower every person and every organization on the planet to achieve more. As employees we come together with a growth mindset, innovate to empower others, and collaborate to realize our shared goals. Each day we build on our values of respect, integrity, and accountability to create a culture of inclusion where everyone can thrive at work and beyond.
Responsibilities
- Lead data-platform architecture: Design and evolve ETL workflows, data pipelines, and data models using Spark, Python, and SQL. The platform supports experimentation and scorecards.
- Drive hands-on, end-to-end implementation: Build data solutions using Spark or similar distributed systems, open table formats such as Delta Lake, and Parquet files; develop scalable, secure APIs and data access controls, including RBAC; provision resources through Infrastructure as Code; and deliver production-grade deployments through CI/CD with observability and operational runbooks.
- Think beyond immediate requirements: Own complex system-design decisions, evaluate alternatives, and account for cross-team dependencies, scalability, resiliency, disaster recovery, and cost.
- Make evidence-based decisions: Proactively gather facts, validate assumptions, test hypotheses, and use telemetry and investigation to resolve complex problems rather than relying on unverified assumptions.
- Advance AI-native engineering: Apply AI across design, coding, testing, and troubleshooting; validate AI-generated outputs and help the team adopt effective, responsible practices.
- Own production outcomes: Establish monitoring and operational readiness, drive incident resolution and root-cause analysis, and implement preventive improvements. Automate safe deployments, targeting zero-touch deployment where possible.
- Build data agents that investigate pipeline failures and performance using permitted logs, metrics, traces, lineage, and execution plans. Detect anomalies and regressions, gather evidence, and propose root causes, actionable fixes, and tuning.
- Evaluate and validate agents in production, measuring diagnosi
