Overview
Are you a Senior Software Engineer who thrives in a cutting-edge environment where you get to work with some of the largest data volumes processing? Are you excited to craft the most reliable data processing pipelines with Spark and Flink to process PiB level data every day? Are you passionate about using artificial intelligence tools and practices across the software development lifecycle and data processing?
The Data and Insight team in Search Foundation organization leverages open-source technology to process our petabyte scale data of Bing and MSN logs and provide comprehensive insight into experiments. The team also supports the grounding business with Realtime signals and insight information and protects the Bing search through offline bot protection analysis and model training.
Microsoft’s mission is to empower every person and every organization on the planet to achieve more. As employees we come together with a growth mindset, innovate to empower others, and collaborate to realize our shared goals. Each day we build on our values of respect, integrity, and accountability to create a culture of inclusion where everyone can thrive at work and beyond.
Responsibilities
You are a seasonal leader in building scalable data analytics/insights platform with PiB level data processing every day. You have a growth mindset to learn the cut-edge technology and apply it in daily work to change the status quo. You dive deep and resolve the challenging technical problem. You insist on the highest standard for system performance and reliability. You will be working on Bing mission critical analytical data and support the grounding business; processing at big data scale and need for generating real-time insights. You will be working in a hybrid technology space that combines the best from Azure and open-source world. Other responsibilities may include:
- Design, build and own the optimal data processing architecture and systems for new data and analytics pipelines, dashboards and metrics leveraging Azure and open-source technologies.
- Build core datasets as well as scalable and fault-tolerant pipelines
- Build data anomaly detection, data quality checks, and optimize pipelines for ideal compute and storage systems while guaranteeing high quality and SLA datasets
- Understand the data compliance and data governance
- Collaborate with cross functional partners including data scientists and product managers to design and develop end-to-end data assets that help improve our products.
- Hand on at least one data process technology like Spark, Flink, and message queue technology like Kafka, EventHub
- Hand on data storage technology like HDFS, Azure blob etc
- Leverage AI technology in coding/testing/Data processing/data validation.
Qualifications
Required Qualifications:
- Bachelor's Degree in Computer Science or related technical field AND 4+ years technical enginee
