Staff Software Engineer, Stream Compute

We're transforming the grocery industry At Instacart, we invite the world to share love through food because we believe everyone should have access to the food they love and more time to enjoy it together. Where others see a simple need for grocery delivery, we see exciting complexity and endless opportunity to serve the varied needs of our community. We work to deliver an essential service that customers rely on to get their groceries and household goods, while also offering safe and flexible earnings opportunities to Instacart Personal Shoppers. Instacart has become a lifeline for millions of people, and we’re building the team to help push our shopping cart forward. If you’re ready to do the best work of your life, come join our table. Instacart is a Flex First team There’s no one-size fits all approach to how we do our best work. Our employees have the flexibility to choose where they do their best work—whether it’s from home, an office, or your favorite coffee shop—while staying connected and building community through regular in-person events. Learn more about our flexible approach to where we work. OVERVIEW Instacarts Data Infrastructure organization builds and operates the systems that power our company’s data ecosystem, including a modern open data lakehouse on Apache Iceberg, a multi-engine compute platform for stream and analytical workloads, and self-serve tooling that helps Product, Data Science, ML, Ads, Finance, and engineering teams move fast with data. We’re looking for a Staff Software Engineer to join our Data Governance and Foundations Team. In this role, you’ll serve as a senior technical leader owning the architecture and delivery of our open lakehouse foundation, governance and access patterns, and multi-engine compute strategy—balancing today’s reliability with the next three to five years of scale, maturity, and cost efficiency. You’ll collaborate closely with engineering leadership and stakeholders across Data Science, ML Platform, Ads Infrastructure, Finance Engineering, Product Engineering, and Security. You’ll operate with a high degree of ownership in a fast-paced environment where architectural decisions have real technical and financial consequences. If you thrive on complex, high-scale challenges and roll-up-your-sleeves execution, this is a chance to shape the backbone of Instacart’s data platform. Our stack includes technologies such as Apache Iceberg, Apache Flink, Trino, ClickHouse, Apache Kafka, Apache Spark, Snowflake, Databricks, Confluent, Airflow, dbt, Delta Lake, Scala, Python, Postgres, and AWS. You’ll join a focused team of 7 engineers that values pragmatism, clarity, and impact. ABOUT THE JOB * Translate Instacart’s data strategy (e.g., monetization, federated access, real-time) into an actionable multi-year architecture roadmap; align with leadership while evolving the platform for scale, maturity, and cost efficiency. * Own the open lakehouse foundation: define and deliver unified table formats, storage governance, and a multi-engine compute portfolio (interactive, batch, streaming) that enables portability and prevents lock-in. * Drive real-time and streaming infrastructure for critical use cases (Ads, Fraud, ML): set deployment patterns, SLAs, and operational practices that balance performance, availability, and spend. * Pioneer AI-native data infrastructure engineering by applying LLM/AI tools to the platform lifecycle—accelerating development, automation, observability, and cost optimization—and partnering to embed AI-powered capabilities into the platform. * Elevate engineering excellence: lead architecture reviews, mentor senior/staff engineers, influence hiring, and clearly communicate complex trade-offs to both technical and executive audiences to ensure cross-org alignment. ABOUT YOU MINIMUM QUALIFICATIONS * 10+ years of software engineering experience building and operating data infrastructure or distributed systems at production scale. * Hands-on expertise with modern data lakehouse architectures and open table formats (e.g., Apache Iceberg, Delta Lake, Hudi) and with distributed query/compute engines (e.g., Trino, Spark, ClickHouse), including performance tuning and production reliability. * Experience with event-driven and streaming infrastructure (e.g., Kafka, Flink) for real-time pipelines and serving systems. * Proven ownership of major platform transitions or migrations (build vs. buy, migration design, risk management) delivered to production. * Ability to build cost/benefit and TCO models for infrastructure investments and to drive alignment via clear architecture docs and strategy memos across multiple teams and leadership levels. PREFERRED QUALIFICATIONS * Experience designing platform-level governance controls and familiarity with compliance frameworks (e.g., SOX, CPRA, GDPR). * FinOps experience optimizing data platform spend, including managing multi-million dollar infrastructure budgets and negotiating vendor contracts. * Deep SQL proficiency and strong skills in Python or Scala for systems-level development. * Experience with orchestration (e.g., Apache Airflow) and data transformation pipelines (e.g., dbt) in large-scale production environments. * Bachelor’s, Master’s, or PhD in Computer Science, Computer Engineering, Electrical Engineering, or equivalent practical experience. #LI-Remote   Instacart provides highly market-competitive compensation and benefits in each location where our employees work. This role is remote and the base pay range for a successful candidate is dependent on their permanent work location. Please review our Flex First remote work policy here. Offers may vary based on many factors, such as candidate experience and skills required for the role. Additionally, this role is eligible for a new hire equity grant as well as annual refresh grants. Please read more about our benefits offerings here. For US based candidates, the base pay ranges for a successful candidate are listed below. CA, NY, CT, NJ $265,000—$279,500 USD WA $254,000—$268,000 USD OR, DE, ME, MA, MD, NH, RI, VT, DC, PA, VA, CO, TX, IL, HI $243,000—$256,500 USD All other states $221,000—$233,000 USD

About Pinterest: Millions of people around the world come to our platform to find creative ideas, dream about new possibilities and plan for memories that will last a lifetime. At Pinterest, we’re on a mission to bring everyone the inspiration to create a life they love, and that starts with the people behind the product. Discover a career where you ignite innovation for millions, transform passion into growth opportunities, celebrate each other’s unique experiences and embrace the flexibility to do your best work. Creating a career you love? It’s Possible. At Pinterest, AI isn't just a feature, it's a powerful partner that augments our creativity and amplifies our impact, and we’re looking for candidates who are excited to be a part of that. To get a complete picture of your experience and abilities, we’ll explore your foundational skills and how you collaborate with AI. Through our interview process, what matters most is that you can always explain your approach, showing us not just what you know, but how you think. You can read more about our AI interview philosophy and how we use AI in our recruiting process here. We're seeking an exceptional Staff Software Engineer to join our Observability team at Pinterest. This role combines deep technical expertise in distributed systems and data engineering with a product-oriented mindset to build world-class observability solutions that empower our engineering organization. As a Staff Engineer on the Observability team, you'll be responsible for designing and building the infrastructure and tools that provide visibility into Pinterest's large-scale distributed systems, helping thousands of engineers understand, debug, and optimize their services.   What you'll do: * Define and execute the observability roadmap, treating it as a product. Understand engineering team needs and translate them into technical solutions with measurable impact * Architect, build, and scale distributed observability infrastructure (metrics, logs, traces) to handle massive volumes across Pinterest's distributed systems * Build high-performance data pipelines and storage for real-time and historical telemetry analysis at Pinterest scale * Champion Best Practices: Establish observability standards and patterns across the organization, making it easy for teams to instrument their services and gain actionable insights * Technical Leadership: Mentor engineers, lead architectural reviews, and influence technical decisions across teams to improve overall system reliability and performance * Cross-functional Collaboration: Partner with SRE, Infrastructure, Product Engineering, and other teams to understand pain points and deliver solutions that improve developer productivity and system reliability * Innovation: Stay current with observability trends and technologies, evaluating and adopting cutting-edge tools and techniques to keep Pinterest at the forefront   What we’re looking for: * Bachelor’s degree in Computer Science, Engineering, or a related field, or equivalent experience * Product Mindset: Demonstrated ability to work backwards from customer needs —understanding user needs, prioritizing features, measuring success, and iterating based on feedback. Experience building internal platforms or tools with strong adoption * Distributed Systems Expertise: 7+ years of experience designing and operating large-scale distributed systems with deep understanding of consistency, availability, scalability, and failure modes * Data Engineering Skills: Strong background in building data pipelines, working with time-series databases, columnar storage, stream processing (Kafka, Flink, etc.), and data modeling at scale * Observability Domain Knowledge: Hands-on experience with modern observability tools and practices including metrics, logging, tracing, and profiling. Familiarity with OpenTelemetry, Prometheus, Grafana, or similar technologies * Programming Proficiency: Expert-level coding skills in languages like Java, Python, Go, or Scala with ability to write production-quality code * Systems Thinking: Ability to see the big picture while managing complex technical details, balancing trade-offs between cost, performance, and reliability * Experience building observability platforms from the ground up or significantly scaling existing solutions * Familiarity with cloud-native architectures and technologies (Kubernetes, service mesh, etc.) * Track record of driving adoption of internal platforms through excellent documentation, UX, and developer advocacy * Experience with machine learning or anomaly detection applied to observability use cases * Strong communication skills with ability to influence stakeholders at all levels * Contributions to open-source observability projects, a plus   In-Office Requirement Statement:  * We let the type of work you do guide the collaboration style. That means we're not always working in an office, but we continue to gather for key moments of collaboration and connection. * This role will need to be in the office for in-person collaboration 1-2 times/quarter and therefore can be situated anywhere in the country.    Relocation Statement: * This position is not eligible for relocation assistance. Visit our PinFlex page to learn more about our working model.   #LI-REMOTE #LI-JT1 At Pinterest we believe the workplace should be equitable, inclusive, and inspiring for every employee. In an effort to provide greater transparency, we are sharing the base salary range for this position. The position is also eligible for equity. Final salary is based on a number of factors including location, travel, relevant prior experience, or particular skills and expertise. Information regarding the culture at Pinterest and benefits available for this position can be found here. US based applicants only $177,185—$364,795 USD Our Commitment to Inclusion: Pinterest is an equal opportunity employer and makes employment decisions on the basis of merit. We want to have the best qualified people in every job. All qualified applicants will receive consideration for employment without regard to race, color, ancestry, national origin, religion or religious creed, sex (including pregnancy, childbirth, or related medical conditions), sexual orientation, gender, gender identity, gender expression, age, marital status, status as a protected veteran, physical or mental disability, medical condition, genetic information or characteristics (or those of a family member) or any other consideration made unlawful by applicable federal, state or local laws. We also consider qualified applicants regardless of criminal histories, consistent with legal requirements. If you require a medical or religious accommodation during the job application process, please complete this form for support.   By submitting this application, I certify that all information submitted in my application and throughout the hiring process is true, accurate, and complete to the best of my knowledge. I understand that any false statement, omission, or misrepresentation may disqualify me from employment consideration or result in termination if discovered after hire.