Principal Site Reliability Engineer

Ready to be a Titan?

We're looking for a Principal Site Reliability Engineer to join our Site Reliability & Infrastructure Engineering team. We run entirely on the cloud, and this team owns production intelligence, reliability, and operational automation — moving from reactive incident response toward systems that observe, correlate, reason, and act. We're building the next generation of SRE: not dashboards and runbooks, but agents and automation that make our platform self-aware and self-healing at scale.

At Principal level, you won't just operate the platform — you'll define how we think about reliability across the entire engineering organization. You'll set the technical direction, establish the standards, and be the person other engineers turn to when the hardest infrastructure problems need to be solved.

We make a huge impact on thousands of companies in the U.S. and abroad by enabling them to be more efficient and effective at running their business. We have a cultural foundation built on diversity, inclusion, and innovation, and we want you and your ideas to thrive at ServiceTitan. Come join us.

What You'll Do

• Define the technical vision and long-term architecture for ServiceTitan's reliability and infrastructure platform — setting the standard for how we operate at scale.

• Lead the design and delivery of major platform initiatives: Kubernetes infrastructure, observability frameworks, SLO programs, and incident management systems.

• Partner with Engineering Managers, Staff Engineers, and product engineering leadership to review architecture and infrastructure decisions before they ship — and hold the bar on non-functional requirements across the org.

• Identify systemic reliability risks before they become incidents, and drive remediation across teams with org-wide impact.

• Define and operationalize SLIs, SLOs, and error budgets across ServiceTitan's platform — not just within the SRE team, but as a standard every engineering team adopts.

• Drive adoption of reliability and observability best practices across engineering — through documentation, design reviews, and direct partnership with product teams.

• Design and build AI-assisted operational systems — agents that correlate production telemetry, diagnose failure patterns, recommend or execute remediation, and participate safely in deployment decisions. Move routine investigation and triage from humans to automation, while preserving human judgment for high-risk changes.

• Partner with Infrastructure Engineering on deployment safety — progressive delivery, canary analysis, and AI-assisted promotion decisions that answer "is there enough production evidence to continue this deployment?" not just "did the deployment complete?"

• Mentor Staff and Senior SREs, raising the technical ceiling of the team through code reviews, architecture feedback, and direct pairing on complex problems.

• Contribute to the technical hiring bar — participate in system design interviews and help define what Principal-level looks like at ServiceTitan.
   

What You'll Bring

•  Kubernetes (must-have): deep, expert-level understanding of Kubernetes as a system — not just operations, but internals, failure modes, capacity planning, and large-scale cluster management.

• SRE principles: proven experience defining and operationalizing SLIs, SLOs, and error budgets across an engineering organization — able to speak to how you drove adoption, not just how you implemented them on your own team.

• Observability at scale: deep expertise with a modern observability stack (OpenTelemetry, Prometheus, Grafana, Datadog, or Elasticsearch) and the ability to design observability frameworks that work org-wide, not just per-service.

• Cloud engineering & networking: expert grounding in AWS, GCP, or Azure — including networking fundamentals, security, and cost optimization at scale.

• Distributed systems depth: able to reason through complex failure modes (retry storms, cascading failures, thundering herds, timeout propagation) and architect systems that degrade gracefully under pressure.

• Incident management: track record of leading major incident response, driving blameless postmortems, and implementing systemic fixes — not just resolving individual incidents.

• CI/CD and delivery safety: deep experience with CI/CD systems, progressive delivery, and the ability to improve pipeline reliability and safety at org scale.

• AI in operations: experience building or deploying AI-assisted operational capabilities in production — automated investigation, anomaly correlation, remediation agents, or deployment safety systems. We're not looking for hackathon prototypes; we want engineers who have shipped AI tooling that production systems depend on.

• 12+ years of relevant hands-on experience, with a clear track record of technical leadership on complex, cross-team infrastructure initiatives at product companies operating at scale.

About You

You've operated at Staff or Principal level before and know what it means to own a technical domain — not just execute within it. You're the person Engineering Managers and VPs call when a system needs to be rethought, not just fixed. You can set direction with incomplete information, make architectural trade-offs explicit, and build consensus across teams without needing organizational authority to do it.

You care about reliability the way a product engineer cares about user experience — it's not overhead, it's the product. You measure your success not by what you personally shipped, but by how much better the entire engineering organization runs because of the standards and systems you put in place.

You're drawn to the question of what operations looks like when AI agents handle the repetitive analysis and humans own the judgment calls — and you want to be the engineer who designs that boundary.

‌

Be Human With Us: 

Being human isn’t about checking every box on a list. It’s about the experiences we have, people we meet, and the perspectives we share. So, if you have the skills but are hesitant to apply because of your background, apply anyway. We need amazing people like you to help us challenge the conventional and think differently about the problems that we’re solving. We’re in this together. Come be human, with us. 

Use of AI Technology:

We use technology, including automated and AI-assisted tools, to support certain aspects of our recruitment process. These tools are designed to improve efficiency and enhance the candidate experience. AI tools are not used to make hiring decisions; all hiring decisions are made by our hiring teams.

At ServiceTitan, we celebrate individuality and uniqueness. We believe that the convergence of fresh perspectives and experiences from all walks of life is what makes our product and culture so great. We do not discriminate against employees based on race, color, religion, sex, national origin, gender identity or expression, age, disability, sexual orientation, or any other characteristic protected by applicable laws.