Senior Technical Lead

Job Summary

• Administer and operate enterprise Databricks workspaces across development, QA, production, and disaster recovery environments.
• Manage Databricks workspace configuration including clusters, jobs, pools, policies, secrets, and init scripts.
• Design, configure, and enforce cluster policies to ensure security, stability, and cost control.
• Manage Databricks compute including autoscaling, instance pools, job clusters, and all‑purpose clusters.
• Administer Databricks Jobs, Workflows, and scheduling for batch and streaming workloads.
• Configure and manage Databricks security including workspace access controls, IAM integration, permissions, and secrets management.
• Implement and manage Databricks Unity Catalog for data governance, access control, and lineage.
• Configure and support integration with cloud storage (Azure Data Lake, S3, GCS) and external data sources.
• Administer Delta Lake features including table management, versioning, optimization, and retention policies.
• Monitor platform health, performance, and availability using logs, metrics, and alerts.
• Troubleshoot Spark workloads including job failures, performance degradation, memory issues, and scaling problems.
• Support ML and AI workloads using Databricks ML Runtime, Model Registry, Feature Store, and notebooks.
• Implement CI/CD processes for Databricks notebooks, jobs, and configurations using Git-based workflows.
• Manage Databricks REST APIs, CLI tools, and automation scripts for platform operations.
• Perform capacity planning, usage analysis, and cost optimisation across Databricks environments.
• Apply platform patches, runtime upgrades, and configuration changes following change management best practices.
• Act as the escalation point for Databricks production incidents and lead root cause analysis.
• Maintain operational documentation, standards, and platform runbooks.
• Collaborate with data engineers, data scientists, MLOps, cloud, and security teams.
• Provide on‑call production support through a rotation schedule.

Key Responsibilities

Skill Requirements

• Databricks Administrator or Data Engineer certification.
• Experience supporting ML and AI workloads on Databricks.
• Experience with streaming technologies such as Spark Structured Streaming or Kafka.
• Familiarity with Infrastructure as Code tools such as Terraform for Databricks.
• Experience working in regulated or security‑sensitive environments.
• Familiarity with ITIL‑based enterprise service management processes.

Other Requirements