Senior Technical Lead

Job Summary

Job Summary

We are seeking a highly experienced, strategic, and hands-on Lead Data Engineer to join our enterprise data engineering team. With over 12 years of professional expertise, you will act as a bridge between complex business requirements and high-performance technical solutions.

In this role, you will architect resilient, cloud-agnostic Databricks and Lakehouse environments, design enterprise-grade streaming and batch ingestion pipelines, and implement modern architectures such as Data Mesh. You will also drive innovation by integrating Generative AI capabilities (including RAG frameworks and LLM fine-tuning) into modern data platforms, while establishing robust, enterprise-wide data governance and compliance.

Key Responsibilities

1. Architecture & Lakehouse Design

• Architect and optimize highly scalable, resilient cloud-agnostic Databricks Lakehouse environments to convert multi-terabyte batch and streaming data into actionable intelligence.
• Lead complex, large-scale database and process migrations from legacy estates (e.g., SQL Server, on-premise systems) to modern cloud environments (Azure, AWS, Snowflake, MS Fabric), improving query performance and reducing infrastructure overhead.
• Design and deploy modern semantic data models and high-performance BI reporting solutions (such as Power BI or Tableau) to enable executives with real-time KPI insights.

2. Data Pipeline & API Engineering (ETL/ELT)

• Design, build, and dynamically parameterize reusable ETL/ELT pipelines using platforms like Azure Data Factory (ADF), AWS Glue/Lambda/Step Functions, Snaplogic, Ab Initio, or Microsoft Fabric.
• Develop robust stream-processing pipelines utilizing Apache Kafka, Azure Event Hubs, or Amazon Event Bridge alongside REST APIs to build synchronous and event-driven data workflows.
• Implement automated data-loading accelerators (such as Databricks Auto Loader) to ingest varied, heterogeneous data structures (structured, semi-structured, APIs, and flat files) with minimal manual overhead.

3. Data Governance, Compliance & Security

• Devise and enforce unified data governance frameworks incorporating data quality (DQ) metrics, Attribute-Based Access Control (ABAC), data lineage, and compliance tracking.
• Ensure rigorous protection of PII and adherence to international regulatory standards (e.g., ISO, GDPR, HIPAA) using tools like Microsoft Purview or Informatica MDM/IDD.

4. Generative AI & Innovation

• Lead the adoption of emerging AI technologies by designing RAG (Retrieval-Augmented Generation) virtual assistants and integrating them natively into Databricks platforms.
• Collaborate on fine-tuning Large Language Models (LLMs) on domain-specific datasets to deliver high-quality, context-aware responses in real time.

5. Leadership & Project Delivery

• Provide technical leadership and mentorship to cross-functional, onshore-offshore engineering teams.
• Manage stakeholder relationships, collaborating with business users, functional analysts, and executive teams to scope and prioritize key deliverables.

Enforce agile practices (Scrum/Kanban) and DevOps standards (CI/CD pipelines via Azure DevOps/GitHub) to maintain delivery excellence

Skill Requirements

Required Technical Skills:

• Cloud & ETL/ELT: Deep expertise in Azure (ADF, Azure Synapse, ADLS), Azure Databricks, MS Fabric, AWS (S3, EMR, DMS, Glue, Lambda), Snowflake, Snaplogic, or SSIS.
• Big Data & Databases: Strong experience in Postgres, Oracle, SQL Server, Snowflake, Teradata, MongoDB, and Hadoop/Hive ecosystems.
• Programming & Scripting: Mastery in Python, PySpark, T-SQL, and Unix Shell Scripting. Experience in C# or Java is highly desirable.
• Streaming & Integration: Apache Kafka, Azure Event Hubs, REST API integration, and Change Data Capture (CDC) technologies (e.g., Qlik Replicate).
• AI/BI Tools: Power BI, Tableau, RAG implementations, and LLM orchestration.
• DevOps & Tools: Azure DevOps, GitHub, Jira, and SFTP automation.

Qualifications & Education:

• Education: Bachelor’s Degree in Engineering (B.E./B.Tech) in Computer Science, Information Technology, or a closely related field.
• Certifications (Highly Valued):
• Databricks Certified Data Engineer Associate / Professional
• Databricks Generative AI Fundamentals or Lakehouse Fundamentals
• Scrum Fundamentals Certified (SFC)
• Snaplogic or cloud provider certifications (Azure/AWS)

 

Other Requirements