T&T - Engineering - Consultant/Senior Consultant | Databricks |Coimbatore

The Team

Deloitte’s Technology & Transformation: EAD: Engineering practice can help you uncover and unlock the value buried deep inside vast amounts of data. Our global network provides strategic guidance and implementation services to help companies manage data from disparate sources and convert it into accurate, actionable information that can support fact-driven decision-making and generate an insight-driven advantage. Our practice addresses the continuum of opportunities in business intelligence & visualization, data management, performance management and next-generation analytics and technologies, including big data, cloud, cognitive and machine learning.

Job Title: Senior Databricks Engineer

Experience: 4-9 Years

Location: Coimbatore

Job Summary

We are seeking an experienced Databricks Developer with 6-9 years of expertise in designing, building, and supporting modern Data Lakehouse solutions. The ideal candidate will have strong hands-on experience in Databricks, PySpark, Delta Lake, cloud platforms, and large-scale data processing. The role involves building scalable data pipelines, optimizing distributed data workloads, and enabling advanced analytics and AI-driven initiatives.

Key Responsibilities

• Design, develop, and maintain scalable data engineering solutions using Databricks.

• Build and optimize batch and real-time data processing pipelines.

• Develop ETL/ELT frameworks using PySpark and Spark SQL.

• Implement Delta Lake architecture and data governance standards.

• Process large-scale structured, semi-structured, and streaming datasets.

• Optimize Spark jobs and cluster performance for maximum efficiency.

• Develop reusable frameworks, notebooks, workflows, and data products.

• Collaborate with business teams, data scientists, analysts, and architects to deliver enterprise-grade solutions.

• Implement monitoring, alerting, security, and compliance requirements.

• Participate in code reviews, deployment activities, and production support.

• Develop technical documentation and mentor junior engineers.

Required Technical Skills

Databricks & Big Data

• Azure Databricks

• Databricks Lakehouse Platform

• Delta Lake

• Spark SQL

• PySpark

• Databricks Workflows

• Unity Catalog

• Delta Live Tables (DLT)

• Structured Streaming

Data Engineering

• Data Lake Architecture

• Lakehouse Architecture

• ETL/ELT Development

• Data Modeling

• Data Integration

• Performance Tuning

• Data Quality Frameworks

Programming Languages

• Python

• PySpark

• SQL

• Scala

• Java

• Shell Scripting

Cloud Platforms

• Microsoft Azure

• Azure Data Factory (ADF)

• Azure Data Lake Storage (ADLS)

• AWS

• Amazon S3

• AWS EMR

• Google Cloud Platform (GCP)

Databases

• SQL Server

• Oracle

• PostgreSQL

• MySQL

• MongoDB

• Cassandra

Streaming & Messaging

• Kafka

• Event Hubs

• Spark Streaming

• Azure Stream Analytics

DevOps & Tools

• Git

• Jenkins

• Azure DevOps

• CI/CD Pipelines

• Terraform

• Unix/Linux

Required Qualifications

• Bachelor's or Master's degree in Computer Science, Engineering, Information Technology, or related field.
• 4-9 years of IT experience with at least 4+ years of hands-on Databricks and Spark development experience.
• Strong expertise in distributed data processing and Lakehouse architecture.
• Experience building enterprise-scale data engineering solutions.
• Strong knowledge of PySpark, Spark SQL, and performance optimization.
• Experience with Agile development methodologies.
• Excellent analytical, troubleshooting, and communication skills.

Preferred Qualifications

• Experience with cloud-native analytics platforms on Azure, AWS, or GCP.
• Knowledge of Machine Learning workflows within Databricks.
• Experience implementing Delta Lake and Unity Catalog.
• Exposure to Data Governance and Data Security frameworks.
• Relevant Databricks, Azure, or AWS certifications.

Key Competencies

• Data Engineering Excellence

• Performance Optimization

• Solution Design

• Stakeholder Management

• Collaboration and Leadership

• Ownership and Accountability

Nice to Have

• Snowflake

• Azure Data Factory (ADF)

• Delta Live Tables (DLT)

• Unity Catalog

• MLflow

• Airflow

• Kafka

• Data Mesh Architecture

• Generative AI / AI Engineering exposure

• dbt