Tech Lead - Databricks + AWS Job

Experience

10+ Years

Job Description

Key Responsibilities

• Note: Resources should be familiar with Iceberg Tables.
• Design, develop, and maintain scalable data pipelines using AWS services.
• Build and optimize ETL/ELT workflows for structured and unstructured data.
• Implement data ingestion solutions using AWS Databricks.
• Develop and manage data lakes and data warehouse solutions on AWS.
• Create and maintain data models, transformations, and processing frameworks.
• Implement data governance, data quality, security, and compliance controls.
• Monitor and optimize data platform performance, scalability, and cost.
• Develop reusable frameworks and automation for data orchestration and deployment.
• Troubleshoot data-related issues and provide production support.

Data Engineering and Data Builder :
•    Build Bronze, Silver and Gold data assets 
•    Develop reusable enterprise data products 
•    Create pipelines, transformations and data integrations 
•    Implement CI/CD, monitoring and operational support of use cases through MVP 
•    Operationalize MVP solutions into production-ready assets 

Required Technical/ Functional Competencies

Domain/ Industry Knowledge:

• Basic knowledge of customer's business processes- relevant technology platform or product.
• Able to prepare process maps, workflows, business cases and simple business models in line with customer requirements with assistance from SME and apply industry standards/ practices in implementation with guidance from experienced team members.

Requirement Gathering and Analysis:

• Working knowledge of requirement management processes and requirement analysis processes, tools & methodologies.
• Able to analyse the impact of change requested/ enhancement/ defect fix and identify dependencies or interrelationships among requirements & transition requirements for engagement.

Product/ Technology Knowledge:

• Working knowledge of technology product/platform standards and specifications.
• Able to implement code or configure/customize products and provide inputs in design and architecture adhering to industry standards/ practices in implementation.
• Analyze various frameworks/tools, review the code and provide feedback on improvement opportunities.

Architecture tools and frameworks:

• Working knowledge of architecture Industry tools & frameworks.
• Able to identify pros/ cons of available tools & frameworks in market and use those as per Customer requirement and explore new tools/ framework for implementation.

Architecture concepts and principles:

• Working knowledge of architectural elements, SDLC, methodologies.
• Able to provides architectural design/ documentation at an application or function capability level and implement architectural patterns in solution & engagements and communicates architecture direction to the business.

Analytics Solution Design:

• Knowledge of statistical & machine learning techniques like classification, linear regression modelling, clustering & decision trees.
• Able to identify the cause of errors and their potential solutions.

Tools & Platform Knowledge:

• Familiar with wide range of mainstream commercial & open-source data science/analytics software tools, their constraints, advantages, disadvantages, and areas of application.

Required Behavioral Competencies

Accountability:

• Takes responsibility for and ensures accuracy of own work, as well as the work and deadlines of the team.

Collaboration:

• Shares information within team, participates in team activities, asks questions to understand other points of view.

Agility:

• Demonstrates readiness for change, asking questions and determining how changes could impact own work.

Customer Focus:

• Identifies trends and patterns emerging from customer preferences and works towards customizing/ refining existing services to exceed customer needs and expectations.

Communication:

• Targets communications for the appropriate audience, clearly articulating and presenting his/her position or decision.

Drives Results:

• Sets realistic stretch goals for self & others to achieve and exceed defined goals/targets.

Resolves Conflict:

• Displays sensitivity in interactions and strives to understand others’ views and concerns.

Certifications