RE

Relanto

Lead Data Engineer

Bengaluru · Senior

Recently posted - highest visibilityVerified listingPosted 2h ago

Applicants who checked fit first are 3.1× more likely to hear back

What we know about this role

Hiring pulse

MEDIUM

Relanto is reviewing applications at a steady pace. Expect a standard response time as they evaluate the current pool.

Apply window

First 72 hours

Still inside it - posted 2h ago

Early applicants get seen before the pile builds.

Not a repost

The first time we've seen this listing - it hasn't been closed and reopened.

Skills required

28 listed

CI/CDData StreamingApache FlinkApache KafkaApache AirflowResource UtilizationQuery PerformanceData Interfaces+20 more

You almost certainly match several of these already. Unlock your skill map to see the matches, the gaps, and what to fix first.

Job description

Role: Lead Data Engineer

Experience: 6–8 years

Location: Bengaluru / Hybrid

Employment: Full-time

Role Summary

We are looking for a Lead Data Engineer with strong hands-on experience in real-time data streaming, event processing, CDC, data integration, and modern data engineering.

The Lead Data Engineer will be responsible for designing, developing, and maintaining highperformance data pipelines using Apache Flink, Apache Kafka, Debezium, CDC, ClickHouse, and Apache Airflow.

This is a hands-on engineering role requiring strong practical experience in building productiongrade streaming applications, developing data pipelines, troubleshooting distributed systems, and optimizing data processing workloads.

The ideal candidate should be comfortable working across the full data pipeline—from source systems and CDC ingestion through Kafka and Flink processing to analytical storage, APIs, dashboards, and downstream integrations.

Key Responsibilities

Real-Time Data Engineering
Design, develop, and maintain real-time data pipelines using Apache Flink and Apache Kafka.
Develop production-grade streaming applications for high-volume and low-latency workloads.
Implement data transformation, filtering, enrichment, aggregation, and event processing.
Build reliable event-processing pipelines with appropriate error handling and recovery mechanisms.
Consume and publish events across Kafka topics. • Implement appropriate partitioning, consumer groups, offsets, and delivery mechanisms.
Troubleshoot streaming pipeline failures and performance issues.

Apache Flink

Develop and maintain Apache Flink jobs forreal-time data processing.
Implement:
- Stream transformations
- Filtering
- Mapping
- Aggregations
- Joins
- Windows
- Event-time processing
- Watermarks
- State management
Implement Flink checkpointing and recovery mechanisms.
Optimize Flink jobs for performance, scalability, and resource utilization.
Monitor Flink jobs for latency, throughput, failures, and resource consumption.
Troubleshoot state, checkpointing, backpressure, and processing issues.

Kafka

Develop Kafka-based ingestion and streaming pipelines.
Create and manage Kafka topics and event streams.
Workwith partitions, offsets,consumer groups, replication, and retention.
Implement reliableproducer and consumer applications.
Handle message ordering, retries,duplicate events, and replay scenarios.
Monitor Kafka performance and troubleshoot consumerlag and throughput issues.
Work with Kafka schemasand serialization formats.

CDC & Debezium

BuildCDC-based ingestion pipelines using Debezium.
Configure and maintain Debeziumconnectors.
Capture source-system inserts, updates,and deletes.
Publish CDC events into Kafka.
Handle initial snapshots and incremental CDC processing.
Manage schemaevolution and changesin source systems.
Implement data reconciliation and consistency checks.
Troubleshoot CDC failures and source-to-target data issues.

Data Orchestration

Develop and maintain data workflows using Apache Airflow orequivalent orchestration frameworks.
Buildreusable DAGs for:
- Data ingestion
- CDC workflows
- Data validation
- Flink job execution
- Data transformation
- ClickHouse loading
- Downstream integrations
Implement workflowdependencies, scheduling, retries,backfills, SLAs, and alerting.
Integrate Airflow workflows with Kafka, Flink, Debezium, ClickHouse, APIs, and cloud services.
Monitor workflow execution and troubleshoot failures.
Develop reusable operators, sensors,and workflow components where appropriate.
Useevent-driven triggers where real-time workflows require them.

ClickHouse & Analytical Data

Integrate streaming data pipelines with ClickHouse.
Design efficient analytical data models.
Develop and optimize SQL queries.
Implement appropriate partitioning, sorting, indexing,and retention strategies.
Optimize data ingestion and query performance.
Support downstream analytical use cases,dashboards, and reporting requirements.

Data Quality& Reliability

Implement data validation and quality checksthroughout the pipeline.
Buildreconciliation mechanisms betweensource and target systems.
Monitor data freshness, completeness, accuracy, and consistency.
Implement error handling, retry,replay, and recoverymechanisms.
Establish loggingand observability for critical pipelines.
Support incident investigation and root-cause analysis.

Integration & APIs

Integrate streaming and analytical data with APIs, endpoints, dashboards, and downstream applications.
Develop data interfaces and integration components.
Workwith application teams to define data contractsand integration requirements.
Support future integrations and additional data consumers.

Engineering Practices

Follow modernsoftware engineering practices around:
- Git
- Code reviews
- Unit testing
- Integration testing
- CI/CD
- Logging
- Monitoring
- Documentation
Develop reusable and maintainable data engineering components.
Participate in technical designdiscussions and architecture reviews.

Mentor other Data Engineers and contribute to engineering standards.

Required Skills& Experience

6–8 years ofexperience in Data Engineering.
Strong hands-onexperience with Apache Flink — mandatory/core requirement.
Strong hands-on experience with Apache Kafka.
Hands-on experience with Debeziumand CDC.
Strong programming experience in Java or Scala.
Python experience is an advantage.
Strong SQL skills.
Experience with analytical databases; ClickHouse is highly preferred.
Hands-on experience with Apache Airflowor another data orchestration framework.
Strong understanding of distributed systemsand real-time data processing.
Experience developing and supporting production-grade streaming pipelines.
Strong understanding of Kafka:
- Topics
- Partitions
- Offsets
- Consumer groups
- Replication
- Retention
Experience with data transformation, enrichment, filtering, aggregation, and event processing.
Experience troubleshooting performance and reliability issues.
Familiarity with cloud platforms and containerized environments.

Preferred Skills

Advanced experience with Apache Flink, including:
- State management
- Checkpoints
- Savepoints
- Watermarks
- Event time
- Windows
- Backpressure
- State backends
Experience with Apache Airflow, Dagster,Prefect, or Apache NiFi.
Experience with Kafka Schema Registry.
Experience with Avro, Protobuf, or JSON.
Experience with Kubernetes.
Experience with AWS / Azure / GCP.
Experience with CI/CD pipelines.
Experience with Docker and containerized applications.
Experience with Terraform or other Infrastructure as Code tools.
Experience with data observability and monitoring tools.
Experience buildinghigh-volume, low-latency real-time data platforms.
Experience with REST APIs and systemintegrations.

More Data Engineer roles in Bengaluru

See all

Data Architect

Relanto · Bengaluru

Data & Analytics Engineer - Consultant

KPMG India · Bengaluru

Staff Data Engineer

Jll · Bengaluru

Data & Analytics Engineer - AM