Data Platform Engineer - II

Are you passionate about solving roadblocks and challenges faced by MSMEs in India?

MSMEs contribute significantly to India’s total GDP. Around 90% of India’s ~$1 trillion retail market is controlled by Micro, Small & Medium Enterprises (MSMEs). This means approximately $900 billion worth of commerce flows through nearly 60 million MSMEs across the country.

At Khatabook, we envision empowering MSMEs and helping them increase their incomes. We have built a product that brings efficiency to MSME operations by providing easy-to-use tools to manage receivables, inventory, and billing, creating transparency in cash flow. We have also enabled access to short-term working capital loans for select users demonstrating good credit behavior.

Our app has been downloaded over 100 million times, with 8 million+ monthly active users generating 220 million+ transactions with a transaction value of $18 billion.

Why work with us

People are our biggest asset! At Khatabook, every one of us is a dynamic superstar. We have carefully built an ecosystem that brings together incredibly talented people who can dream, collaborate, experiment, and break new ground. We are a strong team that looks out for each other.

Your role

We, at Khatabook, are looking to hire a Data Engineer II for our Data Engineering team.

As a Data Engineer II, you will build, maintain, and scale reliable, secure, and cost-efficient data infrastructure that enables data-driven decision-making across the organisation. You will work on production-grade data pipelines, cloud data warehouses, CDC systems, streaming platforms, and large-scale data processing.

You will play a key role in ensuring data reliability, quality, security, and performance while contributing to AI-ready data infrastructure that supports AI/ML and Generative AI use cases.

What would you do at Khatabook?

• Design, develop, and maintain scalable batch and streaming data pipelines using Python and Apache Airflow.

• Build reliable data ingestion workflows with appropriate error handling, retries, monitoring, and recovery mechanisms.

• Develop incremental data processing workflows and Change Data Capture (CDC) pipelines to ensure timely and consistent data availability.

• Build, maintain, and optimize cloud data warehouses, preferably Snowflake, with a focus on scalability, performance, and cost efficiency.

• Develop and maintain SQL-based data models and transformation workflows using tools such as DBT.

• Build event-driven data workflows using Kafka or other messaging and queuing technologies.

• Develop and optimize Apache Spark jobs for large-scale data processing.

• Build and operate data engineering workloads on AWS and work with Kubernetes and containerised environments where appropriate.

• Optimize SQL queries, data models, Spark jobs, warehouse utilisation, and pipeline execution to improve performance and reduce infrastructure costs.

• Implement data quality checks, validation frameworks, monitoring, alerting, and failure-recovery mechanisms to ensure data accuracy, freshness, and reliability.

• Implement data access controls, role-based access control (RBAC), secure data sharing, and appropriate security practices for sensitive data.

• Support data governance, metadata management, and data lineage to improve data discoverability and trust.

• Build and maintain data pipelines that prepare structured and unstructured data for AI/ML and Generative AI applications.

• Support AI-powered search and analytics use cases involving Large Language Models (LLMs), embeddings, vector databases, and Retrieval-Augmented Generation (RAG).

• Collaborate with engineering, product, and analytics teams to understand data requirements and deliver trusted, actionable data.

• Take ownership of production workloads, troubleshoot pipeline failures, investigate incidents, and continuously improve data platform reliability and operational efficiency.

• Explore AI-assisted development tools to improve Python and SQL development, debugging, and code quality while validating generated code for correctness and security.

• Participate in a shared production on-call rotation approximately once every three weeks to ensure the reliability of critical data workflows.

What are we looking for?

• You have 3–7 years of experience in Data Engineering or a related technical role.

• You have strong Python programming skills and advanced SQL knowledge, including complex joins, window functions, query optimisation, and performance tuning.

• You have hands-on experience building and maintaining production-grade data pipelines using Apache Airflow.

• You have experience working with modern data warehouses such as Snowflake or Amazon Redshift.

• You have experience with CDC pipelines, incremental data ingestion, or database replication.

• You have working knowledge of AWS cloud services and experience operating data engineering workloads.

• You have experience with Apache Spark or PySpark for large-scale data processing.

• You have experience with Kafka or other messaging and queuing technologies.

• You have a strong understanding of relational databases such as PostgreSQL or MySQL.

• You understand data modeling, data quality, pipeline reliability, monitoring, troubleshooting, and production support.

• You have strong analytical, problem-solving, and debugging skills.

• You understand distributed systems, data security, and performance optimisation.

• You have a strong sense of ownership and can independently troubleshoot and improve production systems.

• You can collaborate effectively with engineering, product, and analytics teams to deliver scalable data solutions.

Good to have:

• Strong knowledge of Snowflake security, RBAC, secure data sharing, and compute and storage cost optimisation.

• Familiarity with DBT for data transformation, testing, documentation, and data model management.

• Understanding of dimensional modeling and modern data warehouse architecture.

• Familiarity with Docker, Kubernetes, CI/CD, and automated testing.

• Experience with data observability, metadata management, and data lineage.

• Understanding of Generative AI concepts, including LLMs, embeddings, vector databases, and RAG.

• Exposure to AI-ready data pipelines for structured and unstructured data.

• Familiarity with AI-assisted development tools for Python and SQL.

• Exposure to AI application platforms such as AWS Bedrock or Snowflake Cortex.

• Familiarity with BI and analytics platforms such as Tableau, Redash, or similar tools.