Job Description: Data Quality Engineer (Data QE)

About Anblicks

Role Overview

Key Responsibilities

• Design and execute data quality testing strategies for end-to-end data pipelines.
• Perform ETL/ELT testing, source-to-target validation, data reconciliation, and business rule verification.
• Develop advanced SQL queries and Python scripts for data validation and test automation.
• Validate Databricks workflows, Delta tables, and Bronze, Silver, and Gold data layers.
• Test incremental loads, CDC, schema changes, data integrity, and failure-recovery scenarios.
• Build reusable, configuration-driven data quality automation frameworks.
• Integrate automated tests into CI/CD pipelines and workflow orchestration tools.
• Investigate data discrepancies, perform root cause analysis, and collaborate with data engineering teams.
• Leverage AI-assisted tools to improve test case generation, SQL development, and failure analysis.

Required Technical Skills

• Strong SQL skills, including complex joins, CTEs, window functions, and aggregations.
• Proficiency in Python for data validation, reconciliation, and automation.
• Exposure to PySpark for large-scale data processing and validation.

• Strong understanding of ETL/ELT testing, data warehousing, data modelling, and data profiling.
• Experience with source-to-target validation, business rules, and data integrity checks.
• Knowledge of batch/streaming pipelines and formats such as JSON, CSV, and Parquet.

• Hands-on experience with Databricks notebooks, Spark SQL, jobs, and workflows.
• Knowledge of Delta Lake, schema evolution, and Medallion Architecture.
• Exposure to Snowflake, Unity Catalog, and cloud data platforms is desirable.

• Experience building reusable SQL/Python-based test automation frameworks.
• Knowledge of PyTest or similar testing frameworks.
• Experience with Git, Jenkins, Azure DevOps, or equivalent CI/CD tools.
• Familiarity with Databricks Workflows or Apache Airflow.

AI & Emerging Technologies

• Understanding of Generative AI, LLMs, and AI-assisted testing.
• Exposure to RAG pipeline validation, including document quality, metadata, retrieval relevance, and response accuracy.
• Familiarity with LLM evaluation, AI agents, and responsible AI practices is an advantage.

Good to Have

• Experience with Great Expectations, Soda, or Deequ.
• Knowledge of Kafka, data observability, and anomaly detection.
• Exposure to AWS, Azure, GCP, RAG evaluation tools, and data governance.