Agent Evaluation & Operations Engineer
Pune · Staff/Principal
Applicants who checked fit first are 3.1× more likely to hear back
Your score for this role already exists
ASAI compared this JD against 41 signals - skills, seniority, domain, stack overlap etc. Add a resume and it unlocks in about 30 seconds.
Always free · no credit card · 1 tap with Google
What we know about this role
HIGH
Our tracked job listings for NCSi show strong recent activity.
First 72 hours
Window passed - posted 62d agoEarly applicants get seen before the pile builds.
Not a repost
The first time we've seen this listing - it hasn't been closed and reopened.
Skills required
26 listedYou almost certainly match several of these already. Unlock your skill map to see the matches, the gaps, and what to fix first.
Job description
| Category | Platforms / Frameworks / Tools |
|---|---|
| Programming & APIs | Python, FastAPI, REST APIs, Pydantic, pytest |
| Agent Orchestration | LangGraph, LangChain, OpenAI Agents SDK, Google ADK, Pydantic-AI |
| Agent Observability | OpenTelemetry, OpenInference, Langfuse, LangSmith |
| LLM Evaluation, Experiment Lifecycle | MLflow Evaluation, DeepEval, Ragas, promptfoo, custom pytest evaluations |
| Azure AI Stack | Azure OpenAI, Microsoft Foundry, Azure Monitor, Application Insights |
| Databricks | Unity Catalog, Databricks Workflows/Jobs, SQL Warehouses |
| CI/CD & Version Control | Git, Gitlab, automated testing pipelines |
| Production Monitoring | Grafana, Prometheus, Azure Monitor, Kibana, alerting platforms |
| Cloud & Containers | Docker, Kubernetes, AKS, Terraform/Bicep |
| Security & Identity | Microsoft Entra ID, managed identities, OAuth2, Key Vault, RBAC |
Department
AIDA
Open Positions
1
Skills Required
Python,Langchain,Langgraph,OpenAI,Azure Databricks
Role
| Category | Platforms / Frameworks / Tools |
|---|---|
| Programming & APIs | Python, FastAPI, REST APIs, Pydantic, pytest |
| Agent Orchestration | LangGraph, LangChain, OpenAI Agents SDK, Google ADK, Pydantic-AI |
| Agent Observability | OpenTelemetry, OpenInference, Langfuse, LangSmith |
| LLM Evaluation, Experiment Lifecycle | MLflow Evaluation, DeepEval, Ragas, promptfoo, custom pytest evaluations |
| Azure AI Stack | Azure OpenAI, Microsoft Foundry, Azure Monitor, Application Insights |
| Databricks | Unity Catalog, Databricks Workflows/Jobs, SQL Warehouses |
| CI/CD & Version Control | Git, Gitlab, automated testing pipelines |
| Production Monitoring | Grafana, Prometheus, Azure Monitor, Kibana, alerting platforms |
| Cloud & Containers | Docker, Kubernetes, AKS, Terraform/Bicep |
| Security & Identity | Microsoft Entra ID, managed identities, OAuth2, Key Vault, RBAC |
Years Of Exp
8 to 11 years
Designation
Agent Evaluation
tags
Python,Langchain,Langgraph,OpenAI,Azure Databricks
Tools to land this role
PremiumYour match score, skill map and tracker stay free.
Ground Reality
PremiumSpot the dealbreakers in this 239-word post before you apply
Skip jobs with hidden dealbreakers. See what they actually need, what's easy to miss, and how they describe the team.
Ground Reality on every job, plus Insider Path, practice interviews and resume tailoring.
Insider Path
PremiumFind someone who can refer you here
We help you find people there who worked or studied where you did, and draft your first message and follow-up.
Free · no signup
Get tomorrow's jobs before you have to search
Daily job drops, skill trends and free resources - posted straight to the group. Leave any time.
No spam. Just jobs and resources.
Why people use ASAI
Someone shared one job with you. ASAI keeps finding the rest.
Scored, not searched. Every role ranked against your actual profile.
Alerts as often as hourly. Reach new roles while the pile is still small.
Skill gaps, spelled out. See exactly which requirements you don't meet yet.
Verified jobs, only. Say no to ghost jobs. Your time deserves respect.
More AI Engineer roles in Pune
See all
Keep browsing