Data Engineer (Web Scraping/Crawling)
HyderabadOn-siteMid Level
Applicants who checked fit first are 3.1× more likely to hear back
Before you apply
The post's words + ASAI's dataOn-site in Hyderabad.
OK with that?3-5 years asked. Data Engineer roles in Hyderabad usually ask 4-8; almost all ask 3+.
You have them?No pay range in the post.
Not a repost. None of OSI Digital's last 18 posts was; all 18 of its open roles are here.
69 Data Engineer roles posted in Hyderabad this week, 684 open. See them →
Skills listed in the post · tap the ones you have · 0 of 8
You almost certainly match several of these already.
This post sets 2 stated bars and lists 8 skills. Tap where you stand - nothing is saved, no account needed.
ASAI checks this on every job for you, automatically, so you apply where you fit.
Fit Breakdown
ExampleHow an ASAI match score looks. Yours is built from your resume, not a guess.
Your score for this role already exists - sign in to view it
ASAI compared this JD against 41 signals - skills, seniority, domain, stack overlap etc. Add a resume and it unlocks in about 30 seconds.
Always free · no credit card · 1 tap with Google
Tools on this job
60-Day PassStarts free · 60-Day Pass ₹149 for 60 days, GST included
Free account, no card needed.
The job post
251 words · from OSI Digital's careers siteJob Title: Web Crawling Engineer
Experience:3-5 Years
Location: Hyderabad
We are looking for a skilled Web Crawling Engineer with strong expertise in Python and hands-on experience building scalable crawlers and scrapers. The ideal candidate should have deep knowledge of crawling frameworks, anti-blocking techniques, and proxy management, along with experience solving challenges like captchas and rate-limiting.
Key Responsibilities
- Design, develop, and maintain scalable and efficient web crawlers for data extraction.
- Work with Scrapy, Requests, and other Python libraries to implement crawling solutions.
- Handle anti-scraping measures, including IP blocks, rate limits, and captchas.
- Implement and manage proxy rotation and session management strategies.
- Ensure high-quality and structured data extraction, cleaning, and storage.
- Monitor crawler performance, troubleshoot issues, and optimize for reliability and efficiency.
- Collaborate with the data engineering team to integrate crawled data into pipelines.
- Stay updated with the latest tools, techniques, and best practices in web crawling.
Required Skills & Qualifications
- Strong programming skills in Python.
- Proven experience with Scrapy, Requests, BeautifulSoup, Selenium (if required).
- Hands-on experience solving blocking issues, handling
- Good understanding of proxy usage, rotation, and fingerprinting avoidance techniques.
- Knowledge of HTTP, cookies, headers, sessions, and request/response cycles.
- Experience with databases (SQL/NoSQL) for storing crawled data.
- Strong debugging, problem-solving, and analytical skills.
Good to Have
- Experience with cloud environments (AWS, GCP, Azure).
- Knowledge of data pipelines and ETL processes.
Education & Experience
- Bachelor’s degree in Computer Science, Information Technology, or equivalent practical experience.
- 3+ years of experience in web crawling or related fields
Free · no signup
Get tomorrow's jobs before you have to search
