Senior Python Web Scraping Developer

Description:

Responsibilities:

• Develop and maintain efficient web scraping tools and scripts to collect structured and unstructured data from websites.
• Analyze and understand website structures to implement effective data extraction strategies.
• Handle large-scale scraping projects, ensuring scalability, speed, and accuracy.
• Identify and resolve challenges such as CAPTCHAs, Cloudflare, dynamic content, and IP blocking using advanced techniques.
• Work with linux environment and run scripts through scheduler
• Clean, transform, and store scraped data in structured formats or databases as required.
• Monitor and troubleshoot scraping scripts to ensure optimal performance.
• Stay updated on the latest web scraping technologies, tools, and trends.

Qualifications:

• Bachelor’s degree in Computer Science, IT, or related fields (or equivalent experience).
• Strong proficiency in Python and experience with web scraping libraries such as Scrapy, BeautifulSoup, and Selenium.
• Strong leadership skills with the ability to manage and mentor junior team members.
• Knowledge of HTML, CSS, and JavaScript for interacting with web content.
• Familiarity with handling APIs and data formats like JSON, XML, and CSV.
• Experience with database systems (SQL/NoSQL) for storing and managing scraped data.
• Experience with headless browsers (e.g., Puppeteer, Playwright).