Senior Social Media Scraping & Automation Engineer

Role Description

This is a remote position.

We are looking for a highly skilled Senior Social Media Scraping & Automation Engineer to build a scalable platform for collecting and processing publicly available data from social media platforms and other web sources. You will design reliable scraping infrastructure, browser automation, API integrations, and data pipelines capable of processing large volumes of data efficiently.

Key Responsibilities

• Design and develop scalable web scraping solutions for social media platforms and websites.
• Build browser automation using Playwright, Selenium, or Puppeteer.
• Integrate official platform APIs wherever available.
• Develop systems to collect publicly available information, including user/business profiles, posts, comments, public engagement metrics, hashtags, keywords, company information, and public contact information.
• Build scheduled data collection and monitoring systems.
• Design distributed scraping architecture capable of handling high-volume workloads.
• Implement proxy rotation, session management, cookie handling, browser fingerprinting basics, and rate-limiting strategies.
• Optimize scraper performance, reliability, and fault tolerance.
• Develop ETL pipelines to clean, validate, deduplicate, and enrich collected data.
• Deploy and maintain scraping services on cloud infrastructure.
• Monitor scraper health with logging, alerting, and automated recovery.
• Collaborate with product, data, and engineering teams.
• Maintain technical documentation and follow engineering best practices.

Qualifications

• Bachelor's degree in Computer Science, Engineering, or equivalent practical experience.
• 4-8+ years of experience in Python development and web automation.
• Strong problem-solving and system design skills.
• Excellent communication and documentation skills.

Requirements

• Programming: Python (Expert), JavaScript / TypeScript, SQL, Git
• Scraping & Automation: Playwright (Preferred), Selenium, Scrapy, BeautifulSoup, Requests / Asyncio & Multiprocessing
• Social Media APIs: X (Twitter) API, LinkedIn API, Meta Graph API (Facebook & Instagram), Reddit API, YouTube Data API, Telegram API, Discord API
• Web Technologies: HTML, CSS, JavaScript, GraphQL, AJAX, WebSockets, Chrome Developer Tools
• Databases: PostgreSQL, MySQL, MongoDB, Redis
• Cloud & Infrastructure: Docker, Kubernetes (Preferred), AWS, Azure, or GCP, Linux
• Data Engineering: Pandas, ETL Pipelines, Apache Airflow (Preferred), Kafka or RabbitMQ

Benefits

• Experience building distributed scraping infrastructure.
• Knowledge of proxy management, browser fingerprinting, and session handling.
• Experience with OCR and document processing.
• Experience integrating AI/LLMs into automation workflows.
• Experience building monitoring dashboards and alerting systems.