About the Role

Join a small team to architect, build, and maintain the core of our high-scale web scraping engine, handling over 1 billion monthly requests. You'll directly influence our technology decisions while tackling complex, exciting challenges in browser automation, advanced proxy management, distributed computing, and scalable microservices architecture.

Key Responsibilities

Core Engine Development

• Architect, develop, and optimize our Python-based distributed scraping engine
• Design advanced browser automation capabilities (Playwright/CDP)
• Enhance proxy management, rotation strategies, and anonymity layers
• Implement robust and scalable distributed task processing systems
• Develop efficient HTTP middleware, controllers, and networking layers
• Continuously identify and implement performance optimizations

System Architecture

• Lead the design and implementation of scalable microservices architecture
• Ensure reliability and performance at high scale (1B+ requests/month)
• Develop robust error handling, fault tolerance, and recovery systems
• Create and maintain seamless API integrations

Quality Assurance

• Write comprehensive unit and integration tests to ensure reliability
• Implement automated CI/CD testing pipelines
• Conduct code reviews to maintain high-quality coding standards
• Debug complex systems and optimize performance bottlenecks