We run a business in the pet industry with products listed on eBay. We purchase products from suppliers as orders come in, but our eBay store isn't directly synced with supplier websites, which sometimes causes stock issues and order cancellations.
To solve this, we've set up an automation using n8n and AWS, which:
1.Scrapes supplier websites daily to check stock availability and updates our eBay store automatically.
2.Updates product prices based on market pricing.
The system is functional but frequently breaks, likely due to n8n handling too many automations or large data volumes. Some suppliers with large product ranges need stock checks every 6–12 hours. Overall, there are 5,000–8,000 products that need to be scraped daily.
We're looking for an experienced automation engineer who can:
•Diagnose why the current system is breaking.
•Suggest improvements or alternative solutions for reliability and scalability.
•Optimize the existing AWS/n8n setup or recommend a better approach, such as Apache Airflow, to handle high-volume scraping and updates more reliably.
•Ensure the automation for stock and price updates runs smoothly without frequent failures.
Skills required:
•Strong experience with n8n or similar workflow automation tools.
•AWS experience (Lambda, EC2, serverless architecture).
•Apache Airflow experience is a plus, especially for orchestrating large-scale workflows.
•Web scraping at scale and handling large datasets.
•Experience with eBay API integrations is a plus.
•Problem-solving skills to optimize existing systems and ensure stability.
Deliverables:
•Review of the current automation and identification of failure points.
•Recommendations to fix or improve the system, including feasibility of using Airflow if needed.
•Implementation or guidance for a more stable setup.
•Documentation for the updated workflow for future maintenance.
If you've handled large-scale e-commerce automation before and can help us ensure reliability in our stock and pricing updates, please get in touch with your approach and relevant experience.
Contract duration of less than 1 month.
Mandatory skills: Automation, Apache Airflow, n8n, AWS Amplify