Parallelizing web scraping to improve performance and scalability

• Main Author: Na, Yi Chun
• Format: Final Year Project / Dissertation / Thesis
• Published: 2023
• Subjects: T Technology (General); TD Environmental technology. Sanitary engineering
• Online Access: http://eprints.utar.edu.my/6039/1/fyp_CS_2023_NYC.pdf; http://eprints.utar.edu.my/6039/

Web scraping is the process of extracting data from websites, usually for analysis or other purposes. Web scraping is increasingly important in e-commerce because it enables businesses to extract valuable data from competitors' websites, especially product prices. This data can be used to gain insights into market trends, optimize pricing strategies, and improve product offerings. However, web scraping can be a challenging task due to some reasons such as the sheer volume of data available, network bandwidth, etc., when monitoring product prices across numerous e-commerce platforms. The main purpose of this project is to help whoever is suffering from the long waiting time to scrape desired information much faster than usual regardless of whether the data is small-scale or large-scale from the internet through the integration of web scraping technologies and distributed computer system. The proposed solution requires user interaction to be configured and initialized. It employs a message queuing algorithm to divide scraping tasks into smaller units and utilizes multiple worker nodes for concurrent web data extraction. In the scraping process, Selenium Web Driver interacts with specified web elements based on user-defined selectors or XPATH, allowing asynchronous HTTP requests and responses. Performance metrics such as response times, bandwidth data, and task status will be monitored for benchmarking and error handling. After finishing the scraping process, the scraped results are stored in a CSV file for further analysis.