As the retail industry increasingly shifts online, the growing number of online platforms, including e-commerce sites, second-hand trading boards, retailer-owned sites, and online catalogues, has made it challenging for consumers to compare prices across all options and find the best deals. In response, Indium Software developed a solution that provides real-time visibility of products and dynamically changing prices across multiple sites. The solution utilizes an efficient data handling system that processes real-time data requests and responses with minimal delay. Additionally, it effectively crawls numerous e-commerce sites to extract and present the most relevant information in the correct taxonomy.

The client operates an internet-based price comparison website, providing consumers with quick comparison information for e-commerce products to help them find the best deals. Their platform aggregates data from over 1500+ retailer websites, enabling users to have real-time visibility of products and their dynamically changing prices. To meet this challenge, the client required an efficient data handling system capable of processing real-time data requests and responses.

• Categorizing and redefining the gathered data from multiple eCommerce sources into a standardized format before storing it, to address the product taxonomy challenge.
• Achieving a response time of 1 second or less for processing data refresh requests, in order to meet the end user’s expectation of receiving price updates within 2 seconds.
• Addressing the issue of huge data storage loads resulting from collecting data from over 1500+ retailer websites. The current single line indexing approach for storing data is causing inefficiencies, with the average data storage process taking approximately 25 hours.