For pages which load more data on the same page when scrolled down (infinite scroll) or when a load-more button/link is clicked, if only the first page data is scraped during mining, please follow the steps below.

  1. 1. Open WebHarvy Settings and click on Advanced Miner Options button. Change the value of Data selection accuracy setting to Low and apply the change.
  2. 2. Load the page containing the items which you need to scrape within WebHarvy. Manually scroll down the page and/or click the load-more link/button, so that multiple pages (more than 3) of listings are loaded.
  3. 3. Scroll back up to the first item on the page and Start Configuration.
  4. 4. Follow the normal method of configuration and see if the problem is solved.

When initial navigation is required (form submission, button/link clicks) to load the listings page after starting configuration, please follow the steps given below in the exact same order to make sure that pagination is configured correctly.

  1. 1. Start configuration
  2. 2. Perform initial navigation (input text, select dropdown, click etc.) required to load the data listings page
  3. 3. Configure scroll to load or load more data pagination.
  4. 4. Start selecting data and continue with the rest of the configuration