During mining data is extracted only from the first page. Mining stops after first page extraction, or more pages are loaded but no more data is extracted.
Try the following:
- 1. Increase the 'Script Load Wait Time' value in Miner Settings. Default value of this setting is 5 seconds. Increase it to 10, 15 or 20 seconds and try mining again.
- 2. Soon after starting configuration', click anywhere on the page and select More Options > Page > Scroll Down. Then continue with the configuration. This will make sure that the whole page (including pagination links) is loaded before data extraction begins.
You may also try by clicking on the title of the first item on page and by selecting More Options > Scroll List soon after starting configuration.- 3. Make sure that the first item which you click after starting configuration does not belong to an advertised or sponsored listing. Always select data from a normal (non ad, non sponsored) listing.
- 4. Sometimes the first page of listings has a slightly different layout than the rest of the pages. In such cases the first and rest of pages should be scraped separately. Load the second page and then start configuration. See if data from subsequent pages (3,4 etc.) can be extracted during mining stage.
- 5. Try setting the next page link via both the methods explained (see the images) at Selecting pagination links. The next link can be set either by clicking the next link/arrow or by clicking the direct link to load page number 2. Try both these methods.
- 6. In case the direct links (URLs) to each page of listings has the page number embedded in it you can try the URL based pagination method.
- 7. Configure pagination for pages which load more data on the same page when scrolled down (infinite scroll) or when a button/link is clicked (load more).