Part 4: Using the URL Generator

This article picks up from the end of Part 3: Creating a Listings Extractor.

With the listings extractor created for a search page, you might want to add more URLs to your listings extractor since the search returns more than one page. To do this, you can use the URL Generator.

From the Import.io Dashboard, you can go to your listings extractor and then under the Setting tab, first start with an empty list by clicking Remove all URLs and then click the Generate URLs button to reveal the URL generator.

In the URL generator, you can create parameters, which allows you to use a list of values when generating URLs. To start, try highlighting and then clicking the 0, which controls the pagination. This will create a value called PARAMETER-1. Since the pagination is index based, it starts at 0 and then skips to 10, 20, 30, and so on. You can generate the first 5 pages by setting PARAMETER-1 as 0 to 40 skip 10.

Besides a range of numbers, you can add a list of comma separated values to search the other locations. To do this, select the find_loc value and then add your list of locations. After adding the locations, click Add to list to insert the the new URLs generated.

After adding the URLs, you can use the various options available to clean up the URLs, such as de-duplicating. When it looks good, you can go ahead and click Save and then Run URLs to get our list of business pages.

With the setup nearly completely, you will want to continue to Part 5: Chaining Two Extractors.