Web Scraping with Screenshot

The scraping agent allows you to capture the webpage screenshot automatically while crawling a website, as it is very common in web scraping to capture a screenshot of a website for testing or as a proof of data being scraped is exactly same as it’s shown on the website.

There are 3 types of webpage screenshot capture options available in Agenty

1. Small screenshot : Visible portion on screen or viewport set
2. Full screenshot: Full web page screenshot
3. HD screenshot: High definition screenshot of full web page

Step by step guide: Let’s make them easy to understand how to capture automated website screenshot with this tool:)

Small screenshot

To capture the visible webpage screenshot only, follow these steps:

• Go to your agent page
• Click on the Configuration tab and scroll down to Fields section
• Add a new field with configuration like this
• Field Name : Any name of your choice
• Field Type:DEFAULT`
• From : screenshot

visible webpage screenshot1684×699 99.6 KB

• Save and re-run your agent
• The scraping agent will capture the visible webpage screen.

screenshot webpage in scraping result1699×756 141 KB

This option allow you to capture the above the fold or visible part of website on computer screen without scrolling. The webpage will be captured as per the viewport width and height given in the web scraping engine. So, if you are looking to capture the visible page screenshot with web scraping, use this option. Here is an example :-

https://s3.amazonaws.com/data.agenty.com/Screenshots/2022/7/21/books.toscrape.com-i547w4ehnl.png

Full page screenshot

The full page screenshot option allow us to capture automated website’s full page screen, which make the task easier to diagnose to view full content of website.

To make screenshot of an entire portion of a webpage - We need to follow the same steps as followed above, except change the from type to full_screenshot:

• Agent page > Configuration Tab
• Add new Field
• Select Default as Type
• Select full screenshot option in system fields

Capture entire page of website1397×737 135 KB

• Save this and Click on Run button

Now, when the scraping job is executed, Agenty will capture the full page screenshot.

Here is an example of full webpage screenshot captured in this agent with viewport : full web page screenshot with web scraping

HD Screenshot

The High Definition screenshot produce larger, wider and clearer picture of a website with increased pixels and wider scaled viewport. So, the HD screenshot option allows you to get the better experience to diagnose the web page during web scraping.

To capture the HD screenshot of web page, follow these steps:

• Agent Page > Configuration tab
• Add a default field
• Select hd_screenshot option from system field
  
  

  HD screenshot of webpage1102×480 38.7 KB
• Save the configuration and Run it

Now you can see the high definition quality screenshot of all the pages of the website will be captured by Agenty automated scraping agent.