Scheduling an agent online

Agenty offers a powerful and flexible scheduling capabilities for your agents to run them automatically on minutes, hourly, daily, weekly basis or any particular day(s) you want to scrape data from websites on a scheduled time and day.

The scheduling feature in Agenty is most interesting to automate the data collection service running on the server. And, using the scheduling feature you can configure your scraping agent to run it on fixed interval automatically to refresh the data, for examples here are some use cases:

• Extract stock price from a website every 5 minutes and send an email alert when price drops/increases.
• Extract price from competitor’s websites every hour and import output to SQL using webhook.
• Scrape new jobs from jobs portal on Monday, Wednesday and Friday at 10:00 AM and send to a Google spreadsheet in your account.
• And lots more…

Note : All the time in scheduler are in UTC - Coordinated Universal Time

Let’s see how to automate web scraping agents

Create Schedule

1. Go to your agent details page, which you want to schedule

2. Click on top right side Manage this agent button and select the Schedule this agent option as in this screenshot
   

   

   agent scheduler steps in Agenty1145×423 63.4 KB

3. The scheduler dialog box will open where you can schedule your agent as per your requirement on minutes, hourly, daily, weekly, monthly or yearly basis
   

   

   create new schedule1041×497 62.4 KB

4. In this example, I scheduled my scraping agent on a daily basis - So I clicked on the Daily tab and selected ‘1’ from the day’s dropdown and selected the start time as 8:25 AM to run my agent.

5. And now, save this.

Once, the agent has been scheduled- you can see the scheduling description on Next Scheduled Run option on the agent page.

Note : A particular agent can have only one schedule and each time you click on the “Save” button, the existing schedule will be overwritten if any

CRON Expression

CRON is a UNIX tool that has been very known for its powerful scheduling capabilities. You can use the CRON expression in Agenty to create an advanced scheduler for your agent.

For example, we were talking to a web scraping customer last week who wanted to schedule his web scraping agents every 15 minutes, between 07:00 AM - 10:59 PM, only on Monday to Saturday which was not possible with scheduling UI. So, we end-up using following expression for him to meet the requirement from back-end:

0 0/15 7-22 ? * MON,TUE,WED,THU,FRI,SAT *

And that gave us a new idea to build a feature for all, see the CRON expression here

CRON scheduler for web scraping1555×363 75 KB

Here are some sample CRON expression, and more details about how to create these CRON expressions can be found here.

Update Schedule

1. Go to the agent page for which you want to update the schedule

2. Click on scheduled time hyperlink as in below screenshot or you may also go “Manage this agent > Schedule option” to change the schedule
   

   

   update agent schedule1082×457 61.4 KB

3. The scheduler dialog box will open where you can edit to change your scheduled day or time as per your need

4. For example, If I want to change my agent schedule time to every Friday - I just need to go to Weekly tab and click on Friday checkbox
   

   

   update agent schedule confirm1092×477 64.7 KB

5. And then click on Save button to update the schedule

Delete Schedule

If you want to delete the scheduling, just open the schedule dialog box and click on the Delete button.

delete scraping agent schedule946×458 52 KB