How to combine 2 or more scraping agents result in one agent?

To combine multiple web scraping agent results with exact same fields, we can use the Agenty post-processing script feature to merge multiple data tables and then set the agent result back to Agenty.

// Get the current job output as DataTable
var table = Agenty.Cloud.GetAgentResult("first agent id");

// Get running job id
var jobId = table.ExtendedProperties["job_id"];

// Get other agents data
var table2 = Agenty.Cloud.GetAgentResult("second agent id");

// merge both tables
table.Merge(table2);

// set job id
table.ExtendedProperties["job_id"] = jobId;

// set the result back to table
table.SetAgentResult();

Remember to run this agent in the end, as it will download the 1st agent result to combine. So, the first agent must be completed before the post-processing script is executed.

combine web scraping data1487×800 71 KB

For example, I have 2 agents to scrape electronics and hardware categories from an eCommerce website and want to combine the final result in a single CSV.

I can use this script to meet my needs.

• Go to the agent page
• Click on the Scripting tab
• Switch on the post-processing script
• Enter the script with and change the agent id GetAgentResult("here") in get result function
• Save and run