How to Run an Automation in a Loop
As many of us are aware, automation in the Salesforce Marketing Cloud can only be scheduled to run at minimum once per hour. While this frequency may suffice for most automations created within the Automation Studio, there are instances where we may require an automation to run continuously, immediately after it completes its previous run.
Not forever, though. Nobody wants that, but at least it will, for example, process a huge number of records (millions).
To accomplish this, we will utilize a while loop and create two automations. The primary automation will act as the worker and execute the necessary tasks, while the secondary automation will solely serve to trigger the worker automation to restart the processing cycle.
This approach is particularly beneficial when handling Data Extensions with an extensive number of rows, often resulting in a run timeout if processed in one go. Alternatively, this method can be utilized in any other applications that necessitate immediate execution.
Prerequisites
Add following columns to data extension that will be processed in batches:
- processed as Boolean
- processed_at as date
Worker automation
Main automation that will process Data extension with a lot of data will consists of following activities.
- query activity to take portion of data from the data extension you want to process in loop
- verification activity – to finish automation while loop when there is no data to process
- activities to process actual records taken into actual loop
- mark records as processed so they wont be processed again in the original data extension
- call the caller – this is script using to call other automation. For this you can find wast examples on the internet. You can achieve same with WSProxy or with platform function InvokePerform. Example can be found on stackexchange.
//InvokePerform simplified example
var obj = Platform.Function.CreateObject("Automation");
Platform.Function.SetObjectProperty(obj, "ObjectID", ObjectID);
var po = Platform.Function.CreateObject("PerformOptions");
var performResult = [0,0,0];
var performStatus = Platform.Function.InvokePerform(obj, "start", performResult, po);
//ws proxy sipmlified example
var WSProxy = new Script.Util.WSProxy();
WSProxy.performItem("Automation", { CustomerKey: customerKey }, "start");
Caller automation
Caller automation will consist of wait activity to make sure our worker automation is finished before we called it back to run again. And the script itself that runs the worker automation with method described earlier.
GOOD TO KNOW: Do not configure scheduler leave it blank. If you do so the automation could not be called from script
