How to fix SSJS timeout when retrieving objects
We usually encounter this type of error when using SQL activities in Salesforce Marketing Cloud Engagement (MCE), and we already know how to handle the 30-minute runtime restriction. However, with the Script activity, I have experienced this issue for the first time. Let me show you how to tackle this problem with ease.
This never happened to me when I was retrieving Data Extension records or other objects, but for some mysterious reason, I encountered an error when retrieving a list of all Data Extensions from the current business unit. That said, this timeout can surprise you during any of the operations mentioned above.
The main issue wasn’t the timeout itself but the fact that I couldn’t process all the records I needed. Unfortunately, the retrieve method doesn’t offer an offset feature to paginate through records, which would allow me to traverse them slowly and manageably.
I have used the script retrieve all data extensions but also with few modification you can use advanced retrieves via WSProxy script directly from the salesforce help pages. While you’re reading this, you can also take a look at the list of all objects from which you can retrieve information using WSProxy and retrieve method.
After searching the vast internet for any mentions of an offset feature for the retrieve method, if there is any please let me know, the solution quickly presented itself. The records retrieval loop uses a request ID that is passed to the next query, automatically offsetting the records. So, I thought, why not use this information?
A simple modification to the script allowed me, even after a timeout, to continue from where it left off by rerunning the automation. So, what is the magic that lets me continue processing batches of data even after a timeout?
I simply stored the last request ID in the data extension. Why didn’t I think of this sooner? It was so simple, right in front of me, and yet I hadn’t used it before!
var result = [],
moreData = true,
reqID = data = null,
/* my addition to get request id from a data extension */
reqIDDB = Platform.Function.Lookup('request_id','reqid','automation','Get the query def and DE info');
/* if request id has been retrieved use that to continue retrieve request */
if (reqIDDB) reqID = reqIDDB;
/* script same as on https://www.ssjsdocs.xyz/contact-builder/dataextensions/retrieve/all.html*/
while(moreData) {
moreData = false;
if(reqID){
props.ContinueRequest = reqID;
/* my addition to save request id into a data extension */
if(reqID!=reqIDDB)
Platform.Function.UpsertData(
"request_id",
["automation"],
["Get the query def and DE info"],
["reqid"],
[reqID]
);
}
/* follows script same as on https://www.ssjsdocs.xyz/contact-builder/dataextensions/retrieve/all.html*/
I added the reqIDDB variable to use the request ID from the Data Extension, if available. I save it every time a new request ID is assigned within the loop. As you might guess, when the automation times out, the last request ID is safely stored in the Data Extension, ready for the next batch.
I will use it to retrieve the next batch of data in the following retrieve method when the automation runs again. This helped me resolve the issue of not having an offset feature in my retrieve function. However, my automation failed every time, requiring me to manually rerun it, so you can imagine this is just a duct-tape solution to the problem I had to tackle. The next level of this script would be to prevent timeout from even happening and adding automation loop just before we hit timeout and let the loop run until no more records are to be retrieved, that means no timeout is reached and automation finishes itself in success.
