With the introduction of SuiteCloud Processor into NetSuite, the batch job management with NetSuite took a new turn. With a simple use case let us explore how Map/Reduce brought a significant impact in this area.
On a day to day basis, the client’s NetSuite system must process several thousands of records to manage and recompute the data based on complex business logic. Each process might involve looking up and writing into several other records which are costly operations.
The core business revolves around Subscriptions, Events, and Certifications, and the client manages 300000+ Subscriptions and several thousands of ongoing events. This whole feature was implemented with NetSuite’s scheduled script in SuiteScript 1.0.
Using schedule scripts to process high volume data comes with several drawbacks.
Limited Governance Units
Scheduled Script in NetSuite has an available governance unit of 10,000. Taking into consideration the volume of data and complex business logic, processing governance units were not sufficient. It required an extra rescheduling logic which might not run as expected based on the available queue.
Performance Pull Down
When processing data in the Scheduled script each record goes through the business logic sequentially consuming the scheduled script queue for a longer time. This might hold off any other scheduled scripts in the queue.
Complex Error Handling
Plus, there’s the challenge of error handling, due to the Sequential Execution of Data, the errors needed to be handled or the whole scheduled script would stop and any further records will not be processed.
FORTUNATELY, THERE’S A BETTER WAY…
Introducing Team Incept, we provide expertise in NetSuite customization, so you can relax. Our experts provided a solution involving migrating Scheduled Script-based design to Map Reduce based design using SuiteScript 2.0.
Map/Reduce Scripts come with a structured way to handle large amounts of data, in different stages to the process and helps to navigate data. Let’s look into the key features of Map/Reduce that solved our problems.
Map/Reduce’s allows parallel processing at multiple stages significantly cutting down the time of execution and allowing the queue to other awaiting jobs.
To ice the cake NetSuite gives control of what level of parallelism a user would want to choose.
(The only notable caution would be to take care of the concurrency issue with records. This had to be handled at the input stage making sure no duplicate records are made to process.)
Unlike the scheduled script where the developer had to take the responsibility of passing each data for processing, Map/Reduce gives us robust context objects which are passed to each stage which can access data from the previous stage and write to the next stage.
The context object also includes information about the remaining governance units, errors, and other information. This also helps in bringing out better error handling and flagging failed processes.
As Map/Reduce scripts processing is divided into several stages each of these stages have their governance units. The jobs invoked with the Map and Reduce stage also get their governance units to each computation hence providing sufficient units to process huge volumes of data if structured properly.
Map/Reduce also allows the yielding of jobs if it goes beyond the allocated units and takes care to create a job with the same priorities.
Through our solution, we saw a drastic reduction in processing time and improvement in the overall performance of the system. As an added benefit, we migrated all the SuiteScript 1.0 to SuiteScript 2.0 building a modular, structured, and maintainable code- base.
To learn how Incept can help you implement such NetSuite customization visit us as inceptdatasolution.com.