Handling big data with Purge-it

We’re sometimes asked by JD Edwards organizations whether Purge-it can handle archiving (and if required purging) really large volumes of JD Edwards data. In this article we address how to run a big data Purge-it project based on many years of successful company implementation experiences.

When running an archive project where the database size is extremely large, it’s important that the correct approach is agreed and followed from the outset.  This is especially advisable in the General and Item Ledgers of organizations with large data volumes.

We advise an approach that targets the following points:

• Get the archive process running in Production as quickly as possible by taking a module by module implementation approach
  
  
• Reduce testing to levels that can validate and signoff the text results without necessarily running over 100’s of millions of transactions
  
  
• Use Purge-it archive automation to automate the process over a period of time to achieve the required results. This has the added benefit of minimising user involvement and only requiring occasional monitoring via the Purge-it environment and session dashboards.

Two key points to keep in mind:

1. Set realistic expectations of the time it’ll take to complete the data archiving project
   
   
2. Don’t over manage the project, because in practice, data archiving with Purge-it is a fairly simple process.

What does a Purge-it project consist of?

Phase 1 of a typical data archiving project

Get set up and started

• This starts with a kick off meeting to discuss the process and ensure everyone is aligned in the process and the project objectives
  
  
• Refresh the Test environment if necessary
  
  
• Install and configure the Purge-it software (performed by Klik IT inhouse JD Edwards data archiving experts)
  
  
• Get your team trained (performed by Klik IT’s inhouse JD Edwards trainer)

Phase 2 of a typical data archiving project

• Running the project
  
  
• Agree the archiving rules for your project.  For example, data retention dates, validations to use etc.
  
  
• Review custom tables to see which need to be archived (and/or purged if required)
  
  
• Look to automate the archive process
  
  
• Address company specific documentation requirements
  (each organization knows its own internal project processes)
  
  

How to set up your data archiving project
for a successful outcome

Questions and Answers

Question:  What should we be thinking about to get ready for our data archive project?

Answer:  The dates used for archiving JD Edwards data are key. One area to start thinking about and discussing is data retention dates across your business.  It’s entirely up to each organization how many years of data is retained.  Remember that archive data is easily accessible via Purge-it apps.

Question:  Is it necessary to run everything in test before moving to production?

Answer:  How the archive project is ordered, approached and executed is critical to the success of the project itself.  It would be easy to think about approaching the different tasks in a sequential order and running everything in test before moving to production.  Experience has shown that this is often the wrong approach for organizations with large amounts of data.

Question:  What action should we take if we see a lot of exceptions in our test results due to poor data quality?

Answer:  It’s important to remember that if you see exceptions that are due to poor data quality you don’t have to fix these.  This data exists in your system now and it might do after your data archiving project, there is no difference.  It doesn’t have to be the job of this project to clean the data.

It’s possible to take this opportunity to clean the data but it might have a disproportionate effect on the time to run the data archiving project.

Remember!

• Look for quick wins
  
  
• Look at the archive efficiency and remember the 80/20 rule.
  If you get rid of 80% of the data for 20% of the effort then this is a good place to be, so move on.
  
  
• It’s about running jobs.  Be pragmatic in your approach. Aim to not over analyse or over manage the data archiving project.

Running the project with large data

• When testing, use a reduced data set in order to get this phase of the project completed quickly.
  
  Example:
  If you decide to archive 10 years of JD Edwards historic data, then consider running the tests over a few months of this data. Let’s say 3 months of the data or if you prefer, as much as 12 months data, but do not run over the full 10 years.
  
  It’s perfectly possible to check and verify the results adequately on a smaller dataset.
  
  
• Automate your data archiving.  Set it and forget it!

For the General Ledger and Inventory modules, due to the number of indexes on the tables and the need to summarise them, the initial runs of these jobs can take some time.  When an organization has a large amount of JD Edwards data (500M plus records) then we suggest setting up daily jobs on the scheduler and leaving them to run. Each of these jobs will archive one days’ data for a specific year in the archive.

Consider this scenario:

If you have 10 jobs running daily, individually on a daily basis they may run for a few hours, but over the course of a year, 10 years data will be removed from the system.  These jobs will be background tasks requiring minimal monitoring.

There will be no system downtime. System maintenance and processes are unaffected.

This approach has been incredibly successful with a number of our clients including those with extremely big data.