Delta Feed Automation :
Current Customer operates their business across Mexican country and had different fulfillment centers for each region.
They need us to do continues updating of their inventory, price and catalogs product data based on demand for each region.
As of now business users are uploading their product data into SAP systems and SAP is sharing that data to commerce system as data feed.
But the challenge is we faced here is data need to continuously update into WCS system which is impacting the business with up to 4-6 % in loss and end user also getting hugely impacted.
We from WCS system manually need to run the delta data loads in periodically in Commerce servers and updating the same in Database.
So, we have come up with a solution of writing a custom script to automate the process . we provided the Folder structure for All Data Feeds which is having the data feeds file in the format of csv , this feeds uploaded by SAP System of script or we can upload manually on demand base to Data Feed Folder .
we need to follow below to implement the delta load automation
- Automating delta loads using shell script.
- Types of data considered to run for delta loads
- Folder Structure for delta loads
- Path to upload the delta load files
- Running script by cron jobs at certain intervals of time (operating system level)
- Logic developed to perform automation of delta loads
- Success scenario of delta loads
- Failure scenario of delta loads
- Script path
- Acknowledge through mail
- Cron jobs
- Delta loads the data loads that are run after running the complete initial data loads.
- Using Delta loads, data can be updated (if exist) or inserted if the data is new.
- Delta loads are run by cron jobs which will execute a shell script.
We are considering only three types of data for delta loads
This file needs to be uploaded using ftp to the path: /data/ecom/dataload/Price/updatePrice.csv
This file needs to be uploaded using ftp to the path: /data/ecom/dataload/CatalogEntry/File.csv
This file needs to be uploaded using ftp to the path: /data/ecom/dataload/Inventory/ Inventory.csv
- This script will be triggered by a cron job for every 15 minutes
- Cron jobs are configured at operating system level
- These jobs look for the CSV files for Price, CatalogEntry and Inventory delta loads
- If CSV files are available, cron does the following logic execution
- Both Price and Catalog Entries are available:
- It checks for the updatePrice.csv, if its available the it checks for CatalogEntries.csv
- If both catalog and price files are available, it first performs the delta load for catalog entries then loads price data.
- Only Price is available and catalog entries are not available:
- Only is price is found by cron job, it performs the delta load for price only.
- Both Inventory and Catalog Entries are available:
- If cron job finds files for Inventory and catalog entries, it first performs delta load for catalog entries then performs inventory delta load.
- Only Inventory is available and catalog entries are not available:
- If cron finds only Inventory files, it performs delta load for Inventory only.
- Only CatalogEntry is found.
- If cron job finds only CatalogEntry files, it performs delta load for catalog entry only.
script take care of three combinations into consideration to perform delta load
- Price and CatalogEntry or only Price
- Inventory and CatalogEntry or only Inventory
- Only CatalogEntry
- If cron job does the delta load successfully, it moves the csv files to Archive folder
- Archive folder is located at /data/ecom/util/Archive
If cron job fails to perform delta load, it moves the csv files to at /data/ecom/util/failures
Logs are found at this location: /data/ecom/util/logs
- Cron job sends a mail to configured user email.
- It sends a mail on delta load success, delta load fail.
- If there are no files to run, then it won’t trigger a mail
- Below is the cron configuration details on Dev environment.
[wcsuser@HOSTNAME]$ crontab -l
#*/15 /data/ecom/util/Scripts/dataload.sh 2>/dev/null