Commerce, DevOps & Cloud Solutions

WebSphere Commerce DeltaFeed Automaiton

Delta Feed Automation : 

Challenge:

Current Customer operates their business across Mexican country and had different fulfillment centers for each region.
They need us to do continues updating of their inventory, price and catalogs product data based on demand for each region.
As of now business users are uploading their product data into SAP systems and SAP is sharing that data to commerce system as a data feed.
But the challenge is we faced here is data need to continuously update into WCS system which is impacting the business with up to 4-6 % in loss and end user also getting hugely impacted.

Problem:

We from WCS system manually need to run the delta data loads in periodically in Commerce servers and updating the same in Database.

Implemented Solution:

So, we have come up with a solution of writing a custom script to automate the process. we provided the Folder structure for All Data Feeds which is having the data feeds file in the format of csv , this feeds uploaded by  SAP System of script  or we can upload manually on demand base  to  Data Feed Folder .

we need to follow  below to implement the delta load automation

  • Automating delta loads using shell script.
  • Types of data considered to run for delta loads
  • Folder Structure for delta loads
  • Path to upload the delta load files
  • Running script by cron jobs at certain intervals of time (operating system level)
  • Logic developed to perform automation of delta loads
  • Success scenario of delta loads
  • Failure scenario of delta loads
  • Logs
  • Script path
  • Acknowledge through mail
  • Cron jobs

Delta Loads:

  • Delta loads the data loads that are run after running the complete initial data loads.
  • Using Delta loads, data can be updated (if exist) or inserted if the data is new.
  • Delta loads are run by cron jobs which will execute a shell script.

Types of data considered for delta load:

We are considering only three types of data for delta loads

  1. CatalogEntry
  2. Price
  3. Inventory

Folder Structure

Path: /data/ecom/dataload/

CSV Files for each data type and path to upload:

Price:

This file needs to be uploaded using ftp to the path: /data/ecom/dataload/Price/updatePrice.csv

catalog entry:

This file needs to be uploaded using ftp to the path: /data/ecom/dataload/CatalogEntry/TelcelCatalogEntries.csv

Inventory:

This file needs to be uploaded using ftp to the path: /data/ecom/dataload/Inventory/ Inventory.csv

Script:

  • /data/ecom/util/Scripts/dataload.sh
  • This script will be triggered by a cron job for every 15 minutes

Logic developed to perform automation of delta loads

  1. Cron jobs are configured at the operating system level
  2. These jobs look for the CSV files for Price, CatalogEntry and Inventory delta loads
  3. If CSV files are available, cron does the following logic execution
  4. Both Price and Catalog Entries are available:
    • It checks for the updatePrice.csv, if its available the it checks for TelcelCatalogEntries.csv
    • If both catalog and price files are available, it first performs the delta load for catalog entries then loads price data.
  5. Only Price is available and catalog entries are not available:
    • Only is price is found by cron job, it performs the delta load for price only.
  6. Both Inventory and Catalog Entries are available:
    • If cron job finds files for Inventory and catalog entries, it first performs delta load for catalog entries then performs inventory delta load.
  7. Only Inventory is available and catalog entries are not available:
    • If cron finds only Inventory files, it performs delta load for Inventory only.
  8. Only CatalogEntry is found.
    • If cron job finds only CatalogEntry files, it performs delta load for catalog entry only.

 script  take care of  three combinations into consideration to perform delta load

  1. Price and CatalogEntry or only Price
  2. Inventory and CatalogEntry or only Inventory
  3. Only CatalogEntry

Success Scenarios:

  1. If cron job does the delta load successfully, it moves the csv files to Archive folder
  2. Archive folder is located at /data/ecom/util/Archive

Failure Scenario:

If cron job fails to perform delta load, it moves the csv files to at /data/ecom/util/failures

Logs Location:

Logs are found at this location: /data/ecom/util/logs

Acknowledge with a mail:

  • Cron job sends a mail to configured user email.
  • It sends a mail on delta load success, delta load fail.
  • If there are no files to run, then it won’t trigger a mail

Cron Jobs:

  1. Below is the cron configuration details on Telcel Dev environment.

[wcsuser@QAE120001 logs]$ crontab -l

#*/15     /data/ecom/util/Scripts/dataload.sh 2>/dev/null

About The Author

Leave a Reply

*