StorePick AUI Processors

Overview

This repository contains a tasks which will be running a Lambdas. 1. AUI Product Import File Processor 2. AUI Product Import Load Processor 3. AUI Product Availability File Processor 4. AUI Product Availability Load Processor 5. AUI Product Capping File Processor 6. AUI Product Capping Load Processor

AUI Product Import File Processor -This lambda function invoke by S3 event. From AUI portal user uploaded file for product import then this lambda will trigger and read files. It process 2 types of files csv and xlsx. It will read file and prepare batches based on configuration and then it will add prepared batch into SQS for further process and move files.

AUI Product Import Load Processor - This lambda function is invoked by the SQS event. This will read message then call product API if MIN is valid then call overrides API from OSP to get all overrides for product. After that it will save details into product, masking tables.

AUI Product Availability File Processor - This lambda function is invoked by S3 event; when files are added in S3 bucket. It moves these files to processing folder on S3 bucket. Process the MIN, maskFrom, maskUntil, stores fields, adds them in batches of size specified by SQS_MSG_Limit. If it masking date fields it adds the prepared batches to SQS, for loading into masking tables. Invalid files are moved to Error folder in S3 bucket.

AUI Product Availability Load Processor - This lambda function is invoked by the SQS event. This will read message then it will check MIN in body is present in RDS or not firstly; If present then it will fetch masking details from RDS table and append provided masking details from SQS, Else call product API if MIN is valid then call overrides API from OSP to get all overrides for product and append provided overrides from SQS and prepare new payload. Later it will add message into priority 2 SQS. After that it will save details into product, masking, audit, bulk_upload tables.

AUI product Capping File Processor - This lambda function is invoked by S3 event ; when files are added in S3 bucket. It moves these files to processing folder on S3 bucket. Process the MIN, fromDate,toDate,capAlertSet,productCap, alertEmail, this fields will be added in batches of size specified by SQS_MSG_Limit. If its capping is reached then the notification will be sent to table . invalid files are moved to Error folder in S3 bucket

AUI Product Capping Load Processor - This lambda function is invoked by SQS event.This will read message then it will check MIN in body is present in RDS or not firstly; if present then it will fetch masking details from RDS database and append provided capping details from SQS , Else call product API if MIN is valid then call overrides API from OSP to get all overrides for product and append provided overrides from SQS and prepare new payload. then it update each product id in product capping table if min is not there it will raise an error. it will update the bulk upload table on the success return if it fails it raises an exception

Accounts Needed

AWS accounts will be required for DEV/SIT/PROD.

References

  1. Python Service with infrastructure as code.

  2. Unit tests & code coverage at about 90%+

  3. Configuration for multiple environments, DEV/SIT/PRD

Document

Ref

Architecture Document

Solution Architecture -.

Jira Story

Technical Elaboration -SPDP-4695.

Bitbucket repository

storepick-osp-file-processor-lambda.

Testing Approach

We will be using a BDD approach where the behaviour will be tested first the code will be developed to match the tested behaviour.

Deployment strategy

The deployment is handled by a jenkins multibranch pipeline, https://jenkins.p.morconnect.com/sonarqube/dashboard?id=aui-osp-productavailability-processor-feature. The commit in the develop branch will automatically deploy the application to the aws environments. The release is done by standard process of tagging the release branch, and by building and deploying from the tags. The tag format is sit-v*-<commit> for nonprod and prd-v*-<commit> for the production release. These tags are manually built on jenkins after the change request has been approved. The release branch is merged to the stable master branch after the successful production deployment.

Backout

To back out we will build off the last release in the master branch and redeploy an old tag version, this shall take around 5 minutes. This will minimise the data loss in case a problem is discovered.