AUI OOS ECS Infrastructure¶

Components¶

Component	Description	Ref
<<Config Service>>	Configuration details used for OSP Event Handler Service, S3 to SQS lambda	Configuration(from config service)
<<ECS Task>>	Configuration details used for OSP Event Handler Service, to load aui_to_bq component	osp-productavailability-Ecs-Task
<<S3 Bucket>>	S3 Bucket to store pick details received	AWS S3 Bucket
<<SQS>>	SQS to add MINs in batches from file processor and then load processor will read that data for further processing	AWS SQS
<<RDS>>	RDS to save information of products , capping details, masking details ,bulk-upload,aui-audit,	RDS TABLES
<<Lambda>>	Lambda filters input customers list based on whether customers exist and return filter list of existing customers.	AWS Lambda

Configuration(from config service)¶

Config File Name
storepick-aui-productavailability-service.json

Value:

{
   "S3_BUCKET_NAME": "m$ENV.xxwmm.storepick.aui.product-import.eeeeeeeeee",
    "S3_PRODUCT_IMPORT_PATH": "productimport/pending",
    "S3_PRODUCT_AVAILABILITY_PATH": "product-availability/pending",
    "S3_PRODUCT_CAPPING_PATH": "product-capping/pending",
    "PRODUCT_SQS_NAME": "sqs-euw1-$ENV-storepick-aui-product-import-002.fifo",
    "AVAILABILITY_SQS_NAME": "sqs-euw1-$ENV-storepick-aui-product-availability-002.fifo",
    "CAPPING_SQS_NAME": "sqs-euw1-$ENV-storepick-aui-product-capping-002.fifo",
    "CHECKOUT_SQS_NAME": "sqs-euw1-$ENV-storepick-aui-checkout-002",
    "PRIORITY_ONE_SQS_NAME": "sqs-euw1-$ENV-storepick-aui-oos-priority-one-002.fifo",
    "PRIORITY_TWO_SQS_NAME": "sqs-euw1-$ENV-storepick-aui-oos-priority-two-002.fifo",
    "NOTIFICATION_SNS": "sns-euw1-$ENV-storepick-aui-email-notifier",
    "NO_OF_THREADS": 1,
    "BUFFER_SIZE": 1,
    "VISIBILITY_TIMEOUT": 120,
    "FLUSH_TIMEOUT": 120,
    "WAIT_TIME": 20,
    "WAIT_FOR_NEXT_SQS_CALL": 10,
    "READ_NO_OF_MESSAGES": 10,
    "DATABASE": "aui",
    "SQS_MSG_LIMIT": 100,
    "DEFAULT_FETCH_LIMIT": 1,
    "FILE_TYPES": [
        "csv",
        "xlsx"
    ],
    "PRODUCT_DYNAMODB_TABLE": "$ENV.xxwmm.storepick.aui.products",
    "PRODUCT_CAP_DYNAMODB_TABLE": "$ENV.xxwmm.storepick.aui.product_cap",
    "PRODUCT_MASKING_DYNAMODB_TABLE": "$ENV.xxwmm.storepick.aui.product_masking",
    "BULK_UPLOAD_DYNAMODB_TABLE": "$ENV.xxwmm.storepick.aui.bulk_upload",
    "AUDIT_DYNAMODB_TABLE": "$ENV.xxwmm.storepick.aui.audit",
    "USER_ROLES_DYNAMODB_TABLE": "$ENV.xxwmm.storepick.aui.user_roles",
    "ORDERS_DYNAMODB_TABLE": "$ENV.xxwmm.storepick.aui.orders",
    "DYNAMO_TTL": 90,
    "MORI_BASE_URL": "https://sit-api.morrisons.com",
    "PRODUCT_URL": "/product/v1/items/",
    "OSP_BASE_URL": "https://mocks.dev.spt.digital.morconnect.com/osp",
    "OSP_AVAILABILITY_URL": "v4/products",
    "OSP_ORDER_SERVICE_URL": "v3/morrisons/orders",
    "OSP_TIMEOUT_SECONDS": 61,
    "OSP_API_CONCURRENT_THREADS": 10,
    "KAFKA_BROKER": "b-1.picktask-test.k8bixh.c5.kafka.eu-west-1.amazonaws.com:9094,b-2.picktask-test.k8bixh.c5.kafka.eu-west-1.amazonaws.com:9094,b-3.picktask-test.k8bixh.c5.kafka.eu-west-1.amazonaws.com:9094",
    "KAFKA_TOPIC_MASKING": "kafka-euw1-$ENV-digital-storepick-aui-product-masking",
    "BIG_QUERY_DATASET": "store_pick_real_time",
    "BIG_QUERY_TABLE": "product_masking",
    "KAFKA_TOPIC_CAPPING": "kafka-euw1-$ENV-digital-storepick-aui-product-capping",
    "BIG_QUERY_MASKING_TABLE": "product_masking",
    "BIG_QUERY_CAPPING_TABLE": "product_capping",
    "TIME_OUT": 30,
    "KFK_MAX_READ_MESSAGES": 1000,
    "BQ_KAFKA_CONSUMER_GROUP_ID": "AUI-LIVE-FEED-GRP",
    "BQ_KAFKA_MAX_MESSAGES_IN_ONE_BATCH": 10,
    "KFK_MAX_POLL_INTERVAL_MS": 90000,
    "KFK_AUTO_COMMIT_INTERVAL_MS": 90

}

Attribute	Description
S3_BUCKET_NAME	S3 bucket to upload product import, capping, masking files for process and some predefined templates also
S3_PRODUCT_IMPORT_PATH	Import path for product import file
S3_PRODUCT_AVAILABILITY_PATH	Import path for product availability file
S3_PRODUCT_CAPPING_PATH	Import path for product capping file
PRODUCT_SQS_NAME	Product SQS name in this file-processor lambdas will add MINs from uploaded file into batches
AVAILABILITY_SQS_NAME	Product availability SQS name in this product-availability-file-processor lambdas will add MINs, maskfrom maskUntil, stores from uploaded file, in batches
CAPPING_SQS_NAME	Product capping SQS name in this product-availability-file-processor lambdas will add MINs, fromDate toDate, capAlertSet, productCap, alertEmail, from uploaded file, in batches
SQS_MSG_LIMIT	This is SQS message limit, number of MINs in one message
FILE_TYPES	Supported file types csv and xlsx
DATABASE	this is the RDS database that we are using to store information
MORI_BASE_URL	This is morrisons base URL
PRODUCT_URL	This is product service URL
OSP_BASE_URL	This is OCADO base URL
OSP_AVAILABILITY_URL	This is OCADO availability URL to GET/POST product overrides
BIG_QUERY_DATASET	This is the name of big query dataset
BIG_QUERY_TABLE	This is name of the big query table
KFK_MAX_READ_MESSAGES	number of message read at one batch
KFK_MAX_POLL_INTERVAL_MS	this is a poll interval for kafka consumer

AWS S3 Bucket¶

Environment	S3 Bucket
dev	mdev.xxwmm.storepick.aui.product-import.eeeeeeeeee
sit	msit.xxwmm.storepick.aui.product-import.eeeeeeeeee
prd	mprd.xxwmm.storepick.aui.product-import.zzzzzzzzzz

AWS SQS¶

Environment	SQS
dev	sqs-euw1-dev-storepick-aui-product-import-002.fifo
sit	sqs-euw1-sit-storepick-aui-product-import-002.fifo
prd	sqs-euw1-prd-storepick-aui-product-import-002.fifo

RDS TABLES¶

RDS Name - rds-euw1-dev-storepick-pgr-aui-001¶

RDS Name - rds-euw1-sit-storepick-pgr-aui-001¶

RDS Name - rds-euw1-prd-storepick-pgr-aui-001¶

RDS TABLES¶
Table Name
storepick_aui.product_capping
storepick_aui.bulk_upload
storepick_aui.audit
storepick_aui.product_masking

AWS Lambda¶

Environment	Lambda
dev	lmb-euw1-dev-storepick-aui-product-import-file-processor-001
sit	lmb-euw1-sit-storepick-aui-product-import-file-processor-001
prd	lmb-euw1-prd-storepick-aui-product-import-file-processor-001

AWS Lambda¶

Environment	Lambda
dev	lmb-euw1-dev-storepick-aui-product-import-load-processor-001
sit	lmb-euw1-sit-storepick-aui-product-import-load-processor-001
prd	lmb-euw1-prd-storepick-aui-product-import-load-processor-001