OR Big Query Writer Infrastructure¶

Component	Description	Ref
<<Config Service>>	Configuration details used for OR Big Query Writer	Configuration(from config service)
<<SNS>>	SNS for Cloudwatch Alarm.	SNS Topic name for Cloudwatch alarm
<<CLOUDWATCH RULE>>	CLOUDWATCH RULE to trigger task after every 1 hour.	Cloudwatch Rule Name
<<CLOUDWATCH LOG GROUP>>	CLOUDWATCH LOG to show logger messages .	Cloudwatch Log Group Name
<<SSM>>	SSM for Online Returns.	SSM for online returns
<<SSM>>	SSM for accssing GCP Big Query Table.	SSM for GCP Big Query
<<RDS>>	RDS will contain three tables scanned_product,return_product and online_product under storepick_online_returns schema.	AWS RDS
<<Ecstask>>	This Task is used to fetch scanned product and persist into big query table.	AWS ECS Task
<<GCP Big Query Projects>>	Big Query Projects which has big query table ‘store_pick.online_returns’.	GCP Big Query Projects

Configuration(from config service)¶

Config File Name
storepick-online-returns.json

Value:

config = {
 "DATABASE": "online_returns",
 "MORI_BASE_URL": "https://api.morrisons.com",
 "PRODUCTELASTICSEARCH_URL": "product/v1/items/",
 "STOCKTRANSFERSCREATE_URL": "stock/v2/transfers",
 "STOCK_TRANSFER_REQUEST": {
     "VIRTUAL_LOC_PREFIX": "8",
     "USER_NAME": "StorePickUser",
     "CONTAINER_ID": 1
 },
 "STOCKADJUSTMENTCREATE_URL": "stock/v2/locations",
 "GETITEMPRICE_URL": "price/v2/locations",
 "GETLOCATION_URL": "location/v2/stores",
 "OSP_BASE_URL": "https://api.morrisons.osp.tech/externalapi",
 "OSP_USER_URL": "v4/users/find-by-username",
 "CALL_LOCATION_V2": True,
 "STOCK_ADJUSTMENT_REASON": "31",
 "STOCK_ADJUSTMENT_LIMIT": 100,
 "STOCK_TRANSFER_LIMIT": 1000,
 "NO_OF_DAYS_FOR_HISTORY": 2,
 "IS_CATCHWEIGHT_REQUIRED": False,
 "BIG_QUERY_DATASET": "store_pick",
 "BIG_QUERY_TABLE": "online_returns",
 "BIG_QUERY_LIMIT": 100,
 "PRICE_V2_CACHE_MAX_LENGTH": 99999,
 "PRICE_V2_CACHE_TTL": 100,
 "API_RETRY_COUNT": 5,
 "API_RETRY_EXPONENTIAL_BACKOFF": 1000,
 "RDS_RETRY_COUNT": 5,
 "KAFKA": {
     "CONSUMER_GROUP_ID": "return_products_from_sales_adapter",
     "BOOTSTRAP_SERVER": "b-2.msk-euw1-prd-digital-v.7a52by.c1.kafka.eu-west-1.amazonaws.com:9094,b-1.msk-euw1-prd-digital-v.7a52by.c1.kafka.eu-west-1.amazonaws.com:9094,b-3.msk-euw1-prd-digital-v.7a52by.c1.kafka.eu-west-1.amazonaws.com:9094",
     "TOPICS": [
         "kafka-euw1-$ENV-digital-storepick-online-returns-sales-items"
     ],
     "READ_MAX_MESSAGES": 10000,
     "TIME_OUT": 20
 },
 "STORE_ID_LIST": [],
 "AUTHORIZED_USER_ROLES": [
     "shift_manager",
     "driver_manager"
 ]

}

SNS Topic name for Cloudwatch alarm¶

Environment	RDS
dev	sns-euw1-dev-storepick-online-return-big-query-writer-002-001-Alarm
sit	sns-euw1-sit-storepick-online-return-big-query-writer-002-001-Alarm
prd	sns-euw1-prd-storepick-online-return-big-query-writer-002-001-Alarm

Cloudwatch Rule Name¶

Environment	Cloudwatch Rule
dev	cwr-euw1-dev-storepick-online-return-big-query-writer-002
sit	cwr-euw1-sit-storepick-online-return-big-query-writer-002
prd	cwr-euw1-prd-storepick-online-return-big-query-writer-002

Cloudwatch Log Group Name¶

Environment	Cloudwatch Log Group
dev	clg-euw1-dev-storepick-online-return-big-query-writer-002
sit	clg-euw1-sit-storepick-online-return-big-query-writer-002
prd	clg-euw1-prd-storepick-online-return-big-query-writer-002

SSM for online returns¶

SSM Name
storepick_online_returns_secret

Value:

{
    "apikey": "__EXISTS__",
    "resourceArn": "__EXISTS__",
    "secreteArn": "__EXISTS__",
    "hostUrl": "__EXISTS__",
    "userName": "__EXISTS__",
    "password": "__EXISTS__"
}

SSM for GCP Big Query¶

SSM Name
storepick_google_secret

Value:

{
    "type": "__EXISTS__",
    "project_id": "__EXISTS__",
    "private_key_id": "__EXISTS__",
    "private_key": "__EXISTS__",
    "client_email": "__EXISTS__",
    "client_id": "__EXISTS__",
    "auth_uri": "__EXISTS__",
    "token_uri": "__EXISTS__",
    "auth_provider_x509_cert_url": "__EXISTS__",
    "client_x509_cert_url": "__EXISTS__"
}

AWS RDS¶

Environment	RDS
dev	rds-euw1-dev-digital-apsql-onl-returns-002
sit	rds-euw1-sit-digital-apsql-onl-returns-002
prd	rds-euw1-prd-digital-apsql-onl-returns-002

AWS ECS Task¶

Environment	ECS Task
dev	storepick-ecs-online-return-big-query-writer-dev-002
sit	storepick-ecs-online-return-big-query-writer-sit-002
prd	storepick-ecs-online-return-big-query-writer-prd-002

GCP Big Query Projects¶

Environment	Big Query Project
dev	mrw-storepick-dev
sit	mrw-sit-storepick
prd	mrw-prod-storepick