OSP Picks to Kafka Infrastructure¶

Components¶

Component	Description	Ref
<<Config Service>>	Configuration details used for OSP Event Handler Service, BigQuery Writer, Streams to S3 lambda and topic segregator lambda	Configuration(from config service)
<<S3 Bucket>>	S3 Bucket to store pick details received from dynamodb streams for audit purposes	AWS S3 Bucket
<<AWS Role>>	AWS role for OSP Event Handler Service, BQ Writer, Stream to S3 lambda and topic segregator lambda	AWS Role
<<AWS ECS Service>>	ECS Service to handle events from realtimepicktaskupdated webhook. It then posts the message to DynamoDB for downstream processing	AWS ECS Service
<<AWS ECS Service>>	ECS Service to post pick events to Big Query.	AWS ECS Service
<<AWS Lambda>>	Lambda to store pick event details received from the dynamodb stream to S3 for audit	AWS Lambda
<<AWS Lambda>>	Lambda to post pick details received from the dynamodb stream to Kafka topic based on the type ie. old/new image	AWS Lambda

Configuration(from config service)¶

Config File Name
osp-picks-to-kafka.json

Value:

{
    "DYNAMODB_TABLE": "$ENV.xxwmm.storepick.osp-eh-v2",
    "DYNAMODB_TTL": 366,
    "ORDERS_SQS_QUEUE_NAME": "sqs-euw1-$ENV-storepick-wos-ord-hdlr-ord-agg-002",
    "ORDERS_SQS_ERROR_TABLE": "$ENV.xxwmm.storepick.osp-eh-v2-error-db",
    "ERROR_QUEUE_NAME": "sqs-euw1-$ENV-storepick-osp-eh-v2-order-picks-service-error-db-002",
    "MSK_CLUSTER_ARN": "arn:aws:kafka:eu-west-1:928369729144:cluster/picktask-test/2e94ea2f-ed50-4c4b-bf25-2f81aca8b00a-5",
    "PICKS_NEW_TOPIC": "picktasks-new",
    "PICKS_OLD_TOPIC": "picktasks-old",
    "S3_BUCKET_NAME": "m$ENV.xxwmm.storepick.picks-cdc-events.eeeeeeeeee",
    "S3_ROOT_FOLDER": "stream-to-s3",
    "BROKERS": "b-1.picktask-test.k8bixh.c5.kafka.eu-west-1.amazonaws.com:9094,b-2.picktask-test.k8bixh.c5.kafka.eu-west-1.amazonaws.com:9094,b-3.picktask-test.k8bixh.c5.kafka.eu-west-1.amazonaws.com:9094",
    "TIME_OUT": 30,
    "KFK_MAX_READ_MESSAGES": 1000,
    "BIG_QUERY_DATASET": "store_pick_real_time",
    "BIG_QUERY_TABLE": "pick_events",
    "BQ_KAFKA_CONSUMER_GROUP_ID": "KFK-BQ-CONSUMER-GRP",
    "BQ_KAFKA_MAX_MESSAGES_IN_ONE_BATCH": 1000,
    "KFK_MAX_POLL_INTERVAL_MS": 900000,
    "KFK_AUTO_COMMIT_INTERVAL_MS": 900000,
    "OSP_PICKS_TO_KAFKA_LOG_LVL": "INFO",
    "OSP_PICKS_TO_KAFKA_PYMORRI_LOG_LVL": "INFO",
    "ignorerecords": True,
    "ignorefollowing": [
        {
            "ignoreKey": "siteId",
            "ignoreValue": "999",
            "condition": "equals"
        }
    ]
}

Attribute	Description
DYNAMODB_TABLE	Dynamodb table name where the pick task details are stored
DYNAMODB_TTL	time till which a record can exist in DynamoDB table
ORDERS_SQS_QUEUE_NAME	Order SQS to send the pick details further to the SMS2 via wholesale order service
ORDERS_SQS_ERROR_TABLE	Error table to maintain the failed records occurred during Order SQS insert
ERROR_QUEUE_NAME	DLQ to maintain failed records occurred during dynamodb insertion
MSK_CLUSTER_ARN	Kafka cluster arn where the osp picks are transferred to various kafka topics
PICKS_NEW_TOPIC	Kafka topic where new/updated pick event is stored
PICKS_OLD_TOPIC	Kafka topic where old pick event is stored
S3_BUCKET_NAME	Name of the S3 bucket to store the pick event details for audit purposes
S3_ROOT_FOLDER	Name of the folder in S3 bucket to store the pick event details for audit purposes
BROKERS	Kafka brokers to fetch pick event details from Kafka topics
TIME_OUT	Max time until which the records are accumulated for writing to the BigQuery
KFK_MAX_READ_MESSAGES	Max number of messages to accumulate in a batch for writing to the BigQuery
BIG_QUERY_DATASET	Name of the dataset for the below big query table
BIG_QUERY_TABLE	BigQuery table used to store pick task event
BQ_KAFKA_CONSUMER_GROUP_ID	kafka consumer group id for BigQuery writer
BQ_KAFKA_MAX_MESSAGES_IN_ONE_BATCH	Max number of messages to read in a batch from Kafka
KFK_MAX_POLL_INTERVAL_MS	Timeout for reading from Kafka topics
KFK_AUTO_COMMIT_INTERVAL_MS	Timeout for writing to Kafka topics
OSP_PICKS_TO_KAFKA_LOG_LVL	Logging level for this component
OSP_PICKS_TO_KAFKA_PYMORRI_LOG_LVL	Logging level for pymorri
ignorerecords	Flag to indicate whether picks need to be filtered for test data
ignorefollowing	Filter conditions to use if filtering is enabled

AWS Role¶

Component	Role Name
OSP Event Handler Service	rol-glob-p-digital-ecsTaskSsmRestrictAccess
Big Query Writer	rol-glob-p-digital-ecsTaskSsmRestrictAccess
Storepick Stream to S3 Lambda	rol-glob-p-digital-restricted-lambda-001
Storepick topic segregator Lambda	rol-glob-p-digital-restricted-lambda-001

AWS S3 Bucket¶

Environment	S3 Bucket
dev	mdev.xxwmm.storepick.picks-cdc-events.eeeeeeeeee
sit	msit.xxwmm.storepick.picks-cdc-events.eeeeeeeeee
prd	mprd.xxwmm.storepick.picks-cdc-events.zzzzzzzzzz

AWS ECS Service¶

Environment	Aws ECS Service	Aws ECS Task
dev	svc-euw1-storepick-osp-eh-v2-dev-002	tsk-euw1-storepick-osp-eh-v2-dev-002
sit	svc-euw1-storepick-osp-eh-v2-sit-002	tsk-euw1-storepick-osp-eh-v2-sit-002
prd	svc-euw1-storepick-osp-eh-v2-prd-002	tsk-euw1-storepick-osp-eh-v2-prd-002

AWS ECS Service¶

Environment	Aws ECS Service	Aws ECS Task
dev	svc-euw1-dev-storepick-picks-to-bq-002	storepick-ecs-picks-bq-dev-002
sit	svc-euw1-sit-storepick-picks-to-bq-002	storepick-ecs-picks-bq-sit-002
prd	svc-euw1-prd-storepick-picks-to-bq-002	storepick-ecs-picks-bq-prd-002

AWS Lambda¶

Environment	Aws Lambda
dev	lmb-euw1-dev-storepick-stream-to-s3-001
sit	lmb-euw1-sit-storepick-stream-to-s3-001
prd	lmb-euw1-prd-storepick-stream-to-s3-001

AWS Lambda¶

Environment	Aws Lambda
dev	lmb-euw1-dev-storepick-picks-topic-segregator-001
sit	lmb-euw1-sit-storepick-picks-topic-segregator-001
prd	lmb-euw1-prd-storepick-picks-topic-segregator-001