Pick Task Event Handler

Overview

This repository contains 2 ECS tasks which will be running as a service and 2 Lambdas.

1. OSP Events Handler Service

2. Storepick Streams to S3 lambda

3. Storepick topic segregator lambda

4. Storepick BigQuery Writer

OSP Events Handler Service - The service will bring up the end-points for consuming Ocado pick task and order delivered events. The pick end point receives the order pick event, validates it and inserts the same into Dynamodb. The order delivered end-point takes the order id and retailer banner id. This data is pushed to the order sqs for further processing.

Storepick Streams to S3 lambda - This lambda function is invoked by the dynamodb data stream. When there is a insert/update to the record in the dynamodb. The lambda function then creates the S3 file and stores it.

Storepick topic segregator lambda - This lambda function is invoked by the dynamodb data stream. When there is a insert/update to the record in the dynamodb. This lambda then segregates the data and insert it into Kafka. New records is pushed to the Kafka New Topic and old image of the record is pushed to the old kafka topic.

Storepick BigQuery Writer - This service consumes the records from the Kafka and inserts it into the BigQuery.

Accounts Needed

AWS accounts will be required for DEV/SIT/PROD.

References

1. Python Flask Service with infrastructure as code.

2. An end-point to consume pick-task events from ocado web-hook.

3. An end-point to consume order-delivered events from ocado web-hook.

4. An end-point for health check of the service.

5. Unit tests & code coverage at about 90%+

6. Configuration for mutliple environments, DEV/SIT/PRD

Document	Ref
Architecture Document	Solution Architecture.
Jira Story (Pick Task)	Technical Elaboration - PickTask: SPDP-3781.
Bitbucket repository	storepick-osp-picks-to-kafka.

Testing Approach

We will be using a BDD approach where the behaviour will be tested first the code will be developed to match the tested behaviour.

Deployment strategy

The deployment is handled by a jenkins multibranch pipeline, https://jenkins.p.morconnect.com/digital/job/storepick-osp-picks-to-kafka/. The commit in the develop branch will automatically deploy the application to the aws environments. The release is done by standard process of tagging the release branch, and by building and deploying from the tags. The tag format is sit-v*-<commit> for nonprod and prd-v*-<commit> for the production release. These tags are manually built on jenkins after the change request has been approved. The release branch is merged to the stable master branch after the successful production deployment.

Backout

To back out we will build off the last release in the master branch and redeploy an old tag version, this shall take around 5 minutes. This will minimise the data loss in case a problem is discovered.

Components

Components

• OSP Picks to Kafka Infrastructure
• OSP Picks to Kafka Runbook
• OSP Picks to Kafka Code

Swagger

Swagger for application is found in the docs section of the repository.

Diagrams

Picktask & Order delivered diagram gives the overview from the point of realtimepicktaskupdated webhook. The webhook sends the pick task & order delivered events payload from ocado to osp-event handler service. The events are posted to the downstream components for further processing.

Order delivered component diagram shows the components developed as part of order event handling process.

Order delivered overview diagram allows to peek into the overall and component level details of the pick task order delivered handler operation.

OSP Picks to Kafka & BigQuery Writer Sequence Diagram

OSP Picks to kafka Overview This sequence diagram gives the overview from the point realtimepicktaskupdated webhook from Ocado calls osp-eventhandler service and how pick task events are received and processed further downstream.