OSP Picks to Kafka Runbook¶

Overview¶

This is the runbook for OSP Picks to Kafka which covers below components. 1. OSP Events Handler Service 2. Storepick BigQuery Writer 3. Storepick Streams to S3 lambda 4. Storepick topic segregator lambda

This documents (and references) the component documentation. Below are a set of scenarios that may happen in production along with diagnosis and resolution information.

References¶

Document	Ref
Architecture Document	Solution Architecture.
Jira Story (OSP Picks to Kafka)	Technical Elaboration - PickTask: SPDP-3781.
Bitbucket (OSP Picks to Kafka)	https://bitbucket.org/morrisonsplc/storepick-osp-picks-to-kafka/src/

Runtime Issues List¶

Issue	Component	Description
Issue 1	Main Controller	Error - content type as application/json
Issue 2	Pick_task_validator	PayloadValidationException - when validation fails for one of the fields in the request json
Issue 3	Pick_task_validator	PayloadValidationException - Mandatory field eventId cannot be empty or null
Issue 4	ApplicationConfiguration	ApplicationConfigurationException - application configuration missing
Issue 5	ApplicationDependencies	Exception - Exception while loading configs
Issue 6	Main Controller	PickTaskHandlerException - key details missing for inserting errors into Dynamodb
Issue 7	Process_picks_to_bq Task	MoriConfigurationException - error while writing records into BigQuery table
Issue 8	Storepick_stream_to_s3 Lambda	botocore.exceptions.ClientError - ClientError - Unexpected client error. Please check config and bucket exists
Issue 9	Storepick_topic_segregator Lambda	OSP_KAFKA_PRODUCER_ERROR - Probably due to bad config
Issue 10	Storepick_topic_segregator Lambda	OSP_KAFKA_PRODUCER_ERROR - Error while writing to Kafka topic
Issue 11	Order delivered event end-point	405 Method Not Allowed - The method is not allowed for the requested URL.
Issue 12	Order delivered event end-point	400 BAD REQUEST - orderId is either blank or not found in request payload.
Issue 13	Order delivered event end-point	400 BAD REQUEST - Failed to decode JSON object: Expecting value: line 2 column 16 (char 18).

Runtime Issues¶

Issue 1¶

Scenario¶

When unsupported Media Type is set / used in the request json. Unsupported Media Type. Request data unreadable, content type was not set as application/json error code - 415.

Error¶

Error

Symptoms¶

ospeventhandler will respond with response code 415 .

Corrective Action¶

call ospeventhandler with correct request json

Issue 2¶

Scenario¶

when validation fails for one of the fields in the request json.

Error¶

PayloadValidationException raised when validation fails for one of the fields in the request json

Symptoms¶

ospeventhandler will respond with response code 400 and log it to dynamodb.

Corrective Action¶

call ospeventhandler with correct request json

Issue 3¶

Scenario¶

when the mandatory field event_id is not present in the request header.

Error¶

PayloadValidationException raised when the mandatory field event_id is not present in the request header

Symptoms¶

ospeventhandler will respond with response code 400 and log it to dynamodb.

Corrective Action¶

call ospeventhandler with a valid event_id in request header

Issue 4¶

Scenario¶

This happens when application configuration file is missing in the config service or when config service is not reachable.

Error¶

ApplicationConfigurationException raised when application configuration is missing in config service

Symptoms¶

ospeventhandler will respond with response code 5XX.

Corrective Action¶

make sure file storepick-real-time-event-handler.json is available in the config service and SSM is configured correctly.

Issue 5¶

Scenario¶

This happens when there is an error while loading all the dependencies needed for the application to run.

Error¶

Exception raised when loading dependencies

Symptoms¶

ospeventhandler will respond with response code 5XX.

Corrective Action¶

make sure DYNAMODB_TTL, PICK_TASK_SNS_NAME or DYNAMODB_ERROR_TABLE fields are present in the config file.

Issue 6¶

Scenario¶

This happens when pickTaskId or eventId is missing from the request which is required for inserting errors into Dynamodb.

Error¶

PickTaskHandlerException raised when pickTaskId or eventId is missing from the request

Symptoms¶

ospeventhandler will respond with response code 5XX.

Corrective Action¶

call ospeventhandler with a valid event_id and pickTaskId in the request header.

Issue 7¶

Scenario¶

This happens when there is an error while writing records into BigQuery table.

Error¶

Exception raised when there is an error while writing records into BigQuery table

Symptoms¶

ospeventhandler will respond with response code 5XX.

Corrective Action¶

Check logs.

Issue 8¶

Scenario¶

This happens when there is an error while writing records into S3.

Error¶

Exception raised when there is an error while writing records into S3

Symptoms¶

ClientError - Unexpected client error. Please check config and bucket exists.

Corrective Action¶

Check config file for S3 configuration. Correct the configuration and reload Lambda function.

Issue 9¶

Scenario¶

This happens when there is an error while writing records into Kafka topics.

Error¶

Exception raised when there is an error while writing records into Kafka topics.

Symptoms¶

OSP_KAFKA_PRODUCER_ERROR - Probably due to bad config.

Corrective Action¶

Check config file for Kafka configuration. Correct the configuration and reload Lambda function.

Issue 10¶

Scenario¶

This happens when there is an error while writing records into Kafka topics.

Error¶

Exception raised when there is an error while writing records into Kafka topics.

Symptoms¶

OSP_KAFKA_PRODUCER_ERROR - Error while writing to Kafka topic.

Corrective Action¶

Check the kafka cluster. Check the cluster logs. Analyse the issue using cluster logs and lambda logs.

Issue 11¶

Scenario¶

This happens when there is an error while doing GET instead of POST.

Error¶

Exception raised when there is an error in the use of HTTP method.

Symptoms¶

405 Method Not Allowed - The method is not allowed for the requested URL.

Corrective Action¶

Please change the HTTP method to POST.

Issue 12¶

Scenario¶

This happens when there is blank no orderId or invalid key (like orderID) in request payload.

Error¶

The respo return 400 bad request.

Symptoms¶

400 BAD REQUEST - orderId is either blank or not found in request payload.

Corrective Action¶

Please POST the valid orderId.

Issue 13¶

Scenario¶

This happens when there is non-text orderId(other params) in request payload.

Error¶

The respo return 400 bad request.

Symptoms¶

400 BAD REQUEST - Failed to decode JSON object: Expecting value: line 2 column 16 (char 18).

Corrective Action¶

Please POST the valid orderId quoted as string.