OSP Stock Transfer Adapter Runbook¶

Overview¶

This is the runbook for OSP Stock Transfer Adapter for Storepick.

This documents (and references) the component documentation. Below are a set of scenarios that may happen in production along with diagnosis and resolution information.

References¶

Document	Ref
Architecture Document	Solution Architecture.
Jira Story (Forward Location Aggregator, Reverse Location Aggregator, Stock Transfer Adapter)	Technical Elaboration - PickTask: `https://morrisons.atlassian.net/browse/SPDP-3019>`_.
Bitbucket (Forward Location Aggregator, Reverse Location Aggregator, Stock Transfer Adapter)	https://bitbucket.org/morrisonsplc/storepick-osp-stock-transfer-adapter/src

Runtime Issues List¶

Issue	Component	Description
Issue 1	Forward Location Aggregator: stock_transfer_agg.picks_aggregator: process	OSP_STOCK_FORWARD_AGGREGATOR_ERROR_CRITICAL - Unrecoverable error occurred. Task will exit now.
Issue 2	Forward Location Aggregator: stock_transfer_agg.picks_aggregator: _write_to_kafka	OSP_STOCK_FORWARD_AGGREGATOR_ERROR - Rollback in progress.
Issue 3	Reverse Location Aggregator: stock_transfer_agg.picks_aggregator: process	OSP_STOCK_REVERSE_AGGREGATOR_ERROR_CRITICAL - Unrecoverable error occurred. Task will exit now.
Issue 4	Reverse Location Aggregator: stock_transfer_agg.picks_aggregator: _write_to_kafka	OSP_STOCK_REVERSE_AGGREGATOR_ERROR - Rollback in progress.
Issue 5	Stock Transfer Adapter: rms_stock_transfer.rms_stock_transfer_handler: process	OSP_STOCK_TRANSFER_ADAPTER_ERROR_CRITICAL Unrecoverable Error occurred while processing message
Issue 6	Stock Transfer Adapter: rms_stock_transfer.rms_stock_transfer_handler: save_to_dynamodb	OSP_STOCK_TRANSFER_ADAPTER_ERROR Exception while saving records into DynamoDB table
Issue 7	Stock Transfer Adapter: rms_stock_transfer.rms_stock_transfer_handler: _post_stock_transfer	OSP_STOCK_TRANSFER_ADAPTER_ERROR Exception while Posting to Stock transfer Service
Issue 8	Stock Transfer Adapter: rms_stock_transfer.config: ApplicationKafkaConfig	MoriSSMException - SSM is missing values. OSP_STOCK_TRANSFER_ADAPTER_ERROR Exception while Posting to Stock transfer Service

Runtime Issues¶

Issue 1¶

Scenario¶

The Forward Location Aggregator ECS task errors and some non recoverable error occurs. The task will exit and a new task should start.

Error¶

Exception raised due to commit and abort transaction of Kafka are part of this. Any data related and config issues are also part of this error.

Symptoms¶

From checking the CloudWatch logs you should see a similar log within traceback:

|   OSP_STOCK_FORWARD_AGGREGATOR_ERROR_CRITICAL

Alarm will also be raised as soon as this error occurs.

All kafka related Errors and Exception are part of this https://docs.confluent.io/platform/current/clients/confluent-kafka-python/html/index.html#kafkaerror

Corrective Action¶

For Kafka commit related failures: Commit failed: Local: No offset stored kafka - Normally, this is a intermittent error and no action is required. Check the min offsets and max offsets and also check for any 3rd party errors. Scan the logs, if there are 3rd party errors and the messages from kafka are not re-processed then these messages need to be reprocessed manually using Kafka command line.

Logs for Mix and Max offsets: (Partition:offset) Min Offsets processed - {1:22} Max Offsets processed - {1:23}

For config related failures: Check the config for stock-transfer-adapter.json and compare it with that in Configuration (from config service). If any fields are obviously missing then add them back with those expected in the PROD environment. If you are not certain of the required PROD values then contact the business with the details. Also check that the config .json file mentioned in the logs is the one expected.

Issue 2¶

Scenario¶

The Forward Location Aggregator ECS task errors when the connecting to kafka / external service is called.

Error¶

OSP_STOCK_FORWARD_AGGREGATOR_ERROR raised and seen in ECS task logs

Symptoms¶

From checking the CloudWatch logs you should see the following log in the traceback:

|   OSP_STOCK_FORWARD_AGGREGATOR_ERROR - Rollback in progress.

Alarm will also be raised if 10 occurrences of this occurs continuously.

Corrective Action¶

If this error occurs task will automatically retry the same offsets until it is successful. If Alarm is raised then this needs to be investigated and fixed.

Issue 3¶

Scenario¶

The Reverse Location Aggregator ECS task errors and some non recoverable error occurs. The task will exit and a new task should start.

Error¶

Exception raised due to commit and abort transaction of Kafka are part of this. Any data related and config issues are also part of this error.

Symptoms¶

From checking the CloudWatch logs you should see a similar log within traceback:

|   OSP_STOCK_REVERSE_AGGREGATOR_ERROR_CRITICAL

Alarm will also be raised as soon as this error occurs.

All kafka related Errors and Exception are part of this https://docs.confluent.io/platform/current/clients/confluent-kafka-python/html/index.html#kafkaerror

Corrective Action¶

Logs for Mix and Max offsets: (Partition:offset) Min Offsets processed - {1:22} Max Offsets processed - {1:23}

Issue 4¶

Scenario¶

The Reverse Location Aggregator ECS task errors when the connecting to kafka / external service is called.

Error¶

OSP_STOCK_REVERSE_AGGREGATOR_ERROR raised and seen in ECS task logs

Symptoms¶

From checking the CloudWatch logs you should see the following log in the traceback:

|   OSP_STOCK_REVERSE_AGGREGATOR_ERROR - Rollback in progress.

Alarm will also be raised if 10 occurrences of this occurs continuously.

Corrective Action¶

If this error occurs task will automatically retry the same offsets until it is successful. If Alarm is raised then this needs to be investigated and fixed.

Issue 5¶

Scenario¶

The Stock Transfer Adapter ECS task errors and some non recoverable error occurs. The task will exit and a new task should start.

Error¶

Exception raised due to commit and abort transaction of Kafka are part of this. Any data related and config issues are also part of this error.

Symptoms¶

From checking the CloudWatch logs you should see a similar log within traceback:

|   OSP_STOCK_TRANSFER_ADAPTER_ERROR_CRITICAL

Alarm will also be raised as soon as this error occurs.

All kafka related Errors and Exception are part of this https://docs.confluent.io/platform/current/clients/confluent-kafka-python/html/index.html#kafkaerror

Corrective Action¶

Logs for Mix and Max offsets: (Partition:offset) Min Offsets processed - {1:22} Max Offsets processed - {1:23}

Issue 6¶

Scenario¶

The message has been process however whilst attempting to save the message to DynamoDB the task fails.

Error¶

Exception raised and seen in ECS task logs and OSP_STOCK_TRANSFER_ADAPTER_ALARM raised.

Symptoms¶

From checking the CloudWatch logs you should see the following log in the traceback:

|   OSP_STOCK_TRANSFER_ADAPTER_ERROR Exception while saving records into DynamoDB table

The OSP_STOCK_TRANSFER_ADAPTER_ERROR Alarm will also be raised to give an alert of this occurrence on 10 occurrences.

Corrective Action¶

Check the config for stock-transfer-adapter.json and compare it with that in Configuration (from config service). Ensure the configuration for DynamoDB present and that the table exists. The record which was not processed will be logged after following log — : Pick Task Before saving to DynamoDB : — and needs to be inserted manually. This error can only occur when DynamoDB is facing an outage or the table is missing.

Issue 7¶

Scenario¶

Whilst attempting to post the pick to stock transfer service the request will fail as the task cannot make a connection to the service.

Error¶

Exception raised and seen in ECS task logs and OSP_STOCK_TRANSFER_ADAPTER_ERROR raised.

Symptoms¶

From checking the CloudWatch logs you should see the following log in the traceback:

|   Exception: OSP_STOCK_TRANSFER_ADAPTER_ERROR Exception while Posting to Stock transfer Service

The OSP_STOCK_TRANSFER_ADAPTER_ERROR ALARM will also be raised to give an alert of this occurrence on 10 occurrences.

Corrective Action¶

In this scenario check the Stock transfer service and ensure it is running correctly. Check the config for stock-transfer-adapter.json and compare it with that in Configuration (from config service) ensuring the configuration for the stock transfer service is correct. The Kafka commits will be reset and the message will be reprocessed until successful.

Issue 8¶

Scenario¶

The Stock Transfer Adapter ECS task is initialised successfully and the stock_transfer_handler.py is called, however, when the _post_stock_transfer function is called, the stock transfer API request fails with an SSM exception.

Error¶

MoriSSMException raised and seen in ECS task logs

Symptoms¶

From checking the CloudWatch logs you should see the following log in the traceback:

|   MoriSSMException: Invalid or missing SSM config