OR Return Processor Service Runbook

Overview

This is the runbook for OR Return Processor Service which covers below given components.

  1. Read returns data from kafka server.

  2. Return Processor stores data to return Product RDS Table.

  3. Calls Product V1 Service to get the barcode, description and catchweight information.

This documents (and references) the component documentation. Below are a set of scenarios that may happen in production along with diagnosis and resolution information.

This component will generate cloudwatch alarm for any ERROR or EXCEPTION. Prefix ONLINE_RETURN_PROCESSOR_ERROR, ONLINE_RETURN_PROCESSOR_DB_ERROR and ONLINE_RETURN_PROCESSOR_API_ERROR has been configured to raise alarm for common error, DB error and API call error respectively.

Reference

Docuement

Ref

Jira Story (OR Retrun Processor Service)

https://morrisons.atlassian.net/browse/SPDP-5943

Bitbucket (OR Retrun Processor Service)

https://bitbucket.org/morrisonsplc/storepick-online-returns-return-processor/src/master/

Runtime Issue List

Issue

component

Description

Issue 1

Service Call Access Failure

Failed to Access Price V2, Product V1 API it raises MoriException.

Issue 2

Data Inconsistent in Database

RDS session is down or DB table columns missing it raises Exception.

Issue 3

ApplicationDependencies

Exception while loading storepick-online-returns.json config.

Issue 4

Kafka Connection refused

Kafka Connection refused.

Issue 5

Incorrect Config details

Exception when details given in input data is not valid.

Issue 6

Invalid kafka message

Exception while kafka read list object has no attribute.

Runtime Issues

Issue 1 (P2)

Scenario

  • If the request failed to access Price v2, Product V1 API it raises MoriException.

Error

  • MoriException
    _images/Processor_Product_V1_Error.PNG

Symptoms

  • Price V2, Product V1 service will respond with response code 400 / 500.

Corrective Action

  • Check if price V2, product V1 Service is up and running.

  • Check both endpoint URLs are configurred correctly in config file storepick-online-returns.json.

  • Check SSM keys storepick_product_service_secret and storepick_price_service_secret has proper URL credentials.

Issue 2 (P1)

Scenario

  • If RDS session is down it raises Exception.

Error

  • sqlalchemy.exc.OperationalError
    _images/Processor_DB_Failed.PNG

Symptoms

  • Raise ERROR:ecstask.return_processor.service.return_processor_service: ONLINE_RETURN_PROCESSOR_ERROR - Failed to fetch records and aggregate.

Corrective Action

  • Check Postgres is up and username, password, databaseName is correct in SSM Key storepick_online_returns_secret.

  • check if Return product and Online Product table in RDS has valid data.

Issue 3 (P1)

Scenario

  • This happens when there is an error while loading all the dependencies needed for the application to run.

Error

  • Exception raised when loading dependencies.

Symptoms

  • Config service will respond with response code 404.

Corrective Action

  • Check config file name (storepick-online-returns.json) is correct.

  • Check if config fields are present in the config file.

Issue 4 (P1)

Scenario

  • This happens when incorrect kafka server name or topic name has configured in config file.

Error

  • %3|1641798785.484|FAIL|rdkafka#consumer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT, 30 identical error(s) suppressed)

Symptoms

  • Kafka connection refused

Corrective Action

  • Check kafka server and topic name are configurred correctly in config file storepick-online-returns.json.

Issue 5 (P1)

Scenario

  • This happens when the config details in the input is not correct.

Error

  • TypeError: ‘NoneType’ object is not subscriptable.

Symptoms

  • Raise Error pymorri.util.serverless_utils:resolve_env - input data is not: str, list or dict.

Corrective Action

  • Check config file name (storepick-online-returns.json) has correct data fields.

  • Check if config fields are having input value with correct details.

Issue 6 (P2)

Scenario

  • This happens when the kafka message is not correct.

Error

  • Invalid Kafka message format.
    _images/Invalid_kafka_Message.PNG

Symptoms

  • Raise Error pymorri.util.kafka.reader:Function reset_all_topics_to_previous_commit started with arguments.

Corrective Action

  • Kafka Message format should be like this

Value:

{

    "orderId": "453",
    "item": "98765496",
    "locationId": "132",
    "transactionDate": "2021-12-30T00:00:00",
    "quantity": 1,
    "routeName": "route-157043"
}