Picktask-Order Writer Service Runbook

Overview

This is the runbook for Storepick WOS Order Writer Service component. This documents (and references) the component documentation. Below are a set of scenarios that may happen in production along with diagnosis and resolution information.

Status: Datadog Metrics Dashboard

Runtime Issues List

Issues	Related Error Codes in logs	Description
Issue 1		The bootstrap server name is incorrect.
Issue 2		The Kafka Topic name is incorrect
Issue 3	500	DynamoDB Requested resource not found

Runtime Issues

Issue 1

Component

Wholesale Order Writer Service

Scenario

Incorrect bootstrap server name in storepick-wos-aggregator.json config

Error

“Connection timed out”

Symptoms

The log files for Wholesale Order Writer Service would have an error message similar to below.

Example error:

%3|1623924872.207|FAIL|rdkafka#consumer-1| [thrd:ssl://b-1.picktask-test.k8bixh.c5.kafka.eu-west-1.amazonaws.com]:
ssl://b-1.picktask-test.k8bixh.c5.kafka.eu-west-1.amazonaws.com:9094/bootstrap:
Connect to ipv4#10.250.192.247:9094 failed: Connection timed out
(after 32153ms in state CONNECT, 1 identical error(s) suppressed)

Corrective Actions

Check the bootstrap server name in storepick-wos-aggregator.json config

Issue 2

Component

Wholesale Order Writer Service

Scenario

Incorrect kafka topic in storepick-wos-aggregator.json config

Error

“KafkaError”

Symptoms

The log files for Wholesale Order Writer Service would have an error message similar to below.

Example error:

[ERROR] (MainThread) kafka Topics ['picktasks-new123'] does not exist.
Traceback (most recent call last):
File "/app/ecstask/wos_order_writer/order_writer.py", line 85, in process
  raise exc
File "/app/ecstask/wos_order_writer/order_writer.py", line 47, in process
  for order_data in kafka_consumer.read():
File "/app/pymorri/util/kafka/reader.py", line 42, in read
  for msg in self.read_with_metadata(max_messages=max_messages, timeout=timeout):
File "/app/pymorri/util/kafka/reader.py", line 61, in read_with_metadata
  raise MoriKafkaError(msg.error())
pymorri.util.kafka.model.MoriKafkaError: KafkaError{code=UNKNOWN_TOPIC_OR_PART,val=3,str="Subscribed topic not available: picktasks-new123: Broker: Unknown topic or partition"}
[ERROR] (MainThread)  OSP_WOS_ORD_HNDLR_WRITER_ALARM Exception while proccesing kafka messages

Corrective Action

Check the kafka topic name in storepick-wos-aggregator.json config.

Check kafka topic confiuration in AWS MSK

Issue 3

Component

Wholesale Order Writer Service

Scenario

There are multiple possible reasons like AccessDenied or the Dynamo Table being accessed does not exist.

Error

“http_status_code: 500”

Symptoms

The log files for Wholesale Order Writer Service would have an error message similar to below.

Example error

[ERROR] (MainThread) Table dev.xxwmm.storepick.wos-ord-hdlr-orders123 does not exist.
Traceback (most recent call last):
File "/app/ecstask/wos_order_writer/order_writer.py", line 80, in process
  raise ce
File "/app/ecstask/wos_order_writer/order_writer.py", line 48, in process
  order_item = dynamodb_get_rows(
File "/app/pymorri/util/aws_utils.py", line 689, in dynamodb_get_rows
  response = table.query(KeyConditionExpression=Key(query_key).eq(query_value), **kwargs)
File "/usr/local/lib/python3.9/site-packages/boto3/resources/factory.py", line 520, in do_action
  response = action(self, *args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/boto3/resources/action.py", line 83, in __call__
  response = getattr(parent.meta.client, operation_name)(*args, **params)
File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 357, in _api_call
  return self._make_api_call(operation_name, kwargs)
File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 676, in _make_api_call
  raise error_class(parsed_response, operation_name)
botocore.errorfactory.ResourceNotFoundException: An error occurred (ResourceNotFoundException) when calling the Query operation: Requested resource not found
[ERROR] (MainThread)  OSP_WOS_ORD_HNDLR_WRITER_ALARM Exception while proccesing kafka messages

Corrective Action

Check the storepick-wos-aggregator.json config for AWS DynamoDB table name and path.

Check whether IAM role has permission to access AWS DynamoDB table.