Club Leavers/Joiners Runbook¶

Overview¶

This is the runbook for Club Leavers/Joiners (NHS Discount). This documents (and references) the component documentation. Below are a set of scenarios that may happen in production along with diagnosis and resolution information.

Status: Datadog Metrics Dashboard

References¶

Document	Ref
Architecture Document	Solution Architecture.
Main Jira Story	NHS Discount Workflow: SPDP-2137.

Runtime Issues List¶

Issue	Component	Related Error Codes in logs	Description
Issue 1	Club Leavers Joiners	400	Bad Request on DynamoDB due to credentials or object does not exist.
Issue 2	Step Function Trigger	404	MoriConfigurationException while triggering Step Function.
Issue 3	OSP Add Customers to Group	404	Customer Group does not exist in OSP
Issue 4	OSP Remove Customers from Group	404	Customer/Customer Group does not exist in OSP
Issue 5	Club Leavers Joiners	500	No events returned from Club.
Issue 6	Club Leavers Joiners		New watermark lower than previous.
Issue 7	Club Leavers Joiners		Max no of the events processed has reached.
Issue 8	Club Leavers Joiners	500	Incorrect configuration value for COLLEAGUE_DISCOUNT_ENABLED attribute.
Issue 9	Club Leavers Joiners		Mapping for club_id not found in the config file’s group_mapping.
Issue 10	Club Leavers Joiners		Operation value in Club event not ‘JOIN’ or ‘LEAVE’.
Issue 11	Club Leavers Joiners		Incorrect event_type received from Club events.

Runtime Issues¶

Issue 1¶

Component¶

Club Leavers Joiners

Scenario¶

Lambda fails to connect to Dynamodb table to fetch the watermark.

There are multiple possible reasons like AccessDenied or the Dynamo Table being accessed does not exist.

Error¶

“400 Bad Request”

Symptoms¶

The log files for lambda would have an error message similar to below.

Example error:

HTTP/1.1 400 Bad Request
x-amzn-RequestId: LDM6CJP8RMQ1FHKSC1RBVJFPNVV4KQNSO5AEMF66Q9ASUAAJG
Content-Type: application/x-amz-json-1.0
Content-Length: 240
Date: Thu, 15 Mar 2020 23:56:23 GMT

{"__type":"com.amazonaws.dynamodb.v20120810#ResourceNotFoundException",
"message":"Requested resource not found: Table: tablename not found"}

Corrective Actions¶

Check the access level of AWS account being used to connect to DynamoDB. Check the required Table and column (“Watermark”) exist in the database on the given environment.

Issue 2¶

Component¶

Step Function Trigger

Scenario¶

Lambda fails to trigger the Step function as the config file is missing step_functions_arn_template.

Error¶

“MoriConfigurationException”

Symptoms¶

The log files for lambda would have an error message similar to below.

Example error:

MoriConfigurationException: exception_code: 404; detail: MoriConfigurationException: Empty or missing key 'step_functions_arn_template' in dict in file: storepick-nhs-discount.json
Full Dict: {....};
http_status_code: 404

Corrective Action¶

Make sure the structure of event matches one given here Configuration(from config service)

It should contain “step_functions_arn_template” without which the lambda will not know which Step function needs to be triggered on current environment.

Issue 3¶

Component¶

OSP Add Customers to Group

Scenario¶

The OSP Customer Group PUT endpoint fails with 404 error.

The Customer group in which Customer is being added does not exist.

Error¶

“MoriException”

Symptoms¶

The log files for lambda would have an error message similar to below.

Example error:

MoriException: exception_code: 1500; detail: MoriException: CC-2600 - Banner not found, CC-2000 - Customer group not found. args: {...};
http_status_code: 404

Corrective Action¶

Make sure the Customer group exists using the OSP Customer Group GET endpoint.

Make sure the Club_id and OSP Customer group mapping is correctly provided in the config file. Refer group_mapping attribute in config file.

E.g. Configuration(from config service)

Issue 4¶

Component¶

OSP Remove Customers from Group

Scenario¶

The OSP Customer Group DELETE endpoint fails with 404 error.

The Customer group from which Customer is being deleted does not exist. Else the Customer being deleted does not exist in the given group.

Error¶

“MoriException”

Symptoms¶

The log files for lambda would have an error message similar to below.

Example error:

MoriException: exception_code: 1500; detail: MoriException: CC-2600 - Banner not found, CC-2000 - Customer group not found, CC-2006 - Customer group membership is not found. args: {...};
http_status_code: 404

Corrective Action¶

Customer Group does not exist¶

Make sure the Customer group exists using the OSP Customer Group GET endpoint.

Make sure the Club_id and OSP Customer group mapping is correctly provided in the config file. Refer group_mapping attribute in config file.

E.g. Configuration(from config service)

Customer does not exist¶

In case the CustomerId does not exist then we will have to back trace why it was present in Club service but not OSP service. Check watermark in DynamoDB is working correctly.

It is possible that Customer was deleted in previous run but DynamoDB watermark pulled the old event again from Club service. This case will need investigation on such lines.

Issue 5¶

Component¶

Club Leavers Joiners

Scenario¶

The Club service does not return any Leavers/Joiners events

Error¶

“500 - No events found”

Symptoms¶

The response returned by the lambda will have following information. Also error message will be published to SNS with below details

TopicArn = sns-euw1-prd-digital-storepick-failure
eventType = “failureReportedToSNS”
Message = “No events found in club service”

Example Lambda response

{
   "status": 500,
    "message": "No events found",
    "results":  {"error": "No events found in club service"},
    "headers": {"content-type": "application/json"},
}

Corrective Action¶

Make sure that genuinely there are no events of Leavers/Joiners, also check if watermark in DynamoDB is correct as events are fetched w.r.t the watermark.

Issue 6¶

Component¶

Club Leavers Joiners

Scenario¶

The Club service eventId is lower than the previous watermark stored in DynamoDB.

Processing will stop with ERROR message in log.

Error¶

No Error raised but only logged

Symptoms¶

Below ERROR message will appear in the lambda logs.

Example error

ERROR:root:UNEXPECTED ERROR => stopping processing events as detected new watermark is lower than prev watermark

Corrective Action¶

Make sure new watermark is higher than the previous watermark. There can be some issue in Club service or with update process of DynamoDB. Will need investigation.

Issue 7¶

Component¶

Club Leavers Joiners

Scenario¶

The number of maximum events to be processed in 1 lambda trigger has reached.

Processing will stop with WARNING message in log.

Error¶

No Error raised but only WARNING logged

Symptoms¶

Below WARNING message will appear in the lambda logs.

Example error

WARNING:root:MAX_NUMBER_OF_EVENTS = 6000 reached. Stopping further processing as configured limit exceeded

Corrective Action¶

Reprocessing will be required for remaining events. Make sure watermark is correctly updated before starting next batch process.

Issue 8¶

Component¶

Club Leavers Joiners

Scenario¶

Incorrect configuration value for COLLEAGUE_DISCOUNT_ENABLED attribute.

It should be ‘yes’ or ‘no’, if it is anything else then Exception will be raised.

Error¶

“500 - No events found”

Symptoms¶

The response returned by the lambda will have following information. Also error message will be published to failure SNS sns-euw1-prd-digital-storepick-failure

Example Lambda response

{
   "status": 500,
    "message": "Something bad happened",
    "results": {"error":"Incorrect configuration value set for COLLEAGUE_DISCOUNT_ENABLED attribute"},
    "headers": {"content-type": "application/json"},
}

Corrective Action¶

Correct the COLLEAGUE_DISCOUNT_ENABLED value in config or make sure correct config is being used.

Issue 9¶

Component¶

Club Leavers Joiners

Scenario¶

Mapping for club_id not found in the config file’s group_mapping.

The event will be skipped and processing will continue.

Error¶

No Error raised but only WARNING logged

Symptoms¶

Below WARNING message will appear in the lambda logs.

Example error

WARNING:root:SKIPPING event_id: 221252: mapping for club_id: 2 not found in the config file's 'group_mapping' for customer_id 111111111

Corrective Action¶

If club_id is valid and still missing in the group_mapping then Config should be updated else event will get skipped.

Count of such skipped events (due to this or other reason) will logged and displayed.

Issue 10¶

Component¶

Club Leavers Joiners

Scenario¶

Operation value in Club event not ‘JOIN’ or ‘LEAVE’

The event will be skipped and processing will continue.

Error¶

No Error raised but only WARNING logged

Symptoms¶

Below WARNING message will appear in the lambda logs.

Example error

WARNING:root:SKIPPING event_id: 2252: Operation value was not 'JOIN' or 'LEAVE', but 'FILL' for customer: 223225665

Corrective Action¶

Only events with operation ‘JOIN’ or ‘LEAVE’ should flow in from Club service. Make sure config at Club service side has not changed.

Count of such skipped events (due to this or other reason) will logged and displayed.

Issue 11¶

Component¶

Club Leavers Joiners

Scenario¶

Event_type received from Club service is not in event_type_map

The event will be skipped and processing will continue.

Error¶

No Error raised but only WARNING logged

Symptoms¶

Below WARNING message will appear in the lambda logs.

Example error

WARNING:root:SKIPPING event_id: 2575 for customer: 114565 - event_type for operation JOIN is not set in event_type_map

Corrective Action¶

If the event_type is important then get it added into event_type_map so that such events are processed.

Count of such skipped events (due to this or other reason) will logged and displayed.