Colleague Discount API Runbook¶

Overview¶

This is the runbook for Colleague Discount v1 API

This documents (and references) the component documentation. Below are a set of scenarios that may happen in production along with diagnosis and resolution information.

Status: Datadog Metrics Dashboard

References¶

Document	Ref
Business Analysis	Business Analysis.
Architecture Document	Solution Architecture.
Jira Epic	Colleague & Customer Discounts: SPDP-2138.

Runtime Issues List¶

Issue	UI Error Code	Related Error Codes in logs	Description
Issue 1	500	409	POST request to Club Service fails as customer already within the Club.
Issue 2	500	404	PUT request to Club Service fails as customer not already within the Club.
Issue 3	400	6104	Payload body or customerId missing from request
Issue 4	400	6104	Number of additionalDiscountMembers greater than the limit
Issue 5	404	404	DELETE request given for a customer that cannot be found in Discount
Issue 6	500	1500 or any others	Any other 500 error not already detailed

Runtime Issues¶

Issue 1¶

Scenario¶

Unable to register from portal. The system is confused and the colleague discount API does not have an entry for the user in the dynamoDB table but the club service believes they are a member of the club. A missing colleague in the dynamoDB table suggests this colleague is not a member of club 6 in club service and we are therefore attempting to create the colleague. In this case the data is mismatched and the call to club service fails as per the log file examples. The important thing here is the “POST”. A 409 conflict error would be expected in this situation as it is trying to create a customer in a club within Club Service that has already been created there.

Error¶

UI reports “Request failed with error code 500”, 409 code found in logs

Symptoms¶

Log files for discount api contain the following lines within the boundaries of a single request:

|   POST https://api.morrisons.com/club/v1/customers/{customer_number}/clubs/6
|   Employee was not already in the table. Creating club membership.
|   {'httpResponseCode': 409, 'errorCode': '404.20.003', 'errorMessage': 'Customer is already a member of this club'}

A single request starts with the entry looking like:

| START RequestId: 999b06fa-16f2-4b37-a816-ee0bf917a357 Version: $LATEST

and ends with the entry looking like:

| END RequestId: 999b06fa-16f2-4b37-a816-ee0bf917a357

The GUID in start and end should be the same

Corrective Action¶

Colleague registration was previously successful for club service but an entry was not written to dynamoDB table for this employee id. Check the logs for the customer id in the payload, this 409 error is expected behaviour if this customer id is already associated with a different employee id, therefore no action would be needed.

If the customer id cannot be found for a different employee id contact the business to explain that a customer is unexpectedly missing from the table but in the Club Service with all the details you have. That employee id may need a DELETE request to the Club Service so that the customer can try register again without this issue. We want a proper registration, otherwise when the colleague leaves the business their discount will not be removed.

Issue 2¶

Scenario¶

Unable to register from portal. The system is confused as a result of previous failures and the colleague discount API has an entry for the user in the dynamoDB table but the club service believes they are NOT a member of the club. The presence of a colleague in the dynamoDB table suggests this colleague is a member of club 6 in club service and we are therefore updating the colleague’s additional members. In this case the data is mismatched and the call to club service fails as per the log file examples. The important thing here is the “PUT”. Distinct from Issue 1 because Issue 1 refers to a “POST” and has different Corrective action.

This issue previously occurred frequently with an older version of this API, where a customerId given in the request body did not match the one associated with the employee ID in the dynamoDB table. However, as shown in the PUT Endpoint Diagram in Colleague Discount Handler, in this scenario we now would replace the existing customerId with the new one provided in the body.

Error¶

UI reports “Request failed with error code 500”

Symptoms¶

Log files for discount api contain the following lines within the boundaries of a single request:

|   PUT https://api.morrisons.com/club/v1/customers/{customer_number}/clubs/6
|   Employee was already in the table. Updating club membership.
|   {'httpResponseCode': 404, 'errorCode': '404.20.003', 'errorMessage': 'Customer is not a member of this club'}

A single request starts with the entry:

| START RequestId: 999b06fa-16f2-4b37-a816-ee0bf917a357 Version: $LATEST

and ends with the entry:

| END RequestId: 999b06fa-16f2-4b37-a816-ee0bf917a357

Corrective Action¶

Colleague registration failed for club service but an entry was written to dynamoDB table. Check the frequency of the error with this employee id, and investigate the cause of the failure in Club Service. A potential fix would be to raise a change and remove the entry from dynamoDB that is for that {employeeID}, therefore when the colleague attempts this again, it should create the new entry and attempt to add to CLub Service. If the colleague attempts again and the error consistently occurs then investigate further with the Club Service and contact the business.

Issue 3¶

Scenario¶

Unable to register from portal. The wrong payload is given to the API.

Error¶

UI reports “Request failed with error code 400”

Symptoms¶

Log files for discount api contain the following lines within the boundaries of a single request:

|   ERROR:handlers.colleague_discount_handler:The supplied mandatory field (customerId) is missing, invalid or blank.
|   {'status': 6104, 'message': 'The supplied mandatory field (customerId) is missing, invalid or blank.', 'body': []}

Similar issue could be found with missing body of the payload instead of missing customerId.

Corrective Action¶

Check the Datadog dashboard (400 Bad Request errors such as these would be sent to the colleague_register_invalid_count metric) as given at the top of this page as well as the CloudWatch logs, to see how common this issue is. Contact the business, specifically mentioning the team responsible for the Front-End of the discount portal as it is very likely an issue with the Front-End code which is not sending either the payload body or customer ID to the API.

Issue 4¶

Scenario¶

Unable to register from portal. The additionalDiscountMember limit is exceeded due to mismatch between front end and API code.

Error¶

UI reports “Request failed with error code 400”

Symptoms¶

Log files for discount api contain the following lines within the boundaries of a single request:

|   ERROR:handlers.colleague_discount_handler:The number of 'additionalDiscountMembers' was 2, which is greater than the limit of 1
|   {'status': 6104, 'message': "The number of 'additionalDiscountMembers' was 2, which is greater than the limit of 1", 'body': []}

Corrective Action¶

Contact the business, specifically mentioning the team responsible for the Front-End of the discount portal as it is very likely an issue with the Front-End code which is allowing more additional discount members than specified in the code.

Issue 5¶

Scenario¶

Unable to delete colleague from portal. System is confused as it is given a request to delete an employee/customer that is not found within the dynamodb table, therefore it cannot be deleted as it’s not found.

Error¶

UI reports “Request failed with error code 404”

Symptoms¶

Log files for discount api contain the following lines within the boundaries of a single request:

|    INFO:handlers.colleague_discount_handler:Record not found for employee_id... 1112223456
|   {'statusCode': 404, 'headers': {'Content-Type': 'application/json', 'Access-Control-Allow-Origin': '*'}, 'body': '{"status": 6104, "message": "Customer not found", "body": []}'}

Or:

|   INFO:handlers.colleague_discount_handler:Error deleting record from club service MoriException: exception_code: 1500; detail: MoriException: Unexpected error or exception occured on the server. Please check server logs; http_status_code: 404
|   WARNING:handlers.colleague_discount_handler:HTTP 404 - customer id related to employee id 1112223456 not found
|   {'statusCode': 404, 'headers': {'Content-Type': 'application/json', 'Access-Control-Allow-Origin': '*'}, 'body': '{"status": 6104, "message": "Customer not found", "body": []}'}

Corrective Action¶

Check the logs for the history of this Employee ID within Discount API, if this employee was not previously created with a PUT request then no corrective action is required as this would be expected - meaning that the colleague has not previously applied for a Colleague Discount online, in most cases this is the cause.

However, if the employee ID given was previously created and attempted to add to dynamodb, then further investigation is required as to why it could not be found. The situation where we would need action would be if the colleague was added to Club Service but not to the dynamoDB table, therefore if the DELETE fails as in this case then the colleague would continue to have the Colleague Discount after leaving the company. If this is the case indicated by the logs, then the business should be informed and a DELETE request for the customer id should be sent to Club Service.

Issue 6¶

Scenario¶

Unable to process on portal, due to an unexpected error.

Error¶

UI reports “Request failed with error code 500”

Symptoms¶

Log files for discount api contain the following lines (or other errors not mentioned in the previous scenarios) within the boundaries of a single request:

|   ERROR:handlers.colleague_discount_handler: exception_code: 1500; detail: 'Unexpected error or exception occured on the server. Please check server logs'; http_status_code: 500"

Corrective Action¶

Check the Datadog dashboard (500 errors such as these would be sent to the colleague_register_fail_count metric) as given at the top of this page as well as the CloudWatch logs, to see how common this issue is. Investigate the logs, config file (checking if any values are missing or incorrect based on expected config) and code to see if you can determine the issue, then contact the business if no obvious fix or details are found, with any information you have.