Colleague Handler Runbook¶

Overview¶

This is the runbook for Storepick Colleague Handler repository and its lambdas. This documents (and references) the component documentation. Below are a set of scenarios that may happen in production along with diagnosis and resolution information.

Status: Datadog Metrics Dashboard

References¶

Document	Ref
Architecture Document	Solution Architecture.
Jira Epic	Colleague Leavers Handler: SPDP-2613.

Runtime Issues List¶

Issue	Component	Related Error Codes in logs	Description
Issue 1	Create OSP Shell Account	404	Customer does not exist in Morrisons Customer v2
Issue 2	Create OSP Shell Account	400	Error while adding customer to OSP Customer API
Issue 3	Create OSP Shell Account	500	Empty or missing key
Issue 4	Colleague Leavers Handler	404	Customer could not be deleted from Discount v1
Issue 5	Ocean File Generator	500 or other	Not able to connect to GCP bucket
Issue 6	Ocean File Generator	500 or other	No file (with specific prefix) found in Kevin a/c
Issue 7	Ocean File Generator	500 or other	More than 1 file (with specific prefix) found in Kevin a/c
Issue 8	Process Ocean events	500 or other	The source_system_customer_id is null or empty for the Morrisons customer

Runtime Issues¶

Issue 1¶

Component¶

Create OSP Shell Account

Scenario¶

The customer to be added to OSP is not a valid Morrisons customer.

The Customer v2 API returns 404 not found for the passed customerId.

Error¶

“MoriException”

Symptoms¶

The Lambda will return following response. Exception will not be raised due to the use of build_response_or_throw utility from pymorri.

Example error response

{
  'eventData': {
      'customerId': '1014860340',
      'groupId': '1111-2222'
  },
  'config_file': 'storepick.json',
  'bq_date': '2020-11-03T16:58:53.265025',
  'club_id': '6',
  'club_name': 'Colleague',
  'DATADOG_APP_NAME': 'storepick_create_shell_account',
  'DD_CHECK_METRIC_NAME': 'create_shell_ac_check',
  'response_message': 'Morrisons customer not found for id 1014860340'
}

Corrective Action¶

Make sure the Customer is a valid one using Customer v2. If the customer has been Deleted from Customer API it needs to be removed from OCEAN as well.

Make sure Leavers process is working correctly.

Issue 2¶

Component¶

Create OSP Shell Account

Scenario¶

The customer cannot be added to OSP as the email id being used already exists.

The OSP Customer POST endpoint returns 400.

Error¶

“MoriException”

Symptoms¶

The Lambda will return following response. Exception will not be raised due to the use of build_response_or_throw utility from pymorri.

Example error response

{
  'eventData': {
      'customerId': '1014860340',
      'groupId': '1111-2222'
  },
  'config_file': 'storepick.json',
  'bq_date': '2020-11-03T16:58:53.265025',
  'club_id': '6',
  'club_name': 'Colleague',
  'DATADOG_APP_NAME': 'storepick_create_shell_account',
  'DD_CHECK_METRIC_NAME': 'create_shell_ac_check',
  'response_message': 'Error creating shell account for 1014860340'
}

Corrective Action¶

Make sure the email id in OSP Customer API is up to date. It might be the case that Customer has updated his email id but is not reflected correctly in OSP.

Check the email and customer mapping in Morrisons Customer v2 and compare with OSP Customer.

Issue 3¶

Component¶

Create OSP Shell Account

Scenario¶

The customer cannot be added to OSP as it is missing required name values.

Error¶

“MoriConfigurationException”

Symptoms¶

The Lambda will return following response, which should be seen within the logs for create shell account.

Example error response :: | MoriConfigurationException: Empty or missing key ‘firstName’ in dict in file: /var/task/app/lamdas/create)shell_account.py or | MoriConfigurationException: Empty or missing key ‘lastName’ in dict in file: /var/task/app/lamdas/create)shell_account.py:106

Corrective Action¶

This is a data input issue, first we need to know the customer first and last name, then update the firstName and lastName fields to the correct values in the user’s Morrisons account using the customer API. Then when this is run again, it should be successful. However, if these names cannot be found, the business should be contacted with all the information, as the OSP account cannot be created without these names.

Issue 4¶

Component¶

Colleague Leavers Handler

Scenario¶

The Discount v1 DELETE endpoint throws error that the customer cannot be deleted.

Error¶

“MoriException”

Symptoms¶

Log files for discount api contain the following lines within the boundaries of a single request:

|    INFO:handlers.colleague_discount_handler:Record not found for employee_id... 1112223456
|   {'statusCode': 404, 'headers': {'Content-Type': 'application/json', 'Access-Control-Allow-Origin': '*'}, 'body': '{"status": 6104, "message": "Customer not found", "body": []}'}

Or:

|   INFO:handlers.colleague_discount_handler:Error deleting record from club service MoriException: exception_code: 1500; detail: MoriException: Unexpected error or exception occured on the server. Please check server logs; http_status_code: 404
|   WARNING:handlers.colleague_discount_handler:HTTP 404 - customer id related to employee id 1112223456 not found
|   {'statusCode': 404, 'headers': {'Content-Type': 'application/json', 'Access-Control-Allow-Origin': '*'}, 'body': '{"status": 6104, "message": "Customer not found", "body": []}'}

Corrective Action¶

Colleague registration was previously successful for club service but an entry was not written to dynamoDB table. Raise a change and add the entry to the dynamoDB that is for the {employeeID} that matches {customer_number}, along with the relevant {additionalMembers} (if this information of additional members is not available then do NOT add to the dynamoDB table and contact the business for these details). Check the Datadog dashboard to understand how common this issue is.

Issue 5¶

Component¶

Ocean File Generator

Scenario¶

Not able to connect to OCEAN’s GCP bucket due to one of following reasons related to the Credentials.

GOOGLE_APPLICATION_CREDENTIALS not found in ssm

GOOGLE_APPLICATION_CREDENTIALS is not valid json

GOOGLE_APPLICATION_CREDENTIALS is not valid

Error¶

“Exception”

Symptoms¶

Log files would contain error messages like below:

|    FATAL:GOOGLE_APPLICATION_CREDENTIALS not found in ssm
|    GOOGLE_APPLICATION_CREDENTIALS not found in ssm

Or:

|    GOOGLE_APPLICATION_CREDENTIALS is not valid, it is missing one or more of the required properties
|    Missing: ....

Or:

|    GOOGLE_APPLICATION_CREDENTIALS is not valid json, the service account file should be stored as a string with valid json structure.
|    Missing: ....

Corrective Action¶

Make sure the SSM key has correct structure. It contains “GOOGLE_CREDENTIALS” and that they are in correct Json format. Refer SSM Key

If the credentials exist but do not work then probably needs to be updated or Access level on service account needs to be checked.

Issue 6¶

Component¶

Ocean File Generator

Scenario¶

Could not find a file with the specified prefix, in the Kevin a/c S3 bucket

Error¶

MoriException: file not found in Kevin account S3 bucket

Symptoms¶

({source_file_prefix} will be the value of “S3_SOURCE_FILE_PREFIX” with a timestamp suffix, see: Configuration(from config service)).

Log files for Ocean File Generator contain the following lines within the boundaries of a single request:

|    ERROR:handlers.colleague_discount_handler:file with prefix {source_file_prefix} not found in Kevin account S3 bucket

Or:

|   MoriException: file not found in Kevin account S3 bucket

Corrective Action¶

Further investigate the logs to see if the file was added in the S3 bucket in Kevin Account, and that the config values are correct so that it is looking in the correct place. If the file was added in the wrong place, then raise a change to correct the config and then add the file to the correct location. If config is correct, then check the code for any potential error there and inform the business of any useful information found.

Issue 7¶

Component¶

Ocean File Generator

Scenario¶

Multiple files were found with the specified prefix in the Kevin a/c S3 bucket

Error¶

MoriException: More than 1 file found in Kevin account S3 bucket

Symptoms¶

({source_file_prefix} will be the value of “S3_SOURCE_FILE_PREFIX” with a timestamp suffix, see: Configuration(from config service)).

Log files for Ocean File Generator contain the following lines within the boundaries of a single request:

|    ERROR:handlers.colleague_discount_handler:Multiple files with prefix {source_file_prefix} found in kevin account

Or:

|   MoriException: More than 1 file found in Kevin account S3 bucket

Corrective Action¶

This scenario should not technically happen as the file suffix is based on the current timestamp, therefore if this scenario occurs it is likely due to a code error. If logs for this error are found, investigate config and further logs for more details and read through the code to see if you can find the cause. Contact the business with any useful information you find.

Issue 8¶

Component¶

Process Ocean events

Scenario¶

The source_system_customer_id is null or empty for the Morrisons customer

Error¶

“MoriException”

Symptoms¶

The Lambda will return following response. MoriException is raised.

Example error response

{
 'status': 500,
 'message': 'Error',
 'results': {
             'error': 'MoriException: exception_code: 1500; detail: MoriException: source_system_cust_id is null for customer_id 1025974240; http_status_code: 500'
            },
 'headers': {
             'content-type': 'application/json'
            }
}

Corrective Action¶

A manual record needs to be created or deleted in the prd.xxwmm.storepick.ocean_colleague_discount dynamodb table with the following detail. Proceed only when the correct source_system_customer_id is available.

for event_type = addCustomerToOcean db statement Item={ “customerID”: {“S”: source_system_cust_id}, }

for event_type = removeCustomerFromOcean db statement Key={ “customerID”: {“S”: source_system_cust_id}, }