OR Big Query Writer Runbook

Overview

This is the runbook for OR Big Query Writer Task which covers below given components.

  1. Fetches scanned products with status=TRANSFFERED or (status=FAILED and retry count>=5) and sent_to_biq_query=false and their respective item details from online Product table.

  2. Call GCP Big Query and store all these details in to Big Query Table.

  3. Response from the Big Query Table is used to update sent_to_biq_query column (from false to true) in Scanned Product.

This documents (and references) the component documentation. Below are a set of scenarios that may happen in production along with diagnosis and resolution information.

This component will generate cloudwatch alarm for any ERROR or EXCEPTION. Prefix ONLINE_RETURN_BIQ_QUERY_WRITER_ERROR has been configured to raise alarm for common error or DB error.

Reference

Docuement

Ref

Jira Story (OR Big Query Writer Service)

https://morrisons.atlassian.net/browse/SPDP-6916.

Bitbucket (OR Big Query Writer Service)

https://bitbucket.org/morrisonsplc/storepick-online-returns-big-query-writer/src/master/.

Runtime Issue List

Issue

component

Description

Issue 1

AWS Environment Variables Invalid

Error connecting to SSM using key.

Issue 2

Data Inconsistent in Database

RDS session is down or DB table columns missing it raises Exception.

Issue 3

ApplicationDependencies

Exception while loading storepick-online-returns.json config.

Issue 4

Incorrect Config details

Exception when details given in input data is not valid.

Runtime Issues

Issue 1

Scenario

  • This happens when the security token included while requesting is expired or invalid.

Error

  • pymorri.exceptions.util_exceptions.SSMConnectionError.

Symptoms

  • Task will stop and raise Error connecting to SSM using key: storepick_config_service, http_status_code: 500.

Corrective Action

  • Check AWS credentials with correct values of AWS Access Key Id, AWS Secret access key and AWS session token.

  • The credentials are not expired ones and are refreshed.

Issue 2 (P1)

Scenario

  • If RDS session is down or scanned product data is invalid it raises Exception.

Error

  • sqlalchemy.exc.OperationalError.

Symptoms

  • Task will stop and raise ERROR:ecstask.big_query_writer.service.online_return_bq_writer: ONLINE_RETURN_BIQ_QUERY_WRITER_ERROR Failed to fetch scanned products from scanned table.

Corrective Action

  • Check Postgres is up and username, password, databaseName is correct.

  • Check if Scanned product table in RDS has valid data.

Issue 3 (P1)

Scenario

  • This happens when there is an error while loading all the dependencies needed for the application to run.

Error

  • Exception raised when loading dependencies.

Symptoms

  • Config service will respond with response code 404.

Corrective Action

  • Check config file name (storepick-online-returns.json) is correct.

  • Check if config fields are present in the config file.

Issue 4 (P1)

Scenario

  • This happens when the config details in the input is not correct.

Error

  • TypeError: ‘NoneType’ object is not subscriptable.

Symptoms

  • Task will stop and raise Error pymorri.util.serverless_utils:resolve_env - input data is not: str, list or dict.

Corrective Action

  • Check config file name (storepick-online-returns.json) has correct data fields.

  • Check if config fields are having input value with correct details.