Common Serverless Repo Runbook¶
Overview¶
This is the runbook for Common Serverless Repo
This documents (and references) the component documentation. Below are a set of scenarios that may happen in production along with diagnosis and resolution information.
This is common code referenced by many repos hence there are no dedicated Business Analysis or Architecture diagrams.
Apart from below scenarios more scenarios may be covered in respective repositories where this code has been used. Common errors which can occur are covered here. If error you have faced is not covered here please have a look in runbook of repository where the error occurred.
Runtime Issues¶
Issue 1¶
Component¶
BigQuery update lambda
Scenario¶
Lambda fails to connect to BigQuery due to incorrect access level of Service account being used for the operation. The access might be denied on project/dataset/table.
Error¶
“accessDenied with error code 403”
Symptoms¶
The log files for lambda would have an error message similar to below.
Example error:
{
"error": {
"errors": [
{
"domain": "global",
"reason": "accessDenied",
"message": "Access Denied: Job abcd:job_LyHmZIvlY1_0J8JQ3pxThEBf19I: The user does not have permission to run a query in project ********"
}
],
"code": 403,
"message": "Access Denied: Job abcd:job_LyHmZIvlY1_0J8JQ3pxThEBf19I: The user does not have permission to run a query in project ********"
}
}
Corrective Action¶
Check the access level of service account being used to connect to BigQuery. The service account should have access to target project, dataset, table and to create bq jobs. It should also have access to Read/Write data in the table as required.
Issue 2¶
Component¶
BigQuery update lambda
Scenario¶
Lambda fails to update/insert in the table as the table/column names passed from the event does not exist.
Error¶
“notFound with error code 404”
Symptoms¶
The log files for lambda would have an error message similar to below.
Example error:
{
"error": {
"errors": [
{
"domain": "global",
"reason": "notFound",
"message": "Not Found: Table proj.dataset.table_name"
}
],
"code": 404,
"message": "Not Found: Table proj.dataset.table_name"
}
}
Corrective Action¶
Check that the table name and dataset name provided in the config exists. Check if the structure of table has changed and the column being updated exist or not. Make changes in the query or BigQuery table accordingly after consulting the SME.
Issue 3¶
Component¶
BigQuery update lambda
Scenario¶
The event being passed the the lambda does not have correct structure or information.
The pymorri submodule will raise MoriConfigurationException if the event does not have “config_file” or “bq_update_data”.
Error¶
“MoriConfigurationException”
Symptoms¶
The log files for lambda would have an error message similar to below.
Example error:
MoriConfigurationException: exception_code: 404; detail: MoriConfigurationException: Empty or missing key 'config_file' in dict in file: event
Full Dict: {....};
http_status_code: 404
Corrective Action¶
Make sure the structure of event matches one given here Code
It should contain “config_file” and “bq_update_data” without which the update/insert in BigQuery cannot be carried out.
Issue 4¶
Component¶
Datadog event writer lambda
Scenario¶
One or more of the arguments being passed to the Datadog is Null. We will get an error if any of below argument is passed as Null.
metric_type, metric_value, metric_name, application_name, env, options
Error¶
“MoriException”
Symptoms¶
The log files for lambda would have an error message similar to below.
Example error:
MoriException: exception_code: 1500; detail: MoriException: one of the args provided is None. args: {...};
http_status_code: 500
Corrective Action¶
Make sure the structure of event matches one given here Code
Make sure the SSM being used is valid and as expected SSM Key
All of the below should have valid values for the event to be posted correctly in Datadog dashboard. metric_type, metric_value, metric_name, application_name, env, options
Issue 5¶
Component¶
Datadog event writer lambda
Scenario¶
Metric type or Metric value passed on from the event is not as expected leading the an exception.
Error¶
Custom Exception
Symptoms¶
The log files for lambda would have an error message similar to below.
Example error:
Invalid datadog event value