Skip to content

README.md

Latest commit

 

History

History
52 lines (29 loc) · 2.61 KB

README.md

File metadata and controls

52 lines (29 loc) · 2.61 KB

gp2gp-enhanced-mi-infrastructure

Setting up a python development environment

To setup the python development environment use the make env command.

This will setup your python virtual environment and install required packages.

Applying terraform

Rolling out terraform against each environment is managed by the GitHub Actions pipeline. The workflow files can be found in .github/workflows.

Manually testing the Lambdas

For more information on testing the Lambdas please see the EMI Lambdas documentation

Degrades Reporting

To collect data around the number and different types of degrade events received from GP2GP Degrades events received by event-enrichment lambda are sent to a Degrades SQS queue which triggers degrades_message_receiver_lambda, which writes these events to a DynamoDB table to be later queried for reports generation. These reports are written to CSV and stored in an S3 bucket, under paths reports/ and reports/daily/, to be consumed by Power BI for the creation of dashboards.

degrades_architecture.svg

As we already have a check in the gp-registrations-mi lambda against whether an event is a degrade event or not, it was decided to use this lambda to send any degrades events to an SQS queue to be processed. The alternative was to read every gp2gp event received each day from S3 to determine its event type, and only store or process degrades found.

Scripts

There are 3 adhoc scripts that have been used to prepopulate the dev environment with historical data held within S3. Each will need environmental variables defined, which can be found in AWS, bucket name, queue name, table name, and region. The terminal they are run from/tool used to run them will also need to be granted permission to be able to interact with these AWS resources.

populate_table.py

Loops through the S3 bucket where enriched gp registration events are stored by date prefix, on dev this can be YYYY/MM as there is relatively little data held there, sending degrade events to a queue to be written to the degrades event dynamo table.

generate_daily_report.py

Reads degrades dynamo table for a given date YYYY-MM-DD to generate a daily summary of this data and store in S3 reports/daily/YYYY-MM-DD.csv

Headers are:

  • Type
  • Reason
  • Count

generate_weekly_report.py

Collects reports for a week, 7 days, starting from date YYYY-MM-DD, summarises the data within these reports and appends new rows to an existing report, creates new report if one does not exist.

Headers are:

  • Week_beginning
  • Type
  • Reason
  • Count