Dataplattform: Developer Guide

Filoppsett:

• requirements-test.txt pip avhengigheter kun brukes i tester

• requirements.txt pip avhengigheter

• serverless.yml Serverless konfigurasjon

• test_mylambda.py Tester

• /lib/service_name/mylambda.py modul med lambda handler entrypoint

tillegg:

• pytest.ini test konfigurasjon

• package.json npm manifest for serverless plugin avhengigheter

• tox.ini Konfigurering av tox for separare virtuelle miljø brukt i test

• setup.py Trengs for å definere pakke for tox

Fellespakker

Se packages

Github actions

For å kjøre automatiske tester i github så må du også opprette en ny github action workflow under .github/workflows

name: Dataplattform - poller - myPoller

on: 
  pull_request:
    paths:
      - 'services/ingestion/poller/myPoller/**'
      - 'packages/common/**'


jobs:
  lint:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2
    - name: Set up Python 3.7
      uses: actions/setup-python@v1
      with:
        python-version: 3.7
    - name: Install dependencies
      run: |
        python -m pip install --upgrade pip
    - name: Lint with flake8
      run: |
        pip install flake8
        # stop the build if there are Python syntax errors or undefined names
        flake8 ./services/ingestion/poller/myPoller --count --select=E9,F63,F7,F82 --show-source --statistics
        flake8 ./services/ingestion/poller/myPoller --count --max-line-length=120 --statistics
  
  test:
    runs-on: ubuntu-latest
    container:
      image: lambci/lambda:build-python3.7

    steps:
    - uses: actions/checkout@v2
    - name: Install dependencies
      working-directory: './services/ingestion/poller/myPoller'
      run: |
        python -m pip install --upgrade pip
        pip install -r requirements.txt
        pip install -r requirements-test.txt
    - name: Test with pytest
      working-directory: './services/ingestion/poller/myPoller'
      run: |
        pip install pytest
        pytest 

Boilerplate

requirements.txt

file:../../../../packages/common

requirements-test.txt

file:../../../../packages/testing

mylambda.py

from dataplattform.common.handler import Handler
from dataplattform.common.schema import Data, Metadata
from datetime import datetime
from typing import Dict


handler = Handler()


@handler.validate()
def validate(event) -> bool:
    #  Validate event, used in webhooks to validate caller
    return True


@handler.ingest()
def ingest(event) -> Data:
    #  get data from the event or some external source
    some_data = get_data_from_some_source()

    return Data(
        metadata=Metadata(
            timestamp=datetime.now().timestamp()
        ),
        data=some_data
    )


@handler.process(partitions=['a', 'b'])
def process(data) -> Dict[str, pd.DataFrame]:
    # process the raw data into a pandas dataframe (tabular data) 
    df = convert_raw_data_to_dataframe(data)

    return {
        # name of table to append the data to
        'my_table_name': df
    }

test_mylambda.py

from mylambda import handler

def test_a_test():
    assert True

serverless.yaml

TODO