Miljødata - Data Sync

This repo is used to create scripts for fetching data from different services. The common denominator is that the various data is being fetched with the Python library dlt which simplifies the process of downloading data from RestAPI's.

This CLI-tool is used in github.com/NINANor/miljodata-tasks.

The scripts are CLI-based, and the usage can be understood by typing:

uv run datasync --help

For example to understand how the CLI works for fetching data from ubw you can do:

uv run datasync ubw --help

Environment variables

To set up the correct environment variables a .env.example has been added to the project. Copy this template and rename it to .env and set the values.

Setup

Install uv: https://docs.astral.sh/uv/getting-started/installation/

git init
uv sync --dev
git add .
git commit -m "Initial commit"
uv run pre-commit install # optional

Run

To execute your software you have two options:

Option 1: Direct execution

uv run main.py --help

Option 2: Run as installed package

uvx --from . datasync

Development

Just run uv run main.py and you are good to go!

Update from template

To update your project with the latest changes from the template, run:

uvx --with copier-template-extensions copier update --trust

You can keep your previous answers by using:

uvx --with copier-template-extensions copier update --trust --defaults

(Optional) pre-commit

pre-commit is a set of tools that help you ensure code quality. It runs every time you make a commit.

First, install pre-commit:

uv tool install pre-commit

Then install pre-commit hooks:

pre-commit install

To run pre-commit on all files:

pre-commit run --all-files

How to install a package

Run uv add <package-name> to install a package. For example:

uv add requests

Visual studio code

If you are using visual studio code install the recommended extensions

Tools installed

• uv
• pre-commit (optional)

What is an environment variable? and why should I use them?

Environment variables are variables that are not populated in your code but rather in the environment that you are running your code. This is extremely useful mainly for two reasons:

• security, you can share your code without sharing your passwords/credentials
• portability, you can avoid using hard-coded values like file-system paths or folder names

you can place your environment variables in a file called .env, the main.py will read from it. Remember to:

• NEVER commit your .env
• Keep a .env.example file updated with the variables that the software expects