Skip to content

perf: use atlite streaming#2135

Closed
FabianHofmann wants to merge 6 commits intomasterfrom
feat/atlite-streaming
Closed

perf: use atlite streaming#2135
FabianHofmann wants to merge 6 commits intomasterfrom
feat/atlite-streaming

Conversation

@FabianHofmann
Copy link
Copy Markdown
Contributor

Use atlite's faster matrix calculation and streaming backend for converison proposed in PyPSA/atlite#497. At this stage this is still experimental and serves as a reference for benefits of a new backend in conversion functions.

Performance gains for matrix calculation and renewable profiles:

22:10 min -> 4:20 min

I verified that the exact results are reproduced with this pr (max numerical discrepancies are < 0.001 rtol)

Other changes

  • remove all logging from determine_availability_matrix and build_renewable_profiles to not spoil the terminal log (parallel execution makes them non-readable)
  • use 1 core per atlite-based rule (config default)

Checklist

Required:

  • Changes are tested locally and behave as expected.
  • Code and workflow changes are documented.
  • A release note entry is added to doc/release_notes.rst.

If applicable:

  • Changes in configuration options are reflected in scripts/lib/validation.
  • For new data sources or versions, these instructions have been followed.
  • New rules are documented in the appropriate doc/*.rst files.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant