Feat/copy assets

[joshuaunity]

Description

This PR introduces an API that allows users to copy assets from one account to another (or the same account), using the original asset and its children as templates.

##TODO

• Successfully Copy Asset to target asset
• Write a test to confirm the above feature
• Write a test to confirm that asset parents and children are accurate
• Successfully copy assets, asset sensors, and children to the target asset
• Write a test to confirm the above feature
• Successfully copy asset flexconfig
• Write a test to confirm the above feature

Look & Feel

None

How to test

1. Pick an asset you'd like to copy
2. Using the new copy API (/assets/<asset_id>/copy). Put in your destination account (the account you want to copy to) and your parent asset (the asset you want to copy over to)—all as query params. See Swagger docs for more details on the API.
3. Confirm that the asset (and its flex-config with graphs), as well as its children (if any), are copied correctly.
4. Confirm Graph and flex-config sensor references are properly mapped to their new sensors

Further Improvements

None

Related Items

This PR closes #1966

Sign-off

• I agree to contribute to the project under Apache 2 License.
• To the best of my knowledge, the proposed patch is not based on code under GPL or other license that is incompatible with FlexMeasures

[joshuaunity]

CC @nhoening

[read-the-docs-community]

Documentation build overview

📚 flexmeasures | 🛠️ Build #32254162 | 📁 Comparing 14cb654 against latest (7dee659)

  🔍 Preview build  

Show files changed (10 files in total): 📝 9 modified | ➕ 0 added | ➖ 1 deleted

File	Status
changelog.html	📝 modified
genindex.html	📝 modified
http-routingtable.html	📝 modified
index.html	📝 modified
_autosummary/flexmeasures.api.common.utils.api_utils.html	📝 modified
_autosummary/flexmeasures.api.v3_0.assets.html	📝 modified
_autosummary/flexmeasures.data.services.data_sources.html	📝 modified
api/v3_0.html	📝 modified
dev/scripting.html	➖ deleted
host/installation.html	📝 modified

[Flix6x]

Thank you for requesting a review. It looks like the scope of this PR may be mistaken? You are creating an endpoint to copy all assets under a given account*, rather than an endpoint to copy a given asset, including all assets under it.

[nhoening]

We can also discuss the test strategy. Joshua chose to implement a part of the solution, then tests for that, then the next part, test for that etc.

I believe we suggested initially to have one test case that tests a complete successful copy, then work on making it pass.

• The test can assert the first part first (e.g. assets and subassets are copied), and others later(sensors, flex-config are copied), so you can still follow along how far you are in kaking the test pass, layer by layer.
• This way would be more helpful in understanding together if the scope is agreed between all of us. The test is easier to write (I suggest using AI to help) and to read than the actual implementation.

[Flix6x]

Thanks @joshuaunity this is now heading in the right direction. See my comments for the next challenge: a deep copy. Just let me know if any decisions come up.

[Flix6x]

A few comments on the schema. I'll move to testing now.

[Flix6x]

I was hoping to use Swagger for testing, but I don't see the new endpoint listed on http://localhost:5000/api/v3_0/docs/. I'm delaying testing until I can use that. If you need me to test without it, please provide instructions.

[Copilot]

@copilot please fix the failing pipeline on this PR branch. Do not make a new PR. Then, review this PR.

Fixed in commit 3b8dd82. The CI failure was caused by unused Annotation and get_or_create_annotation imports (F401 flake8 errors). These were unused because the post_annotation function that uses them was accidentally dropped during the merge at commit 4e84d1f. I've restored the function, and all pre-commit hooks (flake8, black, mypy, OpenAPI spec generation) now pass.

[Flix6x]

While testing, a lot works very well now. A few things left:

• Regression: on opening the Edit Graphs modal, I get toast notifications showing "Unknown graph format: 124". It doesn't happen on main. Can you reproduce it?
• I also had copilot fix the failing pipeline, but I still see the POST annotations endpoint being deleted from openapi-specs.json in this PR. I don't know why, please investigate.
• When copying an asset twice, the second copy gets the same name as the first copy. I think subsequent copies should get a number, so:
  • Home
  • Home (Copy)
  • Home (Copy 2)
  • Home (Copy 3)
  • etc.
• Please update the PR description.
• Missing changelog entry.

[joshuaunity]

Regression: on opening the Edit Graphs modal, I get toast notifications showing "Unknown graph format: 124". It doesn't happen on main. Can you reproduce it?

What does your sensors_to_show look like on your DB

[Flix6x]

(I don't know how feasible this would be yet, maybe 80% ?)

Are we now at 100% or are there still cases where you expect copying flex-config to fail?

[Flix6x]

Regression: on opening the Edit Graphs modal, I get toast notifications showing "Unknown graph format: 124". It doesn't happen on main. Can you reproduce it?

What does your sensors_to_show look like on your DB

> select sensors_to_show from generic_asset where id=46;
   sensors_to_show    
----------------------
 [121, 122, 123, 124]

[joshuaunity]

Regression: on opening the Edit Graphs modal, I get toast notifications showing "Unknown graph format: 124". It doesn't happen on main. Can you reproduce it?

What does your sensors_to_show look like on your DB

> select sensors_to_show from generic_asset where id=46;
   sensors_to_show    
----------------------
 [121, 122, 123, 124]

thanks

[joshuaunity]

Are we now at 100% or are there still cases where you expect copying flex-config to fail?

we are at 100% i would say, as of my tests on Friday, flex config was copied over nicely.

[joshuaunity]

Regression: on opening the Edit Graphs modal, I get toast notifications showing "Unknown graph format: 124".

This is now fixed.

@Flix6x

[Flix6x]

Two observations from testing:

• I found an interesting case that seems to hang the server: setting the parent ID to the asset ID being copied.
• The ID of a new asset (leaf node in an asset tree) seems to jump up by several thousand. I don't understand why. Isn't the next available integer chosen?

Are we now at 100% or are there still cases where you expect copying flex-config to fail?

we are at 100% i would say, as of my tests on Friday, flex config was copied over nicely.

Why don't you update the PR description?

[joshuaunity]

Why don't you update the PR description?

I see, I forgot to add the "how to test". Sorry about that

[joshuaunity]

I found an interesting case that seems to hang the server: setting the parent ID to the asset ID being copied.

Nice find. The problem is a recursion bug. Here is the issue.

When copying an asset, the function first creates and flushes the new copied asset under the requested target parent. After that, it queries for all children of the source asset to continue recursive copying.

If the target parent is the same asset being copied (for example, copy asset 5 to parent 5), the just-created copy now appears in that child query.

The recursion then treats this new copy as another source child to copy, creates another copy, and repeats indefinitely, causing a recursion loop until Python raises RecursionError.

The first thing that pops into my head is to add validation to prevent this. What do you think?

@Flix6x

[joshuaunity]

The ID of a new asset (leaf node in an asset tree) seems to jump up by several thousand. I don't understand why. Isn't the next available integer chosen?

After multiple tests, I noticed it, but I can't pinpoint a part of the code that can do this, except for the "flush" part. The logic can inflate the sensor sequence, especially during recursive copies or failed recursion loops.

Now, maybe it was caused by the recent bug you caught, and at 'flush time," when the endless loop was running, thousands of DIs were consumed.

Use the below to reset your sensor and generic_asset IDs and test again

SELECT setval('sensor_id_seq', , false);
SELECT setval('generic_asset_id_seq', , false);

@Flix6x

[Flix6x]

Thanks, I'll test with that.

And adding validation to prevent the recursion sounds like a good plan. I did not yet test setting the parent to a child of the original asset, but I would imagine that would lead to the same kind of recursion.

[joshuaunity]

Thanks, I'll test with that.

And adding validation to prevent the recursion sounds like a good plan. I did not yet test setting the parent to a child of the original asset, but I would imagine that would lead to the same kind of recursion.

Ill also test that case

[joshuaunity]

And adding validation to prevent the recursion sounds like a good plan. I did not yet test setting the parent to a child of the original asset, but I would imagine that would lead to the same kind of recursion.

This has now been implemented