Bulk loading rev#2412
Closed
fabianmurariu wants to merge 28 commits intodb_v4from
Closed
Conversation
fabianmurariu
force-pushed
the
bulk_loading_rev
branch
from
7cb45f3 to
8f1da08
Compare
# Conflicts: # db4-graph/src/lib.rs # db4-storage/src/pages/layer_counter.rs # db4-storage/src/pages/node_store.rs # raphtory-api/src/core/entities/properties/prop/prop_array.rs # raphtory-api/src/core/entities/properties/prop/prop_enum.rs # raphtory-core/src/storage/mod.rs # raphtory-storage/src/graph/graph.rs # raphtory/src/db/api/view/graph.rs # raphtory/src/db/graph/views/deletion_graph.rs # raphtory/src/errors.rs # raphtory/src/io/arrow/df_loaders.rs # raphtory/src/io/parquet_loaders.rs # raphtory/src/search/mod.rs # raphtory/src/serialise/graph_folder.rs # raphtory/src/serialise/parquet/mod.rs # raphtory/src/serialise/serialise.rs # raphtory/src/vectors/db.rs
fabianmurariu
force-pushed
the
bulk_loading_rev
branch
from
dc248bb to
b4a23e4
Compare
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
⚠️ Performance Alert ⚠️
Possible performance regression was detected for benchmark 'Rust Benchmark'.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold 2.
| Benchmark suite | Current: 8cc546d | Previous: db9bce1 | Ratio |
|---|---|---|---|
large/1k fixed edge updates with varying time and numeric string input |
773906 ns/iter (± 11872) |
364880 ns/iter (± 1280) |
2.12 |
large/1k fixed edge updates with varying time and string input |
843654 ns/iter (± 9602) |
418788 ns/iter (± 3188) |
2.01 |
large/1k random edge additions |
944450 ns/iter (± 120975) |
467869 ns/iter (± 38115) |
2.02 |
lotr_graph/num_edges |
4 ns/iter (± 0) |
0 ns/iter (± 0) |
+∞ |
lotr_graph/num_nodes |
413 ns/iter (± 3) |
1 ns/iter (± 0) |
413 |
lotr_graph/iterate nodes |
5779 ns/iter (± 4) |
1828 ns/iter (± 4) |
3.16 |
lotr_graph_materialise/materialize |
8963210 ns/iter (± 94017) |
1493744 ns/iter (± 4485) |
6.00 |
lotr_graph_window_100/num_edges |
28 ns/iter (± 0) |
8 ns/iter (± 0) |
3.50 |
lotr_graph_window_100/num_nodes |
431 ns/iter (± 3) |
5 ns/iter (± 0) |
86.20 |
lotr_graph_window_100/iterate nodes |
9497 ns/iter (± 38) |
4349 ns/iter (± 14) |
2.18 |
lotr_graph_window_100_materialise/materialize |
9414846 ns/iter (± 55106) |
1648521 ns/iter (± 83100) |
5.71 |
lotr_graph_window_10/iterate nodes |
33430 ns/iter (± 543) |
10286 ns/iter (± 20) |
3.25 |
lotr_graph_window_10/iterate edges |
105066 ns/iter (± 425) |
45357 ns/iter (± 230) |
2.32 |
lotr_graph_window_10_materialise/materialize |
6112132 ns/iter (± 51941) |
1002929 ns/iter (± 2666) |
6.09 |
lotr_graph_subgraph_10pc/has_edge_existing |
508 ns/iter (± 7) |
87 ns/iter (± 0) |
5.84 |
lotr_graph_subgraph_10pc/num_nodes |
321 ns/iter (± 5) |
4 ns/iter (± 0) |
80.25 |
lotr_graph_subgraph_10pc/has_node_existing |
232 ns/iter (± 9) |
34 ns/iter (± 0) |
6.82 |
lotr_graph_subgraph_10pc/iterate nodes |
4800 ns/iter (± 14) |
826 ns/iter (± 18) |
5.81 |
lotr_graph_subgraph_10pc_windowed/has_node_existing |
129 ns/iter (± 9) |
62 ns/iter (± 12) |
2.08 |
lotr_graph_subgraph_10pc_windowed/iterate nodes |
4577 ns/iter (± 10) |
1403 ns/iter (± 20) |
3.26 |
lotr_graph_window_50_layered/num_nodes |
106544 ns/iter (± 1450) |
22514 ns/iter (± 1885) |
4.73 |
lotr_graph_window_50_layered/has_node_existing |
1554 ns/iter (± 208) |
129 ns/iter (± 15) |
12.05 |
lotr_graph_window_50_layered/max_id |
118067 ns/iter (± 6028) |
28069 ns/iter (± 1272) |
4.21 |
lotr_graph_window_50_layered/max_degree |
2033381 ns/iter (± 14153) |
300064 ns/iter (± 17214) |
6.78 |
lotr_graph_window_50_layered/iterate nodes |
242760 ns/iter (± 5538) |
18124 ns/iter (± 68) |
13.39 |
lotr_graph_window_50_layered/iterate edges |
220217 ns/iter (± 2427) |
82399 ns/iter (± 1747) |
2.67 |
lotr_graph_window_50_layered/max_neighbour_degree |
3814658 ns/iter (± 28032) |
383667 ns/iter (± 2307) |
9.94 |
lotr_graph_window_50_layered_materialise/materialize |
20353820 ns/iter (± 240840) |
3459987 ns/iter (± 27050) |
5.88 |
lotr_graph_persistent_window_50_layered/num_edges_temporal |
591282 ns/iter (± 29341) |
204425 ns/iter (± 1699) |
2.89 |
lotr_graph_persistent_window_50_layered/num_nodes |
140156 ns/iter (± 2281) |
32598 ns/iter (± 582) |
4.30 |
lotr_graph_persistent_window_50_layered/has_node_existing |
2021 ns/iter (± 445) |
180 ns/iter (± 61) |
11.23 |
lotr_graph_persistent_window_50_layered/max_id |
153678 ns/iter (± 6357) |
40064 ns/iter (± 3384) |
3.84 |
lotr_graph_persistent_window_50_layered/max_degree |
2822059 ns/iter (± 19939) |
458997 ns/iter (± 7079) |
6.15 |
lotr_graph_persistent_window_50_layered/iterate nodes |
330499 ns/iter (± 958) |
35321 ns/iter (± 1219) |
9.36 |
lotr_graph_persistent_window_50_layered/iterate edges |
196143 ns/iter (± 329) |
95091 ns/iter (± 933) |
2.06 |
lotr_graph_persistent_window_50_layered/max_neighbour_degree |
5189599 ns/iter (± 77795) |
517570 ns/iter (± 4540) |
10.03 |
lotr_graph_persistent_window_50_layered_materialise/materialize |
48919843 ns/iter (± 1727079) |
5203292 ns/iter (± 23745) |
9.40 |
lotr_graph/proto_decode |
10347615 ns/iter (± 102166) |
1941404 ns/iter (± 31833) |
5.33 |
lotr_graph/proto_encode |
9482173 ns/iter (± 91436) |
1196445 ns/iter (± 62942) |
7.93 |
This comment was automatically generated by workflow using github-action-benchmark.
…ith async_graphql 7.1.0
…ith async_graphql 7.1.0 (#2438)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
redo the way node ids are distributed, no longer sequential
Why are the changes needed?
parallel loading
Does this PR introduce any user-facing change? If yes is this documented?
No
How was this patch tested?
throw a chicken in the air then hit it with pytest and cargo test
Are there any further changes required?
yes