Adding node layers to tests and loaders#2597
Draft
arienandalibi wants to merge 111 commits into
Draft
Conversation
…re-compute new IDs and turn them into RecordBatches
…ock the graph to get parallel iterators over edges. We filter to respect GraphView filtering behaviour.
…ill use ArrowWriter<File> for now, but we will add support for loading into a graph
# Conflicts: # raphtory/src/serialise/parquet/mod.rs
…ng explode_layers() on each EdgeView.
… function can now be passed to these functions to determine how the sinks will be created. This will allow us to pass a sink which is a crossbeam_channel to send RecordBatches elsewhere.
# Conflicts: # raphtory/src/serialise/parquet/mod.rs
…w materialize function
…dge_meta and node_meta.
…k and reusing the old one.
…f encoding everything and then ingesting everything (which would keep everything in memory at once).
… when run on a big graph.
…another thread pool.
…rage so that it doesn't run out of memory
…anning each segment for each row. Now using this path in the new materialize_using_recordbatches function.
… as much as possible
…separate out running materialize and parquet decoding. Test using SF10 for now.
…odes_from_df call. We can actually pass a column of layer names to the "layer_id_col" parameter, the name is misleading
…ze) and in proptests. currently fails
… was in persistent_semantics.rs, in fn node_updates_window. Proptests still fail.
…p_dst_id". GIDS are now "rap_src_id" and "rap_dst_id". This is inconsistent with other column's naming scheme, but it is backwards compatible with already encoded parquet files.
# Conflicts: # raphtory/src/arrow_loader/df_loaders/nodes.rs # raphtory/src/db/api/view/graph.rs # raphtory/src/io/parquet_loaders.rs # raphtory/src/parquet_encoder/edges.rs # raphtory/src/parquet_encoder/mod.rs # raphtory/src/parquet_encoder/model.rs # raphtory/src/parquet_encoder/nodes.rs # raphtory/src/python/graph/io/arrow_loaders.rs # raphtory/src/serialise/parquet.rs # raphtory/tests/df_loaders.rs # raphtory/tests/test_materialize_sf10.rs
…'re now back to ingesting using VIDs instead of resolving GIDs.
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
⚠️ Performance Alert ⚠️
Possible performance regression was detected for benchmark 'Rust Benchmark'.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold 2.
| Benchmark suite | Current: 279336f | Previous: 9823ef7 | Ratio |
|---|---|---|---|
lotr_graph/num_edges |
4 ns/iter (± 0) |
0 ns/iter (± 0) |
+∞ |
lotr_graph/num_nodes |
4 ns/iter (± 0) |
1 ns/iter (± 0) |
4 |
lotr_graph/has_node_nonexisting |
5 ns/iter (± 0) |
2 ns/iter (± 0) |
2.50 |
lotr_graph/iterate_exploded_edges |
614287 ns/iter (± 7679) |
285473 ns/iter (± 2920) |
2.15 |
lotr_graph/graph_latest |
3 ns/iter (± 0) |
0 ns/iter (± 0) |
+∞ |
lotr_graph_materialise/materialize |
8433913 ns/iter (± 27678) |
1564816 ns/iter (± 35303) |
5.39 |
lotr_graph_window_100/num_nodes |
14 ns/iter (± 0) |
5 ns/iter (± 0) |
2.80 |
lotr_graph_window_100/iterate_exploded_edges |
862699 ns/iter (± 1936) |
325242 ns/iter (± 847) |
2.65 |
lotr_graph_window_100_materialise/materialize |
8894575 ns/iter (± 40226) |
1669150 ns/iter (± 10700) |
5.33 |
lotr_graph_window_10/has_node_existing |
156 ns/iter (± 9) |
62 ns/iter (± 11) |
2.52 |
lotr_graph_window_10/iterate nodes |
36040 ns/iter (± 449) |
11339 ns/iter (± 40) |
3.18 |
lotr_graph_window_10/iterate edges |
104287 ns/iter (± 678) |
48684 ns/iter (± 211) |
2.14 |
lotr_graph_window_10/iterate_exploded_edges |
416800 ns/iter (± 3801) |
155788 ns/iter (± 1001) |
2.68 |
lotr_graph_window_10_materialise/materialize |
3764745 ns/iter (± 10836) |
971980 ns/iter (± 4278) |
3.87 |
lotr_graph_subgraph_10pc/has_node_nonexisting |
5 ns/iter (± 0) |
2 ns/iter (± 0) |
2.50 |
lotr_graph_subgraph_10pc_materialise/materialize |
1740553 ns/iter (± 24143) |
334634 ns/iter (± 1287) |
5.20 |
lotr_graph_subgraph_10pc_windowed/has_node_existing |
162 ns/iter (± 10) |
62 ns/iter (± 14) |
2.61 |
lotr_graph_subgraph_10pc_windowed/iterate nodes |
5616 ns/iter (± 89) |
1365 ns/iter (± 3) |
4.11 |
lotr_graph_subgraph_10pc_windowed_materialise/materialize |
1052497 ns/iter (± 9388) |
230399 ns/iter (± 2617) |
4.57 |
lotr_graph_window_50_layered/num_edges |
78579 ns/iter (± 1949) |
38714 ns/iter (± 1017) |
2.03 |
lotr_graph_window_50_layered/num_edges_temporal |
164464 ns/iter (± 2353) |
70121 ns/iter (± 7586) |
2.35 |
lotr_graph_window_50_layered/num_nodes |
43473 ns/iter (± 1412) |
21435 ns/iter (± 536) |
2.03 |
lotr_graph_window_50_layered/has_node_existing |
496 ns/iter (± 28) |
129 ns/iter (± 12) |
3.84 |
lotr_graph_window_50_layered/has_node_nonexisting |
5 ns/iter (± 0) |
2 ns/iter (± 0) |
2.50 |
lotr_graph_window_50_layered/iterate nodes |
82404 ns/iter (± 212) |
19308 ns/iter (± 47) |
4.27 |
lotr_graph_window_50_layered/iterate edges |
213439 ns/iter (± 2431) |
83616 ns/iter (± 1318) |
2.55 |
lotr_graph_window_50_layered/graph_latest |
82871 ns/iter (± 1848) |
36649 ns/iter (± 916) |
2.26 |
lotr_graph_window_50_layered_materialise/materialize |
34126410 ns/iter (± 320201) |
3488825 ns/iter (± 24948) |
9.78 |
lotr_graph_persistent_window_50_layered/num_edges_temporal |
662732 ns/iter (± 6412) |
192686 ns/iter (± 1569) |
3.44 |
lotr_graph_persistent_window_50_layered/has_node_existing |
511 ns/iter (± 380) |
174 ns/iter (± 83) |
2.94 |
lotr_graph_persistent_window_50_layered/has_node_nonexisting |
5 ns/iter (± 0) |
2 ns/iter (± 0) |
2.50 |
lotr_graph_persistent_window_50_layered/iterate nodes |
112424 ns/iter (± 2432) |
35886 ns/iter (± 191) |
3.13 |
lotr_graph_persistent_window_50_layered/iterate edges |
195632 ns/iter (± 1628) |
84161 ns/iter (± 596) |
2.32 |
lotr_graph_persistent_window_50_layered/iterate_exploded_edges |
4946269 ns/iter (± 23255) |
1659940 ns/iter (± 19402) |
2.98 |
lotr_graph_persistent_window_50_layered/graph_latest |
121074 ns/iter (± 2539) |
57549 ns/iter (± 4809) |
2.10 |
lotr_graph_persistent_window_50_layered_materialise/materialize |
63242405 ns/iter (± 632400) |
5298035 ns/iter (± 147912) |
11.94 |
lotr_graph/proto_encode |
10343401 ns/iter (± 174295) |
1157897 ns/iter (± 73709) |
8.93 |
This comment was automatically generated by workflow using github-action-benchmark.
# Conflicts: # raphtory/src/db/api/view/graph.rs
…c. resolve_layer fast path when layer ids are present is gone temporarily while debugging, will bring it back. fix node_updates_window in persistent_semantics.rs to account for the entire timestamp at the windows beginning for persisting properties properly.
…ders, and bringing back the fast path that uses these when resolving layers.
… back to Option, if it's not there then we imply STATIC_GRAPH_LAYER
… added layer and layer id to nodes_t
…ing both functions into one. updating callsites. fixing variable names. cleaning up comments
# Conflicts: # raphtory/src/db/api/view/internal/time_semantics/persistent_semantics.rs # raphtory/src/parquet_encoder/nodes.rs
…the disk graphs are unreadable or data is loaded incorrectly
…the disk graphs are unreadable or data is loaded incorrectly
…the source graph doesn't change each time the test runs. Added .gitkeep empty files so empty directories are picked up by git.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Node layers were previously not tested rigorously. They are now being added to tests, proptests, the loaders, and the parquet encoders. The loaders and parquet encoders are also used by materialize.
Why are the changes needed?
Fix node layer related bugs that we find.
Does this PR introduce any user-facing change? If yes is this documented?
It shouldn't
How was this patch tested?
proptests
Are there any further changes required?
There shouldn't be