Better documentation/code example for "clone" operation

[codekrolik2]

Would be great to see a reference implementation of doing something as typical with cloning as:

1. Deserialize an existing object from a buffer using clone_as_root;
2. Update a field in root object, or a field in child object or an array (add/remove elements);
3. Assemble everything back together and serialize it back into a buffer.

While making sure we're not creating any garbage data in the buffer along the way.

[codekrolik2]

Tests show some manipulations with refmpas, which are not exactly self-explanatory and therefore a bit scary, so a good example in the docs (or two) would make it much easier to get things right with cloning.

[codekrolik2]

BTW does it even work? I'm trying to figure out that by guessing, and nothing seems to work.
E.g.:
this works:

  phraser_FoldersBlock_start_as_root(&builder);

  phraser_StoreBlock_t* store_block;
  store_block = phraser_FoldersBlock_block_start(&builder);
  store_block->block_id = 123;
...
  phraser_FoldersBlock_block_end(&builder);
  phraser_FoldersBlock_end_as_root(&builder);

but this doesn't:

phraser_FoldersBlock_table_t folders_block = phraser_FoldersBlock_as_root(buffer);
phraser_FoldersBlock_clone_as_root(&builder, folders_block));

  phraser_StoreBlock_struct_t storeblock = phraser_FoldersBlock_block_get(folders_block);
  phraser_FoldersBlock_block_create(&builder, store_block->block_id, store_block->version, store_block->entropy);

nor does this:

phraser_FoldersBlock_table_t folders_block = phraser_FoldersBlock_as_root(buffer);
phraser_FoldersBlock_clone_as_root(&builder, folders_block));

  phraser_StoreBlock_t* store_block = phraser_FoldersBlock_block_start(&builder);

What's the intended way for this to work?

[codekrolik2]

I mean even this didn't work for me:

  phraser_FoldersBlock_clone_as_root(&builder, folders_block);
  phraser_FoldersBlock_end_as_root(&builder);

I'm probably not getting something about what clone's purpose is. (due to not much details in docs)

[codekrolik2]

I guess in the meanwhile I'll just fall back to modifying by copying every field manually.
If there's a way to optimize that with clone, please let me know.

[codekrolik2]

Here's the official test example:

    refmap_old = flatcc_builder_set_refmap(B, &refmap);
    if (!ns(Monster_clone_as_root(B, ns(Monster_as_root(buffer))))) {
        printf("Cloned union vector Monster didn't actually clone.\n");
        goto failed;
    };
    flatcc_builder_set_refmap(B, refmap_old);
    cloned_buffer = flatcc_builder_finalize_aligned_buffer(B, &size);

from monster_test.c,
and it seems to be just doing nothing. (similarly to this one)
Not very helpful to understand the api design.

[codekrolik2]

OK, something worked for me:

  phraser_FoldersBlock_table_t folders_block = phraser_FoldersBlock_as_root(buffer);

  flatcc_builder_t builder;
  flatcc_builder_init(&builder);

  phraser_FoldersBlock_start_as_root(&builder);
  phraser_FoldersBlock_folders_clone(&builder, phraser_FoldersBlock_folders(folders_block));
  phraser_FoldersBlock_block_clone(&builder, phraser_FoldersBlock_block(folders_block));

  phraser_FoldersBlock_end_as_root(&builder);

But I'll just leave this here to serve as a pro-better documentation argument.
thanks!

[codekrolik2]

Spoke too soon - this one didin't work:

  phraser_FoldersBlock_start_as_root(&builder);
  phraser_FoldersBlock_folders_clone(&builder, phraser_FoldersBlock_folders(folders_block));

  phraser_StoreBlock_struct_t old_store_block = phraser_FoldersBlock_block(folders_block);

  phraser_StoreBlock_t* store_block;
  store_block = phraser_FoldersBlock_block_start(&builder);
  store_block->block_id = old_store_block->block_id;
  store_block->version = next_block_version();
  store_block->entropy = random_uint32();
  phraser_FoldersBlock_block_end(&builder);

  phraser_FoldersBlock_end_as_root(&builder);

Looks like clone is cursed maybe.

[codekrolik2]

Ok, just to summarize - my goal was to update 2 fields in store_block. (version and entropy)
In order to achieve that, I had to write the following code:

  phraser_FoldersBlock_table_t folders_block = phraser_FoldersBlock_as_root(buffer);

  phraser_StoreBlock_struct_t old_store_block = phraser_FoldersBlock_block(folders_block);
  phraser_Folder_vec_t old_folders = phraser_FoldersBlock_folders(folders_block);
  size_t folders_vec_length = flatbuffers_vec_len(old_folders);

  flatcc_builder_t builder;
  flatcc_builder_init(&builder);

  phraser_FoldersBlock_start_as_root(&builder);

  phraser_FoldersBlock_folders_start(&builder);
  for (int i = 0; i < folders_vec_length; i++) {
    phraser_Folder_table_t folder_fb = phraser_Folder_vec_at(old_folders, i);
    phraser_FoldersBlock_folders_push_create(&builder, 
      phraser_Folder_folder_id(folder_fb),
      phraser_Folder_parent_folder_id(folder_fb),
      flatbuffers_string_create_str(&builder, phraser_Folder_folder_name(folder_fb)));
  }
  phraser_FoldersBlock_folders_end(&builder);

  phraser_StoreBlock_t* store_block;
  store_block = phraser_FoldersBlock_block_start(&builder);
  store_block->block_id = phraser_StoreBlock_block_id(old_store_block);
  store_block->version = next_block_version();
  store_block->entropy = random_uint32();
  phraser_FoldersBlock_block_end(&builder);

  phraser_FoldersBlock_end_as_root(&builder);

And that's nearly the simplest flat buf structure I've got.
More complex and nested structures would require more code to copy all fields, of course.
In my mind before I tried it was doable with 1 clone and 2 sets, or something like that, which is a function of the changes to be made, as opposed to a function of the complexity of the structure itself.

Please if the above code is optimizable, let me know ASAP, I'd apreciate that information a lot!
(Especially if it happens before I've covered all the structures in my schema with transforming code like the above :) )

Excerpt from my schema for the reference:

table FoldersBlock {
  block: StoreBlock;
  folders: [Folder];
}

table Folder {
  folder_id: uint16;
  parent_folder_id: uint16;
  folder_name: string;
}

struct StoreBlock {
  block_id: uint16;
  version: uint32;
  entropy: uint32;
}

[mikkelfj]

The monster test cases that you have found are the best sources of documentation right now.

I can't dig into your problem right away, but I will get back to you.

The problem being solved is if you have a DAG, that, is, two different paths from root to the same object, do you want to make or copy or make it the same. To make that decision, you need to remember if you have already visited an object.

Because there are different situations where choices differ, at different levels in the graph, a bit more control is needed, and refmaps give you some degree of control.

However, it's been several years since I worked with this, so that is the best I can for now.

If you figure something out, consider contributing to the documentation.

[mikkelfj]

BTW: normally I would suggest printing JSON to see what happens, but this does not work in this case.

Instead, try writing a program that recursively follows you data structure in the source, and separately in the result. Print each node with its buffer offset (ref) as a name, maybe with some extra info like the table type.

Use the dot graph syntax when logging the traversal, and visualize the result.

There is also variant of clone that only injects a single field from an external buffer. I don't recall its name, but you will find it if you look. It might be better for targeted updates while you add other fields to a table.