Add the option to keep the path prefix in the generated include path

[lfluer]

Hello,

I've noticed that flatcc always strips away the prefix of an included file when formatting the include path in the header file.

For instance, if in my flatbuffers schema I have the following:

include "path/to/included_schema.fbs";

then in the generated header file the included path will always strip away the directories:

#ifndef INCLUDED_SCHEMA_READER_H
#include "included_schema_reader.h"
#endif

I know that flatbuffers has an option called --keep-prefix that allows you to keep the prefix of the path so that the generated header file looks like this:

#ifndef INCLUDED_SCHEMA_READER_H
#include "path/to/included_schema_reader.h"
#endif

This would be very useful for me as I have a quite big project where the schemas are spread across multiple directories and, without the prefix, I would need to add quite a lot of include paths in the cmake files.

Would it be possible to add a --keep-prefix option for flatcc as well? I could also help with the implementation. I've taken a look at the place where the include gets generated and that part wouldn't be too hard to modify to keep the full path instead of the stripped basename. However, I'm not sure exactly how to add a new option and how difficult it would be to integrate that.

Thank you!

[mikkelfj]

I have not looked closely at the source code, but it should be possible to do.

The reason the prefix is removed is because there can be multiple relative or absolute paths to the same include file. To prevent infinite recursion and redefinitions, the base name is used as the internal name. This concept is similar to how include guards behave in C files. It is assumed that an include guard is unique in the given context. Incidentally, flatcc also needs to define include guard names, and conveniently use the same name mapping for that purpose. Finally, the generated files form a cluster around the base name xxx_reader, xxx_builder etc.

There is an option to generate all header files as one single file. This works by concatenation. The include guards, and the include order, ensures that this works.

So, yes, we can have a longer prefix, but it must map to a flat non-hierarchical name, and it must be the same for all references to the same include file, which is not necessarily easy to guarantee.

In the current solution you might get errors if you reuse a schema file name in that manner, and this is probably where your problem originates from. This is a fail safe.

Given this information, what would you suggest to do? Would you be willing to implement it if there is sane solution?

Also note a separate issue on - not being mapped correctly, this would be obvious to address at the same time.

Most of the code pertinent to this is driven by the flatcc.c file, and some included headers, and some calls into parse logic where it keeps a stack of include definitions. The core mapping should be trivial to change with an option setting. It is all the secondary semantics that are problematic.

[lfluer]

Thanks for your answer!

I think I understand the issue but let's take a simple example to make sure. Let's say we have the following directory structure:

• a/
  • file1.fbs ---> This file has the include statement include "b/file1.fbs";
  • b/
    • file1.fbs
• c/
  • file2.fbs ---> This file has two include statements: include "../a/file1.fbs"; and include "../a/b/file1.fbs";

With a --keep-prefix option, the issue is that the file a/b/file1.fbs would be included twice (once in file2.fbs and once in a/file1.fbs) because the prefix is different, so flatcc would detect it as two different files.

As a side note, this setup would anyway fail silently at the moment because, when compiling file2.fbs, flatcc would strip both prefixes and generate a single include statement for file1.fbs.

I'm not sure what the best fix for this behavior would be besides... warning the user that using the --keep-prefix option along with includes with different root paths might lead to issues?

In all fairness, I haven't tested what happens at the moment in flatbuffers in this situation.

Regarding dashes or other special characters (like .. in relative paths), we could replace them with valid characters when generating the include guards, so I think that wouldn't be a big issue.

[mikkelfj]

To your example, yes this is the issue, or concern rather.

This doesn't fail silently, but works as intended because there is only one file2.fbs. So the first time is is included, it is processed, and a second time, regardless of path, it is skipped. This works as intended because the generated output lands in the same output directory (which you can control with -o).

Now, if you keep prefixes, you pretend that the same file is two different files, in the general case, as in your example. This would then cause file2.fbs to be processed twice, with the same symbols, and likely all sorts of problems because the include guards would also be different.

When stripping prefix, as is the case now, a problem can happen when you have two different paths to two functionally different fbs files with the same base name, e.g. file2.fbs. In that case one of the instances will not be processed, and it if were important, you would get a compile error, and if not, well, you saved the effort. The solution is to make sure the base name of the file is distinct instead of only relying on paths. E.g. instead of myproject/foobar.fbs, use myproject/myproject_foobar.fbs
This solves all problems, if can tolerate that convention.

This is the dash issue: #293

[mikkelfj]

Note that you can have the options --prefix to add a prefix to generated names, which is not related to file prefixes, but name files, which is not directly related, but may be useful.

You and also avoid the -a (--all) option which includes -r (--recursive). By omitting -r, you have to call flatcc for every file individually, which may also help to control conflicts.

So how does on file now which include to use for another if doing so? It works because the prefix is stripped, so it can guess the include guard (I'm not 100% sure, maybe you have to manually include).

So, one could imagine a new option that is --file-prefix, which does not map the include path, but uses a prefix you provide instead of directly naming files. You may have to manually include these files and perhaps create stub files for the basename include guard, or provide another option to not automatically include files.

Whatever is done, it needs to be simple, clean and predictable.

Maybe you can solve your problem without adding new code.

Alse see my prev. response if you missed it.

[mikkelfj]

@lfluer Please let me know if the above addresses your concerns, or if you are considering providing a PR.

[lfluer]

Hi @mikkelfj ,

Sorry for my late response, I was on holidays.

When stripping prefix, as is the case now, a problem can happen when you have two different paths to two functionally different fbs files with the same base name, e.g. file2.fbs. In that case one of the instances will not be processed, and it if were important, you would get a compile error, and if not, well, you saved the effort. The solution is to make sure the base name of the file is distinct instead of only relying on paths. E.g. instead of myproject/foobar.fbs, use myproject/myproject_foobar.fbs

Yes, this is actually what I was referring to by "failing silently" -- that the two files with the name file1.fbs would be treated as the same file, even if they are actually two different files.

Note that you can have the options --prefix to add a prefix to generated names, which is not related to file prefixes, but name files, which is not directly related, but may be useful.

I think I tried this at one point but, if I remember correctly, there is a problem if the prefix contains characters like forward slashes (which are needed for paths) because the prefix will be added to some C type names as well and it will generate invalid C code. So it can be useful for some things but unfortunately not for paths.

So, one could imagine a new option that is --file-prefix, which does not map the include path, but uses a prefix you provide instead of directly naming files. You may have to manually include these files and perhaps create stub files for the basename include guard, or provide another option to not automatically include files.

If I understand correctly this solution, I would need to provide only one --file-prefix for all includes, which unfortunately would not solve my problem as I might include different files from different directories, which would therefore need different prefixes. Or if I misunderstood this solution please let me know.

[mikkelfj]

As to --prefix, I don't really recall the details, but I don't think has to do with paths, it should just be a name you choose, obviously a valid C identifier.

--file-prefix : you can build each .fbs file individually, the -r option is recursive, but you don't have to use it. Just make a build script. Even without --file-prefix that might be enough if you rename the files are after building the output. You might have to change the include guards with sed or similar substitution, as part of your build, or as stated before, manually include the new file names and add stub files for the files already being included.

Additionally, even if the base name is being used for different projects, if you generate the files separately to separate directories, you don't even have to rename files, you just have to add the output path to the include.

FlatCC is explicitly designed such that the output files are the same regardless of whether they are included by another schema file or built directly.