Cbh file format integration

[rolandlo]

Hi Fulvio

In https://sourceforge.net/p/scid/feature-requests/24/ you expressed your reluctance and caution to write code for cbh databases. I don't quite share it. Legally it seems perfectly fine to reverse engineer a file format for the purpose of interoperability. There are already different proprietary and free apps and tools which read the Chessbase .cbh file format and as far as I know, Chessbase hasn't acted against it.

Would you consider integrating the Cbh file format if someone else wrote the code? In fact I have already done so, see my add-cbh-codec branch. It works quite reliably, even though there is some cleanup work to do. The move decoding, which is the most difficult part, is tested with automatic tests. I followed the scidb-beta implementation relatively closely, making adjustments to Scid's data structures and methods where necessary. Chess960 games, which use a different encoding, are decoded successfully as well. This was missing in scidb-beta. There are some other minor improvements with respect to some changes and additions that Chessbase must have made after the scidb-beta implementatoin was written, and some other minor fixups. So it works fine with recent databases including the MegaBase 2026.

Performance-wise I can say that reading a .cbh database is quite a bit faster than cbh2si4 from scidb-beta. For example it takes a bit less than 3 minutes for reading the > 10 Million games from the MegaBase 2026 into memory on my convertible, whereas cbh2si4 would take more than an hour (and produces invalid dates without applying an additional patch). Still the idea is that large databases are converted by the user into the si5-format, which can be done conveniently once the .cbh database is loaded.

[benini]

Hi,
The responsibility would still lie with me as the software distributor, regardless of who the author is.
Which other chess apps are able to read .cbh files?
I’ve only had a quick look at the code (which already looks quite clean): is it possible to turn it into a standalone independent library? For instance, CodecPgn uses pgn::parse_game, which accepts a generic 'visitor' and is therefore completely agnostic regarding the file format. This way, you would be both the author and the distributor of the library and would hold full responsibility (similar to how Scid uses the Tcl/Tk library, but I am not responsible for that code).
Another option would be to write to ChessBase and let them know that, for interoperability purposes, we plan to add a .cbh decoder to Scid, giving them a reasonable amount of time (30 days?) to voice any objections.

[rolandlo]

Thanks for your kind response.

Apps and tools reading .cbh files (other than Chessbase products):

1. Hiarcs Chess Explorer Pro (for Windows and MacOS)
2. Chess Assistant (for Windows only)
3. ChessStudio (for iPhone)
4. Scidb (Unfortunately hard to compile on modern distributions)
5. Cbh2Pgn (A Python module for converting cbh files to pgn)

I doubt it's feasible to turn the decoder into an independent library without duplicating quite some of Scid's code. Chessbase moves are stored in a way like e.g.

1. 1st remaining knight moves by offset (+2,+1) (potentially capturing a piece) or
2. 2nd remaining rook moves by offset (+6, 0) (potentially capturing a piece and also possibly moving two squares to the left, depending on which square it is on)
3. 4th original pawn captures left or
4. pawn promotes from this square to that square (potentially capturing a piece) and promotes into that piece.

It takes some chess to figure out what that means for the current position and the whole sequence of moves that proceeded it. What would be the library's responsibility and what would be Scid's responsibility? If the library must basically figure out a SAN representation of the move, it needs a lot of chess knowledge, thus duplicated work. If it doesn't then Scid's part would involve internals from the file format.

I wouldn't mind writing to ChessBase and giving them the chance to voice objections if they have any.

[benini]

Ok then, please go ahead and contact ChessBase.

Let me also highlight a couple of quirks that can be misleading:

• game.SetStartPos(position) doesn't work as one might expect because the order of pieces is important for move encoding. It is necessary for position to have been created using ReadFromFEN or ReadFromFENorUCI. The best approach is to use position.PrintFEN() followed by game.SetStartFEN().
• PgnVisitor::visitPGN_SANMove() actually accepts moves in UCI format as well (from_square to_square promo). In general, using PgnVisitor instead of directly manipulating the Game object helps avoid surprises. For example, using visitPGN_TagPair("FEN", "...") already prevents the SetStartPos pitfall.

[rolandlo]

I did write to Chessbase (in German) about it and gave them 30 days for a response/voicing any objections. So let's wait for their response or for the 30 days to pass.

[rolandlo]

I haven't got an answer from Chessbase yet. In the meantime I managed to turn the decoding into a standalone library: https://github.com/rolandlo/libcbh

See my use-libcbh branch for how it is integrated into Scid. There is still work to be done, but I like the approach. There is clear separation and it will make it much easier for other projects to use it. It's still roughly as fast as the previous approach. It uses some of the code from Scid (around 1300 LOC from the header files and 600 LOC from the source files). The communication between library and Scid adds a little overhead which is compensated by some stripping I made to the Position class. So reading the Mega2026 (with > 10 million games) still takes a bit less than 3 minutes, of which a bit more than half is spent in the library code.

What do you think about that approach @benini ?

[benini]

It's the best approach, imho.
I also think it should also be possible to move everything into just codec_cbh.h
For example:

const std::string squares[64] = {
    "a1", "b1", "c1", "d1", "e1", "f1", "g1", "h1", // 1st row
};

static inline std::string squareName(byte sq) {
	if (0 <= sq && sq < 64) {
		return squares[sq];
	}
	return "";
}

could be:

namespace {
std::string_view squareName(size_t sq) {
  constexpr static std::string_view squares = "a1b1c1....";
  return squares.substr(sq * 2, 2);
}
}

Also, beware of ODR, I would wrap everything into a namespace cbh.
The library's CMakeLists.txt can export the public headers with target_include_directories PUBLIC.
Long, imprecise, AI explanation (sorry):

---

1. Modern CMake: Using PUBLIC target includes

In "Modern CMake" (Target-Based CMake), targets (libraries) should define their own usage requirements. If a library needs headers to compile, and executables linking to that library also need those headers, the library should use PUBLIC in its target_include_directories.

The Result for scid:

Because the libraries now define their own include paths (via PUBLIC or INTERFACE), your executable configuration becomes incredibly clean. You delete the target_include_directories lines for the executable entirely:

# scid automatically inherits the include paths from polyglot and cbh!
target_link_libraries(scid PRIVATE 
    polyglot 
    cbh 
    ${CMAKE_THREAD_LIBS_INIT} 
    ${TCL_LIBRARY}
)

---

2. The One Definition Rule (ODR)

The One Definition Rule (ODR) is a core rule in C++. In simple terms, it states:

1. Within a single translation unit (a single .cpp file), a variable, class, or function can be defined only once.
2. Across the entire program (all compiled files and linked libraries), an externally visible variable or non-inline function can be defined exactly once.
3. Types (like classes/structs) and inline functions can be defined in multiple translation units (usually via header files), but they must be token-for-token identical everywhere.

Why it matters: If your polyglot library defines a global function named void init_board(), and scid also defines a global function named void init_board(), you have violated the ODR. Depending on the linker and platform, this will either cause a linker error ("multiple definition") or, worse, silently pick one definition over the other, leading to highly unpredictable Undefined Behavior.

---

3. How to check for Global Namespace Pollution

To prevent ODR violations, good C++ libraries wrap everything in namespaces (e.g., namespace polyglot { ... }). To verify that a compiled library isn't leaking symbols into the global namespace, you can inspect its symbol table using the nm tool (available on Linux/macOS/MinGW).

Run this command on your compiled library:

nm -gC libpolyglot.a
# or libpolyglot.so

• -g : Only show external/global symbols (ignores internal static variables).
• -C : Demangle C++ names so they are readable.

What to look for:
Look at symbols marked with a T (Text/Code section), D (Data), or B (BSS/Uninitialized data).

• Good output:

  00000000000001a0 T polyglot::Board::initialize()
  00000000000002f0 T polyglot::evaluate_position()
  

  (Notice how everything is inside the polyglot:: namespace).

• Bad output (Global Pollution):

  00000000000005b0 T calculate_score()
  00000000000008d0 D global_board_state
  

  (No :: present. These are in the global namespace and could clash with scid or cbh).

How to fix pollution:
If you find leaked C++ symbols, wrap them in a named namespace. If they are internal helper functions that shouldn't be exposed at all, wrap them in an anonymous namespace (namespace { ... }) or mark them static in the .cpp file, which gives them internal linkage and hides them from the linker entirely.