make DuckDB::TableFunction work in multi thread duckdb

[suketa]

Currently we must set duckdb thread to 1 to use DuckDB::TableFunction functionality.

make DuckDB::TableFunction work multi thread duckdb.

con.execute('SET threads=1') we need this to use DuckDB::TableFunction

[otegami]

Hi, I'd like to try tackling this issue.

My understanding

DuckDB invokes table function callbacks from its own worker threads rather than Ruby threads. Because Ruby's GVL cannot be safely acquired from non-Ruby threads (rb_thread_call_with_gvl crashes with rb_bug), we currently require SET threads=1.

If that understanding is correct, I'd like to explore an approach based on per-worker proxy threads for DuckDB >= 1.5.0, if that version constraint is acceptable. Each DuckDB worker thread would communicate with a dedicated Ruby thread that acquires the GVL and executes the callback on its behalf. This seems similar to the proxy thread pattern used in the FFI gem (ext/ffi_c/Function.c).

To get there cleanly, I'd first like to extract the executor logic into a shared component, then apply that infrastructure to table function callbacks.

Proposed incremental steps

I'd like to proceed in small, mergeable steps:

• Extract the global executor from scalar_function.c into shared executor.{c,h}
• Add per-worker proxy thread infrastructure to the executor
• Integrate the per-worker proxy into scalar function callbacks
• Add three-path dispatch to table function callbacks
• Use the per-worker proxy for the table function execute callback
• Remove the SET threads=1 requirement for DuckDB >= 1.5.0

I don't have deep experience with multi-threaded programming, so I'd especially appreciate feedback on whether this approach seems sound. If it does, I'm happy to work through it in small, reviewable steps.

I've also been experimenting with a draft implementation here for reference: otegami#6

[suketa]

Sorry for being late to reply.
Thank you. Your steps looks good to me,
But I prefer ruby-duckdb supports DuckDB >= 1.5.0 and DuckDB 1.4.x because 1.4.x is LTS version.
And per-worker proxy into scalar function callbacks might need DuckDB >= 1.5.0.
So it is best to care which version be used when building ruby-duckdb and enable per-worker proxy into scalar function callbacks if DuckDB >= 1.5.0 is used.

Feel free to open PR.

[suketa]

@otegami

Maybe I think I have already changed DuckDB::TableFunction works with SET threads > 0 while trying to fix #1158.
But per-worker proxy integration is not implemented yet (See #1308).

I'm glad if you open PR to implement per-worker proxy integration.

[otegami]

Thanks for the update, and no worries at all.
I’m a bit tied up preparing for RubyKaigi at the moment, but I expect I’ll be able to start working on this for sure after April 25.
I’ll keep in mind the need to support both DuckDB 1.4.x and >= 1.5.0, and I’ll look into the per-worker proxy integration with version-aware handling.

[suketa]

Enjoy RubyKaigi!!