Crashing on Ubuntu 26.04 (PG18 plus Python 3.14)

[ShaheedHaque]

I upgraded to Ubuntu 26.04 (which comes with PG18 plus Python 3.14) and unfortunately, my FDW is crashing with the following in syslog (manually unwrapped, hopefully without too many typos):

2026-05-18T12:30:49.614536+01:00 baba2 systemd-coredump[4037]: Process 4034 (postgres) of user 123 dumped core.
#012
#012
Module /usr/lib/postgresql/18/bin/postgres from deb postgresql-18-18.3-1.amd64#012
Module linux-vdso.so.1 from deb linux-7.0.0-15.15.amd64#012
Module libmd.so.0 from deb libmd-1.1.0-2build4.amd64#012
Module libbsd.so.0 from deb libbsd-0.12.2-2build2.amd64#012
Module libtinfo.so.6 from deb ncurses-6.6+20251231-1.amd64#012
Module libedit.so.2 from deb libedit-3.1-20251016-1.amd64#012
Module libLLVM.so.21.1 from deb llvm-toolchain-21-1:21.1.8-6ubuntu1.amd64#012
Module llvmjit.so from deb postgresql-18-18.3-1.amd64#012
Module _bcrypt.cpython-314-x86_64-linux-gnu.so from deb python-bcrypt-5.0.0-3build1.amd64#012
Module _cffi_backend.cpython-314-x86_64-linux-gnu.so from deb python-cffi-2.0.0-3build1.amd64#012
Module legacy.so from deb openssl-3.5.5-1ubuntu3.amd64#012
Module _rust.abi3-x86_64-linux-gnu.so from deb python-cryptography-46.0.5-1ubuntu2.amd64#012
Module libsqlite3.so.0 from deb sqlite3-3.46.1-9.amd64#012
Module _sqlite3.cpython-314-x86_64-linux-gnu.so from deb python3.14-3.14.4-1.amd64#012
Module termios.cpython-314-x86_64-linux-gnu.so from deb python3.14-3.14.4-1.amd64#012
Module netifaces.cpython-314-x86_64-linux-gnu.so from deb netifaces-0.11.0-2build7.amd64#012
Module _utils.cpython-314-x86_64-linux-gnu.so from deb python3.14-3.14.4-1.amd64#012
Module _zoneinfo.cpython-314-x86_64-linux-gnu.so from deb python3.14-3.14.4-1.amd64#012
Module _setproctitle.cpython-314-x86_64-linux-gnu.so from deb python-setproctitle-1.3.7-2build1.amd64#012
Module backend_c.cpython-314-x86_64-linux-gnu.so from deb python-zstandard-0.25.0-1build1.amd64#012
Module libyaml-0.so.2 from deb libyaml-0.2.5-2build3.amd64#012
Module _yaml.cpython-314-x86_64-linux-gnu.so from deb pyyaml-6.0.3-1build1.amd64#012
Module resource.cpython-314-x86_64-linux-gnu.so from deb python3.14-3.14.4-1.amd64#012
Module libffi.so.8 from deb libffi-3.5.2-4.amd64#012
Module _ctypes.cpython-314-x86_64-linux-gnu.so from deb python3.14-3.14.4-1.amd64#012
Module _asyncio.cpython-314-x86_64-linux-gnu.so from deb python3.14-3.14.4-1.amd64#012
Module libuuid.so.1 from deb util-linux-2.41.3-3ubuntu2.amd64#012
Module _uuid.cpython-314-x86_64-linux-gnu.so from deb python3.14-3.14.4-1.amd64#012
Module _decimal.cpython-314-x86_64-linux-gnu.so from deb python3.14-3.14.4-1.amd64#012
Module liblzma.so.5 from deb xz-utils-5.8.3-1.amd64#012
Module _lzma.cpython-314-x86_64-linux-gnu.so from deb python3.14-3.14.4-1.amd64#012
Module libbz2.so.1.0 from deb bzip2-1.0.8-6build2.amd64#012
Module _bz2.cpython-314-x86_64-linux-gnu.so from deb python3.14-3.14.4-1.amd64#012
Module _hmac.cpython-314-x86_64-linux-gnu.so from deb python3.14-3.14.4-1.amd64#012
Module _hashlib.cpython-314-x86_64-linux-gnu.so from deb python3.14-3.14.4-1.amd64#012
Module _zstd.cpython-314-x86_64-linux-gnu.so from deb python3.14-3.14.4-1.amd64#012
Module _ssl.cpython-314-x86_64-linux-gnu.so from deb python3.14-3.14.4-1.amd64#012
Module _queue.cpython-314-x86_64-linux-gnu.so from deb python3.14-3.14.4-1.amd64#012
Module _interpreters.cpython-314-x86_64-linux-gnu.so from deb python3.14-3.14.4-1.amd64#012
Module plpython3.so from deb postgresql-18-18.3-1.amd64#012
Module libexpat.so.1 from deb expat-2.7.4-1.amd64#012
Module libpython3.14.so.1.0 from deb python3.14-3.14.4-1.amd64#012
Module multicorn.so from deb postgresql-18-18.3-1.amd64#012
Module libkeyutils.so.1 from deb keyutils-1.6.3-6ubuntu3.amd64#012
Module libcap-ng.so.0 from deb libcap-ng-0.8.5-4build5.amd64#012
Module libicudata.so.78 from deb icu-78.2-2ubuntu1.amd64#012
Module libgcc_s.so.1 from deb gcc-16-16-20260322-1ubuntu1.amd64#012
Module libstdc++.so.6 from deb gcc-16-16-20260322-1ubuntu1.amd64#012
Module libsasl2.so.2 from deb cyrus-sasl2-2.1.28+dfsg1-9ubuntu3.amd64#012
Module liblber.so.2 from deb openldap-2.6.10+dfsg-1ubuntu5.amd64#012
Module libkrb5support.so.0 from deb krb5-1.22.1-2ubuntu4.amd64#012
Module libcom_err.so.2 from deb e2fsprogs-1.47.2-3ubuntu4.amd64#012
Module libk5crypto.so.3 from deb krb5-1.22.1-2ubuntu4.amd64#012
Module libkrb5.so.3 from deb krb5-1.22.1-2ubuntu4.amd64#012
Module libaudit.so.1 from deb audit-1:4.1.2-1build1.amd64#012
Module libxxhash.so.0 from deb xxhash-0.8.3-2build1.amd64#012
Module libsystemd.so.0 from deb systemd-259.5-0ubuntu3.amd64#012
Module liburing.so.2 from deb liburing-2.14-1.amd64#012
Module libicuuc.so.78 from deb icu-78.2-2ubuntu1.amd64#012
Module libicui18n.so.78 from deb icu-78.2-2ubuntu1.amd64#012
Module libldap.so.2 from deb openldap-2.6.10+dfsg-1ubuntu5.amd64#012
Module libnuma.so.1 from deb numactl-2.0.19-1build1.amd64#012
Module libz.so.1 from deb zlib-1:1.3.dfsg+really1.3.1-1ubuntu3.amd64#012
Module libgssapi_krb5.so.2 from deb krb5-1.22.1-2ubuntu4.amd64#012
Module libcrypto.so.3 from deb openssl-3.5.5-1ubuntu3.amd64#012
Module libssl.so.3 from deb openssl-3.5.5-1ubuntu3.amd64#012
Module libpam.so.0 from deb pam-1.7.0-5ubuntu3.amd64#012
Module libxml2.so.16 from deb libxml2-2.15.2+dfsg-0.1.amd64#012
Module liblz4.so.1 from deb lz4-1.10.0-8.amd64#012
Module libzstd.so.1 from deb libzstd-1.5.7+dfsg-3.amd64#012
Stack trace of thread 4035:
#012#0  0x000076497537fa95 n/a (libpython3.14.so.1.0 + 0x17fa95)
#012#1  0x000076497537fed5 PyObject_GenericGetDict (libpython3.14.so.1.0 + 0x17fed5)
#012#2  0x00007649753cd359 n/a (libpython3.14.so.1.0 + 0x1cd359)
#012#3  0x00007649753bc9b4 n/a (libpython3.14.so.1.0 + 0x1bc9b4)
#012#4  0x0000764975324cde PyObject_Vectorcall (libpython3.14.so.1.0 + 0x124cde)
#012#5  0x00007649754859e0 _PyEval_EvalFrameDefault (libpython3.14.so.1.0 + 0x2859e0)
#012#6  0x00007649754940f7 n/a (libpython3.14.so.1.0 + 0x2940f7)
#012#7  0x0000764975324cde PyObject_Vectorcall (libpython3.14.so.1.0 + 0x124cde)
#012#8  0x00007649753354fb n/a (libpython3.14.so.1.0 + 0x1354fb)
#012#9  0x000076497539ad17 _PyObject_GenericSetAttrWithDict (libpython3.14.so.1.0 + 0x19ad17)
#012#10 0x0000764975398688 PyObject_SetAttr (libpython3.14.so.1.0 + 0x198688)
#012#11 0x000076497548ab75 _PyEval_EvalFrameDefault (libpython3.14.so.1.0 + 0x28ab75)
#012#12 0x00007649754940f7 n/a (libpython3.14.so.1.0 + 0x2940f7)
#012#13 0x0000764975326c75 n/a (libpython3.14.so.1.0 + 0x126c75)
#012#14 0x00007649753dbd4d n/a (libpython3.14.so.1.0 + 0x1dbd4d)
#012#15 0x00007649753bfe10 n/a (libpython3.14.so.1.0 + 0x1bfe10)
#012#16 0x0000764975324b0c _PyObject_MakeTpCall (libpython3.14.so.1.0 + 0x124b0c)
#012#17 0x0000764975486946 _PyEval_EvalFrameDefault (libpython3.14.so.1.0 + 0x286946)
#012#18 0x00007649754940f7 n/a (libpython3.14.so.1.0 + 0x2940f7)
#012#19 0x000076497532607b n/a (libpython3.14.so.1.0 + 0x12607b)
#012#20 0x00007649754b5fc3 n/a (libpython3.14.so.1.0 + 0x2b5fc3)
#012#21 0x0000764975324cde PyObject_Vectorcall (libpython3.14.so.1.0 + 0x124cde)
#012#22 0x00007649754859e0 _PyEval_EvalFrameDefault (libpython3.14.so.1.0 + 0x2859e0)
#012#23 0x00007649754940f7 n/a (libpython3.14.so.1.0 + 0x2940f7)
#012#24 0x000076497532607b n/a (libpython3.14.so.1.0 + 0x12607b)
#012#25 0x0000764975667f9a n/a (libpython3.14.so.1.0 + 0x467f9a)
#012#26 0x000076497555b83c n/a (libpython3.14.so.1.0 + 0x35b83c)
#012#27 0x000076498f8a40da start_thread (libc.so.6 + 0xa40da)
#012#28 0x000076498f9377ac __clone3 (libc.so.6 + 0x1377ac)
#012
#012
Stack trace of thread 4034:
#012#0  0x000076498f8acae2 __syscall_cancel_arch (libc.so.6 + 0xacae2)
#012#1  0x000076498f8a05ec __internal_syscall_cancel (libc.so.6 + 0xa05ec)
#012#2  0x000076498f8a09c7 __futex_abstimed_wait_common64 (libc.so.6 + 0xa09c7)
#012#3  0x000076498f8a3662 __pthread_cond_wait_common (libc.so.6 + 0xa3662)
#012#4  0x00007649754d74b8 n/a (libpython3.14.so.1.0 + 0x2d74b8)
#012#5  0x00007649754d7855 PyEval_AcquireThread (libpython3.14.so.1.0 + 0x2d7855)
#012#6  0x000076497552df39 _PyParkingLot_Park (libpython3.14.so.1.0 + 0x32df39)
#012#7  0x0000764975516912 _PyMutex_LockTimed (libpython3.14.so.1.0 + 0x316912)
#012#8  0x000076497566927b n/a (libpython3.14.so.1.0 + 0x46927b)
#012#9  0x00007649753338c5 n/a (libpython3.14.so.1.0 + 0x1338c5)
#012#10 0x0000764975324cde PyObject_Vectorcall (libpython3.14.so.1.0 + 0x124cde)
#012#11 0x00007649754859e0 _PyEval_EvalFrameDefault (libpython3.14.so.1.0 + 0x2859e0)
#012#12 0x00007649

I presume I need to try to get a stack trace with symbols...it has been a while since I helped look after Multicorn code, and my C is now VERY rusty...can somebody remind me where to get started?

[ShaheedHaque]

I tried running the tests. But I cannot get them to run:

$ make easycheck
Python version is 3.14
echo "# +++ regress check in  +++" && PATH="/tmp_install/usr/lib/postgresql/18/bin:/home/srhaque/multicorn2-main:$PATH" LD_LIBRARY_PATH="/tmp_install/usr/lib/x86_64-linux-gnu" INITDB_TEMPLATE=''/tmp_install/initdb-template  /usr/lib/postgresql/18/lib/pgxs/src/makefiles/../../src/test/regress/pg_regress --temp-instance=./tmp_check --inputdir=./ --bindir=     --inputdir=test-3.14 --encoding=UTF8 --host=localhost multicorn_cache_invalidation multicorn_column_options_test multicorn_error_test multicorn_logger_test multicorn_planner_test multicorn_regression_test multicorn_sequence_test multicorn_test_date multicorn_test_dict multicorn_test_limit multicorn_test_list multicorn_test_sort write_savepoints write_test import_test write_batch_test
# +++ regress check in  +++
# initializing database system by copying initdb template
# copying of initdb template failed
# Examine "/home/srhaque/multicorn2-main/log/initdb.log" for the reason.
# Command was: cp -RPp "/tmp_install/initdb-template" "/home/srhaque/multicorn2-main/tmp_check/data" > "/home/srhaque/multicorn2-main/log/initdb.log" 2>&1
Bail out!make: *** [Makefile:156: easycheck] Error 2

The content of "/home/srhaque/multicorn2-main/log/initdb.log" highlights the fact that I don't have a /tmp_install root-level directory:

cp: cannot stat '/tmp_install/initdb-template': No such file or directory

But what is that directory supposed to contain? What am I missing?