Spurious query failures with concurrent database connections

[stapelberg]

Thanks for the txdb package, it’s a nice optimization when running many database tests :)

I noticed one issue, though, and wanted to ask if the issue is known or to be expected.

I’m working on a server which maintains multiple caches in the background (i.e. separate goroutines query the database and store the result in memory). My test code starts up a server and makes some database updates/queries itself.

Sometimes, the background goroutine which queries the database reports an error like update: pq: unexpected DataRow in simple query execution and the test itself errors out with pg.Close(): driver: bad connection.

These issues disappear once I add a pg.SetMaxOpenConns(1) call in my test setup.

So I was wondering: Are concurrent database connections supposed to work with txdb?

[flimzy]

TL;DR; No, txdb likely won't work with concurrent connections.

Each database transaction is tied to a specific connection. So if your actual connections are shared, that will obviously break txdb.

Setting max open conns to 1 is a good work-around, although that can lead to contention or deadlocks in certain applications.

In your case, where a caching layer is involved, I would try to disable the caching layer during tests, to reduce the number of variables at play in your testing scenario.

[stapelberg]

Thanks for confirming!

Setting max open conns to 1 is a good work-around, although that can lead to contention or deadlocks in certain applications.

Yes. Given that Go’s database/sql.Open results in a concurrent setting by default (max open conns defaults to 0, i.e. unlimited), would it make sense to mention pg.SetMaxOpenConns(1) in the README? I couldn’t find any reference to it before reporting this issue.

[flimzy]

That probably would be a good addition to the README and/or GoDoc. Feel like submitting a PR?

[stapelberg]

That probably would be a good addition to the README and/or GoDoc. Feel like submitting a PR?

Sure thing: #70