EntityWorkItemRepository.RunInTransaction leads to InvalidOperationException - the connection was not closed

[TimLovellSmith]

We've seen a lot of errors showing up in our logs like this.

System.InvalidOperationException: The connection was not closed.
at System.Data.EntityClient.EntityConnection.Open()
at WebBackgrounder.EntityWorkItemRepository.RunInTransaction(Action query)
at WebBackgrounder.WebFarmJobCoordinator.ReserveWork(String workerId, IJob job)
at WebBackgrounder.WebFarmJobCoordinator.GetWork(IJob job)
at WebBackgrounder.JobManager.DoNextJob()
at WebBackgrounder.JobManager.OnTimerElapsed(Object sender)

[aplcr]

Hi, I've been seeing this a lot too, and it seems to kill the job / stop it from recurring. Any thoughts?

[TimLovellSmith]

So, read through EntityWorkItemRepository code a little.
EntityWorkItemRepository takes a factory thunk for creating contexts (IWorkItemsContext) which it

1. steals the connection from
2. runs a query
3. disposes of the context and regenerates it IF no exception was thrown during the query

The problem scenario is likely that
a) first time we run a query we threw an exception
b) second time we run a query, the same context item gets reused, but since we already opened the connection for the previous query, boom.

[TimLovellSmith]

Just found evidence to confirm the theory, so I'll work on the pull request.

Before the first InvalidOperationException in the logs we have a stack trace going through RunInTransaction.

System.Data.Entity.Infrastructure.DbUpdateException: An error occurred while updating the entries. See the inner exception for details. ---> System.Data.UpdateException: An error occurred while updating the entries. See the inner exception for details. ---> System.Data.SqlClient.SqlException: Transaction (Process ID 495) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action1 wrapCloseInAction) at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose) at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady) at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString) at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite) at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource1 completion, Int32 timeout, Task& task, Boolean asyncWrite)
at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource1 completion, String methodName, Boolean sendToPipe, Int32 timeout, Boolean asyncWrite) at System.Data.SqlClient.SqlCommand.ExecuteNonQuery() at StackExchange.Profiling.Data.ProfiledDbCommand.ExecuteNonQuery() in C:\Users\sam\Desktop\MiniProfiler\StackExchange.Profiling\Data\ProfiledDbCommand.cs:line 182 at System.Data.Mapping.Update.Internal.DynamicUpdateCommand.Execute(UpdateTranslator translator, EntityConnection connection, Dictionary2 identifierValues, List`1 generatedValues)
at System.Data.Mapping.Update.Internal.UpdateTranslator.Update(IEntityStateManager stateManager, IEntityAdapter adapter)
--- End of inner exception stack trace ---
at System.Data.Mapping.Update.Internal.UpdateTranslator.Update(IEntityStateManager stateManager, IEntityAdapter adapter)
at System.Data.Objects.ObjectContext.SaveChanges(SaveOptions options)
at System.Data.Entity.Internal.InternalContext.SaveChanges()
--- End of inner exception stack trace ---
at System.Data.Entity.Internal.InternalContext.SaveChanges()
at WebBackgrounder.EntityWorkItemRepository.SetWorkItemFailed(Int64 workItemId, Exception exception)
at WebBackgrounder.WebFarmJobCoordinator.<>c__DisplayClass7.b__6()
at WebBackgrounder.EntityWorkItemRepository.RunInTransaction(Action query)