Problem with WCF Distributed Transactions - Crash on transaction rollback
Description
Environment
Activity
OrenE May 15, 2009 at 11:27 PM
The problem is that no one (including the original reporting) cannot reproduce this issue.
At that point, I would say that this is a non issue.
Trying to reproduce multi threading issues is notoriously difficult, which is why I included the reference to NH-1769, as the probably cause.
Fabio Maulo May 15, 2009 at 11:16 PM
But I can't recreate the issue before apply patch even in MultiThreadedTransaction
OrenE May 15, 2009 at 10:09 PM
Cannot reproduce this issue
Reporter cannot reproduce this issue anymore
was the likely cause, and that was fixed.
Fabio Maulo May 13, 2009 at 8:09 PM
Denis, I can't recreate the issue even using our MultiThreadRunner.
check the NHibernate.Test.NHSpecificTest.DtcFailures.DtcFailuresFixture.MultiThreadedTransaction()
The test run a similar tests commited by Ayende in 20 threads.
Denis Shevchenko March 26, 2009 at 10:33 AM
I have not found way how to write simple test for this issue.
Currently I am able to reproduce the problem only when WCF is in use in my current project.
The most interesting...
I am unable to reproduce the problem in my solution when NHibernate revision 4151 or later is used.
I found that the problem was fixed with the following line of the code (see rev. 4151):
ambientTransation.EnlistVolatile(this, EnlistmentOptions.EnlistDuringPrepareRequired);
[see AbstractSessionImpl.cs, class NHibernate.Impl.AbstractSessionImpl, method void EnlistInAmbientTransactionIfNeeded(), last line of the method]
The problem depends on what enlistment option is used. If EnlistmentOptions.EnlistDuringPrepareRequired is used everything works fine but if EnlistmentOptions.None is used the problem appears.
I tried to change enlistment option in revision before 4151 to EnlistDuringPrepareRequired and it fixes the problem. I also tried to change enlistment option to EnlistmentOptions.None in the latest revision (4154) and it leads to the problem.
Let me know if you can explain this behavior or if you still need the code to reproduce the problem.
Details
Details
Assignee
Reporter
Denis ShevchenkoComponents
Affects versions
Priority
Major
WCF + DTC + NHibernate (rev. 4148)
Crash with InvalidOperationException: "Collection was modified; enumeration operation may not execute." in
NHibernate.dll!NHibernate.Engine.ActionQueue.AfterTransactionCompletion(bool success = false) Line 177.
The problem happens when client distributed transaction is used.
When transaction is completing session receives two notifications:
1. Handler of TransactionCompleted event (anonymous delegate defined in NHibernate.Impl.AbstractSessionImpl.EnlistInAmbientTransactionIfNeeded is called)
2. Implementation of IEnlistmentNotification.Rollback in NHibernate.Impl.AbstractSessionImpl is called
Both methods call AfterTransactionCompletion(bool success, ITransaction tx) method that is implemented in the SessionImpl class. This method calls actionQueue.AfterTransactionCompletion(success) where loop (foreach) over "executions" elements is executing with clear of the "executions" collection after it.
The problem that both notifications happen in asynchronous fashion on thread pool threads and IN THE SAME TIME. The code doesn't contain multi-thread protection for "executions" collection. Under some circumstances it leads to cleaning of the "executions" collection in Thread B (see call stacks below) when Thread A is executing some iteration over "executions" collection. When Thread A starts next iteration "Collection was modified" exception is thrown.
The problem is also described here :
http://groups.google.com/group/nhusers/browse_thread/thread/a7c7ce9ccc6c4564
Relative issue (current issue is reproduced on my computer in about 20% of attempts but sometimes crash happens instead of it):
NH-1676: Bug in WCF Distributed Transactions - NHibernate should implement workaround
(https://nhibernate.jira.com/browse/NH-1676)
Call stacks:
------------------- Thread A (InvalidOperationException: "Collection was modified; enumeration operation may not execute.")
------------------- > NHibernate.dll!NHibernate.Engine.ActionQueue.AfterTransactionCompletion(bool success = false) Line 177 + 0x1ba bytes C#
NHibernate.dll!NHibernate.Impl.SessionImpl.AfterTransactionCompletion(bool success = false, NHibernate.ITransaction tx = null) Line 371 + 0x2f bytes C#
NHibernate.dll!NHibernate.Impl.AbstractSessionImpl.EnlistInAmbientTransactionIfNeeded.AnonymousMethod(object sender = {System.Transactions.Transaction}, System.Transactions.TransactionEventArgs e = {System.Transactions.TransactionEventArgs}) Line 305 + 0x1b bytes C#
[Native to Managed Transition]
[Managed to Native Transition]
System.Transactions.dll!System.Transactions.TransactionStatePromotedAborted.EnterState(System.Transactions.InternalTransaction tx) + 0x60 bytes
System.Transactions.dll!System.Transactions.InternalTransaction.DistributedTransactionOutcome(System.Transactions.InternalTransaction tx, System.Transactions.TransactionStatus status) + 0x9a bytes
System.Transactions.dll!System.Transactions.Oletx.RealOletxTransaction.FireOutcome(System.Transactions.TransactionStatus statusArg) + 0x1cc bytes
System.Transactions.dll!System.Transactions.Oletx.OutcomeEnlistment.InvokeOutcomeFunction(System.Transactions.TransactionStatus status = Aborted) + 0xb0 bytes
System.Transactions.dll!System.Transactions.Oletx.OletxTransactionManager.ShimNotificationCallback(object state, bool timeout) + 0x59f bytes
mscorlib.dll!System.Threading._ThreadPoolWaitOrTimerCallback.PerformWaitOrTimerCallback(object state, bool timedOut) + 0x5e bytes
------------------- Thread B:
------------------- [In a sleep, wait, or join]
System.Transactions.dll!System.Transactions.Enlistment.Done() + 0xe2 bytes
> NHibernate.dll!NHibernate.Impl.AbstractSessionImpl.System.Transactions.IEnlistmentNotification.Rollback(System.Transactions.Enlistment enlistment = {System.Transactions.Enlistment}) Line 283 + 0x12 bytes C#
System.Transactions.dll!System.Transactions.InternalEnlistment.System.Transactions.IEnlistmentNotificationInternal.Rollback(System.Transactions.IPromotedEnlistment enlistment) + 0x28 bytes
System.Transactions.dll!System.Transactions.Oletx.OletxVolatileEnlistment.Rollback() + 0x14e bytes
System.Transactions.dll!System.Transactions.Oletx.OletxPhase1VolatileEnlistmentContainer.Aborted() + 0x12d bytes
System.Transactions.dll!System.Transactions.Oletx.OletxTransactionManager.ShimNotificationCallback(object state, bool timeout) + 0x5fe bytes
mscorlib.dll!System.Threading._ThreadPoolWaitOrTimerCallback.PerformWaitOrTimerCallback(object state, bool timedOut) + 0x5e bytes