[聚合问答] SqlException from Entity Framework - New transaction is not allowed because there are other threads running in the session

After much pulling out of hair I discovered that the foreach loops were the culprits. What needs to happen is to call EF but return it into an IList<T> of that target type then loop on the IList<T>.

Example:

IList<Client> clientList = from a in _dbFeed.Client.Include("Auto") select a;
foreach (RivWorks.Model.NegotiationAutos.Client client in clientList)
{
   var companyFeedDetailList = from a in _dbRiv.AutoNegotiationDetails where a.ClientID == client.ClientID select a;
    // ...
}

As you've already identified, you cannot save from within a foreach that is still drawing from the database via an active reader.

Calling ToList() or ToArray() is fine for small data sets, but when you have thousands of rows, you will be consuming a large amount of memory.

It's better to load the rows in chunks.

public static class EntityFrameworkUtil
{
    public static IEnumerable<T> QueryInChunksOf<T>(this IQueryable<T> queryable, int chunkSize)
    {
        return queryable.QueryChunksOfSize(chunkSize).SelectMany(chunk => chunk);
    }

    public static IEnumerable<T[]> QueryChunksOfSize<T>(this IQueryable<T> queryable, int chunkSize)
    {
        int chunkNumber = 0;
        while (true)
        {
            var query = (chunkNumber == 0)
                ? queryable 
                : queryable.Skip(chunkNumber * chunkSize);
            var chunk = query.Take(chunkSize).ToArray();
            if (chunk.Length == 0)
                yield break;
            yield return chunk;
            chunkNumber++;
        }
    }
}

Given the above extension methods, you can write your query like this:

foreach (var client in clientList.OrderBy(c => c.Id).QueryInChunksOf(100))
{
    // do stuff
    context.SaveChanges();
}

The queryable object you call this method on must be ordered. This is because Entity Framework only supports IQueryable<T>.Skip(int) on ordered queries, which makes sense when you consider that multiple queries for different ranges require the ordering to be stable. If the ordering isn't important to you, just order by primary key as that's likely to have a clustered index.

This version will query the database in batches of 100. Note that SaveChanges() is called for each entity.

If you want to improve your throughput dramatically, you should call SaveChanges() less frequently. Use code like this instead:

foreach (var chunk in clientList.OrderBy(c => c.Id).QueryChunksOfSize(100))
{
    foreach (var client in chunk)
    {
        // do stuff
    }
    context.SaveChanges();
}

This results in 100 times fewer database update calls. Of course each of those calls takes longer to complete, but you still come out way ahead in the end. Your mileage may vary, but this was worlds faster for me.

And it gets around the exception you were seeing.

EDIT I revisited this question after running SQL Profiler and updated a few things to improve performance. For anyone who is interested, here is some sample SQL that shows what is created by the DB.

The first loop doesn't need to skip anything, so is simpler.

SELECT TOP (100)                     -- the chunk size 
[Extent1].[Id] AS [Id], 
[Extent1].[Name] AS [Name], 
FROM [dbo].[Clients] AS [Extent1]
ORDER BY [Extent1].[Id] ASC

Subsequent calls need to skip previous chunks of results, so introduces usage of row_number:

SELECT TOP (100)                     -- the chunk size
[Extent1].[Id] AS [Id], 
[Extent1].[Name] AS [Name], 
FROM (
    SELECT [Extent1].[Id] AS [Id], [Extent1].[Name] AS [Name], row_number()
    OVER (ORDER BY [Extent1].[Id] ASC) AS [row_number]
    FROM [dbo].[Clients] AS [Extent1]
) AS [Extent1]
WHERE [Extent1].[row_number] > 100   -- the number of rows to skip
ORDER BY [Extent1].[Id] ASC

We have now posted an official response to the bug opened on Connect. The workarounds we recommend are as follows:

This error is due to Entity Framework creating an implicit transaction during the SaveChanges() call. The best way to work around the error is to use a different pattern (i.e., not saving while in the midst of reading) or by explicitly declaring a transaction. Here are three possible solutions:

// 1: Save after iteration (recommended approach in most cases)
using (var context = new MyContext())
{
    foreach (var person in context.People)
    {
        // Change to person
    }
    context.SaveChanges();
}

// 2: Declare an explicit transaction
using (var transaction = new TransactionScope())
{
    using (var context = new MyContext())
    {
        foreach (var person in context.People)
        {
            // Change to person
            context.SaveChanges();
        }
    }
    transaction.Complete();
}

// 3: Read rows ahead (Dangerous!)
using (var context = new MyContext())
{
    var people = context.People.ToList(); // Note that this forces the database
                                          // to evaluate the query immediately
                                          // and could be very bad for large tables.

    foreach (var person in people)
    {
        // Change to person
        context.SaveChanges();
    }
} 

Just put context.SaveChanges() after end of your foreach(loop).

I was getting this same issue but in a different situation. I had a list of items in a list box. The user can click an item and select delete but I am using a stored proc to delete the item because there is a lot of logic involved in deleting the item. When I call the stored proc the delete works fine but any future call to SaveChanges will cause the error. My solution was to call the stored proc outside of EF and this worked fine. For some reason when I call the stored proc using the EF way of doing things it leaves something open.

FYI: from a book and some lines adjusted because its stil valid:

Invoking SaveChanges() method begins a transaction which automatically rolls back all changes persisted to the database if an exception occurs before iteration completes; otherwise the transaction commits. You might be tempted to apply the method after each entity update or deletion rather than after iteration completes, especially when you're updating or deleting massive numbers of entities.

If you try to invoke SaveChanges() before all data has been processed, you incur a "New transaction is not allowed because there are other threads running in the session" exception. The exception occurs because SQL Server doesn't permit starting a new transaction on a connection that has a SqlDataReader open, even with Multiple Active Record Sets (MARS) enabled by the connection string (EF's default connection string enables MARS)

Sometimes its better to understand why things are happening ;-)

So in the project were I had this exact same issue the problem wasn't in the foreach or the .toList() it was actually in the AutoFac configuration we used. This created some weird situations were the above error was thrown but also a bunch of other equivalent errors were thrown.

This was our fix: Changed this:

container.RegisterType<DataContext>().As<DbContext>().InstancePerLifetimeScope();
container.RegisterType<DbFactory>().As<IDbFactory>().SingleInstance();
container.RegisterType<UnitOfWork>().As<IUnitOfWork>().InstancePerRequest();

To:

container.RegisterType<DataContext>().As<DbContext>().As<DbContext>();
container.RegisterType<DbFactory>().As<IDbFactory>().As<IDbFactory>().InstancePerLifetimeScope();
container.RegisterType<UnitOfWork>().As<IUnitOfWork>().As<IUnitOfWork>();//.InstancePerRequest();

Always Use your selection as List

Eg:

var tempGroupOfFiles = Entities.Submited_Files.Where(r => r.FileStatusID == 10 && r.EventID == EventId).ToList();

Then Loop through the Collection while save changes

 foreach (var item in tempGroupOfFiles)
             {
                 var itemToUpdate = item;
                 if (itemToUpdate != null)
                 {
                     itemToUpdate.FileStatusID = 8;
                     itemToUpdate.LastModifiedDate = DateTime.Now;
                 }
                 Entities.SaveChanges();

             }

I was also facing same issue.

Here is the cause and solution.

http://blogs.msdn.com/b/cbiyikoglu/archive/2006/11/21/mars-transactions-and-sql-error-3997-3988-or-3983.aspx

Make sure before firing data manipulation commands like inserts, updates, you have closed all previous active SQL readers.

Most common error is functions that read data from db and return values. For e.g functions like isRecordExist.

In this case we immediately return from the function if we found the record and forget to close the reader.

In my case, the problem appeared when I called Stored Procedure via EF and then later SaveChanges throw this exception. The problem was in calling the procedure, the enumerator was not disposed. I fixed the code following way:

public bool IsUserInRole(string username, string roleName, DataContext context)
{          
   var result = context.aspnet_UsersInRoles_IsUserInRoleEF("/", username, roleName);

   //using here solved the issue
   using (var en = result.GetEnumerator()) 
   {
     if (!en.MoveNext())
       throw new Exception("emty result of aspnet_UsersInRoles_IsUserInRoleEF");
     int? resultData = en.Current;

     return resultData == 1;//1 = success, see T-SQL for return codes
   }
}

I needed to read a huge ResultSet and update some records in the table. I tried to use chunks as suggested in Drew Noakes's answer.

Unfortunately after 50000 records I've got OutofMemoryException. The answer Entity framework large data set, out of memory exception explains, that

EF creates second copy of data which uses for change detection (so that it can persist changes to the database). EF holds this second set for the lifetime of the context and its this set thats running you out of memory.

The recommendation is to renew your context each batch.

So I've retrieved Minimal and Maximum values of the primary key- the tables have primary keys as auto incremental integers.Then I retrieved from the database chunks of records by opening context for each chunk. After processing the chunk context closes and releases the memory. It insures that memory usage is not growing.

Below is a snippet from my code:

  public void ProcessContextByChunks ()
  {
        var tableName = "MyTable";
         var startTime = DateTime.Now;
        int i = 0;
         var minMaxIds = GetMinMaxIds();
        for (int fromKeyID= minMaxIds.From; fromKeyID <= minMaxIds.To; fromKeyID = fromKeyID+_chunkSize)
        {
            try
            {
                using (var context = InitContext())
                {   
                    var chunk = GetMyTableQuery(context).Where(r => (r.KeyID >= fromKeyID) && (r.KeyID < fromKeyID+ _chunkSize));
                    try
                    {
                        foreach (var row in chunk)
                        {
                            foundCount = UpdateRowIfNeeded(++i, row);
                        }
                        context.SaveChanges();
                    }
                    catch (Exception exc)
                    {
                        LogChunkException(i, exc);
                    }
                }
            }
            catch (Exception exc)
            {
                LogChunkException(i, exc);
            }
        }
        LogSummaryLine(tableName, i, foundCount, startTime);
    }

    private FromToRange<int> GetminMaxIds()
    {
        var minMaxIds = new FromToRange<int>();
        using (var context = InitContext())
        {
            var allRows = GetMyTableQuery(context);
            minMaxIds.From = allRows.Min(n => (int?)n.KeyID ?? 0);  
            minMaxIds.To = allRows.Max(n => (int?)n.KeyID ?? 0);
        }
        return minMaxIds;
    }

    private IQueryable<MyTable> GetMyTableQuery(MyEFContext context)
    {
        return context.MyTable;
    }

    private  MyEFContext InitContext()
    {
        var context = new MyEFContext();
        context.Database.Connection.ConnectionString = _connectionString;
        //context.Database.Log = SqlLog;
        return context;
    }

FromToRange is a simple structure with From and To properties.

Here are another 2 options that allow you to invoke SaveChanges() in a for each loop.

The first option is use one DBContext to generate your list objects to iterate through, and then create a 2nd DBContext to call SaveChanges() on. Here is an example:

//Get your IQueryable list of objects from your main DBContext(db)    
IQueryable<Object> objects = db.Object.Where(whatever where clause you desire);

//Create a new DBContext outside of the foreach loop    
using (DBContext dbMod = new DBContext())
{   
    //Loop through the IQueryable       
    foreach (Object object in objects)
    {
        //Get the same object you are operating on in the foreach loop from the new DBContext(dbMod) using the objects id           
        Object objectMod = dbMod.Object.Find(object.id);

        //Make whatever changes you need on objectMod
        objectMod.RightNow = DateTime.Now;

        //Invoke SaveChanges() on the dbMod context         
        dbMod.SaveChanges()
    }
}

The 2nd option is to get a list of database objects from the DBContext, but to select only the id's. And then iterate through the list of id's (presumably an int) and get the object corresponding to each int, and invoke SaveChanges() that way. The idea behind this method is grabbing a large list of integers, is a lot more efficient then getting a large list of db objects and calling .ToList() on the entire object. Here is an example of this method:

//Get the list of objects you want from your DBContext, and select just the Id's and create a list
List<int> Ids = db.Object.Where(enter where clause here)Select(m => m.Id).ToList();

var objects = Ids.Select(id => db.Objects.Find(id));

foreach (var object in objects)
{
    object.RightNow = DateTime.Now;
    db.SaveChanges()
}