Starting with string in the first post we continue to study examples with collection transformations and how they affect our applications.

• Part 1: string to array of chars
• Part 2: automatic diagnostics
• Part 3: collections

Collections

.NET contains hundreds of different collections for any purpose, but it looks like developers used to use List<T> or Array so much that they convert anything into an array just in case. In most cases, this is unnecessary. Sometimes it's harmful as we found out earlier. Let's look at some examples:

Count elements

Sometimes we don't need all elements, but only count them. And we can see the code like this:

return certificates.Where(x => certificateStatesByFormId.SafeGet(x.Id) == Form.CertificateState.Revoked || x.ValidToTicks <= nowTicks - 1)
   .ToArray().Length;

Or another example:

public int GetTotalAmount(Guid managerId)
{
   return filterSettingsHandler
       .Select<ClientFilterDbo>(managerId)
       .Where(IsVisible)
       .ToArray().Length;
}

Creating an array to count elements is unnecessary. LINQ provides a Count() method that does what we need without harmful overhead. And we can easily rewrite examples:

return certificates.Count(x => certificateStatesByFormId.SafeGet(x.Id) == Form.CertificateState.Revoked || x.ValidToTicks <= nowTicks - 1);

public int GetTotalAmount(Guid managerId)
{
   return filterSettingsHandler
       .Select<ClientFilterDbo>(managerId)
       .Count(IsVisible);
}

We get the cleaner and more readable code. Besides that we got some performance improvements.

namespace ToCharArrayBenchmark
{
    using System.Linq;
    using BenchmarkDotNet.Attributes;
    using BenchmarkDotNet.Jobs;

    [HtmlExporter]
    [RPlotExporter]
    [SimpleJob(RuntimeMoniker.Net48, baseline: true)]
    [SimpleJob(RuntimeMoniker.NetCoreApp31)]
    [SimpleJob(RuntimeMoniker.Net50)]
    [MemoryDiagnoser]
    public class CollectionsCountBenchmark
    {
        [Params(10, 20, 50, 100, 500)]
        public int N { get; set; }
        
        private string[] array;
        
        [GlobalSetup]
        public void SetUp()
        {
            array = new string[N];
            for (int i = 0; i < N; i++)
            {
                array[i] = i.ToString();
            }
        }

        [Benchmark(Baseline = true)]
        public int Count()
        {
            return array.Where(o => o != "1").Count();
        }
        
        [Benchmark]
        public int ArrayLength()
        {
            return array.Where(o => o != "1").ToArray().Length;
        }
    }
}

BenchmarkDotNet=v0.13.0, OS=Windows 10.0.19043.1348 (21H1/May2021Update)
AMD Ryzen 7 4700U with Radeon Graphics, 1 CPU, 8 logical and 8 physical cores
  [Host]             : .NET Framework 4.8 (4.8.4420.0), X64 RyuJIT
  .NET 5.0           : .NET 5.0.7 (5.0.721.25508), X64 RyuJIT
  .NET Core 3.1      : .NET Core 3.1.20 (CoreCLR 4.700.21.47003, CoreFX 4.700.21.47101), X64 RyuJIT
  .NET Framework 4.8 : .NET Framework 4.8 (4.8.4420.0), X64 RyuJIT

// * Legends *
  N         : Value of the 'N' parameter
  Mean      : Arithmetic mean of all measurements
  Error     : Half of 99.9% confidence interval
  StdDev    : Standard deviation of all measurements
  Ratio     : Mean of the ratio distribution ([Current]/[Baseline])
  RatioSD   : Standard deviation of the ratio distribution ([Current]/[Baseline])
  Gen 0     : GC Generation 0 collects per 1000 operations
  Allocated : Allocated memory per single operation (managed only, inclusive, 1KB = 1024B)
  1 us      : 1 Microsecond (0.000001 sec)

The following diagram shows execution time of both methods (in microseconds) for .NET 5 platform:

As we see converting to array is about 2.5 times slower even for the most optimized .NET 5. Other frameworks are up to 4 times slower.

And if this is not enough let's look at memory allocations (in bytes):

The LINQ method does not create many objects and does not lead to additional GC regardless of the size of the original collection.

We can confidently say that creating an array to count elements is a harmful transformation.

ToArray() with LINQ

The very common pattern - to call ToArray() (just in case) before LINQ-methods.

var correctPaymentsByBillId = correctPayments
   .GroupBy(x => x.BillId)
   .ToDictionary(x => x.Key, x => x.ToArray().Last());

Here GroupBy returns IGrouping<out TKey, out TElement> interface and it, in turns, implements IEnumerable<TElement>. Based on this we can simply remove this ToArray():

var correctPaymentsByBillId = correctPayments
   .GroupBy(x => x.BillId)
   .ToDictionary(x => x.Key, x => x.Last());

Array is often used as a return type of methods like this:

private OrderPackage[] ReadRootPackages(params Guid[] packageIds)
{
    var rootPackages = new Dictionary<Guid, OrderPackage>();
    //...
    return rootPackages.Values.ToArray();
}

And the result doesn't have to be an array or any specific collection:

public OrderPackageDto[] SelectTariffForestByPackageIds(Guid[] packageIds)
{
	var rootOrderPackages = ReadRootPackages(packageIds);
	var masterPackageIds = rootOrderPackages.Select(package => package.Id).ToArray();
	...
}

As we see, ReadRootPackages can return ICollection<OrderPackage> or IEnumerable<OrderPackage> and ToArray() is redundant:

private IEnumerable<OrderPackage> ReadRootPackages(params Guid[] packageIds)
{
    var rootPackages = new Dictionary<Guid, OrderPackage>();
    //...
    return rootPackages.Values;
}

The most obvious example is declaring method return type as IEnumerable but returning an array:

public IEnumerable<Division> GetDivisions(HashSet<SupportUserAccessRight> accessRights)
{
   return divisions.Where(d => d.AccessRights.Intersect(accessRights).Any()).OrderBy(d => d.Priority).ToArray();
}

Like the previous example GetDivisions is used in LINQ-methods chain and we can remove ToArray() call:

public IEnumerable<Division> GetDivisions(HashSet<SupportUserAccessRight> accessRights)
{
   return divisions.Where(d => d.AccessRights.Intersect(accessRights).Any()).OrderBy(d => d.Priority);
}

Usually, you don't need a collection between LINQ-methods, because of the lazy evaluation nature of LINQ. How does ToArray() affect performance depending on different collections? Let's compare two versions - with ToArray() and without it.

namespace ToCharArrayBenchmark
{
    using System.Collections.Generic;
    using System.Linq;
    using BenchmarkDotNet.Attributes;
    using BenchmarkDotNet.Engines;
    using BenchmarkDotNet.Jobs;

    [HtmlExporter]
    [RPlotExporter]
    [SimpleJob(RuntimeMoniker.Net48, baseline: true)]
    [SimpleJob(RuntimeMoniker.NetCoreApp31)]
    [SimpleJob(RuntimeMoniker.Net50)]
    [MemoryDiagnoser]
    public class CollectionToArrayBenchmark
    {
        [Params(10, 20, 50, 100, 500)]
        public int N { get; set; }

        private List<string> list;
        private string[] array;
        private HashSet<string> hashSet;
        private Dictionary<long, string> dictionary;
        private readonly Consumer consumer = new Consumer();

        [GlobalSetup]
        public void SetUp()
        {
            array = new string[N];
            list = new List<string>(N);
            dictionary = new Dictionary<long, string>(N);
            hashSet = new HashSet<string>(N);

            for (int i = 0; i < N; i++)
            {
                list.Add(i.ToString());
                array[i] = i.ToString();
                hashSet.Add(i.ToString());
                dictionary[i] = i.ToString();
            }
        }

        [Benchmark(Baseline = true)]
        public void FilterArray()
        {
            GetFilteredEnumerable(array).Consume(consumer);
        }

        [Benchmark]
        public void FilterList()
        {
            GetFilteredEnumerable(list).Consume(consumer);
        }

        [Benchmark]
        public void FilterHashSet()
        {
            GetFilteredEnumerable(hashSet).Consume(consumer);
        }

        [Benchmark]
        public void FilterDictionary()
        {
            GetFilteredEnumerable(dictionary.Values).Consume(consumer);
        }

        [Benchmark]
        public void FilterArray_ToArray()
        {
            GetFilteredArray(array).Consume(consumer);
        }

        [Benchmark]
        public void FilterList_ToArray()
        {
            GetFilteredArray(list).Consume(consumer);
        }

        [Benchmark]
        public void FilterHashSet_ToArray()
        {
            GetFilteredArray(hashSet).Consume(consumer);
        }

        [Benchmark]
        public void FilterDictionary_ToArray()
        {
            GetFilteredArray(dictionary.Values).Consume(consumer);
        }

        private static IEnumerable<string> GetFilteredEnumerable(IEnumerable<string> strings)
        {
            return strings.Where(o => o.Length > 1).Select(o => o);
        }

        private static string[] GetFilteredArray(IEnumerable<string> strings)
        {
            return strings.Where(o => o.Length > 1).ToArray();
        }
    }
}

BenchmarkDotNet=v0.13.0, OS=Windows 10.0.19043.1288 (21H1/May2021Update)
AMD Ryzen 7 4700U with Radeon Graphics, 1 CPU, 8 logical and 8 physical cores
  [Host]             : .NET Framework 4.8 (4.8.4420.0), X64 RyuJIT
  .NET 5.0           : .NET 5.0.7 (5.0.721.25508), X64 RyuJIT
  .NET Core 3.1      : .NET Core 3.1.20 (CoreCLR 4.700.21.47003, CoreFX 4.700.21.47101), X64 RyuJIT
  .NET Framework 4.8 : .NET Framework 4.8 (4.8.4420.0), X64 RyuJIT

// * Legends *
  N         : Value of the 'N' parameter
  Mean      : Arithmetic mean of all measurements
  Error     : Half of 99.9% confidence interval
  StdDev    : Standard deviation of all measurements
  Ratio     : Mean of the ratio distribution ([Current]/[Baseline])
  RatioSD   : Standard deviation of the ratio distribution ([Current]/[Baseline])
  Gen 0     : GC Generation 0 collects per 1000 operations
  Allocated : Allocated memory per single operation (managed only, inclusive, 1KB = 1024B)
  1 us      : 1 Microsecond (0.000001 sec)

The following diagram shows execution time of both methods (in microseconds) for .NET 5 platform:

The results are quite comparable, but .NET Framework is not so good:

Additional ToArray() call is about 2.5 times slower.

.NET made huge improvements in Core 3.1 and 5 versions. The following diagram shows execution time (in microseconds) of different .NET versions depending on number of elements:

Array arguments

We used to use arrays as arguments in methods, but this can be a place where we can make another unnecessary transformation. Let's look at the example:

private async Task<OrderPackage[]> ReadRootPackagesAsync(params Guid[] packageIds)
{
	var queryPackageIds = new HashSet<Guid>(packageIds);
	...
	var packageIds = queryPackageIds.ToArray();
	var orderPackages = await orderPackageService.SelectByIdsAsync(packageIds).ConfigureAwait(false);
	...
}

public Task<OrderPackage[]> SelectByIdsAsync(params Guid[] packageIds)
{
	...
	return GetTable().Where(o => packageIds.Contains(o.Id));
}

GetTable() is a method that returns IQueryable<OrderPackage>, so the Where expression will be converted to a SQL code and it's not necessary to use any specific collection. Here we can use IEnumerable<Guid> again, but I prefer an interface IReadOnlyCollection<T> that was introduced in .NET 4.5 Framework. The IReadOnlyCollection<T> interface indicates that collection is materialized unlike IEnumerable<T> but we are not interested in a collection's structure (HashSet, List, Array, ... - does not matter). And on the other hand using the IReadOnlyCollection<T> prevents the possibility of multiple enumeration of IEnumerable<T>. So, our modified example:

private async Task<OrderPackage[]> ReadRootPackagesAsync(params Guid[] packageIds)
{
	var queryPackageIds = new HashSet<Guid>(packageIds);
	...
	var orderPackages = await orderPackageService.SelectByIdsAsync(queryPackageIds).ConfigureAwait(false);
	...
}

public Task<OrderPackage[]> SelectByIdsAsync(IReadOnlyCollection<Guid> packageIds)
{
	...
	return GetTable().Where(o => packageIds.Contains(o.Id));
}

We pass the HashSet<Guid> directly to the SelectByIdsAsync method without using intermediate variables.

But we should be careful using common interfaces like IEnumerable<T> or IReadOnlyCollection<T> to avoid antipatterns like this:

public UserSearchModel BuildUserSearchModel(IEnumerable<IUserProps> userPropses)
{
	var users = new List<UserModel>();
	var allUserPropses = userPropses.ToArray();
	foreach (var userProps in allUserPropses)
	{
		...
	}
	
	return new UserSearchModel
	{
		Users = users.ToArray(),
		QueryModel = queryModel,
		TotalCount = total,
		AvailableCount = users.Count,
		HasMore = allUserPropses.Length >= queryModel.PageSize,
		QueryForm = queryFormBuilder.Build(queryModel)
	};
}

The argument userPropses converted to a local array allUserPropses to avoid multiple enumeration as we learned before. userPropses has type IEnumerable<IUserProps>, so it deserves our attention. But what we will see, if we look at the BuildUserSearchModel call?

public async Task<SearchModel> SearchAsync(SearchQuery searchQuery)
{
	var userPropses = new List<IUserProps>();
	...
	return searchModelBuilder.BuildUserSearchModel(userPropses);
}

It becomes obvious that IEnumerable<IUserProps> is not the best choice. It confuses developers because it says that userPropses is not a collection, but it is in fact. The better variant - replace it with IReadOnlyCollection<IUserProps>:

public UserSearchModel BuildUserSearchModel(IReadOnlyCollection<IUserProps> userPropses)
{
	var users = new List<UserModel>();
	foreach (var userProps in userPropses)
	{
		...
	}
	
	return new UserSearchModel
	{
		Users = users.ToArray(),
		QueryModel = queryModel,
		TotalCount = total,
		AvailableCount = users.Count,
		HasMore = userPropses.Count >= queryModel.PageSize,
		QueryForm = queryFormBuilder.Build(queryModel)
	};
}

Code became cleaner, no conversions, no overhead.

Fluent assertions in unit tests

I would like to make a small note about the Fluent Assertions library. We write many tests where we need to compare two collections:

[Test]
public void TestSingleWriteInSameTimeline()
{
	var testData = new List<TestTimelineData>();
	...
	
	var actual = timeline.Select<TestTimelineData>(timelineId, ticks, TestWriteCount);

	actual.Select(x => x.Data).ToArray().Should().BeEquivalentTo(testData.ToArray());
}

But the method BeEquivalentTo works fine with enumerations. We can verify this by looking at its definition:

public AndConstraint<TAssertions> BeEquivalentTo<TExpectation>(IEnumerable<TExpectation> expectation,
	string because = "", params object[] becauseArgs)
{
	return BeEquivalentTo(expectation, config => config, because, becauseArgs);
}

Fluent Assertions will create arrays anyway, so you won't get much performance benefits, but still can keep your code cleaner:

[Test]
public void TestSingleWriteInSameTimeline()
{
	var testData = new List<TestTimelineData>();
	...
	
	var actual = timeline.Select<TestTimelineData>(timelineId, ticks, TestWriteCount);

	actual.Select(x => x.Data).Should().BeEquivalentTo(testData));
}

Analyzer

If you are worried about performance and memory management or just a purity of code, you should be very attentive with collection manipulations. Sometimes, it's difficult to notice a problem, especially after refactorings: you change a method signature from int[] to IEnumerable<int> but don't check all callings - it can lead to some issues like the ones we reviewed earlier.

Unfortunately, there is not a built-in tool to detect problems like this, because many of them need complex analysis. I started Collections.Analyzer project that can warn about potential problems:

And more obvious cases:

Links

• IReadOnlyCollection<T>
• Code Inspection: Possible multiple enumeration of IEnumerable
• Fluent Assertions
• Collections.Analyzer