[Question] Best practices for injecting huge amounts of data (.NET 9)

[tovich37]

Hi,

Here's my business case:
I want to inject 500,000 leads into Dynamics 365 in the most optimized way possible.
This injection will be performed by a .NET 9 application with the latest version of the NuGet packages.

I have 5 App Regs available to perform this task (and only this task), and I thought I'd do the following:

1. Register the 5 ServiceClients (via their IOrganizationServiceAsync2 interface) as "scoped" (the connection strings are different) and disable the affinity cookie for all 5 services.

builder.Services.AddKeyedScoped<IOrganizationServiceAsync2>("app-reg-01", (_, __) =>
{
    // Create the service client
    ServiceClient.MaxConnectionTimeout = TimeSpan.FromMinutes(3);
    var client = new ServiceClient(appReg01ConnectionString)
    {
        EnableAffinityCookie = false
    };
    if (!client.IsReady)
        throw new InvalidOperationException($"Failed to connect to Dataverse (Message: {client.LastError}).", client.LastException);
    return client;
});

2. Create a DI scope.
3. Split my leads into groups of 15 ExecuteMultipleRequests and add them to a ConcurrentQueue.
4. Create 50 tasks (Task.Run) per App Reg (so 5 * 50 = a list of 250 tasks).
   All 50 tasks for the same App Reg would use the same ServiceClient (eg: sp.GetRequiredKeyedService<IOrganizationServiceAsync2>("app-reg-01");)
5. The tasks execute a TryDequeue (until the queue is empty) and use the ExecuteAsync method to create the leads.
6. Wait for all tasks to complete with the Task.WhenAll(my list of 250 tasks) method.

My questions are:

• Is this optimized? After some testing, I've gotten good results, with no memory leaks or other issues.
• Should the ServiceClient Clone() method be used?

Thanks!

[JimDaly]

Why aren't you using CreateMultipleRequest instead of ExecuteMultipleRequest?

See the guidance here Optimize performance for bulk operations.