Table of Contents

Best practices for data importing

Introduction

Data importing can be a complex process, especially when dealing with large volumes of data or data from multiple sources. In this article, we will discuss some best practices for data importing to ensure that the process is efficient, accurate, and reliable.

File type

When importing data, it is important to consider the file type. The most common file types for data importing are CSV, Excel, and XML. Each file type has its own advantages and disadvantages, so it is important to choose the right file type for your specific needs. For example, CSV files are lightweight and easy to work with, but they do not support complex data structures. Excel files, on the other hand, support complex data structures but can be heavy and difficult to work with. XML files are flexible and support complex data structures, but they can be verbose and difficult to read. In case of large files, it is recommended to use CSV files because they are lightweight and easy to work with.

Task runner service

One of the key components of a data importing process is the task runner service. To use the task runner service, you need to define a background server methods that will be executed by the task runner service. The user will trigger the import, but the request will be transmitted to the task runner service, and notifications can be defined to inform the user of the progress and success/failure of the import. You can also develop a supervision screen based on the data recorded in the history tables (not provided by the framework).

Batch import

In the case of large volumes, data import should be done in batches. You should not attempt to save the data in a single transaction. To each batch, you need to :

  • recreate the scope for each batch of data. This resets the Entity Framework Core context and frees up memory.
  • save data in the database in batches.
using IServiceScope scope = serviceProvider.CreateScope();
IUnitOfWork unitOfWork = scope.ServiceProvider.GetRequiredService<IUnitOfWork>();
IImportFileLineRepository repository = scope.ServiceProvider.GetRequiredService<IImportFileLineRepository>();

foreach (ImportFileLine importFileLine in importFileLines)
{
    repository.Add(importFileLine);
}

await unitOfWork.SaveAsync();

Save options for bulk imports

For high-volume imports, you can tune the save pipeline with SaveOptions.

Typical optimization strategy:

  • Keep validation rules enabled when you need data quality checks.
  • Disable event rules for technical imports that do not require business side effects.
  • Disable SQL query logging to reduce log volume and overhead during large batches.

In some technical workflows, disabling validation rules can provide a significant performance gain when validation logic executes SQL queries (directly or indirectly through repositories/services). Treat this as an exceptional optimization only.

SaveOptions saveOptions = new()
{
    EnableValidationRules = true,
    EnableEventRules = false,
    EnableQueryLogging = false,
};

Result result = await unitOfWork.SaveAsync(saveOptions, cancellationToken);
if (result.IsFailed)
{
    // Handle errors and stop import if needed.
}

You can also combine save mode and options:

Result result = await unitOfWork.SaveAsync(
    SaveMode.NeverThrow,
    new SaveOptions
    {
        EnableValidationRules = true,
        EnableEventRules = false,
        EnableQueryLogging = false,
    },
    cancellationToken);

Use this carefully: disabling rules changes behavior and can bypass business logic.

For a complete workflow example, see Large-volume import and data adjustment workflows.

Warning

Disabling validation or event rules can allow invalid business states to be persisted. This should be treated as an exceptional operation and never as a default optimization. Use it only when you have strict functional controls, operational supervision, and post-processing verification. The risk is higher when validation rules include SQL calls: disabling them may improve throughput a lot, but it also removes important integrity safeguards.

A common advanced scenario is post-migration data adjustment: after database migration, technical treatments may need to update existing data while avoiding temporary blocks from business rules. In that case, apply rule bypasses only for the treatment window, and restore standard behavior immediately after completion.

Warning

In case of a background server method, the current tenant service need to be configured manually. To read more about this, please refer to the documentation.

File memory consumption

If you load the entire file into memory, you may run out of memory if the file is too large. To avoid this, you can read the file line by line or in chunks. This will reduce memory consumption and improve performance. Some libraries like Sep can help you read large files efficiently.

See also