DataSource ModelMapping

Overview

DataSourceModelMapping – model and field mapping

This document describes how the Data Service converts data from external sources (DataSource) into internal models (DataModel, DataObjectRecord) using the DataSourceModelMapping configuration. The goal is to give integrators and developers a clear, consistent guide on how to design, configure, and debug mappings.

1. Data flow – where the mapping comes from

1. An external system sends a batch of data for a specific DataSource (IncomingDataRecord, individual rows IncomingDataObject).
2. TransformationIncomingDataContext for each row:
   • determines entityMappingId (IncomingDataRow.GetEntityMappingId()),
   • selects the corresponding DataSourceModelMapping based on DataSource.Mappings and/or MappingDirectives,
   • sorts mappings by Priority.
3. EntityTransformer for each selected DataSourceModelMapping:
   • decides based on Predicate whether to apply the mapping (EvaluatePredicate),
   • computes externalId and optionally referenceId using FieldValueConverterFactory,
   • performs Delete (if the row is marked as deleted) or Upsert to unified data (TransformationOutputDataContext.DeleteDataObjectAsync / UpsertDataObjectAsync).
4. TransformationOutputDataContext:
   • looks up an existing DataObjectRecord (by ModelId, externalId, mandantCode),
   • creates or updates the record and persists changes,
   • can compute expression fields after transformation (ComputeExpressionFieldsAsync).

2. DataSourceModelMapping – property overview

Class: ASOL.DataService.Domain.Model.DataSourceModelMapping

Property	Type	Description
EntityMappingId	string?	Primary identifier of the entity mapping (used mainly in root mapping).
EntityId	string?	Entity identifier; if EntityMappingId is not provided, it is also used as the mapping identifier.
GetEntityMappingId()	string?	Returns EntityMappingId or EntityId (for compatibility with older data).
Priority	int	Order of mapping application (ascending). Lower priority = processed earlier.
EntityOwnership	DataSourceModelEntityOwnership?	Entity ownership settings in tenant/mandant context (uses EntityOwnershipResolver).
ModelId	Guid?	Target model ID (IDataModel) to which data is mapped. Without ModelId, the mapping is incomplete.
Options	ICollection<KeyValuePair<string, object?>>?	Optional KVP for fine-tuning behavior (e.g., index key for nested entities).
TransformationType	SourceExternalIdTransformationType	Method of computing externalId (default, special modes for nested entities, etc.).
AutoMapFields	bool	Automatically map fields by name/type (where supported by field type).
Mappings	ICollection<DataSourceFieldMapping>?	Explicit mapping of individual fields at the level of this model mapping.
Expressions	ICollection<DataSourceModelMappingExpression>?	Definitions of derived (expression) fields.
Predicate	DataSourceMappingPredicate?	Condition under which this mapping is applied.
IsSupplementary	bool (Obsolete)	Historical flag for supplementary mappings (no longer used).
CustomProperties	ICollection<KeyValuePair<string, object?>>?	Additional metadata used in transformation logic.

2.1 Mappings – mapping of individual fields

Mappings contains a list of DataSourceFieldMapping describing how individual input fields are projected into the target model:

• scalar fields (e.g., Name, Status, Amount),
• navigation fields (nested entity, collections) – format such as Items.Name or Items[KEY].Code.

Internally, the structure is analyzed in EntityTransformerStrategyBuilder and split into:

• IScalarSourceFieldsGroup – simple target fields,
• INavigationSourceFieldsGroup – nested/collections (NestedEntityFieldType).

2.2 Expressions – derived fields

Expressions define fields that are not directly mapped from a single input value but are computed from multiple inputs.

Use cases:

• calculations (e.g., amount with VAT),
• composed values (e.g., FullName = FirstName + " " + LastName),
• logical derivation.

They are evaluated in TransformationOutputDataContext (ComputeExpressionFieldsAsync, ComputeExpressionFieldsOfSingleEntityAsync).

2.3 Predicate – when to apply the mapping

Predicate controls when a mapping is applied:

• If Predicate is empty → the mapping is always applied.
• If Predicate.Value is provided, EntityTransformer.EvaluatePredicate evaluates the expression against IncomingDataCache (common/custom fields of the input row).

Typical use:

• distinguishing by record type (e.g., only for Type = "ORDER"),
• distinguishing by version, state, etc.

2.4 Options – additional mapping behavior

Options is an optional set of KVPs primarily for:

• special keys for nested entities (e.g., how to determine indexKey),
• fine-tuned modes for a specific data source.

It is used for example in EntityTransformer.TransformSingleNestedEntityAsync and helper methods like TryResolveOptionsIndexKeyValue.

3. JSON configuration examples

Below are simplified JSON examples of how the DataSourceModelMapping configuration can look. The actual storage (Mongo, config files) may differ, but the value structure corresponds to the class properties.

3.1 Simple root mapping (1:1 fields)

{
  "entityMappingId": "Customer",
  "entityId": "Customer",
  "priority": 0,
  "modelId": "11111111-1111-1111-1111-111111111111",
  "transformationType": "Default",
  "autoMapFields": false,
  "mappings": [
    {
      "sourceFieldName": "CustomerId",
      "targetFieldName": "Code"
    },
    {
      "sourceFieldName": "Name",
      "targetFieldName": "Name"
    },
    {
      "sourceFieldName": "Email",
      "targetFieldName": "Email"
    }
  ]
}

Explanation:

• Each row with entityMappingId = "Customer" is mapped to the customer model.
• autoMapFields = false → all mappings are explicitly defined.

3.2 Mapping with AutoMapFields + override

{
  "entityMappingId": "Product",
  "entityId": "Product",
  "priority": 0,
  "modelId": "22222222-2222-2222-2222-222222222222",
  "transformationType": "Default",
  "autoMapFields": true,
  "mappings": [
    {
      "sourceFieldName": "Id",
      "targetFieldName": "Code"
    },
    {
      "sourceFieldName": "PriceGross",
      "targetFieldName": "Price"
    }
  ]
}

Explanation:

• autoMapFields = true → if the input and target model fields have the same name/type, the mapping is automatically added.
• Explicit mappings override automatic mapping (e.g., conversion of Id → Code).

3.3 Nested entity (items collection)

Example of an order where the target model contains a collection of items Items as a nested entity.

{
  "entityMappingId": "Order",
  "entityId": "Order",
  "priority": 0,
  "modelId": "33333333-3333-3333-3333-333333333333",
  "transformationType": "Default",
  "autoMapFields": false,
  "mappings": [
    {
      "sourceFieldName": "OrderNumber",
      "targetFieldName": "Number"
    },
    {
      "sourceFieldName": "CustomerCode",
      "targetFieldName": "CustomerCode"
    },
    {
      "sourceFieldName": "Items[].Sku",
      "targetFieldName": "Items.Sku"
    },
    {
      "sourceFieldName": "Items[].Quantity",
      "targetFieldName": "Items.Quantity"
    },
    {
      "sourceFieldName": "Items[].Price",
      "targetFieldName": "Items.Price"
    }
  ],
  "options": [
    {
      "key": "ItemsIndexKey",
      "value": "LineId"
    }
  ]
}

Explanation:

• The input contains a collection Items[] (e.g., IncomingNestedObject).
• The target model has Items as NestedEntityFieldType.
• Each element of the collection is mapped to one nested entity in the Items array.
• options.ItemsIndexKey can control the field used to identify the items (index key).

3.4 Mapping with Predicate and Expression field

{
  "entityMappingId": "Invoice",
  "entityId": "Invoice",
  "priority": 10,
  "modelId": "44444444-4444-4444-4444-444444444444",
  "transformationType": "Default",
  "autoMapFields": false,
  "predicate": {
    "value": "Type == \"INVOICE\" && Status != \"CANCELLED\""
  },
  "mappings": [
    {
      "sourceFieldName": "InvoiceNumber",
      "targetFieldName": "Number"
    },
    {
      "sourceFieldName": "CustomerCode",
      "targetFieldName": "CustomerCode"
    },
    {
      "sourceFieldName": "TotalAmount",
      "targetFieldName": "TotalAmount"
    }
  ],
  "expressions": [
    {
      "targetFieldName": "TotalWithVat",
      "expression": "TotalAmount * 1.21"
    }
  ]
}

Explanation:

• predicate.value ensures that the mapping is applied only to records of type INVOICE that are not CANCELLED.
• expressions defines a derived field TotalWithVat, computed from TotalAmount.

4. Typical scenarios and procedures

4.1 How to add a new model mapping

1. In the DataSource configuration (e.g., in Mongo or migration script) add a new DataSourceModelMapping record:
   • set EntityMappingId / EntityId,
   • set ModelId of the target model,
   • set Priority (if there are multiple mappings for the same entityMappingId),
   • set TransformationType (usually Default),
   • decide on AutoMapFields vs. pure Mappings.
2. Define Mappings:
   • for scalar fields direct mapping,
   • for nested entities use ParentField.ChildField format and/or index syntax.
3. Optionally add:
   • Predicate, if the mapping applies only to a subset of records,
   • Expressions, if you need derived fields,
   • Options, if you need special behavior (e.g., index key for nested entities).

4.2 How to add a new field to an existing mapping

1. Extend Mappings with a new DataSourceFieldMapping:
   • sourceFieldName: name of the field in the input data,
   • targetFieldName: name of the existing field in the target model.
2. If the target field is nested:
   • use ParentField.ChildField format,
   • ensure the parent field is defined as NestedEntityFieldType.
3. Test the change using existing tests (e.g., EntityTransformerTests) or create a new integration/unit test that uses EntityTransformer and contexts.

4.3 Troubleshooting mapping issues

• Missing ModelId:
  • TransformationIncomingDataContext.DetectMappingsWithoutModelAsync logs a warning "Empty transformations with ...".
• No available mappings for the given entityMappingId:
  • DetectNoAvailableMappingsAsync logs a warning "No transformation for '...' entity-mapping identifier of 'DataSource.Id' data-source.".
• Field in the target model does not exist:
  • EntityTransformerStrategyBuilder.TransformFieldsAnalysis... logs information/warning for unknown targetTypeFieldName or nested field name.
• Incorrect externalId:
  • TransformationOutputDataContext.CheckRecordSource verifies that externalId belongs to the current DataSource.Id.
• Predicate never triggers:
  • check the expression in predicate.value and the availability of fields in IncomingDataCache (common/custom data).

5. Usage via REST API – DataTransformationsController (Create)

In addition to controlling transformation via mapping at the DataSource level and IncomingDataMappingDirective, you can define data-transformation definitions using the REST API DataTransformationsController. These definitions are separate entities (not the same as DataSourceModelMapping), but in practice, they are often used together – e.g., for preprocessing data before the actual mapping to the model.

Endpoint to create a new definition:

• URL: POST /api/v1/DataTransformations?accessLevel={level}
• Body: DataTransformationDefinitionCreate

5.1 Example – simple transformation definition

POST /api/v1/DataTransformations?accessLevel=Auto HTTP/1.1
Content-Type: application/json

{
  "code": "NormalizeCustomerName",
  "name": "Normalize customer name",
  "description": "Trims and normalizes customer name before mapping.",
  "sourceId": "00000000-0000-0000-0000-000000000123",
  "agentId": null,
  "enabled": true,
  "steps": [
    {
      "order": 0,
      "type": "Trim",
      "fieldName": "Name"
    },
    {
      "order": 1,
      "type": "ToUpper",
      "fieldName": "Name"
    }
  ]
}

Explanation:

• A transformation NormalizeCustomerName is created, which can then be linked to a specific data-source/scenario.
• By combining these transformations with DataSourceModelMapping, you can control both data preprocessing and their final mapping to the unified model.

5.1 Basic v1 call – EnqueueDataBySourceId/start

Endpoint (v1):

• URL: POST /api/v1/SourcingData/EnqueueDataBySourceId/{sourceId}/start
• Body: EdgeDataCollection<EdgeDataRecord> – collection of records with Items (IncomingDataObject).

Simple example without adhoc directives (relies on static DataSource.Mappings):

POST /api/v1/SourcingData/EnqueueDataBySourceId/00000000-0000-0000-0000-000000000123/start HTTP/1.1
Content-Type: application/json

{
  "items": [
    {
      "entityId": "Customer",
      "fields": {
        "CustomerId": "C-001",
        "Name": "Customer A",
        "Email": "customer.a@example.com"
      }
    },
    {
      "entityId": "Customer",
      "fields": {
        "CustomerId": "C-002",
        "Name": "Customer B",
        "Email": "customer.b@example.com"
      }
    }
  ]
}

Explanation:

• sourceId in the URL identifies the DataSource where DataSourceModelMapping are configured.
• Each IncomingDataObject (items[]) has an entityId, by which TransformationIncomingDataContext looks up suitable mappings.

5.2 Example – transformation specific to data-source and agent

POST /api/v1/DataTransformations?accessLevel=Tenant HTTP/1.1
Content-Type: application/json

{
  "code": "ComputeTotalWithDiscount",
  "name": "Compute total amount with discount",
  "description": "Applies discount before mapping to unified invoice model.",
  "sourceId": "11111111-1111-1111-1111-111111111111",
  "agentId": "AGENT-CRM-01",
  "enabled": true,
  "steps": [
    {
      "order": 0,
      "type": "Expression",
      "fieldName": "TotalWithDiscount",
      "expression": "TotalAmount - DiscountAmount"
    }
  ]
}

Explanation:

• The transformation is restricted to a specific sourceId and agentId.
• The resulting field TotalWithDiscount can then be mapped in DataSourceModelMapping.Expressions or directly in field mappings.

Note: The specific schema of DataTransformationDefinitionCreate may vary slightly depending on the implementation of contracts in the project (ASOL.DataService.Contracts). The above examples are to illustrate how to call the REST API, not as an exact reference.

5.2 Call with MappingDirectives at the batch level

You can add mappingDirectives (IncomingDataRecord.MappingDirectives) to the request body, which control the mapping for the entire batch:

POST /api/v1/SourcingData/EnqueueDataBySourceId/00000000-0000-0000-0000-000000000123/start HTTP/1.1
Content-Type: application/json

{
  "operationId": "b5c7b8b4-3f2a-4e3f-9e2f-111111111111",
  "mappingDirectives": [
    {
      "modelId": "11111111-1111-1111-1111-111111111111",
      "priority": 0,
      "transformationType": "Default",
      "replaceCommonFields": false,
      "replaceCustomFields": false,
      "mandantCodeIsRequired": false
    }
  ],
  "items": [
    {
      "entityId": "Customer",
      "fields": {
        "CustomerId": "C-001",
        "Name": "Customer A"
      }
    },
    {
      "entityId": "Customer",
      "fields": {
        "CustomerId": "C-002",
        "Name": "Customer B"
      }
    }
  ]
}

Explanation:

• All items in the batch (items) will be mapped to the same model (modelId from the directive).
• AutoMapFields will be enabled on generated DataSourceModelMapping, so you don’t need to explicitly define Mappings if field names match.

6. Usage via REST API – EnqueueDataBySourceId

This chapter shows how to send data using REST API endpoints in SourcingDataController and simultaneously control mapping via IncomingDataMappingDirective and/or preconfigured DataSourceModelMapping.

7.3.2 Directive at IncomingDataObject level (overriding record-level mapping)

POST /api/v1/SourcingData/EnqueueDataBySourceId/00000000-0000-0000-0000-000000000123/start HTTP/1.1
Content-Type: application/json

{
  "operationId": "c6d8e9f0-4a5b-6c7d-8e9f-222222222222",
  "mappingDirectives": [
    {
      "modelId": "11111111-1111-1111-1111-111111111111",
      "priority": 0,
      "transformationType": "Default"
    }
  ],
  "items": [
    {
      "entityId": "Customer",
      "fields": {
        "CustomerId": "C-001",
        "Name": "Standard Customer"
      }
    },
    {
      "entityId": "Customer",
      "mappingDirectives": [
        {
          "modelId": "55555555-5555-5555-5555-555555555555",
          "priority": 10,
          "transformationType": "Default",
          "replaceCommonFields": true,
          "replaceCustomFields": true,
          "mandantCodeIsRequired": true
        }
      ],
      "fields": {
        "CustomerId": "C-002",
        "Name": "VIP Customer",
        "Type": "VIP"
      }
    }
  ]
}

Explanation:

• The first item uses the directive from IncomingDataRecord → mapped to the basic customer model (modelId from the record).
• The second item has its own mappingDirectives:
  • mapped to a different model (modelId),
  • replaceCommonFields and replaceCustomFields set “replace” strategy (i.e., not patch-update but replace fields),
  • mandantCodeIsRequired = true ensures that EntityOwnership will require MandantCode.

7.3.3 Combining with existing DataSource.Mappings

If the DataSource has predefined Mappings (static DataSourceModelMapping), directives work as adhoc extension/override:

• for the given ModelId, a temporary mapping is created from the directive,
• then merged with the original set via UnionBy(x => x.ModelId),
• in case of ModelId match, the configuration created from the directive takes precedence.

This allows temporary changes in transformation behavior without modifying static configurations (e.g., for one-off migration or special batch).

---

8. Related classes and files

For a deeper understanding of how mapping is used at runtime, we recommend reviewing:

• ASOL.DataService.Domain.Model.DataSourceModelMapping – definition of the model mapping entity.
• ASOL.DataService.Domain.Model.IncomingDataMappingDirective – adhoc mapping directives at input level.
• ASOL.DataService.Domain.Model.IncomingDataRecord – batch of incoming data (MappingDirectives, Items).
• ASOL.DataService.Services.Transformations.EntityTransformer – main orchestrator of transformations (delete/upsert, nested, lookup entity).
• ASOL.DataService.Services.Transformations.EntityTransformerStrategyBuilder – analysis and evaluation of field mappings.
• ASOL.DataService.Services.Contexts.TransformationIncomingDataContext – selection of suitable mappings (ResolveInputMappingsAsync, GenerateSystemMapping, ResolveFieldsStrategy).
• ASOL.DataService.Services.Contexts.TransformationOutputDataContext – storing and managing target records (UpsertDataObjectAsync, DeleteDataObjectAsync).

This document should serve as a starting point for designing and maintaining mappings between external data sources and internal data models of the Data Service.