You are viewing documentation for Data Trust v7.6 (latest).
Looking for the previous version? View v2024.06 documentation →

Data Trust › Scenario Studio › Scenario Builder

Validation with Duplicates Check Business Rule

Identify duplicate records in a dataset using the Duplicates Check business rule.

Overview

The Duplicates Check business rule identifies records that appear more than once in the dataset based on a combination of specified key columns. This guide walks through configuring a Duplicates Check validation scenario from source selection to execution.

Note: Ensure all required connection profiles are configured in the Administration module before building this scenario.

Prerequisites

• You are logged in to Data Trust with a Pro-User or Admin Pro-User role.
• An RDBMS connection has been configured and the target table is accessible.

Steps

Path: Data Trust › Scenario Studio › Scenario Builder › Validation with Duplicates Check Business Rule

Figure 1: Validation Option

Navigate to the Scenario Studio module and select the Validation with Duplicates Check Business Rule option from the left sidebar or main menu. This opens the scenario builder canvas where you will configure the data source connections, transformation rules, and execution settings.

Figure 2: Drag and drop RDBMS Widget

Drag the Rdbms Widget from the widget palette and drop it onto the scenario canvas. Positioning the widget opens its configuration popup where you will set the connection, table, and any additional options.

Figure 3: Select Connection and click on Navigator

Select the appropriate connection from the dropdown to link this widget to its data source. Choosing the correct connection profile ensures the scenario reads from or writes to the intended database or file system.

Figure 4: Select table

The Navigator panel displays the available databases, schemas, and tables for the selected connection. Expand the tree to locate the required table or view, then click Select to add it to the widget configuration.

Figure 5: Click on Trasformations Tab

This step involves clicking the Click on Trasformations Tab within the Validation with Duplicates Check Business Rule workflow. Follow the on-screen prompts to complete this step before proceeding to the next configuration stage.

Figure 6: Drag and drop Filter widget

Drag the Filter Widget from the widget palette and drop it onto the scenario canvas. Positioning the widget opens its configuration popup where you will set the connection, table, and any additional options.

Figure 7: Filter Expression

The Transformations tab lists all available data transformation widgets including Filter, Derive, Aggregate, and Lookup. Add transformation widgets between data source and reconciliation or validation widgets to cleanse or reshape data before comparison.

Figure 8: Click on Validation Rules Tab

The Validation Rules tab lists all available business rule widgets that can be applied to the selected dataset. Drag the required rule widget onto the canvas, configure its threshold or pattern, then save before moving to the next step.

Figure 9: Drag and drop Duplicate check widget

Drag the Duplicate Check Widget from the widget palette and drop it onto the scenario canvas. Positioning the widget opens its configuration popup where you will set the connection, table, and any additional options.

Figure 10: Rule Expression

This step involves interacting with the Rule Expression within the Validation with Duplicates Check Business Rule workflow. Follow the on-screen prompts to complete this step before proceeding to the next configuration stage.

Figure 11: Click on Additional Options Tab

Navigate to the Scenario Studio module and select the Validation with Duplicates Check Business Rule option from the left sidebar or main menu. This opens the scenario builder canvas where you will configure the data source connections, transformation rules, and execution settings.

Figure 12: Drag and drop Email Widget

Drag the Email Widget from the widget palette and drop it onto the scenario canvas. Positioning the widget opens its configuration popup where you will set the connection, table, and any additional options.

Figure 13: Configure Email

The Email widget sends an automated result notification to configured recipients after the scenario finishes executing. Provide the recipient addresses, subject, and choose whether to attach the exception report or include a summary in the email body.

Figure 14: Click on Execute Button

Click the Execute button to trigger the scenario and begin processing data from the configured sources. The scenario transitions to Monitor mode where you can track real-time execution progress and view interim results as each step completes.

Figure 15: Validation Execution Summary

The Validation Execution Summary page presents the full outcome of the scenario execution including record counts, match rates, and any detected exceptions. Drill into exception records to inspect individual mismatches and use the Export to CSV option to share results with stakeholders.

Figure 16: 15Click on Exception Count

The 15Click on Exception Count page presents the full outcome of the scenario execution including record counts, match rates, and any detected exceptions. Drill into exception records to inspect individual mismatches and use the Export to CSV option to share results with stakeholders.