You are viewing documentation for Data Trust v7.6 (latest).
Looking for the previous version? View v2024.06 documentation →

Bulk Data Validation (BDV)

Apply multiple data quality checks to one or more tables in a single scenario.

Overview

Bulk Data Validation (BDV) applies a configurable set of data quality checks — Null Check, Duplicate Check, Range Check, Regex, List Value, Row Count Check, and more — to one or more tables simultaneously. BDV scenarios are ideal for profiling large datasets against a defined set of quality rules.

Note: Ensure all required connection profiles are configured in the Administration module before building this scenario.

Prerequisites

You are logged in to Data Trust with a Pro-User or Admin Pro-User role.
At least one database connection has been configured and the target tables are accessible.

Steps

Path: Data Trust › Scenario Studio › Scenario Builder › Bulk Data Validation (BDV)

Bulk Data Validation Option

Figure 1: Bulk Data Validation Option

Navigate to the Scenario Studio module and select the Bulk Data Validation (BDV) option from the left sidebar or main menu. This opens the scenario builder canvas where you will configure the data source connections, transformation rules, and execution settings.

Select Connection

Figure 2: Select Connection

Select the appropriate connection from the dropdown to link this widget to its data source. Choosing the correct connection profile ensures the scenario reads from or writes to the intended database or file system.

BDV Create Page

Figure 3: BDV Create Page

The Bulk Data Validation (BDV) create page is the starting point for building a new scenario. Provide a name for the scenario, choose the appropriate folder, and then proceed to configure data sources and execution logic on the canvas.

Select Database and Schema

Figure 4: Select Database and Schema

The Navigator panel displays the available databases, schemas, and tables for the selected connection. Expand the tree to locate the required table or view, then click Select to add it to the widget configuration.

Enter Multiple Inpits Popup

Figure 5: Enter Multiple Inpits Popup

The Enter Multiple Inpits Popup dialog exposes all available configuration options for this step. Review each field carefully and fill in the required values before clicking Save or OK to apply the configuration.

Click on Get Metadata

Figure 6: Click on Get Metadata

Click Get Metadata to automatically retrieve the column structure and data types from the connected data source. Review the retrieved metadata and deselect any columns that are not relevant before proceeding to configure validation rules.

Select Required tables and click on Configure Manually Button

Figure 7: Select Required tables and click on Configure Manually Button

click on add objects button

Figure 8: click on add objects button

This step involves clicking the click on add objects button within the Bulk Data Validation (BDV) workflow. Follow the on-screen prompts to complete this step before proceeding to the next configuration stage.

Apply Rules and Click on Add Objects button

Figure 9: Apply Rules and Click on Add Objects button

This step involves interacting with the Apply Rules and Click on Add Objects button within the Bulk Data Validation (BDV) workflow. Follow the on-screen prompts to complete this step before proceeding to the next configuration stage.

BDV Create Page

Figure 10: BDV Create Page

Select new table along with old table

Figure 11: Select new table along with old table

This step involves selecting the Select new table along with old table within the Bulk Data Validation (BDV) workflow. Follow the on-screen prompts to complete this step before proceeding to the next configuration stage.

Skip and Replace popup

Figure 12: Skip and Replace popup

The Skip and Replace popup dialog exposes all available configuration options for this step. Review each field carefully and fill in the required values before clicking Save or OK to apply the configuration.

New Tables are added

Figure 13: New Tables are added

This step involves interacting with the New Tables are added within the Bulk Data Validation (BDV) workflow. Follow the on-screen prompts to complete this step before proceeding to the next configuration stage.

Create BDV page

Figure 14: Create BDV page

This step involves interacting with the Create BDV page within the Bulk Data Validation (BDV) workflow. Follow the on-screen prompts to complete this step before proceeding to the next configuration stage.

Select Tables

Figure 15: Select Tables

Replace option

Figure 16: Replace option

Query Selection Option

Figure 17: Query Selection Option

Select Queries and click on Add to Metadata

Figure 18: Select Queries and click on Add to Metadata

Click on Configure Manually Button

Figure 19: Click on Configure Manually Button

Both Query and Object Selection

Figure 20: Both Query and Object Selection

This step involves interacting with the Both Query and Object Selection within the Bulk Data Validation (BDV) workflow. Follow the on-screen prompts to complete this step before proceeding to the next configuration stage.

Select required columns to add

Figure 21: Select required columns to add

This step involves selecting the Select required columns to add within the Bulk Data Validation (BDV) workflow. Follow the on-screen prompts to complete this step before proceeding to the next configuration stage.

Added Selected Columns

Figure 22: Added Selected Columns

This step involves interacting with the Added Selected Columns within the Bulk Data Validation (BDV) workflow. Follow the on-screen prompts to complete this step before proceeding to the next configuration stage.

Null Check selection and Threshold

Figure 23: Null Check selection and Threshold

Configure the Null Check selection and Threshold parameters including the target column(s), threshold values, and any custom expressions. After saving, the rule is added to the scenario and will be evaluated against every record when the scenario executes.

Null Check

Figure 24: Null Check

Configure the Null Check parameters including the target column(s), threshold values, and any custom expressions. After saving, the rule is added to the scenario and will be evaluated against every record when the scenario executes.

Duplicate check and Combination toggle button

Figure 25: Duplicate check and Combination toggle button

Configure the Duplicate check and Combination toggle button parameters including the target column(s), threshold values, and any custom expressions. After saving, the rule is added to the scenario and will be evaluated against every record when the scenario executes.

Zero Value Check

Figure 26: Zero Value Check

This step involves interacting with the Zero Value Check within the Bulk Data Validation (BDV) workflow. Follow the on-screen prompts to complete this step before proceeding to the next configuration stage.

Numeric type Check

Figure 27: Numeric type Check

This step involves interacting with the Numeric type Check within the Bulk Data Validation (BDV) workflow. Follow the on-screen prompts to complete this step before proceeding to the next configuration stage.

Regex Check

Figure 28: Regex Check

Configure the Regex Check parameters including the target column(s), threshold values, and any custom expressions. After saving, the rule is added to the scenario and will be evaluated against every record when the scenario executes.

List Value Popup

Figure 29: List Value Popup

The List Value Popup dialog exposes all available configuration options for this step. Review each field carefully and fill in the required values before clicking Save or OK to apply the configuration.

Enter Multiple Inputs popup

Figure 30: Enter Multiple Inputs popup

The Enter Multiple Inputs popup dialog exposes all available configuration options for this step. Review each field carefully and fill in the required values before clicking Save or OK to apply the configuration.

List Value check

Figure 31: List Value check

This step involves interacting with the List Value check within the Bulk Data Validation (BDV) workflow. Follow the on-screen prompts to complete this step before proceeding to the next configuration stage.

Min Value Check

Figure 32: Min Value Check

This step involves interacting with the Min Value Check within the Bulk Data Validation (BDV) workflow. Follow the on-screen prompts to complete this step before proceeding to the next configuration stage.

Max Value Check

Figure 33: Max Value Check

This step involves interacting with the Max Value Check within the Bulk Data Validation (BDV) workflow. Follow the on-screen prompts to complete this step before proceeding to the next configuration stage.

Condition Check popup

Figure 34: Condition Check popup

The Condition Check popup dialog exposes all available configuration options for this step. Review each field carefully and fill in the required values before clicking Save or OK to apply the configuration.

Condition Check is populated

Figure 35: Condition Check is populated

Configure the Condition Check is populated parameters including the target column(s), threshold values, and any custom expressions. After saving, the rule is added to the scenario and will be evaluated against every record when the scenario executes.

Row Count Check

Figure 36: Row Count Check

This step involves interacting with the Row Count Check within the Bulk Data Validation (BDV) workflow. Follow the on-screen prompts to complete this step before proceeding to the next configuration stage.

All checks Applied

Figure 37: All checks Applied

This step involves interacting with the All checks Applied within the Bulk Data Validation (BDV) workflow. Follow the on-screen prompts to complete this step before proceeding to the next configuration stage.

Save BDV

Figure 38: Save BDV

Click the Save button to persist the current widget or scenario configuration. A confirmation indicator appears on the canvas once the save is successful, and the widget thumbnail updates to reflect the configured state.

Execution Summary Page

Figure 39: Execution Summary Page

The Execution Summary Page page presents the full outcome of the scenario execution including record counts, match rates, and any detected exceptions. Drill into exception records to inspect individual mismatches and use the Export to CSV option to share results with stakeholders.

Field Rule Info Button

Figure 40: Field Rule Info Button

This step involves interacting with the Field Rule Info Button within the Bulk Data Validation (BDV) workflow. Follow the on-screen prompts to complete this step before proceeding to the next configuration stage.

Detailed View

Figure 41: Detailed View

This step involves interacting with the Detailed View within the Bulk Data Validation (BDV) workflow. Follow the on-screen prompts to complete this step before proceeding to the next configuration stage.

Exception Result

Figure 42: Exception Result

The Exception Result page presents the full outcome of the scenario execution including record counts, match rates, and any detected exceptions. Drill into exception records to inspect individual mismatches and use the Export to CSV option to share results with stakeholders.