Skip to main content

Technical Data Recon

You are viewing archived documentation for Data Trust v2024.06 (previous version).
Go to the latest v7.6 documentation →

Technical Data Recon

Technical Data Reconciliation (TDR) is a part of the SCENARIO STUDIO module. Using TDR, users can perform a row count of data comparison between multiple pairs of data sets.

To learn about the TDR, let us walk through the below procedure:

  • Click the ‘SCENARIO STUDIO’ module.
  • Hover on the ‘Scenario Builder’ snippet from the left pane options.
  • Click the ‘Technical Data Recon’ option.

Note: Either clicking the Technical Data Recon option or Create TDR Scenario button navigates to a new TDR create and edit page

Path 01: SCENARIO STUDIO > Scenario Builder > Technical Data Recon

Path 02: SCENARIO STUDIO > Create TDR Scenario

  • The user is navigated to a new TDR session page in create and edit mode.
  • Under the Data Source tab, the user can view different Data Source options that are supported by The RightData product such as RDBMS Tables, SalesForce, SAP BW Info, SAP Tables, SAP Bex, SAP Bobj, Azure Data Lake, Amazon S3, and Apache Drill.

  • From the Data Source tab, let’s drag and drop an ‘RDBMS Tables’ icon on to the TDR session page.
  • Click the ‘RDBMS Tables’ widget.

  • An ‘RDBMS Tables’ pop-up window is displayed.
  • Let’s select ‘PostgreSQL’ from the Database Type drop-down list.
  • Select ‘PostgreSQL Northwind_QA’ from the Connection Name drop-down list.

Note: As the user selects the ‘Connection Name’ (PostgreSQL Northwind_QA here) the Parallelism, Packet Size along with Server, Database, Port, and User Id are populated.

  • Click the ‘Additional Settings’ button.

  • An ‘Additional Settings’ pop-up window is displayed with Parallelism, Packet Size, and Packetize by field options.

Note: By default, Parallelism is set to 1 and Packet Size by 50000 records

  • User can modify the Parallelism and Packet Size based on their requirements and click the ‘Save’ button to save the changes.
  • A ‘Reset to default’ button is used to reset the ‘Parallelism’ and ‘Packet Size’ to default values when changed.
  • When hovered on the info button against the text fields the user is assisted with information.
  • Click the ‘X’ button to close the ‘Additional Settings’ pop-up window.

  • The user is navigated to the ‘RDBMS Tables’ session page.
  • Click the ‘Select’ button.

  • The ‘RDBMS Table’ widget is closed saving the widget data and further collapsing into the TDR session page in create and edit mode.

  • From the Data Source tab, drag and drop an ‘RDBMS Tables’ widget as Target Data Source.

Note: The ‘RDBMS Tables’ icon including the remaining icons from the Data Source tab gets disabled as the user drags the ‘Target Data Source’ onto the TDR session page in create and edit mode.

  • Click the ‘RDBMS Tables’ widget.

  • An ‘RDBMS Tables’ pop-up window is displayed.
  • Let’s suppose, select ‘Database Type’ as ‘Redshift’ and ‘Connection Name’ as ‘Redshift-QA.’

Note: As the user selects the ‘Connection Name’ (Redshift-QA here) the Parallelism, Packet Size along with Server, Database, Port, and User Id will be populated

  • Click the ‘Additional Settings’ button.

  • An ‘Additional Settings’ pop-up window is displayed and has Parallelism, Packet Size, and Packetize by text field options.

Note: By default, Parallelism is set to 1 and Packet Size by 50000 records.

  • Users can also modify the Parallelism and Packet Size based on their requirements and click the ‘Save’ button to save the changes.
  • A ‘Reset to default’ button is used to reset the ‘Parallelism’ and ‘Packet Size’ to default values when changed.
  • When hovered on the info button against different text fields the user is assisted with information.
  • Click the ‘X’ button to close the ‘Additional Settings’ pop-up window.

  • The user is navigated to the ‘RDBMS Tables’ session page.
  • Click the ‘Select’ button.

  • The ‘RDBMS Tables’ widget is closed saving the data and collapsing into the TDR session page in create and edit mode.
  • Click the ‘Compare’ tab.

  • Two icons namely ‘Count Compare’ and ‘Data Compare’ are displayed.

Note: ‘Count compare’ compares the Row Count from the Source and Target Data sources whereas ‘Data Compare’ compares the Data from both the Source and Target profiles.

  • Let’s suppose, from the ‘Compare’ tab, drag and drop the ‘Count Compare’ widget onto the TDR session page in create and edit mode.

Note: The ‘Count Compare’ and also the ‘Data Compare’ icon is disabled as the user drags any icon onto the TDR session page in create and edit mode.

  • Now, connect one end of the Source ‘Data Source’ and the other end of the Target ‘Data Source’ to the Count Compare widget.
  • Click the ‘Count Compare’ widget.

  • A Count Compare pop-up window is displayed.
  • The user can view two Connection Name columns namely ‘PostgreSQL Northwind_QA’ and ‘Redshift-QA’ are placed beside each other.
  • In-between these two Connection Name columns ‘PostgreSQL Northwind_QA’ and ‘Redshift-QA’ a couple of options such as Upload Mappings, Parallelism for Pairs, Threshold, Propose, Delete selected mappings, Delete all mappings, Object mapping view, and Object list entry view are displayed.
  1. Upload Mappings:

The user can upload mappings through an Import from file button. Apart from this, he/she can also download a sample mapping template using the ‘Download mapping template button.

  1. Parallelism for Pairs:

The user can add Parallelism for Pairs and by default 2 is added in the Parallelism field box. Click the Ok button to save the modified Parallelism values.

  1. Threshold:

This button is used to add Threshold for comparison. By default, 0 is added in the ‘Threshold for comparison’ field. The user notices two special characters for this comparison namely % for threshold in percentage and # for threshold in value comparisons. Click the Apply button to save the changes.

  1. Propose:

Propose button is used to map the common Table fields of the schemas that appear both in the source field to the target field.

  1. Delete Selected Mappings:

The user can delete already selected mappings fields.

  1. Delete all mappings:

The user can delete all the selected mapping fields.

  1. Object mapping View:

The user can view both the Schema and table associated with that particular schema.

  1. Object list entry view:

The user can only view the object lists without schemas.

  • The user views two similar sets (left and right) with different button options beside each Connection Name (PostgreSQL Northwind_QA and Redshift-QA) such as Advanced Options, Object list entry view, Object mapping view, Delete selected entries, Add new entry, Filter, and Search.
  • Click the ‘Advance Options’ button.

  • An ‘Advanced Options’ pop-up slide screen is displayed with different text fields as shown here.
  1. Exclude Columns
  2. Columns Filter
  3. Convert date format from

Note: A ‘Multiple Selection’ button is available beside ‘Exclude Columns’ field.

  • Click the ‘Multiple Selection’ button.

  • An ‘Enter Multiple Inputs’ pop-up window has ‘Import from file’ button where the user can add different field values using this button.
  • A ‘Submit’ button is used to save the field values uploaded using an ‘Import from file button.’
  • Click the ‘X’ button to close the ‘Enter Multiple Inputs’ pop-up window.

  • The user is navigated to the ‘Count Compare’ session page.
  • Click the ‘X’ button to close the ‘Advanced Options’ slide screen.

  • Now, click the ‘Filter’ button of ‘PostgreSQL Northwind_QA.’

  • A ‘Filter’ pop-up slide screen with Schema / Owner, Object Name, and Max. No. of Values field options are displayed.

Note: By default, Max. No. of Values is set to 500.

  • Let’s, select ‘pg_catalog’ from Schema / Owner drop-down list.
  • Click the ‘Get Metadata’ button.

  • ‘Get Metadata’ button populates the selected Schema tables (pg_catalog).
  • Click the Filter button of Redshift-QA Connection Name.

  • A ‘Filter’ slide screen is displayed with different field options such as Schema / Owner, Object Name, and Max. No. of Values.
  • Let’s again, select a similar Schema ‘pg_catalog’ from the drop-down list.
  • Click the ‘Get Metadata’ button.

  • ‘Get Metadata’ button populates the selected Schema tables (pg_catalog) of Redshift-QA.
  • Now, click the ‘Propose’ button.

  • Mapping similar fields for both the Connection Names (PostgreSQL Northwind_QA and Redshift-QA) are displayed.
  • 52 similar fields have been mapped between the two Connections (PostgreSQL Northwind_QA and Redshift-QA)

Note: There are 102 field values in PostgreSQL Northwind_QA and 277 field values in Redshift-QA

  • Click the ‘Save’ button.

  • The ‘Count Compare’ widget is closed saving the widget data and collapsing into the TDR session page in create and edit mode.

  • From the Additional Options tab, drag and drop the ‘Email widget’ onto TDR create and edit page.
  • Click the ‘Email Settings’ widget.

  • An ‘Email Settings’ pop-up window is displayed with the following check box options such as Email when Found Exception, Email when no exceptions found, Enable External Scenario Exceptions Link, Include download link to the exceptions data file, and Attach execution summary result PDF file.
  • The user also notices different field options like Email Subject, Email From, Email Priority, and Email Template.
  • Select the ‘Email when Found Exception’ checkbox and enter an email address in the first field.
  • Click the ‘Save’ button.

  • The Email Settings widget is closed saving the data and collapsing into the TDR session page in create and edit mode.
  • Click the ‘Execute Now’ button.

  • The user is navigated to a new tab displaying the ‘Scenario Execution Summary’ results of that particular TDR Scenario (Sample TDR Execution).
  • The Row Count is displayed under ‘Compare Counts’ results tab.

Note: This is a Failed Scenario because the failed percentage is high (43%).

  • The TDR execution results are forwarded to the email address as shown here.

Was this article helpful?