Capability to view the source/target duplicate records from FDR execution summary screen

You are viewing archived documentation for Data Trust v2024.06 (previous version).
Go to the latest v7.6 documentation →

Functional Data Reconciliation

Functional Data Reconciliation is a part of the SCENARIO STUDIO module. Using FDR, users can compare a pair of data sets (source and target) to the field by field level and this option is the most comprehensive way to reconcile data between all right data supported data sources.

To learn about FDR, let us walk through the below procedure:

• Click the ‘SCENARIO STUDIO’ module.
• Hover on the ‘Scenario Builder’ snippet from the left pane options.
• Click the ‘Functional Data Recon’ option.

Path 01: QUERY STUDIO > Scenario Builder > Functional Data Recon

Path 02: QUERY STUDIO > Create FDR Scenario

• The user is navigated to the FDR Session page in create and edit mode.
• In the ‘Data Source’ tab, the user can view different Data Source options such as Queries, Query Chain, Reconciliation Output, Custom SQL, RDBMS Table, Tableau, Sales Force, CSV, REST API, SAP Table, SAP Infoprovider, Gem Fire, PowerBI, SSRS, Qlik, and SOAP API which are supported by the RightData.
• Let’s, click the header widget (FDR Sample Scenario).

• An ‘FDR Sample Scenario’ pop-up window is displayed.
• By default, the header widget name is included under the ‘Scenario Name’ and ‘Description’ text fields.

Note: The user can modify these details if required.

• By default, ‘Retain data only for the last 2 Days’ is displayed for the ‘Data Retention’.

Note: The user can also change the ‘Data Retention’ settings if required.

• Click the ‘Browse Folders’ button option.

• A ‘Choose Folder’ pop-up window is displayed.
• The user can choose the specific folder from the ‘Root’ path here.
• Let’s, click the ‘Add New Folder’ button.

• An ‘Add Folder’ pop-up window is displayed.
• Let’s add ‘RDBMS Folder’ under the ‘Name’ text field.
• Click the ‘Add’ button.

• This action adds the ‘RDBMS Folder’ folder under the ‘Root’ path.
• Click the ‘X’ button from the right corner of the page.

• This action navigates the user to the header (FDR Sample Scenario) pop-up window.
• ‘Delete original data after reconciliation?’ is checked in by default.

Note:

1. If the user checks this option, the Source and Target data are not shown after the reconciliation.
2. If the user unchecks this option, the Source and Target data are shown after the reconciliation.

• Close this window and navigate back to the FDR session page.

• From the ‘Data Source’ tab, let’s drag and drop an RDBMS table widget.
• Click the RDBMS Table widget.

• A ‘Read data from Table’ pop-up window is displayed.
• A ‘Database Type’ and ‘Connection Name’ drop-down lists are displayed.
• The user can ‘Pick Table’ through a ‘Navigator’ button option.

Note: By default, the ‘Pick Table’ field option is disabled but when the user selects the ‘Database Type’ and ‘Connection Name’ from the drop-down list, the ‘Pick Table’ field option is enabled.

• Let’s, select the Database Type as ‘PostgreSQL’ from the drop-down list.
• Select Connection Name as ‘Northwind PostgreSQL’ from the drop-down list.

• A green tick mark appears beside the ‘Northwind PostgreSQL’ indicating the ‘Connection Name’ selection is successful.
• An ‘Additional Settings’ button is displayed beside the green tick mark.
• Click the ‘Additional Settings’ button.

• An ‘Additional Settings’ pop-up window is displayed with different text field options such as Parallelism, Packet Size, and Packetize By.

Note: By default, Parallelism is set to ‘1’ and Packet Size to ‘50000’.

• A ‘Reset to Default’ button is available if the user wants to reset ‘Parallelism’ and ‘Packet Size’ to default values.
• When hovered on the ‘info’ button on the ‘Additional Settings’ pop-up window the user is assisted with information.
• Click the ‘Save’ button to save the ‘Additional Settings’.
• The moment the settings are saved the ‘Additional Settings’ pop-up window is closed and the user is navigated back to the ‘Read data from Table’ window.

• Click the ‘Navigator’ button to pop up a ‘Navigator’ slide screen from the right-hand side.
• A ‘Refresh’ button is used to refresh the ‘Navigator’ session page.
• The ‘Navigator’ displays the Schemas and Tables.

• To add a table, click the plus ‘(+) bankdetails table available under the Tables folder.

Path: Northwind PostgreSQL > Public > Tables> bankdetails

• The ‘Pick Table’ displays the ‘public. bankdetails’ table selected from the tabular grid.
• Click the ‘Preview Data’ button.

• An ‘Existing Query’ pop-up window is displayed.
• Close the ‘window and navigate back to the ‘Read data from Table’ window.

• Click the ‘Select’ button.

• The user is navigated to the ‘FDR’ session page saving and closing the ‘RDBMS Table’ widget.

• From the Data Source tab, let’s drag and drop another RDBMS Table widget onto FDR create and edit page.

Note: All the Data Source icons including ‘RDBMS Table’ get disabled as the user drags and drops another comparison widget on to FDR create and edit page.

• Click the RDBMS Table widget.

• A ‘Read data from Table’ pop-up window is displayed.
• A ‘Database Type’ and ‘Connection Name’ drop-down lists are displayed.
• The user can ‘Pick Table’ through a ‘Navigator’ button option.

Note: By default, the ‘Pick Table’ field option is disabled, but when the user selects the ‘Database Type’ and ‘Connection Name’ from the drop-down list, the ‘Pick Table’ field option is enabled.

• Let’s, select Database Type as ‘PostgreSQL’ from the drop-down list.
• Select Connection Name as ‘Northwind PostgreSQL’ from the drop-down list.
• Click the Navigator button.

• A green tick mark appears beside the ‘Northwind PostgreSQL’ indicating the ‘Connection Name’ is successful.
• An ‘Additional Settings’ button is displayed beside the green tick mark.
• Click the ‘Additional Settings’ button.

• An ‘Additional Settings’ pop-up window is displayed with different text field options such as Parallelism, Packet Size, and Packetize By.

Note: By default, Parallelism is set to ‘1’ and Packet Size to ‘50000’.

• A ‘Reset to Default’ button is available if the user wants to reset ‘Parallelism’ and ‘Packet Size’ to default values.
• When hovered on the ‘info’ button on the ‘Additional Settings’ the user is assisted with information.
• Click the ‘Save’ button to save the ‘Additional Settings’.
• The moment the settings are saved the ‘Additional Settings’ pop-up window is closed and the user is navigated back to the ‘Read data from Table’ window.

• Click the ‘Navigator’ button to pop up a ‘Navigator’ slide screen from the right-hand side.
• A ‘Refresh’ button is used to refresh the ‘Navigator’ session page.
• The ‘Navigator’ displays the Schemas and Tables.

• To add a table, click the plus ‘(+) bankdetails table under the ‘Tables’ folder.

Path: Northwind PostgreSQL > Public > Tables> bankdetails

• The ‘Pick Table’ displays the ‘public. bankdetails’ table selected from the tabular grid.
• Click the ‘Preview Data’ button.

• An ‘Existing Query’ pop-up window is displayed.
• Close the ‘window and navigate back to the ‘Read data from Table’ window.

• Click the ‘Select’ button.

• The user is navigated to the ‘FDR’ session page saving and closing the ‘RDBMS Table’ widget.
• Click the ‘Transformations’ tab.

• Projection, Aggregation, and Filters icons are displayed.

• From the Transformations tab, let’s drag and drop the ‘Filters’ icon on to FDR session page.
• Click the ‘Filters’ widget.

• A ‘Filter data from Data Source’ pop-up window is displayed.
• Within the Filter session, Available Fields, Available Operators, input, and add variables tabs are displayed.
• Below the ‘Filter’ session, an ‘Expression’ session with an ‘Edit Expression’ button preference is displayed.

• Let’s, drag the Available Fields, Available Operators, and input.
• Click the ‘Select’ drop-down list from the ‘Available Fields’ filter.

• The different columns from the ‘RDBMS Table’ widget data are displayed.
• Let’s select the ‘paymentsource’ option from the Select drop-down list under the Available Fields tab.
• Then, click the ‘Select’ drop-down list under the ‘Available Operators’ tab.

• The functions such as Equals to, Not equals to, Greater than, Greater than or equals to, Less than, Less than or equals to, Like, Between, Not between, AND, OR, Is null, Is null not, and IN are displayed.
• Let’s, select the ‘Less than or equals to’ option from the Select drop-down list under the ‘Available Operators’ tab.

• Let’s add a value of ‘100000’ in the name field under the ‘input’ tab.
• The user can notice the ‘Expression’ field is updated according to the Filter’ condition.
• Click the ‘Save’ button.

• The ‘Filter data from Data Source’ pop-up window is closed saving the ‘Filters’ widget condition.
• The user is navigated to the FDR session page in create and edit mode.
• Click the ‘Reconciliations’ tab.

• ‘Map source & target fields’ icon is displayed.

• Let’s drag and drop the ‘Map source & target fields’ icon onto the FDR session create and edit page.

• Join the first ‘RDBMS Table’ widget and the ‘Filters’ widget to the ‘Map source & target fields’ widget.
• Click the ‘Map source & target fields’ widget.

• A ‘Map source & target fields’ pop-up window is displayed.

• The user can view different buttons such as Sort Options, Smart Mappings, Delete all mappings, Delete selected mappings, Upload mappings, Fuzzy mapping, Historical mapping, Search, Additional options, and Reporting characteristics.
• Click the ‘Sort Options’ button.

• A list of check box options like Default Order, Sort by Field Name, Sort by Description, Show Mapped Fields Only, Show Keys + All Measures, Show Keys + All Attributes, Show Keys + First ( ) Fields, Show Keys + Last ( ) Fields.

Note: By default, ‘Default Order’ is checked.

• Click the ‘Smart Mapping’ button to enable smart mapping.

• Click the ‘Delete all mappings’ button to delete all the selected mappings.

• Click the ‘Delete selected mappings’ button to delete the selected mappings.

• Click the ‘Upload Mappings’ button.

• An ‘Upload Mappings’ pop-up window is displayed.
• A ‘Download mapping template’ button is used to download a sample template, so that the users can add the Source and Target data and upload the same using an ‘Import from file’ button.

• Let’s suppose import a sample downloaded template and upload it through an ‘Import from file’ button.
• Click the ‘Import from file’ button.
• Click the ‘Open’ button.

• The sample data is loaded as shown here.
• Click the ‘Verify’ button.

• The ‘Status’ column is updated showing the ‘Valid’ or ‘Invalid’ mapping status.
• As the user clicks the ‘Map’ button, the ‘Upload Mappings’ widget is closed saving the data.

Note: It is shown as a sample to upload local mapping data here, but let’s close this window and navigate back to the ‘Map source & target fields’ pop-up window.

• Click the ‘Fuzzy mapping’ button for a Fuzzy Matcher algorithm linkage.

Note: Click here to view the Fuzzy mapping algorithm.

• Click the ‘Historical mapping’ button to check the mapping history.

• Click the ‘Search’ button.

• A ‘Search’ box text field is displayed to search for any columns.

• An individual ‘Search’ icon button is also present for the ‘Source Dataset Fields’ and ‘Target Datasets Fields.’
• Click the ‘Additional Options’ button.

• An ‘Additional Options’ pop-up window is displayed with different checkbox options such as Allow duplicates in the comparison, Save Matching data, Automatically map new columns by name.
• Close this window and navigate back to the ‘Map source & target fields’ session page.

• Click the ‘Report Characteristics’ button.

• A ‘Reporting Characteristics’ pop-up window is displayed with different reporting characteristics drop-down lists such as Quality Dimensions, Risk Categories, Data Domains, Team, PROJECT, Layer, and GROUP.
• Click the ‘Select’ button to save the ‘Reporting Characteristics.’
• Close this window.

• Click the ‘Save’ button to save changes.

• A ‘Confirm action’ pop-up window is displayed.
• Click the ‘Verified’ button.

• The ‘Map source & target fields’ widget is closed saving the widget data.
• The user is navigated to the FDR session page in create and edit mode.
• Click the ‘Additional Options’ tab.

• Various icons such as Email, HPALM, IMDB, Jira, XRAY, Azure Dev Ops Board, Q-Test Execution, and Q-Test Defect are displayed.

• From the ‘Additional Options’ tab, let’s drag and drop an ‘Email’ widget onto the FDR session in create and edit page.

Note: As the user drags the ‘Email’ icon from the ‘Additional Settings’ tab, it gets disabled.

• Click the ‘Email Settings’ widget.

• An ‘Email Settings’ pop-up window is displayed with different checkbox options listed below.

1. Email when Found Exception
2. Email when no exceptions are found
3. Enable External Scenario Exceptions Link
4. Include download link to the exceptions data file
5. Attach execution summary result PDF file

• Along with these checkbox options, other text field options such as Email Subject with Show Variables button, Email From, Email Priority, and Email Template are displayed.

• Let’s select ‘Email when found Exception’ checkbox and add an email address.
• Click the Save button.

Note: Email is greyed due to confidential data.

• The ‘Email Settings’ widget is closed saving the data.
• The user is navigated to the FDR session page in create and edit mode.
• Click the ‘Execute Now’ button.

• The FDR execution is ‘Failed’ here showing the Exceptions data.
• A Monitor mode is enabled immediately after the execution process.
• Click the ‘Map source & target fields’ count icon.

• The user is navigated to the ‘Functional Data Reconciliation Exceptions’ page.

• The user receives exceptions found email as shown here.