Skip to main content

Release Notes - 2021.06

Features:

  1. Query Performance Testing (QPT).

QPT is a new feature introduced in the release 2021.06, using which users test the performance of SQL queries against a baseline expected runtime. The actual query runtime can be compared with the expected runtime and categorize the scenario outcome into – Pass, Information, Warning, or Fail

        • Pass – When actual query execution time is below or equal to the expected runtime
        • Information – When actual query runtime is more than the Pass threshold and less than the Information threshold
        • Warning – When actual query runtime is more than Information threshold and less than the Warning threshold
        • Fail – When actual query runtime is greater than or equal to Fail threshold
  1. Restart middleware and download logs from UI.
      • This feature is only available for users with admin privileges. Prior to the 2021.06 version, restarting the middleware and displaying the middleware logs is only possible from the RightData (DataTrust) application server. Now, the admin user can restart the middleware from UI as well as downloading the logs
  1. Fuzzy join transformation in the query chain.
      • A new transformation type, Fuzzy join has been introduced as part of 2021.06
      • Fuzzy join allows users to join two datasets based on one or more columns using proximity matching instead of exact matching.
      • We support two types of algorithms “Fuzzy Matcher”, “Record Linkage”
  1. Scenario execution history report (pivot style).
      • A new report “Scenario Execution History” allows users to get a quick snapshot of each scenario result, bucketed into daily buckets. Report by default, shows all the scenarios for the past 10 days which can be edited as per the users choice.
  1. Create different report views using filters in Dashboards.
      • Users can save different views of the pre-delivered reports by customizing the report filters, layout, and title.
  1. Access controls for report views.
      • Access to the new views created for the pre-delivered reports can be controlled by mapping them to one or more user groups.
  1. New Enterprise License option.
      • When the enterprise license option is purchased by the customers – they now can create users in RightData (DataTrust) without any user license key
      • At the time of enterprise license key generation, RightData (DataTrust) license key support team will encode the max number of allowed user licenses. With this setting, the customer’s administrator can create up to the max number of allowed users.
  1. Push-down reconciliation.
      • When the source and target of a scenario is using the same connection profile – users now can perform push-down reconciliations.
      • This option can improve the performance of the reconciliation process by 5X – 100X. This option avoids the need for the data to be ingested into RightData (DataTrust) and pushes the reconciliation logic to the source system itself.
      • As of version 2021.06 the connection profiles supported for pushdown reconciliation are – Azure SQL, Synapse, SQL Server, Big Query, Redshift and Snowflake.

Enhancements:

  1. Bulk Data Validation (BDV) support for Snowflake, redshift.
      • Now users can perform pushdown validations for one or more datasets from snowflake and redshift databases. Prior to 2021.06, only Bigquery, SQL Server, Azure SQL, Synapse, and Oracle are supported.
  1. Support of variables in the CMS (Content Management Console).
      • Now users can export and import the queries, query chains, scenarios, and jobs along with associated static and query-based variables.
  1. Redesign "Data Quality Metrics" dashboard.
      • The existing “Data Quality Metrics” Dashboard has been redesigned to give a user-friendly experience with drill-down capabilities.
  1. Ability to select connection profile icon for generic JDBC profile type.
      • Besides the pre-built connectors, RightData (DataTrust) supports any database with a JDBC connection. When a generic JDBC connection is used, prior to 2021.06 – the scenario summary report only shows the generic JDBC icon which doesn’t give a clear indication of the type of database. Starting from version 2021.06, users now can upload the database icon that corresponds to the JDBC connection.
  1. New status-based filters in the Job explorer.
      • Three new options were added to the job explorer. One is to show only “scheduled jobs”, the second is to show “running jobs” and the third is to show “All jobs”.
      • Along with these three options, filtering capability has been added to the job explorer, which allows users to easily filter one or more jobs matching to a specific filter criterion in the explorer.
  1. New status-based filters in the session monitors.
        • Four new options were added to scenario session monitors.
      • Four buttons (Running, Passed, Failed, and Errored) which are acting as filters are enabled in the Scenario studio session monitors to show only running/passed/ failed/errored scenario sessions, based on the user’s choice.
  1. Where-used option in the Query chain, scenario, and job designers and explorers.
      • The where-used option allows the user to identify the lineage of the respective RightData (DataTrust) component. For example, when a query chain is used in the two scenarios and one job; this dependency can be easily tracked by click on the where-used button on the toolbar of the respective design canvas or the explorer.
  1. Scope definition for the folders.
      • Prior to version 2021.06, the folders created in either query studio, scenario studio or Batch Console are automatically shown in other modules. There is no option to define the scope of the folder.
      • Now users can select the scope of the folder at the time of folder creation.
  1. Ability to add a user to one or more user groups from user edit screen/explorer.
      • Now, the RightData (DataTrust) administrator can easily add a user to one or more user groups from user edit screen.
  1. Object count under a folder and pagination.
      • Users can now see the number of objects available in each folder from the explorer screen. Additionally, a load more button has been enabled at the top right side in all explorer pages to enhance loading performance.
  1. Re-design the UI for system preferences.
      • A user-friendly enhancement has been made in the UI for the system preferences page which is located in the Admin Console tab.
  1. Optimizing Azure DevOps Bug Creation.
      • When user selects an option to attach the exceptions dataset to the DevOps defect – now user can limit the number of exception records to be saved. This will avoid any error while creating the defect due to the attachment being too big.
  1. Access controls to the user-defined variables.
      • The user now have the ability to restrict access to variables to one or more user groups
  1. Identifying the variables (query-based) that are part of the queries/scenarios and add them to export/import process.
      • If users wanted to export any variables using Queries or Query chains, Scenarios, and Jobs from one system to another, then they can go through the content management console of one system and select any specific variables used in Queries or Query chains, Scenarios, or Jobs that they wanted to export into another system.
  1. BDV support of Apache drill profile (AWS S3, GCS, Azure Data lake, etc.).
      • Users can now perform pushdown validations of the files stored in the cloud BLOB storages (AWS S3, GCS, and AZURE Datalake) using the BDV scenario.
  1. Compare type switch in the TDR (Technical Data Reconciliation).
      • Prior to version 2021.06, when a TDR is created using Rowcount compare setting, there is no option to switch to data compare and vice versa. Users were required to create a new scenario from scratch for data comparison.
      • With this new feature, users now can easily switch between “Count compare” and “Data compare”

    17. Deletion of any object shouldn't disturb the downstream objects.

      • If a user deletes an object in a job, prior to version 2021.06 – the downstream steps are used to turn it into draft mode. Now deleting a step doesn’t affect the existing configuration flow of its downstream objects including additional options.
  1. Replace Profiles and Variables in Rightdata (DataTrust) rest API.
      • Prior to 2021.06, users can generate rest API for RightData (DataTrust) components like Query Chain, Scenario, and Job. This allows users to execute these components from external systems.
      • Starting 2021.06, users can change the connection profile or variable values at runtime. This feature allows the user to templatize the jobs/scenarios.
      • Example:
        • There is an FDR created to compare data between source and target from dev systems. Without touching the scenario configuration, if you want to run the same scenario to compare source and target from QA systems, you can now simply change the connection profile in the rest API input and trigger rest API to compare QA systems.
        • If there is an FDR to compare 100 million records between source and target, instead of comparing all 100 million records in one go – you can apply a where-clause filter using variables for a subset of data.
  1. Auto-generate FDR scenarios from excel for SAP profiles.
      • Auto-generate FDR feature allows users to create one or more scenarios by uploading the configuration from excel. This feature was only available for RDBMS profiles.
      • Starting 2021.06, this feature has been extended to SAP ECC profiles.
  1. Security Token and Key encryption for Power BI Profiles.
      • Two new parameters “Client_id” and “Client_secret” have been added to Power BI connection profile configuration.
  1. Power BI visualization filters support.
      • Users can now save the visualization filter values for Power BI reports in the RightData (DataTrust) queries. With this feature - users can perform reconciliation and validations for the filtered datasets from the Power BI report visual.  
  1. TDR Exceptions persistency limit for data comparison.
      • Often, customers may end up incorrectly configuring the field mapping between source and target; returning in too many exceptions persisted in the RightData (DataTrust) exception database. This could lead to the filling up of the database and hog the processing resources.
      • This feature limits the number of exception records to be persisted. This will significantly improve the performance of the scenario execution.
      • For example, if you are comparing the source table with approximately 100 M records with 100 M records from the target tables and configured to persist 10,000 records as exceptions limit. If comparison results in 500,000 records – instead of saving all 500,000 records only top 10,000 records will be saved, but the actual mismatch count will be reported.

    23. Jira and X-ray Integration.

      • Ability to associate a Scenario with a Jira Test.
      • In FDR Scenarios, ability to create a separate Jira Bug for each of the exceptions categories (missing in source, missing in target, key exist in both but data mismatch).
      • Any Jira Bug created by RightData (DataTrust) will be labeled "Created-by-RightData".
      • Option to give the Jira Bug closure Id for auto-closure for the Bug in case of a PASS result in a subsequent execution.
      • Ability to choose a Test Execution along with a Test in X-ray configuration of a Scenario.
      • If a Test Execution is selected in X-ray configuration, no new Test Execution will created by RightData (DataTrust). If no Test Execution is selected, RightData (DataTrust) creates a new Test Execution for every Scenario or Job Execution, irrespective of the result.

 

 

Was this article helpful?