Looking for the previous version? View v2024.06 documentation →
Query Studio — Data Profiling
Build a Query Chain with a Data Profiling widget to analyse column statistics, null rates, and value distributions.
Overview
Data Profiling in Query Studio allows you to analyse the statistical characteristics of your data directly within a Query Chain. By attaching a Data Profiling widget to your chain, you can view column-level statistics such as null counts, distinct values, min/max, cardinality, mean, and frequency — helping you understand data quality before building reconciliations or validations.
Navigating to Query Chain
Figure 1: Navigating to Query Studio — Query Chain
In the left sidebar, click Query Studio. The Explorer opens showing your folders and assets. In the left panel sub-navigation, click Query Chain to open the Query Chain canvas editor.
Adding a Data Source — RDBMS Widget
Figure 2: Dragging the RDBMS Table widget onto the canvas
The Query Chain canvas has a widget palette at the top under the Data Sources tab. Drag the RDBMS Table widget onto the canvas. This creates a data source node that reads from a relational database table.
Figure 3: Read data from Table popup
Double-clicking the RDBMS widget (or it opens automatically after dropping) shows the Read data from Table popup. Select the Database Type (e.g. Oracle) and the Connection Name from the dropdowns. Then use Pick Table to choose the source table.
Figure 4: Show Row Count
Click Show Row Count to fetch the total number of rows in the selected table. This helps you gauge the dataset size before running the full chain.
Figure 5: Closing the row count popup
After reviewing the row count, click the X to close the count display and return to the table configuration popup.
Figure 6: Previewing sample data
Click Preview Data to load a sample of the table's rows directly in the popup. This lets you confirm you have the right table and understand its column structure before proceeding.
Figure 7: Returning from Preview Data
Click X on the preview panel to return to the main RDBMS configuration popup.
Figure 8: Confirming the table selection
Once you have selected the correct table and connection, click Select to confirm. The RDBMS widget on the canvas updates to show the selected table name.
Adding a Transformation — Filter
Figure 9: Switching to the Transformations tab
Click the Transformations tab in the canvas toolbar to see the available transformation widgets — including Filters, Joins, Aggregations, and more.
Figure 10: Adding a Filter widget
Drag the Filters widget from the Transformations palette onto the canvas. Connect it to the RDBMS widget by drawing a line from the RDBMS output port to the Filter input port.
Figure 11: Opening the Filter widget
Double-click the Filter widget to open its configuration popup. The filter lets you define conditions to restrict which rows flow through the chain.
Figure 12: Selecting a field to filter on
Click the Available Fields dropdown to see all columns from the connected data source. Select the column you want to filter on.
Figure 13: Available Fields list
The dropdown expands to show all available columns from the upstream data source. Click the column name you want to apply the filter condition to.
Figure 14: Selecting the filter operator
After selecting a field, choose the comparison operator from the Operator dropdown (e.g. equals, greater than, less than, contains). The available operators depend on the selected field's data type.
Figure 15: Entering the filter value
Enter the filter value in the input field. In this example, the value 10530 is entered to filter rows where ORDERID is greater than 10530. Click Apply or confirm to save the filter condition.
Figure 16: Query Chain with Data Source and Filter connected
The canvas now shows two connected widgets: the RDBMS widget reading from the source table, and the Filter widget applying the row condition. The connection line between them shows the data flow.
Adding an Output Widget
Figure 17: Switching to the Output tab
Click the Output tab in the canvas palette. The available output widgets appear: RD Output (stores results in Data Trust), Data Profiling, and Offload File.
Figure 18: Adding the RD Output widget
Drag the RD Output widget onto the canvas and connect it to the Filter widget output. RD Output stores the query results as an extract that can be viewed in Explore Extracts.
Figure 19: RD Output configuration popup
The RD Output popup lets you configure how the output data is stored, including naming and storage options. Configure as needed and confirm.
Adding a Data Profiling Widget
Figure 20: Query Chain pipeline ready for Data Profiling
The Query Chain now has: RDBMS source → Filter → RD Output. Return to the Output tab to add the Data Profiling widget to the same chain.
Figure 21: Adding the Data Profiling widget
Drag the Data Profiling widget from the Output palette and connect it downstream. The full chain now reads: RDBMS → Filter → RD Output → Data Profiling, providing both data storage and statistical profiling in one execution.
Figure 22: Data Profiling popup
The Data Profiling popup opens automatically. Configure the profiling options as needed (e.g. selecting which columns to profile) and confirm.
Executing and Viewing Profiling Results
Figure 23: Executing the Query Chain
Click the Execute Now button (play icon in the toolbar) to run the Query Chain. Data Trust reads the source data, applies the filter, stores the output, and runs the profiling analysis.
Figure 24: Monitor mode — execution in progress
After clicking Execute Now, the Query Chain enters Monitor mode. The status of each widget updates in real time. Wait for all widgets to show a green completed state.
Figure 25: Data Profiling results page
Once execution completes, the Data Profiling results page opens. It displays a statistics grid with one row per column. For each column you can see: Data Type, Null %, Distinct, Non-Distinct, Value (Min → Max), Length (Min → Max), Avg Length, Cardinality, Sum, Mean value, High Frequency value, and Standard Deviation.
Figure 26: Selecting columns for trend charts
Check the boxes next to specific column names in the profiling grid to select only those columns for the trend chart analysis. This focuses the visualisation on the data dimensions you care about most.
Figure 27: Re-executing with selected columns
After selecting the required columns, click Execute Now again to regenerate the trend charts filtered to only your selected columns.
Figure 28: Trend charts — selected columns only
The trend charts update to show only the selected columns. Each chart visualises the value distribution over the dataset, making it easy to spot outliers, skewed distributions, or data quality issues.
Figure 29: Distribution graph
Click on a column's Distribution graph icon to see a detailed histogram of value frequencies for that column. This helps identify the most common values and the spread of the data.