Bulk Data Ingestion is a new widget introduced in Query Chain. Using BDI, we can select and ingest multiple tables of the same RDBMS data type. Simultaneously, we can also check the row count and preview the metadata of the selected table.
BDI is followed by BDP (Bulk Data Profiling) which is similar to the existing data profiling functionality but here we can view multiple tables profiling details one after the other as a dropdown list pattern.
To learn more about BDI, let us walk through the below steps:
- Click the QUERY STUDIO module.
- Hover on Query Builder snippet and click on Query Chain.
- A new Query Chain session page is displayed.
- From the Data Source tab, drag and drop Bulk RDBMS Table icon.
- Double tap Read data from Table widget.
- A Read data from Table pop-up window is displayed.
- Select Database Type and Connection Name parameters.
- Click the Navigator.
- From the Navigator, select the desired tables.
Note: Here, four tables from TPCH1 folder are selected.
- Metadata from TPCH1.SUPPLIER is only displayed.
- Click the Show Row Count.
- A Data for TPCH1.SUPPLIER pop-up window is displayed showing the overall records count.
- Click the X icon to close the pop-up window.
- Click the Preview Data button.
- A Data for TPCH1.SUPPLIER pop-up window is displayed showing the metadata.
- Click the X icon to navigate back to the Read data from Table.
- Click the Dropdown icon from the Choose Table.
- All four selected tables are displayed here.
Note: A user can select only one table at a time to check its row count and preview data.
- Click the Select button to save the Read data from Table widget.
- Click the Transformations tab.
- All the transformations are disabled here.
Note: Transformations cannot be performed using multiple tables.
- Click the Output tab.
- Drag and drop RD Output icon and double-tap it.
- An RD Output pop-up window is displayed.
Note: By default, the metadata of the table TPCH1.CUSTOMER is displayed.
- Click on the Table Name dropdown icon.
- Here, all the table names are displayed.
Note: Only one table can be selected at a time to view its metadata.
- Click the Select button.
- Click the Execute Now icon.
- A Monitor mode is enabled.
- Here, the Query Chain is Passed.
- Click the 4/4 RD Output count.
- A Records Ingested tab is displayed that shows the Table Name and Total Records Ingested count details.
- Click the TPCH1.CUSTOMER table records count 150000.
- A new tab Results for ‘TPCH1.CUSTOMER’ tab is opened which displays all its records.
- Click the Records Ingested tab.
- All the other tables’ records count is displayed.
Note: Here, click on the records count of any selected table to display its records.
Bulk Data Profiling
Bulk Data Profiling helps in profiling the bulk ingested data. Data Domain Prediction is a new feature introduced as a part of BDP that assists in predicting the data domains for all columns. Simultaneously, the system-predicted data type can also be overridden by the end user.
Note: Currently, BDP works only with MSSQL, MySQL, Redshift, Oracle, and PostgreSQL profiles.
- From the Output tab, drag and drop the Data Profiling icon and double tap it.
- A Bulk Data Profiling pop-up window is displayed.
Note: Data domain prediction started for all tables message is displayed immediately after opening the Bulk Data Profiling widget.
- By default, one table is selected for which the metadata is displayed.
Note: A Preview Data button is used to preview the data of the selected table.
- Click the Output View Name dropdown icon.
Note: A message Data domain prediction completed is displayed after finishing the domain prediction process.
- Click the Save button to save and close the Bulk Data Profiling widget.
Note: One table can be selected each time to either preview the data or view the metadata of the selected table.
- Click the Execute Now icon.
- A Monitor mode is enabled.
- The execution is Passed here.
- Click the Data Profiling green tick mark icon.
- A BDI and BDP Data Profiling page is displayed.
Note: By default, one table name is selected for data profiling.
- Click the Table Names dropdown icon.
- All the tables list is displayed.
Note: Only one table can be selected at a time to view the selected table Data Profiling details.