This Integration process establishes a connection between the Databricks Platform and DataTrust
Steps for Integration
1. Navigate to System Preferences:
- Click the ADMIN CONSOLE module.
- Select Menu and click System Preferences.
It will take to the System preferences page, under this we have different tiles
2. Instance and Processor Configuration:
- On the System Preferences page, click the Instance & Processors tile.
- In the Instance & Processors section:
- Click the Instance tab
- Click the ADD button
3. Add New Instance:
- A pop-up window will appear:
- Enter the Instance name.
- Provide the workspace URL (Databricks connection URL).
- Enter the Token (authentication token)
- Click Validate token URL
4. Catalog and Schema Configuration:
- If the token is validated successfully, fields for catalog name and schema name will appear.
- The default selection is Create new catalog
- For a new catalog, provide the catalog name and schema name.
- For an existing catalog:
- Select the Existing catalog radio button
-
- Choose from the catalog drop down
- Select either Create new schema or Existing schema
- Enter the schema name if creating a new schema
- For an existing Schema
- Select the schema name from dropdown
- Click the confirm button
5. Confirmation:
- A confirmation pop-up will appear for creating either a new catalog and schema or using an existing catalog
- Confirm to create pop up will appear like this with existing catalog
- Click Proceed to complete the setup:
- This creates a new catalog and schema for internal purposes.
- Create three workflow jobs: Ingestion, Reconciliation, and Transformation.
- A SQL warehouse serverless for JDBC connection to catalogs will also be created.
6. Completion:
- Once completed, green tick marks will indicate successful creation
- Click OK to finalize
- The created instance will be displayed in the Instance tab
- Now the connection being created between Databricks and Data trust
Processor Creation
To execute scenarios with the Databricks processor engine, create a processor using the existing instance.
Steps to Create a Processor
1. Navigate to Processor Configuration:
- Click the Processor tab
- Click the ADD button
2. Add New Processor
- A pop-up window will appear:
- Enter the Processor name
- Select Databricks from the Type dropdown
- Add a User group from the available list
- Select the Instance from the dropdown
- This will populate parameters related to the selected instance
3. Cluster Configuration:
- Select the Cluster type:
- Job cluster or Shared cluster
- Select Job cluster as cluster type
- For an shared cluster, additional cluster fields will appear
- Select the cluster from the dropdown
- Click Save
4. Completion:
- The created processor will be displayed in the Processors tab
- Use the job ID of each workflow for the Ingestion, Reconciliation, and Transformation pipelines in the DataTrust application
- Now we will use the job id of each work flow to perform respective Ingestion pipeline, Reconciliation pipeline and Transformation pipeline in Data trust application.
Add Processor to Folder
To execute scenarios with the Databricks processor engine, add the processor at the folder level.
Steps to Add Processor to Folder
1. Navigate to Query Studio:
- Click Query Studio/Scenario Studio/Batch Console module
- Select Query Studio module.
- Click the Menu button at the folder
- Select Security and choose the user group assigned during processor creation
- Click Save
2. Assign Processor:
- Click Processors to add processors to the folder
- A pop-up will display for selecting the default processor
- Choose the processor from the dropdown
- Click Update to add the processor to the folder
3. Execute Scenario:
- Open any scenario within the folder
- Execute the scenario and verify it runs with the Databricks processor