Configure GCS as Data Source

This document explains how to create a connection profile for GCS (Google Cloud Storage) within DataTrust. This connection profile can help you access a large amount of data stored on Google Cloud easily.

How to Configure GCS:

As GCS is cloud storage, we cannot setup its connection directly. In order to setup a GCS connection we need to

• Configure the tool called Apache drill.
• Configure Apache Drill on Web Console.
• Create Connection Profile on DataTrust.

How to Configure Apache Drill Tool:

1. Steps to Download the Apache Drill

1. First go to the link https://drill.apache.org/download/ to download the Apache drill tool.
2. Extract the original file from the tar.gz file saved in the download folder.
3. Download the third party jar called the gcs-connector-hadoop3-2.2.0-shaded.jar from the following link - https://repo1.maven.org/maven2/com/google/cloud/bigdataoss/gcs-connector/hadoop3-2.2.0/
4. Save that folder in the location: – Download\Apache\jars\3rdparty

2. Steps to Select the Service Account

1. Login to your Google Cloud console.
2. Go to Admin
3. Select the service account you want to setup the GCS connection for.
4. Setup the necessary permissions for that service account.

3. Steps to Create the JSON File

1. Go to Service Accounts tab under Google Cloud Admin
2. Click the Service Account with which you want to setup the connection
3. Go to Add Key option under the heading “Keys”
4. From the dropdown select the option “Create New Key”
5. Select JSON
6. Click on Create
7. You also have to ensure that the service account has the following permission GCS Storage Owner role for the buckets where the files are stored.
8. After you complete setting up the service account and the Apache Drill You need to download the credential JSON File
9. Save the JSON file in the location: Downloads\Apache\ bin.

4. Configuration Steps

After saving the credentials you need to complete the configuration. To complete the configuration do the following.

1. Go to the conf folder in the location: Downloads\apache\conf
2. Open the core site example.xml file.
3. Paste the content of the configuration file provided to you in this file.
4. Edit the ‘Project id’ and ‘credential JSON file’ name with the relevant data you just created.

5. Restart the Apache drill from the task scheduler to apply all the changes.

How to Configure Apache Drill on Web Console:

1. Open web console - http://localhost:8047
2. Click on the Storage tab
3. Under Plugin Management click on Create Button and a new window will pop -up.
4. Copy the content from ApacheDrillStoragePlugin file provided and paste in the config text box. Ensure to give the correct bucket name and name the Plugin “GCS”.
5. Click on Create to create the Plugin.

How to Create Connection Profile:

1. Login to DataTrust and Go to Admin Console Tab.
2. Click on the Profile button on the extreme right side.
3. From The dropdown Select “Apache Drill”
4. Ensure to give the IP address of the system where Apache Drill is running. (if it’s the same system give localhost.)
5. In the text box labeled Schema, enter the name of the new Plugin ( In this example ‘gcs’)
6. Click on Test Connection to ensure the setup is done correctly.
7. Click on Save to save the profile.

8. You can even provide additional settings for the profile.