Configure Azure Data Lake Storage as data source

This document explains how to create a connection profile for ADLS (Azure Data Lake Storage) within DataTrust. This connection profile can help you access files with a large amount of data stored on ADLS easily.

How to Configure ADLS?

DataTrust connects to ADLS storage account via a third-party tool, Apache Drill. The following steps will help you configure both Apache Drill and ADLS connection from DataTrust via Apache Drill:

• Install and Configure Apache Drill.
• Configure Apache Drill on Web Console.
• Create Connection Profile on DataTrust.

How to Configure Apache Drill Tool?

1. Steps to Download the Apache Drill

• First, go to the link https://drill.apache.org/download/ to download the Apache drill tool.
• Extract the original file from the tar.gz file saved in the download folder.
• Download the following jar files from the Maven repository that support ADLS connection in Apache Drill and save the files in the location: <apache-drill-home_dir>\jars\3rdparty.
  • https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-common/3.3.0
  • https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-azure/3.3.0
  • https://mvnrepository.com/artifact/com.microsoft.azure/azure-storage/8.6.5
• Delete any older versions of the jars in the 3rdparty folder.
• Install the latest 32-bit MapR Drill ODBC driver for Windows OS from the link http://package.mapr.com/tools/MapR-ODBC/MapR_Drill

2. Steps to Get the ADLS authentication details

1. Login to your Azure portal.
2. Navigate to the storage account.
3. Navigate to Access Keys section under Settings.
4. Copy the key1 value.

3. Configuration Steps

After saving the credentials you need to complete the configuration. To complete the configuration do the following.

• Navigate to the conf folder in the location: <apache-drill-home-dir>\conf
• Rename the core-site-example.xml to core-site.xml file and enter the following config:

In case of Access Key:

<property>
     <name>fs.azure.account.auth.type</name>
     <value>SharedKey</value>
</property>
<property>
     <name>fs.azure.account.key.storage_account.dfs.core.windows.net</name>
     <value>access_key</value>
</property>

• Make sure the correct storage_account and access_key are updated.
• Start the Apache Drill application by running the drill-embedded batch file in <apache-drill-home-dir>\bin

In case of Service Principal:

<property>
  <name>fs.azure.account.auth.type</name>
  <value>OAuth</value>
</property>
<property>
  <name>fs.azure.account.oauth.provider.type</name>
  <value>org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider</value>
</property>
<property>
  <name>fs.azure.account.oauth2.client.id</name>
  <value>CLIENT_ID</value>
</property>
<property>
  <name>fs.azure.account.oauth2.client.secret</name>
  <value>CLIENT_SECRET</value>
</property>
<property>
  <name>fs.azure.account.oauth2.client.endpoint</name>
  <value>https://login.microsoftonline.com/TENNANT_ID/oauth2/token</value>
</property>

How to Configure Apache Drill on Web Console

• Open web console - http://localhost:8047
• Click on the Storage tab
• Under Plugin Management click on Create Button and a new window will pop -up.
• Copy the following content in the config text box:

{ 
"type":"file",
"enabled":true,
"connection":"abfs://container@storage_account.dfs.core.windows.net/root",
"config":null,
"workspaces":{ 
"root":{ 
"location":"/",
"writable":false,
"defaultInputFormat":null
}
},
"formats":{ 
"psv":{ 
"type":"text",
"extensions":[ 
"tbl"
],
"delimiter":"|"
},
"csv":{ 
"type":"text",
"extensions":[ 
"csv"
],
"extractHeader":true,
"delimiter":","
},
"tsv":{ 
"type":"text",
"extensions":[ 
"tsv"
],
"delimiter":"\t"
},
"parquet":{ 
"type":"parquet"
},
"json":{ 
"type":"json",
"extensions":[ 
"json"
]
},
"avro":{ 
"type":"avro"
},
"sequencefile":{ 
"type":"sequencefile",
"extensions":[ 
"seq"
]
},
"csvh":{ 
"type":"text",
"extensions":[ 
"csvh"
],
"extractHeader":true,
"delimiter":","
}
}
}

• Give the plugin an appropriate name and click on Create to create the Plugin.

How to Create Connection Profile:

• Login to DataTrust and navigate to Admin Console Tab.
• Click on the Profile button on the extreme right side.
• From the dropdown Select “Apache Drill”
• Ensure to give the IP address of the system where Apache Drill is running. (if it’s the same system you can give localhost.)
• In the text box labeled Schema, enter the name of the new Plugin ( In this example ‘azure_data_lake’)
• Click on Test Connection to ensure the setup is done correctly.
• Click on Save to save the profile.

• You can even provide additional settings for the profile.