Skip to content

added mounting method for passthrough #1

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
flomader wants to merge 1 commit into
base: master
Choose a base branch
from
Open

added mounting method for passthrough #1

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ To download all sample notebooks, [here](notebooks/connect_storage.dbc) is the D
| | |AD Passthrough|**User has correct RBAC role assigned OR ACLs permissions to files/folders in ADLS Gen2|[Python](notebooks/directconnect_adpassthrough_py.py), [SQL](notebooks/directconnect_adpassthrough_sql.sql)| [ADLS Gen2](https://docs.azuredatabricks.net/spark/latest/data-sources/azure/azure-datalake-gen2.html#access-adls-automatically-with-your-aad-credentials)|
|**Mount on DBFS**|Databricks Workspace|Storage Key|All rights|[Python](notebooks/mount_storagekey_py.py)|[Blob](https://docs.azuredatabricks.net/spark/latest/data-sources/azure/azure-storage.html#mount-azure-blob-storage-containers-with-dbfs), [ADLS Gen2](https://docs.azuredatabricks.net/spark/latest/data-sources/azure/azure-datalake-gen2.html#access-an-adls-account-directly-using-the-storage-account-access-key)|
| | |OAuth via Service Principal (SP)|**SP has correct RBAC role assigned OR ACLs permissions to files/folders in ADLS Gen2|[Python](notebooks/mount_oauthsp_py.py)| [ADLS Gen2](https://docs.azuredatabricks.net/spark/latest/data-sources/azure/azure-datalake-gen2.html#mount-an-adls-filesystem-to-dbfs-using-a-service-principal-and-oauth-2-0)|
| | |AD Passthrough|**User has correct RBAC role assigned OR ACLs permissions to files/folders in ADLS Gen2|[Python](notebooks/mount_adpassthrough_py.py)|[ADLS Gen2](https://docs.azuredatabricks.net/spark/latest/data-sources/azure/adls-passthrough.html#mount-your-data-with-dbfs)|
|---|---|---|---|---|---|


Expand Down
Binary file modified notebooks/connect_storage.dbc
View file
Open in desktop
Binary file not shown.
21 changes: 21 additions & 0 deletions notebooks/mount_adpassthrough_py.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
# Databricks notebook source
# MAGIC %md
# MAGIC ## Azure DataLake Gen2
# MAGIC
# MAGIC 1. Azure Databricks Premium Plan.
# MAGIC 2. Azure Data Lake Storage Gen2: Databricks Runtime 5.3 or above.
# MAGIC 3. High concurrency clusters, which support only Python and SQL. [Enabled AD Passthrough checkbox under Advanced Options](https://docs.azuredatabricks.net/spark/latest/data-sources/azure/adls-passthrough.html#enable-passthrough-for-a-cluster)
# MAGIC 4. User needs to have [Storage Data Blob Owner/Contributor/Reader role](https://docs.microsoft.com/en-us/azure/storage/common/storage-auth-aad-rbac-portal#rbac-roles-for-blobs-and-queues) OR [appropriate ACL permissions (R/W/E) on ADLA Gen2](https://docs.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-access-control#access-control-lists-on-files-and-directories) is granted

# COMMAND ----------

configs = {
"fs.azure.account.auth.type": "CustomAccessToken",
"fs.azure.account.custom.token.provider.class": spark.conf.get("spark.databricks.passthrough.adls.gen2.tokenProviderClassName")
}

# Optionally, you can add <your-directory-name> to the source URI of your mount point.
dbutils.fs.mount(
source = "abfss://<STORAGE_CONTAINER>@<STORAGE_ACCOUNT>.dfs.core.windows.net/",
mount_point = "/mnt/my_adls_gen2_mount_path/",
extra_configs = configs)