Azure Databricks with Azure Key Vaults

Image for post
Image for post

Why?

Image for post
Image for post
We can’t restrict a user to view a particular notebook if she/he has access to the workspace.

What is the Solution?

Step 1: Create a new Key Vault

Image for post
Image for post
Open Azure Key Vault, click on ‘Add’
Image for post
Image for post
Input Name, select Subscription, Resource Group and other settings
Image for post
Image for post
Note the DNS Name & Resource Id of the newly created Key Vault

Step 2: Create a Secret

Image for post
Image for post
Select Secrets under Settings at the left blade & create a new secret
Image for post
Image for post
We can store the userid, password or access keys inside the Azure Key Vault

Step 3: Create a Secret Scope at Databricks

Image for post
Image for post
Input a Scope Name, add the DNS Name & Resource ID taken from Step 1

Step 4: Use the Secrets from Azure Databricks

Image for post
Image for post
The actual output is obscured

That’s all?

Image for post
Image for post
Databricks notebook — Revision history

Cautions

Update — 25th June 2020

Image for post
Image for post
Pass the secrets from parent notebook to a called child notebook.
Image for post
Image for post
Once run, the secrets will be revealed in the Parameters section!

Written by

Tech enthusiast, Azure Big Data Architect.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store