How to Protect Your Databricks Code Without the Azure Premium Plan

Have you ever been in the situation of building a Data Lake, everything working… until you realized your credentials were exposed in the code? 😬

Recently I needed to mount Azure Data Lake containers inside a Databricks notebook — a seemingly simple task. But when I looked at the code carefully, I noticed that the access keys were in plain text.

That's when I understood: it's not enough for the code to run, it has to run securely.

And the best part: there is a way to do it without the Azure Premium plan, using only Secret Scopes created via the Databricks CLI.

In this article I share the complete step-by-step and what I learned about protecting credentials in Databricks — a simple process that can save you big headaches down the road.

🚀 Why this matters

In Data Engineering, during development it is common to store keys, tokens or credentials directly in the code. Does it work? It works. But it also opens huge security holes: anyone with access to the notebook can see sensitive data.

That is exactly what motivated me to look for a solution that was: ✅ Secure ✅ Simple to implement ✅ Compatible with the Azure Standard plan

The result was the setup with Secret Scopes via CLI, which I share next.

🧩 Understanding the main components

Before getting your hands on the code, it's worth understanding the role of each part of this setup:

• Azure Data Lake → the central data repository, with containers such as bronze, silver and gold.
• Storage Account → the storage account where the containers are hosted.
• Databricks CLI → a command-line tool that lets you automate actions in the workspace (create scopes, tokens, jobs, etc.).
• Secret Scope → an “internal vault” in Databricks used to store passwords, keys or tokens securely.

🔑 And what is Azure Key Vault?

Azure Key Vault is Microsoft's secure secret storage service. It keeps keys, passwords, certificates and tokens in a centralized and encrypted way, with built-in access control and auditing.

In other words: it is Azure's official vault — the right place to keep any sensitive credential.

Creating a Key Vault is simple:

1. In the Azure portal, go to Create a resource → Security → Key Vault
2. Choose the Resource Group and the vault name
3. Set the region and the access policies
4. Then add your secrets in the Secrets section

💡 Tip: in Databricks, you can integrate Key Vault directly by creating a Key Vault-backed Secret Scope — which lets you access secrets automatically, without storing them in Databricks.

In this article, however, the focus is the Databricks-backed Secret Scope, which works even without the Premium plan.

🛠️ Step by step: secure access to Azure Data Lake in Databricks

✅ Step 1: install and configure the Databricks CLI

In your Windows terminal, install and configure the Databricks CLI:

pip install databricks-cli
databricks configure --token
# Host: https://<your-instance>.azuredatabricks.net
# Token: <your access token>

🔍 How to get the Host and the Token

1. Open your Azure Databricks workspace
2. In the top-right corner, click your profile (your name) → User Settings
3. Go to the Access Tokens tab
4. Click Generate New Token, set an expiration (e.g. 90 days) and copy the token
5. The Host is simply your workspace URL:

https://<your-workspace>.azuredatabricks.net

6. Paste this information when the CLI asks for it

✅ Step 2: create the local Secret Scope

When the Premium plan is not enabled, you need to use the --initial-manage-principal "users" parameter to allow the creation of the local scope.

databricks secrets create-scope \
  --scope datalake-scope \
  --initial-manage-principal "users"

✅ Step 3: add the secrets manually

# Storage Account name
databricks secrets put --scope datalake-scope --key STORAGE-NAME

# Storage access key
databricks secrets put --scope datalake-scope --key STORAGE-KEY

These commands open an editor in the terminal so you can type the values safely.

✅ Step 4: mount the containers in the notebook

A reusable function to mount multiple containers (in this example, the bronze, silver and gold layers):

def mount_container(containerName):
    scope_name = "datalake-scope"
    storageAccountName = dbutils.secrets.get(scope=scope_name, key="STORAGE-NAME")
    storageAccountAccessKey = dbutils.secrets.get(scope=scope_name, key="STORAGE-KEY")
    blobContainerName = containerName
    mountPoint = "/mnt/" + blobContainerName

    if not any(m.mountPoint == mountPoint for m in dbutils.fs.mounts()):
        try:
            dbutils.fs.mount(
                source = f"wasbs://{blobContainerName}@{storageAccountName}.blob.core.windows.net",
                mount_point = mountPoint,
                extra_configs = {
                    f"fs.azure.account.key.{storageAccountName}.blob.core.windows.net": storageAccountAccessKey
                }
            )
            print(f"✅ Mount successful: {blobContainerName}")
        except Exception as e:
            print("❌ Mount error:", e)
    else:
        print(f"ℹ️ Container already mounted: {blobContainerName}")

containers = ["bronze", "silver", "gold"]
for c in containers:
    mount_container(c)

✅ Step 5: test the mount of the bronze container

mount_path = "/mnt/bronze"
if any(m.mountPoint == mount_path for m in dbutils.fs.mounts()):
    print("✅ The 'bronze' container is mounted.")
else:
    print("❌ The 'bronze' container is not mounted.")

🔐 Lessons learned

• 💡 1. Avoid exposing keys directly in the code. Use Secret Scopes or, in production, Azure Key Vault.
• 💡 2. Mount only what you need. Fewer mounts = fewer risks.
• 💡 3. Avoid logs with sensitive information. A simple print() can reveal more than it should.

🌱 Conclusion

Although setting up a Data Lake may seem simple, doing it securely completely transforms the reliability of the environment. Small decisions, like moving a key into a vault, can make all the difference.

And the best part: you can implement all of this in just a few minutes, with the tools you already use every day.

Got any questions? Get in touch and we'll sort them out 😉