This is an updated video and writeup on setting up and using Log Analytics with your Azure Databricks logs. Some of the content overlaps with what I shared in the past, but these instructions are valid for Databricks Runtimes 11.3+.

Log Analytics provides a way to collect and query logs in Azure. For teams that run other workloads in Azure beyond Databricks, they often want to collect their Databricks logs in the same Log Analytics workspace. In this video I walk through the setup steps and quick demo of this capability for the log4j output and the Spark metrics. I include high level steps, links to the README, and troubleshooting guidance in this post to help you set this up yourself.

Typically you will want to get the latest from branch l4jv2 and build using the profile that matches the Databricks Runtime version you will use. However, if you want to skip building for yourself, you can get the last jars I have built for each profile here: https://github.com/dvannoy/spark-monitoring/tree/l4jv2_jars/target.

If your goal is to get to Log Analytics then this library is effective for the supported runtimes and with the README and video you should have what you need to get it working. However, there is an important note from README: This library and GitHub repository are in maintenance mode. There are no plans for further releases, and issue support will be best-effort only.

For more details on what Databricks provides and supports for viewing logs, you can browse the following resources.

• Cluster logs from Cluster UI
• Cluster log delivery to DBFS
• Databricks System Tables
• Azure Databricks Diagnostic logs

Setup steps

For Databricks Runtime 11.3 and above I recommend using the README from the github repository and consulting the video if part of it is not clear.

The high level of steps in order:

1. Clone repository (or just download jars and skip to step 3)
2. Build jars using Maven
3. Create log analytics workspace (if doesn’t exist)
4. Get log analytics workspace id and key (from “Agents management” pane)
5. Add log analytics workspace ID and key to a Databricks secret scope
6. Modify spark-monitoring.sh script as needed
7. Run commands to upload files using Databricks CLI
8. Add environment configs to cluster environment variables
   

9. Add the spark-monitoring.sh init script in the cluster advanced options. I recommend choosing either Workspace or Volume options. Note: Workspace option is not available for Shared access mode.
   

10. Start cluster and confirm Event Log shows successful cluster init
    

11. Confirm custom logs are created in Log Analytics and messages are flowing to it
    

Troubleshooting

What if custom logs do not show up in Azure Log Analytics?

There are a few things to look at to try and see what has gone wrong.

1. Start cluster and watch event log to confirm you see an INITS_SCRIPTS_FINISHED message for spark-monitoring.sh.
2. Give it a little time when running for the first time in a Log Analytics workspace. Tables should be created within 10 minutes but may not show up in the first couple of minutes.
3. Confirm cluster environment variables were set and that they reference secret names in a Databricks secret scope. To check what is in your databricks secret scope, replace demo with your secret scope name and run the following script from a notebook: dbutils.secrets.list(scope=”demo”)
4. Confirm the init script was added properly.
5. If logs are setup to be saved to DBFS, view the latest init logs to find details about any errors that happened during init.