Ingest tables in parallel with an Apache Spark notebook using multithreading

By dustinvannoy May 6, 2022 / 1 Comment

If we want to kick off a single Apache Spark notebook to process a list of tables we can write the code easily. The simple code to loop through the list of tables ends up running one table after another (sequentially). If none of these tables are very big, it is quicker to have Spark load tables concurrently (in parallel) using threads. There are some different options of how to do this, but I am sharing the easiest way I have found when working with a notebook in Databricks, Azure Synapse Spark, Jupyter, or Zeppelin.

Spark Streaming join to Slow Changing Data

By dustinvannoy Jun 9, 2021 / Leave a comment

f you are building data pipelines for a video streaming site, you would need to consume analytics about video views in real time. Assume you need to look up additional user attributes like the subscription level, that information will change very infrequently. However, once that change happens its important to start tying usage to the correct subscription right away. So you need to find the best way to lookup that info in Apache Spark. With Delta Lake format, the batch data frame will update in memory without restarting the stream. The video in this post shows an example of this in action. Delta Lake supports updates via the merge statement so you keep the data up to date in your file system and Spark will also update its in memory data frame.

Which language for Apache Spark

Best Language for Apache Spark

By dustinvannoy Apr 7, 2021 / 1 Comment

The question is raised often, “What programming language should we choose for our Apache Spark project?” The short answer I give is to choose between Scala or Python. I admit, this is only slightly more helpful than saying it depends, which I try to avoid. The real question is what are the tradeoffs between the… Continue Reading

Spark Summit Takeaways

By dustinvannoy Jun 26, 2020 / Leave a comment

Wrapping up my attendance at Spark + AI Summit 2020 and I found a lot of value. Here are my quick takeaways to try and save you time. To keep it real, some sessions were a big miss for me either due to too much detail or not enough focus, but some were awesome. If… Continue Reading

Data Lake Introduction

By dustinvannoy Mar 5, 2020 / Leave a comment

Hearing a lot of mention of Data Lakes but still not sure what that means or why anyone cares? This video will cover a brief introduction to what a Data Lake is and why so many organizations are adding them to their analytics ecosystem. To show what interacting with a data lake may look like for a typical data analyst, I included a demo of how you would use Spark SQL to query the data lake from Azure Databricks.

Stream image

Spark Streaming with Azure Databricks

By dustinvannoy Dec 4, 2019 / Leave a comment

In the world of data science we often default to processing in nightly or hourly batches, but that pattern is not enough any more.  Our customers and business leaders see information is being created all the time and realize it should be available much sooner.  While the move to stream processing adds complexity, the tools we have available make it achievable for teams of any size.

This presentation covers why we need to shift some of our workloads from batch data jobs to streaming in real-time.  We dive into how Spark Structured Streaming in Azure Databricks enables this along with streaming data systems such as Kafka and EventHub. We will discuss the concepts, how Azure Databricks enables stream processing, and review code examples on a sample data set.

Delta Lake on Azure Databricks

By dustinvannoy Nov 21, 2019 / Leave a comment

With the shift to data lakes that use distributed file storage as the foundation, we have been missing the reliability that relational databases provides. Databricks Delta is a data management system focused on bringing more reliability and performance into our data lakes. It sits on top of existing storage and the API is very similar to reading and writing to files from Spark already. This session will present the overview of Delta Lake, why it may be a better option than standard data lake storage, and how you can use it from Azure Databricks.

Azure Databricks from PyCharm IDE

By dustinvannoy Sep 4, 2019 / 4 Comments

I am pleased to share with you a new, improved way of developing for Azure Databricks from your IDE – Databricks Connect! Databricks Connect is a client library to run large scale Spark jobs on your Databricks cluster from anywhere you can import the library (Python, R, Scala, Java). It allows you to develop from your computer with your normal IDE features like auto complete, linting, and debugging. You can work in an IDE you are familiar with but have the Spark actions send out to the cluster, with no need to install Spark locally.