The tech industry has evolved rapidly and AI coding tools are changing how we develop. For Databricks developers, tools like Cursor IDE offer significant productivity gains when used correctly. The difference between frustration and success comes down to providing the proper context.
In this article and video, I explain recommendations to use Cursor with Databricks. This includes some development tools that I have shared about before: Databricks Connect and the VS Code Extension for Databricks.
If you prefer reading, continue on for the highlights of my suggestions and links to related content that helped me build out my recommendations. If you prefer video, watch this presentation and demo of how to setup and use Cursor with Databricks.
Why Context Makes or Breaks AI Development
Before we get into specifics about Cursor, we need to talk about context. Without proper context, AI tools will not create the code you want to use and maintain. When you ask an AI assistant to write code without specifying your technology preferences, project structure, or coding standards, it is unpredictable what the finished codebase will look like. You’ll end up complaining that AI coding is overhyped while fighting with poorly structured code.
Context means providing AI tools with the correct information at the right time. This includes:
- Technology stack decisions (pytest vs unittest, UV vs pip)
- Project structure conventions (where tests, notebooks, and libraries belong)
- Code style guidelines (documentation standards, naming conventions)
- Environment configuration (virtual environments, Databricks profiles)
Think of it as teaching a new team member your development standards upfront rather than correcting their work repeatedly. Investing in context early will quickly pay off as you get much cleaner code and can reuse these configurations across projects.
For Databricks specifically, proper context means the AI understands to use Databricks Connect sessions instead of standard Spark sessions, knows your cluster configuration approach, and follows your patterns for structuring data pipelines and notebooks.
Setup Cursor with Databricks Connect
Databricks Connect allows you to develop locally while executing Spark workloads on Databricks compute. Your local machine builds the Spark execution plan and submits only Spark operations to Databricks, while other code runs locally. This architecture provides the best of both worlds: local IDE capabilities with cloud compute resources.
Installation and authentication:
- Start by installing Cursor from their website and creating an account. The 14-day trial includes the features you need to get started. Once that expires, I found it valuable to get the paid tier. You can start with Pro for learning and occasional use. If you use Cursor a lot, you may want a higher (more expensive) tier.
- Next, install the Databricks extension within Cursor. This extension helps manage your Databricks connection, provides specific run options, and provides additional functionality like Asset Bundles integration.
- Configure your Databricks profile using the extension’s setup wizard. You’ll need your workspace URL (from your Databricks workspace address) and an authentication method. OAuth is the preferred method, but an access token will also work well.
The critical configuration detail:
Name your profile “DEFAULT” in all caps. This simple decision eliminates countless issues with AI tools. Databricks Connect automatically looks for the default profile, and when your AI assistant runs code, it won’t need explicit instructions about which profile to use. As you switch between development environments, simply change which profile is the default rather than reconfiguring everything.
Add your cluster ID to the profile configuration file (located at ~/.databrickscfg). For serverless compute, use serverless_compute_id = auto instead.
Virtual environment setup:
Always use a virtual environment for Python development. You can create one manually or let Cursor create it for you. Install Databricks Connect in your virtual environment. The Databricks extension can handle this automatically, though you may need to adjust the version to match your Databricks runtime.
Run simple Databricks Connect code to verify everything works. The built-in Cursor run button works well, mimicking how the AI agent executes code during development sessions.
Leveraging Cursor Rules for Consistent Results
Cursor rules are configuration files that guide AI behavior across your project. They transform chaotic, inconsistent AI output into structured, professional code that follows your standards.
Rules live in a hidden .cursor/rules directory within your project. Each rule file contains instructions for the AI and can be configured to apply globally or only to specific file types.
Essential rule categories:
Python Development Rules: Specify your testing framework preference (pytest over unittest, for example), dependency management approach (UV, pip, poetry), code formatting tools (black, ruff), and documentation standards. These rules prevent the AI from mixing testing frameworks or making arbitrary tooling choices.
Example guideline: “Always use pytest for testing. Never use unittest. Create fixtures for reusable test data. Use descriptive test function names that explain what part of the code is validated by the test.”
Project Structure Rules: Define your directory organization upfront. Specify that library code belongs in a src/{library_name} directory, notebooks and Python scripts go in src/, tests live in tests/, and development scripts belong in scripts/. Include information about where configuration files, documentation, and sample data should reside. This prevents the AI from creating tests in the same file as implementation code or inventing its own organization scheme that differs from your standards.
Testing Guidelines: Control test generation by specifying how many tests to create per function (typically one happy path test with multiple assertions and one failure case), whether to use test classes or standalone functions, and how to structure integration tests. Without these guidelines, AI tools often generate too many tests for a single function.
Creating and refining rules:
Start with minimal rules covering your basic preferences. As you work, you’ll notice patterns in what the AI does wrong. Each mistake becomes an opportunity to refine your rules. The process looks like this:
- AI makes an unwanted decision (uses unittest instead of pytest)
- You add a rule specifying your preference
- Future code generation follows your standard
- Rules accumulate into a comprehensive guide
You can even use Cursor to generate rules. When the AI creates something you don’t like, submit a prompt like: “/Generate Cursor Rules to prevent this pattern and requires [your preferred approach] instead.”
Rules are reusable across projects. Develop a good set of Python development and testing rules once, then copy them to new projects. Over time, you build a personal library of rules that accelerate every new project.
Model Context Protocol (MCP) Integration
Model Context Protocol (MCP) provides standardized interfaces between AI tools and external services. For Databricks development, MCPs are not strictly necessary for productivity but they offer convenient access to documentation and services.
Context7 – The essential MCP:
Context7 indexes documentation for major technology stacks and makes it queryable by AI tools. Instead of manually providing documentation links or hoping the AI remembers syntax correctly, Context7 lets your AI assistant search current documentation in real-time.
For Databricks development, Context7 includes Databricks SDK documentation, PySpark references, and related libraries. When your AI needs to understand proper Databricks Connect session initialization or the latest SDK methods, it queries Context7 rather than relying on potentially outdated training data.
Setup involves installing the Context7 MCP and configuring it in Cursor’s settings. Once configured, your AI automatically has access to these documentation sources.
Databricks-specific MCPs:
Several experimental MCPs exist for Databricks-specific functionality:
- Unity Catalog functions MCP for accessing UDFs and catalog metadata
- Genie workspace MCP for querying datasets through natural language
- Vector Search MCP for semantic search capabilities
There are some options for unofficial Databricks MCPs with different focuses. Research current options before adopting one, as the landscape changes frequently.
The practical reality:
You don’t need MCPs to be productive with AI-assisted Databricks development. Proper Cursor rules, Databricks Connect configuration, and occasional manual documentation links get you most of the way there. MCPs add convenience and polish but aren’t prerequisites. Start without them, then add MCPs as you identify specific pain points they address.
Summary
Effective AI-assisted development for Databricks requires layering several concepts:
Foundation: Databricks Connect properly configured with the DEFAULT profile provides reliable connection to Databricks compute from your local environment.
Structure: Cursor rules ensure consistent code organization, appropriate testing, and adherence to your development standards without constant manual correction.
Enhancement: MCPs like Context7 give your AI access to current documentation, reducing errors from outdated information.
Workflow: Commit frequently, test incrementally, and review all AI-generated code carefully. The AI is a powerful assistant, not an autonomous developer (though this is debatable if you provide REALLY good context).
The developers who succeed with these tools aren’t necessarily more skilled at coding, they’re better at understanding what they need to build and providing context so that AI can make good decisions from the start. It’s best to invest time in creating comprehensive rules, understanding how to structure effective prompts, and knowing when to let AI handle work versus doing it themselves.
So what now? Start small: configure Databricks Connect, create basic Python development rules, and work on a simple project. As you encounter issues, refine your rules. As your rules improve, your productivity compounds. What initially takes hours of iteration eventually happens correctly on the first attempt because you’ve taught your AI assistant your standards.
AI-assisted development is changing what data engineering looks like. It’s hard to predict where things will go, but I believe you will need the skill of leveraging AI to build out solutions quicker. The skill is shifting from writing every line of code yourself to architecting systems, providing effective context, and reviewing AI-generated implementations. Master context engineering now, and you’ll find yourself building in hours what previously took days.
Resources
https://github.com/datakickstart/ai-coding-tools
https://docs.databricks.com/aws/en/generative-ai/mcp
https://context7.com/?q=databricks
Excellent video on Databricks Connect + Cursor. I assume that you could probably extend the cursor rules to .claude.local.md and use Claude Code to achieve the same results.