In the rapidly advancing field of agentic artificial intelligence, a key hurdle often lies not in the sophistication of AI models but in the intricate 'glue code' needed to bridge these models with vast data repositories. Developers frequently encounter repetitive tasks, such as defining tools, fetching schemas, and wrapping SQL execution logic, whenever an agent requires interaction with a Databricks Lakehouse.
This persistent challenge of brittle and unscalable agent integrations has prompted the creation of a custom Databricks Model Context Protocol (MCP) server. This innovative server functions as a standardized conduit, allowing any MCP-compliant agent—be it a custom LangGraph bot or other intelligent systems—to autonomously discover Unity Catalog metadata, inspect table lineage, and execute Spark SQL queries. Crucially, this is achieved without embedding specific tool definitions directly within the agent's codebase.
Understanding the Model Context Protocol (MCP)
Before delving into the technical specifics, it is important to grasp the essence of MCP. This open standard facilitates the decoupling of AI models from the tools they utilize. Rather than manually crafting function schemas, such as those used for OpenAI function calling, within a chatbot's code, a lightweight server is deployed to expose these capabilities. The agent then simply connects to this server and 'learns' about the available tools dynamically. Conceptualized as a universal connector for AI tools, MCP standardizes how models interface with data and execution environments.
Architectural Overview and Implementation
The architecture central to this solution involves an AI agent connecting via the MCP Protocol to the custom Databricks server. This server, in turn, acts as a proxy, routing requests to the Unity Catalog for metadata access and to Serverless Compute for SQL execution. Python was chosen for the implementation due to the maturity and rich features of the Databricks SDK for Python. To efficiently manage the MCP protocol details, the mcp library, particularly its FastMCP class, was leveraged.
The server's design incorporates a critical 'LLM Translation' layer. API responses from Unity Catalog, often voluminous JSON objects filled with identifiers and timestamps, are unsuitable for direct consumption by Large Language Models (LLMs). To conserve tokens and enhance model comprehension, these responses are systematically converted into clean, LLM-friendly Markdown format. LLMs, extensively trained on Markdown text, demonstrate superior understanding of structured information presented with headers and bullet points, thereby significantly improving their ability to interpret table structures.
Core Functionalities and Tools
The server currently exposes four primary tools, designed to guide an agent through a logical workflow of discovery, exploration, inspection, and execution:
fetch_schemas_in_catalog(catalog: str): This initial tool allows an agent to discover available schemas within a specified catalog, acting as a high-level entry point.fetch_tables_in_schema(catalog: str, schema: str): Once a relevant schema is identified, this tool retrieves a list of tables it contains, facilitating targeted exploration.fetch_table_info(table_names: List[str]): A comprehensive tool that provides detailed metadata, including column names, data types, comments, and crucially, upstream and downstream lineage dependencies for specified tables.execute_spark_sql_query(query: str): The final tool enables the agent to retrieve actual data by executing read-only Spark SQL queries via the Databricks SQL Statement Execution API. This is explicitly restricted to SELECT statements to prevent unintended data modifications.
Security and Governance Principles
Integrating AI agents with sensitive data demands robust security measures. The SQL execution tool is designed strictly for data retrieval, with actual enforcement occurring at the credential level. The MCP server operates under a Service Principal identity possessing narrowly scoped permissions.
Leveraging Databricks Unity Catalog (UC) provides a significant advantage. The solution avoids the need for custom row-level security or access control lists within the Python code. Instead, the agent inherits the powerful governance capabilities of Unity Catalog. If the Service Principal lacks SELECT permissions on a table containing Personally Identifiable Information (PII), the query is blocked at the Databricks layer, with Unity Catalog handling masking, filtering, and access denial, mirroring the security experience for human users.
Real-World Application: The AgenticLakehouse
This MCP server extends beyond a mere proof of concept; it serves as a critical component of a larger initiative known as AgenticLakehouse. This multi-agent system, constructed with LangGraph, employs a 'Router' agent that intelligently delegates tasks. For instance, in response to a user query like “Who are our top customers in the retail sector?”, the agent dynamically utilizes the four tools: discovering schemas, exploring tables, inspecting table metadata and lineage, and finally executing a precise SQL query to retrieve and rank customers. This modular design, by separating tools via MCP from the reasoning process in LangGraph, enables agents to dynamically navigate the Lakehouse structure, moving beyond static, hardcoded assumptions.
Future Enhancements
This initial version lays a strong foundation, with future plans for expanding the server's capabilities. These include integrating MLflow to provide tools for listing experiments and fetching optimal runs to assist with MLOps workflows, incorporating the Jobs API to facilitate debugging of failed workflows through run log retrieval, and adding standard system prompts to the MCP server to guide LLMs in interpreting Databricks-specific nuances more effectively.
This article is a rewritten summary based on publicly available reporting. For the original story, visit the source.
Source: Towards AI - Medium
