Sequel2SQL

An Agentic Framework for SQL Error Correction

The Team:

Photo	Bio
	Akshay Ravi (Data Scientist) Akshay is a Data Scientist specializing in finding and fixing system inefficiencies. His expertise in process optimization and high-speed algorithms directly supports the Sequel2SQL mission of generating efficient, corrected queries. He also brings practical experience in building streamlined, automated data pipelines that are required to log and analyze query performance.
	Aravindh Manavalan (Team Coordinator / Data Scientist) Aravindh is a Data Science professional with experience in data analytics, data engineering and agentic AI. His proven track record in leading data projects and hands-on experience with LLMs and AI Agents directly supports Sequel2SQL’s vision of building a robust error correction system. Aravindh aims to focus on product architecture and agentic system design.
	Jay Sanghavi (Data Scientist) Jay Sanghavi is a data science graduate student with experience in building scalable ML and analytics pipelines. He has worked on real-time model deployment, predictive maintenance, and SQL-based data workflows, and is skilled in Python, cloud data tools, and model evaluation. Jay supports the team by focusing on reliable, production-minded ML and clear, data-driven problem solving aligned with Microsoft-scale standards.
	Smeet Dedhia (Data Scientist) Smeet will work primarily as a data scientist, while also supporting general software development tasks. He has previous experience with RAG systems & Generative AI which would be helpful in this project. He has also worked with large codebases shared across teams, and would bring this experience for setting up efficient code collaboration for this project.
	Vijay Balaji S (Data Scientist) Vijay will work as a Data Scientist on the team, bringing a blend of analytical insight and technical depth to the project. Vijay will lead the design of the model pipeline that enables the system to understand, debug, and learn from SQL errors, combining retrieval, language modeling, and reinforcement learning into a self-improving query correction framework.

Project Sponsor: Dhruv Relwani : Software Developer, Microsoft.

1 Introduction

Structured Query Language (SQL) is the primary language used for interacting with relational databases and is a critical tool for data engineers, analysts, and backend developers. In real-world data workflows, engineers frequently write complex SQL queries involving multiple joins, aggregations, and nested subqueries. As database schemas grow in size and complexity, even experienced practitioners often encounter errors while writing or modifying SQL queries.

Debugging SQL queries can be time-consuming because many errors arise from subtle issues such as incorrect join conditions, missing aggregation clauses, schema mismatches, or misuse of SQL functions. These issues become especially common when queries are adapted across different database schemas or SQL dialects. As a result, SQL debugging remains an important but often repetitive task in modern data engineering workflows.

Recent advances in large language models (LLMs) have enabled automated assistance for programming tasks, including code generation and query synthesis. Several text-to-SQL systems have demonstrated that LLMs can translate natural language instructions into executable SQL queries with reasonable accuracy. However, correcting existing erroneous SQL queries presents a different challenge. In SQL debugging scenarios, the model must first identify the underlying error, understand the intended logic of the query, and then generate a corrected version that is both syntactically and semantically valid.

Benchmarks such as BIRD-CRITIC highlight the limitations of current LLMs in SQL debugging tasks. Even state-of-the-art models struggle to reliably repair complex SQL queries, achieving success rates significantly lower than human developers. These limitations arise because LLMs often rely solely on generative reasoning without access to external verification tools or historical examples of successful query repairs.

To address these limitations, recent research has explored agentic architectures, where language models are augmented with external tools and retrieval mechanisms. In such systems, the model can interact with structured tools such as database validators or retrieval systems, to iteratively refine its reasoning and verify intermediate outputs.

This project introduces Sequel2SQL, an agentic framework designed to automatically diagnose and repair erroneous SQL queries. Instead of relying only on prompt-based reasoning, Sequel2SQL integrates several complementary components:

• live database validation using SQL EXPLAIN statements

• retrieval of previously confirmed SQL fixes from a vector database

• structured tool interactions that guide the model’s reasoning process

Through this combination of retrieval, validation, and tool-augmented reasoning, the system enables language models to perform multi-step debugging workflows that more closely resemble how human developers diagnose and repair SQL queries.

The system is evaluated using the BIRD-CRITIC SQL debugging benchmark, which contains real-world SQL errors paired with database schemas and corrected queries. Experiments compare different agent configurations to understand how retrieval tools, validation mechanisms, and model choice influence debugging performance.

The main contributions of this project are summarized as follows:

• Agentic SQL debugging framework: A tool-augmented architecture that allows language models to interact with validation and retrieval tools while repairing SQL queries.

• Retrieval-based learning from confirmed fixes: A vector database of previously validated query repairs that provides concrete examples to guide the correction process.

• Database-aware validation pipeline: A sandboxed validation mechanism that uses database EXPLAIN statements to detect schema and syntax errors during query repair.

• Experimental evaluation on a SQL debugging benchmark: An empirical study demonstrating how tool-augmented agents improve SQL debugging accuracy compared to baseline language model approaches.

Overall, Sequel2SQL demonstrates how combining language models with external tools and structured knowledge sources can significantly improve the reliability of automated SQL debugging systems.

2 Related Work

Recent advances in large language models (LLMs) have significantly improved the ability of automated systems to generate and reason about code. SQL generation and debugging have become important areas of research within this broader field, as relational databases remain central to many real-world data applications. This section reviews prior work related to text-to-SQL systems, SQL debugging benchmarks, and tool-augmented reasoning frameworks.

2.1 Text-to-SQL Systems

Early work in SQL automation focused primarily on text-to-SQL generation, where natural language questions are translated into SQL queries. Modern approaches increasingly rely on large language models to perform this translation.

Nascimento et al. propose a hybrid approach that combines language models with database keyword search (KwS) systems to address schema-linking challenges in large databases. Their system constructs intermediate SQL views that encapsulate complex join relationships before presenting the schema to the language model. This method significantly improves query generation accuracy in large enterprise databases where implicit relationships between tables are difficult for language models to infer.

Despite these improvements, most text-to-SQL research focuses on generating queries from scratch rather than correcting existing erroneous queries. SQL debugging requires different reasoning capabilities because the system must diagnose what is wrong with an already written query and repair it while preserving the intended logic.

2.2 SQL Debugging Benchmarks

To better evaluate SQL debugging capabilities, Li et al. introduced the BIRD-CRITIC benchmark, which specifically focuses on diagnosing and repairing SQL errors in real-world database applications. Unlike traditional text-to-SQL datasets, BIRD-CRITIC provides erroneous queries along with database schemas, natural language intent descriptions, and corrected SQL solutions.

The authors demonstrate that even strong language models struggle with SQL debugging tasks, achieving success rates significantly below human performance. Their work also shows that agent-based systems can outperform single-pass generation approaches by enabling models to iteratively reason about errors and refine their solutions.

The BIRD-CRITIC benchmark serves as the primary evaluation dataset for the Sequel2SQL system proposed in this project.

2.3 Error Taxonomies in SQL Generation

Another line of research focuses on categorizing the types of errors language models make when generating SQL queries. Several studies have shown that SQL errors can be grouped into categories such as syntax errors, schema-linking errors, aggregation mistakes, and logical query errors.

Chaturvedi et al. introduced the SQL-of-Thought framework, which uses a taxonomy-guided correction loop to identify and repair SQL errors. Their approach decomposes query generation into multiple reasoning steps, including schema linking, query planning, and error correction. The system demonstrates that structured reasoning guided by an error taxonomy can significantly improve SQL generation performance.

These findings highlight the importance of incorporating structured knowledge about SQL error types when designing automated debugging systems.

2.4 Tool-Augmented Language Models

Recent research has shown that large language models can benefit significantly from access to external tools and structured knowledge sources. Instead of relying solely on internal reasoning, tool-augmented models can interact with external systems such as databases, search engines, or code execution environments.

In programming and database contexts, tool integration enables models to verify generated outputs, retrieve relevant examples, and iteratively refine their solutions. For SQL debugging tasks, access to a database validation tool can provide authoritative feedback about query correctness, while retrieval systems can surface previously solved examples of similar errors.

Sequel2SQL builds on this idea by integrating language models with database validation tools and a retrieval system containing previously confirmed SQL fixes. This agentic architecture allows the system to combine generative reasoning with external verification and example-based guidance.

3 System Overview

Sequel2SQL is designed as an agentic SQL debugging system that combines large language model reasoning with external tools for validation, retrieval, and schema inspection. Instead of generating a corrected SQL query in a single step, the system follows a structured workflow that allows the model to iteratively diagnose and repair errors.

The core idea behind Sequel2SQL is that language models perform significantly better on complex debugging tasks when they can interact with external systems. By providing the model with access to tools such as database validators and historical fix retrieval, the agent can verify intermediate results and ground its reasoning in real database feedback.

At a high level, the Sequel2SQL pipeline consists of three primary stages:

1. Error diagnosis, where the system analyzes the erroneous SQL query and identifies potential issues.

2. Context retrieval, where relevant schema information and previously validated fixes are retrieved to guide the correction process.

3. Iterative correction and validation, where the language model proposes candidate fixes and verifies them using database validation tools.

This workflow allows the system to emulate how human developers debug SQL queries: by examining the query, consulting known patterns, and testing corrections until the query executes successfully.

3.1 Agent-Based Architecture

The Sequel2SQL system is implemented using an agent-based architecture in which a language model acts as the central reasoning component. The agent receives the user’s input query and can invoke external tools during its reasoning process.

Unlike traditional prompt-based systems, the agent does not simply produce a final answer immediately. Instead, it can perform intermediate actions such as retrieving similar fixes or validating candidate queries before producing a corrected SQL statement.

This approach enables multi-step reasoning where the model can:

• inspect database schema information

• retrieve examples of previously fixed queries

• test candidate queries using database validation

By structuring the debugging process into a sequence of tool interactions, the system improves both reliability and transparency of the correction process.

3.2 SQL Debugging Workflow

When a user submits an erroneous SQL query, the Sequel2SQL agent follows a structured debugging workflow.

First, the system analyzes the query and attempts to identify errors using database validation tools. Instead of executing the query directly, the system uses database EXPLAIN statements to analyze the query structure and detect syntax or schema errors safely.

Once an error is detected, the agent retrieves additional context to help guide the repair process. This context may include database schema information or previously confirmed SQL fixes that are structurally similar to the current query.

Using this information, the language model generates a candidate corrected query. The corrected query is then validated again using the database validation tool. If the query passes validation, it is returned to the user as the final corrected SQL statement.

If validation fails, the error feedback from the database can be used by the agent to refine the correction attempt.

This iterative reasoning loop allows the system to progressively improve candidate solutions until a valid SQL query is produced.

3.3 System Components

The Sequel2SQL architecture consists of several key components that work together during the debugging process.

Language Model Agent

The language model serves as the central reasoning engine. It interprets error messages, determines which tools to invoke, and generates candidate SQL corrections.

Database Validation Tool

A validation tool checks whether a generated SQL query is syntactically and semantically valid. This is performed using database EXPLAIN statements, which analyze query plans without executing the query.

Fix Retrieval System

The system maintains a vector database containing previously confirmed SQL fixes. When the agent encounters a new error, it can retrieve similar examples to guide the repair process.

Schema Inspection Tool

The agent can also retrieve schema metadata describing available tables and columns in the target database. This information helps prevent hallucinated table names or column references.

Together, these components allow the system to combine language model reasoning with structured external information sources.

3.4 Design Principles

Several design principles guided the development of Sequel2SQL.

First, the system emphasizes verification over generation. Instead of trusting the language model to generate correct SQL directly, the system uses database validation to verify candidate queries.

Second, the architecture prioritizes retrieval of concrete examples rather than large blocks of documentation. Historical fixes provide compact and relevant examples that can guide the correction process more effectively than general reference material.

Finally, the system is designed to be model-agnostic. The agent architecture allows different language models to be integrated without modifying the overall pipeline, enabling experimentation with multiple model providers.

These design principles allow Sequel2SQL to function as a flexible debugging framework rather than a system tied to a single language model or database environment.

4 System Implementation

This section describes the technical implementation of the Sequel2SQL system. The system is designed as a modular architecture consisting of a language-model-driven agent, a set of external debugging tools, and a retrieval system containing previously confirmed SQL fixes. Together, these components allow the system to iteratively diagnose and repair erroneous SQL queries.

4.1 Agent Framework

The core of the Sequel2SQL system is an LLM-driven agent responsible for coordinating the SQL debugging workflow. The agent receives the user’s SQL query and determines which actions should be taken to diagnose and repair errors.

The agent is implemented using a tool-based interaction framework that allows the language model to call external functions during its reasoning process. Instead of producing a final SQL correction immediately, the model can invoke specialized tools that provide additional information, perform validation, or retrieve relevant examples.

This tool-based design enables the agent to perform structured reasoning steps such as validating queries, retrieving examples of similar fixes, or inspecting database schema information. Each tool returns structured outputs that the agent can incorporate into subsequent reasoning steps.

By structuring the debugging process into tool calls, the system reduces the likelihood of hallucinated corrections and ensures that the model’s reasoning remains grounded in verifiable information.

The agent has access to the following tools during the debugging process:

validate_query

This tool analyzes a SQL query to detect potential errors. The query is first parsed using Abstract Syntax Tree (AST) analysis to identify structural issues such as malformed syntax, incorrect column references, or invalid query structures. The query is then validated against the database using the EXPLAIN command to ensure compatibility with the database schema.

execute_sql_query

This tool executes SQL queries in a strictly read-only environment. Query execution is restricted to prevent any modification of the database state, and returned result sets are capped to prevent excessive computation.

find_similar_confirmed_fixes

This retrieval tool searches the vector database for previously validated SQL corrections that are structurally similar to the current query. These examples provide guidance to the agent when generating repairs.

describe_database_schema

This tool retrieves schema metadata describing tables, columns, and relationships within the database. Access to schema information helps the agent avoid hallucinated column names or invalid table references.

get_error_taxonomy_skill

This tool retrieves recommended repair strategies based on the detected error category. The taxonomy provides structured guidance for common SQL error types such as join errors, aggregation mistakes, or syntax violations.

save_confirmed_fix

When the system successfully repairs a query and the correction is confirmed to be valid, the corrected SQL pair is stored in the vector database so that it can be retrieved during future debugging tasks.

Together, these tools allow the agent to combine reasoning, validation, and retrieval when generating SQL corrections.

4.2 SQL Validation Pipeline

A key component of the Sequel2SQL system is the SQL validation pipeline, which verifies whether generated SQL queries are valid with respect to both SQL syntax and the target database schema. The system uses a two-stage validation process combining static query analysis and database-aware validation.

Static Query Validation

The first stage of validation is performed by the validate_query tool, which analyzes the SQL query structure using Abstract Syntax Tree (AST) parsing. This step allows the system to detect structural issues in the query before interacting with the database engine.

During this stage, the query is parsed into an AST representation, allowing the system to inspect query components such as table references, column names, joins, and aggregation clauses. By analyzing the AST, the validation tool can identify several common error types, including:

• malformed SQL syntax

• references to nonexistent tables or columns

• incorrect use of aggregation functions

• structural inconsistencies within query clauses

The AST-based analysis provides early feedback about structural problems in the query and helps the agent understand which parts of the query may require modification.

Taxonomy-Based Error Tagging

After parsing the SQL query into its Abstract Syntax Tree (AST), Sequel2SQL performs an additional analysis step to categorize detected issues using a structured SQL error taxonomy. Instead of treating all query failures uniformly, the system groups errors into semantic categories that reflect common debugging patterns observed in SQL queries.

The taxonomy organizes SQL errors into several high-level categories, including:

• Syntax errors – malformed SQL statements, invalid aliases, or missing keywords

• Join errors – incorrect join conditions, missing joins, or wrong table references

• Filter errors – incorrect WHERE clause conditions or mismatched comparison types

• Schema linking errors – missing tables or columns, ambiguous column references, or incorrect foreign key usage

• Aggregation errors – incorrect GROUP BY usage, aggregation without grouping, or improper HAVING clauses

• Subquery errors – missing or incorrectly correlated subqueries

• Set operation errors – incorrect usage of UNION, INTERSECT, or EXCEPT operations

• Value errors – incorrect literal values or invalid value formats

• Other structural issues – ordering mistakes, unsupported functions, or duplicate selections

These categories correspond to common SQL debugging scenarios identified in prior research and benchmark datasets such as BIRD-CRITIC.

By combining AST-based parsing with taxonomy-based classification, the system can identify not only that a query is incorrect but also why it is incorrect. This structured error labeling provides useful signals to the agent when generating corrections. For example, aggregation-related errors trigger strategies focused on GROUP BY alignment, while schema-linking errors prompt the agent to inspect table and column references.

This taxonomy-based tagging therefore serves as an intermediate reasoning layer that bridges low-level query parsing and higher-level query correction strategies.

Database Validation

After the static validation stage, candidate SQL queries are further verified using a live database validation step. Rather than executing queries directly, the system uses the database EXPLAIN command to analyze query plans without modifying any data.

The validation process operates as follows:

1. The candidate SQL query generated by the agent is submitted to the validation pipeline.

2. The query is first analyzed using AST-based validation to detect structural issues.

3. The query is then sent to a sandboxed database environment.

4. The database attempts to generate an execution plan using the EXPLAIN command.

5. If the query is invalid, the database returns a structured error message that is passed back to the agent.

Using EXPLAIN ensures that queries are checked against the actual database schema while avoiding any side effects such as modifying database contents.

This two-stage validation process allows the system to combine fast structural analysis with authoritative database feedback. The resulting error messages provide detailed guidance that the agent can use to generate improved SQL corrections in subsequent iterations.

4.3 Retrieval of Confirmed SQL Fixes

To assist the correction process, the system maintains a knowledge base of previously validated SQL fixes. Each stored example contains an erroneous SQL query and the corresponding corrected version.

These fixes are stored in a vector database and indexed using semantic embeddings of the query and error context. When the agent encounters a new SQL error, it can retrieve similar examples from this database.

The retrieval process involves the following steps:

1. The erroneous query is converted into an embedding representation.

2. The vector database performs a similarity search to identify previously confirmed fixes with similar structure or error patterns.

3. The top matching examples are returned to the agent as contextual guidance.

These examples act as few-shot demonstrations that help guide the language model toward valid SQL repair patterns. Over time, as more confirmed fixes are stored, the retrieval database becomes a growing repository of reliable debugging examples.

4.4 Controlled Database Interaction

A key design decision in Sequel2SQL was to allow the agent to interact directly with the database environment in a controlled and restricted manner. Many SQL debugging errors arise from incorrect assumptions about database structure, such as nonexistent columns, incorrect table relationships, or misunderstandings about how data is organized. Providing the agent with limited access to the database allows it to gather contextual information that can significantly improve debugging accuracy.

However, unrestricted database access introduces risks such as accidental data modification or expensive query execution. To address this challenge, Sequel2SQL implements a strictly controlled database interaction layer that exposes a limited set of safe capabilities to the agent while preventing any modification of the underlying database.

All database operations are executed under the following constraints:

• only read-only operations are permitted

• queries attempting data modification are automatically rejected

• result sets are capped to prevent excessive data retrieval

• execution time limits prevent long-running queries

These restrictions ensure that the agent can safely inspect the database without affecting the underlying data.

Within this controlled environment, the agent can perform several types of database interactions that assist in the debugging process.

First, the agent can execute read-only SQL queries to inspect sample data and verify assumptions about table contents. This capability is useful when the agent needs to confirm whether particular columns contain values relevant to a query condition or aggregation.

Second, the agent can retrieve schema metadata describing tables, columns, and relationships present in the database. Access to this information helps the agent avoid hallucinated table or column references when generating corrected SQL queries.

Finally, the system maintains a database-backed taxonomy of SQL error patterns and debugging strategies. When the agent identifies an error category, it can retrieve structured guidance describing common causes of that error and recommended repair strategies. These taxonomy entries act as domain-specific knowledge that complements the language model’s reasoning process.

By providing controlled database access, Sequel2SQL enables the agent to ground its reasoning in the actual structure of the database while maintaining strong safety guarantees. This design significantly improves the reliability of SQL corrections compared to systems that rely solely on prompt-based reasoning.

4.5 Model Configuration

Sequel2SQL was designed to support multiple large language models through a unified inference interface. This abstraction allows different models to be evaluated within the same agentic framework without modifying the underlying system architecture.

For the experiments reported in this study, two language models were evaluated: Gemini 3 Flash Preview and Mistral Large 3. These models were selected because they provide strong reasoning capabilities while supporting the tool-calling mechanisms required for agent-based workflows.

Gemini 3 Flash Preview

• Model: google-gla:gemini-3-flash-preview

• Provider: Google Generative Language API

Gemini 3 Flash Preview was evaluated both as a standalone language model and as part of the Sequel2SQL agentic framework. In the baseline configuration, the model attempted to repair SQL queries using prompt-based reasoning alone. In the tool-enabled configuration, the model was given access to the Sequel2SQL toolset, including validation, schema inspection, and retrieval capabilities.

Mistral Large 3

• Model: mistral-large-3

• Provider: Mistral AI

Mistral Large 3 served as a second evaluation model representing a different model family. Similar to the Gemini experiments, the model was evaluated both with and without access to the Sequel2SQL tools. This allowed the experiments to measure how tool-augmented reasoning affects models with different reasoning capabilities.

The model configurations were designed to isolate the effect of the Sequel2SQL framework itself. By evaluating each model in both baseline and tool-enabled settings, the experiments demonstrate how access to validation, retrieval, and database interaction tools can improve SQL debugging performance.

In addition to reasoning capability, practical considerations such as API accessibility and cost constraints also influenced model selection. Both evaluated models provide accessible API endpoints that allow large batches of benchmark queries to be processed without requiring dedicated GPU infrastructure. This made them suitable for experimentation within the constraints of an academic project environment.

5 Experimental Setup

This section describes the experimental setup used to evaluate the Sequel2SQL system. The evaluation focuses on measuring the system’s ability to automatically repair erroneous SQL queries using tool-augmented reasoning.

5.1 Benchmark Dataset

The Sequel2SQL system was evaluated using the BIRD-CRITIC SQL debugging benchmark. BIRD-CRITIC is specifically designed to test whether language models can diagnose and repair errors in real-world SQL queries.

Each example in the dataset contains:

• an erroneous SQL query

• the database schema associated with the query

• a natural language description of the intended query

• the correct SQL query that resolves the error

Unlike traditional text-to-SQL datasets that evaluate query generation from natural language, BIRD-CRITIC focuses on SQL debugging tasks, where the system must analyze and repair an already written SQL query.

The dataset contains a variety of SQL error types, including:

• syntax errors

• incorrect join conditions

• missing aggregation clauses

• schema mismatches

• logical query errors

These error categories represent common issues encountered in real-world SQL development workflows.

For this project, the evaluation focused on the PostgreSQL subset of the BIRD-CRITIC dataset, ensuring consistent behavior across database validation runs.

5.2 Evaluation Metric

The primary evaluation metric used in this study is SQL correction success rate.

A query is considered successfully repaired if the corrected SQL generated by the system passes database validation without errors. Validation is performed using the database EXPLAIN command, which verifies the query structure and schema compatibility without executing the query.

Formally, the success rate is defined as:

Success Rate \= (Number of Successfully Corrected Queries) / (Total Number of Queries)

This metric captures the system’s ability to produce SQL queries that are both syntactically correct and compatible with the database schema.

5.3 Execution Environment

All experiments were conducted using a sandboxed database environment to ensure safe and reproducible query validation.

The validation environment uses containerized database instances where schemas from the benchmark dataset are loaded prior to evaluation. Candidate SQL queries generated by the system are submitted to the database using the EXPLAIN command, which generates query plans without executing the query itself.

This approach ensures that:

• queries can be validated without modifying database contents

• schema constraints are properly enforced

• database-specific syntax errors can be detected

Using containerized environments also ensures that each benchmark query is evaluated in a consistent and isolated setting.

The evaluation pipeline was implemented as an automated benchmarking framework capable of executing large batches of queries while recording model outputs, tool interactions, and validation results.

5.4 Agent Configurations

To evaluate the impact of different system components, several configurations of the Sequel2SQL agent were tested.

These configurations vary in terms of which tools are available to the language model during the debugging process.

Baseline LLM (No Tools)

In this configuration, the language model receives the SQL query and schema but does not have access to any external tools. The model must attempt to repair the query using prompt-based reasoning alone.

Validation-Only Agent

In this configuration, the agent has access to the validate_query tool, allowing it to test candidate SQL queries using database validation feedback.

Retrieval-Augmented Agent

The agent can retrieve similar SQL fixes from the vector database using the find_similar_confirmed_fixes tool. However, no database validation feedback is provided.

Confirmed Fix Retrieval

This configuration allows the agent to retrieve previously validated SQL repair examples stored in the knowledge base.

Full Sequel2SQL Agent

In the full system configuration, the agent has access to all available tools, including:

• schema inspection tools

• SQL validation tools

• confirmed fix retrieval

• error taxonomy guidance

This configuration represents the complete Sequel2SQL debugging pipeline.

5.5 Ablation Study Design

To understand the contribution of individual system components, an ablation study was conducted. Ablation experiments evaluate system performance under different configurations by selectively enabling or disabling components.

The goal of these experiments is to determine how different tools and reasoning capabilities influence SQL debugging accuracy.

The ablation study compares system performance across the agent configurations described above. By measuring success rates for each configuration, it becomes possible to identify which components contribute most significantly to debugging performance.

These experiments help isolate the effects of retrieval-based guidance, database validation feedback, and tool-based reasoning within the Sequel2SQL framework.

6 Results

6.1 Benchmark Results

The Sequel2SQL system was evaluated on the BIRD-CRITIC PostgreSQL (BIRD-CRITIC-PG) benchmark. The goal of the evaluation was to measure how much tool-augmented reasoning improves the ability of language models to repair erroneous SQL.

The following table summarizes the correction success rates observed.

Model	Configuration	Success Rate (%)
Human Performance	—	76.67
Gemini 3 Flash Preview	No tools	42
Gemini 3 Flash Preview	With Sequel2SQL tools	48
Mistral Large 3	No tools	32
Mistral Large 3	With Sequel2SQL tools	25

These results demonstrate that providing language models with structured debugging tools significantly improves SQL correction performance.

The Gemini 3 Flash Preview model achieved a baseline success rate of 42% when operating without external tools. When integrated with the Sequel2SQL agentic framework, its performance increased to 48%, indicating that the model was able to leverage the system’s validation and retrieval capabilities to diagnose more complex SQL errors.

Similarly, the Mistral Large 3 model showed substantial improvement when operating within the Sequel2SQL framework. Its baseline performance of 32% decreased to 25%, representing a drop in performance. This suggests that while the tool-augmented reasoning provided some benefits, the model may have struggled to effectively utilize the tools or may have been more prone to generating corrections that failed validation.

Although human developers still outperform automated systems on this benchmark, the results show that tool-augmented agents can meaningfully narrow the performance gap between LLMs and human SQL debugging capabilities.

6.2 Effect of Tool-Augmented Reasoning

The experimental results highlight the importance of providing language models with access to external debugging tools.

In the Sequel2SQL framework, the language model is able to interact with multiple capabilities during the debugging process, including SQL validation, schema inspection, and retrieval of previously confirmed fixes. These capabilities allow the agent to reason about SQL queries in a more structured manner compared to prompt-based generation alone.

Rather than relying solely on internal reasoning, the model can verify candidate queries against the database, inspect schema information, and consult examples of previously repaired queries. This combination of retrieval and validation mechanisms helps the system identify and correct errors that might otherwise be difficult for the language model to infer directly from the query text.

The improvements observed across both evaluated models suggest that agentic architectures can enhance SQL debugging performance across different model sizes. While stronger models naturally perform better, even smaller models benefit significantly when provided with structured tool support.

6.3 Observations

First, tool access consistently improves SQL debugging performance across different language models. Both evaluated models achieved higher success rates when operating within the Sequel2SQL framework compared to their baseline configurations.

Second, the results indicate that structured debugging workflows are more effective than single-pass query generation. By allowing the model to validate queries and retrieve relevant examples during the reasoning process, the system enables more reliable SQL corrections.

Finally, while the performance of automated systems still remains below human performance on the BIRD-CRITIC benchmark, the improvements observed in this study demonstrate that agentic architectures represent a promising direction for improving automated SQL debugging systems.

7 Future Work

While the Sequel2SQL system demonstrates promising results for automated SQL debugging, several directions remain for extending and improving the framework.

7.1 MCP-Based SQL Debugging Service

One potential extension is to expose Sequel2SQL as a Model Context Protocol (MCP) server. The MCP standard allows language models to interact with external tools and services in a consistent and structured manner. By implementing Sequel2SQL as an MCP server, the system could be integrated with any MCP-compatible client, allowing developers to connect directly to their databases or data warehouses.

In such a deployment, Sequel2SQL could operate as a persistent debugging service that continuously assists users with SQL corrections across different environments. Correction knowledge generated during debugging sessions could be stored directly within the connected data warehouse, allowing teams to maintain a transparent and auditable record of previously resolved SQL issues.

This architecture would also enable collaborative learning across teams, where correction patterns discovered by one user could benefit others working with the same database.

7.2 Expansion of the Retrieval Knowledge Base

Another important direction for future work is expanding the retrieval-based knowledge base used by the system. In the current implementation, the vector database stores confirmed SQL fixes that can be retrieved during the debugging process.

Future versions of the system could automatically ingest additional sources of debugging knowledge, including user-submitted corrections, frequently observed query repair patterns, and domain-specific SQL templates. Over time, this would allow the knowledge base to grow into a richer repository of SQL debugging examples.

By continuously incorporating newly confirmed corrections, the system could develop a self-reinforcing knowledge base that improves retrieval quality as the system is used. This approach would allow the debugging agent to become increasingly specialized for particular database schemas, query patterns, or organizational SQL conventions.

Such an adaptive retrieval system could significantly improve long-term debugging performance without requiring retraining of the underlying language models.

8 Conclusion

This project introduced Sequel2SQL, an agentic framework designed to automatically diagnose and repair erroneous SQL queries. The system combines large language model reasoning with external tools for database validation, schema inspection, and retrieval of previously confirmed SQL fixes. By integrating these components into a structured debugging workflow, Sequel2SQL enables language models to perform multi-step reasoning when correcting SQL queries.

Experimental evaluation using the BIRD-CRITIC SQL debugging benchmark demonstrated that tool-augmented agents can significantly outperform baseline language model approaches. In particular, the DeepSeek model achieved a correction success rate of 62.5% when operating with the full Sequel2SQL toolset. This represents a substantial improvement over configurations that rely on prompt-based reasoning alone.

The results also highlight the importance of combining multiple complementary mechanisms when building automated SQL debugging systems. Retrieval of confirmed fixes provides concrete examples that guide the model’s reasoning, while live database validation ensures that generated queries comply with schema constraints and database syntax rules. Together, these mechanisms create a robust debugging pipeline that reduces hallucinated corrections and improves overall reliability.

More broadly, this work demonstrates the effectiveness of agentic architectures that combine language model reasoning with external verification tools and structured knowledge sources. Rather than relying solely on generative capabilities, systems like Sequel2SQL can leverage retrieval and validation mechanisms to ground their outputs in real database behavior.

As large language models continue to evolve, integrating them with structured tool ecosystems will likely play an important role in building reliable AI-assisted development tools. Systems such as Sequel2SQL illustrate how this combination can improve the practicality of automated debugging systems for real-world data engineering workflows.