thealper2/codet5p-sql2text

SQL-to-Text (Salesforce/codet5p-220m)

Salesforce/codet5p-220m fine-tuned to explain a SQL query in plain English.

The direction is SQL -> natural language: the model takes a query (and, optionally, the DDL of the tables it touches) and returns a sentence describing what that query does. It does not generate SQL from a question.

Prompt format

Inputs follow one fixed template; training, evaluation and inference all build it with the same function, so they cannot drift apart. The schema block is dropped when no DDL is supplied, and when it is supplied only CREATE TABLE ... statements are kept.

Explain the following SQL query.

Schema:
<CREATE TABLE statements>

SQL:
<the query>

Usage

from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

model_id = "thealper2/codet5p-sql2text"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSeq2SeqLM.from_pretrained(model_id)

schema = "CREATE TABLE employees (id INT, name TEXT, salary INT, dept_id INT);"
sql = "SELECT dept_id, AVG(salary) FROM employees GROUP BY dept_id;"
prompt = f"Explain the following SQL query.\n\nSchema:\n{schema}\n\nSQL:\n{sql}"

inputs = tokenizer(
    prompt,
    return_tensors="pt",
    truncation=True,
    max_length=256,
)
outputs = model.generate(
    **inputs,
    num_beams=4,
    max_new_tokens=128,
    min_new_tokens=5,
    early_stopping=True,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Training data

gretelai/synthetic_text_to_sql, mapping sql + sql_context to sql_explanation.

Preprocessing drops rows that are too short to be a real explanation, removes exact duplicates and duplicate inputs, and removes any training row whose input also appears in the official test split, so the reported test scores are not inflated by leakage. The validation split is 3% of the cleaned train split (seed 42).

Sequence lengths were chosen from the measured token-length distribution: source 256 tokens, target 128 tokens.

Training procedure

Hyper-parameter	Value
Epochs	3.00
Learning rate	0.0001
LR schedule	linear
Warmup ratio	0.0500
Weight decay	0.0100
Optimiser	adamw_torch
Per-device train batch size	16
Gradient accumulation	4
Max gradient norm	1.00
Model selection	eval_rougeL
Seed	42
Effective batch size	64

Trained on a single NVIDIA GeForce RTX 5060 Ti (15.9 GB) with torch 2.11.0+cu128, bf16 mixed precision.

Wall-clock training time: 84 minutes.

Evaluation

Scored by evaluate.py on the full splits with beam search (num_beams=4).

Generation quality

Metric	Validation	Test	_meta
Examples	2989	5850	-
BLEU	33.65	33.10	-
ROUGE-1	67.15	66.86	-
ROUGE-2	45.76	45.11	-
ROUGE-L	57.58	57.06	-
Mean generated length	36.37	35.69	-
Loss	0.5829	0.5923	-

SQL-aware faithfulness

Recall metrics ask whether the explanation mentions what the query actually does; the rate metrics are error rates, where lower is better -- they measure claims the query does not support.

Metric	Validation	Test
Examples	2989	5850
Operation recall	98.53	98.40
Aggregation recall	98.72	98.57
Join mention recall	98.21	98.74
Join table coverage	98.34	98.56
Condition column coverage	82.98	81.99
Condition value coverage	86.18	87.24
Operation over-claim rate	5.47	5.29
Unsupported number rate	3.98	3.18
Unsupported quoted-string rate	3.20	3.18
Unsupported entity rate	1.38	1.72

Limitations

• Trained on synthetic queries and synthetic explanations, so the phrasing reflects that generator's style rather than how a particular team documents its own queries.
• Explanations are grounded in the query text, not in the data: the model cannot know what a column means beyond its name.
• Condition coverage is the weakest area -- long WHERE clauses lose some columns and literals -- so an explanation may describe a filter less precisely than the query applies it. Do not rely on it as an audit of what a query returns.
• Inputs are truncated past the configured source length, so very large schemas are only partially visible to the model.
• English only.

Reproduction

make preprocess
make train
make evaluate

Base model: Salesforce/codet5p-220m.