← Research

How the Relational Transformer predicts over a database

What replaces feature engineering, and what it costs.

1 Aug 2026 · Architecture · 7 min

Your data lives in a rich graph: a customer table, the orders pointing at it, the tickets those orders spawned, and the timestamps holding all of it in order. However, historically most approaches have predicted over a rectangular shape of data rather than retaining that original graph structure.

Architecture

When first looking at the Relational Transformer architecture, the striking thing is how ordinary it is: pre-norm RMSNorm, QK-norm on the queries and keys, a SwiGLU feed-forward, zero-initialised residual branches. All standard modern transformer hygiene. Similar to LLMs, it takes some context of tokens, applies attention, normalizes, and then outputs tokens.

context cells · ~8,192embed: col_name + typed valueone cell, one token — and no positionRelationalBlock× 12RMSNormattn · col+same columnRMSNormattn · feat+own row + parentsRMSNormattn · nbr+rows pointing backRMSNormFFN · SwiGLU+512 → 2048 → 512RMSNormdecoders · one per semantic typenumberLinear → 1textLinear → 384datetimeLinear → 1booleanLinear → 1loss is taken at the masked cells only
The published architecture. Dimensions are the pretraining defaults: 12 blocks, d_model 512, 8 heads of 64, d_ff 2048.

On top this, RelativeDB adds a task head to support multiclass and ranking over the backbone. Each example is encoded once and a [outputs × 512] layer is fitted on the results, about 2 KB.

published architecture ends hereRelativeDBtarget cell · 512task head · [outputs × 512] + bias≈ 2 KB, fitted on labelsbinary → 1finetuneregression → 1finetunemulticlass → Kfit_headranking → 1fit_head

The Data

RT-J was pretrained on The Join: 650 real-world relational databases spanning e-commerce, sports, media, finance, healthcare, government, and more. Together they supply roughly 6,000 forecasting tasks, plus autocomplete tasks over ordinary database rows. Every database keeps its tables, schema names, and primary-foreign-key graph, and any overlap with the seven RelBench evaluation databases was excluded.

The Join · 650 databasescommercesportsmediafinancehealthgovernmentpretraining signals≈ 6,000 forecasting tasksrare events · cold starts · heavy tailsautocomplete tasksmasked cells in ordinary database rows7 RelBench databasesheld out from pretraining
The pretraining corpus stays relational: tables, names, and foreign-key structure all survive the trip into the model.

Cell tokenisation

Each cell is tokenized with its column name, the type, and its data. Embedding the column name is what lets a model work with schemas it has never seen. For example, a column named sentiment can give sentiment analysis results despite only ever being trained on product review data.

enc_dict = {
    "number":   Linear(1,      d_model),
    "text":     Linear(d_text, d_model),   # 384, MiniLM
    "datetime": Linear(1,      d_model),
    "boolean":  Linear(1,      d_model),
    "col_name": Linear(d_text, d_model),
}
orderscustomer_idamountplaced_at4118.0003-024242.5003-09427.2503-11one tokenembed("amount")+Linear_number(42.50)no position — a database has no reading order

Masked relational attention

Each block runs three attentions in sequence, each with its own weights and mask: feat (own row and its parents), nbr (rows pointing back), col (the same column elsewhere). These masks turn the database structure into the attention pattern: foreign keys are not another feature, but the rules that decide which cells can exchange information. As blocks stack, evidence can travel beyond one row or one join without opening attention across the entire context.

featown row + parentscustomersordersitemsnbrrows pointing backcustomersordersitemscolsame column, same tablecustomersordersitems

Context construction

Relational Transformers work with just a few thousand cells rather than your whole dataset. Which cells is decided by a sampler. The target cell goes first. Then a breadth-first shell around it; then peers of the same table, found by random walks across the foreign keys; then random rows to top up. This is configurable from 256-8192 cells. Like LLMs, more context doesn't always mean more accuracy.

one context windowthe target cell, always first — and the only one maskedtier 0 · the target’s own neighbourhoodtier 1 · peers, answers visibletier 2 · random same-table rowsevery tier filtered to timestamps at or before the anchor
Roughly 8,192 cells, spent. The split between tiers is illustrative — it is sampled per item, not fixed.

Learning by hiding cells

Pretraining turns each database into its own supervision: hide known cells, reconstruct them from the visible relational context, and compare the predictions with the original values. Numbers, text, and dates use Huber loss; booleans use cross-entropy. At inference, an unknown target is presented in the same shape as a masked cell, its value is absent, while its schema and surrounding data remain visible.

pretraininghide a cell that exists18.0003-02?truerecover it · loss against the true valueidenticalforecastingaim at a cell that does not exist yet42.5003-09true?no task head · nothing retrainedanchorevery cell is a legitimate target, so a database with ten million cellsis ten million supervised examples nobody had to label
Same weights, same forward pass. Only the position of the mask changes.

The Future

PluRel's result is that synthetic databases can unlock scaling laws for Relational Transformers. Pretraining loss follows a power law in the number of synthetic databases. Those databases are generated, not acquired through buying anyone's private data, which is a usual toll on this road.

acquiredpartnerships, licences, lawyersbounded by what existsand who will sell itgeneratedsynthesised schemas, on demandbounded by computepretraining loss follows a power law in the number of synthetic databases
The scaling input is manufactured, so the path up the curve does not run through acquiring anyone's private data.

Small model, large database

RT-J scales by being small and selective. An 85-million-parameter model reads at most 8,192 cells per prediction, so the database behind that window can hold millions of rows without making every inference enormous. On held-out RelBench tasks, it matches or beats much larger in-context pipelines, including LLM agents that write SQL, while using far fewer labels.

Sources