← Research

Forty years of flattening

Where tabular prediction came from, and what changed recently.

1 Aug 2026 · History · 8 min

Tabular prediction has been a solved-enough problem for a decade, so it is worth asking why anything new would appear now. The answer is not a better model. It is a different assumption about the shape of the input.

flatten the schemakeep the schema1984CART2001Random forests · GBM2003Multi-relational data mining2015Deep feature synthesis2016XGBoost2017LightGBM · feature stores2020GNNs on databases2021Trees still win2022TabPFN2024RelBench2025Relational Transformer2026PluRel
Ordinally spaced, not to scale: the gap from 1984 to 2001 is seventeen years and the gap from 2024 to 2025 is one.

The tree era

XGBoost's own paper reports the number that made its reputation: of 29 winning Kaggle solutions published in 2015, 17 used XGBoost. The second most popular method, deep neural networks, appeared in 11. That result set the default for the decade that followed.

2015 · published Kaggle winning solutionsall published29used XGBoost17used deep nets11the number that set the default for the decade that followed
Of 29 winning Kaggle solutions published in 2015, 17 used XGBoost. The counts overlap — a solution could use both — so they are not stacked.

The flattening tax

By 2017 the flattening step had become infrastructure. Uber's Michelangelo post popularised the feature store — a central place to publish canonical features — and Feast and Tecton turned it into a category. The step nobody liked had acquired a budget, a team and a roadmap.

the schemacustomerordersticketspaymentsrows, timestamps, free text,order of events, who wrote whataggregateone row, engineeredn_orders40mean_basket18.20days_since7n_tickets3four numberschosen before the model saw anything
Forty tickets become the number 40. Everything the aggregate did not ask for leaves here.

Deep learning's false start

Deep tabular models arrived on schedule — TabNet in 2019, TabTransformer in 2020, SAINT in 2021 — and each claimed to beat the trees. Then two papers checked. Both found that gradient boosting still won, with far less tuning, on the challengers' own datasets.

2 Jun 2021SAINTclaims wins over XGBoost, CatBoost and LightGBM6 Jun 2021Deep Learning is Not All You Needre-runs them on their own datasets; XGBoost ahead, with less tuning22 Jun 2021FT-Transformerconcludes no method is universally superior18 Jul 2022Inria benchmarkasks why rather than whether: three inductive biasesnone of it has been overturned — on one table, tree ensembles are still the thing to beat
Four days between the claim and the rebuttal. The 2022 benchmark is the one that asked why rather than whether.

Learning without fitting

TabPFN in 2022 changed the question. Instead of fitting a model to your table, it was trained offline on millions of synthetic datasets and then read your table as context — no gradient steps, no tuning, under a second. The 2025 Nature version scaled it; TabICL pushed the context to hundreds of thousands of rows.

gradient boostingone engineered row, fittedtabular in-contextexample rows, nothing fittedthe schema is still whatever you flattened it into
Both units are rows of one flat table. Removing the fitting step did not change what the model was allowed to see.

Someone kept the joins

The other lineage was quieter and older. Multi-relational data mining framed the problem in 2003. In 2020 a paper made the obvious move explicitly — treat the primary-foreign-key structure as a graph and run a GNN over it — and positioned it against automated feature engineering. It beat DFS on two of three datasets, which is a real result honestly reported.

flatten itcustomerCOUNTone rowthe structure is spent on the way inkeep ittargetordersticketspaymentsthe structure is what the model readsmessage passing, then attention: two ways of doing the same refusal to flatten
The foreign keys are the wiring, not an input. Nothing is aggregated on the way in.

What actually changed

The Relational Transformer changed the unit. Not the row, not the table — the cell, tagged with its column, table, row and timestamp, with attention masked to follow foreign keys. There is no flattening step because there is nothing to flatten into.

before · the unit is a rown_ordersmean_basketdays_sincen_ticketsassembled by a person, before the model saw anythingafter · the unit is a cell42.50column: amounttable: ordersrow: #8814time: 03-09tagged, not aggregated —nothing is decided in advancethe models kept improving for forty years; this is the part that had never moved
Every era before it modelled a row that something else had to assemble. This one models the cell, and assembles nothing.

What this buys you

What disappears is a job. No feature table to design, no pipeline to maintain, no model to retrain per question — and no feature store standing between a question and its answer. A first result takes minutes because there is no training run in front of it, and the same checkpoint answers the next question without being asked to learn anything.

beforeschemafeature engineeringa training runa model per taskan answerrebuilt for every new question, and maintained by someonemonthsafterschemaPREDICT … FROM customersan answerminutesno feature table, no training run, no model to keepthe pretrained model reads the schema you already have
The steps that disappear are the ones that took the months. What is left is a question and an answer.

What this does not settle

The Relational Transformer is still relatively unoptimised, yet it can already beat XGBoost when the relationships carry signal. It does not win everywhere today, particularly on a single wide table, but that looks more like a starting point than a ceiling. If your data genuinely is a rectangle, forty years of work on rectangles is available to you.

What happens next

The architecture is the part to watch. LLMs keep getting better through changes to attention, retrieval, memory and efficient inference, with an enormous concentration of research and compute behind them. Relational Transformers can inherit that progress while evolving the parts specific to databases — their masks, samplers and objectives. The current model is a first point on that curve, not its final form.

settlednot yetbinary classificationregressionmulticlass · rankingzero-shot on unseen schemasfine-tuning on your own dataquantiles · prediction intervalscalibrated multiclass probabilitiesmulti-label without tuninga single wide tablethe last line is not a defect — it is the case where forty years of work on rectangles wins
Rejected at parse time rather than approximated: the right-hand column is refused honestly, not silently.

Sources