How to measure synthetic data: the metrics that matter

"Synthetic" is not a guarantee. A generator can produce data that looks realistic but is useless for modeling, or data that models beautifully but quietly memorizes real people. The only way to know which you have is to measure both sides — utility and privacy — on every dataset. Here's how.

Utility: is it still worth training on?

Fidelity (does it match the statistics?). Start simple: compare per-column distributions, means, and cardinalities between real and synthetic. Then check the relationships — correlation matrices, pairwise dependencies, and the joint distributions that actually carry signal. Good synthetic data matches not just each column but how columns move together.

Train-synthetic, test-real (TSTR). The gold standard. Train your model on synthetic data, evaluate it on held-out real data, and compare to a model trained on real data. If the synthetic-trained model comes within a few points, the data carries the signal that matters. TSTR cuts through pretty charts and answers the only question that counts: will a model built on this work in production?

Downstream task performance. Beyond a single model, check the specific tasks you care about — the fraud classifier's recall, the forecast's error, the segment counts your analysts rely on.

Privacy: does anyone real leak through?

Distance to closest record (DCR). For each synthetic row, how close is the nearest real row? If synthetic records are suspiciously close to (or identical to) real ones, the generator is copying. A healthy DCR distribution looks like the distance between two independent real samples — not smaller.

Membership inference. Can an attacker tell whether a specific person was in the training data by looking at the synthetic output? Run the attack yourself: if a membership-inference model does no better than a coin flip, you're in good shape. If it succeeds, the generator is leaking.

Attribute disclosure. Given some columns, can an adversary predict a sensitive one better from your synthetic data than from public priors? That gap is your disclosure risk.

The tradeoff is real — so make it explicit

Utility and privacy pull against each other. A generator tuned for maximum fidelity leaks more; one tuned for maximum privacy blurs signal. The goal isn't to max one metric — it's to land in a defensible spot and document where you landed. A one-line "it's synthetic, so it's fine" is how privacy incidents happen. A short report with TSTR and DCR numbers is how audits pass.

Make it a gate, not an afterthought

The teams that trust synthetic data are the ones who never ship a dataset without its scorecard. Every generation run should produce a utility-and-privacy report automatically, the same way every code change runs tests. That's the difference between hoping and knowing — and it's exactly what the Syntheticore platform does on every run.