Practical EDA for Synthetic Datasets

English עברית

Run this before training any model on a synthetic dataset. Each objective answers a different question about dataset quality, from whether the data is technically correct to whether it will tell weak and strong models apart. Pairs with the Synthetic Data Generation guide.

Eight objectives to check before training

Each card gives what the objective verifies and the practical EDA that checks it.

1

Sanity

Verify the dataset is technically correct.

Check for missing values, duplicates, invalid records, the label distribution, the schema, and impossible values, and read a random sample of records by hand.

2

Coverage

Ensure every intended scenario is represented.

Count samples per class and per attribute, verify that every planned scenario and edge case exists, and inspect cross-tabulations of the important variables.

3

Realism

Confirm the synthetic data resembles real-world data.

Compare feature distributions with real data where available, inspect random samples, visualize embeddings or clusters, and ask domain experts to review examples.

4

Variability (diversity)

Prevent repetitive or mode-collapsed data.

Detect duplicates and near-duplicates, cluster embeddings, measure the diversity of vocabulary, features, or images, and verify that multiple styles and conditions are present.

5

Label quality

Ensure labels are accurate and consistent.

Manually review random samples, validate labels with rules or a second model, inspect confusing cases, and check annotation consistency.

6

Leakage

Prevent models from learning shortcuts.

Search for duplicate train and test samples, inspect metadata and filenames, identify features highly correlated with the label, and verify the label cannot be inferred from unintended information.

7

Bias and balance

Detect skewed or unfair distributions.

Plot class frequencies, compare demographic or subgroup distributions, and verify that important attributes are not over-represented or under-represented.

8

Difficulty

Ensure the benchmark meaningfully evaluates models.

Inspect easy versus hard examples, verify the level of ambiguity, include challenging edge cases, and compare a baseline against a stronger model to confirm there is room for improvement.

Rule of thumb

Before training, ask the eight questions in one line each.

  • SanityIs the data correct?
  • CoverageIs everything important included?
  • RealismDoes it look real?
  • VariabilityIs it diverse enough?
  • Label qualityCan the labels be trusted?
  • LeakageCan the model cheat?
  • Bias and balanceIs the dataset representative?
  • DifficultyWill this dataset tell weak and strong models apart?

Quick Links