EDA — Variation

PSY 410: Data Science for Psychology

Dr. Sara Weston

2026-04-29

What is EDA?

Know your data before you test it

Before you run a single statistical test, you should know your data well enough to predict what the results will look like.

That’s what EDA gives you — no surprises.

It’s also how you catch problems early: rows that didn’t load, codes that weren’t recoded, “999” sneaking in as a real age.

Exploratory vs. confirmatory

Exploratory (EDA) Confirmatory
Goal Discover patterns Test hypotheses
Attitude Curiosity Rigor
Questions Open-ended Pre-specified
Output Visualizations, hunches p-values, conclusions

The two roles get mixed up if you do them on the same data. Preregistration keeps them honest: you write your confirmatory hypotheses down before you peek.

A cleaner pattern: split your data

If you’re worried about exploring and testing on the same dataset, use multiple datasets:

  • Dataset 1 → EDA. Look around. Generate ideas.
  • Dataset 2 → Confirm ideas from Set 1. Then EDA on Set 2 → new ideas.
  • Dataset 3 → Confirm ideas from Sets 1 & 2. Then EDA on Set 3 → new ideas.
  • … and so on.

Each new dataset gets one shot at confirmation, then becomes fair game for exploration.

Good EDA means looking before you test

“EDA is an attitude, not a technique.” — John Tukey

  • Ask questions
  • Answer them by visualizing data
  • Use what you learn to ask new questions
  • Repeat

There is no single “right” way to do EDA. The goal is understanding.

What we’re doing today

Exploring variation — what does a single variable look like?

  • Distribution shape
  • Center and spread
  • Outliers and unusual values
  • Missing data

Tomorrow: covariation — how do variables relate to each other?

Our dataset: Big Five Personality

dim(bfi)
[1] 2800   28
names(bfi)
 [1] "A1"        "A2"        "A3"        "A4"        "A5"        "C1"       
 [7] "C2"        "C3"        "C4"        "C5"        "E1"        "E2"       
[13] "E3"        "E4"        "E5"        "N1"        "N2"        "N3"       
[19] "N4"        "N5"        "O1"        "O2"        "O3"        "O4"       
[25] "O5"        "gender"    "education" "age"      

25 items measuring five personality factors (Agreeableness, Conscientiousness, Extraversion, Neuroticism, Openness) + demographics.

What’s in here

Variable Description
A1–A5 Agreeableness items (1–6 scale)
C1–C5 Conscientiousness items (1–6 scale)
E1–E5 Extraversion items (1–6 scale)
N1–N5 Neuroticism items (1–6 scale)
O1–O5 Openness items (1–6 scale)
gender 1 = male, 2 = female (binary coding)
education 1–5 (HS incomplete through graduate)
age Age in years

Exploring distributions

Start simple: summary statistics

bfi |>
  select(age, gender, education) |>
  summary()
      age            gender        education   
 Min.   : 3.00   Min.   :1.000   Min.   :1.00  
 1st Qu.:20.00   1st Qu.:1.000   1st Qu.:3.00  
 Median :26.00   Median :2.000   Median :3.00  
 Mean   :28.78   Mean   :1.672   Mean   :3.19  
 3rd Qu.:35.00   3rd Qu.:2.000   3rd Qu.:4.00  
 Max.   :86.00   Max.   :2.000   Max.   :5.00  
                                 NA's   :223   

But summary stats can hide a lot. Always visualize.

Histograms: the workhorse

bfi |>
  ggplot(aes(x = age)) +
  geom_histogram(binwidth = 5, fill = "#2c7fb8", color = "white") +
  labs(x = "Age (years)", y = "Count") +
  theme_minimal(base_size = 14) +
  theme(panel.grid = element_blank(),
        axis.line = element_line(color = "gray70"))

Histograms: the workhorse

Histogram of participant ages with 5-year bins showing a strong right skew, with most participants in their teens to thirties and a long tail extending past age 60.

Choosing binwidth

The binwidth matters a lot:

Three histograms of age side by side comparing binwidths. Binwidth 1 is noisy with erratic spikes, binwidth 5 clearly shows the right-skewed shape, and binwidth 20 oversimplifies into just a few bars that hide the skew.

There’s no “right” answer — try a few and pick the one that shows the shape clearly.

Density plots: smooth alternative

bfi |>
  ggplot(aes(x = age)) +
  geom_density(fill = "#2c7fb8", alpha = 0.6, color = "#2c7fb8") +
  labs(x = "Age (years)", y = "Density") +
  theme_minimal(base_size = 14) +
  theme(panel.grid = element_blank(),
        axis.line = element_line(color = "gray70"))

Density plots: smooth alternative

Density plot of participant ages showing a smooth right-skewed curve that peaks around age 20 and gradually tapers off toward age 80.

Exploring survey items

# All Extraversion items
item_labels <- c(
  E1 = "E1\nDon't talk a lot. (R)",
  E2 = "E2\nFind it difficult to\napproach others. (R)",
  E3 = "E3\nKnow how to\ncaptivate people.",
  E4 = "E4\nMake friends easily.",
  E5 = "E5\nTake charge."
)

bfi |>
  select(E1:E5) |>
  pivot_longer(everything(), names_to = "item", values_to = "response") |>
  ggplot(aes(x = response)) +
  geom_bar(fill = "#2c7fb8") +
  facet_wrap(~ item, nrow = 1, labeller = as_labeller(item_labels)) +
  scale_x_continuous(breaks = 1:6) +
  labs(
    title = "Distribution of responses to Extraversion items",
    x = NULL,
    y = NULL,
    caption = "Scale: 1 = Very inaccurate, 6 = Very accurate.   (R) = reverse-scored."
  ) +
  theme_minimal(base_size = 12) +
  theme(panel.grid = element_blank(),
        axis.line.x = element_line(color = "gray70"),
        axis.line.y = element_blank(),
        axis.text.y = element_blank(),
        axis.ticks.y = element_blank(),
        strip.text = element_text(lineheight = 1.1),
        plot.caption = element_text(hjust = 0, size = 10))

Exploring survey items

Five small bar charts side by side showing the response distribution for each Extraversion item on a 1 to 6 scale. Item descriptions appear above each panel. The two reverse-scored items (E1, E2) skew toward lower values while E3, E4, and E5 skew toward higher values.

A quick aside: factor()

Education is stored as numbers (1–5), but the numbers are codes, not measurements. factor() turns them into ordered categories with readable labels:

factor(education,
       levels = 1:5,
       labels = c("Some HS", "HS grad", "Some college",
                  "College grad", "Graduate"))
  • levels = the values currently in the data (in the order you want them)
  • labels = what to show instead

We’ll come back to factors in detail in a later session — for now, just know this is how we attach labels to numeric codes.

Categorical variables

bfi |>
  filter(!is.na(education)) |>
  mutate(education = factor(education,
    levels = 1:5,
    labels = c("Some HS", "HS grad", "Some college", "College grad", "Graduate"))) |>
  ggplot(aes(x = education)) +
  geom_bar(fill = "#2c7fb8") +
  labs(x = "Education", y = "Count") +
  theme_minimal(base_size = 14) +
  theme(panel.grid = element_blank(),
        axis.line = element_line(color = "gray70"))

Categorical variables

Bar chart of education level distribution showing the most common levels are some college and college graduate in the BFI sample.

Counts and proportions

# Counts
bfi |>
  filter(!is.na(education)) |>
  mutate(education = factor(education,
    levels = 1:5,
    labels = c("Some HS", "HS grad", "Some college", "College grad", "Graduate"))) |>
  count(education)
# A tibble: 5 × 2
  education        n
  <fct>        <int>
1 Some HS        224
2 HS grad        292
3 Some college  1249
4 College grad   394
5 Graduate       418

Counts and proportions

# Proportions
bfi |>
  filter(!is.na(education)) |>
  mutate(education = factor(education,
    levels = 1:5,
    labels = c("Some HS", "HS grad", "Some college", "College grad", "Graduate"))) |>
  count(education) |>
  mutate(proportion = n / sum(n))
# A tibble: 5 × 3
  education        n proportion
  <fct>        <int>      <dbl>
1 Some HS        224     0.0869
2 HS grad        292     0.113 
3 Some college  1249     0.485 
4 College grad   394     0.153 
5 Graduate       418     0.162 

Don’t round inside the data frame

Keep the full precision in your data. Round only when you’re displaying it:

bfi |>
  count(education) |>
  mutate(proportion = n / sum(n)) |>
  knitr::kable(digits = 2)

If you round in the data frame, you’ve thrown away precision you can’t get back. If you round at display, the underlying data is still exact.

Outliers

What are outliers?

Values that are unusual compared to the rest of the data. They might be:

  • Real — genuinely extreme values (a 95-year-old in a college study)
  • Errors — data entry mistakes (age = 999)
  • Interesting — worth investigating

Never delete an outlier without understanding it.

Spotting outliers visually

bfi |>
  ggplot(aes(x = age)) +
  geom_histogram(binwidth = 5, fill = "#2c7fb8", color = "white") +
  labs(x = "Age (years)", y = "Count") +
  theme_minimal(base_size = 14) +
  theme(panel.grid = element_blank(),
        axis.line = element_line(color = "gray70"))

Spotting outliers visually

Histogram of participant ages with 5-year bins highlighting the right-skewed distribution, with a handful of participants beyond age 60 that could be considered outliers.

Spotting outliers with boxplots

bfi |>
  ggplot(aes(x = age)) +
  geom_boxplot(fill = "#2c7fb8", alpha = 0.7) +
  labs(x = "Age (years)") +
  theme_minimal(base_size = 14) +
  theme(panel.grid = element_blank(),
        axis.line.x = element_line(color = "gray70"),
        axis.text.y = element_blank(),
        axis.ticks.y = element_blank())

Points beyond the whiskers are flagged as outliers by the 1.5×IQR rule.

Spotting outliers with boxplots

Horizontal boxplot of participant ages with the median around age 28, the box spanning roughly 20 to 40, and individual points plotted beyond the upper whisker indicating older outliers.

Investigating outliers programmatically

# Find unusually old or young participants
bfi |>
  filter(age > 60) |>
  select(age, gender, education)
# A tibble: 22 × 3
     age gender education
   <int>  <int>     <int>
 1    68      1         5
 2    64      2         5
 3    74      1         5
 4    63      2         3
 5    62      1         2
 6    86      2         2
 7    61      1         2
 8    67      1         5
 9    67      1         5
10    63      1         5
# ℹ 12 more rows

What to do with outliers

Outlier type What to do
Likely error (age = 999) Fix or set to NA
Real but extreme Note it, keep it, mention in write-up
Suspicious Investigate further
Doesn’t affect conclusions Keep it, move on

Document your decisions. Future you will want to know.

Let’s explore YOUR data

Our class survey

Remember that survey you filled out Week 1? Let’s do some EDA on it.

class_survey <- read_csv(
  "https://raw.githubusercontent.com/sjweston/datasci410/refs/heads/main/data/class_survey.csv"
)
glimpse(class_survey)
Rows: 34
Columns: 15
$ response_id       <chr> "R_7Fkq5bRdEYWENih", "R_7mni7E59hbZtnha", "R_7BKhLj1…
$ major             <chr> "Psychology", "Neuroscience", "Psychology", "Psychol…
$ year              <chr> "Junior", "Senior", "Junior", "Junior", "Senior", "S…
$ chronotype        <chr> "Evening person", "Morning person", "Morning person"…
$ sleep_hrs         <dbl> 8.0, 6.0, 9.0, 9.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.…
$ caffeine_per_day  <dbl> 2.00, 1.00, 2.00, 1.00, 2.00, 1.00, 0.15, 1.50, 1.30…
$ social_media_hrs  <dbl> 7.00, 2.00, 1.00, 3.00, 2.00, 8.00, 0.33, 1.00, 1.00…
$ tabs_open         <dbl> 363, 10, 2, 11, 6, 26, 7, 27, 12, 9, 6, 6, 246, 22, …
$ stress            <dbl> 4, NA, 3, 3, 2, 6, 6, 6, 2, 5, 3, 6, 7, 6, 7, 7, 7, …
$ coding_excited    <dbl> 7, 6, 8, 6, NA, 7, 7, NA, 5, 5, 6, 8, 8, 5, 5, 6, 7,…
$ coding_anxious    <dbl> 8, NA, 8, 8, 5, 7, 6, 5, 4, 5, 8, 7, 3, NA, 8, 9, 4,…
$ personality_neur  <dbl> 9, NA, 9, 7, NA, NA, NA, 7, 8, NA, NA, 9, NA, NA, NA…
$ personality_extra <dbl> 8, NA, 2, 6, 5, 4, 8, 7, 9, 7, 6, 9, 6, 5, 7, 5, 7, …
$ personality_consc <dbl> 8, 3, NA, 7, NA, 7, NA, 6, 8, 8, 8, 7, 4, 5, 5, 9, 7…
$ data_words        <chr> "frustrating and overwhelming", "novel", "novice", "…

How many tabs do you have open?

ggplot(class_survey, aes(x = tabs_open)) +
  geom_histogram(binwidth = 5, fill = "#2c7fb8", color = "white") +
  labs(x = "Number of tabs", y = "Count") +
  theme_minimal(base_size = 14) +
  theme(panel.grid = element_blank(),
        axis.line = element_line(color = "gray70"))

How many tabs do you have open?

Histogram of browser tabs open showing a right-skewed distribution, with most students reporting fewer than 20 tabs and a few extreme values stretching the distribution.

How does our class sleep?

ggplot(class_survey, aes(x = sleep_hrs)) +
  geom_histogram(binwidth = 1, fill = "#2c7fb8", color = "white") +
  labs(x = "Average hours of sleep per night", y = "Count") +
  theme_minimal(base_size = 14) +
  theme(panel.grid = element_blank(),
        axis.line = element_line(color = "gray70"))

How does our class sleep?

Histogram of average hours of sleep per night reported by students, showing the distribution centered around 7 hours.

Morning people or night owls?

ggplot(class_survey, aes(x = chronotype)) +
  geom_bar(fill = "#2c7fb8") +
  labs(x = "Chronotype", y = "Count") +
  theme_minimal(base_size = 14) +
  theme(panel.grid = element_blank(),
        axis.line = element_line(color = "gray70"))

Morning people or night owls?

Bar chart of chronotype distribution showing counts of students who identify as morning person, evening person, or neither.

Pair coding break

Your turn: 10 minutes

Using the class_survey data, look at the distribution of social_media_hrs (self-reported hours per day on social media):

  1. Plot the distribution — what shape is it?
  2. Check for outliers. Are any values suspicious?
  3. What does this tell you about our class?

Tip

Try both a histogram and a boxplot. They show different things. Use summary(class_survey$social_media_hrs) first to get oriented.

Solution: Social media hours

class_survey |>
  ggplot(aes(x = social_media_hrs)) +
  geom_histogram(
    binwidth = 2,
    fill = "navy",
    color = "white") +
  scale_x_continuous(
    breaks = c(0, 2, 4, 6, 8)
  ) +
  theme_minimal()

class_survey |>
  ggplot(aes(x = social_media_hrs)) +
  geom_boxplot(
    fill = "lightgrey",
    color = "navy"
  ) +
  theme_bw()

Solution: histogram

Histogram of self-reported social media hours per day with binwidth of 2, navy fill and white borders, x-axis breaks at 0, 2, 4, 6, 8, and a minimal theme.

Solution: boxplot

Horizontal boxplot of self-reported social media hours per day with light grey fill and navy outline, on a black-and-white theme. Outliers appear as individual points beyond the upper whisker.

Missing data — a first look

Missing data are everywhere

# How much missing data do we have?
bfi |>
  summarize(across(everything(), ~ sum(is.na(.)))) |>
  pivot_longer(everything(), names_to = "variable", values_to = "n_missing") |>
  arrange(desc(n_missing))
# A tibble: 28 × 2
   variable  n_missing
   <chr>         <int>
 1 education       223
 2 N4               36
 3 N5               29
 4 O3               28
 5 A2               27
 6 A3               26
 7 C4               26
 8 E3               25
 9 C2               24
10 E1               23
# ℹ 18 more rows

Visualizing missingness

bfi |>
  summarize(across(everything(), ~ mean(is.na(.)))) |>
  pivot_longer(everything(), names_to = "variable", values_to = "pct_missing") |>
  mutate(pct_missing = pct_missing * 100) |>
  filter(pct_missing > 0) |>
  arrange(desc(pct_missing)) |>
  ggplot(aes(x = reorder(variable, pct_missing), y = pct_missing)) +
  geom_col(fill = "#2c7fb8") +
  coord_flip() +
  labs(x = "Variable", y = "% Missing") +
  theme_minimal(base_size = 14) +
  theme(panel.grid = element_blank(),
        axis.line = element_line(color = "gray70"))

Visualizing missingness

Horizontal bar chart showing the percentage of missing values for each variable that has any missingness. Education has the highest rate at around 7 percent, followed by several personality items with lower rates.

Why it matters for EDA

Missing data can bias your exploration:

  • If missingness is related to the variables you’re studying, your distributions are wrong
  • If certain groups are more likely to have missing data, you might miss important patterns

We’ll cover this in depth in Session 14. For now: always check for missing data before exploring.

The EDA workflow

A systematic approach

  1. Look at the structure — glimpse(), summary()
  2. Check for missing data — how much? Which variables?
  3. Explore each variable — histograms, bar charts, summaries
  4. Look for outliers — boxplots, programmatic checks
  5. Ask questions — what surprised you? What do you want to know more about?
  6. Repeat — EDA is iterative

Putting it together: a mini-EDA

# Education distribution — what does it look like?
bfi |>
  filter(!is.na(education)) |>
  mutate(education = factor(education, labels = c(
    "HS incomplete", "HS grad", "Some college",
    "College grad", "Graduate degree"
  ))) |>
  ggplot(aes(x = education)) +
  geom_bar(fill = "#2c7fb8") +
  labs(x = "Education level", y = "Count") +
  theme_minimal(base_size = 14) +
  theme(panel.grid = element_blank(),
        axis.line = element_line(color = "gray70"),
        axis.text.x = element_text(angle = 30, hjust = 1))

Putting it together: a mini-EDA

Bar chart of education levels in the BFI sample showing the most common category is some college, followed by college graduate and high school graduate, with fewer participants at the extremes of high school incomplete and graduate degree.

Wrapping up

EDA toolkit so far

Tool When to use
glimpse() First look at structure
summary() Quick numeric summaries
geom_histogram() Continuous distributions
geom_density() Smooth distribution shape
geom_bar() Categorical distributions
geom_boxplot() Outlier detection
count() Frequency tables
is.na() / sum(is.na()) Missing data check

Before next class

📖 Read:

✅ Practice:

  • Explore at least 3 variables in bfi
  • Look for outliers and missing data
  • Start generating questions

Key takeaways

  1. EDA is an attitude — curiosity, not confirmation
  2. Always visualize — summary stats hide patterns
  3. Outliers are information — investigate, don’t delete
  4. Check missing data first — it affects everything
  5. Ask questions, then answer them — that’s the loop

The one thing to remember

EDA isn’t a step you finish. It’s the habit of looking at your data before you believe anything about it.

Next time: EDA — Covariation

Get a head start

Assignment 4: Part 2 (Distributions)

Open Assignment 4 and start Part 2. You have everything you need:

  1. 2.1 — Histogram of penguin body mass. Try ≥3 binwidths in your scratch work; submit the one that best shows the shape, with a comment explaining why.
  2. 2.2 — Overlapping density plots of body mass, colored by species, with transparency.
  3. 2.3 — Boxplot of body mass by species, then violin plot. Comment on what each shows that the other doesn’t.

Due Sunday, May 3 at 11:59 PM.