The fit takes one line of code. Knowing what it is worth takes the rest of the lecture.
Part 1 · Concepts
Part 2 · The Iowa River
Part 3 · Group Lab 2
Seven measures, each worked by hand on five numbers.
Definition
The mean is the sum divided by the count. The median is the middle value once sorted.
Equation
Five days of flow, hundreds of cfs
Sorted: 2, 3, 3, 4, 18
mean = (2+3+3+4+18) / 5
= 30 / 5 = 6
median = 3rd of 5 sorted = 3
Definition
Skew measures how lopsided values are: 0 is symmetric, positive means a long right tail. log10 turns equal ratios into equal steps.
Equation
Flow that doubles each step
Raw: 100, 200, 400, 800, 1600 cfs
g = 0.89
log10: 2.0, 2.3, 2.6, 2.9, 3.2
g = 0
+0.30 in log10 = ×2 in cfs; +1 = ×10
Definition
r measures how closely points fall along one straight line, from −1 to +1. It says nothing about why.
Equation
Five weeks of rain (x) and flow (y)
x = 1, 2, 3, 4, 5 y = 1, 3, 2, 5, 4 mean x = mean y = 3 sum (x-mx)(y-my) = 4+0+0+2+2 = 8 sum (x-mx)² = 10 sum (y-my)² = 10 r = 8 / sqrt(10 × 10) = 0.80
Definition
A residual is observed minus predicted. Least squares picks the line with the smallest sum of squared residuals, SSE.
Equation
Same five points, line yhat = 0.6 + 0.8x
Residuals:
−0.4, +0.8, −1.0, +1.2, −0.6
SSE = 0.16+0.64+1.00+1.44+0.36
= 3.60
Every other straight line gives a larger SSE
Definition
The slope is the change in y for one more unit of x. The intercept is the predicted y at x = 0.
Equation
Same five points
b1 = 8 / 10 = 0.8 b0 = 3 − 0.8 × 3 = 0.6 If y is log10 flow: +0.8 in log10 = ×10^0.8 = ×6.3 in cfs
Definition
r² is the share of the spread in y that the line accounts for, from 0 to 1.
Equation
Same five points
SST = 10.0 misses from the flat mean SSE = 3.6 misses from the line r² = 1 − 3.6 / 10 = 0.64 = 0.80²
Definition
RMSE, root mean squared error, is the typical size of a residual, in the units of y.
Equation
Same five points
RMSE = sqrt(3.6 / 5)
= sqrt(0.72)
= 0.85
If y is log10 flow:
×10^0.85 = about ×7 in cfs
| Measure | What it tells you | Five numbers | Iowa River, 2019–2025 |
|---|---|---|---|
| Mean and median | Typical value; gap means a tail | 6 and 3 | 2,248 and 1,170 cfs |
| Skew, log10 | Lopsidedness; log evens it out | raw lopsided, log even | 1.76 raw, −0.02 log10 |
| Correlation, r | How straight, not why | 0.80 | 0.31, 7-day rain vs log10 flow |
| Slope and intercept | Effect per unit; value at zero | 0.8 and 0.6 | 0.0082 per mm and 2.945, in log10 |
| r² | Share of spread explained | 0.64 | 0.099 |
| RMSE | Typical size of a miss | 0.85 | 0.49 log10, a factor of 3.1 |
The same measures on 2,549 days of rain and flow.
| File or table | Source | Coverage | Used for |
|---|---|---|---|
| iowa_streamflow_daily.csv | USGS NWIS, site 05454500, Iowa River at Iowa City | Daily, 2015–2025 | The response: discharge |
| iowa_city_weather_daily.csv | Iowa Environmental Mesonet, station IOW | Daily, 2019–2025 | Rain, temperature, dew point |
| merged, then s | The two files joined on date (Week 6) | 2,549 days; 2,537 with a full 7-day window | Every fit today |
| train / test | s split on 2024-01-01 | 1,820 days / 717 days | The honest score |
Raw units
log10 units
68% of days sit below the mean. The wettest 10% of days carry 38% of the water. A straight line fitted in raw units spends its effort on those few days.
Iowa River at Iowa City, 2,549 days with weather, 2019–2025.
Python
q = sample["discharge_cfs"]
print(q.describe().round(0).to_string())
print(f"skew, raw: {q.skew():.2f}")
print(f"skew, log10: {np.log10(q).skew():.2f}")Output
count 2549.0 mean 2248.0 std 2584.0 min 89.0 25% 474.0 50% 1170.0 75% 2900.0 max 12600.0 skew, raw: 1.76 skew, log10: -0.02
?
A report says typical Iowa River flow is 2,248 cfs. Two thirds of days are below that. What should it report instead?
Shared physics: warm air holds more water. Strong, and no discovery.
Weak: the river answers rain days later, and most days have no rain.
Same two records, chosen with physics: three times the correlation.
Both records rise in spring and fall in late summer. Part of their correlation comes from that shared calendar, not from one driving the other day by day.
Removing each month's normal (Week 6 anomalies) takes out the season and shrinks r. What is left is closer to the rain-to-river link itself.
Left: mean of each calendar month, 2019–2025, rain at Iowa City (top) and Iowa River discharge (bottom).
?
Ice cream sales and drownings rise and fall together. What drives both, and what plays that role for rain and river flow?
Python
fit = linear_regression_summary(s["precip_7day_mm"],
s["log10_discharge"])
for key in ["slope", "intercept", "r_squared",
"p_value", "stderr", "rmse"]:
print(f"{key:>10}: {fit[key]:.4g}")Output
slope: 0.008213
intercept: 2.945
r_squared: 0.09887
p_value: 2.486e-59
stderr: 0.0004925
rmse: 0.4939?
The slope is 0.0082 log10 cfs per mm. One inch (25.4 mm) of rain falls in a week. By what factor does the line say flow changes?
Python
a, b = fit["intercept"], fit["slope"]
print(f"dry week: {10 ** a:,.0f} cfs")
print(f"+25.4 mm rain: x {10 ** (b * 25.4):.2f}")
print(f"+50 mm rain: x {10 ** (b * 50):.2f}")
print(f"typical miss: x {10 ** fit['rmse']:.1f}")Output
dry week: 881 cfs +25.4 mm rain: x 1.62 +50 mm rain: x 2.57 typical miss: x 3.1
Same records, four choices of x and y (all years)
Longer rain windows, log10 flow (Group Lab 2, Module A)
x = merged["precipitation_mm"]
y = merged["discharge_cfs"]
b, a = np.polyfit(x, y, 1)
r2 = np.corrcoef(x, y)[0, 1] ** 2
fig, ax = plt.subplots(figsize=(6, 4))
ax.scatter(x, y)
ax.plot(x, a + b * x, color="tab:orange")
ax.set_title(f"Rain drives river flow (R² = {r2:.2f})")
Same 2,549 days as the rest of the lecture. Work in pairs for two minutes: find five problems, from the data choices to the title.
Problems 1 and 2 are about the science, 3 and 4 about the claim, 5 about the evidence. The next three slides fix them in that order.
s = merged.sort_values("date").set_index("date")
s["precip_7day_mm"] = (s["precipitation_mm"]
.rolling("7D", min_periods=7).sum())
s["log10_discharge"] = np.log10(s["discharge_cfs"])
s = (s.dropna(subset=["precip_7day_mm", "log10_discharge"])
.reset_index())
fit = linear_regression_summary(s["precip_7day_mm"],
s["log10_discharge"])
Fixes 1 and 2. Same two files; r² rises from about 0.01 to 0.099. Both choices came from physics before the fit: rivers integrate rain over days, and flow varies by factors.
Fixed: the river's memory and its skew
fig, ax = plt.subplots(figsize=(8, 5))
ax.scatter(s["precip_7day_mm"], s["discharge_cfs"],
s=6, alpha=0.15, color="0.2")
xx = np.linspace(0, 150, 100)
ax.plot(xx, 10 ** (fit["intercept"] + fit["slope"] * xx))
ax.set_yscale("log")
ax.set_xlabel("Rain in the previous 7 days (mm)")
ax.set_ylabel("Discharge (cfs)")
ax.set_title("Wetter weeks bring higher flow;\n"
"rain explains about 10% of the variation")
Fixes 3 and 4. The axis is labelled in cfs on a log scale, which a reader can check against a gauge. The title states the pattern and its size; it claims no cause.
Fixed: units a reader can check, a title that matches r²
train, test = train_test_split_by_time(s, "date", "2024-01-01")
fit_train = linear_regression_summary(train["precip_7day_mm"],
train["log10_discharge"])
b0, b1 = fit_train["intercept"], fit_train["slope"]
pred = b0 + b1 * test["precip_7day_mm"]
mean_only = train["log10_discharge"].mean()
print(rmse(test["log10_discharge"], pred)) # the model
print(rmse(test["log10_discharge"], mean_only)) # a flat line
Fixed: evidence from years the fit never saw
Black: observed. Red: the model. Dashed: the flat line. The model follows the rainy spells but misses the size of most floods and droughts; it beats the flat line by only 0.047 log10 units.
The caption a reader needs, assembled from the three steps.
Daily discharge of the Iowa River at Iowa City against rain at Iowa City summed over the previous 7 days, 2,537 days, 2019–2025, log10 flow.
Least-squares line fitted on 2019–2023 (1,820 days); scored on 2024–2025 (717 days) that the fit never saw.
Wetter weeks bring higher flow (slope 0.0066 log10 cfs per mm), but rain explains about 10% of the variation, and the test error is a factor of 2.9 in discharge.
?
The test RMSE (0.467) came out lower than the training RMSE (0.505). Celebrate, or investigate?
The line took what a line can. The rest is noise or missing variables.
The relationship bends: a straight line is the wrong shape.
Spread grows with the fitted value. A log transform usually cures it.
Our data, 7-day rain fit: in raw cfs the residual spread grows from 1,469 to 2,997 cfs from the lowest to the highest third of fitted values, a funnel. In log10 units it stays between 0.44 and 0.53.
+0.42 in log10 units is a factor of 2.6; −0.47 is a factor of 2.9 the other way.
The same week of rain does more in May than in September. Wet soils send rain to the river; dry soils and growing crops keep it. One rain predictor cannot hold that, so the residuals do.
| Date | 7-day rain | Observed | Line expects | Residual | Hypothesis to check |
|---|---|---|---|---|---|
| 2023-10-17 | 47.8 mm | 173 cfs | 2,173 cfs | −1.10 | Dry autumn soils took up the rain |
| 2023-10-18 | 45.5 mm | 169 cfs | 2,081 cfs | −1.09 | The same storm, a day later |
| 2023-12-09 | 9.1 mm | 89 cfs | 1,047 cfs | −1.07 | Lowest flow of 2019–2025 |
| 2025-08-07 | 0.0 mm | 10,300 cfs | 881 cfs | +1.07 | Rain upstream, routed through Coralville Lake |
| 2025-08-08 | 0.0 mm | 10,300 cfs | 881 cfs | +1.07 | The same flood wave |
Investigate, never delete silently: each hypothesis can be checked against another gauge or station.
Each day is almost a copy of its neighbours. Draw test days at random and nearly every one has its yesterday and tomorrow in the training set: the test is not new data.
Our two-parameter line has little to memorise, so a random split scores about the same (0.503). A flexible model would not; the time split stays honest either way.
An error means little until it is set beside something simple. Our model beats the flat line by 0.047 and the monthly normals by 0.028. Yesterday's flow is 7 times better: the river's own memory outpredicts rain. The model describes a relationship; it is not a forecast.
Python
lo = fit["slope"] - 1.96 * fit["stderr"]
hi = fit["slope"] + 1.96 * fit["stderr"]
print(f"slope {fit['slope']:.4f} log10 cfs per mm")
print(f"95% CI {lo:.4f} to {hi:.4f}")
print(f"p {fit['p_value']:.1e}")
print(f"r² {fit['r_squared']:.3f}")Output
slope 0.0082 log10 cfs per mm 95% CI 0.0072 to 0.0092 p 2.5e-59 r² 0.099
| Verb | Evidence it needs | Our sentence |
|---|---|---|
| suggests | A pattern in one dataset | Wetter weeks suggest higher flow (r² = 0.10) |
| shows | A check that could have failed | The residuals show a seasonal cycle: +0.42 in June, −0.47 in September |
| predicts | Skill on unseen data, beyond a baseline | Not yet: 0.047 better than a flat line, far behind yesterday's flow |
| proves | A mechanism plus a controlled test | Nothing here proves cause |
Two partners, one honest regression.
Both partners, 20 minutes: same-day fit, then 7-day rain against log10 flow; set split_date.
Partner A chooses the predictor: correlation matrix, rain windows of 3, 7 and 14 days.
Partner B tests honestly: residual panels, train and test RMSE with evaluate_window.
Run A's window through B's function, plot test residuals by month, cross-check one number each.
Swap letters from Group Lab 1, so each partner leads both kinds of work this semester.
| Function | Returns | Use it for | Today |
|---|---|---|---|
| s.describe() | Count, mean, quartiles | Summarize before modeling | Daily discharge |
| s.skew() | Asymmetry of the distribution | Whether to transform | 1.76 raw, −0.02 log10 |
| np.log10(s) | Values by order of magnitude | Multiplicative variables | log10 discharge |
| s.rolling("7D").sum() | A trailing window total | Memory of past days | 7-day rain |
| df.corr() | Pearson r for every pair | Screen candidate predictors | Temperature vs dew point |
| linear_regression_summary(x, y) | Slope, intercept, r², p, stderr | Fit and read a line | 7-day rain vs log10 flow |
| np.polyfit(x, y, 1) | Slope and intercept only | A quick line | The critique slide |
| train_test_split_by_time(df, ...) | Train before, test after a date | An honest score | 2019–2023 / 2024–2025 |
| rmse(y_true, y_pred) | Typical miss, in y units | Comparing models | 0.467 log10 cfs |
| s.autocorr(1) | Correlation with yesterday | How much a day repeats | 0.99 |
linear_regression_summary, rmse and train_test_split_by_time live in src/earthcourse/stats.py.
?
Yesterday's flow predicts today's seven times better than rain does. Is the rain model useless?