Implement once what you usually import, and every future import becomes an informed choice.
| File or value | Source | Coverage | Used for |
|---|---|---|---|
| iowa_streamflow_daily.csv | USGS NWIS, site 05454500, Iowa River at Iowa City | Daily, 2015–2025: 4,018 days, none missing | Every algorithm today |
| flow | stream["discharge_cfs"].to_numpy() | One NumPy array | Loops index it by position |
| The largest flood | np.argmax(flow) | 14,800 cfs on 2018-10-06 | Peak timing, the hardest gap |
One line of pandas, three decisions you did not make.
smooth = stream["discharge_cfs"].rolling(14).mean()
Grey: daily flow. Red: the one-line 14-day mean. Both defaults, center=False and min_periods=14, are decisions someone else made; the output looks like data either way.
?
rolling(14).mean() returned NaN for the first 13 days of the record. Bug, or decision?
Plain language before Python, so the logic is wrong on paper and not in code.
Twelve days, a 5-day window (half = 2). Red: the day being smoothed. Pink: the days averaged for it. Near the ends the window shrinks, so those values rest on fewer days; that shrinking is an edge policy, chosen in the pseudocode.
Pseudocode
for each position i in the series:
start = i minus half the window (not below 0)
end = i plus half the window (not past the end)
output[i] = mean of values from start to end?
Rewrite the pseudocode for a trailing window. Which lines change?
From English to a loop, and why the timing of a window is science.
Python
def moving_average_centered(values, window):
half = window // 2
out = []
for i in range(len(values)):
start = max(0, i - half)
end = min(len(values), i + half + 1)
out.append(np.mean(values[start:end]))
return np.array(out)
print(moving_average_centered(np.array([1., 2., 3., 4., 5.]), 3))Output
[1.5 2. 3. 4. 4.5]
Grey: daily. Red: centered mean. Black: centered median. Dashed: trailing mean. The flood is a one-day spike on a long plateau: the next two weeks stay near 12,500 cfs, while eight days earlier the river ran at 1,490. Even a symmetric window leans toward the plateau.
?
Your 14-day trailing mean puts the 2018 flood peak 13 days late. Bug, or design decision? Defend both answers.
Storms and floods survive.
Single storms blur; wet months stay.
Floods merge into seasons.
Only wet and dry years remain.
Centered means over the whole 2015–2025 record, shown for 2018–2019. "Spread kept" is the standard deviation of the smoothed series as a share of the raw one. Pick the width from the signal the question is about.
Every window meets the end of the record; the policy there is a choice.
Dots: daily flow. One 15-day centered mean, three answers at the edges. Away from the edges all three agree to within 4e-12 cfs. Zero padding pulls the first week toward a dry river that never existed.
?
How many values of a 14-day trailing mean rest on fewer than 14 days? When does that matter?
A draft with four problems, and the three checks that catch them.
Five numbers with a hand-checked answer: the middle of [1..5] with window 3 is 3.0.
Agreement with an independent implementation, pandas, on all 4,018 days.
Hide data you have, rebuild it, score the result: the gap experiment.
Each rung catches what the one below misses. The next five slides climb the first two.
def moving_average_draft(values, window):
half = window // 2
out = np.zeros(len(values))
for i in range(half, len(values) - half):
out[i] = values[i - half:i + half].mean()
return out
smooth = moving_average_draft(flow, 7)
plt.plot(stream["date"], flow, color="0.8")
plt.plot(stream["date"], smooth)
It runs, and its line hugs pandas (r = 0.9986). Work in pairs for two minutes: find four problems in the code, not in the plot.
Red: the draft. Black: pandas, centered, 7 days.
def moving_average_draft(values, window):
half = window // 2 # 3
out = np.zeros(len(values)) # 2
for i in range(half, len(values) - half): # 2
out[i] = values[i - half:i + half].mean() # 1
return out
# 4
Problems 1 and 3 are arithmetic; 2 is an edge policy nobody chose; 4 is why the other three survived. The largest error inside the record is 1,133 cfs, and 6 edge days read 0 cfs.
Python
check = moving_average_draft(np.array([1., 2., 3., 4., 5.]), 3)
print(check)
assert check[2] == 3.0, f"middle value is {check[2]}, not 3.0"Output
[0. 1.5 2.5 3.5 0. ] AssertionError: middle value is 2.5, not 3.0
Python
mine = moving_average_centered(flow, 7)
trusted = (stream["discharge_cfs"]
.rolling(7, center=True, min_periods=1)
.mean().to_numpy())
print("largest difference:", np.max(np.abs(mine - trusted)), "cfs")
print("agree:", np.allclose(mine, trusted))Output
largest difference: 1.4210854715202004e-14 cfs agree: True
Python
mine = moving_average_centered(flow, 14)
trusted = (stream["discharge_cfs"]
.rolling(14, center=True, min_periods=1)
.mean().to_numpy())
print("largest difference:", round(np.max(np.abs(mine - trusted))), "cfs")
print("values in my window:", 2 * (14 // 2) + 1)
print(pd.Series(range(6)).rolling(4, center=True).mean().tolist())Output
largest difference: 508 cfs values in my window: 15 [nan, nan, 1.5, 2.5, 3.5, nan]
Python
import timeit
q = stream["discharge_cfs"]
tasks = {
"loop": lambda: moving_average_centered(flow, 15),
"pandas": lambda: q.rolling(15, center=True, min_periods=1).mean(),
"convolve": lambda: np.convolve(flow, np.ones(15) / 15, mode="same"),
}
for name, task in tasks.items():
ms = 1000 * min(timeit.repeat(task, number=1, repeat=5))
print(f"{name:>8}: {ms:6.2f} ms")Output
loop: 12.73 ms pandas: 0.11 ms convolve: 0.14 ms
?
What can a tiny test catch that agreement with pandas cannot?
Hide data we have, rebuild it, and score the rebuild against the truth.
One position, 2020-06-23, 2020. Black: the true flow we hid. Red: the straight line from the last good day to the next one. Pink: the gap. Because we made the gap, we know the truth and can score the fill; with a real outage the truth is gone.
Mean RMSE in cfs, with the range over 20 evenly spaced positions. Dots: each position. Pink: lowest to highest. The mean sits above the median at every length: most gaps are easy, and a few hide a storm.
Python
line, last_week = [], []
for s0 in np.linspace(31, len(flow) - 61, 20).astype(int):
hidden = flow[s0:s0 + 7]
with_gap = flow.copy()
with_gap[s0:s0 + 7] = np.nan
line.append(rmse(hidden, fill_gap_linear(with_gap, s0, 7)[s0:s0 + 7]))
last_week.append(rmse(hidden, np.full(7, flow[s0 - 7:s0].mean())))
line, last_week = np.array(line), np.array(last_week)
print(f"straight line: {line.mean():4.0f} cfs")
print(f"last week's mean: {last_week.mean():4.0f} cfs")
print(f"the line wins at {(line < last_week).sum()} of 20 positions")Output
straight line: 292 cfs last week's mean: 426 cfs the line wins at 15 of 20 positions
A 7-day gap, 09-28 to 10-04, ending two days before the peak. Its error beats the worst 30-day gap of the random test (2,348 cfs): position matters as much as length. Flag filled values as estimated.
?
The mean error of a 30-day gap is seven times that of a 3-day gap. What does that tell a sensor-network manager?
Two partners, two algorithms, one flood.
Both partners, 15 minutes: load flow, write the trailing moving average, check it against pandas.
Partner A smooths without shifting the peak: moving_median_centered and peak_table.
Partner B fills sensor gaps: gap_error_table for 3- to 30-day gaps at 20 positions.
Fill a gap before the largest flood, smooth it, run each other's tiny test, explain.
Swap letters from Group Lab 2, so each partner leads both kinds of work this semester.
| Function | Does | Use it for | Today |
|---|---|---|---|
| s.rolling(w).mean() | Trailing mean; NaN until w values | Real-time smoothing | Peak +13 days |
| s.rolling(w, center=True) | Window on both sides | Finished records | Peak +3 days |
| min_periods=1 | Shrink the window at the edges | Report every day | First value from 8 days |
| s.rolling(w, ...).median() | Middle value of each window | Spiky records | 13,000 cfs peak |
| np.convolve(x, k, "same") | Weighted sum; zeros past the edges | Fast, any kernel | 642 cfs on day 1 |
| s.interpolate() | Straight line across NaN | Short sensor gaps | Matches fill_gap_linear |
| np.argmax(x) | Position of the largest value | Find the flood | 2018-10-06 |
| assert cond, msg | Stop when cond is False | Tiny tests | [1..5] middle is 3 |
| np.allclose(a, b) | All equal within a tolerance | Trusted-twin checks | 1e-14 cfs apart |
| rmse(y_true, y_pred) | Typical miss, in y units | Scoring a fill | 884 cfs, 30 days |
rmse lives in src/earthcourse/stats.py; the loops you write in the lab sit beside these as their twins.
?
Name one tool you have imported this semester whose hidden decisions you could now list. What are they?