Videos
Six short videos, 6 to 9 minutes each, with questions to test yourself.
Each video shows one idea from the sessions in VS Code, with real code and real output. Watch it before or after the session it belongs to. Then answer the questions below it without AI and without rewatching, and only then open the answer. Trying to remember is what makes it stick. All videos have English subtitles (click CC on YouTube).
| # | Video | Length | Session |
|---|---|---|---|
| 1 | From Stata to pandas: objects, not commands | 8 min | 1 |
| 2 | The intake checklist, worked on one variable | 6 min | 2 |
| 3 | Anatomy of a merge audit | 8 min | 3 |
| 4 | Wide and long | 7 min | 3 |
| 5 | An API call, start to finish | 9 min | 4 |
| 6 | run_all and git in eight minutes | 7 min | 4 |
1. From Stata to pandas: objects, not commands
Watch on YouTube (8 min). For Stata users: each dataset is an object with a name, you ask it to act, and nothing changes unless you assign.
1. You run japan = cars[cars["origin"] == 3].
japan has 79 rows. How many rows does cars
have now, and how is that different from Stata’s
keep if origin == 3?
Answer
Still all 392. The subset is a new object with its own name, andcars is untouched. In Stata, keep throws the
other cars away, and you must load the file again (or use preserve and
restore) to get them back.
2. You run cars.sort_values("mpg") and then
cars["mpg"].head(3). The values are 18, 15, 18. Why are
they not sorted, and what is the fix?
Answer
sort_values made a sorted copy and threw it away;
cars itself never changed. Assign the result: cars =
cars.sort_values(“mpg”). In pandas, nothing changes unless you
assign.
3. Why check cars.shape every time you load a
file?
Answer
Python does not tell you the size of what you loaded, as Stata does. A wrong number of rows is often the first sign that something went wrong.2. The intake checklist, worked on one variable
Watch on YouTube (6 min). One wage variable through the checklist: numbers stored as text, a sentinel code for “refused”, and one extreme value.
1. A wage column has the type str. Why is it
dangerous to convert it with
pd.to_numeric(..., errors="coerce") and move
on?
Answer
errors=“coerce” turns everything it cannot convert into
missing, without a word. In the video, 12 real wages written with a
space between the thousands (“50 100”) would have become missing. Use
coerce only to find the problem values, fix them (remove the spaces),
then convert without errors=, so the script crashes if
anything else is wrong.
2. The documentation says −9 means “refused”.
isna() shows 4 missing wages. What has it missed, and what
happens to the mean if you leave it?
Answer
The sentinel codes: 6 people coded −9, whichisna() cannot
see because −9 is a number. Left in, six people earn −9 kronor a month
and every mean is wrong. Set them to missing with
.where(wages[“wage”] != -9): then 10 are missing.
3. The median wage is about 33,000 kronor and the mean is over 49,000. What do you do?
Answer
Look at the maximum: one person at 3.5 million a month. It may be a typing error or a real top earner, so do not delete it. Flag it in a new column, check the source, and write the decision down. Without it the mean is about 34,000.3. Anatomy of a merge audit
Watch on
YouTube (8 min). Check the key, avoid the silent inner
merge, stop duplicate keys with validate=, and read the
_merge column.
1. You write
pd.merge(workers, firms, on="firm_id") with 1,000 workers
and get 950 rows. What happened, and how do you see the missing
workers?
Answer
pandas’ default merge is inner: it keeps only keys found in both tables, so the 50 workers whose firm is not in the firm table were dropped without a warning. Merge withhow=“left” (or
“outer”) and indicator=True; the 50 have
_merge == “left_only”. Stata’s merge would
have kept them, with _merge == 1. (Not “50 workers had a
missing wage”: the merge never looks at wages.)
2. A left merge of 6 workers onto a firm table returns 8 rows. What is the cause, and which argument would have stopped it?
Answer
A firm id appears more than once in the firm table, so every worker at that firm is matched to each copy (in the video, firm 10 twice: its 2 workers become 4 rows).validate=“many_to_one” (or
“m:1”) stops the merge with a MergeError that
names the duplicated key. (Not “a worker is duplicated”: a left merge
keeps one row per worker unless the key repeats on the right.)
3. After indicator=True, which value of
_merge marks the workers with no matching firm, and which
marks the firms with no workers?
Answer
Workers without a firm:left_only (the workers are the left
table, the one before .merge). Firms without workers:
right_only. In a left merge right_only is
always 0, because rows only in the right table are not kept.
4. Wide and long
Watch on
YouTube (7 min). melt and pivot,
their Stata reshape equivalents, and why panel data are
kept long while you work.
1. A wide table has 290 municipalities and 10 year columns.
How many rows does melt(id_vars="municipality")
give?
Answer
2,900: one row per municipality and year (290 × 10). Melting multiplies the rows by the number of columns melted. (Not 290 or 10.)2. After melt, long["year"] + 1
stops with a TypeError. Why, and what is the
fix?
Answer
The years came from column names, and column names are text, soyear is a text column (pandas shows its type as
str). Convert it: long[“year”] =
long[“year”].astype(int). The values are fine; only the type of
the year column is wrong.
3. pivot stops with
ValueError: Index contains duplicate entries, cannot reshape.
What do you check, and with which line?
Answer
Whether some unit and year appear twice, so pandas does not know which value to put in the cell:long.duplicated([“region”,
“year”]).sum(). It is the same key check as before a merge. (Not
“pivot needs sorted data”: the order does not matter; the key must be
unique.)
5. An API call, start to finish
Watch on YouTube (9 min). Sweden’s monthly consumer price index from Statistics Sweden’s open API: save the raw answer, check every month arrived, read the dates, aggregate to quarters and compute inflation.
1. The table’s metadata lists 560 months, and your data have 548 rows. What do you do?
Answer
Stop and find the missing months before any analysis: theassert
len(rows) == len(months) line should fail exactly here. Perhaps
the answer was capped or split into pages and you read only part of it.
Carrying on is the tempting wrong answer: missing months are rarely
random (usually the latest ones, or a whole block), and they change the
results.
2. Why does the script save SCB’s answer to
data/raw/scb/ before doing anything else with
it?
Answer
The answer is the raw data. SCB revises and rebases its series, and addresses can change; the saved file records exactly what you received and lets you rerun the analysis without the internet (the Session 4 lab’sUSE_SAVED = True does just that).
3. You turn monthly job-ad counts into quarters with
.resample("QE").mean(). What is wrong, and what is the
right rule for the consumer price index?
Answer
Job ads are a flow, so a quarter’s number is the sum of its three months; the mean gives a monthly average, a third of the quarterly total. For a price index (or a rate) the mean is right. The rule is economics, not code: write it down.4. The latest quarter shows only two months in the count. What do you do with it, and why?
Answer
Drop it: keep only quarters with all three months (quarterly[n_months == 3]). A mean of two months is not a
quarter, and a year-on-year growth rate built on it is quietly wrong.
5. Code from an old answer online uses
.resample("Q") and stops with a ValueError.
Why, and what is the fix?
Answer
pandas 3 no longer accepts the old alias: write“QE”
(quarter, labelled by its end date), and “ME” for
month-end. The dates were read correctly; only the frequency code is out
of date, as the error message itself says.
6. run_all and git in eight minutes
Watch on YouTube (7 min). One script that rebuilds a whole project, the delete-outputs test, a log of rows in and out, git, and one minute on how a researcher checks AI.
1. What is the delete-outputs test, and what does it prove?
Answer
Delete everything the code made (data/processed/,
output/), run run_all.py once, and check that
everything comes back with the same numbers. It proves that every result
comes from the raw data through code. It is not a syntax test: scripts
can run perfectly and still depend on a file you once made by hand,
which is what the test catches.
2. The run log says
01_clean.py: 392 rows in, 385 out. What should that line
also say, and why does it matter?
Answer
Why rows went: here “dropped 7 cars with 3 or 5 cylinders”. The log is the first thing to read when a result looks strange: a step that suddenly drops half the sample shows there before it shows in the regression.3. You add a fourth script to the project. What must you
change in run_all.py?
Answer
One line: add its path to theSCRIPTS list in the CHANGE
HERE block, in the place where it should run. The loop below the list
does not change.
4. In the video’s last minute, how does the researcher use AI in research, in one rule?
Answer
The AI drafts; the researcher checks; every number can be traced to the saved script that made it, and every check is written down with its date. The same rule as in the course: try first, test on a case you know, and own the result.5. Why are output/ and
data/processed/ in .gitignore?
Answer
run_all.py rebuilds them from the raw data and the code, so
saving them adds nothing and risks keeping stale copies. They are
important, but because they are reproducible you save the recipe, not
the dish.