Open orders_messy.csv in a spreadsheet and the gaps are easy to miss: an empty quantity cell here, N/A in the channel column there. Load it into pandas and those gaps become NaN, which then flows silently through your calculations: sums skip it, means ignore it, and a fillna(0) written in a hurry turns “we don’t know” into “nothing was sold”. Missing values are not a technical nuisance to clear away; each one is a question about the business.
import pandas as pd\n\nmessy = pd.read_csv(\"orders_messy.csv\")\nprint(messy.shape)\nprint(messy.isna().sum())\nprint(messy[messy.isna().any(axis=1)].head().to_string())isna() returns a DataFrame of booleans, and summing it counts the gaps per column: 9 quantities and 7 channels. isna().any(axis=1) asks, per row, whether any column is missing (axis=1 means “across the columns”), which lets you look at the actual damaged rows. Looking at them matters: the 9 missing quantities belong to ordinary orders across both years, not to one broken day.
The raw file has no empty channel cells. It has the text N/A, and pandas converted it to NaN because N/A is on its default list of missing markers, along with an empty cell, NA, NaN, null, None and a few others. Usually that is helpful. Sometimes it is wrong: a column of country codes would lose Namibia, whose ISO code is NA. Load the file without the conversion to see what is really there:
import pandas as pd\n\nraw = pd.read_csv(\"orders_messy.csv\", keep_default_na=False)\nprint(raw[\"channel\"].value_counts())\nprint((raw[\"quantity\"] == \"\").sum(), \"empty quantity cells\")Now N/A shows up as a real value with a count of 7, and the quantity column is all text, because the empty cells stayed empty strings. To control the list precisely, pass na_values=["N/A", ""] together with keep_default_na=False.
There is no universal fix. Ask three questions, in this order, and keep the value missing if every answer is no:
\nRecovery comes first because it is the only option that adds information. The export’s order ids also exist in Cartwheel’s clean orders and order_items tables, so the missing quantities could be looked up there; you will learn the join that does it in Lesson 4.2. Until you can recover a value, do not invent one.
Applied to the export: the missing channel cannot be guessed from anything else in the row, but “we don’t know the channel” is still a fact worth keeping visible, so it gets an explicit label. A blank coupon code in the clean orders table is different again: there, missing means “no coupon used”, which is a real answer.
import pandas as pd\n\nmessy = pd.read_csv(\"orders_messy.csv\")\nmessy[\"channel\"] = messy[\"channel\"].fillna(\"unknown\")\nprint(messy[\"channel\"].value_counts())\nprint(len(messy.dropna(subset=[\"quantity\"])), \"rows with a known quantity\")dropna(subset=["quantity"]) removes only the rows missing a quantity, and only in the result you store; plain dropna() with no subset removes a row if any column is missing, which on the raw export would throw away 16 rows, including perfectly good totals. Drop for a specific analysis, not from your master copy.
One more trap: you cannot find gaps with ==. NaN is not equal to anything, not even itself, so messy["channel"] == np.nan is False on every row. Always use isna() and notna().
The quantity column loaded as float64, so quantities print as 1.0 and 2.0. That is the same effect you saw with referred_by in Lesson 2.2: classic NaN is a float. pandas also has nullable dtypes, written with a capital letter, which store a separate missing marker, pd.NA:
import pandas as pd\n\nmessy = pd.read_csv(\"orders_messy.csv\")\nqty = messy[\"quantity\"].astype(\"Int64\")\nprint(qty.dtype, qty.iloc[32:35].tolist())\nprint(qty.sum(), qty.count(), len(qty))\nprint(round(qty.mean(), 2))Int64 keeps whole numbers as integers and shows gaps as <NA>. Aggregations skip missing values by default, so sum() adds the 159 known quantities, and count() versus len() tells you exactly how many are missing. The same family includes Float64, boolean and string, and pd.read_csv(..., dtype_backend="numpy_nullable") loads every column that way. One behaviour to know: comparisons with pd.NA return <NA> rather than False, because “is an unknown quantity equal to 1?” honestly has no answer.
In the exercise you count, label and retype the gaps. Next you clean the text columns, where the country names come in four different spellings.
\n","toc":[{"id":"sec-1","text":"Finding the gaps","level":2},{"id":"sec-2","text":"What read_csv decided for you","level":2},{"id":"sec-3","text":"Deciding what to do","level":2},{"id":"sec-4","text":"Nullable types: integers that can be missing","level":2}],"takeaways":["read_csv turns empty cells and strings like N/A, NA and null into missing values by default, so check the raw file when counts look surprising.","df.isna().sum() counts blanks per column; df[df.isna().any(axis=1)] shows the rows that have any.","Every missing value needs a decision - recover it, label it, drop the row, or keep it missing - and filling with 0 is rarely right.","The nullable Int64 dtype keeps whole numbers as integers and shows gaps as <NA>, instead of turning the column into floats.","Sums, means and counts skip missing values by default, so count() and len() differ exactly by the number of gaps."],"further":[{"title":"Working with missing data","url":"https://pandas.pydata.org/docs/user_guide/missing_data.html"},{"title":"Nullable integer data type","url":"https://pandas.pydata.org/docs/user_guide/integer_na.html"},{"title":"pandas.DataFrame.fillna","url":"https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.fillna.html"}],"quiz":[{"q":"The messy export has 9 blank quantities. A colleague runs fillna(0) on the column before computing the average quantity per order. What happens?
mean(), not zeros. Filling turned “unknown” into “zero”, which is a claim about the data."},{"text":"pandas raises an error because the column is now mixed.","why":"Filling with 0 keeps the column numeric; there is no error, which is exactly why the mistake goes unnoticed."},{"text":"The average drops, because 9 unknown quantities are now counted as orders of zero items.","why":"Correct. Every order has at least one item, so 0 is not a plausible value. Leaving the gaps as missing gives the honest average of the known quantities."}],"answer":3},{"q":"The raw CSV has the text N/A in the channel column for 7 rows. After pd.read_csv, what does value_counts() show for it?
N/A with count 7","why":"N/A is on pandas’ default list of missing-value markers, so it is converted to NaN and dropped from value_counts()."},{"text":"Nothing; the 7 rows are missing values and are excluded unless you pass dropna=False","why":"Correct. Pass dropna=False to see them, or keep_default_na=False when loading if N/A should stay a real category."},{"text":"An error, because N/A is not a valid channel","why":"read_csv does not validate categories; it either keeps text or converts known missing markers."}],"answer":1},{"q":"q = messy["quantity"].astype("Int64"). What do q.count() and len(q) return for 168 rows with 9 gaps?
159 and 168","why":"Correct. count() counts non-missing values; len() counts rows, gaps included."},{"text":"168 and 168","why":"count() never counts missing values, whatever the dtype."},{"text":"159 and 159","why":"Missing values are still rows; len() includes them."}],"answer":0}],"exercise":{"kind":"python","files":["orders_messy.csv"],"starter":"import pandas as pd\n\nmessy = pd.read_csv(\"orders_messy.csv\")\n\n# 1. A Series: number of missing values in each column\nmissing_by_col = None\n\n# 2. Replace missing channels with the label \"unknown\" (in messy itself)\n\n# 3. Convert quantity to the nullable integer dtype \"Int64\" (in messy itself)\n\n# 4. Average quantity of the rows where it is known, rounded to 2 decimals\navg_quantity = None\n","solution":"import pandas as pd\n\nmessy = pd.read_csv(\"orders_messy.csv\")\n\nmissing_by_col = messy.isna().sum()\nprint(missing_by_col)\n\nmessy[\"channel\"] = messy[\"channel\"].fillna(\"unknown\")\nmessy[\"quantity\"] = messy[\"quantity\"].astype(\"Int64\")\n\navg_quantity = round(float(messy[\"quantity\"].mean()), 2)\n","checks":[{"id":"missing-counts","test":"isinstance(missing_by_col, pd.Series) and int(missing_by_col['quantity']) == 9 and int(missing_by_col['channel']) == 7 and int(missing_by_col.sum()) == 16"},{"id":"channel-labelled","test":"messy['channel'].isna().sum() == 0 and (messy['channel'] == 'unknown').sum() == 7"},{"id":"quantity-int64","test":"str(messy['quantity'].dtype) == 'Int64' and int(messy['quantity'].isna().sum()) == 9"},{"id":"avg","test":"round(float(avg_quantity), 2) == 1.43"},{"id":"rows-kept","test":"len(messy) == 168"}],"title":"Count, label and retype the gaps","prompt":"Load the messy export (done for you). Set missing_by_col to a Series counting missing values per column. In messy itself, replace missing channels with "unknown" and convert quantity to the nullable "Int64" dtype, keeping its gaps as missing. Then set avg_quantity to the mean of the known quantities, rounded to 2 decimals. Keep all 168 rows: no dropping, and no filling quantities.
messy.isna().sum() returns one count per column.","Assign the results back: messy["channel"] = messy["channel"].fillna("unknown") and messy["quantity"] = messy["quantity"].astype("Int64").","mean() skips missing values on its own: round(float(messy["quantity"].mean()), 2)."],"explanation":"The channel gaps get an explicit label because “unknown channel” is useful information in a report, while the quantity gaps stay missing because, until you recover them with the join from Lesson 4.2, no defensible value exists. Int64 (capital I) keeps the column as whole numbers with <NA> for the 9 gaps, instead of the 1.0, 2.0 floats you started with. mean() skips missing values by default, so the average of 1.43 describes the 159 known quantities; filling with 0 first would have produced 1.36, a number that describes no real order.
missing_by_col finds 9 missing quantities and 7 missing channels","channel-labelled":"Missing channels are now "unknown"","quantity-int64":"quantity is Int64 and still has its 9 gaps","avg":"avg_quantity is 1.43","rows-kept":"All 168 rows are still there"}},"words":710,"readMinutes":3}