Nine rows in the messy export appear twice. Each duplicate is a real order counted again, so every total you have calculated from the export so far is too high: 23,500.67 dollars instead of 22,486.69, an overstatement of 4.5%. Duplicates come from re-run exports, copy-pasted rows and systems that retry a failed upload. They are the last problem in the file, and fixing them lets you turn this whole section into a single function.
\nimport pandas as pd\n\nmessy = pd.read_csv(\"orders_messy.csv\")\ndupes = messy.duplicated()\nprint(dupes.sum(), \"duplicate rows\")\n\nboth_copies = messy[messy.duplicated(keep=False)].sort_values(\"order_id\")\nprint(both_copies.head(4).to_string())duplicated() returns a mask that is True for every row identical to an earlier one. The first appearance is not flagged (keep="first"), which is what you want for removal. For inspection, keep=False flags every copy, and sorting by the id puts each pair together. Look at them before deleting anything: rows 22 and 113 are the same order, same product, same total, almost a hundred rows apart. That pattern, identical rows far apart, is typical of an export that was appended to itself.
import pandas as pd\n\nmessy = pd.read_csv(\"orders_messy.csv\")\ndeduped = messy.drop_duplicates()\nprint(len(messy), \"->\", len(deduped))\nprint(deduped[\"order_id\"].is_unique)After dropping exact duplicates, every order_id appears once, which confirms that the duplicates were full copies and not two different versions of the same order. If is_unique had been False, you would have conflicting rows: the same id with, say, two different totals. Those need a decision, such as keeping the latest version, and drop_duplicates(subset=["order_id"], keep="last") implements it once you have made that decision.
Duplicate detection compares values exactly. "Ali Evans" and "Ali Evans " are different strings, and $22.00 and 22.00 are different totals. In this export the copies happen to be byte-for-byte identical, but in general you strip text, standardise case and convert numbers before looking for duplicates, or near-identical copies survive.
You now have five cleaning steps, written across five lessons. Scattered across a notebook, they are fragile: run a cell twice or skip one, and the result changes. Put them in one function, in the right order, with the checks at the end:
\nimport pandas as pd\n\nCOUNTRIES = {\"Egypt\", \"United Arab Emirates\", \"Saudi Arabia\", \"Jordan\",\n \"United Kingdom\", \"Germany\", \"United States\", \"Canada\"}\n\ndef parse_money(values):\n cleaned = (values.astype(\"string\").str.strip()\n .str.replace(\"$\", \"\", regex=False).str.replace(\",\", \"\", regex=False))\n return pd.to_numeric(cleaned, errors=\"coerce\")\n\ndef parse_dates(raw):\n iso = pd.to_datetime(raw, format=\"%Y-%m-%d\", errors=\"coerce\")\n dmy = pd.to_datetime(raw, format=\"%d/%m/%Y\", errors=\"coerce\")\n mdy = pd.to_datetime(raw, format=\"%m/%d/%Y\", errors=\"coerce\")\n ambiguous = dmy.notna() & mdy.notna() & (dmy != mdy)\n return iso.fillna(dmy).fillna(mdy), ambiguous\n\ndef clean_orders(path):\n \"\"\"Load the messy export and return a clean, deduplicated DataFrame.\"\"\"\n df = pd.read_csv(path)\n df[\"customer\"] = df[\"customer\"].str.strip()\n df[\"country\"] = df[\"country\"].str.strip().str.title()\n df[\"channel\"] = df[\"channel\"].str.strip().str.replace(\" \", \"_\").fillna(\"unknown\")\n df[\"quantity\"] = df[\"quantity\"].astype(\"Int64\")\n df[\"total\"] = parse_money(df[\"total\"])\n df[\"order_date\"], df[\"date_is_ambiguous\"] = parse_dates(df[\"order_date\"])\n df = df.drop_duplicates().reset_index(drop=True)\n\n assert df[\"order_id\"].is_unique, \"duplicate order ids remain\"\n assert df[\"order_date\"].notna().all(), \"unparsed dates\"\n assert df[\"total\"].notna().all() and (df[\"total\"] > 0).all(), \"bad totals\"\n assert set(df[\"country\"]) <= COUNTRIES, f\"unknown countries: {set(df['country']) - COUNTRIES}\"\n return df\n\nclean = clean_orders(\"orders_messy.csv\")\nprint(clean.shape)\nprint(clean.dtypes)\nprint(round(clean[\"total\"].sum(), 2), \"total sales,\", clean[\"date_is_ambiguous\"].sum(), \"ambiguous dates\")df["order_date"], df["date_is_ambiguous"] = parse_dates(...) unpacks the two values the helper returns into two columns at once. reset_index(drop=True) renumbers the rows 0 to 158 after the drop, so positions and labels agree again. Notice the dtypes too: quantity is Int64 and total is Float64, both nullable, because parse_money converts through pandas’ string type and to_numeric keeps the nullable family. They behave like ordinary numbers in every calculation you will do.
The function reads its own file, so it never depends on what you happened to run earlier in a notebook. Same input, same output, every time. When next month’s export arrives, clean_orders("orders_messy_2026_01.csv") runs the whole section in one line, and if the new file contains a ninth country spelling or a total in euros, an assertion stops it with a message naming the problem.
Where should the function live? While you are exploring, in the first cell of your notebook. Once a second notebook or a colleague needs it, move it to its own file, say cleaning.py next to your notebooks, and write from cleaning import clean_orders wherever you need it. One definition, imported everywhere, means a fix made once reaches every analysis that depends on it.
The export is clean: 159 orders, 22,486.69 dollars of sales, and 25 dates flagged as ambiguous (30 before the duplicates went). Section 4 moves from cleaning to answering, starting with the most used tool in pandas: groupby.
df.duplicated() flags exact repeats after their first appearance; keep=False flags every copy so you can inspect them side by side.","drop_duplicates(subset=[...]) deduplicates on key columns, but only use a key that is truly unique per row, never order_id in a table of order lines.","Clean text before looking for duplicates, because "Ali Evans" and "Ali Evans " are different strings.","A cleaning function turns a one-off notebook session into a repeatable step - same input, same output, every month.","End the function with assert statements that encode what clean means, so bad data stops the pipeline instead of reaching a report."],"further":[{"title":"pandas.DataFrame.drop_duplicates","url":"https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.drop_duplicates.html"},{"title":"pandas.DataFrame.duplicated","url":"https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.duplicated.html"},{"title":"The assert statement","url":"https://docs.python.org/3/reference/simple_stmts.html#the-assert-statement"}],"quiz":[{"q":"A table has two identical rows for order 1026. What does df.duplicated().sum() count for them?
keep="first", the first copy is not flagged; only repeats are."},{"text":"0, because the rows have different index labels","why":"duplicated() compares the column values, not the index."},{"text":"1, the second copy only","why":"Correct. The first appearance is kept as the original, and each later identical row is flagged. Use keep=False to flag both."}],"answer":2},{"q":"Why is items.drop_duplicates(subset=["order_id"]) dangerous on order_items?
order_item_id."},{"text":"It raises an error because order_id is numeric.","why":"Any column type can be used as a subset key; the risk is silent data loss, not an error."},{"text":"It changes the original table in place.","why":"drop_duplicates returns a new DataFrame; the danger is in what that new table is missing."}],"answer":0},{"q":"What is the main benefit of ending a cleaning function with assert df["order_id"].is_unique?
return hands back the data."},{"text":"If a future export breaks the rule, the script stops with an error instead of producing a wrong report.","why":"Correct. The assertion turns an assumption into a tested fact every time the function runs."}],"answer":3}],"exercise":{"kind":"python","files":["orders_messy.csv"],"starter":"import pandas as pd\n\ndef parse_money(values):\n cleaned = (values.astype(\"string\").str.strip()\n .str.replace(\"$\", \"\", regex=False).str.replace(\",\", \"\", regex=False))\n return pd.to_numeric(cleaned, errors=\"coerce\")\n\n\ndef clean_orders(path):\n \"\"\"Return the messy export cleaned and deduplicated.\"\"\"\n df = pd.read_csv(path)\n # strip customer; strip + title-case country\n # channel: \"mobile app\" -> \"mobile_app\", missing -> \"unknown\"\n # total: parse_money\n # order_date: pd.to_datetime(..., format=\"mixed\", dayfirst=True)\n # drop exact duplicates and reset the index\n # assert that order_id is unique\n return df\n\n\nclean = clean_orders(\"orders_messy.csv\")\n\n# Sales per cleaned country, as a dict {country: total rounded to 2 decimals}\nsales_by_country = None\n","solution":"import pandas as pd\n\ndef parse_money(values):\n cleaned = (values.astype(\"string\").str.strip()\n .str.replace(\"$\", \"\", regex=False).str.replace(\",\", \"\", regex=False))\n return pd.to_numeric(cleaned, errors=\"coerce\")\n\n\ndef clean_orders(path):\n \"\"\"Return the messy export cleaned and deduplicated.\"\"\"\n df = pd.read_csv(path)\n df[\"customer\"] = df[\"customer\"].str.strip()\n df[\"country\"] = df[\"country\"].str.strip().str.title()\n df[\"channel\"] = df[\"channel\"].str.replace(\" \", \"_\").fillna(\"unknown\")\n df[\"total\"] = parse_money(df[\"total\"])\n df[\"order_date\"] = pd.to_datetime(df[\"order_date\"], format=\"mixed\", dayfirst=True)\n df = df.drop_duplicates().reset_index(drop=True)\n assert df[\"order_id\"].is_unique, \"duplicate order ids remain\"\n return df\n\n\nclean = clean_orders(\"orders_messy.csv\")\n\nsales_by_country = {}\nfor country in clean[\"country\"].unique():\n sales_by_country[country] = round(float(clean.loc[clean[\"country\"] == country, \"total\"].sum()), 2)\nprint(sales_by_country)\n","checks":[{"id":"rows","test":"isinstance(clean, pd.DataFrame) and len(clean) == 159 and clean['order_id'].is_unique"},{"id":"index","test":"list(clean.index) == list(range(159))"},{"id":"text","test":"clean['country'].nunique() == 8 and set(clean['channel']) == {'web', 'mobile_app', 'marketplace', 'unknown'} and (clean['customer'] == clean['customer'].str.strip()).all()"},{"id":"types","test":"pd.api.types.is_numeric_dtype(clean['total']) and pd.api.types.is_datetime64_any_dtype(clean['order_date'])"},{"id":"total","test":"round(float(clean['total'].sum()), 2) == 22486.69"},{"id":"by-country","test":"isinstance(sales_by_country, dict) and len(sales_by_country) == 8 and round(sum(sales_by_country.values()), 2) == 22486.69"},{"id":"reusable","test":"len(clean_orders('orders_messy.csv')) == 159"}],"title":"Write clean_orders()","prompt":"Complete clean_orders(path) so it loads the export and returns it cleaned: strip customer; strip and title-case country; turn mobile app into mobile_app and missing channels into "unknown"; convert total with the provided parse_money; parse order_date with pd.to_datetime(..., format="mixed", dayfirst=True); drop exact duplicates and renumber the index from 0; and assert that order_id is unique before returning. Then set sales_by_country to a dictionary mapping each cleaned country to its total sales, rounded to 2 decimals.
df[...]. Do the text and money cleaning before drop_duplicates().","df = df.drop_duplicates().reset_index(drop=True) removes the copies and renumbers rows 0 to 158. Then assert df["order_id"].is_unique, "duplicate order ids remain".","For the dictionary, loop over clean["country"].unique() and sum clean.loc[clean["country"] == country, "total"], as in Lesson 1.4. (Lesson 4.1 shows a one-line groupby for this.)"],"explanation":"The function reads its own file and applies every fix from this section in a fixed order, so running it twice, or on next month’s file, gives the same kind of result. Cleaning happens before deduplication so near-identical copies would also match; here the nine copies were exact. One shortcut to be honest about: format="mixed", dayfirst=True still misreads the five month-first dates you caught in Lesson 3.4, so a production version would keep the ambiguity flag or repair those rows from the order system. The assertion is the function’s contract: if a future export contains two different rows for one order, it stops instead of returning bad data. The country totals add back up to 22,486.69, the deduplicated sales figure, which is the parts-equal-the-whole check from Lesson 1.4 again. In the next lesson, clean.groupby("country")["total"].sum() replaces the loop.
clean has 159 rows, one per order","index":"The index is renumbered 0 to 158","text":"Countries, channels and customer names are cleaned","types":"total is numeric and order_date is a datetime","total":"Total sales are 22,486.69","by-country":"sales_by_country has 8 countries adding up to 22,486.69","reusable":"Calling clean_orders again gives the same 159 rows"}},"words":689,"readMinutes":3}