The messy export writes dates three ways: 2024-03-05, 05/03/2024 and 03/05/2024. The first is unambiguous. The other two are the same day written by a European and an American spreadsheet, and the trouble is that 03/05/2024 might also be the European way of writing 3 May. Dates are the column where “it parsed without errors” and “it is correct” are furthest apart.
import pandas as pd\n\nmessy = pd.read_csv(\"orders_messy.csv\")\nprint(messy[\"order_date\"].sample(8, random_state=1).tolist())\ntry:\n pd.to_datetime(messy[\"order_date\"])\nexcept ValueError as error:\n print(str(error).splitlines()[0])pandas 2 infers one format from the first value, 2024-01-02, and applies it to the whole column. Row 6, 08/04/2024, does not fit, so the conversion stops. That strictness is a feature: older versions guessed row by row and could silently mix up days and months.
You know which formats exist, so say so. Format codes describe the layout: %Y is a four-digit year, %m a two-digit month, %d a two-digit day. Parse the column once per format with errors="coerce", which turns non-matching rows into NaT (pandas’ “not a time”), then fill the gaps in order of preference:
import pandas as pd\n\nmessy = pd.read_csv(\"orders_messy.csv\")\nraw = messy[\"order_date\"]\niso = pd.to_datetime(raw, format=\"%Y-%m-%d\", errors=\"coerce\")\ndmy = pd.to_datetime(raw, format=\"%d/%m/%Y\", errors=\"coerce\")\nmdy = pd.to_datetime(raw, format=\"%m/%d/%Y\", errors=\"coerce\")\nprint(iso.notna().sum(), \"ISO |\", dmy.notna().sum(), \"fit day-first |\", mdy.notna().sum(), \"fit month-first\")\n\nparsed = iso.fillna(dmy).fillna(mdy)\nprint(parsed.isna().sum(), \"unparsed\")\nprint(parsed.min(), parsed.max())Every row parsed, and the range is right: January 2024 to December 2025. Day-first comes before month-first in the chain because most of Cartwheel’s countries write dates that way. pandas can do the same in one call with pd.to_datetime(raw, format="mixed", dayfirst=True), which parses each value on its own; the explicit version is longer but shows you exactly which rule decided each row.
But look at the counts. 57 values fit day-first and 44 fit month-first, from only 68 slashed values. So 33 of them fit both.
\nA value that parses both ways to different dates is ambiguous. Three of the 33 are dates like 05/05/2025, where both readings agree, so they are harmless:
import pandas as pd\n\nmessy = pd.read_csv(\"orders_messy.csv\")\nraw = messy[\"order_date\"]\ndmy = pd.to_datetime(raw, format=\"%d/%m/%Y\", errors=\"coerce\")\nmdy = pd.to_datetime(raw, format=\"%m/%d/%Y\", errors=\"coerce\")\nambiguous = dmy.notna() & mdy.notna() & (dmy != mdy)\nprint(ambiguous.sum(), \"ambiguous dates\")\nprint(raw[ambiguous].head(6).tolist())Thirty rows depend on an assumption. You can make the assumption, as the fillna chain did, but you should know you made it, and the person reading your report should too.
Cartwheel’s clean orders table is the system of record: every order id in the export also appears there with its real timestamp. A lookup by order id lets you test the assumption directly. set_index("order_id") turns the dates into a Series labelled by id, and map() looks each export row up in it:
import pandas as pd\n\nmessy = pd.read_csv(\"orders_messy.csv\")\nraw = messy[\"order_date\"]\niso = pd.to_datetime(raw, format=\"%Y-%m-%d\", errors=\"coerce\")\ndmy = pd.to_datetime(raw, format=\"%d/%m/%Y\", errors=\"coerce\")\nmdy = pd.to_datetime(raw, format=\"%m/%d/%Y\", errors=\"coerce\")\nparsed = iso.fillna(dmy).fillna(mdy)\n\norders = pd.read_csv(\"orders.csv\", parse_dates=[\"order_date\"])\ntrue_dates = orders.set_index(\"order_id\")[\"order_date\"].dt.normalize()\ntruth = messy[\"order_id\"].map(true_dates)\n\nwrong = parsed != truth\nprint(wrong.sum(), \"dates disagree with the order system\")\nprint(messy.loc[wrong, [\"order_id\", \"order_date\"]].assign(parsed=parsed[wrong].dt.date, real=truth[wrong].dt.date))dt.normalize() drops the time of day so the comparison is date to date. Five dates are wrong, and every one is an ambiguous value that was really month-first: 02/08/2025 is 8 February, not 2 August. The day-first assumption was right for 25 of 30 ambiguous rows, which sounds good until you realise it moves five orders into the wrong month, and some into the wrong quarter.
The export carries dates only, but the orders table has full timestamps such as 2025-12-30 21:19:03, and none of them say which time zone they are in. Cartwheel records everything in one store clock, so you can treat them as local times and move on. When a file does include offsets like +04:00, pass utc=True to pd.to_datetime so every value lands on one common clock before you compare or group them; leaving zones mixed is how two orders placed at the same moment in Dubai and London end up on different dates, or even in different years around midnight on 31 December.
Once the column is datetime64, the .dt accessor exposes its parts:
import pandas as pd\n\norders = pd.read_csv(\"orders.csv\", parse_dates=[\"order_date\"])\nd = orders[\"order_date\"]\nprint(d.dt.year.value_counts().sort_index())\nprint(d.dt.day_name().value_counts().head(3))\nprint(d.dt.to_period(\"M\").value_counts().sort_index().tail(3))Year, month, weekday name, calendar month as a period: these are the keys you will group by in Section 4, where Lesson 4.4 covers time series properly.
\nIn the exercise you parse, flag and repair the export’s dates. Then one problem remains before the export is clean: rows that appear twice.
\n","toc":[{"id":"sec-1","text":"Letting pandas guess","level":2},{"id":"sec-2","text":"Parse each format explicitly","level":2},{"id":"sec-3","text":"Flag the ambiguity","level":2},{"id":"sec-4","text":"Verify against the source of truth","level":2},{"id":"sec-5","text":"Times and time zones","level":3},{"id":"sec-6","text":"What a real date column gives you","level":2}],"takeaways":["Withoutformat=, pandas 2 guesses the format from the first value and fails on any row written differently.","Parse each known format explicitly with pd.to_datetime(col, format=..., errors="coerce"), then combine the results with fillna.","A date like 05/08/2025 parses under both day-first and month-first rules; no parser can tell you which one is true.","Flag ambiguous values and resolve them from a trusted source (the system of record, another column, the person who made the file), never by assumption alone.","Once a column is datetime64, the .dt accessor gives you year, month, day_name() and to_period() for grouping."],"further":[{"title":"pandas.to_datetime","url":"https://pandas.pydata.org/docs/reference/api/pandas.to_datetime.html"},{"title":"strftime() and strptime() format codes","url":"https://docs.python.org/3/library/datetime.html#strftime-and-strptime-format-codes"},{"title":"Time series / date functionality","url":"https://pandas.pydata.org/docs/user_guide/timeseries.html"}],"quiz":[{"q":"pd.to_datetime(messy["order_date"]) raises ValueError: time data "08/04/2024" doesn't match format "%Y-%m-%d". Why?
2024-01-02, set it."},{"text":"Dates with slashes are not supported by pandas.","why":"Slashed dates parse fine with the right format, for example %d/%m/%Y."},{"text":"The column contains missing values.","why":"Missing values become NaT (not a time); they do not trigger a format error."}],"answer":1},{"q":"Which of these values is ambiguous, meaning both day-first and month-first readings give a valid but different date?
","options":[{"text":"25/10/2025","why":"There is no month 25, so only the day-first reading is valid."},{"text":"10/25/2025","why":"There is no month 25 in the day-first reading either, so this must be month-first."},{"text":"2025-10-05","why":"ISO dates always put the year, then month, then day, so there is only one reading."},{"text":"05/08/2025","why":"Correct. It could be 5 August or 8 May. Only outside information can decide."}],"answer":3},{"q":"Your parser assumes day-first for ambiguous dates. How should you check whether that assumption is right?
","options":[{"text":"Compare the parsed dates with a trusted source for the same records, such as the order system, and count disagreements.","why":"Correct. Here the cleanorders table shows the assumption is wrong for 5 of the 30 ambiguous rows."},{"text":"Check that no parsed date is NaT.","why":"An ambiguous date never fails to parse; it parses to the wrong date, which this check cannot see."},{"text":"Check that all dates fall between 2024 and 2025.","why":"Swapping day and month inside the same year stays inside the range, so this cannot catch the error."}],"answer":0}],"exercise":{"kind":"python","files":["orders_messy.csv","orders.csv"],"starter":"import pandas as pd\n\nmessy = pd.read_csv(\"orders_messy.csv\")\norders = pd.read_csv(\"orders.csv\", parse_dates=[\"order_date\"])\nraw = messy[\"order_date\"]\n\n# 1. Parse raw three ways (ISO, day-first, month-first) and combine: ISO, then day-first, then month-first\nparsed = None\n\n# 2. Column is_ambiguous: True where raw parses both day-first and month-first to DIFFERENT dates\n\n# 3. The real date of each export row, looked up by order_id in orders (date only, no time)\ntruth = None\n\n# 4. How many parsed dates disagree with truth\nn_wrong = None\n\n# 5. messy[\"order_date\"]: parsed dates, but use truth for the ambiguous rows\n","solution":"import pandas as pd\n\nmessy = pd.read_csv(\"orders_messy.csv\")\norders = pd.read_csv(\"orders.csv\", parse_dates=[\"order_date\"])\nraw = messy[\"order_date\"]\n\niso = pd.to_datetime(raw, format=\"%Y-%m-%d\", errors=\"coerce\")\ndmy = pd.to_datetime(raw, format=\"%d/%m/%Y\", errors=\"coerce\")\nmdy = pd.to_datetime(raw, format=\"%m/%d/%Y\", errors=\"coerce\")\nparsed = iso.fillna(dmy).fillna(mdy)\n\nmessy[\"is_ambiguous\"] = dmy.notna() & mdy.notna() & (dmy != mdy)\n\ntrue_dates = orders.set_index(\"order_id\")[\"order_date\"].dt.normalize()\ntruth = messy[\"order_id\"].map(true_dates)\n\nn_wrong = int((parsed != truth).sum())\n\nmessy[\"order_date\"] = parsed.where(~messy[\"is_ambiguous\"], truth)\nprint(n_wrong, messy[\"is_ambiguous\"].sum())\n","checks":[{"id":"parsed","test":"isinstance(parsed, pd.Series) and pd.api.types.is_datetime64_any_dtype(parsed) and int(parsed.isna().sum()) == 0"},{"id":"ambiguous","test":"'is_ambiguous' in messy.columns and int(messy['is_ambiguous'].sum()) == 30"},{"id":"wrong","test":"int(n_wrong) == 5"},{"id":"fixed-column","test":"pd.api.types.is_datetime64_any_dtype(messy['order_date']) and messy.loc[messy['order_id'] == 2401, 'order_date'].iloc[0] == pd.Timestamp('2025-02-08')"},{"id":"all-correct","test":"(messy['order_date'] == messy['order_id'].map(orders.set_index('order_id')['order_date'].dt.normalize())).all()"}],"title":"Parse, flag and repair the dates","prompt":"Using raw = messy["order_date"]: build parsed by parsing ISO (%Y-%m-%d), then day-first (%d/%m/%Y), then month-first (%m/%d/%Y) and combining them in that order. Add a boolean column is_ambiguous that is True where the day-first and month-first parses both succeed and give different dates. Set truth to each row’s real date from orders, looked up by order_id with the time removed, and n_wrong to how many parsed dates disagree with it. Finally, set messy["order_date"] to parsed, except on ambiguous rows, where it takes the value from truth.
pd.to_datetime(raw, format=..., errors="coerce"), then chain .fillna() in order of preference.","Ambiguous: dmy.notna() & mdy.notna() & (dmy != mdy). For the truth: orders.set_index("order_id")["order_date"].dt.normalize() is a lookup Series; messy["order_id"].map(...) uses it.","parsed.where(condition, other) keeps parsed where the condition is True and takes other elsewhere, so the condition is ~messy["is_ambiguous"]."],"explanation":"Explicit formats with errors="coerce" parse every row and record which rule matched. Thirty values fit both slash formats with different results; the day-first preference gets 25 of them right and 5 wrong, which you can only discover by checking against the order system. Series.where(cond, other) keeps values where the condition holds and substitutes elsewhere, so only the ambiguous rows are replaced, and the repaired column now matches the source of truth on all 168 rows. In real projects, keep the is_ambiguous flag even after repairing: it documents which values depended on an outside lookup.
parsed is a datetime Series with no unparsed values","ambiguous":"is_ambiguous flags 30 rows","wrong":"n_wrong is 5","fixed-column":"Order 2401 is dated 8 February 2025, not 2 August","all-correct":"Every repaired date matches the order system"}},"words":701,"readMinutes":3}