{"id":"l-pda-4-3","courseId":"python-data-analysis","lang":"en","kind":"lesson","summary":"Reshape data between long and wide formats - pivot_table for cross-tab summaries, crosstab for counts and rates, and melt to turn spreadsheet-style wide tables back into analysable long ones.","html":"

A manager does not want 3,991 rows. They want a grid: channels down the side, quarters across the top, revenue in the cells. The finance team sends targets the same way, one column per quarter. Your analysis, meanwhile, works best on long tables with one row per observation, because groupby, merge and filtering all expect that shape. Reshaping moves data between the two forms, and it is the last core skill before you start answering questions in earnest.

\n
\n On the left, a long table with columns quarter, channel and revenue has six rows. An arrow labelled pivot_table points to a wide table with quarters as rows and channels web and app as columns. A return arrow labelled melt points back.\n long\n quarter channel revenue\n Q1 web 27,868\n Q1 app 16,814\n Q2 web 30,110\n Q2 app 15,840\n one row per observation\n \n pivot_table\n \n melt\n wide\n quarter web app\n Q1 27,868 16,814\n Q2 30,110 15,840\n one column per channel\n
Long and wide hold the same data in different shapes
\n

pivot_table: summarise and reshape at once

\n

Start from the sales table you built in Lesson 4.2 (here rebuilt in a few lines) and ask for revenue by channel and year:

\n
Python
import pandas as pd\n\nitems = pd.read_csv(\"order_items.csv\")\norders = pd.read_csv(\"orders.csv\", parse_dates=[\"order_date\"])\nitems[\"revenue\"] = items[\"quantity\"] * items[\"unit_price\"] * (1 - items[\"discount\"])\nsales = items.merge(orders, on=\"order_id\", how=\"left\", validate=\"many_to_one\")\nsales = sales[sales[\"status\"] != \"cancelled\"]\nsales = sales.assign(year=sales[\"order_date\"].dt.year, quarter=sales[\"order_date\"].dt.quarter)\n\ntable = sales.pivot_table(index=\"channel\", columns=\"year\", values=\"revenue\",\n                          aggfunc=\"sum\", margins=True, margins_name=\"Total\")\nprint(table.round(0))
\n

Every argument maps to a spreadsheet pivot: index is the rows, columns the columns, values the field, aggfunc the summary. margins=True adds the totals row and column, and they confirm the result: 446,666 dollars overall, the revenue definition from the last lesson. Revenue grew in every channel from 2024 to 2025, and the web roughly tripled.

\n

pivot_table is the same computation as groupby(["channel", "year"])["revenue"].sum().unstack(). Use groupby when you will keep computing, and pivot_table when the grid itself is the output. For counts of distinct orders, change the aggregation: values="order_id", aggfunc="nunique". And fill_value=0 replaces the NaN of empty cells when zero is the honest answer, as it is for “no sales in that cell”.

\n

From amounts to shares

\n

A grid of amounts invites the next question: how did the mix change? Divide each column by its total and the grid shows shares instead. by_year.sum() returns one total per column, and dividing a DataFrame by that Series lines each total up with its own column by label, the same alignment rule you met with two Series in Lesson 2.1. To divide each row by its row total instead, use by_year.div(by_year.sum(axis=1), axis=0).

\n
Python
import pandas as pd\n\nitems = pd.read_csv(\"order_items.csv\")\norders = pd.read_csv(\"orders.csv\", parse_dates=[\"order_date\"])\nitems[\"revenue\"] = items[\"quantity\"] * items[\"unit_price\"] * (1 - items[\"discount\"])\nsales = items.merge(orders, on=\"order_id\", how=\"left\", validate=\"many_to_one\")\nsales = sales[sales[\"status\"] != \"cancelled\"]\n\nby_year = sales.pivot_table(index=\"channel\", columns=sales[\"order_date\"].dt.year,\n                            values=\"revenue\", aggfunc=\"sum\")\nprint((by_year / by_year.sum()).round(3))
\n

The web’s share of revenue rose from 53% in 2024 to 59% in 2025, while the app’s fell from 35% to 31%. Amounts tell you everything grew; shares tell you where the growth concentrated. Report both, because each hides what the other shows.

\n\n

crosstab: counts and rates

\n

For two categorical columns, pd.crosstab counts combinations, and normalize converts counts to shares. Here is Cartwheel’s web traffic:

\n
Python
import pandas as pd\n\nsessions = pd.read_csv(\"web_sessions.csv\")\nprint(pd.crosstab(sessions[\"source\"], sessions[\"device\"]))\nprint(pd.crosstab(sessions[\"device\"], sessions[\"purchased\"], normalize=\"index\").round(3))
\n

The second table answers a real question. normalize="index" makes each row sum to 1, so the 1 column is the purchase rate per device: 12.4% of desktop sessions end in a purchase, but only 6.7% of mobile sessions, although mobile brings the most traffic. That gap is worth a conversation with whoever owns the mobile checkout. Choose the normalisation by the question: "index" for rates within each row group, "columns" for composition of each column, "all" for shares of the grand total.

\n

melt: from wide back to long

\n

The finance team’s 2025 targets arrive as a spreadsheet, one column per quarter. To compare them with actual revenue you need them long, one row per channel and quarter, so they can be merged:

\n
Python
import pandas as pd\n\ntargets = pd.DataFrame({\n    \"channel\": [\"marketplace\", \"mobile_app\", \"web\"],\n    \"Q1\": [5000, 15000, 28000],\n    \"Q2\": [5000, 16000, 30000],\n    \"Q3\": [6000, 20000, 40000],\n    \"Q4\": [9000, 30000, 60000],\n})\nlong = targets.melt(id_vars=\"channel\", var_name=\"quarter\", value_name=\"target\")\nlong[\"quarter\"] = long[\"quarter\"].str[1].astype(int)\nprint(long.head(6))\nprint(long.shape)
\n

id_vars are the columns that identify a row and stay as they are; every other column becomes a pair of values in two new columns, the old column name in var_name and its value in value_name. Three rows times four quarters gives twelve. str[1] takes the second character of "Q1", so the quarter becomes a number that matches sales["quarter"], ready for a merge like the ones in Lesson 4.2.

\n\n

In the exercise you build the quarterly grid for 2025 and check performance against those targets. Next comes time itself: resampling, periods and growth rates.

\n","toc":[{"id":"sec-1","text":"pivot_table: summarise and reshape at once","level":2},{"id":"sec-2","text":"From amounts to shares","level":3},{"id":"sec-3","text":"crosstab: counts and rates","level":2},{"id":"sec-4","text":"melt: from wide back to long","level":2}],"takeaways":["Long data has one row per observation; wide data spreads one variable across columns. pandas analyses long data best, and people read wide tables best.","pivot_table(index=..., columns=..., values=..., aggfunc="sum") aggregates and reshapes in one step, like a spreadsheet pivot table.","pivot only reshapes; it raises ValueError: Index contains duplicate entries when several rows share a cell, so use pivot_table for raw data.","pd.crosstab(a, b, normalize="index") turns two columns into row percentages, which is the quickest way to compare rates between groups.","melt(id_vars=..., var_name=..., value_name=...) turns wide columns such as Q1…Q4 back into rows."],"further":[{"title":"Reshaping and pivot tables","url":"https://pandas.pydata.org/docs/user_guide/reshaping.html"},{"title":"pandas.DataFrame.pivot_table","url":"https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.pivot_table.html"},{"title":"pandas.melt","url":"https://pandas.pydata.org/docs/reference/api/pandas.melt.html"}],"quiz":[{"q":"

sales.pivot(index="quarter", columns="channel", values="revenue") raises ValueError: Index contains duplicate entries, cannot reshape. What is the fix?

","options":[{"text":"Sort sales by quarter first.","why":"Order is not the problem; many lines share the same quarter and channel, and pivot has no rule for combining them."},{"text":"Drop duplicate rows with drop_duplicates().","why":"The rows are different order lines, not duplicates. Dropping them would delete real revenue."},{"text":"Convert quarter to text.","why":"The type of the key is irrelevant; the problem is several values per cell."},{"text":"Use pivot_table with aggfunc="sum", which combines the values that land in the same cell.","why":"Correct. pivot_table aggregates first, then reshapes. pivot only works when every index and column pair appears once."}],"answer":3},{"q":"

A finance sheet has columns channel, Q1, Q2, Q3, Q4. Which call makes one row per channel and quarter?

","options":[{"text":"targets.melt(id_vars="channel", var_name="quarter", value_name="target")","why":"Correct. channel stays as an identifier, and the four quarter columns become rows with their names in quarter and values in target."},{"text":"targets.pivot_table(index="channel")","why":"The table is already wide by quarter; pivoting would not move the quarter columns into rows."},{"text":"targets.T","why":"Transposing swaps rows and columns, giving channels as columns, which is still wide."}],"answer":0},{"q":"

pd.crosstab(sessions["device"], sessions["purchased"], normalize="index") shows 0.067 in the mobile row, column 1. What does it mean?

","options":[{"text":"6.7% of all purchases happened on mobile.","why":"That would be normalize="columns". With "index", each row sums to 1."},{"text":"6.7% of mobile sessions ended in a purchase.","why":"Correct. Normalising by index divides each count by its row total, so the value is the conversion rate within mobile sessions."},{"text":"Mobile has 6.7% of all sessions.","why":"The share of sessions by device would come from value_counts(normalize=True) on device."}],"answer":1}],"exercise":{"kind":"python","files":["order_items.csv","orders.csv"],"starter":"import pandas as pd\n\nitems = pd.read_csv(\"order_items.csv\")\norders = pd.read_csv(\"orders.csv\", parse_dates=[\"order_date\"])\nitems[\"revenue\"] = items[\"quantity\"] * items[\"unit_price\"] * (1 - items[\"discount\"])\nsales = items.merge(orders, on=\"order_id\", how=\"left\", validate=\"many_to_one\")\nsales = sales[sales[\"status\"] != \"cancelled\"]\nsales = sales.assign(year=sales[\"order_date\"].dt.year, quarter=sales[\"order_date\"].dt.quarter)\nsales_2025 = sales[sales[\"year\"] == 2025]\n\ntargets = pd.DataFrame({\n \"channel\": [\"marketplace\", \"mobile_app\", \"web\"],\n \"Q1\": [5000, 15000, 28000],\n \"Q2\": [5000, 16000, 30000],\n \"Q3\": [6000, 20000, 40000],\n \"Q4\": [9000, 30000, 60000],\n})\n\n# 1. grid: 2025 revenue with quarters (1-4) as rows and channels as columns\ngrid = None\n\n# 2. targets_long: columns channel, quarter (int 1-4), target - 12 rows\ntargets_long = None\n\n# 3. vs_target: targets_long with an added \"revenue\" column (actual 2025 revenue\n# for that channel and quarter)\nvs_target = None\n\n# 4. Number of channel-quarters where revenue reached the target\nbeat = None\n","solution":"import pandas as pd\n\nitems = pd.read_csv(\"order_items.csv\")\norders = pd.read_csv(\"orders.csv\", parse_dates=[\"order_date\"])\nitems[\"revenue\"] = items[\"quantity\"] * items[\"unit_price\"] * (1 - items[\"discount\"])\nsales = items.merge(orders, on=\"order_id\", how=\"left\", validate=\"many_to_one\")\nsales = sales[sales[\"status\"] != \"cancelled\"]\nsales = sales.assign(year=sales[\"order_date\"].dt.year, quarter=sales[\"order_date\"].dt.quarter)\nsales_2025 = sales[sales[\"year\"] == 2025]\n\ntargets = pd.DataFrame({\n \"channel\": [\"marketplace\", \"mobile_app\", \"web\"],\n \"Q1\": [5000, 15000, 28000],\n \"Q2\": [5000, 16000, 30000],\n \"Q3\": [6000, 20000, 40000],\n \"Q4\": [9000, 30000, 60000],\n})\n\ngrid = sales_2025.pivot_table(index=\"quarter\", columns=\"channel\", values=\"revenue\", aggfunc=\"sum\")\n\ntargets_long = targets.melt(id_vars=\"channel\", var_name=\"quarter\", value_name=\"target\")\ntargets_long[\"quarter\"] = targets_long[\"quarter\"].str[1].astype(int)\n\nactual = sales_2025.groupby([\"channel\", \"quarter\"], as_index=False)[\"revenue\"].sum()\nvs_target = targets_long.merge(actual, on=[\"channel\", \"quarter\"], how=\"left\", validate=\"one_to_one\")\n\nbeat = int((vs_target[\"revenue\"] >= vs_target[\"target\"]).sum())\nprint(grid.round(0))\nprint(vs_target.round(0))\n","checks":[{"id":"grid","test":"isinstance(grid, pd.DataFrame) and grid.shape == (4, 3) and abs(float(grid.loc[4, 'web']) - 87530) < 1"},{"id":"long","test":"isinstance(targets_long, pd.DataFrame) and len(targets_long) == 12 and {'channel', 'quarter', 'target'} <= set(targets_long.columns) and sorted(set(targets_long['quarter'].tolist())) == [1, 2, 3, 4]"},{"id":"vs-target","test":"isinstance(vs_target, pd.DataFrame) and len(vs_target) == 12 and abs(float(vs_target['revenue'].sum()) - 326808) < 1"},{"id":"beat","test":"int(beat) == 8"}],"title":"2025 against target","prompt":"

sales_2025 holds 2025’s non-cancelled order lines, with quarter and channel columns, and targets is the finance team’s wide sheet. Build grid, a pivot table of revenue with quarters 1 to 4 as rows and channels as columns. Melt targets into targets_long with the columns channel, quarter (an integer 1 to 4) and target. Then make vs_target: targets_long plus a revenue column with the actual revenue for each channel and quarter. Finally set beat to the number of rows where revenue reached the target.

\n","hints":["sales_2025.pivot_table(index="quarter", columns="channel", values="revenue", aggfunc="sum").","targets.melt(id_vars="channel", var_name="quarter", value_name="target"), then turn "Q1" into 1 with .str[1].astype(int).","Aggregate actuals to the same grain with groupby(["channel", "quarter"], as_index=False)["revenue"].sum(), merge on both keys, and count (vs_target["revenue"] >= vs_target["target"]).sum()."],"explanation":"

The grid is for people; the long table is for computing. Melting the targets and aggregating the actuals to the same grain (one row per channel and quarter) makes the comparison a plain merge on two keys, and validate="one_to_one" confirms each target met exactly one actual. Cartwheel hit 8 of 12 targets; all four misses came in the first half of the year, and then it beat Q4 everywhere by a wide margin: web Q4 revenue was 87,530 dollars against a 60,000 target. Another clue that something unusual happened in Q4.

\n","checkLabels":{"grid":"grid is 4 quarters × 3 channels, with web Q4 at about 87,530","long":"targets_long has 12 rows with integer quarters","vs-target":"vs_target has 12 rows and actual revenue adding up to 2025’s total","beat":"beat is 8"}},"words":707,"readMinutes":3}