{"id":"l-pda-5-1","courseId":"python-data-analysis","lang":"en","kind":"lesson","summary":"Describe a numeric column honestly with the median, quantiles and spread, recognise skewed distributions and outliers, and read correlations without mistaking them for causes.","html":"<p>Someone asks: “What’s a typical Cartwheel order?” You compute the mean, 228 dollars, and say so. Then the marketing team designs a free-shipping threshold around 228 dollars, and discovers that two thirds of orders never reach it. The mean was correct and the answer was wrong, because order values are not spread evenly around their average. Describing data honestly means describing its shape, not only its centre.</p>\n<h2 id=\"sec-1\" tabindex=\"-1\">Mean, median and quantiles</h2>\n<div class=\"code-block\" dir=\"ltr\" data-lang=\"python\" data-run=\"python\" data-index=\"0\"><div class=\"code-head\"><span class=\"code-title\">Python</span><span class=\"code-actions\"><button type=\"button\" class=\"code-btn code-run\" data-action=\"run\" data-lang=\"python\" data-index=\"0\" aria-label=\"Run this example\"><svg class=\"icon \" viewBox=\"0 0 24 24\" fill=\"none\" stroke=\"currentColor\" stroke-width=\"2\" stroke-linecap=\"round\" stroke-linejoin=\"round\" aria-hidden=\"true\" focusable=\"false\"><path d=\"M5 5a2 2 0 0 1 3.008-1.728l11.997 6.998a2 2 0 0 1 .003 3.458l-12 7A2 2 0 0 1 5 19z\"/></svg><span>Run</span></button><button type=\"button\" class=\"code-btn\" data-action=\"copy\" aria-label=\"Copy code\"><svg class=\"icon \" viewBox=\"0 0 24 24\" fill=\"none\" stroke=\"currentColor\" stroke-width=\"2\" stroke-linecap=\"round\" stroke-linejoin=\"round\" aria-hidden=\"true\" focusable=\"false\"><rect width=\"14\" height=\"14\" x=\"8\" y=\"8\" rx=\"2\" ry=\"2\"/><path d=\"M4 16c-1.1 0-2-.9-2-2V4c0-1.1.9-2 2-2h10c1.1 0 2 .9 2 2\"/></svg><span>Copy</span></button></span></div><pre class=\"shiki\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#CF222E;--shiki-dark:#FF7B72\">import</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\"> pandas </span><span style=\"color:#CF222E;--shiki-dark:#FF7B72\">as</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\"> pd</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">items </span><span style=\"color:#CF222E;--shiki-dark:#FF7B72\">=</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\"> pd.read_csv(</span><span style=\"color:#0A3069;--shiki-dark:#A5D6FF\">\"order_items.csv\"</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">)</span></span>\n<span class=\"line\"><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">orders </span><span style=\"color:#CF222E;--shiki-dark:#FF7B72\">=</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\"> pd.read_csv(</span><span style=\"color:#0A3069;--shiki-dark:#A5D6FF\">\"orders.csv\"</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">, </span><span style=\"color:#953800;--shiki-dark:#FFA657\">parse_dates</span><span style=\"color:#CF222E;--shiki-dark:#FF7B72\">=</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">[</span><span style=\"color:#0A3069;--shiki-dark:#A5D6FF\">\"order_date\"</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">])</span></span>\n<span class=\"line\"><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">items[</span><span style=\"color:#0A3069;--shiki-dark:#A5D6FF\">\"revenue\"</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">] </span><span style=\"color:#CF222E;--shiki-dark:#FF7B72\">=</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\"> items[</span><span style=\"color:#0A3069;--shiki-dark:#A5D6FF\">\"quantity\"</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">] </span><span style=\"color:#CF222E;--shiki-dark:#FF7B72\">*</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\"> items[</span><span style=\"color:#0A3069;--shiki-dark:#A5D6FF\">\"unit_price\"</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">] </span><span style=\"color:#CF222E;--shiki-dark:#FF7B72\">*</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\"> (</span><span style=\"color:#0550AE;--shiki-dark:#79C0FF\">1</span><span style=\"color:#CF222E;--shiki-dark:#FF7B72\"> -</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\"> items[</span><span style=\"color:#0A3069;--shiki-dark:#A5D6FF\">\"discount\"</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">])</span></span>\n<span class=\"line\"><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">orders </span><span style=\"color:#CF222E;--shiki-dark:#FF7B72\">=</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\"> orders.merge(items.groupby(</span><span style=\"color:#0A3069;--shiki-dark:#A5D6FF\">\"order_id\"</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">, </span><span style=\"color:#953800;--shiki-dark:#FFA657\">as_index</span><span style=\"color:#CF222E;--shiki-dark:#FF7B72\">=</span><span style=\"color:#0550AE;--shiki-dark:#79C0FF\">False</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">)[</span><span style=\"color:#0A3069;--shiki-dark:#A5D6FF\">\"revenue\"</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">].sum(), </span><span style=\"color:#953800;--shiki-dark:#FFA657\">on</span><span style=\"color:#CF222E;--shiki-dark:#FF7B72\">=</span><span style=\"color:#0A3069;--shiki-dark:#A5D6FF\">\"order_id\"</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">, </span><span style=\"color:#953800;--shiki-dark:#FFA657\">validate</span><span style=\"color:#CF222E;--shiki-dark:#FF7B72\">=</span><span style=\"color:#0A3069;--shiki-dark:#A5D6FF\">\"one_to_one\"</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">)</span></span>\n<span class=\"line\"><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">orders </span><span style=\"color:#CF222E;--shiki-dark:#FF7B72\">=</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\"> orders[orders[</span><span style=\"color:#0A3069;--shiki-dark:#A5D6FF\">\"status\"</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">] </span><span style=\"color:#CF222E;--shiki-dark:#FF7B72\">!=</span><span style=\"color:#0A3069;--shiki-dark:#A5D6FF\"> \"cancelled\"</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">]</span></span>\n<span class=\"line\"><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">value </span><span style=\"color:#CF222E;--shiki-dark:#FF7B72\">=</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\"> orders[</span><span style=\"color:#0A3069;--shiki-dark:#A5D6FF\">\"revenue\"</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">]</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#0550AE;--shiki-dark:#79C0FF\">print</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">(value.describe(</span><span style=\"color:#953800;--shiki-dark:#FFA657\">percentiles</span><span style=\"color:#CF222E;--shiki-dark:#FF7B72\">=</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">[</span><span style=\"color:#0550AE;--shiki-dark:#79C0FF\">0.1</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">, </span><span style=\"color:#0550AE;--shiki-dark:#79C0FF\">0.25</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">, </span><span style=\"color:#0550AE;--shiki-dark:#79C0FF\">0.5</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">, </span><span style=\"color:#0550AE;--shiki-dark:#79C0FF\">0.75</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">, </span><span style=\"color:#0550AE;--shiki-dark:#79C0FF\">0.9</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">, </span><span style=\"color:#0550AE;--shiki-dark:#79C0FF\">0.99</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">]).round(</span><span style=\"color:#0550AE;--shiki-dark:#79C0FF\">2</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">))</span></span>\n<span class=\"line\"><span style=\"color:#0550AE;--shiki-dark:#79C0FF\">print</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">(</span><span style=\"color:#0A3069;--shiki-dark:#A5D6FF\">\"skew:\"</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">, </span><span style=\"color:#0550AE;--shiki-dark:#79C0FF\">round</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">(value.skew(), </span><span style=\"color:#0550AE;--shiki-dark:#79C0FF\">2</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">))</span></span></code></pre><div class=\"run-output\" data-run-output=\"0\" hidden></div></div>\n<p>Read it from the middle out. The <strong>median</strong> (50%) is 147 dollars: half of all orders are smaller. The quartiles say the middle half of orders falls between 68 and 303 dollars. The 90th percentile is 505 and the 99th is 1,168, while the maximum is 2,117. The mean, 228, sits far above the median because of that long right tail, and the positive skew of 2.43 confirms it. For a typical value in skewed data, quote the median; for totals and forecasts, the mean is the right number, because mean × count = total.</p>\n<p><code dir=\"ltr\">std</code>, the standard deviation, is 241 dollars, larger than the median itself. That is another sign of a spread-out, skewed distribution; in that situation quantiles describe the spread far better than “mean plus or minus one standard deviation”.</p>\n<h2 id=\"sec-2\" tabindex=\"-1\">The shape, without a chart</h2>\n<p><code dir=\"ltr\">value_counts</code> with bins counts how many values fall in each range, which is a histogram in text form:</p>\n<div class=\"code-block\" dir=\"ltr\" data-lang=\"python\" data-run=\"python\" data-index=\"1\"><div class=\"code-head\"><span class=\"code-title\">Python</span><span class=\"code-actions\"><button type=\"button\" class=\"code-btn code-run\" data-action=\"run\" data-lang=\"python\" data-index=\"1\" aria-label=\"Run this example\"><svg class=\"icon \" viewBox=\"0 0 24 24\" fill=\"none\" stroke=\"currentColor\" stroke-width=\"2\" stroke-linecap=\"round\" stroke-linejoin=\"round\" aria-hidden=\"true\" focusable=\"false\"><path d=\"M5 5a2 2 0 0 1 3.008-1.728l11.997 6.998a2 2 0 0 1 .003 3.458l-12 7A2 2 0 0 1 5 19z\"/></svg><span>Run</span></button><button type=\"button\" class=\"code-btn\" data-action=\"copy\" aria-label=\"Copy code\"><svg class=\"icon \" viewBox=\"0 0 24 24\" fill=\"none\" stroke=\"currentColor\" stroke-width=\"2\" stroke-linecap=\"round\" stroke-linejoin=\"round\" aria-hidden=\"true\" focusable=\"false\"><rect width=\"14\" height=\"14\" x=\"8\" y=\"8\" rx=\"2\" ry=\"2\"/><path d=\"M4 16c-1.1 0-2-.9-2-2V4c0-1.1.9-2 2-2h10c1.1 0 2 .9 2 2\"/></svg><span>Copy</span></button></span></div><pre class=\"shiki\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#CF222E;--shiki-dark:#FF7B72\">import</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\"> pandas </span><span style=\"color:#CF222E;--shiki-dark:#FF7B72\">as</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\"> pd</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">items </span><span style=\"color:#CF222E;--shiki-dark:#FF7B72\">=</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\"> pd.read_csv(</span><span style=\"color:#0A3069;--shiki-dark:#A5D6FF\">\"order_items.csv\"</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">)</span></span>\n<span class=\"line\"><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">orders </span><span style=\"color:#CF222E;--shiki-dark:#FF7B72\">=</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\"> pd.read_csv(</span><span style=\"color:#0A3069;--shiki-dark:#A5D6FF\">\"orders.csv\"</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">, </span><span style=\"color:#953800;--shiki-dark:#FFA657\">parse_dates</span><span style=\"color:#CF222E;--shiki-dark:#FF7B72\">=</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">[</span><span style=\"color:#0A3069;--shiki-dark:#A5D6FF\">\"order_date\"</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">])</span></span>\n<span class=\"line\"><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">items[</span><span style=\"color:#0A3069;--shiki-dark:#A5D6FF\">\"revenue\"</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">] </span><span style=\"color:#CF222E;--shiki-dark:#FF7B72\">=</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\"> items[</span><span style=\"color:#0A3069;--shiki-dark:#A5D6FF\">\"quantity\"</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">] </span><span style=\"color:#CF222E;--shiki-dark:#FF7B72\">*</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\"> items[</span><span style=\"color:#0A3069;--shiki-dark:#A5D6FF\">\"unit_price\"</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">] </span><span style=\"color:#CF222E;--shiki-dark:#FF7B72\">*</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\"> (</span><span style=\"color:#0550AE;--shiki-dark:#79C0FF\">1</span><span style=\"color:#CF222E;--shiki-dark:#FF7B72\"> -</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\"> items[</span><span style=\"color:#0A3069;--shiki-dark:#A5D6FF\">\"discount\"</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">])</span></span>\n<span class=\"line\"><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">orders </span><span style=\"color:#CF222E;--shiki-dark:#FF7B72\">=</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\"> orders.merge(items.groupby(</span><span style=\"color:#0A3069;--shiki-dark:#A5D6FF\">\"order_id\"</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">, </span><span style=\"color:#953800;--shiki-dark:#FFA657\">as_index</span><span style=\"color:#CF222E;--shiki-dark:#FF7B72\">=</span><span style=\"color:#0550AE;--shiki-dark:#79C0FF\">False</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">)[</span><span style=\"color:#0A3069;--shiki-dark:#A5D6FF\">\"revenue\"</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">].sum(), </span><span style=\"color:#953800;--shiki-dark:#FFA657\">on</span><span style=\"color:#CF222E;--shiki-dark:#FF7B72\">=</span><span style=\"color:#0A3069;--shiki-dark:#A5D6FF\">\"order_id\"</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">, </span><span style=\"color:#953800;--shiki-dark:#FFA657\">validate</span><span style=\"color:#CF222E;--shiki-dark:#FF7B72\">=</span><span style=\"color:#0A3069;--shiki-dark:#A5D6FF\">\"one_to_one\"</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">)</span></span>\n<span class=\"line\"><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">orders </span><span style=\"color:#CF222E;--shiki-dark:#FF7B72\">=</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\"> orders[orders[</span><span style=\"color:#0A3069;--shiki-dark:#A5D6FF\">\"status\"</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">] </span><span style=\"color:#CF222E;--shiki-dark:#FF7B72\">!=</span><span style=\"color:#0A3069;--shiki-dark:#A5D6FF\"> \"cancelled\"</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">]</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#0550AE;--shiki-dark:#79C0FF\">print</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">(orders[</span><span style=\"color:#0A3069;--shiki-dark:#A5D6FF\">\"revenue\"</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">].value_counts(</span><span style=\"color:#953800;--shiki-dark:#FFA657\">bins</span><span style=\"color:#CF222E;--shiki-dark:#FF7B72\">=</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">[</span><span style=\"color:#0550AE;--shiki-dark:#79C0FF\">0</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">, </span><span style=\"color:#0550AE;--shiki-dark:#79C0FF\">50</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">, </span><span style=\"color:#0550AE;--shiki-dark:#79C0FF\">100</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">, </span><span style=\"color:#0550AE;--shiki-dark:#79C0FF\">200</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">, </span><span style=\"color:#0550AE;--shiki-dark:#79C0FF\">400</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">, </span><span style=\"color:#0550AE;--shiki-dark:#79C0FF\">800</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">, </span><span style=\"color:#0550AE;--shiki-dark:#79C0FF\">3000</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">]).sort_index())</span></span></code></pre><div class=\"run-output\" data-run-output=\"1\" hidden></div></div>\n<figure class=\"diagram\"><svg class=\"diagram-svg\" focusable=\"false\" viewBox=\"0 0 700 250\" role=\"img\" aria-labelledby=\"l-pda-5-1-t1\"><defs><marker id=\"arrow-l-pda-5-1-1\" viewBox=\"0 0 10 10\" refX=\"9\" refY=\"5\" markerWidth=\"7\" markerHeight=\"7\" orient=\"auto-start-reverse\"><path d=\"M0 0 L10 5 L0 10 z\" class=\"d-arrowhead\"/></marker></defs>\n  <title id=\"l-pda-5-1-t1\">A bar chart of order counts by value band: 338 orders up to 50 dollars, 398 from 50 to 100, 476 from 100 to 200, 420 from 200 to 400, 256 from 400 to 800 and 73 above 800. A marker shows the median at 147 dollars in the third band and the mean at 228 dollars in the fourth.</title>\n  <path class=\"d-line\" d=\"M40 200 L680 200\"/>\n  <rect class=\"d-box-accent\" x=\"60\" y=\"129\" width=\"80\" height=\"71\"/>\n  <rect class=\"d-box-accent\" x=\"160\" y=\"116\" width=\"80\" height=\"84\"/>\n  <rect class=\"d-box-accent\" x=\"260\" y=\"100\" width=\"80\" height=\"100\"/>\n  <rect class=\"d-box-accent\" x=\"360\" y=\"112\" width=\"80\" height=\"88\"/>\n  <rect class=\"d-box-accent\" x=\"460\" y=\"146\" width=\"80\" height=\"54\"/>\n  <rect class=\"d-box-accent\" x=\"560\" y=\"185\" width=\"80\" height=\"15\"/>\n  <text class=\"d-label\" x=\"100\" y=\"122\" text-anchor=\"middle\">338</text>\n  <text class=\"d-label\" x=\"200\" y=\"109\" text-anchor=\"middle\">398</text>\n  <text class=\"d-label\" x=\"300\" y=\"93\" text-anchor=\"middle\">476</text>\n  <text class=\"d-label\" x=\"400\" y=\"105\" text-anchor=\"middle\">420</text>\n  <text class=\"d-label\" x=\"500\" y=\"139\" text-anchor=\"middle\">256</text>\n  <text class=\"d-label\" x=\"600\" y=\"178\" text-anchor=\"middle\">73</text>\n  <text class=\"d-label-muted\" x=\"100\" y=\"220\" text-anchor=\"middle\">0-50</text>\n  <text class=\"d-label-muted\" x=\"200\" y=\"220\" text-anchor=\"middle\">50-100</text>\n  <text class=\"d-label-muted\" x=\"300\" y=\"220\" text-anchor=\"middle\">100-200</text>\n  <text class=\"d-label-muted\" x=\"400\" y=\"220\" text-anchor=\"middle\">200-400</text>\n  <text class=\"d-label-muted\" x=\"500\" y=\"220\" text-anchor=\"middle\">400-800</text>\n  <text class=\"d-label-muted\" x=\"600\" y=\"220\" text-anchor=\"middle\">800+</text>\n  <path class=\"d-line d-dashed\" d=\"M318 40 L318 200\"/>\n  <text class=\"d-label-strong\" x=\"318\" y=\"32\" text-anchor=\"middle\">median 147</text>\n  <path class=\"d-arrow d-dashed\" d=\"M374 60 L374 200\"/>\n  <text class=\"d-label-strong\" x=\"420\" y=\"56\" text-anchor=\"middle\">mean 228</text>\n  <text class=\"d-label-muted\" x=\"350\" y=\"244\" text-anchor=\"middle\">order value, dollars (bands are not equal width)</text>\n</svg><figcaption>Order values are skewed: most orders are small, a long tail is large</figcaption></figure>\n<p>Most orders sit between 50 and 400 dollars, and a thinning tail runs past 800. Notice the bands get wider as they go right; equal-width bands of 100 dollars would squash nearly everything into the first few bars.</p>\n<h2 id=\"sec-3\" tabindex=\"-1\">Outliers: flag, then decide</h2>\n<p>A common rule of thumb flags values more than 1.5 interquartile ranges (IQR, the distance between the quartiles) beyond the quartiles:</p>\n<div class=\"code-block\" dir=\"ltr\" data-lang=\"python\" data-run=\"python\" data-index=\"2\"><div class=\"code-head\"><span class=\"code-title\">Python</span><span class=\"code-actions\"><button type=\"button\" class=\"code-btn code-run\" data-action=\"run\" data-lang=\"python\" data-index=\"2\" aria-label=\"Run this example\"><svg class=\"icon \" viewBox=\"0 0 24 24\" fill=\"none\" stroke=\"currentColor\" stroke-width=\"2\" stroke-linecap=\"round\" stroke-linejoin=\"round\" aria-hidden=\"true\" focusable=\"false\"><path d=\"M5 5a2 2 0 0 1 3.008-1.728l11.997 6.998a2 2 0 0 1 .003 3.458l-12 7A2 2 0 0 1 5 19z\"/></svg><span>Run</span></button><button type=\"button\" class=\"code-btn\" data-action=\"copy\" aria-label=\"Copy code\"><svg class=\"icon \" viewBox=\"0 0 24 24\" fill=\"none\" stroke=\"currentColor\" stroke-width=\"2\" stroke-linecap=\"round\" stroke-linejoin=\"round\" aria-hidden=\"true\" focusable=\"false\"><rect width=\"14\" height=\"14\" x=\"8\" y=\"8\" rx=\"2\" ry=\"2\"/><path d=\"M4 16c-1.1 0-2-.9-2-2V4c0-1.1.9-2 2-2h10c1.1 0 2 .9 2 2\"/></svg><span>Copy</span></button></span></div><pre class=\"shiki\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#CF222E;--shiki-dark:#FF7B72\">import</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\"> pandas </span><span style=\"color:#CF222E;--shiki-dark:#FF7B72\">as</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\"> pd</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">items </span><span style=\"color:#CF222E;--shiki-dark:#FF7B72\">=</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\"> pd.read_csv(</span><span style=\"color:#0A3069;--shiki-dark:#A5D6FF\">\"order_items.csv\"</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">)</span></span>\n<span class=\"line\"><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">orders </span><span style=\"color:#CF222E;--shiki-dark:#FF7B72\">=</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\"> pd.read_csv(</span><span style=\"color:#0A3069;--shiki-dark:#A5D6FF\">\"orders.csv\"</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">, </span><span style=\"color:#953800;--shiki-dark:#FFA657\">parse_dates</span><span style=\"color:#CF222E;--shiki-dark:#FF7B72\">=</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">[</span><span style=\"color:#0A3069;--shiki-dark:#A5D6FF\">\"order_date\"</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">])</span></span>\n<span class=\"line\"><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">items[</span><span style=\"color:#0A3069;--shiki-dark:#A5D6FF\">\"revenue\"</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">] </span><span style=\"color:#CF222E;--shiki-dark:#FF7B72\">=</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\"> items[</span><span style=\"color:#0A3069;--shiki-dark:#A5D6FF\">\"quantity\"</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">] </span><span style=\"color:#CF222E;--shiki-dark:#FF7B72\">*</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\"> items[</span><span style=\"color:#0A3069;--shiki-dark:#A5D6FF\">\"unit_price\"</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">] </span><span style=\"color:#CF222E;--shiki-dark:#FF7B72\">*</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\"> (</span><span style=\"color:#0550AE;--shiki-dark:#79C0FF\">1</span><span style=\"color:#CF222E;--shiki-dark:#FF7B72\"> -</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\"> items[</span><span style=\"color:#0A3069;--shiki-dark:#A5D6FF\">\"discount\"</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">])</span></span>\n<span class=\"line\"><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">orders </span><span style=\"color:#CF222E;--shiki-dark:#FF7B72\">=</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\"> orders.merge(items.groupby(</span><span style=\"color:#0A3069;--shiki-dark:#A5D6FF\">\"order_id\"</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">, </span><span style=\"color:#953800;--shiki-dark:#FFA657\">as_index</span><span style=\"color:#CF222E;--shiki-dark:#FF7B72\">=</span><span style=\"color:#0550AE;--shiki-dark:#79C0FF\">False</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">)[</span><span style=\"color:#0A3069;--shiki-dark:#A5D6FF\">\"revenue\"</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">].sum(), </span><span style=\"color:#953800;--shiki-dark:#FFA657\">on</span><span style=\"color:#CF222E;--shiki-dark:#FF7B72\">=</span><span style=\"color:#0A3069;--shiki-dark:#A5D6FF\">\"order_id\"</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">, </span><span style=\"color:#953800;--shiki-dark:#FFA657\">validate</span><span style=\"color:#CF222E;--shiki-dark:#FF7B72\">=</span><span style=\"color:#0A3069;--shiki-dark:#A5D6FF\">\"one_to_one\"</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">)</span></span>\n<span class=\"line\"><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">orders </span><span style=\"color:#CF222E;--shiki-dark:#FF7B72\">=</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\"> orders[orders[</span><span style=\"color:#0A3069;--shiki-dark:#A5D6FF\">\"status\"</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">] </span><span style=\"color:#CF222E;--shiki-dark:#FF7B72\">!=</span><span style=\"color:#0A3069;--shiki-dark:#A5D6FF\"> \"cancelled\"</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">]</span></span>\n<span class=\"line\"><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">value </span><span style=\"color:#CF222E;--shiki-dark:#FF7B72\">=</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\"> orders[</span><span style=\"color:#0A3069;--shiki-dark:#A5D6FF\">\"revenue\"</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">]</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">q1, q3 </span><span style=\"color:#CF222E;--shiki-dark:#FF7B72\">=</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\"> value.quantile([</span><span style=\"color:#0550AE;--shiki-dark:#79C0FF\">0.25</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">, </span><span style=\"color:#0550AE;--shiki-dark:#79C0FF\">0.75</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">])</span></span>\n<span class=\"line\"><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">upper </span><span style=\"color:#CF222E;--shiki-dark:#FF7B72\">=</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\"> q3 </span><span style=\"color:#CF222E;--shiki-dark:#FF7B72\">+</span><span style=\"color:#0550AE;--shiki-dark:#79C0FF\"> 1.5</span><span style=\"color:#CF222E;--shiki-dark:#FF7B72\"> *</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\"> (q3 </span><span style=\"color:#CF222E;--shiki-dark:#FF7B72\">-</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\"> q1)</span></span>\n<span class=\"line\"><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">big </span><span style=\"color:#CF222E;--shiki-dark:#FF7B72\">=</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\"> value </span><span style=\"color:#CF222E;--shiki-dark:#FF7B72\">></span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\"> upper</span></span>\n<span class=\"line\"><span style=\"color:#0550AE;--shiki-dark:#79C0FF\">print</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">(</span><span style=\"color:#CF222E;--shiki-dark:#FF7B72\">f</span><span style=\"color:#0A3069;--shiki-dark:#A5D6FF\">\"fence </span><span style=\"color:#CF222E;--shiki-dark:#FF7B72\">{</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">upper</span><span style=\"color:#CF222E;--shiki-dark:#FF7B72\">:.2f}</span><span style=\"color:#0A3069;--shiki-dark:#A5D6FF\">: </span><span style=\"color:#CF222E;--shiki-dark:#FF7B72\">{</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">big.sum()</span><span style=\"color:#CF222E;--shiki-dark:#FF7B72\">}</span><span style=\"color:#0A3069;--shiki-dark:#A5D6FF\"> orders, </span><span style=\"color:#CF222E;--shiki-dark:#FF7B72\">{</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">value[big].sum() </span><span style=\"color:#CF222E;--shiki-dark:#FF7B72\">/</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\"> value.sum()</span><span style=\"color:#CF222E;--shiki-dark:#FF7B72\">:.1%}</span><span style=\"color:#0A3069;--shiki-dark:#A5D6FF\"> of revenue\"</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">)</span></span></code></pre><div class=\"run-output\" data-run-output=\"2\" hidden></div></div>\n<p>125 orders are above 656.62 dollars, and together they bring in 25.9% of revenue. These are not errors to clean away: they are multi-item orders of desks, chairs and dumbbells, real and important. An outlier rule tells you where to look. Data entry errors, such as a total with a missing decimal point, get fixed; genuine extremes stay, and you check whether your conclusion survives with and without them.</p>\n<aside class=\"callout callout-mistake\" role=\"note\"><p class=\"callout-title\"><svg class=\"icon \" viewBox=\"0 0 24 24\" fill=\"none\" stroke=\"currentColor\" stroke-width=\"2\" stroke-linecap=\"round\" stroke-linejoin=\"round\" aria-hidden=\"true\" focusable=\"false\"><path d=\"m21.73 18-8-14a2 2 0 0 0-3.48 0l-8 14A2 2 0 0 0 4 21h16a2 2 0 0 0 1.73-3\"/><path d=\"M12 9v4\"/><path d=\"M12 17h.01\"/></svg><span><span class=\"callout-kind\">Common mistake</span><span class=\"callout-sep\" aria-hidden=\"true\"> · </span>Comparing means of small, skewed groups</span></p>\n<p>One 2,000-dollar order added to a 30-order group moves its mean by about 60 dollars. Before saying “marketplace customers spend less”, put the count, the mean and the median side by side: <code dir=\"ltr\">orders.groupby(&quot;channel&quot;)[&quot;revenue&quot;].agg([&quot;count&quot;, &quot;mean&quot;, &quot;median&quot;])</code>. Here the means run from 214 (marketplace) to 236 dollars (web), but the highest median belongs to the app, 153 against the web’s 147: the two measures do not even agree on which channel has the bigger typical order.</p>\n</aside>\n<h3 id=\"sec-4\" tabindex=\"-1\">The same lens over time</h3>\n<p>Distributions can be compared across periods too, and this one carries a clue for the case study. If Q4 2025 revenue jumped because customers bought bigger baskets, the typical order should have grown:</p>\n<div class=\"code-block\" dir=\"ltr\" data-lang=\"python\" data-run=\"python\" data-index=\"3\"><div class=\"code-head\"><span class=\"code-title\">Python</span><span class=\"code-actions\"><button type=\"button\" class=\"code-btn code-run\" data-action=\"run\" data-lang=\"python\" data-index=\"3\" aria-label=\"Run this example\"><svg class=\"icon \" viewBox=\"0 0 24 24\" fill=\"none\" stroke=\"currentColor\" stroke-width=\"2\" stroke-linecap=\"round\" stroke-linejoin=\"round\" aria-hidden=\"true\" focusable=\"false\"><path d=\"M5 5a2 2 0 0 1 3.008-1.728l11.997 6.998a2 2 0 0 1 .003 3.458l-12 7A2 2 0 0 1 5 19z\"/></svg><span>Run</span></button><button type=\"button\" class=\"code-btn\" data-action=\"copy\" aria-label=\"Copy code\"><svg class=\"icon \" viewBox=\"0 0 24 24\" fill=\"none\" stroke=\"currentColor\" stroke-width=\"2\" stroke-linecap=\"round\" stroke-linejoin=\"round\" aria-hidden=\"true\" focusable=\"false\"><rect width=\"14\" height=\"14\" x=\"8\" y=\"8\" rx=\"2\" ry=\"2\"/><path d=\"M4 16c-1.1 0-2-.9-2-2V4c0-1.1.9-2 2-2h10c1.1 0 2 .9 2 2\"/></svg><span>Copy</span></button></span></div><pre class=\"shiki\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#CF222E;--shiki-dark:#FF7B72\">import</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\"> pandas </span><span style=\"color:#CF222E;--shiki-dark:#FF7B72\">as</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\"> pd</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">items </span><span style=\"color:#CF222E;--shiki-dark:#FF7B72\">=</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\"> pd.read_csv(</span><span style=\"color:#0A3069;--shiki-dark:#A5D6FF\">\"order_items.csv\"</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">)</span></span>\n<span class=\"line\"><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">orders </span><span style=\"color:#CF222E;--shiki-dark:#FF7B72\">=</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\"> pd.read_csv(</span><span style=\"color:#0A3069;--shiki-dark:#A5D6FF\">\"orders.csv\"</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">, </span><span style=\"color:#953800;--shiki-dark:#FFA657\">parse_dates</span><span style=\"color:#CF222E;--shiki-dark:#FF7B72\">=</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">[</span><span style=\"color:#0A3069;--shiki-dark:#A5D6FF\">\"order_date\"</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">])</span></span>\n<span class=\"line\"><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">items[</span><span style=\"color:#0A3069;--shiki-dark:#A5D6FF\">\"revenue\"</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">] </span><span style=\"color:#CF222E;--shiki-dark:#FF7B72\">=</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\"> items[</span><span style=\"color:#0A3069;--shiki-dark:#A5D6FF\">\"quantity\"</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">] </span><span style=\"color:#CF222E;--shiki-dark:#FF7B72\">*</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\"> items[</span><span style=\"color:#0A3069;--shiki-dark:#A5D6FF\">\"unit_price\"</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">] </span><span style=\"color:#CF222E;--shiki-dark:#FF7B72\">*</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\"> (</span><span style=\"color:#0550AE;--shiki-dark:#79C0FF\">1</span><span style=\"color:#CF222E;--shiki-dark:#FF7B72\"> -</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\"> items[</span><span style=\"color:#0A3069;--shiki-dark:#A5D6FF\">\"discount\"</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">])</span></span>\n<span class=\"line\"><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">orders </span><span style=\"color:#CF222E;--shiki-dark:#FF7B72\">=</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\"> orders.merge(items.groupby(</span><span style=\"color:#0A3069;--shiki-dark:#A5D6FF\">\"order_id\"</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">, </span><span style=\"color:#953800;--shiki-dark:#FFA657\">as_index</span><span style=\"color:#CF222E;--shiki-dark:#FF7B72\">=</span><span style=\"color:#0550AE;--shiki-dark:#79C0FF\">False</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">)[</span><span style=\"color:#0A3069;--shiki-dark:#A5D6FF\">\"revenue\"</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">].sum(), </span><span style=\"color:#953800;--shiki-dark:#FFA657\">on</span><span style=\"color:#CF222E;--shiki-dark:#FF7B72\">=</span><span style=\"color:#0A3069;--shiki-dark:#A5D6FF\">\"order_id\"</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">, </span><span style=\"color:#953800;--shiki-dark:#FFA657\">validate</span><span style=\"color:#CF222E;--shiki-dark:#FF7B72\">=</span><span style=\"color:#0A3069;--shiki-dark:#A5D6FF\">\"one_to_one\"</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">)</span></span>\n<span class=\"line\"><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">orders </span><span style=\"color:#CF222E;--shiki-dark:#FF7B72\">=</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\"> orders[orders[</span><span style=\"color:#0A3069;--shiki-dark:#A5D6FF\">\"status\"</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">] </span><span style=\"color:#CF222E;--shiki-dark:#FF7B72\">!=</span><span style=\"color:#0A3069;--shiki-dark:#A5D6FF\"> \"cancelled\"</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">]</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">quarter </span><span style=\"color:#CF222E;--shiki-dark:#FF7B72\">=</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\"> orders[</span><span style=\"color:#0A3069;--shiki-dark:#A5D6FF\">\"order_date\"</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">].dt.to_period(</span><span style=\"color:#0A3069;--shiki-dark:#A5D6FF\">\"Q\"</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">)</span></span>\n<span class=\"line\"><span style=\"color:#0550AE;--shiki-dark:#79C0FF\">print</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">(orders.groupby(quarter)[</span><span style=\"color:#0A3069;--shiki-dark:#A5D6FF\">\"revenue\"</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">].agg([</span><span style=\"color:#0A3069;--shiki-dark:#A5D6FF\">\"count\"</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">, </span><span style=\"color:#0A3069;--shiki-dark:#A5D6FF\">\"median\"</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">]).round(</span><span style=\"color:#0550AE;--shiki-dark:#79C0FF\">1</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">).tail(</span><span style=\"color:#0550AE;--shiki-dark:#79C0FF\">4</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">))</span></span></code></pre><div class=\"run-output\" data-run-output=\"3\" hidden></div></div>\n<p>It did not. The median order in Q4 2025 was 151.70 dollars, slightly <strong>below</strong> Q3’s 159.20, while the number of orders doubled. Whatever happened in Q4, it was about how many orders arrived, not how big they were. One table of medians has already ruled out a popular explanation.</p>\n<h2 id=\"sec-5\" tabindex=\"-1\">Correlation: moving together is not causing</h2>\n<p><code dir=\"ltr\">corr()</code> measures how strongly two numeric columns move together, from -1 (perfectly opposite) through 0 (no linear relation) to 1 (perfectly together). <code dir=\"ltr\">churn.csv</code> has one row per customer and a <code dir=\"ltr\">churned</code> flag:</p>\n<div class=\"code-block\" dir=\"ltr\" data-lang=\"python\" data-run=\"python\" data-index=\"4\"><div class=\"code-head\"><span class=\"code-title\">Python</span><span class=\"code-actions\"><button type=\"button\" class=\"code-btn code-run\" data-action=\"run\" data-lang=\"python\" data-index=\"4\" aria-label=\"Run this example\"><svg class=\"icon \" viewBox=\"0 0 24 24\" fill=\"none\" stroke=\"currentColor\" stroke-width=\"2\" stroke-linecap=\"round\" stroke-linejoin=\"round\" aria-hidden=\"true\" focusable=\"false\"><path d=\"M5 5a2 2 0 0 1 3.008-1.728l11.997 6.998a2 2 0 0 1 .003 3.458l-12 7A2 2 0 0 1 5 19z\"/></svg><span>Run</span></button><button type=\"button\" class=\"code-btn\" data-action=\"copy\" aria-label=\"Copy code\"><svg class=\"icon \" viewBox=\"0 0 24 24\" fill=\"none\" stroke=\"currentColor\" stroke-width=\"2\" stroke-linecap=\"round\" stroke-linejoin=\"round\" aria-hidden=\"true\" focusable=\"false\"><rect width=\"14\" height=\"14\" x=\"8\" y=\"8\" rx=\"2\" ry=\"2\"/><path d=\"M4 16c-1.1 0-2-.9-2-2V4c0-1.1.9-2 2-2h10c1.1 0 2 .9 2 2\"/></svg><span>Copy</span></button></span></div><pre class=\"shiki\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#CF222E;--shiki-dark:#FF7B72\">import</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\"> pandas </span><span style=\"color:#CF222E;--shiki-dark:#FF7B72\">as</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\"> pd</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">churn </span><span style=\"color:#CF222E;--shiki-dark:#FF7B72\">=</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\"> pd.read_csv(</span><span style=\"color:#0A3069;--shiki-dark:#A5D6FF\">\"churn.csv\"</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">)</span></span>\n<span class=\"line\"><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">cols </span><span style=\"color:#CF222E;--shiki-dark:#FF7B72\">=</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\"> [</span><span style=\"color:#0A3069;--shiki-dark:#A5D6FF\">\"tenure_days\"</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">, </span><span style=\"color:#0A3069;--shiki-dark:#A5D6FF\">\"orders\"</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">, </span><span style=\"color:#0A3069;--shiki-dark:#A5D6FF\">\"days_since_last_order\"</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">, </span><span style=\"color:#0A3069;--shiki-dark:#A5D6FF\">\"support_tickets\"</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">, </span><span style=\"color:#0A3069;--shiki-dark:#A5D6FF\">\"churned\"</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">]</span></span>\n<span class=\"line\"><span style=\"color:#0550AE;--shiki-dark:#79C0FF\">print</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">(churn[cols].corr().round(</span><span style=\"color:#0550AE;--shiki-dark:#79C0FF\">2</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">)[</span><span style=\"color:#0A3069;--shiki-dark:#A5D6FF\">\"churned\"</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">])</span></span>\n<span class=\"line\"><span style=\"color:#0550AE;--shiki-dark:#79C0FF\">print</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">(churn.groupby(</span><span style=\"color:#0A3069;--shiki-dark:#A5D6FF\">\"churned\"</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">)[</span><span style=\"color:#0A3069;--shiki-dark:#A5D6FF\">\"days_since_last_order\"</span><span style=\"color:#1F2328;--shiki-dark:#E6EDF3\">].median())</span></span></code></pre><div class=\"run-output\" data-run-output=\"4\" hidden></div></div>\n<p>Customers who churned had gone far longer since their last order: a median of 214 days against 46, and a correlation of 0.45. That makes the gap a good warning signal. It does not show that the gap <strong>causes</strong> churn; a customer who has already lost interest stops ordering and churns, and both measurements reflect that one fact. Treat correlations as leads to investigate, and keep the word “causes” for experiments.</p>\n<p>In the exercise you describe Cartwheel’s order values yourself. Next you turn numbers like these into charts.</p>\n","toc":[{"id":"sec-1","text":"Mean, median and quantiles","level":2},{"id":"sec-2","text":"The shape, without a chart","level":2},{"id":"sec-3","text":"Outliers: flag, then decide","level":2},{"id":"sec-4","text":"The same lens over time","level":3},{"id":"sec-5","text":"Correlation: moving together is not causing","level":2}],"takeaways":["In skewed data such as order values, the mean sits well above the median; report the median (and a few quantiles) when you describe a typical value.","<code dir=\"ltr\">quantile([0.25, 0.75])</code> gives the quartiles; values beyond 1.5 times the interquartile range from them are worth a look, not automatic deletion.","<code dir=\"ltr\">value_counts(bins=[...])</code> is a quick text histogram that shows the shape of a distribution without a chart.","Compare groups with both their mean and median and their sizes, so one big order or one tiny group does not drive the conclusion.","<code dir=\"ltr\">corr()</code> measures how two columns move together; a strong correlation is a lead to investigate, never proof that one causes the other."],"further":[{"title":"Descriptive statistics in pandas","url":"https://pandas.pydata.org/docs/user_guide/basics.html#descriptive-statistics"},{"title":"pandas.Series.quantile","url":"https://pandas.pydata.org/docs/reference/api/pandas.Series.quantile.html"},{"title":"pandas.DataFrame.corr","url":"https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.corr.html"}],"quiz":[{"q":"<p>Cartwheel’s order values have a mean of 228 dollars and a median of 147. What does that tell you about the distribution?</p>","options":[{"text":"It is skewed to the right; a minority of large orders pulls the mean above the typical order.","why":"Correct. Half of all orders are below 147 dollars, and a long tail of big orders lifts the mean by 81 dollars."},{"text":"The data contains errors, because mean and median should be equal.","why":"They are equal only for symmetric distributions. Money, durations and counts are usually skewed, with no error involved."},{"text":"Most orders are close to 228 dollars.","why":"The median says half the orders are below 147, so 228 is not where most orders sit."},{"text":"There are more small orders than large ones, so the mean is too low.","why":"Many small orders and a few huge ones push the mean <strong>up</strong>, not down."}],"answer":0},{"q":"<p>You find 125 orders above the 1.5 × IQR upper fence. What should you do with them?</p>","options":[{"text":"Delete them; outliers distort averages.","why":"These are real orders worth 26% of revenue. Deleting them would misrepresent the business."},{"text":"Replace them with the median.","why":"Replacing real sales with a typical value is inventing data, and it would erase most of that quarter of revenue."},{"text":"Inspect them, check they are genuine, and report results with and without them if they change the conclusion.","why":"Correct. The fence flags values for review. Genuine large orders are part of the story, often the most interesting part."}],"answer":2},{"q":"<p>In <code dir=\"ltr\">churn.csv</code>, <code dir=\"ltr\">days_since_last_order</code> has a correlation of 0.45 with <code dir=\"ltr\">churned</code>. Which statement is justified?</p>","options":[{"text":"Long gaps between orders cause customers to churn.","why":"Correlation cannot establish cause. Both may reflect a third factor, such as the customer losing interest."},{"text":"Customers who churned tended to have longer gaps since their last order, so the gap is a useful warning signal.","why":"Correct. A moderate positive correlation supports using it as an indicator, which is what the ML course builds on, without claiming a cause."},{"text":"45% of customers with long gaps churned.","why":"A correlation coefficient is not a percentage of customers; it measures the strength of a linear relationship between -1 and 1."},{"text":"The relationship is too weak to be useful.","why":"0.45 is a moderate correlation, and the medians (46 days for active customers, 214 for churned) show a large practical difference."}],"answer":1}],"exercise":{"kind":"python","files":["order_items.csv","orders.csv"],"starter":"import pandas as pd\n\nitems = pd.read_csv(\"order_items.csv\")\norders = pd.read_csv(\"orders.csv\", parse_dates=[\"order_date\"])\nitems[\"revenue\"] = items[\"quantity\"] * items[\"unit_price\"] * (1 - items[\"discount\"])\norders = orders.merge(items.groupby(\"order_id\", as_index=False)[\"revenue\"].sum(), on=\"order_id\", validate=\"one_to_one\")\norders = orders[orders[\"status\"] != \"cancelled\"]\nvalue = orders[\"revenue\"]\n\n# 1. Median and 90th percentile of order value, each rounded to 2 decimals\nmedian_value = None\np90_value = None\n\n# 2. Share of orders below the mean order value, rounded to 3 decimals\nbelow_mean_share = None\n\n# 3. Upper outlier fence (Q3 + 1.5 * IQR) and the number of orders above it\nupper_fence = None\nn_outliers = None\n\n# 4. A DataFrame indexed by channel with columns count, mean, median of order value\nby_channel = None\n","solution":"import pandas as pd\n\nitems = pd.read_csv(\"order_items.csv\")\norders = pd.read_csv(\"orders.csv\", parse_dates=[\"order_date\"])\nitems[\"revenue\"] = items[\"quantity\"] * items[\"unit_price\"] * (1 - items[\"discount\"])\norders = orders.merge(items.groupby(\"order_id\", as_index=False)[\"revenue\"].sum(), on=\"order_id\", validate=\"one_to_one\")\norders = orders[orders[\"status\"] != \"cancelled\"]\nvalue = orders[\"revenue\"]\n\nmedian_value = round(value.median(), 2)\np90_value = round(value.quantile(0.9), 2)\n\nbelow_mean_share = round((value < value.mean()).mean(), 3)\n\nq1, q3 = value.quantile([0.25, 0.75])\nupper_fence = q3 + 1.5 * (q3 - q1)\nn_outliers = int((value > upper_fence).sum())\n\nby_channel = orders.groupby(\"channel\")[\"revenue\"].agg([\"count\", \"mean\", \"median\"])\nprint(by_channel.round(1))\n","checks":[{"id":"median","test":"abs(float(median_value) - 147.26) < 0.01"},{"id":"p90","test":"abs(float(p90_value) - 505.0) < 0.01"},{"id":"below-mean","test":"round(float(below_mean_share), 3) == 0.663"},{"id":"fence","test":"abs(float(upper_fence) - 656.625) < 0.01 and int(n_outliers) == 125"},{"id":"by-channel","test":"isinstance(by_channel, pd.DataFrame) and list(by_channel.columns) == ['count', 'mean', 'median'] and int(by_channel.loc['web', 'count']) == 1089 and abs(float(by_channel.loc['marketplace', 'median']) - 132.2) < 0.1"}],"title":"Describe a typical order","prompt":"<p><code dir=\"ltr\">value</code> holds the revenue of every non-cancelled order. Set <code dir=\"ltr\">median_value</code> and <code dir=\"ltr\">p90_value</code> (the 90th percentile), each rounded to 2 decimals. Set <code dir=\"ltr\">below_mean_share</code> to the share of orders whose value is below the mean, rounded to 3 decimals. Compute <code dir=\"ltr\">upper_fence</code> as Q3 + 1.5 × (Q3 − Q1) and <code dir=\"ltr\">n_outliers</code> as the number of orders above it. Finally build <code dir=\"ltr\">by_channel</code>, a DataFrame indexed by channel with the columns <code dir=\"ltr\">count</code>, <code dir=\"ltr\">mean</code> and <code dir=\"ltr\">median</code> of order value, in that order.</p>\n","hints":["<code dir=\"ltr\">value.median()</code> and <code dir=\"ltr\">value.quantile(0.9)</code>; wrap each in <code dir=\"ltr\">round(..., 2)</code>.","A share is the mean of a boolean Series: <code dir=\"ltr\">(value &lt; value.mean()).mean()</code>. <code dir=\"ltr\">value.quantile([0.25, 0.75])</code> returns both quartiles, which you can unpack into two names.","<code dir=\"ltr\">orders.groupby(&quot;channel&quot;)[&quot;revenue&quot;].agg([&quot;count&quot;, &quot;mean&quot;, &quot;median&quot;])</code> gives the three columns in that order."],"explanation":"<p>The median order is 147.26 dollars, and 66.3% of orders are below the 227.77 mean: the clearest possible demonstration that the mean is not the typical order in skewed data. The fence at 656.62 flags 125 large orders, which you would inspect rather than delete. The channel table puts sizes next to both averages, so you can see that the means and the medians do not even rank the channels the same way, and that the marketplace group is the smallest at 208 orders.</p>\n","checkLabels":{"median":"<code dir=\"ltr\">median_value</code> is 147.26","p90":"<code dir=\"ltr\">p90_value</code> is the 90th percentile","below-mean":"66.3% of orders are below the mean","fence":"The fence is 656.62 with 125 orders above it","by-channel":"<code dir=\"ltr\">by_channel</code> has count, mean and median per channel"}},"words":684,"readMinutes":3}