{ "cells": [ { "cell_type": "markdown", "id": "a7009d86", "metadata": {}, "source": [ "# 第 11 讲｜pandas 数据清洗：缺失值、重复值与数据转换\n", "\n", "**参考教材：** `da_7f_clean.pdf` \n", "**适用基础：** 已学过 pandas 数据读取、列选择、布尔筛选和基础统计。 \n", "**本讲目标：** 识别并处理缺失、重复、异常和不规范文本，把原始数据整理成可以继续统计分析的表格。\n", "\n", "数据清洗不是“让表格看起来整齐”，而是让每一列的含义、类型和取值规则一致。进行删除或替换前，应先统计问题数量，并保留可核对的原始数据。" ] }, { "cell_type": "markdown", "id": "53c647c4", "metadata": {}, "source": [ "## 训练路线\n", "\n", "1. 检测、删除和填充缺失值；\n", "2. 检测重复记录，并按业务字段去重；\n", "3. 使用 `map`、`replace`、`rename` 统一数据和值标签；\n", "4. 使用 `cut` 分箱，检测和限制异常值；\n", "5. 完成随机抽样与分类虚拟变量转换；\n", "6. 使用 `.str` 批量清理、替换、筛选和拆分文本。\n", "\n", "每个知识点均按“示例代码 → 对应解读 → 动手练习”展开。练习单元格保留空白，补全后再运行。" ] }, { "cell_type": "markdown", "id": "e697a7a4", "metadata": {}, "source": [ "## 教材内容取舍\n", "\n", "本课件使用 Conda `skills` 环境中的 pandas 3.0 运行验证。主线保留竞赛和日常数据处理中高频、易解释的清洗操作，并尽量让每段代码只承担一个明确任务。\n", "\n", "复杂正则表达式、`qcut` 分位数分箱、随机置换、可空扩展类型、Categorical 内部编码及分类方法放在末尾“阅读拓展”。所有清洗结果默认保存到新变量；条件修改时应继续使用单次 `.loc[...] = ...`，避免链式赋值。" ] }, { "cell_type": "markdown", "id": "38684052", "metadata": {}, "source": [ "## 1. 检测缺失值：isna 与 notna\n", "\n", "pandas 使用 `NaN`、`None` 或 `pd.NA` 表示缺失数据：\n", "\n", "- `.isna()`：缺失位置返回 `True`；\n", "- `.notna()`：非缺失位置返回 `True`；\n", "- `.isna().sum()`：按列统计缺失值数量。\n", "\n", "`pd.NA` 是 pandas 提供的统一缺失值标记。" ] }, { "cell_type": "markdown", "id": "610b138e", "metadata": {}, "source": [ "### 示例代码" ] }, { "cell_type": "code", "execution_count": 1, "id": "63a3ac96", "metadata": { "execution": { "iopub.execute_input": "2026-07-30T13:37:06.546308Z", "iopub.status.busy": "2026-07-30T13:37:06.546171Z", "iopub.status.idle": "2026-07-30T13:37:07.197620Z", "shell.execute_reply": "2026-07-30T13:37:07.197108Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "缺失位置：\n", " 姓名 城市 成绩\n", "0 False False False\n", "1 False True False\n", "2 False False True\n", "3 False False False\n", "各列缺失数量：\n", "姓名 0\n", "城市 1\n", "成绩 1\n", "dtype: int64\n", "成绩非缺失的位置：\n", "0 True\n", "1 True\n", "2 False\n", "3 True\n", "Name: 成绩, dtype: bool\n" ] } ], "source": [ "import pandas as pd\n", "\n", "students = pd.DataFrame({\n", " \"姓名\": [\"小林\", \"小周\", \"小郑\", \"小王\"],\n", " \"城市\": [\"杭州\", None, \"宁波\", \"温州\"],\n", " \"成绩\": [82, 76, pd.NA, 68]\n", "})\n", "\n", "print(\"缺失位置：\")\n", "print(students.isna())\n", "print(\"各列缺失数量：\")\n", "print(students.isna().sum())\n", "print(\"成绩非缺失的位置：\")\n", "print(students[\"成绩\"].notna())" ] }, { "cell_type": "markdown", "id": "269eac19", "metadata": {}, "source": [ "### 代码解读（对应上方示例）\n", "\n", "- 城市列的小周位置是 `None`，因此 `isna()` 结果为 `True`。\n", "- 成绩列的小郑位置是 `pd.NA`，同样被识别为缺失。\n", "- `.isna().sum()` 把 `True` 按 1 统计，所以城市列和成绩列各有 1 个缺失值。\n", "- `.notna()` 与 `.isna()` 相反，可用于保留已有成绩的记录。" ] }, { "cell_type": "markdown", "id": "393a6792", "metadata": {}, "source": [ "### 动手练习" ] }, { "cell_type": "code", "execution_count": 2, "id": "3818c600", "metadata": { "execution": { "iopub.execute_input": "2026-07-30T13:37:07.199462Z", "iopub.status.busy": "2026-07-30T13:37:07.199321Z", "iopub.status.idle": "2026-07-30T13:37:07.201412Z", "shell.execute_reply": "2026-07-30T13:37:07.200821Z" } }, "outputs": [], "source": [ "# 统计下表每一列的缺失数量，并判断哪些销量不是缺失值\n", "# sales = pd.DataFrame({\n", "# \"城市\": [\"A\", None, \"C\"], \"销量\": [100, pd.NA, 120]\n", "# })\n", "# print(__________)\n", "# print(__________)" ] }, { "cell_type": "markdown", "id": "20f77566", "metadata": {}, "source": [ "## 2. 用 dropna 删除缺失记录\n", "\n", "`.dropna()` 返回删除含缺失值行后的新 DataFrame。\n", "\n", "`subset=[列名列表]` 只检查指定的关键列。这样可以删除关键字段缺失的行，同时允许备注等非关键列为空。" ] }, { "cell_type": "markdown", "id": "28338186", "metadata": {}, "source": [ "### 示例代码" ] }, { "cell_type": "code", "execution_count": 3, "id": "703f2ba2", "metadata": { "execution": { "iopub.execute_input": "2026-07-30T13:37:07.203381Z", "iopub.status.busy": "2026-07-30T13:37:07.203235Z", "iopub.status.idle": "2026-07-30T13:37:07.214626Z", "shell.execute_reply": "2026-07-30T13:37:07.214149Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "所有列都完整的行：\n", " 姓名 成绩 备注\n", "1 小周 76.0 转专业\n", "只要求成绩不缺失：\n", " 姓名 成绩 备注\n", "0 小林 82.0 NaN\n", "1 小周 76.0 转专业\n", "3 小王 68.0 NaN\n" ] } ], "source": [ "students = pd.DataFrame({\n", " \"姓名\": [\"小林\", \"小周\", \"小郑\", \"小王\"],\n", " \"成绩\": [82, 76, None, 68],\n", " \"备注\": [None, \"转专业\", \"补考\", None]\n", "})\n", "\n", "complete_rows = students.dropna()\n", "score_required = students.dropna(subset=[\"成绩\"])\n", "\n", "print(\"所有列都完整的行：\")\n", "print(complete_rows)\n", "print(\"只要求成绩不缺失：\")\n", "print(score_required)" ] }, { "cell_type": "markdown", "id": "d8115204", "metadata": {}, "source": [ "### 代码解读（对应上方示例）\n", "\n", "- `students.dropna()` 要求一行中所有列都非缺失，因此只保留小周。\n", "- `dropna(subset=[\"成绩\"])` 只把成绩看作必填字段，删除小郑，保留其他三人。\n", "- 两次调用都返回新表，原来的 `students` 没有被修改。" ] }, { "cell_type": "markdown", "id": "82066328", "metadata": {}, "source": [ "### 动手练习" ] }, { "cell_type": "code", "execution_count": 4, "id": "c3d9fea3", "metadata": { "execution": { "iopub.execute_input": "2026-07-30T13:37:07.216366Z", "iopub.status.busy": "2026-07-30T13:37:07.216254Z", "iopub.status.idle": "2026-07-30T13:37:07.218188Z", "shell.execute_reply": "2026-07-30T13:37:07.217684Z" } }, "outputs": [], "source": [ "# 只删除“销量”缺失的行，不考虑“备注”是否缺失\n", "# sales = pd.DataFrame({\n", "# \"城市\": [\"A\", \"B\", \"C\"],\n", "# \"销量\": [100, None, 120],\n", "# \"备注\": [None, \"待核对\", None]\n", "# })\n", "# cleaned = __________\n", "# print(cleaned)" ] }, { "cell_type": "markdown", "id": "ca0684bd", "metadata": {}, "source": [ "## 3. 控制 dropna 的删除规则：how 与 thresh\n", "\n", "`dropna` 还可以控制一行缺失到什么程度才删除：\n", "\n", "- `how=\"all\"`：只有整行全部缺失时才删除；\n", "- `thresh=数量`：至少有指定数量的非缺失值才保留。\n", "\n", "`thresh` 统计的是非缺失值数量。" ] }, { "cell_type": "markdown", "id": "0e687d78", "metadata": {}, "source": [ "### 示例代码" ] }, { "cell_type": "code", "execution_count": 5, "id": "4a878424", "metadata": { "execution": { "iopub.execute_input": "2026-07-30T13:37:07.220078Z", "iopub.status.busy": "2026-07-30T13:37:07.219915Z", "iopub.status.idle": "2026-07-30T13:37:07.226156Z", "shell.execute_reply": "2026-07-30T13:37:07.225604Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "原表：\n", " A B C\n", "0 1.0 2.0 3.0\n", "1 NaN 5.0 NaN\n", "2 NaN NaN NaN\n", "3 4.0 NaN 6.0\n", "只删除整行全缺失：\n", " A B C\n", "0 1.0 2.0 3.0\n", "1 NaN 5.0 NaN\n", "3 4.0 NaN 6.0\n", "至少有 2 个非缺失值：\n", " A B C\n", "0 1.0 2.0 3.0\n", "3 4.0 NaN 6.0\n" ] } ], "source": [ "data = pd.DataFrame({\n", " \"A\": [1, None, None, 4],\n", " \"B\": [2, 5, None, None],\n", " \"C\": [3, None, None, 6]\n", "})\n", "\n", "remove_all_missing = data.dropna(how=\"all\")\n", "keep_at_least_two = data.dropna(thresh=2)\n", "\n", "print(\"原表：\")\n", "print(data)\n", "print(\"只删除整行全缺失：\")\n", "print(remove_all_missing)\n", "print(\"至少有 2 个非缺失值：\")\n", "print(keep_at_least_two)" ] }, { "cell_type": "markdown", "id": "0141c164", "metadata": {}, "source": [ "### 代码解读（对应上方示例）\n", "\n", "- 索引 2 的 A、B、C 全部缺失，所以 `how=\"all\"` 会删除这一行。\n", "- `thresh=2` 要求至少两个位置非缺失，因此保留索引 0 和 3。\n", "- 索引 1 只有 B 一个非缺失值，达不到阈值 2。" ] }, { "cell_type": "markdown", "id": "943995c4", "metadata": {}, "source": [ "### 动手练习" ] }, { "cell_type": "code", "execution_count": 6, "id": "1836c595", "metadata": { "execution": { "iopub.execute_input": "2026-07-30T13:37:07.228061Z", "iopub.status.busy": "2026-07-30T13:37:07.227931Z", "iopub.status.idle": "2026-07-30T13:37:07.230025Z", "shell.execute_reply": "2026-07-30T13:37:07.229591Z" } }, "outputs": [], "source": [ "# 只删除整行都缺失的记录，再尝试只保留至少 2 项非缺失的记录\n", "# data = pd.DataFrame({\n", "# \"A\": [1, None, None], \"B\": [2, 3, None]\n", "# })\n", "# result_all = __________\n", "# result_two = __________\n", "# print(result_all)\n", "# print(result_two)" ] }, { "cell_type": "markdown", "id": "2a52faac", "metadata": {}, "source": [ "## 4. 用 fillna 填充缺失值\n", "\n", "`.fillna(填充值)` 返回填充后的新对象。传入字典时，可以为不同列指定不同填充值。\n", "\n", "数值列常根据业务规则填 0、平均值或中位数；文本列常填“未知”。`.mean()` 用于计算非缺失数值的平均值。" ] }, { "cell_type": "markdown", "id": "21af062e", "metadata": {}, "source": [ "### 示例代码" ] }, { "cell_type": "code", "execution_count": 7, "id": "78827a79", "metadata": { "execution": { "iopub.execute_input": "2026-07-30T13:37:07.231682Z", "iopub.status.busy": "2026-07-30T13:37:07.231579Z", "iopub.status.idle": "2026-07-30T13:37:07.238860Z", "shell.execute_reply": "2026-07-30T13:37:07.238352Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "平均成绩： 75.33333333333333\n", "填充后：\n", " 姓名 城市 成绩\n", "0 小林 杭州 82.000000\n", "1 小周 未知 76.000000\n", "2 小郑 宁波 75.333333\n", "3 小王 温州 68.000000\n", "原表仍保留缺失值：\n", " 姓名 城市 成绩\n", "0 小林 杭州 82.0\n", "1 小周 NaN 76.0\n", "2 小郑 宁波 NaN\n", "3 小王 温州 68.0\n" ] } ], "source": [ "students = pd.DataFrame({\n", " \"姓名\": [\"小林\", \"小周\", \"小郑\", \"小王\"],\n", " \"城市\": [\"杭州\", None, \"宁波\", \"温州\"],\n", " \"成绩\": [82, 76, None, 68]\n", "})\n", "\n", "average_score = students[\"成绩\"].mean()\n", "filled = students.fillna({\n", " \"城市\": \"未知\",\n", " \"成绩\": average_score\n", "})\n", "\n", "print(\"平均成绩：\", average_score)\n", "print(\"填充后：\")\n", "print(filled)\n", "print(\"原表仍保留缺失值：\")\n", "print(students)" ] }, { "cell_type": "markdown", "id": "4f501283", "metadata": {}, "source": [ "### 代码解读（对应上方示例）\n", "\n", "- `.mean()` 自动跳过缺失成绩，使用 82、76、68 计算平均值。\n", "- 字典指定城市列填“未知”，成绩列填计算出的平均成绩。\n", "- `fillna` 默认返回新表；最后打印原表可确认它没有被修改。" ] }, { "cell_type": "markdown", "id": "3ddc6c27", "metadata": {}, "source": [ "### 动手练习" ] }, { "cell_type": "code", "execution_count": 8, "id": "4958c575", "metadata": { "execution": { "iopub.execute_input": "2026-07-30T13:37:07.240501Z", "iopub.status.busy": "2026-07-30T13:37:07.240401Z", "iopub.status.idle": "2026-07-30T13:37:07.242645Z", "shell.execute_reply": "2026-07-30T13:37:07.242090Z" } }, "outputs": [], "source": [ "# 把缺失城市填为“未知”，把缺失销量填为销量平均值\n", "# sales = pd.DataFrame({\n", "# \"城市\": [\"A\", None, \"C\"], \"销量\": [100, None, 120]\n", "# })\n", "# average = __________\n", "# filled = sales.fillna({\n", "# \"城市\": __________,\n", "# \"销量\": __________\n", "# })\n", "# print(filled)" ] }, { "cell_type": "markdown", "id": "2cf70071", "metadata": {}, "source": [ "## 5. 用 duplicated 检测重复行\n", "\n", "`.duplicated()` 返回布尔 Series：某行内容在前面已经出现过时，该行标记为 `True`。\n", "\n", "`subset=[列名列表]` 只根据指定业务字段判断重复。默认保留第一次出现，把后续重复项标为 `True`。" ] }, { "cell_type": "markdown", "id": "048951a3", "metadata": {}, "source": [ "### 示例代码" ] }, { "cell_type": "code", "execution_count": 9, "id": "7070bb58", "metadata": { "execution": { "iopub.execute_input": "2026-07-30T13:37:07.244281Z", "iopub.status.busy": "2026-07-30T13:37:07.244173Z", "iopub.status.idle": "2026-07-30T13:37:07.253597Z", "shell.execute_reply": "2026-07-30T13:37:07.252880Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "整行是否重复：\n", "0 False\n", "1 False\n", "2 True\n", "3 False\n", "dtype: bool\n", "姓名是否重复：\n", "0 False\n", "1 False\n", "2 True\n", "3 False\n", "dtype: bool\n", "重复的完整记录：\n", " 姓名 成绩\n", "2 小林 82\n" ] } ], "source": [ "students = pd.DataFrame({\n", " \"姓名\": [\"小林\", \"小周\", \"小林\", \"小郑\"],\n", " \"成绩\": [82, 76, 82, 82]\n", "})\n", "\n", "full_duplicate = students.duplicated()\n", "name_duplicate = students.duplicated(subset=[\"姓名\"])\n", "\n", "print(\"整行是否重复：\")\n", "print(full_duplicate)\n", "print(\"姓名是否重复：\")\n", "print(name_duplicate)\n", "print(\"重复的完整记录：\")\n", "print(students[full_duplicate])" ] }, { "cell_type": "markdown", "id": "e4924d1a", "metadata": {}, "source": [ "### 代码解读（对应上方示例）\n", "\n", "- 索引 2 与索引 0 的姓名和成绩都相同，因此整行重复。\n", "- 只按姓名判断时，索引 2 仍然是重复项。\n", "- 小郑的成绩也为 82，但姓名不同，所以不属于重复记录。" ] }, { "cell_type": "markdown", "id": "de2d9b20", "metadata": {}, "source": [ "### 动手练习" ] }, { "cell_type": "code", "execution_count": 10, "id": "e940483b", "metadata": { "execution": { "iopub.execute_input": "2026-07-30T13:37:07.255473Z", "iopub.status.busy": "2026-07-30T13:37:07.255332Z", "iopub.status.idle": "2026-07-30T13:37:07.257312Z", "shell.execute_reply": "2026-07-30T13:37:07.256939Z" } }, "outputs": [], "source": [ "# 分别检查整行重复和只按“订单号”重复\n", "# orders = pd.DataFrame({\n", "# \"订单号\": [101, 102, 101], \"金额\": [80, 120, 90]\n", "# })\n", "# print(__________)\n", "# print(__________)" ] }, { "cell_type": "markdown", "id": "39be81ef", "metadata": {}, "source": [ "## 6. 用 drop_duplicates 删除重复记录\n", "\n", "`.drop_duplicates()` 返回去重后的 DataFrame。\n", "\n", "- `subset=[列名列表]`：指定判断重复的字段；\n", "- `keep=\"first\"`：保留第一次出现，默认规则；\n", "- `keep=\"last\"`：保留最后一次出现。" ] }, { "cell_type": "markdown", "id": "06473e79", "metadata": {}, "source": [ "### 示例代码" ] }, { "cell_type": "code", "execution_count": 11, "id": "08b00e8c", "metadata": { "execution": { "iopub.execute_input": "2026-07-30T13:37:07.258866Z", "iopub.status.busy": "2026-07-30T13:37:07.258776Z", "iopub.status.idle": "2026-07-30T13:37:07.263916Z", "shell.execute_reply": "2026-07-30T13:37:07.263504Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "订单号重复时保留第一次：\n", " 订单号 金额\n", "0 101 80\n", "1 102 120\n", "3 103 60\n", "订单号重复时保留最后一次：\n", " 订单号 金额\n", "1 102 120\n", "2 101 90\n", "3 103 60\n" ] } ], "source": [ "orders = pd.DataFrame({\n", " \"订单号\": [101, 102, 101, 103],\n", " \"金额\": [80, 120, 90, 60]\n", "})\n", "\n", "keep_first = orders.drop_duplicates(\n", " subset=[\"订单号\"],\n", " keep=\"first\"\n", ")\n", "keep_last = orders.drop_duplicates(\n", " subset=[\"订单号\"],\n", " keep=\"last\"\n", ")\n", "\n", "print(\"订单号重复时保留第一次：\")\n", "print(keep_first)\n", "print(\"订单号重复时保留最后一次：\")\n", "print(keep_last)" ] }, { "cell_type": "markdown", "id": "1631efbb", "metadata": {}, "source": [ "### 代码解读（对应上方示例）\n", "\n", "- 订单号 101 出现两次，但金额不同，所以要先明确业务上用哪条记录。\n", "- `keep=\"first\"` 保留金额 80 的记录。\n", "- `keep=\"last\"` 保留金额 90 的记录。\n", "- 去重规则必须依据字段含义，不能看到重复就直接删除。" ] }, { "cell_type": "markdown", "id": "b65f57fd", "metadata": {}, "source": [ "### 动手练习" ] }, { "cell_type": "code", "execution_count": 12, "id": "8697fcdc", "metadata": { "execution": { "iopub.execute_input": "2026-07-30T13:37:07.265572Z", "iopub.status.busy": "2026-07-30T13:37:07.265475Z", "iopub.status.idle": "2026-07-30T13:37:07.267636Z", "shell.execute_reply": "2026-07-30T13:37:07.267081Z" } }, "outputs": [], "source": [ "# 按学号去重并保留最后一次记录\n", "# students = pd.DataFrame({\n", "# \"学号\": [\"S01\", \"S02\", \"S01\"], \"成绩\": [80, 75, 85]\n", "# })\n", "# result = students.drop_duplicates(\n", "# subset=__________,\n", "# keep=__________\n", "# )\n", "# print(result)" ] }, { "cell_type": "markdown", "id": "99d87b2a", "metadata": {}, "source": [ "## 7. 用 map 按映射表转换一列\n", "\n", "Series 的 `.map(映射字典)` 按字典中的对应关系转换每个值。\n", "\n", "字典的键是原值，字典的值是转换结果。原值不在字典中时，结果为缺失值，因此应检查映射表是否覆盖全部类别。" ] }, { "cell_type": "markdown", "id": "f7f3a408", "metadata": {}, "source": [ "### 示例代码" ] }, { "cell_type": "code", "execution_count": 13, "id": "27cb030d", "metadata": { "execution": { "iopub.execute_input": "2026-07-30T13:37:07.269539Z", "iopub.status.busy": "2026-07-30T13:37:07.269395Z", "iopub.status.idle": "2026-07-30T13:37:07.274931Z", "shell.execute_reply": "2026-07-30T13:37:07.274395Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " 姓名 方向 方向代码\n", "0 小林 数据 DA\n", "1 小周 金融 FI\n", "2 小郑 营销 MK\n", "3 小王 设计 NaN\n", "未映射数量： 1\n" ] } ], "source": [ "students = pd.DataFrame({\n", " \"姓名\": [\"小林\", \"小周\", \"小郑\", \"小王\"],\n", " \"方向\": [\"数据\", \"金融\", \"营销\", \"设计\"]\n", "})\n", "\n", "direction_codes = {\n", " \"数据\": \"DA\",\n", " \"金融\": \"FI\",\n", " \"营销\": \"MK\"\n", "}\n", "students[\"方向代码\"] = students[\"方向\"].map(direction_codes)\n", "\n", "print(students)\n", "print(\"未映射数量：\", students[\"方向代码\"].isna().sum())" ] }, { "cell_type": "markdown", "id": "ebb86baa", "metadata": {}, "source": [ "### 代码解读（对应上方示例）\n", "\n", "- “数据”“金融”“营销”分别被映射成 DA、FI、MK。\n", "- 映射字典中没有“设计”，所以小王的方向代码是缺失值。\n", "- 最后一行统计未映射数量，提醒我们补充映射规则或处理未知类别。" ] }, { "cell_type": "markdown", "id": "be5e9bff", "metadata": {}, "source": [ "### 动手练习" ] }, { "cell_type": "code", "execution_count": 14, "id": "662fd27e", "metadata": { "execution": { "iopub.execute_input": "2026-07-30T13:37:07.276481Z", "iopub.status.busy": "2026-07-30T13:37:07.276355Z", "iopub.status.idle": "2026-07-30T13:37:07.278416Z", "shell.execute_reply": "2026-07-30T13:37:07.277874Z" } }, "outputs": [], "source": [ "# 把 A、B、C 等级映射为“优秀”“良好”“合格”\n", "# grades = pd.Series([\"A\", \"B\", \"C\", \"A\"])\n", "# mapping = {__________}\n", "# result = grades.map(__________)\n", "# print(result)" ] }, { "cell_type": "markdown", "id": "8d957a1a", "metadata": {}, "source": [ "## 8. 用 replace 替换指定值\n", "\n", "`.replace(替换字典)` 根据“原值: 新值”替换 Series 或 DataFrame 中的完整值。\n", "\n", "它适合把特殊编码转成缺失值，或统一类别名称。`replace` 返回新对象，不会自动修改原数据。" ] }, { "cell_type": "markdown", "id": "f79bbaaf", "metadata": {}, "source": [ "### 示例代码" ] }, { "cell_type": "code", "execution_count": 15, "id": "2d2c1072", "metadata": { "execution": { "iopub.execute_input": "2026-07-30T13:37:07.279986Z", "iopub.status.busy": "2026-07-30T13:37:07.279873Z", "iopub.status.idle": "2026-07-30T13:37:07.288810Z", "shell.execute_reply": "2026-07-30T13:37:07.288298Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "清理后的成绩：\n", "0 82\n", "1 \n", "2 91\n", "3 \n", "4 76\n", "dtype: object\n", "清理后的状态：\n", "0 正常\n", "1 已停止\n", "2 NaN\n", "3 正常\n", "dtype: str\n" ] } ], "source": [ "scores = pd.Series([82, -999, 91, -1, 76])\n", "\n", "cleaned_scores = scores.replace({\n", " -999: pd.NA,\n", " -1: pd.NA\n", "})\n", "\n", "status = pd.Series([\"正常\", \"停用\", \"未知\", \"正常\"])\n", "cleaned_status = status.replace({\n", " \"停用\": \"已停止\",\n", " \"未知\": pd.NA\n", "})\n", "\n", "print(\"清理后的成绩：\")\n", "print(cleaned_scores)\n", "print(\"清理后的状态：\")\n", "print(cleaned_status)" ] }, { "cell_type": "markdown", "id": "9a09dbaf", "metadata": {}, "source": [ "### 代码解读（对应上方示例）\n", "\n", "- 特殊数值 `-999` 和 `-1` 被统一替换为 pandas 能识别的缺失值。\n", "- 状态“停用”改为“已停止”，“未知”改为缺失值。\n", "- 这里替换的是整个元素；字符串内部替换要使用后面介绍的 `.str.replace()`。" ] }, { "cell_type": "markdown", "id": "c6e8dde3", "metadata": {}, "source": [ "### 动手练习" ] }, { "cell_type": "code", "execution_count": 16, "id": "ff1eb267", "metadata": { "execution": { "iopub.execute_input": "2026-07-30T13:37:07.290530Z", "iopub.status.busy": "2026-07-30T13:37:07.290415Z", "iopub.status.idle": "2026-07-30T13:37:07.292404Z", "shell.execute_reply": "2026-07-30T13:37:07.291887Z" } }, "outputs": [], "source": [ "# 把销量中的 -1 和 -999 替换为缺失值\n", "# sales = pd.Series([100, -1, 120, -999])\n", "# cleaned = sales.replace(__________)\n", "# print(cleaned)" ] }, { "cell_type": "markdown", "id": "ee0260dd", "metadata": {}, "source": [ "## 9. 用 rename 重命名行列标签\n", "\n", "`.rename(columns=列名映射, index=行标签映射)` 返回重命名后的 DataFrame。\n", "\n", "只需写出要修改的标签；没有出现在映射字典中的列名或行标签保持不变。" ] }, { "cell_type": "markdown", "id": "38c55781", "metadata": {}, "source": [ "### 示例代码" ] }, { "cell_type": "code", "execution_count": 17, "id": "bcceaeb7", "metadata": { "execution": { "iopub.execute_input": "2026-07-30T13:37:07.294423Z", "iopub.status.busy": "2026-07-30T13:37:07.294254Z", "iopub.status.idle": "2026-07-30T13:37:07.299673Z", "shell.execute_reply": "2026-07-30T13:37:07.299144Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "重命名后：\n", " 姓名 成绩 城市\n", "S01 小林 82 杭州\n", "S02 小周 76 宁波\n", "原表：\n", " name score city\n", "0 小林 82 杭州\n", "1 小周 76 宁波\n" ] } ], "source": [ "raw = pd.DataFrame({\n", " \"name\": [\"小林\", \"小周\"],\n", " \"score\": [82, 76],\n", " \"city\": [\"杭州\", \"宁波\"]\n", "})\n", "\n", "renamed = raw.rename(\n", " columns={\n", " \"name\": \"姓名\",\n", " \"score\": \"成绩\",\n", " \"city\": \"城市\"\n", " },\n", " index={0: \"S01\", 1: \"S02\"}\n", ")\n", "\n", "print(\"重命名后：\")\n", "print(renamed)\n", "print(\"原表：\")\n", "print(raw)" ] }, { "cell_type": "markdown", "id": "9af337c2", "metadata": {}, "source": [ "### 代码解读（对应上方示例）\n", "\n", "- `columns` 字典把三个英文列名改成中文列名。\n", "- `index` 字典把默认行标签 0、1 改为 S01、S02。\n", "- `rename` 返回新表，因此原来的 `raw` 仍保留英文列名。" ] }, { "cell_type": "markdown", "id": "da5fee4b", "metadata": {}, "source": [ "### 动手练习" ] }, { "cell_type": "code", "execution_count": 18, "id": "a587d616", "metadata": { "execution": { "iopub.execute_input": "2026-07-30T13:37:07.301262Z", "iopub.status.busy": "2026-07-30T13:37:07.301136Z", "iopub.status.idle": "2026-07-30T13:37:07.303424Z", "shell.execute_reply": "2026-07-30T13:37:07.302882Z" } }, "outputs": [], "source": [ "# 把 product、qty 重命名为“商品”“数量”\n", "# raw = pd.DataFrame({\"product\": [\"A\", \"B\"], \"qty\": [3, 5]})\n", "# renamed = raw.rename(columns=__________)\n", "# print(renamed)" ] }, { "cell_type": "markdown", "id": "d4f60970", "metadata": {}, "source": [ "## 10. 用 pd.cut 把连续数值分组\n", "\n", "`pd.cut(数据, bins=边界列表, labels=标签列表)` 把连续数值划分为若干区间。\n", "\n", "边界列表比标签列表多一个元素。默认区间右端包含、左端不包含；`include_lowest=True` 让第一个区间包含最小边界。" ] }, { "cell_type": "markdown", "id": "04498b4f", "metadata": {}, "source": [ "### 示例代码" ] }, { "cell_type": "code", "execution_count": 19, "id": "a8aef873", "metadata": { "execution": { "iopub.execute_input": "2026-07-30T13:37:07.304873Z", "iopub.status.busy": "2026-07-30T13:37:07.304771Z", "iopub.status.idle": "2026-07-30T13:37:07.312673Z", "shell.execute_reply": "2026-07-30T13:37:07.312142Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " 成绩 等级\n", "0 55 不及格\n", "1 60 及格\n", "2 72 及格\n", "3 80 良好\n", "4 88 良好\n", "5 90 优秀\n", "6 96 优秀\n", "各等级人数：\n", "等级\n", "及格 2\n", "良好 2\n", "优秀 2\n", "不及格 1\n", "Name: count, dtype: int64\n" ] } ], "source": [ "scores = pd.Series([55, 60, 72, 80, 88, 90, 96])\n", "bins = [0, 59, 79, 89, 100]\n", "labels = [\"不及格\", \"及格\", \"良好\", \"优秀\"]\n", "\n", "levels = pd.cut(\n", " scores,\n", " bins=bins,\n", " labels=labels,\n", " include_lowest=True\n", ")\n", "\n", "result = pd.DataFrame({\"成绩\": scores, \"等级\": levels})\n", "print(result)\n", "print(\"各等级人数：\")\n", "print(result[\"等级\"].value_counts())" ] }, { "cell_type": "markdown", "id": "0e488a5b", "metadata": {}, "source": [ "### 代码解读（对应上方示例）\n", "\n", "- 0 到 59 分归为“不及格”，60 到 79 分归为“及格”。\n", "- 80 到 89 分归为“良好”，90 到 100 分归为“优秀”。\n", "- `value_counts()` 统计每个等级包含多少人。\n", "- 分箱前必须先明确边界是否符合题目规则。" ] }, { "cell_type": "markdown", "id": "b906ba77", "metadata": {}, "source": [ "### 动手练习" ] }, { "cell_type": "code", "execution_count": 20, "id": "91c69d27", "metadata": { "execution": { "iopub.execute_input": "2026-07-30T13:37:07.314126Z", "iopub.status.busy": "2026-07-30T13:37:07.314034Z", "iopub.status.idle": "2026-07-30T13:37:07.316097Z", "shell.execute_reply": "2026-07-30T13:37:07.315572Z" } }, "outputs": [], "source": [ "# 把销量分为“低”“中”“高”：0-99、100-199、200-300\n", "# sales = pd.Series([80, 120, 200, 260])\n", "# bins = __________\n", "# labels = __________\n", "# levels = pd.cut(\n", "# sales,\n", "# bins=bins,\n", "# labels=labels,\n", "# include_lowest=True\n", "# )\n", "# print(levels)" ] }, { "cell_type": "markdown", "id": "51f48b38", "metadata": {}, "source": [ "## 11. 用 abs 与 any 检测异常行\n", "\n", "`.abs()` 计算数值的绝对值，`> 阈值` 逐元素判断是否超出范围。\n", "\n", "对布尔 DataFrame 使用 `.any(axis=1)`，可以判断每一行是否至少有一项为 `True`，从而找出包含异常值的整行。" ] }, { "cell_type": "markdown", "id": "231e3dfa", "metadata": {}, "source": [ "### 示例代码" ] }, { "cell_type": "code", "execution_count": 21, "id": "fab12528", "metadata": { "execution": { "iopub.execute_input": "2026-07-30T13:37:07.317613Z", "iopub.status.busy": "2026-07-30T13:37:07.317504Z", "iopub.status.idle": "2026-07-30T13:37:07.322322Z", "shell.execute_reply": "2026-07-30T13:37:07.321788Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "每个位置是否超过绝对值 50：\n", " 指标A 指标B\n", "0 False False\n", "1 False True\n", "2 True False\n", "3 False False\n", "每行是否包含异常值：\n", "0 False\n", "1 True\n", "2 True\n", "3 False\n", "dtype: bool\n", "包含异常值的行：\n", " 指标A 指标B\n", "1 12 -80\n", "2 100 6\n" ] } ], "source": [ "data = pd.DataFrame({\n", " \"指标A\": [10, 12, 100, 11],\n", " \"指标B\": [5, -80, 6, 7]\n", "})\n", "\n", "cell_mask = data.abs() > 50\n", "row_mask = cell_mask.any(axis=1)\n", "outlier_rows = data[row_mask]\n", "\n", "print(\"每个位置是否超过绝对值 50：\")\n", "print(cell_mask)\n", "print(\"每行是否包含异常值：\")\n", "print(row_mask)\n", "print(\"包含异常值的行：\")\n", "print(outlier_rows)" ] }, { "cell_type": "markdown", "id": "dc8d07e2", "metadata": {}, "source": [ "### 代码解读（对应上方示例）\n", "\n", "- 指标 A 中的 100 和指标 B 中的 -80，其绝对值都超过 50。\n", "- `cell_mask` 标记具体异常单元格。\n", "- `any(axis=1)` 跨列检查每一行，只要一列异常，该行就标记为 `True`。\n", "- 结果保留索引 1 和 2 两行，便于进一步核对数据来源。" ] }, { "cell_type": "markdown", "id": "0cd0b8f5", "metadata": {}, "source": [ "### 动手练习" ] }, { "cell_type": "code", "execution_count": 22, "id": "3f9dc170", "metadata": { "execution": { "iopub.execute_input": "2026-07-30T13:37:07.323848Z", "iopub.status.busy": "2026-07-30T13:37:07.323740Z", "iopub.status.idle": "2026-07-30T13:37:07.325732Z", "shell.execute_reply": "2026-07-30T13:37:07.325181Z" } }, "outputs": [], "source": [ "# 找出任意一项绝对值超过 30 的行\n", "# data = pd.DataFrame({\n", "# \"A\": [10, 35, 20], \"B\": [5, 8, -40]\n", "# })\n", "# cell_mask = __________\n", "# row_mask = __________\n", "# result = __________\n", "# print(result)" ] }, { "cell_type": "markdown", "id": "f99c9032", "metadata": {}, "source": [ "## 12. 用 clip 限制数值上下界\n", "\n", "`.clip(lower=下界, upper=上界)` 把小于下界的值改为下界，把大于上界的值改为上界，范围内的值保持不变。\n", "\n", "限制异常值前应保留原数据并说明规则，不能把所有极端值都默认当成错误。" ] }, { "cell_type": "markdown", "id": "a239226a", "metadata": {}, "source": [ "### 示例代码" ] }, { "cell_type": "code", "execution_count": 23, "id": "1b276e6c", "metadata": { "execution": { "iopub.execute_input": "2026-07-30T13:37:07.327346Z", "iopub.status.busy": "2026-07-30T13:37:07.327235Z", "iopub.status.idle": "2026-07-30T13:37:07.331273Z", "shell.execute_reply": "2026-07-30T13:37:07.330820Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " 原成绩 限制后\n", "0 -5 0\n", "1 55 55\n", "2 80 80\n", "3 105 100\n", "4 92 92\n" ] } ], "source": [ "scores = pd.Series([-5, 55, 80, 105, 92])\n", "limited_scores = scores.clip(lower=0, upper=100)\n", "\n", "changes = pd.DataFrame({\n", " \"原成绩\": scores,\n", " \"限制后\": limited_scores\n", "})\n", "\n", "print(changes)" ] }, { "cell_type": "markdown", "id": "3d845edb", "metadata": {}, "source": [ "### 代码解读（对应上方示例）\n", "\n", "- 原值 -5 小于下界 0，因此变成 0。\n", "- 原值 105 大于上界 100，因此变成 100。\n", "- 55、80、92 已在范围内，保持不变。\n", "- `clip` 返回新 Series，原来的 `scores` 不会被自动覆盖。" ] }, { "cell_type": "markdown", "id": "0eed86bd", "metadata": {}, "source": [ "### 动手练习" ] }, { "cell_type": "code", "execution_count": 24, "id": "fa16a5a9", "metadata": { "execution": { "iopub.execute_input": "2026-07-30T13:37:07.332854Z", "iopub.status.busy": "2026-07-30T13:37:07.332726Z", "iopub.status.idle": "2026-07-30T13:37:07.334763Z", "shell.execute_reply": "2026-07-30T13:37:07.334207Z" } }, "outputs": [], "source": [ "# 把销量限制在 0 到 500 之间\n", "# sales = pd.Series([-20, 120, 520, 300])\n", "# limited = __________\n", "# print(limited)" ] }, { "cell_type": "markdown", "id": "6d972fc4", "metadata": {}, "source": [ "## 13. 用 sample 随机抽取记录\n", "\n", "`.sample(n=数量, random_state=种子)` 从 Series 或 DataFrame 随机抽取指定数量的记录。\n", "\n", "`random_state` 固定随机结果，便于复现和核对。默认不重复抽取同一行。" ] }, { "cell_type": "markdown", "id": "ae3127fc", "metadata": {}, "source": [ "### 示例代码" ] }, { "cell_type": "code", "execution_count": 25, "id": "1625549d", "metadata": { "execution": { "iopub.execute_input": "2026-07-30T13:37:07.336727Z", "iopub.status.busy": "2026-07-30T13:37:07.336588Z", "iopub.status.idle": "2026-07-30T13:37:07.341646Z", "shell.execute_reply": "2026-07-30T13:37:07.341107Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " 姓名 成绩\n", "3 小王 68\n", "4 小赵 85\n", "5 小钱 79\n" ] } ], "source": [ "students = pd.DataFrame({\n", " \"姓名\": [\"小林\", \"小周\", \"小郑\", \"小王\", \"小赵\", \"小钱\"],\n", " \"成绩\": [82, 76, 91, 68, 85, 79]\n", "})\n", "\n", "sample_students = students.sample(\n", " n=3,\n", " random_state=2026\n", ")\n", "\n", "print(sample_students)" ] }, { "cell_type": "markdown", "id": "e4f0d139", "metadata": {}, "source": [ "### 代码解读（对应上方示例）\n", "\n", "- `n=3` 表示从 6 行中抽取 3 行。\n", "- 使用相同的 `random_state=2026` 重新运行，会得到相同的抽样结果。\n", "- 抽样后的行保留原索引，可以追溯到原表位置。" ] }, { "cell_type": "markdown", "id": "1fd960fa", "metadata": {}, "source": [ "### 动手练习" ] }, { "cell_type": "code", "execution_count": 26, "id": "4be3c6e8", "metadata": { "execution": { "iopub.execute_input": "2026-07-30T13:37:07.343147Z", "iopub.status.busy": "2026-07-30T13:37:07.343033Z", "iopub.status.idle": "2026-07-30T13:37:07.345211Z", "shell.execute_reply": "2026-07-30T13:37:07.344682Z" } }, "outputs": [], "source": [ "# 从下表随机抽取 2 行，并使用种子 100\n", "# sales = pd.DataFrame({\n", "# \"城市\": [\"A\", \"B\", \"C\", \"D\"], \"销量\": [80, 120, 100, 95]\n", "# })\n", "# result = sales.sample(n=__________, random_state=__________)\n", "# print(result)" ] }, { "cell_type": "markdown", "id": "c810b5b7", "metadata": {}, "source": [ "## 14. 用 get_dummies 把分类值转成 0/1 列\n", "\n", "`pd.get_dummies(分类数据, prefix=\"前缀\", dtype=int)` 为每个分类创建一列。\n", "\n", "当前行属于某个分类时对应列为 1，否则为 0。这种结果常用于统计建模和机器学习。" ] }, { "cell_type": "markdown", "id": "9afb8cfe", "metadata": {}, "source": [ "### 示例代码" ] }, { "cell_type": "code", "execution_count": 27, "id": "5f2d2680", "metadata": { "execution": { "iopub.execute_input": "2026-07-30T13:37:07.346813Z", "iopub.status.busy": "2026-07-30T13:37:07.346699Z", "iopub.status.idle": "2026-07-30T13:37:07.351964Z", "shell.execute_reply": "2026-07-30T13:37:07.351514Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " 方向_数据 方向_营销 方向_金融\n", "0 1 0 0\n", "1 0 0 1\n", "2 1 0 0\n", "3 0 1 0\n" ] } ], "source": [ "directions = pd.Series([\"数据\", \"金融\", \"数据\", \"营销\"])\n", "\n", "dummy_columns = pd.get_dummies(\n", " directions,\n", " prefix=\"方向\",\n", " dtype=int\n", ")\n", "\n", "print(dummy_columns)" ] }, { "cell_type": "markdown", "id": "a8d0220e", "metadata": {}, "source": [ "### 代码解读（对应上方示例）\n", "\n", "- 原数据有“数据”“金融”“营销”三个不同值，因此生成三列。\n", "- 第一行方向是“数据”，所以“方向_数据”为 1，其余两列为 0。\n", "- `prefix=\"方向\"` 让生成的列名含义更明确。\n", "- `dtype=int` 让结果明确显示为整数 0 和 1。" ] }, { "cell_type": "markdown", "id": "7a57f843", "metadata": {}, "source": [ "### 动手练习" ] }, { "cell_type": "code", "execution_count": 28, "id": "25286615", "metadata": { "execution": { "iopub.execute_input": "2026-07-30T13:37:07.354085Z", "iopub.status.busy": "2026-07-30T13:37:07.353956Z", "iopub.status.idle": "2026-07-30T13:37:07.355948Z", "shell.execute_reply": "2026-07-30T13:37:07.355453Z" } }, "outputs": [], "source": [ "# 把“东”“西”“南”三个区域转换为带“区域”前缀的 0/1 列\n", "# regions = pd.Series([\"东\", \"西\", \"东\", \"南\"])\n", "# result = pd.get_dummies(\n", "# __________,\n", "# prefix=__________,\n", "# dtype=__________\n", "# )\n", "# print(result)" ] }, { "cell_type": "markdown", "id": "dac0981d", "metadata": {}, "source": [ "## 15. 批量清理文本：str.strip 与 str.lower\n", "\n", "pandas 的 `.str` 是字符串批量操作入口：\n", "\n", "- `.str.strip()`：删除每个字符串两端空白；\n", "- `.str.lower()`：把英文字母转为小写；\n", "- `.str.upper()`：把英文字母转为大写。\n", "\n", "多个字符串方法可以依次连接使用。" ] }, { "cell_type": "markdown", "id": "d20705b6", "metadata": {}, "source": [ "### 示例代码" ] }, { "cell_type": "code", "execution_count": 29, "id": "60d45c04", "metadata": { "execution": { "iopub.execute_input": "2026-07-30T13:37:07.357584Z", "iopub.status.busy": "2026-07-30T13:37:07.357461Z", "iopub.status.idle": "2026-07-30T13:37:07.363691Z", "shell.execute_reply": "2026-07-30T13:37:07.363183Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "原数据：\n", "0 Alice \n", "1 BOB\n", "2 Carol \n", "dtype: string\n", "清理后：\n", "0 alice\n", "1 bob\n", "2 carol\n", "dtype: string\n" ] } ], "source": [ "raw_names = pd.Series(\n", " [\" Alice \", \"BOB\", \" Carol \"],\n", " dtype=\"string\"\n", ")\n", "\n", "cleaned_names = raw_names.str.strip().str.lower()\n", "\n", "print(\"原数据：\")\n", "print(raw_names)\n", "print(\"清理后：\")\n", "print(cleaned_names)" ] }, { "cell_type": "markdown", "id": "f17a802a", "metadata": {}, "source": [ "### 代码解读（对应上方示例）\n", "\n", "- 第一个 `.str.strip()` 去掉 Alice 和 Carol 两端多余空格。\n", "- 第二个 `.str.lower()` 把所有英文字母统一成小写。\n", "- 方法按从左到右的顺序执行，结果保存在新的 Series 中。" ] }, { "cell_type": "markdown", "id": "df23a60f", "metadata": {}, "source": [ "### 动手练习" ] }, { "cell_type": "code", "execution_count": 30, "id": "144ac4e4", "metadata": { "execution": { "iopub.execute_input": "2026-07-30T13:37:07.365227Z", "iopub.status.busy": "2026-07-30T13:37:07.365112Z", "iopub.status.idle": "2026-07-30T13:37:07.367188Z", "shell.execute_reply": "2026-07-30T13:37:07.366615Z" } }, "outputs": [], "source": [ "# 去掉两端空格，并把英文城市名统一为大写\n", "# cities = pd.Series([\" hangzhou \", \"NINGBO\", \" Wenzhou\"], dtype=\"string\")\n", "# cleaned = __________\n", "# print(cleaned)" ] }, { "cell_type": "markdown", "id": "32ed47a1", "metadata": {}, "source": [ "## 16. 替换字符串内部内容：str.replace\n", "\n", "`.str.replace(旧文本, 新文本, regex=False)` 替换每个字符串内部的指定内容。\n", "\n", "`regex=False` 表示按普通文本替换，不把旧文本解释为正则表达式。它与替换完整元素的 Series `.replace()` 不同。" ] }, { "cell_type": "markdown", "id": "6111fa96", "metadata": {}, "source": [ "### 示例代码" ] }, { "cell_type": "code", "execution_count": 31, "id": "1d9a274c", "metadata": { "execution": { "iopub.execute_input": "2026-07-30T13:37:07.368940Z", "iopub.status.busy": "2026-07-30T13:37:07.368774Z", "iopub.status.idle": "2026-07-30T13:37:07.372432Z", "shell.execute_reply": "2026-07-30T13:37:07.372059Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0 1380001\n", "1 1390020\n", "2 1370300\n", "dtype: string\n" ] } ], "source": [ "phone_numbers = pd.Series(\n", " [\"138-0001\", \"139-0020\", \"137-0300\"],\n", " dtype=\"string\"\n", ")\n", "\n", "cleaned_numbers = phone_numbers.str.replace(\n", " \"-\",\n", " \"\",\n", " regex=False\n", ")\n", "\n", "print(cleaned_numbers)" ] }, { "cell_type": "markdown", "id": "60e45cc4", "metadata": {}, "source": [ "### 代码解读（对应上方示例）\n", "\n", "- 每个电话号码中的连字符 `-` 被空字符串替换，相当于删除连字符。\n", "- `.str.replace` 对 Series 中每个字符串分别处理。\n", "- `regex=False` 让本例只执行普通字符替换，规则更容易理解。" ] }, { "cell_type": "markdown", "id": "3e5a59d3", "metadata": {}, "source": [ "### 动手练习" ] }, { "cell_type": "code", "execution_count": 32, "id": "beee430a", "metadata": { "execution": { "iopub.execute_input": "2026-07-30T13:37:07.374267Z", "iopub.status.busy": "2026-07-30T13:37:07.374162Z", "iopub.status.idle": "2026-07-30T13:37:07.376201Z", "shell.execute_reply": "2026-07-30T13:37:07.375644Z" } }, "outputs": [], "source": [ "# 删除商品编码中的空格\n", "# codes = pd.Series([\"A 001\", \"B 020\", \"C 300\"], dtype=\"string\")\n", "# cleaned = codes.str.replace(\n", "# __________,\n", "# __________,\n", "# regex=False\n", "# )\n", "# print(cleaned)" ] }, { "cell_type": "markdown", "id": "3af876c8", "metadata": {}, "source": [ "## 17. 按文本内容筛选：str.contains\n", "\n", "`.str.contains(目标文本, na=False, regex=False)` 检查每个字符串是否包含目标文本，返回布尔 Series。\n", "\n", "`na=False` 让缺失字符串按“不包含”处理；`regex=False` 表示普通文本匹配。" ] }, { "cell_type": "markdown", "id": "2083b1a6", "metadata": {}, "source": [ "### 示例代码" ] }, { "cell_type": "code", "execution_count": 33, "id": "a8597e92", "metadata": { "execution": { "iopub.execute_input": "2026-07-30T13:37:07.378073Z", "iopub.status.busy": "2026-07-30T13:37:07.377965Z", "iopub.status.idle": "2026-07-30T13:37:07.382726Z", "shell.execute_reply": "2026-07-30T13:37:07.382135Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "是否包含 Python：\n", "0 True\n", "1 False\n", "2 False\n", "3 True\n", "dtype: boolean\n", "筛选结果：\n", "0 Python 基础\n", "3 Python 统计\n", "dtype: string\n" ] } ], "source": [ "course_names = pd.Series(\n", " [\"Python 基础\", \"pandas 清洗\", pd.NA, \"Python 统计\"],\n", " dtype=\"string\"\n", ")\n", "\n", "python_mask = course_names.str.contains(\n", " \"Python\",\n", " na=False,\n", " regex=False\n", ")\n", "selected = course_names[python_mask]\n", "\n", "print(\"是否包含 Python：\")\n", "print(python_mask)\n", "print(\"筛选结果：\")\n", "print(selected)" ] }, { "cell_type": "markdown", "id": "469ce8eb", "metadata": {}, "source": [ "### 代码解读（对应上方示例）\n", "\n", "- 第一项和第四项包含完全相同的文本“Python”，所以条件为 `True`。\n", "- 缺失项通过 `na=False` 处理为 `False`，不会让筛选条件出现缺失。\n", "- `course_names[python_mask]` 只保留包含目标文本的课程名。" ] }, { "cell_type": "markdown", "id": "dd0eec9f", "metadata": {}, "source": [ "### 动手练习" ] }, { "cell_type": "code", "execution_count": 34, "id": "cfa3f59c", "metadata": { "execution": { "iopub.execute_input": "2026-07-30T13:37:07.384207Z", "iopub.status.busy": "2026-07-30T13:37:07.384088Z", "iopub.status.idle": "2026-07-30T13:37:07.386356Z", "shell.execute_reply": "2026-07-30T13:37:07.385772Z" } }, "outputs": [], "source": [ "# 筛选出名称中包含“数据”的项目\n", "# names = pd.Series([\"数据分析\", \"市场营销\", pd.NA, \"数据清洗\"], dtype=\"string\")\n", "# mask = names.str.contains(\n", "# __________,\n", "# na=__________,\n", "# regex=False\n", "# )\n", "# result = __________\n", "# print(result)" ] }, { "cell_type": "markdown", "id": "af96b118", "metadata": {}, "source": [ "## 18. 拆分结构化文本：str.split\n", "\n", "`.str.split(分隔符, n=次数, expand=True)` 按分隔符拆分每个字符串。\n", "\n", "`n=1` 表示最多拆分一次，`expand=True` 表示把拆分结果展开为 DataFrame 的多列。" ] }, { "cell_type": "markdown", "id": "479cd10a", "metadata": {}, "source": [ "### 示例代码" ] }, { "cell_type": "code", "execution_count": 35, "id": "84cc9d30", "metadata": { "execution": { "iopub.execute_input": "2026-07-30T13:37:07.387879Z", "iopub.status.busy": "2026-07-30T13:37:07.387761Z", "iopub.status.idle": "2026-07-30T13:37:07.391919Z", "shell.execute_reply": "2026-07-30T13:37:07.391395Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " 类别 编号\n", "0 A 001\n", "1 B 020\n", "2 C 300\n" ] } ], "source": [ "product_codes = pd.Series(\n", " [\"A-001\", \"B-020\", \"C-300\"],\n", " dtype=\"string\"\n", ")\n", "\n", "parts = product_codes.str.split(\n", " \"-\",\n", " n=1,\n", " expand=True\n", ")\n", "parts.columns = [\"类别\", \"编号\"]\n", "\n", "print(parts)" ] }, { "cell_type": "markdown", "id": "f16c9298", "metadata": {}, "source": [ "### 代码解读（对应上方示例）\n", "\n", "- 每个编码在第一个连字符位置拆成两段。\n", "- `expand=True` 让左半部分和右半部分分别进入两列。\n", "- 最后一行把默认列名 0、1 改成含义明确的“类别”“编号”。" ] }, { "cell_type": "markdown", "id": "f7a6be31", "metadata": {}, "source": [ "### 动手练习" ] }, { "cell_type": "code", "execution_count": 36, "id": "721417f9", "metadata": { "execution": { "iopub.execute_input": "2026-07-30T13:37:07.393460Z", "iopub.status.busy": "2026-07-30T13:37:07.393333Z", "iopub.status.idle": "2026-07-30T13:37:07.395468Z", "shell.execute_reply": "2026-07-30T13:37:07.394906Z" } }, "outputs": [], "source": [ "# 把“城市:销量”拆成“城市”“销量”两列\n", "# records = pd.Series([\"杭州:120\", \"宁波:98\"], dtype=\"string\")\n", "# parts = records.str.split(\n", "# __________,\n", "# n=__________,\n", "# expand=__________\n", "# )\n", "# parts.columns = __________\n", "# print(parts)" ] }, { "cell_type": "markdown", "id": "2617567a", "metadata": {}, "source": [ "## 教材中的阅读拓展\n", "\n", "以下内容在 `da_7f_clean.pdf` 中有介绍，但不列入本次必须掌握范围：\n", "\n", "- **前向和后向填充**：`.ffill()`、`.bfill()`，常用于有明确顺序的时间数据；\n", "- **分位数分箱 `qcut`**：根据样本分位数让各组数量大致相等；\n", "- **随机置换与有放回抽样**：`permutation`、`sample(replace=True)`；\n", "- **复杂正则表达式**：`findall`、`extract`、分组和替换引用；\n", "- **可空扩展类型**：`Int64`、`boolean`、`string` 等类型中的 `pd.NA`；\n", "- **Categorical 分类类型**：分类编码、排序、重命名和移除未使用分类；\n", "- **多成员虚拟变量**：一个单元格同时包含多个分类时的拆分与编码。\n", "\n", "完成一次清洗后，应再次检查：缺失数量、重复数量、关键列类型、取值范围、分类取值和行数变化，并保存清洗规则。" ] }, { "cell_type": "markdown", "id": "a5201cf7", "metadata": {}, "source": [ "## 离场检验\n", "\n", "1. `.isna().sum()` 的作用是什么？\n", "2. `dropna(subset=[\"成绩\"])` 会不会因为备注缺失而删除一行？\n", "3. `dropna(thresh=2)` 中的 2 表示什么？\n", "4. `duplicated` 与 `drop_duplicates` 分别返回什么？\n", "5. 使用字典 `map` 时，字典中没有的原值会得到什么？\n", "6. Series 的 `replace` 与 `.str.replace` 有什么区别？\n", "7. `any(axis=1)` 在异常值检测中表示什么？\n", "8. `str.contains(..., na=False)` 为什么要设置 `na=False`？\n", "\n", "
\n", "点击查看参考答案\n", "\n", "1. 按列统计缺失值数量。\n", "2. 不会；它只检查“成绩”列。\n", "3. 一行至少需要 2 个非缺失值才保留。\n", "4. 前者返回重复标记的布尔 Series；后者返回删除重复记录后的数据。\n", "5. 缺失值。\n", "6. `replace` 替换完整元素；`.str.replace` 替换每个字符串内部的内容。\n", "7. 检查每一行是否至少有一列满足异常条件。\n", "8. 让缺失字符串按 `False` 处理，保证筛选条件是完整布尔值。\n", "\n", "
" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (skills)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.14.3" } }, "nbformat": 4, "nbformat_minor": 5 }