house_price_inspection
A local file has been selected for analysis:
"/Users/marklong/learning/UOM/bitbucket/miniproject1/code/Week_11_Graded_Mini_Project_Dataset_houseprice.csv"
Load and inspect this file now.
I'll load the selected CSV file and show a quick preview.
</>Load house price data
import pandas as pd
file_path = "/Users/marklong/learning/UOM/bitbucket/miniproject1/code/Week_11_Graded_Mini_Project_Dataset_houseprice.csv"
df = pd.read_csv(file_path)
print(f"Shape: {df.shape}")
columns = df.columns.tolist()
print(f"Columns ({len(columns)}):", columns if len(columns) <= 100 else columns[:100] + ["..."])
df.head()Shape: (1460, 81) Columns (81): ['Id', 'MSSubClass', 'MSZoning', 'LotFrontage', 'LotArea', 'Street', 'Alley', 'LotShape', 'LandContour', 'Utilities', 'LotConfig', 'LandSlope', 'Neighborhood', 'Condition1', 'Condition2', 'BldgType', 'HouseStyle', 'OverallQual', 'OverallCond', 'YearBuilt', 'YearRemodAdd', 'RoofStyle', 'RoofMatl', 'Exterior1st', 'Exterior2nd', 'MasVnrType', 'MasVnrArea', 'ExterQual', 'ExterCond', 'Foundation', 'BsmtQual', 'BsmtCond', 'BsmtExposure', 'BsmtFinType1', 'BsmtFinSF1', 'BsmtFinType2', 'BsmtFinSF2', 'BsmtUnfSF', 'TotalBsmtSF', 'Heating', 'HeatingQC', 'CentralAir', 'Electrical', '1stFlrSF', '2ndFlrSF', 'LowQualFinSF', 'GrLivArea', 'BsmtFullBath', 'BsmtHalfBath', 'FullBath', 'HalfBath', 'BedroomAbvGr', 'KitchenAbvGr', 'KitchenQual', 'TotRmsAbvGrd', 'Functional', 'Fireplaces', 'FireplaceQu', 'GarageType', 'GarageYrBlt', 'GarageFinish', 'GarageCars', 'GarageArea', 'GarageQual', 'GarageCond', 'PavedDrive', 'WoodDeckSF', 'OpenPorchSF', 'EnclosedPorch', '3SsnPorch', 'ScreenPorch', 'PoolArea', 'PoolQC', 'Fence', 'MiscFeature', 'MiscVal', 'MoSold', 'YrSold', 'SaleType', 'SaleCondition', 'SalePrice']
| Id | MSSubClass | MSZoning | LotFrontage | LotArea | Street | Alley | LotShape | LandContour | Utilities | ... | PoolArea | PoolQC | Fence | MiscFeature | MiscVal | MoSold | YrSold | SaleType | SaleCondition | SalePrice | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 1 | 60 | RL | 65.0 | 8450 | Pave | NaN | Reg | Lvl | AllPub | ... | 0 | NaN | NaN | NaN | 0 | 2 | 2008 | WD | Normal | 208500 |
| 1 | 2 | 20 | RL | 80.0 | 9600 | Pave | NaN | Reg | Lvl | AllPub | ... | 0 | NaN | NaN | NaN | 0 | 5 | 2007 | WD | Normal | 181500 |
| 2 | 3 | 60 | RL | 68.0 | 11250 | Pave | NaN | IR1 | Lvl | AllPub | ... | 0 | NaN | NaN | NaN | 0 | 9 | 2008 | WD | Normal | 223500 |
| 3 | 4 | 70 | RL | 60.0 | 9550 | Pave | NaN | IR1 | Lvl | AllPub | ... | 0 | NaN | NaN | NaN | 0 | 2 | 2006 | WD | Abnorml | 140000 |
| 4 | 5 | 60 | RL | 84.0 | 14260 | Pave | NaN | IR1 | Lvl | AllPub | ... | 0 | NaN | NaN | NaN | 0 | 12 | 2008 | WD | Normal | 250000 |
5 rows × 81 columns
The file contains 1,460 property records and 81 columns, matching a typical house-price modeling dataset. It includes a unique Id field, numerical home and lot characteristics, categorical features such as neighborhood and zoning, and substantial missingness in some optional-property fields (for example, Alley, pool-related fields, and fence-related fields) as visible in the preview.
The first rows show mixed data types and variables covering location, lot dimensions, construction quality, living area, basement, bathrooms, garage, and sale timing—suitable for house-price analysis.
Do an EDA on this dataset
Thinking
Considering EDA steps
I need to choose the first step for exploratory data analysis (EDA). It makes sense to start with a data quality overview, focusing on missing values and types, along with a summary of targets. I could create a compact table showing data types and missing values, while avoiding initial inspection. I'll generate a summary DataFrame to highlight the primary missing data, since the user wants a general and iterative approach to EDA. I'll start with the missing data profile.
Start with data completeness: this identifies fields that need cleaning or special handling before deeper price analysis.
</>Summarize missing values
missing = (
df.isna()
.sum()
.to_frame("missing_count")
.assign(missing_pct=lambda x: (x["missing_count"] / len(df) * 100).round(1))
.query("missing_count > 0")
.sort_values("missing_count", ascending=False)
)
missing| missing_count | missing_pct | |
|---|---|---|
| PoolQC | 1453 | 99.5 |
| MiscFeature | 1406 | 96.3 |
| Alley | 1369 | 93.8 |
| Fence | 1179 | 80.8 |
| MasVnrType | 872 | 59.7 |
| FireplaceQu | 690 | 47.3 |
| LotFrontage | 259 | 17.7 |
| GarageType | 81 | 5.5 |
| GarageYrBlt | 81 | 5.5 |
| GarageFinish | 81 | 5.5 |
| GarageQual | 81 | 5.5 |
| GarageCond | 81 | 5.5 |
| BsmtFinType2 | 38 | 2.6 |
| BsmtExposure | 38 | 2.6 |
| BsmtFinType1 | 37 | 2.5 |
| BsmtCond | 37 | 2.5 |
| BsmtQual | 37 | 2.5 |
| MasVnrArea | 8 | 0.5 |
| Electrical | 1 | 0.1 |
Several columns have missing values that are likely structural, meaning the feature does not apply rather than being an unknown value. This is especially clear for PoolQC (99.5%), MiscFeature (96.3%), Alley (93.8%), and Fence (80.8%). FireplaceQu is absent for nearly half of homes, plausibly because many homes lack fireplaces. Garage-related columns share exactly 81 missing records, suggesting those properties have no garage.
LotFrontage has 17.7% missingness and likely needs statistical imputation rather than a “None” category. Only Electrical has a single missing record, while basement and masonry measurements have relatively limited missingness.
Next, summarize the sale-price target and identify its strongest numerical relationships.
</>Show sale price summary
numeric_summary = df.select_dtypes(include="number").corr(numeric_only=True)["SalePrice"].sort_values()
display(df["SalePrice"].describe().to_frame("SalePrice"))
display(numeric_summary.tail(11).to_frame("correlation_with_SalePrice"))| SalePrice | |
|---|---|
| count | 1460.000000 |
| mean | 180921.195890 |
| std | 79442.502883 |
| min | 34900.000000 |
| 25% | 129975.000000 |
| 50% | 163000.000000 |
| 75% | 214000.000000 |
| max | 755000.000000 |
| correlation_with_SalePrice | |
|---|---|
| YearRemodAdd | 0.507101 |
| YearBuilt | 0.522897 |
| TotRmsAbvGrd | 0.533723 |
| FullBath | 0.560664 |
| 1stFlrSF | 0.605852 |
| TotalBsmtSF | 0.613581 |
| GarageArea | 0.623431 |
| GarageCars | 0.640409 |
| GrLivArea | 0.708624 |
| OverallQual | 0.790982 |
| SalePrice | 1.000000 |
Sale prices are right-skewed: the median is 163,000 USD, below the mean of 180,921 USD, with a wide range from 34,900 USD to 755,000 USD. This suggests a small number of high-priced homes may influence averages and modeling.
The strongest numeric relationships with price are:
OverallQual($r=0.791$): overall construction/material quality is the clearest price driver.GrLivArea($r=0.709$): larger above-ground living space is strongly associated with higher prices.- Garage capacity and area ($r=0.640$ and $r=0.623$), basement area ($r=0.614$), and first-floor area ($r=0.606$) are also important.
- Newer or recently remodeled homes tend to sell for more (
YearBuilt, $r=0.523$;YearRemodAdd, $r=0.507$).
Missingness is concentrated in amenity fields. PoolQC, MiscFeature, Alley, and Fence are missing for most homes, likely indicating the amenity is absent. Garage fields are jointly missing for 81 properties, consistent with homes without garages. LotFrontage has meaningful missingness (17.7%) and should be imputed carefully if used in modeling.
A useful remaining EDA view is how sale price varies across the highest-impact categorical feature, overall quality.
</>Plot sale price by quality
import seaborn as sns
import matplotlib.pyplot as plt
plt.figure(figsize=(10, 5))
sns.boxplot(data=df, x="OverallQual", y="SalePrice", color="#72a6c7")
plt.title("Sale Price by Overall Quality")
plt.xlabel("Overall Quality")
plt.ylabel("Sale Price (USD)")
plt.ticklabel_format(style="plain", axis="y")
plt.show()
The price distribution and correlations provide a strong initial view of numeric drivers, but the latest quality chart has no visible summary values. The table below quantifies the median sale price and sample size at each quality level, allowing the pattern to be interpreted reliably.
</>Summarize sales by quality
quality_summary = (
df.groupby("OverallQual")["SalePrice"]
.agg(homes="size", median_sale_price="median", mean_sale_price="mean")
.round(0)
)
quality_summary| homes | median_sale_price | mean_sale_price | |
|---|---|---|---|
| OverallQual | |||
| 1 | 2 | 50150.0 | 50150.0 |
| 2 | 3 | 60000.0 | 51770.0 |
| 3 | 20 | 86250.0 | 87474.0 |
| 4 | 116 | 108000.0 | 108421.0 |
| 5 | 397 | 133000.0 | 133523.0 |
| 6 | 374 | 160000.0 | 161603.0 |
| 7 | 319 | 200141.0 | 207716.0 |
| 8 | 168 | 269750.0 | 274736.0 |
| 9 | 43 | 345000.0 | 367513.0 |
| 10 | 18 | 432390.0 | 438588.0 |