dtypes in pandas

A NumPy array has one dtype. A DataFrame is a collection of columns, so it has one per column — which is exactly why it can hold a date, a name and a price side by side.

df.dtypes is the first thing to look at after loading anything, and the most common source of "why isn't this working".

Text

In pandas 3, text columns have dtype str. In older versions you'll see object, which meant the column held pointers to Python objects — flexible, slow, and memory-hungry. If you meet object in someone's notebook, that's what it is, and .convert_dtypes() will usually fix it.

The integer-plus-gap problem

NumPy integers have no way to represent "missing". So one blank in an integer column turns the whole column into float64:

pd.Series([1, 2, 3])          # int64
pd.Series([1, 2, None])       # float64  — ids become 1.0, 2.0, NaN

This is why CSVs so often load with float columns you didn't ask for. pandas' answer is a parallel family of nullable dtypes, spelled with a capital letter:

pd.Series([1, 2, None], dtype="Int64")    # Int64 — stays an integer
# missing value is pd.NA, not np.nan

Int64, Float64 and boolean all behave this way. The capital letter is the whole difference, which is unfortunate but easy to remember once you've been caught by it.

category

When a column has few distinct values repeated many times — city, status, product, country — category stores the distinct values once and one small integer code per row:

orders["city"] = orders["city"].astype("category")
orders.memory_usage(deep=True)

Ten-fold memory savings are normal on a big table, groupby gets faster, and you can give the categories an order — so "low < medium < high" sorts correctly instead of alphabetically.

Dates

A date read from a CSV is a string until you say otherwise, and string dates sort wrongly, can't be subtracted, and don't expose .dt.month:

orders["date"] = pd.to_datetime(orders["date"])
orders["date"].dt.day_name()

That's the whole of Module 6, so it gets proper treatment there.

Converting

df["cups"].astype("Int64")               # explicit
pd.to_numeric(s, errors="coerce")        # bad values become NaN, no exception
pd.to_datetime(s, errors="coerce")       # same idea for dates
df.convert_dtypes()                      # let pandas pick sensible nullable types

pd.read_csv(f, dtype={"id": "Int64"}, parse_dates=["date"])   # best: on the way in

errors="coerce" is worth remembering. Without it a single bad value raises and you lose the whole load; with it, the bad values become NaN and you can go and look at them.