shift, diff and change over time

"How does today compare with yesterday" needs each row to see the row above it. In a loop that's values[i] - values[i-1]. Vectorised, it's one move: slide the column down by a row and then do ordinary arithmetic.

shift

sales.shift(1)     # every value moves down one row
sales.shift(-1)    # ...and up, for the next value
sales.shift(7)     # a week ago

After shift(1), each row sits beside the previous row's value. The first row has nothing above it, so it gets NaN — which also promotes an integer column to float.

diff and pct_change

sales.diff()          # the same as sales - sales.shift(1)
sales.diff(7)         # change against a week ago
sales.pct_change()    # as a proportion: -0.208 means down 20.8%

Note pandas' diff keeps the length and puts a NaN at the front, unlike NumPy's np.diff which came back one shorter. Same idea, different convention, and the pandas one is easier to assign back as a column.

pct_change is undefined where the previous value was zero — you'll get inf rather than an error, which is worth knowing before it reaches a chart.

Comparing like with like

For anything with a weekly rhythm, comparing today with yesterday is misleading — Saturday is always up on Friday, and that tells you nothing. shift(7) compares each day with the same day last week:

week_on_week = sales.pct_change(7)

That's the everyday version of seasonal adjustment, and it's one argument.

Two things that go wrong

Shifting across groups. On a frame with several cities, a plain shift(1) makes the last row of one city the "previous" row of the next. Shift within the group:

orders.groupby("city")["cups"].shift(1)
orders.groupby("city")["cups"].diff()

Leakage. shift(-1) pulls the future into the current row. As a prediction target that's exactly right. As a feature it means your model can see the answer, and it'll score beautifully in testing and fail completely in production. Be deliberate about which you're building.

And both depend on row order. shift trusts the order it's given without checking whether it means anything, so .sort_index() first.

The related family

sales.cumsum()                   # running total
sales.cummax()                   # running peak
sales.rank(pct=True)             # percentile rank of each value
sales.diff().gt(0).sum()         # how many days went up

That last idiom is worth having: diff to get the changes, a comparison to get a mask, sum to count it. Three operations, no loop, and it reads as the question you asked.