Upgrade?

The last few times I have flown, I've gotten upgraded to first class.
This is because of my frequent flyer status. I'm a BIG fan of the
window seat and I can recognize a lot of things from the air - being
that I've cross-crossed the country by car countless times.

My recent upgrades however have been ailse seats and since my flight
have been under 2 hours, I'm starting to wonder (since they take me
away from the window) if they are an upgrade at all?

Certainly if the flight were longer, then the wider chair and much
more leg room would be worth it, but I really miss looking out that
window.

I wonder how often people refuse an upgrade if its not a window?