How are different sized buildings affected by earthquakes?

The simple answer.

The more mass you have further from the bottom (ground level in this case) the stronger the earthquake affects the building.

So, a huge one story (low) building will withstand the same earthquake better than a forty story building with a small footprint. All other things being equal.

That being said, the massing, the shape (yes, those are different), what the structure is composed of, what the non-structural components are composed of, and many more factors all go into the mix, but the most basic mass vs height.