How strong does an earthquake have to be to cause damage?

It is highly dependant on the magnitude, and your distance from the quake origin (surface and depth of the quake).

A shallow 3.0 can significantly damage a building located at the epicenter.