How come magical creatures only infest wizard homes?

In OotP they’re cleaning out Grimmauld Place and there are infestations of different magical creatures. How is it that those things only infest magical homes? If they’re drawn to magic then wouldn’t they also be in the homes of muggle borns before they know they’re wizards?