left joystick rubbing against housing

when I rotate my left joystick around the outside of the housing, it rubs and seems to be wearing material off, the right joystick is all smooth and has no signs of wear. is there something I can do to stop this? do I need a replacement stick topper?