Feeling overwhelmed

I have just started playing the game and noticed that the amount of collectible items and side quests in this game is massive, and I try to pick up as many items and clear as many quests as possible fearing that I might be missing out on something and my character might not get enough experience and good equipment but doing so is kinda exhausting and time wasting, so I was wondering how much side tracking is enough?