Lime tree vs lemon

I have a lemon and lime tree, both bought a year and a half ago at Home Depot. So all summer my lime tree was fine, we got about 8 limes but the lemon tree was very yellow and not really thriving. So I brought them inside early November. And the lemon tree is now thriving and the lime tree has lost over half its trees. The lime tree has had flowers and they just fell off and there are branches that are completely bare. Any thoughts on perking up the lime tree. Pics of both trees.