“Adopted” a stray cat 2 years ago

Hello all, I need some advice. I started feeding a stray from the street 3 years ago and brought him inside 2 years ago. He still won't let me touch him and will run away from me and hide. I think I should let him back outside and just give him food if he comes back but idk. I WANT to keep him but not if it's torture for him living here. He gets along with the cat I got from the shelter if that helps.