Leaving new cat alone… I feel so bad

My husband and I adopted the most adorable 1 year old cat from the shelter 6 days ago. We bonded with him very fast and I’ve been staying with him in the apartment (3 bed, 2 bath + upstairs loft) every day since we got him. My husbands left here and there throughout the week and he hasn’t had any anxiety. We have no children, no roommates, and no other animals

On our first day back from the shelter we setup a base camp for him in my husbands office, and within a few hours we were all laying on the floor together and he was sleeping belly up. He was confident enough so we let him explore the rest of the apartment and we all slept in bed together in our bedroom that night. Every day since then our bond and affection with him has grown so much, it feels like we’ve had him for weeks now.

Tonight (day 6) was the first time he’s been alone in the apartment. We went to a concert tonight and were out of the house for 5.5 hours (5:30-11pm). When we got back home, he was hiding on the kitchen island stool staring at the door (kitchen is right next to the door), which was right where we left him. I unplugged everything and tucked away any hazards before we left, but after looking around he clearly didn’t go anywhere and the robot litter was unused. He also did a big stretch when he came down. Within minutes of resettling with us he used the litter. We’re pretty sure he stayed in that spot for the whole time we were out just staring at the door.

Tonight he’s very skiddish and it’s like he’s a bit unfamiliar/nervous with parts of the apartment now. I feel so bad for him, thinking that he was sitting there that whole time. He also clearly hates my perfume smell (never using perfume again lol). Thankfully we’re in bed together now and we got lots of pets in, but he bolts at anything that spooks him and I have to coax him back. Laying in bed now feeling so guilty…

We’re both first time cat owners. How do I prevent separation anxiety? Is this normal? We have to be out of the house tomorrow again for 3.5 hours and I’m feeling so bad for him. I want him to be his confident self again walking around from room to room, and I also don’t want to accidentally develop any unhealthy dependencies on us/me.