AITA for strictly following the visitation plan instead of letting my ex have extra time with our son?

I (34) have custody of my son (13). His mother (33) and I broke up when he was 18 months old. We were on good terms at that point and had co-parented well. When our son was in second grade he had a horrific time with his teacher. For whatever reason this teacher, who I'll call him Sam for the sake of the post, singled out my son and treated him like shit. He was a bully. My son had no trouble with the two previous teachers but this guy always had complaints about my son. He'd send home notes complaining that our son was annoying because he was too quiet and he needed to speak up and stop being quiet, that my son wouldn't ask questions when he needed to and that he got upset a lot in class. He'd correct my son's homework and mark as wrong when it wasn't and when I'd reach out to bring this up he'd ignore me.

I was in the school a lot about this. I tried talking to the teacher but his attitude toward my son was awful. He even said my son was too old to be crying to daddy about issues with his teacher. My ex was involved at the start too and had made complaints. We spoke to the principal and I requested several times to have my son placed in another classroom but I was told it wasn't an option. It took far too long for them to agree to this and it came when other kids confirmed issues my son brought up. Like how the teacher would go out of his way to embarrass my son by telling him boys are loud and can be heard and don't sound like babies, that he'd always say my son got something wrong (even when he didn't) while the others would just get corrected on paper or privately and how he mocked my son whenever my son would ask questions, which explains why he wouldn't ask questions when he needed to. He also called my son names in front of the rest of the class. It was heartbreaking to hear just how badly he treated my son.

At some point in all of this my ex decided that Sam, the man who treated our son like shit, would be a good partner. She hid the relationship for a while and then she told our son one day when he was with her. She wanted him to join them for a day out and he became hysterical. There was a lot of discussion that happened and a lot of it was us angry with each other because I could not understand. She told me I didn't know Sam the man and our son and I would need to get over it. She moved him in weeks later and sprung that on our son like it was nothing. This led our son to run from her house a number of times, he was having emotional issues, he was even shutting his therapist out who he'd been seeing since the issues with Sam in school had become a source of pain for him. She refused to give up Sam and insisted they would live together and get married, which they later did. This led to me asking for custody of our son and filing with the courts. It wasn't straightforward or quick but it was agreed that the strain of living with Sam was not good for my son and I was given custody and my ex visitation.

This visitation does not include overnights. It also cannot include Sam.

My ex has not been the most consistent with the visitation. There are times she has dragged Sam along and I turned them away and she refused to take the time with our son alone. She does see him most of the time but she has not given up on trying to make our son get over the trauma Sam caused. Sam still has a dislike for our son and I'm putting that very mildly. He still gives me the vibe that he'd bully my son happily.

My ex resents losing custody and she resents that she can't see our son whenever she wants to. My son doesn't want to see his mom but he has to if she shows up alone for the visitation. The judge reinforced that only recently when she tried to get more custody time and my son said he'd like to have the choice to say no. She's asked for more visits on a few occasions but I always stated we needed to follow the visitation plan. But then she learned her family was spending more time with our son than she was and she wanted him for her birthday. That wasn't one of her days and I said no. She told me it wasn't right to deny them time and that I was punishing her for falling in love with Sam. She accused me of not wanting our son to change his opinion of Sam and engaging in alienation of Sam. I refused to engage but she told me I never say yes to extra time and I shouldn't be so strict about following the plan and nothing extra because our son needs his mom.

I feel like I'm doing the right thing and not just because of the plan but my son's wishes are also on my mind. But maybe I'm letting my disgust about this turn me into an asshole. Am I?