Feeling pressured from realtor because I want to change lender after offer has been signed

I had my offer accepted on my first house as of today (woohoo!) But am running into a situation where I feel quite pressured from my realtor. We moved really fast on the house and closing is only 2 weeks from now. Since I've been talking with my realtor, he immediately suggested Lender A that he's been working with for a while. I went forward with him as I was getting ready for my offer because of this. However, I was never informed that I would have to "commit" to a lender once the contract was made and my realtor had just assumed that I was going to go with Lender A.

Come today, a family member had suggested Lender B who I phoned up and got a .5% lower rate because of the FTHB program that has a lenient income restriction. That's almost a $2k yearly difference than the previous lender I had been working with. Realtor thinks that a difference of .5% means something sketchy is going on, but Lender B has been very communicative all afternoon and assured me that he can get me to the finish line on time (who knows if this is true though).

In the contract, we had waived the financial contingency and I can't find anywhere on the contract that states that I am going with the previous lender. However, my realtor claims that if I choose a different lender now, I will have to re-open the contract and seller's agent can back out/refuse to work with Lender B since neither of them are familiar.

I understand that I am very pinched with closing being so soon, but I am definitely feeling pressured with this. How does it make sense that it's expected to commit to a lender BEFORE being under contract if neither of them can even give me a final estimate until AFTER the contract is signed? What would you do in this situation? Should I risk potentially not closing in time/turning off the seller from going through with this to get the better rate?