I can't change my schedule without freaking out

For all my life I have always relied on a schedule. I eat at the same time everyday, I sleep at the same time every day, reading a book 30 minutes before bed. I go to work at same time every day, despite being able to start whenever.

I can't go to any events or parties that interfere with my schedule - even though it's a event i would like to attend. This has made me lost friends over the years. I can eat later, but it makes me really stressed. But when I can't go to bed at my usual time I freak out, I get extremely anxious. There has been times when I was stuck in train and got home late and I am totally panicking after getting home 20-30 minutes later then usual.

I have tried to change things but it's so difficult with all the anxiety that it causes. It should not be the end of the world to go to bed a bit later.