Orlando local and first-time home buyer. Question: Is Sanford really that bad?

Local here! My husband and I are finally looking to purchase our first house, but unfortunately we have a limited budget. We are looking at single-family homes, largely due to needing the FHA mortgage which doesn't finance condo/townhomes. Anyway.. In our price range, most houses are gone before we even get to see them. Sanford has a lot of houses in our price range, but I've heard awful things about the area. Even as a long time Orlando resident, I haven't explored Sanford very much beyond going to the Zoo and Hollerbach's. Right now, our number 1 option in the area is actually within walking distance of the downtown area (think Hollerbach's, the park, etc). But, would this be a horrible place to live? I'd love being close to a downtown area that's walkable, but would it even be safe to walk around? Any feedback is much appreciated! Thanks all!