I'm soon to read Progress and Poverty

So far, my only reference for a LVT has been the disastrous Zamindari system under the East India Company. I understand that this system was inherited from the Mughals (and somewhat functional), but when exposed directly to the international markets, the land assessments grew beyond the capacity of the farmers on the land.

What prevents something similar from happening in a Georgist system? If we distinguish "best use" from "best returns", how will we quantify the difference and keep from inventing arbitrary valuations for "a library that faces the sun at dawn" or for a large plot of land growing wasted, subsidized victuals due to a lobbying scheme?