Shennong quest Part 2

I can't trigger Shennong's quest where you go to the village. I'm trying to get true ending and I have everything else done. I've given Shuan Shuan all the artifacts including the amulet and grown the tree fully, I've given Kuafu the bomb, I've got all of shennongs poison. But I can't get Shennong to move out of the pavilion. Any help would be greatly appreciated, thanks!