Why didn’t Voldemort go after Petunia pre-PS

We see the Dursleys at the start of PS and they were just living normal life while Voldemort would have been actively hunting down the Potters. Probably just plot convenience though.