Should I be concerned with my speed during roadwork?

I run usually anywhere between 2 and 4 miles twice a week as told by my coach, but I am admittingly pretty slow when I run. I still am working fairly hard when running, and my heart rate seems consistently high. Should I be concerned with how fast I run, or just whether or not my hearts doing work?