I have never found my poly bushes to be noisy, certainly wouldn't be swapping out stock ones all the time. Having had the car 10 years, I reckon we'd be up to the 3rd set.

There's a lot of movement in the front end of these cars with the stock rubber.

I'd not be critical of pull under braking while on a loose surface. Differing levels of grip and camber on the gravel could be the main culprits.