I have done quite a fair bit of scraping over the last year, and I have to say that the combo of PhantomJS / CasperJS is really unbeatable. I have had to navigate some fairly awful DOM structures replete with errors, confirm dialogs, IE-only features, horrendous endless iframe trees, and more fun stuff. There's nothing I haven't been able to plow through yet using Phantom/Casper.