Searching & Scraping

I am trying to automate some “fact answer and verification”, where given a search, find some number of sources, pull referenced pages, extract results,and rank multiple sources based on the site reputation. For example, let’s say for sport S, who won X vs Y on date Z. Or what movies were nominated/won for award A in year Y. That sort of thing. This is straightforward in many cases and individual cases could be automated with a custom script. But let’s say for some particular fact list the results aren’t indexed. I wanted to have an agent basically do what a human would do, “following leads”, and essentially exploring and navigating to wherever was necessary. I got this to work using Claude when the sources were “open”, but it refused to crawl anything where robots policy denied access. I’m not doing bulk robot operations, it’s low-rate/low-traffic web use like what a human would do, so I’m not feeling the robots policies really apply. So I turned my attention to local models, and have qwen running on a strix halo under llama-swap. Not quite sure what the best flow here is; should I build a custom agent loop and give it tool access and a sandbox, and task it to curate a library of “scraper tools” as needed to answer questions? Or should I just use existing bulk crawl/scrape tools (I’m not familiar with these) and then try to pluck the results from the file tree? I’m running completely async and distributed, based on a queue, so this isn’t a “run inside Hermes” type situation. Speed isn’t a major priority, but accuracy is. These are such majorly different architectures that I figured I’d ask for advice before diving in. I have no experience with scraping results off the web, and all the advice I’ve seen is for the “pre-AI” era. Thanks for any suggestions!