Technically the Palantiri were a force for good in the hands of Elves and Men, and could still be used for good, like Aragorn using it to challenge Sauron and forcing Sauron’s hand. So that’s a defense to the self-awareness argument. In fact that ambiguity is likely intentional.
Btw I always wondered why I was seeing droves of Palantir swag on Stanford campus back in early 2010s. I wouldn’t wear something that has a 50%+ chance of being interpreted as evil.
That's not the perfect defense you think it is. Plenty of robots.txts[1] technically allow scraping their main content pages as long as your user-agent isn't explicitly disallowed, but in practice they're behind Cloudflare so they still throw up Cloudflare bot check if you actually attempt to crawl.
And forget about crawling. If you have a less reputable IP (basically every IP in third world countries are less reputable, for instance), you can be CAPTCHA'ed to no end by Cloudflare even as a human user, on the default setting, so plenty of site owners with more reputable home/office IPs don't even know what they subject a subset of their users to.
[1] E.g. https://www.wired.com/robots.txt to pick an example high up on HN front page.