Interested to know more about your inference start up? How you guys operating, do you own hardware or use the cloud?

https://gitlab.com/usecaliper/caliper-python-sdk

An LLM observability SDK that let's you store pre and post request metadata with every call in as lightweight an SDK as possible.

Stores to S3 in batched JSON files, so can easily plug into existing tooling like DuckDB for analysis.

It's designed to answer questions like; "how do different user tiers of my services rate this two different models and three different systems prompts?". You can capture all the information required to answer this in the SDK and do some queries over the data to get the answers.

That's still the same crawler system though. And it's lazy engineering to not build in something to track when you last requested a url.

And it's quite a trivial feature at that.

Tbh, for most companies/orgs the cost/complexity of multi region just isn't worth it.

The cost of a work days worth of downtime is rarely enough to justify the expense of trying to deploy across multiple regions or clouds.

Esp if you are public facing and not internal. You just go 'well everyone else was down to because of aws' and your customers just go 'ah okay fair enough'