I think the best example I've seen of the pipeline approach: ASR(STT) -> LLM -> TTS is still from about 10 months ago in HN https://github.com/KoljaB/RealtimeVoiceChat

Just a pity the code is undecipherable :(

Don't be so hard on yourself :P

what do you mean "infer"? how does the LLM get anything it of this as input?

I guess if I'd done all that work on lobster and then bipolar built a ray traced voxel engine on top of that, end game would be to licence the engine. You have something here that the other game engines don't have, and something Minecraft is not likely to have (they are pretty stagnant development wise). This is easily 100 times better than Roblox. If you focus on making the world building tools easy to use and modding/game design you could easily be the next big thing. Don't get too bogged down in game design other than to use it as a proof of concept to help you understand what game designers will need.

Try norther Ireland.