We are far from that point for abstract tasks

We are very very far from that point

I've never lived in Japan but I've spent a good deal of time there (probably about 3-4 months total over the course of the last 6-7 years), and I have friends who are Japanese that I have met up with and even traveled with.

I think you'll find that for most of these things, you can get them in New York City. Lots of small local shops, everything is within walking distance or a few stops away on the subway. It is not as safe as most parts of Japan, but its still pretty safe compared to most big cities in the US, and I've seen many people walking around alone at night in Central Park, which is surprisingly quiet and peaceful then.

They'll be a lot of things you'll miss. The Golden Gai is not a tourist hot spot for nothing; that's not to say that New York does not have a million cool local bars you can run into by accident, great food prepared with care by chefs, diverse groups for any interest you could think of, and certainly more than Japan every single possible cuisine from everywhere in the world at high quality.

And while you might miss the Japanese rent, you are not going to miss the Japanese wages. Its true, you have to work in tech or finance to live in Manhattan for the most part, and Brooklyn seems to be filled with trust-fund babies struggling to launch their career in arts (though I've met a few who are doing decently well as video fx artists and UX designers), but if you do manage to secure a decent wage, life will be similar enough, and things will feel affordable, even if they are not by any other standard.

Just like anything else, you either think "that's definitely wrong" or "huh, I guess that's probably it." If its really serious, you have to pause and make of a judgement call of course.

The fact that it can, and often does, get things working, sometimes even well, is evidence enough. It can't do some things, it can easily do others, and knowing which is which is very important nowadays.