Hacker News

Python 3.11 vs 3.10 performance

2022-07-0615:05715444github.com

Contribute to faster-cpython/ideas development by creating an account on GitHub.

Benchmark 2022-05-05_17-47-3.10-9b47252d5433 2022-05-05_16-04-main-ae553b3561f0
deltablue 12.4 ms 6.35 ms: 1.96x faster
go 381 ms 229 ms: 1.66x faster
logging_silent 281 ns 169 ns: 1.66x faster
scimark_sor 324 ms 199 ms: 1.63x faster
scimark_monte_carlo 177 ms 113 ms: 1.57x faster
raytrace 782 ms 505 ms: 1.55x faster
richards 123 ms 79.5 ms: 1.54x faster
pyflate 1.08 sec 699 ms: 1.54x faster
crypto_pyaes 193 ms 126 ms: 1.54x faster
chaos 174 ms 114 ms: 1.53x faster
float 184 ms 125 ms: 1.48x faster
pickle_pure_python 740 us 502 us: 1.48x faster
scimark_lu 274 ms 187 ms: 1.47x faster
nbody 229 ms 158 ms: 1.45x faster
hexiom 15.4 ms 10.6 ms: 1.45x faster
mako 25.1 ms 17.6 ms: 1.43x faster
spectral_norm 248 ms 174 ms: 1.43x faster
xml_etree_process 125 ms 94.4 ms: 1.33x faster
logging_simple 13.5 us 10.2 us: 1.33x faster
django_template 73.5 ms 56.0 ms: 1.31x faster
logging_format 14.5 us 11.0 us: 1.31x faster
unpickle_pure_python 490 us 382 us: 1.28x faster
tornado_http 212 ms 165 ms: 1.28x faster
regex_compile 286 ms 223 ms: 1.28x faster
2to3 533 ms 419 ms: 1.27x faster
chameleon 14.5 ms 11.6 ms: 1.25x faster
fannkuch 784 ms 642 ms: 1.22x faster
sqlalchemy_imperative 35.0 ms 29.1 ms: 1.20x faster
sqlalchemy_declarative 267 ms 222 ms: 1.20x faster
scimark_fft 677 ms 568 ms: 1.19x faster
scimark_sparse_mat_mult 8.87 ms 7.46 ms: 1.19x faster
sqlite_synth 4.80 us 4.06 us: 1.18x faster
dulwich_log 125 ms 106 ms: 1.18x faster
nqueens 164 ms 139 ms: 1.18x faster
xml_etree_generate 157 ms 134 ms: 1.17x faster
unpack_sequence 96.9 ns 82.9 ns: 1.17x faster
sympy_integrate 39.2 ms 34.2 ms: 1.15x faster
sympy_expand 869 ms 772 ms: 1.13x faster
sympy_sum 303 ms 270 ms: 1.12x faster
sympy_str 527 ms 472 ms: 1.12x faster
regex_v8 39.7 ms 35.8 ms: 1.11x faster
regex_effbot 5.30 ms 4.79 ms: 1.10x faster
python_startup 15.4 ms 13.9 ms: 1.10x faster
json_loads 48.5 us 44.1 us: 1.10x faster
pathlib 33.6 ms 30.8 ms: 1.09x faster
unpickle 23.7 us 21.9 us: 1.08x faster
pickle_list 7.55 us 7.04 us: 1.07x faster
json_dumps 22.6 ms 21.3 ms: 1.06x faster
xml_etree_iterparse 176 ms 167 ms: 1.05x faster
regex_dna 353 ms 337 ms: 1.05x faster
meteor_contest 192 ms 185 ms: 1.04x faster
pickle 17.2 us 16.7 us: 1.03x faster
xml_etree_parse 255 ms 248 ms: 1.03x faster
telco 11.4 ms 11.1 ms: 1.02x faster
pickle_dict 50.4 us 49.6 us: 1.02x faster
python_startup_no_site 9.98 ms 10.4 ms: 1.04x slower
pidigits 316 ms 346 ms: 1.10x slower
Geometric mean (ref) 1.25x faster

Benchmark hidden because not significant (1): unpickle_list

Intel(R) Xeon(R) D-2141I CPU @ 2.20GHz 41-Ubuntu SMP Mon Sep 20 16:39:20 UTC 2021


Read the original article

hyperbovine

Karma: 10404
...
 @Hacker__News
@hacker._news

Comments

  • By antman 2022-07-0618:128 reply

    There was always a denial of removing the Global Interpreter Lock because it would decrease single threaded Python speed for which most people din’t care.

    So I remember a guy recently came up with a patch that both removed GIL and also to make it easier for the core team to accept it he added also an equivalent number of optimizations.

    I hope this release was is not we got the optimizations but ignored the GIL part.

    If anyone more knowledgable can review this and give some feedback I think will be here in HN

    • By jart 2022-07-0618:402 reply

      I use a beautiful hack in the Cosmopolitan Libc codebase (x86 only) where we rewrite NOPs into function calls at runtime for all locking operations as soon as clone() is called. https://github.com/jart/cosmopolitan/blob/5df3e4e7a898d223ce... The big ugly macro that makes it work is here https://github.com/jart/cosmopolitan/blob/master/libc/intrin... An example of how it's used is here. https://github.com/jart/cosmopolitan/blob/5df3e4e7a898d223ce... What it means is that things like stdio goes 3x faster if you're not actually using threads. The tradeoff is it's architecture specific and requires self-modifying code. Maybe something like this could help Python?

      • By pm 2022-07-0619:37

        I would love to see this implemented purely for curiosity's sake, even if it's architecture-specific.

        Personally, cosmo is one of those projects that inspires me to crack out C again, even though I was never understood the CPU's inner workings very well, and your work in general speaks to the pure joy that programming can be as an act of creation.

        Thanks for all your contributions to the community, and thanks for being you!

      • By jfim 2022-07-0715:53

        That's a pretty clever hack, nicely done!

    • By BerislavLopac 2022-07-0618:432 reply

      The GIL removal by that guy reverted some of the improvement done by other optimisations, so the overall improvement was much smaller.

      And most people do care for single-threaded speed, because the vast majority of Python software is written as single-threaded.

      • By metadat 2022-07-0619:024 reply

        > the vast majority of Python software is written as single-threaded.

        This is a self-fulfilling prophecy, as the GIL makes Python's (and Ruby's) concurrency story pretty rough compared to nearly all other widely used languages: C, C++, Java, Go, Rust, and even Javascript (as of late).

        • By tyingq 2022-07-0619:241 reply

          Getting rid of the GIL will also immediately expose all the not-thread-safe stuff that currently exists, so there's a couple of waves you would need before it would be broadly usable.

          • By tempest_ 2022-07-0622:234 reply

            Cool, they should start now.

            As a python dev, pythons multiprocess/multithreading story is one the largest pain points in the language.

            Single threaded performance is not that useful while processors have been growing sideways for 10 years.

            I often look at elixir with jealousy.

            • By IceWreck 2022-07-0623:577 reply

              Or maybe keep things the way they are.If you really need performance python is not the language you should be looking for.

              Instead of breaking decades of code, maybe use a language like Go or Rust for performance instead.

              • By Mehdi2277 2022-07-071:292 reply

                Python is also the dominant language for machine learning which does care for performance. The person who made recent nogil work is one of the core maintainers of key ML library. The standard workaround is ML libraries, the performance sensitive stuff is written in C/C++ (either manually or with cython) and then uses python bindings. But it would be much friendlier if we could just use python directly.

                It's also commonly used language for numerical work in general. Most of the time numpy is enough and then occasionally you'll need something not already implemented and then have to do your own bindings.

                • By cycomanic 2022-07-076:10

                  > Python is also the dominant language for machine learning which does care for performance. The person who made recent nogil work is one of the core maintainers of key ML library. The standard workaround is ML libraries, the performance sensitive stuff is written in C/C++ (either manually or with cython) and then uses python bindings. But it would be much friendlier if we could just use python directly.

                  Multithreading is not really the reason why things get written in cython etc., you can easily see 100x improvements in single threaded performance (compared to maybe a factor of 2-8x for multithreading). If you care about performance you'd definitely write the performance critical stuff in cython/pythran/c.

                • By pjmlp 2022-07-0710:15

                  Nope, C++ and Fortran are.

                  The bindings available in Python can also be used from other languages.

              • By xwdv 2022-07-071:282 reply

                I would have thought convincing people they’ll just have to use Go or Rust or Elixir would have been an easy sell around here.

                Turns out they just want a better Python.

                • By fnord123 2022-07-077:531 reply

                  >Turns out they just want a better Python.

                  That's Go. It gives actual types[1] and structs (so you dont have to wonder about dict, class, class with slots, dataclasses, pydantic, attrs, cattrs, marshmallow, etc). It removes exceptions and monkeypatching. It's async-first (a bit like gevent). It's inherently multicore. And you can package and distribute it without marking a pentagram on the ground and sacrificing an intern to Apep.

                  You just need to stop being friends with C FFI. Which is fine for web and sysadmin tools. For data science and ML/AI/DL, it's less ok.

                  [1] And the types are actually checked! I think a lot of people aren't really using python type checking given how slow it is and no one seems to complain. Or maybe everyone is using pyright and pyre and are satisfied with how slow these are.

                  • By kaba0 2022-07-0714:041 reply

                    Going to the very unexpressive Go from the expressivity of python is a goddamn huge jump though.

                    Going to JS or even TS for performance would be saner, and it has a same-ish object model even.

                    • By fnord123 2022-07-0715:34

                      The expressivity in Python is a problem that needs to be solved though. Moving to JS goes the wrong way.

                • By pjmlp 2022-07-0710:16

                  It already exists, but they don't want to learn other languages.

              • By shepardrtc 2022-07-072:28

                > Instead of breaking decades of code

                Pin your version.

              • By bayindirh 2022-07-0712:55

                Concurrency is not solely required for performance. I'm designing a small tool (a file ingester/processor) for myself, which gonna need several, really concurrent threads. I love Python, but I can't use that, so I'm learning Go.

              • By kaba0 2022-07-0714:02

                Why put Go and Rust in the same category? I never really understood that.

                Either include like almost every language from JS, Java, C# to Haskell, or just list C++ and Rust. But Go is in the former category.

              • By chaps 2022-07-0715:06

                Python has basically already done exactly that with 2.7 to 3 and we came out of that relatively fine.

                I say bring it.

              • By Dangisshit 2022-07-074:33

                So you are essentially saying Python is obsolete. It's used for decade old code and for new code you should use go or rust.

            • By Galanwe 2022-07-071:042 reply

              > As a python dev, pythons multiprocess/multithreading story is one the largest pain points in the language.

              Hmm, how is that so?

              As a python dev as well, I don't have much complaint with multiprocessing.

              The API is simple, it works OK, the overall paradigm is simple to grok, you can share transparently with pickle, etc.

              • By nine_k 2022-07-073:221 reply

                Multiprocessing is fine, but the cost of starting another interpreter is pretty visible, so you need to create a pool, and it may not be an overall speedup if the run time is short.

                It takes more careful planning than async in JS?, say, or goroutines.

                • By geysersam 2022-07-076:191 reply

                  Yeah but for JS style async you'd use probably use an event loop in Python, not multi processing.

                  • By nine_k 2022-07-078:571 reply

                    Yes. But, frankly, async is also simpler in JS than in Python: e.g. no need to start a reactor loop.

                    • By Too 2022-07-084:311 reply

                      Starting the event loop is no worse than any setup of a main function, it’s a oneliner: asyncio.get_event_loop().run_until_completion(my_async_main)

                      • By BerislavLopac 2022-07-1011:30

                        Errr no, that has been replaced with asyncio.run quite some time ago.

              • By plonk 2022-07-0912:03

                Pickle isn't transparent though, custom objects that wrap files or database sessions need to override serialization.

                The ProcessPoolExecutor is nice but shouldn't be necessary.

            • By oofbey 2022-07-0623:441 reply

              On the flip side, if your workload can be parallelized across thousands of cores, python has about the best CUDA support anywhere.

              • By pjmlp 2022-07-0710:161 reply

                That would be C++ and Fortran actually.

                • By p10_user 2022-07-0716:441 reply

                  But the python bindings are great and useful to many, so Python gets to be added to the list.

                  • By pjmlp 2022-07-0720:00

                    Just like any language with FFI capabilities to call the same libraries.

            • By Nimitz14 2022-07-0623:52

              As another python dev, it has basically never been a painpoint for me.

              Your anecdote adds little.

        • By SiempreViernes 2022-07-0619:191 reply

          Pretty sure it is actually a statement that comes from the "python is a scripting language" school, and not because the huge horde of programmers that craves concurrency when they write a script to reformat a log file to csv keeps being put off by the python multiprocessing story.

          • By metadat 2022-07-0619:221 reply

            Not sure I understand your point, can you clarify? Python is used across many different domains, being able to really take advantage of multiple cores would be a big deal.

            I'd really appreciate if python included concurrency or parallelism capabilities that didn't disappoint and frustrate me.

            If you've tried using the thread module, multiprocessing module, or async function coloring feature, you probably can relate. They can sort of work but are about as appealing as being probed internally in a medical setting.

            • By pclmulqdq 2022-07-0623:482 reply

              I'm not the person you responded to, but I think the gist of it is: what it is is not defined by how it is used. Python, at its core, is a scripting language, like awk and bash. The other uses don't change that.

              Occasionally, a technology breaks out of its intended domain. Python is one of these - it plays host to lots of webservers, and even a filesystem (dropbox). Similarly, HTML is a text markup language, but that doesn't stop people from making video games using it.

              The developers of the technology now have to make a philosophical decision about what that technology is. They can decide to re-optimize for the current use cases and risk losing the heart of their technology as tradeoffs get made, or they can decide to keep the old vision. All of the design choices flow down from the vision.

              They have decided that Python is a scripting language. Personally, I agree (https://specbranch.com/posts/python-and-asm/). In turn, the people using the language for something other than its intended purpose have choices to make - including whether to abandon the technology.

              If instead Python moves toward competing with languages like go, it is going to need to make a lot of different tradeoffs. Ditching the GIL comes with tradeoffs of making Python slower for those who just want to write scripts. Adding more rigorous scoping would make it easier to have confidence in a production server, but harder to tinker. Everything comes with tradeoffs, and decisions on those tradeoffs come from the values of the development team.

              Right now, they value scripting.

              • By metadat 2022-07-070:113 reply

                Python has already become a lot more than a scripting language. To say today that scripting is it's core identity seems naive at best. Yes, it has roots but has object oriented and functional facets which do not exist in awk or bash.

                Pandas, numpy, scipy, tensorflow. All of these go way beyond what is possible with a scripting language.

                Since when is the runtime performance of a script a serious concern? Why is it a problem if this aspect gets slightly slower if it brings real concurrency support?

                • By pclmulqdq 2022-07-070:382 reply

                  Awk is a functional language with a surprising number of features. Bash, not so much. Developers really do care about runtime performance of scripting languages: They often wait for scripts to finish running before doing other work, and if the sum of the time it takes to write and execute a script is too long, they will look for an alternative.

                  All of the libraries you have cited are from the ML and stats communities, and they are not core language features. From what I understand, ML folks like Python because it is fast to play with and get results. In other words, they like it because it is a scripting language.

                  Personally, I like that Python has kept the GIL so far because I would never run a 24/7 server in Python and I am happy to use it very frequently for single-threaded scripting tasks.

                  Edit: I didn't decide that Python was a scripting language. The Python maintainers did. The point is that the identity of a project doesn't flow down from its use cases.

                  Edit 2: I should have said "the identity of a project doesn't flow down from its users."

                  • By ghshephard 2022-07-074:251 reply

                    "Personally, I like that Python has kept the GIL so far because I would never run a 24/7 server in Python and I am happy to use it very frequently for single-threaded scripting tasks."

                    Just as a side-note - my prior gig used Python on both the Server and Data Collection industrial systems. It was very much a 24x7x365 must-never-go-down type of industrial application, and, particularly when we had a lot of data-sources, was very multi-processing. Was not unusual to see 32 processes working together (we used sqlite and kafka as our handoff and output of processes) running on our data collection appliances.

                    Our core data modelling engine would routinely spin up 500 worker pods to complete the work needed to be done in 1/500th of the time, but we would still see some of the long term historian runs take upwards of a week to complete (many hundreds of machines for multiple years with thousands of tags coming in every 5 seconds is just a lot of data to model).

                    I say this mostly to demonstrate that people and companies do use python in both large-processing intensive environments as well as industrial-must-never-stop-24x7 mission critical appliances.

                    I don't ever recall any of our engineers looking at Python as "a scripting language" - it was no different to them than Java, C#, Rust, Go, C++ or any other language.

                    • By pclmulqdq 2022-07-0710:572 reply

                      I know that people do use python for 24/7 "must not fail" applications. I'm just not smart enough to write python that I would trust like that. Python comes with a tremendous number of foot guns and you have to find them because there is no compiler to help, and it can be a real pain to try to understand what is happening in large (>10,000 line) python programs.

                      • By ghshephard 2022-07-0922:20

                        I think the key here is to see using the Python language as an engineering discipline like any other, and just take the classes, read the literature, learn from more Senior Engineers, and projects, before attempting to develop these types of systems on your own.

                        I don't think anybody expects a recent graduate from computing science, with maybe 4 or 5 classes that used python under their belt (and maybe a co-op or two) to be writing robust code (perhaps in any language).

                        But, after working with Sr. Engineers who do so, and understanding how to catch your exceptions and respond appropriately, how to fail (and restart) various modules in the face of unexpected issues (memory, disk failures, etc...) - then a python system is just as robust as any other language. I speak from the experience of running them in factories all over the planet and never once (outside of power outages - and even there it was just downtime, not data loss) in 2+ years seeing a single system go down or in any way lose or distort data. And if you want more performance? Make good use of numpy/pandas and throw more cores/processes at the problem.

                        Just being aware of every exception you can throw (and catching it) and making robust use of type hinting takes you a long way.

                        Also - and this may be more appropriate to Python than other languages that are a bit more stable, an insane amount of unit and regression testing helps defend quite a bit from underlying libraries like pandas changing the rules from under you. The test code on these project always seemed to outweigh the actual code by 3-5x. "Every line of code is a liability, Every test is a golden asset." was kind of the mantra.

                        I think that what makes python different from other languages, is that it doesn't enforce guardrails/type checking/etc... As a result, it makes it trivial for anyone who isn't an engineers to start blasting out code that does useful stuff. But, because those guardrails aren't enforced in the language, it's the responsibility of the engineer to add them in to ensure robustness of the developed system.

                        That's the tradeoff.

                      • By plonk 2022-07-0912:07

                        It's not that hard, we have compute-intensive servers running 24/7 in production and written entirely in Python on our side, using C++ libraries like PyTorch.

                        You just have to isolate the complicated parts, define sensible interfaces for them, and make sure they are followed with type hints and a good type checker.

                  • By Supermancho 2022-07-071:07

                    >> To say today that scripting is it's core identity seems naive at best. Yes, it has roots but has object oriented and functional facets which do not exist in awk or bash. Pandas, numpy, scipy, tensorflow. All of these go way beyond what is possible with a scripting language.

                    No, it is not. It's what you can do with a scripting language, which is a fairly straightforward functional categorization. If you are very familiar with it and really want to just ignore the performance issues and don't mind strange workarounds (eg elaborate libraries like Twisted), you probably use python.

                    > The point is that the identity of a project doesn't flow down from its use cases.

                    Use cases are what determines the identity of a language. It takes a massive and ongoing marketing campaign to convince people otherwise, with limited success without use-cases. Python is popular because it's a scripting language with a relatively low learning curve (one true way to do things philosophy). That's it. It will improve over time, but it's been slow going...and that's in comparison to the new Java features!

                    Haskell is a shining example of how the language identity is foisted upon the public until you are convinced enough to try it. It doesn't take long to learn it's a nightmare of incomplete features and overloaded idioms for common tasks, so it isn't used. There aren't good use-cases for a language with haskell's problems, so the developer community and industry-at-large avoids it.

                • By p10_user 2022-07-0716:52

                  But all the “magic” that makes the scientific stack so great is largely due to numpy (and many other Fortran or c driven code) being fantastic. Python is the glue scripting language for organizing and calling procedures. That’s,IMO, it’s original and core purpose.

                  Now the fact that you can do some fun metaprogramming shenanigans in Python just speaks to how nice it is to write.

                • By pjmlp 2022-07-0710:17

                  Except most of them are bindings written in native languages, being scripted from Python.

              • By FridgeSeal 2022-07-070:121 reply

                If we’re going to leave Python as a scripting language (fine by me), can we get the machine learning community to swap to something better suited?

                It strikes me as a bit of a waste of resources to keep stapling engineering effort into the Python ML/data ecosystem when it’s basically a crippled language capable of either: mindlessly driving C binaries, or scripting simple tasks.

                What other performance, feature and technique advancements are we leaving on the table because the only “viable” ecosystem is built on a fundamentally crippled language?

                • By pclmulqdq 2022-07-070:262 reply

                  From what I can tell, the ML community is moving toward Julia. I don't think anyone predicted that they would end up locked into Python so heavily.

                  • By Mehdi2277 2022-07-071:311 reply

                    That's not been my experience much at all work or research wise. Tensorflow, pytorch, jax are still very dominant. I've worked at several companies and interviewed at several dozen for ML roles. They have 100% been python/c++ for ml. I'd be impressed if even 2% of ML engineers used Julia.

                    • By disgruntledphd2 2022-07-0717:06

                      I feel like Julia will take more of the R people than the Python people, to be honest.

                  • By plonk 2022-07-0912:10

                    I wanted to use Julia for some experiments but it's so confusing. I would call a function with VSCode and get "potential function call error" or something, with no details. Is it valid or not?

                    Also, I hate the idea of re-building all the code when the program starts. Python's JIT can at least ignore the performance-critical code that's written in C++.

        • By citrin_ru 2022-07-0620:481 reply

          IMHO majority of Python software doesn't use threads because it is easier to write single threaded code (for many reasons), not because of GIL.

          • By metadat 2022-07-0621:223 reply

            In Golang you can spawn a green thread on any function call with a single keyword: `go'.

            The ergonomics are such that it's not difficult to use.

            Why can't or shouldn't we have a mechanism comparably fantastic and easy to use in Python?

            • By lmm 2022-07-0622:131 reply

              Because making it easy to write C/C++ extensions that work the way you expect (including for things like passing a Python callback to a C/C++ library) has always been a priority for Python in a way that it isn't for Golang?

              • By liuliu 2022-07-0622:581 reply

                Any C/C++ extensions that wants to enable more efficient Python has to learn GIL and how to manipulate that as well. Including not limited to: how to give up GIL (so that other Python code can progress); how to prepare your newly initiated threads to be Python GIL friendly etc.

                Personally, GIL is more surprising to me when interop with Python.

                • By lmm 2022-07-074:18

                  > Any C/C++ extensions that wants to enable more efficient Python has to learn GIL and how to manipulate that as well. Including not limited to: how to give up GIL (so that other Python code can progress); how to prepare your newly initiated threads to be Python GIL friendly etc.

                  Sure, but the easy cases are easy (in particular, you can usually do nothing and it will be correct but slow, which is much better than fast but incorrect) and the hard cases are possible.

                  > Personally, GIL is more surprising to me when interop with Python.

                  Any GCed language, including Go, will oblige you to integrate with its APIs if you want to handle complex cases correctly.

            • By Twirrim 2022-07-0621:31

              https://docs.python.org/3/library/concurrent.futures.html sort of gives you that, syntax works with threads or processes.

            • By metadat 2022-07-0621:35

              @Twirrim How would you rate the user experience of the concurrent.futures package compared to the golang `go' keyword?

              It is architecturally comparable to async Javascript programming, which imho is a shoehorn solution.

        • By dahart 2022-07-070:19

          I agree with your point, but the vast majority of C, C++, Java, and Javacript code is also written as single-threaded. It’s fair to acknowledge that the primary use case of most languages is single threaded programming, and also that improving the ergonomics of concurrency in Python would be a huge boon.

      • By magic_hamster 2022-07-070:18

        A lot of python runs in celery and possibly even more python runs parallelized cuda code. Not sure at all the majority of python code is single threaded, especially for more serious projects.

    • By nmstoker 2022-07-0618:42

      I think you mean the work by Sam Gross:

      https://github.com/colesbury/nogil/

      Interesting article about it here:

      https://lukasz.langa.pl/5d044f91-49c1-4170-aed1-62b6763e6ad0...

    • By int_19h 2022-07-0619:553 reply

      Removing GIL also breaks existing native packages, and would require wholesale migration across the entire ecosystem, on a scale not dissimilar to what we've seen with Python 3.

      • By icegreentea2 2022-07-0621:06

        They believe that the current nogil approach can allow the vast majority of c-extension packages to adapt with a re-compile, or at worst relatively minor code changes (they thought it would be ~15 lines for numpy for example).

        Since c-extension wheels are basically built for single python versions anyways, this is potentially manageable.

      • By smcl 2022-07-0620:14

        The various trade-offs and potential pitfalls involved in removing the GIL are by now very well known, not just here on HN but among the Python core developers who will ultimately do the heavy lifting when it comes to doing this work.

      • By josefx 2022-07-074:512 reply

        > on a scale not dissimilar to what we've seen with Python 3.

        If only people had asked for it before the Python 3 migration so it could have been done with all the other breaking and performance harming changes. But no, people only started to ask for it literally yesterday so it just could not ever be done, my bad. /sarcasm

        If anything the whole Python 3 migration makes any argument against the removal of the GIL appear dishonest to me. It should have been gone decades ago and still wasn't included in the biggest pile of breaking changes the language went through .

        • By ModernMech 2022-07-0715:58

          I think probably the reason is that in 2008, consumer chips were still mostly single or dual core. Today we have consumer grade chips that have dozens of cores, so the calculus has changed.

        • By gpderetta 2022-07-0713:08

          We didn't get any performance improvements, but at least now we can call print from lambda functions, that's was certainly worth the massive breakage.

    • By didip 2022-07-0618:32

      I agree, please don't just accept the optimization and sweep the GIL removal under the rug again.

    • By lrtap 2022-07-0619:08

      > I hope this release was is not we got the optimizations but ignored the GIL part.

      I would not be surprised. It is highly likely that the optimizations will be taken, credit will go to the usual people and the GIL part will be extinguished.

  • By alexfromapex 2022-07-0615:394 reply

    One of the biggest features I'm looking forward to is the more specific error messages. I can't tell you how much time I've wasted with cryptic errors that just point to a line that has a list comprehension or similar.

    • By CobrastanJorji 2022-07-0618:422 reply

      I've been learning Rust recently. There are a number of things about the language that I dislike, but its error messages are an absolute joy. They clearly put an incredible amount of effort into them, and it really shows.

      • By jmcgough 2022-07-0620:231 reply

        I wrote 2-3 of those and I really wish more languages had a similar approach with their standard library.

        When you get an error there's a verbose explanation you can ask for, which describes the problem, gives example code and suggests how you can fix it. The language has a longer ramp-up period because it contains new paradigms, so little touches like this help a lot in onboarding new devs.

        • By HdS84 2022-07-0716:57

          I am a hug fan of ravendb's founder ayende. He always stresses how useful good error messages are, especially for self service. And he is certainly on spot with that, e.g "could not execute Mode None" is much less helpful than " environment variable X not set, could not determine Mode. Please set this as MODE=Agressive or MODE=Passive".

      • By dralley 2022-07-0622:063 reply

        The exception is std::io::Error, which doesn't provide any information about which file failed to be opened/closed/read/whatever. I know this is because doing so would require an allocation, but it's still painful.

        • By tialaramex 2022-07-0622:46

          Your parent is thinking about the compiler errors, whereas std::io::Error (which I'm guessing you mean) is an Error type your software can get from calling I/O functions at runtime.

          To be fair, if decorating the error with information about a filename is what you needed, since Rust's Error types are just types nothing stops you making your function's Error type be the tuple (std::io::Error, &str) if you have a string reference, or (though this seems terribly inappropriate in production code) leaking a Box to make a static reference from a string whose lifetime isn't sufficient.

        • By estebank 2022-07-0622:40

          The GP comment is referring to the compile time diagnostics and not the runtime diagnostics, which certainly could do with some out-of-the-box improvements but that you can extend in your own code (as opposed to compile errors which crate authors have little control over, at least for now).

        • By rowanG077 2022-07-0622:40

          huh a stack trace should tell you that.

    • By dec0dedab0de 2022-07-0618:394 reply

      This is the first time I ever heard someone complain about Python's error messages in the 10+ years I've been using it. Even people who just learned it, pick up reading tracebacks after about a day of practice. The only problem I ever see is if a library swallows too much, and gives a generic error, but that's not something the language can fix. I really hope they don't change things too much.

      • By joshmaker 2022-07-0619:421 reply

        The new error messages are much better. Take a look at this example:

          Traceback (most recent call last):
    File "calculation.py", line 54, in <module>
      result = (x / y / z) * (a / b / c)
                ~~~~~~^~~
  ZeroDivisionError: division by zero
        In this new version it's now obvious which variable is causing the 'division by zero' error.

        https://docs.python.org/3.11/whatsnew/3.11.html#enhanced-err...

        • By bredren 2022-07-0622:27

          For a lot of folks dealing with getting something working, its way more useful to have the most likely spot where the error is occurring spit out for them like this.

          Stack trace is useful, especially in understanding how the code is working in the system.

          But if the goal is solve the problem with the code you've been working on, existing traces are way too verbose and if anything add noise or distract from getting to productive again.

          I could see tracebacks get swifty terminal UI that shows only the pinpointed error point that can be accordion'd out to show the rest.

      • By dx034 2022-07-076:38

        You've clearly been working with the right people. I constantly get screenshots of error messages (not even the text of the traceback so that I could copy it) with questions about what that means. It takes some training for people to be able to read tracebacks correctly, especially if they have no other programming experience.

      • By e-master 2022-07-0622:322 reply

        I only rarely delve into the python world, but as a .net developer I always found it odd and confusing that the error message comes after the stack trace in python. I don’t see people complaining about it, so maybe it’s just a matter of habit?

        • By kzrdude 2022-07-0623:23

          I guess it's just a habit even if I think it's mildly confusing too.

          But: it's kind of a virtue that the most important error message comes last. Far too many (ahead of time) compilers output too many errors. The most likely error to matter is the first one, but it is often scrolled way off screen; the following errors might even just be side effects that mislead or confuse the new users.

        • By AdamN 2022-07-0713:12

          On the CLI, you'll see the error just above the prompt and then the stack trace above that. It makes sense since the user is always reading up the terminal in reverse order.

      • By actuallyalys 2022-07-0621:071 reply

        I don’t think they’re that bad, but I think the bar has (fortunately) become higher.

        • By urthor 2022-07-073:07

          Exactly right.

          The old Python stack trace, circa 2015, was the best in the world.

          The new ones are even better.

          The gap between Python and Rust, and the JVM languages/C/C++, is increasingly widening.

          Stack trace is one of the areas Python and Rust are making the case for why they're the languages of the future.

    • By kirillbobyrev 2022-07-077:091 reply

      Not saying that there isn't room for improvement but I wouldn't call the Python error messages "cryptic". C++, on the other hand...

      • By gpderetta 2022-07-0710:06

        I program in both C++ and Python every day and I'm fluent in both. I would prefer 10000 lines template instantiation backtraces every time to the average python error message.

        Clang and GCC error messages have come a loooong way in the last 10 years or so and their quality is impressive.

        To be fair, I'm currently stuck at python 3.6 ATM and I hear that python has also improved a lot since.

    • By bilsbie 2022-07-0615:474 reply

      When is that coming? Yes the error messages are confusing in a big list comprehension.

      • By alexfromapex 2022-07-0616:17

      • By teekert 2022-07-0617:153 reply

        Whenever I nest a list comprehension I feel dirty anyway, I wonder if a better error message will help solve something that is just hard to grasp sometimes.

        • By 8organicbits 2022-07-0617:471 reply

          "Debugging is twice as hard as writing the code in the first place. Therfore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Rajanand

          Although, personally, I enjoy python list comprehensions.

        • By mikepurvis 2022-07-0618:14

          Not even with nesting— as soon as it's long enough that I'm breaking it into multiple lines, I'm immediately like, okay this is long enough to just be a normal for-loop, or maybe a generator function. But it doesn't need to be a list comprehension any more.

        • By polski-g 2022-07-0617:273 reply

          All list comprehensions should error with message "FATAL PARSE ERROR: Will you really understand this in six months?"

          • By teekert 2022-07-078:52

            Something like this I find ok (just a filter):

                [x for x in y if x is not z]
            Sometimes though, facilitated by Jupyter's notebooks making executing as you build the code very easy, I create a beast like this:

                [os.path.join([x for x in y if x is not z]) + '/bin' for a in b if 'temp' not in a]
            Yes this is fictional and probably contains an error but you get the point, you can filter lists of lists like this, but it's really unfriendly to anyone trying to understand it later.

          • By wheelerof4te 2022-07-0619:11

            List comprehensions should be used for making quick lists that contain simple calculations or filters.

            Anything more will come back to bite you later.

          • By TremendousJudge 2022-07-0617:42

            If people complain about Rust's borrow checker already...

      • By psalminen 2022-07-0617:491 reply

        Big list comprehensions cause confusion in many ways. KISS is crucial with them.

        • By mcronce 2022-07-0618:081 reply

          I agree. The rule of thumb I follow is that if the list comp doesn't fit on one line, whatever I'm doing is probably too complicated for a list comp.

          • By gpderetta 2022-07-0710:141 reply

            But why? The only reason is that error messages are awful and intermediate values can't be named. There is in principle no reason why list comprehension need to be worse than straight line for loops. The deciding factor to chose betwee the two should be whether I'm writing something for side effects as opposed to for its value.

            • By mcronce 2022-07-096:49

              You just listed two very good reasons: Intermediate values can't be named, and error messages are awful.

      • By agumonkey 2022-07-0617:291 reply

        In a way it's a nice incentive not to nest too much (I have that tendency too)

        • By yantrams 2022-07-0618:043 reply

          • By ChadNauseam 2022-07-0618:383 reply

            Hold mine :D https://github.com/anchpop/genomics_viz/blob/master/genomics...

            That's one expression because it used to be part of a giant comprehension, but I moved it into a function for a bit more readability. I'm considering moving it back just for kicks though.

            My philosophy is: if you're only barely smart enough to code it, you aren't smart enough to debug it. Therefore, you should code at your limit, to force yourself to get smarter while debugging

            • By int_19h 2022-07-0620:103 reply

              Yours is nice and readable. Parents' one is not, but it feels like the indentation is deliberately confusing. I'd lay it out like so:

                 tags = list(set([
     nel
     for subli in [
       mel
       for subl in [
         [[jel.split('/')[2:] for jel in el] 
         for el in classified
       ]
       for mel in subl 
     ]
     for nel in subli
     if nel
   ]))
              Still not very readable, tho. But that's largely due to Python's outputs-first sequence comprehension syntax being a mess that doesn't scale at all.

              Side note: one other thing I always hated about those things in Python is that there's no way to bind an intermediary computation to a variable. In C# LINQ, you can do things like:

                 from x in xs
   let y = x.ComputeSomething()
   where y.IsFoo && y.IsBar
   select y
              In Python, you have to either invoke ComputeSomething twice, or hack "for" to work like "let" by wrapping the bound value in a container:

                 y
   for x in xs
   for y in [x.ComputeSomething()]
   if y.IsFoo and y.IsBar
              It's not just about not repeating yourself or not running the same code twice, either - named variables are themselves a form of self-documenting code, and a sequence pipeline using them even where they aren't strictly needed can be much more readable.

              • By dragonwriter 2022-07-072:20

                > But that's largely due to Python's outputs-first sequence comprehension syntax being a mess that doesn't scale at all.

                Choices like wrapping a set constructor around a list comprehension rather than just using a set comprehension don't help, neither does using multiple nested no-transformation comprehensions laid out that way.

                By hand, on mobile, I think this reduces to (assuming “from itertools import chain” in the file header):

                   tags = list({
     nel
     for subli in chain.from_iterables(chain(
       [[jel.split('/')[2:] for jel in el] 
       for el in classified))
     for nel in subli
     if nel
   })

              • By andrewaylett 2022-07-0714:56

                Since it's all generator expressions internally anyway, an intermediate doesn't actually add appreciable (any?) runtime overhead but does make the code more readable:

                    ys = (x.ComputeSomething() for x in xs)
    result = [y for y in ys if y.isFoo and y.isBar]

              • By texaslonghorn5 2022-07-0620:152 reply

                Does the new walrus := in python solve your problem?

                • By roelschroeven 2022-07-0622:291 reply

                  The official name is assignment expression, which can be good to know to find documentation. Here's the PEP: https://peps.python.org/pep-0572/

                • By int_19h 2022-07-071:252 reply

                  Not really; where would you put it inside the []?

                  • By dragonwriter 2022-07-071:332 reply

                    In the example from the end of the earlier post:

                       y
   for x in xs
   if (y:=x.ComputeSomething()).IsFoo and y.IsBar

                    • By int_19h 2022-07-073:20

                      I suppose that works, although readability is not great, to put it mildly.

                    • By texaslonghorn5 2022-07-073:211 reply

                      Much better would be

                      [(y:=x.ComputeSomething()) for x in xs if y.IsFoo and y.IsBar]

                      • By int_19h 2022-07-077:01

                        That doesn't work, because that first subexpression is evaluated last, after "if".

                  • By ChadNauseam 2022-07-073:37

                    For another example of how to use the walrus operator, I use it multiple times in the code I linked a few comments up

            • By teddyh 2022-07-0711:52

            • By agumonkey 2022-07-0620:02

              That code is fine in my book, there's more static code than for-comp and it's at work 3 level deep with no hard coupling between levels. Also well spaced and worded.

          • By wvh 2022-07-0619:12

            I don't think this is bad code. Maybe not the most Pythonic in the imperative sense, but certainly you could come across such constructs in languages that are more oriented towards functional programming. This could also be solved in a more readable manner by having a generator pipeline, though it would be a good idea to see what the performance of chained generators is like in your version and flavour of Python.

          • By kzrdude 2022-07-0623:26

            It's “not even” using for nesting in list comprehensions. This kind of thing:

               [x for stop in range(5) for x in range(stop)]

  • By woodruffw 2022-07-0615:251 reply

    This is really incredible work.

    The "What's New" page has an item-by-item breakdown of each performance tweak and its measured effect[1]. In particular, PEP 659 and the call/frame optimizations stand out to me as very clever and a strong foundation for future improvements.

    [1]: https://docs.python.org/3.11/whatsnew/3.11.html

HackerNews

About