It's not necessary, it was a design choice that made sense back in the 90ies. In a multithreaded environment you can lock at a fine grained level or on a coarse grained level - or you can crash, but let's ignore that as an option. Python chose coarse grained, giving up parallel interpreter computations, but gaining a lot of thread sync overhead. All attempts so far to remove the GIL have resulted in a (usually much) slower interpreter, but the latest attempt shows some promise and it's thinkable (but not guaranteed) that in a few years there will be an official GIL-less cPython.
Removing Python's GIL will never make much sense. Not today and not in future. If you need CPU-fast code and would bother to multi-thread, it's much more worth it to write the code in Cython.
If your code is CPU bound and you're using native Python, you're going to be making a tonne of heap allocations and pointer dereferences. This will be very slow.
If you implement the relevant stuff in Cython, even without using multi-threading you'll likely see 10x performance improvement, and can often see up to 100x.
Removing the GIL makes Python worse at the stuff it's good at, for questionable improvements in the areas Python is really terrible. This is not a good trade.
Do you mean waiting for I/O? It's already possible to do I/O asynchronously or with separate locks. The interpreter lock only applies to the interpreter.
http://pyparallel.org is one of the more interesting experiments currently going on in the GIL area . They're basically working on removing all the practical limitations of the GIL without actually removing the entire GIL.
Please remember that threads are not the only way. If you can simply break your function/routine into a smaller piece that is independent, you can easy get by with a fork. (well, unless you are on windows..)
Been a since I tried to use the multiprocessing module. But, last time I did try, I ran into issues with it interacting poorly with pyodbc. It's been years, so I don't recall what the problem was, but I spent a few days trying to resolve or work around the issue with no satisfaction.
Also, most of my Python scripts run on both Linux and Windows, so I have that restriction, as well.
I'm writing a transpiler that uses global information from codebases, and so it transpiles potentially hundreds of files at once and creates rather complex data structures. Compute bound for quite a while, so I tried speeding it up with multiprocessing (since multithreading would be useless). But with multiprocessing it took longer to serialize/deserialize the complex datastructures for each process, so I had to give up. Next time I have time for this I'd probably try to use Jython as a drop-in replacement and see whether I can get it to run with GIL-less multithreading.
It sounds like you have a couple of hot paths and are not optimizing them. I can't tell for sure without seeing any code but nothing in your post screams out "this will be slow" or "I need parallism/concurrency". Perhaps it's the data structures you are using?
I already did extensive profiling and performance improvements, at this point I'm quite sure that if I could do multithreading on my lab's 24 core Xeon Haswell machines I'd be getting a nice speedup.
Isn't that up to the RDBMS whether than's multithreaded or not? Unless the RDBMS is implemented in Python, CPython doesn't force extension code to be single-threaded. Just Python bytecode.
That's pretty much the point, isn't it? If I need true multithreading, then I am forced to write an extension in C or offload the multithreading work to another process (such as RDBMS in your case). It would have been nice if true multithreading was possible in Python itself. It would immediately make Python more useful in a variety of scenarios where splitting the work into multiple processes is not optimal or more convoluted.
Sure, I guess I'm just used to writing my performance bottlenecks in a lower level language already, so I'm used to the GIL not actually being held most of the time in any intensive computation.
So if I want to call two Fourier transform functions at the same time in Python I can, because neither of them is implemented in Python and so they don't hold the GIL.
That's the kind of parallelism use case I most often see come up, so although the GIL dismayed me early on I've come to see it as pretty irrelevant.
But maybe it makes more sense for other applications, for the performance critical parts of the code to be actual pure Python. I do mostly numerical simulations, so pure Python is usually a non-starter, you fix that long before you think about parallelism.
It's not that I write the whole program in another language, it's that I either write the bottleneck in another language (usually Cython), or it turns out that the Python package I'm calling already has its bottlenecks written in another language, whether I wrote it or not.
Day to day, I'm writing Python code which is actually parallel because a large fraction of the run time is dominated by the by things that aren't pure Python. I suspect this is true even for people who are not going out of their way to make it true. It's simply the case that most RDBM systems, Fourier transforms, etc with Python bindings are not written in Python.
The GIL sounds scary, but I think people overestimate the fraction of time it is actually held in their code.
