Python will gradually remove GIL from the programming language over the next few years. This is an important development to keep Python relevant in the area of AI model development. The transition period is longer than five years but is a conscious decision by the Python Steering Council because compatibility issues caused major delays in the last significant change.
The Global Interpreter Lock (GIL) has been ingrained in the Python programming language since 1992. The programming language itself had existed for one year at that time.
Despite GIL’s long existence, the Python community is joyful that GIL will disappear. The feature is obsolete because multi-core CPUs have been available to the general public for a long time. The pressure is now increased as GIL puts a brake on AI development.
From necessity to obstacle
In 1992, however, GIL proved itself necessary for Python. At the time, multi-core CPUs were still a scarce commodity. Multi-threading could therefore not be performed by devices that lacked multi-core CPUs. GIL ensured that only one thread at a time was processed by the Python interpreter.
The advent of multi-core CPUs put pressure on the function. However, the pressure never got high enough to completely remove GIL from Python. After all, many programs did not run on multiple CPUs and worked much faster because of GIL. In addition, the feature simplified the security story in development. Other programming languages did already remove the sharp edges from GIL. Thus a technique of “fine-grained locking” emerged.
In January, GIL again became the Python community’s conversation topic. Sam Gross, a research engineer at Meta, then proposed making GIL optional for developers in CPython with PEP 703. With AI development in mind, GIL presents an obstacle. AI models work faster by addressing several CPU cores simultaneously.
Five-year process
The Python Steering Council already polled developers’ attitudes toward GIL in a poll. The community showed a clear majority in favor of the idea of making Python free-threaded. 87 percent agreed with the idea.
The council now agrees with the proposal from PEP 703. It will only take more than five years to effectively eliminate GIL from Python. In the first step, Python without GIL will be released only as an experiment. Once the community has been able to test the experience, it will be possible to disable GIL in CPython, but it will still be on by default. The council wants to evaluate this situation for at least two years. The effective removal of GIL will then follow after ‘probably 5+ years’.
The interim period in which both a CPython with GIL and without it exists promises to be an unpleasant period for developers. Developers will then have to double-test their code. In other words, the situation will have to worsen before improving.
‘Avoiding new Python 3 situation’
However, the decision to spread the transition period over more than five years was made deliberately. After all, the transition from Python 2 to Python 3 ran into huge delays when it was found that Python 3 was not backwards compatible with Python 2.
“We want to be able to change our minds if it turns out, at any time before we make no-GIL the default, that it will just be too disruptive for too little benefit,” he said. Such a decision could mean reversing all work,” the council wrote. It later added to the blog, “We want to be able to discontinue PEP 703 and want to find another solution if it seems to become problematic.”
Python is the most popular programming language among developers at the moment, according to the TIOBE index. The move to Python without GIL actually comes just in time as more developers begin to focus on AI development. The programming language may well lose its popularity quickly if a no-GIL future is blocked.
Tip: Hugging Face, GitHub, others fight to protect open-source in AI Act