It turned out that the problem is caused by two hard coded bugs that limit the amount available to Firefox to only 4MB. Version 126.96.36.199 should fix this and more. Furthermore, most of the fixes are flagged as critical which underline the amount of work that was needed.
Amongst the other welcomed improvements are native support for Intel Core processors and Apple mac as well as improvements to product stability as well as speed. Interestingly though, Firefox plug-ins, which are at least as popular as Firefox itself and provide more functionality to Firefox, come with their own set of problems.
As you can see there, the amount of plugins that have memory leaks is truly awful and the very idea of having to upgrade all your plugins to erase those leaks is more than enough to dissuade me from using Firefox for the time being.