Over nine years ago, on April 27, 2016, an issue was opened on the Ninja GitHub issue tracker requesting support for the GNU Make jobserver. Now, after multiple failed pull requests and a few soft forks, it has finally been added and released.

And I promise you: this is awesome!

What does this solve?

Ninja is a build system like GNU Make: you give it the list of files you want created (the outputs), how it should create them (the rules), and what each one’s dependencies are (the inputs). But you need to ensure it creates files in parallel when possible. It must not re-create files that already exist, but must re-create files when inputs change. Many other tiny annoying details need to be handled as well.

Let’s focus on one specific detail: parallelism. Almost all build systems run in parallel to maximize resource utilization and minimize execution time. Ninja specifically defaults to running one process per CPU thread. This works great most of the time!

Except… what if one of those processes is another instance of Ninja? Now in the worst-case scenario (assuming an 8-core/16-thread CPU), you have the parent instance of Ninja managing 16 processes and another child instance managing 16 more threads! That adds up to 32 processes total on an eight-core CPU. You can see how this can quickly get out of hand and lead to massive resource over-utilization and even system freezes.

There are ways to work around this. For instance, you can disable parallelism on the child Ninja instance(s), but while that will fix over-utilization, it will lead to resource under-utilization and slower execution times. You can also try manually tweaking parallelism levels. However, that will cause inconsistent and inefficient behavior when using multiple build machines.

And this is a real problem affecting real projects. Features like CMake’s ExternalProject often lead to recursive Ninja calls. These recursive calls can easily cause the parallelism problem described above. For reference, see these discussion posts.

But what if I told you this problem has already been solved… back in 1999?

Introducing: The GNU Make Jobserver

GNU Make 3.78 introduced a new feature: the jobserver.

This was created because GNU Make experienced the same problem as Ninja: recursive GNU Make calls could easily cause resource over-utilization and system freezes. Previously, this had been “solved” by disabling parallelism in child GNU Make calls, but as I mentioned earlier, this led to resource under-utilization and longer build times.

The jobserver was a proper solution. The parent GNU Make instance would act as the server. All child instances would be clients. Anytime a client wanted to launch a process, it would ask the server to tell it when it was allowed to. This allowed the server to ensure resources were not over- or under-utilized.

So, if GNU Make had this problem solved for such a long time, why did people experiencing it keep using Ninja? Well, unfortunately many tools like Meson did not support GNU Make. This meant developers still needed Ninja.

Over the years, this has led to a few soft forks of Ninja where the sole change was merging jobserver support (including one from Kitware, the company behind CMake).

Thankfully, as of this blog post, Ninja now supports the exact same tried-and-tested jobserver protocol. That means the problem is completely solved, right? …Right?

What’s the catch(es)?

As you might have guessed, there are multiple catches.

For one, Ninja only implements support for the jobserver client. Without a corresponding server, it cannot actually do anything. This means you either need to run your Ninja instance inside an instance of GNU Make or some other server implementation.

Another issue is that on Linux it only supports the named pipe/FIFO implementation of the jobserver. This was released with GNU Make 4.4 back in October 2022, which was only added to Debian Trixie just a few months ago, in December 2024. This means you will need an extremely recent OS. Otherwise, GNU Make’s jobserver will not be compatible with Ninja.

And finally, as of writing this post, Ninja v1.13.0 (the version containing jobserver support) is less than a week-old. It will take a while before it is included in major package repositories. Until then, you will probably have to compile it from source code or use third-party binaries.

Conclusion

While I might have been nitpicky, this is a major improvement. This will give many Ninja-based projects an immediate performance improvement with minimal required changes. I certainly cannot complain too much about that.

Fun fact about me: I am the weirdo who always updates the UEFI even though it is not technically necessary. It just makes me feel better to know everything is up to date. Particularly because every update’s changelog always lists an impressive collection of security bug fixes.

The problem is that my laptop does not support updating the UEFI from within Linux. Lenovo does not submit updates to LVFS for my laptop. Additionally, they do not release a Linux version of the update tool or a generic UEFI shell version.

This is especially annoying because Lenovo does all of that for other laptops. Unfortunately, my laptop uses InsydeH2O UEFI, and it seems that Lenovo laptops with this firmware do not support updates without Windows.

This leaves me in quite a predicament if I want to update my laptop’s UEFI.

Why not just use Windows?

Now, I am sure you are thinking “just swallow your pride and use Windows.” And that is what I did… the first time.

Sadly, because the InsydeH2O update tool does not support running in Windows PE, I had to create a full-blown Windows installation. This was a lengthy process that involved setting up a Windows VM and then using Rufus (a Windows-only tool) to create a Windows To Go USB.

This worked! I had successfully updated my laptop’s UEFI.

Then, an update broke my Windows To Go USB and I was not willing to go through that whole process again. So, I began to search for a better solution.

Attempt #1: The Ubuntu Wiki

The first resource I found was this page on the Ubuntu Community Help Wiki. My hopes were quickly dashed when I realized the page was last updated in 2021 and the instructions section only contained a placeholder: “This paragraph is yet to be written.” This is, frankly, pretty consistent with the rest of the wiki.

It did give me an idea though: the page revealed that a Linux version of the update tool existed. I thought that if I could find a copy, I could use it on my laptop! I was eventually able to find a copy of this tool (version 200.01.00.10) in the Steam Deck’s software. That is when I learned a devastating truth: a Linux version of the updater exists, but only for the “server/embedded” platform.

You see, there are two versions of the InsydeH2O update tool. There is the “server/embedded” platform (with versions like 200.02.00.08) for devices like the Steam Deck, and the “client” platform (with versions like 6.60) for devices like my laptop. Additionally, the “client” platform only supports updates via Windows or the UEFI shell.

Attempt #2: InterToolx64.efi

After that failure, I continued searching for a solution. Eventually, I found a glimmer of hope!

A Reddit comment claimed that if I extracted the InterToolx64.efi file from the Windows update tool, I could make my own bootable UEFI update USB.

Unfortunately, this did not work. The InterTool.Log file reported an issue with platform.ini, and I could not determine the cause. I found a forum post from someone experiencing a similar issue, but Lenovo support was… less than helpful.

Attempt #3: Scavenging

My last hope was simple: find a manufacturer that offered the UEFI shell version of the “client” platform update tool and use that.

The Framework Laptop also uses InsydeH2O and provides a UEFI shell version of the update tool. So, I extracted H2OFFT-Sx64.efi from their update package and combined it with firmware extracted from my laptop’s update package to create my own UEFI update USB.

Finally, I had managed to update my laptop’s UEFI!

However, I was not out of the woods yet…

Enter: Intel ME

You see, the Intel ME requires its own firmware, which must be updated separately. Once again, Lenovo only provided the Windows update tool.

Thankfully, this time the Arch Wiki provided a solution! Unfortunately, this solution is highly problematic. However, I did not have any better options.

The recommended solution is as follows:

1. Go to this specific internet forum.
2. Find the thread that corresponds to your version of Intel ME (v16.1 for me).
3. Find someone in that thread offering a download link to the Linux version of FWUpdLcl.
4. Download it and run it as root with the firmware extracted from the Windows version of the update tool.

It is well known that downloading binaries from unverified sources (and running them as root) is a significant security risk. Sadly, Intel does not allow downloading the update tool standalone, so I had no other option.

This could potentially lead to severe system compromise, including malware or unauthorized access. I was lucky not to encounter any issues, but I would strongly advise against this approach.

Thankfully, in this case, it worked.

Conclusion

Normally, I understand why companies do not support Linux, even though I may not like it. Because it does cost money to maintain and support a Linux port.

However, in this case, I do not understand it. Especially because they already maintain one! Intel already has a Linux version of the Intel ME update tool! And Insyde has a UEFI shell version of their update tool!

Ultimately, I successfully updated my laptop’s UEFI. However, this required scouring the internet for update tools and downloading them from potentially untrustworthy sources. Nothing prevents Lenovo from adding these tools to their driver download page.

But Lenovo is not solely responsible for this situation. Intel could also allow downloading their update tool standalone! And so could Insyde! Any of these three companies could make this situation better by just adding a download link to their website. An official download, even if it was not supported, would be a vast improvement simply because it would nullify the risk of malware.

There is no technical reason this situation exists. There is no port that has to be written or compatibility issues to fix. This situation exists solely because these companies do not care.

Sometimes, software changes and bug fixes can have unintended consequences. Usually this results in things breaking. However, rarely it can result in things working when you least expect it. This tale is about both of these situations.

Specifically, this tale is about OpenGL ES v1.1 on Debian.

What is OpenGL ES v1.1?

OpenGL ES v1.1 is a modified subset of OpenGL v1.3 for mobile devices. In other words, it sucks. Mainly, because:

• It lacks many desktop OpenGL features.
• It is fixed-function only, no shaders at all!
• It is ancient. (Released in 2003!)

But like many terrible technologies, sometimes you have to use them anyway.

In my case, I maintain a modding project for a game that uses OpenGL ES v1.1. Normally, I use a compatibility layer that ports the game to OpenGL ES v2, but I wanted to make sure the project worked without it. And that is where this tale begins.

December 2023: I encounter an unusual error

It was December 25th, 2023, and I was ready to test my project. I had quickly built a version without the compatibility layer. However, when I started it, I was blindsided by the most peculiar error:

GLX: Failed to create context: BadAlloc (insufficient resources for operation)

This error made no sense! My laptop had 32 GB of RAM and was nowhere near running out!

I tried everything I could think of:

• I tried running a bare-bones OpenGL ES v1.1 demo program¹.
• I tried using EGL instead of GLX.
• I double-checked that libgles1 was installed correctly.
• I tried using EGL with debug logging.

But nothing worked.

Ultimately, I concluded this was probably a bug in Mesa. Especially since it had worked perfectly before I upgraded to Ubuntu 23.10. Oh, how naive I was.

So, I built a copy of upstream Mesa (using the same version as Ubuntu), ran the OpenGL ES v1.1 test program, and… it worked?

Hours of testing later, I eventually understood why Ubuntu 23.10 broke OpenGL ES v1.1. And to explain that, we have to go all the way back to 2017!

February 2017: Debian disables OpenGL ES v1.1 support in Mesa

On February 2nd, 2017, Debian disabled OpenGL ES v1.1 support in the Mesa package. I do not know why it was disabled, because no explanation was given in either the Git commit or the associated change-log². But --disable-gles1³ was added to the Debian package regardless.

This might seem like it explains my situation, except for one crucial detail: OpenGL ES v1.1 had worked perfectly on distributions like Debian Bookworm and Ubuntu 23.04, both of which had included this change.

For some reason, this change had not actually disabled OpenGL ES v1.1!

My question had changed from “why is OpenGL ES v1.1 not working” to “why was OpenGL ES v1.1 working in the first place?”

The answer to that comes only one year later.

August 2018: Enter GLVND

Fun (and relevant) fact: if you use Debian, many OpenGL-related shared libraries (like libGL.so.1, libOpenGL.so.0, libGLESv2.so.2, libEGL.so.1, and many more) are not actually provided by your graphics driver. Instead they are provided by a project called GLVND. GLVND’s job is to provide a driver-agnostic OpenGL interface, that then forwards any API calls to your actual driver. This is beneficial for reasons beyond the scope of this blog post.

Anyways, on August 8th, 2018, OpenGL ES v1.1 support was enabled in Debian’s GLVND package⁴. Reading the linked Launchpad bug is actually quite interesting because:

• It seems the maintainers believed that Mesa itself no longer supported OpenGL ES v1.1, which is patently false.
• And it reveals that OpenGL ES v1.1 support was enabled in GLVND anyways to support non-Mesa drivers (like NVIDIA’s⁵).

Debian’s use of GLVND interacted with a quirk of Mesa’s build-system in a really interesting way: you see, --disable-gles1 did not actually disable OpenGL ES v1.1. Instead it only disabled building the libGLESv1_CM.so.1 library, Mesa itself still included support for OpenGL ES v1.1. But Debian systems did not need Mesa’s libGLESv1_CM.so.1 because GLVND provided one!

So GLVND’s libGLESv1_CM.so.1 forwarded API calls directly to Mesa, completely bypassing the effects of --disable-gles1! Therefore, OpenGL ES v1.1 worked. Completely. By. Accident.

But it did not just work. It worked well. In fact, there was no sign that it was not supported!. All you had to do was install libgles1 and it just worked.

Now that I knew why OpenGL ES v1.1 used to work, I could finally answer my first question: why did it stop working?

May 2023: Mesa makes an objectively good change

Remember when I said “--disable-gles1 did not actually disable OpenGL ES v1.1?” Well, that was past-tense for a reason.

On May 19th, 2023, Mesa 23.1.0 was released. And among its many changes, it made --disable-gles1 actually remove the associated code. This version of Mesa was eventually included in Ubuntu 23.10 and Debian Trixie.

This is what broke OpenGL ES v1.1. Now that Mesa properly disables OpenGL ES v1.1, the GLVND workaround is broken.

Conclusion

This situation annoyed me for multiple reasons. It annoyed me that OpenGL ES v1.1, which appeared to be fully supported, was actually only working by chance and had been unsupported for years. It annoyed me that Debian dropped support for OpenGL ES v1.1 seemingly without an explanation. And it really annoyed me that almost none of this information was written down anywhere accessible!

But ultimately, the damage has been done.

There is no reason for Mesa to revert what truly is a good change. Making disabling OpenGL ES v1.1 actually remove the associated code is the sensible thing to do. So the GLVND workaround is well and truly dead.

And considering that almost nobody uses OpenGL ES v1.1, I highly doubt Debian will ever re-enable it.

So, here we are. If you want to use OpenGL ES v1.1 on Debian, get ready to compile Mesa from source. It sucks, but at least it’s documented now.

(If you want to skip to the tutorial, click here.)

If your household uses Xfinity for cable, you have almost definitely heard of Xfinity Stream at some point. If you haven’t, Xfinity Stream is a service that allows you to access Xfinity’s on-demand TV over the internet. This can be useful if you want to watch something on Xfinity’s on-demand TV but do not want to go to your actual TV.

This is nice because some media still goes to on-demand before its respective streaming service (if it even has one). And of course, if you are already paying for Xfinity, it is nice not to have to pay for another streaming service if you can avoid it.

However, it has some limitations as well, mainly that you can only access it on your local Xfinity network… and its completely arbitrary Linux block. That’s right, if you have ever tried to load Xfinity Stream on Linux, you will have seen this lovely error message:

And if you look up that error message, you will find a common theme: Xfinity Stream does not support Linux.

Also of note is that Xfinity Stream will refuse to load at all on Google Chrome without changing your user-agent, but it will load on Firefox, just for it to fail when you try to watch anything.

How does it block Linux?

One of the first methods I tried to use to bypass this block was changing my user-agent… just for the block to persist anyways. Which raised the question “how does Xfinity know I am running Linux?” Because if I can trick it into thinking I am running Windows or MacOS, then it should not be able to block me anymore.

So I dug into Firefox’s network logs, which led me to the source of the error:

This POST request was generated using the MediaKeySession.generateRequest() API, which isn turn used the browser’s built-in Widevine CDM to generate a DRM license request. At this point, I had formed a hypothesis: the Linux build of the Widevine CDM was embedding metadata into this request which Xfinity was using to block Linux.

How do I bypass that?

Unfortunately, just patching the Widevine CDM’s binary is beyond my skill set, especially since it is ridiculously obfuscated.

However, I eventually came to a realization: Xfinity Stream probably does not block Chrome OS, and Chrome OS is basically GNU/Linux internally. So I formed a plan: extract Chrome OS’s Widevine CDM from a recovery image, load it into Firefox, and try to use Xfinity Stream.

It did not work, at all.

The Widevine CDM immediately crashed, and when I tried to load it into a standalone program using dlopen(), it caused a segmentation fault before it even finished loading. This incredibly vague error stumped me for a while until I found this incredibly helpful GitHub issue from a project that was trying to do something similar.

This GitHub issue revealed the problem: Chrome OS was not exactly GNU/Linux. Namely, it was complied with a feature that was not included in my version of glibc that Google had patched into their version: the DT_RELR relocation format.

With some more research, I learned that glibc had actually merged support for this feature in version 2.36! So I loaded Ubuntu 22.10 on a USB flash drive and tried loading Chrome OS’s Widevine CDM into my testing program again.

And it still did not work.

This time the error was slightly more helpful though: DT_RELR without GLIBC_ABI_DT_RELR dependency.

What in the world is a GLIBC_ABI_DT_RELR dependency?

It turns out, the glibc maintainers actually agreed with my earlier assessment: immediately causing a segmentation fault on older versions of glibc just because a program used the DT_RELR relocation format was not good behavior.

They decided that for every binary compiled with the DT_RELR relocation format, the linker would include a version dependency called GLIBC_ABI_DT_RELR on libc.so.6. This would cause any program using the DT_RELR relocation format to give a more sane error on older versions of glibc: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_ABI_DT_RELR' not found (required by ./program).

However, the glibc maintainers also decided to implement this functionality in reverse as well: glibc refuses to load any binary containing the DT_RELR relocation format that does not also contain the GLIBC_ABI_DT_RELR version dependency. I do not know why they did this, but this is what caused the previous error. This is because Chrome OS’s toolchain evidently predates GLIBC_ABI_DT_RELR so it was never included when linking the Widevine CDM.

Originally, I tried to patch the Widevine CDM binary to include the GLIBC_ABI_DT_RELR version dependency using LIEF. However, the resulting binary was invalid and caused a segmentation fault when started. Ultimately, I decided to instead remove the check itself from 'glibc manually.

The GLIBC_ABI_DT_RELR check was quite trivial to patch out of glibc (although waiting for glibc to compile was certainly tedious). And once done, I was able to load up Firefox with Chrome OS’s Widevine CDM… and Xfinity Stream finally worked on Linux!

How do I actually do this?

1. Install a Linux distribution that contains glibc v2.36 or higher. I would also recommend taking a backup or using a more disposable installation medium such as a USB flash drive or a VM.
   • You could use an older version of glibc if you really want to, but you would have to patch in support for the DT_RELR relocation format yourself.
2. Use this script to patch glibc (this script is Debian-specific, but it should not be too hard to adapt it to other distributions):

   #!/bin/sh
   
   set -e
   
   # Create Working Directory
   rm -rf glibc-work
   mkdir glibc-work
   cd glibc-work
   
   # Download Sources
   apt-get source glibc
   cd glibc-*
   
   # Patch
   patch -p1 << EOF
   --- glibc-master-old/elf/dl-version.c
   +++ glibc-master/elf/dl-version.c
   @@ -362,7 +362,7 @@
      /* When there is a DT_VERNEED entry with libc.so on DT_NEEDED, issue
         an error if there is a DT_RELR entry without GLIBC_ABI_DT_RELR
         dependency.  */
   -  if (dyn != NULL
   +  if (0 && dyn != NULL
          && map->l_info[DT_NEEDED] != NULL
          && map->l_info[DT_RELR] != NULL
          && __glibc_unlikely (!map->l_dt_relr_ref))
   EOF
   
   # Commit Patch
   EDITOR='/bin/true' dpkg-source -q --commit . disable-GLIBC_ABI_DT_RELR-check.patch
   
   # Update Package Version
   EMAIL='example@example.com' dch -n 'Disable GLIBC_ABI_DT_RELR Check'
   
   # Disable Testing
   export DEB_BUILD_OPTIONS=nocheck
   
   # Build
   dpkg-buildpackage -us -uc
   
   # Install
   sudo apt-get install -y ../*.deb
   

3. Download an x86_64 Chrome OS recovery image. I used the stable version of atlas, which was Chrome OS v107 at the time of writing.
4. Extract /opt/google/chrome/WidevineCdm/_platform_specific/cros_x64/libwidevinecdm.so from the image.
5. Browser-specific steps:
   • Google Chrome
     1. Copy libwidevinecdm.so to /opt/google/chrome/WidevineCdm/_platform_specific/linux_x64/libwidevinecdm.so.
     2. You will need to use the Chrome Developer Tools user-agent switcher when using Xfinity Stream. Other user-agent switcher extensions do not appear to work.
   • Firefox
     1. Install a non-Snap/Flatpak version of Firefox. This is needed because we need Firefox (and therefore the Widevine CDM) to use the host’s glibc.
     2. If you are using the Firefox build from ppa:mozillateam/ppa, you will also need to modify Firefox’s AppArmor configuration so that the Widevine CDM does not crash.
6. Use Xfinity Stream!
7. You will need to re-patch glibc every time your distribution releases an updated version.

Conclusion

All in all, this whole situation is ridiculous. It should not be this much harder to access media legally than it is to pirate. It is insane how much effort we as a species waste on this DRM that does not stop anybody. If I wanted to pirate anything, I could probably pull up a high-resolution copy in under an hour with minimal effort. But instead, because I tried to access this media the legal way, I have to either go through this whole complicated process or set up an entirely different operating system solely for that purpose. That is so unbelievably silly. The only reason I can remotely justify the time spent on this is because it was more enjoyable than watching whatever it was I was originally planning to watch!

In all honesty, if you want to use Xfinity Stream, I would recommend just setting up a Windows VM. It will probably be much more resistant to whatever nonsense Xfinity or Google try to pull in the future. But if you would rather not watch it at all than set up a Windows VM like me, well here is your method.