devlog: let's set up mumble in 2024

posted on 6.10.2024

Recently I heard a rumor that discord is going to get banned in russia by the russian government (for spreading iconoclasm and shit). I also heard a rumor that it’s not going to get banned, since it’s being used by russian military in the field. Which is kind of crazy! It’s a foreign closed-protocol closed-source tool with public by default access; how come the other militaries are not already listening to all the ongoing communications? Anyway, if this is true, then rather discord will ban russian accounts. The end result in both cases is the same still.

I still have some friends in russia, and sometimes I use discord to play games with them. As an opponent of closed platforms, I decided this is my chance to act preemptively: I shall create and advertise the alternative before it’s even required, so that the switch to it is the first thing on their mind.

For the alternatives, I had my mind on mumble immediately: it’s something I used with my geek friends to play minecraft and arma in my youth, back when people unironically used skype to call each other. No idea why we stopped using it (mumble I mean, skype just got worse and worse each year). But for the sake of keeping my mind open, I took a glance at some other alternatives:

  1. Matrix. I remembered matrix having some call features, but it’s kind of hard to find info on them, and from what I gather, they are not core protocol, but just element-the-client extensions. In my humble opinion element is a crap client and I won’t force my friends to use it.

  2. revolt.chat - looks just like discord, but opensource

  3. jami at https://jami.net

Well it was only a glance and I didn’t see anything mind-changing in either. There’s also very little info on revolt and jami protocols and federation; maybe they don’t have any? Aaanyway, mumble it is.

the easy part

Mumble is a well-written program with minimal dependencies and a sane config file. What’s better is that it has a nixos module!

    services.murmur = {
      enable = true;
      openFirewall = true;
      welcometext = "мур мур мяу";
      registerHostname = "no-entry.morj.men";
    };

What’s worse is that https://wiki.mumble.info has died, but a lot of links online still lead to it. It has been replaced with https://mumble.info/documentation, but not all articles have yet been migrated. For example, the one on certificates.

the complicated part

Mumble self-signs a server certificate on first launch. This is used to authenticate the server, and it has to be accepted on first connection. But, it still uses RSA (boo), and it doesn’t respect my registerHostname: when people connect to the address, the client complains that the certificate doesn’t match the domain name used. After looking through the web-archive of the wiki (please donate to webarchive!), I found that you can use the same ssl certificates that nginx uses, and that using the exact certificate that nginx uses is a supported use-case. That is, I can have some website on https://no-entry.morj.men with tls being terminated by nginx, and I can have a mumble server under the same hostname and it would use the exact same certificate and private key. So this is exactly what I did:

    services.nginx.virtualHosts."no-entry.morj.men" = {
      enableACME = true;
    };
    services.murmur = {
      extraConfig = ''
        sslCert=/var/lib/acme/no-entry.morj.men/fullchain.pem
        sslKey=/var/lib/acme/no-entry.morj.men/key.pem
      '';
    };
    # enable murmur to read nginx acme certificates
    users.users.murmur.extraGroups = [ "nginx" ];

This way, nixos will automatically issue and renew the certificate for this domain, and I can simply use it.

But rather, I think this is a kind of dirty approach. Nixos has a better way to manage certificates, and by better I mean more logical and the one nginx module uses under the hood. But I couldn’t find a simple way to make nginx use the certificates produced in this way other than a manual extraConfig. And I would rather add extra config to murmur than to the monster that is nginx.

the impossible part

Why do I want to have the same domain for mumble and for a website? Because we’re living in 2024 man, everything now is done on a web app. I think it would be a more seamless process to first ask people to open a website to talk to me, and only push them to the client later (even if mumble installation is incredibly simple). So, I remembered that back in 2019 I tried a mumble web app - and would you look at that, it hasn’t been updated since 2020! And as we all know, the web world has a half-life of a quarter of a millisecond, and I was not looking forward to trying to build this.

Small side note: while googling for the https://github.com/Johni0702/mumble-web, I found several other mumble web projects, some under the same name, some a fork of this one but trying to hide it. Neither built as well.

    > nix-shell -p nodejs
    > npm install
    npm warn old lockfile
    npm warn old lockfile The package-lock.json file was created with an old version of npm,
    npm warn old lockfile so supplemental metadata must be fetched from the registry.
    npm warn old lockfile
    npm warn old lockfile This is a one-time fix-up, please be patient...
    npm warn old lockfile

Well that’s already a bad sign.
Then I get a hundred deprecation warnings for library versions. Yawn, tell me something I don’t know.
Then I get an error.
What I don’t get is a fixed-up lockfile: it just remains unchanged. Does it try to succeed in installing dependencies before updating the lockfile? But why? Can’t it simply succeed in resolving them? Aren’t all the versions needed already written down?

Anyway, the error. It’s a big one, an envy of C++ compilers.

Oh boy 
npm error code 1
npm error path /home/morj/projects/temp/american-mumble/node_modules/node-sass
npm error command failed
npm error command sh -c node scripts/build.js
npm error Building: /nix/store/w78sh036dyn14f29w8za9ni9syrmwm3q-nodejs-20.17.0/bin/node /home/morj/projects/temp/american-mumble/node_modules/node-gyp/bin/node-gyp.js rebuild --verbose --libsass_ext= --libsass_cflags= --libsass_ldflags= --libsass_library=
npm error gyp info it worked if it ends with ok
npm error gyp verb cli [
npm error gyp verb cli   '/nix/store/w78sh036dyn14f29w8za9ni9syrmwm3q-nodejs-20.17.0/bin/node',
npm error gyp verb cli   '/home/morj/projects/temp/american-mumble/node_modules/node-gyp/bin/node-gyp.js',
npm error gyp verb cli   'rebuild',
npm error gyp verb cli   '--verbose',
npm error gyp verb cli   '--libsass_ext=',
npm error gyp verb cli   '--libsass_cflags=',
npm error gyp verb cli   '--libsass_ldflags=',
npm error gyp verb cli   '--libsass_library='
npm error gyp verb cli ]
npm error gyp info using [email protected]
npm error gyp info using [email protected] | linux | x64
npm error gyp verb command rebuild []
npm error gyp verb command clean []
npm error gyp verb clean removing "build" directory
npm error gyp verb command configure []
npm error gyp verb check python checking for Python executable "python2" in the PATH
npm error gyp verb `which` failed Error: not found: python2
npm error gyp verb `which` failed     at getNotFoundError (/home/morj/projects/temp/american-mumble/node_modules/which/which.js:13:12)
npm error gyp verb `which` failed     at F (/home/morj/projects/temp/american-mumble/node_modules/which/which.js:68:19)
npm error gyp verb `which` failed     at E (/home/morj/projects/temp/american-mumble/node_modules/which/which.js:80:29)
npm error gyp verb `which` failed     at /home/morj/projects/temp/american-mumble/node_modules/which/which.js:89:16
npm error gyp verb `which` failed     at /home/morj/projects/temp/american-mumble/node_modules/isexe/index.js:42:5
npm error gyp verb `which` failed     at /home/morj/projects/temp/american-mumble/node_modules/isexe/mode.js:8:5
npm error gyp verb `which` failed     at FSReqCallback.oncomplete (node:fs:197:21)
npm error gyp verb `which` failed  python2 Error: not found: python2
npm error gyp verb `which` failed     at getNotFoundError (/home/morj/projects/temp/american-mumble/node_modules/which/which.js:13:12)
npm error gyp verb `which` failed     at F (/home/morj/projects/temp/american-mumble/node_modules/which/which.js:68:19)
npm error gyp verb `which` failed     at E (/home/morj/projects/temp/american-mumble/node_modules/which/which.js:80:29)
npm error gyp verb `which` failed     at /home/morj/projects/temp/american-mumble/node_modules/which/which.js:89:16
npm error gyp verb `which` failed     at /home/morj/projects/temp/american-mumble/node_modules/isexe/index.js:42:5
npm error gyp verb `which` failed     at /home/morj/projects/temp/american-mumble/node_modules/isexe/mode.js:8:5
npm error gyp verb `which` failed     at FSReqCallback.oncomplete (node:fs:197:21) {
npm error gyp verb `which` failed   code: 'ENOENT'
npm error gyp verb `which` failed }
npm error gyp verb check python checking for Python executable "python" in the PATH
npm error gyp verb `which` failed Error: not found: python
npm error gyp verb `which` failed     at getNotFoundError (/home/morj/projects/temp/american-mumble/node_modules/which/which.js:13:12)
npm error gyp verb `which` failed     at F (/home/morj/projects/temp/american-mumble/node_modules/which/which.js:68:19)
npm error gyp verb `which` failed     at E (/home/morj/projects/temp/american-mumble/node_modules/which/which.js:80:29)
npm error gyp verb `which` failed     at /home/morj/projects/temp/american-mumble/node_modules/which/which.js:89:16
npm error gyp verb `which` failed     at /home/morj/projects/temp/american-mumble/node_modules/isexe/index.js:42:5
npm error gyp verb `which` failed     at /home/morj/projects/temp/american-mumble/node_modules/isexe/mode.js:8:5
npm error gyp verb `which` failed     at FSReqCallback.oncomplete (node:fs:197:21)
npm error gyp verb `which` failed  python Error: not found: python
npm error gyp verb `which` failed     at getNotFoundError (/home/morj/projects/temp/american-mumble/node_modules/which/which.js:13:12)
npm error gyp verb `which` failed     at F (/home/morj/projects/temp/american-mumble/node_modules/which/which.js:68:19)
npm error gyp verb `which` failed     at E (/home/morj/projects/temp/american-mumble/node_modules/which/which.js:80:29)
npm error gyp verb `which` failed     at /home/morj/projects/temp/american-mumble/node_modules/which/which.js:89:16
npm error gyp verb `which` failed     at /home/morj/projects/temp/american-mumble/node_modules/isexe/index.js:42:5
npm error gyp verb `which` failed     at /home/morj/projects/temp/american-mumble/node_modules/isexe/mode.js:8:5
npm error gyp verb `which` failed     at FSReqCallback.oncomplete (node:fs:197:21) {
npm error gyp verb `which` failed   code: 'ENOENT'
npm error gyp verb `which` failed }
npm error gyp ERR! configure error 
npm error gyp ERR! stack Error: Can't find Python executable "python", you can set the PYTHON env variable.
npm error gyp ERR! stack     at PythonFinder.failNoPython (/home/morj/projects/temp/american-mumble/node_modules/node-gyp/lib/configure.js:484:19)
npm error gyp ERR! stack     at PythonFinder.<anonymous> (/home/morj/projects/temp/american-mumble/node_modules/node-gyp/lib/configure.js:406:16)
npm error gyp ERR! stack     at F (/home/morj/projects/temp/american-mumble/node_modules/which/which.js:68:16)
npm error gyp ERR! stack     at E (/home/morj/projects/temp/american-mumble/node_modules/which/which.js:80:29)
npm error gyp ERR! stack     at /home/morj/projects/temp/american-mumble/node_modules/which/which.js:89:16
npm error gyp ERR! stack     at /home/morj/projects/temp/american-mumble/node_modules/isexe/index.js:42:5
npm error gyp ERR! stack     at /home/morj/projects/temp/american-mumble/node_modules/isexe/mode.js:8:5
npm error gyp ERR! stack     at FSReqCallback.oncomplete (node:fs:197:21)
npm error gyp ERR! System Linux 6.11.0-1-default
npm error gyp ERR! command "/nix/store/w78sh036dyn14f29w8za9ni9syrmwm3q-nodejs-20.17.0/bin/node" "/home/morj/projects/temp/american-mumble/node_modules/node-gyp/bin/node-gyp.js" "rebuild" "--verbose" "--libsass_ext=" "--libsass_cflags=" "--libsass_ldflags=" "--libsass_library="
npm error gyp ERR! cwd /home/morj/projects/temp/american-mumble/node_modules/node-sass
npm error gyp ERR! node -v v20.17.0
npm error gyp ERR! node-gyp -v v3.8.0
npm error gyp ERR! not ok 
npm error Build failed with error code: 1

Holy hell, python2? Wasn’t it already deprecated in 2020, when this thing last updated? And why does node tooling require python anyway? In my opinion they are about equally painful to write (python gets ahead because of sane defaults, but node clenches the victory because of typescript).

This made me remember how I got my first friend hooked on nix: apple just removed python2 from the system, and I showed them how they could install python on their macbook with nix. So let’s remember how it was for them, even if I’m not on mac.

> nix-shell -p python27
       error: Package ‘python-2.7.18.8’ in /nix/store/5l4zx8mprka9k4n2wpnaaspvf00qj764-nixpkgs/nixpkgs/pkgs/development/interpreters/python/cpython/2.7/default.nix:341
       is marked as insecure, refusing to evaluate.
> # Of course.
> NIXPKGS_ALLOW_INSECURE=1 nix-shell -p python27

What follows is it building python from sources. I like the reproducibility of nix: I can just build it myself and it will be the same as on another machine. And I completely support python2 not being in binary cache: python2 users must suffer.

Armed with the most deprecated of deprecated tools, I start the build again:

Round 2

Haha shit, when trying to build it again today while writing this log, I get a different error! Thankfully, I had my webdev friend help me debug it yesterday and I sent them all the logs.

npm ERR! Traceback (most recent call last):
npm ERR!   File "/home/morj/projects/temp/american-mumble/node_modules/node-gyp/gyp/gyp_main.py", line 16, in <module>
npm ERR!     sys.exit(gyp.script_main())
npm ERR!   File "/home/morj/projects/temp/american-mumble/node_modules/node-gyp/gyp/pylib/gyp/__init__.py", line 545, in script_main
npm ERR!     return main(sys.argv[1:])
npm ERR!   File "/home/morj/projects/temp/american-mumble/node_modules/node-gyp/gyp/pylib/gyp/__init__.py", line 538, in main
npm ERR!     return gyp_main(args)
npm ERR!   File "/home/morj/projects/temp/american-mumble/node_modules/node-gyp/gyp/pylib/gyp/__init__.py", line 514, in gyp_main
npm ERR!     options.duplicate_basename_check)
npm ERR!   File "/home/morj/projects/temp/american-mumble/node_modules/node-gyp/gyp/pylib/gyp/__init__.py", line 130, in Load
npm ERR!     params['parallel'], params['root_targets'])
npm ERR!   File "/home/morj/projects/temp/american-mumble/node_modules/node-gyp/gyp/pylib/gyp/input.py", line 2783, in Load
npm ERR!     variables, includes, depth, check, True)
npm ERR!   File "/home/morj/projects/temp/american-mumble/node_modules/node-gyp/gyp/pylib/gyp/input.py", line 399, in LoadTargetBuildFile
npm ERR!     includes, True, check)
npm ERR!   File "/home/morj/projects/temp/american-mumble/node_modules/node-gyp/gyp/pylib/gyp/input.py", line 271, in LoadOneBuildFile
npm ERR!     aux_data, includes, check)
npm ERR!   File "/home/morj/projects/temp/american-mumble/node_modules/node-gyp/gyp/pylib/gyp/input.py", line 308, in LoadBuildFileIncludesIntoDict
npm ERR!     LoadOneBuildFile(include, data, aux_data, None, False, check),
npm ERR!   File "/home/morj/projects/temp/american-mumble/node_modules/node-gyp/gyp/pylib/gyp/input.py", line 251, in LoadOneBuildFile
npm ERR!     None)
npm ERR!   File "/home/morj/.node-gyp/20.10.0/include/node/common.gypi", line 1
npm ERR!     nerate ',
npm ERR!             ^
npm ERR! SyntaxError: EOL while scanning string literal
npm ERR! gyp ERR! configure error 
npm ERR! gyp ERR! stack Error: `gyp` failed with exit code: 1
npm ERR! gyp ERR! stack     at ChildProcess.onCpExit (/home/morj/projects/temp/american-mumble/node_modules/node-gyp/lib/configure.js:345:16)
npm ERR! gyp ERR! stack     at ChildProcess.emit (node:events:514:28)
npm ERR! gyp ERR! stack     at ChildProcess._handle.onexit (node:internal/child_process:294:12)
npm ERR! gyp ERR! System Linux 6.10.9-1-default
npm ERR! gyp ERR! command "/nix/store/yc5bicc5zs5czjp5q5aygz79km84rmd4-nodejs-20.10.0/bin/node" "/home/morj/projects/temp/american-mumble/node_modules/node-gyp/bin/node-gyp.js" "rebuild" "--verbose" "--libsass_ext=" "--libsass_cflags=" "--libsass_ldflags=" "--libsass_library="
npm ERR! gyp ERR! cwd /home/morj/projects/temp/american-mumble/node_modules/node-sass
npm ERR! gyp ERR! node -v v20.10.0
npm ERR! gyp ERR! node-gyp -v v3.8.0
npm ERR! gyp ERR! not ok 
npm ERR! Build failed with error code: 1

npm ERR! A complete log of this run can be found in: /home/morj/.npm/_logs/2024-09-27T19_56_00_704Z-debug-0.log

Which is kind of crazy. It autogenerated a python file, tries to import it, and this file contains invalid python syntax. It looks like it was somehow just cut from the beginning.

Somewhere between rounds 1 and 2 I complained to my webdev friend, and after just one look they told me I won’t finish it with the nowadays node, which is version 20 for me. They also told me that gyp and sass are always like this and everyone hates them. (Which didn’t change my mind in any direction because I already hate all javascript tooling). What followed is us trying to get old node from an old nixpkgs (which I succeeded in), and trying to build with it and some other downgrades. No luck as well. The errors were getting weirder and weirder. Somewhere in the process of googling for one of them, I found the light: https://git.termer.net/termer/mumble-web-prebuilt - someone managed to build it on a very old debian with some very old tools. With this, I gave up on node: I’m not going to change the mumble-web code anyway (although I wanted to a little), so I’ll just use the prebuilt files.

Side note: remember how docker was supposed to be used for reproducibility? The mumble-web project even has a dockerfile with build instructions. It starts with a punchline:

FROM alpine:edge

Everybody laugh.

What did I learn? If I have to write a complicated web application, I’m not going to use any node-adjacent tooling, even my favourite rescript. Maybe I’ll use purescript-without-node if it’s still alive, maybe I’ll use typescript with type annotations in comments only and everything (EVERYTHING) in one file so there isn’t a build step at all. Node infrastructure can’t be trusted.

the annoying part

Mumble protocol works over UDP+TCP instead of HTTP or grpc or something. If it were written now, I would call it based, but it was written in a better era when this was just normal. The downside is that web pages can’t use them. I myself is still not sure if it’s a good thing for webpages or not. But as a consequence, mumble-web requires a server that it connects to via websocket that will translate it into the mumble protocol. The server is written in rust, and rust has a better build infrastructure, I hoped. Well.

I start with several rounds of running cargo build followed by sudo zypper install somelib-devel. Until I install openssl-devel and get:

'libopenssl-devel' providing 'openssl-devel' is already installed.
Nothing to do.

Hah, shit. This means that someone broke backwards compatibility. So starts our first patching. But because I have faith in rust core libs, I’m going to patch the dependencies to use a newer openssl-sys and openssl. And honestly, it was a simple process: cargo tree | less, find who needs openssl, clone them and patch the Cargo.toml. Thankfully noone changed the cargo format and was telling me to “sit for a while while it’s getting migrated”. Thankfully it just built after that.

Then I had to do it several more times, basically for all the libs written or patched by the author of mumble-web-proxy themself. And again it just builds after upgrading the deps, even if with warnings.

Rust is pretty good. Not ideal, but good. As usual its biggest downside is the C shared libraries. I hope one day static linking and vendoring your C deps into sources will become the norm. Meanwhile, I wrote a nix derivation with a flake lockfile that will download and build them all for you.

the stupid part

Let me complain really quick that in nixos the build-run-debug-rebuild loop is way too long. Just running nix-build already makes me want to kill myself, and rebuilding the whole system is just like putting on the noose while pointing a gun at your head.

Aaaanyway, now I have everything built, even if I haven’t built everything, and I try it out locally: a websocket-proxy running on localhost, and an index.html file I open in my web-browser. If you’ve worked with websockets, you might already realise where this is going. And yes, the page uses wss protocol unconditionally. As soon as it failed to connect, I was 80% sure that that was the reason. The other 20% was me seeing the error messages in both services. Here’s the one in the web app:

What terrors await us here 
[1:32:48 PM] Connecting to server localhost
[1:32:48 PM] Connection error: [object Event]

My favourite. This obviously doesn’t help, but my knowledge of websockets does: it’s very pissy about about ws vs wss distinction. Ws can only be used on a page opened via http, and wss only on a page opened with https. It doesn’t support auto-renegotiation to a secure connection, nor auto selecting the encryption type. Which protocol string should I use with a page opened via file:// to an unencrypted websocket? Only one answer left after we ruled out the other one.

The stupid part here is that our JS app is minified, and we have to patch it there. After this experience, I’m not sure if I believe in JS minification anymore. I mean, what purpose does it serve? Saving a couple of hundred bytes of online traffic? Can’t gzipping solve this? More benchmarking needed.

Thankfully wss in only mentioned two times in index.js and after replacing it everything worked locally. Great!

the easy: part two

NixOS modules are easy to write after you have written one: you take it and modify slightly to suit your new service. This is the one I use every time.

Another simple thing I skipped above is writing a derivation for mumble-web-prebuilt, because god save me from just putting files in /srv and pointing nginx, oh no. I love this web page with a list of builders for nixos that allow you to produce derivations to use basically as dependencies to other derivations. But somehow while there is writeFile, there is no writeMultipleFiles so I have to do it myself:

{ pkgs ? import <nixpkgs> {}, ... }:

pkgs.stdenv.mkDerivation {
  name = "mumble-web";
  src = ./mumble-web;
  phases = [ "unpackPhase" "installPhase" ];
  installPhase = ''
    mkdir -p $out
    cp $src/* $out
  '';
}

Writing shell in 2024? Ew.

the obtuse part

Now comes my favourite activity in the world: writing nginx configs without writing nginx configs. You see, I want the websocket app and the static files webapp to be available on the same domain, so basically have both proxy_pass and root in one route. How do I do it?

  1. Just list them both as would be obvious? Nope, this gives me proxy-pass but no files
  2. Virtual location + tryFiles, the first suggestion in google: for me this gives me files but websocket is not proxied
  3. A gosh-darned if statement which turned out to be working

Here’s the idea: the websocket negotiation begins with an HTTP request with a header of Upgrade: websocket. Nginx has an automatic variable $http_upgrade set in a request and we can dynamically switch our handler based on it. The result looks something like:

locations."/" = {
  # This one is for files
  root =
    let mumble-web-prebuilt = import (pkgs.fetchgit {
        url = "https://github.com/maurges/mumble-web-prebuilt";
        deepClone = false;
        rev = "fee150a652cb75121573ff909e2c79a2cf6ff94e";
        hash = "sha256-MHFMQAieFMhJH6TYLBBnV0fvlIJ1/LLNXIh7ipf4ttg=";
      }) { inherit pkgs; };
    in "${mumble-web-prebuilt}/";

  # Allow websockets on this route
  proxyWebsockets = true;

  # And if websocket is requested, we proxy_pass
  extraConfig = ''
    if ($http_upgrade = "websocket") {
      proxy_pass http://localhost:64737;
    }
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
  '';
};

I graciously skip fighting with “proxy_set_header is not allowed here” errors.

And, I’m done; finally I can go and play some CS2. There won’t be a “what did we learn” section today because we learned too much, from rust to nginx. Ciao!