Ejabberd certificates

I know that jabber/XMPP protocol has been out of fashion for some time, at least in the sense that most users wouldn’t be using a direct XMPP client these days. Of course, the protocol is not dead, there are a lot of conversations that take place without us users even realizing that this protocol is there somewhere down the stack in some form or shape.

And I certainly understand why other chat options took hold over time. Especially the ability to deliver offline messages to multiple devices was something that new messaging application, especially on mobile phones, did inherently better from the start while XMPP was just catching up.

That being said, I’m still fond of jabber/XMPP and still find it useful even in today’s landscape. Sadly, there’s hardly anyone left from my contacts who still goes online. But since it’s self-hosted, it can pull a lot of weight when it comes to what can be best described as service messages. For instance, Jenkins can use jabber to send messages when something goes wrong and it would pop up immediately on my phone thanks to Conversations, which is a really great client for Android phones. Another use case would be a tablet that can’t accommodate the likes of Signal as a phone number is needed there. Conversations client can step in here easily. But most importantly, even if it’s currently just a minor thing, it’s good to know that in this world of perpetual attempts at surveillance (like Chat Control at the time of writing this), there is a self-hosted fallback option that my family can use if we want to.

Now, besides the above mentioned reasons for using XMPP, there’s another reason for me – self-hosting a jabbber/XMPP service without prior experience brings about a lot of fun moments and learning opportunities. I decided to go for ejabberd as the server of my choice and it took me some time to wrap my head around it, certificate management being especially tricky. But as usual, with enough time for documentation, it turns out that it works just as was intended to, the only obstacle was lack of understanding on my part at first. I’m just going to leave these notes here because I’m pretty sure that it might come handy once I forgot it all and need to do a refresh or a migration at some point in the future.

The first trick was the part of ejabberd configuration in /etc/ejabberd/ejabberd.yml that listens for http connections on port 5280 by default, which including any incoming acme requests. This is the part in question:

 -
    port: 5280
    ip: "::"
    module: ejabberd_http
    tls: false
    protocol_options: 'TLS_OPTIONS'
    request_handlers:
      /admin: ejabberd_web_admin
      /.well-known/acme-challenge: ejabberd_acme

The thing with acme verification is that it needs to go to port 80 and can’t be forced (to the best of my knowledge) to any other port. Here, the port was already being used by Apache so I had to stop it temporarily so that I could let ejabberd.yml use it just for a short while:

  -
    port: 80
    ip: "::"
    module: ejabberd_http
    tls: false
    protocol_options: 'TLS_OPTIONS'
    request_handlers:
      /admin: ejabberd_web_admin
      /.well-known/acme-challenge: ejabberd_acme

After restarting ejabberd service, I was able to successfully request a certificate for my domain using ejjaberctl command:

# ejabberdctl request-certificate example.com

Then I changed it back to 5280 so that I could bring Apache back. Of course, this was a rather clumsy solution that I only used because I was exploring how ejabberd certificates worked. Since then, I’ve set up a reverse proxy on this Apache instance pointing to ejabberd’s 5280 so that any future acme verification requests could be accommodated without this manual workaround.

Existing certificates that are handled by ejabberd can be listed using ejabberdctl and ejabber takes them into account automatically without me having to list them in the yaml configuration in any way:

# ejabberdctl list-certificates

A domain that’s pointing somewhere else

With my second domain (say example.org), I ran into a completely different situation. While the first one was pointing to the same machine that ejabberd was running on, with the other domain, I was not able to use the built-in ejabberd certificate management system simply because the domain in question had the A and AAAA records pointing somewhere else and so any acme request would be going there rather than to the machine with ejabberd.

It would be fine be me if the example.com certificate could be used instead of the example.org certificate with some kind of a warning. And indeed, when using Pidgin or Thunderbird as a client, it could work like that. But Conversations client on phone was more strict about it and was adamantly refusing to connect stating that it wasn’t able to verify the domain example.org.

As the domain example.org was pointing somewhere else where certificates were actually generated, I was able to take the existing keypair from the other machine and feed it to ejabberd.

cat example.org.key > /etc/ejabberd/example.org.pem 
cat fullchain.cer >> /etc/ejabberd/example.org.pem 
chmod 640 /etc/ejabberd/example.org.pem
chgrp ejabberd /etc/ejabberd/example.org.pem

This takes the key, the certificate and the chain and puts them all together in a single .pem file. In this case, the certificate and the chain files were already concatenated in a file called fullchain.cer in the correct order, otherwise it would have had to be fed into the .pem file individually in this specific order: key, cert, chain.

Then, I just had to let ejabberd know about its existence by adding it to the cerfiles section of the /etc/ejabberd/ejabberd.yml file:

certfiles:
  - "/etc/ejabberd/ejabberd.pem"
  - "/etc/ejabberd/example.org.pem"

Having done all this and restarting ejabberd one more time, I was able to connect my user@example.org in the Conversations client finally.

That’s all, whatever you use, happy chatting everyone.