System Administration

Two years ago I heard from a friend that they wanted to send an email to my wife, but the email never went through. Instead the friend received an automated message from their email provider (Gmail) informing them that the email delivery had permanently failed. The automated message contained this error information:

TLS Negotiation failed: FAILED_PRECONDITION: starttls error (71):
6945907163592:error:10000417:SSL routines:
OPENSSL_internal:SSLV3_ALERT_ILLEGAL_PARAMETER:
third_party/openssl/boringssl/src/ssl/tls_record.cc:594:SSL alert number 47

I had never seen something like that before, also none of the contacts that are sending us private or business messages on a regular basis had ever reported any problems with sending us emails. I did some tests, such as checking the certificates being used by the Exim MTA on my dedicated Linux server (they were in order), but also sending myself an email from my own Gmail account (the email arrived). So nothing seemed amiss.

I then tried to find out more about the root cause by searching the net for keywords from the error message above, but was ultimately unsuccessful. As a side effect I found out the following about BoringSSL from their website:

BoringSSL is a fork of OpenSSL that is designed to meet Google’s needs.

So at this point it looked like this was a) a Gmail-specific issue that b) mysteriously affected only our friend (remember: I had been able to send myself emails from my own Gmail account). Lacking the time for further investigation, I dropped the case for the moment, after which it lay dormant for the next two years - until I was jolted by its reoccurrence a few days ago!

OpenLDAP 2.3 (released in 2005) introduced a new way for configuring the slapd daemon. The traditional method was a configuration file (/etc/ldap/slapd.conf on Debian) that could simply be edited with a text editor. The new way follows the Eat your own dog food maxim: The configuration is stored in a set of LDIF files (stored under /etc/ldap/slapd.d in Debian) which cannot be edited directly with a text editor. Instead, all changes must be done via LDAP operations. Funny enough: In order to configure the daemon, the daemon must already be running.

slapd-config, as the new configuration method is called, may be a technically cool feature, but from a casual sysadmin's point of view it is nothing but a major pain in the butt! So you want to quickly change slapd's log level to diagnose some authentication problem? OK, first check the documentation to see where the log level option is located in the configuration schema, and how its attribute is called. Then query the running daemon to look at the current value(s). Then write an .ldif file that contains the change. Then issue a complicated ldapmodify command that requires more cryptic options than tar and cpio combined. Of course the .ldif file contains an error, so diagnose & repeat. After maybe 20 minutes the job is done. Phew, only 20 minutes to change the daemon log level, I am such an LDAP wizard!

Actually when I did this just now it took me more like an hour because I am so not used to the procedure (another explanation might be that I'm just stupid, but hey, I think that's not it). Since I don't want to repeat the experience, I have started to write up some recipes on my wiki. Here's the link in case you are interested.

Two good news from the Mediawiki battle front of a "grizzled Debian veteran" :-) Read the whole article if you are interested.

The following story is probably not very interesting to anyone out there, but it serves as a reminder (at least to myself) why backups are so important. Today I was lucky enough to spot disaster creeping up on me in time, but tomorrow I might be too late...

The whole thing started when my wife told me that "there are a lot of funny error messages on our homepage". "Error messages on our homepage, what do you mean?" I replied, and then pointed my browser at the website to see for myself. This is what I saw:

I have written a script that regularly updates a list with IP addresses that have committed one or another offence in regard to the network security rules on my Internet gateway osgiliath.herzbube.ch. If you are interested to have a look, follow this link (link is now defunct)

Finally there is a Debian package for Bugzilla 2.20. I have upgraded the site to the 2.20 version.