This is an old post I had in draft since… I don’t know, maybe years.

Anyway, CentOS 8 is old and out of support, but still I think running several instances of Apache httpd server through systemd can be useful today in other modern Linux distributions, so I think it’s time to clean up the drafts and publish it.

—————————————————

Recently a customer asked me to setup a webdav access to a vm to change some files inside a couple of java web applications deployed on a Tomcat instance.
My first choice was to configure the webdav servlet already available with Tomcat, which sounds like a nice and elegant solution, but that went wrong because webdav http methods were blocked by some Spring waf protection, and change the java applications for that was not an option (for various reasons I will not explain but not technical ones).

At this point I thought to create a new virtualhost for the webdav access, but in that case file ownership would be a problem, and running the frontend webserver with write permission on all the applications resouces was not a good idea.
The solution was simple, setup a new webserver running as tomcat user (Tomcat file owner) on a different port (and also only available on lan) but I found no documentation with some of the latest distros using systemd.
Yes I know, I could install nginx or another webserver, but running a new Apache configuration felt much more elegant to me.

First of all, create a new httpd systemd unit copying the old one with a new name

cp -v /usr/lib/systemd/system/httpd.service /usr/lib/systemd/system/httpd-dav.service

Copy the main httpd config file for the new webserver

cp -v /etc/httpd/conf/httpd.conf /etc/httpd/conf/httpd-dav.conf

Enable the new systemd unit to start at boot

systemctl enable httpd-dav.service

To check if it’s ok run this

systemctl list-unit-files | grep httpd-dav

Now you must edit the new systemd unit to use the new httpd-dav.conf file

systemctl edit httpd-dav.service

…and add this

[Service]
Environment=OPTIONS="-f /etc/httpd/conf/httpd-dav.conf"

Now you must edit the new httpd-dav.conf changing some basic directives to not overlap the main Apache configuration:
In my case I changed Listen PidFile, User, Group, ErrorLog, CustomLog, I removed “IncludeOptional conf.d/*.conf” and added a new virtualhost with mod_dav active, basic authentication, etc etc…
Adjust your Apache configurations as you need, but at least you have to change the Listen and PidFile directives to avoid conflicts with the other httpd process.

When you’re ready you only have to start it with

systemctd start httpd-dav.service

I hope this can be helpful.

I love Bookstack, actually I think it’s one of the best wiki project existing.

It’s well documented, it works like charm, the developer is very active (and he’s also a very kind person, which has nothing to do with the software, but it’s always a pleasure interact with him) and it has very nice features:

• a nice and responsive design
• drafts autosave
• MFA out of the box
• diagrams.net integration

It also works perfectly fine in a docker container, technically the official project do not offer a container image, but there are two groups building them and they’re referred directly in the official documentation.

Recently I started to sort things out on my beloved Raspberry PI 5, in particular I’m moving services so I can reverse proxy them on a single Apache httpd instance (you know I still love Apache :D ), today I moved around Bookstack, in particulare I did two things:

1. change Bookstack hostname (for example from https://site.domain.tld to https://newsite.domain.tld )
2. make Bookstack work under a specific url context (for example https://site.domain.tld/bookstack instead of https://site.domain.tld ).

On my environment I’m using the LinuxServer.io docker image, so check the project site for details, and also I’m using docker compose, if you’re not familiar with it start using it for Reorx’s sake.

Backup

First of all take a damn backup, it’s mandatory.

Seriously I’m not joking.

Stop the containers

cd /data/docker/bookstack ; docker compose down

Backup files with a simple tar, restic, kopia, whatever you want, but DO IT!

cd /data/docker/ ; tar -cpzf /backup/bookstack-backup.tar.gz bookstack

Change Bookstack hostname

This process is documented on the Bookstack documentation (LINK), but still I decided to mention it because the procedure is a little bit different on a docker container, so it’s worth spent a few words about it.

First of all you have to change the APP_URL configuration variable, in case of a docker container it’s enough to change the environment variable on the docker-compose.yaml file, so open the file and change the variable to the new url

Now you must replace the old url from the database record with the new one using the bookstack:update-url command, in case of a docker container you must identify where’s the Laravel framework artisan file and launch it accordingly to the documentation.

docker exec -it bookstack php /app/www/artisan bookstack:update-url https://site.domain.tld https://newsite.domain.tld

After that clear the cache using

docker exec -it bookstack php /app/www/artisan cache:clear

Restart the docker container to change the environment variable you previously changed with the new url.

cd /data/docker/bookstack ; docker compose down ; docker compose up -d

Done, now your Bookstack instance should be reachable to the new url.

Change Bookstack root context

This change is a little bit tricky, because it involves some webserver changes.

First of all you must repeat the same process used for changing the url hostname of your Bookstack instance, this time including the context you want to use (for example /bookstack ).

Let’s quickly review the steps:

1) Change the APP_URL environment variable in the docker-compose.yaml (APP_URL=https://newsite.domain.tld/bookstack in this case)

2) Replace the url in the database using the bookstack:update-url

docker exec -it bookstack php /app/www/artisan bookstack:update-url https://newsite.domain.tld https://newsite.domain.tld/bookstack

3) Clear Bookstack cache

docker exec -it bookstack php /app/www/artisan cache:clear

4) Restart the docker container to change the environment variable you previously changed with the new url.

cd /data/docker/bookstack ; docker compose down ; docker compose up -d

Now you must review the webserver configuration inside your docker container, in case of the LinuxServer.io container there’s a nginx instance running inside the container, you can fine its configuration inside the /config/nginx/ directory inside the container.

If you followed the LinuxServer.io recommendations the /config directory should be a persistent volume (or a persistent path on your docker host), so any changes in the nginx configuration files should not be lost in case of a container restart.

In my case the config persistent volume is located in the /data/docker/bookstack/bookstack-config directory, so the nginx configuration is located in the file /data/docker/bookstack/bookstack-config/nginx/site-confs/default.conf.

Apply this patch

wget https://tasslehoff.burrfoot.it/pub/bookstack-nginx.patch ; \
patch /data/docker/bookstack/bookstack-config/nginx/site-confs/default.conf < bookstack-nginx.patch

Reload nginx configuration

docker exec -it bookstack nginx -s reload

Done, now your Bookstack instance should work at the new url https://newsite.domain.tld/bookstack

Recently I had a nice discussion on a forum on Kubernetes, for me the topic was crystal clear since the first time I used it and understood a little bit about it, but sometimes this discussion repeats on the web, so I decided to save here my thoughts on this, so I can copy and paste it in the future to save some time…

I have to be honest, my experience with K8s always been the 2nd image…

When I first came in contact with K8s my first impression was: “this is a fantastic tool to solve a problem… that almost nobody has”.

Now after two years working as a sysadmin in a company extremely involved in K8s, where I learned, teached, installed, configured, fixed countless K8s cluster my conclusion is: “this is a fantastic tool to solve a problem… that almost nobody has”.

At the end of the day, despite all the buzzwords the main reason to use K8s is scalability, horizontal scalability, there are no other advantages.

Some people thinks the main advantage is the declarative approach to infrastructure, but I disagree; first of all K8s does not force you to use a declarative approach, you can do everything with an imperative approach (kubect create bla bla bla… instead of using manifests).
On the other hand also docker (with docker-compose) can do the same, so why use a more complex tool (K8s, which is much more complex) to get the same result?

No K8s approach, its objects, its object relations everything is built with one purpose: scalability.

But honestly who cares about scalability?

If you’re a Facebook or Amazon or Microsoft of Google obviously it’s important and necessary.
If you’re a big campus lab with thousands of services or the Cern datacenter or other examples like that it’s important and almost necessary.

But even if you’re a big company honestly nobody gives a damn about scalability from a technical point of view.

Ok HA is important, maybe distribute load on a couple of nodes makes things easier for maintenance, but in most of the case if you plan a good maintenance window and announce it correctly to your users nobody cares.

On top of that scalability works well with stateless applications, but in reality in my experience (25 years in the IT industry) stateless applications are the exception, the vast majority of applications are stateful, so it’s not that simple to scale up and make everything work ok, you have to deal with a lot of details (persistent data, sticky sessions, concurrency and so on…).

So what’s the end result in so many companies?

K8s cluster everywhere (in a lot of case single node “clusters”) requested by some manager who don’t understand a thing about them, installed by some consultant company, completely abandoned and managed by nobody, with a couple of applications made with statefulsets or deployments with only one replica.

Logging management is a pain in the a$$, backups are a paint in the a$$, and everybody hopes it will run forever because nobody in the company knows how to deal with problems, exceptions, certificates expirations and so on…

The very opposite of the basic principle that should be applied in the IT, the KISS principle.

As I said in the previous post I’m in love with Cygwin… but I’m also in love with Keepass… and I also live with ssh… so how to make my life easier?

The answer is simple: KeeAgent

KeeAgent is a Keepass plugin that can work as a ssh agent, in this way:

1. you can save you ssh private keys in a safe and encrypted Keepass database –> NO MORE PRIVATE KEYS ON FILESYSTEM
2. you don’t have to deal with ssh private keys passphrases, because they’re stored in the Keepass entries where you’ll save private keys –> NO MORE UNENCRYPTED PRIVATE KEYS
3. you can easily retrieve public keys to authorize hosts access without searching like a hound on your filesystem, in a couple of clicks KeeAgent will provide them

Apparently the setup of all this thing may seem complex, in reality it’s quite simple:

1. download and install KeeAgent from https://github.com/dlech/KeeAgent/releases
2. in KeeAgent options (Tools -> Options -> KeeAgent) put this string into “Cygwin/MSYS Integration” Path

   %UserProfile%/.keeagent.sock

3. restart Keepass and check if there’s the file .keeagent.sock in your home and it’s a socket file

   file ~/.keeagent.sock

4. export SSH_AUTH_SOCK variable and put it on your .bashrc file, the value of the variable should be the path of the socket file you set at the point number 2.

   echo "export SSH_AUTH_SOCK=~/.keeagent.sock" >> ~/.bashrc

5. make sure you have installed Cygwin OpenSSH and you’re not using Windows ssh

The setup is complete, now you only have to create key entries in your Keepass database, attach private key files, fill the password field and enable those entries to be loaded with KeeAgent.

For more details check the super clear KeeAgent documentation.

Working as a GNU/Linux sysadmin means working all day long on a terminal, it doesn’t matter if you’re dealing with a server, a container or a Kubernetes cluster, and for me terminal means Cygwin.

Microsoft made huge changes since Windows 10 to make it easy to work with GNU/Linux on a Windows workstation, think about OpenSSH available by default or WSL, but for me Cygwin is still the 1st choice, no virtualization, no  WSL network issues, (almost) all the GNU/Linux utils available, fast and native to Windows.

But sometimes having two home directories (one for Cygwin, one for your Windows user) is not convenient at all… but there’s a solution for that, it’s easy and it’s fast and do not involve any workaround or symlinks, you can tell Cygwin to use the Windows home directory as a Cygwin home.

For doing it you simply have to edit the /etc/nsswitch.conf file and change the db_home variable, using as value “/%H”, here’s the result:

tas@tas ~
$ grep ^db_home /etc/nsswitch.conf
db_home: /%H

The next terminal you’ll open will have its home into “C:\Users\<user>” or if you are a Cygwin lover “/cygdrive/c/Users/<user>” ;)