Five minutes ago I was posting a reply on Reddit about monitoring GNU/Linux systems and I suddenly realized something very basic but extremely useful that most of the people and most of the professional sysadmins ignore: this wonderful OS will tell you everything if you let it talk to you.

In more technical terms, every GNU/Linux distribution out of the box has everything it needs to send you notifications via email if something goes wrong, you don’t have to search anything, you don’t have to parse logs, you don’t have to look for clues.

To reach this objective you only need two things:

1. install an MTA (aka an SMTP server)
2. configure the OS to forward root emails to your email address

Let’s see how to reach this simple but fundamental goal.

First of all install Postfix as SMTP (forget about Sendmail), to do this you only need to use the OS package managers:

on Debian based distribution use

sudo apt install postfix

on RedHat based distribution use

sudo dnf install postfix

Once done make sure you started Postfix and configured it to start at boot with

sudo systemctl enable --now postfix.service

Ok you’re half of the way, another little step.

Once Postfix is started you have to tell the OS to forward root email to your email address, to do this you only have to put a like like this in the /etc/aliases file

root: [email protected]

If you find a line with only “root:” remove it, or simply change it as the example I wrote before. Obviously change [email protected] with your email address…

Now one last step, saving /etc/aliases with your new alias is not enough, you have to rebuild the aliases database, and then reload Postfix, to do these things you only have to launch this command.

sudo newaliases ; sudo postfix reload

Done, now if you try to send an email to the root user Postfix will forward it to your email address, for example:

echo "lorem ipsum" | mail -s "email test" root

Obviously your server must be able to reach the MX record host of your email address domain on TCP port 25, on top of that if you used a public domain email address you have to face several other problems regarding spam or sender email domain, or SPF validation… but these are topics for another more advanced post.

Make GNU/Linux work for you, end of part 1.

I love restic, I use it on every my server and since the developers added the backup compression feature (previously it has only deduplication) I think it’s the best backup software exisisting.

Using restic on GNU/Linux is a piece of cake and the peace of mind, but on Windows it’s a total different story…

The thing the bothers me the most on Windows is the awful Windows task scheduler and the total absence of notifications (who the hell regularly take a look to the Windows Event Viewer?).

Fortunately there’s a solution for those problem, which is Backrest, a nice web frontend to the restic that brilliantly solves both problems.

In this post I would like to show a nice solution for backup notification using Backrest and Telegram, in this way you’ll get a Telegram notification using Backrest Hooks every time a backup is successful, returns a warning or an error.

First of all open Telegram and find a user called @BotFather

Start a chat with /start and create a new bot with the command /newbot

Give a name and a username to yourbot (for example Backrest Buddy and backrest_buddy)

BotFather will give you a reply like this, the most important information is the API token

Use this token to access the HTTP API:
1234235:069:AAHgPEM2HJIh2Ci1G1l345u2QbbCs876CA
Keep your token secure and store it safely, it can be used by anyone to control your bot.

Open a browser and go to this url
https://api.telegram.org/bot<API TOKEN>/getUpdates

Then open Telegram, open a chat with your bot and write something to it, then get back to the browser and refresh the page, you’ll see some json output with a chat section with a chat id, copy that data, it’s the most important data with the API token.

Now create a powershell script (for example c:\Users\tas\backrest-buddy.ps1) with this syntax, just be careful to insert the API token and the chat id the right spots.

param(
[int]$ExitCode = 0
)

$BotToken = "<API TOKEN>"
$ChatId = "<CHAT ID>"

$HostName = $env:COMPUTERNAME
$Date = Get-Date -Format "dd-MM-yyyy HH:mm:ss"

if ($ExitCode -eq 0) {
$Text = "Backup OK`n------------------------------`n`nHost: $HostName`nDate: $Date"
} else {
$Text = "Backup FAILED`n------------------------------`n`nHost: $HostName`nData: $Date`nExit code: $ExitCode"
}

$Uri = "https://api.telegram.org/bot$BotToken/sendMessage"

$Body = @{
chat_id = $ChatId
text = $Text
parse_mode = "Markdown"
}

Invoke-RestMethod -Uri $Uri -Method Post -Body $Body

Now you can test the notification using this command for successfull backup (change the Powershell script path accordingly to your script)

powershell.exe -NoProfile -ExecutionPolicy Bypass -File c:\Users\tas\backrest-buddy.ps1 0

or this command for a failed backup

powershell.exe -NoProfile -ExecutionPolicy Bypass -File c:\Users\tas\backrest-buddy.ps1 1

If you receive Telegram messages from your bot for a successfull or failed backup everything works as expected.

Now one last step, let’s make Backrest use these powershell script to send us notifications.

Open your Backrest plan settings and go to the Hooks section.
Create three Hooks commands:
– on the first Hook choose CONDITION_SNAPSHOT_SUCCESS as condition
– on the secondo Hook choose CONDITION_SNAPSHOT_WARNING as condition
– on the first Hook choose CONDITION_SNAPSHOT_ERROR as condition

Now insert the same commands you used to test the Telegram notification as Hook commands:
– on the first Hook insert “powershell.exe -NoProfile -ExecutionPolicy Bypass -File c:\Users\tas\backrest-buddy.ps1 0”
– on the second Hook insert “powershell.exe -NoProfile -ExecutionPolicy Bypass -File c:\Users\tas\backrest-buddy.ps1 1”
– on the third Hook insert “powershell.exe -NoProfile -ExecutionPolicy Bypass -File c:\Users\tas\backrest-buddy.ps1 1”

Insert ON_ERROR_FATAL as Error Behavior on all the three hooks.

Ok you did it, now you only have to launch a backup and check your Telegram notifications, if you want to test a failed backup simply add a non existent path to the backup and launch it.

I always been a RedHat boy, the first GNU/Linux distro I seriously used was a RedHat Linux 5.0 (please note I’m not talking about RedHat Enteprise Linux, I’m talking about the old RedHat Linux distribution, before RHEL was born) and since then I always tried to stick to the RedHat side of the Linux world, even now If I have to choose a distro I’ll choose Rocky Linux over Debian or Ubuntu.

When I started working on AWS many years ago I tried Amazon Linux 2, and It was good, more or less It was like CentOS 7 and everything was ok… then came Amazon Linux 2003.

I didn’t chose It, someone else did and passed the instance to me, and since the beginning something was not right…

You can’t use EPEL…

You can’t find a lot of RHEL/CentOS/Fedora/Rocky/Alma packages on it…

You can’t even run dnf-automatic or yum-cron to automatically install updates on a scheduled base… updates are shipped in batches and you have to update the whole OS with all the updated packages… manually.

And today I found that even the damn flippin’ cron do not work, you have to manually install it with

sudo yum install cronie -y
sudo systemctl enable crond.service

What on earth?!?!?!?!

No, seriously if you have to choose a RHEL based GNU/Linux distributio choose Fedora, CentOS Stream, Rocky Linux, Alma Linux, Oracle Linux but DO NOT CHOOSE AMAZON LINUX 2003…

Amazon Linux 2003 is the devil of GNU/Linux distros, probably one of the worst distros ever made.

Last autumn I posted my2cents on Stalker 2, finally after 143 hours of play I finished it… and I’m disappointed.

Despite patches, despite almost 50 more hours of play I basically can confirm everything I wrote before, plus an even more deep disappointment about the ending of the game.

Boss fights?!?! Really?!?!

What is the meaning of putting boss fights like some stupid console game?

I chose the Strelok ending (as a fan of the series I can’t choose anything else…) and just at the end of the game you have more and more CoD fights, close quarters fight with enemies spawning from thin air and without any cover.

Fights that that end up dying over and over again until you get so frustrated to lower the difficulty level only to finish this agony.. and then boss fights?!?

Flippin’ heck who got this stupid idea? This ain’t some Metal Gear Solid game, this is Stalker for God’s sake…

I admit I always hated the “boss fight” concept in any game, It’s something inherited from console games that makes no sense with PC games, but Stalker… Stalker… is the last game on earth where you can put a boss fight, it simply makes no sense…

In the end, Stalker 2 can be considered a good FPS, nothing more nothing less, but has nothing in common with the Stalker series; Stalker 2 is like a CoD console game settled in the Zone.

If you played the real Stalker games you’ll be pleased when you start, you’ll be pleased when you enter some old facilities and locations rebuilt in this game, but sooner or later you’ll realize Stalker is another story, this game is a CoD style spinoff, period.

For those who live under a rock, during the last year one company emerged like the new villain in town, and this company is Broadcom.

Among various stupid things Broadcom management did, recently they screwed up a very nice project made by Vmware (which is now part of Broadcom) called Bitnami.

Bitnami project essentially produced some very cool and well made container images with tons of very useful software and services with a nice and clean setup and documentation, basically the stupid monkeys in charge of Broadcom management decided to introduce more and more restrictions on Bitnami images to push people to pay subscriptions for images which were free since then… and please notice that those images are based on software and services which are totally free, so Broadcom is charging a fee on something they get for free.

They have to rot in hell…

Anyway, back to our topic; one of the Bitnami images I used a lot is the Apache Tomcat servlet container one, and I love Tomcat. It’s light it’s powerful, it’s one of the pillars of the IT industry, way better than many enterprise Java application servers.

So because of Broadcom stupidity I started to get rid of my beloved Bitnami image and get back to the official Tomcat docker image.

Here’s a quick list of commands to setup a simple Tomcat server with docker containers with Tomcat manager applications and 1 GB of heap memory.

On this setup I used Tomcat 11.0.10, feel free to change the tag to whatever version of Tomcat you prefer, check the official Tomcat page on dockerhub for your desired tag.

mkdir -p /data/docker/tomcat ; cd /data/docker/tomcat
docker run -d --rm --name tomcat tomcat:11.0.10
docker cp -a tomcat:/usr/local/tomcat/webapps.dist ./webapps
docker cp tomcat:/usr/local/tomcat/conf/tomcat-users.xml .
docker stop tomcat
rm -rf webapps/docs webapps/examples/ webapps/sample
sed -i '$d' tomcat-users.xml
echo '<role rolename="manager-gui"/>' >> tomcat-users.xml
echo '<user username="tomcat-admin" password="CHANGEME" roles="manager-gui"/>' >> tomcat-users.xml
echo '</tomcat-users>' >> tomcat-users.xml
sed -i 's/127/\\d+/g' webapps/manager/META-INF/context.xml

cat << 'EOF' > docker-compose.yaml
services:
  web:
    image: tomcat:11.0.10
    container_name: tomcat
    ports:
      - "8080:8080"
    environment:
      - JAVA_OPTS=-Xms1024m -Xmx1024m
    volumes:
      - ./webapps:/usr/local/tomcat/webapps
      - ./tomcat-users.xml:/usr/local/tomcat/conf/tomcat-users.xml:ro
    restart: unless-stopped
EOF

docker compose up -d

That’s all

I know the pros will argue that I should build an image with my tomcat-users.xml file and the modified context.xml file, but I hate building images for no reasons or when I have simpler and more clean alternatives.

And by the way obviously you should not expose to the http connector to the web, use a simple reverse proxy with Apache httpd or Nginx to publish ONLY your applications contexes.