Five minutes ago I was posting a reply on Reddit about monitoring GNU/Linux systems and I suddenly realized something very basic but extremely useful that most of the people and most of the professional sysadmins ignore: this wonderful OS will tell you everything if you let it talk to you.

In more technical terms, every GNU/Linux distribution out of the box has everything it needs to send you notifications via email if something goes wrong, you don’t have to search anything, you don’t have to parse logs, you don’t have to look for clues.

To reach this objective you only need two things:

1. install an MTA (aka an SMTP server)
2. configure the OS to forward root emails to your email address

Let’s see how to reach this simple but fundamental goal.

First of all install Postfix as SMTP (forget about Sendmail), to do this you only need to use the OS package managers:

on Debian based distribution use

sudo apt install postfix

on RedHat based distribution use

sudo dnf install postfix

Once done make sure you started Postfix and configured it to start at boot with

sudo systemctl enable --now postfix.service

Ok you’re half of the way, another little step.

Once Postfix is started you have to tell the OS to forward root email to your email address, to do this you only have to put a like like this in the /etc/aliases file

root: [email protected]

If you find a line with only “root:” remove it, or simply change it as the example I wrote before. Obviously change [email protected] with your email address…

Now one last step, saving /etc/aliases with your new alias is not enough, you have to rebuild the aliases database, and then reload Postfix, to do these things you only have to launch this command.

sudo newaliases ; sudo postfix reload

Done, now if you try to send an email to the root user Postfix will forward it to your email address, for example:

echo "lorem ipsum" | mail -s "email test" root

Obviously your server must be able to reach the MX record host of your email address domain on TCP port 25, on top of that if you used a public domain email address you have to face several other problems regarding spam or sender email domain, or SPF validation… but these are topics for another more advanced post.

Make GNU/Linux work for you, end of part 1.

I love restic, I use it on every my server and since the developers added the backup compression feature (previously it has only deduplication) I think it’s the best backup software exisisting.

Using restic on GNU/Linux is a piece of cake and the peace of mind, but on Windows it’s a total different story…

The thing the bothers me the most on Windows is the awful Windows task scheduler and the total absence of notifications (who the hell regularly take a look to the Windows Event Viewer?).

Fortunately there’s a solution for those problem, which is Backrest, a nice web frontend to the restic that brilliantly solves both problems.

In this post I would like to show a nice solution for backup notification using Backrest and Telegram, in this way you’ll get a Telegram notification using Backrest Hooks every time a backup is successful, returns a warning or an error.

First of all open Telegram and find a user called @BotFather

Start a chat with /start and create a new bot with the command /newbot

Give a name and a username to yourbot (for example Backrest Buddy and backrest_buddy)

BotFather will give you a reply like this, the most important information is the API token

Use this token to access the HTTP API:
1234235:069:AAHgPEM2HJIh2Ci1G1l345u2QbbCs876CA
Keep your token secure and store it safely, it can be used by anyone to control your bot.

Open a browser and go to this url
https://api.telegram.org/bot<API TOKEN>/getUpdates

Then open Telegram, open a chat with your bot and write something to it, then get back to the browser and refresh the page, you’ll see some json output with a chat section with a chat id, copy that data, it’s the most important data with the API token.

Now create a powershell script (for example c:\Users\tas\backrest-buddy.ps1) with this syntax, just be careful to insert the API token and the chat id the right spots.

param(
[int]$ExitCode = 0
)

$BotToken = "<API TOKEN>"
$ChatId = "<CHAT ID>"

$HostName = $env:COMPUTERNAME
$Date = Get-Date -Format "dd-MM-yyyy HH:mm:ss"

if ($ExitCode -eq 0) {
$Text = "Backup OK`n------------------------------`n`nHost: $HostName`nDate: $Date"
} else {
$Text = "Backup FAILED`n------------------------------`n`nHost: $HostName`nData: $Date`nExit code: $ExitCode"
}

$Uri = "https://api.telegram.org/bot$BotToken/sendMessage"

$Body = @{
chat_id = $ChatId
text = $Text
parse_mode = "Markdown"
}

Invoke-RestMethod -Uri $Uri -Method Post -Body $Body

Now you can test the notification using this command for successfull backup (change the Powershell script path accordingly to your script)

powershell.exe -NoProfile -ExecutionPolicy Bypass -File c:\Users\tas\backrest-buddy.ps1 0

or this command for a failed backup

powershell.exe -NoProfile -ExecutionPolicy Bypass -File c:\Users\tas\backrest-buddy.ps1 1

If you receive Telegram messages from your bot for a successfull or failed backup everything works as expected.

Now one last step, let’s make Backrest use these powershell script to send us notifications.

Open your Backrest plan settings and go to the Hooks section.
Create three Hooks commands:
– on the first Hook choose CONDITION_SNAPSHOT_SUCCESS as condition
– on the secondo Hook choose CONDITION_SNAPSHOT_WARNING as condition
– on the first Hook choose CONDITION_SNAPSHOT_ERROR as condition

Now insert the same commands you used to test the Telegram notification as Hook commands:
– on the first Hook insert “powershell.exe -NoProfile -ExecutionPolicy Bypass -File c:\Users\tas\backrest-buddy.ps1 0”
– on the second Hook insert “powershell.exe -NoProfile -ExecutionPolicy Bypass -File c:\Users\tas\backrest-buddy.ps1 1”
– on the third Hook insert “powershell.exe -NoProfile -ExecutionPolicy Bypass -File c:\Users\tas\backrest-buddy.ps1 1”

Insert ON_ERROR_FATAL as Error Behavior on all the three hooks.

Ok you did it, now you only have to launch a backup and check your Telegram notifications, if you want to test a failed backup simply add a non existent path to the backup and launch it.

I always been a RedHat boy, the first GNU/Linux distro I seriously used was a RedHat Linux 5.0 (please note I’m not talking about RedHat Enteprise Linux, I’m talking about the old RedHat Linux distribution, before RHEL was born) and since then I always tried to stick to the RedHat side of the Linux world, even now If I have to choose a distro I’ll choose Rocky Linux over Debian or Ubuntu.

When I started working on AWS many years ago I tried Amazon Linux 2, and It was good, more or less It was like CentOS 7 and everything was ok… then came Amazon Linux 2003.

I didn’t chose It, someone else did and passed the instance to me, and since the beginning something was not right…

You can’t use EPEL…

You can’t find a lot of RHEL/CentOS/Fedora/Rocky/Alma packages on it…

You can’t even run dnf-automatic or yum-cron to automatically install updates on a scheduled base… updates are shipped in batches and you have to update the whole OS with all the updated packages… manually.

And today I found that even the damn flippin’ cron do not work, you have to manually install it with

sudo yum install cronie -y
sudo systemctl enable crond.service

What on earth?!?!?!?!

No, seriously if you have to choose a RHEL based GNU/Linux distributio choose Fedora, CentOS Stream, Rocky Linux, Alma Linux, Oracle Linux but DO NOT CHOOSE AMAZON LINUX 2003…

Amazon Linux 2003 is the devil of GNU/Linux distros, probably one of the worst distros ever made.

Here’s a quick tip to update your Windows applications using Winget.

If you don’t know this tool winget is the Windows Package Manager available since Windows 10 1809 (build 17763). It works like charm and can get your life waaaaaaayyy easier to keep your system updated.

Basically to update all your Windows applications available through winget you have to open a command prompt as Administrator and launch

winget upgrade --all --silent --accept-source-agreements --accept-package-agreements

If you want to keep some of your applications pinned to a specific version and avoid to update them use

winget pin add <Vendor.Packagename> --blocking

To see the list of pinned packages use

winget pin list

To see the list of your packages use

winget list

This is an old post I had in draft since… I don’t know, maybe years.

Anyway, CentOS 8 is old and out of support, but still I think running several instances of Apache httpd server through systemd can be useful today in other modern Linux distributions, so I think it’s time to clean up the drafts and publish it.

—————————————————

Recently a customer asked me to setup a webdav access to a vm to change some files inside a couple of java web applications deployed on a Tomcat instance.
My first choice was to configure the webdav servlet already available with Tomcat, which sounds like a nice and elegant solution, but that went wrong because webdav http methods were blocked by some Spring waf protection, and change the java applications for that was not an option (for various reasons I will not explain but not technical ones).

At this point I thought to create a new virtualhost for the webdav access, but in that case file ownership would be a problem, and running the frontend webserver with write permission on all the applications resouces was not a good idea.
The solution was simple, setup a new webserver running as tomcat user (Tomcat file owner) on a different port (and also only available on lan) but I found no documentation with some of the latest distros using systemd.
Yes I know, I could install nginx or another webserver, but running a new Apache configuration felt much more elegant to me.

First of all, create a new httpd systemd unit copying the old one with a new name

cp -v /usr/lib/systemd/system/httpd.service /usr/lib/systemd/system/httpd-dav.service

Copy the main httpd config file for the new webserver

cp -v /etc/httpd/conf/httpd.conf /etc/httpd/conf/httpd-dav.conf

Enable the new systemd unit to start at boot

systemctl enable httpd-dav.service

To check if it’s ok run this

systemctl list-unit-files | grep httpd-dav

Now you must edit the new systemd unit to use the new httpd-dav.conf file

systemctl edit httpd-dav.service

…and add this

[Service]
Environment=OPTIONS="-f /etc/httpd/conf/httpd-dav.conf"

Now you must edit the new httpd-dav.conf changing some basic directives to not overlap the main Apache configuration:
In my case I changed Listen PidFile, User, Group, ErrorLog, CustomLog, I removed “IncludeOptional conf.d/*.conf” and added a new virtualhost with mod_dav active, basic authentication, etc etc…
Adjust your Apache configurations as you need, but at least you have to change the Listen and PidFile directives to avoid conflicts with the other httpd process.

When you’re ready you only have to start it with

systemctd start httpd-dav.service

I hope this can be helpful.