I love restic, I use it on every my server and since the developers added the backup compression feature (previously it has only deduplication) I think it’s the best backup software exisisting.

Using restic on GNU/Linux is a piece of cake and the peace of mind, but on Windows it’s a total different story…

The thing the bothers me the most on Windows is the awful Windows task scheduler and the total absence of notifications (who the hell regularly take a look to the Windows Event Viewer?).

Fortunately there’s a solution for those problem, which is Backrest, a nice web frontend to the restic that brilliantly solves both problems.

In this post I would like to show a nice solution for backup notification using Backrest and Telegram, in this way you’ll get a Telegram notification using Backrest Hooks every time a backup is successful, returns a warning or an error.

First of all open Telegram and find a user called @BotFather

Start a chat with /start and create a new bot with the command /newbot

Give a name and a username to yourbot (for example Backrest Buddy and backrest_buddy)

BotFather will give you a reply like this, the most important information is the API token

Use this token to access the HTTP API:
1234235:069:AAHgPEM2HJIh2Ci1G1l345u2QbbCs876CA
Keep your token secure and store it safely, it can be used by anyone to control your bot.

Open a browser and go to this url
https://api.telegram.org/bot<API TOKEN>/getUpdates

Then open Telegram, open a chat with your bot and write something to it, then get back to the browser and refresh the page, you’ll see some json output with a chat section with a chat id, copy that data, it’s the most important data with the API token.

Now create a powershell script (for example c:\Users\tas\backrest-buddy.ps1) with this syntax, just be careful to insert the API token and the chat id the right spots.

param(
[int]$ExitCode = 0
)

$BotToken = "<API TOKEN>"
$ChatId = "<CHAT ID>"

$HostName = $env:COMPUTERNAME
$Date = Get-Date -Format "dd-MM-yyyy HH:mm:ss"

if ($ExitCode -eq 0) {
$Text = "Backup OK`n------------------------------`n`nHost: $HostName`nDate: $Date"
} else {
$Text = "Backup FAILED`n------------------------------`n`nHost: $HostName`nData: $Date`nExit code: $ExitCode"
}

$Uri = "https://api.telegram.org/bot$BotToken/sendMessage"

$Body = @{
chat_id = $ChatId
text = $Text
parse_mode = "Markdown"
}

Invoke-RestMethod -Uri $Uri -Method Post -Body $Body

Now you can test the notification using this command for successfull backup (change the Powershell script path accordingly to your script)

powershell.exe -NoProfile -ExecutionPolicy Bypass -File c:\Users\tas\backrest-buddy.ps1 0

or this command for a failed backup

powershell.exe -NoProfile -ExecutionPolicy Bypass -File c:\Users\tas\backrest-buddy.ps1 1

If you receive Telegram messages from your bot for a successfull or failed backup everything works as expected.

Now one last step, let’s make Backrest use these powershell script to send us notifications.

Open your Backrest plan settings and go to the Hooks section.
Create three Hooks commands:
– on the first Hook choose CONDITION_SNAPSHOT_SUCCESS as condition
– on the secondo Hook choose CONDITION_SNAPSHOT_WARNING as condition
– on the first Hook choose CONDITION_SNAPSHOT_ERROR as condition

Now insert the same commands you used to test the Telegram notification as Hook commands:
– on the first Hook insert “powershell.exe -NoProfile -ExecutionPolicy Bypass -File c:\Users\tas\backrest-buddy.ps1 0”
– on the second Hook insert “powershell.exe -NoProfile -ExecutionPolicy Bypass -File c:\Users\tas\backrest-buddy.ps1 1”
– on the third Hook insert “powershell.exe -NoProfile -ExecutionPolicy Bypass -File c:\Users\tas\backrest-buddy.ps1 1”

Insert ON_ERROR_FATAL as Error Behavior on all the three hooks.

Ok you did it, now you only have to launch a backup and check your Telegram notifications, if you want to test a failed backup simply add a non existent path to the backup and launch it.

I always been a RedHat boy, the first GNU/Linux distro I seriously used was a RedHat Linux 5.0 (please note I’m not talking about RedHat Enteprise Linux, I’m talking about the old RedHat Linux distribution, before RHEL was born) and since then I always tried to stick to the RedHat side of the Linux world, even now If I have to choose a distro I’ll choose Rocky Linux over Debian or Ubuntu.

When I started working on AWS many years ago I tried Amazon Linux 2, and It was good, more or less It was like CentOS 7 and everything was ok… then came Amazon Linux 2003.

I didn’t chose It, someone else did and passed the instance to me, and since the beginning something was not right…

You can’t use EPEL…

You can’t find a lot of RHEL/CentOS/Fedora/Rocky/Alma packages on it…

You can’t even run dnf-automatic or yum-cron to automatically install updates on a scheduled base… updates are shipped in batches and you have to update the whole OS with all the updated packages… manually.

And today I found that even the damn flippin’ cron do not work, you have to manually install it with

sudo yum install cronie -y
sudo systemctl enable crond.service

What on earth?!?!?!?!

No, seriously if you have to choose a RHEL based GNU/Linux distributio choose Fedora, CentOS Stream, Rocky Linux, Alma Linux, Oracle Linux but DO NOT CHOOSE AMAZON LINUX 2003…

Amazon Linux 2003 is the devil of GNU/Linux distros, probably one of the worst distros ever made.

Last autumn I posted my2cents on Stalker 2, finally after 143 hours of play I finished it… and I’m disappointed.

Despite patches, despite almost 50 more hours of play I basically can confirm everything I wrote before, plus an even more deep disappointment about the ending of the game.

Boss fights?!?! Really?!?!

What is the meaning of putting boss fights like some stupid console game?

I chose the Strelok ending (as a fan of the series I can’t choose anything else…) and just at the end of the game you have more and more CoD fights, close quarters fight with enemies spawning from thin air and without any cover.

Fights that that end up dying over and over again until you get so frustrated to lower the difficulty level only to finish this agony.. and then boss fights?!?

Flippin’ heck who got this stupid idea? This ain’t some Metal Gear Solid game, this is Stalker for God’s sake…

I admit I always hated the “boss fight” concept in any game, It’s something inherited from console games that makes no sense with PC games, but Stalker… Stalker… is the last game on earth where you can put a boss fight, it simply makes no sense…

In the end, Stalker 2 can be considered a good FPS, nothing more nothing less, but has nothing in common with the Stalker series; Stalker 2 is like a CoD console game settled in the Zone.

If you played the real Stalker games you’ll be pleased when you start, you’ll be pleased when you enter some old facilities and locations rebuilt in this game, but sooner or later you’ll realize Stalker is another story, this game is a CoD style spinoff, period.

For those who live under a rock, during the last year one company emerged like the new villain in town, and this company is Broadcom.

Among various stupid things Broadcom management did, recently they screwed up a very nice project made by Vmware (which is now part of Broadcom) called Bitnami.

Bitnami project essentially produced some very cool and well made container images with tons of very useful software and services with a nice and clean setup and documentation, basically the stupid monkeys in charge of Broadcom management decided to introduce more and more restrictions on Bitnami images to push people to pay subscriptions for images which were free since then… and please notice that those images are based on software and services which are totally free, so Broadcom is charging a fee on something they get for free.

They have to rot in hell…

Anyway, back to our topic; one of the Bitnami images I used a lot is the Apache Tomcat servlet container one, and I love Tomcat. It’s light it’s powerful, it’s one of the pillars of the IT industry, way better than many enterprise Java application servers.

So because of Broadcom stupidity I started to get rid of my beloved Bitnami image and get back to the official Tomcat docker image.

Here’s a quick list of commands to setup a simple Tomcat server with docker containers with Tomcat manager applications and 1 GB of heap memory.

On this setup I used Tomcat 11.0.10, feel free to change the tag to whatever version of Tomcat you prefer, check the official Tomcat page on dockerhub for your desired tag.

mkdir -p /data/docker/tomcat ; cd /data/docker/tomcat
docker run -d --rm --name tomcat tomcat:11.0.10
docker cp -a tomcat:/usr/local/tomcat/webapps.dist ./webapps
docker cp tomcat:/usr/local/tomcat/conf/tomcat-users.xml .
docker stop tomcat
rm -rf webapps/docs webapps/examples/ webapps/sample
sed -i '$d' tomcat-users.xml
echo '<role rolename="manager-gui"/>' >> tomcat-users.xml
echo '<user username="tomcat-admin" password="CHANGEME" roles="manager-gui"/>' >> tomcat-users.xml
echo '</tomcat-users>' >> tomcat-users.xml
sed -i 's/127/\\d+/g' webapps/manager/META-INF/context.xml

cat << 'EOF' > docker-compose.yaml
services:
  web:
    image: tomcat:11.0.10
    container_name: tomcat
    ports:
      - "8080:8080"
    environment:
      - JAVA_OPTS=-Xms1024m -Xmx1024m
    volumes:
      - ./webapps:/usr/local/tomcat/webapps
      - ./tomcat-users.xml:/usr/local/tomcat/conf/tomcat-users.xml:ro
    restart: unless-stopped
EOF

docker compose up -d

That’s all

I know the pros will argue that I should build an image with my tomcat-users.xml file and the modified context.xml file, but I hate building images for no reasons or when I have simpler and more clean alternatives.

And by the way obviously you should not expose to the http connector to the web, use a simple reverse proxy with Apache httpd or Nginx to publish ONLY your applications contexes.

It’s been years since I started working with K8s, I lost count on how many cluster I worked on… and really I don’t understand why people is so mad at it, It’s really simple…

You can run services on K8s… except they are not what you think.

Your services in reality run on pods… which are not containers, but sandboxes that contain containers…

And inside each pod you can have how many containers as you want… but not all of those are used to run services, some are sidecars, others are temporary containers, other run services on them…

And you should not create pods and containers inside them, you’re supposed to create other objects called deployments and statefulsets… which create other objects again called replicasets… which create your pods… which have your containers inside…

And deployments and statefulsets are similar in some sense and different in others, your supposed to use statefulsets if you want to run services with persistent data and deployments if you are running “ephemeral” services… however you can run deployments with persistent data anyway…

Deployment can be configured to run several instances to your pod and scale your applications… and also statefulsets can do the same… except you end up screwing everything because your create a split brain architecture…

But let’s back to services, K8s services, which are not services but network endpoints that balance requests on your pods… but there are also ingresses that can do the same… but you must use services for some case and ingresses for others and use specific types of services for specific use cases…

And as I said you can persist data with persistent volumes… which you can’t manage yourself because you need persistent volume claims… which can do nice things like delete your data without any warning because they depend on storage classes… which are some sort of driver to use some kind of storage to save your data, but also dictates how storage must be used and which policies and rules have to be followed while interacting with your data and storage…

And then you have backups… except nobody gives a shit about backups in the K8s world… so you don’t have any backup measure or strategy out of the box and you have to rely on third party tool you have to trust…

And you can’t either mount your persistent volume on your pc and do a cold backup, because if you start your containers your applications will access data and that would be an inconsistent hot backup, and if you kill your application processes the pod cease to exist and you can’t access your data… so you have to create a temporary container inside a temporary pod which mounts the persistent volume while your application pod is down to manually create a cold backup… isn’t it simple?

So yes, Kubernetes is simple