/var/log/tzermias

Deploying a software RIPE atlas probe and ppp

Thu, 08 Apr 2021 23:03:00 +0000

On 2010 RIPE Network Coordination Centre (RIPE NCC) established a distributed platform for Internet measurements called “RIPE Atlas”.
The project consists of multiple hardware devices, called probes, that conduct measurements such as ping and traceroute to specific destinations.
The concept is to hand over RIPE Atlas probes free of change to volunteers, which in turn install them to their networks and contribute a small portion of their bandwidth to the project.
As an incentive, RIPE provides “credits” to volunteers based on their probe uptime. So the more the uptime your probe has, the more the credits you get.
Credits can be redeemed, by creating and conducting custom measurements on the platform.
Currently, more than 11.000 probes have been deployed globally and many network tools have been developed, based on this platform.
One of those probes is hosted on my parent’s house in Heraklion, Crete since 2013.

In February 2020, RIPE announced the availability of software probes, to improve platform coverage even on smaller networks. So instead of installing a network device (the probe) to your network, you install the software package for the probe in a machine connected to your network.
RIPE provides packages to install the probe at CentOS and Debian distributions. There is even a Docker image developed by the community.
I used the latter option, to host another probe on my home’s OrangePI.

One might reasonably wonder though. “Hey, do you trust that this device won’t intercept your local network traffic, or tamper with your other devices in any way?”.
As stated on the FAQ, the probe executes measurement commands against the public internet only.
For the security cautious, it is advised that the probe should be install outside of one’s local network (e.g on a DMZ network). I had followed this recommendation on my hardware probe before, but how am I going to perform such isolation in Docker?
To make matters worse, I am using the CPE provided by my ISP, which has very limited capabilities in terms of configuring new networks.
One possible solution is to reject any traffic from probe’s network to internal networks via iptables.
However, this would require additional steps when redeploying the atlas probe. Ideally, my goal is to deploy the software probe out-of-the-box using a Docker compose manifest, without any additional commands.

So what if network traffic of our software probe container can be proxied to another service, which in turn tunnels it to the public Internet?
In this post we are going to describe how to achieve this, by spinning up a dedicated PPP connection with our ISP in Docker and use it as a “gateway” to our software probe container.

Design

The overall design can be depicted on the following figure

Our goal is to initiate a new PPPoE connection from a container inside an isolated network. Thus our ripe-atlas probe can only perform its measurements only via ppp connection, and without interfering with adjacent containers on the same host.

PPP client

There are two prerequisites so that ppp is established correctly. First of all you need to ensure that the CPE (modem/router) of your ISP has “PPPoE Passthrough” enabled. Some ISPs have this setting enabled by default.
Next, is to retrieve your ISP username and password. Some ISPs (like mine) had handed over the username and password on paper along with my CPE equipment. On other circumstances, you might want to fetch a configuration backup from your CPE and retrieve credentials from it. YMMV. With those in hand we can proceed with configuring ppp client.

Required ppp configuration can be easily generated using pppoeconf tool. It is an ncurses based tool that helps you populate /etc/ppp/peers/dsl-provider and /etc/ppp/chap-secrets with the correct values.
Next step, is to dockerise the ppp client. I created a small Docker image (tzermias/ppp) that it simply invokes ppp client.
Providing that our configuration is under a file named “dsl-provider” and located under /etc/ppp/peers directory, we could invoke the container as follows:

$ docker run --name ppp -d --privileged \
     --device=/dev/ppp \
     - v/etc/ppp:/etc/ppp \
     tzermias/ppp call dsl-provider

ppp client requires access to /dev/ppp device to work properly, so we need to expose it accordingly.
Furthermore, you need to use macvlan network driver for the ppp container to work properly.
A dedicated macvlan network can be created using the command below.

$ docker network create -d macvlan \
     --subnet=172.16.86.0/24 \
     --gateway=172.16.86.1
     -o parent=eth0 ppp

The ppp container can be attached to the network by adding the --network ppp flag on the docker run command above.

If everything went well, logs of ppp container would indicate a successful call is performed and a new IPv4 address is assigned to ppp0 interface of our container.

$ docker logs ppp
Plugin rp-pppoe.so loaded.
PPP session is 394
Connected to d8:67:d9:48:XX:XX via interface eth0
Using interface ppp0
Connect: ppp0 <--> eth0
PAP authentication succeeded
peer from calling number D8:67:D9:48:XX:XX authorized
local  LL address fe80::dead:beef:dead:beed
remote LL address fe80::beef:beef:dead:dead
replacing old default route to eth0 [172.16.86.1]
local  IP address XX.XX.XX.XX
remote IP address YY.YY.YY.YY

ppp client will also create a resolv.conf file under /etc/ppp, containing DNS nameservers obtained from our peer.
For this, you need to have the usepeerdns option in your /etc/ppp/peers/dsl-provider file.

Container networking

Now comes the interesting part. How can we ensure that any network traffic from our ripe-atlas container will only pass through the ppp container we started earlier?
The answer is to share the network stack of this container. Apart from the usual bridge, host or none options that one can pass to --network setting, there a special option named container which allows you to use the network stack of another container.
According to docker run reference:

With the network set to container a container will share the network stack of another container. The other container’s name must be provided in the format of –network container:<name|id>.
…
Example running a Redis container with Redis binding to localhost then running the redis-cli command and connecting to the Redis server over the localhost interface.

With ppp container running, we can easily validate that our setup works, just by spinning up a test container as follows, and perform some tests (e.g a traceroute).
Don’t forget to mount /etc/ppp/resolv.conf file in this container in order to have proper DNS resolution.

$ docker run -it --network ppp -v /etc/ppp/resolv.conf:/etc/resolv.conf:ro busybox:latest traceroute google.com

Now we are ready to spin up our ripe-atlas probe!

RIPE Atlas in Docker

I used ripe-atlas image by jamesits for this purpose. Probe installation and registration are straightforward and well-documented.
On first run, probe generates an SSH keypair and saves it under /var/atlas-probe/etc/probe-key.pub.
Simply create a new directory where any relevant ripe-atlas files will be stored and mount it to the container.

$ mkdir -p atlas-probe

$ docker run --network "container:ppp" \
        --detach --restart=always --log-opt max-size=10m \
	--cpus=1 --memory=64m --memory-reservation=64m \
	--cap-add=SYS_ADMIN --cap-add=NET_RAW --cap-add=CHOWN \
	--mount type=tmpfs,destination=/var/atlasdata,tmpfs-size=64M \
	-v ${PWD}/atlas-probe/etc:/var/atlas-probe/etc \
	-v ${PWD}/atlas-probe/status:/var/atlas-probe/status \
        -v /etc/ppp/resolv.conf:/etc/resolv.conf:ro \
	-e RXTXRPT=yes --name ripe-atlas \
	jamesits/ripe-atlas:latest

After uploading the probe’s public key to RIPE Atlas WebUI, probe is ready to accept new measurements.

IPv6 support

For those who wish the probe run IPv6 measurements, you can enable IPv6 support by performing the following modifications:

Firstly you need to enable IPv6 support in Docker daemon. Ensure that your /etc/docker/daemon.json contains the following:

{
  "ipv6": true,
  "fixed-cidr-v6": "2001:db8:1::/64"
}

Then, ensure that Docker daemon is aware of those settings

$ systemctl restart docker

Next step is to make sure that our ppp container has IPv6 support enabled. For this we need to start our container with net.ipv6.conf.all.disable_ipv6=0 set.

And the last step (eventually) is to enable IPv6 support when performing the PPP call. This can be achieved by adding the following two options in our /etc/ppp/peers/dsl-provider config.

+ipv6
ipv6cp-use-persistent

Putting it together

To sum up, one should execute the following commands to have the probe and the ppp client up and running:

# Create network docker network create -d macvlan --subnet=172.16.86.0/24 --gateway=172.16.86.1 -o parent=eth0 ppp # Create ppp container docker run --name ppp --network ppp --restart unless-stopped\ -d --privileged --device=/dev/ppp \ --sysctl net.ipv6.conf.all.disable_ipv6=0 \ -v /etc/ppp:/etc/ppp \ tzermias/ppp call dsl-provider # Create directory for atlas-probe config mkdir -p atlas-probe # Create ripe-atlas container docker run --network "container:ppp" \ --detach --restart=always --log-opt max-size=10m \ --cpus=1 --memory=64m --memory-reservation=64m \ --cap-add=SYS_ADMIN --cap-add=NET_RAW --cap-add=CHOWN \ --mount type=tmpfs,destination=/var/atlasdata,tmpfs-size=64M \ -v ${PWD}/atlas-probe/etc:/var/atlas-probe/etc \ -v ${PWD}/atlas-probe/status:/var/atlas-probe/status \ -v /etc/ppp/resolv.conf:/etc/resolv.conf:ro \ -e RXTXRPT=yes --name ripe-atlas \ jamesits/ripe-atlas:latest

I also created the following docker-compose.yaml file to spin up the network and the containers to avoid typing those commands again and again.

version: '2.1' networks: ppp: driver: macvlan driver_opts: parent: eth0 ipam: config: - subnet: 172.16.86.0/24 gateway: 172.16.86.1 services: ppp: container_name: ppp image: tzermias/ppp:latest restart: always privileged: true sysctls: - net.ipv6.conf.all.disable_ipv6=0 volumes: - /etc/ppp:/etc/ppp devices: - /dev/ppp:/dev/ppp networks: - ppp command: - "call" - "dsl-provider" ripe-atlas: container_name: ripe-atlas image: jamesits/ripe-atlas:latest restart: unless-stopped volumes: - ${PWD}/atlas-probe/etc:/var/atlas-probe/etc - ${PWD}/atlas-probe/status:/var/atlas-probe/status - /etc/ppp/resolv.conf:/etc/resolv.conf:ro tmpfs: - /var/atlasdata:size=64M environment: - RXTXRPT=yes cap_add: - SYS_ADMIN - NET_RAW - CHOWN network_mode: "service:ppp" depends_on: - ppp

For those who wish to use this probe for their measurements, just use probe #1001487.

Software probes can increase the reach of RIPE Atlas platforms on more networks and geographic locations.
From 260 probes currently deployed in Greece, we can clearly observe a large portion of them being deployed in Athens metro area.
Wouldn’t be nice to have Internet measurements on less densely populated areas, like islands?
Furthermore, the option to share one container’s network stack with another can be leveraged on other applications, like proxying traffic through a VPN tunnel.
If you have a RaspberryPi sitting idle on your network, I’d suggest you give this option a try!

Creating a timelapse

Mon, 27 Apr 2020 00:00:00 +0000

Few days ago, I noticed that you can enjoy a scenic view of the Attica basin as shown from the foothills of Mount
Penteli via a set of web cameras. In particular, National Observatory of Athens/meteo.gr has installed two cameras on the Penteli Astronomical Station premises.

One camera has a view to the south of the basin with Mt. Hymettus on the left, while the other covers the
southwestern part (Mt. Aigaleo and Mt. Parnitha). As with other weather cameras, only still images are provided
that are refreshed at a one-minute interval, rather than a live feed.
Resolution of each image is 2688x1520. Feed for both cameras is available here. To make a step further, meteo.gr is courteous enough a “panorama” that is synthesized from the images of the cameras. Each “panorama” has a resolution of 4190x1394 pixels. Contrary to the actual cameras, panorama is refreshed every two minutes.

So, I thought that this panorama is a prime source of creating a timelapse video of Athens.
Collecting the images is a rudimentary process.
Every two minutes a script is invoked that fetches the panorama image and saves it locally following a predetermined naming scheme:

Then, I set a cronjob to my humble Orange Pi, in order to invoke the script every two minutes.
24 hours and 700 images later, raw material for the video has been collected!

So moving to the next step of creating a video out of these still images. For simplicity, I opt using ffmpeg to render
the whole video. FFmpeg can render a video using a sequence of images by specifying the following
flags; -pattern_type glob -i <IMG_FOLDER>/*.jpg. Also, you need to specify the -r flag to set to a reasonable frame rate. Here, I used 12 fps.
Moreover, FFmpeg provides a variety of audio/video filters that can be applied during video rendering. The complete documentation of FFmpeg filters is available here.
Thus, I used fade and afade filters to add fade-in and fade-out effects on video and audio sources respectively.

Rendering took approximately 2-3 minutes. The final result is shown below:

The glitch at the middle of the image is due to an webcam image being sightly misplaced during the merging of the two sources.
I was astonished with the fact that you can create a complete video just by using two commands; curl and ffmpeg!
Hope this small post will help you creating your own timelapses, either from readily available sources (like
web cameras) or even your own captures!

Revisiting an Ansible role and integrate testing with molecule

Wed, 25 Mar 2020 00:00:00 +0000

Five years ago, I had developed an Ansible role that facilitates the
installation and basic configuration of Plesk for Linux. Plesk is a commercial web hosting control
panel and it is widely adopted in various shared web hosting companies.
According to Plesk documentation, the installation process is straightforward; Download and run an installation script locally, that would guide you through the installation process.
By passing some parameters to the installation script, installation is performed unattended.

The goal of this Ansible role was to perform an unattended installation for pre-determined Plesk version and also perform some post-installation tasks (such as run some initial configuration steps for Plesk or install a Plesk license).
The role, named ansible-plesk is available in Ansible Galaxy, and its source code is available in
Github. I made this role just for experimentation in order to learn Ansible.

In order to test the role, I used to spin a virtual machine (usually CentOS 7) and run the role against this machine.
During its installation, Plesk performs numerous changes on the underlying host (install services such as dovecot, httpd, mysql etc.) hence the only option to ensure that the role works as expected, was using a full-blown virtual machine.
This method was cumbersome and also required some resources.
In particular, the virtual machine should be created, destroyed, recreated across tests.
Not to mention the possibility of using a different OS version (CentOS 6) or even a different linux distribution.
In other words I would seek for a solution to test Ansible roles, where the testing environment would be as close as a virtual machine but without all the maintenance hassle.

Recently, I came across a blog post by Jeff Geerling, with regards to test his Ansible roles with Molecule.
Jeff has developed a variety of Ansible roles available on Ansible Galaxy as well as is the author of the “Ansible for DevOps” book. As I wanted to give molecule a try, I thought that the ansible-plesk role is a good candidate.
Besides, the code had been left untouched for years, so it was a perfect opportunity to perform a refactoring, if needed. Following instructions omit lots of details for brevity.

I started with installing molecule on my laptop using pip

$ pip install molecule molecule[docker]

Then I created the molecule/default directory in the role. molecule directory contains all tests (called “scenarios”) for that role.
Molecule expects that each role has a default scenario, hence the default directory. Core configuration exists under molecule.yml.

---
driver:
  name: docker
platforms:
  - name: instance
    image: geerlingguy/docker-centos7-ansible:latest
    command: ""
    volumes:
      - /sys/fs/cgroup:/sys/fs/cgroup:ro
    privileged: true
    pre_build_image: true
provisioner:
  name: ansible
  lint:
    name: ansible-lint

Molecule can use Docker containers as test platforms. In a nutshell, you can use a pre-built Docker container (e.g a Centos container with systemd) where your role would run. Jeff also maintains some base Docker images tailored for the purpose.
Setting privileged: true is inevitable when our role is performing changes on systemd units (which in our case, it does).

The next step was to create a playbook that invokes the role. Molecule expects a specific playbook called converge.yml to be present.
This playbook is invoked by molecule and is run against the instance defined on molecule.yml.
It could be as simple as the following

---
- name: Converge
  hosts: all
  roles:
    - ansible-plesk

We are now ready to run our Ansible role with molecule. Simply run

molecule test

This would create the testing environment, invoke converge.yml to run the role, re-invoke converge.yml to test whether our role is idempotent or not and finally destroy the testing environment.
testalso performs other tasks as well (such as linting).

By incorporating these changes to ansible-plesk, I managed to successfully migrate my role testing to Molecule. Moreover, I can now test my changes locally with minimal effort.
Incorporating molecule with an existing role was also really simple, if you wish some basic testing to your role.
I even found an issue when the role was running on Ansible 2.9 and pushed the relevant fix to upstream.

All in all, molecule combined with pre-built Docker containers by Jeff Geerling is a compelling option when it comes to test Ansible roles.
I would suggest reading the blog post for a more detailed approach on how to use molecule.

Watching Nagios/Icinga status in tmux status line

Mon, 06 Nov 2017 00:00:00 +0000

For the past few months I use Tmux in a daily basis as my terminal multiplexor. It is a very handy tool that solves limitations of screen as well as provides some new features.
I even integrated a tmux session in Guake and completely disable Guake tabs!

As a systems administrator, it is essential to have a monitoring/alerting system in your infrastructure.
When something happens on one of your systems (i.e. server exhibits an ever-increasing CPU load or Apache stops responding) you should be the first to know.
For this, I use Icinga. Icinga started as a Nagios fork and introduced features like high availability that make it a compelling alternative in large environments in contrast to Nagios.
For visual notifications in desktop, Nagstamon is used that connects to a monitoring system (supports a wide variety of systems) and displays which services need attention (how many are in CRITICAL/WARNING state etc). Here is a typical image on my work desktop:

That means that 10 services are in CRITICAL state and you should take care of them while 173 have set a warning.
You can move this widget anywhere on your desktop. By clicking to it, a screen with detailed information about states is shown.

Despite the fact that Nagstamon is always present on my desktop, I would like this information to be available in my Tmux status line.
Most of my time, I am in a terminal trying to troubleshoot an issue.
The question here, is how to integrate this kind of information in the status line.
Towards this, I made a simple script that checks how many services are in CRITICAL, WARNING or UNKNOWN state and shows it in the status line. Below is the complete code

#!/bin/bash CGI_URL="https://server/icinga/cgi-bin" USERNAME="" PASSWORD="" out=$(curl --silent -u $USERNAME:$PASSWORD "$CGI_URL/status.cgi?jsonoutput&servicestatustypes=28" |jq .status.service_status[].status 2>&1) echo $out | awk 'BEGIN {RS=" ";d=0;c=0;u=0;w=0} /DOWN/ {d++} /CRITICAL/ {c++} /UNKNOWN/ {u++} /WARNING/ {w++} END { if (d>0) printf d" " if (c>0) printf "#[bg=red]"c" #[default]" if (u>0) printf "#[bg=magenta]#[fg=black]"u" #[default]" if (w>0) printf "#[bg=yellow]#[fg=black]"w" #[default]" }'

The only requirement is the presence of jq for JSON parsing. It is just a crude try.
In a nutshell it leverages JSON output provided by the Icinga Classic Web UI. The parameter servicestatustypes=28 in the request, tell Icinga to return services that are in one of the WARNING, UNKNOWN or CRITICAL states.
More info is provided in the documentation here. To use the script, simple change the variables $CGI_URL, $USERNAME and $PASSWORD to match your Icinga monitoring server and you are ready to go.

The complete result is shown on the above image.
Feel free to comment below on how to improve it.

From Wordpress to Jekyll

Wed, 07 Dec 2016 00:00:00 +0000

When I finally decided to start my personal blog back in 2013, I choose Wordpress as its blogging engine. Despite the fact that there is a big hype with Wordpress right now, I decided that its time to move to a more simpler engine.
In this post, I will explain the reasons for moving from Wordpress to Jekyll and describe my migration experience.

Wordpress

Wordpress is one of the top-3 platforms to choose (the other two are Joomla and Drupal) when it comes to develop a new website.
It has a strong community of developers, a multitude of plugins to add functionality to your website (from photo sliders to more sophisticated plugins like Wordfence) and many web developers around the world use it as the basis for their end-products.
In my experience with Wordpress, I wouldn’t say that it suits me pretty well. The WYSIWYG interface always puzzles me and it hinders me from writing. Hence, I prefer a more simpler editor.
Of course, Wordpress has support for Markdown (that I am currently using to write this post), but for a couple of reasons I couldn’t use this option.
Another reason was that Wordpress, like any other CMS, needs regular updates both in its core and on its plugins. Wordpress has done great steps on improving its stability and security on its core. But when it comes to its themes or plugins, things are not the same.
Many users dread to update to a new version of a theme with the fear of breaking the entire site. The same lies to plugins, plus some of them are prone to vulnerabilities (remember the RevSlider vulnerability back in late 2014?).
Along with some minor limitations that my Wordpress installation had (that could be resolved with some effort), I decided I need a simpler engine for my blog.
Hence, I chose Jekyll. Besides, I thought that would be interesting to tackle with something new and I had a hunch that Jekyll would be better for me.

Jekyll

Jekyll, is a static site generator written in Ruby. It uses a simple template to convert posts in text files to a complete site with links.
Jekyll is widely used and it is incorporated in Github Pages as its rendering engine. No database is required, just some files representing your posts.
The basic workflow for writing a post to Jekyll is the following; Open your favorite text editor, write your post (for example in Markdown), generate your entire site using jekyll build and upload the generated content to the document root of your site.
Jekyll functionality can be extended with the use of plugins, so Jekyll can be tailored to your needs.
Jekyll has a great documentation to start with. You can also watch some introductory tutorials and casts in jekyll.tips.

The fact that posts are flat files gives you the opportunity to track changes using a Versioning Control System (VCS), like Git. This eliminates (up to a point) the need of regular backups to your site.
If something happens, you can re-render it and upload it again. Just ensure that VCS repo is replicated, by maintaining a remote copy in e.g GitHub or BitBucket.
Jekyll offers a simple web-server out-of-the-box, to test your changes. Type jekyll serve inside your Jekyll directory and visit the generated site on your browser at http://127.0.0.1:4000.
This test environment is great for testing new layouts, themes without the hassle of configuring a full-blown web server on your PC. You are just a command away to see how your site will look like!
You can write posts without the need to be online (OK, with the ubiquity of Internet connectivity nowadays this isn’t much of a strong argument), not to mention the potential of testing new layouts and writing posts in different branches in Git, without touching the live site.
Another advantage of Jekyll lies to its static nature; A page or a post in Jekyll is generated during build. In contrast, dynamic websites generate a post or a page during visit.
This of course induces a small delay of running the PHP script that connects to the database server, performs the appropriate queries to fetch the post and renders the HTML output during each visit.
Despite the fact that this delay is usually small (and sometimes negligible), it can increased when for instance the query on the database is taking a couple of seconds (either due to load on the database server or due to the fact that the database itself doesn’t have proper indexing on its tables).

Migration process

After reasoning about my choice on Jekyll, its time to perform the actual migration from Wordpress.
Jekyll has a built-in mechanism for the migration from various platforms. You can find more details on the jekyll-import documentation.
But first, lets create a Jekyll instance.

jekyll new myblog

This command will create the basic structure of a Jekyll site inside the directory myblog. If we just want to see it, just type;

jekyll serve

The check the result at http://127.0.0.1:4000. It is just a simple site.
To perform the actual migration from my Wordpress, I used the “Export” option, located in My Site → Settings. For the lazy, just go to http://mysite.com/wp-admin/export.php .
This generates an XML file with the entire content of your site. It is widely used from migrations between Wordpress installations from different providers.
The import of your data to Jekyll, is performed using the following command.

ruby -rubygems -e 'require "jekyll-import";
    JekyllImport::Importers::WordpressDotCom.run({
      "source" => "wordpress.xml",
      "no_fetch_images" => false,
      "assets_folder" => "assets"
    })'

This eventually will parse wordpress.xml file that was exported from Wordpress, download any image contained on posts and save it to assets folder.
You can find more information in the Wordpress section of the documentation.
If no errors are presented, the _posts directory (the directory that contains the actual posts of your site), would have the following structure:

$ ls -l _posts
total 88
-rw-rw-r-- 1 aris aris  402 Dec  7 16:15 2013-11-29-hello-world-2.html
-rw-rw-r-- 1 aris aris 3756 Dec  7 16:15 2014-10-31-busybox-sagem-routers.html
-rw-rw-r-- 1 aris aris 4861 Dec  7 16:15 2016-01-26-migrating-to-qemu.html
-rw-rw-r-- 1 aris aris 3144 Dec  7 16:15 2016-06-28-disk-replacement-notes.html
-rw-rw-r-- 1 aris aris 2694 Dec  7 16:15 2016-07-05-a-simple-reminder-in-ubuntu.html
-rw-rw-r-- 1 aris aris 2114 Dec  7 16:15 2016-07-26-create-a-vm-with-virt-install-with-no-hassle.htm

As simple as that!

In this blog, I used the Jekyll Clean Dark theme from Pavel Makhov and just added my _posts director.
I performed a couple of minor modifications to the posts (e.g replaced <code> tags with Jekyll highlighting), and deployed the site to the server.
The site is currently served entirely from Nginx (hence leveraging HTTP/2) and its main page loads in slightly more that 1.5 seconds. Not so bad!

All things considered, I think that Jekyll suits me better. I can continue to use tools like git, vim etc. rather than visiting a page with an editor.
Nevertheless, I still do consider Wordpress as an excellent blog engine and CMS in general, but somehow it doesn’t suits me. But that’s just me!

Create a VM with virt-install with no hassle

Tue, 26 Jul 2016 09:04:28 +0000

Occasionally, I come to the point where I need to create a Virtual Machine for testing purposes. Either, to check something new (a new version of Plesk or cPanel, for instance) or just create a test environment for my Ansible playbooks. Virtualbox has been removed from my computer completely, in favour of QEMU/KVM. Despite the fact that you can use a GUI application like virt-manager (similar to VirtualBox GUI) to create a VM in KVM, I would prefer a more automated way of creating one.

The only prerequisite that I put on this, is that VM creation and installation must be unattended.
I resort to virt-install, part of virtinst package in Ubuntu. Thus, we end up with the following:

virt-install -n centos-test -r 768 --vcpus=2 --os-variant=rhel7 --accelerate \
-v -w network=default --disk path=/path/to/disk.img,size=4.5 \
-l ftp://ftp.cc.uoc.gr/CentOS/7.2.1511/os/x86_64/ --nographics \
-x "ks=http://192.168.122.1:8000/ks.cfg console=ttyS0"

The above command creates a new Virtual Machine, named "centos-test", uses a CentOS 7 local mirror (here ftp.cc.uoc.gr) as its installation source, creates (if it does not exist) a new disk with size 4.5 GiB and installs CentOS 7 using ks.cfg kickstart file. Note that kickstart is served from a web server in 192.168.122.1, the IP address of the host when you use the default NAT configuration. In case you don't have a webserver in handy, you can start the simplest one, just by typing python -m SimpleHTTPServer in a different console window. A ks.cfg that I use is available on GitHub, here.

A simple reminder in Ubuntu

Tue, 05 Jul 2016 08:41:17 +0000

As far as I know Ubuntu does not have a built-in reminder application. Surely there are some third-party applications that do the job well, but I didn't want to install extra packages etc.
So, I tried to build mine!

This reminder is based on at(1) command, that allows you to run a command at specified time.
In case it has not been installed on your system, just install it via:

sudo apt-get install at

at reads commands from standard input, so it is handy in case where we need an one-liner. For example the above command

echo "firefox" | at now + 1 minute

starts Firefox after one minute. You can find more examples in the following link.

So, we found a way to schedule one-time jobs but how are going to be notified?
notify-send to the rescue. This simple program sends notifications to the user via command line. For example the following

notify-send "Hello World"

shows this notification on the desktop!

Notifications are fully customizable: You can set summary, body or add an image to your notification. Just man notify-send for more information.

So with these two pieces, we can create the following bash function and incorporate it on our .bashrc file:

function notifymeat() { 
  echo 'notify-send -i /path/to/image.png "Impossible Mission robot" ' "\"$2\"" | at $1
}

If we run the following

notifymeat "now +1 minute" "Hello World"

this notification will appear on your screen

The name as well as the image of the pop-up window are from the infamous "Impossible Mission" game in Commodore 64.

Disk replacement notes

Tue, 28 Jun 2016 08:43:19 +0000

Recently, I bought a Samsung 850 Evo 500 GB SSD drive, as an upgrade to my Thinkpad X220. This post is yet another post of installing and aligning SSD partitions in ext4 filesystems. I have followed kargig's post on installing and aligning his SSD. This is simply a bunch of notes I kept during the procedure and not an explanatory post.

Before you start, you must find the Erase Block Size and Page size of the SSD. With a couple of searches in Google, I discovered that Samsung 850 EVO has the following values:
Erase Block Size: 1536kb
Page size: 8kb

I followed the same formula that kargig used to align the partitions. In my case, I used heads=96 and sectors=32

sudo fdisk -H96 -S32 /dev/sdb

The next step is to align the LVM accordingly.
Create a physical volume aligned to the Erase Block Size using the command

pvcreate --dataalignment 1536k /dev/sdb2

Then check the alignment of the physical volume:

pvs /dev/sdb2 -o+pe_start

The output of this command should be something like this

PV VG Fmt Attr PSize PFree 1st PE
/dev/sdb2 thinkpad lvm2 a-- 465,26g 0 1,50m

Note that the 1st PE starts from 1536k(1,50m), which is our goal here. Then proceed to create the VG and LVs you need:

vgcreate thinkpad /dev/sdb2
lvcreate -L 4G -n swap thinkpad
lvcreate -l 100%FREE -n root thinkpad

To align the ext4 partition, you have to set the proper values on stripe-width and stride options. I followed this recipe. In a nutshell:

stride = Page size / Filesystem block
stripe-width = Erase Block / Filesystem block

Filesystem block is 4kB (or 4096 bytes), so stride = 8 kB/4 kB = 2 kB and stripe-width=1536 kB /4 kB = 384 kB.
We create the ext4 partition using this command:

mkfs.ext4 -b 4096 -E stride=2,stripe-width=384 /dev/mapper/thinkpad-root

The last step is to copy data from the HDD to the SSD and change /etc/fstab on the SSD

mkdir -p /mnt/ssd
mount /dev/thinkpad/root /mnt/ssd/
mkdir -p /mnt/ssd/{dev,proc,sys,mnt}
cp -avp /dev/{console,random,urandom,zero} /mnt/ssd/dev/
rsync -aPEHv --exclude=/dev --exclude=/proc --exclude=/sys --exclude=/mnt --exclude=/var/cach/apt/archives/ /mnt/ssd
vi /mnt/ssd/etc/fstab #Update /etc/fstab with the UUID of the root partition.
update-grub

Migrating to QEMU

Tue, 26 Jan 2016 10:22:19 +0000

I use virtualization for various tasks. Either for running photo-editing software in Windows or just run a test CentOS in a safe environment. But for quite a while, for all of my virtualization chores, Virtualbox was the answer when you need something that works really quick. VM management is very easy and you could deploy a VM within seconds. Despite the fact that I use it flawlessly for years, I was tricked into migrating to QEMU. I thought that it is interesting to write down my experience.

The goal was to migrate my existing Windows 7 VM to a QEMU, without compromises in usability and without (extensively) modifying the image. Moreover I wanted to tackle with some KVM features (like memory ballooning). For some people it seems more reasonable to perform a clean Windows installation, rather to struggle migrating an existing one. I chose the hard way, because of fun primarily and I didn't want to reinstall programs from scratch, customize it etc.

Before migrating, make sure you have the required packages for QEMU and KVM:

sudo apt-get install qemu qemu-system-x86_64 qemu-kvm

The first step for the migration is to convert the Virtualbox VDI image into raw format. This can be performed using the following command:

VBoxManage clonehd --format raw /path/to/WindowsVM.vdi
/path/to/WindowsVM.raw

As the disk image is cloned, the free space in your disk should be at least the size of the VM. Let's say the image is 30 GB, you will need at least 30 GB of free space in order the conversion to be successful. Furthermore, cloning takes a considerable amount of time, so be patient.
Upon successful conversion, I tried to run the image with QEMU and no special arguments.

qemu-system-x86_64 -m 1024M /path/to/WindowsVM.raw

The -m 1024 specifies the amount of memory that the VM should have. The result in my case was a BSOD!

Let's try to run it with KVM support

qemu-system-x86_64 --enable-kvm -m 1024M -boot c /path/to/WindowsVM.raw

Here, the VM started successfully, despite the fact that it was a bit slow. Hence, I download the VirtIO drivers ISO and mount it in the VM. Let's start the VM again with virtio network and virtio disk to see what happens.

qemu-system-x86_64 --enable-kvm -m 1024M  -net nic,model=virtio -net user \
-boot c -drive file=/path/to/WindowsVM.raw,if=virtio

This resulted in a BSOD again, as the VirtIO disk driver was not actually installed. The solution was found in Archlinux wiki. You need to make a "dumb" image and boot the VM with that.

dd if=/dev/zero of=dumb.file bs=1M count=100
qemu-system-x86_64 --enable-kvm -m 1024M  -net nic,model=virtio -net user \
-cdrom /path/to/virtio-win-0.1.102.iso -boot c \
-drive file=/path/to/dumb.file,if=virtio /path/to/WindowsVM.raw

As stated in the Archlinux wiki, Windows will detect the dumb image as a new device and will try to find a driver. Go to Control Panel > Device Management, detect the "Unknown device" and Install the VirtIO SCSI driver from the virtio-win iso. Then shutdown the VM, and boot it with virtio support:

qemu-system-x86_64 --enable-kvm -m 1024M  -net nic,model=virtio -net user \
-boot c -drive file=/path/to/WindowsVM.raw,if=virtio

Works flawlessly!

Another feature I would like to tackle with, was memory ballooning. In short, you can change the VM memory on the fly, without rebooting it. More information for this feature is available on the linux-kvm.org page.

qemu-system-x86_64 --enable-kvm -m 1024M  -net nic,model=virtio -net user \
-boot c -drive file=/path/to/WindowsVM.raw,if=virtio -balloon virtio

When run, the Windows VM, will detect the balloon device and will install the respective driver.
You can change the memory size from the QEMU monitor (press Ctrl+Alt+2) using the command "balloon". Then run "info balloon" to list its current status.

Hope this information will help you perform a migration from Virtualbox to QEMU/KVM without a hassle!

Busybox access on Sagem F@ST 2404 routers

Fri, 31 Oct 2014 09:07:26 +0000

I stumbled accross a Sagem F@ST 2404 ADSL modem recently. While I was experimenting with some of its features, I found an interesting feature (or a bug :P) in version 3.21a4

Usually, many home ADSL modems provide command line access (usually via telnet) along with their fancy Web interface.
If you try to connect via telnet to ths ADSL modem, you will be provided with the follwing screen of options:

So you just type an appropriate number and navigate to the respective subcategory.
But what if you try to type a command instead of a number? Let's try with ls /bin first.

Nope. Let's try with a simple echo "This is a test" or something similar.

Bingo! That means that command line interface does not sanitize input properly and allows the user to execute arbitrary commands!
Let's incorporate our command a bit. Let's try to invoke busybox sh

Nope! Let's try to combine the last two commands to a single one and see the result. So our new command would beecho "This is a test"; busybox sh

The busybox shell is spawned! Great!
You can explore you router internals via typical ls and cat commands. You can also view running processes using ps command.
Be careful not to run the cfm executable! I speculate that this is the router's bootstrap script and if you try to re-execute, you will probably lose connectivity. In that case just try a hardware reboot of the router.