At Green Coding Solutions (GCS), one goal is to enable reproducible runs on our cluster. An important step towards accurate measurements was the creation of NOP Linux, our custom Linux distro that disables as many background processes as possible to avoid interruptions during measurements. Another crucial step was ensuring the reliable operation of the PowerSpy2, so we could measure the entire power consumption.

We wanted to create a cluster that allowed users to select the server on which they’d like to run the benchmark. Initially, we aimed for full automation and looked at the excellent tool from Canonical, MAAS. As we use Ubuntu as our reference system, this seemed to be the logical choice. Although the tool was impressive, it required a daemon running on the machine, which created multiple interruptions during our measurements. This led us to reevaluate our tooling, and we decided to try a simpler approach using PXE. While there is a great description [1], and the general flow worked very well, we invested a significant amount of time and effort in configuring the machines correctly. Getting the entire installation flow working with reboots, different configurations like PowerSpy, and the multitude of different servers we wanted to use presented a considerable overhead. Additionally, we have our machines distributed across various data centers, and we needed to set up a complex networking layer for the DHCP discovery to work. While this was a scalable solution, it required substantial overhead that had to be maintained. Moreover, our tool develops quite rapidly, so we would have to keep updating the installation process. As a small company, this was not feasible in our scenario. Consequently, we decided to sacrifice scalability in favor of simplicity. In the meantime, we had built a complex test setup with various servers and a complicated setup that we could now disassemble. The main lesson learned for the future is to start with the simplest solution that solves the problem and continually reevaluate your assumptions and needs.

We are aware that there are a multitude of configuration systems out there that don’t require a client running on the machine to be configured and that automate some of the tasks we will now do manually. But we decided to keep it very simple for now and not invest more time into another solution.

At Green Coding Solutions, we are committed to not only creating efficient and reproducible programming solutions but also sharing our findings and tools with the wider community. We firmly believe in the principles of open-source, the power of shared knowledge, and the benefits of collaborative development. Our aim is to create tools and systems that can be utilized by anyone, without the restrictions of proprietary licenses. We don’t just want to make our solutions better - we want to make programming better, for everyone. One of the exciting initiatives that align with our philosophy is the Blue Angel for Software. We support this cause and believe that our tools and systems should be made available for such uses. By making our developments publicly available, we hope to contribute to the broader objective of creating software that is efficient, effective, and transparent.

The system we are using now

As previously mentioned, the current system will not scale to accommodate thousands of machines, but it will suffice for a considerable amount of time in our situation.

The files shown in this article might already be outdated when you read it as we will not update the article! For a detailed discussion please check out our documentation under https://docs.green-coding.io/

We have now opted for quite a simple solution. You will need a server that exposes the database externally and all results will be written to this server. We then have a client.py script that runs on every server that periodically queries the server for jobs and if so executes the measurement undisturbed. After a job is finished the client does some cleanup tasks and checks if there is an update for the GMT and also for the operating system. It then retries to get a job till there are no more jobs left on which the client sleeps for 5 minutes and retries. On every wake up we send a message to the server that the client is up and functional. So we can check server side that all clients are up and working.

To create your own GCS cluster, you can follow these steps:

1) Install Ubuntu

These configurations are testes with Ubuntu 22.04 LTS, but newer versions should also work. Older versions are discouraged.

Use this cloud config file to install your client machine:

#cloud-config
autoinstall:
  apt:
    disable_components: []
    geoip: true
    preserve_sources_list: false
    primary:
    - arches:
      - amd64
      - i386
      uri: http://de.archive.ubuntu.com/ubuntu
    - arches:
      - default
      uri: http://ports.ubuntu.com/ubuntu-ports
  drivers:
    install: false
  identity:
    hostname: gc
    password: $6$exDY1mhS4KUYCE/2$zmn9ToZwTKLhCw.b4/b.ZRTIZM30JZ4QrOQ2aOXJ8yk96xpcCof0kxKwuX1kqLG/ygbJ1f8wxED22bTL4F46P0
    username: ubuntu
EOF
touch meta-data
    realname: gc
    username: gc
  kernel:
    package: linux-generic
  keyboard:
    layout: us
    toggle: null
    variant: ''
  locale: en_US.UTF-8
  network:
    ethernets:
      enp1s0:
        dhcp4: true
    version: 2
  source:
    id: ubuntu-server-minimal
    search_drivers: true
  ssh:
    allow-pw: true
    authorized-keys: []
    install-server: true
  storage:
    layout:
      name: direct
    match:
     ssd: yes
  updates: security
  version: 1
  shutdown: poweroff

You can use the descriptions how to create a custom iso Using another volume to provide the autoinstall config . We then put the iso on a usb stick and boot the machine by hand. As we have physical access this is ok for now. In this example the password is ubuntu. Obviously change this in your case. Once the install has finished you can pull the usb stick and reboot.

2) Install NOP Linux

You can now ssh into the machine and start configuring. This is mainly done by copy pasting scripts manually. As we don’t install machines that often this is totally ok for now. Please note that most of these commands need to be run as root.

#!/bin/bash
set -euox pipefail

# this is a patch. Firefox seems to have a trick to remove read-only filesystem. We need to unmount that first
sudo umount /var/snap/firefox/common/host-hunspell || true

# remov all snaps first as they mount read only filesystem that only snap itself can find and unmount
for i in {1..3}; do # we do this three times as packages depends on one another
    for snap_pkg in $(snap list | awk 'NR>1 {print $1}'); do sudo snap remove --purge "$snap_pkg"; done
done


# Remove all the packages we don't need
sudo apt purge -y --purge snapd cloud-guest-utils cloud-init apport apport-symptoms cryptsetup cryptsetup-bin cryptsetup-initramfs curl gdisk lxd-installer mdadm open-iscsi snapd squashfs-tools ssh-import-id wget xauth unattended-upgrades update-notifier-common python3-update-manager unattended-upgrades needrestart command-not-found cron lxd-agent-loader modemmanager motd-news-config pastebinit packagekit
sudo systemctl daemon-reload
sudo apt autoremove -y --purge

# Get newest versions of everything
sudo apt update

sudo apt install psmisc -y

# on some versions killall might be missing. Please insta
sudo killall unattended-upgrade-shutdown

sudo apt upgrade -y

# These are packages that are installed through the update
sudo apt remove -y --purge networkd-dispatcher multipath-tools

sudo apt autoremove -y --purge

# These are user running services
systemctl --user disable --now snap.firmware-updater.firmware-notifier.timer
systemctl --user disable --now launchpadlib-cache-clean.timer
systemctl --user disable --now snap.snapd-desktop-integration.snapd-desktop-integration.service


# Disable services that might do things
sudo systemctl disable --now apt-daily-upgrade.timer
sudo systemctl disable --now apt-daily.timer
sudo systemctl disable --now dpkg-db-backup.timer
sudo systemctl disable --now e2scrub_all.timer
sudo systemctl disable --now fstrim.timer
sudo systemctl disable --now motd-news.timer
sudo systemctl disable --now e2scrub_reap.service
sudo systemctl disable --now tinyproxy.service
sudo systemctl disable --now  anacron.timer


# these following timers might be missing on newer ubuntus
sudo systemctl disable --now systemd-tmpfiles-clean.timer
sudo systemctl disable --now fwupd-refresh.timer
sudo systemctl disable --now logrotate.timer
sudo systemctl disable --now ua-timer.timer
sudo systemctl disable --now man-db.timer

sudo systemctl disable --now sysstat-collect.timer
sudo systemctl disable --now sysstat-summary.timer

sudo systemctl disable --now systemd-journal-flush.service
sudo systemctl disable --now systemd-timesyncd.service

sudo systemctl disable --now systemd-fsckd.socket
sudo systemctl disable --now systemd-initctl.socket

sudo systemctl disable --now cryptsetup.target

sudo systemctl disable --now power-profiles-daemon.service
sudo systemctl disable --now thermald.service
sudo systemctl disable --now anacron.service



# Packages to install for editing and later bluetooth. some of us prefer nano, some vim :)
sudo apt install -y vim nano bluez

# Setup networking
NET_NAME=$(sudo networkctl list "en*" --no-legend | cut -f 4 -d " ")
cat <<EOT | sudo tee /etc/systemd/network/en.network
[Match]
Name=$NET_NAME

[Network]
DHCP=ipv4
EOT

# Disable the kernel watchdogs
echo 0 | sudo tee /proc/sys/kernel/soft_watchdog
echo 0 | sudo tee /proc/sys/kernel/nmi_watchdog
echo 0 | sudo tee /proc/sys/kernel/watchdog
echo 0 | sudo tee /proc/sys/kernel/watchdog_thresh

# Removes the large header when logging in
sudo rm /etc/update-motd.d/*

# Remove all cron files. Cron shouldn't be running anyway but just to be safe
rm -R /etc/cron*

sudo apt autoremove -y --purge

# Desktop systems have NetworkManager. Here we want to disable the periodic check to Host: connectivity-check.ubuntu.com.
if [ -f "/etc/NetworkManager/NetworkManager.conf" ]; then
    echo "[connectivity]" >> /etc/NetworkManager/NetworkManager.conf
    echo "uri=" >> /etc/NetworkManager/NetworkManager.conf
    echo "interval=0" >> /etc/NetworkManager/NetworkManager.conf
else
    echo "NetworkManager configuration file seems not to exist. Probably non desktop system"
fi

# List all timers and services to validate we have nothing left
sudo systemctl list timers
systemctl --user list-timers

echo "All done. Please reboot system!"

Now you should have a machine that only runs a minimal amount of services and hence should not create a significant amount of interrupts that disturb measurements. We can measure this by starting NOP Linux in an virtual machine and checking CPU statistics.

3) Install the Green Coding Tool

Now we need the tooling installed on the client to start the measurements.

#!/bin/bash
set -euox pipefail

apt update
apt install -y make gcc python3 python3-pip libpq-dev uidmap git iproute2
apt remove -y docker docker-engine docker.io containerd runc
apt install -y ca-certificates curl gnupg lsb-release

su gc << 'EOF'
git clone https://github.com/green-coding-solutions/green-metrics-tool ~/green-metrics-tool
cd ~/green-metrics-tool
git submodule update --init
python3 -m pip install -r ~/green-metrics-tool/requirements.txt
python3 -m pip install -r ~/green-metrics-tool/metric_providers/psu/energy/ac/xgboost/machine/model/requirements.txt
EOF

mkdir -p /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg

 echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

apt update

apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
systemctl disable --now docker.service docker.socket
apt install -y docker-ce-rootless-extras dbus-user-session

shutdown -r now

#
# You need to reboot here
#

systemctl disable --now docker.service docker.socket

su - gc -c "dockerd-rootless-setuptool.sh install"

cat <<EOT >> /home/gc/.bashrc
docker context use rootless
EOT

su - gc -c 'systemctl --user enable docker; loginctl enable-linger $(whoami)'

apt install -y lm-sensors libsensors-dev libglib2.0-0 libglib2.0-dev
sensors-detect --auto

# You only need these commands if you are planning to use the PowerSpy2.
pip install pyserial==3.5
apt install -y bluez

This might also change, please refer to the GMT Documentation.

If you want to use the PowerSpy2 device please follow the installation under https://docs.green-coding.io/docs/measuring/metric-providers/psu-energy-ac-powerspy2/

4) Configure the GMT

Now that you installed the GMT you need to configure it to run in client mode. You can run the install script with the following parameters to give you the first version of the config file. You will need to change the api/ metrics endpoint to the url of your server.

./install_linux.sh

It is important that you don’t run the GMT server or database on the same machine as you are doing your benchmarks on, as this will create additional load and falsify your measurements.

Now please also edit the following points in the config.yml:

• modify the postgresql section so that the host points to your server. You will need to replace the green-coding-postgres-container value. This should be the same url as you specified when running the install script. Check that the password is correct.
• Enable and configure the metric providers that make sense for your client. See the GMT documentation for more details
• Set the machine_id to the number you gave the client when adding it to the machines table on the server.

In this setup we only have one machine configured to send emails, the server. You can add email sending capabilities to any client if you want by adding the smtp data in the configuration. Don’t forget to also set the admin values (email and no_emails=False) at the end of the file. Also you will need to set up a cron job for this. Please see the GMT documentation for details.

5) Add the cleanup script to the sudoers file

chmod a+x /home/gc/green-metrics-tool/tools/cluster/cleanup.sh
echo "ALL ALL=(ALL) NOPASSWD:/home/gc/green-metrics-tool/tools/cluster/cleanup.sh" | sudo tee /etc/sudoers.d/green_coding_cleanup

6) Start the client service

To make sure that the client is always running you can create a service that will start at boot and keep running.

Create a file under: /etc/systemd/system/green-coding-client-service.service with following content

[Unit]
Description=The Green Metrics Client Service
After=network.target

[Service]
Type=simple
User=gc
Group=gc
WorkingDirectory=/home/gc/green-metrics-tool/
ExecStart=/usr/bin/python3 /home/gc/green-metrics-tool/tools/client.py
Restart=always
RestartSec=30s

[Install]
WantedBy=multi-user.target

• Reload the systemd configuration by running:

sudo systemctl daemon-reload

• Start your new service:

sudo systemctl start green-coding-client-service

• Enable the service to run at boot:

sudo systemctl enable green-coding-client-service

• To check the status of your service, run:

sudo systemctl status green-coding-client-service

You should now see the client reporting it’s status on the server.

[1] Setup PXE Boot Server using cloud-init for Ubuntu 20.04