Ever since it was first released in August, I had been using Copperhead OS on my Nexus 5. It's a custom Android distribution that integrates PAX and libc hardening. The intention of the developers from the start was to upstream as much of their work as possible and rebase on top of AOSP. Until about a week ago, however, it was based on CyanogenMod 12.1. This was great. I got a generally open-source-friendly firmware with the only free-software superuser implementation, with a bunch of extra security features added.

Then Android 6.0 "Marshmallow" made its debut earlier this fall, introducing several minor features, such as breaking push IMAP synchronization. Apparently it also included enough of the Copperhead work that they decided to go ahead and switch over earlier this month. Unfortunately, this made Copperhead no longer fit my use case. I use root for about three things on my phone: AdAway, a terminal, and DriveDroid. For the first two, I can generally get by with ADB in recovery mode. The third requires an actual app. Yes, all it does is run a couple of commands and write to a file in sysfs, but on a touchscreen, with a touchscreen keyboard, that needs a user interface.

The problem with the new AOSP Copperhead is that it has no superuser access. I could install SuperSU, but as I alluded to earlier, it is important for me to have the program that manages access to, well, everything be free software. SuperSU is not. That means going back to CyanogenMod. Now, CYNGN is not my favorite company, but CyanogenMod is generally the most stable and least "script-kiddie" of the rooted custom firmwares.

Thankfully, CyanogenMod 13.0 had just been released when Copperhead broke. This version was based on Marshmallow, so it presumably included all of the userspace hardening that Copperhead got into AOSP. Since I had needed to put PAX into soft mode for DriveDroid to work on Copperhead anyway, I figured I was not losing much security by switching. Why not just stay on the latest CyanogenMod-based Copperhead version? Because, again, security. Not staying up to date is not really an option.

So I upgraded. Everything appeared to work fine, until I went through the DriveDroid USB setup wizard. For some reason, I couldn't read the partition table from the emulated drive:

Dec 20 20:31:19 sodium kernel: scsi 9:0:0:0: Direct-Access     Linux    File-CD Gadget   0000 PQ: 0 ANSI: 2
Dec 20 20:31:19 sodium kernel: sd 9:0:0:0: Attached scsi generic sg4 type 0
Dec 20 20:31:19 sodium kernel: sd 9:0:0:0: [sdd] 8192 512-byte logical blocks: (4.19 MB/4.00 MiB)
Dec 20 20:31:19 sodium kernel: sd 9:0:0:0: [sdd] Write Protect is off
Dec 20 20:31:19 sodium kernel: sd 9:0:0:0: [sdd] Mode Sense: 0f 00 00 00
Dec 20 20:31:19 sodium kernel: sd 9:0:0:0: [sdd] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
Dec 20 20:31:19 sodium kernel: sd 9:0:0:0: [sdd] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08
Dec 20 20:31:19 sodium kernel: sd 9:0:0:0: [sdd] tag#0 Sense Key : 0x3 [current]
Dec 20 20:31:19 sodium kernel: sd 9:0:0:0: [sdd] tag#0 ASC=0x11 ASCQ=0x0
Dec 20 20:31:19 sodium kernel: sd 9:0:0:0: [sdd] tag#0 CDB: opcode=0x28 28 00 00 00 00 00 00 00 08 00
Dec 20 20:31:19 sodium kernel: blk_update_request: critical medium error, dev sdd, sector 0
Dec 20 20:31:19 sodium kernel: Buffer I/O error on dev sdd, logical block 0, async page read
Dec 20 20:31:19 sodium kernel: sd 9:0:0:0: [sdd] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08
Dec 20 20:31:19 sodium kernel: sd 9:0:0:0: [sdd] tag#0 Sense Key : 0x3 [current]
Dec 20 20:31:19 sodium kernel: sd 9:0:0:0: [sdd] tag#0 ASC=0x11 ASCQ=0x0
Dec 20 20:31:19 sodium kernel: sd 9:0:0:0: [sdd] tag#0 CDB: opcode=0x28 28 00 00 00 00 00 00 00 08 00
Dec 20 20:31:19 sodium kernel: blk_update_request: critical medium error, dev sdd, sector 0
Dec 20 20:31:19 sodium kernel: Buffer I/O error on dev sdd, logical block 0, async page read
Dec 20 20:31:19 sodium kernel:  sdd: unable to read partition table
Dec 20 20:31:19 sodium kernel: sd 9:0:0:0: [sdd] Attached SCSI removable disk

I attributed it to something about the image file in use, and carried on. But when I copied over a UEFI update ISO for my laptop and it also was unreadable, I realized the problem was the interaction of DriveDroid and something in CyanogenMod 13. To the dmesg!

[  247.848373] type=1400 audit(1450744742.272:6): avc: denied { use } for pid=91 comm="file-storage" path="/storage/emulated/0/Download/ISO/gjuj23us.iso" dev="fuse" ino=167 scontext=u:r:kernel:s0 tcontext=u:r:sudaemon:s0 tclass=fd permissive=0
[  247.870718] type=1400 audit(1450744742.282:7): avc: denied { use } for pid=91 comm="file-storage" path="/storage/emulated/0/Download/ISO/gjuj23us.iso" dev="fuse" ino=167 scontext=u:r:kernel:s0 tcontext=u:r:sudaemon:s0 tclass=fd permissive=0
[  247.871447] type=1400 audit(1450744742.292:8): avc: denied { use } for pid=91 comm="file-storage" path="/storage/emulated/0/Download/ISO/gjuj23us.iso" dev="fuse" ino=167 scontext=u:r:kernel:s0 tcontext=u:r:sudaemon:s0 tclass=fd permissive=0
[  247.881682] type=1400 audit(1450744742.302:9): avc: denied { use } for pid=91 comm="file-storage" path="/storage/emulated/0/Download/ISO/gjuj23us.iso" dev="fuse" ino=167 scontext=u:r:kernel:s0 tcontext=u:r:sudaemon:s0 tclass=fd permissive=0

That'll do it. So SELinux is blocking the kernel from accessing the file. I opened a root shell in ADB, ran setenforce 0, and sure enough: I could boot off the ISO.

Now that I know the general problem, how can I fix it? I could leave my device in permissive mode, but that would be poor for overall security. What I need to do is create a policy that allows this specific action, and graft it into the existing policy. The common solution is to use a tool that comes with SuperSU, supolicy, but since I am using CyanogenMod, I don't have it. And since it is closed-source, I am not interested in installing it. (Plus, it adds a bunch of other policies when run, and is not compatible with the policies used to implement CyanogenMod's superuser support.)

After some searching, I found the free "alternative supolicy", sepolicy-inject. (See also here.) Unfortunately, it does not work with Android 6.0. It dies trying to read the /sepolicy I pulled from my phone:

libsepol.policydb_read: policydb version 30 does not match my version range 15-29

Apparently, Android uses a newer SELinux policy version than available in any released version of libsepol. So you have to get this version with prebuilt libraries. It works, on the desktop, but using the prebuilts makes it hard to cross-compile for native use. In any case, I added the appropriate policy rules:

$ ./sepolicy-inject -s kernel -t sudaemon -c fd -p use -P ./sepolicy -o ./fixed

Then I pushed the file on to my phone and and loaded the policy from a shell.

# load_policy ./fixed

Hooray! It works! Now I just have to get that policy loaded on boot, so I don't have to run the command every time I reboot my phone. Android has a method for loading SELinux policy from the /data partition, meant for OEMs to push policy OTA updates. To load the policy via an intent, it has to be signed, but you can just push the files in the right place with ADB and they will be loaded. So I put my fixed sepolicy file in /data/security/current, rebooted, and... it wasn't loaded. It turns out you have to put a selinux_version file in that directory, to avoid loading policy designed for an older system image.

# cat /selinux_version > /data/security/current/selinux_version
# restorecon /data/security/current/selinux_version

I copy that file into the right place, reboot, and... everything crashes. The RIL (radio interface layer) can't access the radio, so everything relating to cellular connectivity force closes. It's not obvious from the SEAndroid page I referenced above, but all of the files used for specifying security policy are loaded from /data/security when selinux_version exists.

12-21 23:13:55.842  2381  2381 W SELinuxMMAC: java.io.FileNotFoundException: /data/security/current/mac_permissions.xml: open failed: ENOENT (No such file or directory)
12-21 23:13:55.674  2381  2381 E SELinuxMMAC: java.io.FileNotFoundException: /data/security/current/seapp_contexts: open failed: ENOENT (No such file or directory)

Some of those other files are used to put certain apk packages in a more priveleged context. Therefore, you have to copy the rest from the initramfs to /data/security/current as well (they do not need any modifications).

# umask 022
# for f in /file_contexts /seapp_contexts /property_contexts /service_contexts /selinux_version /mac_permissions.xml
# do
#   cp $f /data/security/current$f
#   restorecon /data/security/current$f
# done
# chown -R system.system /data/security/current

I reboot again, and cellular service works again. That's good. But the replacement policy is not loaded. Surely now we are getting into "this is a bug" territory (from "this is an unintended use case"). Looking in the dmesg again:

[   62.824360] init: SELinux:  Could not load policy:  Permission denied
[   62.824755] type=1400 audit(66564380.419:5): avc: denied { load_policy } for pid=1 comm="init" scontext=u:r:init:s0 tcontext=u:object_r:kernel:s0 tclass=security permissive=0

This legitimately makes no sense made no sense at the time of my investigation. Apparently, CyanogenMod's GitHub was outdated (yes, I was looking at the right branch), because this commit was not merged and pushed. But that explains things now. So Google removed this support in Marshmallow because nobody used it. Well, I used it. It actually worked, albeit intermittently. It would work once after running load_policy on a file in /data, but never after running load_policy /sepolicy. This explains why it did not work when run during boot. (If you're wondering how I can even load policy with that neverallow rule being there, CyanogenMod puts superuser access in permissive mode.)

It seems, then, the only real solution is to compile my own CyanogenMod builds, with that rule added in at the source level. I do a bit of customization of my system image with an addon.d script anyway. I was trying to avoid that, as it increases the lag time for getting security updates and other bugfixes. Security is hard.

It is often said that any data you do not have backed up is data you do not care about. I agree. This adage is used to encourage people to start keeping copies of their data. However, simply making copies is not enough to make data safe. Those copies must be invulnerable to whatever catastrophe you are trying to protect your data from. Thus, when choosing a method of backing up your data, you must take into account your threat model: that is, what could go wrong, and how will your data survive it?

For example, if you are only worried about hard drive failure, RAID1 is "backup" enough. If you just want to recover from accidental file deletion or an application corrupting your files (bugs, malware, etc.), then filesystem snapshots are a reasonable solution. On the other hand, if your family photos need to survive a house fire, you need some sort of offsite backup, like giving a hard disk to a friend, or uploading to a cloud service.

My main backup solution is to synchronize my home directory between my laptop and my home server with unison. This works great for protecting against several failure modes: accidental deletion, filesystem corruption, disk failure, stolen laptop, and natural disasters at my house. I consider this to be effectively an offsite backup because my laptop is rarely in the same building as the server. My workstation mounts its home directory with Samba, so it has nothing to back up. Operating system configuration is version controlled and stored in my home directory (I currently use Ansible). The only thing left is data on my server outside my home directory, like my web site.

I use encryption on all of my systems for various reasons. For my laptop, it is to prevent data access in case it is lost or stolen (physical security is especially hard with small self-contained machines you carry around with you). For my workstation, it is because I am not the only person with a key to my abode, so I cannot guarantee that nobody else has gained physical access to my machine. For my server, it is so I can RMA a failed disk without worrying about its contents. On all of my systems, and additional reason is to prevent someone booting a live system and installing a rootkit or keylogger. Therefore, I must use full-disk encryption and not simply use PGP with individual files.

Since I use encryption on the main copy of my data, to maintain its security I must also encrypt my backups. However, it would be a good idea to encrypt offsite backups even if my main storage was not encrypted. Data at rest is kept confidential in one of two ways: by ensuring the physical security of the computer it is stored on or by encryption. This first method is generally not possible when it comes to offsite backups (unless you, e.g., work for a large company with multiple datacenters), so that leaves you with only encryption.

Unfortunately, encrypted offsite backups present a few problems:

1. They are long-lived due to the effort required to make them (slow uploads for cloud services, or having to carry a physical disk to a secure location)
2. They are rarely used, since all but the most catastrophic failures can be recovered from locally
3. They must be encrypted, since you cannot guarantee physical security
4. The password must be unusually strong, since any attacker will have a long time to mount a brute-force attack (because, from point 3, they could just image your disk, which they will have in possesion for a while due to point 1, and you won't notice because of point 2) The combination of points 2 and 4 above mean you have to create a strong password that you will rarely use. This sounds like the perfect recipe for a password you will forget.

And here we get to the personal application--the motivation for the story. I had a blog. It had several posts. In a way, I still have the blog. It is stored on a laptop hard disk in my desk. However, that disk is encrypted with dm-crypt and a rather long passphrase. It is an offsite backup made August 2014. I have tried brute forcing anything I thought the password might be, but I have not been successful. You see, at one point, I was shuffling data around on my disks and ran out of space. I had the idea that "well, everything is backed up anyway, so I can just delete it now and restore it later when I need it." Then I forgot for a year, and by the time I needed the data again, I could not access it. So all of my previous website content is lost, probably forever, to the mathematical void. I keep the disk in hope that one day I will remember the passphrase, but I realize my chances are slim.

What did I learn from this? That an untested backup is no backup at all! Test your backups!