Exciting! Initial backup complete. The associated B2 bucket now has about 170 GB uploaded in about 346,000 files. I expect the file count to drop, because I added additional path exceptions after observing it uploading things I don't really need backed up – but the size probably won't significantly drop, because the new exceptions contain mostly tiny files.
The code is really maturing. It's developing all those tiny wrinkles that only come from actually running it and discovering the tricky and unexpected combinations of things that can happen.
- The filesystem enumerator now artifically puts /home mounts first, because you want your home directory to be one of the first things backed up.
- Added a work-around for a bug in a bunch of B2 APIs. Basically, they converted parts of their API to use HTTP GET requests instead of HTTP POST to be more idiomatic (you're "getting" data, so it should be HTTP GET – which isn't wrong, per se), and in the process they accidentally introduced additional filename format limitations into a bunch of their endpoints. So now you can e.g. upload a file whose filename contains a comma, a square bracket or an ampersand, but when you try to download that file, the download endpoint rejects the filename. Sigh. 😛
- When you enumerate ZFS mounts, at least the way it's configured on recent Ubuntu versions, you find volumes like rpool/USERDATA/logiclrd_ltzio4 for /home/logiclrd and bpool/BOOT/ubuntu_znaqup for /boot. But, the way I'm gathering the mount info somehow also sees rpool and bpool devices mounted to the same mount points. This causes problems, because you can create a snapshot on these and it isn't a snapshot on the actual volume. So, the code now ignores ZFS devices that don't contain slashes. Seems right. 🙂
- There was a bug in the code that tested whether the Backup Agent was busy, to determine whether an Initial Backup operation has completed. But, I only just now completed an Initial Backup operation for the bug to manifest. The bug was that I wrote "queue size >= 0" instead of "queue size > 0" – basically, "it's busy if the queue size is greater than 0 or if the queue size is 0", when a queue size of 0 means it's not busy!
- Fixed a performance issue that arises with this sequence of batch uploads in a thing called the Remote File State Cache. The state files start to get quite large after a while, such that the background thread that uploads them can't keep up with their creation during Initial Backup. The result is a logged series of actions that uploads file #1, then deletes file #1 and uploads file #2, then deletes file #2 and uploads file #3. This is a simplification, but the essence is there: files get uploaded, and then pretty much immediately get replaced by a newer file. This presented an opportunity for optimization: If the file is going to be deleted anyway, don't bother uploading it!
- I've just added a proper error logging infrastructure, intended to be only used for logging important exceptions that might have some bearing on consistency or completeness.
- If the filesystem monitor detects a Move event, and the "from" path isn't being tracked, that's actually not the end of the world. Previously, it would just bomb out of the Process Backup Queue thread (and probably take the whole process down in the process), but now it just says, "Okay, well, let's just queue the "to" path as though it is a newly-created file." 🙂
- Watching it run in filesystem monitoring mode, there is pretty much a continuous stream of activity from Firefox in its .mozilla and .cache folders. This is categorically content that does not need to be backed up, so I've added a filter mechanism to ignore paths that have certain components in them.
My laptop running Ubuntu 24.04 now has real-time cloud backup!
Line count: 12,535
No comments:
Post a Comment