Computers Sure Have Come A Long Way

It's amazing how far computers have come :-) My first computer was a 486 DX2 with a 100 MB hard drive. That's no typo, it was measured in megabytes, not gigabytes! It, along with the CD-ROM drive we got later on, was connected to the computer via an IDE bus. This bus used a ribbon cable about 3 inches wide with 80 separate tiny wires running through it. If you needed to reconfigure the system, there were no two ways about it -- you had to completely shut the system down and disconnect the power before you started changing what wires were plugged into what. It seems awkward, but the reason the name is "IDE" -- "Integrated Drive Electronics" -- is that in even earlier systems, hard drives _didn't know how to access themselves_. You needed another specialized controller in between the computer and the disk for it to even work! But I digress. Today, I had a minor annoyance with a server. That server provides Internet for the house I grew up in. It is a _virtualized_ server, which means that the "computer" it's running on doesn't even exist, not physically. It's just a program running on _another_ computer. That other computer ran into a problem: it ran out of hard drive space. The hard drive in it was only 120,000 MB, and the virtual hard drive for the actual server just kept growing for whatever reason. (I say that because the actual filesystem inside of it is only about 6 GB, so why is the virtual disk over 87 GB? :-P) So, I needed to upgrade it to a bigger hard drive. In the "good ol' days", that would have meant shutting the computer down, installing the second hard drive (which probably would have meant disconnecting something else to free up a slot, because you could only connect two IDE devices at a time), then starting the system up in a special mode running a program to copy data from one disk to another. That process would take hours to complete, after which you'd then shut it down again, reorganize the disks, plug that optical drive back in or what have you, and then cross your fingers that the boot sector copied over properly and is still properly configured for the new drive, otherwise you have a whole new category of problems to solve! I was able to do it without shutting the computer down at all! The first thing that made this possible is the way hard drives connect to computers now. Back in the day when home users were struggling with IDE hard drives, fancy high-end systems used a different technology to connect called SCSI ("Small Computer System Interface"). With SCSI, you could connect up to 8 things to the same cable, but one of those things was the computer, so you could only connect 7 hard drives. Except you could do fancy tricks, like for instance you could connect 6 hard drives and 2 computers, and the computers could both access the hard drives at the same time. And, you could connect new devices at runtime, without having to shut everything down. If you needed to disconnect something, you'd need to make sure it was fully "released" by the software first, but that was also possible. Well, SCSI eventually fell by the wayside, because its cables, like IDE cables, are these huge fat wide ribbon cables with, in the case of SCSI, 68 separate tiny wires in them, twisted into pairs. It poses some limitations, and as electronics really properly matured, it became possible to blast data down a single wire really really fast. You couldn't send 16 bits in one go like with a SCSI cable, but you could send 16 bits one at a time in less time overall! This led to a new standard called SAS -- Serially-Attached SCSI. With SAS, the devices were all still SCSI devices, but the cables were much thinner. Like SCSI, SAS was made for higher-end situations, and full SAS support makes a device more complex and more expensive. But, us little guys struggling with IDE got an upgrade as well. A subset of the SAS standard called SATA -- "Serial ATA" (ATA itself is a very boring acronym :-) ) -- was created. It doesn't do all of the fancy tricks SAS can do, but it does do some of them. One of those things is that, like with the full SCSI standard, devices can be attached/detached at runtime. It's built into the standard. To support SATA, a system must give at least token support to it. Today's computers are pretty good at this. :-) The second thing that made this possible is a thing that is very commonly done to decrease the chance of losing data due to a hard drive failure. That thing is ... store the data twice, on two hard drives! This sounds almost banal and silly, but is literally a thing. It's a concept that has been given the fancy name "RAID" -- Redundant Array of Inexpensive Disks. RAID has different "levels" to it, and RAID level 1 is just straight-up mirroring. I use that pretty much everywhere, and it has saved my bacon multiple times. Back in the day, if you wanted to use RAID, you needed a specialized add-on circuit board between the computer and the drives. You _could_ just plug two hard drives in and then use the operating system to interact with multiple drives, but it was horrendously slow doing it that way. With IDE, each cable could connect two devices, and you'd get two cables, one per "channel". Each channel could only talk to one of the drives at a time. In addition, cheaply-made consumer controllers could only operate one channel at a time. So if you had two hard drives connected to the system and you wanted to mirror the data, so that when you hit Save, that file gets written twice, once to each drive, it now takes twice as long to save the file. These days, though, the SATA cables that connect the hard drives can transfer data considerably faster than any hard drive can actually process it, and even cheap controllers can send data down more than one cable at the same time anyway. The bottleneck is gone. So, you don't need fancy hardware, you just need the right software, and all modern operating systems come with that software. You just tell the operating system, "Hey, you see those two disks? Please pretend they're one disk, and whenever I save a file, just do exactly the same write on both of them at the same time." (There are fancier RAID schemes as well, but this one is the simplest and easiest to maintain.) So, back to my server. My server is actually two servers, because there's the server that's running on the hardware, and then there's the server running inside a program running on the first server (which is where the actual important work is taking place :-) ). The "outer" computer, which people have decided should be called a "hypervisor" because it manages virtual computers running inside of it, is running Slackware Linux. It is using a feature of the Linux operating system called "md" to mirror its (real, physical) hard drives. Anything that is written to disk is written to two disks. If/when a disk fails, it can be replaced without losing anything, and in some cases without even shutting the server off! The process of replacing the disks with larger disks is actually modelled as pretending that a hard disk has failed and needs to be replaced. Following instructions I found online, I ran a health check just to make sure, and then I told md that one of the drives had failed. (It hadn't really.) MD then took that disk out of rotation, and with another command, I was able to tell Linux to actually fully detach from the disk, so that the operating system no longer considered it to even exist any more. At that point, then, without shutting the computer off, I was able to unplug the first of the two disks from the computer, which continued running just fine on the second disk (just without redundancy at this point). As soon as I plugged the second disk in, the system immediately detected that it was connected (because connecting new devices at runtime is a core feature of SATA), and then I was able to tell md, "Hey, here's your replacement disk". MD brought it into the array and proceeded to copy the entirety of the remaining original drive onto it. This is another area where computers have advanced so insanely far. Suppose I had had to copy that old 486 computer's 100 MB hard drive to a replacement disk. That copy process would have taken about 15 minutes to complete. As computers progressed into the gigabytes of storage, the time required for shuffling data around increased correspondingly. A few years later, if you set up a computer with a 1 GB IDE drive in 1995, well, the IDE standard was still the same standard, it wasn't any faster, and duplicating that could now take something like 3 hours to complete! Fast-forward to today, replacing 120 GB hard drives with 512 GB hard drives. When I told md that it had a second disk again, and it needed to duplicate 120 GB of data from the remaining original disk onto the new disk, that process took ... a bit under 10 minutes. :-D So to summarize, with off the shelf consumer grade components and a completely free, community-made operating system, today's computer technology allowed me to: - Trivially use RAID-1 mirroring without any hardware support needed - Hotswap the hard disks (that is, plug/unplug them while the computer is powered on) - Remove and add RAID member devices at runtime (without having to shut down the operating system) - Regenerate a 120 GB RAID-1 array in 10 minutes And the virtual server, the one doing the actual work? It never even knew anything took place! It just kept chugging along the entire time blissfully unaware that the foundations of its existence were being shuffled about.