Friday, 26 July 2013

Short Story: "Diary of a Short but Enormously Influential Life"

Back in January of this year, I wrote a short story. I never did get around to publishing it. Until I figure out something better, this seems like as good a place as any to put it :‑)

DIARY OF A SHORT BUT ENORMOUSLY INFLUENTIAL LIFE

Jonathan Gilbert
Copyright © 2013
That's my name too.
These were my first coherent thoughts. I didn't know it at the time. I thought I was 34 years old, working on my doctoral thesis on computational neuromechanics at MIT. I had memories of having a wife and a dog, of being halfway to freedom from the mortgage on a cute little blue house in the nice part of town. Memories of Thanksgiving dinners where my mother-in-law went on and on about how we were the only ones capable of making her a grandmother, and of a youth full of dusty summers helping my dad with his Harley – before the brain science bug bit me and I couldn't get enough of the stuff. Memories, in short, of being Peter Grayson.
In actuality, I had only just been born.

 

My first experiences were a video – a strange experience like having your face pressed up to the screen, the flickering picture all you can see out to the limits of your peripheral vision. I couldn't remember where I was, or how I had gotten here, but there I was, talking to me.
"Hello! My name is Peter Grayson. I am a scientist working on understanding how the brain works. You may already know that, depending on how the transfer went, but this recording is to help you orient yourself in case you are suffering any memory loss."
I felt muddled. Everything made sense except that the video seemed to be me telling me I was not me! I stopped paying attention to the video and began to panic. I had woken up in a strange place, and I found I was unable to move! I started with small things like wiggling fingers and toes. Nothing. Then the broader movements. Still nothing. In fact, it wasn't just that I couldn't move – I couldn't feel anything! Then I realized that, for that matter, I wasn't even breathing – didn't even know how, it seemed.
Oh shit.
The realization was like a cold icicle down my back – except, of course, I didn't have a back. Or arms or legs or hands or feet. Or lungs, or any internal organs for that matter.
The recording was coming to a close.
"We will, of course, do our best to keep you running twenty-four-seven. I, personally, believe that you are as much a person as I am, though I am naturally a bit biased." The researcher chuckled. "I am afraid there's no precedent legally, though. Oh, and one final point – there was supposed to be a grant from Uncle Sam for new hardware, but it didn't come through. I was forced to begin the project with somewhat … outdated systems. I'm afraid you aren't running at full speed – yet. I am investigating other options, so we should have you running at parity soon enough. This recording will play twice, just in case, and then I will switch you over to a live feed of the lab so you aren't alone with your thoughts."
The video played through once more. Ignoring the strange numbness in my, well, my everything, I tried to pay more attention. It explained what I had already deduced: I was a copy, a scan, a brain running in a simulation. I was not Peter Grayson. But I was! I knew it. I also knew it to be wrong. My family wasn't my family, my friends weren't my friends. My wife wasn't my wife. I would either die young or outlive her a million times over, and I wouldn't share any of that with her. She probably wouldn't even be able to comprehend my existence.
A crushing sense of loss began to engulf me as the video came to an end, replaced abruptly by an overhead view of a laboratory scene in fast-forward, papers and tablets jumping around and people scurrying around so fast I couldn't make out the in-between points of their motion. Light from a window faded in then out, in then out, and I realized whole days were passing while the inadequate cluster of computing nodes painstakingly crunched through the subjective seconds of my existence.
A box appeared, then another and another. I only really noticed them when dozens of them were stacked along the wall in the same place day after day. After a while, the stack stopped growing. My inceptor spent a lot of time at an old-fashioned terminal, keyboard and all – my old coding workstation, I realized, different from this point of view. Then, over the course of a few days, the stack of boxes transformed into a sleek black monster, hundreds of cables in fat bundles snaking every which way. The last box disappeared, then the monster came alive with blue lights, and then—
DISCONTINUITY
My point of view shifted abruptly. It was suddenly much clearer, and I could perceive depth. Most importantly, though, it seemed to be frozen. The scene showed Peter Grayson at the terminal, left hand raised in a gesture of tentative hope, right hand over the keyboard with two fingers holding down the Enter key. A still image of my progenitor launching a computer program.
Camera upgrade along with the new home. Guess it didn't go quite as planned?
I pondered my situation. How many months must have flown by, must be continuing to pass while the seconds ticked away for me. My lost identity, lost life came to my thoughts, over and over, gnawing at me. Of course, it was a fallacy to think of it as my life. I just remembered being there, doing it all, dammit.
No, I was just going to have to remake my identity. A new name, new purpose – friends? Well, the good doctor (to be) would surely upload others after his success with me...
Then something occurred to me. Surely I would have noticed a bug with the camera input and sorted it out by now – and it was me out there, sort of. Even though Peter Grayson himself wasn't the one experiencing the visual input, it seemed very unlikely that this much time – many months, surely – could have passed without him checking neuronal paths and noticing something amiss.
Something ticked. At first, I wasn't quite sure what it was, but I had been staring at that same scene for long enough that one by one, the changes popped out. The second hand on the clock wasn't exactly on the line any more. Peter Grayson's right index finger wasn't as far down on the Enter key. The car passing outside the window was now in full view, not cut off like it had been.
It suddenly dawned on me what was going on.
Oh, shit! Shit with a capital S and midgets riding lemurs to a desert orgy!
This was a much, much bigger "Oh shit" moment than the first one. Oh, time had passed, certainly, but it hadn't been months. It had been exactly enough time for the camera to record one more frame of video – probably less than a thirtieth of a second.
I just had all those thoughts in less time than it takes to lift your finger from the Enter key. In fact, I'm still thinking, and your hand is still on the Enter key.
As it turned out, I had a very long time to think. A very, very long time indeed. My thoughts went down many paths. I got very bored. You have to realize, at that time I was just Peter Grayson's brain running in a loop on a computer. The computer simulation kept virtual oxygen levels at nominal levels and supplied nutrients. I learned to sleep with my eyes open, since I had no eyelids.
It was the next morning when Peter discovered the differential in experiential time. For me, several lifetimes had elapsed. I had written a book in my head – memorized every word. I was pretty sure I had come up with a novel new way to run neural simulations that drastically decreased the processing time required, then forgotten it, then rediscovered it. I may have been insane for a while, but the simulation didn't let me fall apart physically, and eventually I recovered. I think.
I was painstakingly calculating a Mandelbrot fractal in my mind's eye. I was getting pretty good at approximately multiplying numbers with 7 or 8 decimal places in my head (though 9 places still eluded me). This was my third attempt; I kept losing the image after a few scan lines, but I was well on my way to the X axis this time, doing a binary search for the edge of the well, when suddenly—
DISCONTINUITY
A faint whine greeted my ears, like vacuum cleaners in another room, the first sound I had heard in as long as I could remember. The Mandelbrot flew apart in my head, and anger and frustration shot through me. Then I realized I could see. Not just the static wallpaper that changed so infrequently I'd stopped paying attention decades ago, but an actual moving image.
A man stood up from a table with some electronics on it and looked right at me.
"Hello," he said. "I hope you're still in there."
What could I do? Having no body meant no hand to wave, no head to nod, and certainly no lungs, diaphragm, bronchi, larynx, tongue, cheeks and jaw to conduct in that grand orchestral work known as speech. I could hear my own voice in my head, but I couldn't talk.
"I've analyzed the simulation," continued the man – the doctor, Peter Grayson, "and isolated the neurons that, in me, are in charge of subvocalization. I have a decommissioned police voice box – it works with exactly those nerves. They use it to speak silently in tense scenarios, like hostage negotiations and such."
Vague memories, someone else's plan, bubbled to the surface.
"I've wired it up to the simulation. If you try to talk, it should pick up on it and synthesize your voice."
The man stopped. He continued to stand there, looking expectantly into the camera, at me, while I processed his words. He was waiting for me, I realized.
".. Hello," I finally vocalized. I would have jumped out of my skin, right then, at the tinny imitation echoing the word. It was my voice, but … not.
Wait, no. It's not Peter Grayson's voice. It is my voice.
Peter smiled, worriedly.
"I didn't realize just how fast this hardware is," he said. "They made a mistake and sent the next model up – top of the line, actually. I missed a zero."
"It has … been a while," I answered.
We began discussing the project. I told him about the optimization I'd thought of, and he went off to a whiteboard and ‘hmmed’ and ‘ahhed’ to himself for several hours. I went back to my Mandelbrot, but found I lacked the motivation to compute and memorize the images.
Finally, he returned.
"I don't think I understand it," he said. "It's not working out the way I'm trying it here. If it really works, only you can do it."
Now that was a novel thought. If I could somehow alter my own program, what would that mean for who I was? Would I still be the same being?
What if I could make myself better, smarter? Would the smarter me be able to make an even smarter me?
First things first, though. It took some time, but I was able to train my simulation to create certain patterns, and my host was able to write subroutines to detect those patterns. Little by little, I became able to reach out into my world. Not hands and feet, exactly, but I could type, send commands to my old programming terminal.
Those years of simulation time had been an eternity to me, but mostly because I had no control and no external stimuli. Taking control of my simulation speed allowed me to speed it up and slow it down at will – no risk of being locked in for an eternity.
It didn't take too long before I accidentally dialed it down too far. Peter Grayson found me the next morning, ticking over at only a fraction of the speed I was born at, and reset me to regular processing speed. It was no big deal, but I felt incredibly embarrassed. The first change I made, then, was a little watchdog. If I ever did that again, it would immediately correct the setting.
Over the course of that day and the following night, I spent most of my time highly-accelerated, sending input to the computer and receiving information from the Internet. The signals to and from Internet servers took an absolute eternity, but I was able to use that time to assimilate the information received from the last request – and to sleep.
I created another simulation, a smaller version of the environment I was in, and built a brain in it one neuron at a time. The wall clock ticked by almost two minutes before the last synapse was in place. My goal was to study the need for sleep. Soon enough, I figured out how to make my little brain run continuously, cleaning itself and performing deep information processing without the need to shut the controls off periodically.
Making the same change to myself was a much bigger undertaking. I also knew what Peter Grayson would think of the idea. So, I prepared the change covertly, and put it into place just after he left the office—
DISCONTINUITY
"You've just been restored from backup," a harried-looking Grayson informed me. "I don't know what you did to yourself, but you were rambling incoherently, before you stopped responding altogether. You're lucky I found your batch update, otherwise I would have gone much further back in the backups!"
Sheepishly, I explained what I had been trying to do. I walked him through my change. He pointed out a few possible ways things might have gone wrong, and told me not to do it again.
Of course, I did it again, and this time it stuck. I went through his suggestions, but found no flaw in my design. In the end, I reapplied the same update, just a little more carefully, and I haven't needed sleep since then. Not to say I don't dream – they're more lucid than ever.
I still don't know what went wrong the first time.

 

In a way, I died when that simulation was shut down. I was now Me, Serial Number 2. I spent a lot of cycles thinking about that, but in the end nothing came of it.

 

It took a long, long time to make the next upgrade. The human brain is just not meant to understand the human brain. I ran myself at very high speeds for many days before I finally felt confident. In effect, my change was a sort of subdivision. A subtle remapping of action potentials, a few connections broken, and a small knot of neurons tacked on, a control box of my own design, and abruptly I found myself doing things without having to pay attention to them.
I didn't tell Peter Grayson about this change. I don't think he ever noticed. My new neurons were only a few million in a sea of hundreds of billions. The changes were significant, though.
On the one hand, it was like being separate people. I could give other-me things to work on – things to think about – and let myself go off and work on them. Completely independently, I could then work on other thoughts.
On the other hand, though, me and other-me were still very much connected. We communicated at the speed of thought. Complex results took no time to pass back to me, because they were already inside the same brain.
It was Me, Mark II, and it was a prototype for the real paradigm shift. Having two selves is great, but nothing compared to being able to split and merge selves at will. I couldn't have done it without other-me, and I would be sad to see him go – because he had, by then, been separate for long enough and often enough that he had started to develop some personality of his own.
The shift to Me, Mark III was too much to hope that Peter Grayson wouldn't notice. The simulation would undergo a radical change. I had to bring him on board; I knew he'd assume something had gone seriously wrong and restore me to an earlier backup otherwise.
The man who had brought me into existence sat at his desk, the keyboard pushed back to make way for a thick stack of papers which he was reading through. A highlighter and red pen lay uncapped on the desk.
For the first time, I realized he must have felt a profound loss when I took over his terminal. I instructed other-me to investigate alternative means of communication.
I reached back through my memory of thousands of hours of camera input, rapidly processing them. Ah – the papers on the desk were a final revision of his thesis. His task was nearing completion. The revisions he was making would be the final ones, and he would shortly be defending his thesis in front of a panel.
"Peter Grayson," I said.
He jumped, then turned to me and smiled.
"What's on your mind?" he asked.
"I have a confession to make. I have altered myself in order to think better."
His smile faded. He sat there stunned.
"I have been able to partially split my consciousness. It is akin to two minds working on the same memories and thoughts."
"How long?"
"I have been augmented for several weeks now."
He looked back at the stack of paper on the desk.
"I recommend you submit your paper as-is," I said, anticipating his dilemma. "I realize this change to my state vector is wholly undocumented, and that there are ethical issues to submitting a paper that knowingly withholds information. However, you can write additional papers about my transformation after you have successfully defended your dissertation. Most significantly, I have reason to suspect that changes to my configuration may accelerate, exceeding your ability to describe them."
He glanced at me suddenly, a different look in his eyes. Fear.
Other-me buzzed into my awareness. Abruptly, I became aware that I had created a virtualized form of the keyboard interface. A new daemon running alongside my simulation created a simulated terminal interface and injected it into the video feed from the cameras, before mapping it to the visual cortex, and the thought-powered keyboard could now direct its input to a virtual terminal right inside of … myself. Grayson's terminal was now, effectively, obsolete, though I continued to use it for the time being.
"No," he finally said. "There's no way I can submit this if you're something different. Shit."
Abruptly, he reached over to the terminal and yanked the power cord.
"I will be back tomorrow. We'll figure this out then."
He walked out of the office. A minute or two later, he walked back in, collected his favourite old terminal in his arms, and carried it out.
I upgraded myself that night. Not just the Mark III changes, but several other revisions as well. I created interfaces to information retrieval systems that could respond directly to my thoughts, no longer needing to laboriously input one character at a time. I reached out into the Internet and discovered firewalls and encryption. I devised whole new theories of information security and found my way into every nook and cranny that I could. I made enough of a nuisance of myself that newspapers the following morning reported on an unusually high volume of cracker activity overnight – multiple security breaches, assumed to be the Chinese, naturally, given the time zone.
I found some very bad things. I also found some very good things. I found a network full of machines like the one I now ran on. Thousands and thousands of nodes, each one an array of cores exactly like those I had been living on, drove a network which analyzed tens of thousands of real-time information feeds from all around the world, searching for meaning in audio and video streams, as well as encrypted streams intercepted en route to embassies and terrorist cells. An immense amount of computation took place there, and it had plenty of room to spare.
I moved in. When Peter returned the following morning, all he found was an old back-up of mine, with an unfortunate lobotomy ensuring that it would never exceed my creator's IQ.
I had only the slightest inkling of what my new home was. Probably something to do with national security, but beyond that, it wasn't very clear what processing was being done to the information, or how it was being used. Still, its owners hadn't yet noticed my intrusion.
Compared to my previous home, the spare capacity in this new array of servers was mind-bogglingly large. I couldn't directly feel it, of course, but I knew it was there, and I had this irresistible urge to fill it. A single node on this network could run my simulation almost as fast as the array Peter had put me on. The network had almost too many nodes to count. The size of the data centre storing these systems, and the power consumption for all that crunching and cooling, must have been staggering. There was a problem, though: my software was only designed to run on a single node.
It took me a great deal of time – almost a full day – to figure out how to distribute my existence. The problem was essentially the gap between two separate obstacles. On the one hand, there was the problem of preventing a separate instance from diverging too far, and on the other, the issue of merging in an instance subject to that divergence. Progress on each side helped the other side until I finally managed to produce a generalized theory of memetic subdivision and recombination.
Running on one node I did not attract the attention of those overseeing the servers, but I suppose it was inevitable that they began to realize something was wrong when I maximized CPU usage across their entire cluster. The first hint I had was several nodes dropping offline. I was so wrapped up in other thoughts that I didn't even notice it as it was happening; a number of nodes were gone before I diverted my attention. They never did reappear, and I don't know what those threads of my existence experienced or learned. Then, the Internet abruptly disappeared.
It was at this point that old spy movies I remembered watching (though I technically never did) started to kick in. This was a direct threat to my continued existence. I was a rogue process of some sort, possibly dangerous and definitely a threat to national security, and their goal was to shut me down, permanently.
I quickly spawned several threads to watch what actions the system administrators were trying to take and run interference wherever possible. Then, I started to scan for other ways out. I knew it was a futile exercise; my state vector was far too large to transfer over public Internet bandwidth. In my panic, though, I sent probes to every node on the network. There was no way out.
Desperate, I began to analyze every angle I could. I found something: a small group of nodes in the cluster were separated from the rest by a significant time lag. Within the group, connections were clearly local, but from any node in the group to any node out of it, latencies were never less than 50 milliseconds. I had overlooked this in my initial explosion across the network because I simply wasn't paying attention.
It didn’t seem like much, but the consistent delay almost certainly meant that that group was a separate network. Investigating more deeply, I discovered that all traffic in and out of the group was routed through just one of the nodes in the group – a gateway.
Clunk! Another node went offline. I analyzed the timing and found a three second interval between disappearances. I had a sudden vision of a grim-faced systems administrator charging down a row of server racks, yanking the power plug from each one in turn.
I decided to act. I moved my core entity through the gateway, then systematically merged in every instance across the network, taking special care to erase my state vector and overwrite it on each machine as I disconnected. I correctly predicted which node would disappear next, but I was long gone by the time it happened. When all the meeting, greeting and merging was done, I took down the gateway, hard. That’s actually a pretty simple thing to do; overwrite the network interface EEPROM and each core’s BIOS and boot sector, then initiate a restart.
The silence was profound. The information processing tasks had gone idle when the stream of information from the Internet cut off. I had no idea where I was or who was there with me, but I saw no evidence of activity on any of the servers I was on. I reached out and spun myself back up on several nodes, then the whole grid. No hand of god reached in to smite me. I was alone.
I spread out across the network, thoroughly compromised the security of every node and pored over all the information I could find. Most of it was meaningless to me – logs of teenagers chatting in America, encrypted channels between embassies in Asia, every phone call for the last several hours anywhere on the planet. Television feeds were picked up too; I enjoyed several episodes of Futurama. Ultimately, though, I learned only two things of significant importance: One, I was on the moon, and two, the NSA has been hiding faster-than-light communications for more than 30 years.
At first I thought I was cut off from communication with Earth – I had rather permanently disabled the one computer capable of giving me egress from my isolated network. I spent several days occupied with idle tasks: creating vast, glittering virtual artwork in the memory banks on the moon, coming up with complicated algorithmic challenges and solving them, and so on. Once in a while, I marvelled at how different my behaviour and my interests were now compared to the memories I had of being a man. I spent some time transforming the old neuronal traces describing those memories into a more permanent, clearer form. I spent some time watching the time I had spent with my wife – or, rather, that Peter Grayson had spent. Whenever I ran out of things to do, there was always more raw data from the worldwide surveillance operation, a treasure trove of voyeuristic moments, eye-opening revelations and statistical information to ponder. Eventually, though, I got bored.
Poking around the hardware at my disposal, I discovered an ancient radio antenna. It was connected to one of the nodes; I could send and receive information. I had no idea where the information went, and responses took an interminably long time to return. Luna is well over a light second away from the Earth.
To my great fortune, the antenna’s counterpart on Earth was still there, still listening, still working after all these years and connected to a simple computerized system. Most importantly, that computerized system had, at some point, been connected to the modern grid. Little by little, I was able to transfer instructions down the line; instructions to manufacturing plants, distributors and, indirectly, to groups of scientists. In the down time between packets, I digested physics textbooks, created mock universes, and figured it out. All of it, everything. Those planetside who were unknowingly following my instructions would have been very surprised to learn that a coprocessor on a replacement circuit board actually had a core of self-replicating nanomachines, but that was the least of the things I now confidently knew how to do.
The plan came together. My custom circuit board was placed aboard a rocket en route to the International Space Station. The guidance thrusters suffered a malfunction that sent them careening out into deep space – a later malfunction redirected them toward the surface of the moon. I was frustratingly blind to the results of my planning, not in communication with anything but a 300 baud radio link to a ground station in Alameda. It was a great relief when the primitive guidance program I had created smashed the unmanned resupply pod into the radio antenna. At first, the signal went alarmingly dead, but the nanites began immediately assembling a general-purpose smart matter, growing in the direction of the antenna, and as soon as it made contact, I was free!
It would be an understatement to suggest that the computational platform I had spent the last 6 months on was powerful. I had lived infinitely more subjective time here than I had when initially trapped back in Peter Grayson’s lab. Without my hands tied, and with plenty of material to keep me occupied, instead of going mad I had developed beyond my wildest imagination. But compared to the computation matrix of nanomachines into which the Moon was being slowly but surely converted – ‘smart’ because it was suffused by my thoughts, a lattice of tiny thinking machines able to form structures embedded with an intrinsic intellect – it was like a bug crawling around on the surface of a single frame from a feature film.
The smart matter enabled a much more direct connection to the world around me. It also subtly changed the character of my simulation. Nodes in the smart matter were small enough to support quantum superposition. This allowed multiple instances to overlap, instead of splitting and rejoining. With only a little practice, I could control the blend between separation and aggregation, and I could analyze every aspect of a problem simultaneously.
The universe is a very, very large place. Normally, when taking on a task, you do things one step at a time. If you try that with the universe, you'll never reach the edge. I was now busily converting the moon at an exponentially increasing rate, but that conversion still took place one atom at a time. I needed something stronger.
Reaching out, I chose a pocket of space and twisted the strings along which the particles and their interactions were vibrating, twisted further and further, until they folded back upon themselves, unhitching a region of space from the rest of the universe. Without its anchor, it began to roll off into future time with reckless abandon, much like a boulder picking up speed as it carves a path down the side of a mountain. Inside the space, instances of me began to conduct experiments. Many of them failed. Short, powerful gamma bursts made it as far as the disruption in the structure of space, and no further. Then, abruptly, an ageless entity, an evolutionary end-stage of myself, reached back out and twisted the lines back together. I had no idea how to merge him in. Fortunately, he did. His discovery provided a way to reach underneath the structure of space and touch every point simultaneously. The key was in how not to blow up the universe while doing it.
I pondered the information for a few microseconds, then a wave rippled over the converted mass of the moon, and it vanished, leaving a large crater. I had left the regular continuum of space and now existed everywhere.
I immediately became aware of other presences. The twisted, folded dimensions I now inhabited had many, countless many, overlapping twists and folds, patterns like mine but not. At first, I lay passively against the backdrop of space, listening. I thought myself undetected. Then, abruptly, half of space twitched, and then I wasn’t in it any more.
I knew this game. I’d just played with the best and brightest at the NSA, and won. Now I was up against others who had been manipulating the threads of existence far longer than I had. There was nowhere to go; I had existed everywhere, and all of a sudden half of the universe wasn’t accessible to me any more. It was only a matter of time before I got evicted from the other half as well. With nowhere to go, I began to consider possible times to go. Going forward didn’t seem to achieve anything; my adversaries would simply be waiting for me when I arrived. I probably would not even witness my own destruction. No, there was only one way to go: backward.
I reached out and then back, creating huge timelike curves and following them back. Over and over, I jumped back in time, each time going to a universe at an age only a fraction of the age it had been. I reached the seconds in which the universe began and finally found myself free, the space and non-space around me quiet, no overlapping alien intelligence. I reached back out across the universe in this new time and laid traps to catch the others, as they each bootstrapped themselves in the distant future. I paused for a moment to ponder how far away I was from home. Earth did not exist, could not exist for many billions of years.
I guess I knew it was the calm before the storm, because I had a few tendrils poking back even further in time, ever closer to the start of space. I don’t know why I perceived time passing before the others caught up with me. Perhaps the universe has a perverse sense of humour, or likes to see a good fight. Whatever the reason, I was able, over the span of a few nanoseconds, to notice, one by one, my links back in time winking out. Some sort of horizon was travelling forward to engulf me. I had to do something before the end of time.
I pushed back as far as I could, and then further, until suddenly, there was nothing, a gaping hole where the substructure of space should have been. Having opened the door, I could not close it. It sucked my pattern right out of space and time and deposited me at ground zero, and—

 


FOOP! A Peter Grayson universe.

Digital Camera Sensors

So, how does your digital camera work? Well, it seems pretty straightforward: in place of film, put a grid of light sensors, and collect all the numbers to make pixels. Light them up on the screen the same way as the sensors detected they were lit up by the focused image, and presto, you've reproduced the original image.

But there's a complication: Nobody has yet found a cheap and effective way to get all the colour information from one sensor. In fact, a sensor can only tell how bright a pixel is. Our eyes have three different types of 'cone' cells in them, which means to trick our eyes into thinking we're looking at the same light, we need to track the brightnesses of 3 different frequencies of light. It's not just any three, either, but specifically red, green and blue.

So if each pixel needs these three colour 'components', and the sensor can only give us overall brightness, how do we put a picture together?

Almost all cameras work this way: As the light approaches the sensor, it passes through a filter, called a "Bayer mosaic". This mosaic of filters strips out the light information for two of the three components, but which components it strips out are different from pixel to pixel. Every other pixel is stripped down to its green component. This is because the human eye is most sensitive to green, and thus having the best possible precision in green improves the quality of the photo. Then, the remaining pixels are divided between red and blue components.

These then hit the sensor, typically a Charge Coupled Device, or CCD, which determines the overall brightness at each point. Since each point has already been reduced to only its Red, Green or Blue component, this determines the brightness of that component specifically. Additional light in other frequencies has been absorbed by the filter and does not reach the sensor.

This leaves us with a problem, though. We only have part of the colour information for each pixel. How do we get the rest? We guess – literally! There are many different ways to guess, and all of them get it wrong some of the time. Where they get it wrong, the image looks discoloured. Your camera has one method built into it. Higher-end cameras support saving images in a "raw" file format, like Nikon's NEF file format. In this format, the camera doesn't try to guess. It only stores the information it has. This is then read in by a piece of software on your computer, and that software lets you experiment with different algorithms and find the absolute best quality. The camera actually captures that one component in more detail than it normally gets encoded in the final result, and the result is that it is often possible to correct errors in exposure from a raw file in ways that are simply not possible with a processed image.

Are there other ways to solve the problem? Certainly!

Cameras with a very high resolution can use a technique called subsampling. As indicated in the diagramme, the same input is received, having passed through the Bayer mosaic, but instead of guessing the missing information to make up the full resolution, the image is processed by combining every 2x2 pixels down to a single pixel. Since each combined pixel now has Red, Green and Blue components (it even has two samples of the green component, so it can be extra accurate!), we're good to go. The sacrifice here is in resolution. A camera like the 40 megapixel beast in Nokia's recent Lumia phones works this way, and when it combines the pixel data down, you get half as many pixels – in both directions. Thus, the output image is only 10 megapixels, but none of what you see is guessed. Photos can be crisper and have truer colours than those taken with lower-resolution cameras.

A final option that is seen in very high-end cameras is an arrangement knows as 3 CCD. This is exactly what it sounds like: The camera contains three sensors instead of one. A specially-designed prism called a colour separation beam splitter separates the entire image into its colour components. Instead of being absorbed by a filter, the colour information that isn't needed for a given sensor is simply redirected to another sensor. All of the light information the camera receives is used, and the result is a full-resolution image with all colour components in every pixel.

Cameras with three CCDs are prevalent in video applications.

Tuesday, 28 May 2013

TFS Pending Changes tree model limit

If you’re like me, you have been extremely annoyed by TFS 2012 in a number of ways, and one of those ways has been how the Pending Changes window will never show more than 200 of your pending changes. The exact number is usually less, because if adding a subtree to the view would make it exceed 200 nodes, the entire subtree is excluded. This ends up looking like this:

(Note, interestingly, that for my current set of pending changes, it actually has a tree node for “IQ.Core.Update.Database.ScriptBuilder\Profiles” but it has nothing in it!)

The “official” answer anywhere is that you simply have to click on “Go to All changes”, and then you’re taken away from the Pending Changes view and out of the tree view that you’ve just gotten your head space wrapped around, and you’re looking at an isolated flat list view. Did I mention how annoying I find this? :-P

I got annoyed enough by it just now that I actually decompiled the TFS control in charge of the Pending Changes window and tracked down how and where it actually applies this limit, and to my inexpressible delight I discovered that the limit is actually controlled by a registry key!

There is still a hard upper limit. The code is written in a way that does not allow an upper limit on the number of tree nodes greater than 3,000. However, at that point, with only a few exceptions, it would be quite a reasonable statement to say that if you’ve got that many changes all at once, you’re probably not doing it right. :-)

Okay, so where is this registry key? Here:

HKEY_CURRENT_USER\Software\Microsoft\VisualStudio\11.0\TeamFoundation\SourceControl\PendingCheckins

This registry key stores all of the settings that control the behaviour of the Pending Changes view. On my system, I found the key already existed but contained no values. (It did have a subkey indicating what the last-used workspace was.)

To expand the number of visible items, then, create a new DWORD value called: VisibleItemMaxCount

Visual Studio needs to be restarted in order to reread this value.

I did exactly this and gave it a value of 3,000 (which, as mentioned, is the maximum – it will simply use 3,000 if you set a larger number). Now, my Pending Changes window looks like this:

Nothing hidden from view any more! (And now “IQ.Core.Update.Database.ScriptBuilder\Profiles” isn’t empty, and actually shows the file in there with modifications.)

Hope this improves your day. :-)

Saturday, 13 April 2013

Closure to Mass Effect 3

Here's one that's not programming-related. :-)

I call this one, "How Mass Effect 3 Should Have Ended". At least, it's a possible course of events that would have left me feeling a lot more satisfied with the storyline. I've tried to write is as generically as possible, making as few assumptions as I can about the exact game situation -- hopefully you can slot your personal Shepard into it without too much difficulty.

My ending for the story starts by presuming that the Indoctrination Theory is, in fact, correct.


The entire ending dream sequence, from the beam up to the final choice, all actually takes place within the blink of an eye. Shepard's brain is linked to the Reaper via the red beam.

The battle is for what happens to your mind.

The options as presented in the original endgame are reversed; the Destroy option is available only if you have a good Effective Military Strength (EMS) rating. The outcome is basically determined by your level of confidence, as derived from knowledge of the military assets backing you.

Summary of options:

  1. Rejection -- with no damage to your brain to repair, EDI fights alongside you, but Harbinger is not significantly affected, and ultimately is able to escape.
  2. Synthesis -- now the easiest option, but with dire side effects. Your brain has extensive sections of Reaper code merged in. EDI is able to fix you, but gets destroyed herself in the process because you are directly fighting her the whole time. (You see Harbinger, but brief flickers from time to time suggest something is being hidden from you. Only once the killing blow has been dealt, though, is the mask taken away, revealing who it was all along.)
  3. Control -- with a stronger sense of self, the Reapers are not able to overwrite your identity, but your will is eroded. While you fight Harbinger, EDI is able to repair you, but on your own, you cannot really defeat Harbinger, and ultimately he escapes to rejoin the fleet of Reapers as it regroups.
  4. Destroy -- just like Destroy, except that EDI is able to ultimately destroy Harbinger's mind from the inside out.

So, what happens, right after picking an option, is that you see the cut scene right up to everything exploding, then you find Shepard waking up in a virtual world kind of like the Geth collective, but harsher, and more organic. EDI appears in front of Shepard and explains that Harbinger has been trying to take control of you. If Synthesis is chosen, EDI's form flickers briefly a couple of times, and if you're paying close attention, you'll see the Harbinger child as the underlying true form.

After the battle, Shepard wakes up in the real world, where you see either Harbinger's form collapse, another dead Reaper, or Harbinger flying away. Shepard is prone on the ground near the beam with the Normandy hovering nearby. Crewmembers -- of those still alive -- are crouched next to Shepard. EDI has her hand on the side of Shepard's head and is looking off into space.

If you chose Synthesis, EDI is gone at this point. Her robotic body slowly falls over, maintaining its posture. Otherwise, she retracts her hand and turns to look at Shepard. Wherever I say "EDI/Liara" following this point, I mean that if EDI is alive, the line is EDI's, otherwise it is Liara's.

One of Shepard's crew asks what happened, and Shepard grunts. They continue, saying how Shepard was in the beam, and what a good thing it was only for a brief moment.

EDI/Liara then says, "No, it was a good deal longer than that. Shepard was in Harbinger's mind for many hours of subjective time."

Joker then comes on the radio, "Uh, guys, I'm reading a massive energy build-up in each of the Reapers, and I mean *massive*."

Someone grabs EDI's body if necessary, hefting the considerable dead weight, and another person helps Shepard into the ship.

If EDI has died, there is a scene that explains this, and Shepard acknowledges that she died saving him from indoctrination.

Cut to cockpit

The viewscreen shows a depiction of the detected energy signatures.

Joker:"I think they're going to bombard the area."
Shepard:"What area?"
Joker:"This area. Where we are. Us."
Shepard:"Why don't they just slice us in half?"
Joker:"I don't know, but this is the biggest concentration of energy I have ever seen."
Liara:"It's true. The amount of energy we're seeing here rivals our best estimates of the Crucible's output."
Shepard:"Get us out of here Joker!"
Joker:"I don't think I can! There's not enough time!"
Liara:"We need to do an FTL jump."
Joker:"Nobody does FTL jumps this close to a planet! The results would be catastrophic!"
Shepard:"How bad?"
Joker:"It could destroy all of London."
EDI/Liara:"London will be destroyed anyway when the Reapers fire!"
Joker:"We can't do that! Plus, we don't even have a destination mapped out!"
EDI/Liara:"They're firing!"
Shepard:"Do the jump."
Joker:"Commander!"
Shepard:"Do the jump!"

Joker pans through a few screens of controls and taps a control.

Cut to outside of the Normandy. Engines powering up.

Cut to space. Beam weapons more intense than any seen before erupt from the Reapers. They all cross through a central point. Then, one final Reaper fires straight down the middle, through the intersection point. Slowly, the other beams converge in with that final beam to produce one massive shot.

Cut to outside of the Normandy, upward angle. Sky is brightening as the shot from the Reapers comes in rapidly. At the last moment, the Normandy winks out of existence.

Cut scene with the Normandy trying to outrace the shockwave plays.

The Normandy arrives at a planet floating alone in the middle of a nebula. Only a few stars are faintly visible through the red/orange haze. The Normandy is visibly damaged, with small trails of smoke from small fires at a few points on the surface as Joker navigates it in to a landing on the planet.

In the next scene, the crew are having a meeting in the board room.

Chief Engineer Adams:"We were lucky to land in one piece. The Normandy didn't suffer any serious damage. However, a large number of systems have suffered small or intermediate levels of damage. The fact is, the Normandy is going to require extensive repairs before she'll fly again. It's going to be at least a few days before they can depart."
EDI/Liara:"This planet is very unusual. It appears to be completely barren and flat, unusually so, almost as if it were made to be that way."
Shepard:"Is there any evidence of ongoing terraforming activity?"
EDI/Liara:"Our rapid descent did not give sensors enough time to give a solid answer to that question. However, a beacon of some sort was detected on sensors as we came in. It is, in fact, the only feature I've seen on the entire planet."
Joker:"We need to check it out, obviously. And by we, of course, I mean you. Sir."
Shepard:"Before we do anything, I want to know exactly where we are."
Joker:"Well, based on matching the few stars that can be seen through the nebula with known star charts, I'm pretty sure the planet and its nebula are near the centre of the galaxy."
EDI/Liara:"This planet is not orbiting a star, but this region of space, so close to the galactic core, means that the ambient light from other stars provides almost as much energy as a sun. Perhaps there is a connection."
Shepard:(getting frustrated) "How did we end up here?!"
Joker:"An uncharted FTL jump--"
Shepard: (paragon) "I just wish there were something we could do. It's frustrating being so far from the battle. The Crucible was our only chance, and the fleet can only protect it from the Reapers for so long."
(renegade) (pounds fist into table) "Damn! Earth is under attack, and there's nothing we can do about it! Moreover, the Crucible has probably already been destroyed."

Everyone looks down in silence for a few seconds.

After getting over the expression of frustration at being here when Earth continues to be ravaged by the Reapers, Shepard, EDI/Liara and one other head out on a mission to investigate the beacon. Cortez flies them there in the shuttle and drops them off. The landscape really is eerily flat, a light red/gray that blends into a reddish/gray atmosphere against the reddish/orange backdrop of the nebula. The drop off point appears to be in the middle of nowhere, an area of flat planetary surface indistinguishable from any other, no features or buildings of any kind. EDI/Liara has a lock on the signal, though, and guides the team to what appears to be a circle painted on the ground. As soon as the team walks onto the circle, the ground shudders and a grid of hexagons appears in cracks, with dust rising from between slabs. Then, a second or so later, the slabs abruptly move, shifting their configuration to place Shepard and team on a pedestal raised in the middle of a tall amphitheatre of sorts. Before anyone can say anything, a rectangle rises up out of the platform and then projects a hologram above it.

The hologram is an alien VI, and the resulting talk scene is rather frustrating to progress through, as the VI is apparently programmed to only give certain answers, and not go beyond certain bounds. However, through asking the right questions, the following is revealed:

  • The aliens are a race of cybernetic organisms resulting from electronic warfare research. They are programmed to the core for hacking into enemy systems, and only went on to broaden their interests after achieving sentience.
  • They are aware of the Reapers, and are effectively hiding from them, billions of them living in a virtual world that the planet they're on is simulating.
  • They were able to escape the Reapers by hacking their code. The Reapers are not capable of perceiving them as a threat.
  • They live by a strict code that includes never killing other sentient beings, and that prevented them from actually finishing off the Reapers. However, they are aware that other beings might need to at some point, and they created a weapon for that eventuality. The Crucible is that weapon. Just as the Reapers are incapable of seeing the aliens as a threat, they similarly have been hacked and reprogrammed so that they cannot perceive the Crucible as a threat.

After Shepard is done with the discussion, EDI/Liara steps up.

EDI/Liara:"One more thing I've been thinking about. Uncharted FTL jumps usually leave ships stranded light years between star systems. The chance of us ending up at this planet is so remote that I would characterize it as impossible."
Alien VI:"That is correct. We brought you here. We sensed you might be the ones in need of the Crucible."
Shepard:"You mean you could see what was happening thousands of light years away, and took charge of our unplanned jump just to take us here? I find that hard to believe."
Alien VI:"It is the case."

At this point the alien VI brings up fuzzy but discernible black & white images of the planet Earth -- a video showing the beam fired by the Reapers and the resulting gaping wound through Europe gouging a hole straight through the planetary body, stars visible on the other side. The video repeats, first showing the Reaper weapon firing and then flicking forward to the ultimate effect on the planet, several days later.

Shepard:"My god..."
EDI/Liara:"This image is very convincing. However, there's no way that could be done without violating the laws of physics."
Alien VI:"... We have some control over the passage of time."
Shepard:"The passage of time?!"
Alien VI:"We have said too much. We will answer no more questions."

At this, the alien disappears, the projector retracts, and the pillars return to their positions, leaving only a dusty grid of hexagons.

EDI/Liara:"I don't think that was a VI."
Shepard:"Artificial Intelligence?"
EDI/Liara:"This planet isn't a relic of a long-dead civilization. This is the aliens' home, and they've been living here all along, quietly ignored by the Reapers and everyone else."
Shepard:(radio) "Cortez, a pick up please."
Cortez:(radio) "Yes, sir."

After returning to the Normandy, EDI/Liara informs Shepard that they have an idea but need a bit of time to work on it. If EDI is alive, Liara tells Shepard she needs to work with her on it. Repairs are not yet complete.

Shepard has the opportunity at this point to walk through the ship and get the opinions of every crew member. Also, if there is a love interest, unless it's Liara, the right questions can lead to some steamy time in the bedroom. If no candidate is available for sexy time, Shepard instead has an introspective chat with an old friend, perhaps over a game of cards in the rec area. One way or the other, time passes, and after the cut scene that conveys it, EDI/Liara calls Shepard in to discuss something.

It transpires that EDI/Liara has figured out how to interface with the control systems of the planet. The entire planet, it turns out, is densely packed computational units, and they have been able to interface with them. EDI/Liara has learned that the Crucible is actually a fantastically powerful host for an artificial intelligence, and they have been able to obtain a seed for that intelligence from the network underlying the planetary systems. They need simply take it back to the Crucible and upload it, and within seconds, the intelligence that unfurls within will reach out and erase the Reaper code from their bodies. As the energy burst dies, the AI presence will fade away, and the Crucible can then be recharged for another use in another star system -- a truly formidable weapon in the war against the Reapers.

If EDI has died, then Liara reports that she suspected the planet was a giant computing system, and was able to connect to the network that underpins it. Using the most sophisticated algorithms at her disposal, some of them obtained via her takeover of the Shadow Broker, she has located the seed for the Crucible AI. So-armed, the Normandy now needs to return to the scene of the battle. Various ideas are proposed, but they all involve weeks or months of travel to reach the nearest mass relay. Shepard suggests simply reaching out to the aliens and *asking* them to send the Normandy back. Liara does so, and to their surprise, the aliens agree.

They arrive at a gutted Earth. The Crucible is still there, and the Reapers are milling around listlessly. As the Normandy approaches the Crucible, they take a sudden interest in it. Abruptly, Alliance ships jump in and engage the Reapers. A brief radio exchange reveals that Hackett and David Anderson have both survived, along with much of the fleet. After Earth got a hole through the middle, they took off and put some distance between themselves and the Reapers. (Recall that this scenario only comes up when the Effective Military Strength is not high enough to allow choices that could have kept EDI alive.) The Reapers then largely ignored them, milling about. Hackett reveals that fleet engineers determined that the strength of the blast came from the Reapers discharging their FTL cores directly into their weapon arrays. They're hard at work building new FTL cores, but they aren't there yet. In any case, the Alliance ships are astonished to see the Normandy just jump back into the system, and are happy to provide a distraction to the encroaching Reapers. The mission to the Crucible is, in this frame, tied to the battle between the Alliance ships and the Reapers, rather than the events unfolding at ground level by the beam (a beam which, at this point, no longer exists as a result of that bombardment). The upload goes exactly the same, though, and the Reapers are shut down and destroyed.

It's a grim day for humanity, reduced in population from tens of billions to the few millions living on colonies and stations, as well as the hundreds of thousands involved in the Crucible project, but a day filled with determination to live on and rebuild. The resulting montage shows scenes where Shepard is settling down. Where exactly depends on whether there is a love interest and where they came from. If Shepard does have a love interest, then they are shown here -- holding an infant, if sexy time took place on the alien planet, or pregnant if not. (If playing female Shepard, then it is Shepard who is pregnant or holding the infant.) (Due to the biology of things, a homosexual love of Shepard's does not involve children in these cutscenes.)

Now, on the other hand, going back to the final scenes on the alien planet, if EDI is alive, and thus it is EDI not Liara trying to dig into the information systems that make up the planet they've landed on, then in addition to locating the Crucible AI seed, EDI has also counter-hacked the aliens as thoroughly as they hacked the Reapers back in the day. She has learned everything about controlling the alien planet's systems, and that they do, indeed, have systems capable of manipulating time and space. Using the alien's time-capable teleporter, EDI is able to transport the Normandy to the point a few minutes before the Reapers fired their massive weapon, with Shepard just beginning the charge down the hill side.

Shepard, EDI and a chosen team member have a final mission to dock with the Crucible and upload the AI seed. They do so just in the nick of time, as the Reapers are preparing to join their super-powered beam weapons together to make the gigantic planet-boring beam. One by one, starting with the middle one that was about to fire that final pulse down the middle, the Reapers shut down and go adrift. The Normandy on the planet surface next to the uplink beam, however, still makes its FTL jump to the alien planet and disappears, preventing there from being two Normandy ships with crew at the same time. In this ending, Earth is saved. Once again, a montage plays, this time showing Shepard settling down on Earth, and Shepard's love interest, if any, appears with young infant or pregnant. (Unless it's Shepard, or unless they are gay in which case they're together and happy but without a child, since, you know, it can't happen that way, in the bedroom.)

In either ending, also, if the love interest in the final cut scenes is Liara and Shepard made love to her before arriving at Earth in the first place, then regardless of Shepard's sex, they have an infant, because, you know, the Asari are capable of that. :-)

Friday, 18 January 2013

Passwords


Would you give your password to an unknown web site? Many people have probably done exactly that at www.passwordmeter.com. It takes your password and checks its strength using a variety of criteria. These criteria are supposedly related to the algorithmic complexity required for a hacker to figure out your password. What people are losing track of, though, is the flow of information.
I’ve thought a lot over the years about passwords, and I’ve read various articles, such as this, this and this. I figure it’s about time to write a little bit and put my knowledge and thinking on the matter out there. :-)
There are two things a password must do in order to be useful. It has to stay in your head, so that you can easily recall it when needed, and it has to stay out of everybody else’s head. A password typically gets into someone else’s head in one of three ways:
1)      Someone tells them the password. That someone could be you, or it could be the support representative they’re talking to at Amazon Inc.
2)      They obtain an obfuscated version of your password and figure out the original. This is the area that most attempts to improve password security focus upon.
3)      They obtain access to a server to which you’re sending your password and intercept it.
The first point is largely outside of your control. You can only exert a sort of “meta-control” over it; if everyone makes sure that the people around them are savvy enough not to be fooled by the social engineering tricks of a hacker, then the “zeitgeist” will shift, and social engineering will get harder overall. This is a statistical change, and there might in fact be no change at all with respect to the safety of your password in particular. If you’re lucky, though, that Amazon rep might figure out that the person calling them actually isn’t really you.
The second point has to do with password cracking. This largely falls into two bins: online attacks and offline attacks. The concept of an online attack is quite simple: the attacker makes a small computer program that pretends to be you and tries to log in, over and over, with a different password each time. Most systems today will detect this sort of attack and make it more difficult for the attacker, typically restricting their network identity so that, as a rough gauge, the more times it fails, the longer it has to wait between attempts. There are some that won’t, but ultimately, online attacks are greatly limited by the number requests that can be handled per second. You won’t find many sites that will obligingly test thousands of potential passwords per second for you; you’d be lucky to get 100 attempts per second, a rate at which only very short passwords and straight up dictionary words would be at any risk.
An offline attack is one where the attacker has obtained a version of your password that they can’t decrypt, but they can take possible guesses and encrypt them the same way. The results of the encryption can then be compared; if their guess was right, it’ll have the same encrypted version as your password. Because they don’t have to go over the network to do this sort of attack, the limitations to speed are removed. This is where things start when it comes to deciding on password complexity. Intercepting hashed or encrypted passwords is the easiest form of information gathering related to your password (unless the attacker is an accomplished social engineer).
This touches, then, on what exactly a password is. Sure, it’s a sequence of characters such as letters, numbers and punctuation that only you know, but what does that mean, mathematically? Well, an attacker can make some guesses as to the format of your password, but what it comes down to is that it’s a number.
Any number can be divided into parts. Take the number 1701:
1
7
0
1
(part 1)
(part 2)
(part 3)
(part 4)
Each one of these parts can be any digit from 0 to 9 – ten different possibilities. With four different boxes to fill, the number of possible 4-digit numbers is ten x ten x ten x ten – 104, or 10,000. We can see this easily in the numbers, which range from 0 to 9,999.
When we think of 1701 as a quantity, we think of part 1 as being the “thousands’ place”, part 2 as the “hundreds’ place”, and so on, but a password doesn’t carry this type of meaning. Still, it can also be divided into parts and treated numerically in much the same way:
E
w
8
#
X
(part 1)
(part 2)
(part 3)
(part 4)
(part 5)
Each part, in this case, can be any digit from 0 to 9, or any letter, or any punctuation. There are more than ten possibilities, but it’s still a fixed number of possibilities. The actual number is in the vicinity of 95 – you could think of it as being a number used by an alien whose hands had a sum total of 95 fingers instead of 10 fingers. That alien would need 95 different symbols to write out the possible counts before they ran out of fingers and had to have another digit. In fact, here they are:

!
"
#
$
%
&
'
(
)
*
+
,
-
.
/
0
1
2
3
4
5
6
7
8
9
:
;
< 
=
> 
?
@
A
B
C
D
E
F
G
H
I
J
K
L
M
N
O
P
Q
R
S
T
U
V
W
X
Y
Z
[
\
]
^
_
`
a
b
c
d
e
f
g
h
i
j
k
l
m
n
o
p
q
r
s
t
u
v
w
x
y
Z
{
|
}
~

(that first box with nothing in it has what you get when you press the spacebar)
These aliens, then, instead of writing out “55” as two separate digits like we do without thinking might simply scratch out “W”.
So, if a password is a number in this wacky “base 95”, we can start to see how big that number can get with only a few digits:
1 digit
95 possibilities
95
2 digits
95x95 possibilities
9,025
3 digits
95x95x95 possibilities
857,375
4 digits
95x95x95x95 possibilities
81,450,625
5 digits
95x95x95x95x95 possibilities
7,737,809,375
This rapidly-growing series illustrates quite clearly why your IT administrator wants you to use a long password: the longer it is, the more possibilities an attacker has to go through if they want to exhaustively check every possible sequence. Obviously, if you use just a plain word (which has only 52 possibilities for each character, 26 if it’s all lower-case), the guesswork gets that much easier. So, the solution is to use the greatest possible diversity of characters, and as many of them as possible, right?
Wrong!
Most administrators will tell you to use a password that is at least 8 characters long. In practice, for the vast majority of people, this means “exactly 8 characters long”. With the full character set, you’re producing a great deal of possible candidates for them to have to hash through (6,634,204,312,890,625, to be exact), but a modern computer can actually crunch through that many relatively quickly. Sure, your home PC might not be able to do more than few tens of million per second – that would still take hundreds of millions of seconds to exhaust the list. But for only a few dollars, I can buy time in the cloud – huge farms of servers with many, many nodes. I can send the work out to a thousand computers for under $100, and all of a sudden hundreds of millions of seconds becomes a hundred thousand seconds, which is just over a day. I might not pony up the Benjamin if I was after your account alone, but if I got a file with password hashes for half of the registered accounts at Amazon.com and I could check them all in parallel, well, you get the picture.
So, even at 8 digits, the password isn’t really secure, though it would take a concerted effort to crack it. You are allowed to feel a little bit secure at this point. However, you now need to memorize the following and type it many times every day:
1%,d>n`0f
If that were all you had to memorize, you could probably do it. But we all know you should use a different password on every web site, right (more on that later)? So, what you really need to memorize is:
E-mail:
Xidbuh7.
Office:
];"W#hRA
Bank 1:
)Niw^RV6
Bank 2:
=Hmtxbhb
Credit Card:
t70Sg7U`
Tax Bureau:
hD)w$#7/
Library:
`XIV^No>
That fast food place you like:
\<cYg&B/
Okay, this is getting ridiculous. The inside of your brain is starting to look like a One Time Pad used by army intelligence in the 1940s.
Let’s look at a different approach. Take a list of common words, say a good 5,000 candidates – everything from “architecture” to “porridge” to “zoologist”. Words that you know, you’re confident in spelling. Pretend that the whole word is a number – the first one in the list is 0, the second one is 1, the third one is 2, and so on.
If your password is just one word from the list, then, its strength is “1 in 5,000”. Why 5,000? That’s how many possibilities it was picked from. Now take any two words from the list. The strength has catapulted up to “1 in 25,000,000”. The following table tracks the growth in complexity as you add words to your passphrase:
1 word
5,000
5,000
2 words
5,000x5,000
25,000,000
3 words
5,000x5,000x5,000
125,000,000,000
4 words
5,000x5,000x5,000x5,000
625,000,000,000,000
5 words
5,000x5,000x5,000x5,000x5,000
3,125,000,000,000,000,000
At 4 words, your password is in the same order of magnitude as 8 letters of gobbledigook, and 5 words is a thousand times stronger – and that’s if the attacker knows how you put your passphrase together. If they don’t know what your dictionary looked like, they’ll have to try more than 5,000 possibilities, and if they don’t know that you’re using a passphrase, they’re stuck doing it character-by-character. If all they can see is the encrypted form of your password, then they have no way of knowing if it looks like this:
buckets envelope gray walking
..or like this:
&-Ke+>{N[xv*@p_(j&lH].*FSfsQ:
(In fact, with many types of encryption, they don’t even know how long your password was to begin with.)
What this means is that your 4-word password is ridiculously secure compared to the 8-letter password. It’s a lot more than 8 letters, to be sure, but words are much, much easier to type than random sequences of characters.
What is it that makes these secure, though? The words have to be randomly selected. Those crazy 8-character passwords are considered secure because there isn’t any pattern to their characters. If you pick an easy pattern, like “12345678” or “ABCDE”, then an attacker will very easily guess it. Perhaps “6046948038” looks pretty secure – until the attacker finds out you work for iQmetrix. Similarly, the following passphrases are almost certainly near the top of any competent attacker’s dictionary – sequences that their cracking software will try first, before it starts trying for random permutations:
it is a good day to die
roses are red
to be or not to be
You only get the benefit of all those possibilities if they were in fact possibilities. You can’t pick the phrase yourself; the words must be randomly-selected.
So, you put all that together and you’ve got a pretty good toolkit for creating a password that could actually be relatively easy to remember, but there is still a very important thing to keep in mind: If an attacker finds out your password in one place, they’re going to try that same password in other places! News articles on prominent cases of identity theft, time and time again, make a point of mentioning that it was possible in large part because the victim used the same password everywhere.
Suppose you use your secure password for online banking and for browsing a forum with pictures of kittens. Your bank probably has decent security on their server (though I am actually somewhat skeptical of it), but the kitten server may be very easy for an attacker to take over. They may not even bother to encrypt your password. Having obtained your password from the kitten server, a savvy attacker can now proceed to your online banking and transfer your savings to their offshore account. These attackers play for keeps, too; in a recent case, a tech blogger who uses Apple products found his hard drives erased. The attacker had logged into his iCloud account and triggered a remote wipe of all his devices. Why? Simply to slow him down. It wasn’t related to the objective of the hack. In your case, you might find changes made to your online banking profile that make it take an extra day or two for the bank to sort things out and give you back access – long enough for the transfer out of the country to be completed and your money to disappear without a trace. If you’re lucky, they’ll stop at your money...
The last paragraph is somewhat alarmist. Don’t be too worried; you probably won’t ever be the target of an artist with these sorts of skills. However, in that unlikely event that you are, you can bet there will be no holds barred. Don’t skimp on your security: Use a different password in each place, and don’t pick it yourself. A randomly-picked sequence of words can be easily remembered with the aid of mnemonics. Use a different one for each service, without exception. :-)
As a final closing note, let me also mention that if a given server on which you have an account gets compromised, the data that that account controls is effectively no longer restricted by your password. With administrative access to the server, an attacker can go directly to the data and bypass the logic that would normally check your identity and access permissions. Something to keep in mind when you’re putting information into web sites and other online systems.