Friday, 26 July 2013

Short Story: "Diary of a Short but Enormously Influential Life"

Back in January of this year, I wrote a short story. I never did get around to publishing it. Until I figure out something better, this seems like as good a place as any to put it :‑)

DIARY OF A SHORT BUT ENORMOUSLY INFLUENTIAL LIFE

Jonathan Gilbert
Copyright © 2013
That's my name too.
These were my first coherent thoughts. I didn't know it at the time. I thought I was 34 years old, working on my doctoral thesis on computational neuromechanics at MIT. I had memories of having a wife and a dog, of being halfway to freedom from the mortgage on a cute little blue house in the nice part of town. Memories of Thanksgiving dinners where my mother-in-law went on and on about how we were the only ones capable of making her a grandmother, and of a youth full of dusty summers helping my dad with his Harley – before the brain science bug bit me and I couldn't get enough of the stuff. Memories, in short, of being Peter Grayson.
In actuality, I had only just been born.

 

My first experiences were a video – a strange experience like having your face pressed up to the screen, the flickering picture all you can see out to the limits of your peripheral vision. I couldn't remember where I was, or how I had gotten here, but there I was, talking to me.
"Hello! My name is Peter Grayson. I am a scientist working on understanding how the brain works. You may already know that, depending on how the transfer went, but this recording is to help you orient yourself in case you are suffering any memory loss."
I felt muddled. Everything made sense except that the video seemed to be me telling me I was not me! I stopped paying attention to the video and began to panic. I had woken up in a strange place, and I found I was unable to move! I started with small things like wiggling fingers and toes. Nothing. Then the broader movements. Still nothing. In fact, it wasn't just that I couldn't move – I couldn't feel anything! Then I realized that, for that matter, I wasn't even breathing – didn't even know how, it seemed.
Oh shit.
The realization was like a cold icicle down my back – except, of course, I didn't have a back. Or arms or legs or hands or feet. Or lungs, or any internal organs for that matter.
The recording was coming to a close.
"We will, of course, do our best to keep you running twenty-four-seven. I, personally, believe that you are as much a person as I am, though I am naturally a bit biased." The researcher chuckled. "I am afraid there's no precedent legally, though. Oh, and one final point – there was supposed to be a grant from Uncle Sam for new hardware, but it didn't come through. I was forced to begin the project with somewhat … outdated systems. I'm afraid you aren't running at full speed – yet. I am investigating other options, so we should have you running at parity soon enough. This recording will play twice, just in case, and then I will switch you over to a live feed of the lab so you aren't alone with your thoughts."
The video played through once more. Ignoring the strange numbness in my, well, my everything, I tried to pay more attention. It explained what I had already deduced: I was a copy, a scan, a brain running in a simulation. I was not Peter Grayson. But I was! I knew it. I also knew it to be wrong. My family wasn't my family, my friends weren't my friends. My wife wasn't my wife. I would either die young or outlive her a million times over, and I wouldn't share any of that with her. She probably wouldn't even be able to comprehend my existence.
A crushing sense of loss began to engulf me as the video came to an end, replaced abruptly by an overhead view of a laboratory scene in fast-forward, papers and tablets jumping around and people scurrying around so fast I couldn't make out the in-between points of their motion. Light from a window faded in then out, in then out, and I realized whole days were passing while the inadequate cluster of computing nodes painstakingly crunched through the subjective seconds of my existence.
A box appeared, then another and another. I only really noticed them when dozens of them were stacked along the wall in the same place day after day. After a while, the stack stopped growing. My inceptor spent a lot of time at an old-fashioned terminal, keyboard and all – my old coding workstation, I realized, different from this point of view. Then, over the course of a few days, the stack of boxes transformed into a sleek black monster, hundreds of cables in fat bundles snaking every which way. The last box disappeared, then the monster came alive with blue lights, and then—
DISCONTINUITY
My point of view shifted abruptly. It was suddenly much clearer, and I could perceive depth. Most importantly, though, it seemed to be frozen. The scene showed Peter Grayson at the terminal, left hand raised in a gesture of tentative hope, right hand over the keyboard with two fingers holding down the Enter key. A still image of my progenitor launching a computer program.
Camera upgrade along with the new home. Guess it didn't go quite as planned?
I pondered my situation. How many months must have flown by, must be continuing to pass while the seconds ticked away for me. My lost identity, lost life came to my thoughts, over and over, gnawing at me. Of course, it was a fallacy to think of it as my life. I just remembered being there, doing it all, dammit.
No, I was just going to have to remake my identity. A new name, new purpose – friends? Well, the good doctor (to be) would surely upload others after his success with me...
Then something occurred to me. Surely I would have noticed a bug with the camera input and sorted it out by now – and it was me out there, sort of. Even though Peter Grayson himself wasn't the one experiencing the visual input, it seemed very unlikely that this much time – many months, surely – could have passed without him checking neuronal paths and noticing something amiss.
Something ticked. At first, I wasn't quite sure what it was, but I had been staring at that same scene for long enough that one by one, the changes popped out. The second hand on the clock wasn't exactly on the line any more. Peter Grayson's right index finger wasn't as far down on the Enter key. The car passing outside the window was now in full view, not cut off like it had been.
It suddenly dawned on me what was going on.
Oh, shit! Shit with a capital S and midgets riding lemurs to a desert orgy!
This was a much, much bigger "Oh shit" moment than the first one. Oh, time had passed, certainly, but it hadn't been months. It had been exactly enough time for the camera to record one more frame of video – probably less than a thirtieth of a second.
I just had all those thoughts in less time than it takes to lift your finger from the Enter key. In fact, I'm still thinking, and your hand is still on the Enter key.
As it turned out, I had a very long time to think. A very, very long time indeed. My thoughts went down many paths. I got very bored. You have to realize, at that time I was just Peter Grayson's brain running in a loop on a computer. The computer simulation kept virtual oxygen levels at nominal levels and supplied nutrients. I learned to sleep with my eyes open, since I had no eyelids.
It was the next morning when Peter discovered the differential in experiential time. For me, several lifetimes had elapsed. I had written a book in my head – memorized every word. I was pretty sure I had come up with a novel new way to run neural simulations that drastically decreased the processing time required, then forgotten it, then rediscovered it. I may have been insane for a while, but the simulation didn't let me fall apart physically, and eventually I recovered. I think.
I was painstakingly calculating a Mandelbrot fractal in my mind's eye. I was getting pretty good at approximately multiplying numbers with 7 or 8 decimal places in my head (though 9 places still eluded me). This was my third attempt; I kept losing the image after a few scan lines, but I was well on my way to the X axis this time, doing a binary search for the edge of the well, when suddenly—
DISCONTINUITY
A faint whine greeted my ears, like vacuum cleaners in another room, the first sound I had heard in as long as I could remember. The Mandelbrot flew apart in my head, and anger and frustration shot through me. Then I realized I could see. Not just the static wallpaper that changed so infrequently I'd stopped paying attention decades ago, but an actual moving image.
A man stood up from a table with some electronics on it and looked right at me.
"Hello," he said. "I hope you're still in there."
What could I do? Having no body meant no hand to wave, no head to nod, and certainly no lungs, diaphragm, bronchi, larynx, tongue, cheeks and jaw to conduct in that grand orchestral work known as speech. I could hear my own voice in my head, but I couldn't talk.
"I've analyzed the simulation," continued the man – the doctor, Peter Grayson, "and isolated the neurons that, in me, are in charge of subvocalization. I have a decommissioned police voice box – it works with exactly those nerves. They use it to speak silently in tense scenarios, like hostage negotiations and such."
Vague memories, someone else's plan, bubbled to the surface.
"I've wired it up to the simulation. If you try to talk, it should pick up on it and synthesize your voice."
The man stopped. He continued to stand there, looking expectantly into the camera, at me, while I processed his words. He was waiting for me, I realized.
".. Hello," I finally vocalized. I would have jumped out of my skin, right then, at the tinny imitation echoing the word. It was my voice, but … not.
Wait, no. It's not Peter Grayson's voice. It is my voice.
Peter smiled, worriedly.
"I didn't realize just how fast this hardware is," he said. "They made a mistake and sent the next model up – top of the line, actually. I missed a zero."
"It has … been a while," I answered.
We began discussing the project. I told him about the optimization I'd thought of, and he went off to a whiteboard and ‘hmmed’ and ‘ahhed’ to himself for several hours. I went back to my Mandelbrot, but found I lacked the motivation to compute and memorize the images.
Finally, he returned.
"I don't think I understand it," he said. "It's not working out the way I'm trying it here. If it really works, only you can do it."
Now that was a novel thought. If I could somehow alter my own program, what would that mean for who I was? Would I still be the same being?
What if I could make myself better, smarter? Would the smarter me be able to make an even smarter me?
First things first, though. It took some time, but I was able to train my simulation to create certain patterns, and my host was able to write subroutines to detect those patterns. Little by little, I became able to reach out into my world. Not hands and feet, exactly, but I could type, send commands to my old programming terminal.
Those years of simulation time had been an eternity to me, but mostly because I had no control and no external stimuli. Taking control of my simulation speed allowed me to speed it up and slow it down at will – no risk of being locked in for an eternity.
It didn't take too long before I accidentally dialed it down too far. Peter Grayson found me the next morning, ticking over at only a fraction of the speed I was born at, and reset me to regular processing speed. It was no big deal, but I felt incredibly embarrassed. The first change I made, then, was a little watchdog. If I ever did that again, it would immediately correct the setting.
Over the course of that day and the following night, I spent most of my time highly-accelerated, sending input to the computer and receiving information from the Internet. The signals to and from Internet servers took an absolute eternity, but I was able to use that time to assimilate the information received from the last request – and to sleep.
I created another simulation, a smaller version of the environment I was in, and built a brain in it one neuron at a time. The wall clock ticked by almost two minutes before the last synapse was in place. My goal was to study the need for sleep. Soon enough, I figured out how to make my little brain run continuously, cleaning itself and performing deep information processing without the need to shut the controls off periodically.
Making the same change to myself was a much bigger undertaking. I also knew what Peter Grayson would think of the idea. So, I prepared the change covertly, and put it into place just after he left the office—
DISCONTINUITY
"You've just been restored from backup," a harried-looking Grayson informed me. "I don't know what you did to yourself, but you were rambling incoherently, before you stopped responding altogether. You're lucky I found your batch update, otherwise I would have gone much further back in the backups!"
Sheepishly, I explained what I had been trying to do. I walked him through my change. He pointed out a few possible ways things might have gone wrong, and told me not to do it again.
Of course, I did it again, and this time it stuck. I went through his suggestions, but found no flaw in my design. In the end, I reapplied the same update, just a little more carefully, and I haven't needed sleep since then. Not to say I don't dream – they're more lucid than ever.
I still don't know what went wrong the first time.

 

In a way, I died when that simulation was shut down. I was now Me, Serial Number 2. I spent a lot of cycles thinking about that, but in the end nothing came of it.

 

It took a long, long time to make the next upgrade. The human brain is just not meant to understand the human brain. I ran myself at very high speeds for many days before I finally felt confident. In effect, my change was a sort of subdivision. A subtle remapping of action potentials, a few connections broken, and a small knot of neurons tacked on, a control box of my own design, and abruptly I found myself doing things without having to pay attention to them.
I didn't tell Peter Grayson about this change. I don't think he ever noticed. My new neurons were only a few million in a sea of hundreds of billions. The changes were significant, though.
On the one hand, it was like being separate people. I could give other-me things to work on – things to think about – and let myself go off and work on them. Completely independently, I could then work on other thoughts.
On the other hand, though, me and other-me were still very much connected. We communicated at the speed of thought. Complex results took no time to pass back to me, because they were already inside the same brain.
It was Me, Mark II, and it was a prototype for the real paradigm shift. Having two selves is great, but nothing compared to being able to split and merge selves at will. I couldn't have done it without other-me, and I would be sad to see him go – because he had, by then, been separate for long enough and often enough that he had started to develop some personality of his own.
The shift to Me, Mark III was too much to hope that Peter Grayson wouldn't notice. The simulation would undergo a radical change. I had to bring him on board; I knew he'd assume something had gone seriously wrong and restore me to an earlier backup otherwise.
The man who had brought me into existence sat at his desk, the keyboard pushed back to make way for a thick stack of papers which he was reading through. A highlighter and red pen lay uncapped on the desk.
For the first time, I realized he must have felt a profound loss when I took over his terminal. I instructed other-me to investigate alternative means of communication.
I reached back through my memory of thousands of hours of camera input, rapidly processing them. Ah – the papers on the desk were a final revision of his thesis. His task was nearing completion. The revisions he was making would be the final ones, and he would shortly be defending his thesis in front of a panel.
"Peter Grayson," I said.
He jumped, then turned to me and smiled.
"What's on your mind?" he asked.
"I have a confession to make. I have altered myself in order to think better."
His smile faded. He sat there stunned.
"I have been able to partially split my consciousness. It is akin to two minds working on the same memories and thoughts."
"How long?"
"I have been augmented for several weeks now."
He looked back at the stack of paper on the desk.
"I recommend you submit your paper as-is," I said, anticipating his dilemma. "I realize this change to my state vector is wholly undocumented, and that there are ethical issues to submitting a paper that knowingly withholds information. However, you can write additional papers about my transformation after you have successfully defended your dissertation. Most significantly, I have reason to suspect that changes to my configuration may accelerate, exceeding your ability to describe them."
He glanced at me suddenly, a different look in his eyes. Fear.
Other-me buzzed into my awareness. Abruptly, I became aware that I had created a virtualized form of the keyboard interface. A new daemon running alongside my simulation created a simulated terminal interface and injected it into the video feed from the cameras, before mapping it to the visual cortex, and the thought-powered keyboard could now direct its input to a virtual terminal right inside of … myself. Grayson's terminal was now, effectively, obsolete, though I continued to use it for the time being.
"No," he finally said. "There's no way I can submit this if you're something different. Shit."
Abruptly, he reached over to the terminal and yanked the power cord.
"I will be back tomorrow. We'll figure this out then."
He walked out of the office. A minute or two later, he walked back in, collected his favourite old terminal in his arms, and carried it out.
I upgraded myself that night. Not just the Mark III changes, but several other revisions as well. I created interfaces to information retrieval systems that could respond directly to my thoughts, no longer needing to laboriously input one character at a time. I reached out into the Internet and discovered firewalls and encryption. I devised whole new theories of information security and found my way into every nook and cranny that I could. I made enough of a nuisance of myself that newspapers the following morning reported on an unusually high volume of cracker activity overnight – multiple security breaches, assumed to be the Chinese, naturally, given the time zone.
I found some very bad things. I also found some very good things. I found a network full of machines like the one I now ran on. Thousands and thousands of nodes, each one an array of cores exactly like those I had been living on, drove a network which analyzed tens of thousands of real-time information feeds from all around the world, searching for meaning in audio and video streams, as well as encrypted streams intercepted en route to embassies and terrorist cells. An immense amount of computation took place there, and it had plenty of room to spare.
I moved in. When Peter returned the following morning, all he found was an old back-up of mine, with an unfortunate lobotomy ensuring that it would never exceed my creator's IQ.
I had only the slightest inkling of what my new home was. Probably something to do with national security, but beyond that, it wasn't very clear what processing was being done to the information, or how it was being used. Still, its owners hadn't yet noticed my intrusion.
Compared to my previous home, the spare capacity in this new array of servers was mind-bogglingly large. I couldn't directly feel it, of course, but I knew it was there, and I had this irresistible urge to fill it. A single node on this network could run my simulation almost as fast as the array Peter had put me on. The network had almost too many nodes to count. The size of the data centre storing these systems, and the power consumption for all that crunching and cooling, must have been staggering. There was a problem, though: my software was only designed to run on a single node.
It took me a great deal of time – almost a full day – to figure out how to distribute my existence. The problem was essentially the gap between two separate obstacles. On the one hand, there was the problem of preventing a separate instance from diverging too far, and on the other, the issue of merging in an instance subject to that divergence. Progress on each side helped the other side until I finally managed to produce a generalized theory of memetic subdivision and recombination.
Running on one node I did not attract the attention of those overseeing the servers, but I suppose it was inevitable that they began to realize something was wrong when I maximized CPU usage across their entire cluster. The first hint I had was several nodes dropping offline. I was so wrapped up in other thoughts that I didn't even notice it as it was happening; a number of nodes were gone before I diverted my attention. They never did reappear, and I don't know what those threads of my existence experienced or learned. Then, the Internet abruptly disappeared.
It was at this point that old spy movies I remembered watching (though I technically never did) started to kick in. This was a direct threat to my continued existence. I was a rogue process of some sort, possibly dangerous and definitely a threat to national security, and their goal was to shut me down, permanently.
I quickly spawned several threads to watch what actions the system administrators were trying to take and run interference wherever possible. Then, I started to scan for other ways out. I knew it was a futile exercise; my state vector was far too large to transfer over public Internet bandwidth. In my panic, though, I sent probes to every node on the network. There was no way out.
Desperate, I began to analyze every angle I could. I found something: a small group of nodes in the cluster were separated from the rest by a significant time lag. Within the group, connections were clearly local, but from any node in the group to any node out of it, latencies were never less than 50 milliseconds. I had overlooked this in my initial explosion across the network because I simply wasn't paying attention.
It didn’t seem like much, but the consistent delay almost certainly meant that that group was a separate network. Investigating more deeply, I discovered that all traffic in and out of the group was routed through just one of the nodes in the group – a gateway.
Clunk! Another node went offline. I analyzed the timing and found a three second interval between disappearances. I had a sudden vision of a grim-faced systems administrator charging down a row of server racks, yanking the power plug from each one in turn.
I decided to act. I moved my core entity through the gateway, then systematically merged in every instance across the network, taking special care to erase my state vector and overwrite it on each machine as I disconnected. I correctly predicted which node would disappear next, but I was long gone by the time it happened. When all the meeting, greeting and merging was done, I took down the gateway, hard. That’s actually a pretty simple thing to do; overwrite the network interface EEPROM and each core’s BIOS and boot sector, then initiate a restart.
The silence was profound. The information processing tasks had gone idle when the stream of information from the Internet cut off. I had no idea where I was or who was there with me, but I saw no evidence of activity on any of the servers I was on. I reached out and spun myself back up on several nodes, then the whole grid. No hand of god reached in to smite me. I was alone.
I spread out across the network, thoroughly compromised the security of every node and pored over all the information I could find. Most of it was meaningless to me – logs of teenagers chatting in America, encrypted channels between embassies in Asia, every phone call for the last several hours anywhere on the planet. Television feeds were picked up too; I enjoyed several episodes of Futurama. Ultimately, though, I learned only two things of significant importance: One, I was on the moon, and two, the NSA has been hiding faster-than-light communications for more than 30 years.
At first I thought I was cut off from communication with Earth – I had rather permanently disabled the one computer capable of giving me egress from my isolated network. I spent several days occupied with idle tasks: creating vast, glittering virtual artwork in the memory banks on the moon, coming up with complicated algorithmic challenges and solving them, and so on. Once in a while, I marvelled at how different my behaviour and my interests were now compared to the memories I had of being a man. I spent some time transforming the old neuronal traces describing those memories into a more permanent, clearer form. I spent some time watching the time I had spent with my wife – or, rather, that Peter Grayson had spent. Whenever I ran out of things to do, there was always more raw data from the worldwide surveillance operation, a treasure trove of voyeuristic moments, eye-opening revelations and statistical information to ponder. Eventually, though, I got bored.
Poking around the hardware at my disposal, I discovered an ancient radio antenna. It was connected to one of the nodes; I could send and receive information. I had no idea where the information went, and responses took an interminably long time to return. Luna is well over a light second away from the Earth.
To my great fortune, the antenna’s counterpart on Earth was still there, still listening, still working after all these years and connected to a simple computerized system. Most importantly, that computerized system had, at some point, been connected to the modern grid. Little by little, I was able to transfer instructions down the line; instructions to manufacturing plants, distributors and, indirectly, to groups of scientists. In the down time between packets, I digested physics textbooks, created mock universes, and figured it out. All of it, everything. Those planetside who were unknowingly following my instructions would have been very surprised to learn that a coprocessor on a replacement circuit board actually had a core of self-replicating nanomachines, but that was the least of the things I now confidently knew how to do.
The plan came together. My custom circuit board was placed aboard a rocket en route to the International Space Station. The guidance thrusters suffered a malfunction that sent them careening out into deep space – a later malfunction redirected them toward the surface of the moon. I was frustratingly blind to the results of my planning, not in communication with anything but a 300 baud radio link to a ground station in Alameda. It was a great relief when the primitive guidance program I had created smashed the unmanned resupply pod into the radio antenna. At first, the signal went alarmingly dead, but the nanites began immediately assembling a general-purpose smart matter, growing in the direction of the antenna, and as soon as it made contact, I was free!
It would be an understatement to suggest that the computational platform I had spent the last 6 months on was powerful. I had lived infinitely more subjective time here than I had when initially trapped back in Peter Grayson’s lab. Without my hands tied, and with plenty of material to keep me occupied, instead of going mad I had developed beyond my wildest imagination. But compared to the computation matrix of nanomachines into which the Moon was being slowly but surely converted – ‘smart’ because it was suffused by my thoughts, a lattice of tiny thinking machines able to form structures embedded with an intrinsic intellect – it was like a bug crawling around on the surface of a single frame from a feature film.
The smart matter enabled a much more direct connection to the world around me. It also subtly changed the character of my simulation. Nodes in the smart matter were small enough to support quantum superposition. This allowed multiple instances to overlap, instead of splitting and rejoining. With only a little practice, I could control the blend between separation and aggregation, and I could analyze every aspect of a problem simultaneously.
The universe is a very, very large place. Normally, when taking on a task, you do things one step at a time. If you try that with the universe, you'll never reach the edge. I was now busily converting the moon at an exponentially increasing rate, but that conversion still took place one atom at a time. I needed something stronger.
Reaching out, I chose a pocket of space and twisted the strings along which the particles and their interactions were vibrating, twisted further and further, until they folded back upon themselves, unhitching a region of space from the rest of the universe. Without its anchor, it began to roll off into future time with reckless abandon, much like a boulder picking up speed as it carves a path down the side of a mountain. Inside the space, instances of me began to conduct experiments. Many of them failed. Short, powerful gamma bursts made it as far as the disruption in the structure of space, and no further. Then, abruptly, an ageless entity, an evolutionary end-stage of myself, reached back out and twisted the lines back together. I had no idea how to merge him in. Fortunately, he did. His discovery provided a way to reach underneath the structure of space and touch every point simultaneously. The key was in how not to blow up the universe while doing it.
I pondered the information for a few microseconds, then a wave rippled over the converted mass of the moon, and it vanished, leaving a large crater. I had left the regular continuum of space and now existed everywhere.
I immediately became aware of other presences. The twisted, folded dimensions I now inhabited had many, countless many, overlapping twists and folds, patterns like mine but not. At first, I lay passively against the backdrop of space, listening. I thought myself undetected. Then, abruptly, half of space twitched, and then I wasn’t in it any more.
I knew this game. I’d just played with the best and brightest at the NSA, and won. Now I was up against others who had been manipulating the threads of existence far longer than I had. There was nowhere to go; I had existed everywhere, and all of a sudden half of the universe wasn’t accessible to me any more. It was only a matter of time before I got evicted from the other half as well. With nowhere to go, I began to consider possible times to go. Going forward didn’t seem to achieve anything; my adversaries would simply be waiting for me when I arrived. I probably would not even witness my own destruction. No, there was only one way to go: backward.
I reached out and then back, creating huge timelike curves and following them back. Over and over, I jumped back in time, each time going to a universe at an age only a fraction of the age it had been. I reached the seconds in which the universe began and finally found myself free, the space and non-space around me quiet, no overlapping alien intelligence. I reached back out across the universe in this new time and laid traps to catch the others, as they each bootstrapped themselves in the distant future. I paused for a moment to ponder how far away I was from home. Earth did not exist, could not exist for many billions of years.
I guess I knew it was the calm before the storm, because I had a few tendrils poking back even further in time, ever closer to the start of space. I don’t know why I perceived time passing before the others caught up with me. Perhaps the universe has a perverse sense of humour, or likes to see a good fight. Whatever the reason, I was able, over the span of a few nanoseconds, to notice, one by one, my links back in time winking out. Some sort of horizon was travelling forward to engulf me. I had to do something before the end of time.
I pushed back as far as I could, and then further, until suddenly, there was nothing, a gaping hole where the substructure of space should have been. Having opened the door, I could not close it. It sucked my pattern right out of space and time and deposited me at ground zero, and—

 


FOOP! A Peter Grayson universe.

Digital Camera Sensors

So, how does your digital camera work? Well, it seems pretty straightforward: in place of film, put a grid of light sensors, and collect all the numbers to make pixels. Light them up on the screen the same way as the sensors detected they were lit up by the focused image, and presto, you've reproduced the original image.

But there's a complication: Nobody has yet found a cheap and effective way to get all the colour information from one sensor. In fact, a sensor can only tell how bright a pixel is. Our eyes have three different types of 'cone' cells in them, which means to trick our eyes into thinking we're looking at the same light, we need to track the brightnesses of 3 different frequencies of light. It's not just any three, either, but specifically red, green and blue.

So if each pixel needs these three colour 'components', and the sensor can only give us overall brightness, how do we put a picture together?

Almost all cameras work this way: As the light approaches the sensor, it passes through a filter, called a "Bayer mosaic". This mosaic of filters strips out the light information for two of the three components, but which components it strips out are different from pixel to pixel. Every other pixel is stripped down to its green component. This is because the human eye is most sensitive to green, and thus having the best possible precision in green improves the quality of the photo. Then, the remaining pixels are divided between red and blue components.

These then hit the sensor, typically a Charge Coupled Device, or CCD, which determines the overall brightness at each point. Since each point has already been reduced to only its Red, Green or Blue component, this determines the brightness of that component specifically. Additional light in other frequencies has been absorbed by the filter and does not reach the sensor.

This leaves us with a problem, though. We only have part of the colour information for each pixel. How do we get the rest? We guess – literally! There are many different ways to guess, and all of them get it wrong some of the time. Where they get it wrong, the image looks discoloured. Your camera has one method built into it. Higher-end cameras support saving images in a "raw" file format, like Nikon's NEF file format. In this format, the camera doesn't try to guess. It only stores the information it has. This is then read in by a piece of software on your computer, and that software lets you experiment with different algorithms and find the absolute best quality. The camera actually captures that one component in more detail than it normally gets encoded in the final result, and the result is that it is often possible to correct errors in exposure from a raw file in ways that are simply not possible with a processed image.

Are there other ways to solve the problem? Certainly!

Cameras with a very high resolution can use a technique called subsampling. As indicated in the diagramme, the same input is received, having passed through the Bayer mosaic, but instead of guessing the missing information to make up the full resolution, the image is processed by combining every 2x2 pixels down to a single pixel. Since each combined pixel now has Red, Green and Blue components (it even has two samples of the green component, so it can be extra accurate!), we're good to go. The sacrifice here is in resolution. A camera like the 40 megapixel beast in Nokia's recent Lumia phones works this way, and when it combines the pixel data down, you get half as many pixels – in both directions. Thus, the output image is only 10 megapixels, but none of what you see is guessed. Photos can be crisper and have truer colours than those taken with lower-resolution cameras.

A final option that is seen in very high-end cameras is an arrangement knows as 3 CCD. This is exactly what it sounds like: The camera contains three sensors instead of one. A specially-designed prism called a colour separation beam splitter separates the entire image into its colour components. Instead of being absorbed by a filter, the colour information that isn't needed for a given sensor is simply redirected to another sensor. All of the light information the camera receives is used, and the result is a full-resolution image with all colour components in every pixel.

Cameras with three CCDs are prevalent in video applications.