The C♯ programming language is full of nice features that make code more concise and easier to read. Things like 'foreach' statements and 'using' blocks are examples of these. It also has core language features that are necessary to provide the basic level of functionality needed to write a program in the .NET environment, such as basic flow control like 'if' and 'while' and error handling with 'try', 'catch' and 'finally'.
Some of these features are implemented directly. When you write an 'if' statement, the compiler translates it to a conditional branch. For instance, this:
..becomes a somewhat more verbose equivalent to this:if (condition) truePart(); else falsePart();
The compiler doesn't translate your 'if' into a series of 'goto's, though. The flow control is the native level of the code in its conceptual representation. Your code is represented as a directed graph, where each node is a block of code that can only be run one way (no "branches", a sequential sequence of operations), and then nodes are joined to one another by graphs that say, "if this condition is met, do this node next". Statements like 'if', 'while', 'for', 'switch' and so on get translated directly to this form. Similarly, 'try', 'catch' and 'finally' are actually represented by metadata in the final compiler output that says, "If an exception happens between these two byte offsets of MSIL code, then jump to this other byte offset to handle it, and this fourth byte offset is the 'finally' block that needs to be executed whether flow control exits the 'try' range via an exception or a 'leave' opcode."if (!condition) goto _else; truePart(); goto _endif; _else: falsePart(); _endif: ;
(I mentioned MSIL code in that last metaphorical sentence -- this is the "machine code" upon which .NET operates. In order to be truly multi-platform, code written for .NET doesn't actually get translated for the CPU -- not immediately, anyway. MSIL is an intermediate language, modelled after Java's bytecode, which is designed to make it relatively easy to quickly produce efficient machine code for a variety of actual hardware. Thus, the same compiled EXE can be used on PCs, tablets, phones, PDAs, even Macs. :-)
So, some features are core to how the language gets translated to the underlying machine code. These are called primitives. Other features, though, are implemented in terms of these underlying primitives. The 'using' statement, for instance, is really a 'try' block with a 'finally' that always calls .Dispose() on the object being "used". Similarly, the 'foreach' statement is really a 'while' loop that calls the .MoveNext() method of an enumerator and, as long as it gets a next element, retrieves it with the .Current property before running the body of the loop. The 'while' statement is inside of a 'using' block to ensure that the enumerator gets disposed when it's done.
I'm going to call these features that are based on other features composed.
Some of these composed features are so complicated, the compiler needs to synthesize entire classes to support their operation. Most features that work this way are more recently introduced into C♯.
The e-mail I initially responded to was an educational thread about one of these more advanced composed features called "closures". To be sure, C♯ does not have true lexical scopes, but it does nevertheless allow the implicit sharing of variables between scopes that have the superficial look & feel of first-class functions being passed around. As a practical tool, it's still very useful, even if it doesn't satisfy language purists.
The question that was asked is as follows:
Hey guys, I told the group today that I’d send out a better example for variable scoping when using delegates, so here it is. I’m sending it to all developers because it’s useful for everyone. Without running this code, what will be the output? Why? What’s the scope of the variable x?public class Program { private static void Main(string[] args) { var resultOfFoo = Foo(); resultOfFoo(); resultOfFoo(); resultOfFoo(); }
public static Action Foo() { int x = 0; return () => Console.WriteLine(x++); } }
To understand this, I find it simplest to understand what the compiler does when it sees this kind of code.
Roughly speaking, when interpreting the code, the compiler turns this (repeated for alternative highlighting):..into this:public class Program { private static void Main(string[] args) { var resultOfFoo = Foo(); resultOfFoo(); resultOfFoo(); resultOfFoo(); } public static Action Foo() { int x = 0; return () => Console.WriteLine(x++); } }
When C♯ was first released, it didn't have support for the type of construct shown in the upper listing, where a function is apparently created and returned directly in the last line of the "Foo()" method. Using this trick of throwing together a class to hold the scope of the variable (and the method!) allowed them to implement this new language feature in a way that didn't require any changes to the runtime. If you're writing a plug-in for a program written back before .NET supported this feature, you can still use this feature in your implementation; your method looks the same as if it were written out the long way, and can be called by code that has no knowledge that this feature ever existed.public class Program { private static void Main(string[] args) { var resultOfFoo = Foo(); resultOfFoo(); resultOfFoo(); resultOfFoo(); } public static Action Foo() { VariableScopeClass scope = new VariableScopeClass(); scope.x = 0; return (Action)scope.AnonymousMethod; } private class VariableScopeClass { public int x; public void AnonymousMethod() { Console.WriteLine(x++); } } }
Another feature that involves this sort of translation, though one that is considerably more involved, is the definition of enumerators as coroutines. To be sure, as with the not-quite-lexical-scopes, these are not-quite-coroutines, but they are close enough to be a useful tool. The idea behind a coroutine is to have two methods running simultaneously, taking turns (i.e., not concurrently), and each method passes control to the other thread by what it thinks is a method call. When it makes that call, the other function sees a call that it made earlier return. Eventually, method #2 will make a call back to method #1, which has been patiently waiting for its function call to return, and, mirror image, method #2's call becomes method #1's return. Kind of wacky stuff to wrap your head around!
The way C♯ has exposed this functionality is designed to be somewhat easier to wrap the head around. Instead of two general-purpose methods bouncing back and forth, swapping out their stacks and generally causing mayhem (if not in the execution environment, then in the brains of the programmers having to deal with them), C♯ models one half of the coroutine as generating a sequence of objects. To generate the next object in the sequence, you use the statement 'yield return'. Conceptually, you can simply consider your function as continuing execution after the 'yield return', doing one after another until finally the function completes execution and returns.
Enumerations in .NET, though, are a form of "lazy evaluation". The caller has an object representing a sequence of objects, but it only sees one of them at a time. When it's done with the current object, it probes to see if there is another object by calling 'MoveNext()'. If MoveNext() returns true, then there is another object waiting in the 'Current' property, and if it returns false, there is not. How is this compatible with the idea of a method whose execution straightforwardly produces the entire list?
The C♯ compiler is doing a bit of rewriting on your behalf. The example here is a bit longer, and I ran out of nice colours to use for the highlighting, so my apologies in advance for any eyesight damage that may occur due to the highlighting.
Before I show how the enumerator itself is generated, I'll show how the caller is generated, because the majority of C♯ code never directly refers to the 'MoveNext()' method and 'Current' property I've been talking about.
To access the sequence of objects from an enumeration, you write code something like this:
The compiler pretends you've written code like this, which is semantically identical, a reduction of the 'foreach' compound feature to its underlying implementation:static void Main(string[] args) { foreach (string str in TestEnumerator(true)) Console.WriteLine(str); }
In turn, reducing the 'using' compound feature to the true underlying primitives, the compiler treats this code as though it were this:static void Main(string[] args) { using (var enumerator = TestEnumerator(true).GetEnumerator()) { while (enumerator.MoveNext()) { string str = enumerator.Current; Console.WriteLine(str); } } }
This is the code that, finally, gets turned into MSIL for execution. You can see here the way a conceptual stream of objects is turned into a discrete sequence of function calls, each one retrieving the next in sequence. A key point here is that you could, at any time, decide not to call MoveNext again, and no further elements would be retrieved from that enumerator. Let's take a look at what the TestEnumerator method might look like:static void Main(string[] args) { var enumerator = TestEnumerator(true).GetEnumerator(); try { while (enumerator.MoveNext()) { string str = enumerator.Current; Console.WriteLine(str); } } finally { if (enumerator != null) enumerator.Dispose(); } }
This looks like a straightforward method that will run through and generate a sequence of objects. This isn't compatible, though, with the idea of a MoveNext() call that may or may not even take place. To support the MoveNext() mechanic, and a Current property that holds its value between calls, some sort of "stop-and-go" mechanism is clearly needed, and that is exactly what the compiler supplies. The above code, which uses 5 different highlight colours twice, gets translated into something rather like this behind the scenes:static IEnumerable<string> TestEnumerator(bool bigger) { yield return "starting"; if (bigger) { yield return "BIG"; yield return "GER"; } else yield return "smaller"; for (int i = 1; i <= 10; i++) yield return "iterating: #" + i; yield return "finishing"; }
Wow, that's a lot of code! When this particular type of code happens to be exactly what you need, it's very nice to have a compiler that can interpret the 10-line version using 'yield return' and produce this in an automated way, no risk of errors, typos or silly bugs.static IEnumerable<string> TestEnumerator(bool bigger) { var ret = new TestEnumeratorStateMachine( TestEnumeratorState.ParentIEnumerable); ret.bigger = bigger; return ret; } enum TestEnumeratorState { ParentIEnumerable = -2, // means that the IEnumerator<string>.MoveNext() method won't do anything for this instance Interrupted = -1, // means that flow didn't leave the MoveNext() method the expected way, and the machine is in an error state Initial, AfterStarting, AfterBIG, AfterBIGGERorsmaller, TopOfForLoop, BottomOfForLoop, AfterForLoop, EndOfEnumeration, } class TestEnumeratorStateMachine : IEnumerable<string>, IEnumerator<string> { public bool bigger; public TestEnumeratorStateMachine(TestEnumeratorState initialState) { _state = initialState; } private TestEnumeratorState _state; private string _current; private int i; public string Current { get { return _current; } } public bool MoveNext() { switch (_state) { case TestEnumeratorState.Initial: _state = TestEnumeratorState.Interrupted; _current = "starting"; _state = TestEnumeratorState.AfterStarting; break; case TestEnumeratorState.AfterStarting: _state = TestEnumeratorState.Interrupted; if (this.bigger) { _current = "BIG"; _state = TestEnumeratorState.AfterBIG; } else { _current = "smaller"; _state = TestEnumeratorState.AfterBIGGERorsmaller; } break; case TestEnumeratorState.AfterBIG: _state = TestEnumeratorState.Interrupted; _current = "GER"; _state = TestEnumeratorState.AfterBIGGERorsmaller; break; case TestEnumeratorState.AfterBIGGERorsmaller: _state = TestEnumeratorState.Interrupted; i = 1; // for loop initialization goto case TestEnumeratorState.TopOfForLoop; break; case TestEnumeratorState.TopOfForLoop: _state = TestEnumeratorState.Interrupted; if (i <= 10) { _current = "iterating: #" + i; _state = TestEnumeratorState.BottomOfForLoop; } else goto case TestEnumeratorState.AfterForLoop; break; case TestEnumeratorState.BottomOfForLoop: _state = TestEnumeratorState.Interrupted; i++; goto case TestEnumeratorState.TopOfForLoop; case TestEnumeratorState.AfterForLoop: _state = TestEnumeratorState.Interrupted; _current = "finishing"; _state = TestEnumeratorState.EndOfEnumeration; break; default: // Handles states ParentIEnumerable, Interrupted, and // EndOfEnumeration. Returning false here tells the caller // that Current is now indeterminate and there is nothing // further in the sequence. return false; } return true; } public IEnumerator<string> GetEnumerator() { var enumerator = new TestEnumeratorStateMachine(TestEnumeratorState.Initial); enumerator.bigger = this.bigger; return enumerator; } void IDisposable.Dispose() { // This would handle any 'using' blocks in the original // method. If a 'yield return' occurred in the middle of a // 'using', we'd have an object waiting to be .Dispose()d. // Continuing the enumeration and letting it flow out of the // 'using' block would achieve that, but if the caller // decided to stop enumerating and dispose the enumerator, // it'd still need to be cleaned up. That would go here :-) } // IEnumerable<string> and IEnumerator<string> both extend the // older, non-generic IEnumerable and IEnumerator interfaces. // So, to implement the generic methods, we also have to supply // non-generic methods. System.Collections.IEnumerator System.Collections.IEnumerable.GetEnumerator() { return this.GetEnumerator(); } object System.Collections.IEnumerator.Current { get { return this.Current; } } void System.Collections.IEnumerator.Reset() { throw new NotSupportedException(); } } }
As you can see, each section of the code between 'yield return' blocks (even if it's empty, like between 'BIG' and 'GER') becomes its own 'case', so that a subsequent call to 'MoveNext()' can jump straight back there. Also notice how every case block starts off by setting the state to 'Interrupted'. Before returning, it sets it to the actual next state. If an exception occurs, though, the state is left at 'Interrupted', and henceforth, the 'switch' block always jumps to the 'default' case that returns false: Nothing more to come from this enumeration.
(The more astute reader may note that the “TopOfForLoop” state is not externally reachable. As such, it doesn’t need to set the state to ‘Interrupted’; it will already be in that state when it gets there. I’ve left it in for consistency. The actual compiler output does not use a separate state in this way, but I found it instructive, as it mirrors the conceptual flow control of the bits in a “for (init; cond; iter) stmt;” block: “init; top: if (cond) { stmt; iter; goto top; }”.)
I should mention that I've omitted certain aspects of the true compiler output. For instance, in both this case and the first example with the anonymous method, the compiler uses names that aren't valid C♯ identifiers (but are valid in MSIL). This guarantees that there's no chance of you typing up a variable of your own that unwittingly steals the name the compiler wants to use. The true enumerator code also has a small optimization in it, where if GetEnumerator() is called from the same thread that created the object in the ParentIEnumerable state, it reuses the same object, transitioning it to the Initial state in the GetEnumerator() call. (Why the same thread? Simply to ensure that there aren't two threads doing it at the same time. It doesn't matter which thread it is -- if you only let one thread do that transition, then you can't ever end up with two threads holding a transitioned initial object.)
These transformations, reductions from compound features to primitive features during the compilation phase, are taking place countless times in our codebase and any other piece of complex C♯ code.
