def euler(x); cos(x) + i*sin(x); end

Is Node.js Better?

2012-04-09T00:00:00-07:00

NOTE: This is essentially a transcript of (though mostly written before) my JSConf 2012 talk, “Is Node.js better?”. It’s long, I know. There is no tl;dr, sorry.

Is this better than that?

This is a question that we confront constantly. Numerous times a day, in fact. And sometimes a lot rides on the answer. Is this job offer better than that one? Is this car better than that one? Is this home for my ailing mother better than that one? Is this billing system better than that one? Is this tax plan better than that one?

This question is everywhere. We cannot escape it. Yet for the frequency with which we confront it, our approach to resolving the question is quite likely not objective. Which is to say, we are probably not answering the question in a way that is mostly likely to benefit us. Many used car sales, software system sales, and political campaigns, to mention a few things, profit from our incompetence answering this question. We’ll consider some of the reasons why that is true later. But just recognizing this is rather distressing.

But we’re not just asking, “is this better than that?” in general; we’re here at JSConf and this talk is about Node.js. Why? You probably know that I’m not a notable member of the Javascript community. I have not authored a single JS library. I’ve never given a talk at a JSConf. Why am I here? And why am I talking about Node.js?

I met Chris Williams at CodeConf 2011, organized by Github. Chris sat down next to me during the Node.js talk by Ryan Dahl. I found the talk fascinating because much of it focused on improving the non-blocking I/O facilities in V8 for both Windows and Unix-ish platforms. This was something that we have a lot of interest in for the Rubinius project and I had then recently started porting Rubinius to Windows.

At the time, I had no idea who Chris was. But we chatted a bit and at the break, Chris stood up and said, “I’m going to get some more beer, would you like some?” Holy shit. That’s why I had been smelling beer for the last 45 minutes! Chris had a coffee cup full of beer. Hilarious. One person’s moderation is another person’s excess.

I declined the beer then but we exchanged contact info and Chris put me in touch with folks from Microsoft who were assisting with getting Node.js running well on Windows. Chris also informed me of the upcoming JSConf that was being held in Portland, where I live. It was too late to get tickets but I was able to attend the opening party and made a point to hang out as much as I could with the other attendees at conference events. At the time of CodeConf, I was merely curious about Node.js, but I didn’t have much of an opinion.

Fast-forward almost nine months. I was seeing a flurry of tweets about Node.js. This got my attention, in part because of the people who where tweeting about it. I use Twitter as one of my main sources of information about developing technology. The more tweets I saw, the more uneasy I felt. Finally, one morning I posted this:

My intent was not to be antagonistic. Rather, my challenge was focused on finding out what sort of problems people were solving with Node.js. But any challenge, even a sincere one, carries a connotation of aggressiveness. And it is a fact of human nature that when pushed, people tend to push back. Some of the responses to my tweet attempted to share knowledge but others were understandably of the genre, “if you’re looking for a fight, you’ll find one here.”

It wasn’t long before Chris DM’d me and basically said, “What’s up with this ‘challenge’? You are working on important stuff, why are you wasting time on this?” I answered that I was genuinely interested in understanding why people are using Node.js. We exchanged a few comments over a several days and Chris asked if I was willing to do a talk on the subject. So that is how I got here, talking about Node.js at JSConf. It is a honor and privilege to be here and I thank Chris and the conference organizers, as well as all of you.

So who am I? I’ve been working on the Rubinius project for the past five years, and four of those while employed full-time by Engine Yard. I started the RubySpec project as part of my work on Rubinius. Through my work on Rubinius, I have learned a lot about compilers, virtual machines, garbage collectors, and importantly, concurrency. One of the notable points about Rubinius is we have removed the global interpreter lock (GIL) so that multiple native threads can run Ruby code in parallel on multi-core or multi-CPU hardware. We see this ability as vital to the success of Rubinius.

Now that you know how I came to be speaking at JSConf and you know a little about what I do, the question remains, “Why am I speaking at JSConf? Why do I care about Node.js?” The answer to that will take us on a journey through some interesting territory. I only ask that you suspend prejudice and follow along. If you ultimately disagree with me, there is nothing wrong with that.

Organizations tend to perpetuate the problem they were created to solve.

I don’t remember exactly when I was introduced to this idea, but it had a significant effect on me. There are a lot of difficult problems to solve in the world and one of the first things we tend to do is create an organization that is dedicated to some aspect or another of a solution. Notice that there is a difference between people organizing to solve a problem and an organization. As soon as we have an organization, it will tend to take actions to perpetuate itself. Of course, these actions are really the decisions of people in the organization. The organization has no effective power independent of the people who comprise it, yet the organization as a whole is a system, and will tend to exhibit life-preserving actions. If an organization exists to solve a problem, and that problem is solved, the organization ceases to have a reason to exist.

Every time we organize for any reason, there is a tendency for structures to solidify. The rigidity of those structures tend to inhibit movement and change as circumstances change. Therefore, a vital force in social organization is the force that opposes established order. There is a name for this force: subversive, something seeking or intending to subvert an established system or institution. People who participate in subversive activities are called subversives.

There is another aspect to this that is important. People tend to be in one of two groups: those who fear change and those who create change. When we think of “change”, it usually has a positive connotation, and “subversive” has a negative connotation. But they are often the same thing. Change can be positive or negative. But as it has been said, “there is only one constant: change.”

Javascript has been an amazing technology wielded by people who have radically and fundamentally changed how we interact on the web. This is fascinating and tremendously valuable for societies all over the world. So part of the reason why I’m speaking here is that people enthusiastic about Javascript are people who are creating change. I want to acknowledge, assist, encourage, collaborate with, and influence people who are actively working to change the world for the better.

This talk is about conflict resolution. Conflict is inherent in the question, “Is this better than that?”

It’s important to note that any contrast implies conflict. Sometimes people assert that you should be able to promote something without saying anything bad about something else. This idea is impossible to comprehend. If I say, “water is good,” it is inherent, even when implicit, that I’m saying water has value and value is a concept that relies on a context and a standard. There is no concept of value that is not relational. Something is only good relative to what you are judging it against.

Criticism is advocacy; advocacy is criticism. Criticism is also controversy and controversy is entertaining. However, despite the entertainment value, different ways of resolving conflict can have very negative consequences. Usually when there is conflict, we address it with aggression. We talk about having a “shootout”, “fight”, “throw down”, etc. But are fights a healthy, beneficial way to resolve controversy?

Further, when a person comes out as a strong advocate for some technology, people opposed to the technology will hurl the epithet “fanboi” as an attempt to discredit the idea by attacking the person. From the other direction, if we don’t like someone’s criticism, we may call the person a “troll”.

Other times, we will simply attempt to avoid conflict entirely. In Ruby, there is this idea, MINASWAN: Matz is nice and so we are nice. There is nothing wrong with being nice, but why is “nice” contrasted with “criticism”? Are we supposed to never challenge someone because being “nice” is more important and challenging them is inherently not nice? What if someone is doing something that from our experience is not beneficial to themselves or others?

For the most part, this is how I see us dealing with conflict. We either avoid it or we attempt to devalue the person rather than discuss the idea. This is dysfunctional. Basically, we suck at conflict.

How can we improve how we deal with conflict?

I assert that we need to use science. But how? What does that mean? To understand this, we need to look more deeply at how we as humans think. But before we get to that, let’s consider a very common aspect of social organization: the tendency to surrender our own judgment to that of an “expert”.

In 1948, Alex Osborn, who was an ad man at B.B.D.O., wrote a book, Your Creative Power, where he introduced “brainstorming”. Essentially, a group of people toss out solutions to a problem under consideration. The emphasis is on generating as many ideas as possible. A key element of the activity is that everyone is told not to criticize their own or other’s ideas. It was asserted that creativity was too delicate a process to withstand the harsh light of a critical challenge.

Of course, that made a lot of sense to people. Over the years, brainstorming has been used extensively in problem solving. It is still heavily used. The problem is, the method is flawed. The proscription against criticism happens to be wrong.

There were two studies that tested the two main components of the brainstorming methodology for discovering creative solutions. The first focused on individuals versus groups. The result was that people working alone produced not just more but better ideas than the groups.

The second study focused on the suspension of criticism. The subjects were divided into three groups. The first group were instructed to use brainstorming and told not to criticize their own or each other’s ideas. The second group were instructed to challenge and debate one another. The last group was allowed to organize themselves without any instruction in a particular method. The results from this experiment were unambiguous. The debaters significantly outscored both other groups. The act of criticizing other’s ideas causes both the questioner to understand the idea more fully and the person proposing the idea to go more deeply into it.

The important lesson is two-fold. Criticism is an important aspect of creative, intellectual effort. Despite the proscription against criticism having superficial validity, it was rather easily disproved.

Another interesting study related to creativity examined whether there was a correlation between the success of Broadway musicals and how well the team producing the musical knew each other. The measure of team familiarity was named the Q factor. A team where all the members had worked together before would have a high Q factor. A team where no members had worked together before would have a low Q factor. The study found that there was indeed a significant statistical correlation between Q factor and success of a musical. In other words, a certain range of Q factor was a good predictor of success. The value of Q that was most likely to predict success was from a team where most members had worked together previously but some members had not. A team where no one had worked together wasn’t able to communicate effectively enough. A team where everyone had worked together didn’t benefit from an outsider’s perspective challenging ideas.

These results are reported in Groupthink: The Brainstorming Myth by Jonah Lehrer, published in New Yorker magazine, January 30th, 2012. The lesson we can take is that criticism is important to creative solutions to problems and we should be seeking people who are unfamiliar with our favorite language or framework to join us and help our understanding of problems and the solutions we are building.

Another lesson is that there is nothing that prevents a well-meaning and experienced expert from being wrong. One thing that we should all be challenging is “appeals to authority” in decision making. As you can see from the following tweet, Ward Cunningham thinks Node.js is the future of the server side. However, that assertion is most likely not taken by readers of the tweet as a neutral assertion of fact about Ward’s opinion. It is taken as meaningful because Ward said it. Further, it is unremarkable that the assertion is unaccompanied by any evidence with which to help determine the validity of the assertion. Assertions by experts with no accompanying evidence are the status quo in our industry, and that should be extremely embarrassing.

The next part of our journey takes us to one of the most interesting books I have read. Thinking Fast and Slow by Daniel Kahneman (Farrar, Straus and Giroux, 2011) is a book about how our minds work. There are basically two modes that have distinct and surprising features in the way we think. These two modes influence one another.

One mode is called fast thinking because our brains activate an entire network of associations in a sub-second burst of activity. For example, if I display the word “bacon”, you will immediately have numerous memories and associations with bacon coming into awareness. This fast mode is all about pattern matching and the associations activated can be extensive and surprising. In this mode, the brain over-achieves, if you will. It activates far more associations than may be needed. And some associations may be surprising. It is up to the second mode to use those associations depending on the task we are confronted with.

The other mode is called slow thinking and is related to such activities as concentration, judgment, monitoring, and selecting. If I ask, “what is 23 x 47?” Answering this requires deliberate application of a set of tasks to arrive at the answer, unless the problem is one we are quite familiar with, in which case the fast thinking would just produce the answer as an association.

The book is filled with fascinating knowledge about how our thinking works and I cannot recommend it highly enough. It discusses numerous ways in which our thinking can produce erroneous beliefs and ideas. One of the very interesting results is that our deliberate, slow thinking can be easily fooled into accepting a wrong answer provided by our fast thinking mode if we feel at ease. But if we feel anxious, our slow thinking mode will be more critical and less likely to just accept an answer provided by our fast thinking mode.

Reading Thinking Fast and Slow and reflecting on the complex and inter-related behaviors that scientists have been able to discover facts about led me to the following conclusion:

Programming is a behavioral science.

Behavioral science is a very broad category that encompasses disciplines that explore the activities and interactions among organisms through systematic analysis and investigation using controlled observation and scientific experimentation.

There is a distinction between programming and computer science. Writing software is an activity that is mostly centered on human behaviors. Most software will interact with people at some point. Basically, it is written by people, for people, and funded by people. Of course, there are mathematical foundations of computing and algorithms, but those are rarely the most important elements of the process of creating software.

If you are familiar with the many methodologies under the “Agile” umbrella, you know that dealing with changing requirements is one of the most complex aspects of software construction. But note that those changes are almost entirely due to human behaviors. If we build a bridge, an environmental study will determine most of the forces that the bridge design must account for. Further, the laws of physics are quite well established. Engineering the bridge is not a terribly complex activity. However, in software, there is not a good way to clearly establish the constraints for the system we are building. People are responsible for most of the complexity in programming. That is why I assert it is a behavioral science.

What is curious to me is that a typical undergrad psychology student will likely have more exposure to research methods than a typical computer science undergrad or even graduate student. Research methods are activities directed at determining the validity of assertions. They are what enable us to separate knowledge from opinion. Fundamentally, they are related to two aspects of existence: the nature of the world (meta-physics) and the theory of knowledge (epistemology). To learn more about this, I would recommend Research Methods: the basics by Nicholas Walliman (Routledge 2011).

Turning from this general information about applying science to thinking and learning about research methods, let’s look at Node.js again. One of the major assertions about Node.js is that it permits writing efficient web servers that must deal with many concurrent connections. So, let’s look at some basic features of concurrency, including a possible definition of concurrency.

People are selfish, lazy, and easily bored.

I say that without any moral judgment. I think these are all biologically desirable features of organisms. We are selfish because we are responsible for our own well-being. We are lazy so we don’t waste precious and costly-to-obtain energy. We are easily bored because we must constantly incorporate new material to maintain our existence. Of course, each of these attributes has an opposite and any individual will exhibit a combination of these behaviors. But these three attributes are interesting to consider in this context.

Scarcity is a fact of life. Infinity is a mathematical fantasy. There are a finite number of computers.

Combine scarcity with the attributes above about people and we have a reason for concurrency in computation.

When a single CPU with a single core is running, it can basically execute one instruction at a time. Of course, this is a simplification, but it is basically true. If we consider a program to be a sequence of instructions, I₁..I_n, every possible ordering of those instructions is a plausible definition of concurrency. For example, a four instruction program could be I₁, I₂, I₃, I₄ or I₃, I₁, I₂, I₄, etc. Not all orderings are going to be meaningful given the semantics we expect from the program.

Now, consider that the program would be 100% efficient if all the instructions are dedicated to solving the problem for which the program exists. However, if any instructions are dedicated to changing the order of instructions, those would reduce the efficiency of the program by doing work that is not directly related to solving the problem.

Once upon a time, computers solved problems one at a time. The jobs were processed from start to finish one at a time. This is great for efficiency but not great for people. People don’t want to wait for everyone else’s job to finish before theirs. Nor are they willing to do a lot of extra work to make the computer most efficient. So the idea of time-sharing systems was born. Essentially, everyone got a little slice of time to use the CPU. Even though some efficiency was lost, and so everyone’s program would ultimately take longer to complete, the average time to wait went from hours (a whole day, perhaps) to possibly a few minutes. Computers became interactive.

There are only a few mechanisms to re-order the instructions of a sequential program. (Note that we can generalize from the instructions of a single program to the instructions of a set of programs concatenated together. The fundamental considerations do not change.) One way would be to do a little work and then voluntarily yield control of the CPU to another program. At some later time, another program would yield control back to us. This would implement cooperative multitasking. Another way would be for some sort of supervisor to allocate a short amount of time to each program in sequence. This would implement pre-emptive multitasking. Once your time-slice is over, you simply have to wait until everyone else gets a time-slice before you can run again. Yet another way would be to allow a program to run until it performed some action, like writing to the display or reading from the disk.

Each of these mechanisms for interleaving the instructions of a serial program may have different trade-offs, and consequently, different efficiencies. However, there is nothing inherent in any method that dictates it would be more efficient than any other. I consider this a very important point missing from almost every assertion about the uniqueness of Node.js. People are saying, and others are repeating, that Node.js enables solutions that are not possible with other programming languages and frameworks. This is absolutely false.

There is a lot more to understand about concurrency, modern CPUs, and efficiency than we can cover here. However, if you are concerned with the validity of the assertion that Node.js is good technology for writing efficient web servers, then you may want to understand more about concurrency.

Besides concurrency, there are a number of justifications given for using Node.js. Are these justifications valid? I think consistency is important. If I say X is better than some Y because of some reason J, and there is some X’ that is basically equivalent to X and Y’ equivalent to Y, then X’ should probably be better than Y’ for the same reason J. That idea is expressed in symbols below, where the long horizontal line means that the statements above logically imply the statement below the line.

One reason given for using Node.js is that the same language, Javascript, can be used on both the client and the server. If that is true, then why shouldn’t someone use GWT and use Java on both the client and server? Since the JVM works very well on the server, certainly the effort put into GWT could be seen as the equivalent of the effort put into Node.js to make a browser technology work well on the server. At the least, when you hear the “same language” justification, you should be looking for places where the same justification could be made but you don’t find it convincing. If the justification isn’t convincing when you substitute analogous elements, then either the justification is flawed or it is missing something that would differentiate the two situations.

Often, the same language justification for using Node.js is promoted by people who are also bragging about being polyglot programmers. Not only is knowing more than one programming language beneficial, it’s basically required these days. But if knowing more than one language is offered as a positive, why is it important to use the same language on client and server? I’m not saying this argument is invalid, but rather that the justification should examined for consistency.

Another reason given for using Node.js is that there are many Javascript programmers. However, statistically, that means there are many mediocre or poor quality Javascript programmers. And if concurrent applications are so hard, do you want a lot of mediocre or poor programmers building them? Again, many Javascript programmers may be a justification for using Node.js but by itself, the justification is problematic. It’s likely an extreme over-simplification. As such, it has dubious value as more than just a convenient dismissal of someone who challenges the rationale for using Node.js.

It may be that using Node.js is just more fun for many people. This is a great reason to use Node.js. However, it is not one that can be used to evaluate the technical merit of the technology. Which is fine. Recall that I’m asserting programming is a behavioral science. We should be studying and giving adequate weight to considerations that are not merely technical. Humans are using Node.js.

Finally, despite being controversial, I’m going to give some opinions about Node.js based on my experience. The point is not whether you believe me or not. These opinions should be examined critically with the objective of determining their validity using methods suggested above.

More CPU cores is the trend for the future. Typical programs have a huge amount of state. Every library loaded is program state. Running multiple processes demands duplicating the memory used by each process times the number of running processes. As long as the total memory needed to saturate the CPU cores is less than the memory available, using processes should not be a problem. However, we have seen in Ruby, at least, that it is not typically the case. Memory pressure is a constant problem. Solutions to this memory pressure like copy-on-write (CoW) friendly garbage collection are hard and not guaranteed to be that effective. If the mutable part of the heap is small then there is also very likely little shared state that would complicate simply using threads. If the mutable part is large, you get the same memory pressure over time regardless of CoW-friendly GC.

Since it is extremely unlikely that Javascript will add threads to the standard, Node.js will either be essentially single-threaded or programs will use non-standard libraries to introduce threading. This forces Node.js to use process concurrency to achieve parallelism. In this respect, I think that Node.js rejects the multi-core reality that confronts us.

In fact, Node.js is already confronting the issue of process concurrency. In Ruby, we have Passenger for managing multiple worker processes. There is now a similar system for Node.js called Cluster.

Another concern is that an entire ecosystem of libraries and tools must be built up for Node.js. Javascript was confined to the browser of a long time, and that environment is quite impoverished. In building those libraries, many mistakes that have been make in other languages will be made again. That is simply the nature of humans doing work. We make mistakes. Ruby and other languages have repeated mistakes from languages that came before them. I’m not saying that the work that must be done won’t be offset by its future value. I am saying the work must be done and its value should be questioned. There is always an opportunity cost for choosing to do an activity. It is the cost of not doing another activity. In attempting to answer the question whether Node.js is better for our problem, we must consider these issues.

Concurrency is definitely a concern for practically every currently popular programming language. If you have heard that Node.js enables solutions not possible in other languages, I encourage you to look more deeply into the situation. In Ruby, I recommend checking out the Celluloid project by Tony Arcieri. The project’s aim is to bring various concurrency mechanisms to Ruby. He is someone with a lot of experience and accomplishments and I’m confident his projects will advance the state of the art in Ruby and hopefully other languages as well. If you are a Clojure enthusiast, you may want to look at Carl Lerche’s momentum project. Of course, there are many languages and frameworks that handle concurrency and more are under active development. Node.js is not new technology; it’s merely new for Javascript.

Ultimately, the answer to, “Is Node.js better?” is not that important. People will use Node.js even if it’s not the best solution to their problem. Other people will not use it even if it is great for their particular problem. Both results limit the benefits available to all of us. Further, we will continue to confront the question, “is this better than that?” If we improve our ability to answer that question, hopefully we will increase the benefit to us all.

Slides for "Nikita - The Ruby Secret Agent"

2011-10-03T00:00:00-07:00

ED. I’ve converted the slides to PDF and uploaded them to speakerdeck.com.

ED. The except from the 2011 OSCON Twitter talk is below.

A VM by any other name

2009-03-18T00:00:00-07:00

With the recent switch away from the spaghetti stack execution model, Rubinius has also acquired native threads. A big part of understanding something is syncing up our mental model with reality. If you’ve ever tried to explain what an OS is to your mom, you know that can be a challenge. So let’s peel back a few layers and see where these native thread critters fit into Rubinius.

Rubinius is an implementation of the Ruby programming language. One of the bigger components is the virtual machine. But what is that? Unfortunately, virtual machine is a label for a category of software (and maybe hardware) that, well, does a bunch of different things. Virtual machines are often thought of as virtual computers or virtual CPUs. The problem with trying to equate two things is that you look at the one you know about and try to understand the one you do not by looking for analogous structures. Therein lie the seeds of misunderstanding. Since Rubinius has this vm/ directory, we’ve got to try to nail some of this gelatin to the wall.

Thinking about threads running inside a physical computer, you might visualize the relationship something like the following Ruby-ish pseudo code. As the program starts up, it creates a virtual machine.

1 def main
2   vm = VM.new
3   # do some setup
4   vm.run
5 end

Sometime later in the program, you add a new thread, which might be implemented something like this.

1 def Thread.new
2   thr = VM.threads.create
3   # ...
4   return thr
5 end

You might imagine the VM instance having a Scheduler component that would supervise the threads and arrange for running them on one or more processors or cores. In this model, the VM is really like a virtual computer in which all execution is occurring. In other words, the VM is composed of multiple threads of execution.

The point of this mental exercise is to expose the tacit assumptions we might have about our mapping between a real computer and the virtual machine. Now let’s delve into Rubinius.

The main function is located in vm/drivers/cli.cpp. The first thing it does is create an instance of Environment, which is composed of an instance of VMManager, SharedState, and VM. In the Environment constructor, the command line is parsed for configuration options. Then the manager creates a new shared state. The shared state creates a vm. And finally the vm is initialized. During initialization, the ObjectMemory is created. The object memory in turn is composed of garbage collected heaps for the young and mature generations.

Back in main, a platform-specific configuration file is loaded, the “vm” is booted, the command line is loaded into ARGV, the kernel is loaded (i.e. the compiled versions of the files located in the kernel/ directory), the preemption and signal threads are started, and finally the compiled version of kernel/loader.rb is run, which will process the command line arguments, run scripts, start IRB, etc. When your script, IRB, -e command, etc. finish running, loader.rb finishes, main finishes, resources are cleaned up, and finally the process exits.

Whew. The point of this whirlwind tour is to illustrate that VM is a rather fuzzy concept, even though we have a class named VM. Now let’s take a look at how threads fit in.

Rubinius has a 1:1 native thread model. In other words, each time you do Thread.new in your Ruby code, the instance returned maps to a single native thread. In fact, let’s look at the code for Thread.new in kernel/common/thread.rb.

1 class Thread
2   def self.new(*args, &block)
3     thr = allocate
4     thr.initialize *args, &block
5     thr.fork
6 
7     return thr
8   end
9 end

The calls to allocate and fork are implemented as primitives in C++ code. They are short, so we’ll take a look at them, too.

 1 Thread* Thread::allocate(STATE) {
 2   VM* vm = state->shared.new_vm();
 3   Thread* thread = Thread::create(state, vm);
 4 
 5   return thread;
 6 }
 7 
 8 Object* Thread::fork(STATE) {
 9   state->interrupts.enable_preempt = true;
10 
11   native_thread_ = new NativeThread(vm);
12 
13   // Let it run.
14   native_thread_->run();
15   return Qnil;
16 }

The call to allocate creates a new instance of VM as thread local data. The call to fork creates the new native thread. The call to native_thread_->run() will eventually call the __run__ method in kernel/common/thread.rb. Something to note about this snippet of C++ code is the nice consistency between the primitives and the Ruby code that calls them.

We’ve encountered the VM class in two contexts: 1) when starting up the Rubinius process, and 2) when creating a new Thread. We can consider the VM instance to be an abstraction of the state of a single thread of execution, and in fact, state is the name most often given to an instance of VM in the Rubinius source.

As we’ve seen, Rubinius as a running process is composed of various abstractions, including the Environment, SharedState, NativeThread, and VM to name a few. While it is accurate to call Rubinius a virtual machine, it is apparent that concept can cover a fair bit of complexity. But breaking it into parts makes it fairly easy to understand. Let us know what things you’d like to understand better. We have the doc/ directory in the source that we’re (slowly) building out. If you’re interested in contributing, docs would be a great way to help everyone.

Come to the Open Source Bridge conference

2009-03-06T00:00:00-08:00

If you live in or would like to visit beautiful Portland, OR, consider signing up for the Open Source Bridge conference. I will (probably) be giving a talk on RubySpec and Rubinius 1.0. There’s lots of interesting folks giving great talks. This is an opportunity to hear how people are developing the open-source community.

Hope to see you there!

When describe'ing it ain't enough

2009-03-03T00:00:00-08:00

One of the things I like about the RSpec syntax is that it packs a lot of information into a few concise, consistent constructs. It’s relatively easy to read through a spec file and pick out what I am looking for. The use of blocks both enable flexible execution strategies and provide simple containment boundaries.

Perhaps the most valuable aspect, though, is the ability to extend the RSpec syntax constructs easily and consistently. No need to grow a third arm here. In Rubinius, we recently encountered a situation needing some extra sauce when fixing our compiler specs.

A compiler can be thought of as something that chews up data in one form and spits it out in another, equivalent, form. Typically, these transformations from one form to another happen in a particular order. And there may be several of them from the very beginning to the very end of the compilation process.

To write specs for such a process, it would be nice to focus just on the forms of the data (that’s what we care about) with as little noise as possible about how they got there. Here’s what we have in Rubinius:

 1 describe "An And node" do
 2   relates "(a and b)" do
 3     parse do
 4       [:and, [:call, nil, :a, [:arglist]], [:call, nil, :b, [:arglist]]]
 5     end
 6 
 7     compile do |g|
 8       g.push :self
 9       g.send :a, 0, true
10       g.dup
11 
12       lhs_true = g.new_label
13       g.gif lhs_true
14 
15       g.pop
16       g.push :self
17       g.send :b, 0, true
18 
19       lhs_true.set!
20     end
21   end
22 end

The relates block introduces the Ruby source code and contains the blocks that show various intermediate forms. A single word like parse and compile encapsulates the process of generating that particular form, as well as concisely documenting the specs.

The format is sufficiently flexible to allow for other forms. For instance, ast for generating an AST directly from the parse tree rather than using the sexp as an intermediate form. Or llvm to emit LLVM IR directly from our compiler.

Another interesting aspect of this, it was possible with only a few custom extensions to MSpec. Recently, I had added custom options to the MSpec runner scripts to enable such things as our --gc-stats. I didn’t know how easy it would be to add something more extensive. Turns out it was pretty easy. You can check out the source in our spec/custom directory.

What is RubySpec?

2009-03-03T00:00:00-08:00

According to the folks over at http://rubyspec.org, “RubySpec is a project to write a complete, executable specification for the Ruby programming language.” As with any sufficiently concise summary, there’s some opportunity for misunderstanding here, so let’s explore a few aspects of this definition.

Since this post is a bit long, here’s a summary:

There is one standard definition of Ruby.
RubySpec benefits the whole Ruby ecosystem.
RubySpec does not guarantee that program A will run on implementation B.
RubySpec includes only specs for the correct behavior when bugs are discovered in MRI.
Help your favorite Ruby implementation by contributing to RubySpec.

First, what does it mean to be “the Ruby programming language”? The answer to this question used to be a little simpler before Ruby 1.9. Originally, there was the single Ruby implementation that Matz and friends built, referred to as MRI (Matz’s Ruby Interpreter). Now there is also RubyVM or KRI (Koichi’s Ruby Implementation) or simply Ruby 1.9. Either way, MRI 1.8.x and [KM]RI 1.9.x are the standard or reference implementations for Ruby. Everyone else is making an alternative implementation that either complies with the standard or deviates from it.

This is the way RubySpec is written. I realize that it’s possible to consider “the Ruby programming language” to be an abstract thing and all the Ruby projects as merely more or less equal attempts to implement the language. I won’t try to convince you that either way of viewing this is more or less correct. I just want to be clear about the way RubySpec views it.

At the time I began writing what eventually became RubySpec, the only Ruby implementation in wide-spread use was MRI. My biggest concern when getting involved with Rubinius was that the project would be consistent with Ruby the way Matz defines it and not cause fragmentation of the language. My reason was quite selfish. I loved programming in Ruby and I wanted to see the language thrive.

So, RubySpec’s over-arching value proposition is a single, comprehensive definition of the Ruby programming language. To see if the value proposition is actually universal, let’s examine the three categories of people involved with Ruby: consumers, programmers, and implementers.

I define consumers as anyone who depends on a product or service written in Ruby. Consumers may use these products and services directly, or they may own or work for companies that provide them, or they may use products and services that are themselves supported by software written in Ruby. Consumers are the biggest part of the Ruby ecosystem.

Interacting closely with the consumers are programmers, the men and women who write software or frameworks in Ruby. In fact, the same folks may be both consumers and programmers.

The implementers are the people writing Ruby implementations for the programmers and consumers. They want to experiment with ways to better support programmers without worrying that they are breaking their programs in unknown ways. Again, the implementers may be programmers or consumers themselves.

If you’ve ever used a proprietary implementation of a programming language or know what vendor lock-in means, then I am preaching to the choir. If not, then consider that system requirements change, hardware changes, services change, customers change, development teams change, everything changes.

Consumers want assurance that the products and services on which they depend will remain available and will grow with their needs. Programmers want assurance that they will be able to meet their customers’ needs. Implementers want to provide programmers the ability to do so. A single, consistent Ruby language really is a win-win-win situation.

With everything seeming so sunny, one may be tempted to think: “If implementation A and B perform the same on RubySpec, then a program running on A now should run just fine on B.” Unfortunately, it is not quite that simple. Just as a program may not run on both OS X and Linux simply because MRI runs on both. Once in a while, RubySpec finds bugs in MRI, though not that often considering the tens of thousands of expectations. Since a lot of the code in MRI has been around for years, this illustrates that even running thousands of programs (RubySpec is just another program) is no guarantee that there are not bugs lurking around the corner.

What can tell you whether a program will (likely) run on both A and B is the program’s own test suite. In this regard, the program’s test suite and RubySpec are complementary. If the tests discover something that RubySpec did not, it is an opportunity to enhance RubySpec. If running on another implementation expose issues not uncovered by the program’s tests, it is an opportunity to enhance the tests.

Ruby versioning is complex affair and another area where possible misunderstandings exist about RubySpec. It seems pretty simple that Ruby 1.8 and Ruby 1.9 are different. But then there is 1.8.7, which is like 1.8.6 and 1.9. And there may be 1.8.8, which will likely be different than 1.8.6 and 1.8.7 and 1.9. It is dizzying. RubySpec handles the different versions by using the ruby_version_is guard. (Read the guard documentation for full details.)

Generally, the version guards work fine. Each implementation provides the RUBY_VERSION constant and based on its value, the guards permit the correct specs to run. Some have assumed this means RubySpec will tell you that alternate implementation A is just like MRI version X.Y.Z “bugs and all” because A says it is version X.Y.Z.

The principles RubySpec strives for are correctness, clarity, and consistency. There is no way to provide clear and consistent results if RubySpec included specs for the wrong behavior as well as specs for the correct behavior. Either clarity or consistency suffer, badly. Matz is the one who ultimately determines whether a behavior is a bug or not. RubySpec simply includes specs for only correct behaviors.

These are some of the factors that complicate this issue:

There is no definition of what is a bug. It may be a segfault, an incorrect value computed, a frozen state not set or respected, the wrong exception class raised. The list is endless and impossible to consistently categorize.
All implementations, including MRI, have their own release processes and schedules.
RubySpec is a social as well as technical project. An aspect of the value-added proposition for any given implementation is the quality that they provide. There is no way the alternative implementations will consistently agree to defer fixing bugs until MRI releases a fix. Rather than supporting cherry-picking which bugs to fix, RubySpec only includes correct specs.
Bugs discovered by RubySpec in MRI are quite rare.

Finally, contributing to RubySpec is one of the lowest barriers-to-entry means of supporting your favorite Ruby implementation and the Ruby ecosystem as a whole.

Consider what the view looks like from the outside: Ruby has a vibrant community of implementations meeting consumers’ and programmers’ needs on virtually every significant platform, including on Java, .NET, Mac (Obj-C), and semi-platform-agnostic implementations like MRI and Rubinius. Internally, it means that Ruby programmers can focus more on writing their programs using the best tools for the job, confident that if requirements change they can move to a different platform with ease and confidence.

Check out the RubySpec docs if you are interested in helping out.

Boxers or Briefs? -- Neither?!

2009-03-03T00:00:00-08:00

The emperor is wearing clothes and everything looks hunky-dory and sane on the outside. Usually, that’s a good thing. For instance, when running RubySpec on a released version of MRI, it’s good to know that things are behaving as expected and all known issues are accounted for. In other words, you won’t see any failures unless a spec is broken or a new spec has uncovered a bug.

$ mspec library/stringio/reopen_spec.rb 
ruby 1.8.6 (2008-03-03 patchlevel 114) [universal-darwin9.0]
........................

Finished in 0.021797 seconds

1 file, 24 examples, 59 expectations, 0 failures, 0 errors

While the above can be reassuring, it may not tell the whole story. RubySpec uses guards to control which specs are run. This enables the specs to accommodate differences in behavior due to varying platforms, versions, implementations, and bugs.

I’ve added a couple features to MSpec to enable discrete and not-so-discrete peeks under the robes, as it were. The first of these is akin to just yanking down the trousers. By passing the --no-ruby_bug option, all ruby_bug guards are disabled and the guarded specs are run.

$ mspec --no-ruby_bug library/stringio/reopen_spec.rb 
ruby 1.8.6 (2008-03-03 patchlevel 114) [universal-darwin9.0]
............FF.....FF.....FF......

1)
StringIO#reopen when passed [Object, Object] resets self's position to 0 FAILED
Expected 5
 to have same value and type as 0

./library/stringio/reopen_spec.rb:117
./library/stringio/reopen_spec.rb:110:in `all?'
./library/stringio/reopen_spec.rb:61

---- snip ----

Finished in 0.022210 seconds

1 file, 34 examples, 76 expectations, 6 failures, 0 errors

If you cringe a little when blasted by a bunch of failures, don’t worry, So do I. For a more subtle examination, there is also the ability to run the specs and note which specs would have run but did not due to guards.

$ mspec --report library/stringio/reopen_spec.rb 
ruby 1.8.6 (2008-03-03 patchlevel 114) [universal-darwin9.0]
........................

Finished in 0.009809 seconds

1 file, 24 examples, 59 expectations, 0 failures, 0 errors, 10 guards


4 specs omitted by guard: ruby_bug #, 1.8.6.114:

StringIO#reopen reopens a stream when given a String argument
StringIO#reopen reopens a stream in append mode when flagged as such
StringIO#reopen reopens and truncate when reopened in write mode
StringIO#reopen truncates the given string, not a copy

6 specs omitted by guard: ruby_bug #, 1.8.7:

StringIO#reopen when passed [Object, Object] resets self's position to 0
StringIO#reopen when passed [Object, Object] resets self's line number to 0
StringIO#reopen when passed [String] resets self's position to 0
StringIO#reopen when passed [String] resets self's line number to 0
StringIO#reopen when passed no arguments resets self's position to 0
StringIO#reopen when passed no arguments resets self's line number to 0

The guards are reported only if they have altered how the specs were run. Since the ruby_bug guard can only prevent specs from running on the standard implementation, MRI, those guards are not reported when running under JRuby, for instance.

$ mspec -t jruby --report library/stringio/reopen_spec.rb 
jruby 1.2.0RC1 (ruby 1.8.6 patchlevel 287) (2009-02-26 rev 9326) [i386-java]
..................................

Finished in 0.257000 seconds

1 file, 34 examples, 76 expectations, 0 failures, 0 errors, 0 guards

So, if you are wondering what is going on with some specs for a particular library, you can get a quick peek using the --report option before digging into the spec files. There is also a --report-on GUARD option that allows you to narrow the focus of your peeking.

Warning: Includes Known Bugs

2009-02-26T00:00:00-08:00

UPDATE: In further discussions with Jim Deville of IronRuby it appears that there may be a legal issue preventing IronRuby devs from patching Ruby code themselves. However it may be possible for IronRuby to use a commonly maintained and patched version of the standard library.

Reviewing the logs and considering this was Shri’s first major discussion in the IRC channel, I unfortunately grouped him in with Charles’ intolerable behavior and personal attacks which have occurred on a number of occasions in #rubyspec and #rubinius. My apologies to Shri. The struck out text below remains merely for historical accuracy.

UPDATE: Charles response to this post wasn’t exactly positive, but I think it’s fair to have this discussion in public: http://pastie.org/400493 Also, please note that I’ve struck out Shri’s name below as I may have misunderstood him in the earlier discussion.

You, the trusting consumer, would probably like to receive such cautionary advertisement were you to use a product that did, in fact, ship to you code that includes known bugs. And not just known bugs, but known bugs that have fixes for them.

You would like to know this, right? I mean, I’m not just some hard-headed asshole that thinks there’s something a bit whack here, am I? Please, do tell me.

Well, as luck would have it, you can also tell this to Charles Oliver Nutter of JRuby ~~and Shri Borde of IronRuby~~.

Here’s the drama: There’s this project RubySpec. You may have heard of it. It attempts to describe the behavior of the Ruby programming language. All the alternative Ruby implementations use the RubySpec project to attempt to show that they are “Ruby”.

All the alternative implementations also choose to ship some version or other of the Ruby standard library. At least the parts written in Ruby. Makes sense, since they all implement the Ruby programming language.

As is the case with all software, from time to time bugs are discovered in Ruby. Usually, these are fixed soon after they are discovered and the fix is committed to the trunk version of MRI (Matz’s Ruby Implementation). Eventually, trunk becomes another stable release with a particular patchlevel.

The RubySpecs deal with this situation with a ruby_bug guard. You can read the details of RubySpec guards. This particular guard has two functions:

It prevents the guarded spec from executing on any version of MRI less than or equal to the version specified in the guard. This is because MRI cannot re-release a particular patchlevel after it has been released. And the bugs are discovered after a release.
It documents the spec, which shows what the correct behavior should be.

A key feature of the ruby_bug guard is that it does not prevent the spec from running on any alternative implementation. That is because every alternative implementation is expected to have the correct behavior. Additionally, these guards are only added after Matz or ruby-core has stated that the behavior at issue is a bug and the behavior of the spec is the correct behavior.

Now here is the rub, Charles does not want to manage patching the Ruby standard library that he ships with JRuby with the patches that already exist for known bugs. He wants to ship whatever version MRI has most recently released. Further, when you run the RubySpecs with JRuby, he wants to MASK those bugs because he doesn’t think it’s fair that JRuby fails a spec which shows a known bug in the Ruby standard library for which patches are available.

That’s Charles choice of strategies for managing JRuby packaging. I’m strongly of the opinion that you, the user, would like to know that. Charles apparently disagrees.

In fact, he disagrees so vehemently that he takes to calling me names in the #rubyspec IRC channel because I refuse to change the fact that the ruby_bug guard will not silently mask spec failures on JRuby or any other alternative implementation. Aside from being immature, I think there is a real problem with this. Don’t you?

Charlie will argue that it is simply impossible to ship the trunk version of Ruby standard library because it is an unknown quantity? However, the best defense against bugs in the Ruby standard library is better specs. And we’re talking about specs here that show the bugs and for which patches exist. Furthermore, there are actually relatively few bugs noted in the specs and most of those are in older versions of Ruby, not the current stable release.

So, here’s my question to you: Would you like to know that JRuby ~~and possibly IronRuby~~ ship you code that contains know bugs for which patches exist? Would you also like to know that Charles wants you to run RubySpec on JRuby and not know there is a bug?

Caveat Lector

2009-02-20T00:00:00-08:00

I really did double-check this time and I won’t be making any wild claims here. Sorry to disappoint.

We’re going to be running Antonio’s Ruby Benchmark Suite daily to track our progress on performance in Rubinius. The current RBS is a bit of a beast so I imported the files into the Rubinius repository and did some refactoring. You can read the details and up-vote that if you’d like to see this merged back.

Now, for some baseline RBS results. If you want to follow along at home, here’s what I did. I generated these by running the rake bench task using the VM option (see the benchmark/utils/README in the Rubinius repository) for Rubinius on the stackfull and master branch and for MRI using the version installed on Debian lenny, 1.8.7p22. The system is a dual Intel® Xeon™ CPU 2.40GHz. Then I ran the rake bench:to_csv task, imported the CSV file into Google Docs, added the comparison columns and colors, and exported to PDF.

Here’s what I got. The green is faster, the red is slower. The reported time is the minimum time recorded in five “iterations” of each benchmark per input. The maximum time allowed to run five iterations is 300 seconds, or an average of 60 seconds per iteration.

A few notes about these numbers:

We’re still fixing the breakage on the stackfull branch, so it is not surprising, for instance, that all the thread benchmarks errored out. The new native thread support is not 100% done.
There are a couple speed regressions on the stackfull branch, most minor. We’ll fix those.
Most of the benches do run on the stackfull branch.
On most of the benches that run slower in stackfull than MRI, we’re 2x or less slower than MRI.
We are a lot faster than MRI on quite a few benchmarks.
Rubinius on either branch does quite well relative to MRI on benches that MRI times-out on for certain inputs.

Perhaps the biggest point about the stackfull branch is that we haven’t done much optimization at all. Evan’s been coding in the basic new interpreter architecture, fixing the GC interaction, adding the native threading. We’re fixing breakage now so we can get this merged into the master branch. The JIT is not hooked up. The new GC work is not done. There is no inlining. In other words, there is lots of head room. And that is the key point. You can’t just “make it faster”. Architecture is crucial. Since RailsConf 2008, we’ve been working hard to lay the architectural foundations. With those (and the switch away from stackless), we can start focusing on the real dynamic language optimizations.

While the benchmarks tell part of the story, there’s another part that is even more interesting IMO. And this is the part that got me so excited I, um, well I just got excited…

The two biggest pieces of Ruby software that we most often run are the Rubinius compiler and the RubySpecs. The RubySpecs are much more “real-world” than these benchmarks. Here are the results of two complete CI runs on master and stackfull. Note that we are not quite running all the basic CI specs on stackfull, but we’ll figure in that difference in our calculations below.

First, on master:

  $ bin/mspec ci --gc-stats
  rubinius 0.10.0 (ruby 1.8.6) (f4c5576c4 12/31/2009) [i686-apple-darwin9.6.0]

  Finished in 131.248169 seconds

  1430 files, 6927 examples, 23006 expectations, 0 failures, 0 errors

  Time spent in GC: 51.6s (39.3%)

And then on stackfull:

  $ bin/mspec ci --gc-stats
  rubinius 0.11.0-dev (ruby 1.8.6) (e7b6a2d56 12/31/2009) [i686-apple-darwin9.6.0]

  Finished in 66.357996 seconds

  1349 files, 6298 examples, 21344 expectations, 0 failures, 0 errors

  Time spent in GC: 12.7s (19.1%)

Let’s calculate how we do in expectations per second:

  $ irb
  >> master = 23006 / 131.248169
  => 175.286254850534
  >> stackfull = 21344 / 66.357996
  => 321.649255351232
  >> stackfull / master
  => 1.83499416782851

So, compiling and running the specs is about 1.8 times faster on stackfull. This is upside down from the normal results. Normally, we do better on the micro benchmarks and see that invert on “macro” benchmarks. On the RBS benches, stackfull is not 1.8 times faster than master. If I average the “x Master” column, I get 1.39.

There was something else in those spec run numbers I wanted to talk about… oh yeah, GC stats. We have a very simple GC timer stat right now. I’m going to be adding a few more stats. But what we see here is that the overall percentage of time spent in GC drops by half in stackfull. Even so, 19% is too much time to spend in GC. We expect to drop that by half again. Basically, leaning more on structures alloca'd on the C stack reduces a lot of pressure on the GC.

Some would toss out that it’s not hard to be faster than MRI. Perhaps. But it is an accomplishment to write a reasonably good VM, garbage collector, compiler, and Ruby standard library without importing anyone else’s code. And, lest we forget, that is two VM’s in about 27 months of a public project.

Some would also question the sanity of writing a VM and garbage collector when crazy smart people do things like that. Well, crazy smart people write papers that reasonably smart people can read and understand. From the benchmark result above, that is working pretty well.

Here’s the point: Don’t ever let anyone tell you that something is a bad idea. Make your own decisions. We probably wouldn’t have Ruby itself if Matz fretted over whether Larry Wall or Adele Goldberg were smarter than he. My most recent favorites in this space: Factor, Clojure, and yes, tinyrb.

We’re working frantically to get the stackfull branch breakages fixed and the branch merged back into master. Feel free to poke around and ask questions.

This is NOT cold fusion

2009-02-12T00:00:00-08:00

Um, whoops. It was really late last night. Have I mentioned you’re wearing a great outfit today. Ok, already.

There’s this slight matter of DEV=1 rake build in Rubinius. Yes, I was debugging something. Started running some stuff under the stackfull branch, was intrigued by what I was seeing, decided to make some comparisons, could swear I ran rake clean; rake in the master branch, had a lot of green tea yesterday…

All right already. It’s not 4x faster. Here’s some new numbers:

Master branch:

$ bin/mspec ci core/string
rubinius 0.10.0 (ruby 1.8.6) (781eb14d3 12/31/2009) [i686-apple-darwin9.6.0]
.....................................................................

Finished in 10.576468 seconds

69 files, 763 examples, 5632 expectations, 0 failures, 0 errors

Stackfull:

rubinius 0.10.0 (ruby 1.8.6) (325174a8e 12/31/2009) [i686-apple-darwin9.6.0]
..................E.....E...E...F............E..E.EE..E........F..F..

Finished in 6.124444 seconds

69 files, 763 examples, 5545 expectations, 6 failures, 19 errors

That’s about 58% as long, or 42% faster, or creeping up on 2x.

If you rushed out and bought Evan a Valentine’s bear, I do apologize. But send it anyway. All the rest in my previous post about this being the beginning of a very good thing still holds. We’ll be getting more results soon and fixing the spec breakage on the stackfull branch. Stay tuned!

All shiny and new

2009-02-12T00:00:00-08:00

UPDATE 2.0: You really did see the update below, right? You’re getting Charlie all worried with your enthusiasm for Rubinius.

UPDATE: Ahem, you should probably also read: This is NOT cold fusion. No, it’s not April 1st. Sorry about that. Are you still excited? Read on!

It’s a pattern I’m fairly familiar with now. Evan will be pondering an issue with Rubinius. I’ll catch wind of it when he starts asking some questions of smart people, reading academic CS papers, other implementation’s code, and tossing out some “what if…” questions. Next thing you know, he’s frenetically churning out code. Suddenly, Rubinius is much better, and in this case, faster.

Well, it’s happened again and the preliminary results are outstanding_. A couple weeks ago, Evan began coding some changes to the way the Rubinius bytecode interperter works. He changed the stackless execution architecture that implemented an optimized kind of spaghetti stackstack to use the C stack more directly and naturally. This better enables the CPU optimizations of the past dozen years to work. It also significantly simplifies the code for our FFI, C-API for C extensions, JIT, and for potentially leveraging LLVM much more effectively. This change also brings native threads, and a much better GC for the mature generation is also in the works.

Now, for some details. Again, these results are preliminary. There is still a lot of breakage on the stackfull branch but MSpec is already running and many of the CI specs run. I’ll be getting a new CI set in place today and we’ll get the remaining breakage fixed quickly (don’t ya just love those specs).

Here’s some numbers for compiling and running the String specs.

First, on the Rubinius master branch:

    Finished in 25.829773 seconds

    69 files, 763 examples, 5632 expectations, 0 failures, 0 errors

Now, on the Rubinius stackfull branch:

    Finished in 5.834874 seconds

    69 files, 754 examples, 5563 expectations, 6 failures, 19 errors

Here’s the numbers for running after the specs have been compiled.

Again, on the master branch:

    Finished in 5.101799 seconds

    69 files, 763 examples, 5632 expectations, 0 failures, 0 errors

And now the stackfull branch:

    Finished in 1.564942 seconds

    69 files, 754 examples, 5563 expectations, 6 failures, 19 errors

I’ll let that sink in a bit…

The numbers for Hash with compilation are similar.

Master:

    Finished in 5.379050 seconds

    48 files, 195 examples, 425 expectations, 0 failures, 0 errors

Stackfull:

    Finished in 1.295544 seconds

    48 files, 193 examples, 421 expectations, 0 failures, 0 errors

That’s right, ~~between 4.1 and 4.4~~ approaching 2 times faster (see the UPDATE above). And we are just getting started. The significant GC changes are not in yet. We are not yet doing any significant optimizations in the compiler, no profile-directed optimizations at runtime, and our nascent JIT is not hooked up by default. As I said at the outset, these optimizations are made easier by this architecture change.

While I’m breaking the news, Evan deserves the credit for the architecture decisions and generally being courageous enough to try and learn (some would say fail) and try again. Some have doubted that the lofty goals Rubinius has set are realistic. Doubters have a seat.

If you want to try this at home, clone the Rubinius Github repository and do the following:

    $ git branch --track stackfull origin/stackfull
    $ rake build
    $ bin/mspec ci core/string

Thanks to Engine Yard for trusting in Evan’s excellent judgment and system architecture talents and in all our hard work even if it doesn’t look immediately relevant. The path is clear. The goods are in the truck and they will be delivered.

Call of the lemming

2008-11-23T00:00:00-08:00

It’s sometimes hard to gauge troll, tirade, or spirited challenge online. The Opposite of Momentum strikes me as a mix, though in what percentages, I haven’t decided. KirinDave is a little late to the bag-on-Ruby party, and if it’s Javascript he wants, Yegge was there well over a year ago. Browse some of the mind-expanding comments by the bottom feeders at Reddit chewing on KirinDave’s post. It’s a reassuring, if disappointing, commentary on the consistency of human behavior: it is always easier to complain than to fix.

But, anyway.

True, the reference implementation of Ruby has some problems. But, Rubinius has a precise, generational garbage collector. It is definitely not “technically stagnant”, nor is it “in implementation limbo”. It was only in May of this year, before RailsConf, that we first got Rails running on Rubinius. We are close to getting it running again on the new VM. If we do that by year’s end, we will surely have some kind of record. For all the thousands of lines of Ruby code he’s written, KirinDave hasn’t contributed any code to Rubinius. That’s too bad.

Ruby isn’t that extraordinary as a language. But it is a very good language. What is more, Ruby is a humane language, in the spirit of Jef Raskin’s The Humane Interface. Furthermore, now like never before is the age of the dynamic programming language. The technical hurdles are being explored all around us. In fact, before the year is over, you should spend a few quality hours with one or more of the following languages.

Factor. This fascinating concatenative language will definitely get you thinking outside of the box. Rubinius is a stack-based VM. Writing some Factor code is a little like peeling back the syntax layer of Ruby and watching the muscles and tendons work. Check out the quotations and combinators and think about how you write functional code with Ruby. Don’t miss the Google tech talk by Factor’s creator, Slava Pestov.

Clojure. This is another fascinating language by another wicked smart guy, Rich Hickey (check out the video). In fact, this is currently my favorite language to study. It’s the Ruby of Lisps, in my opinion. And it contains a wealth of ideas that could be useful for implementing Rubinius, not to mention, thinking about writing concurrent programs.

Io. Like Javascript? You owe it to yourself to play with this language a bit. I’d be interested to see about an implementation of Io on SquirrelFish Extreme

Haskell. Amaze your friends and improve your sex life. Jump stateful, side-effect tangled heaps of code with a single Monad. There is a cool new online book that will soon be released in print as well.

Seriously, if you are a Ruby programmer, you could hardly do poorly by adopting any one of these languages. If Javascript makes your curly braces tingle, who am I to ridicule you. But if Ruby continues to maintain its captivating hold on your attention, don’t be disappointed. There is some awesome sauce in there, and we’re working on the recipe that really makes the flavor pop. Just a little request, instead of bitchin’, pitch in.

Rubinius is a community project

2008-11-18T00:00:00-08:00

Yesterday was a tough day. You can read the details and Evan’s thoughts about it.

It was especially hard for me because Wilson was a mentor of sorts and the first person I talked to when I dropped into the #rubinius channel nearly two years ago. He allayed any fear I had that Rubinius would fragment the Ruby community and helped me get started contributing.

And Eero, among many other things, jumped on board right away and toiled for many hours on the early specs, putting up with my dead ends when the project that is now RubySpec was merely an expedient to fast-track Rubinius development.

So, I want to express my thanks to my teammates and wish them the best in their next endeavors. And I want to emphasize something that is well known to those of us who have worked hard on Rubinius from long before Engine Yard began to support the project: Rubinius is a community project.

Evan’s free-flowing commit bit, the early adoption of Git, and the emphasis on writing specs while developing are just a few aspects that have helped propel Rubinius very quickly in its couple year history. And there is still plenty of exciting and interesting work to be done. We are all very grateful for Engine Yard’s support of the project, and that support remains strong and dedicated.

But the key to Rubinius’ success is the community. In the past year, I feel we have to some extent let you, the community, down. We need to make Rubinius even easier to try out, understand, and contribute to. And we need your feedback on ways we can do this. The recent VM rewrite has been a huge step forward in this regard. But have you opened up the code, poked around, asked us about stuff that doesn’t make sense? If you haven’t, let us know why not. Have you taken a look at a failing spec for the core library? There’s tons of beautiful Ruby code just begging for some love. Let us know what barriers prevent you from spending some free time in there.

If you’re worried that development on Rubinius will slow down, I would challenge you to not let that happen. Evan produces prodigious amounts of code and has a terrific mix of genius and pragmatism. I will certainly not be working any less hard than I have for the past two years. So if you, like me, believe that Rubinius can be a terrific platform for Ruby, jump in. No contribution is too small. And we’re building on a solid foundation of past success.

Hash is all ponies

2008-09-03T00:00:00-07:00

We’ve been hard at work replacing the C virtual machine in Rubinius with one written in C++. Along the way, we decided to use a simpler data structure internally where we had been using a Hash before. This allowed us to rewrite Hash in pure Ruby. Since our goal is to write as much Ruby in Ruby, I was happy to work on this rewrite.

You can check out the resulting code here and here. It hasn’t been polished much yet, but the idea was to use expressive and natural Ruby. I didn’t take any performance-conscious tacks. Hash should provide a great case study for looking at optimizations in our compiler and VM. For example, Hash#keys can be succinctly written as:

1 def keys
2   inject([]) { |ary, entry| ary << entry.first }
3 end

Since #inject comes from the Enumerable module, we know it’s using #each internally. And for Hash, I implemented an external iterator of sorts to help hide the implementation details. That iterator is making a method call for each item. So we’ve piled on two block calls and a method call for each item. The point is, we want that expressive Ruby code in our applications and let the implementation optimize it. So far, we really haven’t had any Ruby implementation to give us that.

Another thing to check out is the structure of our kernel directory. Talking to the folks working on MagLev, we’ve restructured our runtime kernel (aka Ruby standard library) to better allow sharing with other implementations. The bootstrap and delta directories are implementation specific, while the common directory is intended to be shared. The bootstrap directory provides just enough functionality to load the common directory. The delta directory provides a location for implementation specialization where specific methods can be replaced with more performant or otherwise specialized versions.

As always, we love to hear from folks about what’s bugging them with Ruby. Drop by #rubinius on freenode.net or check out the Rubinius project on Github. Even though we’re in the middle of this VM rewrite, there’s plenty still to be done on the Ruby standard library.

Update: Evan points out that this is a better implementation of #keys:

1 def keys
2   map { |key, value| key }
3 end

As del.icio.us as a giant, steaming turd

2008-08-01T00:00:00-07:00

I often dislike the interfaces that I’m forced to use. Banking sites are about the highest on my list of horrid. The pain these things inflict.

However, today I got a new jolt. Clicked on my handy del.icio.us shortcut in Firefox and was greeted by a bizarre interface. Not to worry, I powered on, since of course, the reason I use del.icio.us in the first place is because it was super quick and easy to add a bookmark. Quick is no longer in the vocabulary for del.icio.us. Instead of simply typing a few characters, hitting <tab> and being done with it, I now have to arrow in this stupid drop-down? Amazed, I started taking a closer look at the unwelcome intruder in my nice browse + bookmark flow.

The lame new auto-complete behavior is a breaker for me. However, it’s hard to say if that’s the worst of the UI redesign. The 3-option choice for the main list display goes from bad to worse to horrid. Is the date, in washed out grey, really the most important piece of info that I need to see in my normal scan from left-to-right? I can barely scan any of the options quickly for the information I need to help me select a link. And where did my nice big entry field go for typing in a tag? Oh, there it is, camouflaged in that bar at the top of the list. Don’t really want me to use that, huh?

I could go on, but what’s the use. The amazing thing to me is that someone, somewhere thought this was a good idea. Have these idiots never heard of Alan Cooper, IxD, the rich body of books describing why you don’t design interfaces like this? Harsh? Damn, right. Horrid.

OSCON 2008

2008-07-21T00:00:00-07:00

If you’re headed to OSCON 2008, I hope I’ll have a chance to meet you. I’ll be speaking on Thursday afternoon, something about Ruby performance. Come and heckle me if you wish. Here’s the talk details:

Who Wants a Faster Ruby?
07/24/2008
2:35pm – 3:20pm PDT
Room: F151

Also, please do come check out FOSCON. This is the 4th year for FOSCON. w00t.

UPDATE: Thanks to all who attended my talk. I’ve uploaded the slides. If you attended, please take a moment to send feedback.

Implementers Unite!

2008-05-16T00:00:00-07:00

There’s a lot of things that Rubinius, MatzRuby (including YARV), JRuby, IronRuby, MacRuby, Ruby.NET, MagLev don’t agree on. But that’s ok, because above all, they seem to agree that Ruby is one awesome language. And lately, that apparent agreement has gotten a boost of sorts.

Earlier this week, Rubinius (an Engine Yard project) announced the RubySpec project at rubyspec.org. The effort to write an executable specification using RSpec-style specs started with the Rubinius project in late December 2006. Since then, many people all over the world have contributed to the effort, including core folks from JRuby. Engine Yard has been financially supporting significant development of the RubySpecs since hiring Evan in June 2007 and other full-time developers (myself included) in Jan 2008. All this effort has been to ensure that the Ruby programming language continues to evolve and receive the recognition it deserves.

Another recent development has been a regular meeting of folks working on various Ruby implementations. These Ruby design meetings are bringing together folks to discuss tough issues with Matz and everyone else.

Well, after last night’s meeting, Tanaka Akira (sorry, I don’t have a link to his website; if you do, please let me know) checked in some changes to the ruby-core Makefile to fetch and run the RubySpecs. This is a very proud moment for me.

If you love Ruby, it doesn’t matter what color shirt you wear, what language you play with, or what country you live in. You can play a valuable role in helping make Ruby a wonderful language for everyone. Please help us with the RubySpecs.

Update: I changed the link to the changeset for the Makefile. Thanks Michael.

Ruby in Ruby--Seriously

2008-04-28T00:00:00-07:00

All the attention that the various Ruby implementations are getting these days is a wonderful thing. Competition and lots of people trying out different ideas is a win for everyone. That said, there are still the silly human issues of ego and vested interest, always exacerbated by tossing some green into the mix.

I would definitely assert that JRuby is one of the more impressive stabs at a Ruby implementation. And we have, in large measure, to thank Charles for that. But, at the same time, Charles can write FUD like the best of them. I haven’t made it through the whole article yet (it does ramble on a bit), but I should point out a couple things.

Method dispatch is hard to get fast in Ruby, no matter what. Substitute a VM for an interpreter, C# for C for Java, whatever. And if you read carefully about what Charles is pinning his hopes on for the future of JRuby, it is precisely optimizations in the new JVM for better method dispatch. Rubinius will address these issues for sure. But instead of working on this with something like 143,866 lines of Java code, we have something like this: In the new C++ VM (which is not yet complete but substantially implemented), we have 12,619 lines of C++, and in our kernel directory, we have 23,882 lines of, what now, oh, right Ruby code. What you might miss when you think about the 143,866 lines of Java code is the several hundred thousand lines of C++ that make up the JVM. Hmm.

So, ultimately, Charles hopes to make JRuby fast using the same basic techniques that Rubinius does and will use. Except, it seems he’d rather use a 747 passenger jet to mow his lawn, rather than a comfy John Deere riding mower that your average small engine enthusiast can tinker with when something goes awry.

Seriously, the “Ruby in Ruby” meme must not die. It is an inspiration to a lot of folks and, most importantly, it has great promise. Press Charles on the issue and he will admit he’d rather have more of JRuby written in Ruby. I sense something of “bitter-gate” in Charles’ post. We don’t cling to Ruby because we can’t write C#, C, Java, etc. We don’t just like Ruby when it’s fun and fashionable and hyped. We are demonstrating that Ruby can be sanely implemented and push the state of the art for the language forward.

The C VM (named shotgun) was not our last word. Nor is the next generation C++ VM. They are both pragmatic steps toward a higher goal. And, let’s be very clear. We have not recently implemented a bunch of core methods in C. I’ve done two major pieces of rework recently that introduced a number of primitives (chunks of C code that access the VM directly). One was LookupTable, which was written in C because it is used heavily in the VM. However, it is exposed to Ruby code as well because, oh yes, we write a ton of stuff in Ruby, like stuff related to method and constant lookup. LookupTable acts a lot like a Hash, but separating it from Hash actually made Hash more clear and enabled writing even more of Hash in Ruby.

Another piece of work was optimizing some of String’s methods. You can see the performance enhancements in this ticket. I did introduce a couple new primitives in this rework, mostly to make a more sane set of composable primitives that were useful in many String methods. Primitives are a necessary fact of life in this composite implementation. Something has to bridge Ruby and C/C++. Also, by using a well-defined set of primitives, it makes it very easy to take our Ruby core libs and run them on something else. Ask Charles and he will tell you this. And if we write a little C/C++ to get the performance of a large number of Ruby methods up to par, that’s a huge win.

So again, the “Ruby in Ruby” meme is really important. Don’t let it die. And don’t let Charles tell you otherwise. While he can make snarky comments about our C/C++ VM, he should really look at his own kettle of 143,866 lines of Java code plus hundred thousands more lines of C++ JVM. The goal is a first-rate, powerful, extensible, approachable implementation of Ruby. Everyone in the Ruby ecosystem is contributing to that in various ways. Rubinius just seems to be putting the best pieces together, if I do say so myself.

Rubinius runs RSpec 1.1.2

2008-01-18T00:00:00-08:00

This day has been a long time coming. About a year ago, I started trying to get Rubinius to run RSpec. Hats off to all the wonderful Rubinius contributors and special mention to the cool guys who hung out with me at pdx.rbc meetups in early 2007. (Unfortunately, the page in that link will disappear soon. Maybe I’ll make a pdf out of it and keep it here. You know, nostalgia.)

Everyone’s efforts have paid off and here is the proof:

$ shotgun/rubinius -I ../rspec-1.1.2/lib/ ../rspec-1.1.2/bin/spec spec/ruby/1.8/core/true/
.....

Finished in 0.988572 seconds

5 examples, 0 failures

The TrueClass specs were some of the very first ones I wrote, so I thought it fitting those be the first ones I ran successfully with RSpec.

Congratulations everyone!

Evil can be dangerous

2008-01-17T00:00:00-08:00

The first post I read by Evan about Rubinius had a quote about evil built in. Of course, that caught my eye. What was Evan’s point? Legend has it (I wasn’t there) that during Q&A someone asked about fundamental aspects of the Rubinius system being available for modification from normal user code. The issue was raised that perhaps something should restrict that unlimited power. Evan’s response was that Rubinius sorta has evil built in. Usually, that’s a good thing.

In fact, it’s a very sharp double-edged blade. We all cherish the dynamic nature of Ruby when it allows us to do useful things. For example, the following works great in MRI:

1 class Fixnum
2   alias / quo
3 end

That little snippet and you’ve redefined fixed-point math to return a floating point number. That gem is one of the core pieces of the Mathn library in the Ruby standard library collection. Together with the Rational library, you can do some cool stuff:


$ irb
>> require 'rational'
=> false
>> require 'mathn'
=> true
>> 1 / 2
=> 1/2
>> 1 / 2 + 3 / 8
=> 7/8

Combine that with Matrix and Rational and you have at your disposal some pretty powerful tools for doing more realistic mathematics. Take a look at what some folks do with it.

Fortunately, and unfortunately, Rubinius is very different than every other implementation of Ruby that I’m aware of. Rubinius follows the Smalltalk model and uses the Ruby language itself to build the implementation. That means that Array, for instance, calculates values for its internal operations using none other than the same Fixnum values that you use in your Ruby code. So, what happens when we redefine Fixnum#/ to return a Float instead?

Yes, you guessed right. Things work differently. In fact, not at all.

Since we’re not about to rescind the promise of a better Ruby, we have to figure out some way to deal with this. There are several approaches possible. One would be to code all of the core library with a different set of numeric classes, say BasicFixnum and BasicBignum, that are superclasses of Fixnum and Bignum. That would be really cumbersome. Another approach would be to use different methods in the core library. Again, cumbersome. So, this sounds like there are some basic goals we have in mind. In fact, there are. We want to keep the core library code as simple as possible and provide the full power of Ruby at every level, to the greatest extent possible.

This isn’t necessarily the ultimate approach we’ll use, but I checked in some code today that leverages a very slick feature that Evan built into the compiler. I created the following compiler plugin:

 1 class SafeMathOperators < Plugin
 2   plugin :safemath
 3   
 4   MathOps = {
 5     :/ => :divide
 6   }
 7   
 8   def handle(g, call)
 9     name = MathOps[call.method]
10     if name and call.argcount == 1
11       call.emit_args(g)
12       call.receiver_bytecode(g)
13       g.send name, 1, false
14       return true
15     end
16     return false
17   end
18 end

Essentially, what this does is map any calls to the #/ method to the #divide method. I’ve changed the core library classes Fixnum, Bignum, Float, and Numeric to handle this. So, the code in core looks the same, but is protected from the redefinition that the Mathn library performs. The compiler plugin is activated with the -frbx-safe-math switch only when compiling the Rubinius core libraries.

> UPDATE: Note that you do not have to do anything differently. The compiler flag is passed in automatically when you type rake to build the Rubinius libraries. If you are writing user code, you also do not need to do anything differently. You just write your code as you normally would. If you require the Mathn library, 1 / 2 => 0.5. If you also require the Rational library 1 / 2 => 1/2. Invisible, maybe even a little evil.

Again, this is not necessarily the way we’ll always do it. The lessons are:

Ruby is a great, powerful language
Rubinius makes Ruby even more powerful
We can work around snags pretty dang easily, even if temporarily

Of course, we’re always learning. Any papers, systems, code, etc that you can point me to?

Contra--what?

2008-01-17T00:00:00-08:00

I’m going to be pedantic for a minute. If you are Betrand Russell, feel free to spend your time more productively. Everyone else, don’t be afraid of a few symbols.

If you read my “last post”:http://blog.brightredglow.com/2008/1/17/evil-can-be-dangerous, you might be inclined, as is every human mind, to reduce all those words to something simple you can put in your back pocket (maybe take it out later and show it to a friend). Perhaps something like this:

  Immutable == Security == Good

If you do that, you might get lots of nods and a fair bit of affirmation from folks. Makes perfect sense, right. You can make all sorts of analogies that prove this to you. Consider my front door, if it’s not easily opened, I feel more secure. Perfect. Must be true.

It’s not. Things are not so easily distilled into such nifty little boxes.

Now, once they have this shiny, handy, palm-sized summary in their back pocket, folks tend to go the extra mile. They decide to make one of the most fundamental logic errors. Essentially, trying to think all the way around this hairy problem, they decide:

  If Immutable == Security
  Then Not Immutable == Not Security

Doh. That’s where this contra thing comes in. Read the gory details on “Wikipedia”:http://en.wikipedia.org/wiki/Contraposition, but here’s the summary:

In logic, something like A -> B is read “A implies B” or “if A then B”. It is very tempting to then think, “if not A then not B”. Unfortunately, that is not generally true. Given “A implies B”, the statement “if not A then not B” is called the inverse. Again, these are not generally logically equivalent. The statement “if not B then not A” is called the contrapositive, and it is generally true that a statement and it’s contrapositive are logically equivalent.

Applying this to the above, we can see that IF (a very big if) it is true that “if Immutable then Secure” is true, then the statement “if not Secure then not Immutable” is equivalently true. However, it has not been demonstrated that immutability is equivalent to security. And indeed there are numerous ways to achieve “security” (a word that begs precise definition) in a variety of different systems.

So, before you start touting something like Java because it is statically typed, immutable and “secure”, consider whether you are making the mistake of confusing the inverse for the contrapositive. If you’re feeling extra nimble and up for some mental gymnastics, consider this rather hyperbolic assertion:

Ruby is a very, very sharp knife. Java is a little more like a butter knife. We safely leave butter knives on the table when the kids are around. But, we probably grumble more than a little when trying to cut our steak with a butter knife.

If we want to make Ruby “safer”, it’s possible to do so in the same manner as dealing with sharp knives: we make something like a sheath. We don’t just take a hammer to the blade and dull it.

Irb in the house

2008-01-05T00:00:00-08:00

I’m going to give this live blogging thing a go. As of 14:52 PST, Evan announced that Rubinius runs irb. It hasn’t landed in the repository yet, but this is great news. Just a little while ago, we started with an irb wannabe. I guess a lot can happen in a year.

Congrats to Evan and all the fantastic Rubinius supporters and contributors.

UPDATE: So, I’m not a very good reporter. There’s some other very big news related to getting irb working. Evan checked in a working eval yesterday. Enjoy!

Looking yourself in the eye

2007-12-06T00:00:00-08:00

I’m hanging out this week at the Engine Yard office in South Park hoping to catch a glimpse of Cartman and pals (I wish!). You’ve heard it all before but I would be remiss to not toss in my congratulations to my new (very soon) colleagues. Evan, Wilson, Ryan, and Eric are a stellar group of folks and a great bunch of personalities to boot. I am honored and extremely happy to be working closely with them, as well as the fantastic group of contributors world-wide.

Everyone at Engine Yard deserves big accolades for the vision, daring, and execution in making all this possible. Here’s my opinion (and gauntlet): This is why it is always the little companies that do something great. I see Ruby in general and Rubinius in particular as a powerful antidote to insane lumbering companies that have foisted agony, fear, and self-loathing on legions of otherwise decent folks with their sharp, pointy brackets and stupidly limited language features. Bring your hungry masses to our shores. We have the very ambitious intent to build the best programming language and system to date. Hubris? We’ll see.

Oh yeah, but about this looking at your own eyeball without a mirror thing. Some of you may recall that I’ve been working a bit on writing specs for Ruby as part of this project. Since we could not run RSpec initially because of its extensive use of very advanced Ruby features, we wrote a minimal replacement for it that I affectionately dubbed mini_rspec. At the time I wrote this, we were converting a ton of specs from the early host/target architecture I started with. We may have been able to run miniunit at that time, but my intention was to write specs for mini_rspec as I went along evolving the features of the framework itself. This is a very difficult bootstrap process akin to looking at your own eyeball… you get the picture. So, I must say I’m solely responsible for this little gem:

1 require File.dirname(__FILE__) + '/../spec_helper'
2 
3 # hmm, this is embarrassing
4 
5 describe "mini rspec" do
6 end

Of course, I took a nice bit of friendly needling (or should I say snickering) from Ryan for that one. But let’s not dwell on the sordid details of the past. Currently, in my branch I have a rewritten (and renamed) mSpec. This is a little better:


euler:rubinius brian$ spec mspec/spec
.................................................................................................................................................................

Finished in 0.062927 seconds

161 examples, 0 failures

That’s what I’m talking about. This time, I wisely just used existing RSpec to write the specs because my goal is not necessarily to be able to run the mSpec specs with mSpec, but to be able to evolve the system while still keeping layers of it so simple that nascent Ruby implementations will actually be able to use it.

A final word about the other approach. We wrote thousands of specs for Ruby using mini_rspec. We ran these specs against MRI 1.8.6 and they behaved as expected. So, those specs were essentially an indirect spec for mini_rspec. The challenge of that approach, of course, is that you do not have a simple and clear way to observe the impact of changes you make to the mini_rspec system. Anyway, I should finish up this massive bunch of changes very soon and push it out.

Send my love to Ruby II

2007-09-20T00:00:00-07:00

In the last post, we saw a nice use case for the #send method. Now, we’ll see some coolness that Rubinius adds. First, some code:

1 class Apple
2   def seed
3   end
4 end
5 
6 a = Apple.new
7 
8 a.send :seed
9 a.__send__ :seed

Now, with the convenient and useful describe facility of the Rubinius compiler, we can see something interesting:


$ shotgun/rubinius describe send.rb
Path: send.rb
Size: 79 bytes

Sexp:
  [:block, 
  [:newline, 1, "(eval)", 
    [:class, [:colon2, :Apple], nil, 
      [:scope, 
        [:newline, 2, "(eval)", set_local_fp 1 ; local a

---- snip ----

pop
#line 8
push_literal 1
get_local_fp 1 ; local a
send send 1
pop
#line 9
get_local_fp 1 ; local a
push nil
push_literal 2
dup
is_symbol
git send_lbl1
send to_sym 0
send_lbl1:
push 0
set_args
send_off_stack
pop
push true
ret

---- snip ----

I’ve truncated that output, but feel free to run this at home. (There’s no long lasting side effects other than an itching desire to contribute to the Rubinius project.) If you correlate the assembly with the line numbers in the source, you’ll notice that the two sends are not turning out to be the same. That’s right. We have a special, and fast, __send__ operation.

The #send method is a highly useful bit of Ruby. Unfortunately, there’s some concern that using it can hurt performance. Well, with Rubinius you can have your send and use it, too.

Send my love to Ruby

2007-09-20T00:00:00-07:00

Over at the Rubinius project, in between hatching plots to take over the world, we fit in some time for recreation. For example, we’ve got this masochistic interest in writing RSpec compatible specs for the Ruby core library. One of the challenges there is the large number of aliased methods that Ruby has. Using RSpec’s shared behaviors as an example, I’ve created a flavor of shared behaviors in our mini_rspec implementation. As the code below shows, this makes it straightforward to spec all these aliases.

 1 hash_store = shared "Hash#store" do |cmd|
 2   describe "Hash##{cmd}" do
 3     it "associates the key with the value and return the value" do
 4       h = { :a => 1 }
 5       (h.send(cmd, :b, 2).should == 2
 6       h.should == {:b=>2, :a=>1}
 7     end
 8 
 9     it "duplicates and freezes string keys" do
10       key = "foo"
11       h = {}
12       h.send(cmd, key, 0)
13       key << "bar"
14 
15       h.should == { "foo" => 0 }
16       h.keys[0].frozen?.should == true
17     end
18 
19     it "duplicates string keys using dup semantics" do
20       # dup doesn't copy singleton methods
21       key = "foo"
22       def key.reverse() "bar" end
23       h = {}
24       h.send(cmd, key, 0)
25 
26       h.keys[0].reverse.should == "oof"
27     end  
28 
29     it "raises TypeError if called on a frozen instance" do
30       should_raise(TypeError) { hash.send(cmd, 1, 2) }
31     end
32   end
33 end
34 
35 describe "Hash#[]=" do
36   it_behaves_like(hash_store, :[]=)
37 end

The very cool thing about this is how useful Ruby’s send method is. And in Rubinius, it gets even cooler, as you’ll see in part II

Rubinius sprint retrospective

2007-09-17T00:00:00-07:00

Last week, Evan Phoenix, Wilson Bilkovich and I met up in Denver (ok, near Denver) for some serious Rubinius coding, aka a sprint. Charles Nutter (of JRuby fame) flew in just to hack with us for 15 hours straight and then flew back home. Very cool. When do you get collaboration like that from someone on a project that could be seen as a competitor? That’s some serious goodwill, folks, and much appreciated. Charles had his laptop out on the taxi ride from the airport digging into some issues we have with including modules. If you hang out in #rubinius, you know there’s a lot of collaboration occurring on different fronts.

This was only the second time I’ve had the pleasure to hang out with these folks in person. The sprint was intended to enable us to get some face-to-face time and work out some foundational issues, hopefully speeding other development and contribution as we head toward RubyConf07. Here’s a few things we accomplished (in no particular order):

Evan hooked up Syck (the YAML parser) using our very cool subtend (Ruby C API compatibility) component.
Wilson hammered out a bunch of StringIO specs in a couple hours.
Evan promptly began writing a Ruby StringIO that passed the specs.
Charles whipped out a bunch of def and case specs, the latter being a serious beast to compile.
Wilson added these wicked colorized backtraces that enable you to quickly pick out the ultimate failure line and some significant points in the backtrace.
Wilson implemented the rest of the missing case support in the compiler.
Charles added some basic ObjectSpace support.
I checked in a complete reorganization and a lot of fixes to our Ruby core library specs. We have around 2,800 specs, of which we’re passing over two thirds. We still have nearly 50% of the core library specs to complete.
Evan fixed our Thread support and added some preemption.
I added (with much help from Evan and some serious gdb sessions looking at x86 machine code) some coolness to our foreign function interface (FFI) to support reading and writing to C integers and doubles from Ruby. (The doubles support still needs a magic touch or two from Evan.) With that I was able to finish off our Math methods support and the rest of the Math specs.
Wilson hammered out a ton of the machinery necessary to support compiling eval. If you don’t realize how awesome (and painful) this is, don’t blame me.
I started on an implementation of File.fnmatch in preparation for Dir.glob (or dear glob is it is not-so-affectionately known to me) and cleaned up our fnmatch specs a bunch.

There’s more where that came from. For all the nitty gritty details, browse the git repository. There’s some cool stuff just over the horizon that we didn’t get to. Evan explained in more detail the architecture of our sampling profiler and our debugger support. Let’s just say you’ll be able to profile a Rails app in production with no significant performance penalty. And remote debugging is on it’s way. Seriously, let that sink in for a minute.

Interestingly, we did not consume 50 liters of coffee each and we had some incredible Japanese food one evening, thanks to a recommendation from one of Wilon’s contacts. Overall, I’m quite impressed with our progress and highly impressed with the technical prowess of these guys. The point it, Rubinius is quite accessible. It’s also some very cool and very real technology that’s going to make writing, running, living, breathing, and enjoying Ruby a lot better.

And last but not least, a big thanks to Sun for their sponsorship, and in particular Tim Bray for his advocacy. Rubinius is a small, ambitious project. But at heart, it’s very true to the value of making Ruby fun.

Dear glob

2007-09-17T00:00:00-07:00

At some point, you likely needed to grab some files and do something with them. You probably had a vague recollection about Dir[] from reading the Pickaxe book. To refresh your memory, you crack it open and fire up irb. Ah, easy…


  irb(main):001:0> Dir['spec/**/*_spec.rb'].each do |file|
  irb(main):002:1* puts file
  irb(main):003:1> end

There’s no denying that Ruby makes this task dead simple. So simple, in fact, that you probably don’t think twice about how that nifty Dir[] method does its work. That is, unless you’re trying to implement it.

During the Rubinius sprint, I realized that having this in Rubinius would enable me to make our continuous integration spec runner much better. Now, I know that there’s a glob function in C that provides behavior similar to what you get in the shell using * and ? to match file names. There’s also a function fnmatch that wraps up some of that magic. No problem. We’ve got this nifty foreign-function interface (FFI) that Evan has graciously provided. Evan recommended I take that route first. Yep, took all of 10 minutes to hook everything up.

Of course, it wouldn’t be that interesting were this the end of the story. It’s not. Our fnmatch specs were mostly passing, but when I looked into the failing ones, I discovered something that I’d probably tried to shield my psyche from. Ruby implements its own fnmatch and glob functions. And when I say ‘implements’, it doesn’t really give you any idea of the pain and suffering involved. Do take a peek:

The Ruby source (stable 1.8.6)
The JRuby source

It doesn’t take but a minute to see that Ola Bini’s java code is extraordinarily more readable than the MRI source. But both are daunting to say the least. So, I’ve decided to take a different route.

 1   def self.fnmatch(pattern, path, flags=0)
 2     pattern = StringValue(pattern).dup
 3     path = StringValue(path).dup
 4     escape = (flags & FNM_NOESCAPE) == 0
 5     pathname = (flags & FNM_PATHNAME) != 0
 6     nocase = (flags & FNM_CASEFOLD) != 0
 7     period = (flags & FNM_DOTMATCH) == 0
 8     subs = { /\*{1,2}/ => '(.*)', /\?/ => '(.)', /\{/ => '\{', /\}/ => '\}' }
 9     
10     return false if path[0] == ?. and pattern[0] != ?. and period
11     pattern.gsub!('.', '\.')
12     pattern = pattern.split(/(?<pg>\[(?:\\[\[\]]|[^\[\]]|\g<pg>)*\])/).collect do |part|
13       if part[0] == ?[
14         part.gsub!(/\\([*?])/, '\1')
15         part.gsub(/\[!/, '[^')
16       else
17         subs.each { |p,s| part.gsub!(p, s) }
18         if escape
19           part.gsub(/\\(.)/, '\1')
20         else
21           part.gsub(/(\\)([^*?\[\]])/, '\1\1\2')
22         end
23       end
24     end.join
25     
26     re = Regexp.new("^#{pattern}$", nocase ? Regexp::IGNORECASE : 0)
27     m = re.match path
28     if m
29       return false unless m[0].size == path.size
30       if pathname
31         return false if m.captures.any? { |c| c.include?('/') }
32         
33         a = StringValue(pattern).dup.split '/'
34         b = path.split '/'
35         return false unless a.size == b.size
36         return false unless a.zip(b).all? { |ary| ary[0][0] == ary[1][0] }
37       end
38       return true
39     else
40       return false
41     end
42   end

This code is only passing 80% of our existing specs for File.fnmatch?, so the jury is still out. And I’m sure someone can make this much better. The lesson for me is that 1) Ruby’s implementation is typically not accessible (I already knew that), and 2) writing Ruby code is a good way to handle tough problems.

But then, you already knew that. ;)

UPDATE: I’ve changed the code here to reflect our current version. It’s now passing 100% of the existing specs.

RailsConf 2007: The case of the missing BDD

2007-05-19T00:00:00-07:00

Looks like the RSpec folks dropped 1.0. Awesome! Congratulations!

Hmm, where’s the applause? This is the good shit. People should be knowing about this. So, as I sit here at RailsConf 2007 looking over the schedule, I’m not seeing any talks scheduled for BDD or RSpec. That’s regrettable. So, my goal will be to embarrass, cajole, entice, encourage (pick your emotional poison) you into trying RSpec. Not sure if it’s right for you? Try it. Not sure if it will work well? Try it. Don’t get it yet? Try it, already!

And while you’re trying it, think about this: How can descriptions of the behavior of a system form the common language that spans the spectrum of development from clients to developers. Do you think your clients care about how many def test_should_be_super_duper methods are in your tests? Why would they?

Would your customer care about something like this?

The Home page
- has an entry field for username labelled 'User name'
- has an entry field for password labelled 'Password'
- has a submit button labeled 'Log in'
- has a link to retrieve a lost password FAILED

1) The Home page has a link to retrieve a lost password FAILED
   <insert client comprehensible failure message here>

Do you think this could form the basis of a dialogue with your clients? Would it be easier to trouble shoot this over a client saying, “The lost password link doesn’t work.” Do you see how this could make everyone’s life easier?

Is this all there is to BDD and RSpec? No, not by a long shot. So… yeah, you guessed it. Try it.

Oh, _why!

2007-05-19T00:00:00-07:00

This post has no nutritional value. Feel free to put it over there by the jelly-filled pastry, bearclaw, and glazed pop tart, go back to your granola. You health nuts, you.

Now, to the jelly filling. Seems that _why dropped by #rubinius, not once, but twice yesterday. I missed it, but hopefully he’ll be back. For your viewing pleasure (no names have been changed to protect the innocents).

19:06   _why >> whoa
19:06   _why >> well fellows
19:06   _why >> i am blown away!
19:07   _why >> looks like things are really comin along
...
19:30   _why >> oh backflip
19:30   _why >> interpolation works, splat works, IO works
19:30   Vagabond >> spat is pretty new
19:30   Vagabond >> I think it's not completely tested
19:30   shadowfiend >> Yeah, like a couple of hours old :-)
19:31   Vagabond >> it's remarkable how much is actually implemented though
19:31   _why >> yeah it's got that new splat smell

The Dev'ilz in ur projects, trackin' ur changz

2007-05-08T00:00:00-07:00

The Dev’il is out; get it while it’s hot. What’s that? you’re wondering.

This is the story behind it. The Rubinius project (where The Best Ruby VM of All Time ™ is being built) had tried a couple project collaboration tools before Evan gave me the ok to try something different. My goal was a project collaboration tool that collected the relevant information in one place, making it easy to access and modify. Communication is key, so I decided to build on the excellent Beast forum.

The big question is: What is relevant information? I have some opinions about that. We have a liberal commit bit policy, which encourages a lot of participation. To balance that, we need to always stay on top of what is being committed. In my experience, that means RSS and graphical browsing of Subversion changes. We need tickets to track bugs. However, our tickets are super light. Basically, a ticket has a title, body, and status. A user can take a ticket to indicate that she is working on it. There are no comments, and no priorities. Tickets are versioned, so a user cannot destroy pertinent information. The ticket body should always be the definitive statement of the problem. No browsing through comments and attachments trying to figure out what is relevant. I’ll be writing more about the design decisions in the Dev’il docs.

I do owe a big debt to Trac , Collaboa, Devalot , Retrospectiva, mostly for showing me (especially Trac) what I did not want to do, which can be a great source of inspiration. The name itself came from a combination of pure frustration working on Devalot and Evan’s quote about rubinius having “a little evil built in”. It was mostly a response to the “alot” in Devalot. I wanted something much lighter and easier to understand and use. Anyway, if you’re offended by the name, it’s very easy to change: drop a file named project.rb in your lib directory that contains your project name.

The project is still under development. I’ve released the source because a few folks were clamoring for it. Drop by the Dev’il forums for features in the works and to offer up some feedback.

Ctrl-C is deprecated

2007-05-08T00:00:00-07:00

Well, not exactly. But here’s a nice anecdote that will give you pause then next time you sprint for the ^C escape hatch.

Yesterday, while compiling Subversion 1.4.3 from source on Ubuntu 6.06, I had invoked the command sudo checkinstall make install-swig-rb and, thinking it was going to overwrite my previous .deb package for subversion, hit ^C. Now that’s something I’ve done a million times, I’m sure. It’s the panic, “Oh shit, I don’t really want to do that” button. When you realize in a split second that there may be dire consequences from that ill-thought command, it’s a natural reflex.

But on that fine day, ^C left me with a root directory that had 700 permissions. The consequence was that nothing except a root shell could spawn. I couldn’t even su <user> as root. Yeah, that’s right. Insane. And this was on a VPS. Lucky for me, Quantact has their Xen VPS setup configured so you can actually access a console remotely without needing to go through your VPS’s networking. Tim at Quantact took a look and suggested that the permissions could be causing the issue. He was right and boy, was I happy that’s all it was.

Most likely, something in checkinstall misbehaved, but needless to say, I didn’t try to reproduce the problem. Now, whenever I head for ^C there will undoubtedly be a microsecond pause to evaluate the relative merits. Hopefully, that won’t result in a Buridan’s Ass dilemma.

Needmore Designs

2007-04-12T00:00:00-07:00

That pretty much sums up my feelings about the majority of the interactive gadgets I encounter, soft or hardware. So, I’m delighted to have the good fortune of joining the talented folks at Needmore Designs. It’ll be a challenge to keep the code as beautiful as these designs, but using Ruby makes that a delicious challenge indeed.

One particularly generous perk for a company this size is that I’ll be able to spend around 20% of my time working on, I like to call them, “personal development” projects. Of course, I plan to contribute that to bettering the Ruby community. In particular, I’ll be spending a good portion on the Rubinius project. This alternative Ruby implementation aims to create an accessible, extensible, powerful, and fast virtual machine. We’re implementing as much as possible in Ruby and we have a lot of very cool things either implemented or in the works.

Not only will this project contribute to Ruby being more suitable for a wider variety of computing tasks, but using rubinius will actually be great for the environment, too! Far fetched? Nope. Our continual thirst for more computing power translates to increasing demand for energy. Any type of energy pretty much means some contribution of carbon to the already overloaded atmosphere. So, when rubinius runs your favorite Ruby applications and web 2.0 sites more efficiently, that reduces the required energy and that’s better for the environment. Ok, I’ve never made that pitch before but it’s a great reason join up and help out.

So, your moment of zen: not only has Needmore given me this great opportunity to work with talented design professionals, but they’re also contributing to a better environment. And if you treasure the abundance of green that we are privileged to have all around Portland, you know that’s a very cool thing. Around the circle goes.

Behavior-Driven Collaboration

2007-03-28T00:00:00-07:00

Eegads, how time flies. At the March 6th meeting of pdx.rb, I gave a short talk introducing BDD (Behavior-driven development) with a couple examples from RSpec. Since David Chelimsky has already done a terrific job writing an introduction tutorial, I decided to focus more on the big picture of BDD. My take is that BDD is some rather sweet glue that can foster collaboration across the spectrum of folks involved in software development, from clients to designers to developers to users. So far, we’ve got a lot of turf wars going on, and as in any war, there are casualties. In software there are crappy products, tortured users, unhappy developers, exasperated designers, cost overruns, and misunderstandings galore. BDD isn’t some magic salve that can be slathered on a messy process like butter on a toasty English muffin and make it all better, but I do believe it has promise.

So, here are the slides from my talk. (Right-click and Save As… Sorry, MIME types are not set right at the moment.) Unfortunately, in a tremendously idiotic move born of high stress, I deleted the original slides before I had uploaded them. Fortunately, I had some notes and a partial earlier draft, so I was (mostly I hope) able to reconstruct them. There is unfortunately no audio from the talk. As with most slides, these are marginally useful without it, but browse and feel free to ask questions.

There is one part that seems highly promising to me and easily attainable now. As you know, RSpec allows you to spec the views separately from the controller. This is a fantastic tool that interaction designers could use to communicate with developers. The syntax that an interaction designer would need to learn is fairly simple (most interaction designers are not developers), and you shouldn’t have much (if any) conditional code in your specs anyway (conditionals being one of the more difficult parts of programming to get right in my experience). The specs can accompany wireframes and provide another artifact of communication that bridges the different domains of interaction/interface design and development. Exciting stuff.

Rubinius Hits Primetime: Let the FUD Begin

2007-02-14T00:00:00-08:00

For being a very young project (Evan’s been working on it about a year, but it’s only been in the public view for a few months), Rubinius has quickly hit primetime with the first (that I know of) FUD-ish review The basic premise is that pypy had money, minds, and at least one goal to produce a simpler VM than CPython, but as it nears 1.0 it’s much more complex than CPython. And since Rubinius is at least lacking in official monetary backing, it’s got a real tough row to hoe. Maybe; but I’m unconvinced.

The jury is still out on the mind-share Rubinius will ultimately attract. We’re certainly not sneezing at anyone’s contribution. And if you hang out in the channel, you might notice some rather interesting conversations between the JRuby folks and the Rubinius folks. You’ll definitely hear conversations about PIC, STM, concurrency models that make you go “Ohh” (thanks MenTaLguY), the possibilities of LLVM, even JIT’ing (though, to hear JP tell it, we’re hopelessly unqualified for that). I’d guess a who’s-who on Rubinius would turn up some rather accomplished folks. Personally, It is encouraging that I didn’t have to write the book on compilers to be able to hack on Rubinius. But really, do you have to know everything to jump into a project and start learning and applying it?

Let’s just say, I’m sure glad Linus was not deterred by the fact that IBM, SCO, AT&T, and Micro$oft had billions of dollars to spend on operating system development. Anyway, I’ll bet the “Yes, but does it scale?” posts will be showing up shortly. C’mon JP, live a little.

TSort in sirb

2007-02-13T00:00:00-08:00

They say good things come in threes, or perhaps I’ll bore you with another post about Rubinius, but what the heck. So, the other day I imported Ruby’s standard library from the SVN repository into the Rubinius repository. I’ve been itching to go through and see what successfully loads. Ah, now that we have sirb, the perfect opportunity presents itself.


$ ./shotgun/rubinius apps/irb/sirb.rb
sirb(eval):000> require 'tsort'
Couldn't find tsort.rbc in runtime/compiler.rba (No such file/-1)
Couldn't find tsort.rb in runtime/compiler.rba (No such file/-1)
Unable to find 'tsort' to load
                 main.raise at bootstrap/04kernel.rb:26
               main.require at core/compile.rb:91
       main.__eval_script__ at (eval):1
    CompiledMethod#activate at bootstrap/compiled_method.rb:52
            main.__script__ at apps/irb/sirb.rb:79
                  main.load at core/compile.rb:56
            main.__script__ at core/__loader.rb:50

Hmm. Oh, yes! I think we need to check load path. (I’ve since added stdlib to the default load path.)


sirb(eval):001> p $:
[".", "runtime/compiler.rba", "lib"]
=> nil
sirb(eval):002> $:.push "stdlib"
=> [".", "runtime/compiler.rba", "lib", "stdlib"]
sirb(eval):003> require 'tsort'
Couldn't find tsort.rbc in runtime/compiler.rba (No such file/-1)
Couldn't find tsort.rb in runtime/compiler.rba (No such file/-1)
=> true

Oh baby. Just ignore those couple complaints. It’s that => true that we’re looking for. Nice! Now, we’ll just try the example in the tsort.rb file.


sirb(eval):004> class Hash
sirb(eval):005>   include TSort
sirb(eval):006>   alias :tsort_each_node :each_key
sirb(eval):007>   def tsort_each_child(node, &block)
sirb(eval):008>     fetch(node).each(&block)
sirb(eval):009>   end
sirb(eval):010> end
=> nil
sirb(eval):011> {1=>[2, 3], 2=>[3], 3=>[], 4=>[]}.tsort
=> [3, 2, 1, 4]
sirb(eval):012> {1=>[2], 2=>[3, 4], 3=>[2], 4=>[]}.strongly_connected_components
=> [[4], [2, 3], [1]]
sirb(eval):013>

Excellent! Er, what’s that? Did I hear you say, “So, big deal, tsort.rb isn’t that tough”? Well, ok, it isn’t. (I didn’t even know about TSort before browsing through the source.) But, we can do other stuff, too.


sirb(eval):008> require 'csv'
=> true
sirb(eval):009>

Yep, that’s right, csv.rb compiles! But, I’ll spare you the pain that ensued trying to use it. Baby steps. Ok, enough fun. C’mon kids, there’s a bunch of the core library still to implement!

Sirb: The Rubinius Loupe

2007-02-11T00:00:00-08:00

If my previous post about sirb got you drooling a bit, read on. I’ve added some simple command line arguments to sirb that enable you to request that s-expressions, assembly, or bytecodes (or all of them) are printed each time through the RCEPL loop. You can get a hint at all the goodness by passing the -h switch.


$ ./shotgun/rubinius apps/irb/sirb.rb -h -p -x -s -b
  Usage: sirb [options]
    sirb is a RCEPL (read, compile, execute, print, loop) program for rubinius

    Options: 
            -p   Print the parsed s-expression
            -x   Print the normalized s-expression.
            -s   Print assembly instructions.
            -b   Print bytecode encoding.

sirb(eval):0> puts "Hello, Rubinius. You rock my world!" 

S-exp:
[:newline, 1, "(eval)", [:fcall, :puts, [:array, [:str, "Hello, Rubinius. You rock my world!", 0]]]]

Normalized S-exp:
[:newline, 1, "(eval)", [:call, [:self], :puts, [:array, [:str, "Hello, Rubinius. You rock my world!", 0]]]]

Asm:
#line 1
push_literal 0
string_dup
push self
send puts 1
ret


Bytecodes:
"\v\000\000\000\000:\f)\000\000\000\001\000\000\000\001'\000\000\000"
Hello, Rubinius. You rock my world!
=> nil
sirb(eval):1>

As I mentioned, sirb is rudimentary at the moment. But it shows you how much can be done with Rubinius right now. Swell! And, a big thanks to mae for fixing up how the output of the execution prints (no more ["foo"]).

Irb on Rubinius

2007-02-11T00:00:00-08:00

There are cat people and dog people, but when I say “Man’s best friend”, I don’t mean either. I’m talking about that program that feels like a home away from home: irb. So, I’ve been chomping at the bit to get something resembling irb running with Rubinius.

Today, a few of us pdx.rb folks got together for some Rubinius hacking. Of course, irb was one of the things I proposed that we work on. We had a go at it, but in the end we didn’t get too far. irb is no trivial application. But, I think a good part of it was my fault. I’ve been working on these specs for so long, where at the moment we’re forced to send everything to rubinius in a subprocess, that I failed to grasp that we would have a persistent machine. Anyway, let’s cut this short: Folks, we have a rudimentary sirb!


sirb(eval):0> class Hello
sirb(eval):1> def greet
sirb(eval):2> puts "Hello, irb!"
sirb(eval):3> end
sirb(eval):4> end
=> nil
sirb(eval):5> Hello.new.greet
=> Hello, irb!
["Hello, irb!"]
sirb(eval):6> @s = "Hello, earthling"
=> "Hello, earthling"
sirb(eval):7> p @s
=> "Hello, earthling"
["Hello, earthling"]
sirb(eval):8> Hello
=> Hello
sirb(eval):9> Hello.class
=> Class
sirb(eval):10> Hello.new
=> #<Hello:0x6696cc>
sirb(eval):11>

In the words of Napolean Dynamite: Sweet. Of course, it is pretty rough right now. You’ll notice the use of an instance variable, @a. That’s because I couldn’t figure out how to get local variables to work. Basically, everything is evaluating with the default receiver being an instance of Object.

Well, warts and all, I think it’s a beautiful thing.

Waiter, there's a Tuple in my Array

2007-02-09T00:00:00-08:00

Implementing Ruby in Ruby raises some challenges. In the beginning, Ruby was implemented in C. Well, at least the interpreter and core libraries. In C, you have the standard library that provides things like the math functions, functions for manipulating arrays of chars, which can be considered strings of text, etc. So, what happens when you want to build Ruby with Ruby? Well, you use Ruby, of course. And since Ruby is a class-based language, you probably will use a class. Ladies and Gentlemen, meet Tuple.

In Rubinius, the class Tuple is present for all to see. And it’s put to good use, underlying the implementation of Array for instance. You can use it, too:


sirb:001> t = Tuple.new(3) => #<Tuple: nil, nil, nil>
sirb:002> t.put(1, "foo")  => #<Tuple: nil, "foo", nil>
sirb:003> t.shift          => #<Tuple: nil, "foo">

Here’s another example. Have you noticed the chr method available on instances of Integer. Well, while implementing some String methods, I decided it would be nice to say, some_str[i].isspace. If you’ve used C, you know that there is this thing ctype that has macros or functions like isspace. Stumbling around a bit, I finally decided to add a module CType and mix that in to Integer. Works pretty well.

So, there’s Tuple in all its glory sitting right in the middle of your Array. Is that a problem? I don’t think so. This is Ruby, folks. Duck typing, open classes, mixins, methods appearing and disappearing any old time of the day are the norm. To clarify, the behavior of the core classes, like Array, which have a different implementation in Rubinius from that of the C one used in MRI, will not change. In other words, if you use Array methods, they’ll behave like Array methods.

I say, if your code depends on a class only having certain elements when you invoke Foo.ancestors, for Pete’s sake, refactor. And if you happen to have a ByteArray or Tuple in your application, refactor that, too. We’re the VM, we get certain privileges. Now, suppose you wrote your application to use all the deliciousness of Rubinius, thereby demonstrating you are a wise and thoughtful old soul. (Yes, that’s right, CompiledMethod, Method, MethodContext, BlockContext, BlockEnvironment, and more are classes. Yum. And that is just the tip of the iceberg, I assure you.) But, you heavily built in dependencies on, say, Tuple, and now you’re complaining it doesn’t run on MRI… Well, here’s your warning. In other words, don’t be that silly.

However, if this raises concerns for you, here’s a suggestion: We’re working feverishly on RSpec specs for all of the core and standard Ruby libraries. We also have a directory for incompatibilities in the Rubinius implementation, which is predictably found in spec/core/incompatible. Feel free to join us in writing these specs. That way, you can run them against your favorite Ruby implementation and know what works. Perhaps, you can even integrate this knowledge into your own project’s specs. You are writing specs for your projects, aren’t you?

Happy New Year

2007-02-08T00:00:00-08:00

Sometime between Jan 1 and Chinese New Year it finally dawns on me that we’ve begun a, well, new year. I start writing the correct dates on checks, the bank honors them, the rent gets paid. Things go much more smoothly.

So, while trying to keep up with the homework from my brilliant professor of game theory, I thought I’d write a few predictions for 2007. Sure, some of these you could call resolutions, but since it’s up to me to do them, lumping them into the predictions category might up my percentage correct score.

This blog will finally get a real theme, shorter posts, an articles section, recent and popular links. Also, I’m going to spruce up the RSS served to be more useful in a reader. Finally, to break the asymmetry of time, I’ll be putting titles of upcoming posts in before the post appears. When it does, the title will become a link.

2007 will be the year of the alternate Ruby implementations. Several will be competing with MRI (Matz’s Ruby Interpreter) for your love and admiration. Among these are YARV, JRuby and Rubinius . One of those will have a tremendous impact on enterprise coding and another will have a tremendous impact on CS in general. We’ll see how that pans out. I’ll be posting more about Rubinius soon. In the meantime, don’t miss the wonderful, appreciated, and tireless work of Pat on the On-Ruby Blog to evangelize Rubinius and other cool projects.

I’ll be learning at least one new application this year: Omnigraffle. This comes highly recommended by our Creative Director, Allison. Truth is, design and graphics is way more interesting to me than plain ol’ code, but writing code usually wins out in a duel, so I’m hoping to even things out a bit.

Finally, I hope to collaborate on writing a book; I’ll let you guess the topic. And if things go well, I’ll be co-announcing a currently top secret project. Ahh, I see the eyebrows rising. Also on my list of things to write about are some more thoughts on d3 and our experiences with it at PLANET ARGON.

In the near future, check out What’s the Big Idea and Another Liberating Constraint: Contracts.

Tuesday: partly sunny, chances of showers

2006-11-15T00:00:00-08:00

Maybe it’s all the rain during this season. I sure look forward to those dry, maybe sunny days when I notice them on my dashboard weather widget. Alas, I’m often disappointed. A nice, bright sun a few days away mysteriously turns into a cloud or rain drops. It’s a big disappointment. Often I call up the weather station and complain. Why can’t they be more accurate? Why don’t they develop better processes? Sometimes it gets nasty and I threaten to sue them. I mean, if I’ve planned my picnic because Saturday is forecasted to be sunny, shouldn’t somebody have to pay to clean my soggy picnic blanket?

Actually, that doesn’t happen. In fact, we’d consider somebody a tad looney if they asserted that somehow the weather forecast should be guaranteed. We know that science is always advancing but weather forecasting is imprecise because the complexity of the “weather system” ensures it is affected by processes described by chaos and complexity theory. There is an inherent limit on precision.

The most common analogy for software development is some form of construction. In fact, we talk about system architecture, building components, software construction. Creating software is less like building and more like weather forecasting.

Methods and processes for constructing buildings are quite well established. And all buildings share far more commonalities than differences. Most importantly, interacting with buildings involves human cognition in extremely limited ways. Software, on the other hand, involves a tremendous amount of human cognition and social effects. There is conceptualization, definition, communication, attitudes, opinions, assumptions, the entire spectrum of human complexity. Given this, we might be advised to study meteorology as much as architecture.

Good fences make good neighbors

2006-11-15T00:00:00-08:00

We might as well accept it now: global warming is real. We live in a finite world with finite resources. Given that, why would we want to expend time and effort building fences. They just sit there. Maybe they contribute to the aesthetic but they won’t help the wheat grow any better in the field. Building them diverts productive capacity from planting, tending, harvesting. Be that as it may, long ago folks realized that good fences serve a purpose.

Fences create boundaries and boundaries define areas. And that’s the value: this is my area, that is yours. Positive relationships also depend on boundaries: this is my responsibility, that is yours. Contracts are like fences: they describe boundaries that define areas. We may be tempted to think that nice, intelligent, professional people can work out differences of understanding without the need for lawyers and contracts. But be not tempted. The jury may still be out on the question of whether there exist nice, intelligent, professional people. But the lawyers are here to stay.

Good neighbors build and maintain fences together. They share the responsibility and share the benefits. Contracts are necessary; collaboration is necessary. Contracts through collaboration. It’s less of a prioritization of values and more a matter that good collaboration generates good contracts, and good contracts support good collaboration. Good!

What the hell

2006-11-07T00:00:00-08:00

Warning. Long quote to follow. There, I’ve warned you.

The most useful products are those where the developers have understood what the product is intended to accomplish for its users and how it must accomplish that purpose. To understand these things, you must understand what kind of work the users want to do and how the product will affect that work and fit into the organization’s goals. What the product does for its users and which constraints it must satisfy in this context are the product’s requirements. Apart from a few fortuitous accidents, no product has ever succeeded without prior understanding of its requirements. It does not matter what kind of work the user wishes to do, be it scientific, commercial, e-commerce, or word processing. Nor does it matter which programming language or development tools are used to construct the product. The development process—whether agile, eXtreme Programming, prototyping, the Rational Unified Process, or any other method—is irrelevant to the need for understanding the requirements. One fact always emerges: You must come to the correct understanding of the requirements, and have your client understand them, or your product or project will fail.

Emphasis added. Quoted from Mastering the Requirements Process, 2nd Ed.

To paraphrase: Knowing what you are doing is an absolute prerequisite to successfully doing it. It’s not a guarantee you’ll be successful, but there’s no substitute for it.

Agile means different things to different people. At times, responding to change over following a plan is translated by someone into planning is not agile; just jump in and see what happens because you won’t know until you get there anyway. There’s also the agile concept of not doing more than necessary at each point. So, there’s a tension set up between looking forward to where you’re going and looking down at what you’re doing. Overdoing either get’s you in trouble. Like reading a newspaper while walking through a crowd. Pay too much attention to the people and you don’t get reading done. Read too much and you’ll bump into people.

A few months ago, when Rails was really new, there was this idea circulating that Rails made it so easy, one didn’t even really have to know Ruby to program with Rails. Yeah, uh, right. Unfortunately, claiming to be agile is also often substituted for knowing what you’re doing. It takes a lot of knowledge and experience to balance the tension described above. It takes a lot of judgment to know how much do I need to do right now? Agile is not a substitute for knowledge and experience, although it may guide you to better apply those.

Dine in or take out: it's all free

2006-11-06T00:00:00-08:00

One of the more amazing and wonderful things about the internet is the sheer volume of information available for free. And just like going to a friend’s party, one way to show appreciation is to partake of the feast prepared in honor of the guests. But a lot of the time it’s more like a grocery store than a nice restaurant. Lots of food but not the place to savor the cooking. Once in a while, though, you happen across a gem.

Since everyone has a different background, we’re all learning at our own rates. If you’ve always wanted to understand more about declarative versus imperative languages, syntactic sugar, dynamic versus static typing, tail recursion, lazy evaluation, currying, concurrency, exceptions, computation models, polymorphism, etc. and you’d like to learn about it in a comprehensive, integrated fashion, then definitely grab a copy of Concepts, Techniques, and Models of Computer Programming, here’s a direct link to the pdf. The book teaches the main concepts of computer science using the Mozart Programming System based on the Oz programming language.

Now, for the fun part. If you’re using OS X and have ports installed, getting the Mozart system is as easy as:

sudo port install mozart

There, you’re ready to dive right in. However, one thing will make your experience much easier. Install Aquamacs: Emacs for OS X. Mozart will use it if it finds it, and for me it’s a nicer experience than Emacs in iTerm.

Mephisto trunk + svk + svn + cap = blogtastic

2006-11-05T00:00:00-08:00

One Mephisto feature I really like is saving an article as a draft. Unfortunately, I’m more likely to start an article than finish it. So, I’m looking forward to that auto-write feature that maybe Rick is working on. Until then, I’ll just have to continue finishing them on my own, even if it takes a few weeks. Ahem, like this one.

Inspired by Octopod yet again, I’ve got the best of worlds: Mephisto edge at my chosen version mirrored to my own svn repository for my own hacking, all cap deploy’d to my VPS.

I decided to put my svn repository on my VPS and develop in that. Basically, I use svk as a fancy adapter with the added advantage that I can update from Mephisto edge whenever it’s convenient for me. Installing svk on Ubuntu 6.06 was a breeze. From there, I mostly followed Octopod’s recipe: First, I create a mirror for Mephisto in my svk repository, then pull in Mephisto.


svk mkdir //mirror
svk mkdir //mirror/mephisto
svk mirror http://svn.techno-weenie.net/projects/mephisto/trunk //mirror/mephisto/trunk
svk sync //mirror/mephisto/trunk

That’s the Mephisto side of the cable. Now, for my local svn side.


svk mkdir //svn
svk mkdir //svn/mephisto
svk mirror svn://svn.brightredglow.com/mephisto/trunk //svn/mephisto/trunk
svk smerge --baseless //mirror/mephisto/trunk //svn/mephisto/trunk

Now, I simply use the sync command above to pull new Mephisto changes to my svk repository and the following command to push those changes to my svn repository:


svk smerge --incremental --log //mirror/mephisto/trunk //svn/mephisto/trunk

From here, it’s pretty simple. I check out a working copy of my svn Mephisto trunk and work normally. Using Capistrano, I can cap deploy or cap rollback to my heart’s content.

Finally, Mongrel + Nginx + Mephisto are working flawlessly. My mongrel processes never get over about 35MB and they hum along day after day, week after week. Sometimes load gets too high for my VPS, but I can live with that. Speaking of VPS, Quantact has recently added OpenVZ options. They’re quiet but dependable. And recently I logged in to my control panel to find that I had 33% more RAM and another gig of disk space. I’m not saying they do this all the time, but for whatever reason they did it, I appreciate it.

Inspiration under a rock

2006-11-05T00:00:00-08:00

PLANET ARGON’s office is a hop, skip, and a jump (about 3 city blocks) from Portland’s Pearl District, which is a really nice thing for lunch with one’s girlfriend. Everyone knows that girls love to shop, right? Gross over-generalization or not, it’s true. Anyway, while browsing through Anthropologie, I’m of course drawn to any stack of books I see. On a nice diagonal book shelf, I located two quite interesting books: Modern Glamour: The Art of Unexpected Style and China Chic.

Yeah, both a little different than typical books I read, but they got me thinking. First, they have lots of interesting compositions; colors, shapes, arrangements that are a pleasure to look at. Second, the compositions are, of course, much more than just collections of colors and shapes. It’s the arrangement part, the whole design, that got me thinking.

With software, we’re often confronted with a level of granularity that isn’t very useful. If an application has some sort of access control, we may be required to twiddle read/write/execute-type permissions for each object. That’s just an example, but one that is particularly problematic for non-savvy users. The implementation model is bleeding through worse than a horror-flick bloody corpse. At that level of granularity, it’s like all the fabrics, wall-hangings, window-treatments, and furniture are arranged in a nice matrix of boxes in the middle of the room instead of making up the room as a whole.

For one of our projects, we’re exploring how we can create idioms that allow the user to manipulate objects at a much higher level. For example, adding someone as a co-worker to a project allows them to make changes, while sharing the project with a friend allows the friend to leave comments but not make changes.

So, are you creating stylish, integrated, holistic rooms in your software? And where do you find odd inspiration in your everyday life?

Principles of d3: simplicity

2006-10-12T00:00:00-07:00

If you’ve paid any attention to our discussion of d3, you’ve noticed that we’re just beginning. We’re trying to articulate the principles that we believe define an effective approach to “client” interaction. I say “client” because it’s not necessarily true that the people are clients in the traditional sense. Organizations often address that with a term like “internal client” to refer to another department, for instance. But that’s a different topic.

In the #d3 channel on irc.freenode.net the other day, Brasten posted this:

17:23   brasten >> anecdote--
17:24   brasten >> spend 30 minutes on the phone with a client discussing a project...  going over all the features he 
                   wanted.
17:25   brasten >> weird stuff.... needed to send small emails with outlook-compatible attachments... import CVS 
                   documents, yadda yadda... none of it made any sense...
17:25   brasten >> finally I asked, "what exactly is the PROBLEM you're having that this project needs to address??"
17:26   brasten >> the guy literally sighed.. paused... and said, "I just want to be able to run my business from my 
                   BlackBerry."     ahhh.... now that's a very different project....we can solve that.
17:26   brasten >> and we were the first developers to actually ask him that.
17:27   tacodog >> hah
17:27   brasten >> -- end anecdote.

You may have had this same experience numerous times. It happens frequently enough to make us ask, “what’s the dynamic here?”

I think this exchange illustrates one of the core principles we’re after: we prefer problem statements because they will always be simpler than a solution or implementation. The problem statement is at a different level of abstraction. It is easier to understand, and thus easier to talk about. The problem statement is as close as we can get to the core. Every possible solution or implementation will be layered around this.

Simplicity is an interesting concept. My dashboard dictionary has this as one definition:

the quality or condition of being easy to understand or do.

We often refer to implementations in terms of simplicity, or simple versus complicated. It’s hard to imagine a compelling argument for something being more complicated than it needs to be. In general, we tend to equate “simpler” with “better”.

As a principle of d3, we want our client interaction to be simpler. So we want to talk about problems with our clients. We want these to be the concrete, explicit elements we dialogue about.

I’ll admit, I feel silly sometimes trying to describe these ideas. They seem really basic. But if this whole dialogue thing were that simple (pun intended), we would rarely, rather than frequently, have exchanges like that above. So, drop by #d3 and help us figure out how to make it as simple as it should be, but no simpler.

User interaction blunders

2006-10-08T00:00:00-07:00

I typically conduct all of my personal business online: mortgage, credit card bills, utility bills, cell phone bill, etc. It’s a great convenience. That’s probably the primary reason. It’s simply more convenient than anything else. I can’t think of any other reason I do it. If there were some attendant in a window I frequently walked by, I’d just as soon give the person a check or credit card to pay a bill. With no waiting line, it would take about the same amount of time.

But since online is the most convenient, I usually use it. Now, since paying these bills means I’m parting with hard-earned money, I tend to be pretty critical of the process. Add to that the fact that these folks, bankers, etc. generally have a ton of money, I expect them to present a pretty nice experience when separating me and my cash. If not nice exactly, it should work well.

So after an experience with Chase’s credit card site, I decided to send them a little note of helpful criticism. I will say, that Chase’s newly redesigned site is probably the best of the 8 or so I frequently use. Even so, I don’t understand how stuff like this gets into the website. Maybe I just have too high of expectations.

Hi,

I just went in to update my business phone number today and was required to fill out every piece of information about my address and personal phone number. That is plainly stupid. I wanted to change one small bit of info and I had to retype everything. There was not even a way to load the existing values into the form.

Please consider your user when designing these forms. Making me enter all that information again when I did not want to change it was both a waste of my time and created the potential for me to accidentally enter incorrect information.

Overall, your site redesign is quite nice and I enjoy using this website much more than the last one. However, things like this form are awfully annoying when forced to deal with them. It creates the tendency to not want to update information because of the hassle. That’s surely not the result you’re aiming at.

Hope you can get these things straightened out. Just because they may be rarely used parts of the website doesn’t mean they should not get the same level of intelligent design. And if the folks who are designing the website don’t understand these issues, that’s a big problem.

Cheers,
Brian

So, do you think I’m expecting too much? What are your favorite examples of websites that should be a cut above, but often fall flat on their faces?

What's it worth to me?

2006-10-04T00:00:00-07:00

After you begin practicing TDD, or better BDD, for even a short while, it clicks. Once you get in the habit of thinking about the behavior you expect and describing that first in a test or spec, you realize there is a qualitative difference in how you are writing code. It’s not simply that you are writing code faster, or writing better code. You are working differently. A quantum leap, if you will. It’s a great, “Ah, hah!” experience.

I sketched a couple diagrams to try to describe this difference while chatting with Robby about creating shared meaning with clients. Check them out below.

p. Figure 1. Traditional linear perspective

p. Figure 2. Circular BDD perspective

The, seemingly, traditional view is linear. You start with no feature, do a bunch of whatever, and at the end you have the feature completed. The TDD/BDD view is quite different. Following the mantra—red, green, refactor—you start with a description of your expectations, you generate code that meets those expectations, and then you refactor. For all but the simplest bits of code, you will likely refactor numerous times. Refactoring is an explicit, purposeful, integral, regular part of creating code. And it’s not, typically, something you do to fix errors, which is how someone with a traditional view tends to understand refactoring. You refactor when your code is meeting your expectations (tests, specs) and those ensure that the product of your refactoring continues to meet your expectations. You refactor because the process of writing code changes how you are thinking about and understanding the code you write.

It’s difficult to find an analogue to this process in our everyday world. In construction, we would surely not build a room and then tear it down to rebuild it because our understanding changed. But an architect may well do that many times with a model in the attempt to create a solution that works. There’s no accurate, faithful way to transfer the architecture paradigm to software development. Many have tried, failed, and many will try again. Instead, BDD is an essential part of the software development paradigm and it’s processes should be understood, to some degree, by everyone participating in software development, clients included.

One significant challenge that we encounter is communicating the importance or value of a particular recommendation. There is more than conveying the information. We have experienced explaining to a client why it was necessary to refactor a particularly important part of an application. (In this case, it was more of a redesign, but since the behavior of the application was not intended to changed, we viewed it as a large scale refactoring.) The client acknowledged that our recommendation had technical merit, in some sense, but continued to view it as something of a diversion with little practical value. After all, the “features” were in place. We weren’t asserting that we would significantly change anything that the user interacted with. And the client was feeling the crunch of getting new features in front of the users.

So, the problem is much more than communicating information between developers and clients. It is essential to dialogue in a way that shared meaning evolves. The shared meaning brings our values into alignment. Then a true, rewarding process of collaboration powers the endeavor.

Pardon my dust

2006-10-03T00:00:00-07:00

And now for a public service announcement…

Over the past month I’ve made a lot of changes to my VPS. Plug for Quantact, they’ve been great. Actually, I almost never talk to them. Which is just the way I like it; means everything is running fine.

So, a few highlights if you’re interested:

Debian 3.1 to Ubuntu 6.06 Dapper Drake.
lighttpd+fcgi to lighttpd+mod_proxy
lighttpd+mod_proxy to nginx
Typo to Mephisto

You probably haven’t noticed much but the last one. Hopefully I didn’t put any wrinkles in your day. I’ve been using Feedburner almost since the beginning of this blog. That feed has been updated and I believe is working. I’ve set up a route for the old Typo-style URLs so your links should not break. I’ll be getting tag based feeds going as soon as I can. And I’ll lovingly replace this awful default Mephisto theme. Justin Palmer is awesome, ~~but I think he stubbed his big toe with this one~~. (Oops, the Hemingway theme is by Kyle Neath. Sorry Justin.)

I’ll post soon about setting up Mephisto in svk and svn, thanks to inspiration from Octoblog. Actually, double thanks, since Octopod inspired me to set up my VPS in the beginning.

Why we focus on solutions, not products

2006-09-21T00:00:00-07:00

Developing anything in the digital realm can be fantastically rewarding precisely because the ropes on our wings of fantasy are greatly weakened. We can imagine practically anything, and we can implement a lot of what we can imagine. The whir of excitement in our stomachs at the vision of some excellent idea prompts my favorite utterance, “Wouldn’t it be cool if…”

Unfortunately, our flights of fantasy must interface with the “physical” world, even if through the digital medium. Faster than light travel doesn’t exist. Perpetual motion doesn’t exist. There is a cost for everything, and it’s thousands of orders of magnitude greater than the few molecules of ATP that literally fueled our fantasy.

No great revelations there. But it sure helps to ground ourselves with this frequent reminder.

When it comes to software, I find that the single most difficult thing for anyone to do is think outside of the Feature Fantasy. This applies to everyone, developers, designers, clients, users, people. Everyone. With a breathless hop, skip, and jump, we leap right over the problem, the solution, and into some particular implementation. It’s so automatic I would have assumed that we’d been taught how to do this in specialized classes since grade school, with some folks getting advanced university degrees in it.

At PLANET ARGON, one thing we are trying to develop through our process and our exploration of d3, is how to effectively separate the problem, the solution, and the implementation. I would say better than 90% of our discussions start out at the feature, or implementation level. We actively and tireless attempt to refocus this back on the problem, describing and documenting it so we can think and dialogue about solutions.

As Jeremy has explored a bit in his article on Feature Ecology, the interaction or relation between features actually alters the product as a whole. Again, not an astounding idea, but the consequence of this is that a lot of time is spent chasing your tail if you focus on features without the benefit of well understanding the problem and solution.

Another benefit of focusing on the problem and solution is that it provides a natural meeting ground between clients and developers. Often the clients are not the users of the software. This makes it even more important to explore the problem to be solved rather than the client’s ideas of particular features. One of the most valuable things I took from reading Alan Cooper’s original About Face almost a decade ago is that no one would use your software if they didn’t have to. How’s that for a response to, “wouldn’t it be cool if…”

If you want to create software that people won’t hate using, focus on the problem and a very good description of a solution. Use your solution to implement a product. Get feedback from people on your understanding of the problem, your description of a solution, and only lastly, on your implementation of the product.

Are your tests findable

2006-09-20T00:00:00-07:00

I probably use Google search on average 20-30 times a day. It’s an indispensable tool for me. It’s definitely not the last word in search engines. I can imagine a lot of additional functionality, like contextualizing my searches over time or within categories. But overall, it works reasonably well.

Lately, I’ve been reading Ambient Findability by Peter Morville. He provides this definition for findability:

findability n

The quality of being locatable or navigable.

The degree to which a particular object is easy to discover or locate.

The degree to which a system or environment supports navigation and retrieval.

Peter explains that findability is defined at both the object and system levels. For example, the title of a document or the color of a pen on your desk. At the system level, you can ask questions like, “Can a traveler navigate an airport,” or “Can a user navigate a website.”

While these ideas help us frame the concepts, how do we evaluate findability in concrete situations. Not surprisingly, there are elements of findability that we can examine:

Findability requires definition, distinction, difference. In physical environments, size, shape, color, and location set objects apart. In the digital realm, we rely heavily on words. Words as labels. Words as links. Keywords.

Hmm, interesting. At PLANET ARGON, we have been evaluating using the rspec_for_rails plugin to focus more heavily on BDD. One of the things BDD emphasizes is leveraging the power of words to help us think more effectively (accurately) about our code. We have been incorporating BDD style method naming into our TDD practices, as well as refining our ability to specify application behavior in our tests.

One thing I dislike about the regular Test::Unit framework is that the methods are often organized ad hoc. Whatever makes the most sense to the programmer when adding those particular tests. Sometimes it’s just appending the method to the list of existing methods. Most often, I notice some sort of organization. Which makes sense; that’s how we deal with information. But the framework doesn’t provide any guidance here.

With the RSpec framework (check out RSpec on Rails), you can arrange your methods into contexts. I think this encourages better organization and contributes to the quality of findability as discussed above. Tests are just another type of information we interact with. How much effort does it take to locate what you’re looking for, understand what the tests are saying, understand the output when a test fails? Following along Jeremy’s questions about acceptable tests:

Are your tests findable?

The tangled web: data migrations

2006-09-06T00:00:00-07:00

Migrations are one of my favorite things about Rails. I remember being amazed at the simplicity and usefulness when I first saw them. I also remember a previous boss asserting that we would just use SQL in the migrations because, to paraphrase, that way we would know what was happening. There’s that saying about tossing rubies before chickens, or something. But as I was saying, migrations.

One thing that makes migrations particularly useful is when they are the only means used to change the database. That way, you know the state you are in and unless you have a rare, non-reversible migration, you can migrate up and down at will. That all works great for the database structure. But what about data? We obviously won’t be funneling all the data changes through migrations. This isn’t much of an issue if your application starts as a blank slate and acquires data through use.

Recently we had an application that needed a bunch of preloaded data. Unfortunately, it wasn’t a one-shot loading of data once the structure was mostly solidified. I needed to try out stuff and wanted the super useful migrations to be my tool. Previously, this had been done with some ad hoc `rake` tasks, which turned out to be, frankly, a mess.

The single biggest challenge working with the data was the fact that the state of the data would be changing independent of the migrations. There was no way to ensure the state between one migration and the next.

I decided that the one thing I could assert before and after a migration was that certain records with certain id’s did not exist (pre-) and then did exist (post-migration) (reverse that for migrating down). It turned out this would be good enough for what I needed.

My first stab at implementing this was a couple helper functions that I used in migrations to update the datafiles (in yaml format) with keys for the id’s. This was a big problem because the files would be changing and were tracked in subversion, so they couldn’t really be shared by more than one developer. Playing around a bit, I came across this idea:

 1 class CreateGeographicTypesFacts < ActiveRecord::Migration
 2   def self.up
 3     capture(:features, :details) do
 4 			# do whatever needed to create records
 5     end
 6   end
 7 
 8   def self.down
 9     release :features, :details
10   end
11 end

Basically, as the data migration runs, I want to keep track of what records are created (and which of those are possibly destroyed). Obviously, this assumes that I’ll be adding data on the up migration and removing it on the down. To implement this, I opened up the `ActiveRecord::Migration` class and added a couple methods:

 1 def self.capture(*args, &block)
 2   args.each do |name|
 3     klass = name.to_s.classify.constantize
 4     klass.class_eval <<-end_eval
 5       @@capture_ids = []
 6       def self.capture_ids
 7         @@capture_ids.uniq
 8       end
 9 
10       after_create do |obj|
11         @@capture_ids << obj.id
12       end
13 
14       after_destroy do |obj|
15         @@capture_ids.remove obj.id
16       end
17     end_eval
18   end
19 
20   yield
21 
22   File.open(data_migration_file, 'w') do |f|
23     f.write args.inject({}) { |hash, name|  
24       klass = name.to_s.classify.constantize
25       hash[name] = klass.capture_ids
26       hash
27     }.to_yaml
28   end
29 end

This `capture` method adds callbacks for `after\create` and `after\destroy` to keep track of the id’s. This is written out to a file in a `data` subdirectory of the `db/migrate` directory. These files can be ignored by Subversion so each developer can run the migrations on their own databases without clashing.

Migrating down is basically a matter of removing the records that were created when migrating up.

 1 def self.release(*args)
 2   data_file = data_migration_file(:down)
 3   data = File.open(data_file, 'r+') { |f| YAML.load f }
 4   if block_given?
 5     yield *args.inject([]) { |list, name| list << data[name] 
 6     }.compact if block_given?
 7   else
 8     args.each do |name|
 9       klass = name.to_s.classify.constantize
10       klass.delete data[name] unless data[name].blank?
11     end
12   end
13   File.unlink data_file
14 end
<span class="lineno"> 1</span> <span class="k">def</span> <span class="nc">self</span><span class="o">.</span><span class="nf">release</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">)</span>
<span class="lineno"> 2</span>   <span class="n">data_file</span> <span class="o">=</span> <span class="n">data_migration_file</span><span class="p">(</span><span class="ss">:down</span><span class="p">)</span>
<span class="lineno"> 3</span>   <span class="n">data</span> <span class="o">=</span> <span class="no">File</span><span class="o">.</span><span class="n">open</span><span class="p">(</span><span class="n">data_file</span><span class="p">,</span> <span class="s1">&#39;r+&#39;</span><span class="p">)</span> <span class="p">{</span> <span class="o">|</span><span class="n">f</span><span class="o">|</span> <span class="no">YAML</span><span class="o">.</span><span class="n">load</span> <span class="n">f</span> <span class="p">}</span>
<span class="lineno"> 4</span>   <span class="k">if</span> <span class="nb">block_given?</span>
<span class="lineno"> 5</span>     <span class="k">yield</span> <span class="o">*</span><span class="n">args</span><span class="o">.</span><span class="n">inject</span><span class="p">(</span><span class="o">[]</span><span class="p">)</span> <span class="p">{</span> <span class="o">|</span><span class="n">list</span><span class="p">,</span> <span class="nb">name</span><span class="o">|</span> <span class="n">list</span> <span class="o">&lt;&lt;</span> <span class="n">data</span><span class="o">[</span><span class="nb">name</span><span class="o">]</span> 
<span class="lineno"> 6</span>     <span class="p">}</span><span class="o">.</span><span class="n">compact</span> <span class="k">if</span> <span class="nb">block_given?</span>
<span class="lineno"> 7</span>   <span class="k">else</span>
<span class="lineno"> 8</span>     <span class="n">args</span><span class="o">.</span><span class="n">each</span> <span class="k">do</span> <span class="o">|</span><span class="nb">name</span><span class="o">|</span>
<span class="lineno"> 9</span>       <span class="n">klass</span> <span class="o">=</span> <span class="nb">name</span><span class="o">.</span><span class="n">to_s</span><span class="o">.</span><span class="n">classify</span><span class="o">.</span><span class="n">constantize</span>
<span class="lineno">10</span>       <span class="n">klass</span><span class="o">.</span><span class="n">delete</span> <span class="n">data</span><span class="o">[</span><span class="nb">name</span><span class="o">]</span> <span class="k">unless</span> <span class="n">data</span><span class="o">[</span><span class="nb">name</span><span class="o">].</span><span class="n">blank?</span>
<span class="lineno">11</span>     <span class="k">end</span>
<span class="lineno">12</span>   <span class="k">end</span>
<span class="lineno">13</span>   <span class="no">File</span><span class="o">.</span><span class="n">unlink</span> <span class="n">data_file</span>
<span class="lineno">14</span> <span class="k">end</span>



If you provide a block to the `release` method, you take care of doing whatever you want to the records. If you just want to delete them, don’t pass a block and `release` will delete them. I used `delete` rather than `destroy` so that everything you specify on capture is deleted without running into issues with `:dependent => :destroy`.
This has been useful for add and removing data while trying out different table structures. Obviously, it’s a rather targeted solution. I’ll be packaging it as a plugin and posting it to [PLANET ARGON’s](http://www.planetargon.org) community site soon.

Testing the tester

2006-08-30T00:00:00-07:00

I recently wrote about the assert_cookie custom test assertion plugin I put together. When I started writing the assertion, it was just a helper method to wrap up the funky logic for testing stuff about a cookie. I wasn’t thinking about tests. And my first tries were definitely not test-driven.

When I was wrapping it up as a plugin, this started bugging me a lot. Especially because my first tries at the assertion method weren’t very good. (Ugh! svn diff -r59:72 http://svn.planetargon.org/rails/plugins/assert\_cookie/lib/assert\_cookie.rb) But the idea of testing it was confounding. How do you think about testing the tester?

The first thing I had to get straight was what I expected. It basically reduced to expecting the assertion to pass or fail. When an assertion fails, it raises the AssertionFailedError exception. Excellent, now I was making progress.

I started writing tests like:

1 def test_assertion_cookie_is_secure_should_pass_when_cookie_is_secure
2   assert_nothing_raised { assert_cookie :secret, :secure => true }
3 end
4 
5 def test_assertion_cookie_is_secure_should_fail_when_cookie_is_not_secure
6   assert_raise(AssertionFailedError) { assert_cookie :open_secret, 
7       :secure => true }
8 end

and notices a lot of repetition of assert\_raise(AssertionFailedError) and assert\_nothing\_raised. It was ugly to read. I decided to write a couple helper assertions:

 1 def assert_pass
 2   assert_block("assert_pass must be called with a block.") { block_given? }
 3   yield
 4 end
 5 
 6 def assert_fail
 7   assert_block("assert_fail must be called with a block.") { block_given? }
 8   yield
 9 rescue AssertionFailedError
10 end

The hard part again was figuring out what I was expecting. My first try was to use assert\_raise and assert\_nothing\_raised as before. But those failed miserable because they call yield. I lamented to Andrew that I wished yield would work like raise. After thinking a bit more, I realized I only cared whether an AssertionFailedError exception was raised or not. The previous tests now look like this:

1 def test_assertion_cookie_is_secure_should_pass_when_cookie_is_secure
2   assert_pass { assert_cookie :secret, :secure => true }
3 end
4 
5 def test_assertion_cookie_is_secure_should_fail_when_cookie_is_not_secure
6   assert_fail { assert_cookie :open_secret, :secure => true }
7 end

There is one last thing that bugs me. I wanted the tests to run outside of Rails but function correctly in Rails. Rails adds a method to Test::Unit::Assertions called clean\_backtrace. I couldn’t figure out how to get that required correctly, so I created a stub in test\_helper.rb. Can anyone help me fix this?

Install the plugin with script/plugin install http://svn.planetargon.org/rails/plugins/assert_cookie. Navigate to the vendor/plugins/assert_cookie directory, type rake and watch the tests run. :)

Tracer bullets are about aiming, not firing

2006-08-29T00:00:00-07:00

There’s more than one way to shoot down a moving target. Dave Thomas and Andy Hunt have used the metaphor of tracer bullets to describe an approach to software development. Martin, Badgerism, Peter, Lights Out Production, and Jeff, in no particular order¹ and among many others, have weighed in with their thoughts.

But how does firing at a moving target with tracer bullets work? It’s a gross over-simplification to say you just fire-aim-fire-aim. In fact, the key to the process is how aiming itself provides immediate feedback to refine aiming. As Dave himself says,

Basically, it all comes down to feedback. The more quickly you can get feedback, the less change you need to get back on target.

So, the process actually goes something like this:

you have already acquired a target
you have initially aimed the gun
you fire and watch where the tracers go
you re-aim based on where the target is now in relation to the tracers

That’s a single person aiming, firing, watching, aiming, … That’s a process that is happening over short instants of time. Where you aimed a fraction of a second ago is still visible in the stream of tracer bullets when you adjust your aim.

Now translate that to a team developing software. No longer is it one person. No longer is the feedback loop on the order of fractions of a second. Time stretches to weeks or months between the initial target acquisition and aiming until the feedback comes in. How does that feedback inform the original rationale for the design decisions or even for the target acquisition?

In the process we are refining at PLANET ARGON, we insist that we document the rationale for our design decisions. This includes documenting our discussions with the client and domain experts to identify the goals of the software and its users. This is a very lightweight process, usually about a small paragraph per major design decision. We typically use a wiki to record the documentation and aggregate in the same place useful references, project documentation, interviews, etc.

To some folks, documentation may be a dirty word, something to shun or dread. Not for us. Documentation is simply our way of providing a record of the decisions we make so that feedback can be evaluated in context. From my dashboard dictionary:

documentation, n.

Material that provides official information or evidence or that serves as a record

I’m at a loss to understand how else you would do it. Tracer bullets don’t help you a bit if you are just firing randomly. And I think that’s what people miss about the process. You really do aim first. And if you can’t remember why you aimed where you did, or if the person who did the aiming has left by the time feedback is coming in, that feedback may be interesting, but it certainly isn’t going to enable you to refine your decisions.

¹ Thanks to Robby for providing links to some of these articles

Agile thought 41 years ago

2006-08-29T00:00:00-07:00

Recently, Robby dropped off his copy of The New Utopians by Robert Boguslaw at my desk with a section marked: “For Whom and For What”. Here are a couple interesting quotes on page 45:

This, then, is one of the fundamental problems of application: on the one hand, the temptation is almost overpowering to establish the techniques and practices of practitioners as handbook truths available in situations of the sort we have termed established. It is the temptation to speak with the same degree of assurance under all professional circumstances as does the familiar stereotype of the TV repairman or journeyman carpenter.

On the other hand, there may be found equally strong temptation to revert to unbridled intuitionism–on the part of those who somehow sense the inadequacy of established situation science applied to emergent situation reality.

This reminded me of the recent blog exchange between Peat and Alex regarding the role of common sense and intuition in application development.

It is an attractive and beguiling illusion that attempts to persuade us to think that if we just apply enough analysis from correct principles, we’ll produce something “correct.” And we should do the analysis first. That pretty siren sings so sweetly over the shipwreck mess of many development projects. The allure is so powerful, of course, because sometimes it is correct. The problem is when to do this and at what scale.

Sometimes it is much better to analyze things first. Some people will say, “First make it correct, then optimize it.” But when speaking with Stefan Kaes at RailsConf 2006 after his talk, I thanked him for pointing out that some consideration for performance must be made up front. To paraphrase his reply, “Absolutely”. Here, the scale is large. Broad decisions are made, not detailed ones. And it is necessary to do it earlier rather than later because this broad decision will dictate the confines under which many small decisions are made that create a structure of significant weight that will resist change.

I was a bit disturbed to read this comparison of Tracer Bullet Development versus eXtreme Programming where it was asserted that TBD uses up-front design while XP does not. Obviously, using XP, you don’t just start randomly writing code. Somewhere, somehow a bunch of decisions were made that create the context in which you start writing tests. Disregarding appropriate situations at which to make preliminary decisions is as dangerous as deciding too much.

Even if we know extremely well the technical details of the small bits of test-driven code we are writing, it’s probably guaranteed that we have but a faint understanding of how those pieces will make a complete application. As Robert points out summarizing Kurt Lewin in “Field Theory and Learning”

In short, it is not … that the whole is more than the sum of its parts; it is different from the some of its parts

assert_no_cookie dammit

2006-08-28T00:00:00-07:00

I hate cookies. Or maybe I’m just feeling schizo today. Actually, I do love cookies, just not testing them with Rails.

I’ve added assert\_no\_cookie method to the assert\_cookie plugin. That’s because in your functional tests, cookies['chocolate'] may be something, nil, or an empty array depending on what you’ve just done. For example, in this code I had tried assert\_nil cookies['pass'] but that failed because cookies['pass'] was [] after calling cookies.delete :pass in the controller:

<typo:code lang="ruby">
def test_destroy_should_remove_persistent_login_cookie
  post :create, :username => 'brian.ford', :password => 'secret', 
      :remember_me => '1'
  pass = cookies['pass']
  
  @request.cookies['pass'] = cookies['pass']
  post :destroy, :_method => 'delete'
  
  assert_no_cookie :pass
  user = User.find_by_username 'brian.ford'
  assert_nil user.persistent_logins.find_by_pass(pass)
end
</typo:code>

Anyway, as before, you can get this by script/plugin http://svn.planetargon.org/rails/plugins/assert_cookie.

And thanks yet again to Pluit Solutions for helping me keep my head straight about how to work with cookies in functional tests. Next stop, a patch for Rails (would a generous soul please donate some time to this so I don’t have to).

assert_cookie for ooey gooey fun

2006-08-27T00:00:00-07:00

I love cookies. There are, of course, tons of varieties and I’m no connoisseur but I love the soft chocolate chip right out of the oven, hot and gooey. But, if you’re like me, you don’t want your Rails code to be gooey.

Last Friday, after spending a good frustrating hour trying to figure out why my tests were failing with a nil when a cookie was expected, I finally tried a google search for “assert cookies”. (A lot of more detailed searches pulled up nothing but Java stuff, a not-so-subtle reminder that Ruby is not yet the dominant language out there.) Seems that Pluit Solutions ran into the same problem. This got me thinking that a nice custom assertion might be in order. Actually, I just wrote a quick helper because that was faster than trying to figure out why Rails wasn’t performing as advertised. But it kept bugging me, so I wrapped it up as a plugin. It’s still rough and tests are coming. Perhaps I’m putting too much into a single assertion. Install it with script/plugin install http://svn.planetargon.org/rails/plugins/assert_cookie. And please send feedback.

The idea is to allow various assertions about cookies using a hash of arguments. In particular, you can do such things as:


assert_cookie :pass, 
    :value => lambda { |value| UUID.parse(value).valid? }
assert_cookie :yellow, :value => ['sunny', 'days']
assert_cookie :delight, :value => 'yum'
assert_cookie :secret, :path => lambda { |path| path =~ /secret/ }, 
    :secure => true

Patterns of dialogue

2006-08-22T00:00:00-07:00

Several weeks ago when Robby and I started discussing how to better interact with clients, one of our goals was a simpler, more effective process. From my experiences as a math student, I knew that a few carefully chosen symbols along with adherence to a fairly simple structure enabled folks from the world over, with different native languages and different cultures, to discuss mathematics, learn from each other, and understand the topics.

The news press is another profession that makes excellent, effective use of a simple structure to interview people and elicit facts to reconstruct an event at which the reporter was not present. We have a rather high degree of confidence (depending on the media outlet) in the accuracy of the method.

These, among other things, encouraged us to focus on the dialogue we have with our clients. We want to examine the dialogue without any special props or artifacts from a particular methodology. What can we accomplish as knowledgeable people, potentially having different vocabularies and areas of expertise, sitting around a table perhaps with paper and pencil? So far, we’ve mostly posted about these ideas. But our aim is to extend these ideas into identifying patterns of dialogue.

Here’s one: You probably noticed that my posts tend to be pretty long. That’s because, just like the three paragraphs above, I usually try to take time to lay some groundwork, get on the same page. And this is often what I do when talking with someone as well. Perhaps a good name would be the Misunderstanding Prevention Focused Long Intro dialogue pattern. The problem is, unless you’re pretty dedicated about paying attention, I might lose you before I get done painting the whole picture. A different pattern could focus on one small bit of the relevant topic, frequently check understanding, and build out the background to enhance understanding when necessary. To me, that pattern is much more like BDD and follows the principle of not solving a problem you haven’t yet encountered.

Another thing we frequently observe is that any little feature of the software we’re building acts just like a spark in tinder, setting a client off on a runaway course of, “Wouldn’t it be cool if…”. Attempting to blow out that flame before it derails the discussion, we often use totally irrelevant data that has less provocative affect. For example, in a recent prototype showing tagging features, we used names of food as tags, which have absolutely nothing to do with the application. The intent was to prevent the client from starting to think about the “right tags” and focus instead on the features of the tagging process.

We are attempting that sort of process on a bigger scale with clients who seem naturally to obsess on features. We try to structure the dialogue about the user’s goals without referring to the software at all. At first glance, that may seem to increase the likelihood that we will misunderstand what the client wants. But, we find the reverse is true. Feature focus is more likely to contribute to misunderstanding what the client needs. And it prevents the client from learning as well.

What patterns do you observe in your dialogue? When are they appropriate? What are you trying to change about your client interaction?

Ethical Software Needs Dialogue

2006-08-16T00:00:00-07:00

Growing up in the United States in the late 70’s, early 80’s, futbol was something I knew very little about. Let’s see, we played once in a great while in gym class, my most memorable experience there was taking a hard-kicked ball right in the face. Later in high school, the soccer team was a mere blip on the screen in the shadow of football and baseball. Much later I started playing soccer and, having so much fun, lament that I didn’t play as a kid. The interesting thing to me about soccer (and many games, but you’ll especially appreciate this if you listen to a soccer game on a hispanic radio station) is that the myriad plays and activities that make up the game are all directed toward scoring a goal (and preventing the opposing team from doing so). In other words, the goal of the game is simple and clear, though not easily achieved.

Alex Bunardzic is a champion of what he terms “ethical software”. The idea goes by many names. My favorite is software that doesn’t suck. I highly value Alex’s ideas, so when I read his recent post, Should-Driven Development, I couldn’t easily dismiss his criticisms of Behavior Driven Development. At the same time, I don’t agree that BDD itself is flawed as an approach to creating better software. BDD is a developer’s tool. It’s about coding. Of course, BDD cannot specify the goals of the software. So, if it is expected to, it will fail to produce software that doesn’t suck.

One of the things about Scrum that I think is tremendously valuable, and yet one of those things that makes you say, “Duh!”, is the emphasis on empirical process control. There are three components to that: 1) observability, 2) inspection, and 3) adaptation.

What both Scrum and BDD need to be effective is a clear and accurate statement of the goals. With a goal-directed process you have these components:

somewhere you are trying to get to (using a geographic metaphor)
some process that accurately compares where you are at with where you want to go
a plan to move you incrementally closer to where you want to go
a process to evaluate whether your plan got you any closer

Or more briefly: goal, assessment, plan, implementation, feedback and around again and again.

The practice of BDD captures this well. Write the test to capture the expected behavior of the system relative to the goal. The test fails. Write the minimum code necessary to make the test pass. Refactor. And around. The process alternates between writing new tests to describe expected behavior and the code to make the test pass and then refactoring to make the code better without changing its behavior. The process is powerful.

But as I said earlier, it is a developer’s tool. Developers should never be specifying the goals for the software, just as the client and user should never be specifying the implementation. I’m rarely dogmatic, but about that I will be.

The problem that we face is how we can more accurately capture the goals for the software in our dialogue with the clients and users. Our exploration of that is what we are calling d3. One significant part of that is education. Our experience is that everyone from clients to developers begins thinking and talking in terms of the software implementation way, way too early. It’s like talking to an architect about what finishes to use and what the windows look like long before you have established a vague idea of where the walls will be or if there will even be walls. The process is too often backwards. Talking about the minutiea of text field this and that before we have even established why in the world the user would be doing that in the first place. One approach that we are trying is dialoguing with the client about goals without talking at all about the web site. In other words, for that exploration, the web site doesn’t even exist. Talk about thinking outside the tubes.

Another part of this is fostering a general respect for that fact that software that doesn’t suck is constructed according to certain principles. Just because anyone can create an HTML form doesn’t mean they should. As long as we collectively persist in thinking that anything’s possible and therefore it all boils down to a matter of opinion about software design, we will continue to create software that sucks.

So rather than putting down BDD for failing, we need to recognize that right now in the state of our craft there is a huge chasm between client interaction to establish goals for the software and the much lower level development that can excel through application of BDD.

What we need to develop is how to dialogue to create common understanding. What I am noticing as I try hard to be more aware of how we are communicating with clients is that “process” and “process artifacts” can often be counter-productive and obscure issues. If we are just trying to “fill in” the boxes in our process, are we stopping to really listen? It’s like assertiveness training where you are taught to reframe your expressions using “I” statements. Rather than, “you are really pissing me off” someone might as an initial stab try “I feel like you are really pissing me off”. Yeah, not very effective. Try that with your girlfriend if you don’t believe me. More effective is, “I feel hurt when you say you don’t want to play with my new Mephisto blog.” One approach attempts to conform to the structure of assertiveness, one effectively conforms to the principles. What are the principles of effective dialogue?

not Scrum.equal :product_backlog

2006-08-10T00:00:00-07:00

I think it’s obvious to everyone that Scrum and the product backlog are not the same thing. Sometimes stating the obvious is useful. Scrum is a structured process of which the product backlog is a part. Mechanism Alley has posted an excellent, thoughtful article about Scrum with some references to some of what we’ve been saying about d3. It’s worth a careful read. In fact, I want to personally thank MSM for not perceiving our comments as an invitation to start flaming about this or that pet methodology.

I certainly don’t want our conversations about Scrum to be alienating to any of the excellent folks working hard under the agile umbrella. Further, I don’t want Scrum only to be the focus of these conversations. Finally, I’m not particularly interested in pumping another meme through the well-oiled Rails machine. Are we restating things said by others? Hell yes! We are standing on the shoulders of giants.

What I find a bit disturbing is that while I said a number of things about Scrum, only what Robby and I have said about the product backlog is getting attention. Further, the responses seem to fall generally into three categories:

the literature we read misrepresented the Scrum process
we have misunderstood the literature
the way the commenter uses the product backlog involves a lot of conversation, not a lot of dictation

Firstly, the book I referred to, Agile Project Management with Scrum by Ken Schwaber seems to be a good source. It seems consistent with everything knowledgeable people have posted about Scrum. And it clearly advocates agile methods. Consider this quote:

You can get through almost anything if you don’t try to impose rigid solutions before problems even arise, instead devising solutions as necessary when problems crop up. (pg 131)

That’s from chapter 9, Scaling Projects Using Scrum, which described application of Scrum to a difficult project and discussed solutions implemented in the spirit of Scrum but rather far outside of a “typical” Scrum process (if such could be argued to exist).

Secondly, we are not claiming to be Scrum experts, but that said, I don’t think I’m misunderstanding the literature. What we are aiming at are things that we think could be better understood and articulated. Just like the agile manifesto doesn’t prescribe a “right” way, it’s clear there are many implementations of agile processes. At the same time, the agile manifesto is certainly opinionated about certain processes being preferred for (and I would go as far as saying essential to) effectively creating software.

Thirdly, commenters that relate, in their experience, how much conversation is required around and facilitated by the product backlog are contributing to our confidence that there is a lot to say and understand about the dialogues we are having. Further, we are really interested in dialogues we are having with clients. We are finding this to be very challenging at best.

As Robby recently pointed out in a conversation with me, development processes often seem geared toward making the developers look good based on all the items they’ve checked off the list of the client’s wants. “See, we’ve delivered everything you’ve asked for, and on time even!” Yet through these processes, we (collectively) continue to successfully create software that sucks! We think that’s because it’s extraordinarily challenging to get processes to be goal directed based on requirements versus solutions. The way we get to this is through a lot of dialogue. We want to better understand the process and patterns of this dialogue.

And, ultimately, whether you are using Scrum or some other process under the agile umbrella, knowledge of how to conduct and facilitate this dialogue will be extremely valuable to you. So, please help us understand how you dialogue. If the product backlog is an artifact in this, please help us understand how?

Rails' future is bright indeed, bright like the sun

2006-08-07T00:00:00-07:00

If you don’t get a chance to hang out on our irc channel much (#planetargon on irc.freenode.net), you may have missed some of the innovations on the horizon for Rails: We’ll let you know when the nuclear-powered Rails beta is up for testing.

Anyway, Happy Birthday Rails!

12:24 tacodog hrm. there are 3ppb of uranium in seawater.
12:24 tacodog 4.5b estimated tons of uranium in the ocean
12:24 tacodog even more thorium.
12:25 tacodog the uranium alone could power the world for the next 6500 years.
12:26 * harrisj hatches a world-domination scheme
12:41 tacodog you know how i came across all this? one day i was eating crab with the wife
12:42 tacodog i looked over and saw a pile of crabshells
12:42 tacodog wondered what it could be used for
12:42 tacodog chitin is the second most common organic substance in the world. second only to cellulose
12:42 tacodog and one of the things you can do with chitin is to clean water.
12:42 tacodog sewer water
12:42 harrisj what about keratin?
12:42 tacodog it absorbs metals
12:43 tacodog dunno.
12:43 tacodog but chitin is crabs, insects, etc.
12:43 tacodog so, it absorbs metals in water
12:43 tacodog uranium, titanium adn thorium are dissolved in seawater
12:43 tacodog so… i started doing some research
12:44 tacodog a japanese scientist ‘mined’ yellow cake from the ocean
12:44 tacodog amazing
12:44 harrisj tacodog: wow
12:45 tacodog more thorium than uranium in seawater
12:45 tacodog more expensive to use thorium in a reactor, but it’s possible
12:45 tacodog and… pair this with pebble bed reactors…
12:45 tacodog and you got yourself a bidness
12:46 tacodog sorry, i’m a nerd. i admit it. this fascinates me.
12:47 lanaer bah, no apologies.
12:47 lanaer just keep adding to the interesting tidbits of info I come across in my day :)
12:47 dgibbons yeah this is much better then the talk in my other channels
12:48 lanaer heh
13:49 brixen tacodog: thanks, now #planetargon just bubbled to the top of the list at NSA
13:49 brixen they’re busily researching everybody
13:51 brixen but I like the idea: get your own nuclear powered rails app :P
13:52 brixen rails on uranium, or thorium
13:52 tshine nuclear powered rails aparams
13:52 tshine opps
13:52 tacodog hahaha
13:52 tshine nuclear powered rails on a 9-volt budget
13:52 * tacodog is back from lunch
13:53 tshine my hotkey proggy inserted that little tidbit :(
13:53 dgibbons your rails app generates enough power you can sell it back to the power company

One cuckoo flew over the nest

2006-08-06T00:00:00-07:00

Firstly, a disclaimer in the words of the inimitable Ken Kesey: >Take what you can use and let the rest go by.

Interestingly, when I googled this quote to ensure I was recalling it correctly, I found some other excellent quotes. Another that seems apropos: >You don’t plow under the corn because the seed was planted with a neighbor’s shovel.

If Dialog Driven Development is merely rounded corners to you, please continue to enjoy your square ones. This exploration is not aimed at gathering the true believers. I’m not the evangelist type. So, if you’re worried I’ll show up on your doorstep thumping my bible, rest assured I won’t. I’m an atheist.

That said, I think there is value in creating distinctions to better understand the terrain. Agile is a big umbrella. If the processes were formulaic, all we would need to worry about is a good algorithm.

Recently on a flight to DC to meet with a new client, I delved into a book Robby brought along: Agile Project Management with Scrum by Ken Schwaber, one of the co-founders of the Scrum method. Below are a few notes I jotted down on my index cards while reading the first 4 chapters and browsing the appendices. Since then I’ve browsed some of the other chapters but I haven’t finished the book.

advantages of Scrum

emphasis on empirical process control
iterations
daily team meetings where each member presents what she did in the past 24 hours, what she’ll do in the next 24 hours, and any blockers
frequent feedback loop
clear separation of responsibility using the pigs and chickens metaphor to facilitate more effective decision making and reduce interference by chickens, which can bog down the process
The three legs that hold up the implementation of empirical process control: 1) visibility - aspects that affect the outcome must be visible to those controlling the process; 2) inspection - these aspects must be inspected frequently enough to change course if necessary before too much has elapsed; and 3) adaptation - changing course when new information suggests it is necessary
emphasis on having a vision, it must be a vision that is fairly resistant to change in order to guide the project, but able to change when required

”You can get through almost anything if you don’t try to impose rigid solutions before problems even arise, instead devising solutions as necessary when problems crop up.” (page 131)

drawbacks to Scrum

specifying the product backlog - this suggests too much dictation and not enough collaboration. According to the book, the product owner specifies the backlog and prioritizes the items in it. The team meets to decide which items it can deliver in the next sprint. That process is two monologues not a dialogue.
30-day sprint is too long. I don’t think the feedback cycle is nearly as effective at this scale because we need feedback long before functionality is “deliverable”. More frequent testing is needed to refine features. Imagine getting a haircut where you tell the stylist what you want. She then takes you to a room with no mirrors and starts chopping away. When she’s done she takes you to another room to demo your new haircut. Fortunately, the difference between a bad haircut and a good one is about 2 weeks. The same is hardly true of software.

The process of software development requires a lot of educating of the people involved. The product backlog does not encourage thinking in this way. I dislike the sequential list where the only explicit organizing idea is priority. Where is the context and relation between what is being specified? And how does the process ensure that the product owner doesn’t fall back to specifying things like, “Dialog X needs a dropdown field to display Y”, or “Build a screen to display user stats”? If you haven’t had to deal with this, then I desperately want to come visit your universe. If you don’t know why this is the biggest ingredient in a recipe for disaster, just continue building software this way.

The way to challenge this is with true, effective dialogue. What that looks like varies a lot. But it’s not about particular artifacts. If you read Manley’s criticism of our rounded corners, he relates the necessity of educating product owners. If dialogue is at the center of your process, the product backlog could function as a passable way to document the evolving mutual understanding.

My favorite idiocy in software development is the crystal ball method. Everyone has a crystal ball on their desks from which they can conjure a precise, detailed, perfectly focused picture of any event at any point in the future. Of course, this is an absolute absurdity. But that doesn’t stop people from believing such feats are possible. The product backlog easily falls prey to this delusion.

A much more useful metaphor is based on the simple fact of vision that any sighted person can appreciate. From where I stand, there are a number of things in crisp focus. Beyond that, things are mostly distinguishable by shape and relationship. I’m seeing the forest. Beyond that, only the vaguest, broadest features are distinguishable. I see that mountain, but the dark areas juxtaposing the snow may be forest or exposed rock. I can’t tell.

Attempting to draw on this fact of vision to inform and structure dialogue about a recent project, we set up a wiki with a page to capture descriptions of the software’s behavior in terms of goals. We have three sections: 1) near term goals, 2) medium range goals, and 3) long range goals. These follow the rough divisions of visual clarity described above.

The process of transforming those goals into software features and determining their priority based on architectural constraints is not something a client can dictate. You don’t tell your architect how to construct your house, you describe what you want in it. The reason for that is simple: physics dictates what is a viable structure. In software, we have no appreciation for physics. Clients believe they can do anything. All the rusting hulks of failed software littering the desert notwithstanding.

The essential way to bring the analogue of physics to software development is mutual education mediated by effective dialogue. What that actually looks like needs a great deal of definition. If this is just rounded corners on existing work, please point us in the right direction.

It's all about the dialogue

2006-08-04T00:00:00-07:00

Martin Fowler is a paradox. That is, if you accept one of the common connotations of “enterprise software”: evil. “I concentrate on designing enterprise software – looking at what makes a good design and what practices are needed to come up with good design.”, says Martin. Yet, he is one of the giants of agile software development. Recently, at the second annual foscon, I won a raffle for a book. The one I selected from those available was Martin’s Patterns of Enterprise Application Architecture. If you have not yet listened to his RailsConf 2006 Keynote, I strongly urge you to do so.

A couple days ago, Robby posted about Dialogue Driven Development. We’ll leave it to the art historians to decide if the following artifact proves that while sitting together at Martin’s talk during RailsConf 2006, I first wrote the words, or jotted them down after Robby summarized one of Martin’s points. Robby often encourages “blog early, blog often”. So I can hardly complain if he pre-empted me on publishing DDD since RailsConf was over six weeks ago.

The important thing is that DDD is not just another marketing term. Although, it could be useful as one, so we want to distinguish between dialog, those artifacts from desktop applications that we on the web infrequently encounter, and dialogue, n. a discussion between two or more people or groups, esp. one directed toward exploration of a particular subject or resolution of a problem (from my dashboard dictionary).

For me, the -ue is all about user experience. And that is the meat and potatoes of application development that we should be targeting. Don’t believe me? Think it’s more about business logic, fancy technology, web 2.0, <insert favorite CS concept here>? I don’t think so. But, don’t take my word for it. You too can pre-empt me on a coming post by heading over and reading, if you have not, Alan Cooper, About Face: The Essentials of User Interface Design.

So, DDD. What we’re trying to do is label a concept that is sorely in need of development. To flesh it out a bit from my perspective, here are a few points from Martin’s keynote that I found particularly interesting.

martin_keynote.points.each do |point|

The interesting thing about Rails is that it tries to do things differently than other software over the last few years. It doesn't try to be the one-size-fits-all solution.

What we are seeing is a drive toward simplicity. Conventional wisdom once was "quick necessarily means dirty". Ruby challenges that. As did Smalltalk, which showed that you could be quick and clean.

What's important about technology is the way it affects the whole conversation between customer and developers. It affects how we go about building software. The essence of XP from Kent Beck is this change in relationship so it is not a one way stream of "this is what I want": want -> tell -> build -> all is well. People may know what they want, but not what they need. We can build exactly what they want and still fail because it is not what they need.
end
It was after jotting down that last note that I wrote "dialog-based develepment" and then "dialog driven development" in my notes (I know, I misspelled it, too). The DDD term was prompted by the recent conversations [Jeremy](http://www.jvoorhis.com) and I had with [Steven Baker](http://blog.lavalamp.ca/) and [David Goodlad](http://david.goodlad.ca/) on the train to Chicago about Behavior Driven Development as they worked a bunch on a next-generation RSpec. My pie in the sky is a Ruby DSL something like RSpec that could be used with clients to talk about the software. It would allow a spectrum from less specific-more tolerant to more-specific-less tolerant. At the left end is client dialogue about behavior of the software. Something like, "app should require login". A "failure" at this high level could be something like, _not implemented_. The spectrum continues down to the level of BDD with something like RSpec, where the focus is on behavior of the code. Also on the train, [Josh Knowles](http://joshknowles.com/) was strongly advocating for much better acceptance testing. Of course, using iterative development, acceptance testing is not something done at the end, months or years into the project, but should be done frequently. The left end of the spectrum could support this _expectation based software behavior dialogue_. Of course, these are very rough ideas. What I want to do is build out the understanding behind the word "collaboration" in the [Agile Manifesto](http://www.agilemanifesto.org) and identify how "individuals and interactions" work to create excellent software. Join the fun!

Is there room for a good Samaritan in the face of the spam deluge

2006-08-03T00:00:00-07:00

We all love the spluge (spam deluge), don’t we? Usually spam is so abnoxiously and obviously stupid it is easy to ignore. And, gmail does a wonderful job of ensuring I deal with very little of it. However, once in a while something comes through. A trend I’ve noticed recently are emails that begin with “This is an advertisement…” I’ve never read one careful but it’s possible, but unlikely, that I somehow asked to receive it.

Today, I got an odd email. I first thought to dismiss it, but upon reading it I decided it was either the best crafted bit of spam, or an honest mistake. Then I was faced with a decision: round-file it, or be a good Samaritan and let the guy know. Here’s the email:

Subject: travel to Missoula

Thanks for getting the reservation at Downtown Motel. The fellow I talked to said “no available rooms the entire weekend”. I’ll put >it on my credit card when I get there.

I’ve encountered another problem with scheduling. I ordered my airline tickets through Priceline.Com and now they tell me there is no record of my order. The airlines now want about $400 more because of the short notice. Consequently, I’m going to drive and will pick up Quincy and your mother. We will arrive sometime late Friday and stay through Sunday. See you then.

                                    Love,  Uncle John

Ultimately, I decided to reply:

Hi John,

I’m sorry, but you’ve reached the wrong email box.

Forgive my suspicion, but if you do happen to be a spammer and are using this as a ploy to get a response from this email address, rest assured a real person lives here and I’ll haunt your sleep forever if you continue to spam me. However, also note that, if you are a spammer, all your future spam will flutter totally un-noticed by me into my big circular bit bucket to live out eternity in utter unacknowledgement.

Hope the trip goes well.

Cheers.

What do you think–was it spam? Would you have replied? The analogy is walking down the city street, only talking to our friends, eyeing with suspicion and from a distance any stranger we happen across. Do we need to recreate this dynamic on the tubes (pipes? I forget) as well, or can we allow for greater friendliness?

Good user experience design is like chicken-sexing

2006-07-23T00:00:00-07:00

To paraphrase Jared Spool, who gave the closing keynote at webvisions (2006), designing compelling user experience is something that can be learned, but there’s no good understanding of how it works. To illustrate this phenomenon, Jared refers to chicken-sexing. You can read up on it, but the basic idea is that to run a cost-effective chicken farm, you need to know the gender of the chicks about 2-3 weeks before any known mechanized process can successfully do so. The solution is to rely on professionals that watch the chicks go by and identify them as male or female.

Consider the ipod and netflix as two examples of compelling user experience despite, according to Jared, the former having inferior technology compared to competitor offerings and the latter not having any brick-and-mortar storefronts or any customer service other than through their website. Even so, the ipod is wildly more popular than any other portable music player, and Netflix membership dwarfs that of Blockbusters recent through-the-mail dvd rental service.

Obviously, there is great advantage to being able to learn good user experience design. And that was one of Jared’s main points. It can be learned. This is supported by current research. Consider an article in the August 2006 issue of Scientific American, “The Expert Mind”. Research into chess masters indicates that there is little actual contribution by “talent”. Rather these folks’ mastery consists primarily of categorizing a huge amount of knowledge of fruitful moves given the configuration of piece on the board.

Effortful study is the key to achieving success in chess, classical music, soccer and many other fields. New research has indicated that motivation is a more important factor than innate ability.

One key to effortful study is constantly pushing yourself to reach just beyond your current abilities. This is good news for those of us who have never felt we are natural born chicken-sexers.

Having a blast at webvisions 2006

2006-07-21T00:00:00-07:00

I have had the good fortune of being sent to webvisions (2006) by PLANET ARGON. I have it on good authority that this year it is much larger than last year. It’s two days long and chock-full of great talks.

So far, I have attended:

The “People and Business” panel was a very interesting, if Portland-specific, look at government and industry participating to create and promote business. This was a bit unusual, I thought, since folks have come from all over to attend the conference. At the same time, it was intriguing to hear, especially from the Portland Development Commission, some things that may distinguish Portland as a hospitable environment for technology innovation. By coincidence, I think, the Portland City Council had voted just the night before to move forward with plans to build a free, public Wi-Fi blanket in Portland.

My favorite talk by far was “Designing for Social Sharing”. I recommend checking out Rashmi’s site. She was a vibrant, engaging, knowledgeable speaker. One of the most interesting points is that to create a successful site for social sharing, we need an object around which people can share. Now that we have sites like Flickr, this seems like a no-brainer. But when we view how “social networking” sites have progressed from 1st generation, where the emphasis was only on the social network, we see how important this idea is. To paraphrase her, people don’t just connect for the sake of connecting, they connect over something.

A big component of a successful site that encourages people to share is providing a way for people to contribute their own creativity or personality. In effect, the design of these sites must facilitate the user’s ability to influence the design. Interestingly, I have recently been reading, A Pattern Approach to Interaction Design, by Jan Borchers. The author goes to some length to present a more full exposition of Alexander’s ideas on patterns. There is a lot of use of “patterns” this-or-that in various domains. To some extent, the word is often just stretched to cover a favorite concept. But the following quote seems particularly relevant in the context of a social sharing site:

Every place is given its character by certain patterns of events that keep on happening there. These patterns of events are always interlocked with certain geometric patterns in the space.

Christopher Alexander, The Timeless Way of Building

And this quote is nicely relevant to designers of such sites:

Interestingly, Alexander’s ideas were not always received well by his colleagues. One reason for this is that Alexander’s concepts empowered the inhabitants, supplying them with more ways to influence the building process, and taking much of that power out of the hands of the professional. Obviously, this idea was not very popular among architects.

Jan Borchers, A Pattern Approach to Interaction Design

Another excellent talk was “Design Patterns for the Web” by a guy from eBay and another from Yahoo!. A nice serendipity: I recently have been re-reading, About Face: the essentials of user interface design, by Alan Cooper. This book was awesome when I first read it years ago, and even today I’m amazed it is copyright 1995. Unfortunately, the content seems to be little-known in the web design world. Not at Yahoo! It appears to be very much an inspiration, as I will explain a bit more in a later post.

Okay, that’s it for now. I’ll mine my notes for juicy tidbits and snippet to share with you. Admittedly, I’m not a good liveblogger. :-/

The "real world" can eat my shorts

2006-07-20T00:00:00-07:00

Yesterday, Robby messaged me that he was invited to be a speaker at Sys-Con Rails Conference. (Sorry, I’d have a link for you, but I’m getting a 500 page right now.) Right in the title, it mentions Ruby on Rails in the “real world”. I am unable to suppress a groan every time I hear that quoted phrase “real world”.

The “real world” label is used to legitimize. If something works in the “real world”, that basically says it’s not a toy, it’s for real. But if you look around at what works in the real world, you might ask yourself, “Why on earth would I want to imitate that?” I certainly do. The “real world” is filled with lame, difficult, buggy, costly software.

This relates to a point from DHH’s keynote at RailsConf that I’ve been meaning to reiterate. DHH essentially said that he lives in a fantasy world. I don’t think most of us realize how lucky we are that he does.

Imagine if he didn’t. Imagine DHH working on a nascent ActiveRecord and being closely in touch with the “real world”. Certainly, he would want it to be legitimate, not merely a toy. To be legit, it should solve “real world” problems. The “real world” is full of complex databases. It better support any imaginable scheme for primary keys, even composite ones. Anything less would… hmm, not work well in the “real world”.

The point is, we should be eager to say no to features. But in the “real world”, it is rare to find someone eager to do so. Sure, I can imagine my super-gizmo-mondo-cool coffee cup that is also a mobile phone, razor, electric toothbrush, talking tire gauge, and bottle opener. But why?! Just because it’s possible, does not mean that it’s desirable.

Really, our ability to live in a fantasy world, even briefly, is a terribly two-edged sword. It gives us the ability to imagine a world that does not exist, and execute the steps necessary to manifest it. For example, Rails. At the same time, it gives us the ability to imagine a world that does not exist, and execute the steps necessary to manifest it. For example, J2EE.

One of those two is thriving in the “real world”, according to the connotations normally associated with that phrase. I think we’re trying too hard in the wrong direction if we are chasing the “real world”. Rails is a disruptive technology; and that is something to be celebrated.

Rails does not need to be legitimized.

That’s my opinion. In fact, I think we do a disservice to the disruptive potential of Rails by trying to legitimize it with the “real world” label. I may be reading too much into the “real world” phrase in the Sys-Con event title, but since it is billed as being for CEO’s, CTO’s, etc., I don’t think so.

I think Rails is highly relevant to creating real world applications, meaning those that assist people to achieve their goals by doing much more with far, far less complexity. That requires us to have a vivid imagination and vibrant fantasy life. In other words, a good, healthy distance from the “real world”.

There's a short in my NetNewsWire

2006-07-10T00:00:00-07:00

When I started at PLANET ARGON, I was given a bright new matte black MacBook to use. It’s a beauty. This is the first time I’ve used a Mac extensively since around ‘89 and I love it. Well, mostly. For the last decade or so I’ve used Linux 90% of the time. First RedHat, then Fedora, and most recently Ubuntu, which in my experience is simply excellent.

What I miss most from Linux is the super responsive Gnome desktop, the wonderful Gnome Terminal, and the oodles of very high quality free software. With the Mac, I am quickly annoyed by the frequent appearance of that damned spinning color wheel, Mac-speak for, “uh, hold on, I’m having a moment”; the typing through molasses feel of iTerm; and paying for so many little pieces of software. Mind you, I’m fine trading my money for something of value, but if you are going to force the issue with me, you should probably ensure your process to lighten my wallet doesn’t piss me off.

Enter NetNewsWire. I’ve used it through it’s evaluation period and it’s been passable, nothing spectacular. But today, it wants money from me. So it puts up a friendly dialog, asking nicely. A bit miffed, I go ahead and click the “Purchase Software” radio button. Hmm, nothing happens. Seems that option is disabled. Oh well, sorry NetNewsWire, now I’m evaluating NewsFire

Dilbert wisdom: CRUD is for idiots

2006-07-05T00:00:00-07:00

Last week, my esteemed colleague Jeremy, at his characteristic morning unveiling^[1], brought in a couple books on “Death March” projects. While he tackled the other, he lent me Death March: second edition, by Edward Yourdon. Edward defines a death march project as “one whose ‘project parameters’ exceed the norm by at least 50 percent”. That is, one or more of the following is true: the project has less than half the budget, the staff, the time, or more than double the functionality, features, performance, or technical requirements. He discusses why death march projects are created, why people participate in them, and some things one might do to try to handle such a project (although he’s pretty pessimistic about successfully changing a project of this type).

Under the heading, Why Do Death March Projects Happen, Edward starts off with a great quote by the author of the Dilbert cartoons, Scott Adams:

When I first started hearing these stories [about irrational corporate behavior] I was puzzled, but after careful analysis I have developed a sophisticated theory to explain the existence of this bizarre workplace behavior. People are idiots.

Including me. Everyone is an idiot, not just the people with low SAT scores. The only difference among us is that we’re idiots about different things at different times. No matter how smart you are, you spend much of your day being an idiot.

Hopefully everyone is not totally insulted at this point. As in Dilbert, this idea can be a great source of inspiration. It’s really not as derogatory or as hopeless as it sounds. And for me, it puts some recent events in perspective.

Since DHH’s keynote at RailsConf, there has been noticeable buzz about CRUD. Everything from new acronyms to debates about nouns versus verbs. All good stuff, interesting, thought provoking. But for me, probably the greatest benefit of more CRUDful modeling is better managing of complexity. I think this is the raison d’être of CRUD.

Similar to when OOP was capturing hearts and minds, many people waxed poetic about reuse, more faithful domain modeling, etc. But the golden egg of OOP is encapsulation, and encapsulation is another powerful abstraction that helps manage complexity. If there were one thing I would lobby to be added to the agile manifesto, it would be “Simplicity over big, amazing, mind-boggling complexity”. Certainly, the “less software” mantra is related to this, but even a tiny piece of software does not necessarily mean simple, not complex.

So, if you notice your controller method count is ballooning, or your controllers are more cross-wired than your plate of spaghetti, you might see an opportunity for using CRUD to restore sanity. But you can also ruthlessly adopt Einstein’s wisdom at every stage of development and use CRUD to make your app as simple as possible, but not simpler.

^[1] Jeremy often comes to work with an announce if you will. For instance, one day we were discussing something about languages; I mentioned Erlang. The next day Jeremy announces when he arrived, “I learned Erlang last night.” Sweet!

I be broke

2006-06-19T00:00:00-07:00

So there I am, peacefully logged into my US Bank online account. Everything is going well. We’re having a nice dialog. I click, it responds. I click… now I’m looking at a plain, white page with the following scrawled across the top in courier font:

an exception occurred: Unable to complete your request at this time, ReferenceID = "wl26*2006-71-59-03:21:11:742*XLP3-594861"

Imagine what the scene might be if I were speaking to a human teller and suddenly in our interaction I received a response like this. I imagine the likes of a scene from the Exorcist. The breakdown in the expected flow is extreme.

On the one hand, US Bank is not a little entity. They have plenty of money for technology solutions. On the other, I understand that exceptions occur. Here’s a gentle reminder to degrade gracefully and if you can write something to the screen, make it useful, somewhat reassuring, and perhaps apologetic. There’s no good excuse for barfing your undigested exception in my face.

What I want to be when I grow up

2006-06-16T00:00:00-07:00

I have been rescued from mundane obscurity. Yesterday I met with Robby and Allison and we settled on Interaction Architect as my title. Those are certainly big shoes to fill, boats really, but I’m looking forward to it. The title is broad enough to encompass interaction as it applies to humans-computers, humans-humans, and computers-computers. It’s both justification and impetus to push my knowledge and understanding. For example, with Project Borat, we’re focused on how we can facilitate client interaction throughout the project as well as pushing the envelope with features to support user interaction.

My interest in languages dates at least as far back as about 6 years old when I discovered Chinese writing. I was fascinated to the point of drawing nonsense characters for kids at school, telling them I could write Chinese. This interest extends to programming languages and mathematics. And since first working with one of the early Macintosh computers almost 20 years ago, human-computer interaction has been a persistent curiosity, if somewhat sidelined. In fact, design interests me more than coding, but unfortunately, I don’t possess abundant artistic talent. For me, interaction design allows me to straddle that space between visual design and implementation and I’m excited to further develop these skills along side the other highly talented folks at PLANET ARGON.

G'day Project Borat

2006-06-09T00:00:00-07:00

Yesterday, Project Borat and I were formally introduced. I’ll be the lead developer, picking up the thread and, along with Robby, I’ll continue to entertain you with insights we gain as we delve deeply into the process, knocking some moss off a few rocks and uncovering ideas that will work well for us and our clients.

As this project is just getting under way, one of the first things we’re facing is… hmm, I hate to label it because along with that label comes all sorts of assumptions. Sure, we could call it requirements gathering, but that doesn’t tell the whole story. Then there are stories, use cases, method XYZ, and better mention PQR for completeness. Before you run off googling those last two, I made them up. Point is, even though there is a lot of material on processes, and I’m learning more every day, it’s just not that simple.

So, today Robby and I took a quick jaunt over to one of my favorite places on the planet. How cool to work this close to Powells Technical Books. While he’s deep into today’s acquisition, Mastering the Requirements Process, 2nd Ed, I had my own surprise waiting on my doorstep this evening.

That brings me to one point about gathering requirements: the first value listed on the agile manifesto–“Individuals and interactions over processes and tools”. I’m guessing there’s more than a few reasons why that one is listed first.

Clients are people with varied backgrounds. If we are slaves to a particular methodology, we either 1) talk over the client’s head with unintelligible jargon, 2) try to coerce the client into “learning” our methodology so we can “be on the same page”, or 3) continually translate, in the back of our heads or on our notes, what the client is saying into our method’s jargon. The first two are disrespectful, the last one error prone. (Where’s the feedback that must occur for checking understanding?)

When we, as developers, began focusing on user-centric design and usability, it was a recognition that users deserved respect and need not be rescued, if you will, by our bright ideas. In a big way, we needed to keep our bright ideas out of the user’s way. In the same way, I’m not convinced I should expect a client to care about stories, use cases, or XYZ artifact. I do believe my role is to facilitate a client’s understanding of what they want and capture that as faithfully as possible.

I think plain ol’ natural language is probably the most valuable tool for that. “Ee gads, but natural language is fraught with ambiguities, imprecision, double meanings, and all things evil,” you say. Well, perhaps. But consider this: mathematics is probably the most precise form of “language” humans have yet devised, and plain ol’ natural language suffices to convey mathematics given a couple rules of usage.

First, we do try to eliminate ambiguity and imprecision with definitions. If I’m talking about a metric space, I begin with, “given a set X and a metric d, (X,d)”. There, no more ambiguity. And second, economy; introduce a symbol or name to mean a particularly well-defined concept. Add a dose of discipline, be conscientious about usage, and you can get very far using normal language to discuss very difficult concepts. Trust me, or visit a math class and watch people learning math.

So, that’s the first of my explorations. How far can natural language take me before I start reaching for this or that “representation”? I’ll aim to keep these posts short and pithy in the future, but thanks for ambling along with me on this one.

Where for art thou lighttpd

2006-05-24T00:00:00-07:00

For the second time in a few days, lighttpd went awol on me. All that shows in the logs is ‘graceful shutdown started’, then ‘server stopped’. The VDS is fine and dandy, showing up-time of 26+ days. And the rails fcgi processes are all fine as well.

Perhaps this is a good time to give Mongrel a whirl. There’s been interest expressed around the office, especially after Redhanded’s quote from the interview with Zed Shaw about Mongrel using a parser based on a real grammar and parser generator, Ragel.

Alas, this bit of sysadmin has eaten the few minutes for my intended post, so I’ll leave you with just the teaser of a title for the next post…

The pursuit of perfection

2006-05-24T00:00:00-07:00

The law of diminishing returns: more and more gets you less and less. It’s familiar in many domains: economics, martial arts training, propulsion. Rush Limbaugh would probably agree it even applies to drug use. It certainly applies to developing software. There’s only a certain value derived from polishing a chunk of code. So, while I’ll reserve opinion about whether this is a ‘law’ of nature, it’s definitely something we observe in many contexts.

It’s interesting to me to note where the law of diminishing returns appears to break down.

I have always been intrigued by one of the ad lines used by Lexus: the relentless pursuit of perfection. For me, perfection is spelled Porsche, but I like that Lexus boldly puts perfection as the aim. The idea of pursuing perfection seems to be thumbing one’s nose at the ol’ law of diminishing returns.

While we’re not all engrossed in engineering the ultimate automobile, I think human artistic endeavor in general doesn’t pay much heed to the law of dimishing returns. I imagine an artist relies more on a sense of holistic completeness when deciding to stop working on a piece. Education is another area. Imagine telling your boss, “Er, I think learning more about this is just pushing the law of diminishing returns.”

In the end, it may be that the law of dimishing returns governs all our activities, but intriguing to me is how our perception influences our efforts. In the pursuit of perfection, I think we perceive each additional effort as adding value. We don’t see that value dimishing or we would likely give up. So, despite the ‘law’ of diminishing returns, it may be our perception that makes the difference between accomplishing something mediocre and something great.

PLANET ARGON is my Google

2006-05-03T00:00:00-07:00

Late breaking news (I really need to speak to my publishing staff about getting these out on time): Monday, May 1st was my first day as part of the great team at PLANET ARGON. Everyone extended themselves to get me up to speed on the working of the company. And, in truly classy fashion, the company treated us to a team welcome lunch on my behalf.

I am delighted and honored to be working with these dedicated, enthusiastic, highly skilled and intelligent folks. Innovation in web applications in general and Rails in particular is occuring at such a rapid pace and PLANET ARGON is committed to being in the thick of it. On the day I started, if you pulled up google search, typed ‘ruby rails development hosting’, and hit I’m Feeling Lucky, it would send you straight to the PLANET ARGON website. I’m here to contribute all my expertise to these endeavors, and I expect to learn a tremendous amount along the way.

So, it may seem odd to compare a small company in Portland, OR to that giant of search engines, etc. but here’s the connection: People I know who aspire to work at Google do so because of the company’s perceived technological innovation and excellence. And that’s the connection for me. The PLANET ARGON team is pursuing innovation and excellence in application design, development processes, project management, training and more. I was certain I wanted to be a part of the action.

Now boarding for PLANET ARGON. Stay tuned and I’ll fill you in on the adventures.

The lure of the black diamond

2006-05-02T00:00:00-07:00

One great thing about Portland is the (usually) great snow less than 1.5 hours away. We have season passes this year for SkiBowl and the snow there has been great. But we had free passes for Mt Hood Meadows so we decided to go there this weekend. Turns out April 30th was a free ski and board day at all area ski resorts, so our tickets will be good for another day till the end of the season. Anyway, it was beautiful, sunny weather and a lot of fun.

Now, this is really my first season snowboarding (I went once last season) and I’ve never skied before. In other words, a rank beginner. But in this one season, I’ve gone from often sliding down on my backside, to mastering the falling leaf and stopping on my edge rather than my butt, and finally linking turns. I’ve also drastically reduced the frequency of sudden, bone-jarring endo’s resulting from catching an edge in the wrong direction. Along with the increasing skill, I’ve ventured into more difficult terrain. First the green circles, then the blue squares, and finally a couple black diamonds (but no double-black yet).

There’s a big difference between a green run and a black one. It takes a lot more effort; there’s a much greater chance of wiping out; it’s definitely intimidating to stare down that near vertical (it seems) face at the black run ahead. So, what’s the lure of the black diamond? If we consider some conventional wisdom about users of software, we may recall ideas like “users are lazy”, or “people take the path of least resistence”, or “someone will do the minimum to get by”, or “people are just pleasure seekers”. How does that add up? The path of least resistance is definitely not down the black diamond run.

It seems to me those ideas need refinement. I firmly believe that people are not lazy. Rather, living organisms are experts at balancing costs versus benefits, an expertise refined over hundreds of millions of years of evolution. I think people do what is most rewarding with the least cost. There is a premium on energy, so an organism must conserve it. That doesn’t mean always using the least possible, but rather using the least necessary to accomplish a goal. The difference is significant.

If we conceive of people as being lazy, we’re likely to design software that supports or encourages laziness. On the other hand, if we realize people are quite willing to expend the energy to tackle a black diamond run when the payoff in the thrill of the experience is great enough, we may wonder how we can design and construct software that brings out the best in people as they tackle challenges in their work. We may wonder how we can provoke people to be the best they can be. Does that sound too idealistic, too unrealistic? Or are we just too embedded in these silly over-simplifications that people are just lazy?

Xen and the art of noise (management)

2006-05-01T00:00:00-07:00

While I definitely appreciate the folks at Textdrive for being early, and continuing, supporters of Rails, I recently decided to switch to a VDS for essentially the same reasons as Octopod. Actually, I have him to thank for that little extra nudge provided by his blog entry. I hadn’t given the idea much thought until then because it seemed like overkill for hosting a couple personal sites. Silly me.

After quite a bit of research, I decided to go with Quantact as well. And thanks to Ezra’s great tutorial, I’m up and enjoying myself a lot. Well, mostly. I decided to use Debian 3.1, which I’ve never used before. However, since I’ve been using Ubuntu for the past several months (which was a pretty big switch after RH and Fedora for years), the process was mostly painless. The one area I’m not real comfortable with is the firewall. I’m using Bastille, but I haven’t closed the book on that yet. Actually, the idea that’s pinging around in my head is: I wonder how it would work to use Rake to write the rules for iptables. Has anyone tried something like this? I’ll let you know what I discover.

Okay, you might be wondering how the title fits in here. If you’ve checked out Quantact, you may have noticed they are using Xen for the VDS. Impressive technology. The rest? Well, when evaluating what I was finding most unsatisfactory about shared hosting, I realized it was the “noise”. All these processes, all these directories, all this stuff going on that had nothing to do with me. I was spending a lot of time in that noise, especially that webmin interface when I just wanted to type a few lines and be done. Now ps aux fills less than one screen of my xterm. I don’t think about special cases or file tickets for a port. Mind you, I’m not saying Textdrive has it all wrong. I’m just saying, my Xen VDS is nice and quiet.

My favorite button to push

2006-04-30T00:00:00-07:00

A long time ago, I remember discovering while using Windows 3.1 a button with a question mark on it. Click on it and you got a cursor with a question mark and pointer. It was the “what’s this” button; click on a control and up popped a blurb to explain something about it. Unfortunately, it seemed that developers saw this as mostly an after thought since I don’t recall ever getting anything useful. Usually it was something like “Name field” when the control was already labeled “Name:”. But it was the kernel of a great idea.

Taken further, the idea that software would have responsive features to assist the user manifests in context sensitive help agents and “show me” button activated tutorials. A reasonable argument would be that such features are required by overly complex and difficult software. But even as we strive to construct less software that is easier to use, how do we assess and implement “helpful” features.

The holy grail for me is software that can provide anything from a little to a lot of help, contextually, and with my own data, without intruding terribly on my flow (e.g. opening new windows, obscuring my view, losing my place). Take over the mouse pointer, click here, highlight this, type that. When I move the mouse, the “tutor” silently retreats and I continue with my work. Just like asking another person over to show me how to do something.

That sort of utility would encourage people to be more exploratory and, I think, derive greater satisfaction from using their software tools.

Oh, and I wish people had a “show me” button.