Log in

Previous 10 | Next 10

Oct. 31st, 2009

I'm a Busy Bee

Apis 1.0 is now ready to go. You can get it here.

Apis 1.0 is essentially the same application as Atta 1.0, but written in Common Lisp, using Clozure Common Lisp, instead of being written in Scheme using Gambit-C. The structure, appearance, and behavior of the two applications is nearly identical, so they may provide useful illustrations of the differences between Cocoa development using CCL versus using Gambit.

There are various superficial differences that you might notice right away. One is that Atta.app is about 7.5MB, while Apis.app is about 34MB. The size difference basically reflects the difference in features built into Common Lisp versus those of Scheme. Scheme is a much more compact language, which has its pluses an minuses. On the one hand, the Scheme application is much smaller. On the other hand, a developer will wind up reinventing more wheels (or hunting the net for someone else's version of a wheel).

Another obvious difference is that there are no Objective-C source files in the Apis distribution. It doesn't need them, because CCL's Objective-C bridge provides everything you need to define Cocoa classes and methods. That's an important point to some people; they want to be able to write a Cocoa app without having to see C code. CCL offers you that option, at the cost of having to learn the Objective-C-in-Lisp language of CCL's bridge. With Gambit, you must write your Objective-C code in C, but Gambit makes that easy to do. With CCL, on the other hand, you can write your Objective-C code in Lisp--but don't fool yourself; it's still basically Objective-C code, and you still have to learn Cocoa. Not that I'm complaining, mind you; if you're going to write Cocoa apps, you're going to have to learn Cocoa, no matter what language you use.

A fussier detail is that the build process of the two projects is a little different. Both use Makefiles, and I;ve packaged them to be about as similar in structure as is reasonable, but Atta uses Gambit and, through it, Apple's C compiler. Apis, on the other hand, uses CCL to compile, and Apple's C compiler is never involved. (Don't take that to mean you can build Apis without Apple's developer tools. Even if you replaced the Makefile with a Lisp program, you'd need Apple's ibtool to compile the resources for the bundle.)

The Atta source code is in three Scheme files and four Objective-C files. The Apis code is in five Lisp files. Ata is 260 lines of code total. Apis is 247. Apis launches in about a quarter of a second on my iMac; Atta launches in a bout a third of a second. Atta builds in about six seconds. Apis takes about nine.

For Apis, you'll need a recent install of Apple's developer tools, and a recent release of Clozure Common Lisp. I tested with XCode 3.2 and CCL 1.4.


Atta, Boy!

Last week a reader expressed a hope that I'll release Apis soon, and I intend to. That reader, though, hoped that Apis was written in Gambit Scheme, and it isn't. It's written in Common Lisp, and uses Clozure Common Lisp's very good Cocoa support.

Do not despair, however, Faithful Reader! I've packaged a Gambit Scheme variant of Apis, called Atta, which you can download here.

Atta is an ultra-simple text editor, the kind that you can build by following along with Apple's tutorial example, "Building a Text Editor in 15 Minutes." Instead of using XCode and pure Objective-C code to implement the text editor, though, Atta uses a mix of Objective-C and Scheme, using Gambit-C's spiffy C interface to make that easy.

The net being what it is, I expect to hear complaints that Atta is trivial. Yes, it is; that's the point. It's a nearly-minimal Cocoa application whose main program and parts of the application logic are written in Scheme.

I also expect to hear complaints that there's hardly any Scheme code--not even as much as the Objective-C code (and there's precious little of that). That's also true, but it's because the application is so trivial. There is just barely enough code to do anything at all, and, what with Scheme code tending to be so compact and all, there's hardly anything to see.

Finally, I'm pretty sure that someone is going to read through the code and complain that the implementations of saving and loading text files are unnecessarily complicated. That's also true, but if they weren't, then there really wouldn't be any scheme code at all. Well, apart from the call to "(begin (cocoa:application-main))".

Atta is intentionally minimal, so that it's easy to read through all the code and see what it's doing. It's not completely minimal, because there needs to be enough code that you can see something being done, and in order to serve that purpose, I implemented parts of the save and load methods in Scheme, though it certainly would have been simpler just to write a line of Objective-C code in each place.

The point of doing it the way I did is that you can trace through the program's logic and see how an NSAttributedString value begins in a Cocoa window, is extracted and passed to Scheme, which then passes it back to Objective-C to convert it to an NSData value, then gets the NSData back and hands it to the original Objective-C context, which writes it to a file. A similar process occurs when opening a file, so you can follow the progress of the data all the way from Cocoa window to disk and back again, passing through Scheme's heap on the way.

In this initial version, the Scheme code doesn't actually do anything with the data, except pass it around. It would probably be useful to think of some transformation to do on it as it whizzes by, in order to better illustrate how Gambit can convert C data to Scheme data and vice versa. I don't want to wait around for inspiration to strike on that point, though, so I've gone ahead and packed up Atta 1.0 and made it available.

Apis should be along as well in pretty short order. It's not any more complicated or difficult to package up than is Atta.

Oh, and somebody's sure to wonder what's up with the names "Atta" and "Apis". They're genus names of insects; specifically, "Atta" is the leafcutter ant, and "Apis" is the honeybee. When I first wrote Apis it was part of an experiment in writing a minimal Cocoa application that could serve as the front-end of a system of cooperating processes. Lots of cooperating processes; a swarm that makes something appealing or useful; honeybees; you get the idea. When I decided to port the basic application framework over to Gambit, I debated using the same name, but I thought, "I still want to release the Common Lisp version, and it would be nice if were easy to distinguish them. It would also be nice if it could have a similarly brief and easy-to-remember name, and if it stuck to the eusocial organism theme established by Apis." There are several eusocial organisms to choose from. "Naked Mole Rat" has its own kind of aesthetic, but just didn't seem quite right. After some consideration, I chose "Atta', the leafcutter ants, because they're eusocial, and I admire their invention of agriculture.

So that's it; the clamoring throng of one or two people who want a simple example of how to build a Cocoa app in Gambit can grab Atta 1.0, and Apis will be along shortly. Bear in mind that for Atta to be of any use, you'll need to have a recent version of Apple's Developer Tools , and also a recent version of Gambit-C.

Have fun!

Oct. 23rd, 2009

A peek at Categories

I've begun preparations to package Categories for release. The first thing I'm doing is going over the surface syntax. In previous iterations I've built Categories from the bottom up, which confirmed that the concepts and mechanisms worked, but left a few warts in the surface syntax. This time, I figured I'd start at the surface syntax and work the other way, in hopes that the result will be more congenial.

This rather long post describes the current state of the Categories surface syntax. Those who are interested in such things are invited to tell me what's wrong with it.

Categories has three logical parts: datatypes (representation), functions (behavior), and domains (taxonomy). Following Larry Tesler's maxim that "simple things should be simple; complex things should be possible," I'll start by describing just the simplest and most common use-cases, and then make another pass through all three parts of the system to add in the more complicated options.


Representation is the province of Datatypes. Types are first-class objects in Categories. They can be created by evaluating expressions, passed to and returned from functions, and bound to variables.

Categories defines a set of primitive type objects that represent built-in types of the underlying platform. For example, the Scheme version defines primitive types like <pair> and <symbol>, and it arranges for its reflective operations to return appropriate values. For example:

  ? (type 'foo)

  ? (type? 'foo <symbol>)

Type synonyms

The simplest possible way to define a type is to say that it's the same as some existing type. Of course, even simpler than that is to just go ahead and use the existing type. But sometimes you want a different name for the type, to better communicate its intended use. Sometimes you want to use an existing type for now, but may want to represent the data differently in the future. To make these uses easier, Categories provides type synonyms:

  (define <name> (type-synonym <string>))

By defining <name> as a synonym for <string> you can use <string> objects to represent names, while calling them <name>s in your code to clearly communicate their purpose. Later, if you find that you need to represent names in some other way--say as some structured object that takes into account various different kinds of names that people use--you can replace the definition of <name>. You'll then have to change the implementations of functions that operate on <name>s, of course, but if you structure your APIs appropriately, that's all you'll have to change.


More commonly, we want to create structured types by grouping several simpler types into named fields. In Categories, that kind of type is called a structure:

  (define <cartesian-point> (structure () x y))

Like other types, structures are first class objects. The above code binds a new structure to the variable <cartesian-point>. The new structure has two keys: x and y. Instances of <cartesian-point> will contain two values, one stored on the key x, and the other stored on the key y.

You can create an instance of a structure by applying the function make to it:

  (make <cartesian-point>)

You can get the value associated with a key by applying the function get-key:

  ? (get-key (make <cartesian-point>) 'x)
Error: Unbound key x

Unfortunately, we didn't specify any value for x when we created the instance. You can do that by passing in an initializer for the key:

  ? (get-key (make <cartesian-point> x: 101) 'x)

You can also specify a default value for a key in the definition of the type:

(define <cartesian-point> 
    (structure () 
        (x default: 7)

  ? (get-key (make <cartesian-point>) 'x)

Before I go any farther, I had better explain why there's an empty list after the symbol "structure" in:

(define <cartesian-point> (structure () x y))

When you create a new structure, you can specify zero or more existing structures to include in its definition. So if I wanted to define a new type of 3D point that is just like a cartesian point, but with an extra coordinate, I could define it like so:

(define <3d-point> (structure (<cartesian-point>) z))

"Okay," you're thinking, "that's nothing new. The new type inherits from the old one." Not so fast: there's no such thing as inheritance in types. Including a type just means you don't have to retype all its field definitions when you want to reuse them. Categories does support inheritance, but that belongs to domains, not to datatypes. Remember: datatypes are representation. Inheritance belongs to taxonomy, and that's the province of domains, not of datatypes.

Structure amenities

The structure constructor provides a few more conveniences. You can say that you want Categories to automatically construct getter functions for the new type's keys:

(define <cartesian-point> 
  (structure () 
             (x get: get-x)
             (y get: get-y)))

Now you can fetch the x value from a cartesian point like this:

  ? (get-x (make <cartesian-point> x: 101))

If you leave off the get: argument for a key, then you have to use get-key and the key's name to fetch the value.

You can also specify a setter:

(define <cartesian-point> 
  (structure () 
             (x get: get-x set: set-x!)
             (y get: get-y set: set-y!)))

Categories then constructs a function for setting the value associated with the key:

  ? (define p (make <cartesian-point> x: 101))
#<<cartesian-point> x: 101 y: #<unbound>>

  ? (get-x p)

  ? (set-x! p 1001)
  ? (get-x p)

If you omit the set: argument then Categories makes the key read-only. Categories is generally biased in favor of immutable data.


Now that we know how to define datatypes, what do we do with them? We apply functions to them.

Functions are objects that can be applied to zero or more parameters and return zero or more results. Functions as defined in Categories are polymoprphic. In other words, the same function can have several different definitions. The one that applies to a particular set of inputs depends on the inputs.

This practice of making the particular implementation of a function depend on its inputs is called polymorphism, and is one of the foundations of object-oriented programming. Choosing a specific implementation to run is called dispatching. Categories provides polymorphic functions with programmable dispatching.

We'll look at how you can create your own dispatching schemes below. First, though, let's look at what it's like to define and use functions in the simplest way.

The Default domain

The simplest way to define and use functions in Categories is with the default domain. Categories assumes you are using the default domain unless you tell it otherwise. The features of the default domain generally resemble those of CLOS.

Like types, functions are first-class objects. You can create one and bind it to a variable, like this:

(define serialize
  (function ((s <store>))
     (vector 'store
             (version s)
             (map serialize (columns s))
             (column-order s)
             (sort-column s)
             (sort-reversed? s)
             (map serialize (rows s))
             (notes s))))

The function object that is bound to the variable named "serialize" is created with a single implementation, corresponding to the type <store>. If we want to add another implementation, we can use add-method!:

(add-method! serialize ((d <document>))

Why do the parameter lists have double parentheses? Because each parameter may be type-qualified:

(add-method! serialize ((d <document>)(outs <stream>))

You can define a method to execute when no match is found for the inputs by omitting the type qualifiers:

(add-method! serialize (something somewhere)

It can be an inconvenience to keep track of where the definition of the function first appears, and to make sure that all the add-method! calls appear after it. If you instead use the define-function macro, that bookkeeping is handled for you automatically:

(define-function serialize ((s <store>))

(define-function serialize ((d <document>))

In that case, it doesn't matter which comes first.


So how exactly does dispatching work in Categories? Well, that depends on what domain you use. By default, you use the default domain. The default domain, which is named -c3-, uses a dispatching algorithm very similar to that of CLOS. It's not identical to CLOS; it uses a scheme first described in

That scheme, which is generally called "C3", yields dispatching behavior that is very similar to that of CLOS, but which in a few complicated corner cases produces less surprising results. It's the default in Categories because I like it, and because it has served the general programming community well. It was designed for the Dylan programming language, but has been adopted by implementors outside the Dylan community, including the Python implementors and the designers of the Class::C3 object system for Perl.

Although C3 is the default dispatching scheme, it's not the only dispatching scheme supported by Categories. Two other dispatching schemes are provided with Categories, and by defining your own Domains, you can make your own.

Before we get to that, though, let's just look at how to use something other than the default domain.

When you write:

(define-function serialize ((s <store>))

that expression is really shorthand. It leaves out a reference to the domain you're using. If we wrote it out in full, it would look like this:

(define-function -c3- serialize ((s <store>))

You can see that here we've said explicitly that we're defining a function that operates in the -c3- domain. We could instead use one of the other domains supplied with Categories, the Flat domain:

(define-function -flat- serialize ((s <store>))

The difference between -c3- and -flat- is that -c3- defines supertype relations among types, and -flat- doesn't. Suppose you make an instance of <repository> and call serialize on it.

  (define $r (make <repository>))
  (serialize $r)

What happens? That depends on whether serialize is defined on the -c3- or on the -flat- domain. It's the domain that defines how dispatching is done. If it's defined on -flat- then the answer is simple: serialize finds no match for $r. Unless a default method is defined as in the example above, an error is signaled.

The -c3- domain defines a more complicated--and more familiar--dispatching scheme. It will try to match $r against serialize methods that are defined on <repository>. When it fails to find one, it then tries to find serialize methods defined on supertypes of <repository>.

Didn't I say above that datatypes don't define supertype or inheritance relations? Yes, because domains do. The -c3- domain defines supertype relations with multiple inheritance. It tries to find an inherited method that matches, using the C3 algorithm to determine which matching method is most specific. Once again, if no such method is found then either the default method is called or, if there is no default method, an error is signaled.

More options

That's a first pass across the simplest and most commonly-used features of Categories. Let's go over the parts of the system one more time, and look at a few more options that it provides.


First of all, I left out an important group of types, because it's hard to explain what they're for until you've seen how Categories works with domains. A constructor named "category" creates a kind of type that is different from the others. For example:

(define <vehicle> (category <car> <truck>))

The expression above binds a new type to <vehicle>. The new type, which is called a category, represents any of the types passed as inputs to the "category" constructor. Once <vehicle> is defined, any method matching <vehicle> will also match <car> or <truck>. You can therefore define functions that accept <vehicle> values, and those functions will work on <car> or <truck> values, even if the functions are defined on the flat domain. In other words, even without inheritance, it is possible to define functions that operate the same way on a variety of different types.

Supertypes and subtypes

Okay, but suppose you actually want types that have supertypes and subtypes? Well, then, you can use -c3-,or any domain that supports supertype relations, and you can add your own types to it. For example,

  (add-type! -c3- <truck> 
      supertypes: (list <transportation> <four-wheeled-objects>))

...adds to -c3- a type named <truck>, with supertypes <transportation> and <four-wheeled-objects>.

You can, of course, also add types to the -flat- domain:

  (add-type! -flat- <truck>)

You can't specify any supertypes for it, though, because -flat- doesn't support supertype relations. If you try to, then add-type! will signal an error.

Name collisions

What happens if I define a type that includes two other types that have different definitions for the same key?

For example, suppose I have these two definitions:

(define <monument> 
  (structure () 
       (age type: <historical-epoch>))

(define <property> 
  (structure () 
       (age type: <count-of-years>))

Now suppose I want to define <historical-property> like this:

(define <historical-property> 
    (structure (<monument> <property>)
        (square-footage asking-price))

Well, I can't. The definition of age in <monument> conflicts with the definition of age in <property>. What I can do, though, is rename one or both of the included keys:

(define <historical-property>
  (structure ((<monument> (age as: period))
                     (<property> (age as: how-old)))
    (square-footage asking-price)))

The Predicate domain

Besides the default C3 domain and the flat domain, Categories includes a third domain that works very differently from either. The -pred- domain matches input function arguments with a user-defined predicate function. For example:

(define-function -pred- ((x y z) all-odd?)

The example above shows a function that will match any three inputs, as long as calling the function all-odd? on the sequence of three inputs returns true. You can define functions that match on any predicate you like. You just have to ensure that when the function passes the inputs to your predicate, the predicate correctly returns either true or false.

The -pred- domain also uses the C3 algorithm to support supertype relations among predicates. Using the -pred- domain you can construct whatever relations among predicate functions you like, and Categories will dispatch appropriately. For example, you define all-integers? to be a supertype of all-odd?, and then anything that matches all-odd? will also match all-integers?.

I should probably mention here that -pred- may not be all that useful to anyone. It exists mainly because I wanted to make sure that it was workable to build a dispatching scheme in Categories that was quite a bit different from the usual thing.

One mystery remains with the -pred- domain, though: the syntax for defining a function is different from the syntax used with the -c3- and -flat- domains. How does that work?

Defining domains

The unique define-function syntax of the -pred- domain is provided by the domain itself. It turns out that the operators "function" and "define-function" don't define the syntax of parameter lists; the domain does. The -c3- and -flat- domains both define a syntax that is familiar to users of CLOS and similar object systems. You write define-function forms like this:

(define-function -c3- add ((x <integer>)(y <integer>))

The forms (x <integer>) and (y <integer>) mean that the formal parameters x and y match inputs only if those inputs can be matched with the <integer> type, according to the domain.

A domain like -pred- works quite differently, and requires a different syntax for parameter lists. Instead of piecewise matching input values against types, -pred- matches the parameter list as a whole against predicates applied to it. That makes it possible to dispatch on conditions that -c3- and -flat- can;t check. For example, -pred- can match a sequence of five integers that are monotonically increasing, or a sequence of characters that forms a palindrome.

But if some domains require unique parameter-list syntax, that means Categories must have a way to define that syntax; and it does.

When you define a domain, you are defining three things:

1. a format for function parameter lists

2. a function that determines whether a particular method matches a given sequence of input values

3. a function that can tell you which of two methods better matches a given set of inputs

Here's what a domain definition looks like:

(define -my-domain-
  (domain catalog: some-catalog-data-structure
          parser: (lambda (domain formals) ...)
          matcher: (lambda (domain signature values) ...)
          comparator: (lambda (domain signature1 signature2 values) ...)))

Right away you'll notice that, although I said you have to define three things, there are four inputs to the domain constructor. Categories provides each domain with a data structure called a catalog. The catalog can be anything you like. It can be a false value, or an integer, or a table, or a more complex data structure. It's up to you. Its purpose is to provide storage for any auxiliary data that your domain might need to perform its tasks correctly and efficiently. You can use it to store method caches, or arrangements of supertypes designed to make lookup fast, or whatever you wish. You can also ignore it completely, if you wish.

The other inputs to the domain constructor are essential, though. Each of them is a function that defines one aspect of how the domain works. Because your domain may have crucial data stored in its catalog, all three of the functions accept the domain itself as the first argument. Its up to you what you do with that argument; it's available in case you need it.

Besides the domain parameter, the functions that define a domain deal with three other items: formal parameters, method signatures, and input values.

A formal parameter is an object that represents the formal arguments in a function definition. Categories doesn't define the format of argument lists; that's up to the domain. When it processes a function definition, it extracts the formal parameter from the function or define-function expression, and passes it to the domain's parser. The parser must accept the formal parameter and return a method signature. It's the parser that determines what the argument list of a function definition looks like.

The parser doesn't have to be complicated. Consider a -c3- function definition again:

(define-function -c3- add ((x <integer>)(y <integer>))

All the parser needs to do is collect the type qualifiers from the formal parameter, which is ((x <integer>)(y <integer>)), and make a method signature that matches two inputs that are both of type <integer>.

That brings us to method signatures, and to the matcher. Once again, the method signature can be any data structure you like, as long as it stores the information that the matcher needs. Categories passes to the matcher a method signature and a collection of input values. The matcher must return true if the signature matches the inputs, and false otherwise. For -c3-, a method signature can simply be a sequence of types. For example, the method signature for the add function defined above is just (<integer> <integer>). The -c3- domain will say that any sequence of two values matches that signature, as long as both are of type <integer>, or of some type that, according to -c3-, is a subtype of <integer>.

What if more than one signature matches a set of inputs? How does the domain know what to do?

That's what the comparator is for. After a function determines all the signatures that match a set of inputs, it sorts them. The comparator is the function it uses to determine their order. The comparator must accept two signatures and a sequence of input values. If the first signature is a better match than the second for the inputs, then the comparator returns -1. If the second signature is a better match, it returns 1. If neither is better, it returns 0. It's best to try to arrange for the comparator not to return zero, if that makes sense for the domain you;re designing, because returning zero means you can't tell which method to apply. Categories has to signal a dispatch error when that happens.

Customizing the default domain

It may happen that you want to make your own domain, but you don't really want to totally rewrite how everything works. Maybe, for example, you want to use something other than C3 to order methods, but everything else about how Categories works suits you fine. In order to make it easier to customize just the part you want to customize, Categories provides functions that implement the behavior of the default domain. You can make your own domains using them, and you can replace just one of them, or two of them, or you can use all of the standard functions, but change their inputs or outputs a little using your own code.

As an example, you can make a domain that duplicates the behavior of the default domain like this:

(define -my-c3-domain-
  (domain catalog: (make-c3-catalog)
          parser: (c3-signature-parser)
          matcher: (c3-signature-matcher)
          comparator: (c3-signature-comparator)))

What next?

All of the features described in this overview have been implemented, but the working code doesn't provide this surface syntax. The syntax of the working versions is not too different in most places, but it's a little more complicated and not quite so tidy. I'll be fixing that as part of getting the code ready for release.

The prototype I've most recently been working on is implemented on top of Gambit Scheme 4.5.2. I've tried to avoid making it particularly Gambit-specific, but the surface syntax I chose does use DSSSL keywords, which not all Schemes support. In addition, some of the syntax needs to be defined as macros. For reasons of convenience, those will be defined using Gambit's define-macro form, rather than something more portable. It may turn out that someone wants Categories on some Scheme that lacks DSSSL keywords, or where a different means of defining macros is needed. I'll cross that bridge when I come to it.

I plan to port it back to Clojure, as well. The first few versions of Categories were written in Clojure, but I wanted to move it elsewhere, mainly to gain the perspective of looking at it as something independent of a particular language, rather than as an extension specifically to Clojure. I'm planning a sizeable project in Clojure shortly, though, and I know for sure that I'll want what Categories offers me for it.

It may be that the design of Categories has serious flaws. If so, feel free to enlighten me about them. If they can be repaired, I'll do my best to fix them. If not, I'll have to look for alternatives. I could always port Sheeple to Clojure, or maybe implement PLOT's object system.

Oct. 18th, 2009

Gambit considered helpful

I've just packed up Delectus(tm) 1.0 Alpha 2 and shipped it off to my testers. Delectus is an application for keeping track of collections of things. The inspriation for it was an ancient and hoary program my mother used to keep track of her movies. She has around a thousand movies, the majority of them still on videocassettes. Each cassette has a catalog number on it, and she uses a database program to associate those numbers with information like the title, star, genre, and so forth, so that she can quickly find a particular movie in her stacks of shelves. Recently the software, which is over twenty years old, finally became obsolete. As we were looking for a replacement, I decided I'd just write one, and that became the genesis of Delectus.

The basic design of Delectus is very simple. A Delectus document stores a list of rows and a list of columns. Each column has a text label. Each row contains a list of text fields. The number of fields is the same in every row, and it's also the same as the number of columns. So you can create a document with the columns "Title", "Genre", and "Star", and every row will then have three fields. You can make a new row and put the title of a movie into the "Title" field, the star into the "Star" field, and the genre into the "Genre" field.

Delectus doesn't decide for you how many columns to create or what the names will be. Instead, a new Delectus document has no rows and no columns. You can add new rows and new columns at any time, and you can delete them at any time, too. This flexibility makes it easy to set up your databases any way that is convenient for you. You can sort a document's entries, forward or backward, by clicking the heading of a column. If every entry in a column is a number, Delectus detects that automatically, and sorts by number instead of lexicographically.

Delectus uses a trash can, like OS X's trash can. When you delete a row or column, it's not really gone, but tossed into the trash. Click a trash button, and Delectus shows the trash to you, integrated in the document, but displayed in a different color, so you can easily tell which items are deleted. Empty the trash, of course, and the deleted items are gone forever.

Delectus is mainly written in Scheme. The Scheme compiler I'm using is Gambit-C. A few weeks ago, I wrote a post about the happy discovery that Gambit-C made it easy to work with Mac OS X's Cocoa frameworks. I said Gambit was a great platform for Cocoa development. That opinion drew some fire, but I haven't changed it. To the contrary, I've grown happier with Gambit as I've developed Delectus.

Some people grumbled that I'm not allowed to call it a great platform if I still have to deal with C code. Now, that's silly. You know, I actually had a Lisp system once in which I never had to deal with C code. It was called a Symbolics Lisp Machine, Every other Lisp I've used--and I've used a lot of them in twenty one years of programming--has had to deal with C in one way or another, at one time or another. Heck, even when I was writing Lisp code for an operating system written in Lisp (an unshipped version of Apple's Newton), we had to deal with C code. If your eyes are damaged by the sight of semicolons, a hard road awaits you. C code is in the underpinnings of every system you are ever likely to use. At least Gambit makes it easy to wrap our beloved parentheses around it, so that the hothouse flowers in our midst are protected from the hot exhaust bits spewing forth from the C code beneath our feet.

The other common complaint seems to be that I made a grammatical error in using the adjective "great" to modify a set of development tools that does not include an IDE. To paraphrase Yoda, "IDEs not make one great." Sure, there have been great IDEs. Smalltalk-80 was a great IDE. Xerox Interlisp was a great IDE. Most IDEs are not great. Most IDEs are going to be great, someday. That is, they're going to be great, right up to the point where the weight of the implementation collapses the envelope of optimism providing their buoyancy, and they crumple into a heap of random features, slowly and inexorably sinking under their combined mass.

IDEs are large, complicated systems that require serious design and engineering. They are major cost centers to anyone who is not in the business of selling IDEs, and the market for them is small. It should not be surprising that they're mostly not all that good.

What's more, the benefit of even a great IDE is, let's face it, marginal, as long as the basic tools are comprehensive and good. So yeah, I'd like to have a great IDE. I'm not holding my breath. Maybe I'll like the Factor IDE. It looks pretty good. But I'm working in Scheme, right now.

So, yeah, folks who never want to see a line of C code, you're not going to like the tools I'm using. Ditto for anyone who must have his projects managed by a digital valet in an impeccably-tailored suit of color-coordinated windows. There's nothing for you guys to see here; move along.

For me, though, Gambit has definitely been great. In the development of Delectus, I've ditched XCode (but not Interface Builder). In Alpha 2, 83% of the application code by linecount is Scheme code. The remaining 17% is Objective-C. Basically, there's just enough Objective-C to load the nibfiles that define the app's windows and menus, and hook their event-handlers to the Scheme code that does all the real work.

After some experimentation, I partitioned the application into the following parts:

1. The back-end: this is pure Scheme code that handles all the data structures, as well as sorting, filtering, and updating. It also handles serializing and deserializing data as CSV and a native binary format, and writing to and reading from disk files.

2. The UI: this is Objective-C code that loads the nibfiles, sets up calls into the back-end, responds to notifications that originate in the back-end, and provides the minor customizations I needed to make Apple's UI elements look just the way I wanted. This part is less than 700 lines of code, total.

3. The bridge. This is a small amount of Scheme code, and an even smaller amount of Objective-C code, in a couple of files. It provides simple utilities like a means of converting Scheme strings to NSString objects when the UI needs to fetch a string from the back-end, without being prodigal about allocation of strings. It provides a mechanism for notifying the Objective-C code from Scheme code, so that the pure-Scheme back-end can post error messages, update requests, and other notifications, without any code that is actually aware that an Objective-C component exists.

The last point is maybe interesting. There is a very small amount of mixed Scheme/C code that enables the Objective-C UI to interact productively with the Scheme back-end. I could easily have put that code into the main body of the back-end proper, and at first, that's what I did. After all, there's only a little of it; what's the harm?

Before long though, I refactored the code to completely separate all that code into a bridge layer. The reason, basically, is that I like to have a fairly comprehensive set of unit tests around my major application functionality. With such a set of tests in place, it's very easy to detect when a change in the code makes an application feature fail, and it's easy to find what broke it. By factoring all of the mixed code out into its own small layer, I was able to make the back-end pure Scheme code, with no reference anywhere to any foreign functions. That means that I can load the entire back-end into an interactive repl and exercise it. I can run all the unit tests at once, or run particular ones interactively. If one breaks, I can use the interactive debugger to crawl around inside the back-end's runtime and figure out what's wrong.

Furthermore, by factoring things this way, I create a circumscribed and coherent API for calling into the back-end and getting data from it. That makes it very easy to hook up the Cocoa UI, simply by having event-handlers call the appropriate API functions. UI calls into the back-end are almost always one or two lines of code. That means that, even when a bug is in the UI rather than the back-end, the way the app is factored makes that easy to detect. And since there is so little front-end code, it's easy to find and fix.

Obviously, you could structure any Cocoa app similarly, whether the back-end is Scheme or C or ML or whatever. What makes Gambit great for me, a Lisp hacker by preference, is that over eighty percent of the code I work with is pure Scheme code that I can work with interactively, in the way I prefer.

You could argue that Delectus is really a C program, on the grounds that Gambit-C compiles Scheme to C, before subsequently compiling it to native code. That suits me fine, as long as I can continue to write my C code in Scheme.

I'll say it again: Gambit-C has been a great platform for developing a Cocoa app. I'll enthusiastically use it again in the future to do the same kind of work. I would recommend it to anyone who wants to use Lisp to write a Cocoa app, provided that they're not allergic to C code, and provided that they're okay with using compilers and editors without an IDE to command the servants for them.

Delectus should be out soon. The gating factor right now is the opinion of the testers. When they're happy with the feature set and the stability of the product, I'll release it.

After the release of Delectus, I'll begin work on a web application aimed at MMORPG players. Most likely, the server side of that application will be written in Clojure.

A little while ago, I promised interested people that they would soon be able to get their hands on my Categories object system. The object system had to take a back seat to revenue-generating work, but when Delectus is out, I'll be working on a release of Categories. I intend to use it in my next product anyway, and so I may as well also package it for release to the tiny community of people who are interested in such things.

The current working prototype of Categories runs on Gambit, and it's likely that's the version I'll clean up and release first. Then I'll port it back to Clojure (which is where I developed the first versions of it), and make that available as well. It may be a little while before I do either; we must now wait on the verdict of my testers. When they say Delectus is done, then I can move forward.

I have several other plans in the works, including the resurrection of some old friends. After six years, I still occasionally get fan mail about Alpaca, the text editor. Those few fans may be glad to know that a new version of Alpaca is on the roadmap. Another old project that I'm still sometimes asked about is Bosco, a simple set of tools for building a Cocoa application in Clozure Common Lisp. Well, Bosco was absorbed into Clozure Common Lisp a couple of years ago, but there stlll seems to be a place under the sun for a small package of files that shows how to build a minimal Cocoa application with the smallest possible amount of Lisp code. I have another project called Apis that can serve that purpose, among others. Apis, and other things derived from it, are also on the roadmap.

There are other things, but they're even further down the road. For now, I'm still squarely in the midst of Delectus, and loving Gambit. Turns out it's still a great platform for developing Cocoa apps.

Oct. 12th, 2009

Could Be a Contender

Champions Online is a flawed, brilliant game that dares to try injecting a few really novel ideas into MMORPGs. Its developer, Cryptic Studios, is struggling a little with the issues raised by their departures from the beaten path, but I have to hope they succeed in what they're trying to do.

Fast Fun

Obviously, when you design a game, you want it to be fun. If it isn't fun, a game has no reason to exist. There are a couple of different ways to approach fun in MMORPGs, though, because of the kind of game that they are. On the one hand, the MMORPG developer hopes to seduce players into coming back again and again, and devoting hundreds of hours of leisure time to the product. To accomplish that goal, a game needs fun aspects that emerge over time; it needs surprises; hidden treasures; happy discoveries that a player has to discover through long labor and exploration.

On the other hand, if the first few minutes and hours of play are not fun, then players might never discover any of those lovingly-crafted surprises.

Of course, in an ideal world of games, every MMORPG would be both instantly exciting and lastingly rewarding, but the reality is that every development team has to make tradeoffs in order to get a product out the door. There is never the time, money, or team needed to polish absolutely every aspect of a game to the high gloss that its inventors would wish. Decisions have to be made. Do you focus your efforts primarily on getting buyers' attentions in the first place, or in hanging on to it once you've got it? Different studios go different ways. Although World of Warcraft is certainly a rich, complex, deep game at this point, at launch the emphasis was pretty clearly on that first impression. Blizzard worked hard to make sure you'd have a lot of fun in the first couple of hours. Turbine's Lord of the Rings Online is an example of a game whose design leans the other way. The best things about LOTRO are not obvious in the first couple of hours of play. They take time to find and appreciate.

With Champions Online, Cryptic has gone the same way as Blizzard, and with a vengeance. The new player experience is crammed with lights, whizzy options, and action. The fun starts with the character creator, which provides seemingly unlimited power to build any kind of character you want. Height, weight, and hair, eye, and skin color are only the beginning. You can make hulking green behemoths with hands the size of a small car, or tiny, wispy fairies with butterfly wings. Robots, werewolves, slacker thugs, and little old ladies are all well within the range of the character creator. You can have a good time for hours just picking an idea for a character and seeing how close you can come with Cryptic's tools. Usually, you can get nearly as close as you want to.

The freedom to explore doesn't stop with your character's looks. Abandoning the now-traditional half-dozen classes, Cryptic instead gives us eighteen "Power Sets": collections of abilities that revolve around a common theme and a common pair of supporting statistics. You can choose from options like Fire, Ice, Force, Munitions (guns and bombs), four flavors of Martial Arts, Gadgeteering (robots! attack toys! orbital beam weapons!), Might (brute force) and a slew of magical options. More interestingly, the Power Sets are not mutually exclusive as are the classes in most games. With minor restrictions, you can cherry pick abilities from the full range of Power Sets, creating your own style of character by the simple expedient of picking abilities that match the concept, regardless of the Power Set that provides them.

Once you actually begin to play, the fun kicks into high gear. In no time flat, you're blasting, chopping, capping, stomping, or incinerating hostile aliens in the middle of explosions, wailing sirens, and hysterical NPCs screaming for your help. The game propels you through a whirlwind tutorial that spits you out into one of two quest hubs, where you can then acquire then next big dose of fun: your first Travel Power.

There's no lengthy grind through thirty or forty or fifty levels before you have the opportunity to spend the wealth acquired in hundreds of hours of play to buy some sort of banal mount to ride around the landscape. Nope; in Champions, you get to about level five, whiz off to a new zone, and buy any one of over a dozen different ways to zip around the landscape like a superhero. You can learn to fly, jump a quarter mile, swing from the rooftops, run at formula-one speed, or tunnel through the earth like Bugs Bunny taking a wrong turn at Albuquerque. By itself, the travel power is good for a decent amount of fun, and if you pick the right one for your character and learn to use it well, it adds a new dimension to your fighting abilities.

Truly, Champions delivers a truckload of fun right at the start of play. For the first couple of hours it seems like you'll never stop saying, "wow! that's cool!"

Fast Flaw

Along with this freedom, though, Champions offers us the first serious flaw in the game. Not all combinations of powers work equally well. One of the first questions an experienced MMORPG player wants answered about a new game is: is it possible to gimp your character? Will an unfortunate choice of powers make your character weak and ineffective? Will it hamper your experience of the game? In Champions, the truthful answer has to be yes; you can easily make an unfortunate choice that pretty much kills all the fun.

Balance is always a tough problem in MMORPGs. They're big, complicated games with a lot of variables in play. It's inevitable that some design decisions will have unexpected and unfortunate consequences for players. The players know this, and make some allowances for it. With eighteen Power Sets (and more on the way), each with dozens of powers, there are bound to be a few glitches and gaffes. We all understand that, and it doesn't have to be a serious problem. But this is the first place that Cryptic shoots itself in the foot: Champions Online shipped with a mechanism for rebuilding characters that got in the way more than it helped.

You would think that in a game whose new player experience so much emphasizes freedom of choice--freedom to choose your looks, your theme, your powers and abilities--that it would be dead easy to change your choices if you find that you've made some bad ones. It isn't. You can visit the Powerhouse--the same location where you buy new powers--and remove powers that you'd acquired, to make room for different choices. So far, so good. Unfortunately, in order to remove a power that you acquired a couple of visits back, you have to first remove all the ones you acquired after it. If you decide that the lightning bolts you bought three visits ago are the wrong thing, then you need to get rid of the other things you bought after them before you can touch them. That wouldn't be so bad in itself, but you also have to pay to remove each power, and the cost of each power you remove goes up significantly. If the bad choice you made is more than five or six visits ago, you're looking at a serious chunk of change. Ouch.

To make matter worse, on the day Champions launched, Cryptic hastily issued a patch that drastically reduced the toughness and effectiveness of player characters across the board. Beta testers were gobsmacked by the changes. The same characters who had, one day previously, mopped the floor with hordes of bad guys, now struggled to complete tasks that required defeating two or three mobs. In fairness, there had been some complaint that the game was a little too easy, and the game after the patch is not unplayable. But to issue such a drastically game-changing patch on the day of launch was pretty ham-handed, and when you combine it with the high expense of removing powers at launch, it felt to many players like a slap in the face. In effect, Cryptic took away all that made your character fun, and simultaneously made it so expensive to change builds that the only realistic option was to dump your character and start over. Poor form, Cryptic; poor form.

There were worse effects, too, but they were less obvious. We'll get to those soon enough.

Recently, Cryptic lowered the cost of removing powers significantly, which is a step in the right direction. The mystery is why, in a game where the freedom to experiment with different combinations of powers is so important, they raised that barrier in the first place. It takes experimentation to find the right combination of theme and powers, to create a character that is right aesthetically and that is also fun to play. That Cryptic would put an arbitrary barrier in the way of that discovery is mystifying.

The Color of Funny

Once you're past the first few hours of rapid-fire fun, you want to know that the game is going to remain involving. We play MMORPGs because we like playing in an imaginary world we can come back to. We like to come back to a character that is in some way better to play with than the last time, and where we can roam around and find new things to do, new challenges to face with our increasingly mighty toys.

Champions gives us a brightly-colored comic-book world that is great to look at. Moreover, the responsiveness of the game and the quick-moving style of combat is intensely engaging. The emphasis in Champions is on powers that you can use on the move. You often find yourself hurtling through the air tossing things that light up and explode and zap your targets. With a little practice, combat starts to resemble the ballet of violence in an action movie more than the relatively static slugfests with light shows that are more traditional in MMORPGs. Probably too much has been made of Champions being an "action" MMORPG; the differences rom older games are not all that earth-shaking. Still, Champions does bring something new to the table. It's positive contributions in combat have as much to do with characters that bounce off each other, real line-of-sight, and destructible objects that you can blow up or throw, as with moving combat, but the innovations are real, and they're fun. Champions offers tactics that aren't available in older games, and this new freedom of action feels like more power. Often, if a bad guy was too tough for you on the first try, you can think of ways to exploit terrain and environment to win your second fight with him.

It's a shame that the UI often feels balky. Between a complicated set of independent cooldowns and the occasional bug, you sometimes feel like you're just banging buttons to no effect. The problem doesn't happen all that often, but when it does, it's inevitably when you are in the middle of some complicated fight against multiple enemies, and you really need that ability to fire when you hit the button. If it as just a matter of learning the abilities and the cooldowns and the power requirements, then you'd shrug and say, "Hey, I'll get better." But there really are times, occasionally, when you hit a button, and nothing happens, and there's no discernible reason for it. All the cooldowns are over; you have plenty of power; there are no debuffs on you; the target is in range; and still, you hit the button and nothing happens. Operate some other control or move a millimeter one way or another, and suddenly it works again.

This balkiness is not a deal-breaker. The game is still huge fun, and undoubtedly incidents like this are just the inevitable bugs. Without doubt, Cryptic will find and fix them in due course.

Maybe more troublesome is the issue of leveling content, and the larger, more subtle issue of what makes a good game world.

When Cryptic was adjusting the mightiness of player characters, it was also monkeying with the experience rewards. Late in the Beta testing process, it seems that Cryptic concluded that, between the might of the player characters and the fast rate of leveling, it looked like players might rocket to the endgame before there was any endgame for them to play. We can forgive them for being nervous about that prospect. We've seen that happen in other games, and we've seen the inevitable effect of bored endgamers on subscription numbers. There had better be something for players at maximum level to do. If there isn't, it's only a matter of time before they go looking for another game.

Cryptic's solution was to slow down leveling. A natural enough choice, and it solves the problem of bored endgamers--or rather, it delays it long enough to give the studio a chance to come up with more content--but in this case it led to another problem.

Champions has just about enough missions available to keep a leveling player busy, if you are diligent about finding all of them, and if you complete every single one of them. Missions get a little thin in the low twenties, which leads to lots of people running missions above their level. For people with really strong character builds, that works pretty well, apart from the fact that you're eating into the content that should be there for you at those higher levels. For people with weaker builds, the situation is tougher. In some games, you could address the problem of a lack of content by grinding mobs for experience. That's kind of a rough way to go in Champions, which strongly emphasizes questing as a means of advancement, and therefore awards fairly paltry experience for mob kills.

When you combine just enough missions, a low experience return on kills, and patches that reduce the experience return from completed missions, what was a bit of an inconvenience becomes a real pain. MMORPG players have become accustomed to games that have so many quests available that we can afford to pick and choose. We can run the ones that we find fun, and give the boring or annoying ones a pass. That doesn't really work when you need every last mission just to reach your next level. Maybe we wouldn't mind a gap in the fun quests now and again if we could make up the experience in mob kills. Maybe we wouldn't mind paltry XP returns from mobs if our characters were still the godlike figures of Beta. But when quests are scarce, mobs are stingy, and our superpowers have been reduced to shadows of their former selves, the fun starts to grow scarcer, too.

There's a sameness to a lot of the missions that starts to wear on you after a while, too. By some time in the twenties you begin to realize that any time you accept a mission, there is a better than even chance it's going to be eerily familiar. Go to this address (haven't I been here before? Like, several times?), go inside and find a group of [demon worshippers/paramilitary insurgents/mobsters] trying to do Some Bad Thing, knock twenty or thirty of them down, maybe pick up a thingie from them, and out again, you're done. These missions are basically fun, and there is some variety. I'm a little concerned, though, that the sameness is going to start wearing on us after a couple of months of play.

We need not despair. For one thing, even when you're grumbling around looking for a quest in your level range and not finding one, the basic mechanics of your travel powers and clobbering random bad guys is so much fun that you're willing to forgive. For a while, at least. Surely, Cryptic is using the time and money that launch bought them to prepare more content for us, with greater variety. They'd better be.

The Big, Bright Room

We expect an MMORPG to present us with a virtual world that seems, to a certain extent, real. It's not actually realism we want, of course; it's verisimilitude. We don't really want it to take days of riding to cross a continent. We don't want our characters to come down with an infectious disease and be bedridden for days. We want just enough of the suggestion of reality that we can easily suspend disbelief just enough to enjoy the action and the story of the game.

Different developers devote different levels of effort to creating and maintaining the illusion of a world. Like every other aspect of designing an MMORPG, the game world is a large and complex undertaking, and hard choices have to be made about how much time, effort, and money will be spent on it. Sometimes, as with Lord of the Rings Online, the nature of the property demands an extraordinary effort of craftsmanship, and the result is a rich, varied, subtle, and believable world. In other cases, that level of craftsmanship in world-building is not a priority.

Champions is a game of this latter kind. It gives us, not a world, but three great big, brightly-lit rooms in which to play. Each one is very large--larger than it appears at first--and each one is packed with places to go and things to do. None of them feels much like a world, though, once you get past the initial couple of hours of play.

One reason that Champion's world doesn't seem like a world is that there are really only three places to go (neglecting Monster Island and Lemuria, which only become available after level 30). There is Millennium City, the transformed vision of Detroit where the Tutorial starts; there is the Southwest Desert; and there is the Canadian Wilderness.

Naturally, each of these destinations is spacious and each contains many smaller destinations within it. As with the day-to-day missions, though, there is a sameness about everything in each of these places. The desert, the city, and Canada are distinct enough from each other, but within any of these spaces, one place looks much like another. There is none of the exquisitely subtle care seen in a game like Lord of the Rings Online to make each small place distinctive and different from every other; nor is there the relentlessly organized approach that we see in WoW, where each zone is consciously made to starkly contrast with its neighbors. There is no rounding the crest of a hill to find revealed some new and enchantingly different landscape before us. Instead, there's a great big rectangular room decorated in Urban Modern; a great big rectangular room decorated in Santa Fe Style; and a great big rectangular room decorated in Arctic Tundra; that, in Champions, makes up the world.

It's not only that there are only three scenes, and each of them is a uniform set piece. It's that Champions makes not even a token attempt to conceal the edges of the stage. Blizzard took some care to place the edges of its world out of easy reach, and to discourage players from finding them by placing everything interesting you'd want to do far away from them. Turbine went further with LOTRO, consciously and carefully designing every zone to put the edges not only out of reach, but out of view, and working hard to make every boundary seem a natural part of a believable landscape.

Champions doesn't even try to do any of this. Each zone is a big rectangle and the edge is the edge, full stop. You can be running down a street in Millennium City and suddenly smack into an invisible wall that has no relation at all to the scenery around you. There's no wall, no gate, no unpassable chasm; there's just a street with cars and pedestrians, and many more blocks of street ahead of you, and you can;t go there. When you try, you just bang your head on the invisible wall.

Partly, this design decision has to be an acknowledgement of the problems that travel powers create: how are you going to carefully craft terrain that conceals the world's edges, when players can fly anywhere they want? Well, Blizzard's work on The Burning Crusade provides one answer, but that's a heck of a lot of work, and deadlines have to be met. We can forgive Cryptic the decision to put their efforts elsewhere, especially since they put it into making the first few hours of the game huge fun, and succeeded brilliantly. Prospective players should be aware, though, that if they're looking for a world they can dwell in, a world suited to immersive play, Champions is not it. Those of us looking forward to Cryptic's forthcoming Star Trek Online have to hope they went a different way with that title.

It Ain't Over 'Til It's Over

Okay, Champions has problems. Cryptic has fumbled a little with the gameplay, especially in balancing character power, quest rewards, and customizing characters after they're created. In tuning the gameplay after launch, they've created a bit of a content problem for themselves. Quantity is not the only problem there; they need to work some on variety, too. And if they stick with the aesthetics that drove the design of Millennium City, the Southwest Desert, and the Canadian Wilderness, they're going to have a problem with immersiveness, and with players just plain getting tired of the same thing all the time.

All that said, with all its problems, Champions is still enormous fun to play. All the Power Sets provide a pure, visceral excitement, and advancement is a cafeteria of great abilities that you can use to build nearly anything you can imagine.

Even with all the issues the game has, it's still well worth buying and playing, at least for a few months. Whether it's worth continuing past that point depends on what Cryptic does next. If they can iron out the balance problems, get out of the way of players experimenting with powers, address the sameness and scarceness of mission content, and give us some place more interesting to go than the same three big square rooms, then we'll be playing for a long time.

If not, well, a few years down the road we'll be saying, "Remember how much fun Champions was?" "Yeah, why doesn't somebody give us a real game that's that much fun to start?"

Oct. 1st, 2009

Redefining MMOs: The game no one is making

The "Massively" blog (http://www.massively.com/), which publishes news articles about MMORPGs, is wrapping up a series called "Redefining MMOs", and they've asked readers to tell them what they want to see in next-generation, "redefined" MMORPGs. You can find related articles here:


For Massively's sake, my email address is mevins@me.com.

That post provides me a perfect excuse to summarize the goals of the XG gaming project.

Of course, I've written such articles before; "The Whole Elephant", below, gives a bird's-eye view of the qualities I'm looking for in an MMORPG. That's the long-term goal of the XG software project (http://code.google.com/p/explorersguild/). This time, though, let's do it a different way. Let's talk about the nitty-gritty of game features in my ideal MMORPG.

I'd like to see a very large world with a lot of variety in terrain, flora, and fauna. I don't want it to be chopped up into zones by level. In fact, I'd be happier if there were no levels at all. Let the level of danger be determined by what sort of creature my character is and what sort of creature he faces. I've never been happy with the concept of a level one bear. Bears are big, tough, and dangerous. If the time comes that I don't have to worry about bears, it's because I've become pretty darn big, tough, and dangerous myself. Now the bears can worry about me. I'll worry when I run into something that bears are scared of.

Don't just make the world big; design it to grow. Design it with a lot of space for expansion, and keep the new things coming all the time. Let us find them. Exploration is wonderful. Watch what we like to do and where we like to go, and add places for us to find. XG has plans for worlds that can do a lot of that automatically, with some human help, but that's a topic for another time. The main thing is, make worlds that we can help to create by exploring them, and don't make us wait a dog's age for expansions; design the world to grow a little every month, or every week.

I'd like to see mobs that seem to have their own lives, rather than just being spat out arbitrarily by a spawn point. Darkfall, even with all its problems, had the right idea here. That gang of goblins should be in those ruins for some reason. They should have an agenda. If you kill a bunch of them, they shouldn't just respawn; make them run away and look for reinforcements. Maybe they don't come back until you're gone, and then they fortify the place so that it won't be so easy to drive them away next time. Or maybe they give up and move to a less dangerous hideout, and try to sneak whatever it was they were interested in out of those ruins. Point is, give mobs a reason to do what they do. Doing so will give us reasons to do something with them. If the mobs are interesting enough, we don't need as many quests.

I'd like to lose levels altogether. Let me build a character with something like the freedom that Champions gives us--only moreso. Give us skill curves that model actual skill--skilling up is easy to start and gets harder; stop practicing and your skill decreases--and adjust the curves until you reach the point of maximum fun. Give us a rich enough library of skills that we'll have tradeoffs to make. Then don't stick obstacles in the way of acquiring or changing them, unless they contribute positively to our immersion in the game. Losing skill in an ability because I don't use it enough makes sense. Being unable to learn something new because I don't have enough gold to buy a respec doesn't; it's just an arbitrary barrier.

I want to lose classes, too, at least the normal kind of classes. What I want to see instead is a robust system of skills, with progression trees that depend on skilling up and on combining skills. Skilling up tends to translate directly into macroing; Lord of the Rings Online counters this effect with their Deeds system by capping how much you can advance a Deed per day. That works fine; it not only serves a practical purpose, it also serves as a reasonable model of the human limits on skill improvement that athletes actually run into. There's only so much improvement you can make in a short time, no matter how much you want to work at something.

But I don't want to lose classes completely. Instead, I want to see a set of tools that groups of players can use to build their own classes by assembling abilities into skill trees. Force them to make tradeoffs with their skill trees, sure. Make them earn the privilege of creating classes, sure. But give the players the ability to create factions with unique classes.

That approach represents a larger goal I have for this kind of game. I want MMORPGs in which the world is alive and changing, and where what the players do helps to create the world. Not only should player-created factions be able to come up with new and unique classes, they should also be able to earn the right to create quests and rewards. They should be able to look forward to the factions they create becoming part of the game world, part of the story that all other players see.

I want players to have the opportunity to be world-builders, too. When I play conventional games, I'm always frustrated that I can't add content to the world. I want to take a group of Gnomes into Ironforge, set up a shop, hang a sign, and do business. I want to get together with my Hobbit friends and build a brand new watchtower north of Oatbarton, to keep an eye on those goblins and wolves. I want to found a new order of Disciples of Khaine and build a bone tower where they practice their bloody rites. I want players to be able to create new buildings, new towns, new factions, new storylines, and have all of those things available to all players.

I want one world. I don't mean there should only ever be one, true MMORPG world. I mean I want each world to be like EVE Online's world. If you play EVE, you don't need to know what realm or shard your friends are playing on; there's just one. I want a game where each imaginary world gets exactly one copy, like EVE. It's not only so that players don't have to worry about which shard their friends are on; it's so that the game world can have a timeline and a history. It's so that, when you create a character or a faction or a building or a village or a kingdom, your creation will be encountered by the whole world of players. It's so that the shared experience is really shared; so that we're all talking about the same thing when we mention the Battle of Foobar, or the Fall of the Ogre of the West March.

I want to see a system where a group of players can take over control of a group of NPCs, and play them. That might mean a small player group getting together for a short time to make the goblins in the ruins into something a little more fun. Or it might mean a more ambitious group of players taking over some band of ruffians and turning them into a full-fledged faction. Do we all know about Shadowclan? The guild of players who decided to play orcs in Ultima Online to provide a smarter and more entertaining enemy than the mobs, and how in the process they created the concept of monster RP? I hope we as players don't forget that. I hope we as game developers can make room for a lot more of it to happen in our games in the future. I want players like Shadowclan to come into a game and say, "hey! Those NPCs look fun; let's take them over." Or where they can say, "Geez, you know, that road there looks like a good place for a bunch of bloodthirsty goblins. Let get on that." Or even, "wow, you know, this city would be a lot more interesting if it had a big old Thieves' Guild in a big, spooky, labyrinthine guild hall. Let's get started on that right away."

I want to play an MMORPG where the world is a canvas we can use to create things. That means it needs history; it needs a world that the players can change, and a world that stays changed after the players change it. It means it needs ways to build things--buildings, towns, factions, classes--and have them last and be seen by other players.

It also means that it needs destroyers. It is a law of nature that if you hand out paintbrushes, someone's going to paint something rude on the nearest wall. That's where the destroyers come in: beings of godlike power (that is, operations staff) that go around obliterating things and taking power away from people. It would be nice if we didn't need such a thing, but of course we will.

I want to see a game where we don't all work our way through the same sequence of the same quests, and we don't all grind to get the same items. If I kill Mobnorg the ogre, I want him to be dead. I don't want five other people to come along in the next fifteen minutes and kill Mobnorg the ogre. Who cares if I killed Mobnorg the ogre, when four hundred people already did it today before me? I want to see a world that can accommodate unique events. I want characters whose stories are different from every other character's.

While we're at it, let's lose the epic magical items that roll off the assembly line. I mean, how epic and magical is an item really when there are five thousand copies of it? Besides which, surely it should be more important that a fencer is a master of the art of the sword, than whether his sword is green or purple. There's a place for magical items, but in order for them to be magical, they need to be rare and unique. "Unique" means there is exactly one. If there are two Glowing Purple Maces of the Furious Hedgehog, then it's not unique. If we want magical items, let's take a page from Lord of the Rings Online and make them "legendary", unique, and customized to the particular character, so that only you and perhaps your heir ever have your Bright Orange Spatula of the Titans.

I want to see a good, workable version of permanent death. Why? Mortal characters should age and die. If they don't, we can't really have history and stories that we participate in and create. Permadeath doesn't have to happen often, and it doesn't have to be something that blindside new players so that they shun the game. But when you choose to roll a *mortal* character, it should be clear from the get-go that your character will eventually shuffle off its *mortal* coil, from old age, if nothing else. That way, the game world can have generations of characters with histories. There can be statues of the fallen heroes, and those can be players, instead of purely fictional characters. Real people can visit a statute and remember who it was, and what the world was like before that battle. There's no story without history, and there's no history in a world where everyone and everything is reborn a hundred times a day.

Along with permanent death, I want to see mortals, immortals, and transitions that can take you between their worlds. When a mortal dies, I want the option to keep playing--not as the same character magically sprung back to life, but as an undead, or a ghost. I want to see apotheosis, in which mortals may become (minor) gods or demons. I want to see incarnation, in which immortals can be reborn as mortals. I want to see play options for immortals that reflect their legendary natures. For example, if you are a godling, you should be able to gain power by attracting worshippers. You should be limited by the influence of competing gods, and by the damping effect of unbelief. If you are an undead, you should have unearthly powers, sure, but you should also suffer weaknesses inherent in your magical nature--banes, wards, holy symbols: these things should weaken and overcome you, and you should have to work around them.

I'd like to see an MMORPG game world that can simultaneously accommodate several playstyles. I'd like to see a world where you can go adventuring and come across a battlefield where two grand armies (led by players) are contending to decide the fates of kingdoms; where secluded enclaves of wizards and engineers work furiously behind closed gates to launch RTS-style assaults n other enclaves; where you can advance the the highest levels of wealth, prestige, and skill by exploring the uncharted boundaries of the map and bringing back wonders no one has seen before.

I want an MMORPG that is as much a playground for authors as it is for audiences.

Sep. 21st, 2009

Doing It Wrong

Why does it cost a lot to rebuild a character in an MMORPG?

Most MMORPGs provide various ways to customize characters. The details and terminology vary from one game to the next. The 800-pound gorilla, World of Warcraft, requires that you start by choosing a race and class, and then it grants "Talent" points as you level up. You can spend these talent points to modify the details of how your character plays. In Warhammer Online, the customizations are called "Masteries," rather than talents, and, like WoW, WAR gives you a point to spend on them each time you gain a level.

Like most everything it does, Lord of the Rings Online provides a scheme that is generally similar to those of the other games, but with its own twist on the idea, and, just maybe, more careful thought put into making it fun and rewarding. Instead of granting you a point every time you gain a level, LOTRO offers "Deeds", which you gain by doing any of a wide variety of different things. Many Deeds grant Traits, which serve the same purpose as Talents or Masteries, but which are much more interesting to accrue, because, first of all, the Deed is generally related in some way to the particular activity you had to pursue in order to get it (for example, Staff Mastery is gained by using a staff a lot in a particular way), and secondly, because Deeds are completely unrelated to experience and leveling, so the Deeds system feels like a completely different and independent way to customize and advance your character.

The particular set of customizations that a character has is generally called a "build". One thing that all MMORPGs seem to do the same way is, they charge game currency to change builds. Why is that?

To take the most prominent example, in World of Warcraft, you can at any time visit a trainer and reset your build. All your spent talent points are refunded, and all customizations you've made by spending them are removed. The first time you do this, it costs one gold. That's a lot of money to a brand new character, but very little to anyone above level thirty. The second time you do it, it costs five gold. The third time, it costs ten gold. Each time you do it, the price goes up by five gold, until it reaches fifty, which is the cap.

What is that fee for?

For current players of WoW, a cap of fifty gold is trivial. It's maybe enough to keep endgame players from rebuilding their characters several times a day, but I have to wonder why anyone would care if they did.

In some games, changing builds is harder and more expensive. One current controversy in the new superhero game, Champions Online--a very good game in many respects--is over the cost of a "retcon", which is what Champions calls the process of rebuilding a character's customizations. The question is more important in Champions than in many other games, because of some of the game's unique features, and it's a thornier issue, too, because the current cost of retcons is quite high.

Why is it so high?

In fact, why do MMORPGs put any barrier at all in the way of rebuilding character customizations?

Cryptic has said that retcons "can't be" very cheap, but they haven't said why. So what's the reason? What bad thing would happen to an MMORPG if players could painlessly rebuild their customizations any time they wanted?

One argument is that, in the presence of easy rebuilds, diversity would suffer, as players would quickly gravitate to a handful of popular builds. The first thing that occurs to me is that, if this is a problem, then why haven't we seen hundreds of Champions characters that all look alike? It's easy as pie to completely change your appearance several times a day, and yet there are no hordes of identically-uniformed characters plaguing the game. Why not?

How are builds any different from appearance? Well, builds can strongly influence the effectiveness of a character in play; that's a definite difference. Cosmetic appearances don't have any such effect. Okay, so who cares how effective a character is, and why?

Well, obviously, the player who owns the character cares. The more effective a build is, the easier and more fun it will be to play the game. This fact could indeed result in a loss of diversity, as many players converge on the small number of particularly effective builds--but only if some builds are notably more effective than others. That's not an unusual circumstance--in fact, it's nearly inevitable--but it isn't intentional. It happens accidentally. It's a failure of the game-design process, and a wise game designer wants to know about it as quickly as possible, so that he can fix it. Anything that makes it harder to tell which builds are more effective, and which are less, works against the game designers, making it harder for them to do their jobs. It also works against the players. They of course want to know which options are most effective, so that they don't waste a lot of game time building up a character that is crippled, ineffective, and unfun to play.

If the player base converges on just a few builds, it's because something is wrong with the other builds, and they need to be fixed. The quicker and easier it is to find that out, the better. Making it harder and slower to collect that information, which is what raising the costs of rebuilds does, benefits no one. Well, okay, it benefits players who've found good builds and don't want anyone else to find them. I have to say, I don't worry a lot about keeping those guys happy. If you want to play with a stacked deck, tell it to somebody else.

Some players claim that if other players rebuild their characters all the time, that their experience of the game will suffer, that it will break immersion, and so on. This one is pretty hard to believe, if you've spent much time in MMORPGs. First of all, nobody is anywhere near as interested in anyone else's characters as they are in their own, and the odds that anyone would even notice if you completely remade your character every single day for a month are pretty low--excepting, of course, the friends that you regularly play with. And if we're talking about the friends that you regularly play with, well, you and your friends are perfectly capable of deciding for yourselves how much fluctuation you're comfortable with. Heck, if you want standards, form a guild, and put the standards in the rules.

Secondly, it's pretty darn hard to believe that, in games full of people with names like "Invincable" and "Dethmuthakilla", and "Pwnface", that your immersion is going to fall apart because someone switched from casting fire spells to casting ice spells and back again. I just don't buy it, sorry.

Another argument is that if rebuilds were cheap and easy, then replayability would suffer. Instead of starting another character in order to try out a different experience, players would just rebuild an existing, high-level character and try out the game experience that way. Advocates of this view are not thinking things through. A game is replayable if people want to play through it again. You don't rebuild a high-level character because you want to play through the game again. If you want to play through the game again, you start a new character. You rebuild because you don't want to play through it again; you just want to try a different set of high-level options. They're two different things. If rebuilds are expensive, that just means that one of those two things is less available, less appealing. The more expensive rebuilds are, the less appealing it is to try alternatives. You can't reliably influence more people to replay by raising the cost of rebuilds, any more than you can reliably get more people to buy Fords by raising the cost of champagne. The two goods satisfy different demands.

Even if that strategy did work, would it be a good idea? If you're trying to make your game more replayable by increasing that pain-in-the-ass factor of the endgame, rather than by making the early game more fun and varied, then you're doing it wrong. You haven't made your game more replayable. You've added a hazing ritual, and players just have to decide whether they want to put up with it.

And the cost of rebuilds doesn't affect revenues one way or another...unless it exasperates you enough that you decide to leave the game. If you're interested in playing, you'll play--one character at a time, like always, whether you can easily rebuild that character or not. Cheap, easy rebuilds do not harm replayability. Poor replayability harms replayability. Making rebuilds more of a pain does not somehow magically make it more fun to play the same content again.

What possible reason does that leave?

There's the argument that people in real life, and characters in stories, don't drastically change their natures from day to day, so characters in an MMORPG shouldn't, either. This one is totally ridiculous, but I've seen someone seriously proposing it as an argument for high rebuild costs. I mean, come on. We're playing pretend with made up characters. Who cares how many drafts it takes a storyteller to get his story right? Who cares what the names and personalities and backstories of the characters were in the first draft, or in the tenth? Isn't the quality of the final product what we should be interested in?

Maybe it takes you a few tries, or a lot of tries, to get your character to work the way you want. Why is that anyone else's concern? If your back-story, or your appearance, or your powers are different today from what they were yesterday, who gives a rat's ass? Shouldn't we be more interested in how well you play today? Shouldn't we be more interested in how much fun you are to play with? In how much fun you're having, and how much fun we're having? Gimme a break with this one.

Are there any good reasons to impose a high cost on rebuilding character customizations?

Aug. 13th, 2009


There are a few things I've written that I see cited or quote repeatedly. The oldest is a quip about omnipotent beings that I wrote eighteen years ago in a USENET discussion:

"Perhaps this morning there were only three Euclidean solids, but god changed
its mind retroactively at lunchtime, remaking the whole history of the
universe. That's the way it is with omnipotent beings."

More recent is an explanation I once gave of the concept of "natural law" in politics. There are a couple of basic theories of law; one of them is called "natural law". Because that phrase is also used to mean "law of nature", people sometimes confuse the two concepts, often, I think, because they're not aware that the legal one exists, or are not aware of what it is. This was my attempt to succinctly summarize it in one discussion:

"People will naturally and predictably find some methods for resolving [...] conflicts more congenial than others. There are some classes of conflict for which people will naturally and predictably find certain kinds of resolutions more congenial than others. The procedures people find more congenial will also produce the resolutions people find more congenial. And the procedures and resolutions that people find more congenial will tend to resemble each other across times and cultures."

Two others are more technical articles that I wrote early in the 1990s about programming. They were published in a magazine aimed at Mac developers, called Frameworks (now archived at MacTech).

"Objects Without Classes" was republished by the ACM's Computer magazine in Volume 27 , Issue 3 (March 1994) . It's a basic description of prototype-based object systems, such as those found in languages like Self and Javascript. Languages with prototype-based object systems probably seemed novel to more people then than they do now. Lots of people use Javascript now, but at that time you might never have heard of prototype-based object systems unless you were at Sun working on Self, or in the AI business, working with frame languages, or at Apple working on Newton or SK8.

The topic of this post is the ideas presented in "Protocols," originally published in the March, 1994 issue of Frameworks, and archived here. It's an article about how representation, behavior, and taxonomy are distinct concepts that can be handled separately, despite the fact that object-oriented languages tend to confuse them.

Those points need clarifying, I know.

By "representation," I mean how we concretely lay out data. As an example, you can represent an array as a contiguous sequence of memory locations, or as a table of indexes paired with pointers, or as a tree of cells in which bit patterns are mapped to subtrees, or, of course, in many other ways. There are circumstances in which each of the different approaches might be advantageous. These are different representations of a common abstract concept that I have here called an "array".

By "behavior," I mean the set of operations that is defined on any given set of values. Taking the abovementioned abstract concept of an "array" as an example, you can use any of the mentioned representations to support a common behavior. You can straightforwardly implement functions that fetch an element of an array by index, that iterate over the members of an array, that write a new value to the array at some specified index. The implementation and performance details will differ, of course, because the representation differs, but the behavior—that is, the API and what it accomplishes—is the same.

By "taxonomy," I mean the relationship between one defined set of values and another. Types are categories of values. They are (possibly unbounded) collections of data objects. Types naturally have subtypes; a collection of values has a natural relation to a smaller collection of some of the same values: the smaller set is a subtype of the larger one. It's natural to think of obtaining the smaller set by starting with the larger one and adding restrictions that filter out some values. You can think of taxonomies of types, such as the class hierarchies in languages like Smalltalk and Java, as sets of values that begin with a very inclusive type, like Java's Object, and develops more and more refined types by adding more and more exclusive restrictions. Object is all the values (I know; it's not really all of them.) Number excludes all those objects that aren't representations of magnitudes. Integer excludes all that are not representations of whole numbers. And so on.

These three concepts, representation, behavior, and taxonomy, are quite distinct, but most object-oriented languages conflate at least two of them. Some combine all three. Smalltalk classes, for example, are representation, behavior, and taxonomy all rolled into one. The representation is in the instance variables; the behavior is in the methods; the taxonomy is in the subclass/superclass relations. THe trouble with that approach is that when you ask for one of the three, you get the other two whether you want them or not. For example, if you create a new subclass, you inherit representation, behavior, and taxonomy, even if all you wanted was behavior. If you create some subclass because you want a certain taxonomic relation, you also get a representation and a behavior that may or may not be what you need.

The claim of the "Prototypes" article was that you can treat these three concepts separately, and by doing so develop a discipline that makes object-oriented design and implementation feasible in any language. Furthermore, by doing so, you can develop APIs that are language independent. To take the "array" example, what makes it an "array" is that you can get and set elements by index, and iterate over them in index order. That's behavior. If representation, behavior, and taxonomy are properly separated, you can represent "arrays" any way you like, and you can arrange for them to be subtypes of any type you like, and they'll still be arrays, because they support "array" behavior. Representation and taxonomy can be treated independently as well, but not if your tools insist on conflating them.

I still think that's true, and useful, but I was always interested in codifying the ideas of that article more concretely. I was interested in working with an object system designed around clearly distinguishing representation, behavior, and taxonomy.

One language that pretty much nails it is Haskell. In Haskell, representation is the province of datatypes. Behavior is handled by functions. Taxonomy belongs to typeclasses.

For those unfamiliar with Haskell, the terminology is probably confusing. Without some experience with Haskell, it's probably not at all clear how and why "datatypes" and "typeclasses" are different. Briefly, a Haskell datatype is a named description of an arrangement of data. A typeclass is a named description of a set of operations. You can make a datatype into a member of a typeclass by defining functions that implement the operations specified by the typeclass.

This is pretty close to what I was talking about in "Protocols;" it's probably closer than anything else I've seen to language support for those concepts. The next closest thing, I think, is the style of object system exemplified by CLOS (the Common Lisp Object System). In CLOS, classes describe representation and taxonomy. Generic functions and methods describe behavior. It's easy and convenient to describe behavior in CLOS independently of representation or taxonomy. CLOS does still conflate representation with taxonomy, though it provides facilities for modifying itself that enable you to work around that conflation if you really want to.

In February of 2009 I had been using the new Lisp dialect, Clojure, for about five months on some projects, and was generally pretty happy with it. One area I wasn't happy with, though, was Clojure's approach to types and polymorphic functions. Clojure doesn't really have a type system as such. It has a small number of well-chosen and well-designed types, and it can transparently use Java's types (Clojure's main implementation is built on the JVM). For most uses, these two facilities are more than sufficient, and they work very well. They fell a little short of what I wanted for some of my work, though. One of my projects requires ways to specify a large number of structured data elements with taxonomic relations. The set of elements is open, and expected to grow over time, so I need a convenient way to add descriptions of new ones, and ways to ensure that relevant APIs are defined to work in the proper way over all the values currently described, and over all those that are yet to be described.

Clojure gives me a lot of what I want to support this scenario, but not all of it. Clojure's maps provide a great way to describe structured data, but not a great way to place restrictions on it. I can say that Foo has fields A, B, and C, but there's no convenient way to say that A is an automobile and B is a hypotenuse; nor is there a convenient way to say that a Foo has A,B, and C, and nothing else.

I was even more dissatisfied with the facilities that Clojure provided for defining taxonomy. Clojure's documentation, and its justifiably enthusiastic users, make much of the fact that Clojure's derive function and its polymorphic MultiFns make it possible to define arbitrary taxonomies. That's true for small values of "arbitrary". You can easily construct any taxonomy, as long as it's a taxonomy that Clojure was designed to easily construct. Mine wasn't.

The details of the problems don't matter. My main point is that I ran into a couple of insoluble problems with Clojure's MultiFns, derive, and hierarchies, and, after some discussion, satisfied myself that the Clojure community regarded those obstacles as features rather than bugs. So I did what any sensible Lisp hacker does in a situation like that: I wrote my own object system.

It began as an existence proof of an alternative way of handling polymorphic dispatch. There was a little bit of interest in it from a couple of people, but for the most part the Clojure community reacted with a shrug. On the whole, they're happy with Clojure's approach. More power to them.

As I tweaked a few things to respond to casually-mentioned hypothetical objections, I started to like what I had. I was testing it by reimplementing important parts of some production code I was working on. I realized pretty early that the subsystem I was builiding bore more than a passing resemblance to the ideas in "Protocols". I refactored it a few times. I tried a couple of different implementation strategies. It got faster, simpler, and more appealing (to me, that is). Somewhere along the way, I started calling it Categories.

I have a couple of implementations of it in Clojure now, with somewhat different APIs. More recently, I ported it to Scheme, so that I can try it out in a different application context. The API mutated a little more. I'm still refactoring things to try to make the surface API simpler and easier to understand.

Categories represents the concepts from "Protocols" pretty straightforwardly. The basic concepts are:

Types: descriptions of how data are laid out.

Functions: operations that accept zero or more values as parameters, and that compute and return zero or more values as results.

Domains: descriptions of relations among types.

Functions are polymorphic, and are defined in terms of domains. In other words, a functions looks at its arguments at runtime and decides which actual code to run based on what it sees. This is like any other object-oriented language, as far as it goes. The distinguishing characteristic of the system is how a function choses a method.

When you construct a function, one parameter to its constructor is a domain. A domain contains a catalog of types, and a set of rules (represented as functions) that describe how the types are related. Domains can tell you things like whether a type is a member of the domain, whether one type is a subtype of another, and whether a method can be applied to a particular set of argument values.

Importantly, you can have as many domains as you want, and each one can work differently. How you represent the relations among types is entirely up to you. This means that you can pretty easily implement any kind of dispatching you want. Smaltalk-style, Java-style, CLOS-style, predicate dispatch—it's all good. The Categories subsystem defines a default domain that implements a dispatch scheme very similar to those of CLOS and Dylan (because I like those schemes and am comfortable with them), but the default is just one domain. If you want something different, it's easy enough to build it. Just to make sure, I wrote implementations of Clojure's dispatch and of a predicate-dispatch scheme. They were easy to do.

When I was first discussing these ideas, someone mentioned some concern that it wouldn't be possible to implement such a system efficiently, or to optimize it. In practice, it hasn't been a problem. There is a tradeoff between exposing the API you need to implement a domain efficiently, and making the domain API understandable, and that's been a major issue I'm dealing with in rewrites of Categories. It seems clear at this point, though, that efficient implementations are doable. The trick is setting things up so that you can write domains that are efficient without exposing a confusing array of knobs and switches. I'm still working on striking the right balance.

You can't get a current working version of Categories right now; I still have its guts out on the table so I can tinker with them. Some of its early predecessors are readily available if you really want them, but if you read this and are really interested in Categories, I'd counsel patience. I'm folding it into some product code right now, and making corrections and improvements as that process reveals the need. It's my intention to nail it down and ship a product that uses it, then port that version back over to Clojure to support some other work I'm doing in that language. Once I reach that point, if you ask me for it, I'll give it to you.

Why would you want it? I dunno; you might not. I sure did, though, enough to build it. Soon I'll see if it was a waste of time, or a great new tool for my toolbox.

Aug. 12th, 2009

Sometimes you want to play a game

As the XG group has been working on the organization of our latest project, it occurred to me that we're adopting a structure that has served the XG group well through several games. That got me to thinking about guilds and teams in MMORPGs generally, and about common characteristics of the most successful guilds.

[XG's latest project is "Noss Rhiw", a team-oriented roleplaying and questing guild in Lord of the Rings Online, focused on playing through the Epic storyline. If you'd like to know more, contact me here, or speak to Heirobhan or Rhovellon in-game on LOTRO's Landroval server.]

I'd consider a lot of guilds unsuccessful, even when they last for a long time. I've seen some guilds grow large by recruiting anybody who comes along, and some of these have lasted quite a long time, but I don't consider them successful guilds. To be a successful guild, I think you need to involve the members in some common goals, to create a sense of shared community. Without that, a guild, even if it's large and long-lived, isn't successful; it's just a random bunch of people who happen to have the same guild tag under their names. Such a guild is useful for someone who wants to play alone, and who doesn't want to receive guild invitations, but it serves little purpose beyond that.

A successful guild is actually fun to belong to. It's a group effort designed to create fun for its members. There are a few ways to go about this, and a range of sizes from huge to tiny.

At the huge end, you have groups like Shadow Company, a large gaming guild with active communities in five games presently, and in four more games previously. Shadow Company has been in existence and growing since 2004. (Shadow Company's home)

Moving toward smaller groups, the venerable Shadowclan is still a fairly large group, with hundreds of members in some games, and its own approach to organization and recruitment. Shadowclan may be the most famous of these groups; it's been in existence since 1997, and has even been mentioned by name in the collateral materials of some commercially-successful games. (Shadowclan home)

At the small end of the scale, the XG groups have ranged in size from about five people to about fifteen. Once again, XG has its own unique approach to organization and recruitment. The oldest of the examples, XG has been active since 1975. (XG home)

I'd regard all of these examples as successful guilds. Each one has survived for an extended period, and has continued to conceive and create interesting new projects that attract new players. Each one fosters a strong sense of community, and enthusiasm for the group's projects. With the range of gaming options increasing all the time, members of these groups nevertheless continue to chose participation in the group's activities over other options. All three groups actively watch development of new games, and discuss options for the creation of new guild branches and projects.

Besides their continued successes, are there other things these groups have in common?

All three groups have relatively rigid recruitment and membership policies.

Shadowclan will sign up anyone who wants to join, but they first require you to read their rules and agree, in their peculiar dialect, to abide by them. You must then pass a series of tests administered by more senior members before being admitted to full membership.

Shadow Company also accepts most people who want to join, but with a slightly more involved recruitment process. You must apply to their web site and receive approval to contact a recruitment officer in-game. Once invited, you are assigned a commanding officer to whom you must report, and a training officer who will advise and guide you toward acceptance as a full member. Shadow Company uses their psuedo-military organization to advantage. You are expected to learn and obey a comprehensive set of regulations in order to become, and remain, a full member.

XG groups have some relatively rigid policies as well. Generally, the process for joining an XG project is to contact one of the leading members out-of-game to talk about your goals. If you seem like a good fit, you will be referred to someone in-game. You are then expected to schedule questing time with at least two different members. After those members have each had at least two sessions with you, they report to the general membership on their impressions, and if you receive no thumbs-downs, you will be invited aboard as a full member.

All three groups have some fairly inflexible rules for members.

Shadowclan specifies what to wear, what to carry, what and when to ride, what to do with your loot, and even how to speak. Violate any of these rules and you'll be warned, then tossed out.

Shadow Company has a comprehensive military-style set of regulations. Again, violation of regulations is grounds for dismissal from the guild.

XG rules tell you when to play, what level to be, and what group to play with. Depending on the specific project, they impose additional rules; for example, what race to play, what language to speak, what group role to fill, what crafting professions to take, and so on.

All three guilds have small leadership groups that have guided them over an extended period. In Shadowclan, it's "the Helm"--the shadowy set of individuals who are responsible for administration of Shadowclan as a whole. In Shadow Company, it's Commander Bugbig and his close associates. In XG, it's mainly me, plus whichever of my longtime friends is most active in planning at a given moment.

These examples of successful guilds are highly structured organizations with rigid and detailed rules and policies, administered by small groups of dedicated administrators.

Isn't this a little counter-intuitive? We're talking about gaming guilds here. These are groups of people getting together to have fun with computer games, to play pretend, to indulge in simulated violence and mayhem. Now I'm claiming that the most successful ones are rigid, exclusionary hierarchies run by dictatorial juntas?

I think this result is very interesting, and I probably wouldn't have predicted it. Looking back on my experiences in games, I think that the success of groups like these reveal something interesting bout group dynamics, and about how people assess opportunities, costs, and benefits.

MMORPGs are essentially democratic and egalitarian. Everyone playing has exactly the same opportunities to accrue exactly the same rewards from exactly the same investments. We all come in at level one, choosing from exactly the same set of starting options. Guilds all have the same pools of characters to draw from, and the same sets of options for play. Yet some guilds succeed, and others fail. Some grow rapidly before self-destructing in soap operas. Some fall apart because of member or officer misbehavior. Some become nothing more than warehouses for characters; they continue to exist with full rosters, but the members rarely have anything to do with each other--they are guilds in name only.

Why do some of the most vital and active gaming groups exhibit these common traits of regimentation and restrictive rules? Why do they have in common dictatorial leaderships? Nothing about the games requires these characteristics. Guilds can be organized any way the membership likes. They can have any rules they like, or no rules at all. It's up to the guilds themselves.

One insight is something I said in-game once to someone who was going around inviting all and sundry to join his guild. I declined the invitation. When he wanted to know why, I said "If you're inviting everyone, without knowing anything about them, then you have no standards. I don't want to belong to a group with no standards."

I don't want to belong to a group with no standards because not everyone you meet is nice to play with. If the group has no standards, then it's going to have jerks and idiots in it, because there are no standards to keep them out. Worse, there are other groups that do have standards, which means they're getting the people who aren't jerks and idiots, and the groups with no standards recruit from what's left.

Another point is that guilds that impose arbitrary rules are creating artificial barriers to entry, and that has the effect of improving the group's morale. You have to jump through hoops to join. Sure, they might be arbitrary hoops; they might not have a lot of meaning. They are obstacles, nevertheless. Once you get in, you're with other people who all had to jump through the same hoops. They couldn't just wander in; they had to want to be there.

Furthermore, you can easily find out what the membership requirements are before you start the hoop-jumping process. You have a reasonable chance to consider whether earning membership is worth your effort. If the group does things that sound fun and interesting to you, then there's a fair chance that membership is worth the effort. You jump through the hoops because you think you might like the group. So did everyone else. It's not just a random collection of people. It's a collection of people who put in some effort to join a group that's interested in things they're interested in.

Once accepted into these highly-structured guilds, you find that the rigid rules are not as oppressive as they sounded. For one thing, everyone else has to obey them, too, so it's not like you're being singled out and picked on. For another thing, it turns out that you can see that the rukes have been chosen to make the game more fun in some way. The level requirements and team assignments in XG guarantee that every time you log in to a scheduled session, there is a full team of players, at your level, ready to go. The military ranks in Shadow Company provide a clear chain of command that eliminates much of the time-wasting so common in group efforts in MMORPGs. Shadowclan's rules about what you're allowed to wear make it easy to see where your group is and which ones are the leaders, which is invaluable in group combat, especially in PVP. It turns out that all those pesky rules are features, not bugs!

What about the juntas that rule these guilds? They're gaming enthusiasts who have discovered that group efforts can be enough fun to justify the administrative drudgery and abuse that come with running a gaming group. They love playing with good groups so much that they're willing to put up with the boring work and the occasional insults and drama queens, so that they and others will have the opportunity to play good games with good groups.

If you just drift with the current, often as not you end up snagged somewhere. If you want a fun trip down the river, you probably need to do some actual rowing and steering.

Successful guilds don't just happen by accident. Someone has to set out to make them happen, and they have to keep at it. And good groups don't very often just fall together by accident. A good group needs members with common goals, who set out deliberately to achieve them. Successful guilds are structured because they're trying to have fun on purpose, not just wandering around hoping that fun will run up and find them somehow.

Some players disparage the highly structured guilds, complaining that they don't like the arbitrary rules, or the that restrictions interfere with their freedom of choice, or that they don't like having a boss in a game. I think these complaints are misguided. There's nothing forcing anyone to join any of these guilds. They're no threat to anyone's freedom; if you don't like them, don't join one.

Consider this: football's a fun game for a lot of people. Basketball's another. Then there's soccer, baseball, and so on. But all of those games have rigid rules, organized teams, chains of command. Maybe you'd rather wander out onto a field with a couple dozen other people and just knock some balls around and see what happens. There's nothing wrong with that. It might be pretty fun. It won't be a game of soccer or football, though, not unless someone organizes one. And that means rules, structure, and leadership.

Sometimes kicking a ball around the field is good enough. But sometimes you want to play a game.

Jul. 17th, 2009


I’m coming to the close of a contract, and I’m spending part time doing what is natural in that situation: looking for new contracts, and preparing some of my personal projects for release as products.

The first product scheduled for delivery is a Cocoa application for Mac OS X. Written in Scheme. This, of course, is heresy.

Cocoa applications are supposed to be written in Objective-C. (Well, to be fair, about a third of the code in this one is Objective-C. The rest I wrote in Scheme.)

Why would I do a crazy thing like that?

Well, I like Lisp. I’m a Lisp programmer for a reason, that reason being that Lisp is the clear winner on the Joy-Per-Unit-Effort scale. I’m not really partisan in my choice of Lisps. I like Scheme and Common Lisp and Clojure all just fine. If I ever had a favorite Lisp, it was Dylan, but it’s been a while since Dylan really felt like a Lisp.

So I was planning to release this product, and I was looking at options for packaging it. I could certainly have just written it in Objective-C. I’ve been using Objective-C and Apple’s development tools (and NeXT’s development tools before that) for nigh onto twenty years now. It’s a comfortable and familiar ecosystem. But I figured I’d give Lisp a fair shot before settling on the conventional choice because, you know, I like Lisp.

Clojure? Nah; the quick route there is a Swing app. The Java Virtual Machine has many advantages, but delivering great Cocoa apps is not one of them. Sure, if you spend enough time and effort, you can achieve a sort of approximation of a Cocoa UI, but it’s not a fun process. Less Joy Is Bad.

Common Lisp? Maybe. I love Clozure Common Lisp, and it has a great Cocoa interface. Only trouble is, universal apps are off the table, for now. Lispworks? Another good possibility. But my license is getting a little long in the tooth. If I was going to build a product on it, I should really get a new license, so I can have the latest version of Lispworks. And y’know? Right now, end of contract, looking for new contracts, and other revenue streams...not the best moment to spend a few thousand bucks speculatively. Prudence. One learns a modicum of prudence.

I was experimenting with a few options, and Gambit-C (http://dynamo.iro.umontreal.ca/~gambit/wiki/index.php/Main_Page) pleasantly surprised me. Gambit’s been installed on my systems since, oh, probably since 1990 at least. It’s a great Scheme system that has gotten steadily better over the years. But there’s not really any Cocoa support there, or so I thought. Go ahead, Google around; you won’t find much of anything. Except this:


It describes how this fellow set up remote debugging of a Gambit app running on the iPhone. Wait a second; Gambit on the iPhone? Um. Turns out I know something about iPhone apps. Turns out I’ve written some iPhone products (http://www.iphone4kids.net/2009/06/21/firstwords-learning-how-to-spell/). If this guy has Gambit code running on the iPhone, I’m pretty sure I can get Gambit code to run a Cocoa app, as well.

In fact, it was a piece of cake.

I had allocated a number of hours spread over a week or so to evaluate several alternative possibilities before just settling down and building the app in plain, vanilla Objective-C. After the first session with Gambit, I cancelled all the other evaluations and just wrote the app in Gambit.

It’s true that there is no Objective-C interface in Gambit. Turns out you don’t need one. Gambit has a very good foreign-function interface--a C interface really. Gambit compiles Scheme to C code, and then compiles the resulting C code to native code. One advantage of this approach, which is used by several other Schemes (Chicken and Bigloo, for example), is that you can embed C code in your Scheme, and pretty freely mix the two languages. In point of fact, because Gambit uses the platform’s C compiler, and because Apple’s C compiler handles Objective-C, you can in fact embed Objective-C code in your Scheme.

For example,

(define application-main
  (c-lambda () int
   NSApplication *app = [NSApplication sharedApplication];
   NSAutoreleasePool *pool = [[NSAutoreleasePool alloc] init];
   [NSBundle loadNibNamed: [[[NSBundle mainBundle] infoDictionary] objectForKey: @"NSMainNibFile"]
             owner: app];
   [app run];
   [pool release];
   ___result = 0;

Well, whaddya know? We don’t need no steenking Cocoa interface.

So currently the app is about two thirds Scheme code and one third Objective-C code. It works great. It launches in under half a second and loads its test database in about seventy milliseconds. UI responsiveness and scrolling are as fast as you could possibly want. I built the UI the conventional way, in Interface Builder, and it calls into the Scheme code to display and edit data. The delivered app bundle is 7MB and it’s a universal application that runs on Tiger and later.

Turns out Gambit is a great platform for Cocoa app development. Who knew?

Previous 10 | Next 10