Links: One element of ProWiki that has proved a weakness is the way that we have only a single datatype: the Property. In fact, I believe we need two distinct datatypes: Properties and Links. A Link is specifically a field that refers to another page, by name. (Or, as described below, by Object ID? If Links are always Object IDs, then renaming of objects becomes trivial.) By separating Links out as a distinct datatype, we can impose more rigor on them -- we can do things like produce summaries of dead links and the like.
Enumerations: Related to that, we may want to have a distinction between "fulltext" properties and "enumeration" properties. The concept here is that some properties are really enumerations, with a small number of possible values. We could define those as a separate datatype, and allow them to be specified in the inheritance tree. For example, in TR III, the "Character" archetype could define "World" as an enumeration with a specified default value. This would put it into the Enumeration table instead of the Property table. Individual pages could render that as a drop-down list instead of simple text, reducing the likelihood of error. In order to get this, we might want archetypes to define the full list of possible values for that enumeration; that should, however, be extensible by child objects.
(Subtle note here: extensible enumerations are problematic from an OO point of view, because they are child objects expanding the semantics of the class rather than restricting them. So if we allow child extension of enumerations, think through these implications carefully: it implies that they aren't really classes, and you can't depend on the values based on the supertype. Maybe we need a concept of sub-values? That is, the superclass would define coarse-grained values, but sub-classes can add finer-grained refinement of those? That's much more OO, and might be a good way to split the difference. The question is whether to go all the way, and consider the enumeration values to be parameterizable -- in which case I wind up with Scala case classes plus some UI.)
"Pages": It's important to note that I am drifting away from any concept of a single "page" in the traditional wiki sense -- there is no plain text file that represents a page. I believe that there is a concept of a plaintext page (there should probably be a well-defined property that corresponds to "the unstructured text for this page"), but a page is really composed of a mix of properties and links that are linked by name and version.
We need to keep inheritance in mind as this gets designed. There needs to be an inheritance table, which defines which pages inherit from which parents. There probably should be a root (probably named "Page"), which is the master parent of everything and cannot be reparented, but which is otherwise a perfectly normal page, with modifyable display templates and all. When editing any page, we should check the inheritance paths, and make sure they have no loops -- they need to get back to Page within a reasonable number of hops.
Methods?: More a question than a statement -- if pages are really OO-style objects, can they have methods? What would it mean for them to do so? This is potentially a gigantic rathole that leads to a full FlexWiki?-style scripting language, and I'm not sure we want to get into it any time in the foreseeable future, but we should keep the possibility in mind.
Anonymous Pages: We almost certainly want a concept of "anonymous objects" -- objects that don't really have a page name, but simply an anonymous object ID. The use case for this is comment threads. It would be lovely if all objects could have strongly-threaded comment trees associated with them, but not as part of the page -- as separate data that can be included or not as desired. (Keeping in mind that we will often want to be able to insert numerous comment threads at various places inside a page -- comments are often on specific paragraphs, even specific words.)
To achieve this, and to get really first-class threading, each comment ought to be a separate object, with links between the object and the comment roots, and between parent and child comments. No one really wants to name these things, though. So it would be highly useful to have a simple concept of "Object ID" -- perhaps as simple as an incrementing 64-bit counter that's universal to the system or namespace -- and allow the UI to allocate these things as desired. So when you add a comment, it creates a new object with an allocated ID, and links that in as appropriate. Note that this also buys comment user-naming for free, since the comment objects will be associated with the user who created them.
Aliases vs. Names: In fact, on further reflection, we should go further, and say that names are not an inherent part of pages. Pages always are identified by a unique OID -- that is the permanent identifier for the page. On top of that, we have a concept of "aliases". Any number of aliases can point to a given OID, so there is no need for annoying renames and such: it's entirely legal to have multiple "names" for the same page. Default pages probably have a "title" property that shows up at the top of the page, but that's simply a plaintext string, not an inherent name. We might say that, by default, an alias is created that matches the title, but that's just a default. We definitely should also have a trivial query to show all of the Aliases pointing to a given page; that might even show up on the default page view.
Orphan and Reference Management: A consequence of going with the "name-free" architecture is that it's going to become dangerously easy to orphan pages. This is already too easy in normal wikis, and it's only going to get worse here. This feature can be added later, but we should keep it in mind. Ideally, we'd like to be able to build reference maps that show everything referenced from a given root or list of roots (which would also be lovely for data export); turning that around, we ought to be able to isolate objects that aren't referenced by any roots, and are potentially available for "garbage collection". (Exactly what that means, in the case of a history-managed system, remains to be seen.)
Note that comment threads are probably the things most likely to be orphaned. As mentioned in Querki Comment Threads, we want to be able to remove comments, which basically means orphaning those comment objects. They may make one good use case for thinking about this problem. They illustrate that garbage-collection is very closely related to checkpointing -- something is "orphaned" when no current pages refer to it, but historical objects might still do so. So we really can only garbage-collect when we are cleaning the DB, and removing "old" history (whatever that means).
Namespaces: Many better wikis have a concept of namespaces, and I think Querki should as well. Indeed, with the idea of Aliases in place, Namespaces become really easy -- they're simply a hierarchical organization of the aliases, not of the pages. It should probably be possible to associate ACLs (as mentioned below) with namespaces, such that all objects created initially within a given namespace get the desired ACL, but the ACL is associated with the object, not with the name. That's important, since we want to be able to cross-link objects across namespaces, thereby avoiding the hassles I've had in FlexWiki? when I want to have the same page available in two different namespaces.
Namespace Importing: This leads into the appropriate solution to the shared-schema problem. We should allow namespaces to link to each other. If, say, the Girl Genius namespace imports the LARP Schema namespace, that means that all names in the LARP Schema become available within GG without qualification. (Remember that the LARP "Schema" is merely a collection of named objects.) This is a very easy way to deal with schema-sharing: it means we just tweak the name-resolution process to allow indirections to the imported namespace.
Ownership and Access: Also to support threaded comments, it would be nice to allow them to be truly associated with their creating users, rather than just linked by name. The right thing to do, really, is probably for all object revisions to be associated with at least the user who last edited them, and have that be displayable. They might also have timestamps associated as a first-class column per revision. This would allow comment threads to have a completely consistent display model, rather than having the name and time as plaintext.
That being the case, it suggests a very simple per-object access model. While full ACLs may be more work than they are worth, it may be worthwhile to allow an object to be locked to its creating user, so that only that user can edit it. If we want a midpoint of fanciness, we could have a separate concept of "owner", which could be transferred, and objects can be locked to their owners. This would allow a decently flexible approach to begin with, which could be extended to full-scale Groups down the line if we care.
(Note an important corollary: User objects can never be deleted, since page revisions depend upon them. Keep that in mind.)
LATER: in retrospect, we should probably design for ACLs upfront, even if the initial implementation is fairly simple. Also, keep in mind that we should be able to specify read access, not just write access. I would love to be able to lock down my game-design wikis so that only authorized users can read them. But I want to be able to allow specific pages to be world-readable, and maybe even world-writeable: for instance, I should be able to keep the game's public homepage in the wiki, and preferably even the casting questionnaire. (In fact, the casting questionnaire is a nice simple use case for data-entry stories, which should be well-supported.)
LATER: How fine-grained a permissions scheme do we want? Consider the variety of edge cases: being able to control who can see the source code for a page, or be able to see previous versions, etc. We probably don't want to go too whole-hog and have every permission be a separate DB entry (which could get very expensive), but it's possible that we want the normal permissions to be bit flags and then have an "extra permissions" flag, that points to more permissions, defined by GUID or somesuch.
Identity: We should keep in mind the various identity-management schemes on the Web, and be prepared to incorporate them. In particular, it is likely that OpenID? is going to become a serious standard in the coming months, and WS-Trust is pretty likely as well. (And Yadis, and XRI, etc, etc.) The implication is that our concept of "identity" should be flexible enough to use either of these, as well as having conventional "local" identities. The User table probably incorporates a concept of identity type, as well as the handle and password. We should be able to add additional identity types in-code, without changing the DB, so pay close attention here.
Rendering: Looking at this carefully, we should distinguish between three stages of displaying a page if we want to do this perfectly:
An idea inspired by baavgai: it might be very useful to have the intermediate Parse Tree format be XML. If that were the case, a basic renderer could be simply an XSLT file, with no code required. I think there should still be the ability to write a more-sophisticated renderer with code if necessary, but the XML/XSLT model is pretty mature at this point, and therefore well worth considering.
Ad Hoc Page Construction: One of the desireable features is allowing ad hoc queries. Think about what this means. It has all sorts of interesting ramifications for the rendering stack. It essentially means being able to create a new page on-the-fly, probably directly descending from the root Page, feed that through the render stack, and not actually save it anywhere. This feature is probably worth having just for the rigor it will impose on the code.
Timestamps, not Revisions: A key issue here can be phrased as a question: how do I get a particular version of a page? Ordinary wikis make that easy -- they simply have the previous revisions stored in the text file, or in separate files. But a "page" in Querki is a composite of many objects, not a single one. So what does a "version" mean?
The answer, I believe, is to view the problem quite differently. Things in Querki don't have revision numbers -- they have timestamps instead. You can't simply look for a specific previous version of a page, but you can get the state of that page as of a particular time, by composing it out of objects from that time. You can examine the history of any specific property or link, but the history of the page itself is in terms of time.
Indeed, Querki can have an operating mode that I don't think conventional wikis have: timestamp mode. You can declare a particular time as a lens through which you are examining the system, and it will resolve everything as of that time. This is potentially a very interesting and unique capability.
Playing Devil's Advocate for a second: one might want to look at the evolution of a page, going through it step-by-step and seeing each change. I don't do this in wikis much, but I do it in code all the time, and I think it's a plausible use case. That demands a more discrete view of the changes to the page. We should probably make sure we permit such a thing to happen. I think it's conceptually straightforward -- you build the change list by coalescing the constituent properties, comparing their timestamp histories, and constructing a list of changes from that. But keep it in mind.
Diffing: Most simpler wikis are profligate with space, storing the fulltext of all revisions of all pages. We probably should design for a diff-based mechanism from the beginning, so that we can honestly say that we're reasonably space-efficient. This isn't critical-priority, and we might be able to go indefinitely without it, but we should assume that we will eventually implement diff-based storage, and plan around that.
Property Meta-Flags: Another suggestion from baavgai -- we might want to have some flags that can be set on a property, probably at the archetype level. For example, a "required" flag would indicate that each child is required to explicitly set the value of this property, and is not permitted to simply inherit the default. And a "singleton" flag could indicate that this property may only be specified once per object. (Or that could be the default, and a "multiple" flag indicates that this property can be specified many times.)
Complex Properties: Consider this problem from the Cookbook Use Case. Each "ingredient" line wants to be a tuple of the ingredient itself and a quantity. We want these to be separable so that we can write queries on ingredients without worrying about quantity. But this suddenly implies that we need "property" to be a subtler concept -- potentially a data structure unto itself, not just a string -- and that we want to be able to query on specific fields in such a structure.
This is a huge can of worms, and implies that we need a richer OO model. The obvious implication is that each "ingredient" wants to be essentially a micro-page unto itself (a page section really), composed and rendered in the context of the wrapping page. Thus, a property can be a string, a link or an embedded page with properties. It also implies that we need to be able to query through these indirections in the query language: I need to be able to say "Give me all pages that contain an Ingredient property whose IngredientName? child property is 'asparagus'".
Back End Neutrality: The system definitely shouldn't be wedded to MySQL?, although that's likely to be the dominant implementation. To enforce real neutrality, it would be useful to have a pure flat-file implementation of the back end. It won't necessarily be efficient, but if done well it could provide a nice "entry-level" version for small sites that don't want to deal with a database.
Front End Neutrality: Another feature that would be useful to impose some good code discipline -- allow non-Web front ends. For instance, it might be very useful to have a Tcl front end for this thing, for running locally on a machine that isn't running a web server. Being able to do that will require proper code separation.
Unit Tests: It would be worthwhile to have a variety of unit tests for this system; give that some thought. In particular, it would be a fine thing to have security-focused unit tests, to make sure that access works as desired. Unit tests would also serve a useful architectural role in the usual way -- making this thing testable demands a strong measure of decoupling, and will force the interfaces to be cleaner.
On the end-result side, we should probably check out [Watir], which seems to be a Ruby-based testing framework for examining the actual end results that come out of a webpage. Or maybe use Selenium, which is the bigname answer to such things.
Page Cache: Suggestion from siderea: given that speed is an issue with many wikis (and probably will be with this one as well), it might be very worthwhile to have some sort of page cache, of the post-composition pages. That's not simple, but it's terribly interesting as an idea. Think about how this works in the architecture. It probably implies that the cached page carries a timestamp, and a set of dependencies; if the timestamp is newer than those of all of its dependents, then the cache is still valid, and we can skip the bulk of the Render stack.
To make this efficient enough, the real trick is the timestamp checking. A naive implementation would have to recheck the timestamps of all of the depending objects, which is a big DB hit by itself. Can we improve this with back-linkages? We could put some extra load at "edit" time, such that it finds all cache entries that are dependent on this object and deletes them. (Or just marks them dirty, if that's cheaper.) That could be a huge improvement, I suspect.
Definable UI: This one's a bit more off-the-wall, but worth pondering. siderea was thinking that this project sounds like it has elements of Filemaker Pro, and describes FMP in this note: http://jducoeur.livejournal.com/237235.html?thread=1649587 . I don't think that's where I'm planning on going, but it's worth investigating further, to see if there are clever ideas worth looting from there.
Schema Export: Since it's increasingly clear that Querki is really about applications, it will be important to be able to export an entire application from the DB. This probably means exporting a namespace, but think about it carefully. Obviously, we'll also need to be able to import into another DB. Normally, we only want to export the tip revision of each object, not the full history -- think about whether there is any reason to want to have the ability to export full histories. Think about the maintenance side of the equation as well: we want to be able to upgrade an exported app, while preserving local tweaks as much as possible. This is essentially a classic merge problem: take the originally-imported baseline, the new import, and the local changes since the original import, figure out the differences, and merge them together, reporting clashes if they are found. Not easy, but a long-since-solved problem.
What is the format of an export? It really should export to a single file, for easy transport. Probably XML of some stripe, possibly a zip of a collection of XML files.
Edit Contention: We should take seriously the issue of multiple people editing the same object at the same time.
There are two obvious options of how to do this. The first is to "lock" objects: when one person begins to edit something, others aren't allowed to edit it while they're doing so. The issue there is how long the lock lasts, given that the person editing could simply switch away, "losing" the lock. So we need to probably time out locks in some fashion. We could have a very short timeout, and have the edit screen do an Ajax-style ping periodically to say "I'm still editing". Or we could have a longer (or no) timeout, and have a mechanism to break locks if one seems to be abandoned.
A better mechanism would be a conflict/merge capability. When you go to edit a page, the timestamp of the objects come along with the edit page. When we submit the edits, that timestamp gets checked. If someone else has edited it in the meantime, you are presented with the conflict and maybe a proposed auto-merge. Re-edit (or approve the merge), and re-submit. This is more technical work, but if we wind up needing to have merge capability built-in anyway (for module updates), this would be an ideal way to handle the problem. Also, note that the timestamp comparisons probably need to be built into the actual commit transaction to really make it robust.
Threading: Related to that, but subtler and more important -- the system must be robustly multi-threaded. If it's going to be enterprise-grade, we absolutely can't ignore the threading issue: most web servers will tend to feed it requests multi-threaded. And for efficiency, multi-threading is important, given the number of DB hits that will sometimes be involved. So pay very close attention to places where there is potential thread contention. (Hopefully these will be minimal -- there should only be thread contention when we are trying to work on the same objects.)
In-line Object Rendering: This one is challenging, but probably correct. There are times that what we really want is to render or refer to one object from within another. One special case of this is comment threads; another is links; another is bibliographic page references, as I use extensively in the Girl Genius wiki. In each of these cases, what we really want is to be able to create and insert another object in-line, but probably have it be more semantically rich than simply text. In the case of comments, we want to be able to show or suppress them at will, for instance, and in general we want to be able to control their rendering. An interesting question is whether they want to be self-contained objects or parameterized ones: a natural usage model is to have the data be in-line, but the rendering defined by the object. Hmm...
Query Language: In conversation with Zohane and Tibicen, they had different views on the syntax of the query language. She observed that these things tend to wind up with "for each" semantics, which makes sense -- when stating a query verbally, I often construct it that way. He observed that the problem I'm describing seems to be largely one of set math, and that simple set-math operations ("+" for union, "-" for intersection) might be intuitive; that also seems plausible. In general, this argues for being prepared to evolve the language as we play with it and learn.
One thing that is clear is that the current language, which starts with a single set and winnows down, is insufficient. There is a key type of query that I can't currently do, which is akin to "join". A current example to hand runs like this, "For each Character for whom a Player has that Character assigned and the Player is in the Morning Run, do the following." Or even better, "For each Character for which there exists no Player that has that Character assigned and is in the Morning Run, do the following."
Even without joins per se, we certainly need to be able to do nested queries. That's such a gap that I'm now trying to fix it in ProWiki. That isn't as powerful as real joins, but it's a start. Either way, think about how to specify variables in case of ambiguity -- both joins and nested queries introduce the possibility of having multiple pages involved in a single query, which raises the question of which "" field we are dealing with. Ideally, we want a way to distinguish in these cases, which doesn't make life inconvenient in the default single-level case.
There is also an open question of the syntax of the query language. It's effectively a markup language: the syntax needs to be artificial enough that it's not going to come up in ordinary text. (So something SQL-like is out.) I see two basic alternatives. On the one hand, I can stick with something like the current language, which is highly symbolic and C-ish in nature. Or I can go with XML -- bulkier, but perhaps less opaque and more standard-y. I am torn between these two.
Siderea adds further observations about possible syntax options in this thread: http://jducoeur.livejournal.com/289991.html?thread=2110663#t2110663 . That's well worth thinking about as a very different collection of options.
Query Design and Rendering: A big question is what queries look like in the eventual WYSIWYG interface. I don't want to have to keep writing queries by hand -- that's tricky, and bug-prone. Instead, there should be a query designer at the Ajax layer.
For design, I think it wants to be pretty context-sensitive, providing drop-downs based on the case at hand. For instance, it should know about the current object type and provide the variables available for that type; it should also do the same inside queries, for the variables available in that query.
For rendering, I believe the query per se should be entirely hidden. Instead, we should probably use color to indicate it. When you write a query, the display result of the query is what shows in the design view, with a distinctive background color. The query itself shows in a separate pane when the cursor is inside that display result. You can write the results first, and drag over it to specify what will become the query results, adding the query over that. We should also have syntax coloring for query result variables, to show them distinct from the plain text.
Markup Language: Zohane observes that MediaWiki? has a very good markup language: it is both intuitive for simple operations, and powerful for advanced ones. While we should remain markup-neutral, this may be a good default target.
Tags: A solution in search of a problem, so it's low on the priority list; still, it's been useful in enough contexts that it would be interesting to see what people do with it here. One standard mechanism in modern data-management systems is "tags": arbitrary text fields that you can attach to objects. This is potentially pretty interesting, providing a way to get little reminders of what's what.
Implemented naively, tags aren't really all that useful, because they are simply degenerate properties -- a tag on an object is more or less a property with no value. That might be useful, and it's probably worth adding precisely because it's easy, but it's not all that interesting.
What's potentially more interesting, though, is considering tags to be user-specific properties. That is, an object has a set of inherent properties, but individual users can attach further properties themselves. These could be simple degenerate tags (with no value), suitable for searching, but they could also be value-ful and queryable.
I'm not at all sure of all the implications here, or what the use cases look like, but let's keep these ideas in mind. There seem to be two implications. First, in the short term, we should permit value-less properties on objects -- that gives us the simple idea of tags cheaply. In the longer term, we should bear in mind the idea of having properties that are separate from the page itself: that are a tuple of the object, the user and the property name.
Extensibility: Extensibility is a priority here. What does that mean? How do we make this thing seriously extensible? My usual Ecology-based model will help, but what else should we do? Look into Eclipse deeply, and understand how its model works -- it seems to be the best-reputed extensible system out there right now, so there are probably lessons there to be learned.
Related to this -- think carefully about the composition model. A lot of the behaviour of page rendering is potentially a combination of namespace default, user default, and current user mode. Do these different levels allow you to specify the particular elements that are going to make up the rendering chain?
Binary Inclusion: It should be very easy to upload appropriate binary formats into the wiki, such as images and sound, and use them appropriately. This probably means that you can have binary-formatted properties. Whether the format is a characteristic of the property type or instance isn't entirely clear, although it may make sense to make it part of the type. References to binary types should be easy. Two use cases for images to consider. In the Girl Genius wiki, I should be able to have a character's picture as a standard property. The Character display template should show this character's image. And other characters should be able to refer to another one's image easily.
Keep in mind that references to binary images need to be parameterizable. That is, you don't just give the reference, you can also specify things like alignment and size. So this is an interesting new requirement: the ability to parameterize property references in some reasonably general way.
AJAX Front End: Perhaps badly in tension with the desire for Front End Neutrality, but still worth thinking about, is the question of how to make a really pretty and fancy front end as the usual case. One real possibility is to pick up one of the major web toolkits that are out there. Some (like Yahoo) are just simple collections of a few widgets; some (like Dojo and the Google Web Toolkit) are whole religions. I have to admit, if we decided to break down and write the back end in Java, it would make oodles of sense to write the front end in the GWT, which is designed to integrate nicely with a Java backend and is extremely powerful and easy to use.
This is one of the hardest questions we'll have to deal with. We could write a completely custom front end, and get exactly what we want. But there's something to be said for choosing one of the existing baselines and starting with that. If we can manage to keep a modicum of front end neutrality while doing so, then we aren't over-committing. Indeed, there's something to be said for maintaining two front ends -- a very fancy one for normal browsers, and a highly stripped one for, say, cell phones. That would keep us honest...
Identity Model: Need to make sure we get this right. Querki should have a pluggable concept of identity, where a user's identity can be entirely in our database, but can also be a pointer to one of the larger identity systems, like WS-Trust or OpenID?. Such an identity still gets ACLs in Querki, but we don't attempt to manage all of the identity information. Most importantly, it is explicitly linked to that external identity. This should enable us to do cool integration things that would otherwise be annoyingly difficult. (Might as well be part of the identity solution, not the problem.)
Trust Model: The principal use cases for Querki are internal ones, where the thing sits inside the firewall and everyone is trusted. But it would be nice to use it publically, as well. How do we do that?
Some version of my "Trustulator" model (related to Slashdot's karma system) seems the way to go. Each user has a "current trust level" (infinite for admins, zero for new users and slightly negative for anonymous). Users can express trust to varying degrees in each other, ranging from extremely strong statements of "I know this is a real person who I trust" to "They have done some good things". A user's trust level flows to their edits of data. When you view a page, you can say how much trust you demand from it; the page's rendering depends on the trust of the owner of the underlying data. Users mainly express trust in pages: in particular, when they mark an edit as spam, it has a strong negative trust impact on the person who edited that page. Admins can slam a user to MinTrust? if they find that person to be a spammer. Trust flows upstream -- if you expressed strong confidence in a spammer, that makes it likely that you are also a spammer. It is possible that page views that do not result in a spam rating give a slight trust-uptick: an implicit statement that this page isn't spam.
Such a system might make a wiki reasonably resilient against spamming. Some spam would get through, but you basically can't spam more than once with a given account. And if spammers try to abuse the system by building trust graphs among themselves, they will tend to all go negative. This allows trust to flow down from the trusted admins, to the real users who are editing well. Making it efficient would be a challenge (need to study some graph theory), but the principle is reasonably solid. And if viewing of the system is mainly of pages marked with a trust rating of zero and higher, most viewers simply won't see the spam most of the time.
Need to work through many use cases here. Think about admins, registered users, unregistered viewers and anonymous editors, and spammers, all as the players in this complex graph. How would the spammers try to abuse the system? How do we deal with data composition efficiently in light of this system?
Outlining: [This thread] points out an example of why it would be useful to be able to support numbered outlining decently well, and that the XHTML religion is getting in the way. This is reasonably compatible with Querki -- it's all about data organization, after all -- so think about how it might work in this architecture.
Offline mode: One of the more common and useful mechanisms available today is Google Gears, which is rapidly turning into a standard. Ideally, Querki should have a mechanism that caches pages locally, and allows me to edit them offline and resynchronize later. This is very advanced client-side stuff, but would be incredibly cool.
If we do this, we should expose the cache to the user: it should be clear when we are working offline, and the page should be visibly marked as such. Some pages might be specifically marked by the user as cacheable or not cacheable.
Note that synchronization of conflicting edits becomes a sharper problem if we do this, but it was going to be a problem regardless.
Back to Querki Design Notes.