Let's start with trackback, since it is the most powerful of the new standards we've implemented. Trackback is a piece of technology that works in the background, communicating between servers. It lets one server 'ping' another server to 'tell it' about some action a user has made - provided of course that both servers implement the trackback standard. In the web-logging world, trackback has mostly been used to allow disparate writers - using disparate systems - to create distributed discussions across multiple sites. It allows a user on site-A, who links to a piece of content on site-B, to 'tell' site-B about that link. Trackback has also been used to share common categorical metadata terms for posts across mutliple sites, and has many other potential uses, which we'll touch on later in this document.

An example:

For instance, if a blogger-A encounters an interesting piece of content on Blog-B, and has something to say about it, she could of course leave a simple comment on Blog-B, but then her own readers would only come across it if they happened to read that item on Blog-B. With trackback, she can, instead, write about the Blog-B content on her own Blog-A, and have her blogging server software 'send a trackback ping' to Blog-B to inform it about this. The two entries are now two-way linked; Blog-A's entry includes a link to the Blog-B item it is discussing, while Blog-B's item includes a link back to the item which 'pinged' it (and which, presumably, has something to add to its content).

Impact on web-logging culture:

In practice, this has created an explosion of cross-annotation and link-cascading, and a very interesting culture of mutual linking and distributed conversation. There are two very interesting differences between the kinds of discussion emerging from weblogging trackbacks and those supported by conventional online forums. The discussions which emerge are structurally quite different from the thread-tree discussions found in online forums. Typically, blog entries will either have no trackbacks at all, or will be intensively linked-to, and become part of extremely non-tree-like trackback discussion branchings and loops. The number of potential entry-points into the discussion is also much higher in trackback-nets as opposed to thread trees: In emergent trackback nets, every node - being part of a particular writer's discursive structuring and readership base - will have its own range of ways of being discovered by users from a range of communities andinterests; in tree-structured discussion forums, the top-level discussion topic, on the site hosting the forum, is the only entry point.

What we have done:

We have created server-side software which implements trackbacks 'out' and 'in' for any content on our servers. We have also begun the implementation of user-interfaces for leveraging this power. There are two aspects to trackback: 'out' (pinging other servers, or other content on our own servers) and 'in' (receiving pings and associating their content with ours):

'out':

We can send a trackback ping to any compliant server to 'tell it' about content on our sites which reference it. On our test server, for instance, if a user adds a link to a discussion item, the system will try to contact the target of that link with a trackback ping, so that the content being linked to can link back to our discussion item. Of course, we can also use this within our own sites as a mechanism for connecting two pieces of content.

'in':

We can receive trackback pings from any compliant server, to tell our content that it has been referenced elsewhere, and also 'expose' our content for 'auto-discovery' of trackback urls. On our test server, for instance, if someone 'blogged' a piece of content, the details of their blog entry would be displayed in a little portlet within our site alongside our content.

Many potential new features:

It is important to realise that the examples given above are just a sampling of the new features that can be explored now that we have trackback working behind the scenes. We'll touch on some of the new possibilities in the second half of this document ('Why implement these new ingredients?')

Where to find out more:

Most writing about trackback is rather technical, but you can get the gist by scanning the first document linked-to below, and by visiting some sites which make use of trackback and following some of thetrackback links there ( URLs follow - you can generally find trackbacks for a blog entry either listed, or linked-to, at the bottom of the text for that entry)

• http://www.movabletype.org/trackback/beginners
• http://www.plasticbag.org/archives/2003/03/observation_on_the_trackback_how_to.shtml
• http://weblogs.mozillazine.org/hyatt/

New features based on trackback:

The new features we are planning, based on trackback, are covered separately in the 'Why implement these new ingredients?' section below.

RSS (Really Simple Syndication) is a standard format for delivering any kind of web content as a 'news feed'. There are many 3rd-party software applications for reading and organising news feeds; these allow users to very quickly scan a number of feeds to check for updates or interesting new content. Instead of having to visit every site you want to keep up-to-date with, wading through each web interface, you just browse through the news feeds that you 'subscribe' to; the feeds are displayed very tersely so you can scan quickly.

A user 'subscribes' to a news feed by entering a special URL into her news-reading program. These special URLs are usually available on compliant sites by right-clicking on icons with names like 'RSS', 'XML' or 'syndicate this site' and copying the special link location.

Another popular use for RSS feeds is to share news and updates between websites; one site will display a summary of the latest items from another site - usually in a little portlet or box. In these cases, it is servers which 'read' the news feeds, turning them into html to display in their own pages. Our portals, for instance, can display other sites' news feeds (this is a panesfolder option for all users, and can also be used by admin to make news portlets).

Users of our portals may have noticed that the Plone content-management system the portals are based on already supports RSS pretty thoroughly. Search results, for instance, always have a little RSS icon which links to the RSS URL for subscribing to a 'canned search'. It is also possible (depending on a site-wide preference) to turn syndication on for any content you have permissions over; in that case the content will have an RSS feed widget (in the NGRF site this is one of the tools in the left hand portlets area). However, this is of limited usefulness since it only shows updates to particular folders.

What we have done:

We have made our discussion system very RSS-friendly. All discussion topics have an RSS feed, and we've also exposed links for subscribing to site-wide updates on new discussion items posted. We have also made the RSS feeds for discussions very efficient to serve, so it will not slow the site down if there are a lot of subscriptions. The RSS feed for a discussion topic will include updates anywhere within the thread branching of that topic, however deep. We'll also make sure that we provide easy access to RSS feeds for other containers of discussion, such as team-tasks.

But the advantage of having efficient RSS versions for all discussions goes well beyond the convenience of using a news reader to track discussions. RSS feeds can also be aggregated - which means a number of feeds can be woven together into one list of updates. We will be adding a 'My Discussions' feature to leverage this capability. Users will be able to add a discussion to 'My Discussions', and will be able to go to their own 'My Discussions' area to read an 'aggregated' list of updates across all their discussions, or to subscribe to a special 'My Discussions' newsfeed. We will also be using the RSS feeds as the basis of email updates.

Another use for the RSS feeds from discussions is for displaying overviews of discussion traffic at 'high levels'. For instance, the 'Discussions' area of some of our sites display a little box with the most recent 6 discussion items from anywhere in the site.

We will shortly be providing all site members with their own weblogs (see below); these will also be available for aggregating into 'My Discussions'. We will also be introducing powerful new features for taxonomic categories and resource-repository building; these will also be subscribable as RSS and aggregatable by members into their own personalised feeds.

Also note that, with trackback capability and thorough access to RSS feeds, it will be straightforward to deeply integrate content across sites, and to integrate our sites with content from other systems.

Weblogs have exploded in popularity in the past 5 years. They are almost always single-author websites which publish daily diary entries, notes about news items or short essays, and weblog posts tend strongly to be link-centred; authors will usually post brief notes about other content on the web. Navigating a weblog is very simple: the main view shows recent posts; an archive interface shows previous posts by date-range or category. If you have not already encountered weblogs, brief introductions can be found at http://www.bigwales.com/999458.htm and http://www2.warwick.ac.uk/elearning/tools/blogbuilder/blogpage/.

What has made weblogging powerful is (a) the development of lightweight standards like trackback and RSS for integrating among them and (b) their link-centred nature, which makes them well-connected in mini-webs which cascade links and propagate a multitude of links to popular pieces of content, and makes popular weblog posts come up very high in google's page-ranking algorithms.

Since we have recently implemented all the important standards used by the weblogging world, we can now offer fully-featured weblogs as part of our own sites' content. Since our new discussion system completely supports the blogging standards, discussion posts are, in essence, weblog posts which are also located in a reply-thread structure. There have been many experiments in recent years to try to harness the popularity of the single-author weblog for multi-user collaborations, but with markedly little success.

We will very shortly be making weblogs available in our sites, and for most sites will make the default view of a member's area a weblog. These weblogs will have a view option to show all the posts by that user, whether posted as discussion items or as standalone weblog posts. Because we will be sharing categories across users (and sites), it will also be possible to show weblog-like views of all posts and resources which fall within particular categories. This will amount to a very interesting new kind of experiment in collaborative weblogging, based on having shared content and categories across individual single-author blogs, and making the process of posting much more intuitive by including affordances for blogging through the sites' discussions and content. Later developments will allow users to maintain one weblog as a hinge for their posts across multiple sites, and even to maintain this weblog in a completely different system.

A consistent theme in the knowledge development work done by our users is the creation of resources pertaining to a field of knowledge. The NGRF site, for instance, currently contains 1488 Annotated Reference objects and 236 link objects, while other sites contain 175 detailed literature reviews. Our older portals contain hundreds of very carefully crafted annotatable XML documents and thousands of well-categorised e-learning applets.

However, our portals are not currently doing a good job of leveraging these resources - of providing powerful central interfaces for browsing and searching them, for instance, or rapid tools for creating them. In the case of annotated references and links in the NGRF site, there is the further problem of replicating references and links in order to cite them from multiple content locations; obviously it would be better if the resources were kept centralised and separate from their citation. Another major problem with the resource-base functionality of our current portals is their weak support for categorising and description ('metadata'); the metadata support built into Plone is awkward to use and very difficult to adapt to particular purposes; it more-or-less forces one big 'bag of keywords' on users who want to tag their resources so that they can be sorted, related and discovered in useful ways. This is all the more galling in light of the large amountof work done by KnowNet's principals in the late 90's, specifically directed at powerful large-scale resource repositories, and at standards for structured categorisation and description.

We are now, after much preparatory software-engineering work, in a position to bring our resource-repository work from the late 90's to bear on our current portals. We will shortly be providing interfaces for adding categorical and other metadata to objects, for structured searching and browsing based on these descriptions, and for maintaining vocabularies and taxonomies of descriptive terms. Because we also have an implementation of trackback to base this on, we will be able to make the association of metadata with resources very flexible. We will also be introducing more rapid methods for creating resources, using the MetaWeblog API and 'bookmarklets'. Further work will follow on leveraging the resource aspects of links associated with weblog entries and discussion posts, and on rapid, lightweight 'collecting' and organising of resources into special-purpose repositories (these would complement the repository aspects ofmetadata-based resource-browsing and searching).

Also because of our trackback and other standards implementations, we can straightforwardly distribute / share repositories across sites, and will be in a position to explore doing so across platforms and systems as well.

As an example, when the new repository features are deployed, Annotated References will cease to be 'contained' by the indexFolders which cite them, and duplicate entries will be unnecessary. A central repository of bibliographic references will be maintained, and content which cites them can do so either by a simple trackback ping or a lightweight citation object. Users wanting to cite a reference will be able to rapidly search the repository to see if that reference already exists, and either point to that entry or create a new one. Maintenance issues on the references, such as link-checking and Z39.50 data mining, would then be centralised, since only one copy of the full information would need to be kept in the system.

The weblogging world has developed another mini-standard to allow authors to post entries without doing so through the website. The MetaWeblog API (along with two other minor variants) allows posts to be sent up to a weblog from special little application programs. One of the benefits is to allow drafts to be created offline (while travelling for instance); other advantages include rapid posting without having to open a browser and navigate to a site, and avoiding the awkwardness of browser-based editing tools.

We have implemented the MetaWeblog API. Users of our sites will be able to post and edit their blog entries from 3rd party tools, and to apply categories to them. We also hope to leverage the API support to allow emailing of weblog posts. We will also explore methods for creating discussion topics from the API as well, but this is somehate complicated by the potentially large number of discussions which might be posted to, and which would have to be presented as choices in the editing tool.

This section will concentrate on explaining what trackback brings to the architectural underpinnings of the portals, and what features we plan to base on it in the near future. The benefits of the other recently-implemented mini-standards have been covered to some extent above.

If all we were interested in was adding functionality within our sites, it is conceivable that we could have done so with de novo proprietary techniques. Why have we chosen to underpin these new features by adapting existing standards instead?

Cleaner architecture

Basing the new features on a suite of well-known and clearly delineated standards makes for a more comprehensible, flexible and maintainable system architecture.

3rd-party tools

Using these well-established standards within our portals means that we will be able to leverage existing 3rd-party tools to add value to our interfaces, offer choice to users, and simplify our own codebase.

Integrate with other content and across sites

By fully supporting these open standards, we make it possible to closely integrate not only across our own sites, but with content hosted on other platforms which support the standards.

It's better to be open

Over years of software development work, we have grown to appreciate more and more the merit of architectural open-ness. The benefits of open-ness will become more and more apparent in the future.

Get our google rankings up

Trackbacks and RSS encourage links from other sites, and themselves generate links from cross-site activity (for instance, backlinks from compliant content which our users link to). This will help to raise the search-engine profile of our sites, as well as promoting wider awareness of the sites' content.

We may encourage some new standards-development work

The uses we are making of the standards are for the most part quite novel. As we gain experience with these features, we will certainly think of new functionalities we wish were covered by standards already in place. If our experiments with these functionalities prove to be as interesting as we hope, it is quite possible that other developers will be interested in sharing the task of forging new standards to cover the collaboration actions we will be interested in.

More links: integrate with other content and across sites

The first easily apparent feature which trackback brings is automatic cross-linking with other sites. For instance, if a link to a trackback-compliant site is added to a discussion item, the target site will link back to the discussion. If an offisite blogger writes about a piece of content or discussion on our sites, a link to their entry will appear on our site. This makes it quite easy to carry on discussions and annotations distributed across our own sites, and opens up many potential integrations with other platforms.

Richer branching: non-hierarchical discussion branching and linkage

One of the crucial problems with threaded discussion forums - as our more sophisticated users often point out - is that they branch like simple trees, whereas rich discussions branch in more complex ways. Discusion forums' structure is based on the historical accident of the 'reply' gesture as much as by the dynamic of discussion. For instance, we have repeatedly received the following feature request: user A posts a discussion item which touches on several issues; user B has something to say which is inspired by one of the issues in user A's post, but is not properly a thread from it - user B wants to create a new discussion (raise a new issue etc) which is spawned from user A's post but has a distinct life separate from it.

By offering trackback-enabled methods for spawning new posts which link back to a discussion item (or other piece of content), we can take a large step towards meeting that feature request. For instance, instead of replying to a discussion item, a user might 'blog this' item (spawning a new entry in her blog which is two-way linked with it), or might spawn a new discussion topic (or issue, etc - we will also explore 'typing' of posts), find a suitable location for it to be added in the site, and thus make a new discussion which two-way links to the previous one. As a bonus, this can be made to work across sites as well! (In passing, I note that this approach does not solve another issue we've been eager to be able to branch discussion on - internal structure within a long discussion item. At some point in the future, we'll explore using trackbacks to a specific bit of sub-content rather than an entire post ordocument.)

See figures 1 and 2 for some graphical treatments of non-hierarchical branching.

Multiple contexts: combine the transparency of blog-view with the focus of threaded discussions

By allowing a piece of content to be 'both' a part of a discussion thread 'and' an independent topic or blog post, we can explore having the same content made salient in different contexts for different purposes. In fact, combining this with repository views and possible offsite trackbacks, we can start to explore rich ways of re-using content and widening discussion.

Sharing categories: associate metadata with content in an open manner

By basing the association of metadata with content on trackback, we can enable a cross-site, and even cross-system metadata system. This would be especially useful for maintaining large-scale vocabularies, taxonomies and bibliographic databases.

Relating content: associate content items in an open manner

Trackback can also be used to make explicit 'this is related to that' gestures between two items of content.

We have already completed the implementation of the core engines to support all of the above. What remains to be done is to write specific interfaces to give users the affordances to make use of these features, and the displays to envisage them.

Our expected timetable for writing the interfaces is as follows:

• Fast-Forums view for discussions: October 22
• Trackback from discussion: October 25
• Weblogs: October 28 (with full standards support)
• Multi-branching discussion: October 30
• Editable team tasks: November 1
• 'My discussions' aggregator: November 1
• Structured metadata tagging: November 1
• Trackback from general content: November 1
• Weblog skinning: November 8
• Structured metadata browsing: November 8
• Structured metadata searching: November 8-15
• Structured metadata vocabulary admin: November 15-30
• Blog/discussion integration with offsite blog: November 1-30