In my last post I mentioned the possibility of learning more about the structure(s) and behaviour(s) within the dpreview forums through the process of developing a model to describe it. Well, developing models is hard-going, so I deciding to tackle it from a more interesting angle and spent some time attempting to represent forum activity graphically.

Digging a hole for one's self
How and where to start? Well, there are myriad metaphors for the concept of 'conversations', but I like to frame them as Darwinian competitions between ideas in which the 'fittest' ideas 'breed' more 'successful' responses thus carrying on the line (of conversation ;). Hence, my first attempt to create a graphical representation of forum activity involves representing a single conversation in a quasi-organic sense, to see whether it is reasonable to label such entities as 'growing' or 'evolving' (warning: metaphors may be mixed during the course of this post).

After initially experimenting with the processing visualization environment I abandoned it in favour of 'good old' Flash (lens widget induced trauma now mostly repressed). Next up, choosing the basic metaphor (plane, terrain, tree, radar)? Hmm, let's try a variation on the hyperbolic tree (naturally, you say). What to represent? ... ah, let's go with pedigree, generation and mutation (replies which have different subject line to their precursor). A few hours later and presto! Coloured graphs everywhere.

A thread is born
Let's try out this visualization widget by using it to represent a popular recent thread (To the team of DPreview: Do a more serious test of 50d!!! - Canon EOS 50D - 10D forum) in its early phases (6 hours after OP).

Visualization of the To the team of DPreview: Do a more serious test of 50d!!! thread (6 hours).
Colour represents time. Connections represent lineage. Distance from OP (black dot) represents 'generation'. Dots outlined in black represent changes of subject.

So what are we seeing here? It all begins with the black dot representing the original post (OP). Sprawling out from the OP are several dead-end responses and one dominant response (A) which has sparked some conversation. Interestingly this 'successful' response was the first in the thread to change the subject line (to "DPreview has done retesting 50D... Look").

So, can we predict anything about the ongoing development of the thread by looking at the above image? Well it's a fair bet that conversation is going to follow on from A or one of its descendants, though it's also worth keeping an eye on the more recent responses to the OP too (B & C, also sporting altered titles). Let's take a look at how things have progressed 35 hours on.

 
Visualization of the To the team of DPreview: Do a more serious test of 50d!!! thread (41 hours).
Colour represents time. Connections represent lineage. Distance from OP (black dot) represents 'generation'. Dots outlined in black represent changes of subject.  

Forty-one hours in we can see that the An branch (descending from A) did in fact continue a little while longer but has now run cold, instead branches Bn (descending from B) and Cn (descending from C) dominate the landscape. Note also that several variations to the OP have occurred since B & C, with some small success, suggesting that the OP's potential hasn't been fully exploited. This becomes more apparent as the thread progresses.

Visualization of the To the team of DPreview: Do a more serious test of 50d!!! thread (59 hours).
Colour represents time. Connections represent lineage. Distance from OP (black dot) represents 'generation'. Dots outlined in black represent changes of subject.

Forty-five hours into the thread a response (D) to the OP appeared (with the altered subject line: "Future reviews from dpreview will loose value"). The offshoots of D immediately starving previously dominant branches (Bn, Cn) and remained dominant until the 150 post limit was reached.

Time well spent?
Besides learning more about the flash 9 drawing API (note to self, read up on flash 10 drawing API) this mini-project has given me a better understanding of some of the nuances of how dpreview forum threads typically unfold. Of course these colourful diagrams don't tell the whole story or even a noteworthy fraction of it, but they do depict some latent trends which can be investigated via other means. For example I'm keen to investigate further the idea that responses to 'cold' threads garner more interest if they sport new subject lines versus the standard 'Re: ' prefix (another post perhaps).

More
I've included a few more diagrams of other threads below (click thumbnails for larger versions). I'd be interested to hear your thoughts about any more patterns you think might be worth investigating. Also, if you've got an idea for a visualization approach which may highlight other aspects of the forum (and yield some more eye-candy to boot) let me know.

Hi, Is business so bad in US? (larger version)	4/3 has no focal length advantage-correct him if he is wrong (larger version)
LX3 vs. DSLR: You make the call! (larger version)	G10 is AWFUL, but.... (larger version)

By far the largest content group within dpreview is our discussion forums, with around 24 million posts over a period of almost 10 years from around 320,000 posters (dpreview forum stats). One thing I've noticed in the past year is that everyone (admin & forumite alike) has a different take on the forum's nature and likewise everyone struggles to articulate their mental model of it. This will be an issue for the dev team when forum development resumes, because everyone has a feature request (or five) at the ready but no shared vocabulary or model exists to help concisely explain issues or solutions. My goal then is to produce (over a series of blog posts) such a 'forum model' to facilitate productive discussion, analysis and visualisation.

The knee bone's connected to the ... ?
So what exactly am I looking for? At the very least a comprehensive (interlinked) glossary of dpreview forum-related concepts. Even better would be a suite of diagrams really capturing the low-level actions (posting, replying etc.) and higher-level phenomena (cliques, debates etc.). Geek heaven would be a set of techniques to measure, describe & visualize user activity (individual and aggregate) and overall 'forum health' (based on metrics yet to be defined). So roll up the sleeves, it's time to talk social network theory.

The who and the what
Before we tackle the juicy stuff (factions, reputation, mood) we need to cover the basics. First, let's look at the various players and some of the simple (observable) behaviours actions they can perform within the system:

• anonymous
  
  • view list of forums
  • view list of threads within forum by date updated
  • read posts / threads
• members
  
  • all of the above
  • start thread
  • post reply to thread/post
  • quote user
  • edit own post (within allowed time frame)
  • reference images in their post(s)
  • link to any URL in their post(s)
  • view list of threads they've participated in recently
  • complain about any post (including their own)
• admins
  • all of the above
  • all user actions
  • delete post (and all follow-up posts)
  • delete thread
  • lock thread
  • move thread
  • issue forum-specific ban to user (temporary or permanent)
  • issue all-forums ban to user (temporary or permanent)
  • view complaint list
• bots/spiders
• advertisers
• developers

Constraints
So now we know who's playing this game, so what about the rules? Our forum rules come in two flavours: 'hard' rules, imposed by the unyielding machine, and 'soft' rules whose enforcement depends largely on whether the moderator (a dpreview writer procrastinating mid-review) lost their last foosball game.

• Hard rules
  
  • 3 posts in 15 minutes per user
  • Posts can be edited up to 15 minutes after initial posting
  • Maximum of 150 messages per thread
  • Message length limited to 8kb in length
  • Messages will be blocked if they contain censored words or coupons
  • Messages will be blocked if they contain blacklisted URLs
  • Html tags are be removed from submitted posts (plain text only)
• Soft Rules
  
  • No schilling or classifieds: It detracts from the conversation.
  • No coupons: It puts us & advertisers in a precarious legal position
  • Relevant images only: Technique-related only. Happy-snaps go in samples & galleries.
  • No cross-posting: No posting identical message to multiple forums, it leads to problems
  • No multi-posting: Repeatedly posting the same message frustrates others.
  • Nothing risqué: What is tasteful to some may be pornographic to others
  • No inappropriate child images: Illegal, wrong.
  • No voyeur images: We don't want our community to go in that direction
  • Be civil: (most infringed rule) Insults beget insults, nobody wins.
  • No Off-Topic: It distracts. We move it if can, otherwise it has to go.
  • No bashing: Broken-record, unsupported criticism of brands/individuals wears thin.
  • No bumping: Pushing your pet thread to the top distracts from bona fide popular threads
  • No off-board harassment either: Just because it happens off-board doesn't make it OK
  • No piracy: It's illegal. Enough said

Individual behaviour, group behaviour and user archetypes
Ok, so we've covered the main players and most of the rules they're (theoretically) playing by. Now to the meat of the issue: behaviour. Many existing features and most feature requests (including ports of other forum's features) represent attempts to shape, curb or encourage various individual and group behaviours. Here's a list of a few such behaviours (desirable and undesirable) off the top of my head:

• Conversation: the obvious one. But can conversations exist above and/or below the level of threads (probably)
• Cliques: People who interact with each other to the degree that they can be considered a group. What is the participation threshold which implies clique membership? Can cliques overlap or do they merge?
• Fan-boys: What degree of content analysis is required to detect this behaviour programmatically?
  
• Bumping: I'm pretty sure we could detect this programmatically
  
• 'hit-and-run' posting
  
• Cross-posting
• Flaming
• 'Me too' posting
• Bashing
• Ad nauseam debates

Here's where you come in
The lists above are incomplete, I'm sure of it. There's a huge community of dpreview forum members and readers with a deep understanding of our forums (and other forums) so I'm trying to tap into that here. If you can think of any other individual behaviour patterns, group  patterns, user archetypes, metrics or resources on the subject I'd love to hear from you (no 'I was banned', 'jimbo99 calls me names' type comments please).

Aside: What do I mean by 'visualization' anyway?
The expression 'visualizing the forums' may sound like a buzz-phrase, but it refers to a growing field of research around producing graphical representations of complex structures using statistical analysis and computer generated imagery. Feel free to suggest visualisations we could derive from the dpreview forums dataset. Below are a few social networking visualization examples (courtesy visualcomplexity.com).

The past year has seen a dramatic increase in staff at dpreview, but less new 'features' than many expected. Why? What have we been doing? I thought I'd inaugurate the dpreview dev blog by shedding a little light on the dev situation here at dpreview over the past year and in the immediate future.

Oct '07 to Jan '08 - Lens Reviews

Aside from the addition of Andy, Lars & Richard to the editorial team, October 1st 2007 saw the addition of myself (Jaysen) as dpreview's first ever full-time developer (Phil retains his role as hacker-in-chief). By November 2007 we'd handled the transition and knuckled down to begin the lens review project in earnest. Reviewing lenses is a complicated business and we're pretty proud of the proprietary charts, analysis pipeline and interactive UI we've developed for the job. Andy, Phil, Simon and I put some long hours of R&D into that project, which came to fruition when the first lens reviews debuted on 28th Jan 2008. The majority of the subsequent lens review feedback can be described as polite congratulation, with the remainder divided between demands for more reviews (praise of a kind) and confused folk looking for the pre-2008 lens reviews (there are none!).

Feb to Jun '08 - Infrastructure & Search

When we sat down to plan our followup to lens reviews, it became clear that the majority of our future plans required some serious renovation of the site's foundations. So, with a tear in our eye, we shelved our beloved feature ideas and marched into the salt-mines of infrastructure development. The following 5 months saw dpreview transition to a new hosting provider with more bandwidth, hardware and redundancy. The associated hardware/software upgrades also necessitated an overhaul of our search system with some expanded features to boot (any problems with search? let me know) but went largely unnoticed (aside from the occasional yawn).

Jul to Oct '08 - 'Camera Database' overhaul

With the hardware and hosting infrastructure sorted, we turned our attention to a collection of components loosely referred to as 'the camera database'. In an ironic twist for a public-facing web development team (now a bona fide 'team' with the addition of Jan in September) we've spent the past few months developing a new suite of internal tools for data entry, analysis and reporting. Besides dramatically improving the 'quality' of the specs data, the overhaul saw the addition of the 'live view' and 'USB' query parameters to the feature search.

To the lab!

This week we've begun development work for a new, exciting and very public feature of dpreview. I can't say what the new project is yet, but it will be of interest to all dpreview visitors, especially our forum community. What I can say is that we're planning to change the way in which we launch new features onto the site. While past updates have been rolled out silently (or with a quick forum post), future feature releases will be available (initially) in a new 'labs' area of the site (on an 'opt-in' basis). Corresponding announcements and discussion of labs roll-outs and site tweaks will be coordinated through the dpreview dev blog.

We hope to throw this mysterious new project into labs early Q1 '09 for feedback (but as ever, we're operating on a 'when it's ready' basis). In the meantime, in the spirit of the new blog, I'd like to encourage feedback through the feedback channel and the appropriate dev blog comment threads. (discussing bugs/features in the forums is fine, but may not find the right ear). If you're interested in the progress of the site I'd also encourage you to subscribe to the dev blog feed.