Cosley, Dan, & Frankowski, Dan, & Terveen, Loren, & Ridel, John (2007). SuggestBot: Using Intelligent Task Routing to Help People
Find Work in Wikipedia. In Proceedings in Intelligent Usier Interfaces.
Submitted by Tracy Liu on Mon, 03/17/2008 - 23:06.
1
point
This paper studies benefits of implementing intelligent task routing to help people find work using field experiment in online communities including Movielens and Wikipedia, which supports the theoretical predictions both from economics and social psychology(Public Goods Collective Effort Model).
The result supports that intelligent task routing can reduce the contribution cost, therefore, induce more contribution for online users. However, there is no significant difference on the effect of different kinds of intelligent task routings.
It also claims that this approach would work efficiently in online communities where tasks are numerous and heterogeneous as the contribution cost is very high on these sites.
However, I am confused about one methodological suggestion which is “researchers using filed experiments should deploy multiple designs at once whenever possible(p40)”, in detail, we could deploy both within subjects and between subject design in different treatments? Does it add noise and difficulty in data analysis?
Nice summary about the article. For the research design suggestion, I think multiple design can be made properly, but needs to pay for many more subjects involved. And I think the reason the article suggests it, is because of the time-frame constraint in filed-experiment. Just like it has discussed in its methodological note part, people are changing over time, especially in terms of their behaviors on a particular website. So it is better to do comparison treatment within the same time frame. In particular, we study Wikipedia here and we are interested in contribution rate, it can be very different in different life-time-point of both a user and the website. Like Lian's study on Wikipedia, people who were on the website for long tend to contribute more than those who came later. Thus the comparison between people in different time would be suspicious.
My understanding of running multiple designs at one time is to prevent any external effects that might occur that cannot be controlled in the context of a field experiment.
For example, if the researchers tested the design serially (like they did) and Wikipedia decided to change some aspect of the underlying Mediawiki software during the experiment, it would difficult to make comparisons between the groups.
In this experiment, the potential problem seems to be a self-selection bias. That is, the first group of participants might be more eager than other types that join the experiment in later iterations. An easy criticism would be to say that you can't compare the groups because the members in each group are not random (the most eager members self-selected into the first group).
Submitted by John Blair on Tue, 03/18/2008 - 22:56.
3
points
John Blair
Very interesting research utilizing economic and social science theory of reducing costs to increase contribution - or at least the motivation to contribute. Not altogther surprising, but it is nice to see the research results proving a claim.
I'm not sure how this would apply to non member-maintained communties. I suppose one could consider posting to a group message, blog or wiki as member maintained function. It certainly constitutes member content in much the same way that wikipedia is. Though in these non member maintained sites, the cost of contributing is already fairly low, as one simply needs to type a response to a message. However, getting someone to increase their responses is a parallel activity to increasing recommendations / edits that might benefit from topical areas of interest as the suggestion lists.
Currently, my site (ITtoolbox) has a similar function where you choose the group and topic area within that group to receive posts from to your email account. While there is obviously no filtering or intelligent routing going on as in SuggestBot (as I recieve numerous messages a day on my selected topic), I think the system could benefit from the addition of intelligent filtering to only send those messages that I'm likely to respond to or be interested in. Obviously I'm interested in the topic I chose, so that could be a logical starting place for the filtering mechanisms to work from. Beyond that it could considere the previous posts I have made and factor that into the suggestion filter as well.
I appreciate the option of being notified of things I've chosen as having in interest in, but the volume does make it somewhat less likely that I'll review each message received. I can see some real benefit of utilizing a SuggestBot to not only lower the volume of messages received, but also to make those messages more focused and targeted, thus increasing my interest, attention span and likliness to respond.
I agree with your concerns about volume, John. This likewise lead me to reexamine the authors' conclusion that "intelligent task routing is likely to be most useful for communities where tasks are numerous and heterogeneous" (40). Where John's site may suffer from volume in the number of messages, my site (a group on Facebook) suffers for the opposite reason: lack of volume in terms of tasks to be done. Therefore I'm not sure the SuggestBot would work for my type of community, where the tasks are dependent on what type of text is entered by the user.
For example, I am interested in text production on our site: how often people post, what they say, how they interact with others, etc. This mainly occurs on our Wall or Discussion posts. Otherwise there really isn't much room for other tasks. Instead I would like to see users definitely post more and definitely suppor the community (by responding and creating knowledge) but not necessarily doing anything new. Specifically, I'd like to see the task of posting to the call Wall every week accomplished, but economic models don't necessarily predict this. Perhaps what does predict this are social science theories that ask for users to be motivated to respond/post to anything. Perhaps, then, the next step in intelligence is spurring users to know when to respond to a specific kind of post in a specific kind of way. The SuggestBot seems to currently only care about number of entries, no the quality of them, which doesn't seem to apply to all communities at large.
Submitted by LizBlankenship on Fri, 03/21/2008 - 14:48.
2
points
Interesting point at the end of your post.
SuggestBot is trying to solve one problem, primarily- reducing effort in making contributions. It does this by listing a number of items that are on average more interesting to you than a random list. But does the resulting increase in contributions maintain the level of quality of contributions? And similarly, are contributions made to topics someone is more interested in of higher quality than contributions made to a relatively random article or a really popular one that everyone knows about?
Some interesting questions that could help us understand more about sourcing the wisdom of the masses.
It seems a pretty natural conclusion that users are more likely to (a) already have knowledge of or (b) be willing to research articles about a topic in which they are interested. It then follows that the quality of articles should be higher when the user has an interest in the topic, as they are more highly vested. However, I suppose familiarity also introduces the potential for bias.
In either regard, even with random page suggestions, it seems likely that users are only editing pages about which they have a true interest, imposing the same threat of bias. The only real difference is that, without SuggestBot, users spend more time seeking articles of interest, and less time editing them. I think it's safe to assume that bias is an inherent weakness of Wikipedia (gasp!) and that quality is likely only to improve as a result of an increase in edits per user.
Submitted by Daniel Zhou on Sat, 03/22/2008 - 17:17.
1
point
Good point. Althought the three experiments show that SuggestBot can improve contributions, there's no experiment to show that quality of Wikipedia is improved by SuggestBot.
It would be interesting to see how to design an experiment to test the quality of wikipedia content.
I think the point you make here about volume shifting the degree to which you review messages is insightful. My experience with "SuggestBot" like features is limited to job search bots that suggests internships @ the Red Cross that I might be interested in applying to based on my interests, and initially I was excited about the availability of this feature. Although markedly different from the SuggestBot in a wikipedia setting, I did notice that there was a tipping point after which the messages I received from the bot stopped being useful and quickly transitioned to being considered spam.
This presents an interesting challenge to the following premise from the article: "reducing the cost of contribution will increase members' motivation to participate. One way to do so is to make it easy to find work to do (p.32). "
It is evident that in an effort to reduce the costs of contributions, you run the risk creating new costs of participation that may impede users' desire to participate. You risk getting on the nerves of your users...
This is an very updated study as my own interest. It has many things I can borrow to my studies. Here just a bit comments:
1. Routing is basically a recommendation thing. There is always a lot complexity in the matrix we are looking for. For example, just for the concept of "interest" it is can be positive and negative, because I bet people do want rate a movie they really like or really hate. If we just take rating simply as the indicator of positive correlation, it will mess up things. So at the beginning, when the authors thought about just asking user to work on the entries they had worked on in the Wikipedia, (following the logic in MovieLens), then realized it is not good at this particular situation; which is very important in recommendation algorithm building;
2. The inter-comparison among the three intelligent algorithms: apparently these three are strongly interrelated; it would be better to integrate them so i might suggest to test the inter-correlation as well.
While reading about how SuggestBot works, I thought that perhaps including more information on the provenance of suggestions (where did this suggestion come from? how come this was suggested to me?) would increase contributions even more. Some people might like participating in things that are similar to what they have done in the past, while other users might like to participate in things that their friends are also doing. Providing information about why a particular suggestion comes to you makes salient the outcomes that are relevant to you. In this way, cost of contributions is decreased and value of outcomes is increased. Both of these factors, according to the collective effort model as described on pg 33, would increase contributions.
This is what one online community goodreads.com does. Every day you get a list of the books that people on your friends list either rated or marked as "to read" or "currently reading". This list is a really a suggestion on books to add to your own library or to rate. Implicit in this list is provenance - you know that these suggestions are being given to you because they are books that your friends have read/rated. You even know which specific friend rated them.
Design claim: Giving people tailored recommendations for contribution tasks and telling them why they received those recommendations will increase participation.
Submitted by LizBlankenship on Fri, 03/21/2008 - 14:53.
1
point
I wonder if also, to branch off from your design claim, it would help if people were allowed to add more "interests" to the set upon which recommendations were based. Say I have never edited articles on origami because I never come across the Wikipedia pages on the topic, having other sources I go to for origami tips. Maybe my interest in that subject can be harnessed if I tell the system the websites I frequent most often, or a list of topics I'm interested in, in addition to it looking at my list of edited pages.
Perhaps that's getting a bit outside the scope of SuggestBot, but I think it's worth thinking about anyway. :-)
Your design claim would make for an excellent field experiment. One problem with recommender systems is the "black box" aspect of receiving recommendations. Sure, you know that it is based on people that have similar interests to you (in the case of MovieLens or Netflix), but there is no other fine-grained context given. Giving users more information of the recommendations allows them to understand how it came about, and consequently, how they might adjust to improve ratings further.
I've been a long time Netflix user. Until recently, there was no additional context for recommendation. You clicked on a "Movies you'll love" link and there were a whole host of movies you would supposedly like. But why these particular movies?
Without that additional information, I believe people lose faith in the recommendation algorithms when they receive ones that are "outliers". Recently (within the last 18 months or so), Netflix began listing why a certain movie was recommended – "you loved these 4 movies, and people that loved those movies loved this movie." When an outlier pops up, I can now at least understand why maybe it was recommended. Usually, it's because I forgot about an obscure, but excellent film I rated very high!
I think this would be broadly applicable to other recommender systems, like the suggestbot on Wikipedia.
Submitted by Sean Munson on Sun, 03/23/2008 - 18:08.
1
point
Good design claim. Kristin Swearingen and Rashmi Sinha have done some work on what information to show when presenting recommendations, design suggestions included:
transparent recommendations (what you would call provenance): users can see why an item was recommended
an appropriate level of familiarity (users will be more receptive to ecommendations if they have previously heard of at least some (but probably not too many) of the recommended items before). In the case of movies, if the recommender system shows some movies that the user has already seen and liked (but not rated), that actually helps them trust the other recommendations.
Clear path from recommendation to detailed item information, including enough information about the item presented where the recommendation appears
Make it easy to get more recommendations and to remove recommended items they do not like from their list
Some sort of genre filtering
Include new/just released items
References:
Interaction Design for Recommender Systems. Kirsten Swearingen & Rashmi Sinha.
Beyond Algorithms, An HCI perspective on Recommender Systems. Kirsten Swearingen & Rashmi Sinha. Submitted to the 2001 Workshop on Recommender Systems at SIGIR.
I am sort of convinced by this paper that intelligent tash routing is likely to be most useful for communities where tasks are numerous and heterogeneous as it helps greatly in the information filtering. Systems that solicitcontributions from members can use intelligent task routing to increase contributions. Single algorithm may have large effects. Algorithms based only on the community's needs are less likely to interest members than that considers a person's knowledge and ability. Overall, intelligent task routing could be a very useful strategy for eliciting contributions from members of online communities. Designers should use this to build more valuable communities and better experience for people who inhabit them.
This is an excellent paper, I think it is a showcase of using social footprints online to build new features. Such design are essential to manage the information overload boon that have come with Web2.0
Design claim: Recommend community tasks to members that match members' interests. Instead of randomly assigning tasks to members, recommend tasks that they like to do.
Goal: To reduce the cost of participation for members. If users are recommended tasks that match their interest they will happily complete the tasks quickly. They are also likely to have acquired knowledge in the domain related to task.
Context: This can implemented in large communities where data that indicates member interests are available or can be captured. In Wikipedia, member interest can be inferred from articles read, edited, added to watch list by the user. Each community might have a different method for capturing user interest. But in some communities it may be difficult or impossible to capture any data that will indicate interests of the users. In such communities this design will not be applicable.
This is a very interesting reading. One design claim I found from this reading is that: “Reducing the cost of contributing and increasing the value of outcomes will increase motivation to do work for a community.”
This claim does make sense to me. I remember while I was doing the Wikipedia homework, I spent lots of time to figure out what kinds of issues/topics I can edit. If there is an intelligent task routing helping me to search for articles that I might be interested in, and then the assignment would be much easier for me, since I reduce the cost to search for articles.
It seems that the reading doesn’t pay much attention to the value of the outcome. In addition to the cost, I think people may be motivated to contribute more if the outcome is valued by others, since people care about how others perceive their work. For example, after I edited the page in Wikipedia, I didn’t see any comments or reaction from other people. This makes me feel that my contribution is really minor in the community. On the other hand, if Wikipedia can provide a figure to show how many people viewed this article or benefit from this article, I may be motivated to contribute more, since my work is really useful to some people.
I think an interesting part of the reading described how users were annoyed at recommendations that were exactly what the user would edit (and already had). If you got a chance to see Bernardo Huberman speak a couple of weeks ago he mentioned that one thing required to keep users' attention is novelty. The algorithms that the researchers needed to develop were creative because they looked for articles that would be of interest that weren't already edited by that user. I think an overlying idea in community retention is maintaining that level of user novelty.
P.S. I dont' mean novelty in the negative sort of way that you might equate with, say, quickly disposable Facebook applications.
A very interesting article indeed. The main question that it presented to me is, “What is the work that we want done in our community and how do we make that work obvious to users”. In Wikipedia the tasks are specific and discrete; once and article has been written or improved it doesn’t need more work. Some topics will continue to evolve but many won’t require much work after an initial benchmark of quality is met.
In most of our online communities this will not be the case; responding to new users questions in forums will need to be done and submitting content will continue endlessly. These tasks aren’t known in advanced by the sites administrators but they do have the hope that these tasks will occur. If the goal of the community managers is to motivate users to engage in certain types of tasks it might be complicated to communicate these needs to users concisely. One idea to counter this might be to have visualization systems that allow users to view activity in the network according to a variety of metrics that support the administrators’ goals.
I have always imagined that Paradise will be a kind of library.
"SuggestBot treats multiple edits of an article as a single edit, collapsing the list of remaining edits into a set." (p5)
I don't understand this approach because if a person edits the same article many times, wouldn't this behavior signal an interest in the topic and not just the specific subject? For exaxmple, if you edit an article about your hometown many times, and an article about burritos once, wouldn't you be more likely to edit articles that relate to your hometown?
---
While testing SuggestBot, Cosley et al note two bad decisions: 1) randomly choosing members to receive recommendations (and neglecting Wikipedia's opt-in condition for most activities), and 2) posting links to suggestions rather than posting the actual suggestions (more clicks is bad)
(^p4)
This first gave me the impression that as far as Wikipedia norms go, giving members the option to get ("pull") suggestions is good and pushing suggestions on members is bad. However, there is still some degree of "push" in the revised scenario because "The first sizeable batch of users came from a post an early user made to a Wikipedia mailing list." (p7) The only difference appears to be who's doing the pushing - the stakeholders who created the bot (Cosley and friends), or the stakeholders who want a better Wikipedia ("the early user") I guess the obvious, common conclusion is that people are more willing to accept one of these push messages from a friend than from someone who stands to profit.
---
And what's the difference between a "moderator" and a "meta-moderator"?
Submitted by Rozaidi Rashid on Sun, 03/23/2008 - 16:54.
1
point
Meta-moderators moderate what has been moderated (by moderators). Just like a meta-search engine that searches the results of other search engines. Or a meta-analysis that analyses what others have analysed.
Submitted by Paul Resnick on Mon, 03/24/2008 - 05:47.
1
point
I think in the first version they started putting suggestions in user talk pages without users signing up. In the second version, users had to sign up. Advertisements encouraging people to sign up may have been "pushed" via email.
Submitted by Satyendra on Sun, 03/23/2008 - 17:28.
1
point
Intelligent Task Routing reduces search costs because tasks
can be recommended instead of searched. But at the same time the very notion of
recommending a task also sets a goal
for the user – the achievement of that task. Tying in what we read earlier if
this it is also disclosed that certain other members are also part of the same
task, it can help in two ways: easier and more natural socialization and
creating interdependence because of the teamwork involved in the task.
Design Claim: (the
Task and Team (TnT) effect)
Recommending relevant Tasks and a Team of people to solve
that task with to a newcomer of a community can increase contribution, involve the
member by setting a goal and create a stronger identity attachment to the
community.
It is also important in such a setting to ensure that there
are mechanisms in the system to appreciate incremental progress on the task-
which is especially important if the contributors are newcomers.
The authors begin with:
"social science theory suggests reducing the cost of contribution will increase members’ motivation to participate"
If people would've edited something anyway, routing them to something useful is a good strategy (as is mentioned in the paper) but that definition only addresses on aspect of increasing work done, i.e. individual work done, not gaining more workers.
While reading this, all I could think of was NOT eCommunities, but how a SuggestBot would be valuable to me personally. (NOTE: this could be extended to any group of scholars that are working on a set of topics).
For example,
Tagging: tagging documents, personally or collectively, where some documents are under-tagged and others are over-tagged, could benefit from a SuggestBot that would make these disparities visible to researchers who could then task themselves to correct the metadata (archival practice could benefit from this greatly)
Citation Networks: generating citation networks always leaves dangling citations on the margins. Performing textual analysis (as SuggestBot does) to reccomend which citations should be followed (and thus which papers should be added to the network first).
Interesting:
"Intelligent task routing is likely to be most useful for communities where tasks are numerous and heterogeneous."
Worth exploring when and why this is true/false....
-------------------------------------------------------- PHartzog@umich.edu
--------------------------------------------------------
The Universe is made up of stories, not atoms.
--Muriel Rukeyser
Submitted by Paul Resnick on Mon, 03/24/2008 - 05:51.
1
point
Your comment suggests that people are just shifting their contributions. But in fact they seem to have edited more. I suppose it is worth checking to see whether they edited more overall-- there could have been a substitution for effort on other pages that were not part of the random suggestion list.
While I do see SuggestBot as being beneficial to motivating contributions on Wikipedia, one drawback I see is that it can really only be used by those who already contribute quite a bit. If you are a new user, you won't have any edits from which SuggestBot can make suggestions. If you are just relatively new and have made a few edits, SuggestBot could make suggestions but they might not be very relevant to your actual interests - maybe you edited a few random entries. It seems like the accuracy of suggestions would only really improve once a user had made a significant number of edits already.
If this is the case, then this user has obviously already found some sort of motivation to contribute. Already, a very small percentage (I forget the exact number) is making the majority of edits on Wikipedia - so why focus on getting them to contribute even more? In my opinion, what Wikipedia really needs is not more edits on arcane pop culture references, but edits by content specialists on technical or complex topics. So, the community would be served better by having a SuggestBot that could help you find articles no matter how many articles you had already edited - that way it could help motivate newcomers as well. A possibility could be a SuggestBot which allows a user to submit keywords of their interests, and from this generates a list of articles related to these keywords that need work.
Submitted by Paul Resnick on Mon, 03/24/2008 - 05:54.
1
point
A better way to indicate your areas of editing expertise might be to tag a few articles. That way, you enable full-text matching (can use the whole article text as probe). It also fits with natural browsing behavior.
Submitted by Paul Resnick on Mon, 03/24/2008 - 06:00.
1
point
One thing that no one has mentioned is the power of a personal invitation to motivate. As compared to a suggestion from a bot, a request from a person is harder to turn down, and harder the better that the requestor is known and the more you want the requestor to like you.
A personal request is also more motivating, because it indicates that you are known to someone else (they know you exist and have some mental model of what they think you can do) and thus that they care about you. Of course, bad suggestions make you feel like they don't really know you. I think this is the reason we get so upset by bad recommendations from MovieLens or Netflix, a carry-over from how we would respond to bad suggestions from a friend or family member, where bad suggestions indicate they don't really know or love you.
Good point, Paul. When I read this paper I immediately started thinking of how this could apply on different online communities i use, like SourceForge.net -- it seems like you could feasibly build out a similar recommender, but perhaps even more than wikipedia it seems like having a computer algorithm tell you "Project X has a lot of bugs that need fixing" would be a bit trite. It would be more useful to have a human coordinator who could use this system for a team working on a project aid in making suggestions.
Ideally, this system could help a coordinator manage his own backlog of needed work with suggetsions from the system, and route them to contributors in a way where they can be notified of work and express their preference on working on it.
With this system a coordinator could
- easliy identify what kind of work contributors "specialize" in
- weed out obviously bad suggestions
- prioritize suggestions among the contributors
- route tasks to contributors
The recommender then is less of a "suggestbot" and more of a guidence system for a community management tool.
I can see how suggestbot could be a useful within the context of a
site. As with most systems, it has its limitations (which will be
minimized with time) but I find it a useful feature. On LinkedIn, it
gives a list of "possible" connections, on Amazon it gives "recommended
list", Netflix, Stumbleupon...even Google...all very useful services.
Interesting part is that it becomes better with use and time.
A brief Summary
This paper studies benefits of implementing intelligent task routing to help people find work using field experiment in online communities including Movielens and Wikipedia, which supports the theoretical predictions both from economics and social psychology(Public Goods Collective Effort Model).
The result supports that intelligent task routing can reduce the contribution cost, therefore, induce more contribution for online users. However, there is no significant difference on the effect of different kinds of intelligent task routings.
It also claims that this approach would work efficiently in online communities where tasks are numerous and heterogeneous as the contribution cost is very high on these sites.
However, I am confused about one methodological suggestion which is “researchers using filed experiments should deploy multiple designs at once whenever possible(p40)”, in detail, we could deploy both within subjects and between subject design in different treatments? Does it add noise and difficulty in data analysis?
The research design suggestion
Nice summary about the article. For the research design suggestion, I think multiple design can be made properly, but needs to pay for many more subjects involved. And I think the reason the article suggests it, is because of the time-frame constraint in filed-experiment. Just like it has discussed in its methodological note part, people are changing over time, especially in terms of their behaviors on a particular website. So it is better to do comparison treatment within the same time frame. In particular, we study Wikipedia here and we are interested in contribution rate, it can be very different in different life-time-point of both a user and the website. Like Lian's study on Wikipedia, people who were on the website for long tend to contribute more than those who came later. Thus the comparison between people in different time would be suspicious.
Minimizing additional confounds
My understanding of running multiple designs at one time is to prevent any external effects that might occur that cannot be controlled in the context of a field experiment.
For example, if the researchers tested the design serially (like they did) and Wikipedia decided to change some aspect of the underlying Mediawiki software during the experiment, it would difficult to make comparisons between the groups.
In this experiment, the potential problem seems to be a self-selection bias. That is, the first group of participants might be more eager than other types that join the experiment in later iterations. An easy criticism would be to say that you can't compare the groups because the members in each group are not random (the most eager members self-selected into the first group).
This is my understanding, for what it's worth.
higher interest yields higher contributions
John Blair
Very interesting research utilizing economic and social science theory of reducing costs to increase contribution - or at least the motivation to contribute. Not altogther surprising, but it is nice to see the research results proving a claim.
I'm not sure how this would apply to non member-maintained communties. I suppose one could consider posting to a group message, blog or wiki as member maintained function. It certainly constitutes member content in much the same way that wikipedia is. Though in these non member maintained sites, the cost of contributing is already fairly low, as one simply needs to type a response to a message. However, getting someone to increase their responses is a parallel activity to increasing recommendations / edits that might benefit from topical areas of interest as the suggestion lists.
Currently, my site (ITtoolbox) has a similar function where you choose the group and topic area within that group to receive posts from to your email account. While there is obviously no filtering or intelligent routing going on as in SuggestBot (as I recieve numerous messages a day on my selected topic), I think the system could benefit from the addition of intelligent filtering to only send those messages that I'm likely to respond to or be interested in. Obviously I'm interested in the topic I chose, so that could be a logical starting place for the filtering mechanisms to work from. Beyond that it could considere the previous posts I have made and factor that into the suggestion filter as well.
I appreciate the option of being notified of things I've chosen as having in interest in, but the volume does make it somewhat less likely that I'll review each message received. I can see some real benefit of utilizing a SuggestBot to not only lower the volume of messages received, but also to make those messages more focused and targeted, thus increasing my interest, attention span and likliness to respond.
Quanitity over Quality?
I agree with your concerns about volume, John. This likewise lead me to reexamine the authors' conclusion that "intelligent task routing is likely to be most useful for communities where tasks are numerous and heterogeneous" (40). Where John's site may suffer from volume in the number of messages, my site (a group on Facebook) suffers for the opposite reason: lack of volume in terms of tasks to be done. Therefore I'm not sure the SuggestBot would work for my type of community, where the tasks are dependent on what type of text is entered by the user.
For example, I am interested in text production on our site: how often people post, what they say, how they interact with others, etc. This mainly occurs on our Wall or Discussion posts. Otherwise there really isn't much room for other tasks. Instead I would like to see users definitely post more and definitely suppor the community (by responding and creating knowledge) but not necessarily doing anything new. Specifically, I'd like to see the task of posting to the call Wall every week accomplished, but economic models don't necessarily predict this. Perhaps what does predict this are social science theories that ask for users to be motivated to respond/post to anything. Perhaps, then, the next step in intelligence is spurring users to know when to respond to a specific kind of post in a specific kind of way. The SuggestBot seems to currently only care about number of entries, no the quality of them, which doesn't seem to apply to all communities at large.
substituting quantity for quality
Interesting point at the end of your post.
SuggestBot is trying to solve one problem, primarily- reducing effort in making contributions. It does this by listing a number of items that are on average more interesting to you than a random list. But does the resulting increase in contributions maintain the level of quality of contributions? And similarly, are contributions made to topics someone is more interested in of higher quality than contributions made to a relatively random article or a really popular one that everyone knows about?
Some interesting questions that could help us understand more about sourcing the wisdom of the masses.
Quantity/Quality
It seems a pretty natural conclusion that users are more likely to (a) already have knowledge of or (b) be willing to research articles about a topic in which they are interested. It then follows that the quality of articles should be higher when the user has an interest in the topic, as they are more highly vested. However, I suppose familiarity also introduces the potential for bias.
In either regard, even with random page suggestions, it seems likely that users are only editing pages about which they have a true interest, imposing the same threat of bias. The only real difference is that, without SuggestBot, users spend more time seeking articles of interest, and less time editing them. I think it's safe to assume that bias is an inherent weakness of Wikipedia (gasp!) and that quality is likely only to improve as a result of an increase in edits per user.
experiment result
Good point. Althought the three experiments show that SuggestBot can improve contributions, there's no experiment to show that quality of Wikipedia is improved by SuggestBot.
It would be interesting to see how to design an experiment to test the quality of wikipedia content.
non-reversion as measure of quality
One possible automated measure of quality is % of words that you contribute that survive future edits
The Question of Volume
I think the point you make here about volume shifting the degree to which you review messages is insightful. My experience with "SuggestBot" like features is limited to job search bots that suggests internships @ the Red Cross that I might be interested in applying to based on my interests, and initially I was excited about the availability of this feature. Although markedly different from the SuggestBot in a wikipedia setting, I did notice that there was a tipping point after which the messages I received from the bot stopped being useful and quickly transitioned to being considered spam.
This presents an interesting challenge to the following premise from the article: "reducing the cost of contribution will increase members' motivation to participate. One way to do so is to make it easy to find work to do (p.32). "
It is evident that in an effort to reduce the costs of contributions, you run the risk creating new costs of participation that may impede users' desire to participate. You risk getting on the nerves of your users...
Lisa McLaughlin
This is an very updated
This is an very updated study as my own interest. It has many things I can borrow to my studies. Here just a bit comments:
1. Routing is basically a recommendation thing. There is always a lot complexity in the matrix we are looking for. For example, just for the concept of "interest" it is can be positive and negative, because I bet people do want rate a movie they really like or really hate. If we just take rating simply as the indicator of positive correlation, it will mess up things. So at the beginning, when the authors thought about just asking user to work on the entries they had worked on in the Wikipedia, (following the logic in MovieLens), then realized it is not good at this particular situation; which is very important in recommendation algorithm building;
2. The inter-comparison among the three intelligent algorithms: apparently these three are strongly interrelated; it would be better to integrate them so i might suggest to test the inter-correlation as well.
Include provenance
While reading about how SuggestBot works, I thought that perhaps including more information on the provenance of suggestions (where did this suggestion come from? how come this was suggested to me?) would increase contributions even more. Some people might like participating in things that are similar to what they have done in the past, while other users might like to participate in things that their friends are also doing. Providing information about why a particular suggestion comes to you makes salient the outcomes that are relevant to you. In this way, cost of contributions is decreased and value of outcomes is increased. Both of these factors, according to the collective effort model as described on pg 33, would increase contributions.
This is what one online community goodreads.com does. Every day you get a list of the books that people on your friends list either rated or marked as "to read" or "currently reading". This list is a really a suggestion on books to add to your own library or to rate. Implicit in this list is provenance - you know that these suggestions are being given to you because they are books that your friends have read/rated. You even know which specific friend rated them.
Design claim: Giving people tailored recommendations for contribution tasks and telling them why they received those recommendations will increase participation.
I wonder if also, to branch
I wonder if also, to branch off from your design claim, it would help if people were allowed to add more "interests" to the set upon which recommendations were based. Say I have never edited articles on origami because I never come across the Wikipedia pages on the topic, having other sources I go to for origami tips. Maybe my interest in that subject can be harnessed if I tell the system the websites I frequent most often, or a list of topics I'm interested in, in addition to it looking at my list of edited pages.
Perhaps that's getting a bit outside the scope of SuggestBot, but I think it's worth thinking about anyway. :-)
Netflix does this and it works (at least anecdotally)
Your design claim would make for an excellent field experiment. One problem with recommender systems is the "black box" aspect of receiving recommendations. Sure, you know that it is based on people that have similar interests to you (in the case of MovieLens or Netflix), but there is no other fine-grained context given. Giving users more information of the recommendations allows them to understand how it came about, and consequently, how they might adjust to improve ratings further.
I've been a long time Netflix user. Until recently, there was no additional context for recommendation. You clicked on a "Movies you'll love" link and there were a whole host of movies you would supposedly like. But why these particular movies?
Without that additional information, I believe people lose faith in the recommendation algorithms when they receive ones that are "outliers". Recently (within the last 18 months or so), Netflix began listing why a certain movie was recommended – "you loved these 4 movies, and people that loved those movies loved this movie." When an outlier pops up, I can now at least understand why maybe it was recommended. Usually, it's because I forgot about an obscure, but excellent film I rated very high!
I think this would be broadly applicable to other recommender systems, like the suggestbot on Wikipedia.
what information to show about recommendations
Good design claim. Kristin Swearingen and Rashmi Sinha have done some work on what information to show when presenting recommendations, design suggestions included:
References:
SuggestBot today
SuggestBot can be found here http://en.wikipedia.org/wiki/User:SuggestBot
From the request history (http://en.wikipedia.org/w/index.php?title=User:SuggestBot/Requests&action=history), it seems it is still actively/continuously used by Wikipedia users.
A useful strategy for designers
I am sort of convinced by this paper that intelligent tash routing is likely to be most useful for communities where tasks are numerous and heterogeneous as it helps greatly in the information filtering. Systems that solicitcontributions from members can use intelligent task routing to increase contributions. Single algorithm may have large effects. Algorithms based only on the community's needs are less likely to interest members than that considers a person's knowledge and ability. Overall, intelligent task routing could be a very useful strategy for eliciting contributions from members of online communities. Designers should use this to build more valuable communities and better experience for people who inhabit them.
Design Claim from paper
This is an excellent paper, I think it is a showcase of using social footprints online to build new features. Such design are essential to manage the information overload boon that have come with Web2.0
Design claim: Recommend community tasks to members that match members' interests. Instead of randomly assigning tasks to members, recommend tasks that they like to do.
Goal: To reduce the cost of participation for members. If users are recommended tasks that match their interest they will happily complete the tasks quickly. They are also likely to have acquired knowledge in the domain related to task.
Context: This can implemented in large communities where data that indicates member interests are available or can be captured. In Wikipedia, member interest can be inferred from articles read, edited, added to watch list by the user. Each community might have a different method for capturing user interest. But in some communities it may be difficult or impossible to capture any data that will indicate interests of the users. In such communities this design will not be applicable.
Reduce cost and increase the value of outcomes
This is a very interesting reading. One design claim I found from this reading is that: “Reducing the cost of contributing and increasing the value of outcomes will increase motivation to do work for a community.”
This claim does make sense to me. I remember while I was doing the Wikipedia homework, I spent lots of time to figure out what kinds of issues/topics I can edit. If there is an intelligent task routing helping me to search for articles that I might be interested in, and then the assignment would be much easier for me, since I reduce the cost to search for articles.
It seems that the reading doesn’t pay much attention to the value of the outcome. In addition to the cost, I think people may be motivated to contribute more if the outcome is valued by others, since people care about how others perceive their work. For example, after I edited the page in Wikipedia, I didn’t see any comments or reaction from other people. This makes me feel that my contribution is really minor in the community. On the other hand, if Wikipedia can provide a figure to show how many people viewed this article or benefit from this article, I may be motivated to contribute more, since my work is really useful to some people.
Excellent point
Though it might reduce work on the less popular articles.
Maintaining Novelty
I think an interesting part of the reading described how users were annoyed at recommendations that were exactly what the user would edit (and already had). If you got a chance to see Bernardo Huberman speak a couple of weeks ago he mentioned that one thing required to keep users' attention is novelty. The algorithms that the researchers needed to develop were creative because they looked for articles that would be of interest that weren't already edited by that user. I think an overlying idea in community retention is maintaining that level of user novelty.
P.S. I dont' mean novelty in the negative sort of way that you might equate with, say, quickly disposable Facebook applications.
Task uncertainty
A very interesting article indeed. The main question that it presented to me is, “What is the work that we want done in our community and how do we make that work obvious to users”. In Wikipedia the tasks are specific and discrete; once and article has been written or improved it doesn’t need more work. Some topics will continue to evolve but many won’t require much work after an initial benchmark of quality is met.
In most of our online communities this will not be the case; responding to new users questions in forums will need to be done and submitting content will continue endlessly. These tasks aren’t known in advanced by the sites administrators but they do have the hope that these tasks will occur. If the goal of the community managers is to motivate users to engage in certain types of tasks it might be complicated to communicate these needs to users concisely. One idea to counter this might be to have visualization systems that allow users to view activity in the network according to a variety of metrics that support the administrators’ goals.
I have always imagined that Paradise will be a kind of library.
-Jorge Luis Borges
Trouble with Cosley
"SuggestBot treats multiple edits of an article as a single edit, collapsing the list of remaining edits into a set." (p5)
I don't understand this approach because if a person edits the same article many times, wouldn't this behavior signal an interest in the topic and not just the specific subject? For exaxmple, if you edit an article about your hometown many times, and an article about burritos once, wouldn't you be more likely to edit articles that relate to your hometown?
---
While testing SuggestBot, Cosley et al note two bad decisions: 1) randomly choosing members to receive recommendations (and neglecting Wikipedia's opt-in condition for most activities), and 2) posting links to suggestions rather than posting the actual suggestions (more clicks is bad)
(^p4)
This first gave me the impression that as far as Wikipedia norms go, giving members the option to get ("pull") suggestions is good and pushing suggestions on members is bad. However, there is still some degree of "push" in the revised scenario because "The first sizeable batch of users came from a post an early user made to a Wikipedia mailing list." (p7) The only difference appears to be who's doing the pushing - the stakeholders who created the bot (Cosley and friends), or the stakeholders who want a better Wikipedia ("the early user") I guess the obvious, common conclusion is that people are more willing to accept one of these push messages from a friend than from someone who stands to profit.
---
And what's the difference between a "moderator" and a "meta-moderator"?
meta-things
Meta-moderators moderate what has been moderated (by moderators). Just like a meta-search engine that searches the results of other search engines. Or a meta-analysis that analyses what others have analysed.
clarification
I think in the first version they started putting suggestions in user talk pages without users signing up. In the second version, users had to sign up. Advertisements encouraging people to sign up may have been "pushed" via email.
The (T 'n' T) Task and Team Effect
Intelligent Task Routing reduces search costs because tasks
can be recommended instead of searched. But at the same time the very notion of
recommending a task also sets a goal
for the user – the achievement of that task. Tying in what we read earlier if
this it is also disclosed that certain other members are also part of the same
task, it can help in two ways: easier and more natural socialization and
creating interdependence because of the teamwork involved in the task.
Design Claim: (the
Task and Team (TnT) effect)
Recommending relevant Tasks and a Team of people to solve
that task with to a newcomer of a community can increase contribution, involve the
member by setting a goal and create a stronger identity attachment to the
community.
It is also important in such a setting to ensure that there
are mechanisms in the system to appreciate incremental progress on the task-
which is especially important if the contributors are newcomers.
SuggestBot
The authors begin with:
"social science theory suggests reducing the cost of contribution will increase members’ motivation to participate"
If people would've edited something anyway, routing them to something useful is a good strategy (as is mentioned in the paper) but that definition only addresses on aspect of increasing work done, i.e. individual work done, not gaining more workers.
While reading this, all I could think of was NOT eCommunities, but how a SuggestBot would be valuable to me personally. (NOTE: this could be extended to any group of scholars that are working on a set of topics).
For example,
Tagging: tagging documents, personally or collectively, where some documents are under-tagged and others are over-tagged, could benefit from a SuggestBot that would make these disparities visible to researchers who could then task themselves to correct the metadata (archival practice could benefit from this greatly)
Citation Networks: generating citation networks always leaves dangling citations on the margins. Performing textual analysis (as SuggestBot does) to reccomend which citations should be followed (and thus which papers should be added to the network first).
Interesting:
"Intelligent task routing is likely to be most useful for communities where tasks are numerous and heterogeneous."
Worth exploring when and why this is true/false....
--------------------------------------------------------
PHartzog@umich.edu
--------------------------------------------------------
The Universe is made up of stories, not atoms.
--Muriel Rukeyser
They edit more
Your comment suggests that people are just shifting their contributions. But in fact they seem to have edited more. I suppose it is worth checking to see whether they edited more overall-- there could have been a substitution for effort on other pages that were not part of the random suggestion list.
Potential drawback
While I do see SuggestBot as being beneficial to motivating contributions on Wikipedia, one drawback I see is that it can really only be used by those who already contribute quite a bit. If you are a new user, you won't have any edits from which SuggestBot can make suggestions. If you are just relatively new and have made a few edits, SuggestBot could make suggestions but they might not be very relevant to your actual interests - maybe you edited a few random entries. It seems like the accuracy of suggestions would only really improve once a user had made a significant number of edits already.
If this is the case, then this user has obviously already found some sort of motivation to contribute. Already, a very small percentage (I forget the exact number) is making the majority of edits on Wikipedia - so why focus on getting them to contribute even more? In my opinion, what Wikipedia really needs is not more edits on arcane pop culture references, but edits by content specialists on technical or complex topics. So, the community would be served better by having a SuggestBot that could help you find articles no matter how many articles you had already edited - that way it could help motivate newcomers as well. A possibility could be a SuggestBot which allows a user to submit keywords of their interests, and from this generates a list of articles related to these keywords that need work.
or just mark a few articles
A better way to indicate your areas of editing expertise might be to tag a few articles. That way, you enable full-text matching (can use the whole article text as probe). It also fits with natural browsing behavior.
Personal invitations?
One thing that no one has mentioned is the power of a personal invitation to motivate. As compared to a suggestion from a bot, a request from a person is harder to turn down, and harder the better that the requestor is known and the more you want the requestor to like you.
A personal request is also more motivating, because it indicates that you are known to someone else (they know you exist and have some mental model of what they think you can do) and thus that they care about you. Of course, bad suggestions make you feel like they don't really know you. I think this is the reason we get so upset by bad recommendations from MovieLens or Netflix, a carry-over from how we would respond to bad suggestions from a friend or family member, where bad suggestions indicate they don't really know or love you.
a useful tool for a community coordinator
Good point, Paul. When I read this paper I immediately started thinking of how this could apply on different online communities i use, like SourceForge.net -- it seems like you could feasibly build out a similar recommender, but perhaps even more than wikipedia it seems like having a computer algorithm tell you "Project X has a lot of bugs that need fixing" would be a bit trite. It would be more useful to have a human coordinator who could use this system for a team working on a project aid in making suggestions.
Ideally, this system could help a coordinator manage his own backlog of needed work with suggetsions from the system, and route them to contributors in a way where they can be notified of work and express their preference on working on it.
With this system a coordinator could
- easliy identify what kind of work contributors "specialize" in
- weed out obviously bad suggestions
- prioritize suggestions among the contributors
- route tasks to contributors
The recommender then is less of a "suggestbot" and more of a guidence system for a community management tool.
--------------------------------------------------------------
oostendo@umich.edu
--------------------------------------------------------------
StumbleUpon: [Reposting from blog]
I can see how suggestbot could be a useful within the context of a
site. As with most systems, it has its limitations (which will be
minimized with time) but I find it a useful feature. On LinkedIn, it
gives a list of "possible" connections, on Amazon it gives "recommended
list", Netflix, Stumbleupon...even Google...all very useful services.
Interesting part is that it becomes better with use and time.