How Digg Works

The Internet hosts a wealth of information that no single person could possibly tackle in a lifetime. It would take an army to dig through the mounds of news reports and whittle the field to a manageable size, and then another army to pick and chose from that field the bits of information the average Web-savvy Joe might find interesting. And then another army to report back on whether the average Web-savvy Joe actually does find it interesting. Welcome to Digg.com, a user-driven news Web site that brings together hundreds of thousands of people to do the work of finding, submitting, reviewing and featuring news stories drawn from every corner of the Web.

Digg is sort of like Slashdot without the editors, bringing a democratic approach to the news-submission Web site. At Digg, the site's users make all of the content-related decisions. For the most part, the approach seems to be working very well. Kevin Rose, once an on-air personality for TechTV, co-founded Digg in December 2004 (with the help of Jay Adelson, former Digg CEO, and Owen Byrne). Rose spent $1,000 developing and launching Digg.com, relying heavily on open-source software. Six months later, the site had about 25,000 registered users. After a year, Digg had 80,000 registered users and 500,000 unique visitors per day. In March 2007, Digg hit the 1,000,000 registered user landmark. By 2008, some bloggers estimated Digg's user base at more than 2,700,000 unique accounts.

The huge Digg community is made up of users who play different, often overlapping roles. There are submitters who post news stories that they find in blogs, professional news sites and random postings around the Web. These stories land in the Digg queue. There are casual reviewers who look for interesting stuff in the queue and "Digg it" -- meaning they click a button to let Digg.com know they think it's cool. Once an article gets enough Diggs (and meets a bunch of other secret requirements), it's promoted to the homepage. There are truly dedicated reviewers who spend hours every day combing the queue to actively promote good stories and report bad stories (which will eventually get removed with enough reports against them). These people really drive what ends up on the homepage and therefore what gets thousands and thousands of people clicking through to read the story, sometimes crashing unsuspecting Web servers. Small Web sites and home servers can get crippled when 400 visitors a day suddenly turns into 5,000 in two hours. Even at HowStuffWorks, where our servers can handle the traffic, we can easily tell when we've been Dugg. When our stats show an increase over normal traffic of thousands of clicks per hour to a single article, we check the news-compilation frontrunners -- Slashdot, Fark, Yahoo! Buzz and Digg -- to see who's got it.

And finally there are the Digg readers, who make up the majority of Digg users and reap the benefits of the willing Digg army that promotes the best stories to front page. In return, the readers keep Digg in ad revenue and give the submitters and the Diggers something to do.

While some might call the premise revolutionary, the basic functions of the Web site itself are pretty simple and intuitive. It's easy to get started using Digg. When you go to the Digg.com homepage, you're already looking at the moment's most popular stories. They've been Dugg by enough users to get promoted to the homepage.

In the next section, we'll take a look at the Digg layout and registration process.

The Digg Layout

If you want to go beyond reading the homepage stories, here's a breakdown of the primary actions you'll take at Digg.com:

Register

Creating a Digg account takes only a few seconds. Once you have an account, you can access all of the Web site's features and take an active role in submitting and Digging stories.

Before submitting your story, you should specify what category it fits into.
(HSW Screenshot, no credit)

Browse and Digg

The Popular tab on Digg shows the stories that have received the most Diggs from users. Browse for stories within the Digg "Upcoming Stories" queue, and let Digg know which stories you like by clicking the "Digg" button to the left of each story title. (Every story you Digg gets saved to your account for later viewing, so you end up with a running list of everything you've ever Dugg.) The more Diggs a story gets, the better its chances of making it to the homepage. You can also browse the queue by category.

Digg also has tabs that let you filter feeds into news stories, videos, images and podcasts. You can even customize the categories that show up in your Digg view. Interested in the tech industry, but don't give a fig about motorsports? No problem. Just click a few check boxes and Digg will filter your stories so that you get exactly what you want.

Comment

If you discover a story you find particularly interesting and have something to add or would like to discuss it with other Digg users, just click the "comments" link beneath the story description. You can add your own comment at the bottom of the comments page.

Bury

As a Digg user, your help is appreciated in reporting duplicate stories (not allowed), dead links, incorrect stories, oldness, lameness and spam by clicking the corresponding link in the "problem" drop-down list below each story description. When a story gets enough reports, or "buries," it disappears from the Digg queue and only appears in search results and user profiles.

Submit

Finally, you can post a story to the Digg queue yourself and hope other users find it interesting enough to Digg it straight to the front page. It's actually a lot of fun to see if your story makes it. All you need to do is click "Submit a Story" on the upper right-hand side of the homepage, do a keyword or URL search and, if it appears your story hasn't been submitted yet, provide a title, a link and a short description of the story you're posting.

The submission immediately appears in the Upcoming Stories queue where anybody can see it.

It'll either get enough Diggs to move to the homepage or eventually disappear if it doesn't get enough Diggs or it gets reported by multiple users. In the next section, we'll look at special Digg features.

Special Digg Features

We've covered most of the basics, but there's more you can do with Digg.com. It's kind of an all-in-one news site, blog feeder and "social bookmarking" hub. Some of the additional features you can use at Digg include:

Digg Recommendations - Digg's Recommendation Engine uses a special algorithm to search for other Digg members who Dugg the same stories you've Dugg. You can use the engine to make friends or just keep an eye out for stories you might otherwise miss. You can use the Recommendation Engine by browsing in Digg's Upcoming tab, or you can switch to the "All" view and look at everything.
Digg Spy - Digg Spy offers a real-time view of Digg.com activity. You can watch as stories are submitted, promoted, Dugg and reported.
Social bookmarks - Add friends and keep track of their activities on Digg through your own profile page.
Podcast - Subscribe to the Diggnation podcast to listen to Digg's founder and his buddy discuss the most popular stories on Digg each week and address Digg-related issues. (See How Podcasting Works to learn about this populist medium.)
Blog connections - Digg offers one-click blogging of any story as long as your blog is hosted by Typepad, Blogger, Live Journal, Moveable Type or Wordpress. Just click the "blog story" link below any story description. (You can also add a "Digg this" button to your own blog posts so your readers can instantly submit one of your stories to Digg. It's not a Digg.com-created or -endorsed feature, but Digg doesn't seem to mind that people do it.)
RSS feed - Add Digg news to your own site through an RSS feed. You have several options of which stories to feed -- you can automatically add all Digg homepage stories, all stories you Digg or comment on, and all stories any of your friends Digg or comment on.
E-mail story - E-mail any story to a friend by clicking the "email this" link beneath the story description.
Block/report user - If you find that you don't like a particular person's submissions or comments, you can block that user so nothing he or she does appears in your Digg view. If that user is blocked by enough Digg users, he or she can get banned from Digg.

In addition to these special features, the developers at Digg are always working on new applications. The Digg Labs hosts applications that use Digg in unique ways, such as tracking the use of images across Digg or using different views to display stories. For example, the BigSpy application displays Digg stories using fonts of different sizes -- larger fonts indicate stories that have received more Diggs.

There's a lot happening on the user end of the Digg Web site. On the surface, it's a voluntary group effort that consistently produces a Digg homepage worth checking out -- a brilliant business strategy if you consider how much it would cost to pay people to perform the same jobs. Under the surface, Digg has 75 employees (with plans to double this number by the end of 2009) and hundreds of servers. In the next section, we'll check out some of what goes on the behind the scenes to make Digg work.

Digg Technology

For a Web site that receives more than 230 million page views a month (May 2008), Digg's technology framework is pretty streamlined. As with any proprietary system, Digg's technical department doesn't just put it all out there for everyone to copycat. But there are bits and pieces to be gleaned. The entire setup is based on LAMP (Linux, Apache, MySQL, PHP/Perl/Python), a programming model that ties together server applications running, say, PHP, with something like a MySQL database. It allows multiple languages to converse over a system architecture with minimal translation hold-ups.

In the Diggnation podcast recorded on June 14, 2006, Kevin Rose estimated the total number of servers in the area of 75. A 2008 post in the Digg technical blog stated that the company has between 1.8 and 2.5 times the minimum number of servers it needs to keep Digg running. According to the post, no one at Digg really knows how many servers the company actually has [source: Digg].

In an e-mail interview with HowStuffWorks, former Digg CEO Jay Adelson identifies the servers as Penguin Computing and Sun servers and says the Web servers are running PHP and the Debian Linux. operating system, and the database servers are running MySQL database management software. Digg actually doesn't have to store that much since it deals almost exclusively in text, but Adelson reports that the current setup is infinitely scalable. In a Mad Penguin interview in December 2005, Adelson says that Digg is "doubling the infrastructure every month to keep up with the demand." The basic infrastructure looks something like this:

Within this system, users submit and Digg stories and utilize all of the other features available on Digg.com. Digg doesn't use any cookies, only server-side storage, so all of your user data (past Diggs, friends, comments, etc.) are saved in your Digg profile, which is saved in the Digg database. You need to actually log in when you get to the site, which in practice may act as an initial security measure to ensure user validity for each visit. There are a number of legitimacy checks, which Digg calls "karma checks," built into the system at different points:

When a user attempts to submit an article, the system checks that the submission is legitimate -- that it's not spam (auto-submitted by a computer) and that the user is valid (not banned or otherwise limited). According to Adelson, the karma system also takes into account a variety of factors that includes certain "unique properties to digg's critical mass of users."
At the point when a submitted article is up for promotion to a category homepage or the front page -- which is determined by a number of factors, including the number and velocity of Diggs -- the system checks to make sure the Diggs are valid. One check involves looking for fraudulent accounts created only for the purpose of promoting a story. According to Adelman, "Our karma system knows the difference between users created to just digg a story and a user who has interacted with the site." The system also looks for auto-Digging, the computerized Digging of a story to fraudulently promote it to the homepage.

Adelman confirms that there is zero editorial control happening at Digg behind the scenes, whether in submission, promotion or burying (the removal of a story from the Digg queue). There is no censorship of submissions beyond letting a user turn on a profanity filter that blocks curse words. And Digg manages "buries" the same way it manages everything else -- with a proprietary algorithm. The system runs a "de-promotion algorithm" that determines when a reported story is ready to disappear from the main site pages.

All of this sounds very democratic and forward-thinking, with Digg moving us further down the path of the populist Web that turns regular Joes into entrepreneurs, reporters, editors, stock traders and encyclopedia contributors. But a bit of a hubbub in mid-2006 called Digg's utterly user-driven nature into question -- at least in the minds of a select and verbal few. In the next section, we'll look into the user response to Digg.

Digg Comments and Complaints

To quote the tech blog ForeverGeek's Macgyver in April 2006, "Digg is 99% fantastic." Although the blogger would later change his opinion of the site following a drawn-out battle with Digg founder Kevin Rose, most people will tell you that the original rating holds true. People race to be the first to post a great news story on Digg; Digg routinely features "diamond in the rough" stories that lead to the discovery of a little-known blogger who's doing quality work; the site's users, for the most part, seem to be genuinely and selflessly interested in promoting the best stories and burying the worst. In theory, the user-driven nature of the site creates a news venue that's difficult to corrupt, at least by large corporations or over-zealous editors. Of course, some would disagree, especially about that last point -- the presence vs. absence of editorial control was the basis for Macgyver's complaint, which we'll get into in a moment.

In any Web-based community, there are going to be complaints. In Digg's case, the biggest one for a long time was about the article comments, which are often just rude or silly, not thought-provoking or conversation-starting. But in the typical fashion, as the Web site has grown, concerns regarding the potential for abuse have grown with it. These concerns mostly deal with the fraudulent Digg activities we mentioned in the previous section -- spamming, fraudulent accounts and auto-digging. Another possible abuse involves the marketing potential of a Web site with no editorial control. Unscrupulous Web site owners could post seemingly irresistible stories just to get Digg users to click through to their Web site, generate page views and increase ad revenue.

One particularly odd abuse potential came to Digg's attention in March 2006, when a series of curious posts reported a rumor that Google was buying Sun Microsystems. Such a buy-out would presumably send Sun's stock price sky high. The article submissions appeared in quick succession, there were at least four in a single day, and Diggers promoted several of them to the front page. In each case, the Diggers appeared to be the same people. In fact, there was no truth to the rumor, and some have wondered if those posters were trying to use Digg to spread a false rumor that might boost Sun's stock price in order to make themselves some money.

So, going back to the "99% fantastic" rating, it would make sense to assume the other 1 percent might have something to do with the inevitable failure of Digg.com to catch every bit of spam, auto-Digging and other plain-old-evil uses of the site. It's dealing with more than a million visitors and 1,500 submissions a day. But Macgyver's ultimate complaint -- the one that started the mini-battle that was reported on tech blogs everywhere -- wasn't about any of that.

In the next section, we'll learn more about the Digg controversy.

The Digg Controversy

It started small and quiet. Macgyver, a frequent submitter to ForeverGeek and to Digg, discovered a strange thing on Digg -- two stories submitted by the same user with nearly identical Diggers in nearly identical Digging order got promoted to the front page, and one of the Diggers for each story happened to be Kevin Rose. While Macgyver drew no conclusions, the obvious one to draw was that Digg had missed an instance of auto-Digging. The added strangeness of Kevin Rose being one of the Diggers is more difficult to explain, and Macyger left it at that. He posted his observation on the ForeverGeek blog.

In a somewhat strange move, a ForeverGeek reader (not Macgyver) submitted the blog story to Digg. The next thing they knew, the story was un-Diggable (effectively buried) and the ForeverGeek URL was banned -- users could no longer submit ForeverGeek stories. The buried story soon disappeared from the site entirely.

You can adjust settings on your profile to create a customized view when you log in to Digg.
(HSW Screenshot, no credit)

Macgyver followed up with a blog post titled "Digg Corrupted: Editor's Playground, not User-Driven Website," and the story spread quickly. In the official Digg blog, Kevin Rose posted a response to the controversy, essentially stating that there was nothing funny going on. He said that he Diggs stories he finds interesting, and if auto-Digging was occurring in that instance, he didn't know about it. ForeverGeek was banned, he said, because it was breaking Digg policies against spamming and fraudulent accounts.

Macgyver didn't let it go, calling Rose's response a "(non)response" and continuing to question the very premise of the Web site -- its lack of editorial control -- in a running account of the event on ForeverGeek. Soon and without explanation, ForeverGeek was unbanned from Digg.com, free to submit at will.

This little unresolved episode aside, Digg seems to be doing just fine in terms of traffic and funding, and it has plans. The latest version of Digg includes new categories encompassing all types of news, not just science and technology, as well as a "Top Digg Users" feature that lets you check out the activities of the most active (and therefore influential) people using the Digg Web site. This type of tracking makes it easy to discover people you'd like to add to your friends list so you can keep up with their Digging activities.

For more information on Digg.com and related topics, check out the links on the next page.

How Digg Works