The deep Web is only getting deeper. Its store of human knowledge and trivialities grows more massive every day, complicating our efforts to make sense of it all. In the end, that's perhaps the biggest challenge behind the Internet that we've created.
Programmers will continue to improve search engine algorithms, making them better at delving into deeper layers of the Web. In doing so, they'll help researchers and businesses connect and cross-reference information in ways that were never possible before.
At the same time, the primary job of a smart search engine is not to simply find information. What you really want it to do is find the most relevant information. Otherwise, you're left awash in a sea of cluttered data that leaves you wishing you had never clicked on that search button.
That's the problem of so-called big data. Big data is the name for sets of data that are so large that they become unmanageable and incoherent. Because the Internet is growing so quickly, our whole world is overrun with data, and it's hard for anyone to make sense of it all -- even all of those powerful, all-knowing computers at Bing and Google headquarters.
As the Internet grows, every large company spends more and more money on data management and analysis, both to keep their own organizations functioning and also to obtain competitive advantages over others. Mining and organizing the deep Web is a vital part of those strategies. Those companies that learn to leverage this data for their own uses will survive and perhaps change the world with new technologies. Those that rely only on the surface Web won't be able to compete.
In the meantime, the deep Web will continue to perplex and fascinate everyone who uses the Internet. It contains an enthralling amount of knowledge that could help us evolve technologically and as a species when connected to other bits of information. And of course, its darker side will always be lurking, too, just as it always does in human nature. The deep Web speaks to the fathomless, scattered potential of not only the Internet, but the human race, too.
Author's Note: How the Deep Web Works
The Deep Web is a vague, ambiguous place. But while researching this story, it was easy to conclude at least one thing for sure -- most news headlines tend to sensationalize the dark Web and its seedier side, and rarely mention the untapped potential of the deep Web. Articles about illegal drugs and weapons obviously draw more readers than those detailing the technical challenges of harvesting data from the deep Web. Read the negative, breathless articles with a grain of salt. It's worth remembering that there's a whole lot more to the deep Web than the obvious criminal element. As engineers find better, faster ways to catalog the Web's stores of data, the Internet as a whole could transform our society in amazing ways.
- Bagot, Martin. "Deep Web Criminal Marketplace Completely Disappears After '58 Million Heist'". The Mirror. Dec. 3, 2013. (Dec. 6, 2013) http://www.mirror.co.uk/news/technology-science/technology/sheep-marketplace-deep-web-criminal-2879995
- Bergman, Michael K. "White Paper: The Deep Web: Surfacing Hidden Value." Journal of Electronic Publishing." Aug. 2001. (Dec. 6, 2013) http://quod.lib.umich.edu/cgi/t/text/text-idx?c=jep;view=text;rgn=main;idno=3336451.0007.104
- Berman, Fran. "Deep Web Film Will go Inside the World of Bitcoin, Silk Road and Beyond." Mashable. Nov. 20, 2013. (Dec. 6, 2013) http://mashable.com/2013/11/20/alex-winter-deep-web-documentary/
- Biggs, John. "Deep Web Users are Ready to Launch Silk Road 2.0." Techcrunch. Oct. 4, 2013. (Dec. 6, 2013) http://techcrunch.com/2013/10/04/deep-web-users-are-ready-to-launch-silk-road-2-0/
- Bingham, John and Kirkup, James. "David Cameron Wins FBI Support for 'Dark Web' War on Paedophiles." Telegraph. Nov. 18, 2013. (Dec. 6, 2013) http://www.telegraph.co.uk/news/uknews/crime/10456108/David-Cameron-wins-FBI-support-for-dark-web-war-on-paedophiles.html
- BrightPlanet. "Understanding the Deep Web in 10 Minutes." Mar. 12, 2013. (Dec. 6, 2013) http://www.brightplanet.com/2013/03/whitepaper-understanding-the-deep-web-in-10-minutes/
- BrightPlanet. "How Does Data from the Deep Web go from Results to Actionable Intelligence?" Jan. 31, 2013. (Dec. 6, 2013) http://www.brightplanet.com/2013/01/how-does-data-from-the-deep-web-go-from-results-to-actionable-intelligence/
- BrightPlanet. "The Future of Online Search." Aug. 27, 2013. (Dec. 6, 2013) http://www.brightplanet.com/2013/08/the-future-of-online-search/
- Couts, Andrew. "TorSearch Makes Finding the Next Silk Road a Lot Easier." Digital Trends. Oct. 11, 2013. (Dec. 6, 2013) http://www.digitaltrends.com/web/torsearch-tor-network-hidden-services/
- Couts, Andrew. "That Was Fast: 2 Sites Vie for Silk Road's Criminal Customers." Digital Trends. Oct. 3, 2013. (Dec. 6, 2013) http://www.digitaltrends.com/web/silk-road-fbi-ulbricht-sheep-marketplace-bmr/
- Gallagher, Diane. "Peeling Back the Layers of the 'Dark Web' WCNC." Nov. 19, 2013. (Dec. 6, 2013) http://www.wcnc.com/news/local/The-Dark-Web-232606871.html
- Here and Now. "The Deep Web: Where Google Won't Take You." WBUR.org. Nov. 8, 2013. (Dec. 6, 2013) http://hereandnow.wbur.org/2013/11/08/the-deep-web
- Hockerson, Lauren. "TorSearch Offers Private Search of the Deep Web." Gigaom. Oct. 11, 2013. (Dec. 6, 2013) http://gigaom.com/2013/10/11/torsearch-offers-private-search-of-the-deep-web/
- Lederman, Abe and Lederman, Sol. "Understanding Deep Web Technologies." New Idea Engineering. Jun. 2004. (Dec. 6, 2013) http://deepwebtech.com/PDFs/Understanding%20Deep%20Web%20Technologies.pdf
- Love, Dylan. "There's a Secret Internet for Drug Dealers, Assassins and Pedophiles." Business Insider. Mar. 6, 2013. (Dec. 6, 2013) http://www.businessinsider.com/tor-silk-road-deep-web-2013-3?op=1
- Seymour, Andrew. "Anonymous 'Deep Web' is New Frontier of Child Exploitation, Conference Told." Ottawa Citizen. Nov. 16, 2013. (Dec. 6, 2013) http://www.ottawacitizen.com/business/Anonymous+deep+frontier+child+exploitation+conference+told/9175718/story.html
- Swift, Tim. "What is the 'Deep Web'? And Other Questions About the Shadowy Virtual World of Silk Road." Baltimore Sun. Oct. 3, 2013. (Dec. 6, 2013) http://articles.baltimoresun.com/2013-10-03/business/bal-silk-road-deep-web-explainer-20131003_1_satoshi-nakamoto-bitcoin-silk-road
- University of California, Berkeley. "Invisible or Deep Web: What It Is, How to Find It, and Its Inherent Ambiguity." (Dec. 6, 2013) http://www.lib.berkeley.edu/TeachingLib/Guides/Internet/InvisibleWeb.html