deep web
The deep web and dark web are terms that have enjoyed plenty of headlines in recent years. You could conclude wrongly, it turns out that nothing exists in that seedy cyberalley except for deals that involve contract killing, illegal drugs, human trafficking, weapons, pornography and other not-so-nice products and services.
The Silk Road, a website synonymous with shady dealings, was shut down twice by the FBI and its founder, the Dread Pirate Roberts, also known as Ross Ulbricht, imprisoned for life.
The deep web and dark web aren’t the same thing, though. The dark web occupies only a fraction of the deep web. If you want to visit the deep web, you might already be there. It’s any place you go for content without a permanent URL behind a paywall, password protected or dynamically generated.
The dark web, on the other hand, is kind of shy. You’ll have to go out of your way to find it. It is also invisible to search engines, so you probably won’t end up there by accident. If you do and are seen hanging around a certain type of website, law enforcement might show up with questions and handcuffs. Most websites are protect by Tor or some other brand of crypto.
Getting to them requires a special browser that we’ll discuss in a moment.
The important thing to remember is that, public reputation notwithstanding, the deep web isn’t all bad. Serious researchers love the volumes of archives found there. Others need its level of invisibility just to stay alive. For example, dissident journalists in China, Iran or a dozen other totalitarian states.
The Deep Web
Around 90 percent of online content is blocked from Google’s view because it requires authentication to access. In other words, it needs a username and password. Check out our guide to building a strong password to make a good one.
Examples include an online banking or email account, direct messages on Twitter or photos uploaded to Facebook and marked private. Search engines can’t reach that stuff and a simple Google search of your name won’t bring it into view.
To understand what separates the deep web from the surface web, or viewable search results, you have to understand how search engines, such as Google, discover web pages through the three steps of crawling, indexing and serving.
- Crawling
Since there is no central database of all websites, a search engine constantly looks for new webpages to add to its list through a process called crawling. Website owners would love to know how often Google visits a website but the only hint of an answer mentioned in official statements is the maddeningly vague “regularly.” We know. That’s hard to plan for.
When the Googlebot, which is the part of the algorithm that does the hard work, finds a new webpage, it saves the information to an index. That is where Google goes for results to display when someone types in a search term and hits “enter.”
Without an index to pre-categorize the contents of the internet, it would take so long to perform a real-time, line-by-line search that we’d get bored out of our skulls waiting and develop games, instead.
Google uses different methods to find new webpages. Among them are following links from known webpages to new ones, checking sitemaps or receiving tips from one of our best web hosting providers about a new webpage.
- Indexing
When a page is found, Google decides what it’s about through indexing. This process includes studying and categorizing the content, images and video embedded on the webpage. The goal is to guess the intended topic so that a search for how to knit a llama sweater doesn’t yield webpages on how to make a pig fly.
Google knows that it will stay the world’s most popular search engine only by getting this stuff right, so it’s sort of a big deal. After indexing, a webpage lands in the Google Index, which is a massive database stored across a huge computer network.
Serving
A search engine’s reason for being is to retrieve and present the most relevant information from its database in response to a user query. It does so by using an algorithm to evaluate on-page elements. That part of the process is why some of us lose hair and take heart medication trying to create optimized, keyword-rich content that snags a spot on page one of the search results.
The search engine optimization industry has grown up around gurus trying to guess how Google prioritizes factors to evaluate. Once the crawling, indexing and serving is complete, a webpage joins the tiny percentage of the internet that most of us know. You can easily find this content.
The Dark Web
The purpose of the dark web was to provide an anonymous forum for free expression for people in countries that suffer from oppression, censorship and intimidation. Anonymity is achieved by routing and rerouting online activity through a series of geographically dispersed IP addresses until it is virtually impossible to tell where the traffic originated, much like the best VPNs do.
The idea came from Naval Research Laboratory mathematicians. They called it the Onion Routing Project. Conceived to cloak U.S. intelligence, NRL released it to the public domain in 1997, where it became known as Tor. Accessing the dark web requires the Tor browser or something similar. We’ll talk about that in a moment if you’re still dead set on it.
The terms “Tor” and “torrenting” sometimes cause confusion for those meeting the dark web for the first time. As mentioned, Tor is a centralized, encrypted routing network built to keep users anonymous. Bittorrenting, or torrenting, is an unencrypted, decentralized file-sharing protocol that has little to do with security.
While Tor touts itself as a safe way to explore the deep web, the never ending game of coding and code-breaking should remind us that nothing provides perfect data security. Ask the Dread Pirate Roberts. You’ll find him sitting in a prison cell in Colorado.
The Dark Web Descends
It didn’t take long for the deep web to descend into darker territory from the lofty ideal of freedom through anonymity. The secrecy offered by Tor applies to those with legitimate purposes, as well as criminals. One study estimated that over half of the more than 5,000 Tor websites were involved in illegal activities.
Things you can do on the dark web to draw the attention of authorities include buying and selling malware, stolen credit card info, illegal drugs, weapons or child pornography. You can find assassins there, as well as hackers-for-hire. This may be useful if you want to destroy a particular computer belonging to, say, a former flame.
Perhaps no website better exemplifies the darkness than Silk Road, which flourished from 2011 to 2013. It did about $1 billion in sales, mostly in illegal drugs, before law enforcement arrived. When Ulbricht was sentenced to life in prison without parole, the judge hoped it would dampen the enthusiasm for black market goods on the dark web, but he should have known better.
Those who study this kind of thing have concluded that Ulbricht’s misfortune had the unintended effect of shining a spotlight on the profit potential in online trafficking, spawning a wave of cyber gangsters unfazed by the possible downside of a life sentence.
One “Wired” story describes Ulbricht’s rise and fall in fascinating detail. If you skipped ahead to find out how to access the dark web, you can start reading again now.
Using Tor to Surf the Dark Web
When you visit the dark web, it’s normally done through Tor, though other services, such as the Invisible Internet Project and Freenet, are also available. For Tor, visit the homepage to download and install the browser.
Note that there is no all-inclusive, Google-style search engine to guide your path. Tor websites end in .onion, but you’ll need to know where you’re going beforehand to find anything. Certain subreddits and wikis maintain lists of dark websites.
Only download the Tor browser from the official website. Getting it anywhere else risks inviting all forms of malware, spyware, or the latest nasty virus into your computer. Though Tor calls itself a secure and anonymous way to browse the deep, dark web, we recommend creating a virtual private network and Tor combo first.
There have been enough data-compromising incidents to make a prudent person suspicious that Tor alone might not be safe. A logless VPN paired with the Tor browser offers better identity protection. If you’re headed out for a picnic on the dark web, more security is better than less. Once your VPN is active, you can fire up the Tor browser, in that order.
This is the specific URL of your Facebook Messenger thread with a friend, or the Department of Justice’s public archival material, or Harvard’s internal communications system. The deep web is the majority of the internet as a whole.
About john john