What associations do the words “deep web” conjure in your mind? A place for godlike hackers where mere mortals should fear to tread? A den of iniquity? If so, it may come as a surprise that you use the deep web every single day.
In fact, although the term deep web is easy to confuse with the similar-sounding dark web and darknet — and weirdly, the documentary film Deep Web was in fact about the dark web — we think it’s worth untangling the thread.
Deep web and surface web
By way of illustration, let’s imagine the Internet as one big city. Like any metropolis, it contains public spaces open to everyone, such as the streets, boulevards, and parks you’ll find on the map. Anyone can go there and look around; 360-cam cars can see them; and you can easily find them on a map such as MapQuest or Google Maps.
On the Internet, these public spaces are known as the surface web. They are Web pages, Web applications, and other online elements that search bots — digital analogues of cartographers’ cameras — can index. They may hold documents, media files, and more. Anyone can find them using a search engine and view them without paying, registering, or installing special software.
In addition to public areas, cities have private zones that require a pass, ticket, or invitation for access. These include homes, business centers, private clubs, cinemas, and so on. Usually, no publicly available map will show you what’s going on inside these places.
The Web, too, is home to many corners that Google, Bing and others do not peer into. Collectively, such places are known as the deep web. They consist primarily of all of the Internet pages that cannot be searched and opened by normal means, and bot-cartographers cannot index them.
If a site requires you to enter a CAPTCHA for access, then a search bot won’t be able to learn much about its contents — the point of a CAPTCHA is to weed out bots, after all. If an article is available only by subscription, a bot can’t open and index this article because bots have no accounts or money to pay the fee. Reading a document requires a password? Again, no luck for a bot: it doesn’t know a password.
If something cannot be found, even if it can be opened, then it too belongs to the deep web. If you configure Facebook to hide your profile from search engines, for example, then even if a search bot gets to it, it will have to ignore it. Nor can a search engine process content that a Web page generates only when the page is opened and that varies depending on who opens it. For example, to view personalized offers, you have to be a user with certain digital habits.
Finally, deep web also refers to all content to which no links exist from the visible or surface web. A search bot simply does not know that such content exists; it finds new pages by following links from the pages it’s already indexed. Just as a Google Street View car cannot get into a private courtyard, search bots cannot come across unlinked content.
As you see, the bulk of the deep web is made up of harmless, even useful web pages and documents that most of us use. There’s nothing wrong with them being off-limits to outsiders. Quite the opposite.
Dark web and darknets
Both in town and online, privacy is sought not only by upstanding citizens, but also by those looking to hide their not entirely legal activities. In the physical world, we think of shady business as taking place in slums and criminal dens — places selected for their lack of foot traffic and that are not marked on public maps. The addresses and whereabouts of each den are known to a limited circle of individuals, although many are aware that they exist somewhere.
That is approximately how darknets — restricted-access networks used chiefly for questionable activities — operate. The nodes of each individual darknet (servers, computers, routers) are invisible not only to search engines, but, because they use nonstandard protocols to transfer data, to most browsers as well. Neither a direct link nor a password will get an ordinary user in.
Together, darknets make up the dark web — generally considered a haven for murky characters such as drug traffickers, arms dealers, extortionists, and sellers of stolen data. Many people know that the dark web exists, but few know how to get there.
Of course, hackers and criminals aren’t the only people who need secrecy. Dissidents, free speech activists, whistleblowers helping investigative journalists, and many more people use the dark web to evade persecution and communicate anonymously online. Some people go there for protection from online data collection. Secure and affordable tools exist for that purpose, but some prefer a more radical approach.
All shades of Internet security
It’s no bad thing for data to be buried deep, invisible to those for whom it is not intended. If, for example, corporate correspondence were to find its way into a search index, the consequences could be unfortunate. It’s better to protect your portion of the depths — accounts and documents to which only you have access.
- Always use strong and unique passwords. And if you have too many accounts to remember all of them, use a password manager to help you keep them under control.
- Always confirm you’re exactly where you want to be before entering your credentials online. For example, if the URL is misspelled or looks like a meaningless jumble of letters and numbers, the page is not to be trusted.
- Give access to confidential documents only to those who really need it.
- Avoid the dark web unless you’re completely comfortable telling the difference between a forum for human rights activists and one for hackers.
- Use a trustworthy security solution that will keep you out of trouble wherever you go online.
