Remembering the pre-Google web, when search was an experiment

It was an era when the term “discovery” carried actual weight, when you could authentically be one of the first people viewing a piece of content, regardless of its quality or how long it may have existed—an era of equal parts wonder and frustration. “Then a couple of years later, webrings popped up,” Radziwill said. “You’d get a block of code and put it on the bottom of your site’s HTML page, and it would embed a link to the next website related to this one. Someone else would manage the list of what could come next, so it was a great way to increase your exposure. The people putting together the lists for the webrings were generally pretty upstanding, so we didn’t even think about sabotage.” Webrings are another pre-Google artifact that recalls an era when microcommunities sprang up, not unlike those in the BBS era, from groups of people sharing niche interests, sometimes even local to one another in the real world. Before monolithic aggregators like Reddit, webrings were one of a handful of ways to find sites focused on the topics you were interested in. That mattered because the web had not yet become a pure retrieval machine. It still had a culture of exploration, and many services were designed to help users browse topical categories rather than fire a query into a universal index. In that world, a webring or directory was not a compromise. It was often the main event. “I was a sysadmin for an ‘ecommerce shop’ in 1995 and 1996,” Radziwill told me, “and when we would turn up websites for new clients, the highlight of our process was submitting the site to Yahoo. Yahoo was like the Yellow Pages, but only for websites. There was a form you would fill out, and you had to justify to the real people at Yahoo that this business you were submitting was legit and important enough to be in Yahoo’s main directory.” That kind of human curation is almost unimaginable now. “I remember one time submitting the website for a regional branch of the American Cancer Society and getting rejected because it ‘wasn’t significant enough,’” Radziwill said. “They recommended we contact the main ACS and have them link the site from their page… that they didn’t have yet.” It was a time when getting accepted into a directory like Yahoo by their human moderators was a massive badge of honor. But the model had obvious limits. Human curation couldn’t scale forever, and it became more expensive and less timely as the web ballooned and content outpaced curation. The moment the number of pages outstripped the number of people who could reasonably classify them, the future belonged to crawlers and ranking systems. Search engines of the era confronted the scale problem with software. Many of the first big commercial systems, like AltaVista, Lycos, Excite, and HotBot, used crawlers and indexes to automatically map a growing web rather than relying on editors to hand-classify every site. While these engines didn’t all work the same way, they shared a core ambition: to gather up as much of the web as possible and let the algorithm sort through the mess. It may sound obvious now, but it was a leap at the time. AltaVista in particular represented a major step forward, combining a fast crawler with scalable indexing software, and was already handling millions of HTTP requests per day shortly after launch. It made search feel less like browsing a catalog and more like querying a giant machine. AltaVista became one of the defining search engines of the 1990s because it was fast, broad, and unusually capable for the time. Later versions supported natural language-style searches and gave users the sense that the web could finally be approached as an indexable whole, even if the results were still rough around the edges. In a decade when many people were still learning what the web even was and grappling with its vastness, it felt close to miraculous. But it was still a very limited tool compared to the hyper-sophistication of an evolved engine like modern Google. Mark Friend, director of the IT support firm Classroom365 Limited, worked as a systems operator in the late ‘90s. He remembers that in the pre-Google era, searching the web was both a technical skill and an art form. Just as importantly, Google paired that ranking approach with a stripped-down interface and a results page that got out of the way. The clean design mattered because it reinforced the sense that search should be a utility, not a portal amusement park. In practice, Google made the search box feel like the front door to the whole web, with the same intrinsic, vital importance that a front door serves in a home. In retrospect, the pre-Google era was slower, messier, and in many ways less efficient. But it also had more competing ideas about how information should be found, categorized, and judged, alongside a lawless atmosphere that suggested unlimited potential. Some systems trusted people. Some trusted crawlers. Some trusted directories, and some trusted questions phrased in plain English. “The Internet contained an inherent level of clutter and chaos,” Friend remembered. “But at the same time, it was a much more humanized place before. It’s why many of us feel nostalgic for that time. The Internet had an artisanal feel to it. It was made by real people using text editors such as Notepad and Dreamweaver. You might start at one destination and find yourself at a fan site of an obscure band, and then end up at a forum discussing vintage synthesizers, and ultimately end up on a NASA page.” That diversity is worth remembering because it shows that search was never necessarily destined to look the way it does now. The pre-Google engines weren’t just failed precursors. They were serious attempts to address a problem that the web had made urgent but not yet solvable in one obvious way. Google won by combining technical rank, usability, and scale at exactly the moment the rest of the Internet was ready to abandon the old order.