I don't think it's too late to fix this. We can build better search engines, or better yet, tools more suitable tools for discovery. It doesn't have to be big to be useful.
Right now the big problem as I see it is that there is great content being created but its audience just isn't finding it. There's so much noise, the signal is almost completely drowned out.
Out of largely the same frustration permeating the link, I built search.marginalia.nu alone in less than a single year in my spare time without prior knowledge of search, paying for the hardware out of my own pocket.
I don't say that to promote how great I am. The point is specifically that I'm just some guy on the Internet. Just imagine what could be possible if actual resources were put toward finding solutions to this problem.
> Right now the big problem as I see it is that there is great content being created but its audience just isn't finding it. There's so much noise, the signal is almost completely drowned out.
It's a curation problem. We have billions of pieces of content being created and only a handful of curators to bring that info to us. All the big search engine players want to be "the one" but it's a fools errand IMO.
No one search engine will be able to become "the one" because there is simply too much information. Google with all of their cash in the bank and talented engineers can't even crack the problem. The problem though is even though each search engine offers something different, they all think they offer the same thing and compete to muscle out any smaller players.
So you get a situation where you want a metal roof for your house, metal roofer gives you a quote. Big roofer guy in town hears about this and runs the metal roofer guy out of town, then tells you he cant offer you a metal roof anyways.
PageRank worked when humans were creating content across a huge number of sites. Once you start consolidating the humans (Medium), then the organic content and the machine generated content start to blur together. And the machine sites can collude to make their content look more organic than organic. Today there's 10x as much of it so if only 20% of the organic content is high quality, that's practically in the noise floor for the sorts of graphs I stare at in any given week.
Perhaps what's missing here is a garbage collection algorithm. Whatever they are using as 'roots' is not working, and they can no longer infer trust by association. They need to curate and review sites. Or maybe they already do, but the humans' focus is directed by heuristics and those heuristics have been gamed.
Does anyone know if Google still has a functioning QA department? Perhaps this is the same disease we see everywhere of developers thinking they understand testing when really we know just enough to be dangerous.
PageRank worked when humans were creating content across a huge number of sites.
PageRank worked when there was very little tangible value besides vanity in being number one for a particular search term, and when there was little competition between sites. Once SEO became a key strategy to growing a business PageRank had to shift from finding the most relevant results to removing the things that were pretending to be relevant results. That challenge is much harder.
I contend that practically every update to Google Search over the past 15 years has been to try to defeat SEO spam. There doesn't appear to be anything else. And, unfortunately for all of us, it's a battle Google are not winning.
Even worse, they are completely disincentivised to do anything about it, since presently the ads occupy most or all of the real estate most users see anyway. Buying ads to show up in that section so you aren't drowned out by spam is exactly what benefits GOOG's shareholders. And I say this as a GOOG shareholder, lol.
I've come to believe that the organic results are a niche product that Google could eliminate tomorrow and most consumers wouldn't really realize anything had changed. They'll go to the first result 99% of the time, and 99% of the time that's an ad.
I agree and disagree. From my view it's like they went mole hunting with a bazooka - and kept making better bazookas. The collateral damage is that many good sites were taken out, or downgraded.
They got a lot of bad sites, but they got rid of a lot of good ones too.
Now google is the modern yellow pages imho. It's more shallow than ever, but it's okay for many, and they can shake down legit sites for adwords to show up.
Totally agree with the main article, sadly however it's like what, 90% of people never go past the first page? And many still assume the top ads are vetted results that are better than all the rest.
Looking forward to leaving google as the yellow pages and encouraging more niche search engines, web rings, and similar.
Indeed - and a glance at the ads on even mainstream sites sure back that up. They're mostly shilling casino mobile games, scam tech (like 'portable air conditioners' that are just a fan that blows over a wet sponge for $75) and the big one, unregulated dietary supplements sold with disgusting imagery and "One weird trick" ad copy. It is almost entirely just utter bottom-feeding trash. And this is what's paying the bills for 'reputable' sites like CNN, Bloomberg, CNBC, whatever.
I wonder if it was a veiled reference to the debate between Linus and Andrew Tanenbaum, who wrote MINIX based on OS research that, according to him, showed that microkernels are better than monolithic kernels.
what is the ratio between dynamic and static content amongst say the
top100 web sites on the web? Can we say 0% static pages? [yes, i accept
Dean Gaudet's point that webpages can be split up into dynamic and static
parts, probably that will be the future.]
My point was that as a newcomer in either operating systems (Linux late last century) or a new search engine (Marginalia, Kagi) you cannot always win by repeating the same stuff incumbents do.
Especially not when the incumbents are busy digging deeper holes.
Yeah I'm very much betting on the null hypothesis when it comes to fancy machine learning-based methods and implementing the most bog-standard keyword search you could imagine.
It's not perfect, but it's scary how well it works sometimes.
I assume they're talking about all the OS research done at places like XEROX and Bell Labs that never really came to fruition, but formed the basis for modern OS's.
Ah yes, the amazing hn cynicism - it's not that the problem is really hard, and there is a really complex system here with all sorts of incentives, variables, and issues on all sides.
Instead, everything is easy, and the reason the "right" thing (as defined by the poster) doesn't happen can be reduced to some single cartoon-villain level bullet point like "not in business interest".
Small problem: There are plenty of folks, foundations, companies who are not resource limited in attacking something like this, yet it still doesn't happen.
So maybe it's not as simple as you think?
Please, the cynicism pointed in the Googs direction is well deserved. It is clear and obvious by their behaviour and changes to company ethos that serving ads is priority #1. Any thing that diminishes that ability will not be tolerated.
Just look at the recent (as of yesterday) stories of the changes that Googs is pushing to re-enable tracking that has been previously manually/intentionally disabled by the users. Look at how Maps is forcing users to enable wifi in order to provide routing directions. This one happened to me personally as well.
So you don't at all address whether the problem is hard, and instead deflect completely.
The wifi thing is just a silly example. The wifi is required to do fast start GPS. 99.99999999% of their users probably don't want slow start GPS. If you do, sure, it's no longer the app for you. Yes, they are optimizing for the customers they want and giving them a good experience . I'm not sure exactly how this is something abnormal. Like, you are basically complaining they won't add knobs for you. That's pretty standard in good products - but also means if the knobs they aren't what you want, it's not a good fit for you.
I got the email in the latter, and that is a super weird take. They had one control for literally everything, which was owned by admins. They almost certainly got complaints that turning it off it made search suggestions not work in sheets/docs/etc.
So they introduced a new separate control for that, and made it controllable by the user. Because it's controllable by the user, it also means the admin can't see the doc search history of the user, only the user can. Nor can they delete your history for you - only you can control it. Your auto-delete preferences, etc apply to it.
None of this seems bad or horrible to me. The only thing even worth picking a bone about is whether the new setting should take it's default from the old setting or not. There are issues in there of course, because as mentioned, before admins could turn it on/off for everyone - and this led to lots of complaints. do you take the admin setting? Do you take the setting the user had if it hadn't been overridden by the admin? etc
The claim, in that story, i that it is forced on by default even if you had it off. This claim is wrong, afaict.
It was not worth wading into that clear firestorm of a thread.
The support doc clearly says:
"If you disabled Web & App Activity in the Admin console, it is turned off for Google Workspace users. Users can turn it on if they want."
IE if the admin had it disabled, it will be turned off.
It also says if you had it disabled yourself, it will default to off.
It goes through this quite clearly if you click "what if i had it disabled" and "what happens if i don't do anything".
Overall presenting this as "re-enable tracking that was manually/intentionally disabled by users" is a real stretch.
Except both the comments you reply to are talking about a specific problem and context.
If you are saying that you are just claiming "in general, cynicism is warranted", that seems like a generalized opinion that doesn't really add anything interesting to anything.
It'd be much more interesting if you would engage with the actual conversation being had.
I did. The conversation stated that the cynicism was not warranted. Just because you brought it up, but now don't want to talk about doesn't mean it wasn't part of the conversation. I did not just randomly reply to a comment with spambot level lack of context.
It definitely feels like you did, though you don't seem to believe it.
FWIW - There is literally no claim anywhere that the cynicism was unwarranted.
Only that it was typical of HN.
So like you are making up a strawman and then beating it up, while everyone else is having a conversation about a different thing.
I'm simply inviting you to contribute to the latter rathr than the former.
I have posted in other comments about this subject, and have kept the two topics separate. You seem to be the one unable to handle that people have different views from yours. Not sure why/how you are unable to realize this. Continue the tone deaf hurrahing for the Googs if you must, but it is quite tiresome.
This seems very pedantic, and doesn't change what i said at all.
I'm happy to change it to "wifi helps with a bunch of features they have". AGPS is not even the only one. Wifi scanning is happening in the phone anyway to know when to switch networks, etc, so they are saving power when it is used instead of cell phone.
Not when I've turned off the WiFi to conserve battery while out and about not anywhere near wifi signals. Unfortunately, because I've decided to use Bluetooth earbuds, I now leave the bluetooth radio on as well. Until then, that was kept off specifically to avoid bluetooth tracking.
GPS is just fine for Garmin, TomToms, etc. Why suddenly for a phone that has WiFi in them Maps suddenly "requires" wifi is just not convincing me of anything other than Googs wants all the data.
> I'm happy to change it to "wifi helps with a bunch of features they have".
This does change what you said, because "a bunch of features they have" is not at all the same as "GPS, a feature which pretty much everybody wants to use and wants to start fast".
> Wifi scanning is happening in the phone anyway to know when to switch networks, etc
Not if wifi is turned off. If I'm in my car, for example, I might well turn wifi off to prevent the phone from wasting power scanning for wifi networks that aren't there. But I'm still going to have my cell phone connection on.
> they are saving power when it is used instead of cell phone.
Nonsense. See above. Plus, the cell phone antenna is always scanning too, but on a phone it is far less likely to be turned off to prevent that than wifi is.
What I really can't understand is how any of this is connected to your (correct) claim that search is hard and content curation is hard and so we shouldn't expect Google, or anyone else, to just magically solve it. Google could easily serve the needs of users much better than it does, even if it can't do it perfectly. But it won't because it's not in the business of serving user needs, it's in the business of serving advertiser needs, and serving users is just a by-product.
If Google really wants to solve the problems of users, instead of advertisers, it should have users pay them instead of advertisers. But of course it won't since it has spent more than two decades now getting users used to the idea that search and other "hard" services should magically appear for free, and hoping those users won't notice that they are actually the product, not the customer.
"Google could easily serve the needs of users much better than it does, even if it can't do it perfectly. But it won't because it's not in the business of serving user needs, it's in the business of serving advertiser needs, and serving users is just a by-product."
This is such a uncharitable assumption i don't even know where to begin.
Nobody can serve everyone well. Trying is a quick ticket to failure.
The choice not to serve certain people is clearly the correct one, even if one could theoretically do more than you do now. Why isn't the simpler, more charitable answer that they are just serving the majority of their large userbase, or whoever in that user base they care about? Why is it everyone who feels like they aren't served has to decide the reason is because someone is doing the wrong thing, rather than "guess i'm not the user they are looking for".
That is always going to make someone upset, because yes, you could do something more than you do for them. That doesn't make that path make any sense. Trying to do more of everything for everyone is pretending that you don't need to make choices, and that those choices don't have consequences (for example, what users you will retain or not). That is, as i said, a quick ticket to failure.
I never claimed that anyone could. You are attacking a straw man.
You are completely ignoring the actual point I made, which is that Google is a business, and every business serves the needs of those who pay it money, and in Google's case that is not its users, it's advertisers. So serving user needs has to be a by-product.
> Why isn't the simpler, more charitable answer that they are just serving the majority of their large userbase, or whoever in that user base they care about?
Why isn't the simpler, more obvious answer the one I gave, that Google, like every business, serves its paying customers first, and users are not paying customers?
You seem to be going to extraordinary lengths to ignore this obvious fact.
On the contrary, yours is a befuddlingly charitable explanation. Why on earth would an advertisement company like google, a joint-stock company, be "trying to serve the users' needs"?? They make money by pushing as many ads onto users as possible, almost by definition they must be doing the opposite of serving the users' needs x)
Yes, it's everyone else's fault they don't cater to you. How dare they.
Why do you think you know their users better than they do? What data do you even have?
The much simpler explanation, given that the product doesn't seem to be failing by any measure, is that maybe you just aren't who they are targeting?
One of the main reasons products fail is because they try to please everyone, and have so many knobs that the UI is horrible.
Good product design requires choice. You don't like those choices. Okay, awesome, it's not the product for you. That doesn't make it the wrong choice, or a bad design. It just means you aren't in the user base they are catering to.
Most of the time, if it turns out the user base thy are catering to is unsustainable, the product will fail, and something that does cater to that user base will take over. So how is this not a uniformly good thing?
The market has products for people who like different things - those users will decide what wins or not.
Clearly, being a google hiring person means you will never see the other side of this. So I really feel like every reply is just wasted electrons.
Google does not need WiFi in order to give directions to locations in the Maps application. However, gaining access to WiFi locations is beneficial to Googs in other fashions. They choose to force people to do it for their gain, not the benefit of the Maps user.
Now, you're trying to say that because some people are technically aware about things and have become vocal and push back against the tide that is Googs' overreaching into people's lives is some how in the wrong. You also seem to be implying that the fact that the vast majority of people are not vocal on the point makes it okay even though they have no idea what is actually occurring. Since they are not complaining, everything is okay, right? That's not how that works.
Serious question (I don't mean this as an attack): is online astroturfing part of your job description? It's the only way I can make sense of the tireless apologetics you've been doing all over this thread.
1-99.99999999% of people on earth is 0.8 of a person. Forgive me if I don't buy into your thesis. Seeing as the vast majority of people don't even understand the implications and extent of always-on pervasive ubiquitous tracking and surveillance, an honest metric would be: what percentage of tech-literate people don't want to be forced to have wifi-tracking enabled to access a feature which doesn't need wifi-tracking. As you can see from this forum, that percentage is quite high.
I don't think this is particularly cynical. Ads are how Google makes money. Show results with many ads, make money. Show results with no ads, make no money. I don't think it's an example of mustache twisting villainy, but rather their business model. I can't really blame them, either.
Same reason electrical cars took ages to take off before Tesla. The status quo companies (and most are, because nobody likes a disruptor) dragged their heels to no end before that.
Is it though? There are plenty of examples of businesses or even entire industries trying their level best to slow down changes and innovations that will hurt their profits. And in many cases the products may become worse, but more profitable, over time. In the 1970's, you could buy a washing machine in the United States that would last you 25 years. Try doing that now. I'm sure a lot of folks would like their washing machine to last two and half decades, but that's not that profitable if you're in the business of selling washing machines.
How does that reconcile with google results getting worse over time then, with worse defined as relevancy vs. profit motives? With UI changes that de-emphasize the results and promote the ads? With pulling in cursory summarizes to keep you on a google property and isntead of qucily & easily getting you to the definitive source?
There is lots of support for assuming the worst of google in this case.
Because it's not just Google. Every web search engine has been getting worse; it's why Google remains "the best" in every head-to-head comparison. The whole web has been getting harder to search.
But that's just it - Google's business interest is always about protecting revenue from search advertising, which is never simply aligned with the search user or with search quality, but exactly as you describe it "a really complex system here with all sorts of incentives, variables, and issues on all sides".
It’s perfectly reasonable to speculate about whether search quality is sacrificed in the interest of some incentive Google is trying to balance in favor of its own revenue stream. The fact that it’s complex is immaterial.
...or they are different problems? Google is really good at solving their problems, maybe they only care about quality search results as related to financial incentives?
Who even "gets content" through searching? I'm sure many people here are all basically using the internet through link aggregators (reddit, fb, HN etc).
Has the web ever been much beyond that? Have people ever really been just hitting "give me a random web page about X"?
Of course search engines are very useful, just that these conversations about content and "the web", I feel like there's not much focus on how so much traffic is guided through link aggregation.
I can't tell you the number of times I've wished there was a subreddit or a facebook group for the types of content I'm interested in, only to find that there isn't. Not because the content doesn't exist, mind you.
Link aggregators are also very much part of the problem. Besides the astroturfing, they tend to favor the lowest common denominator in content. Popular vote just doesn't seem to be a good way of surfacing quality content, especially not with the social incentive of winning internet acclaim for submitting a good result.
HN does surface the odd gem, but why can't I pick one of those and say "Hey, this topic interests me, show me more like this"? I'm currently working on build something to that effect, and so far it seems at least reasonably possible.
RSS was good in one way, but it also sort of turns everything it comes into contact with into a blog.
>> I can't tell you the number of times I've wished there was a subreddit or a facebook group for the types of content I'm interested in, only to find that there isn't. Not because the content doesn't exist, mind you.
Why not start it? One of the beauties of reddit subgroups is that not only do you find random very specific ones, but if you don't, you can start your own.
I often search for articles/books about X, or reviews about Y. The top results for those things is a mix of SEO spam, Amazon links etc. which I all try to avoid.
I use search for information, but I often have to dig quite deep to find something worth clicking on or reading. And that's true for Google and DDG (which I mostly use).
When it's raw, basic knowledge, then I'm using !wiki or !mdn etc. in DDG to search those sites directly. Note: Wikipedia and Mozilla Developer Network are two of my favorite sites.
But when it's something more subtle and specialized that I'm not very familiar with then often my only hope of finding something useful is to search via aggregators and forums like HN, Reddit etc. because then I can gauge the quality of the links via the discussion around it and the people who post them. Or in some cases I know about a specialist or journalist and can use their name to filter out spam.
Discovering these kind of things via search is really hard, especially if their authors are not known. It's the spammers and low effort creators that get rewarded the most, partly because of search engines, partly because of visitors who are content with that kind of quality.
Whenever I'm searching for something (using DDG, too) I usually scroll down the results page until I see a website that I don't recognize or one I remember as being small and targeted. The big company websites usually have shortest blurbs on the topic and the worst "NoScript minigames". Smaller websites usually have more information and less reliance on massive ad networks. This is especially true if you are looking for reviews on games.
I don’t get to play video games anymore as often as a couple of years ago. Then I was naturally informed about interesting titles (typically indie). Now when I have a couple of hours to kill and would like to play something new/original I usually give up after 20min of searching and grab a book.
I wish I knew some decent curators or magazines, but most of the stuff I come across is just too far from what I‘m looking for. Search engines don’t help much here either.
If you find yourself looking for a game in the future, just get "Vampire Survivors". Trust me. It's the pinnacle of casual gaming. $3 and as simple as it is plain fun. Can easily spend 10 hours on it if you want to, or 10 minutes if you don't have any more time.
Some of these raw looking indie games have turned out to be gems. Thinking of Faster than Light, Loop Hero etc. It's hard to gauge the game-play from a quick look. Not even "similar to" searches help too much. It's not really about the money spent, but rather about the time both finding and then playing. I wish I had a list of curators that I really dig. So far, hit and miss.
I keep an eye on r/gamingsuggestions on reddit for game discovery. Follow that for long enough and you'll hear some more interesting games talked about upon occasion enough times that you'll seek them out. If you can stand reddit, that is.
A fools errand indeed. But we currently aren't far from "the one" in three world(view)s: China - Baidu and Sogou; Russia - Yandex; Elsewhere - Google and Bing/syndicates. Korea (Naver) and Czechia/Slovakia (Seznam) are not international but local exceptions.
For me it would be enough if RSS feeds could include <recommended-feed href="https://someone-elses/rss.xml" reason="Friend's travel blog" /> and RSS reader allowed to browse through the web of recomendations and add allowed to add selected ones to my feedreader list.
A potentially standard way to find other related feeds, instead of just consuming the content of one author/website. Curated by real humans that produce useful content (that's why I have them subscribed).
Simple thing that could have been very powerful for discovery if this was thought about early enough.
> Google with all of their cash in the bank and talented engineers can't even crack the problem.
Search is a solved problem. But you're starting from the false premise that Google and your expectations are aligned, they're not. Google shows you what they want you to see, typically spam sites carrying their ads. That's not what you're looking for but from their perspective they've 'solved search'.
Google has two problems. One is that they're actually Doubleclick in a trenchcoat, and serving you relevant information isn't their goal. Serving you ads is.
The other is Goodhart's Law: When a measure (being highly-ranked on google) becomes a target, it ceases to be an accurate measure. Pages now optimize for search as measured by Google.
If Marginalia because the #1 search engine in the world tomorrow, I suspect SEO spam would start trying to beat it a few days later, and the results would be crap inside a month.
> If Marginalia because the #1 search engine in the world tomorrow, I suspect SEO spam would start trying to beat it a few days later, and the results would be crap inside a month.
This is only actually a problem if there is one big search engine. Monocultures of all varieties are vulnerable to specialization, this is as true in a country that grows nothing but wheat to a world where every computer runs Windows XP. If there would be two or three big search engines with conflicting means of evaluating a site, it would decimate the effectiveness of SEO.
A lot of what I'm doing to find good results is promoting the sort of sites that Google rejects. While there are search engine optimized sites that aren't SEO spam, there is no SEO spam that isn't search engine optimized.
Whoa. Searching a few random terms you’re hitting your goal of “serendipity” results. Can you share more about the search basics? I’m learning search now
Don't really know what to say, a lot of the algorithms are snore-inducingly standard.
TF-IDF, an approximation of BM-25. Only twist is that I do Personalized PageRank biased toward a secret segment of the internet to promote the right kinds of search results.
> Right now the big problem as I see it is that there is great content being created
Is that so? Where?
We'd have to define 'great'.
If you mean Tik Tok dance videos and instagram photos/art, then yes. If you mean anything related to philosophy, sociology or explaining how the world works at large - no.
You don't need any tools. You just need to pay people, who have a working brain they've accomplished something with.
It's not an algorithm problem, it's a 'fuck you money' problem. Until people with a brain have fuck you money, they aren't going to waste their life fighting braindead Karens and corporate/political interests who don't want people with a brain to explain how the real world works.
Look at the amount of shit Joe Rogan is getting for having accomplished scientists on to talk about their work. He has fuck you money. Do you know how many other people have fuck you money and don't say shit? A lot. You need to spread that fuck you money around thick, so that one in ten thousand will have the guts to put up with braindead Karens of the world.
> If you mean anything related to philosophy, sociology or explaining how the world works at large - no.
I'm the wrong person to ask for sociological links, but here's a whole flock of actively maintained black swans pertaining to philosophy and other topics:
> I don't think it's too late to fix this. We can build better search engines, or better yet, tools more suitable tools for discovery. It doesn't have to be big to be useful.
Over the next few years, things will change. When people don't like the product, it gives us an opportunity to make products people do like. Right now, a lot of the internet is just not very likable.
I'm not sure. This is a problem that affects not just websites, but physical products as well. It's really hard to find what you actually want. Producers are rewarded more for paying for reviews and search results, for reddit astroturfing, for influencer recommendations; than they are for actually making something worthwhile.
It's not that good things aren't being made, it's just that it's incredibly hard to find them among all crap. There doesn't seem to be a good mechanism that lets the actually good products float to the top.
I doubt this is a new phenomenon. I certainly risk buying things just by myself where pre-internet I would find someone that knew about the thing (cars, knitting, lamps, whatever) and tell them my issue and see what they said. now there is some expextation on my part that fifteen minutes of reading will somehow allow me to drop a lot of cash in some specialized area i have no edperience in and not be taken advantage of as a sucker in the market place?
If you are spending money, possibly just having the worlds info at your fingertips doesn't make up for a lack of experience.
Back in the brick and mortar days, you could at least go look at the thing. Not that there weren't lemons, but the sort of extreme low quality alibaba crap that you find everywhere nowdays wouldn't sell at all because it feels so obviously low quality. Even having it on the shelves would stain the reputation of the store.
Simple laissez faire competition leads to many solutions that are more optimal than naively creating a product people want at an attractive price. The game needs to be regulated to force the "right" kind of solutions to the top but that would be a difficult thing to do even if regulators weren't already part of the "wrong" way of competing.
We can still "vote" with our wallets but just like in real elections sometimes it's really hard to even get some worthwhile candidates on the ballot.
Better search is not the issue, it distorts the spirit, google became the focus of attention, now you want to be listed and seen, to monetize too.. before that people spent less time formatting stuff to exist. The word of mouth levels is a different (maybe more fertile) soil.
Yeah I don't think search is actually the solution to the problem of discovery, it was just the first avenue I tried. Although the metadata from a search engine database does open up a lot of options.
Oh sorry btw, it was not an attack against you. And I actually enjoyed your engine and found a new lisp dialect I never heard of (even though I googled that a few hundred times).
none of the real issues breaking the internet (which is not quite broken yet) has technical solutions. No better tool nor "more technology" will solve the problems at hand.
The problems are social and economical, they require "political" solutions, not more technology.
> The problem is that search engine results – especially on mobile – are not how you find that any longer.
I think the biggest story of the web in the last decade is the slow but steady decline of Google.
Google has fallen victim to spam. Goodhart's Law wins in the end. Almost every site on Google's front page is there because people have spent a lot of time and money optimizing it for search engine ranking. But just because a page is good at gaming Google's algorithm does not mean it is the most useful anymore. Google put up a good fight, but it seems like they have finally lost.
It also seems like the company has lost it's way. I was recently using Yandex mobile browser to browse some Russian sites and it can translate text inside images and now even voice inside videos. Chrome on the other hand often fails to detect other languages and therefore does not translate even text. It feels like even 15 years ago (when I was becoming a Google fanboy, I have since grown out of it) I would have expected Google to get there before some obscure Russian company. I get it, Russian web is smaller than English web and therefore it is probably more useful for Yandex, but Google's revenues are many times that of Yandex, they should ideally be getting there by now.
Google is intentionally dropping five year old results from the index and prioritizing five hour old results. They have complete control over whether they're doing this. Part of SEO is making five year old results look like five hour old results, which is something that Google did to spammers, not something that spammers did to google.
Google has decided that it is more profitable to make sure that somebody who types "chicken" into the search box gets either chicken restaurants nearby, or listicles and video covering the story on the news today about a chicken that scared away a bear. They've given up on depth.
lol, on a lark, I did type chicken into Google, and surprise, I got: 4 maps suggestions for local chicken restaurants, 4 news stories suggestions about Wendy's introducing some new chicken food item, a complete sidebar explaining the nutritional properties of chicken, including images of chicken dead, skinned, cooked, fried, battered, recipes for chicken, a link to KFC....and one link buried in the middle: a link to the Wikipedia article on chicken. This is running an adblocker, by the way.
I mean, forgive me for thinking that "chicken" is actually a term referring to live birds. Instead, humanity's view is clearly very distorted by our stomachs.
One does not use the internet for facts. One uses the internet to be directed towards where to spend their money and/or time as sold to the highest bidder, who is just looking to line their pockets with money--a digital sugar rush for digital stomachs. The internet doesn't think with its brain, it doesn't tell you what you want to know, it thinks with its stomach and expects you to as well.
These pages die from starvation. If people can't get to your page even if they specifically google the name of your page, eventually you'll shut it down.
I understand why pages disappear. I'm just saying the expecting a search engine to return a result from 5 years ago is kind of a bad metric to be using in whatever point was attempting to be made.
I don't want results to be dropped arbitrarily. It kills all of the good content. I don't need a book to change every week to remain relevant.
edit: and to directly address what you said, excusing a search engine's loss of content because sites have disappeared because the search engine dropped them is circular.
My feeling is that the Lindy effect largely holds: The expected lifetime of any given document is proportional to its age. Old websites for the most part seem extremely stable. Young websites change drastically and often.
I’d love to agree with you, but are you sure the depth is there? Where is all the quality content about “chicken”? Where are the legions of chicken experts creating high quality content? We might be starting from the (slightly) false premise that the internet is full of quality content.
There's plenty of quality content about chickens on the internet. Permaculture forums especially love poultry. You can read about how to raise, breed, and slaughter chickens, different varieties of chicken, and how to incorporate chickens into your lifestyle. This type of quality content was easy to find back in the day, now you have to know what you're looking for, and where to look for it.
I feel like people on HN are really detached from reality sometimes. For 99% of people, typing "chicken" into Google search means they want chicken, or don't know what a chicken is. Does anyone here actually expect the average person to be looking for in-depth guides on how to raise a chicken? That's like saying the "hotel" query should return compelling information about the supply chain and inner workings of the hotel industry.
Google's "problem" is that they need to cater to the most common intents per query (see: lowest common denominator), and believe it or not, those intents are usually not biased toward in-depth educational content - isn't this why HN is such a nice niche?
Google could think about adding a different search result sorting option that attempts to prioritize high quality, thought provoking, or otherwise educational content above all else, but it makes sense to me why their primary search ranking algorithm works the way it does.
Spot on. I have an in-depth article about a niche topic and ranked on place 1-3 for many years. Everybody who typed the short keyword would want an in-depth explanation.
These days, the search volume went way up and my article went down to 6-10. The first three spots merely give a rough explanation of my in-depth topic. Quite insulting actually.
But upon further reflection I realized that most people these days only want a rough definition and no in-depth explanation. Google gives this to them.
The people who used to search for my keyword now add 2-3 more words to get to my in-depth article. The typical long tail.
What I find worrisome is that sometimes, the long tail query I use to find quality content gives me the same results as the unspecific short query. Some stuff is close to impossible to find without knowing a very topic-specific thing.
Ah, chicken. The quintessential centerpiece of a weeknight meal. I fondly remember my childhood in Kentucky, rolling in a pile of leaves until the warm, scrumptious smell of roast chicken wafted out from the kitchen into my eager young nostrils. I can go on for hours about this, and I will, but I knew that someday billions of people would lovingly read this long, overcooked story before dinner because it was their inner heart's desire.
The second order effect is intense competition to create lowest common denominator content. You're right that it would be nice for Google to separate it from better content, but this leads to the yet deeper problem--there isn't enough money in that idea.
There's no quality content for "chicken," in fact I'd say that local chicken restaurants and funny animal news are the best content for "chicken." That's why I don't want every search to favor new material and drop old material. The problem that I have with Google is that they know that most searches will be simple and timely [edit: and badly phrased and nonspecific], and they've just ignored everything else.
A bad search deserves a bad result. If you walked up to me and just said one word "chicken", I'd be insulted. If you said one word "chicken" while pointing at your mouth, I'd perhaps think you spoke little English and wanted a place to eat chicken. Providing a bit more info is useful to get decent results.
Well, if not all of humanity’s knowledge is freely and openly available on the web, how can we expect a search engine—any search engine—to provide a meaningful answer? I remember when Google aimed at scanning and indexing entire libraries, but then they had to stop due to copyright laws.
You also maybe navigate a larger web with different needs.
What knowledge cant be found with appending "wiki" to any keyword, what person cant you find with linkedin, what movie cant you find on a private bt tracker, what news cant you have from reddit ?
I think people over estimate what the web was before Google and under estimate how much easier it s been to find anything.
Name me one thing you struggled to find on google and Ill probably be able to describe you the nightmare it would have been 20 years ago
Google me up a review of a mechanical keyboard that isn't sponsored, doesn't have ads, isn't on reddit, doesn't have affiliate links, and isn't trying to sell me anything.
Would you expect someone selling a product to give an accurate review? They have a conflict of interest. Give me a review from someone who does not have a conflict of interest.
There are some basic economic realities underpinning why most people take the time to create content like this. I don't think it's reasonable to assume that just because someone expects to make a small amount of money from creating a review like that (with either affiliate links or buy links), that that person's opinion is inherently at conflict with honesty just because there's a monetary element. It may be true for many sites and many people, but it's pretty straightforward to parse whether someone is writing a review earnestly and passionately rather than for $.
> They have links for every single mechanical keyboard they review or list. What is their incentive to give better reviews to some than others?
Given Sturgeon's law (90% of everything is shit), why are most of the reviews 4/5 or above?
> There are some basic economic realties underpinning why people take the time to create content like this. I don't think it's reasonable to assume that just because someone expects to make a small amount of money from creating a review like that, that that person's opinion is inherently at conflict with honesty just because there's a monetary element.
I updated the post with several examples of the type of content I want you to find.
>I updated the post with several examples of the type of content I want you to find.
The point I'm making is that I think your underlying premise, that all sites which have any kind of monetary element to them are fundamentally corrupt in terms of intention or conflict of intention, is flawed. I've now read multiple reviews on https://www.mechtype.com/category/reviews/ that are quite clearly the work of a keen mechanical keyboard hobbyist with sufficient details, knowledge and effort to lead me to believe the work is genuine and useful rather than merely financially motivated alone or spammy. There are many examples of what you suggest, and I agree that many sites with affiliate links and ad popups are disingenuous and fall much closer to the spam end of the spectrum. But there are still useful results if the user is remotely competent at parsing information. If your argument is that users shouldn't need that competence, and search engines should be kinder to less competent users in that regard, then that's a separate (and probably more interesting) conversation.
That site also has a clear affiliate disclosure along with this:
>Paid Reviews/Sponsored Posts. MechType does NOT accept paid reviews or sponsored posts. All thoughts expressed within our reviews are our own opinions. However, MechType may on occasion accept product samples for review. When a product sample is provided for review it will be clearly and fully disclosed in the post along with the company/persons who provided the sample.
You were the one who claimed to be able to find anything on google, so I provided you with something difficult to find. Don't tell me I'm asking for the wrong thing, especially given I've demonstrated that pages matching my criteria do indeed exist.
>You were the one who claimed to be able to find anything on google
I did not in fact claim that. You are confusing me with someone else in this thread. I agree that some things are hard to find on google, but I reject the notion that anything with a financial element is fundamentally corrupt in terms of intention.
Maybe there are good actors but anytime people are spending money the market will arrange for a lot of people to try to fleece uninformed buyers, and on the internet to uninform the buyers.
I don't even mind ads if I get the content.
What I get now however is 90% ads and 10% content, and of those 10% about half of that is actually promoted content.
Oh and the ads are also spying on me and selling my information to more ads.
I recently had to research and compare analytics services for work. Type in Heap vs MixPanel and look at the results, the entire first page is garbage comparison sites. I really struggled to find any impartial, real reviews.
I find myself agreeing with the author. We are regressing towards curated bookmarks. I actually changed my settings to show the bookmarks bar (again) in Chrome and Safari. Search engines, especially Google, are not the panacea they were 10 years ago.
My optimist side tells me we will have a revival of bookmarks in the mobile browsers too (now they are way too hidden). My pessimist side tells me Google will fight this trend to the user's detriment.
I was so frustated to see Facebook/Twitter using the fundamentals of RSS to their advantage whereas everybody should have started to use RSS for the greater good.
RSS isn't some panacea to the problem. A site needing visits for money is not going to put full content in their feeds. At best you'll get elided content and at worst you'll get only titles and links. You'll also get the same SEO practices where some old content has the date changed to show up as "new".
Most CMSes by default don't do a good job generating RSS either. Most of the time a feed is just a grab bag of posts on all topics. I don't know of any that understand inline hash tags for instance. A separate field for adding tags is just more work for an author. As a reader if I'm looking for blog posts on a topic I don't care about Grampa Moshe's goiter. Yet if I put a blog in a category in my reader software that's what I get.
In very specific circles with specific topics RSS can be really useful but outside that narrow range it's no better than the shit show we currently have.
You're kidding if you think RSS is the glory days. There are reasons why it has fallen out of favor. If it were up to me, I'd kill it and put it out of its misery.
Does the "Page 4" thing refer to the few paragraphs at the end of the article? Because here's the thing. The cool, electic web circa 1996 was maybe X in size. Chances are that kind of web content is still here, possibly even 10X in size. It's just drowned out by 990X more of junk. If the search engines can't filter the good stuff from the junk, you have to go back to the pre-search-engine days. Lists of links, web rings and so on. They never went away. They're as eclectic as ever.
Only problem is we've been spoiled by the thought that ALL information is on the web. And a lot of it is buried in the junky, spammy 990X part. I don't like having to watch an ad-riddled Youtube video for a simple thing like where did they hide the last screw you need to remove. But at least the information is there if I need it badly enough.
I also don't like to wade through the "sign up for Amazon Prime" dark patterns. But I do like being able to order a gadget for a good price and reasonably soon delivery.
I thought it referred to search results, page four. Very few people look at the fourth page of results and the first three pages are the types of pages the rest of the article is about.
Is this related to the google default of showing 10 results ?
I have had it show 100 results by default since .. forever.., , much of the stuff I'm actually interrested are often atleast beyond the top 10-20
I remember UX good practice was that the top half (or viewport) is the key for UX design. The info you need should be clear and visible instantly on load.
It's funny that now i have a reflex to immediately skip the top half and scroll to the bottom in a vain attempt to find what i'm looking at/for.
I've became so used to ignore useless stuff on pages that I sometimes don't even know what I came for originally.
The modern web makes me totally mindless.
Couldn't agree more. I'm particularly upset that it's turned our journalists into click-farmers who experience no consequences for the damage and division they're causing. That has been a huge, huge blow to our society in my opinion.
At the end the author talks about curated link collections as a solution - create them using Pinboard.in and consume them via RSS.
I think the part that is missing is how to discover the link collections to subscribe to.
I'm building https://linklonk.com that merges the collection, the consumption and the discovery:
- You add links to content you liked to your "channels".
- LinkLonk connects you to RSS feeds and other user channels that posted the content you liked and shows you new content from them. The more content you have in common with a feed/channel - the more weight their new content gets in the "For you" list.
The problem is see is that without an account, it is impossible to submit links. However, unlike reddit or hn, one needs to submit links to get useful content. Thus I would assume that by the 1% rule [1], it is very difficult to retain visitors.
Could you offer options for those who don't want to make an account? Maybe you could create some ready-made selections to see good content? Or you could allow to post links in a hidden way to get the algorithm going and you and some moderators approve them later-on.
Your assumption is correct, user retention is difficult. I've been running LinkLonk for a year now and out of >1000 users that created accounts we have ~9 daily active users on the site and 43 users are getting recommendations by email. But I think this is in part the chicken and egg problem. The more users join the more useful the system should theoretically become.
If you don't want to create an account, consider creating a guest account - it does not require any personal information from you. The account is just a number that is issued to you and is stored in your browser cookies. If you don't end up using it - all information associated with your guest account will be deleted after 30 days of inactivity.
If you don't want to create a guest account then the best I can do is to show generally popular content - which is what the home page is filled with for brand new users.
I think your best chance of getting adoption is making the service useful for an individual without relying on network affects at first. Maybe you're already working on that. I haven't looked too deeply into linklonk yet.
It seems that mass media, in reaction to the Internet and facing the increased competition, went with the approach where they increased the frequency and churn of the news articles and content, at the expense of quality.
However, this is a very print-centric approach, where you print something and cannot change it afterwards, and maybe throw the old newspaper away that day. I wonder why they cannot adopt a different journalistic paradigm, similar to for example Wikipedia or Wikivoyage, where instead of publishing hundreds of articles about a topic, one a day, they would instead create a sort of curated portals which would be fine-tuned and updated over the longer time periods, focusing on quality of information.
The same goes for most online discussions; they are extremely ephemeral and stupid points are argued over and over. It would be nice to summarize the main points in a particular discussion from time to time and then continue.
So I think there is plenty of space to innovate like this.
This was a big thing in the early days of the blogging boom. I forget who, but one major blogger made a point out of treating posts as living documents that changed as stories developed.
Being ephemeral is both a bug and a feature, though.
Take a random Discord server, where we try to figure out some details of a DAO in the making. People from other timezones have to hop in and out and are welcomed by a wall of text when they come back.
So did the issue from yesterday get resolved already since you were gone? Maybe, but if so, how did that agreement come to fruition? And which part of the above wall of text do we "mark as resolved"?
What part to keep, what to formalize as a rule, etc. Those who were "in the room" when the discussion happened now have to find a way to re-tell the story in another ritual (meeting, weekly summary, task, whatnot). Sounds like the work of an editor or journalist to me.
But most people would rather add onto, instead of editing or removing only [1]. Like with maintaining legacy software or editing existing wikipedia entries, most prefer to instead just "add some more".
But even more important, some people tend to disregard even scientifically proven advice and prefer living through things themselves, no matter how redundant the experience.
The web is now a user hostile place to be. There are dark patterns, clickbait headlines, taboola churn, autoplaying never ending videos, ads that track you, cover content, distract you and occasionally try to force malware onto you.
I pity anyone who's using accessibility tools to help us the web as sites often have a timer before they cover the content with a nag to sign up to their newsletter or they track your mouse and display something else because they don't want you to leave.
I recently was inspired by a hacker news article to try using the web without an adblocker. Holy hell. It's almost unusable. I made a video where I recorded this on my PC, I'm going to do the same on my phone. I had no idea it was this bad.
In contrast, the BBC website still seems pretty usable because they don't need to run ads (at least when you're accessing from the UK). They do, however, have the utterly ridiculous concept of having autoplay on by default so if you've clicked to watch a video about a current event, if you don't click 'cancel' in 5 seconds they'll load you another video about something else (seemingly at random) when there is no reason to do so at all.
The web reminds me of the state of movie and music piracy before Netflix etc. In those days you bought a DVD and were forced to sit through an unskippable FBI warning and in some cases unskippable trailers, whereas if you pirated the movie you got a superior product. You got a copy of the movie that let you watch it on any device, let you make backups and didn't punish you by forcing you to stare at unskippable content.
If you were a music pirate then you didn't need to worry about Sony putting malware onto CDs that installed rootkits in your pc.
Software pirates didn't need to worry about similar rootkits that made the game run slower than the pirated version or generally buggered up your computer.
We've reached this state with the web. I am forced to use an ad blocker for my security and sanity. Most adverts are lies, scams and are too damn distracting.
See how fast it is? See how well it runs on any old device you may have? I'd rather have this than the current web. Surely there's an in between compromise somewhere.
No dark patterns, no clickbait, no videos, no ads, just a labor of love by someone with a lot of time on their hands.
The problem for search engines is that it's trivially easy to copy this list and put it up some garbage blogspam domain, but even if the content is exactly the same, I want this list, not the copy, because if I find this list, I can explore the rest of the site and find more great stuff. Even a spammer copies the entire site, this is the one I want because this is the one with a real person behind it who will be updating it, engaging with readers, appreciating the traffic they get, etc.
It starts to become clear that what I want my web discovery tool to do is less find individual pieces of good content but more find content from good creators.
I don't see any way a search engine can ever do this kind of discovery as well as a human curator.
The web has changed so much from when Google launched. Google is optimized for finding content about some keyword because there was actually a time where the was a legitimate problem. You searched "flowers" and Google helped you find the places to buy flowers online. You searched "guitar tabs" and Google helped you find where to get them. Google is still really good at this. Other search engines too. So good in fact that it's become almost invisible to us how easy this is compared to how it was in the past.
But now instead of buying flowers, I might want to know... where are the communities where professional florists hang out online and talk about their craft? The Google results for that query are pretty much equivalent to a random listing of websites. But I can almost guarantee there is such a community. Is it a subreddit? A facebook group? Some old vbulletin? A mailing list?
Or I might want to know... where are the blogs/channels of the top authorities on web accessibility? What are the best craft coffee roasters in Sweden? What are the essential reads on the history of Botswana? Who are the most consistently good travel vloggers in Spanish? Where can I get help with sourdough starter problems? I can 100% guarantee there are people out there that have great answers to all these questions, but they seem unlikely to be solved by search engines any time in the near future.
If someone can crack the problem of leveraging genuine, interested-motivated human curation to deliver results without falling victim to abusive, quick-buck-motivated spamming... they could replace search engines.
> It starts to become clear that what I want my web discovery tool to do is less find individual pieces of good content but more find content from good creators.
Bingo. What's interesting about this is that while maintaining a list of all useful content on the web is probably intractable, maintaining a list of all useful content producers might be doable.
You could have moderators for each niche, and as the list of producers for a given niche grows, it could split into sub-niches which would get their own moderators responsible for keeping track of quality content producers.
Sort of like an awesome list for content producers.
I'm certain this already exists in some form. The original web was build around curated content like this. But people aren't used to looking for content this way.
EDIT:
Saw dmoz.org mentioned below which sounds like exactly what I'm talking about. Apparently they're called web directories[0]. Sadly DMOZ is dead but appears to have a community fork called curlie[1].
>If someone can crack the problem of leveraging genuine, interested-motivated human curation to deliver results without falling victim to abusive, quick-buck-motivated spamming... they could replace search engines.
That's the tension right there. The moment someone solves the first part is the moment someone will try to hijack it for their own gain. The Internet is dark and full of husslers. They want to exploit people's goodwill and trust---that stuff is costly and doesn't scale, which makes it valuable. If you want genuine interactions, you go to a quiet corner with a handful of folks and be very very careful about who you let in.
The builder would need to load the first few hundred thousand results and then add some sort of user moderation system, where search results get a reputation and site submitters get a reputation as well.
Donation funded, with maybe one sidebar section for curated ads that are tied to search results and not with a tracking cookie or fingerprinting.
With a few tweaks, that would be a really good start, right?
Once it reached a certain point, it would be overrun by the same systems, using human and bot armies to bias results towards the results that would make them the most money.
As long as there is a profit to be made somewhere, people will undermine any obstacle placed in front of them to make it, and I can't imagine any fix for it that isn't "run your own servers and pay for it out of your own pocket".
I share the sentiments. And the first time I got upset about bloat was in 1993 when instead of a <HR> tag, people put a GIF with a line ;-).
Some of it, though, is a natural consequence on the commercial potential of the Web being realized. "If you put up an abreuvoir, the pigs will come", to quote the German supreme court (out of context).
Perhaps one could design another protocol that is designed so that it cannot be commercially useful. Because we live in an "attention society" (attention = power = money) this means in all likelihood that such a WWW successor would by definition not be able to get as big as the WWW (which is an upside and a downside at once).
Decentralized ideas are very appealing, but centralization is used to make the existing Web fast and convenient - this is also something that will have to be addressed somehow (maybe there is a way to have physical centralization but logical distribution and a way so that the two levels - physical and logical - can't see each other).
> This is about me as the customer, not about you and your monthly active user numbers.
Businesses want to retain customers and have repeat customers, not necessarily just active user numbers. Do you think it is not in a businesses interest to do so?
> We don’t create content for the web and for longevity. We create content to show ads around it.
Not just ads, but regarding social, what is more valuable to people, a library or a mechanism to get attention and influence from people?
The broader "rat race" here is how people must earn a living in todays's society
> I didn’t have to wait for a certain time to learn about a new movie – I could read up on it any time I wanted.
Who created that content and if it was from a professional source, did you benefit from it?
"information wants to be free" sure sounds great if you're not the one supplying it.
It doesnt sound like the author was a contributor to the free and open web. I'd be interested to hear a perspective from one of the bloggers he referenced who tirelessly produced content and who's blog died in 2010 with nothing to show for it.
You're supplying information in the form of your opinion on this topic for free. Now imagine that hacker news starts displaying obvious ads here and someone needs the information you freely provided but is frustrated in finding it because of 3 pages of trash results placed above your post.
You would have to agree that this scenario would at the very least be frustrating, not only for you but also for the people who want your free content, right?
> You're supplying information in the form of your opinion on this topic for free.
My "information" here is not "content" like How to fold a Shurgard moving crate or Movie Reviews i.e. produced with intent to inform/educate the general public about specialized topics. My post is simply part of the social web of personal opinions, and he doesn't seem to think twitter/instagram/tiktok are solving his problems.
Forums do exist. Hacker news and Reddit does exist. But I think the OP is not satisfied with Reddit and would not be satisfied if we just had a bunch of additional forum sites.
> I loathe that 90% of image results are pointing to walled gardens that ask me to sign up before seeing it. I really wished search engines would not show me Facebook or Pinterest results unless I am also logged in there.
By search engines they mean google. Other search engines (yandex, bing) have pretty good image search.
The other thing is that the Wikipedia alone is more valuable content than probably the whole web of 25 years ago. And aside from their annual donation drive, junk free. And "X wikipedia" where X is the subject still works on Google, with the desired result at least in the first six or so.
"Starts" is a important word here. I know it's mentioned in the tail of the article, but if you have a site that's not shit - please, please, please fill it to the brim with links to other sites that aren't shit. Gift me the ability to avoid going back to a search engine for as long as possible.
This is why I moved to the Duck 3-4 years ago. I became tired of clicking past Page 1 of my Google results, all content mills and SEO farms.
DuckDuckGo is just a front-end to Bing, though, and as of mid 2021 I've begun seeing more and more low-effort SEOey links in the first page. Google started the trend, but it seems like all major search engines are increasingly disconnecting search results from the actual search terms.
I actually find myself using archive.org to search if I need super-specific old-school AltaVista-style results.
Google results are shockingly bad now. For searching documentation or Stackoverflow it can be fine, but for anything else, good luck. I know that blogspam has floated to the top for years, but it seems like it's gotten even worse in the last couple years or so. Googling for random information, or trying to find a good product recommendation returns absolutely useless blogspam results where it seems to be strings of text copy-pasted into giant "articles" that are just lists of questions and answers that I assume was cobbled together by a bot.
I dread if Reddit going public ruins the site (well, ruins it more than the redesign did), because currently, appending "reddit" to your query is the only way to get results that seem like they're actually written by a human.
edit: oh, and in regards to his bit about web notifications -- I totally agree that they're one of the worst things to happen to the web, and it's the one web standard I'm actually happy Apple refuses to implement in Safari. It's only used maliciously, and it's a horrible UX. The amount of times I've looked at the notification shade of a non-tech-savvy Android user's phone and seen a bunch of ads from random websites, it's obviously not worth it. Users have been so trained to click "ok" and "accept" on the millions of popups that are shoved in their face as they try to navigate the web, it's no wonder they're enabling notifications on every website they enter. Get this stupid popup out of my face so I can see the content!
>For searching documentation or Stackoverflow it can be fine
I have to disagree on that one. If you search for some API or error message, 9 out of 10 results will be these malware-ridden autogenerated clones of Stackoverflow. I can't believe Google engineers can't do something about this. They must be using Google to search for this kind of stuff everyday...
Oh god I forgot about the SO scrapers. They seem somewhat recent, I don't remember seeing them not that long ago. They're usually completely broken. Or just made by someone with no sense of UI
> I miss having the web as a resource. As a library of knowledge and a collection of weird and wonderful entertainment and subculture art.
This is becoming a meme. Yes, I also think Google results are getting worse.
But: The real underlying core reason for this is that the entire society has come online by now. You can't compare the old, elite web to the mainstream web of today.
At the end of the day, Google is just a machine optimizing for the masses.
I hate the new web, especially google news. Its seems to have gone downhill fast the last couple years. So much of what they feed me is based off some random article I clicked on years ago and now they decide that's all I am interested in. Worse all the articles are just clear click farm bait. "This guy clicked on a Marvel movie link, he must love comic books and nothing else..."
> We don’t create content for the web and for longevity. We create content to show ads around it.
And that is the difference between the artisanal/hobbyist web of the pre-Facebook/Youtube/Blogging Gold Rush era, and what came after it. The web is dominated by businesses that consider content to be a means to an end. Kids' cartoons on TV in the 60s were designed to sell toys and merchandise after all.
This isn't just limited to websites that 'review' things with a metric tons of Amazon affiliate links, or gate white papers behind an email signup. Youtube and Instagram went this route years ago. At least YT is still generally good quality. IG feels like a wasteland of influences shilling some product with a discount code that attributes sales to them.
Million Short is a solution. https://millionshort.com/ Good complement for certain popular topics when the default search engine just gives you big blobs of SEO.
I'm not so sure about this article. Yeah the modern web is a commercialized hellscape now, but there are pockets of resistance if you know where to look, and alternative search engines we can leverage to unearth gems. Think Qwant, Mojeek, Marginalia, hell even Tor has plenty of programming blogs/chans & search engines that remind me of the early Internet. The web is still a rabbithole to get lost in. You have to make the web work for you, and try to escape your comfort zone or bubble.
Author here: for sure, but that's not the point. My point is that the initial experience for non-technical people using the web is horrible, and they will never ever get to the places you mention. The promise of the web as a medium that isn't paid for (and I worked in Radio, TV and Newspapers) is not a mainstream thing.
> And when it comes to demands from publishers of PWAs, there is not much about access to hardware or clever new web APIs. The biggest demand was about notifications and making sure people stay in the app or keep coming back to it.
I think this is an interesting observation in the context of Apple adding PWA notification support to iOS. It's also the reason I won't be enabling any notifications from Safari once that feature goes live.
It'd be nice if the feature included a way to disable notifications per site though. I never allow notifications from a website, ever (or any app on my phone), fwiw.
Consider the difference in results between Google and what you might expect if directories rather than search had remained the primary way of finding things on the web. A search for "instant noodles" assumes you either want to buy them, find recipes for them, or read about whether they are healthy. But a directory topic "instant noodles" would find sites about instant noodles. And sure enough, if you hunt through search results, you can find things like https://www.theramenrater.com/ which definitely fits the bill, but it takes some hunting. It's a "page four" type of result (in reality, it was on page five).
So, yes, this is a general topic and you get general results... but that's not how directories worked. A directory listing for "Instant noodles" would never have included Walmart and Amazon searches. It wouldn't have included random news articles on whether or not they're healthy. And in the days of blogrolls and web-rings and such, once you found one site, you usually had a way to get to more human-curated related sites without even returning to the directory.
Not denying there's some rose tint here, but there's a big difference in how human-curated topical browsing differs from search.
> But a directory topic "instant noodles" would find sites about instant noodles. And sure enough, if you hunt through search results, you can find things like https://www.theramenrater.com/ which definitely fits the bill, but it takes some hunting. It's a "page four" type of result (in reality, it was on page five).
Your example of Ramen Rater is the #1 result for the query [instant noodle reviews]. The more broad [instant noodles] has Wikipedia as the #1 result, which covers the "about" intent case you mentioned.
I didn't specifically want instant noodle reviews. I just wanted sites about instant noodles. If they happen to be reviews, cool. If they are photo galleries, that's cool too. Message boards, also cool. But I don't want articles or encyclopedia or dictionary entries about instant noodles. I want full sites, or at least large subsections of sites, where the primary topic is instant noodles. This is what directories list. Sites where you open the site and ask "what's this site about" and the answer is clearly "instant noodles". Not at the page level, but at the site level.
The most widely used directory DMOZ.org was closed in 2018 or so, simply because AOL were fed up with maintaining its content. Google didn't have much to do with that. In fact, the Google search bot benefits from having high-quality curated links available as much as anyone else does.
Agreed. I'm not sure if you meant rose tinted glasses regarding the internet or search, but it's definitely both. People forget that google results felt amazing because they were so much better than other search... But also forgot how often they weren't great.
I see this all the time in home decoration and design. We tried to decorate our home and we had an idea what we wanted but it was almost impossible to escape the "Big Furniture" bubble as almost everything we wanted almost always lead back to the same 5 or 6 mediocre major retailers. Visited any of those big sites? Well then Pinterest is going to flood your searches with content that leads back to them.
We enlisted a designer and she sent us over 30 incredible online stores that we had specifically been looking for that none of our previous searching lead us to. Most were small independent places and had incredible quality/value and originality.
As much as I agree I think you also have to consider that things change, and what once was something that only nerds and weirdos used now it's the mainstream way of consuming information.
Which means that something that for you as the 99% tail is not good anymore doesn't matter for my mom that doesn't even know what you're talking about as the web being different before. It's the digital version of the old "Nobody goes there anymore, it's too crowded".
That being said I believe a major part of the problem is the asymmetry between consumption and creation of content. If creating content was somehow harder you would find way less auto generated attention grabbing websites.
IMO the biggest issue with the web is the lack of a consistent “contract” between content and consumer.
Compare a blog to a podcast. A podcast is long form content. You know clearly how a podcast is monetized, and accept the advertising or Patreon or whatever.
A blog, OTOH, might literally be malicious to your computer, privacy, and attention. You dont know if it’ll try to hijack your browser or what it might track about you. You don't know if you’ll have a pleasant reading experience or get twenty prompts to signup for a newsletter.
Intermittent reinforcement being the strongest kind, we begin to associate the web / blogs with malicious spam that slows down your computer, even if many aren't.
Google in a better era would have noticed this and used it to improve results. I don't think Google cares about good results anymore. Just profitable results.
Why should one expect there to be "good" free content on any platform that permits content for the purposes of advertising?
Is it not the natural result that monetized content dominates free content on any platform that permits both? After all, monetized content, because it leads to some sort of revenue for the creator, has an advertising budget, and free content does not.
The only way to to avoid such advertising is to not permit it on a given platform, or to require it to be explicitly financed (e.g., a subscription to a digital magazine). The internet, as a platform, has no such rules, and so it is dominated by advertising.
Though blatant, obvious advertising isn’t the only form of advertising. And I’m not so sure that it’s the worst. You’re just pitching a platform that only launders ads through content because you only ban the detectable and perhaps even the most honest ads.
The blogroll-style linking culture is still alive in Discord communities. With colorful about pages, and everything. Animated emojis are the new GIFs. Sadly they do not offer much in terms of persistent content.
You mean the "app" that has no search and requires you to sign up to see anything, then pass some obscure initiation ritual by the mods, then be pestered with various bots. And that app doesn't even make money yet, wait until they decide to become profitable.
No-search is the best part. You actually have to talk with people, and get to know them, to discover more communities!
Sadly, it's a proprietary space, yes, but still.
The difference with "social" networks (and now the bad state of search engines) is that you actually get to be part of a community. As opposed to shouting in a loud bar to compete for likes to get heard better maybe if some algorithm likes you.
What do you mean by no search exactly? I think there is a disconnect between the person you replied to and the person that the person you replied to replied to. Discord has a pretty good search function for messages, attachments, and embeds in servers depending on date, author, etc. As for searching for servers to join they didin't have that before and you had to rely on third party aggregators, now they do, although I have never tried it out as I prefer joining servers who advertise themselves on their own websites individually.
I think it's somewhat ironic that when I Google quote "how to fold a Shurgard moving crate" this article shows up on page 1 of the results.
I agree with the author's conclusion, but the example is a poor one. I would expect the sort of content to come directly from the manufacturer. Is it surprising nobody has blogged about a specific moving crate?
Is there any way to fight SEO? That seems like a big problem nobody seems to be working on. Or maybe working on but failing. If we can reduce the efficacy of SEO then we can fight spam right? I understand its a feedback loop where once they improve something, someone will figure out how to beat it. But maybe it can be adaptive in some way?
If it is, its failing miserably. Used to be that if I did a broad search about a concept it would take me to personal blogs and stuff. Now it just takes me to pintrest. It seems like every non technical search I do leads me to pintrest.
I've been using the "small web" much more in the past few days and loving it. Strongly encourage everyone to install a quality Gemini browser like Amfora, Castor, or Lagrange and check it out!
Food for thought/discussion: I don't see what prevents the author from having the experience they had back in the days. Tcp, http and ipv4 are still standing strong. I am sure there are more than plenty people around who would like to retain the old ways as well.
Mentions that lots of links now from search engines take you to walled gardens. Take a peek around the rest of his site and click a link, low and behold its to skillshare, wanting me to sign up :S
Its called the Society of the Spectacle. Here on hackernews we literally call it "Commoditize Your Compliment." The startup ethos produced this web at the behest of capital. Its kind of nuts to see this post discuss all these problems without even mentioning that capitalism, for good or ill, has created them.
How do you define capitalism? I assume it's not 'people having voluntary economic interactions with one another' because that would seem a little redundant.
so if I have a hammer and am using it to build houses when someone pays me, and I refuse to lend out my hammer to everyone who asks, that's capitalism?
> I refuse to lend out my hammer to everyone who asks, that's capitalism
Capitalism in this analogy is your ownership of the hammer, not who you are or are not obligated to lend it to.
Though it’s not really a great example, a lot of things are missing from this example, such as how you came to have ownership of the hammer, what property rights you are entitled to with the hammer, and why you even need to have ownership of said hammer in the first place. Alternatively, there is some sort of public ownership (as opposed to private, not necessarily implying state ownership) of the supply of hammers, and you are able to obtain a hammer to fit your business needs as needed.
Is there any situation in which Alice can refuse Bob access to a tool Bob's declared a need for not capitalism?
I don't care about the word 'ownership' or what makes it valid, I care about limiting access to limited resources, be those resources hammers or the land my ancestors are buried on.
I mean yes, I could literally take the hammer from you and destroy it under any system, thus “limiting your access”. The question is whether or not that is a crime, whether or not you or I have a right to the hammer or control over what’s done with it.
Then what are people referring to when they say "Capitalism caused this"? If capitalism is private ownership of tools, and there are systems other than capitalism with private ownership of tools, what are people diagnosing when they say 'capitalism' in this context?
Well capitalism isn’t just private ownership of tools, it’s also division of labor, wage labor, and market exchange. Possibly some other things that are escaping me at the moment.
The critique ultimately traces back to Marx, who asserted that social relations within a society are dictated by the “mode of production”.
I don’t necessarily alway agree with people when they say “capitalism caused this” but this is by best attempt at relaying the reasoning.
Ultimately the critique that socialists are making, to bring it back to your analogy, is that most people don’t own their own hammer, capitalists do, they just do all the work with it to build houses while the capitalists take the lions share of the profits.
Google is not the victim in this. It is probably the prime tool in the presentation of reality. It fulfils the role that the legacy media used to do - ie newspapers.
Google wants to intermediate reality for you. A perfect example of that is how it tried to kill rss - first by making a fantastic rss client so that most people used it, only to then to stop the service. This was not due to budgetary considerations, - it was by design! RSS is puts you directly in touch with the content you like - this is a threat to a company such as google and whoever runs them.
If you dig deeper, you will see that google has long-standing associations with military intelligence.
"Both Brin and Page were working on the Stanford Digital Library Project (SDLP). The SDLP's goal was "to develop the enabling technologies for a single, integrated and universal digital library" and it was funded through the National Science Foundation, among other federal agencies.[9][10][11][12] Brin and Page were also part of a computer science research team at Stanford University that received funding from Massive Digital Data Systems (MDDS), a program managed for the Central Intelligence Agency (CIA) and the National Security Agency (NSA) by large intelligence and military contractors."
If you look at the CIA's venture capital firm - In-Q-Tel, you will see that they funded google too:
"In-Q-Tel sold 5,636 shares of Google, worth over $2.2 million, on November 15, 2005.[10] The shares were a result of Google's acquisition of Keyhole, Inc, the CIA-funded satellite mapping software now known as Google Earth"
(Why In-Q-Tel require a funding arm at all, is another question you can ponder.)
When the Edward Snowden fiasco was running, I also recollect reading about 3-letter agencies having offices in google's offices - no questions asked.
TLDR
If you think that the 3-letter agencies are about keeping you safe from the world, you have this the wrong way around. They are about managing the people - they cannot allow information to be unrestrained. Google is an essential part of the infrastructure.
Its all very well downvoting me, but say why not take the time to say what is objectionable?
Even if you don't like my assertions, I'm providing sources for them!
I have to say downvoting without engaging with what info is provided is one of the worst elements of this site. Surely downvoting should be used sparingly. It shouldn't be that just because you are a power user (with 500+ upvotes) who feels discomfort with whatever is being raised, right? Perhaps its hard to justify working at a big tech company, but downvoting would be a rather self-serving ideology, don't you think?
Lately I have been wondering about that classic book listing web sites from the early days of the WWW, that everybody was making fun about. I also thought it was funny, but now I wish I had such a book. I don't even know how to discover interesting web sites anymore.
pinboard, interesting, i guess bookmarking is a step in the right direction, but wouldn't it be better if browsers actually wrote functional bookmark systems and provided standard import/export methods?
Right now the big problem as I see it is that there is great content being created but its audience just isn't finding it. There's so much noise, the signal is almost completely drowned out.
Out of largely the same frustration permeating the link, I built search.marginalia.nu alone in less than a single year in my spare time without prior knowledge of search, paying for the hardware out of my own pocket.
I don't say that to promote how great I am. The point is specifically that I'm just some guy on the Internet. Just imagine what could be possible if actual resources were put toward finding solutions to this problem.