Why isn't Google indexing information about the AT Protocol?

Question

I've been observing this for a while, where very basic queries about atproto stuff doesn't show up. But yesterday I found a query that makes it VERY obvious: "list of public atproto relays". Here's DuckDuckGo:1. https://firehose.directory2. https://atproto.at/relays3. https://atproto.wiki/en/wiki/reference/core-architecture/relay4. https://pulsar.feeds.blue5. https://leaflet.pub/12022731-ae4f-4a13-9f7a-5738b7a83c2eOf those results, Google only has 3, the only one on the list that... doesn't have a list of public atproto relays. None of the other sites are present anywhere. Trying not to assume malice instead of incompetence here, but it's really ironic that one of the ecosystems with the strongest ties to the open web, with users creating dozens of new websites every week, isn't getting indexed.

whateverboat · Accepted Answer

https://imgur.com/a/OS7P9jh Seems working for me.

EnglishMobster · Answer

Can confirm that I can also reproduce this. Interesting.

seanhunter · Answer

Use incognito/private browsing mode to get a genuine sense of whether something isn&rsquo;t in the index or whether google decides that even though you&rsquo;re asking for it you probably don&rsquo;t want it.

runningmike · Answer

I got 2300 results. DuckDuckGo Uses google. Google is very personalised. They use many dirty tricks to give personalised results. Use another box and check the results again.

entropyie · Answer

Maybe they should have used a name that wasn't already taken...https://en.wikipedia.org/wiki/Hayes_AT_command_set

wmf · Answer

Google search has generally rotted. Can't get promoted for maintenance.

fragmede · Answer

https://firehose.directory/robots.txt isn't one, but comes up in Google despite that.
https://atproto.at/robots.txt is good and findable with a direct search
https://atproto.wiki/robots.txt is empty but findable on Google.
https://pulsar.feeds.blue/robots.txt is 404 and thus not indexed by Google.
https://leaflet.pub/robots.txt is good. A logged in google search for this finds this for me.
Tell the people at https://pulsar.feeds.blue to fix their robots.txt if they want to get indexed.

ChrisArchitect · Answer

Spare us the conspiracy; atproto.at? firehose.directory? Why would google index any of those sites with no authority/age? Maybe the owners didn't submit them to give them some credibility to start. Maybe the results vary.

alightsoul · Answer

Here's a way to fix that: post the info you're looking for on reddit. Google will index that instead. Happened here: https://www.reddit.com/r/LocalLLaMA/comments/1umokol/meituan... You could post links instead of the actual info to avoid centralizing info on reddit

iameli · Answer

I'm now getting _this HN article_ as the top Google result for "list of public atproto relays". Weird.Yes, I "used an incognito window". Yes, I checked on multiple devices. Yes, Google is personalized. Maybe if you search "list of public atproto relays" from East Timor you get lists of public atproto relays. But you sure don't from Seattle right now.

PaiDxng · Answer

That result set is pretty damning: Google finds the architecture explainer but misses four pages that actually answer the query. Whatever the cause, atproto clearly has a discoverability problem.