agencies

I want to have a local device that will serve web apps to my Android browser over bluetooth or wifi *while my Android is still connected to normal wifi or mobile data*.

I have thought about trying local http proxies over bluetooth serial, the new web bluetooth standards, intercepting xhr in native Android code to send requests over serial, wifi direct, wifi regular mode but w/o route to internet, serving the app over lte so my android connects to it through the internet.

All of these are brittle hacks that have unintended consequences (e.g. can connect to wifi regular mode w/o default gateway but then it only connects to lte, web bluetooth isn't a request/response model, have to build framing to proxy multiple requests over bluetooth serial).

Has anyone done this and it worked??

I recently completed a small web crawl including only pages linked from HN stories. For example I would crawl the lwn article from https://news.ycombinator.com/item?id=32614633. I used archive.org's wayback machine to fetch their copy nearest to the HN submission's timestamp. If archive didn't have a copy, I did a direct fetch. It's about 2.5 million pages.

I would like to publish it for others to use, but I'm not sure how useful it would be. In the HN spirit of validating customers early, I'd like to gauge interest of those would actually download and use such a resource before moving forward. Let me know.

Where do folks keep up with and ask questions about sota performance code (e.g. simd, hashmap tuning, allocators, Dan Lemire, gaming code optimizations, etc.)?

Stackoverflow? Particular forums?

HN links to over 6 million urls in stories and comments. Many domains have expired or content is no longer available. Internet archive has much of the content but throttles requests. What's the fastest way to get the historical content?

How much does it cost to crawl all urls that are linked from hackernews submissions and comments? Ballpark at least 3.5 million story urls.

Several services would likely do this for several thousand dollars but I'm hoping for something cheaper.

Is there an updated dump of hackernews that picks up where pushshift data ends? Specifically asking about web hosted flat files (for much easier access than BigQuery).

https://files.pushshift.io/hackernews/

For a local machine to be able to load and serve say 10 different machine learning models (assume model sizes are 500MB to 1GB in size) at the same time, what are good enough GPUs to run classification without worrying about training?

There is usually smaller versions of models provided that are likely to work on a Nano 4GB, but give up some accuracy. I'm curious to find examples when people found these smaller models not quite good enough.

To improve web search, we need to break down problems and make algorithms, code, and data accessible.

For starters, we need a communal dataset of 50-100 million urls/data/metadata that's "good" (for any value of good) to help people experiment with web search tech.

We need more Kaggle competitions to explore better summarization, reader mode text extraction, boilerplate removal, etc. What other competitions would help web search?

How can we foster a sustainable community and projects to create a bazaar of web search engines?

If you had the 4 billion pages Google had on its index in 2004 in elasticsearch today, what would it take to replicate 2004-era web search quality using that static index?

What relevant research or projects are trying to make these sorts of algorithms and data accessible as a future commodity people can build on top of?

A handful of places are working to make web search better (for any value of better). The clueweb dataset (eg [1]) has been around quite a while but is it actually useful for any real world purposes?

Common crawl is bigger but I've also read still is not good enough to use as input for good web search.

Did the trec web track produce anything useful?

[1] http://www-personal.umich.edu/~kevynct/trec-web-2014/