undefined | Better HN

0 pointssimonw1y ago0 comments

My tooling doesn't measure TPS yet. It feels snappy to me on MLX.

I agree that hosted models are usually a better option for most people - much faster, higher quality, handle longer inputs, really cheap.

I enjoy local models for research and for the occasional offline scenario.

I'm also interested in their applications for journalism, specifically for dealing with extremely sensitive data like leaked information from confidential sources.

0 comments

10 comments · 2 top-level

freeamz1y ago· 7 in thread

>I'm also interested in their applications for journalism, specifically for dealing with extremely sensitive data like leaked information from confidential sources.

Think it is NOT just you. Most company with decent management also would not want their data going to anything outside the physical server they have in control of. But yeah for most people just use an app and hosted server. But this is HN,there are ppl here hosting their own email servers, so shouldn't be too hard to run llm locally.

simonwOP1y ago

"Most company with decent management also would not want their data going to anything outside the physical server they have in control of."

I don't think that's been true for over a decade: AWS wouldn't be trillion dollar business if most companies still wanted to stay on-premise.

ipdashc1y ago

Yeah, this has been confusing me a bit. I'm not complaining by ANY means, but why does it suddenly feel like everyone cares about data privacy in LLM contexts, way more than previous attitudes to allowing data to sit on a bunch of random SaaS products?

I assume because of the assumption that the AI companies will train off of your data, causing it to leak? But I thought all these services had enterprise tiers where they'll promise not to do that?

Again, I'm not complaining, it's good to see people caring about where their data goes. Just interesting that they care now, but not before. (In some ways LLMs should be one of the safer services, since they don't even really need to store any data, they can delete it after the query or conversation is over.)

2 more replies

terhechte1y ago

Or GitHub. I’m always amused when people don’t want to send fractions of their code to a LLM but happily host it on GitHub. All big llm providers offer no-training-on-your-data business plans.

2 more replies

__float1y ago

While none of that is false, I think there's a big difference from shipping your data to an external LLM API and using AWS.

Using AWS is basically a "physical server they have control of".

2 more replies

Tepix1y ago

on-premises

https://twominenglish.com/premise-vs-premises/

mjlee1y ago

AWS has a strong track record, a clear business model that isn’t predicated on gathering as much data as possible, and an awful lot to lose if they break their promises.

Lots of AI companies have some of these, but not to the same extent.

belter1y ago

> "Most company with decent management also would not want their data going to anything outside the physical server they have in control of."

Most companies physical and digital security controls are so much worst than anything from AWS or Google. Note I dont include Azure...but a physical server they have control of is a phrase that screams vulnerability.

triyambakam1y ago· 1 in thread

> specifically for dealing with extremely sensitive data like leaked information from confidential sources.

Can you explain this further? It seems in contrast to your previous comment about trusting Anthropic with your data

simonwOP1y ago

I trust Anthropic not to train on my data.

If they get hit by a government subpoena because a journalist has been using them to analyze leaked corporate or government secret files I also trust them to honor that subpoena.

Sometimes journalists deal with material that they cannot risk leaving their own machine.

"News is what somebody somewhere wants to suppress"

j / k navigate · click thread line to collapse

0 comments

10 comments · 2 top-level

freeamz1y ago· 7 in thread

>I'm also interested in their applications for journalism, specifically for dealing with extremely sensitive data like leaked information from confidential sources.

simonwOP1y ago

"Most company with decent management also would not want their data going to anything outside the physical server they have in control of."

I don't think that's been true for over a decade: AWS wouldn't be trillion dollar business if most companies still wanted to stay on-premise.

ipdashc1y ago

I assume because of the assumption that the AI companies will train off of your data, causing it to leak? But I thought all these services had enterprise tiers where they'll promise not to do that?

2 more replies

terhechte1y ago

Or GitHub. I’m always amused when people don’t want to send fractions of their code to a LLM but happily host it on GitHub. All big llm providers offer no-training-on-your-data business plans.

2 more replies

__float1y ago

While none of that is false, I think there's a big difference from shipping your data to an external LLM API and using AWS.

Using AWS is basically a "physical server they have control of".

2 more replies

Tepix1y ago

on-premises

https://twominenglish.com/premise-vs-premises/

mjlee1y ago

AWS has a strong track record, a clear business model that isn’t predicated on gathering as much data as possible, and an awful lot to lose if they break their promises.

Lots of AI companies have some of these, but not to the same extent.

belter1y ago

> "Most company with decent management also would not want their data going to anything outside the physical server they have in control of."

triyambakam1y ago· 1 in thread

> specifically for dealing with extremely sensitive data like leaked information from confidential sources.

Can you explain this further? It seems in contrast to your previous comment about trusting Anthropic with your data

simonwOP1y ago

I trust Anthropic not to train on my data.

If they get hit by a government subpoena because a journalist has been using them to analyze leaked corporate or government secret files I also trust them to honor that subpoena.

Sometimes journalists deal with material that they cannot risk leaving their own machine.

"News is what somebody somewhere wants to suppress"

j / k navigate · click thread line to collapse