Sail – Unify stream processing, batch processing and compute-intensive workloads (opens in new tab)

(github.com)

79 pointschenxi96491y ago15 comments

15 comments

From the announcement “As of now, we have mined 1,580 PySpark tests from the Spark codebase, among which 838 (53.0%) are successful on Sail. We have also mined 2,230 Spark SQL statements or expressions, among which 1,396 (62.6%) can be parsed by Sail”

Kinda early to call this a drop in replacement with those numbers no?

But, with enough parity this project could be a dream for anybody dealing with spark’s dreadful performance. Kudos to the team

Kydlaw1y ago

The next paragraph explains that: "When looking at the test coverage numbers alone, Sail’s capability may seem limited. But we have found that there is a long tail of failed tests due to formatting discrepancies, edge cases, and less-used SQL functions, which we will continue tackling in future releases."

I am with you that it is still very very early. I'll personally keep an eye on the project.

SpicyLemonZest1y ago

I'll keep an eye on it too, but for a query engine formatting compliance and edge cases tend to be almost all of the work. It's easy to implement SELECT x FROM y WHERE z.

1 more reply

bburnett441y ago

Yeah but the website literally says “zero code changes”. It’s the long tail that’s dangerous since most people don’t understand it as well as a the core functions

anonzzzies1y ago

Bit off topic; we are looking for something like this but with a facility for untrusted users to run sandboxed code instead of trusted code. All that I found (but I am relatively new to this field) are hacky and, worse, slow solutions.

bobbiechen1y ago

What kind of sandboxing are you looking for, and what's the threat model? (My contact info is in my profile as well if you prefer)

anonzzzies1y ago

Our users (who are perhaps untrusted) write the compute for this and so, they cannot just do whatever they want. But it still has to be performant so kicking off containers so far is not great, however even they we couldn't find done.

log4shell1y ago

What is your use case?

log4shell1y ago

It is refreshing to see multiple projects with arrow/datafusion trying to bank on existing and user friendly spark's API instead of reinventing the API all over again.

There is likes of comet and blaze that replace execution backend of spark with datafusion and then you have single process alternatives like sail trying to settle in "not so big data" category.

I am watching evolution of projects powered by datafusion and compatible with spark with keen eye. Early days but quite exciting.

cybergorilla1y ago

This looks interesting, but the docs are really lacking, to the point where it is barely understandable.

I see some potential wins on it, such as it being a Rust-based, Spark-compatible and better suited for single processor environments, but they are just not explained or developed enough.

johanneskanybal1y ago

Been on hacker news long enough to take bold claims like this with a few cups of salt but bookmarked so some kind of interest sparked (hah!).

mikymoothrowa1y ago

Is this for distributed data processing like spark is?

The documentation has nothing to indicate that it is.

rajman1871y ago

From the documentation [1]

> The mission of Sail is to unify stream processing, batch processing, and compute-intensive (AI) workloads. Currently, Sail features a drop-in replacement for Spark SQL and the Spark DataFrame API in single-process settings.

[1] https://docs.lakesail.com/sail/latest/

binary1321y ago

"In single-process settings"

nobody uses Spark for this outside of unit testing

TiredOfLife1y ago

Note. This is not Laravel Sail - the docker environment.

j / k navigate · click thread line to collapse

15 comments

ignoreusernames1y ago

Kinda early to call this a drop in replacement with those numbers no?

But, with enough parity this project could be a dream for anybody dealing with spark’s dreadful performance. Kudos to the team

Kydlaw1y ago

I am with you that it is still very very early. I'll personally keep an eye on the project.

SpicyLemonZest1y ago

I'll keep an eye on it too, but for a query engine formatting compliance and edge cases tend to be almost all of the work. It's easy to implement SELECT x FROM y WHERE z.

1 more reply

bburnett441y ago

Yeah but the website literally says “zero code changes”. It’s the long tail that’s dangerous since most people don’t understand it as well as a the core functions

anonzzzies1y ago

bobbiechen1y ago

What kind of sandboxing are you looking for, and what's the threat model? (My contact info is in my profile as well if you prefer)

anonzzzies1y ago

log4shell1y ago

What is your use case?

log4shell1y ago

It is refreshing to see multiple projects with arrow/datafusion trying to bank on existing and user friendly spark's API instead of reinventing the API all over again.

There is likes of comet and blaze that replace execution backend of spark with datafusion and then you have single process alternatives like sail trying to settle in "not so big data" category.

I am watching evolution of projects powered by datafusion and compatible with spark with keen eye. Early days but quite exciting.

cybergorilla1y ago

This looks interesting, but the docs are really lacking, to the point where it is barely understandable.

I see some potential wins on it, such as it being a Rust-based, Spark-compatible and better suited for single processor environments, but they are just not explained or developed enough.

johanneskanybal1y ago

Been on hacker news long enough to take bold claims like this with a few cups of salt but bookmarked so some kind of interest sparked (hah!).

mikymoothrowa1y ago

Is this for distributed data processing like spark is?

The documentation has nothing to indicate that it is.

rajman1871y ago

From the documentation [1]

[1] https://docs.lakesail.com/sail/latest/

binary1321y ago

"In single-process settings"

nobody uses Spark for this outside of unit testing

TiredOfLife1y ago

Note. This is not Laravel Sail - the docker environment.

j / k navigate · click thread line to collapse