undefined | Better HN

0 pointsdekhn1y ago0 comments

I've worked on quite a few large-scale scientific collaborations like this (and also worked on/talked to the lead scientists of LSST) and typically, the end groups that do science aren't the ones handling the massive infrastructure. That typically goes to well-funded sites with great infrastructure who then provide straightforward ways for the smaller science groups to operate on the bits of data they care about.

Here's the canonical example: https://home.cern/science/computing/grid and a lab that didn't have enough horsepower using a different grid: https://osg-htc.org/spotlights/new-frontiers-at-thyme-lab.ht...

Personally, I have pointed the grid folks (I used to work on grid) towards cloud, and many projects like this have a tier 1 in the cloud. The data lives in S3, metadata in some database, and use cloud provider's notification system. The scientists work in adjacent AWS accounts that have access to those systems and can move data pretty quickly.

0 pointsdekhn1y ago0 comments

0 comments

6 comments · 3 top-level

Tepix1y ago· 2 in thread

If you’re dealing with a fairly constant amount of data every day for years, using the cloud will be way more expensive than necessary.

spacecadet1y ago

The whole thread comes off as an AWS sales pitch...

dekhnOP1y ago

I've setup and built my own machines and clusters, as well as setting up grids, and industrial scale infrastructure. I've seen many closet clusters, and clusters administrated by grad students. Since then, I've gone nearly 100% cloud (with a strong preference for AWS).

In my experience, there are many tradeoffs using cloud but I think when you consider the entire context (people-cost-time-productivity) AWS ends up being a very powerful way to implement scientific infrastructure. However, in consortia like this, it's usually architected in a way that people with local infrastructure (campus clusters, colo) can contribute- although they tend to be "leaf" nodes in processing pipelines, rather than central players.

gregw21y ago· 1 in thread

Why move the data? Why not just enable permissions on cloud sharing a la Snowflake or iceberg?

dekhnOP1y ago

Sure, that also works, although it often leads to problems around cost and scalability and environment customization.

WD-421y ago

The difference with this project is the data from Rubin itself isn’t where most of the scientific value comes from. It’s from follow up observations. Coordinating multiple observatories all with varying degrees of programmatic access in order to get timely observations is a challenge. But hey if you insist on being an “everything is easy” Andy I won’t bother anymore.

j / k navigate · click thread line to collapse