While we're in beta, storage access will be free. As we're thinking about it now, once we're out of beta this wouldn't be included in the base $5/mo plan.
Since there's both a compute component (a Durable Object runs code, like a Worker) and a storage component (for storage operations) to the product, we want the long-term pricing model to mesh those two in a transparent, competitive way.
While we're not finalized on price yet, you can expect that costs for storage will be cheaper than existing services like AWS DynamoDB or Google Cloud Firestore when we move out of beta.
Congrats on launching! It's awesome to see Cap'n Proto and sandstorm's legacy living on :)
I love the migration to nearby edge nodes, but here's a question to any Cloudflare employees around: Have you given any thought to automatically migrating Durable Objects to end user devices?
That has security implications of course, so if you've dismissed the idea previously because the security issues are too hard to surface to the developer, that's reasonable.
We don't have any current plans, but... I was the co-founder of Sandstorm.io before going to Cloudflare, and Durable Objects are very much inspired by parts of Sandstorm's design. So yeah, I've absolutely thought about it. ;)
It would definitely have to be an opt-in thing on the developer's part, due to the security considerations as you mention. But I think the possibilities for solving tricky compliance problems are pretty interesting.
Protip: "Compliance" is how you say "privacy" while sounding like a shrewd business person instead of an activist. ;)
I'm not sure I see many real world applications for this. It seems to sit in the unhappy middle ground between local device storage and central storage. Local storage is the best performance because you eliminate network issues but then you have to deal with sync/consistency issues. Central storage & processing eliminates sync/consistency issues but can have poor performance due to network. Worker Durable Objects sits in the middle. You trade consistency complications for performance but instead of eliminating the network you're shaving some tens of miliseconds off the RTT. It's a level of performance improvement that essentially no one will notice.
To use their examples:
>Shopping cart: An online storefront could track a user's shopping cart in an object. The rest of the storefront could be served as a fully static web site. Cloudflare will automatically host the cart object close to the end user, minimizing latency.
>Game server: A multiplayer game could track the state of a match in an object, hosted on the edge close to the players.
>IoT coordination: Devices within a family's house could coordinate through an object, avoiding the need to talk to distant servers.
>Social feeds: Each user could have a Durable Object that aggregates their subscriptions.
>Comment/chat widgets: A web site that is otherwise static content can add a comment widget or even a live chat widget on individual articles. Each article would use a separate Durable Object to coordinate. This way the origin server can focus on static content only.
The performance benefits for the cart, social feed, and chat are irrelevant. Nobody cares if it takes 50 ms longer for any of those things.
IoT coordination is more promising because you want things to happen instantly. Maybe it's worth it here, but people usually have a device on their local network to coordinate these things.
Game server would definitely be an improvement. But these things are more complex than some JS functions and it would be a large effort to make them work with Durable Objects.
I think this is missing a few points:
1. Yeah they do. If your shopping cart responds 50ms faster when someone clicks "add to cart", you will see a measurable benefit in revenue.
2. It's actually a lot more than 50ms. A chat app built on a traditional database -- in which a message arriving from one user is stored to the database, and other users have to poll for that message -- will have, at best, seconds of latency, and even that comes at great expense (from polling). The benefit from Durable Objects is not just being at the edge but also being a live coordination point at which messages can be rebroadcast without going through a storage layer.
3. Yes, some databases have built-in pub/sub that avoids this problem and may even be reasonably fast, but using Durable Objects is actually much easier and more flexible than using those databases.
No, it's strictly better than both. You get the performance of local storage and the ease of programming of central storage.
Pretty much everyone chooses central storage at the moment, so the advantage of mobile objects manifests as performance.
I don't know about you, but I would find it a much nicer experience if when I clicked "Add Item" in a shopping cart on some website, it happened <5ms regardless of the quality of my network connection, while still being shared between different machines and never encountering consistency issues. The current "wait somewhere from half a second to several seconds for each click" is bad UX, even if users have gotten used to it.
Now in all seriousness, this is super impressive. Congrats to the CF team!
But we also needed a name for the individual instances. We also found that the people who "got" the product were the ones who thought of it in terms of object-oriented programming (an object is an instance of a class). So we ended up gravitated towards "objects".
But I dunno, naming is hard. "Workers State" may in fact have been a better name!
Anyway, thanks again for working on this. I'm sure I'm going to use those once they come out of beta!
https://docs.microsoft.com/en-us/azure/azure-functions/durab...
From what I understand these features are a nice way to implement a serverless Actor Model. I was surprised to see no reference to it on the CloudFlare page.
We actually did call this product "Actors" internally for a long time, but we found that people who had done previous Actor Model work (e.g. in Erlang) ended up more confused than enlightened by this name, so we ditched it.
The read/write limit per second?
That usually the first things I want to know about my cloud primitives...
(Credits for at-least being clear about consistency which is always my very first question)
For Durable Objects, applications should aim to make their objects as fine-grained as they reasonably can, so that the limits on one object are not likely to matter. Meanwhile, the total capacity across all objects is effectively unlimited.
Anyway, we don't have numbers for these questions yet. This is an early beta and we still have a lot of low-hanging fruit optimization to do.
It's nice to know that if "N" chat rooms get started, "N" instances are built, but if 100k people join one chat room, it's going to bog at best, and flame out at worst. Or am I guessing wrong?
How fine-grained?? Which is essentially the question I'm asking.
Would I need a hierarchy of durable objects to store a 100kb collaborative text document? What about 10MB?
You can make things very fine-grained, if you want to.. but at some point you're essentially implementing distributed consensus algorithms -- and then it might be better to just use a single point of failure like a database..
> how security works?
Messages can only be sent to Durable Objects from other Workers. To send a message, you must configure the sending Worker with a "Durable Object Namespace Binding". Currently, we only permit workers on the same account to bind to a namespace. Without the binding, there's no way to talk to Durable Objects in that namespace.
> Is there backpressure on message senders?
Currently, the only message type is HTTP (including WebSocket). There is indeed backpressure on the HTTP request/response bodies and WebSocket streams.
In fact, this is exactly why we added streaming flow control to Cap'n Proto: https://capnproto.org/news/2020-04-23-capnproto-0.8.html
We plan to support other formats for messaging in the future.
> Any ordering guarantees?
Since each object is single-threaded, any block of code that doesn't contain an `await` statement is guaranteed to execute atomically. Any put()s to durable storage will be ordered according to when put() was invoked (even though it's an async method that you have to `await`.)
When sending messages to a Durable Object, two messages sent with the same stub will be delivered in order, i.e.:
let stub = OBJECT_NAMESPACE.get(id);
let promise1 = stub.fetch(request1);
let promise2 = stub.fetch(request2);
await promise1;
await promise2;
> Are messages queued so activated objects can reconstruct state?
No. The only state that is durable is what you explicitly store using the storage interface that is passed to the object's constructor. We don't attempt to reconstruct live object state. We thought about it, but there's a lot of tricky problems with that... maybe someday.
If the machine hosting an object randomly dies mid-request, the client will get an exception thrown from `stub.fetch()` and will have to retry (with a new stub; the existing stub is permanently disconnected per e-order). In capability-based terms, this is CapTP-style, not Ken-style.
> Can passivation warmth be controlled?
Sorry, I don't know what that means.
> Can objects support multiple threads?
No, each object is intentionally single-threaded. It's up to the app to replicate objects if needed, though we might add built-in features to simplify this in the future.
> Can objects move?
This is a big part of the plan -- objects will transparently migrate between datacenters to be close to whatever is talking to them. It's not fully implemented yet, but the pieces are there, we just need to write some more code. This will be done before coming out of beta.
> Failover?
If a machine goes down, we automatically move the object to a different machine. If a colo goes down, we will automatically move to another colo. We still have a little bit of missing code for colo failover -- the data is replicated already, but we haven't fully implemented the live failover just yet. Again, that'll happen before we exit beta.
A variant of the cold start problem. How long after all the messages for an object drain is it passivated? Can you keep it pinned in memory?
Another question. Can objects contain relationships that are themselves references to other Durable Objects? Say a simple reference, or a list, or DAGs?
Perhaps I'm missing something important, but isn't this quite similar to Orleans grains and other distributed actors?
But yes, the basic idea is not entirely new. For me, Durable Objects derive from my previous work on Sandstorm.io, which in turn really derives from past work in Capability-based Security (many implementations of which are Actor-oriented). But while the idea is not entirely new, the approach is not very common in web infrastructure today.
(I'm not familiar with Orleans.)
I haven't done any programming with Actors per se but after skimming over its Wikipedia entry and other blog posts, Durable Objects does sound a lot like Actors to me.
Genuinely curious: What were some glaring differences that you were made aware of that led to not naming it Actors?
Thanks.
- https://edge-chat-demo.cloudflareworkers.com
- A Public Room (to joint test):
hackernews
- Source: https://github.com/cloudflare/workers-chat-demo
Insanely awesome feature add (much needed for truly “serverless” application development). The power to scale here without insane infrastructure headache is amazing.
One day some kid is totally going to build a single-person billion dollar company from his mom’s basement.
On a superficial skim it looks like a tuple space; they were heavily researched in the 80s and 90s. JavaSpaces emerged in the late 90s but never took off.
Scala folks are keen on Actor models (Lightbend have been using the term "Stateful Serverless" for a while now), as are Erlang and Elixir folks.
I guess the key here is "widely-used".
Edit: this sounds even more arrogant than I intended. Sorry. I just feel bad for tuple space researchers (including my Honours supervisor). They laboured mightily in the 80s and 90s and their reward was to be largely ignored by industry.
Edit: oh, here's @kentonv, capnproto author & cloudflare employee, elsewhere in this discussion:
> Each object is essentially an Actor in the Actor Model sense. It can send messages (fetches, and responses to fetches) to other objects and regular workers. Incoming requests are not blocked while waiting for previous events to complete.
Thank you, it was bugging me so much.
Can this result in a deadlock if I access DurableClass(1), then delayed DurableClass(2) in one worker and DurableClass(2) and delayed DurableClass(1) in another worker?
Hence, a block is only atomic if it contains no "await" statements.
In the counter example, the only thing we "await" (after initialization) is the storage put()s. Technically, then, you could imagine that the put()s could be carried out in the wrong order. But, we're able to guarantee that even though put() is an async method, the actual writes will happen in the order in which put()s were called.
(For those with a background in capability-based security: Our system is based on Cap'n Proto RPC which implements something called E-order, which makes a lot of this possible.)
* Disclaimer: At this very moment, there are some known bugs where put()s could theoretically happen out-of-order, but we'll be fixing that during the beta.
However, memory corruption via manual memory management and deadlocks via manual locking are commonly caused by simple and innocent programming mistakes and basically something one has to live with on a day to day basis.
Signed up for beta invite -- does anyone happen to know whether all interested parties are admitted?
With Durable Objects, you instead design your storage model to match your application's logical data model. For example, a document editor would have an object for each document, while a chat app would have an object for each chat. There is no problem creating millions or billions of objects, as each object has minimal overhead.
What does it mean that a document editor will have an object for each document? Will I have to create a new UDO each time a new document is created?
Since the storage is explicit, it's easy to upgrade the class definition. The in-memory object will be reconstructed and will need to refresh its state from storage.
If the connection drops, the Worker will receive an error and can re-establish its connection to the Durable Object. The update may or may not have been successfully persisted by the Durable Object - just like any other remote database operation where the connection drops before you receive the result back.
Is there going to be Jepsen testing for this?
There's no Jepsen testing in the works at the moment, but we'll see if it makes sense in the future.
The chat demo says:
> With the introduction of modules, we're experimenting with allowing text/data blobs to be uploaded and exposed as synthetic modules. We uploaded `chat.html` as a module of type `application/octet-stream`, i.e. just a byte blob. So when we import it as `HTML` here, we get the HTML content as an `ArrayBuffer`[...]
import HTML from "chat.html";
export const IMAGE_DATA =
"iVBORw0KGgoAAAANSUhEUgAAAD8AAAA/..." +
"..."
export const HTML = `
<!-- totally the HTML I wanted to use -->
`;
Ultimately I ended up going with something closer to the latter, and there wasn't even any reluctance about it on my part by the time I made the decision—I was pretty enthusiastic after having an insight verging on a minor epiphany.
I'd been conflicted around the same time also about representing "aliens" (cf Bracha) from other languages and integrating with them. I slapped my head after realizing that the entire reason for my uneasiness about the latter "data islands" approach was because I wasn't truly embracing objects and that these two problems (foreign integration and foreign representation) were very closely related. Usually you don't actually want `HTML`, for example, and focusing on it is missing the forest for the trees. I.e., forget whatever you were planning with your intention to leave it to the caller/importer to define procedures for operating on this inert data. Make it a class that can be instantiated as an object that knows things about itself (e.g. the mimetype) and that you can send messages to, because that's what your program really wants it to be, anyway. Once you're at that point, the "wrapper" approach is much more palatable, because it's really not even a wrapper anymore.
Can the data store only store alphanumeric or can you write blobs? Could a chat app store uploads inside the object?
Oh, well, I'll wait until it's an open beta or generally available.
And today they have given into us a new, powerful bounty of storage with a delicious API!