Python's attempts to remove the GIL are not going anywhere really.
Seems like a guild is just a subprocess with its own resources. And you copy objects over as needed. And when the guild is done it will get garbage collected. Like other objects.
In an ideal world the guild is the interpreter state which would be very far from processes. How far down you can go there largely depends on what promises the API made to C extensions and other things in the past.
That likely would be faster than having OS threads in each guild that use PS locks to prevent running >1 of them concurrently.
* Guild has at least one thread (and a thread has at least one fiber)
* Threads in different guilds can run in parallel
* Threads in a same guild can not run in parallel because of GVL (or GGL: Giant Guild Lock)
A guild can't access the objects of other guilds.
About channels:
* We have Guild::Channel to communicate each other
* 2 communication methods
1. Copy
2. Transfer membership or Move in short
Copy is a deep copy and the object is duplicated into the destination guild. A transfer removes an object from a guild and makes it available to another.
There are also immutable objects that are available to all guilds. An obvious example are numbers, which are objects in Ruby, booleans and symbols. I think that other objects are frozen with https://ruby-doc.org/core-2.3.1/Object.html#method-i-freeze
They already did some encouraging benchmarks.
> 2. Transfer membership or Move in short
How is this enforced? What exactly happens at runtime if a guild tries to manipulate an object that belongs to another? (Absent a compile-time check, this is always a possibility.)
It would seem that:
(1) Guild ownership would have to be tracked in the runtime, obviously.
(2) Any access from Ruby code in the runtime, the runtime would also know what Guild the access request came from as well as the Guild the object belonged to against which access was sought.
(3) The runtime would be required to fail in some well-defined way (presuming, raising an exception in the requester) when the rules were violated.
It should be reasonably straightforward to assure this for all accesses within the runtime, since you can just make sure that there is no method to request access which isn't always attached to the Guild that the request comes from. It may be possible to break the runtime with poorly-behaved extension code that subverts the normal mechanisms, and it may be impossible to fully protect against that, but that's pretty much always a potential with extension code.
Check the code at http://rustbyexample.com/scope/move.html and run it (inside the page). There is a commented out println towards the end. Comment it in, run the code again and see the compiler error.
More about transfer of ownership at https://doc.rust-lang.org/book/ownership.html
Moving data between guilds is cheap because data does not have to be copied. Referencing frozen (immutable) data is cheap to.
It seems it will track ownership of objects to make sure guilds don't access other guilds data. But it doesn't seem that it uses OS-level data protection.
Transfering ownership would probably also mean that Ruby not only needs to move one object but probably all subobjects recursively as well. I assume here that "moving" just means updating the guild field for each object.
Is this really feasible or wouldn't just copying the object be faster... I don't know of any system with gc that uses moving to transfer mutable objects between threads. Do such systems exist? Are there better ways of implementing this?
Ruby is already checking the class of the object on every access. You could combine the guild and the class into a tuple and compare against that instead, so it adds no extra overhead.
There is a paper at OOPSLA this year on doing just that http://2016.splashcon.org/event/splash-2016-oopsla-efficient...
Adding a guild word to each object header is certainly a way to check ownership, and should be a cheap check to perform in the interpreter, but will obviously add some extra overhead to standard program execution.
The thing that concerns me is that explicit ownership passing can introduce as many bugs as it solves. If I have two objects A and B, with A holding a reference to B, then I can freeze A and freely pass it between guilds, but if I try and touch B I'll get an error until that too has been frozen or its ownership transferred. The same problems occurs with explicit ownership transfer of a non-frozen A, which leaves you with the slower option of a deep-copy or a recursive ownership transfer which can have equally unexpected consequences.
The "Ruby global data" slide also gives me the scream heebie-jeebies, as did finding stack overflow answers on how to unfreeze objects in MRI. I'm sure nothing will go wrong. :-)
Having said all that, it probably can work nicely for the common use cases of balancing requests between a group of worker guilds where the request is a simple data structure whose ownership can be safely transferred, but it would be hard to do a general work stealing solution that was always safe.
1. This Ruby 3 proposal says that Ruby 2 compatibility is mission critical, therefore this proposal rejects concurrency solutions from other languages (e.g. Erlang) and concepts (e.g. functions) and data structures (e.g. immutable collections).
2 Instead the proposal is to create a fast copy-on-write with rules to "deep freeze" some kinds of objects and primitives into an immutable sharable state.
Matz has been very public about his fear of a "Python 3" situation occurring in the Ruby community.
I can easily see how people might (rightly or wrongly) say "Ruby 3 broke my code, I'm rewriting in Go".
Since the requests would not be in the "main" guild, it might be painful to call into gems.
It all really depends on how much overhead there is to create and destroy guilds. If it's easy then ideally you could start 100s of guilds or 1000s should your hardware allow it.
I see guilds as a subprocess with its own isolated resources.
Here's my reasoning. Since the GVL is insufficient to guard against data races on Ruby 2, under the guild system, locks would be needed to guard against concurrency issues if multiple threads are present.
It would seem like the intention would be to replace usages of Thread with Guild to avoid the concurrency issues inherent with threaded code. Will there be API support to create a Guild that only allows a single thread?
It seems to me that is the intent; that is, any Ruby code that exists now is single-guild Ruby 3 code -- if its multithreaded, it needs locks, for the same reason it does now.
> It would seem like the intention would be to replace usages of Thread with Guild to avoid the concurrency issues inherent with threaded code
I think that'll be a common use case, though running what amount to multiple "legacy" Ruby 2 multithreaded systems in separate Guilds in the same Ruby 3 process seems also to be an intended supported use case.
> Will there be API support to create a Guild that only allows a single thread?
It certainly sounds like a good idea.
That is correct. You'll still need to use locks if doing multi-threading inside of Guilds.
It looks like Guilds can have 1 to X threads.
On par with That's "crates". Gives the impression some people just want to be remembered as inventing names.
Pretty much. I mean, if there is a standard name for a thing between a process and a thread that is not a thread group, I haven't heard it.
