How does Python class system actually compare to Lisp CLOS?
I've seen arguments for Python class system is the blocker for important code optimizations, AOT or JIT. Are there elaborate explanations on why we get near-machine code speed for compiled SBCL Lisp but we cannot even save the image for PyPy runs? Seems like a problem worse than GIL.
One CLOS feature that is particularly interesting is multi-dispatch. When you call a method, most languages implement polymorphism by looking at the class of the object and call the corresponding method in the class. In CLOS, it looks at all the parameters of the method. This is similar to pattern matching in functional programming languages like Elixir.
Aaaaand... this sort of doubles as a partial answer to the original question, too. When your language is that flexible, and doing all its flexibility at run time, yeah, it's pretty hard to optimize. As a product of its time, Python is a triumph, and don't mistake what I'm about to say for it being "bad" somehow, but if I were designing a language today, even a dynamically typed language, I wouldn't even begin to design it with all the run-time flexibility Python has. As neat as "adding multidispatch" is in some ways, the capabilities in the language that allows you to add that capability at runtime comes at a staggering, staggering performance cost that a good 15 years of fix attempts has only partially and incompletely addressed.
I have not used that Python multidispatch library, or even known it existed until I searched for it just now, but I'd lay money down that it imposes yet some further penalty to Python's already-expensive function call process. I don't think is was HN but I was just mentioning online the other day that dynamic scripting languages aren't just slow themselves, but also tend to encourage you to add yet more slow things on top of the already-slow things without necessarily thinking about the costs, and I'd bet this is a perfect example of what I mean. Yes, yes, very nice, very convenient, probably makes function calls another 5-10x slower than they already are. (I note the source package has benchmarks, and they are not mentioned in the README. I make the obvious inference.)
And to be clear, this is not a criticism of this package, which I'm sure is as nice as it can be within the constraints of Python. It looks very nice to me. No red flags I can see. Compliments to the author on the off chance they see this; I'm not targeting this package. It's just part of the answer to why Python is so slow with this stuff.
I’m not sure that “more sophisticated” is the right term for multiple inheritance. Multiple inheritance create all manner of challenges in understanding code. The simplest form of multiple inheritance as it’s present in languages like Perl or C++ can create hard to debug problems (I’m not entirely certain how it works in Python).
That said, Java has (and has had) a restricted form of multiple inheritance for years through default methods in interfaces which allows the most useful aspect of MI—mixed in methods.
Java forbidden multiple inheritance because of the function overriding or diamond problem.
Also Python OO has a lot to improve.
There are generations of Java programmers who have decided that because Java doesn't have multiple inheritance, it must be bad.
People mostly use frameworks these days, so inheriting from the framework uses up our one opportunity to have inheritance. Then when we would like to use it later to consolidate common code, we can't. Or we use ActiveRecord, but then can't actually, god forbid, model our domain using an inheritance hierarchy.
At this point, I think OO has failed to live up to its promise, and I instead use functional programming and macros.
Nah. Python's metaprotocol is inflexible compared to other similarly-dynamic languages.
Here, let's try to monkeypatch a method onto a hierarchy of existing classes, calling the superclass method from time to time.
First, here's our existing hierarchy:
class Base: pass
class Derived(Base): pass
def extend(cls):
def extender(f):
setattr(cls, f.__name__, f)
return f
return extender
@extend(Base)
def foo(self):
print('I am a base!')
@extend(Derived)
def foo(self):
print('I am a derived!')
super().foo()
print('I am still a derived!')
Derived().foo()
~$ python t.py
I am a derived!
Traceback (most recent call last):
File "/home/tonyg/t.py", line 20, in <module>
Derived().foo()
File "/home/tonyg/t.py", line 17, in foo
super().foo()
RuntimeError: super(): __class__ cell not found
---
Turns out in situations like this you have to hold the runtime's hand by supplying `super(Derived, self)` instead of `super()` in the `foo` in Derived. It's to do with how the compiler statically (!) assigns information about the superclass hierarchy using magic closure variables at compile-time (!).
There may be some insane magic tricks one could do to make this work, maybe. Things like reaching into a closure, rebuilding it, adding new slots, preserving existing bindings, avoiding accidental capture, making sure everything lines up just right. It... didn't seem like a good idea to put insane magic in production, so I did the traditional Python thing: swallowed my discomfort and pressed on with the stupid workaround for the bad design in order to accomplish something like what I was trying to achieve.
And I wasn't even trying metaprogramming. Just expectations for class system as if it was Java or .NET, I wouldn't dream of dealing with it like I do with CLOS.
class Base:
def foo(self):
print('base')
class Mid(Base):
def foo(self):
print('mid')
super(self.__class__, self).foo()
class Derived(Mid):
def foo(self):
print('derived')
super(self.__class__, self).foo()
Derived().foo()Nowhere close even to standard CLOS without using any of the MOP extensions.
Things more or less work as one would expect.
Where it becomes challenging is a library using reflection extensively, like SQLAlchemy.
Who says this? Python does expose a bit of the plumbing of its class/object system - but I can't recall seeing anyone calling it especially powerful or flexible?
I don't think I've seen anyone argue for any object system being strictly more flexible/powerful than CLOS.
Some, like smalltalk, ruby, Dylan or Javascript original prototype based object system might be useful, simpler subsets of CLOS - and might so arguably be "better".
Afaik much of the problem with python has to do with the language being quite dynamic, and also somewhat complex scope handling.
I guess I'd argue that SK8's object system was more flexible and powerful than CLOS. It was an extension of CLOS in Macintosh Common Lisp, implemented with support from low-level modifications in MCL's CLOS implementation.
It added, among other things, persistent objects, constraints, truth-maintenance, and several extensions to CLOS' method combinations and before, after, and around methods.
SK8 supported multiple simultaneous inheritance architectures. I don't mean multiple inheritance, though it did support that. I mean that SK8 supported more than one inheritance architecture at the same time in the same set of objects. It offered support for user-defined inheritance architectures, and support for defining inheritance relationships along multiple different inheritance graphs in the same object at the same time.
In other words, a single given object could inherit structure from one set of parents according to one set of inheritance rules, and behavior from another set of parents according to another set of rules, and truth-maintenance constraints from another set of parents according to another set of rules, and containment-related properties according to another set of parents according to another set of rules, and so forth. It provided APIs for constructing and supporting additional user-defined inheritance architectures.
SK8 supported both class-based and prototype-based inheritance at the same time, with more than one inheritance model active at the same time for different purposes.
SK8 was originally designed as a knowledge-representation system--a frame language in the sense of https://en.wikipedia.org/wiki/Frame_(artificial_intelligence.... Other frame languages and systems (e.g. KEE and KL-ONE and CycL) share similar features. Indeed, if I remember right, SK8's creator, Ruben Kleiman, worked on Cyc before he created MacFrames, which later became SK8.
I guess, strictly speaking, these object-system features were features of the frame language, MacFrames, and SK8 was the authoring application that was built on MacFrames.
But not for CLOS code. CLOS is a subset of Common Lisp and it addresses flexibility, expressiveness, runtime dynamics AND decent speed. But fully dynamic CLOS code (with added meta-level) will not be 'machine level' speed.
See Julia for a more recent approach to multimethods and performance. There are also CLOS optimizations in the past and some newer approaches -> but these usually lead to less runtime dynamics.
Much in the same way as I program Lisp in general, first getting the shape of the task and then honing it by adding type declarations gradually.
Training CLOS caches.
There are attempts to improve Generic Function performance, especially for the SBCL implementation.
...
I for one have never heard that said about Python; if this was on Wikipedia I'd definitely be mumbling Weasel Word[1] now...
I can write OO code as well, this is just personal preference.
However, as far as optimization, the point is that Python's class system is implicated in almost every line of code. Most operators are actually invocations of corresponding "dunder" methods on their operands, which can be potentially changed, and whose invocation is actually surprisingly complicated and difficult to optimize.
Common Lisp, of course, does not have operators in the same sense, just functions. However, the analogous functions, like +, *, aref, etc. are not generic functions in the sense of CLOS. They only take built-in data types as arguments and cannot be overloaded. This lack of extensibility makes it easier for the compiler to know what actual code is being invoked and to optimize their use. Arguably, this makes Common Lisp seem like a less flexible language, and in some ways its design does pay more attention to optimization than its reputation would have you believe. On the other hand, the lisp syntax means that there's no such thing as a finite set of operators that you'd want to overload. If you want a different type of multiplication, you can just use a different function.
Expert may pitch in but it feels like a regression (even though I understand the social dynamics at play)