On the contrary, the dynamic nature of python can make it seem like inaccessible black-magic where you don't even know where to look in the code to see what's even possible. Stronger types will empower users to understand their ecosystems.
I've spent countless hours in rails/django land trying to figure out what's even possible with an object that the ORM or whatever gives me. The human-flavored docs kinda outline it, but if you're in IDE-land, context-switching to prose isn't productive.
Types help humans develop and debug, but they make library authors jump through acrobatics to express what their magic does. This is a limitation of the type-checker implementation(s) not having a strong syntax or notion to capture the runtime-dynamic polymorphism, but it doesn't mean that the concept of types is a flawed idea or not worth using when appropriate.
Don't throw the baby out with the bath-water.
The Python community and the set of people that treat mypy and pylance as dictators rather than tools to be used where appropriate and turned off where not are...not the same thing. (And the latter very much depends on tools built by the rest of the former that internally are wild and woolly, even if they present a nice cleanly typed interface to the consumer.)
class Inject:
def __init_subclass__(cls):
cls.injected = 42
class Child(Inject):
injected: int
print(Child().injected) # 42
import typing as t
class Inject:
injected: t.ClassVar[int]
def __init_subclass__(cls):
cls.injected = 42
class Child(Inject):
pass
print(Child().injected) # 42
Disagree. It's popular because it has an uncluttered syntax - "executable pseudocode" - and simple semantics - "explicit is better than implicit", "there should be one obvious way to do it", and all that. Metaclasses were never particularly Pythonic (if anything they felt like a more Ruby-style way of doing things, and the ascent of Python over Ruby is a reflection of that difference).
I get that it's a dynamic language but if I need metaclasses for something it's more likely than not to be a hidden technical debt generator.
In another language it would be done with code-gen, which would also be fine.
But SQLAlchemy is library code.
I took the time to understand how to use metaclasses, and concluded that I never would, especially not in code I expected another engineer to understand - I expect 90% of people who write Python haven't heard of metaclasses, and 99% have never written one.
Edit: Enums use them as well (makes sense).
For example... you might make a class for each HTML tag in an HTML processor, or each drawing command in an SVG processor. My experience is that it's not hard to debug, because all that you're doing is getting rid of some boilerplate.
As it turns out, since the human mind is the real limitation here, anything that stops runaway complexity ("hiding" functionality behind "magic" is actually worse because you're just moving the complexity "away from your mental model" where it can fester and grow out of sight, even if what you see looks "simpler") results in fewer bugs, and more-debuggable bugs.
I like the way he documents everything in Github issues[0]. I learn a new thing or two with every release of Datasette, because there's so much troubleshooting full of relevant links and solutions. Same with his TIL blog.
i have a text on the python object system and metaclasss here https://github.com/MoserMichael/python-obj-system/blob/maste... as part of my free python course https://github.com/MoserMichael/python-obj-system/blob/maste... (didn't cover __init_subclass__ yet, will have to extend it. You never stop learning with python...)
Edit: another user linked their blog https://news.ycombinator.com/item?id=29813349
class FooModel(metaclass=Base):
x = Field(type=int, primary_key=True)
y = Field(type=reference(BarModel))
The metaclass is basically a class that itself builds classes, which means it can be syntactically convoluted.
However, with __init_subclass__ you can write a thing that looks like a regular class with regular parent methods, but instead just gets a method called each time the interpreter encounters a new subclass, which lets you do things like build up that dictionary for your ORM.
This is from someone who has only ever read about, never used metaclasses, because they are widely regarded similar to git submodules. If you cannot really assert that you need them, you don't. They solve very specific problems, mostly found in library, not user code. A library can then allow user classes to be modified comprehensively. If you control the classes in the first place (not library code), you probably can do without metaclasses.
I’m a pretty good Python dev, but I feel in order to advance my skills I need to get into extensions … which I don’t have the time for.
Any suggestions on advanced Python resources?
* Fluent Python — takes you through Python’s core language features and libraries, and shows you how to make your code shorter, faster, and more readable at the same time
* Serious Python — deployment, scalability, testing, and more
* Practices of the Python Pro — learn to design professional-level, clean, easily maintainable software at scale, includes examples for software development best practices
I've also used metaclasses in the past to apply a decorator to certain methods of all subclasses, without needing to specify that decorator in the subclass declaration itself.
The user of the utility write this:
class Book(objects.Object):
title: Optional[str]
class Book(Base):
__slots__ = ('title', )
title: typing.Optional[str]
def __init__(self, title=None):
self.title = title
@classmethod
def from_data(cls, data):
title = data.get('title', None)
return cls(title=title)
def to_data(self):
data = {}
if self.title is not None:
data['title'] = self.title
return data
@classmethod
def from_oracle_object(cls, obj):
return cls(title=obj.title)
def to_oracle_object(self, connection):
obj = connection.gettype('Book').newobject()
obj.title = self.title
return obj
Another example is the builtin "Typing" module where the type system is used to provide a language for representing itself. "Union" itself is a type, and metaclasses allow you to set "__getitem__" on the "Union" type such that Union[Int|Str] is valid and represents a new type. Not that I don't know if metaclasses are actually used to build this system internally, but you could use it for many of the features implemented by Typing.
There are alternative ways to go about doing that because Python is so flexible, but IMO metaclasses (or __init_subclass__ as seen in this post) are the least "magic" of the magic ways to tackle problems like these.
Interestingly _SpecialForm uses a mixin _Final to prevent subclassing. This mixin is made using __init_subclass__
By creating an IrcMessageHandler metaclass, I can easily subclass it to encapsulate functionality into different classes without having to add boilerplate to load each plugin. There's a Plugins directory with a __init__.py that automatically loads all .py files, and the main bot just has to do `from Plugins import *` to load them all. Each plugin script merely needs a `from ircbot import IrcMessageHandler`, then creates a subclass. So there's a Calc class that adds a "!calc" command, Seen class that adds "!seen", and more. Each of them merely needs to include an "on_message" function that parses IRC messages.
To understand this fully you should try to understand how 'type' is the class of a class (which is also an object in python) and you should read how 'type' creates a new class, which is basically the job of your new metaclass should you create one.
Usually you don't need it. Also it's finicky to get right.
Something like __init_subclass__ has a more specific purpose, which I suppose is good. However it also sounds like the kind of thing you probably shouldn't be doing.
https://stackoverflow.com/questions/392160/what-are-some-con...
The language has flaws. But the function parameters are an joy and incredibly powerful. The validation and type coercion is excellent. The functions are extremely composable.
Overall, it's the epitome of "pit of success".
The same team that has all this powershell also has about 50k lines of Python. The Python has the stench of swamp farts, so I don't think it's team skill that makes the powershell work so well.
However, Python really doesn't shine when compared with other memory safe languages like Go, Rust, and Java.
Ultimately, Python is single-threaded by design. There are some hacks to get around this design contraint (multi-processing), but then you are adding an entire additional Python runtime to your system's memory for every "thread" you want to run. Even with asyncio, Python performs the worst when it comes to I/O.
I love Python, but it's important to understand that due to specific design considerations it is not and will likely never be a performant language.
reference: https://medium.com/star-gazers/benchmarking-low-level-i-o-c-...
Having been a C++ and python developer I see just as many footguns in both languages and have spent many many many hours dealing with slow, buggy code in either.
But I agree about the hordes of data scientists making copy and paste spaghetti. This is evidenced by companies deploying Jupyter notebooks in production, which essentially witnesses the fact that they've given up on getting data people to write quality production Python, and instead they treat their prototype spaghetti as sandboxed blackboxes that can be tested on inputs and deployed if they perform well enough.
Let's say you have this code:
class Creature:
def __init__(self, name: str, hp: int, attack: int):
...
def attack(self, other: Creature):
...
That is the problem with features bolted on to a language where they perhaps don't belong. Now, I know that typing helps when you have simple, functional code. So, this code will work:
from dataclasses import dataclass
@dataclass
class Creature:
name: str
hp: int
attack: int
I was really surprised with this. We need better review in regards to adding additional features that may be broken on certain setups.
def attack(self, other: 'Creature'):> We need better review in regards to adding additional features that may be broken on certain setups.
What does this even mean. The documentation explains that you need to do this, and it’s fairly obvious why you would.
Type hints are one of the best features added to Python.
Except, it is not that simple. It's not even intuitive. Typing is a great feature on paper, but in practice, it is just something we don't need in a scripting and glue language.
Not the most ergonomic, but there is a way to do it.
This is so hard to reason about and fix at scale. If you let other engineers run wild with this and build features with it, the time will eventually come to decom your service and move functionality elsewhere. Having to chase down these rabbits, duplicate magic, and search large code bases without the help of an AST assisted search is slow, painful, and error prone.
I spent several years undoing magic method dispatch in Ruby codebases. Tracing through nearly a thousand endpoints to detect blast radiuses of making schema changes impacted by CRUD ops on lazily dispatched "clever code".
I'm sure there are valid use cases for this, but be exceedingly careful. Boring code is often all you need. It doesn't leave a mess for your maintainers you'll never meet.
Python users tend not to behave this way, but seeing posts like this requires me to urge caution.
That kind of code is great if:
(a) you wrote it
(b) you knew what you were doing when you wrote it and
(c) you also wrote an extensive set of tests to cover the entire domain of whatever you were doing.
Comments are underestimated. I'm not saying they should be used as an excuse to use "magic" esoteric code, but sometimes such solutions are needed and it's much better to document them for future maintainers.
— Tim Peters
This quote ignores the middle ground of users who have a valid use case, but don't yet understand metaclasses. Not great advice for people who are trying to learn metaclasses.
There's a "basic" [1] understanding of Python's object model, one which understands that most of Python is actually syntactic sugar for calling certain special methods. For example, the expression a[b] is "really" just calling a.__getitem__(b). Except that's not actually true; the "real" object model involves yet more dispatching to get things to work. Metaclasses allow you to muck with that "yet more dispatching" step.
So when do you need metaclasses? When you need to do that low-level mucking--the kind of mucking most won't know about until actually needed. If all you know about metaclasses is kind of what they are, then you very likely haven't learned enough about them to use them to actually need them. Conversely, if you've learned those details well enough to need to muck with them, then you've also learned enough to the point that you can answer the question as to whether or not you need metaclasses.
[1] I suspect most Python users don't even have this level of understanding, which is why I put it in scare quotes.
Maybe he was also suggesting that people should generally not be using them. In my experience, it is exceedingly rare to need them. In 29 years of writing Python code, I think I've used them once or twice.
graph = {
key: {
p
for p in inspect.signature(method).parameters.keys()
if p != "self" and not p.startswith("_")
}
for key, method in cls._registry.items()
}
>>> class Test:
... def test(self, a, b, c=5):
... return a + b + c
...
>>> t = Test()
>>> inspect.signature(t.test).parameters
mappingproxy(OrderedDict([('a', <Parameter "a">), ('b', <Parameter "b">), ('c', <Parameter "c=5">)]))The one that bites me a lot is: I want to easily be able to write a decorator that accesses variables on a particular _instance_ of a class (such as a cached return value), and I want to take a Lock when I do it.
But since decorators are defined on the class method, not the instance, it has to do a bunch of work when the function runs: look at the instance, figure out if the lock is defined, if not define it (using a global lock?), take the lock, then do whatever internal processing it wants. It feels like decorators should have a step that runs at `__init__` time to do per-instance setup but instead I have to figure it out myself.
from functools import wraps
from threading import Lock
def my_decorator(f):
# Do whatever you want to set up the lock.
lock = Lock()
@wraps(f)
def wrapper(self, *args, **kwargs):
# Do whatever with the lock.
lock.acquire()
# Do whatever you want with the instance.
print(self.greeting)
return f(self, *args, **kwargs)
return wrapper
class MyClass:
def __init__(self):
self.greeting = "Hello World!"
@my_decorator
def my_method(self):
print("And all who inhabit it.")
>>> MyClass().my_method()
Hello World!
And all who inhabit it.
An example usecase is writing a @cached decorator that behaves well in multithreaded programs and can be used on classes that have many instances, with the caching happening per-instance.
@decorator
class SomeClass:
...
>>> import this
> ...
> Explicit is better than implicit.
> ...
> There should be one-- and preferably only one --obvious way to do it.
I agree that this is a more explicit way to handle metaprogramming in Python. It's analogous to method decorators, unlike metaclasses, which has no "metamethod" equivalent (AFAIK).
Which in most cases are a way saner alternative.
https://twitter.com/dabeaz/status/1466731368956809219
Which is a warning that people shouldn't buy his new book if they want a deep dive on Metaprogramming:
https://www.amazon.com/dp/0134173279/
Simon does a nice job demonstrating how "the __init_subclass__ class method is called when the class itself is being constructed."
class MainWindow(gtk.Window):
__metaclass__ = MetaDescribedUI("MainWindow.ui")
def __init__(self):
# now you can use e.g. self.lvFiles or self.btnOK
pass
"The Jimi Hendrix of programming"
For those curious about what metaclasses actually are, I wrote up an article with some motivating examples about this a little while back. Might be worth a read if you’re interested!
https://www.joe-bergeron.com/posts/Interfaces%20and%20Metacl...
People unironically created interfaces and various overloaded operations using dunder methods. Not only do you need to keep a list of them under your pillow, but you must also keep notes on how to construct the interface for each one.
And they look ugly as hell. In a language that prides itself on readability and user friendlines.
My suggestion is to ditch the OOP syntax completely and make a built-in analogue to a C struct. Take inspiration from dataclasses syntax. OOP in a dynamic glue language is a silly idea anyway. Of course, you would have to bump the version to 4.0.
class MyCustomOptionsFlow(OptionsFlow, DOMAIN="MYPLUGIN"):...
Very neat, not immediately obvious however :/
https://docs.python.org/3/reference/datamodel.html#object.__...