Ty: A fast Python type checker and language server (opens in new tab)

Had you checked it with Pyright previously?

At least for the moment, we aren't planning on a plugin architecture. We do recognize that there are some popular libraries and code patterns that aren't easily (or at all) typeable with the current state of the typing spec. We feel it would be more useful to help drive changes to the typing spec where we can, so that other type checkers can also benefit; and/or implement workarounds for the most popular libraries directly in ty, so that a library author doesn't have to rely on their downstream consumers installing a particular set of plugins to get good type-checker results.

(It's also more difficult to support plugins effectively in a type checker like ty, than in a linter like ruff, since any non-trivial use case would likely require deep changes to how we represent types and how we implement type inference. That's not something that lends itself to a couple of simple hook extension points.)

I also asked this question here: https://github.com/astral-sh/ruff/discussions/15149#discussi....

https://pypi.org/project/django-types/ is compatible with pyright without plugins, so it should theoretically work with ty. It might not check as much as mypy though especially with values() querysets.

This is a weakness of the Python typing system and not necessarily of individual typecheckers. Pyright has a policy of only implementing what's standardized, and the Python type system is simply inadequate to annotate most real world Python code out there. It's been years now and something as basic as properly typing kwargs is still not supported.

Ty could solve this if they rebel and decide to ignore the Python typing standards, which I honestly would appreciate, but if they take the sensible approach and follow the standards, it won't change anything.

My experience is this is nearly impossible, the solution is new packages written after typing was introduced.

I don’t know about SQLAlchemy, but for libraries like pandas I just don’t see how it can be done, and so people are actively replacing them with modern typed alternatives

What version of SQLAlchemy? SQLAlchemy v2 is built with with type-hinting support, I didn't have any issues with it when I used it few months ago

Only tangentially related but does anyone else here get very bothered when looking at the SQLAlchemy documentation? It seems so hard to find what kind of magic incantation you need to do in which order when trying to do a somewhat non-trivial query and I often just write the SQL I want and then tell chatGPT to rewrite it to SQLAlchemy operations but thats not really a sustainable solution.

Have you tried SQLModel?

You mean Python is not a language for production applications ?

For me, until it gets a production quality JIT, or PyPy and GraalPY get more community love, it remains a scripting language for learning on how to program, automating OS and applications tasks.

There is a formal grammar defined in the CPython repo, implemented in a language called ASDL:

https://github.com/python/cpython/blob/main/Parser/Python.as...

ty uses the same AST and parser as ruff. We don't use the ASDL grammar directly, because we store a few syntax nodes differently internally than how they're represented upstream. Our parser is hand-written in Rust. At first, our AST was also entirely hand-written, though we're moving in the direction of auto-generating more of it from a declarative grammar.

https://github.com/astral-sh/ruff/issues/15655

https://github.com/astral-sh/ruff/tree/main/crates/ruff_pyth...

https://github.com/astral-sh/ruff/blob/main/crates/ruff_pyth...

As in, how are we parsing the Python code into an AST?

CPython uses a generated parser. The grammar is defined in https://github.com/python/cpython/blob/main/Grammar/python.g... which is used to generate the specification at https://docs.python.org/3/reference/grammar.html#full-gramma...

We use a hand-written parser, in Rust, based on the specification. We've written that previously at https://astral.sh/blog/ruff-v0.4.0#a-hand-written-parser

It'll be separate (at least to start) — we want to be able to iterate on it rapidly. Long-term, a consistent toolchain is definitely important and something we're thinking about.

+1 for (eventually) baking it ty into ruff. In my mind static type checking is a form of linting.

For years I pushed black for formatting code. Once formatting was baked into ruff I ditched black. Having fewer dependencies to track and update simplifies my life and shortens my dependabot queue.

> Pre-alpha are much better as .dev releases.

No, they are correctly using semantic versioning to indicate pre-alpha releases. https://github.com/astral-sh/ty/releases https://semver.org/

It is the sixth alpha release. They haven't yet released a stable version – this is their sixth alpha release before that.

What am I missing here?

Either something about beanie babies or something riffing on "thank you". Couldn't ever make up my mind then basically forgot about it.

> such as claiming `datetime.UTC` doesn't exist)

This is a known issue — we're currently defaulting to a conservative Python version, and `datetime.UTC` really doesn't exist until Python 3.11!

https://docs.python.org/3/library/datetime.html#datetime.UTC

We will probably change the default to "most recent supported Python version", but as mentioned elsewhere, this is very early and we're still working out these kinds of kinks!

(ty developer here)

Currently we default to our oldest supported Python version, in which `datetime.UTC` really doesn't exist! Use `--python-version 3.12` on the CLI, or add a `ty.toml` with e.g.

``` [environment] python-version = "3.12" ```

And we'll find `datetime.UTC`.

We've discussed that this is probably the wrong default, and plan to change it.

Python is slow for some CPU-intensive operations.

There are some extremely CPU-intensive low-level operations that you can easily write in C and expose as a Python API, like what Numpy and Pandas do. You can then write really efficient algorithms in pure Python. As long as those low-level operations are fast, those Python-only algorithms will also be fast.

I don't think this is necessarily "cheating" or "just calling disguised C functions." As an example, you can write an efficient linear regression algorithm with Numpy, even though there's nothing in Numpy that supports linear regression specifically, it's just one of the ways a Python programmer can arrange Numpy's low-level primitives. If you invent some new numerical algorithm to solve some esoteric problem in chemistry, you may be able to implement it efficiently in Python too, even if you're literally the first person ever writing it in any language.

The actual problem is that it's hard for people to get an intuition of which Python operations can be made fast and which can't, AST and file manipulation are sadly in the latter group.

CPU intensive is not quite the right metric. What python is slow at is all the extra administration that comes with basic stuff like accessing attributes and function calls.

This gives somewhat counterintuitive results where declaring and summing a whole list of integers in memory can be faster than a simple for loop with an iterator.

But yeah writing stuff in a different (compiled) language is often better if that means the python interpreter doesn't need to go through as many steps.

mypy is compiled using mypyc. It does not run as Python code.

It's even worse (for python).

TS might transpile to JS and can always be split into a js and type annotation file but is it's own language developed in tandem with the type check based on a holistisch approach to find how to type check then and then put it into the syntax and type checker.

Thats not true for python at all.

Python types where added as annotations to the language many years ago, but not in a holistic approach but in simplistic approach only adding some fundamental support and then extended bit by bit over the years (and not always consistently).

Furthermore this annotations are not limited to type checking which can confuse a type checker (through Annotated helps a lot, but is also verbose, wonder how long until there is a "Annotated" short syntax e.g. by impl @ on type or similar).

Basically what the type annotation feature was initially intended to be and what it is now differ quite a bit (dump example `list` vs. `List`, `Annotated` etc.).

This is made worse that a bunch of "magic" is deeply rooted in python, e.g. sub-classing `Enum`. Sure you have that in JS too, and it also doesn't work that well in TS (if you don't add annotation on the dynamically produced type).

Lastly TS is structurally typed, which allows handling a bunch of dynamic typing edge cases, while Python is, well, in-between. Duck-typing is (simplified) structural typing but `isinstance` is a common thing in Python and is nominal typing...

So yeah it's a mess in python and to make it worse there bunch of annoyances related to ambiguity or to many ways how to do a thing (e.g. re-exports+private modules, you can do that common coding pattern, but it sucks badly).

And in the process, they ended up creating an extremely powerful type system that ~nobody outside of that original team can (fully) understand.

It's not mypy issue. Comparing to TS python typehints (spec wise) are a joke. It's started as bolted on adhoc solution and evolved quite chaotically. For example [1]. TS doesn't require a special decorator (sic!) to make your custom classes to be picked up by type checkers.

Or how make a wrapper function with args and kwargs to pass through?

[1]: https://docs.python.org/3/library/typing.html#typing.datacla...

through algorithmic improvements can also go a long way and if you are one of the first type checker which have to figure out the mess the python type annotation system is you will vast a lot of time on figuring that out instead of refactoring it's architecture to allow for algorithmic improvements

which brings us to another python issue, python is quite bad at such huge refactoring even with type checkers

but yeah python is by far the slowest widely used language, and for some use cases you can side step it by placing most hot code in C++/Rust extension modules, (or don't care because you are much much more network latency bound) but a type checker probably doesn't belong into that category

The CPython API is such a dumpster fire, even writing very simple modules the reference counting is very difficult to do correctly. The majority of python modules written in C are probably leaking memory somewhere but nobody knows.

Sure, but so far this has been a true criticism of every python type checker that isnt mypy that is production ready today

the thing is most (all) of the type checkers including e.g. mypy _do not_ support most crazy python ...

not because they don't want to or because it's to slow

but because it's not really viable without fully executing module loading in a sandbox, which might seem viable until you realize that you still need to type check `__main__` modules etc. and that its a common trend in python to do configs by loading python modules and grabbing the module locals as keys of the config or loading some things might actually idk. initialize a GPU driver :sob: So it's kinda 100% guaranteed not possible to do fully correct type checking for all project :smh:

But also python is one of the slowest popular languages (and with a large margin to any not also "one of slowest" languages). Only by moving hot code into C++/Rust is it fast, which often is good enough, but a type checker is exactly this kind of software where this approach stops working.

SQLAlchemy 2.x has direct support for mypy, it works out of the box, no longer needing mypy plugins. Many things in SQLAlchemy as are still dynamic and can't be type checked, but the native support works great where it can.

Any rough estimates of how much faster you expect ty to be compared to mypy? I'd be super curious to know!

I was also one of those people who, when first trying Ruff, assumed that it didn't work the first time I ran it because of how fast it executed!

I don't get why so many people go to bat for pyright, my experience with it has been pretty miserable. Open enough instances of it and you're in OOM city. It works, but often gets confused... and of course the absolute audacity of MSFT to say "let's go over to pyright, and by the way we're going to carve up some stuff and put it into pylance instead", meaning that it's totally not within the actual spirit of open source.

I would like to just not use it, but the existence of pyright as a _barely_ functional alternative really sucks the air out of other attempts' continued existence. Real "extend/extinguish" behavior from MSFT.

I tested it side-by-side on my ~100Kloc codebase.

Ty: 2.5 seconds, 1599 diagnostics, almost all of which are false positives

Pyright: 13.6 seconds, 10 errors, all of which are actually real errors

There's plenty of potential here, but Ty's type inference is just not as sophisticated as Pyright's at this time. That's not surprising given it hasn't even been released yet.

Whether Ty will still perform so much faster once all of Pyright's type inference abilities have been matched or implemented - well, that remains to be seen.

Pyright runs on Node, so I would expect it to be a little slower than Ty, but perhaps not by very much, since modern JS engines are already quite fast and perform within a factor of ~2-3x of Rust. That said, I'm rooting for Ty here, since even a 2-3x performance boost would be useful.

Pyright is only for type-checking and it lacks many features you'd expected from a modern LSP (I forgot which). Hence, it was forked and someone created basedpyright to fix it: https://github.com/DetachHead/basedpyright

Pyright is good, but it's quite a memory hog. (Yes, I have plenty of RAM on my machine. No, it has other uses during development, too.)

Pyright is incredibly slow in my experience, I've seen it take over a minute on complex codebases

in my experience pyright is unable to infer many inherited object types (compared to PyCharm's type inference)

I'm not sure if it's using your environment correctly, or are you expecting ~150 errors? Lots of import errors, and I'm guessing most of the other ones are errors because it couldn't infer what was imported.

how does it compare against mypy? is it much faster?

Can also do

   uv tool run ty

If your $PATH sucks

There's a good chance many of the errors we emit are incorrect, at this stage. Lots to do still!

Modular raised $100 million to build heterogeneous AI infrastructure; building a language is just a subset of their goals.

disagree. modular has a different focus, to turn python into a systems language.

you could even say that astral and modular focus on two extreme ends of the developer experience spectrum - just making python tooling faster, vs making python-ish code faster.

Yeah, iirc he was the first to go ‘what if we had cargo for python?’

I mentioned it in my initial comment. Ideally, I would like to use the same type checker/LSP for a language everywhere (any of my local editors, CI/CD).

This is especially important when working in a team setting.

It doesn't feel great to use a forked type checker/LSP that's not enforced in your org's CI/CD. And it also doesn't feel great to force the forked type checker onto the entire organization when only a subset of folks may be using a forked vscode editor.

This might actually be a decent business model. Sell hard goods to fund your habit of making excellent software tools.

Fingers crossed this isn't a joke.

From my experience it's VERY rare for people to pay for anything unless they're forced, I do the same thing even though I'm very aware of the need for compensating creators.

Talk is cheap, and people talk a lot about supporting projects.

Maybe if we could make some kind of statistics over the number of projects that were abandoned because maintainers didn't feel like working and dealing with random people for free anymore. Make the consequences of freeloading visible somehow.

Yes, we've talked; I know a number of the pyrefly devs well. Ty had already been months in development when pyrefly development started. We discussed collaboration, but they decided they needed to do their own thing in order to move quickly and ensure it would serve their needs, which is totally reasonable.

That's a big part of it, but there are also several areas where we're trying to innovate on functionality as well as speed. I'd personally be pretty disappointed if the only thing we had to offer at the end of all this was a type checker where the only value add was speed. We've also got first-class support for intersection types, and quite a different model to other type checkers regarding when and whether redefinitions are allowed, for example. We believe there are significant areas where typing can be made more usable and easily adoptable than it is today.

A type checker is much more algorithmically bound / difficult to parallelise, but I'm sure there are still wins to be had in the future.

Is it really that hard to believe that people like the language and the problems it solves for them?

You guys take your time and enjoy yourselves :). You have a proven record of making good tools, so if this part takes a while, let it take a while.

The code is definitely open source from a licensing perspective, but we are also trying to ensure that a healthy community forms around our tools as well. We've been developing ty in the open for the last year or so, and it already includes significant work from external contributors. This is definitely not a project where only Astral-paid engineers can contribute.

I wonder how they'll handle situations like Django, which don't seem expressible via stubs.

not specific to just your answer but why do people mention ruff?

Ruff is a linter which (intentionally) does close to no type checking.

So you pretty much have to pair it up with a type check to get any even just half way decent static code analysis.

Not supporting plugins for a type checker to me is a plus. It’s quite frustrating that some Python projects only typecheck if you have plugins. That is a major source of frustration.

CI/CD, private repositories, providing hosting. These are the options used by similar companies.

But I like that they’re focussing on creating something useful before chasing revenue. Once they’ve got a single tool that provides a consistent dev experience for Python developers and it’s widely adopted they should be able to pursue monetisation easily.

We're living in the information age. Not sure when or what the next age will be, but you get the idea.

in the tolkien legendarium they may be over 3000 years.

a year (probably)?

An age is a long amount of time.

"An age" is probably an attempted cleaning-up of "a coon's age."

https://grammarist.com/usage/coons-age/

The VS Code extension is linked in one of the above comments. Accepting that it’s early days and all, but … if someone _really_ hated Pylance, could they replace it with ty for hobby projects and get most of the same headline functionality?

Thanks for all your great work! Love ruff, rye/uv.

We'll also discover your venv if you: - Activate it manually (`source .venv/bin/activate`, etc.) - Set the `VIRTUAL_ENV` environment variable - Or use a command such as `uv run` or the equivalent from pdm/poetry/hatch to run ty (these project managers usually implicitly set the `VIRTUAL_ENV` variable to point to the project's virtual environment before executing any commands)

How about put it on the roadmap for v4?