Leaf: Machine learning framework in Rust (opens in new tab)

First of all no one ever expected machines to beat humans at Imagenet. At least not this soon. It's an amazing accomplishment, because Imagenet is high resolution pictures of many different types of objects. Which is very different than tiny photos or pictures of digits.

Second the examples were produced by scraping Flickr. Then mechanical turkers were asked to confirm if the object was in the image or not.

There are many images that are kind of ambigious, or contain multiple objects, so humans don't do perfectly. One researcher tried to estimate human performance, and got about 5%. Which has been beaten by computers now, by a lot.

> how is it possible to learn superhuman performance in anything from examples of mere human performance at the same task? I don't believe in magic.

Computers could be better at assigning probabilities to ambiguous examples. In particular, for an image that is very ambiguous for most humans, maybe a computer would assign 99% probability to it (hence it would be only a little bit ambiguous).

Ensembles of humans can outperform the average human, and in the same way an algorithm trained on data labeled by an ensemble of humans can outperform the average human.

It is much faster than humans

It's code that you write (typically quickly), that you know will need to be re-written at a later stage. It's debt that will need to paid at some stage in the future. You didn't do it right first time.

Technical debt typically arises because the code was poorly structured or the programmer used the wrong tools/libraries (from a longer-term perspective) or didn't abstract when she should have. The current obsession with MVPs has led to an increase in technical debt.

Yes. https://en.wikipedia.org/wiki/Technical_debt

I've seen it firsthand. Basically, it's the accumulation of suboptimal code, over time, usually due to time constraints imposed by management. In short, any time you do a dirty hack just to get something working and meet a deadline, and then don't find the time to refactor that code into a working non-hack, you have piled a bunch of manure onto the technical-debt heap. But it also seems to be a side-effect of normal code accretion to a codebase while on a team- in other words, there seems to be no way to avoid it entirely. It's like cancer, in biology. ;)

TD-ridden code is often not modular, not unit-tested, has many dependencies (spaghetti) which are then difficult to remove or replace and tend to trigger cascading bugs/failures, has too many responsibilities, has very long methods/functions, uses mutable state (changes global state which can then impact other parts of the codebase or make concurrency impossible), or is otherwise difficult to maintain.

An example of "working" tech debt is the "God class" in codebases, the model that the entire business depends on but which is over-laden with responsibilities. The risk to change it is too great (due to the business dependence) so it becomes a constant thorn in the side of maintaining the code.

The "debt" part comes from the fact that at some point you are expected to "repay" it (via costly man-hours of refactoring work). The benefit of doing so is potentially multifold, though: Faster/more modular/better-written code, faster tests (and therefore better productivity), better designs in general, more resilient code, more maintainable code, less buggy code, etc. etc.

The only known resolutions of tech debt are costly refactorings or global rewrites. The way to reduce the risk there is to first unit-test the existing code. These books help:

http://smile.amazon.com/Growing-Object-Oriented-Software-Gui...

http://smile.amazon.com/Refactoring-Improving-Design-Existin...

Think of it as 'code rot'. It's quick and dirty fixes that will have to be fixed later, taking longer overall than if you'd just done it right to start with (hence 'debt').

Tightly coupled code is how I define tech debt

A system like that would be surprisingly hard to build. The problem wouldn't be the ML algorithms - it would be just about everything else. A few things you need to solve robustly to build your counter:

1. The "same background" doesn't really exist for most cameras in most settings. Changes in illumination alone will make segmenting the background tricky. Moving objects in the scene will also be hard - think fountains and trees in the wind. Google for "foreground-background segmentation" to see some papers on this.

2. I haven't seen anyone use recent ML algorithms with less than high quality images. That may not matter, but it could matter a lot.

3. Extending recent ML algorithms to work with video at a high enough frame rate to be useful (10Hz at a minimum) may or may not be easy.

I'm sure that what you're proposing could be done. But I think that the number of small annoyances you'd hit would probably discourage most people who aren't treating the problem as a research exercise in Computer Vision.

In the scale of computer vision problems, the stationary camera case is relatively easy. It's not too hard to isolate moving objects from a background, it's not too hard to decide if an object is a person or not, and it's not too hard to keep track of an object once you've identified it. You would still have to handle overlapping people, scene illumination changes, etc, but these can be solved and have been done before.

If you would like to play with some of this stuff, take a look at OpenCV. http://opencv.org

I actually think this is quite do-able and has been for a while. Although deep learning has revolutionized object recognition, face detection has been working reasonably well for some time, e.g. your cell phone camera or Google street view face blurring.

Yes. The general task of looking at arbitrary images and labeling objects (from a set of known categories) in those images is called "detection." In fact the problem you described is easier, because there's only one category (people), and the system only needs to provide a count, rather than provide bounding box rectangles around each object (which is what the standard "detection" task entails).

Convolutional neural nets are the state of the art for this, specifically deep residual learning (http://arxiv.org/abs/1512.03385). It requires a good deal of background to understand what's going on and tune/implement the models, though, even if you just use the frameworks already out there. You probably don't even need that much data - you can probably grab pre-trained models and train them on a small additional dataset you collect.

They can definitely handle arbitrary backgrounds, although having a standard background makes the problem even easier, again.

Most deep learning computer vision algos are trained on 256x256 images, so having even larger images is just fine (you can downsample, or maybe even add up the predictions of different crops).

The graph in the readme is outdated, you can see the version with Torch/CuDNN v4 here: http://autumnai.com/deep-learning-benchmarks

And I don't believe the first point counts as deceptive; the bars are ordered by Forward ms, not by the sum of Forward and Backward. In both CuDNN v3 and v4, Leaf is faster than Torch by that metric (25 vs 28 for v4, 31 vs 33 for v3).

Yes, on their site they post Torch CuDNN v4 as faster than Leaf [0]. Seems exciting for an early release.

Can it get much faster than something like Torch? I would think if CuDNN is doing most of the computation time it would be hard to see big improvements. Perhaps go the route of Neon and tune your GPGPU code like crazy [1, 2], or MXNet and think about distributed computing performance [3].

[0] http://autumnai.com/deep-learning-benchmarks

[1] https://github.com/soumith/convnet-benchmarks

[2] https://github.com/NervanaSystems/neon

[3] http://alex.smola.org/talks/NIPS15.pdf

> Leaf with CuDNN v3 is a little slower than Torch with CuDNN v3, yet the bar for leaf is positioned to the left of the one for Torch

I think that's because they're sorting by forward time rather than forward+backward. That would also explain why in the Alexnet benchmark Tensorflow (cuDNN v4) is to the left of Caffe (cuDNN v3) despite having a much taller bar overall.

A DSL based format has some advantages as it easy to get going with building networks. However you are then constrained by what the program that interprets/executes the DSL supports in terms of loading/saving data, solvers etc.. If you want to do something more dynamic e.g. AlphaGo then you have to go back to a "real" programming language anyway.

That's not to say that Leaf won't have a DSL at some point, but we will wait until the features of the layers are a bit more stabilized and we have more clearly mapped out what goals we have for a DSL.

By what metric does your CTO consider Rust to be more complex than Scala? A lot of Scala's complexity has to do with interfacing nicely with Java, and Scala has a lot of implicit behavior and TIMTOWTDI-ness that Rust deliberately tries to avoid. Odersky has even said that he's hoping that he can remove many features from Scala in the future.

It has less to do with complexity and more go do with REPL/Jupyter notebook support. Rust is a compiled language and you won't get some of the ease of exploratory data analysis you do with something like ipython.

I can use something like pandas or autograd to experiment with new optimization functions in seconds. For these big NN models it takes hours to days to wait for your model to train so squeezing out more performance is worth a more complex language.

You are right, batchsize is important and we should make that more clear.

The numbers in the benchmark are taken from our deep-learning-benchmarks[1] which we are still in the process of building up. It might actually make sense to test the same model with different batch sizes. The current benchmarks are based on the convnet-benchmarks[2] where the Alexnet model has a batch size of 128. (Alexnet was chosen because out of the benchmarks that's the one I am most familiar with, since it small enough that I can work with it on my Laptop)

In some informal tests Leaf was generally faster than other frameworks in smaller batch sizes, but no benchmarks that we could publish with confidence yet.

[1]: https://github.com/autumnai/deep-learning-benchmarks [2]: https://github.com/soumith/convnet-benchmarks

The interface code to the C libraries (which is written in Rust). We are however optimistic that there will be Rust libraries popping up in the future that outperform the current C implementations. (Optimistic as a Rust user, not as developer as Leaf)

We would love to have them and compare their performance with recurrent layers of other frameworks[1]. There exists an issue for the implementation of recurrent layers in Leaf (#73)[2].

[1]: http://autumnai.com/deep-learning-benchmarks.html

[2]: https://github.com/autumnai/leaf/issues/73

We changed the title to say "in Rust" because someone else complained about "for Hackers". I suppose we could take both of them out, but the project highlights its Rustiness so this seems more representative.