Apple releasing segmentation/pose for humans and animals, embedding for 27 lang (opens in new tab)

There are multiple apps in the App Store that do this. I spent last year implementing pose detection in an exercise app and we used both Apple’s pose detection and a 3rd party’s. The pose (each point of the human form) itself was sent to a machine learning backend at around 30 fps, analyzed, and data returned at about the same speed using gRPC. Each exercise had a set of specific feedback for both positioning (“Stand facing the camera with you arms at your side/Stand sideways to the camera/etc”) and form correction (“Raise your right arm higher above your head etc”). Feedback was spoken out loud to the user and there was a relatively complex set of rules governing which feedback got priority and how often feedback was spoken. I also implemented an on-screen “skeleton” of the user’s human form points that rendered on top of the camera view. Pretty fun project from a tech point of view.

The “there’s an app for that” world where small to medium sized teams can build a very iOS native app that takes advantage of the latest and greatest of the device is long gone.

It’s kinda weird how Apple doesn’t realize this and continues to build for that world. Maybe if they were willing to shift on their % for devs that do build that way but unless they did there just isn’t the audience buying apps outright and the only ways to profit are tricking people into abusive subscriptions or building on ads and their personal data.

Until then no idea why any dev would build just for the Apple ecosystem and not something agnostic.

It’s telling to me that the biggest tech apps of the last 2 years all ran web/desktop first.

The big question is whether it's even capable of making recommendations like that. You'd have to combine it with your own model.

Having read books on strength training and tried to learn stuff like squatting perfectly myself I'm skeptical it could be to grasp the nuance.

But for dancing and other stuff where it doesn't matter as much it could be useful (health/safety wise when carrying load).

There a home gym setup called Tempo that claims to do this. I have the scaled down mobile app version so idk if the full setup is more informative, but it doesn't really give a lot of feedback at all. It basically just tracks the movement of the dumbbell plates that they send you to count your reps, and if you aren't moving the weight to the correct position your rep won't count. It definitely doesn't do anything like correct your form based on your body position like "hey straighten your back" or anything

Hey, I'm the founder of Guru (https://getguru.ai/), a video AI dev platform. Developers are using our movement APIs to build some cool form feedback apps, including NFL coaches.

- Demo: https://www.formguru.fitness/video/c96fa975-fd9e-4912-8f60-1...

- Blog: https://blog.getguru.ai/guru-sports-powering-the-top-prospec...

- Customers: https://www.cadoo.io/, https://www.breakawaydata.com/, https://pharosfit.com/, https://www.producthunt.com/posts/fitx.

We've trained our own models (and customers can finetune them), but it exports cleanly to iOS (and Android!).

One of the reasons that I write native, is so that I can access stuff like this. It usually takes quite a while for hybrid platforms to catch up.

That said, I don’t have an immediate need for this particular SDK, in the project I’m developing. I just like to have the option to integrate stuff like this.

Also, I’m not a “bleeding edge” developer. I’m still using UIKit/AppKit/WatchKit (as opposed to SwiftUI), and my software supports one OS version back, upon release.

I'm part owner of a company (Halterix) that used pose detection (alternately via smartwatch accelerometer) and machine learning to quantify how well exercises are performed.

We built a demo app for use in physiotherapy to improve outcomes and ran a few clinical studies. The detection accuracy was excellent and patient reception was warm.

There are a number of competitors, some with multi-sensor systems targetted to pros, some with vision systems, etc.

We met with all the big fitness app makers and found generally while they weree somewhat interested in pose detection/accuracy assessment and feedback, it's not at the top of their list of priorities to implement (even to incorporate our 3rd party service).

Proper disclosure: I had been formerly involved with them.

Try Kemtai.com.

There's a demo section at https://app.kemtai.com/sample-workouts

We took workout experience super seriously, and in my biased view, got it to be a usability joy.

For a more immediate and quite practical solution, I've had good results from simply taking videos of various movements and watching them right afterwards.

So hire a personal trainer for a session or two?

> it seems developers are mostly porting web apps to all platforms ignoring neat but platform specific apis like this.

I would be weird to have your social app trying to correct your pose ;)

Can download an Xcode project and view documentation here: https://developer.apple.com/documentation/vision/detecting_a...

> static let cat: VNAnimalIdentifier

This feels funny to read, like from one of those inheritance tutorials on object oriented programming

Seems logical, most people interact with their pets which are mostly cats and dogs.

Horses will probably be next

According to the keynote the first party app is going to recognize the family cat/dog as a "person".

Why do you feel that's been dofficult?

CoreML has various models already built-in, although as black boxes that accomplish some task like OCR or rectangle detection. There's also a "feature print" model which I believe are intended to be used as hard-coded features for simple ML tasks. In either case I strongly suspect that when you use them, they're not being embedded in the app.

Another thing to consider is that you don't have to embed the model in your app; at least in CoreML you can download (and update) the model weights over the network.

On Android, you can choose to include the ML model dependency "bundled into your app in compile time" or shipped through Google Play Services. Saves many megabytes. Drawback there: if the device doesn't have Play Services, nothing works. Also, on first download it takes some seconds to work.

There's nothing technically stopping us from doing that even 10 years ago.

MTailor [0] is/was a company that, using your phone camera, could measure you for pants/shorts/shirts/etc... it was a YC company 2014 and also on Shark Tank.

I hope the AI hype brings back some of these sort of use cases that might have been ahead of their time.

[0] https://apps.apple.com/in/app/mtailor-custom-clothing/id8160...

Turns out they have already [1]

https://developer.apple.com/documentation/naturallanguage/nl...