More interesting is "a patent held by the company states that the Facebook app uses voice recognition algorithm, which uses audio recorded by the microphones, to modify the ranking scores of stories in users News Feed." and their speculation that Facebook could soon reveal details about their use of surreptitiously recorded user audio.
Facebook makes a curiously specific denial about audio, which is that it is not used for advertising. Considering their entire business is basically advertising, what does that leave? But all they mean is ad selection. When they were found to be recording audio during the posting of statuses, I believe they claimed it it was so they could recognize the music you were listening to, and know something about your mood. So for a long time, I have thought that they use audio to select other content, like friend suggestions, or to inform the selection of stories that appear on your newsfeed.
However, I think that I disagree with you on whether or not sharing the data is important. If you are heat mapping me, like facebook and probably everyone else from Microsoft to CNN and FoxNews does, or you are recording me like everyone from Facebook to Samsung does, I'm sorry, I've got a problem with it. I don't care if you don't share that data. I don't want Samsung recording what's going on in my living room. Doesn't matter if the data isn't shared. It's just the principle of the thing.
It's gotten to the point where I actually purchased a certain model of Sony TV, because the teardown verified that there is no microphone in it. Then I tossed the remote control and got a generic remote with no voice control.
People joke about me being paranoid, but I'm not paranoid. Sheez... I'm old and boring, I know that no one cares about what's going on in my house or on my computers.
I'm just stubborn.
Why let the privacy invaders win?
I got carried away and rambling but I mean, come on. Mouse movements? Really? I suppose you have to give them credit they are creative in a very perverted sense.
I don't do anything im worried about people finding out, but that doesn't mean I want to let them listen in either. Maybe if someone was upfront about it from the beginning or showed us what they were truly doing, but seeing how shady it all seems to be makes me assume they are doing shady shit with it.
Edit: I wanted to add that I didn't intent to focus on whether the data is shared. I think FB having and using it is bad enough, especially if they're ubiquitous. Also, once anyone creates such data, other entities such as governments will seek to obtain it and likely do so eventually.
The media and general public seem to be 10-15 years behind when it comes to understanding how the things they rely on, and the tools being used to “improve” them, work
Though IMO, a lot of the blame is on Facebook and the whole lot for avoiding discussing openly in order to avoid fallout. Just asking customers too is out of the question, of course. Cause BIG Corps are smarter than their customers
Only once you’re “too big to fail” can you be honest about your shady BS
Does the patent really state that Facebook does that, or Facebook spammed the patent office with obvious ideas about how they could do that. Big tech companies have loads of trivial patents on stuff they have no firm plans to build, just to stake out IP territory.
> I believe they claimed it it was so they could recognize the music you were listening to, and know something about your mood.
When did this happen, and why wasn't it frontpage on HN and all the news sites?
As far as the status audio, I'm sure it has been discussed on HN. I don't have the time to dig up all the info right now but here is FB's take on that: https://newsroom.fb.com/news/2014/05/a-new-optional-way-to-s...
One bad thing about this system for Android is how much control Google has over permissions - for example, their own built-in Shazam...
I'm looking forward to technical solutions to verifiably disable and prevent this kind of tracking.
To those talking about "tracking UI usage": do a UX study. Sit some people down and watch how they use the site. Ask them questions about what works and what doesn't. Stop spying. You got all this damn money and you can't be bothered to actually lift a finger to do some difficult work that involves interacting with all those "dirty" people out there. FFS most people would probably be happy to fill out a survey if it actually would impact the product in a positive way. Creepiness is creepy.
All this leads me to think mouse movements tracking is much more widespread.
However I'm pretty sure this was advertised or at least acknowledged by Google in the launching of reCAPTCHA v2.
https://www.wired.com/2014/12/google-one-click-recaptcha/
> IP addresses and cookies provide evidence that the user is the same friendly human Google remembers from elsewhere on the Web. And Shet says even the tiny movements a user’s mouse makes as it hovers and approaches a checkbox can help reveal an automated bot.
[1] From Book "Silence on the Wire"
However, most biometrics aren't the best single line of verification. You still have to add a backup verification of some kind.
Examples, I was working on my car all weekend and my hands have new calluses on them and my finger print is off. I got a new keyboard and my typing pattern is different, etc.
I agree that this is something people should be more aware of and school is a good place to start. However it is up to browser manufacturers to fix this, not users.
Proof: https://panopticlick.eff.org
(You can always open the browser inspector and check network traffic for each page or, if you are using Chrome, dive into chrome://net-internals/ )
Stupid cat and mouse game. How difficult would it be for a bot to simulate a human's mouse movements? I suppose not very difficult.
Also, doesn't this conflict with rare types of input devices? Or people with a motor function disability?
> to also determine if the window is foregrounded or backgrounded
Shouldn't there be an API for that?
How difficult would it be for a bot
to simulate a human's mouse movements?
And even if the bot author had all that information, it would still be super hard to write an AI that accomplishes a given task in a way that mimicks a human successfully. It would mean to win a 'mouse turing test'.
Shouldn't there be an API for that?
This issue touches on the real privacy problem the net is facing. It's not the wrong cookies or privacy policies. It's fingerprinting. There is no technical solution to it.
Don't forget that Facebook's false positive rate should be very low. There are lots of humans on their platform, and they should all pass the test.
This makes it easier to construct a bot that will pass the test.
Using these metrics you could probably start to draw some characteristics of how your mouse acts based on what you are doing and where you are moving your mouse.
This could then probably be used to build some form of algorithm that moves the mouse for you with noise (accelerating up & down along the way, deviation from a straight line, stopping in the middle of the line, etc.).
You need to break ReCaptcha? Simple, you implement your own captcha on your own site that's frequently used and whenever you need to solve one you copy the challenge and present it to one of your users.
Same with recording mouse data.
It's an old idea even, very similar to https://xkcd.com/792/
You don't need to learn from all humans. You need to learn from very few (or just even one).
Not all problems are machine learning problem.
At least the mouse movements themselves shouldn't be difficult to do given a source of data. Simulating that you click on same FB UI elements as real people with same statistical properties on other hand is where you might be lacking the data to do it properly
But does Facebook track user during everyday session or just during some validation-action?
As for the latter, iirc there are blur/focus(-like) events for the window object. Maybe mouse movements gives them better confidence? Because of course you want to make absolutely sure your users are seeing all the ads.
http://idlewords.com/talks/website_obesity.htm An interesting read on this cat and mouse game.
I suspect this is part of covering their ass for GDPR. Pose everything as a security problem, so you can claim you have legit interest in tracking all of that.
> How difficult would it be for a bot to simulate a human's mouse movements?
Simulating? Extremely difficult. Perturbing pre-recorded paths is a bit easier to do, but requires pre-recording of a lot of paths. One of the fastest ways to get your Poker bot banned is to not fix this one way or the other.
> Also, doesn't this conflict with rare types of input devices? Or people with a motor function disability?
It still tracks the mouse movements, just now being able to classify their users as disabled or using an arcane device (both of which are interesting tidbits to add to your advertisement profile).
There is one : https://developer.mozilla.org/en-US/docs/Web/API/Page_Visibi...
Although I think it can be a privacy concern and somewhat of an anti-feature. For instance, Youtube uses that API to stop playback on mobile when the page or browser is not in the foreground. Of course, there's an extension for that ...
I also suppose not very difficult in principle (imitate any nearby human's movements and Facebook should not complain), but it is not within the focus of a general bot developer and therefore makes the whole project exponentially more difficult.
If I'm interested in the content of a page, I either swipe the mouse off to the edge of the screen or put it on an area of whitespace so I can scroll with the scroll wheel.
The thought that people are hovering their pointers over stuff they're actively looking at strikes me as odd. Oh well.
Being able to aggregate data or inspect individual sessions is a useful tool to learn how users navigate with a site.
Keyboard keystrokes get captured too but the systems are intelligent enough to filter out passwords and payment details.
I don't really like this form of monitoring either but I've seen it in several companies.
Sure. Except they are not.
Citation needed. But i did not realize that before, so thank you very much for this information, i will desactivate js on every page with a password field from now on.
It would also help out Facebook if someone sent them a daily minute-by-minute log of what I was doing. That doesn't mean they should go try and do it.
https://freedom-to-tinker.com/2017/11/15/no-boundaries-exfil...
So, it's for spying on users. What happens then?
A modern social network is a spy agency where you file your own reports on yourself in return for being able to read those of other people.
It's a very anti-privacy environment we live in right now. But it's for our own good or so we are told.
Yes
Centralised social networks are by nature spy networks in that they collect a huge amount of data on a large group of people, both data concerning those people as well as data relating those people to any of the others. As to whether a centralised social network uses these data for nefarious purposes or only to provide a service to the users is up to those currently in control of the network. Ownership can change, a once-benign social network can turn into a nightmare overnight as the data will be there for the new management to exploit.
To be fair, this makes sense from a UX standpoint. You don't want messages to be marked as read if you have your window open but have walked away from the computer.
Do they use some type of client-side library that caches data for a while and asynchronously uploads it occasionally? Or occasionally try to asynchronously sample the mouse position and just get a coarser set of data?
It seems like real-time requests that respond to mouse changes would create huge performance problems and/or be easily stopped with browser extensions.
Personally, I find this even more disturbing. Does this "only" apply to TVs where a user is logged on, or are they also building shadow profiles for any smart TV that comes with the Facebook App preloaded and is connected to the internet?
At least for Samsung and Sony, I can easily see them cooperating with Facebook for a negligible fee.
"But it's not nefarious..."
"But everybody else is doing it too..."
"It's not for surveillance, profiling and shadow profiling, pinky promise, trust us..."
It's a lot easier to get one company (or person) to stop a practice only they do than it is to get them to stop doing something that everyone else does too.
Facebook has ties to and influence over government, it is so big and far reaching that I feel it's right to be more concerned about FB doing something like this than other smaller players.
The scale of FB is what makes it a special case.
However, I think that the proper approach to something like this is educating users. Companies are gonna capture your mouse movements, it's not something we should legislate over, but users should be informed as to what it means to give companies like Facebook information about yourself.
Worrying about mouse movements when you freely send clear text messages to their data pile about your most intimate feelings and thoughts is ass-backwards.
Going after the largest offenders and making a big splash is more effective than doing nothing until you can get every company to simultaneously stop something.
No. Not only does facebook not get cover from "everyone else is doing it," they are actively perpetuating that cover existing for others. They are one of a few entities with the weight to set "industry standard" simply by changing their practices.
>while you can get up in arms about it, you need to get up in arms about the practice industry-wide
I am.
In other news, Google Tag Manager and crazyegg exist.
They said very specifically they don't record what you say for advertising. It was specific enough that it almost seems an admission that they use it for something else.
However, there are plausible explanations for your spouse getting ads for things she's spoken with her phone in the room - for example, the patient googled the medication later or wrote about it on facebook, and facebook knows thay your spouse and the patient likely had a conversation based on location data showing that they walked down a hallway together.
We’ve used inspectlet from time to time to help figure out where there are problems in our UI. It tracks mouse movements as part of a complete session. It’s been really helpful.
What's the easiest way to hack your browser to give no information or dummy information here?
There's also the difference between announcing "hey, look at this cool tech we have to make the web faster!" vs "we are legally required to admit that we've been watching you like a hawk, for... reasons."
It's a very manual process, but probably one of the most powerful tools for improving user experience I've ever seen. And typically for most businesses, you are keeping these sessions for 2 weeks or thirty days at most.
Quick search indicates the tracking information starts on page 84.
Page 86: "Device operations: information about operations and behaviors performed on the device, such as whether a window is foregrounded or backgrounded, or mouse movements (which can help distinguish humans from bots)."
We need a new science of hysterias. The internet provides ample data for robust analysis and prediction. Any takers?
"But Facebook is using it to signal advertising!"
Well. Everything you do on Facebook is used to signal advertising. I thought we were kind of all aware that was what happens with this free service. /shrug
document.body.addEventListener('mouseenter', e => {
console.log('this just in, erickj tracks mouse movements');
})