Fortunately one of our engineers figured out we could get our demo rigs working by setting the clock back a few days. This could have been a huge disaster for our company if we hadn't found that workaround though. Pretty annoyed with Oculus about this
By the way, do you have any links to your surgical training startup? I'm doing some research into VR/AR for surgical telementoring and training and would be interested in seeing how it's in use.
We've had a fair amount of press coverage in specialist press recently so a search for Osso VR should turn up some recent articles too.
It doesn't help that vendors are generally nervous about liability in medical equipment (this fear is often unfounded, but persistent). As a result, vendors of commercial and industrial equipment generally don't want to engage medical device OEMs with engineering and customization support. If there had been that sort of support in this case, Oculus might have made a custom build without the cert check, just as a de-risking measure.
This vendor reluctance is especially present at the FDA Class III (high risk device) level - most vendors outright prohibit use of their devices in these products. It's an open secret that this still happens anyway in a wink-wink nudge-nudge fashion, just without vendor support - which is arguably worse, but it keeps the lawyers happy.
The real MVP of this story. Sometimes a dirty hack is good enough.
This problem is deeper than forgetting to update it. It should never have caused a failure in the first place. Just the fact that the device apparently can't function at all without the internet is a problem too.
On the other hand, maybe this is really a lazy feature. It's probably a good idea for the system to disallow both incoming and outgoing network traffic to any program written in a non-memory-safe language that hasn't been signed in the past couple of years. The lazy version of this feature is just not to run any program not signed in the past couple of years.
Edit: Requiring a timestamped signature on the signature also makes it pretty easy to add auditing functionality to the timestamp server whereby the publisher can detect unauthorized signatures due to their private key being leaked/stolen by criminals or governments. If the timestamp server's logs show a signature by your key that you don't recognize, then something has gone wrong. On the attacker's side, they need to either steal the timestamp server's private key or publish their malicious signatures for scrutiny.
Isn't Oculus owned by Facebook? Of course it has internet-based mandatory data collection, er uh excuse me, license something something.
Except TLS certs. That's the whole point of those certs having an expiration date in the first place.
That’s exactly how they are supposed to work. In the public sector we rely heavily on certificates for inter sector communication for instance, if certificates kept working despite being invalid it would put security at risk.
You’re supposed to build your software with an enterprise certificate store in mind though, meaning you can auto renew and distribute certicates when needed.
I really don’t see the point of adding a certificate to your television though, even if it is a tv that you wear on your head.
Some might call this a feature.
https://msdn.microsoft.com/en-us/library/windows/desktop/bb9...
I bet they use certificate pinning.
Process A launches process B and checks against a pinned certificate. This is even more secure than just using the windows code signing stuff.
Problem is, when their cert expired, they were supposed to renew the same cert. Instead, somebody got a new one and signed the build of process B.
The device automatically downloads process B, but then the certificate pin check fails when it tries to launch it.
All the security guides that tell you to do certificate pinning need flashing neon signs explaining this problem. You can't pin certs if you intend to ever change certs.
A lot of applications and environments seem to be built with the assumption that they can add arbitrary complexity to their interface, since they're only going to be used by "experts" who can be expected to know everything of relevance and work through a thick documentation to understand the system. In truth, the "experts" who use your programs are going to also be using a dozen other applications, each with their own piles of documentation (or equal amounts of lack-of-documentation,) and have little brain-space left for the intricacies of your framework. So, they're going to use your system while knowing the minimum possible amount about it; if that system contains traps that cause problems for this kind of user, that's bad design.
I think we're waaaay past the "wondering" part.
When I read the comment I was immediately flabbergasted: no, someone else fucked up. It's not my fault someone wrote software that sets up undocumented traps for me to fall into. Or provided three ways to do something and two of them are not recommended OOTB. Or is primarily documented by third parties.
Whether you are writing SQL or graphics code you are constantly told "just express what you want to express directly, and the system is smart enough to do things as efficiently as possible".
But that might not be very efficient at all. The people who write "the system" have to write software that does specific things in specific situations and there will be endless cases which cannot be dealt with efficiently. And the more the interface hides the implementation, the less likely it is that those cases will be obvious.
The problem lays in the IP. It's considered to be a vital asset so that when a company goes belly up it will survive kept years or decades in a safe by law firms in the hope someone will buy it, or just to make profits through litigation against infringers. Unfortunately this has a deleterious effect on products derived from that IP, the people who bought them and the people living where the unusable products will be trashed.
Just recently picked up a Rift. I love the hardware and their exclusives are top notch, but this confirms my suspicions that their backend is super goofy.
They sell Rifts at Best Buy and want to pretend that it's a consumer-ready product, but here's why I am recommending people stay away for now:
- Non-existant repair or service out of Warranty.
- Basic things in the platform like changing your name or photo don't exist.
- Lots of non-response over other basic features requested by the community.
- Questionable future investment in the platform or hardware. It sounds like they are moving their efforts towards "lighter" experiences.
In short, it feels like being a legacy customer for a new product.
Would anyone with a straight face say the same about Nintendo's exclusives?
I don't see anyone else in the market funding great things like Medium, Quil, Lone Echo, Robo Recall. Feels it's just cutting off your nose to spite your face to complain about this.
Oculus is producing these at a loss given the current size of the market hoping it will pay out in the long term by growing a healthy ecosystem...it’s the only way
This also suggests that if the decide they stop supporting it, eventually the software will stop working due to these certificate errors for which will then there be no fix.
Still need the sensors, but those aren't that large.
https://www.theverge.com/2017/10/12/16463844/oculus-santa-cr...
It's not bricked, bricking means it's as useful as a brick as in the actual firmware is corrupted beyond repair.
But meanwhile my 10 year old nephew is having life shaping experiences with the Rift I set up for him.
I've been surprised how many investors / technologists view VR as a niche market for enterprise / porn / limited gaming appeal.
- Owned by Facebook, who will use it to gather as much data about you as they can.
That.
My Rift is scratched since first use (I swear it came scratched) and I've never been able to have them acknowledge that the device can be scratched into a blurry mess. "Oh, that's only regular use wear and tear" ...
Lens cannot be replaced or repaired (and they could during rift dk1 era).
Can you explain what this means in practice? I'm not super familiar with Rift but I'm curious.
That's not the future. That's the present! Which is even more worrying.
Generally this attitude doesn't backfire, because individual users loosing access to their data, their accounts or their software can be simply dismissed. But in this case it happened to everyone at once, so it's suddenly a big deal.
If one doesn't, well, she isn't much of a "security expert" after all, is she? Firewalling off TCP port 80/443 at your perimeter firewalls isn't a very good solution if you're an e-commerce company selling your product on your web site -- and the "security experts" know this.
Because Mark Zuckerberg said so, that’s why.
More relevant by the day: http://locusmag.com/2018/03/cory-doctorow-lets-get-better-at...
* data is not tampered
* signing certificate was time valid at signing time: signing time is within signing cert's validity
* Neither certificate was revoked before signature generation
* both, signing and timestamp certificates chain up to trusted root CAs (regardless of their time validity, just must be in trust store).
Nate Mitchell of Oculus posted on Reddit saying "We're working on resolving this issue right now. We'll keep everyone posted on progress here." https://www.reddit.com/r/oculus/comments/82nuzi/cant_reach_o... . Top-level of that thread has a workaround involving setting the clock back or using a utility called RunAsDate to fake the clock for a single application.
https://docs.microsoft.com/en-us/windows-hardware/drivers/da...
EDIT: Looks like the standard TTL for these code signing certificates is none, 1, 2, or 3 years.
https://www.entrustdatacard.com/products/categories/digital-...
https://www.globalsign.com/en/code-signing-certificate/
https://www.instantssl.com/ssl-certificate-products/code-sig...
Fortunately it was a problem with the tests, rather than the code itself, but these things do happen.
At my old job, we had a bunch of tests fail when daylight saving ticked over. For some reason, some things were using local time, rather than UTC. We also had a test that would fail if the minute was the same as the hour.
Time is hard
We furry 'self-reproducing' (YMMV) mammals are simply not ready for all of this.
This was all years ago, so my recollection may be fuzzy, but I spent entirely too much time futzing with SIP traces and certs. Weird, weird things can result from time inconsistencies is my takeaway, however.
Presumably this is what Oculus thought and so how they got in this mess.
> If your antivirus software restricts the file from opening, temporarily disable your AV and continue.
Good Patch Procedure, 2018.
Please, please don't say: "Our teams apologize for any inconvenience this may be causing you"
instead opt for "Our teams apologize for any inconvenience this caused you"
They soft bricked every single headset worldwide.
"Oopsie we may have caused you inconvenience"
vs
"We're sorry for the damage"
Including the word 'may' shifts the sentence from apologising for causing actual inconvenience, to apologising for causing a minor risk of possible inconvenience, which is not what you are trying to convey after selling someone something for a lot of money and then remotely breaking it at no notice.
In many customers, it is likely to elicit a response something along the lines of:
"Any inconvenience this may be causing? I'll give them may be causing. The fucking thing won't boot. May be fucking causing. I wasn't using the damn thing as a doorstop."
EDIT: presumably you need your client apps/libraries in the field write back when they use a cert that is <X months away from expiry.
I use simple Nagios checks for keeping an eye on certificate expiration. It's simple to set up checks for new hosts/services and I have them set to trigger an e-mail alert 30 days before expiration (20 days for certificates from Let's Encrypt). It does the job; I have yet to wake up one day to an expired certificate.
Apparently, this ("send me an e-mail before my certificate expires") is also the sole reason some companies even exist (i.e., it is their only product/service). It amazes me that this is something folks will pay for.
Obviously it should be part of a handoff process when you leave, but companies aren't always good at smoothly handling transitions like this.
(disclaimer: work for Azure, but not on Key Vault)
If it’s a much longer time scale, people start to forget that it’s even possible for stuff to expire.
If my fridge filter can display a little reminder light on a timer every few months, cryptography-dependent devices might need something similar. That way, your customers could know in advance and be asking you for an update.
So many comments agree that (a) security is hard, (b) countersigning with a timestamp server is easy to miss, (c) countersigning makes build processes difficult, and (d) they've done or seen similar things in other apps/companies.
This sounds like a classic UI/UX issue for developers around a literally mandated and mission-critical requirement of the OS.
At the least, MS should provide a validation tool to surface errors or risks before production. Better, signtool.exe should make omissions (like a timeserver) very difficult and make them an override, not a default. Best, they would do both.
I don't agree that the OS should reject non-timestamped signatures as faulty per se (and throw an error), as that puts the burden on the user to understand a developer's mistake. Sometimes running without a timestamp may be desirable - ultimately that's the dev's choice.
It should just be a choice made explicitly.
[0] https://msdn.microsoft.com/en-us/library/windows/desktop/bb9...
However, this affected only one single customer of ours and we had a fix within a couple of hours. -- I certainly learn from this mistake.
I thinks IT is used to managing HTTPS certificates, domain name auto-renewals but app level certs are more of a new thing.
Edit: I don't follow Windows, I'm really curious what the consequences for stuff like this can be generally.
I don't see credit on my Oculus account? Am I supposed to have received it already? Or is this maybe because I don't have payment method added to my account?
It looks like their auto-updater used the same cert though, so they can't distribute it as a normal update. They're probably figuring out the least sketchy/most automated way to distribute it right now.
When this is all said and done, there will be a handful of people who will never, ever forget to use the /t flag in signtool.
[1] http://www.brucebnews.com/2017/11/look-at-the-red-flowers-wi...
It's not just people shrugging it off, many are defending this as being a perfectly fine state of affairs.
https://en.wikipedia.org/wiki/Transport_Layer_Security#Digit...
This includes phones, cars, self-driving cars, watches, farm equipment, computing devices and anything marketed as an IoT appliance.
One glitch, as minor as an improper system time, and you’re dead in the water.
The big question is why on earth can drivers that have been verified and are already installed in your system can suddenly stop working? If this mechanism is intended to protect against malware disguised as drivers then it's already too late. The malware had several years to exploit your system.
Expiration after installation simply doesn't make sense for code signing. The signed executable won't change unlike a website. The driver is always going to have the same file hash, forever.
It makes absolutely no sense to the end user, acting as possessor or potentially a reseller of an object, since the very premise implies that an owner should not be provided total control over their device, that it's never really "theirs", and that a vendor should retain the capacity to take a "sold good" away from the owner, under the guise of expected behavior, built as designed, effectively converting a sale into a rental, in time, perhaps after statutes expire.
It's effectively a back door for manufacturers, so that they can count on well-made products not lasting forever, not in museums, not for resale, not for nostalgia.
