I'll always remember what Mikey told us in December, after the site was back up, could handle a non-trivial amount of traffic, and people who wanted health insurance could finally get it:
"1 in 1000 uninsured people die each year. It's not an exaggeration to say that due to the work we're doing here, 5,000-10,000 people will live to see the end of 2014. You should be proud of what you've done, but we should also all be grateful to have this opportunity."
We're all grateful to be here, but there's a hell of a lot more work to be done.
If any of you out there are an amazing software engineer or SRE, and want to help make our government work better, please shoot me an email: brandon@hcgov.us!
This probably a significant exaggeration. It is based on a 2009 study[1] which examined correlation, not causation. It did not control for many factors that may be relevant (e.g. smoking). The study expressed this in much more careful words: "Lack of health insurance is associated with as many as 44,789 deaths..." (This number was then divided into 45M uninsured in 2009 to get 1 in 1000). Politifact did not rate this claim due to lack of information[2], but they previously rated "Half-true" a number half as big[3]. The latter essay cites work that did control for relevant indicators and found: "the risk of subsequent mortality is no different for uninsured respondents than for those covered by employer-sponsored group insurance."
I'm not sure where to place blame:
- On the authors of the correlation study, who should never have studied this question without looking at extensive control variables or without more specifically studying causation?
- On Alan Grayson and similar folks, who are smart enough to understand the difference but are happy to assert causation?
- On Abbott, who implies causation, pointedly rejecting caveats ("it's not an exaggeration") in order to motivate developers?
I don't want to blame brandonb, particularly. I very much support his recruiting effort. In fact, I would say that the government probably has a disproportionate number of people who can resist unwarranted self-justifications. But I don't think a statistic like this should be left unchallenged on HN.
---
[1] http://www.pnhp.org/excessdeaths/health-insurance-and-mortal...
[2] http://www.politifact.com/truth-o-meter/article/2013/sep/06/...
[3] http://www.politifact.com/truth-o-meter/statements/2009/aug/...
From the abstract: "After additional adjustment for race/ethnicity, income, education, self- and physician-rated health status, body mass index, leisure exercise, smoking, and regular alcohol use"
> On the authors of the correlation study, who should never have studied this question without looking at extensive control variables or without more specifically studying causation?
How do you suggest studying causation in this setting? A randomized controlled trial where we deprive people of health insurance? Even that will likely not yield a true causal estimate, because randomization only helps for pre-randomization differences in the population, and behavior change from lacking health insurance will occur post randomization. The authors do extensively discuss their control variables, and important to remember is the fact that most papers only control for variables which ended up doing something, a subset of all variables that were tried. The NHANES data the study was pulled from includes a staggering number of covariates.
---
While not an ironclad study, I found the paper itself vastly more compelling than the politifact analysis of it, which boils down to "Well, observational studies might be wrong because reasons".
I'm also skeptical of Politifact's competence to adjudicate public health scholarship.
Yeah, thanks for fighting for the cause! #sarcasm
Is HN supposed to be a place where we try to find a flaw on every statement and make sure it doesn't go unnoticed?
What you pointed out doesn't even diminish an ounce of what Brandon and his teammates are doing.
Unless I'm misreading the polifact article you linked to in [2], it says that the 2009 study did control for smoking, and that they did a better job of controlling for such factors than previous studies.
> Still, their work stands out from previous efforts because it used more recent survey data and presented a more apples-to-apples analysis between the uninsured and insured populations. For example, it compared deaths rates for uninsured smokers with insured smokers, as well as other factors such as drinking, obesity, income and education.
My assumption when things melted down so drastically, was that the key problem was integration with the various vendors. Yes we all could look at the Html itself and make assumptions about poor practices but given that you are on the inside - what was the biggest issue?
That's what Mikey and the other Site Reliability Engineers fixed. They set up a war room with an engineer from each and every subcontractor, and the war room had three rules:
Rule 1: "The war room and the meetings are for solving problems. There are plenty of other venues where people devote their creative energies to shifting blame."
Rule 2: "The ones who should be doing the talking are the people who know the most about an issue, not the ones with the highest rank. If anyone finds themselves sitting passively while managers and executives talk over them with less accurate information, we have gone off the rails, and I would like to know about it."
Rule 3: "We need to stay focused on the most urgent issues, like things that will hurt us in the next 24-48 hours."
Once you have that process working, it's the same as optimizing software: you find the current bottleneck, fix it, find the next, etc. The Time article mentions two -- the lack of DB caching and the bad ID generator. There were dozens of things like that. And still are!
Hospitals don't let uninsured people die and insuring people doesn't magically save their lives.
Not sure where you're getting this. A quick Google Scholar or PubMed search shows a consensus that mortality rate is significantly higher for uninsured than for insured. [1, 2, 3]
[1] e.g. http://jpubhealth.oxfordjournals.org/content/32/2/236.short -- On multivariate analysis, uninsured compared with insured patients had an increased mortality risk (odds ratio: 1.60, 95% CI: 1.45–1.76). The excess mortality in uninsured children in the US was 37.8%, or 16 787, of the 38 649 deaths over the 18 period of the study. Children who were hospitalized without insurance have significantly increased all-cause in-hospital mortality as compared with children who present with insurance.
[2] e.g. http://journals.lww.com/jtrauma/Abstract/2012/11000/Undiagno... -- Undiagnosed preexisting comorbidities play a crucial role in determining outcomes following trauma. Diagnosis of medical comorbidities may be a marker of access to health care and may be associated with treatment, which may explain the gap in mortality rates between insured and uninsured trauma patients.
[3] e.g. https://www.sciencedirect.com/science/article/pii/S000296101... -- A total of 1,203,243 patients were analyzed, with a mortality rate of 3.7%. The death rate was significantly higher in penetrating trauma patients versus blunt trauma patients (7.9% vs 3.0%; P < .001), and higher in the uninsured (5.3% vs 3.2%; P < .001). On multivariate analysis, uninsured patients had an increased odds of death than insured patients, in both penetrating and blunt trauma patients.
This said, it is still a shoddy use of statistics. "1 in 1000 uninsured people die each year" by itself tells us pretty much nothing. What is the rate of death for insured people?
Wikipedia tells me that 8.39 in 1000 people die in America every year, so if uninsured people are only dying at a rate of 1 in 1000 every year, it seems to me that either it is beneficial to be uninsured, or uninsured people are not representative of the population (perhaps because many of them are young and healthy?).
I suspect that what is going on is this person actually meant to say something along the lines of "1 in 1000 people die every year in ways that could have been prevented if they had insurance" A subtle but important difference. The actual mortality rate of uninsured people is most likely much higher than 1 in 1000, but the deaths of uninsured people in motorcycle accidents would not be counted in that "1 in 1000" figure.
Either way, it is shoddy.
However, due to lack of regular care, by the time you show up at the ER it might be too late.
They don't do as much to stop them from dying. (Other than public hospitals -- of which there are a limited number with limited capacity -- they are only obligated to stabilize them in the ER and then, if they have further medically-necessary care but are stable, they can discharge them and/or transfer them to a public hospital if one is available, whereas those that have resources, insurance or otherwise, to pay would be admitted.)
> insuring people doesn't magically save their lives.
Strictly true -- it doesn't do so magically. It does so, instead, but the relatively mundane mechanism of providing them the ability to pay for care other than emergency stabilization, which reduces the probability of them having such care denied or delayed.
http://www.reuters.com/article/2012/06/20/us-usa-healthcare-...
The patient with metastatic cancer who needs chemoradiation on an outpatient basis must pay for his/her care.
Irrelevant, even if true. Uninsured people often don't go to the hospital until it's too late, because they know the expense will ruin them and they're hoping it will get better on its own.
I'm trying to reply to each person, but it may take a couple of days. :)
If, for whatever reason, you haven't heard back from me by Monday, please email again!
(The discussion on the 1 in 1000 fact has also been interesting, and I've personally learned a few new facts about the original study and what is controlled for. I'd rather have rigorous analysis of impact for everything, including and especially "good causes," than to give us a free pass.)
I emailed you on Thursday, haven't received a reply back. Is the Monday expect-a-reply-back time frame still good, or are you just buried under all of the replies? :-)
I don't want to spam you with inquiries if you're busy, just wanted to check in on this. Thanks.
To clarify, you don't need to be able to commit long term-- if you can take leave from what you're otherwise doing for a few months, that's enough.
If you want to work on something with real impact and help change the culture of government technology, email Brandon above! We're actively looking for a few good people.
How about sending to prison those that paid nearly 1BILLION for this site as well as the bankers and defense industry gluttons who stole trillions from us in the last ten years?
If I want to make my government modern - I'd put in place a system of accountability.
http://online.wsj.com/news/articles/SB1000142405270230381970...
I'm wondering - how is this perceived within the current team? Are you relieved that you will be getting reinforcements, worried that another big contractor could contribute to the complexity, or even disturbed that they could get the credit for fixing the site after all the hard work you put in?
I used to work at Accenture, so I'm curious as to how this will turn out.
Maybe I don't understand Obamacare well enough but I don't understand this statement (which I hear all the time on the commercials). Who couldn't get insurance before this website? I know the health care law did add guarantees around pre-existing conditions. So other than that, who couldn't call an insurance company directly (or use esurance or other portal) and sign up for insurance? This website is just a marketplace hooking you up with normal private insurance companies correct?
There are millions like me who can take risks for the first time, whether that's working for a small company or starting one. And I don't have to worry about the next 40K hospital bill burying me.
Who couldn't get insurance before this website?
You can only get the Federal subsidies through the insurance exchanges which was supposed to primarily be online, but had small phone support staff too. The phone support staff was completely overwhelmed due to the website problems, which left thousands with no method to get insurance.
I'd say the two most urgent roles we're looking for now are: * Site reliability engineers * Software engineers, including backend and frontend
If you do something else, like design, product management, or data science, feel free to send me an email anyway — I'm certainly happy to check around and see what's needed!
Basically, if you're a citizen and don't lie on applications and resumes and in interviews then there's really not any significant barrier to entry.
Haven't worked at Google, Facebook, whatever, but would love to help.
(I'm not in a federal marketplace state, though, and I ended up having my insurance canceled under ObamaCare and got a 364 day short term $750k policy instead. Still no idea what I'm doing in 2015.)
It's such a high-intensity project that I think the we'll want to keep it to full-time for the time being. Otherwise, it's hard to even keep up, let alone make progress.
(As a technical point, though, even though everybody is working on this for well over 40 hrs/week, we're technically hourly subcontractors. That's just how the government works.)
Is this in addition to the background rate? If true that would be quite astounding. Or is it just rounding?
Using that logic, how many have died due to the botched rollout of the website?
I used to be this guy. The guy with the lofty ideas, but who thought the implementation was "beneath me". The guy who would sit around, waxing poetic about various features, user acquisition, header alignment, etc. Don't get me wrong I had serious technical chops, but fixing that annoying localization bug? Blegh. Form encoding off? Don't wanna get my hands dirty. I had "big ideas"! I was going to change the world! People who change the world don't do the dirty work! So I'm very empathetic to the Obama Administration.
Like them, I needed a real wake up call. In my case, a friend who had implemented an idea I had sold the software for a lot of money. When I confronted him about sharing the profits he started running git blame on files across the project. My name came up maybe once or twice, across a multi-k LOC project, and even then on nearly inconsequential lines. It hit me then that while ideas may have value, the implementation usurps all of it. An idea alone is powerful, but once it's implemented the idea becomes worthless. At that point it's all about rolling up your sleeves and getting shit done. When you focus on that problems like an "ID generator" becoming a bottleneck (I had to read that bit several times over... apparently I need to start raising my rates to the hundreds of millions) disappear. It's a hard lesson I had to learn, and it's one the Obama administration has hopefully learned as well. Of course, I had just turned 17 when I learned my lesson, and Obama is now a lame duck with less than 2 years left on his final term. I guess this exemplifies my greatest struggle with the Obama legacy, in that it has become one defined by squandered potential.
You need both. Implementation people aren't all that useful without a vision. A vision isn't all that useful without implementation. It's a symbiotic relationship.
Steve Jobs without a Steve Wozniak probably wouldn't have been as successful. Steve Wozniak, without Steve Jobs, probably wouldn't have been as successful. But the two put together made some great things happen.
Same with the moon landing. As much as it took a tremendous engineering effort to put a man on the moon, it took someone with the vision and power to make it all work. JFK didn't get involved in the details, I'm sure, but he really helped to set the tone of the whole effort.
There's a fine line between looking at implementation as "beneath" your position and knowing when you're being more of a hindrance than a help. In my experience, nothing has driven me more crazy than a person above me who, while being a great project manager or whatever, tries to get involved with things that end up hindering the effort. If you take a weekend course on programming in Java, that's great! But don't start giving out "helpful" tips in something that is not your domain.
37signals' take on this was the best one I've seen (paraphrased):
Ideas are a multiplier. If your execution is good, a good idea will multiply that. If your execution is minimal, a good idea will scrape by. If your execution is abyssal, a good idea will make it worse.
I'd link the actual page in the book, but I have always had trouble finding it.
That said, obviously they should have brought in the campaign website team to oversee this instead of doing the standard federal government contracting route, which leads to a late, over-budget project literally every single time.
That's called cronyism, and, for the most part, it's either against the law, or will get you destroyed in the media. One of the biggest challenges in government is that, for those of us in startup-land, the process goes something like: "I need to hire someone, I'll call my friend who I know does good work". You can't really do that in the government, or everyone who isn't your friend won't be happy.
The standard federal contracting route is certainly a mess as well. The underlying belief that everything can be reduced to a series of checkboxes, and whoever can check the boxes the cheapest wins leads to a disaster.
Hopefully, one of the outcomes of all of this is we rethink how the government software (and other sorts of procurement) process is done.
Can you suggest some effective mechanisms by which "political opponents" have been able to "sabotage" their efforts? Do these people get depressed when they hear an eeeevil Republican saying what they're doing is wrong, or even evil?
And by my calculations, since we're several months out from the mid-terms, Obama has well over two years left. Not that I'm expecting much from them.
"Rule 1: "The war room and the meetings are for solving problems. There are plenty of other venues where people devote their creative energies to shifting blame."
Rule 2: "The ones who should be doing the talking are the people who know the most about an issue, not the ones with the highest rank. If anyone finds themselves sitting passively while managers and executives talk over them with less accurate information, we have gone off the rails, and I would like to know about it." (Explained Dickerson later: "If you can get the managers out of the way, the engineers will want to solve things.")
Rule 3: "We need to stay focused on the most urgent issues, like things that will hurt us in the next 24--48 hours." "
(long pause ...)
I sit in meetings weekly with mangers discussing issues that I know about the most but I can never get them to hear me out.
Good engineers are able to finish work even if you do not babysit them.
Not really an issue for them since healthcare.gov sort of became the dictionary definition of "crisis mode," but worth keeping in mind if you're on a project with a more normal trajectory.
HealthCare.gov had been constructed so that every time a user had to get information from the website's vast database, the website had to make what's called a query into that database. Well-constructed, high-volume sites, especially e-commerce sites, will instead store or assemble the most frequently accessed information in a layer above the entire database, called a cache.
It also struck me as a little funny that Time had to define the term 'cache'. Even my mom (74 years old) has some idea what a cache is (browser cache).
Especially in areas where I have expert knowledge, a definition like that gives me reference on what the general knowledge is, and teaches me to articulate my knowledge better.
Kudos to your Mom, who I'm guessing knows what a browser cache is from talking with you, but this detail is technical enough that a quick explanation is in order.
This definition is concise and descriptive and isn't long enough to derail the gist of what the article is talking about.
Depends on what industry you work in. A Cache may look more like a giant pile of weapons as opposed to some memory in a computer.
http://en.wikipedia.org/wiki/Cache
http://www.etymonline.com/index.php?term=cache&allowed_in_fr...
[1] "make what's called a query into" instead of just "query"
b. The explanation of cache seemed a bit too plain and didn't give enough context in the article. I was thinking an analogy might have work better, such as:
Caching is way of reducing the amount of work that needs to be done by a system. Suppose you had a system that was hooked up to a thermometer and reading the temperature from this thermometer takes 1 second.
Without caching, every time you asked the system what the temperature was, it would do a new reading from the thermometer. If 1000 people asked the system what the temperature was, it would do 1000 readings, which would take 1000 seconds collectively.
With caching, the first time the system was asked, it would do a reading and it would save the reading somewhere it could get it much faster than 1 second, let's say 0.01 seconds. For all subsequent readings, it would use the saved reading, until enough time had pasted to do warrant doing a new reading.
----
Obviously, my version is 3 paragraphs long so it's not ideal either :D
It's definitely a form of mental dissonance.
The government needs a lot of help. I wish there were more crisis opportunities like this to use as an excuse to hire the right team to build good systems for citizens. Tough to get systems integration above the fold regularly to call attention to the issue.
Someone needs to disrupt the government contracts business pronto.
I know people who are involved in this. Just being able to bid takes decades and a network you either can't buy or need to be very rich to buy into. It takes either already being there or finding excellent bonuses (e.g. the kind of benefits that come with setting up your business in Detroit). And then you have to compete with the entrenched interests.
This is, or at least can be, far down the ladder from Lockheed-Martin. But it's the same kind of environment: If you're already there, it's easy money. If you're not, it almost certainly out of your league. If you know how to play the game, you still need to know the right people. If you do know the right people, you need to grease their palms. And once you've done that, you're part of the problem.
"Someone needs to disrupt x" is a magically simplistic and meaningless point of view.
It's an incredibly difficult thing to do, though. Procurement practices that are enshrined in law are much harder to disrupt than those that are just bad habits that ossified over time.
Awarding work in this way isn't necessarily a mistake. But when the selection process itself is broken, you end up giving a lot of work over time to a contractor that can't do the work.
Even then, a lot of the problems with Healthcare.gov were caused by incompetence on the government side of things.
https://www.gsaadvantage.gov/ref_text/GS35F5457H/0LUS2P.2NVK...
Page 25. Top developer rate for 2014 is $99/hr which requires 10 years of experience.
I also see you're a YC alum! If you're interested in helping out, or just to provide an error report on the problem you encountered, email me at brandon@hcgov.us.
I wonder if they meant that the default caching was turned off, or that all the queries had "no cache" clauses, or that an additional caching layer (Redis, Memcached) was not implemented?
Just for the record, contractor did not wanted to use it and argued the team have no experience with it. Someone in government decided it must be done that way (in spring or summer).
I don't even want to imagine HIPAA compliance for something of this magnitude.
It would not have been "micromanaging" for him to have done what was necessary to be sure that what he was saying matched up with reality. Yes, it was a giant IT project of the sort governments usually suck at. But that was the bill he got passed and signed. If his administration wasn't capable of leading that project to successful, timely completion, he has to shoulder much of the blame.
Cultural problems like this are often caused by leadership. Whether you will be told bad news depends a lot on your management style and on people you decide to work with. You do not have to micromanage, but if they are scared to tell you the truth, then it is often your own fault.
Just a fragment of old media leaking onto the internet.
http://infiniteundo.com/post/25509354022/more-falsehoods-pro...
https://news.ycombinator.com/item?id=4605904 https://news.ycombinator.com/item?id=3763907
This information troubles me, because QSSI was just hired to fix Maryland's broken Health Exchange site. It sounds like the company is cashing in on the donation of brandonb and others experts' time?
brandonb, is your team involved with the Maryland efforts, or is that a completely separate team?
To be clear, it's not like QSSI or anybody else is ripping us off. We're all getting paid the standard contracting rate. But many of us probably would've worked for free if they had let us.
I'm not a regular time magazine reader, am I missing something here?
Using Obama for link bait means your article gets no reads from me.
If this happens the entire contracting landscape of DC would change dramatically.
A clinically-dead patient can be revived, but the condition they're in is far from OK.
how in the heavens would he know that you can deliver? you'll choose the one you'll know at least. and since he has documents to back him up(mcse, mcsa, he might have worked for microsoft for a while). he's the obvious choice.
i worked for the government in healthcare for a while. these are not intentionally malicious people. hell, they even want to make it better, but it's a combination of regulation(bidding system that favors friends), lack of knowledge, and a little bit of ignorance.
but if you want more details feel free to ask. it was kind of fun to see a corruption case close up.
#1: Understand the customer!
In this case it's http://en.wikipedia.org/wiki/Karen_Ignagni (They lobbied a congressman from Chicago w/ donations to elect, and Republican side is pissed their side did not get more.) In EU, it tends to be written for the people, but in USA, it's the lobbyists as customers.
#2: You can't fix political problems with software!
For example we can make the problem 10 times less by allowing each state to manager Healthcare and compete for residents. Like Massachusetts. It's a fools errand. But you can milk the feds if you bribe the right official.
#3. You won't find good and un-employed engineers!
Not in SV anyway. A - type engineers are ~ $240K and up. Y-combinator is mostly n00b growth hackers, not expereinced coders. Plus the rumor is that you have to be loyal to the party of New Democrats as the main req, not software systems.
#4. I heard it's written in .NET and Oracle Access Manager.
A requirement? That's not what google plus, facebook or iTunes use or such use. I don't want to code w/ those people that use that stack ( but I would have a great time w/ them after 5pm, they tend to be nice people ).
Kalvin and Brandon: The most patriotic thing to do is to not try and catch a falling knife so that each state can have a system they own - and we can move to a state that has a good one. Party greetings to you.
How? With another 14 million dollars... that is how. You can do a lot of things with 14 million dollars, this is not especially true when you are talking about the govt spending this kind of money, but still - it is not pocket change to the average company.
http://www.nextgov.com/cloud-computing/2014/02/cost-obamacar...