In hindsight, this shouldn't be too surprising. What might be surprising is that the same logic should apply to ALL government functions, not just software development. Who says government projects and agencies have to be poorly run? Who says we can't devise better systems and/or mechanisms (e.g., market-based mechanisms) for attracting and retaining great people in government?
Throughout history, there are examples of government bureaucracies that have been reasonably well run, in some cases for centuries. The Habsburg empire, in particular, comes to mind: http://www.independent.co.uk/news/world/europe/habsburg-empi...
So in my admittedly short time in the government [0], I've witnessed how all of these problems are due to good intentions. That's what makes this all really tough because everything you think is bonkers actually has a reason.
The 1400 page travel regulations is a result of trying to prevent fraud - every single issue that comes up results in a new rule.
The fact that it takes some projects years to deploy is that we would like to plan and make sure that every resource is well-spent, that it's in a number of languages and accessible to the blind.
It makes it hard for everyone - I've met lots of smart talented civil servants and government contractors who want to do things differently but have their hands tied behind their back.
[0] 2 years feels like forever to me but flash in the pan to many of the dedicated civil servants I've met.
In 22 years, I have never been so hopeful for meaningful improvement in my work life as I am now. Having met a few folks I am all too familiar with DTS and the JFTR (the 1400 pages in the article(1)). I think that's a great choice to start with: like Google going after the mundane problems of every person's life. This will make a difference. I am on travel now and was on the phone and DTS (simultaneously) for an hour today. And for anyone who tries to apologize for the 1400 pages, please don't. I have cut instructions from 238 pages to less than 30. I would argue the major problem is not that people are trying to solve every edge case. The major problem is that people are only in a job for a short period of time, come in, and while they may try to solve the edge cases they encounter, they often do that by trying to simplify things by inserting a new abstraction and taking ownership of that abstraction. So the layers of abstraction accrete like sediment. And as long as there's no direct logic conflicts, they can promote away from the problem.
I will gladly buy any USDS, 18F, or DDS hacker in San Diego a beer. Keep up the good work.
(1) It's actually 1602 pages: https://www.defensetravel.dod.mil/Docs/perdiem/JTR.pdf
This sentence is the simplest explanation for government (and bureaucratic) incompetence.
Think about writing software. Is the optimal solution to every single bug to write more code to deal with that specific situation? Of course not. In many cases, sorting out the underlying cause and fixing that (which may involve new code, rewriting old code, or even deleting outdated code) is the correct approach (assuming that the optimal solution is the desired outcome, there are of course cases where speed of getting something out that works trumps this, but government regulations only take effect once annually in most cases anyway, so they don't have a speed excuse).
Simply writing a new rule to deal with every scenario is an approach that inevitably leads here.
The same logic applies to all those regulations that seem bonkers.
If you think of all those regulations as a type of source code (which dictates what government employees and citizens can and cannot do, when, and under what conditions), it's clear that a lot of regulatory code needs major refactoring.
To use your example with travel regulations, those 1400 pages designed to prevent fraud likely consist primarily of thousands upon thousands of assertions and if-then statements. I wonder if it would be possible to reduce them to, say, a few dozen pages -- by refactoring all that 'regulatory code' to use different, higher-level abstractions.
This seems like a serious inability to understand that no process designed to prevent future things you can't forsee is 100% effective (by definition). At some point, you have to declare "good enough", and live with it until the error rate becomes unacceptable overall again, then modify it.
IE it's likely 50 pages of those regulations gave them a 99.9%+ rate of avoiding fraud. They then added 1350 pages to get to probably 99.99%
This is unlikely to be worth it.
(and yes, before someone points it out, i'm likely being generous with the numbers)
Then wouldn't the succes not be a matter of technical abilities or process, but rather goodwill by the client side to remain reasonable ?
They probably could have given the same resources and authority to the existing staff and they might have turned things around too. It really annoys me that in so many cases the existing guys know what's wrong but are not allowed by management to do anything about it. Then management hires new people, gives them authority, and suddenly these guys are heroes and the existing people look like idiots.
There seems to be this tendency to think that existent workers are all idiots and the only way to do better is to hire new people (preferably young and from ivy league schools). In reality it's a leadership problem because they have set up dysfunctional environments that don't allow people to perform.
We are very proud to be working along side the existing agency and contracting staff. You are right that many times, they do know how to fix it and need some help from us getting management to let them do it. Some times they are skeptical, sometimes there is friction - that's all completely understandable.
Ultimately, my time in government has opened my eyes to the talented folks inside government right now and I'm proud to be working along side them. I hope you didn't get a poor impression of us and the work that is happening.
The problem with the software transition is that they're sticking with the "hire a HIGH TECH contractor" approach rather than the "hire some coders to work inside the business" approach. Compare SAP/Oracle products to internally developed line-of-business software.
I'm not sure how to test this empirically, but it's always seemed to me that these are the inefficiencies that emerge with scale and incredibly broad objectives—and that the federal gov't actually does a decent job considering. Remind which company has 320m consumers, annual revenue of 6.7 trillion, and 2.8m (non-uniformed) employees? The federal government is 10x the size of IBM—it can't be agile. That said, tech is in general not a strong suit for gov't agencies—and it needs to be both 1) for planning, analysis, logistics and 2) for dissemination of information to citizens about what the agency does.
Because lots of people, inside and outside of government, derive significant money, power, or both from government dysfunction. A classic case of goal misalignment.
..two lobbyists who managed to find their way onto the five-person panel testifying. They represented the interests of traditional IT contractors, who seem to believe it is their right to overcharge taxpayers for complex computer systems that don’t work. Even though, as one congressperson at the hearing put it, those special interests are likely to view the USDS and 18F “with a jaundiced eye,” they were given the opportunity to seize on the authority issue as a way to cast doubts on the Obama tech surge.
---
[1] these together are pieces of Yudkowsy's revision of Hanlon's Razor: "Never attribute to malice what you can attribute to an enormous complicated System full of conflicting incentives getting stuck in a weird equilibrium. When that weird equilibrium is crushing people in its gears... We see a broken watch and infer a Watchbreaker."
However, a problem that has been around a long time is that it is difficult to fire government people for under performance so you have some people in positions of power crippling a whole host of IT efforts through their incompetence. That's not so much about money, it's just that the dead wood is hard to get rid of.
Also, the purpose of any bureaucracy is self preservation.
The government is a complex adaptive system with almost no external signaling from it's environment. It's an extreme example of how large companies behave when they have no external pressures (competition).
Every system is perfectly designed to produce the results that it produces. I know gov't waste is appalling but you should expect it until selection pressures are applied from the outside. Contrast NASA 1969 vs. 1986 and you'll get what I mean.
They don't have to, and some are run quite well.
The actual problem is that they don't have to be well run to thrive, since they operate in a monopoly environment.
This the first and presumably most important requirement in you opinion? Interesting.
May I ask, how did you extrpolate it was a property of the team at all let alone responsible for its success let alone the first component of that success?
We hire for experience, which comes in all sorts of shapes and sizes. Based on my personal observations, the team is actually the most diverse team I've ever worked on by a number of dimensions. We've pulled folks from retirement, others who are chronologically younger but have been contributing to Python and Debian for years, those who have uprooted their family. The job is an incredible mix - it requires being able to debug a slow performing database index one moment, briefing a deputy secretary the next, all while empowering the other federal employees and contractors to do great things.
I would be remiss not to take this opportunity to link to our hiring page. https://www.usds.gov/join
I do recognize that there are lots of talented folks that aren't able to move to DC or can't make it work for a variety of reasons. That's okay. There are many other ways to contribute to our country [0] - working for state/local government, volunteering in your community, being a good parent, inventing the next breakthrough, actually using your turn signals, and many more.
[0] Yes I recognize that not everyone here is US based. Though the point actually is probably true for all countries.
That's not bad, but not great for really high talent employees. I wonder how many people get that rate though.
That's right. The US Government is probably not going to win on compensation - salary limits, no stock options, no lunch. However, while it's not a money making enterprise, it's enough to do just fine. I recognize the sacrifices that many of my colleagues have made to be here, not to mention the sacrifices that federal employees and contractors have made (some who are very good and could be making more in the private sector) and that makes this even more worthwhile.
The thing that government can offer is impact. I've always known that government has a big impact on people's lives but not sure if I could personally make an impact. That's what USDS, 18F, a number of other opportunities are offering.
This is not a job for everyone. But if you're a certain type, there's nothing quite like it.
Since we have no power to adjust salaries, we're looking at other avenues, such as doing more "direct from community college" in-training hires as well as non-traditional employment paths.
Unlike USDS and 18F, we don't even have exciting projects, a shortcut around bureaucracy, or cutting edge technologies to sell with. The only sales tool we have for candidates is "if you complete your 6 months probationary, you can essentially never be fired or laid off." Sometimes, that attracts the wrong sort of candidate, as you can probably imagine. :)
Admirable intent, I'm suspicious of its prevalence.
Hopefully this will lead to more substantive change to how government IT is done. It's great to have teams that can drop into problem areas with a clear mandate to get things moving, but the whole system needs some serious reforms so that these IT quagmires don't get created in the first place.
This article heaps a lot of blame on contractors but fact of the matter is, contractors are often hamstrung by what the customer wants them to do and what the customer allows them to do. And some of that comes back to what was specified in the original contract.
Ultimately there are multiple components to the abysmal state of government IT projects:
* The government has moved to a model where very few full time employees are technical people. The idea is that almost all IT will be outsourced to the private sector
* Because fewer and fewer FTE's are technical, procurement and project management decision-making is compromised.
* There is little accountability for government employees, and by extension, contractors. Botched projects where millions of dollars are waste rarely have major career consequences for FTE's, and the bigger contracting companies don't have their ability to get new contracts affected at all.
* Building software for government often involves arduous interpretation of regulations, trying to build to undocumented and byzantine business processes, or waiting an inordinate amount of time for the bureaucracy to make important decisions (then change their mind!).
I feel like empowering the technical people is a very important first step, and getting them executive backing may create the momentum needed to change some of the common pitfalls. But I think there are some big changes that have to happen to the whole ecosystem to really get things to where they need to be.
It's all well and good to give small teams exceptional power to get around the bureaucracy but that approach doesn't fix the underlying problems.
You make a great point - USDS is just one part of the solution. In fact, in almost all of the USDS projects, we work very closely with agency employees and contractors (many of which are just as talented and have chosen to serve their country). Most of the times, I spend very little time hands-on-keyboard and helping empower the existing team.
As for the longer term solution, there is a less publicized version of what folks are doing. USDS has a number of contracting officers who are helping teach others in government how to be savvy customers of technology. 18F and GSA have been doing a lot on this front as well to help bring in really good contractors and writing agile purchasing agreements. The Office of Federal CIO is rewriting and simplifying tech policy and the Office of Federal Procurement Policy has been working on the procurement side. These are the long term changes that I'm personally excited about.
I'm personally seeing some 18F projects come down the pipe that are truly innovative and hopefully will encourage organizations to think differently about IT procurement. Still a lot of silly stuff out there but at least the ship is starting to change heading.
Whether they're contracted or government employees, we need to devolve authority from big top-down specs and towards programmers working alongside the actual users.
Within the contract there is often flexibility as to how the job gets done. But bottom line is that if you have a stubborn customer who insists on doing things the hard/stupid way, there's not a lot you can do. Maybe you go above their head and try to force a change but that is fraught with risk. The prudent thing to do from a financial standpoint is just put your head down and complete the contract, even if the work is compromised because of the customer you're dealing with. And indeed, that's what happens a lot of the time.
Fortunately more and more parts of government are buying into the agile mindset but there's still a lot of outdated thinking out there.
http://www.va.gov/TRM/ToolPage.asp?tid=5692&tab=2
contrast that with Oracle:
http://www.va.gov/TRM/ToolPage.asp?tid=9&tab=2
(Don't miss the difference in tone on the "Analysis" tabs.)
I'm sure some of that is due to lobbyists, but nonetheless it seems to me that there are legitimate challenges in making Postgres meet FIPS 140-2 requirements. I've been able to recompile Postgres to use the FIPS OpenSSL wrapper, but storing passwords with MD5 is a harder issue to fix, and of course there is no definitive list. Does anyone have some experience with this?
There's definitely quite a bit of PostgreSQL in use in government, so this does not need to be a blocker. As someone noted, PAM auth is a good solution; and I think if you use CentOS or RHEL (Use the latest release please!) you'll end up with a FIPS OpenSSL which PostgreSQL is linked against.
More generally, the VA TRM is not a permanent thing, it's precisely the type of existing process which can be improved with the feedback of on the ground software engineers. If this is a blocker for you, I'm sure folks at DSVA would be happy to help!
http://www.va.gov/TRM/SearchPage.asp?catid=83&catName=Inform...
But it doesn't look good in general... :-( http://imgur.com/a/1ALQV
I would assume that the main reason that Oracle is preferred is due to lobbying, but also the fact that they don't really innovate anymore so it is non-trivial for them to provide support and bug fixes for a decade or two.
It's a throwback to this xkcd https://xkcd.com/463/
In my mind, the ideal electronic voting machine:
- Has mechanical buttons which simultaneously punch a paper ballot in a manner observable to the voter
- Runs on a small (open source) microcontroller that's been audited for backdoors.
- Runs (open source) crypto directly on the metal
- Publishes the results in a cryptographically verifiable way.
IMO, anything else is evidence of at least government/contractor ineptitude, and potentially malfeasance.
* Paying a utility bill online
* Registering to vote
* Finding out how to get from place to place using public transportation
* Paying for public transportation passes
* Signing up for unemployment benefits
* Proving residency
* Getting an ID card
* Registering a vehicle
* Paying taxes
* Entering pleas to local courts for minor offenses and infractions without taking a day off work
* Reporting non-emergency code violations (like construction noise after hours)
* Knowing who your representatives are
* Communicating with your representatives when you do know who they are
We actually use and interact with local and state governments far more than we do with the federal government. And yet when it comes down to it, they are the most difficult to work with because their resources are so constrained. I honestly believe that local governments could benefit from some resource allocation on their behalf at the federal level for the creation of highly scalable technological public goods that will only ever be used at the local level.
Its still pretty early days for civic technology. I think we'll see tech provide all the services you listed, eventually. It'll probably be a mix of outside vendors and local governments running their own engineering teams. There are still some legal barriers in the way, procurement for example.
Its happening, just slowly.
If you are interested in speeding up the progress of your city, I'd recommend checking out if there are any civic tech volunteer groups near you. https://www.codeforamerica.org/brigade/
I absolutely agree that the interface that most citizens use to government can be better. President Obama agrees with you (from his sxsw remarks):
"I could change the politics of America faster than just about anything if I could just take control of all the DMVs in the country."
"If their primary interaction with government is the IRS, you just don’t have a good association with government when you’re writing that check."
USDS has been working with the IRS to help them get their Transcript service back online with better identity proofing though admittedly just a small step in a better online experience. (irs.gov/transcript)
A number of state and local governments have been making improvements as well (much harder to see if your state or your local government has not been). I'd encourage you to search them out or even start it.
First I need to save enough money to cover the difference in salary for 2-3 years.
Granted everyone's got their own experiences, so YMMV I guess. Just curious where your data comes from?
The author makes it sound as simple as hiring fresh hackers from Silicon Valley to take over failed projects from stodgy software developers from the large D.C. contracting firms.
Having worked on government projects like those cited in this article for several years, I can tell you that it is the government agencies themselves, and not so much the developers who are to blame. To build a cool iPhone app for a private service, you can pretty much use whatever tools and timetables you want.
Not so with government tech. With even small projects, you are beholden to policies that were written several levels up. Things like no cloud-based hosting, no sites that don't use government-written style guidelines, and in many cases, no development that occurs outside the government facility. If you hamstrung pretty much any private tech company with the same restrictions government contractors have to deal with, I can guarantee you they would look a lot different.
That's not to say that change can't occur, but to attack the contractors is to misunderstand the root cause of why government software is historically so bad. There are actually many, many very talented developers and software engineers steeped in the latest lean startup theories inside the beltway, but they can seldom use any of that because government policies stand in their way. For most projects, you are dealing with literally thousands of regulations throughout the development process, and most software apps have to account for ever-changing laws and policies that are specific to each application.
However I have already seen a great deal of positive change, with many agencies opening their minds to newer, faster methods of development and allowing better tools to be used.
The most useful thing I worked on was a program for calculating air-sea carbon flux: I did the OS X GUI and helped with the core logic (computing air-sea carbon flux is nontrivial). At the time I was appalled that we spent $10k on what could have just been an R package around a wrapper for the C++ core. But since then the program has become a very common tool for ocean acidification research, with ~175 citations, many of these have since been cited in all sorts of climate change policy documents. It's admittedly hard to assess value, but it's definitely more than 10k. And definitely a public good that the market cannot be counted on to deliver.
I also found out at some point that the contractor I was working for (a Fortune 500 engineering and defense company) billed the USGS more than twice what I received (I got no real benefits). The federal employees I worked with detested the contracting company because they knew how much it was skimming from the contract employees, many of whom were full-time and had been there for years. When I figured this all out, I quit, started my own company, and was hired directly as an independent contractor. Unfortunately, this was not an option for most people, as this wouldn't have worked for a full-time, year-round position.
I'm personally appreciative of the work that you've done and your continued work, in spite of the many challenges and frustrations. There are many ways to serve. It's always incredibly rewarding to be working along side dedicated and talented public servants and contractors. Thank you.
Another project involves working on the Global Positioning System. Lynch still can’t believe that he can work on something that cool while serving his country. “I didn’t even know that we ran GPS, right?” he says. “That’s crazy shit.”
This applies top to bottom (and has nothing to do with the current rage between republicans and democrats). The 1400 pages of travel regulations are a classic symptom.
When the hackers leave, who's going to end up doing maintenance on these projects? It's not unusual for a gov't developer to work on a project just long enough to be productive, only to have them leave for greener pastures (often a promotion or relocation). In this case, most will want to stop suffering at lower salary and go back to SV. So it seems like it's going to be USDS holding the bag; unless there's a solid plan to transition the workload back to the VA (either organic or contractor), then USDS will have to maintain every project they start (forever).
On the other hand, I bet there are more than a couple good developers within the VA that would love to take over the projects, if only there were a permanent position they could do it from. Why can't Ash Carter set up more permanent positions?
Worst case scenario, a new President comes in and sunsets USDS, requiring VA to maintain the workload. VA balks, they hand it to Deloitte/BAH/etc, and we're back to square one.
Best case scenario, USDS build quality solutions that last decades and can become the next legacy system that VA builds on. And maybe USDS (if it's still around) can re-revamp the system, etc etc.
Also, I really don't understand the lobbyist's angle on open source: isn't all code written by a federal employee already in the public domain? (with obvious exceptions re: security etc)
"Start Up Life Meets the Compliance Riddled Federal IT Market for 18F and USDS" by A.R. "Trey" Hodgkins (ITI)
https://www.itic.org/news-events/techwonk-blog/start-up-life...
18F’s and USDS’s heavy reliance on open source and custom coding raises serious concerns. While open source is an entirely appropriate option for agencies to consider when looking to acquire software solutions for mission needs, we need to ensure that these are not the only options available as different missions have different demands. Making open source software the only preferred solution doesn’t adhere to the tenets of tech neutrality. It also limits options and competition, which means we may not be achieving best value for the taxpayer.
I'm talking federal, state, and local level. Just like each state has senators and reps; 1 engineer for every 1M residents that live in the state.
each state's teams would tackle problems with software, share solutions to widespread, formulaic issues with one another. it'd be a iconic and fresh-breath-of-air kind of way to speed up efficiency and effectiveness (government and big corps are SO SLOW)
If you're looking for a new job, I'd highly recommend giving it a shot. Definitely felt like an opportunity unmatched elsewhere right now.
That's a very strong accusation from Hodgkins (linked blog post in the article).
https://cdn-images-1.medium.com/max/600/1*Og10aHFHpbFXrwD7cG...
http://www.npr.org/2015/03/31/395829446/after-snowden-the-ns...
