History and details are available at issquareup.com.
We apologize for the downtime; this situation is well outside what we expect of our service and of ourselves.
Please feel free to email me at ewb_at_squareup.com with any questions. Again, we apologize for the outage and take remediation of these issues seriously.
Less technical version: https://www.issquareup.com/incidents/y4mrzjz0nl2d
- Staff at the cafe are extremely frustrated, no real visibility into what's going on.
- Finding the status page was a struggle. Following @square and @sqsupport was insufficient, as both accounts have been publicly silent during the entire outage. The status page, hosted at the non-obvious issquareup.com, is only listed on the profile pages of those social accounts. I located the page and shared it with the cafe staff, which provided some context as to what was going on.
- But, the status page itself was not very useful to them. The information in it is moderately useful for a technical user, but most of Square's POS customers aren't technical? More importantly, most of the hands-on operators of these POS systems are even less technical.
- The only solution offered is to "switch to offline mode", but that only works if your square app hasn't already logged you out, which had happened long before reading about the solution. This behavior corroborated by twitter anecdotes and other comments in this thread.
- There is no other solution path presented.
- Without any other information to share, staff is describing the issue as a "nationwide Square server crash" to all customers.
- Some customers just left when faced with the outage (alternatives are cash or an on-site, fee-based ATM)
- All of this is happening while the staff is continuing to take orders, serve customers, deal with irate customers, and generally be positive and courteous.
- The only reason they retried the app just now is because I read the comment from the Square engineer on this thread announcing service restoration.
Whatever user model Square has of the day-to-day operators of their POS, it seems to be wildly miscalibrated, especially around how to handle incident communication.
I mean, it's clear opening the two Twitter account pages, both have sent tons of replies during this time period.
On @sqsupport specifically they clearly state in the bio that their tweets aren't the right place to check for service outages:
> We're currently working through some issues. For live updates, please check http://issquareup.com
So this doesn't solve the problem of bringing Square online but it also doesn't really sound like the merchant is trying very hard as the right channel was easy to find. Besides adding email / text message alerts to merchants for downtime, Square is doing a lot more than most.
Must be stressful trying to bring it back online.
[0] https://techcrunch.com/2014/01/13/putting-squares-5b-valuati...
> While we continue working to resolve the issue, we recommend that all sellers switch to offline mode, which will enable you to continue taking payments via swiping. Offline mode instructions are available at: squ.re/offlinemode
Though there are some big caveats:
> - Your current swiping rate will be applied to offline transactions, so you’ll see no difference in fees.
> - When operating in Offline Mode, there is additional risk with any payments you accept. Square is not responsible for any loss due to declined cards or expired payments taken while offline or for chargebacks.
> - Square can not contact any customers on your behalf should a payment be declined or expire when taken in Offline Mode.
So if Square is somehow down for 73 hours, a lot of businesses lose a lot of money.
I guess as a business owner one should now consider having a backup credit card reader through a different service.
You can't sign into the square dashboard either so access to square appointments on the browser is a no go.
On that note, I also saw multiple people leave to go to a different coffee shop because they didn't have cash on them.
Seems like they have offline mode. Do their customers know how to use this? What's the chance for increased fraudulent swipes?
If a customer knows the payment processor is offline, they can use an invalid card and it will appear to go through. Merchant will be stuck with the liability after the transaction is later sent and declined.
Today, we went into offline mode automatically and processed a few transactions. Then our app was forcibly logged out of Square service and we couldn't get back in to continue in offline mode.
At least good old hard cash still works.
So, if you are logged in already you can use offline mode.
If you use their point of sale software to track cash sales and are not logged in already, your pretty screwed at this point.
Could this be a DoS of some kind?
As a former POS engineer, this has been my gripe about these services from the get-go. Real payment processors, and POS software/SaaS vendors you... pay for guarantees about stuff like this, and have clear workarounds. Does it screw up sometimes? Yea. But you don't get opaque downtime like this, and you were given a clear workaround(and ALWAYS a clear offline mode you wont get locked out of flipping on, like the case here) in the first place.
This is a failure both on the customers side, and on squares side. They basically scaled a pickup truck up to a delivery truck without considering why a delivery truck was designed differently in the first place, at least in some ways.
