It took us a fair amount of time to figure out that this was the cause of our issue -- the PayPal UI didn't indicate any problems and PayPal business customer support was unable to see the source of the problem either.
Some accounts simply will not send them until an IPN URL is set, even for transactions that include the notify_url in the request. You cannot even see the IPN history until you enter an IPN URL, even if it was used through the notify_url param.
IPNs that include UTF8 characters will not validate by default in most PayPal accounts. PayPal defaults to Win1252, translates the UTF8 content to Win1252, tells you it's Win1252 and then seems to validate the IPN against the original UTF8 data. I have never been able to validate a Win1252 IPN that had UTF8 content. I've tried iconv, stripping bits, and plain substitutions (ü -> u). Nothing seems to work. This seems to be getting better though.
For about 2 months last year they forgot to include a field that had to be added in order to verify the IPN. And you had to iterate through a few different values to see which one was correct. So we got a rash of support tickets asking why our customers were suddenly getting notifications of issues verifying PayPal purchases and were having to manually approve orders. I submitted a ticket after a day of dealing with the issue and got a response about 2 weeks after they fixed the issue. There was no indication of the problem on their status site.
The IPN verification service randomly returns INVALID for valid data or simply doesn't work. Sometimes IPNs are delayed.
Dealing with the PayPal APIs are far, far harder than their sales sites would have you believe.
http://blog.cyberion.net/2014/08/parsing-paypal-ipn-requests...
Allow from .paypal.com
Allow from 173.0.81.1
Allow from 173.0.81.33
Allow from 66.211.170.66
Webhooks are not hard. That Paypal fails at them this badly makes me think their infrastructure is a (much larger) disaster waiting to happen; if I still did freelance work, I would be moving all of my clients to Stripe right now, before the next inevitable mishap.
When my users are redirected back to my site (thanks page, or similar), I check if their transaction is completed, if not, I kick off an every-five-seconds check while asking the user to hold on while we talk with paypal. I will eventually fail after some number of checks of course, but this means PayPal can stop sending IPNs and everything will just keep going along just fine.
If the user might not end up back on your site for some reason, run a cronjob that tries to verify transactions created in the past day/hour/whatever.
An issue like this doesn't have to, and really shouldn't, cripple your business.
PaymentDetails is mandatory for any adaptive payments integration.
https://developer.paypal.com/docs/classic/api/adaptive-payme...
My favorite Paypal annoyance is that for split payments there's no way to verify that a seller's Paypal account qualifies for split payments (has to be a business account and verified), without trying to send a payment. At least, that's the way it was last it mattered to me.
Here's a snippet in Ruby I use to check. It's not perfect, but it catches a lot of cases:
# ...
def initialize
@request = PaypalAdaptive::Request.new
end
# ...
def verified_status( opts={} )
begin
opts.assert_keys! :first_name, :last_name, :paypal_account_email
result = @request.get_verified_status(
emailAddress: opts[:paypal_account_email],
firstName: opts[:first_name],
lastName: opts[:last_name],
matchCriteria: 'NAME'
)
Rails.logger.debug "PAYPAL VERIFICATION: #{result.inspect}"
return result.success? && result['accountStatus'] == 'VERIFIED'
rescue
return false
end
end
What I can't stand is that accounts in some countries (at least India) can't send payments from balance. So you can't get your fee. This is mentioned nowhere in the docs, a customer just hit it one day and that's how I found out. I asked paypal what other countries can't send payments and they said... wait for it... they "don't have a list". I kid you not: https://twitter.com/rfunduk/status/412980259001102336
So the question is if you want these people's money or if you are willing to take a hit in sales just to stick it to PayPal.
I thought this, so when I moved to stripe I retained a pay with paypal option (and braintree makes this very easy now) but it was hardly used so I just got rid of it. I still use paypal for some really old stuff that I can't be bothered to update but all my new stuff is stripe only.
Now, I'd agree if you said that PayPal isn't suitable for a business to use, it should be relegated to the realm of individuals and small businesses who don't know better, but that's different than what you seem to be saying.
Firstly, the degree of marketing bullshit that you have to push through just to get to the truth of the APIs is staggering.
Secondly, they perform backflips to avoid telling the truth about their integration options, which is to say that they're all fundamentally insecure for real time digital goods and services unless you implement IPN plus polling plus duplicate detection / additional round-trip validation calls, ie. you can't trust an IPN notification. The degree of complexity (order state tracking, IPN state tracking and duplicate detection, retry support, running your own IPN-receiving server) and latency requested here of client businesses is immense.
Thirdly, their idea of international support is pathetic. It seems that they've basically duplicated their entire business process to other countries, translated it, and then assumed that everyone in that country requires only one human language in all of their interactions: documentation, support, emails, etc.
Finally, as is widely known throughout the industry they have a shocking reputation for the arbitrary suspension and seizure of accounts and assets, with little to no recourse for those affected.
I am not surprised that Stripe has taken off. Unfortunately, that's still lipstick on a pig: fundamentally, the settlement, risk, transparency and government interaction model of credit card networks means they are unsuitable for an increasingly large volume of business around the world.
What's worse, their system essentially rewrites history over time. This means that rather than keeping a ledger of payment events, they change the dates and amounts depending of payment status / reversal /etc. Querying later in the week and the data is different. For the same charge.
I ended up keeping an event log of everything we ever get from them (IPN, API polling, reports) and synthesizing that into a current snapshot at query time. The code is riddled with comments saying, essentially, "this charge, then return, then recharge and failure means that the payment is OK".
Notice the questions I asked.
IPN hasn't gotten much love in years but still pretty cool that it was (one of) the original webhooks back in 2001 and has worked decently for the past 13 years. In fact, an integration from 2001 should continue to work OK.
It's important to be careful about jumping to drastic conclusions when debugging a connection like this.
E.g., from the OP:
> I called Paypal Technical Support to verify exactly what we were seeing. Sure enough, they had received other calls today about this exact issue. Other vendors were not receiving IPNs for their orders. The ticket for this "critical bug" was created today.
> The problem with that fact is that our customer's order was from 9 days ago. For at least 9 days (we are going through all of our partial orders now to see the full extent of this bug), Paypal was not sending IPNs to merchants and did not know.
This doesn't actually follow logically. The tech support rep mentioned getting other calls about IPNs not going through does not mean NO other IPNs were going through. Likewise, if Tindie has actually not gotten any IPNs for 9 days, that doesn't indicate anything about the other millions of PayPal merchants.
This reminds me a bit of customers who call in a panic that "your entire website is broken!", and after a bit of discussion it becomes clear that their internet connection is down.
The same stream of events could be caused by lots of things, including minor configuration changes on Tindie's end.
Speaking of support, we qualified for the integration reward ($1500 or so; maybe more). We tried to claim the reward for over a year. Every time they'd say "yeah you totally qualify!" and we'd ask how to claim the reward and then silence. Of course every email response took 3-7 days. A few months later we'd go through the same process. Still waiting on that check to arrive.
We supported it for about a year and eventually gave up. Payment notifications could take up to a minute to arrive. You couldn't specify the tax (or shipping from what I remember) for an order in any sane way. For a while it was over 50% of our support tickets while being used by less than 10% of our customers and less than 1% of all transactions.
I was so happy to delete it from our codebase. And we won't support any payment processor they have to offer anytime soon.
We also changed the title (from "If you use Paypal, you should know they aren't sending IPNs") to make it more neutral.
It is a more complex integration, but our product has been much more stable since we made the transition to Express Checkout.
This does not however prevent other problems with IPN delivery on refunds and other post order activity. Paypal's IPN infrastructure is pretty weak in general, very poorly documented and completely untestable.
As other commenters pointed out we didn't have much choice of providers because of our business model (hotel booking). Stripe and Braintree didn't want anything to do with us, so we went with PayPal to start with. I think they are now using RBS WorldPay, but even that had it's issues...
> Disbursements through US Bank Accounts or Debit Cards
Is it only related to US bank accounts and debit cards? Or is it an IPN issue affecting all card?
I received a lot of "hmmm that's weird" and "well i've never seen the API do that" comments from the integration support specialist at PayPal."
What is happening for us is that Paypal is simultaneously sending a authorization and then a second later a capture. the results are weird some transactions process and other get a duplicate invoice it error.
There are many reasons why an IPN will fail. Like having your firewall settings messed up, network issues on your or PayPal's side, PayPal actually sending an IPN but your infrastructure not making use of its information...
b) IPN is clearly going to be flakey, there are several reasons why IPNs might not be delivered and they are not all paypal issues. In the paypal docs it says that if your webhook doesn't return 200 to some IPNs then it will stop sending them. For my new stuff I don't use paypal at all (stripe is just too good) but when I did I had a cron job running every 5 minutes that checked every transaction that had been created but no IPN had come through for. Over the cause of a couple of years I caught several transactions where the IPN had been lost.