German Card Paying Drama - the Verifone H5000 Part 1

One single type of payment terminal (the Verifone H5000), a rather old platform, officially announced End of Life 2019, end of production 2020, with some sort of limited support until 2023, brought down big parts of card payment all over Germany as one of the embedded certificates expired unnoticed on Tuesday, 2022-05-24.

The Verifone H5000

Turns out this terminal is still being installed as new by many local payment service companies. It is cheap (as it is EOLed) and quite robust. But seemingly no one noticed the expiration date of a certificate that is needed to connect and get authorisation from the German payment system for card payments.

Now Verifone is working hard to get an update out but replacing such an important certificate is not a simple act of „install update“ fr reasones I will explain later. I guess they simply didn’t expect this to happen after they announced EOL of this terminal. Oops.

The update will need some kind of manual intervention (as in a service technician physically interacting with the device) for all H5000 that have been restarted after Tuesday, 2022-05-24, so it’s quite a nightmare from a logistical perspective.

Obviously, as this is a specific certificate for the German clearing/payment backend, the responsibility could be with the German company that owns the certification for this device in the German market. The manufacturer however also needs to help to get it fixed.

Sources and context

Sources. “the official PCI 3 expiration date is April 30, 2021 […] Products with expired PCI 3 approvals are not eligible for new sale or new deployments other than repair and replacement of like products.”

EOL announcement from Verifone

EOL announcement, dated 2020-11-19

(Should you wonder - I love to dig deep into this kind of stuff. It really isn’t a lot of work. Needs some Google-Fu and a basic understanding of a specific market. I also LOVE to find out what REALLY is happening myself instead of relying on clickbait, uninformed stuff.)

On page 2: „Verifone is no longer conducting feature enhancements for the PCI 3 products listed above, including any functional improvements or changes, support for existing or new product or service, new EMV kernels or certifications, or any other enhancement.“

EOL announcement page 2

And since January this year only TA7.2 certified payment terminals are allowed to be introduced to the german market. So the H5000 is more or less obsolete for new installations since 2020 and should have been phased out and replaced at existing installations somewhere in the past year, latest. See this page for more details.

Epay Timeline on TA7.1 EOL

Epay Info on H5000 not being TA7.2 compliant

The December Update

It seems an update was made available by Verifone and has been rolled out end of 2021. On christmas day. But many H5000 did not get that update and now, as the certificate has expired, this update cannot be installed in the usual, remote and automated way.

It seems that the certificate update wasn’t prominently mentioned in the release notes for the December 2021 update from Verifone. Which would explain why the update wasn’t installed everywhere - the urgency wasn’t made clear :(

My current best guess as TL;DR: The H5000 was introduced in 2012. The firmware contains a root certificate. Which typically is valid for 10 years. 10 years later …

Every time the device reboots, it does an integrity check using that root certificate. That now fails on devices that didn’t get the December 2021 update, which, again I guess, contained a new root certificate with a new expiry date.

Combined with the recommendation to not switch off running devices, I deduct that a certificate involved in the integrity checking process is the most logical explanation. But I definitely could be wrong. Without details from Verifone it’s all just educated guessing.

Going deeper

I have drifted so deep into this that I am now actually reading the Certification Report and Security Target for the H5000, which you can find here.

The security target document specifies a „secure state“ when internal self-tests, integrity tests of the payment application fails. This „secure state“ means the device deletes all secrets to make sure there cannot be abuse.

What seems to work is ELV Offline. Elektronisches Lastschriftverfahren. As ELV ican run completely offline, a payment authorisation or garantuee is not possible. It’s cheap, but risky. And was mostly phased out for that reason quite some years ago.

And yes, you can add pre-auth to ELV, but for that to work you need a secure connection to the backend system, for which you need a valid certificate. Catch 22 :)

After I’ve read the Security Target document, I guess I understand what’s happened better. The Big Thing is the payment application. That runs sandboxed. And is signed. And gets started during boot. Kind of The Thing that is needed for POS (Point of Sale) Operation.

It seems clear to me now that for some reason, on H5000 that didn’t get that December 2021 update, the verification chain of the app signing fails. Could be an expired DK root certificate or some other intermediate cert.

But when they enter the Secure State (typically through a reboot) they lose all capabilities, even the ELV option and now need physical interaction to be restored to a working state.

So. When Verifone says it is NOT an expired certificate causing these problems - they are kinda sort of correct. When I say it IS a certificate problem, I’m also right. Difference between technical and business/legal logic ;)

But again - my disclaimer. This is all based on my 30+ years of experience in the field, extrapolating from the information I could gather (which I’ve referenced). I have no internal infos from Verifone. It’s up to them to publish a full post mortem.

A Few Days Later

Possibly final update from my side: Verifone has updated their support statement in a few ways:

Verifone updated support statement

So it seems that NFC payments might work, ELV/SEPA too but “normal” card transactions don’t. This still points to a problem with the internal payment application not being able to get authorisation from the auth/backend system.

They have added the info that the H5000 was not being sold by Verifone since the end of 2019, hinting at the EOL situation. There is some sort of update path available, but still no details.

After a bit more discussion - it seems that transactions that don’t need a PIN mostly work. So offline-ELV/SEPA and NFC payments with Apple and Google. But as soon as PIN entry is needed - no chance. Hints at my original thought - something has expired, stopping communication.

If you care to hear my enthusiastic geek voice (in German): I was quoted in this M94.5 interview :)

Here is Part 2