German Card Paying Drama - the Verifone H5000 Part 2

Things are looking better. A short recap. On 2022-05-24 a lot, and I mean a LOT of Verifone H5000 payment terminals stopped working. Rebooting them didn’t help - it actually made things worse. Keeping them running at least saved a chance to get them working again - the manufacturer said. But regardless - card payments didn’t work.

You might want to read part 1 first

TL;DR

Something you shouldn’t call a certificate expired on 2022-05-24, rendering affected H5000 unable to process EMV based transactions that involve PIN entry, physical contact to the card and an online connection to the payment/auth backend. An update is rolled out, fixing the issue.

The Verifone H5000

So on 2022-05-24 chaos ensued. What was happening? What to do? Shops, pharmacies all over Germany put up hand written signs: “CASH ONLY, NO CARDS”. Now, a few weeks later, we know what happened and that it can be fixed.

So. What happened?

Verifone, the manufacturer, was extremely careful in its wording at the beginning. Their support statement said that you should keep the terminals powered on, they are working on an update and no, it definitely isn’t an expired certificate.

Early statement from Verifone

But the log file of affected H5000 clearly had an error message stating “Certificate Expired”. So what was really happening here? Well - turns out, both things are true at the same time. A certficate expired. But it wasn’t exactly, technically a normal certificate.

Error log with “certificate expired” error

The H5000 is a Linux based payment terminal. When it was announced, back in 2011, it was quite a beast. It had a nice touch display with colour. Could handle NFC. State of the art. But as it is a certified payment terminal, it is full of secrets and must be made very secure and tamper proof. Obviously.

The little Linux inside (seemingly running a 2.6.27.39-WR3.0.2.a WindRiver kernel) runs a few binaries to do its job. The most important of those is the Payment Application. It handles the EMV Payload. EMV is a standard for secure authorisation/payment, involving hardware and software. (EMV is, BTW, Europay-Master-Visa - they defined this standard quite some time ago). The H5000 has both. There is a hardware module that handles the magstripe, chip and PIN entry. There is a software part that talks to the hardware and the backend systems.

In order to talk to the hardware and the backend systems, the software needs a set of keys. These are extremely valuable and highly secured. Again - obviously. So these keys come as part of the certification and authorisation process. And are secured (wrapped) in a signed package. You guessed it. Right. That signing part. Yep. That expired.

No CertFile error

And as this signature had expired, the payment application couldn’t run. No EMV Payloads. No communication with the PED (PIN Entry Device), Magswipe or chip reader. Oops. What DID work however was NFC. And transactions that don’t need the EMV flow. Which in Germany is called ELV/SEPA Lastschrift. So affected devices could be put in a state where you could do offline ELV/SEPA and even NFC with Google/Apple Pay - as all these don’t need a PIN for the transaction.

Where are we now

Now comes the good news. Verifone managed to overcome all the problems. Somehow. On their Support Page they now explain that an update is being rolled out. More or less automatically. That restores full functionality for the time being. Remember - the H5000 is officially EOLed and production stopped in 2020. It’s TA7.1 certification will expire in 2024.

Support update

This is good news for the many, many H5000 out there. It is rumoured that some 80000 at least are active as of today.

So they have updated the software signing. EMV Payloads and backend connections work again - all is good.

They also explain what exactly happened (which you already know, dear reader):

Support Update 2

Quite some interesting wording. “eine abgelaufene Zeitstempel Signatur in der Plattform Software des H5000-Terminals” - an expired Timestamp signature in the platform software of the H5000 terminal. Please don’t call it a certificate!

So with all of that said and explained - the good ole H5000 is back in business, with its known flaws, slowness and fickleness. It can run for a while, but not too long. The drama is over. And (at least for me) it was a fun ride, I admit.