Work experiences · Delivery Hero · Payment Smart Routing — STAR account
Payment Smart Routing — STAR account
My role in designing and delivering automatic payment-provider failover.
Read the case study for the system design or the one-screen brief for a short account.
Situation
At Delivery Hero, I worked on Payment Smart Routing in the Payment Orchestration team.
Each country had an ordered PSP list, with a primary provider and backups. Prod-Ops manually disabled the primary when it had problems.
Success-rate reporting reached Prod-Ops after a delay. Manual intervention added more time before routing changed.
During that interval, failed payments could cause lost orders and reduce GMV. We needed to shorten the response to an underperforming PSP.
Task
I took ownership of the RFC and compared technical alternatives. I coordinated stakeholder approval and developed the internal design, implementation plan, and issues.
I also planned the gradual production rollout and designed the metrics and dashboards needed to evaluate it.
My implementation responsibility focused on success-rate calculation and rule evaluation. Other team members also implemented parts of the solution.
Action
Choose scheduled polling
I compared event-driven aggregation with polling the transaction service.
We selected a scheduled job in Insight to avoid additional event infrastructure costs. Insight stored successful and failed transaction counts in one-minute Redis buckets.
Prod-Ops configured the evaluation window and threshold. Below the minimum transaction count, evaluation left the PSP eligible.
Avoid another synchronous service call
I recommended embedding the routing logic in the Payment Integration Gateway through a library.
A separate routing service would add a network hop to payment processing. The library used Redis data and control plane configuration within the gateway.
Select a rule model
I considered condition trees, key-value rules with a custom evaluator, a dedicated rule engine, and SpEL.
I benchmarked SpEL rule evaluation and confirmed that it could handle 50 times our expected workload. This capacity margin supported my selection.
We stored expressions as configuration and exposed a function for those expressions to call.
Define operational safeguards
We maintained a separate health indicator and applied success-rate evaluation only to enabled PSPs.
I asked Prod-Ops to retain at least one fallback outside automatic exclusion.
If the library could not calculate success rate, it bypassed that evaluation and processed the payment through the selected PSP. This applied to any calculation failure, including Redis failures.
The first release required Prod-Ops to restore excluded PSPs manually. It changed routing for subsequent payments and did not retry the failed transaction.
Roll out and observe
I planned a gradual rollout, starting with PedidosYa Argentina across all payment methods.
I compared production performance with the previous week to check normal operation. After several days, we observed automatic failover during a real degradation.
Result
Smart Routing reached production. It could exclude underperforming PSPs from subsequent routing decisions without waiting for manual intervention.
In one production incident, approximately 1,300 orders completed through the backup PSP after automatic failover.
Earlier PSP incidents had affected our overall payment success metric. This history supported our assessment that automatic failover helped protect orders.
We observed the completed orders. We estimated how many would otherwise have failed; we did not establish that every order would have failed.
The first release counted all failures, including insufficient funds. We planned to distinguish technical failures, add automatic recovery, and support selected operating periods.
Details to confirm
- How did the incident’s overall payment success rate compare with earlier PSP incidents?
- What absolute evaluation rate, latency, and test environment supported the SpEL benchmark’s 50-times capacity margin?
- Who configured the minimum transaction count, and which configuration level applied to each rule?