Merchants are no strangers to rising payment-processing costs. Merchant-processing fees often feel unavoidable, driven by interchange rates and card-network rules that are beyond a merchant’s control.

TransFirst Holdings Inc. is trying to make a niche for itself in third-party merchant processing, a business it says regional banks are vacating. This month the Dallas company, which is owned by the ...