May 31, 2011

Executive Insight: Trace is the Light!

From time to time, if you are experiencing an issue, we may ask you to “send us trace” or we may even talk you through “turning on verbose trace” and ask you to send it to us the next day.
Let’s talk about what trace is. Trace is data or logs that are recorded at every step throughout transmissions to and from our Universal Transaction Gateway® (UTG®). We also have trace in our data centers that can be matched up with the UTG’s trace to rapidly resolve issues.

If our support department has a concern that your Point of Sale (POS) system or your Property Management System (PMS) may be handling something in a manner that is not consistent with our Application Program Interface (API), you will be asked to turn on verbose trace for a short period of time. (We don’t leave it on for a very long period because it can store too much information.)

When Shift4 asks to initiate a trace, merchants often assume we’re looking for a problem on our side. Generally, that is not the case. Shift4 uses trace to isolate problems on the network between your POS and Shift4’s UTG or in the telecommunications between the UTG and Shift4’s data center. Basically, we use trace to keep track of what is going on in the areas of the system over which we have no control. Trace merely “turns on the light,” illuminating issues in the vast and complicated world of networking and telecommunications.

With trace, we can usually solve problems in minutes or hours, whereas without it, we may never be able to track down the underlying issue. All software is designed to work in a particular environment and that environment assumes certain capabilities are functioning as expected. Trace can show us what actually happened, when it happened, and the result – illuminating the solution.

We don’t ask for trace to make your job harder; we ask for trace because we don’t want to wander around in the dark when trying to help you. We want to help you to have a system that operates as intended and is not hampered by the problems that can creep into networks, security, and telecommunications.

A perfect example of how trace helps is a situation where a very small number of our merchants were experiencing loss of authorization capability. The first merchant was asked for trace and they happily sent it in. Trace illuminated that UTG was not able to communicate with our data center. Only a few things can cause that situation. 1) The merchant’s connection to their telecommunications (Internet) provider is not working properly, 2) The telecommunications provider is not working, or 3) Shift4’s DOLLARS ON THE NET® is not working properly.

It quickly became obvious that the cause of this problem wasn’t Shift4, because we were successfully processing millions of transactions at the time. It wasn’t the merchant’s Internet provider as both the merchant and Shift4 could validate that it was up and running normally. That left only one thing – the merchant’s connection to their Internet provider. We immediately discovered that for some reason the merchant’s security firewall was only allowing outbound connections to one address at Shift4. The UTG configuration and associated documentation requires that outbound connections be allowed to several spans of IP addresses. This merchant, in trying to be extra secure, locked it down to only one. See our article on the redundancy, resiliency, and reliability of DOLLARS ON THE NET service to understand why this was a problem.

Suffice it to say, by getting trace we were able to solve this merchant’s problem in minutes, giving them the ability to again accept payments and to solve a problem that could have continued to affect them in the future.

Remember: Trace is your friend. Trace is the light. Turn it on!