How to debug network latency in a multi-region data center

Prerequisites

• Access to a Linux server in at least two different data center regions (e.g., US-East and EU-West).
• A stable network connection between the regions with IP connectivity.
• Root or sudo privileges to run network diagnostic tools.
• Installed packages: iputils-ping, traceroute, mtr, and tc (traffic control).

Step 1: Check basic connectivity and round-trip time

Start by verifying that your servers can reach each other and measure the baseline round-trip time (RTT). Run the ping command from the source server to the destination server's public IP address.

ping -c 4 <destination-ip>

You will see output showing the time taken for each of the four packets to travel and return. Look for the rtt min/avg/max/mdev line at the end.

Ping 10.0.0.5 (10.0.0.5) 56(84) bytes of data.
64 bytes from 10.0.0.5: icmp_seq=1 ttl=64 time=12.4 ms
64 bytes from 10.0.0.5: icmp_seq=2 ttl=64 time=11.8 ms
64 bytes from 10.0.0.5: icmp_seq=3 ttl=64 time=12.1 ms
64 bytes from 10.0.0.5: icmp_seq=4 ttl=64 time=11.9 ms
--- 10.0.0.5 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3004ms
rtt min/avg/max/mdev = 11.800/12.050/12.400/0.300 ms

If the packet loss is 0% and the mdev (mean deviation) is low, the connection is stable. High mdev or packet loss indicates instability in the path.

Step 2: Map the route and identify slow hops

Next, determine which network hop is causing the delay. Run the mtr command to combine the functionality of ping and traceroute. This tool shows packet loss and latency for every hop along the path.

mtr -c 20 -r -w <destination-ip>

The -c 20 flag sends 20 probes, -r enables reverse path checking, and -w forces a wide output to show all columns. Review the output table. The columns include Host, Loss%, Snt, Last, Avg, Best, Wrst, StDev.

Host      Loss%   Snt   Last  Avg  Best  Wrst StDev
  1.***    0.0%    20   1.2   1.1   0.9   1.5   0.1
  2.***    0.0%    20   2.3   2.2   2.0   2.6   0.1
  3.***    5.0%    20   3.4   3.3   3.1   4.2   0.2
  4.***    0.0%    20   4.5   4.4   4.2   4.9   0.1
  5.***    0.0%    20   5.6   5.5   5.3   5.8   0.1

Look for rows with high Loss% or high Wrst (worst) latency. In this example, hop 3 shows 5% packet loss and a worst latency of 4.2ms. This hop is likely the bottleneck or experiencing congestion.

Step 3: Analyze packet distribution with tc

If you suspect traffic shaping or queuing delays, use the tc command to inspect traffic control rules on the interface. This helps identify if the network device is dropping packets due to queue saturation.

tc -s qdisc show dev <interface-name>

Check the output for pfifo or htb queues. If the queue length is high, packets are waiting in the buffer, causing latency. High latency combined with packet loss often points to a congested link or a misconfigured router.

Verify the installation

To ensure your diagnostic environment is ready, run a quick sanity check. Verify that all required tools are installed and functional.

which ping mtr traceroute tc

The output should list the paths to the binaries, typically /usr/bin/ping, /usr/bin/mtr, etc. If any command is missing, install it using your package manager.

apt-get install iputils-ping traceroute mtr

Troubleshooting

If you encounter issues during the diagnostic process, follow these steps to resolve them.

1. Packet loss on specific hops: If mtr shows packet loss on a specific hop, check if the router at that hop is overloaded. You may need to contact your ISP or cloud provider to investigate the upstream link.
2. High latency on the first hop: If the first hop shows high latency, the issue is likely within your local data center or the local network interface. Check for CPU saturation or network driver issues.
3. Intermittent connectivity: If the connection drops intermittently, run ping with a longer duration to see if the issue persists. Use tcpdump to capture packets and inspect for retransmissions or timeouts.
4. Firewall blocking ICMP: If you cannot ping a destination, verify that ICMP echo requests are not blocked by a firewall. Temporarily disable the firewall to test connectivity.
5. Asymmetric routing: If latency is high in one direction but not the other, check for asymmetric routing. Packets may take a different path back to the source, causing higher latency.

Always document the baseline metrics before making changes to the network configuration. Compare the new metrics against the baseline to ensure that any optimization efforts improve performance without introducing new issues.