A Missing Timeout, a Synchronised Block, and 205 Stuck Threads
DEV Community

A Missing Timeout, a Synchronised Block, and 205 Stuck Threads

A few days ago our highest-traffic service started returning 504s on every call. There were no errors in the logs, no traces, and Kubernetes swore every pod was healthy.

The root cause turned out to be one HTTP call without a timeout, sitting inside a synchronized block. This is the story of how we found it, and the fixes I would apply to any service that has a cache, a lock and a remote call stacked together.

TL;DR

  • Symptom: every request returned 504 and the server logged nothing.
  • Cause: a token refresh ran inside a synchronized method and called an auth service with no timeout. The call never returned, so all other threads queued behind the lock.
  • Why it stayed up: the pod kept passing its health checks, so Kubernetes never restarted it.
  • Fix: timeouts everywhere, ReentrantLock with tryLock, granular locks, no distributed lock for this case, and never synchronized together with @Transactional.

The symptom: 504s and complete silence

It was the service that handles more API calls than any other in our system. Every request began returning 504 Gateway Timeout. On the client side the call sat in the transfer phase until the gateway gave up.

On the server side there was nothing: no exception, no slow-query warning, no error log. We had no distributed tracing on this path either, so there was no span to open and no trail to follow. From the outside the service looked alive. From the inside it was saying nothing at all.

The wrong suspect: our Kubernetes setup

When requests die before your code logs anything, the instinct is to blame what sits in front of it. So we went deep on the platform:

  • Ingress: configuration and logs were healthy.
  • Load balancer: IP configuration and routing looked correct.
  • Cluster: kubectl describe on pods, services and endpoints showed nothing unusual.

Everything came back clean. That was the first real clue, although we did not read it that way yet. When the infrastructure is innocent and the application is silent, the process is usually alive but no longer doing work.

So we took a heap dump and a thread dump. The heap dump gave us nothing: memory looked normal, and a heap dump shows what objects exist, not what threads are doing. The thread dump did not disappoint.

What the thread dump said

205 platform threads were sitting in a waiting state. That is roughly the size of a default servlet container worker pool (Tomcat ships with 200), so in practice every thread that could serve a request was stuck. New requests had nowhere to run, queued up, and the gateway timed them out with a 504. That also explains the silence: a request that never gets a thread never reaches a log statement.

All 205 stacks looked the same, and they all led to one place:

  • Feign client call -> request interceptor needs an access token -> token provider: cached token expired -> synchronized refresh method <- every other thread queues here -> HTTP GET to the authentication service <- the first thread is stuck here

Two habits made this findable:

  • Take two or three dumps a few seconds apart. If the same threads sit on the same frames each time, you are looking at a stall, not slow work.
  • Find the lock owner. Waiters show waiting to lock <0x...> , and one thread shows locked <0x...> for that same address. That thread is the one to read first.

A thread blocked on a socket read still looks runnable to the JVM, so the culprit is easy to skip if you only scan for blocked threads.

Anatomy of the stall

dev.to does not render diagrams from code, so here is the flow as text:

Incoming request | v Feign call needs a token | v Cached token has expired | v synchronized refresh method | +--> FIRST thread in: holds the lock |HTTP GET to the auth service, no timeout, no reply <-- root cause | +--> EVERY OTHER thread: waits for the same lock queued behind the first thread, which never completes | v All worker threads parked: new requests queue, gateway returns 504

The first thread to find an expired token took the lock and called the authentication service, and with no timeout that call never came back. Every other thread that needed a token joined the queue behind it, until none were left to serve requests.

Why Kubernetes never restarted the pod

The service was completely stalled, and Kubernetes still considered every pod healthy. That is not a Kubernetes bug. A probe answers one narrow question, is this process responding the way I check? , and a JVM with every worker thread parked can still answer a TCP check, or an endpoint served from a separate management port and thread pool.

What we needed was signals that measure ability to do work, not just liveness:

Signal What it would have shown Busy worker threads vs. pool max Pinned at the maximum within seconds of the stall Latency histogram on the outbound auth call One call that never completes, so the p99 grows without bound Distributed tracing A single span stuck inside the token fetch, with 200+ requests queued behind it Alert on in-flight requests A line going up and never coming down

A word of caution: do not make your liveness probe call downstream dependencies. If the auth service is down, you would restart every healthy pod and turn a partial outage into a total one. Alert on saturation instead and let a human or an autoscaler decide.

The second trap: synchronized and @Transactional together

While digging through this code we found a related pattern that deserves its own warning. It looks harmless:

@Service
public class AccountService {
  @Transactional
  public synchronized void debit(Long accountId, BigDecimal amount) {
    Account account = accountRepository.findById(accountId).orElseThrow();
    account.setBalance(account.getBalance().subtract(amount));
  }
}

It reads like "only one thread at a time can debit, inside a transaction". It does not do that, for four reasons:

  • The lock is released before the commit. Spring wraps the method in a proxy. The transaction starts before the method runs and commits after it returns, and synchronized only covers the method body. Thread B can enter, read the old balance, and overwrite thread A's change. The lock gave you a feeling of safety and none of the guarantee.
  • Waiting threads hold database connections. The proxy usually opens the transaction, and takes a connection, before the thread ever reaches the monitor. Threads queued on the lock sit on pooled connections, and a small pool is exhausted by a handful of waiters. Then even endpoints that never touch this lock start timing out.
  • It only works inside one JVM. Two pods have two monitors, so they can still race each other on the same row.
  • Slow work inside the lock stretches the transaction. Put a remote call in there and you have rebuilt our incident, with an open transaction on top.

The correct tool for protecting a row is the database, which is where the cure section ends up.

The cure: five fixes, in the order they matter

The root cause was a missing timeout. The lock only turned one slow call into a service-wide outage. Fix the cause first, then make the lock safe.

1. Put a timeout on every outbound call

Many HTTP clients wait forever by default, including HttpURLConnection and java.net.http.HttpClient. An unbounded wait is a bug waiting for its worst day. Every call needs a connect timeout, a read timeout and an overall deadline.

spring:
  cloud:
    openfeign:
      client:
        config:
          default:
            connectTimeout: 2000 # ms
            readTimeout: 5000 # ms
HttpClient client = HttpClient.newBuilder().connectTimeout(Duration.ofSeconds(2)).build();
HttpRequest request = HttpRequest.newBuilder(tokenUri).timeout(Duration.ofSeconds(3)).GET().build();

Pair this with a circuit breaker such as Resilience4j so that a dead dependency fails fast instead of being retried by every thread.

2. Replace synchronized with ReentrantLock and tryLock

The real weakness of synchronized is not that it is slow. It is that a waiting thread cannot give up. ReentrantLock lets a thread wait for a bounded time, fall back, or fail fast. Reentrancy itself is a minor feature here; tryLock(timeout) is the point.

@Component
public class TokenProvider {
  private final ReentrantLock refreshLock = new ReentrantLock();
  private volatile CachedToken cached;
  private final AuthClient authClient;

  public TokenProvider(AuthClient authClient) {
    this.authClient = authClient;
  }

  public String getToken() {
    CachedToken token = cached;
    if (token != null && !token.nearExpiry()) {
      return token.value(); // fast path, no lock
    }
    boolean locked = false;
    try {
      locked = refreshLock.tryLock(2, TimeUnit.SECONDS);
      if (!locked) {
        // someone else is refreshing and it is taking too long
        if (token != null && !token.expired()) {
          return token.value(); // stale but still valid
        }
        throw new TokenUnavailableException("Token refresh timed out");
      }
      token = cached; // double-check after locking
      if (token == null || token.nearExpiry()) {
        token = authClient.fetchToken(); // has its own read timeout
        cached = token;
      }
      return token.value();
    } catch (InterruptedException e) {
      Thread.currentThread().interrupt();
      throw new TokenUnavailableException("Interrupted while waiting", e);
    } finally {
      if (locked) {
        refreshLock.unlock();
      }
    }
  }
}

Three details carry the design: the unlocked fast path, the double-check after acquiring the lock, and the fallback to a stale-but-valid token. Refreshing a little before expiry (say at 80% of the token lifetime) makes that fallback available almost every time.

3. Use granular locks

One global lock means one slow dependency stalls everything that shares it. Lock per key instead, here per client or scope, so a stuck refresh blocks only the callers that truly need that token.

private final ConcurrentMap<String, TokenSlot> slots = new ConcurrentHashMap<>();

public String tokenFor(String clientId) {
  TokenSlot slot = slots.computeIfAbsent(clientId, id -> new TokenSlot());
  return slot.get(() -> authClient.fetchToken(clientId)); // same tryLock logic, one lock per slot
}

The same principle applies anywhere you lock: keep the critical section as small as you can and never perform network I/O inside it unless that I/O is strictly bounded.

4. Do not reach for a distributed lock here

It is tempting to say "let us coordinate the refresh across all pods with Redis or the database". For a token cache it does not help, and it adds risk:

  • Each pod needs its own valid token anyway, so duplicate refreshes across pods are harmless. The lock only needs to stop a stampede inside one JVM.
  • A distributed lock is a new dependency with its own network calls, and those calls need timeouts too.
  • A lock holder that dies leaves a lock that nobody releases unless it carries an expiry (a lease or TTL). You have swapped one remote call that can hang for another.

Distributed locks are the right tool when pods must not run the same job at once, such as a scheduled task (ShedLock does this well). Always give them a TTL.

5. Never combine synchronized with @Transactional

Let the database arbitrate, because it is the only party that sees all pods and knows when a commit really happened. With JPA you have two options. Optimistic locking lets concurrent writers race and makes the loser retry:

@Entity
public class Account {
  @Id
  @GeneratedValue
  private Long id;
  private BigDecimal balance;
  @Version
  private long version; // an update fails if another writer got there first
}

Pessimistic locking serialises writers on the row, with a bounded wait:

public interface AccountRepository extends JpaRepository<Account, Long> {
  @Lock(LockModeType.PESSIMISTIC_WRITE)
  @QueryHints(@QueryHint(name = "jakarta.persistence.lock.timeout", value = "3000"))
  @Query("select a from Account a where a.id = :id")
  Optional<Account> findByIdForUpdate(@Param("id") Long id);
}
@Service
public class AccountService {
  @Transactional
  public void debit(Long accountId, BigDecimal amount) {
    // no synchronized
    Account account = accountRepository.findByIdForUpdate(accountId).orElseThrow();
    account.setBalance(account.getBalance().subtract(amount));
  }
}

The row lock is held until the transaction commits, which is exactly the guarantee synchronized could not give. It works across pods, and the lock timeout means a waiter gives up instead of hanging.

What to audit in your own services this week

None of the individual pieces here was exotic. A cache, a lock and an HTTP call is a design most services contain somewhere. The failure came from how they stacked, so audit the stack:

  • Every outbound HTTP, Feign, JDBC and message-broker call has an explicit timeout. Search your code for clients built with defaults.
  • No lock is held across network I/O unless that I/O is strictly bounded.
  • Every lock that can queue threads offers a way out: tryLock(timeout), a lock timeout in the database, or a lease on a distributed lock.
  • Token and credential refresh happens ahead of expiry, and a stale-but-valid token is an acceptable fallback.
  • No synchronized method carries @Transactional. Use @Version or a pessimistic lock with a timeout instead.
  • You graph busy threads against pool size and the latency of each dependency, and you alert on saturation.
  • Tracing is on for the highest-traffic paths, so the next stuck call shows up as a span instead of silence.
  • You know the thread-dump command for your runtime and have practised it in a pod: jcmd <pid> Thread.print or kill -3 <pid>.

One last point: virtual threads will not rescue you from this. Millions of cheap threads can all queue behind one lock held across an unbounded call. Timeouts are what end the wait, and the fix works the same on any kind of thread.

We found this one by luck and a thread dump. The goal is to find the next one by a dashboard. Have you been bitten by a missing timeout or a lock held across a remote call? Tell me what it looked like in the comment

Read on DEV Community ↗ ← Back to News

Comments

No comments yet. Start the discussion.