Software Engineer (Spring Boot | Kafka | Microservices | Java | React Js | Gen AI | Agentic AI | 6+ yrs Exp | SDE II @ Airtel Payments Bank)
The $20k Bug Caught by a Java Profiler A few years back, one of our payment APIs suddenly slowed down. Everything looked fine: CPU usage: Normal Database load: Low Error logs: Clean Still… each request took 4+ seconds. We fired up a Java profiler for (String id : customerIds) { sendPaymentReminder(id); // sequential } The profiler showed 90% of total time stuck here. The culprit? Sequential execution of independent calls. CompletableFuture.allOf( customerIds.stream() .map(id -> CompletableFuture.runAsync(() -> sendPaymentReminder(id))) .toArray(CompletableFuture[]::new) ).join(); Result: 4.1s → 300ms Impact: Faster API, happier customers, and a $20k SLA penalty avoided. Lesson: Profiling pinpoints where the real bottleneck is. In Java, CompletableFuture is your friend for independent tasks. Sometimes, the costliest bugs hide in the simplest loops.
Its a performance Issue not a bug,coudnt this thing have worked out better customerIds.parallelStream() .forEach(this::sendPaymentReminder);?
This code might be more performant at small scale, but if you start stepping into the realm of very high throughput your going to run into similar mysterious slowdowns. When you utilize CompletableFuture.runAsync without providing it an executor, you submit to the Common ForkJoin pool which is shared with all operations in the java.util.concurrent package, as well as any parallel streams operations. If you wanted to take this to the next level, you should utilize runAsync with some form of executor using virtual threads.
That is an equally problematic solution if not worse. It may not be apparent at first but the main reason is that you are allocating this work to the common fork join thread pool. Do this from multiple places, and you will not be able to tell when does what task slow down and why some code from some other place unexpectedly starts affecting your throughput for some other task. The common fork join pool is meant for CPU bound task not I/O or network tasks. So a better way will be to create a dedicated threadpool for this task and use the overloaded version runAsync(Runnable runnable, Executor executor).
What did you ensure that any failures in async don't go unnoticed? You can make it highly scalable by using a SQS queue and multiple workers to process the message.
You don't really need a profiler to figure this out. Most concurrency issues are usually along the lines: An API is slow because it hits the database 5 times in a row instead of in parallel, a file upload blocks the web server because processing happens on the main thread, service takes forever because it calls four other services one by one instead of concurrently. Mostly the fix is just CompletableFuture, ExecutorService for performance gain and parallelism
I had similar situation on a project. We had a problem that new added to DB data events were processing very slowly, so we used CompletableFuture plus some additional optimizations and we got about as a result 7-10 seconds processing instead of 30 mins
Very interesting. I've taken note of it!
Nice approach will have it noted
Software Engineer (Spring Boot | Kafka | Microservices | Java | React Js | Gen AI | Agentic AI | 6+ yrs Exp | SDE II @ Airtel Payments Bank)
2moWhat’s the smallest Java change you’ve made that saved big money?