Thread Schedule Pool not Pooling!

Hypothetically one of our services went down due to a memory saturation issue!

Upon jstack analysis, we found that, due to recent increase in usage of that service, several threads were getting accumalated and went into an eternal waiting state, thus blocking their assigned memory.

For these Thread creation a ThreadPool is used.

A thread pool basically has ability to reuse the old threads, which have completed execution of their task; thus saving the overhead required for creation of a thread.

In Java, ExecutorSerivce provides us with a thread pool and various methods to interact with this pool. One of it is, ScheduledThreadPoolExecutor which comes with a delayedQueue. This allows us to schedule tasks to be executed after a specified delay.

You can set a CorePoolSize i.e. the ideal pool size you want to maintain.

• CurrentPoolSize< CorePoolSize :New Threads are created for each task.

• CurrentPoolSize >= CorePoolSize : Queuing of task is preffered.

• CurrentPoolSize > CorePoolSize : Once task execution is completed the thread is killed.

Our flow generated a scheduled task from within a scheduled task if certain conditions were true.

something like this:

So ideally, new task should be assigned to one of the idle thread in task waiting state.

But I observed that new threads were created for each and every thread, even when the number of threads were way past CorePoolSize !

Initially we thought the child task in thread was holding up its parent. As the state of the older threads was WAITING as per our loggers.

There is no concept of parent child threads in Java, and blocking only comes into picture when synchronization is implemented.

But our synchronised method was releasing its lock aptly and the execution flow did continue perfectly beyond that.

So NO BLOCKING from the child thread either.

I scoured the internet for reasons behind this behaviour for hours, none of which matched our scenario.

So in order to better understand what was exactly happening, I created a POC while replicating our behaviour;

I tried

• thread.interrupt()

• thread.stop()

• thread. destroy()

at the end of task, but seemingly nothing could kill those threads!

Well lucky for me, I had blindly recreated the ditto functionality flow in my POC. So I was getting the exact same issue with the threads, but then I thought about zooming out of the problem and noticed that a new instance of object was being created as each time a thread wanted to create a scheduled task!

I had 2 options:

1. Make the class singleton

2. Make the scheduledThreadPool variable static

2nd option would have lesser impact on existing system so we went ahead with that for a hotfix.

Ideally speaking the ThreadPool object should be part of a singleton class which does not have any other responsibility; so that its instance can be summoned wherever needed.

One more clear hint which I missed was the name of threads being printed from within each task:

“thread-pool-67-thread-1”

“thread-pool-68-thread-1”

“thread-pool-69-thread-1”

“thread-pool-70-thread-1”

This obviously meant that new thread pools were created for each task which I initially overlooked.

This issue gave me an opportunity to understand the working of Java threads, scheduled threads, delayed queues and the way they interact with each other in depth.

TLDR; New instance of thread pool was being created for each scheduled timer task thus not allowing reuse of threads and causing memory saturation issues.