My Java Spring @Scheduled methods stopped running in production

Mon, 20 May 2019 18:32:19 -0400

The first time

The first time I encountered this issue was an @Scheduled annotation wrapping a method that performed a long(ish) running query. For example:

java


@Scheduled(fixedRate = 600000)
public void scheduledMethod() {
  System.out.println("I am running");
  performSomeLongRunningQuery();
  System.out.println("I am done");
}

This works great, right? For months in production, my query had been running every 10 minutes as I so clearly told it to until one day I got a call that indicated to me that is was no longer running. Sure enough, after checking the logs of this code above I saw a message stating “I am running”, but there was no “I am done” message. To my knowledge, once this happens to a snippet such as the one above your only option is to restart the application. As expected after a restart we were back up and running every 10 minutes. So what went wrong? I was able to confirm that the query did indeed finish executing on the database, so why didn’t it return? I won’t get into details, but I believe the connection to the database was severed. At this point, I figured we just got lucky (or unlucky) and I forgot about it.

The second time

The second time I encountered an @Scheduled method that stopped running (in a different application) was an @Scheduled annotation wrapping a method that performed an API call. Another developer and I were looking into this and I immediately knew what was wrong and how to fix it, although this time I had no idea what caused it. We restarted the application and everything was back to normal. I should also note, this application has more than just one @Scheduled method and we noticed those were no longer running either.

Time to figure out what happened

Clearly now it was time for me to better understand what is actually going on under the hood of the @Scheduled annotation. As soon as I got home from work I started playing around with the little demo below.

java


public class DemoSchedule {

  @Scheduled(fixedRate = 1000)
  public void scheduledMethod() {
    System.out.println("I am only going to run once");
    sleepy();
  }

  public void sleepy() {
    System.out.println("sleeping");
    try {
      Thread.sleep(60000);
    } catch (InterruptedException e) {
      e.printStackTrace();
    }
    sleepy();
  }

}

This example demonstrates what happens to an @Scheduled method that never returns and more importantly never throws an exception. Simply adding

java


throw new Exception();

anywhere inside the method sleepy() will cause the behavior I want/expect. When the exception is thrown the scheduled method will complete and run again.

Even more to my surprise if you add another @Scheduled method, it won’t run while scheduledMethod() is running forever. I didn’t know why this was the case because I never looked at the documentation for @EnableScheduling.

The documentation

Taken from the Spring EnableScheduling documentation:

By default, will be searching for an associated scheduler definition: either a unique TaskScheduler bean in the context, or a TaskScheduler bean named “taskScheduler” otherwise; the same lookup will also be performed for a ScheduledExecutorService bean. If neither of the two is resolvable, a local single-threaded default scheduler will be created and used within the registrar.

Takeaways from the above documentation:

The default scheduler is only allotted one thread, which is why the method above will only run once.
All @Scheduled annotations share the same thread pool, which means all other @Scheduled annotations would be blocked from executing in the above example.

Remember, a few hours of trial and error can save you several minutes of looking at the README.
— I Am Devloper (@iamdevloper) November 7, 2018

How many @Scheduled methods could be executed simultaneously?

If you only have one @Scheduled method in your application then technically you are fine with leaving it with the default single thread. Although, I’m willing to bet when you add another you won’t remember to increase the thread pool so it is probably worth doing it now.

If more than one @Scheduled method in your application could be overlap, then you should look at implementing SchedulingConfigurer. As you can see in the example below you will implement this on a class that is annotated with @Configuration and @EnableScheduling. One thing to note about the example below is they are using 100 as the thread pool size. To my knowledge, the number here doesn’t really matter as long as it is at least the number of @Scheduled methods that could overlap, that way no @Scheduled method is ever waiting on another to finish before running.

Taken from the Spring EnableScheduling documentation:

java


@Configuration
@EnableScheduling
public class AppConfig implements SchedulingConfigurer {

    @Override
    public void configureTasks(ScheduledTaskRegistrar taskRegistrar) {
        taskRegistrar.setScheduler(taskExecutor());
    }

    @Bean(destroyMethod="shutdown")
    public Executor taskExecutor() {
        return Executors.newScheduledThreadPool(100);
    }
}

Is there any chance your @Scheduled method will never exit?

This one is on you to figure out based on what you are doing in each of your @Scheduled methods. As I showed with the sleepy() example, if the method never returns, or throws an exception, it will never run again. Both times my @Scheduled methods stopped running in production could have been prevented with the proper timeout configurations on the datasource/HTTP client.

The end

If you made it this far, thanks for reading my first technical blog post! I acknowledge this is nothing profound and nothing you couldn’t learn from reading the documentation. I just wanted to share a little story about something that broke in production and what I learned from it.

If you think any of this information is wrong, have questions/comments, or have any feedback then please reach out to me on Twitter!

Hello, World!

Wed, 01 May 2019 07:17:29 -0400

The man behind the blog post

I have been working as a software developer for 3 years now. I write primarily Java and Angular at work but I have been spending my mornings before work learning Go. I have been practicing yoga for just over 2 years and I have my wife to thank for getting me into it. I enjoy two cups of coffee a day and my preferred brewing method is pour over. My local coffee shop introduced me to this method and I haven’t looked back (shout out to The Clay Cup in Altoona). Meditation is the newest addition to my list of favorite things. I have been practicing 20 minutes of meditation a day for the last 102 days.

Why Hugo?

I’m sure a lot of you are wondering why I made a site with Hugo…and by a lot of you I mean nobody because I’ve never written a blog post before. Here is a short list of why I am choosing Hugo because if you have made it this far I’m sure you are just dying to know.

It is written in Go and I can’t get enough of that Gopher.
I get to use a theme which means I don’t have to write any CSS.
It was recommended by a co-worker. Check out his blog here.
Even if I don’t stick with writing blog posts, this is better than my old website.
Markdown is pretty cool I guess.

About me

Mon, 01 Jan 0001 00:00:00 +0000

I am Devon Romanko, a senior software engineer at DICK’s Sporting Goods. I work remotely from Pittsburgh, PA.

I am primarily a Java developer but I am also comfortable on the front end with Angular. I love building resilient applications that keep working when things go wrong, writing tools/scripts that help other developers be more efficient, and reviewing code from my peers. My current hobby language has been Go for the past few years and I look for any excuse to use it.

A few of my favorite things outside of tech include yoga, meditation, video games, and single-origin coffee. I have been practicing yoga since 2017, practicing meditation since 2019, and playing video games my whole life.

dpromanko

My Java Spring @Scheduled methods stopped running in production

The first time

The second time

Time to figure out what happened

The documentation

How many @Scheduled methods could be executed simultaneously?

Is there any chance your @Scheduled method will never exit?

The end

Hello, World!

The man behind the blog post

Why Hugo?

About me

Where you can find me