Step initialization time too long using Partitioner in Spring-Batch?

Question

I'm using Partitioner to parallelize the import of *.csv files. There are about 30k files in the folder.

Problem: the job initialization takes about 1-2h hours until all files are set up. The bottleneck is in SimpleStepExecutionSplitter.split().

Question: is it normal that the step initializations require that much time? Or could I improve it somehow?

@Bean
public Step partitionStep(Partitioner partitioner) {
    return stepBuilderFactory.get("partitionStep")
            .partitioner(step())
            .partitioner("partitioner", partitioner)
            .taskExecutor(taskExecutor())
            .build();
}

@Bean
public TaskExecutor taskExecutor() {
    ThreadPoolTaskExecutor taskExecutor = new ThreadPoolTaskExecutor();
    taskExecutor.setCorePoolSize(4); //run import always with 4 parallel files
    taskExecutor.setMaxPoolSize(4);
    taskExecutor.afterPropertiesSet();
    return taskExecutor;
}


@Bean
public Partitioner partitioner() throws IOException {
    MultiResourcePartitioner p = new MultiResourcePartitioner();
    p.setResources(new PathMatchingResourcePatternResolver().getResources("mypath/*.csv"));
    return p;
}

Community · Accepted Answer · 2017-05-23 12:18:08Z

1

MultiResourcePartitioner creates a partition for each resource. Partition creation process in itself is very fast ( i.e. partitioner returns the executioncontext map very fast) but Spring Batch takes huge time in populating corresponding meta data DB tables and it becomes terribly slow once number of partitions goes beyond 100 ( this is all my personal experience).

As per only answer here, they did some improvements but I am using latest version and its very slow for partitions more than 100.

See this too.

I think, you don't have much of a choice other than reducing number of partitions unless you are ready to rewrite a bunch of API code by yourself.

edited May 23, 2017 at 12:18

CommunityBot

11 silver badge

answered May 5, 2017 at 16:23

Sabir Khan

10.2k7 gold badges51 silver badges105 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

membersound Over a year ago

I'm using spring.batch.initializer.enabled=false and MapJobRepository, thus storing any metadata inmemory only. Anyways the "spring-batch stuff" around the batch job seem to slow it down (for my 30k files I'd have 30k partitions; but I have to stick to Partitioner as I have to define an outputfile name for each file based on the input). So probably I have to drop spring-batch here unfortunately.

Mathieu P · Accepted Answer · 2017-08-17 11:23:49Z

0

I use a custom splitter because in the default splitter (https://github.com/spring-projects/spring-batch/blob/master/spring-batch-core/src/main/java/org/springframework/batch/core/partition/support/SimpleStepExecutionSplitter.java) , you call jobRepository.getLastStepExecution for each StepExecution. I don't use restartability with spring-batch, so i can write my own splitter. Now step initialization takes few seconds for thousand of files (before it was few minutes)

edited Aug 17, 2017 at 11:23

answered Aug 3, 2017 at 9:23

Mathieu P

11 bronze badge

1 Comment

Dov Benyomin Sohacheski Over a year ago

This should be a comment

Collectives™ on Stack Overflow

Step initialization time too long using Partitioner in Spring-Batch?

2 Answers 2

1 Comment

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related