java - Issues with Spring Integration and process taking time and pausing -

i looking @ issue have in our application. spring integration being used poll particular directory , process files in directory. can process 5k 1kb files , there huge pause application doing nothing sitting idle , completes process in 4 minutes. next run take bit longer , 1 after takes longer , on until restart application goes 4 minutes mark. has experienced issue before.

i wrote standalone version without spring integration , dont same issue. have below pasted xml config, incase have done wrong can't spot.

thanks in advance.

 <!-- poll input file directory new files.  if found, send java file object on inputfilechannel -->     <file:inbound-channel-adapter directory="file:${filepath}"         channel="inputfilechannel" filename-regex=".+-ok.xml">         <si:poller fixed-rate="5000" max-messages-per-poll="1" />     </file:inbound-channel-adapter>      <si:channel id="inputfilechannel" />      <!-- call processfile() , start parsing xml inside file -->     <si:service-activator input-channel="inputfilechannel"                           method="splitfile" ref="splitfileservice">     </si:service-activator>      <!-- poll input file directory new files.  if found, send java file object on inputfilechannel -->     <file:inbound-channel-adapter directory="file:${direcotrypath}" channel="inputfilerecordchannel" filename-regex=".+-ok.xml">   <si:poller fixed-rate="5000" max-messages-per-poll="250" task-executor="executor" />     </file:inbound-channel-adapter>      <task:executor id="executor" pool-size="8"                    queue-capacity="0"                    rejection-policy="discard"/>      <si:channel id="inputfilerecordchannel" />      <!-- call processfile() , start parsing xml inside file -->     <si:service-activator input-channel="inputfilerecordchannel"                           method="processfile" ref="processedfileservice">     </si:service-activator>      <si:channel id="wsrequestschannel"/>      <!-- sends messages wsrequestschannel httpsender, , returns responses on          wsresponseschannel.  used once each record found in input file. -->     <int-ws:outbound-gateway uri="#{'http://localhost:'+interfaceservice.getwebserviceinternalinterface().getipport()+'/ws'}"         message-sender="httpsender"         request-channel="wsrequestschannel" reply-channel="wsresponseschannel" mapped-request-headers="soap-header"/>      <!-- handles responses web service (wsresponseschannel).  again          used once each response web service -->     <si:service-activator input-channel="wsresponseschannel"         method="handleresponse" ref="responseprocessedfileservice">     </si:service-activator> 

as surmised in comment question, (default) acceptoncefilelistfilter not scale large number of files because performs linear search on processed files.

we can make improvements there; opened jira issue that.

however, if don't need semantics of filter (i.e. flow removes input file on completion), can replace filter, such acceptallfilelistfilter.

if need accept once semantics need more efficient implementation such large number of files. warn when using such large number of files, if don't remove them after processing, things going slow down anyway, regardless of filter.