The File component provides access to file systems, allowing files to be processed by any other Camel Components or messages from other components to be saved to disk.
or
file://directoryName[?options]Where directoryName represents the underlying file directory.
You can append query options to the URI in the following format, ?option=value&option=value&...
Only directories Camel 2.0 only support endpoints configured with a starting directory. So the directoryName must be a directory. If you want to consume a single file only, you can use the fileName option, e.g. by setting fileName=thefilename . Also, the starting directory must not contain dynamic expressions with ${ } placeholders. Again use the fileName option to specify the dynamic part of the filename.In Camel 1.x you could also configure a file and this caused more harm than good as it could lead to confusing situations.
Avoid reading files currently being written by another application Beware the JDK File IO API is a bit limited in detecting whether another application is currently writing/copying a file. And the implementation can be different depending on OS platform as well. This could lead to that Camel thinks the file is not locked by another process and start consuming it. Therefore you have to do you own investigation what suites your environment. To help with this Camel provides different readLock options and doneFileOption option that you can use. See also the section Consuming files from folders where others drop files directly .Any move or delete operations is executed after (post command) the routing has completed; so during processing of the Exchange the file is still located in the inbox folder.
Lets illustrate this with an example:
from("file://inbox?move=.done" ).to("bean:handleOrder" );When a file is dropped in the inbox folder, the file consumer notices this and creates a new FileExchange that is routed to the handleOrder bean. The bean then processes the File object. At this point in time the file is still located in the inbox folder. After the bean completes, and thus the route is completed, the file consumer will perform the move operation and move the file to the .done sub-folder.
The move and preMove options should be a directory name, which can be either relative or absolute. If relative, the directory is created as a sub-folder from within the folder where the file was consumed.
By default, Camel will move consumed files to the .camel sub-folder relative to the directory where the file was consumed.
If you want to delete the file after processing, the route should be:
from("file://inobox?delete=true " ).to("bean:handleOrder" );We have introduced a pre move operation to move files before they are processed. This allows you to mark which files have been scanned as they are moved to this sub folder before being processed.
from("file://inbox?preMove=inprogress" ).to("bean:handleOrder" );You can combine the pre move and the regular move:
from("file://inbox?preMove=inprogress&move=.done" ).to("bean:handleOrder" );So in this situation, the file is in the inprogress folder when being processed and after it's processed, it's moved to the .done folder.
The move and preMove option is Expression -based, so we have the full power of the File Language to do advanced configuration of the directory and name pattern. Camel will, in fact, internally convert the directory name you enter into a File Language expression. So when we enter move=.done Camel will convert this into: ${file:parent }/.done/${file:onlyname }. This is only done if Camel detects that you have not provided a ${ } in the option value yourself. So when you enter a ${ } Camel will not convert it and thus you have the full power.
So if we want to move the file into a backup folder with today's date as the pattern, we can do:
move=backup/${date:now:yyyyMMdd}/${file:name}The moveFailed option allows you to move files that could not be processed succesfully to another location such as a error folder of your choice. For example to move the files in an error folder with a timestamp you can use moveFailed=/error/${file:name.noext }-${date:now:yyyyMMddHHmmssSSS}.${file:ext }.
See more examples at File Language
The following headers are supported by this component:
This component implements the Batch Consumer .
As the file consumer is BatchConsumer it supports batching the files it polls. By batching it means that Camel will add some properties to the Exchange so you know the number of files polled the current index in that order.
Property Description CamelBatchSize The total number of files that was polled in this batch. CamelBatchIndex The current index of the batch. Starts from 0. CamelBatchComplete A boolean value indicating the last Exchange in the batch. Is only true for the last entry.This allows you for instance to know how many files exists in this batch and for instance let the Aggregator aggregate this number of files.
When Camel is producing files (writing files) there are a few gotchas affecting how to set a filename of your choice. By default, Camel will use the message ID as the filename, and since the message ID is normally a unique generated ID, you will end up with filenames such as: ID-MACHINENAME-2443-1211718892437-1-0 . If such a filename is not desired, then you must provide a filename in the CamelFileName message header. The constant, Exchange.FILE_NAME , can also be used.
The sample code below produces files using the message ID as the filename:
from("direct:report" ).to("file:target/reports" );To use report.txt as the filename you have to do:
from("direct:report" ).setHeader(Exchange.FILE_NAME, constant("report.txt" )).to( "file:target/reports" );... the same as above, but with CamelFileName :
from("direct:report" ).setHeader("CamelFileName" , constant("report.txt" )).to( "file:target/reports" );And a syntax where we set the filename on the endpoint with the fileName URI option.
from("direct:report" ).to("file:target/reports/?fileName=report.txt" );Filename can be set either using the expression option or as a string-based File Language expression in the CamelFileName header. See the File Language for syntax and samples.
Beware if you consume files from a folder where other applications write files directly. Take a look at the different readLock options to see what suits your use cases. The best approach is however to write to another folder and after the write move the file in the drop folder. However if you write files directly to the drop folder then the option changed could better detect whether a file is currently being written/copied as it uses a file changed algorithm to see whether the file size / modification changes over a period of time. The other read lock options rely on Java File API that sadly is not always very good at detecting this. You may also want to look at the doneFileName option, which uses a marker file (done) to signal when a file is done and ready to be consumed.
Available as of Camel 2.6
See also section writing done files below.
If you want only to consume files when a done file exist, then you can use the doneFileName option on the endpoint.
from("file:bar?doneFileName=done" );Will only consume files from the bar folder, if a file name done exists in the same directory as the target files. Camel will automatic delete the done file when it's done consuming the files.
However its more common to have one done file per target file. This means there is a 1:1 correlation. To do this you must use dynamic placeholders in the doneFileName option. Currently Camel supports the following two dynamic tokens: file:name and file:name.noext which must be enclosed in ${ }. The consumer only supports the static part of the done file name as either prefix or suffix (not both).
from("file:bar?doneFileName=${file:name}.done" );In this example only files will be polled if there exists a done file with the name file name .done. For example
hello.txt - is the file to be consumedhello.txt.done - is the associated done fileYou can also use a prefix for the done file, such as:
from("file:bar?doneFileName=ready-${file:name}" ); hello.txt - is the file to be consumedready-hello.txt - is the associated done fileAvailable as of Camel 2.6
After you have written af file you may want to write an additional done file as a kinda of marker, to indicate to others that the file is finished and has been written. To do that you can use the doneFileName option on the file producer endpoint.
.to("file:bar?doneFileName=done" );Will simply create a file named done in the same directory as the target file.
However its more common to have one done file per target file. This means there is a 1:1 correlation. To do this you must use dynamic placeholders in the doneFileName option. Currently Camel supports the following two dynamic tokens: file:name and file:name.noext which must be enclosed in ${ }.
.to("file:bar?doneFileName=done-${file:name}" );Will for example create a file named done-foo.txt if the target file was foo.txt in the same directory as the target file.
.to("file:bar?doneFileName=${file:name}.done" );Will for example create a file named foo.txt.done if the target file was foo.txt in the same directory as the target file.
.to("file:bar?doneFileName=${file:name.noext}.done" );Will for example create a file named foo.done if the target file was foo.txt in the same directory as the target file.
Listen on a directory and create a message for each file dropped there. Copy the contents to the outputdir and delete the file in the inputdir .
Listen on a directory and create a message for each file dropped there. Copy the contents to the outputdir and delete the file in the inputdir . Will scan recursively into sub-directories. Will lay out the files in the same directory structure in the outputdir as the inputdir , including any sub-directories.
inputdir/foo.txt inputdir/sub/bar.txtWill result in the following output layout:
outputdir/foo.txt outputdir/sub/bar.txtIf you want to store the files in the outputdir directory in the same directory, disregarding the source directory layout (e.g. to flatten out the path), you just add the flatten=true option on the file producer side:
from("file://inputdir/?recursive=true &delete=true " ).to("file://outputdir?flatten=true " )Will result in the following output layout:
outputdir/foo.txt outputdir/bar.txtCamel will by default move any processed file into a .camel subdirectory in the directory the file was consumed from.
from("file://inputdir/?recursive=true &delete=true " ).to("file://outputdir" )Affects the layout as follows: before
inputdir/foo.txt inputdir/sub/bar.txtafter
inputdir/.camel/foo.txt inputdir/sub/.camel/bar.txt outputdir/foo.txt outputdir/sub/bar.txtThe body will be a File object that points to the file that was just dropped into the inputdir directory.
By default the file endpoint sends a FileMessage which contains a File object as the body. If you send this directly to the JMS component the JMS message will only contain the File object but not the content. By converting the File to a String , the message will contain the file contents what is probably what you want.
The route above using Spring DSL:
<route> <from uri="file://inputdir/" /> <convertBodyTo type="java.lang.String" /> <to uri="jms:test.queue" /> </route>Camel is of course also able to write files, i.e. produce files. In the sample below we receive some reports on the SEDA queue that we processes before they are written to a directory.
public void testToFile() throws Exception { MockEndpoint mock = getMockEndpoint("mock:result" ); mock.expectedMessageCount(1); mock.expectedFileExists("target/test-reports/report.txt" ); template.sendBody("direct:reports" , "This is a great report" ); assertMockEndpointsSatisfied(); } protected JndiRegistry createRegistry() throws Exception { // bind our processor in the registry with the given id JndiRegistry reg = super .createRegistry(); reg.bind("processReport" , new ProcessReport()); return reg; } protected RouteBuilder createRouteBuilder() throws Exception { return new RouteBuilder() { public void configure() throws Exception { // the reports from the seda queue is processed by our processor // before they are written to files in the target/reports directory from("direct:reports" ).processRef("processReport" ).to("file://target/test-reports" , "mock:result" ); } }; } private class ProcessReport implements Processor { public void process(Exchange exchange) throws Exception { String body = exchange.getIn().getBody(String .class); // do some business logic here // set the output to the file exchange.getOut().setBody(body); // set the output filename using java code logic, notice that this is done by setting // a special header property of the out exchange exchange.getOut().setHeader(Exchange.FILE_NAME, "report.txt" ); } }Using a single route, it is possible to write a file to any number of subdirectories. If you have a route setup as such:
<route> <from uri="bean:myBean" /> <to uri="file:/rootDirectory" /> </route>You can have myBean set the header Exchange.FILE_NAME to values such as:
Exchange.FILE_NAME = hello.txt => /rootDirectory/hello.txt Exchange.FILE_NAME = foo/bye.txt => /rootDirectory/foo/bye.txtThis allows you to have a single route to write files to multiple destinations.
In this sample we want to move consumed files to a backup folder using today's date as a sub-folder name:
from("file://inbox?move=backup/${date:now:yyyyMMdd}/${file:name}" ).to("..." );See File Language for more samples.
Camel supports Idempotent Consumer directly within the component so it will skip already processed files. This feature can be enabled by setting the idempotent=true option.
from("file://inbox?idempotent=true " ).to("..." );By default Camel uses a in memory based store for keeping track of consumed files, it uses a least recently used cache storing holding up to 1000 entries. You can plugin your own implementation of this store by using the idempotentRepository option using the # sign in the value to indicate it's a referring to a bean in the Registry with the specified id .
<!-- define our store as a plain spring bean --> <bean id="myStore" class="com.mycompany.MyIdempotentStore" /> <route> <from uri="file://inbox?idempotent=true&idempotentRepository=#myStore" /> <to uri="bean:processInbox" /> </route>Camel will log at DEBUG level if it skips a file because it has been consumed before:
DEBUG FileConsumer is idempotent and the file has been consumed before. Will skip this file: target/idempotent/report.txtIn this section we will use the file based idempotent repository org.apache.camel.processor.idempotent.FileIdempotentRepository instead of the in-memory based that is used as default. This repository uses a 1st level cache to avoid reading the file repository. It will only use the file repository to store the content of the 1st level cache. Thereby the repository can survive server restarts. It will load the content of the file into the 1st level cache upon startup. The file structure is very simple as it store the key in separate lines in the file. By default, the file store has a size limit of 1mb when the file grew larger Camel will truncate the file store be rebuilding the content by flushing the 1st level cache in a fresh empty file.
We configure our repository using Spring XML creating our file idempotent repository and define our file consumer to use our repository with the idempotentRepository using # sign to indicate Registry lookup:
<!-- this is our file based idempotent store configured to use the .filestore.dat as file --> <bean id="fileStore" class="org.apache.camel.processor.idempotent.FileIdempotentRepository" > <!-- the filename for the store --> <property name="fileStore" value="target/fileidempotent/.filestore.dat" /> <!-- the max filesize in bytes for the file. Camel will trunk and flush the cache if the file gets bigger --> <property name="maxFileStoreSize" value="512000" /> <!-- the number of elements in our store --> <property name="cacheSize" value="250" /> </bean> <camelContext xmlns="http://camel.apache.org/schema/spring" > <route> <from uri="file://target/fileidempotent/?idempotent=true&idempotentRepository=#fileStore&move=done/${file:name}" /> <to uri="mock:result" /> </route> </camelContext>In this section we will use the JPA based idempotent repository instead of the in-memory based that is used as default.
First we need a persistence-unit in META-INF/persistence.xml where we need to use the class org.apache.camel.processor.idempotent.jpa.MessageProcessed as model.
<persistence-unit name="idempotentDb" transaction-type="RESOURCE_LOCAL" > <class> org.apache.camel.processor.idempotent.jpa.MessageProcessed</class> <properties> <property name="openjpa.ConnectionURL" value="jdbc:derby:target/idempotentTest;create=true" /> <property name="openjpa.ConnectionDriverName" value="org.apache.derby.jdbc.EmbeddedDriver" /> <property name="openjpa.jdbc.SynchronizeMappings" value="buildSchema" /> <property name="openjpa.Log" value="DefaultLevel=WARN, Tool=INFO" /> </properties> </persistence-unit>Then we need to setup a Spring jpaTemplate in the spring XML file:
<!-- this is standard spring JPA configuration --> <bean id="jpaTemplate" class="org.springframework.orm.jpa.JpaTemplate" > <property name="entityManagerFactory" ref="entityManagerFactory" /> </bean> <bean id="entityManagerFactory" class="org.springframework.orm.jpa.LocalEntityManagerFactoryBean" > <!-- we use idempotentDB as the persitence unit name defined in the persistence.xml file --> <property name="persistenceUnitName" value="idempotentDb" /> </bean>And finally we can create our JPA idempotent repository in the spring XML file as well:
<!-- we define our jpa based idempotent repository we want to use in the file consumer --> <bean id="jpaStore" class="org.apache.camel.processor.idempotent.jpa.JpaMessageIdRepository" > <!-- Here we refer to the spring jpaTemplate --> <constructor-arg index="0" ref="jpaTemplate" /> <!-- This 2nd parameter is the name (= a cateogry name). You can have different repositories with different names --> <constructor-arg index="1" value="FileConsumer" /> </bean>And yes then we just need to refer to the jpaStore bean in the file consumer endpoint using the [[idempotentRepository}} using the # syntax option:
<route> <from uri="file://inbox?idempotent=true&idempotentRepository=#jpaStore" /> <to uri="bean:processInbox" /> </route>Camel supports pluggable filtering strategies. You can then configure the endpoint with such a filter to skip certain files being processed.
In the sample we have build our own filter that skips files starting with skip in the filename:
public class MyFileFilter implements GenericFileFilter { public boolean accept(GenericFile pathname) { // we dont accept any files starting with skip in the name return !pathname.getFileName().startsWith("skip" ); } }And then we can configure our route using the filter attribute to reference our filter (using # notation) that we have defines in the spring XML file:
<!-- define our sorter as a plain spring bean --> <bean id="myFilter" class="com.mycompany.MyFileSorter" /> <route> <from uri="file://inbox?filter=#myFilter" /> <to uri="bean:processInbox" /> </route>The ANT path matcher is shipped out-of-the-box in the camel-spring jar. So you need to depend on camel-spring if you are using Maven. The reasons is that we leverage Spring's AntPathMatcher to do the actual matching.
The file paths is matched with the following rules:
? matches one character* matches zero or more characters** matches zero or more directories in a pathThe sample below demonstrates how to use it:
<camelContext xmlns="http://camel.apache.org/schema/spring" > <template id="camelTemplate" /> <!-- use myFilter as filter to allow setting ANT paths for which files to scan for --> <endpoint id="myFileEndpoint" uri="file://target/antpathmatcher?recursive=true&filter=#myAntFilter" /> <route> <from ref="myFileEndpoint" /> <to uri="mock:result" /> </route> </camelContext> <!-- we use the antpath file filter to use ant paths for includes and exlucde --> <bean id="myAntFilter" class="org.apache.camel.component.file.AntPathMatcherGenericFileFilter" > <!-- include and file in the subfolder that has day in the name --> <property name="includes" value="**/subfolder/**/*day*" /> <!-- exclude all files with bad in name or .xml files. Use comma to seperate multiple excludes --> <property name="excludes" value="**/*bad*,**/*.xml" /> </bean>Camel supports pluggable sorting strategies. This strategy it to use the build in java.util.Comparator in Java. You can then configure the endpoint with such a comparator and have Camel sort the files before being processed.
In the sample we have built our own comparator that just sorts by file name:
public class MyFileSorter implements Comparator<GenericFile> { public int compare(GenericFile o1, GenericFile o2) { return o1.getFileName().compareToIgnoreCase(o2.getFileName()); } }And then we can configure our route using the sorter option to reference to our sorter (mySorter ) we have defined in the spring XML file:
<!-- define our sorter as a plain spring bean --> <bean id="mySorter" class="com.mycompany.MyFileSorter" /> <route> <from uri="file://inbox?sorter=#mySorter" /> <to uri="bean:processInbox" /> </route> URI options can reference beans using the # syntax In the Spring DSL route about notice that we can refer to beans in the Registry by prefixing the id with # . So writing sorter=#mySorter , will instruct Camel to go look in the Registry for a bean with the ID, mySorter .Camel supports pluggable sorting strategies. This strategy it to use the File Language to configure the sorting. The sortBy option is configured as follows:
sortBy=group 1;group 2;group 3;...Where each group is separated with semi colon. In the simple situations you just use one group, so a simple example could be:
sortBy=file:nameThis will sort by file name, you can reverse the order by prefixing reverse: to the group, so the sorting is now Z..A:
sortBy=reverse:file:nameAs we have the full power of File Language we can use some of the other parameters, so if we want to sort by file size we do:
sortBy=file:lengthYou can configure to ignore the case, using ignoreCase: for string comparison, so if you want to use file name sorting but to ignore the case then we do:
sortBy=ignoreCase:file:nameYou can combine ignore case and reverse, however reverse must be specified first:
sortBy=reverse:ignoreCase:file:nameIn the sample below we want to sort by last modified file, so we do:
sortBy=file:modifedAnd then we want to group by name as a 2nd option so files with same modifcation is sorted by name:
sortBy=file:modifed;file:nameNow there is an issue here, can you spot it? Well the modified timestamp of the file is too fine as it will be in milliseconds, but what if we want to sort by date only and then subgroup by name? Well as we have the true power of File Language we can use the its date command that supports patterns. So this can be solved as:
sortBy=date:file:yyyyMMdd;file:nameYeah, that is pretty powerful, oh by the way you can also use reverse per group, so we could reverse the file names:
sortBy=date:file:yyyyMMdd;reverse:file:nameThe option processStrategy can be used to use a custom GenericFileProcessStrategy that allows you to implement your own begin , commit and rollback logic. For instance lets assume a system writes a file in a folder you should consume. But you should not start consuming the file before another ready file have been written as well.
So by implementing our own GenericFileProcessStrategy we can implement this as:
In the begin() method we can test whether the special ready file exists. The begin method returns a boolean to indicate if we can consume the file or not.in the commit() method we can move the actual file and also delete the ready file.This component has log level TRACE that can be helpful if you have problems.
