What is EIP (Enterprise Integration Patterns)
A pattern language teaches the reader how to solve a limitless variety of problems within a bounded problem space. Because the overall problem that is being solved is different every time, the path through the patterns and how they’re applied is also unique.
Enterprise integration goes beyond creating a single application with a distributed n-tier architecture, which enables a single application to be distributed across several computers. Whereas one tier in a distributed application cannot run by itself, integrated applications are independent programs that can each run by itself, yet that function by coordinating with each other in a loosely coupled way.
Over time, we used to attack the integration problems with FOUR major modes. These patterns were evolved as the technology improves.
- File Transfer (Mainframe era)
- Shared Database (RDBMS era)
- Remote Procedure Invocation (Distributed era)
- Message Oriented Middle ware (Highly distributed heterogeneous era)
The models are depicted in the below diagram
Why EAI (Enterprise Application Integration)
In my experience, on taking any integration project, it ends up to have multiple applications talking over multiple transports on multiple platforms. As we can imagine, in large enterprise applications this can get complex very fast. Much of the complexity stems from THREE issues:
- dealing with the specifics of applications and transports
- coming up with good solutions to integration problems
- adopting to the emerging technology trends
It is represented in the below diagram.
Enterprise application integration is necessary in almost every company due to new products and applications. Integrating these applications creates several problems. New paradigms come up every decade, for example client / server communication, Service-oriented Architecture (SOA) or Cloud Computing.
Instead of storing data in files in the past (many years ago), SQL databases are used often today. Sometimes, even NoSQL databases are required in some usecases. Synchronous remote procedure calls or asynchronous messaging is used to communicate via several technologies such as RMI, SOAP Web Services, REST or JMS. A lot of software silos exists. Nevertheless, all applications and products of these decades have to communicate with each other to work together perfectly.
Apache Camel
Apache Camel is an open source Java framework that focuses on making integration easier and more accessible to developers. It does this by providing:
- concrete implementations of all the widely used EIPs
- connectivity to a great variety of transports and APIs
- easy to use Domain Specific Languages (DSLs) to wire EIPs and transports together
It is the well known open source EAI framework, with the coverage of
Architecture
Logical
Apache Camel architecture comprises of SEVEN listed building blocks as shown below. Camel Context is the heart of this framework; which is the run time platform.
These building blocks are used to orchestrate the core functions.
- Registry : It allows you to look up beans; by default, JNDI registry
- Data formats : Loaded data formats like JSON, RSS, CSV, SOAP, etc.
- Endpoints : Created endpoints like TCP, HTTP, etc.
- Routes : A chain of processors; Core abstraction for Camel
- Languages : Expression languages used to define Domain Specific Languages
- Type Converter : Data format conversion component
- Components : Loading components on the fly
Physical
As CamelContext is the heart of the system, it includes all statements like connectors, transformations, routing, etc. It is a simple bean that is not necessarily unique in the Spring context.
Endpoints realize interfacing with applications, which are described by a URI that information about the protocol used and the address of the component. It can be declared implicitly (like <from uri="uri =" file:="" in="" "=""> ) but in the current version of Camel only the <endpoint> allows outsourcing constants in properties files using PropertyPlaceHolderConfigurer. Each URI may contain additional parameters to tune the behavior like jms: queue: testqueue jmsMessageType = Text; ReceiveTimeout = 5000
As shown in the above diagram, Routes are the communication channels (channels) between two or more endpoints. Each has a single entry point to one or more outputs. It is possible to insert components performing treatments as the Processors; but without necessarily implementing a particular interface, the framework performs introspection and is capable of injecting the body of messages.
Each message consists of a header (HashMap properties) and a body (any serializable Java object) The messages that pass are encapsulated in an object Exchange as shown in the figure below.
Vertical scalability
Scalability is the key parameter in terms of performance and business extension. In terms of vertical scalability, a machine can be set to increase CPU, to add memory (RAM) or to the JVM sizer but in neither case do we configure the framework.
It is still possible to act on its speed component, especially at the pollers. You can redefine the number of threads listening or decrease the latency between two pollings, where settings are configurable in writing URI endpoints.
Horizontal scalability
In terms of horizontal scaling, Runtime Camel can be redundant in more machines or more containers but they are compartmentalized, they do not know and in this sense it is not true clustering. If two runtimes Camel point to the same endPoint for the same application, they are likely to consume twice incorrectly the same message. Apache Camel configuration should be appropriate.
Implementation - Pre Set-up
Project Type
To kick start the implementation of Apache Camel model, the developer needs to create a new Maven project as shown below. In this example, I'm taking a new Maven project with group id as org.codeproject and artifact id as camelsample.
pom.xml
POM stands for Project Object Model, which is fundamental XML file unit in Maven. On executing a task or goal, Maven extracts the needed configuration information either from local or given path to execute the goal. This way, Apache Camel relevant libraries are pulled from the common repository.
Our POM needs to have few fundamental Camel libraries as listed above.
Implementation - File Mode
In the first sample application, Apache Camel is going to deal with File to File processing.
Layout
The problem statement is to scan all the files in the specific folder (c://temp/in), process them using Apache Camel framework, deliver the file content to the output folder (c://temp/out)
Camel Context is the runtime platform for Apache Camel to execute. With DefaultCamelContext, Route is built using RouteBuilder factory. Within the defined Route, the message is embedded to process with the defined custom processor.
Source Code
Camel supports the definition of routing rules using a Java DSL (domain specific language) which avoids the need for cumbersome XML using a RouteBuilder.
The RouteBuilder is a base class which is derived from to create routing rules using the DSL. Instances of RouteBuilder are then added to the CamelContext.
Camel uses a simple lifecycle interface called Service which has a single start() and stop() method. Various classes implement Service such as CamelContext along with a number of Component and Endpoint classes. When you use Camel you typically have to start the CamelContext which will start all the various components and endpoints and activate the routing rules until the context is stopped again.
If noop parameter of File component is set true, then file is not moved or deleted in any way. Here, Camel will set idempotent=true as well, to avoid consuming the same files over and over again. It may be a good fit for ETL type requirements.
private static void camelCopy() throws Exception {
CamelContext context = new DefaultCamelContext();
context.addRoutes(new RouteBuilder() {
public void configure() {
from("file:c://temp/in?noop=true")
.process(new Processor() {
public void process(Exchange exchange) throws Exception {
System.out.println("Within Interim Processor ");
System.out.println("We just processed the file named: "
+ exchange.getIn().getHeader("CamelFileName"));
}
})
.to("file:c://temp/out");
}
});
System.out.println("Context starts here..");
context.start();
Thread.sleep(10000);
context.stop();
System.out.println("Context ends here..");
}
Processor Code
The Processor interface is used to implement consumers of message exchanges or to implement a Message Translator. The process can be used in routes as an anonymous inner class as below:
public void process(Exchange exchange) throws Exception {
System.out.println("Within Processor bean");
System.out.println("We just processed the file named: "
+ exchange.getIn().getHeader("CamelFileName"));
}
This inner pattern is usable for quickly whirling up some code. If the code in the inner class gets a bit more complicated it is of course advised to refactor it into a separate class.
Project Structure
Maven project structure for File based Apache Camel processing, is attached here.
On execution, the given problem statement is run successfully. In the source folder, there are 2 files; txt and xml file each. They are processed by Apache Camel; then delivery to the output folder. The result is reflected in Console window.
Config Driven Process
Spring is the most popular Inversion of Control (IoC) Java container out there. The core framework allows to you “wire” beans together to form applications. This wiring is done through an XML configuration file.
private static void camelConfigCopy() throws Exception {
System.out.println("Context starts here..");
new ClassPathXmlApplicationContext("/FileCopy.xml");
Thread.sleep(10000);
System.out.println("Context ends here..");
}
There are a few different approaches to configuring components and endpoints. When defining routes in Camel using Xml Configuration you may want to define some route. When you use routeContext tag then they are separated, and cannot reuse existing tags onException, intercept, dataFormats and similar cross cutting functionality defined in the camelContext. In other words the routeContext tag is currently isolated.
The routes defined in routeContext can be reused by multiple camelContext. However its only the definition which is reused. At runtime each CamelContext will create its own instance of the route based on the definition.
Implementation - Message Mode
Messages are the entities used by systems to communicate with each other when using messaging channels. Messages flow in one direction from a sender to a receiver. The second sample is driven by the message based processing.
Layout
The second problem statement is to scan all the files in the given folder, packet them into JMS message, routing based on the received content, process the data to complete. The model is shown in the below diagram.
The objective is to show case the message based processing using Apache Camel framework.
Content-Based Router (CBR) is a message router that routes a message to a destination based on its content. The content could be a message header, the payload data type, part of the payload itself—pretty much anything in the message exchange.
In the given example, IncomingOrder message is injected into CBR processing to route via three channels namely xml, csv, bad.
Source Code
To implement the second problem statement, the source code is written with the same methodology of CamelContext, RouteBuilder, Processor, etc.
Here we use the filename extension to determine whether a particular order message should be sent to a queue for CSV orders or a queue for XML orders. This is depicted in the below source code.
private static void FileToMsg() throws Exception {
CamelContext context = new DefaultCamelContext();
ConnectionFactory connectionFactory = new ActiveMQConnectionFactory("vm://localhost");
context.addComponent("jms", JmsComponent.jmsComponentAutoAcknowledge(connectionFactory));
context.addRoutes(new RouteBuilder() {
@Override
public void configure() {
from("file:c://temp/in?noop=true").to("jms:incomingOrders");
from("jms:incomingOrders")
.choice()
.when(header("CamelFileName").endsWith(".xml"))
.to("jms:xmlOrders")
.when(header("CamelFileName").regex("^.*(csv|csl)$"))
.to("jms:csvOrders")
.otherwise()
.to("jms:badOrders").stop()
.end()
.to("jms:continuedProcessing");
from("jms:xmlOrders").process(new Processor() {
public void process(Exchange exchange) throws Exception {
System.out.println("Received XML order: "
+ exchange.getIn().getHeader("CamelFileName"));
}
});
from("jms:csvOrders").process(new Processor() {
public void process(Exchange exchange) throws Exception {
System.out.println("Received CSV order: "
+ exchange.getIn().getHeader("CamelFileName"));
}
});
from("jms:badOrders").process(new Processor() {
public void process(Exchange exchange) throws Exception {
System.out.println("Received bad order: "
+ exchange.getIn().getHeader("CamelFileName"));
}
});
from("jms:continuedProcessing").process(new Processor() {
public void process(Exchange exchange) throws Exception {
System.out.println("Received continued order: "
+ exchange.getIn().getHeader("CamelFileName"));
}
});
}
});
System.out.println("Content Based Route starts here..");
context.start();
Thread.sleep(10000);
System.out.println("Content Based Route ends here..");
context.stop();
}
In the above code, CamelFileName header is set by the FTP consumer to get the filename. To do the conditional routing required by the CBR, Camel introduces a few keywords in the DSL. The choice method creates a CBR processor, and conditions are added by following choice with a combination of a when method and a predicate.
In this sample, Camel’s creators have chosen contentBasedRouter for the method name, to match the EIP, but they stuck with choice because it reads more naturally.
Process Flow
The Process Flow chart provides a visual representation of the steps in a process.
In our message sample program, Runtime layer CamelContext and Message layer JMS/ActiveMQ, are instantiated as the first step. Then, Route and Config are overridden for our use case.
Contend based routing is processed for the incoming file info using JMS message. Content is split either by XML or CSV file types. After executing the relevant processor, the program is directed to proceed towards ContinuedProcessor.
Remaining file types are routed to BAD category and stop the process.
On execution of the given sample program with two files (firstmeet.txt & testxml2.xml) @in folder, the result is shown as above. As per CBR logic, txt file is routed as BAD category and the second file is processed as XML category based on the given file type.
Points of Interest
Apache Camel supports all requirements to be used in enterprise integration projects, such as error handing, transactions, scalability, and monitoring. It should always be evaluated as lightweight alternative to heavyweight ESBs.
This hands-on article helps you to understand the core concepts of EAI, Camel and implementation steps.
History
Initial Version
Ganesan Senthilvel is a passionate IT leader with demonstrated 2 decades’ experience in architecture, design and implementing cutting edge solutions to address business opportunities of Enterprise applications. He earned Masters in Computer Science and Master in Business Administration. Now, he is pursing Doctorate program in Big Data. He is consistent technical contributor via COE, Blog, Whitepaper, Summit, Certification, etc. and also highly skilled at providing coaching and mentoring to internal teams and external institutes. Earned Microsoft Certifications like MCP, MCAD and financial domain certification.
He maintains his weekly blog at http://ganesansenthilvel.blogspot.in
Submit an article or tip
