Introduction

If there is one universal truth about implementing parallelism using Java threads, it is that you will need to pay special attention when writing your code. Writing custom OEP adapters that spawn multiple threads is no different. If you have a strong requirement for handling high-throughput events, you have two choices:

1. Scaling out horizontally using multiple JVMs in cluster.

2. Scaling up a single JVM that spreads the adapter logic into multiple threads.

While scaling out is a good choice, sometimes you just can’t use this approach; or maybe you can but are constrained by lack of resources. When that happens, you need to leverage the scale up approach, especially if the machine where the OEP application will run has enough CPU cores and memory.

The idea behind scaling up using a single JVM is simple: you just need to create two or more threads and equally partition the amount of work between them, so they can process work in parallel. Let’s understand how this is possible considering the built-in JMS adapter available in OEP. The OEP JMS adapter has a property called concurrent-consumers. When you set this property to any positive integer greater than one, you are instructing the adapter to create parallel consumers, each one running on its own thread, which will increase message consumption and also application throughput.

Now consider the scenario where you need your own custom adapter, and you desire to implement parallelism. Implementing multi-threaded code in Java is fairly straightforward once you understand what is happening in the JVM. However, the truth is that even the most well written code in the world, has a good chance of blowing up and doing unexpected things when executing in the OEP runtime due to poor life cycle management of threads. This article will explain some of the possible issues that can occur when you implement threads on your own and, most importantly, how you can leverage the work manager support available in OEP to help ensure that you achieve the scalability and resiliency that you require.

This article will assume that you have some basic understanding about how to create a custom adapter in OEP. If you need information about how to create custom adapters before continue reading this article, I strongly recommend reviewing the product documentation section that covers this topic. Another great source of information is the book Getting Started with Oracle Event Processing 11g, written by some of the folks behind the OEP product at Oracle.

Testing a Simple OEP Application

Consider the following scenario: an event-driven application written in OEP that uses a custom inbound adapter, generates a random amount of events every five seconds. Those events will flow through simple pass-through channels and be queried out by a processor with the following CQL statement: SELECT * FROM inboundChannel [NOW]. Finally, those events will be printed out in the console by a custom outbound adapter. The picture below show the EPN of this application.

Aiming to generate events in parallel, the first version of the custom inbound adapter was written to perform its work using regular Java threads. The listing below shows the custom adapter implementation:

package com.oracle.fmw.ateam.soa;

import java.util.Random;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;

import com.bea.wlevs.ede.api.RunnableBean;
import com.bea.wlevs.ede.api.StreamSender;
import com.bea.wlevs.ede.api.StreamSource;

public class CustomInboundAdapter implements RunnableBean, StreamSource {

	private static final int NUMBER_OF_THREADS = 4;

	private StreamSender streamSender;
	private ExecutorService executorService;
	private boolean suspended;

	@Override
	public void run() {

		executorService = Executors.newFixedThreadPool(NUMBER_OF_THREADS);

		for (int i = 0; i < NUMBER_OF_THREADS; i++) {

			RegularJavaThread thread = new RegularJavaThread();
			thread.setName(RegularJavaThread.class.getSimpleName() + "-" + i);

			executorService.execute(thread);

		}

	}

	@Override
	public synchronized void suspend() throws Exception {

		executorService.shutdown();
		this.suspended = true;

	}

	@Override
	public void setEventSender(StreamSender streamSender) {
		this.streamSender = streamSender;
	}

	private class RegularJavaThread extends Thread {

		@Override
		public void run() {

			final Random random = new Random(System.currentTimeMillis());
			int count = 0;

			try {

				while (!suspended) {

					count = random.nextInt(10);

					for (int i = 0; i < count; i++) {

						streamSender.sendInsertEvent(new Tick(getName()));

					}

					Thread.sleep(5000);

				}

			} catch (Exception ex) {

				ex.printStackTrace();

			}

		}

	}

}

As you can see in the listing above, a regular Java thread is created to perform the main logic of the adapter; which is to generate a random number of events every five seconds. Note that the thread is supposed to finish its work when the OEP application changes its status to suspended. This can happen in two ways: when the application is uninstalled from the server or when the user intentionally set its status as suspended through the OEP visualizer.

According to the run() method of the custom inbound adapter, four instances of the thread are scheduled to work using the java.util.concurrent.Executor service. This code runs perfectly well and produces the desired behavior, as you can see in the console output listed below:

<Aug 6, 2014 9:10:01 PM EDT> <Notice> <Deployment> <BEA-2045000> <The application bundle "wm-driven-threads-in-oep" was deployed successfully to file:/oracle/user_projects/domains/oep-development/defaultserver/applications/wm-driven-threads-in-oep/wm-driven-threads-in-oep.jar with version 1407373801915> 
<Aug 6, 2014 9:10:02 PM EDT> <Notice> <Spring> <BEA-2047000> <The application context for "wm-driven-threads-in-oep" was started successfully> 
Tick [uuid=13074bae-554d-49d1-934f-c2cadfadcaec, dateTime=Wed Aug 06 21:10:07 EDT 2014, threadName=RegularJavaThread-0]
Tick [uuid=377a896e-d26b-4a8c-ad6a-585495ce78c2, dateTime=Wed Aug 06 21:10:07 EDT 2014, threadName=RegularJavaThread-3]
Tick [uuid=488ea3c1-0515-471c-ba46-dd07f446437f, dateTime=Wed Aug 06 21:10:07 EDT 2014, threadName=RegularJavaThread-0]
Tick [uuid=b75c5a24-a82d-49d2-87df-49f3900059e1, dateTime=Wed Aug 06 21:10:07 EDT 2014, threadName=RegularJavaThread-3]
Tick [uuid=69df06ca-a57a-494a-b20b-976047537fce, dateTime=Wed Aug 06 21:10:07 EDT 2014, threadName=RegularJavaThread-0]
Tick [uuid=a5855897-eeef-44d1-85d8-480b4745ac20, dateTime=Wed Aug 06 21:10:07 EDT 2014, threadName=RegularJavaThread-3]
Tick [uuid=f71308ef-797a-4457-a5a6-7f91fe10d773, dateTime=Wed Aug 06 21:10:07 EDT 2014, threadName=RegularJavaThread-3]
Tick [uuid=2164390b-8c74-4f4a-942e-7c60a9656c57, dateTime=Wed Aug 06 21:10:07 EDT 2014, threadName=RegularJavaThread-0]
Tick [uuid=f82a2376-ce56-41e9-90a3-119e4e827ff5, dateTime=Wed Aug 06 21:10:07 EDT 2014, threadName=RegularJavaThread-3]
Tick [uuid=660e074e-991f-4082-8210-8960582aa129, dateTime=Wed Aug 06 21:10:07 EDT 2014, threadName=RegularJavaThread-0]
Tick [uuid=32b2a37e-235d-460d-82b2-8537dddc7703, dateTime=Wed Aug 06 21:10:07 EDT 2014, threadName=RegularJavaThread-2]
Tick [uuid=4bea1680-0d1e-4f7a-9ae4-e78a23bb1229, dateTime=Wed Aug 06 21:10:07 EDT 2014, threadName=RegularJavaThread-2]
Tick [uuid=215ef396-2210-40b5-88f1-4e1c86c4b93f, dateTime=Wed Aug 06 21:10:07 EDT 2014, threadName=RegularJavaThread-2]
Tick [uuid=27cfebbd-649c-4f00-ac8d-33948754298b, dateTime=Wed Aug 06 21:10:07 EDT 2014, threadName=RegularJavaThread-2]
Tick [uuid=cffdab2e-daca-4145-8e37-b808a5f4b412, dateTime=Wed Aug 06 21:10:07 EDT 2014, threadName=RegularJavaThread-2]
Tick [uuid=35a8ea83-130e-48ac-bd47-20bed9c20d82, dateTime=Wed Aug 06 21:10:07 EDT 2014, threadName=RegularJavaThread-1]
Tick [uuid=5c61c10a-e61e-42d6-8931-fbf3ee092b6f, dateTime=Wed Aug 06 21:10:07 EDT 2014, threadName=RegularJavaThread-1]
Tick [uuid=e5d070e2-bdb9-4fb9-bc12-7edf0165d258, dateTime=Wed Aug 06 21:10:07 EDT 2014, threadName=RegularJavaThread-1]
Tick [uuid=a3488aba-00ee-4dd0-93fa-29517f69ce7b, dateTime=Wed Aug 06 21:10:07 EDT 2014, threadName=RegularJavaThread-1]
Tick [uuid=1669f735-35a4-4b2f-bad3-02117ef94991, dateTime=Wed Aug 06 21:10:07 EDT 2014, threadName=RegularJavaThread-1]

If the code runs perfectly well, you are probably wondering what’s wrong with the current implementation. The next section will illustrate the problem.

Problem: Server Cannot Control the Allocated Threads

With the OEP application running on the server, watch the threads running on top of the JVM. There are many ways to accomplish this, from taking thread dumps to using more sophisticated tools. I chose to use the JRockit Mission Control which provides a nice view of the active threads:

As expected, there are four threads allocated in the JVM executing the work defined in the code. Now let’s consider for a moment the idea of the developer being able to define the number of threads used as an instance property that could be changed through a configuration change. Just like the concurrent-consumers property of the JMS adapter, it is scenario perfectly amenable to happen. Let’s not forget that the developer can also hard code the number of threads, making it impossible to change the value even at the configuration level.

The development practices explained above can lead to some potential problems. What if the developer defines a number of threads so high that it affects the performance of other OEP applications running in the same server? Or worse, what if the developer defines a number of threads so high that the JVM itself can’t handle comply due to a lack of resources?

To protect the health of the server and prevent those situations from happening, administrators can use work managers. A work manager is a OEP server feature that controls the threading behavior using constraints. Once created in the server, the administrator can associate the work manager with one or multiple OEP applications, but this is something that should be done manually. When you create a domain for the first time, a default work manager called JettyWorkManager is created, and it is primarily used by the Jetty engine: a Java web server used to deploy HTTP servlets and static resources. You can create and/or change work managers in the domain configuration file found in the config folder of your server.

Back to the scenario, let’s assume that a work manager named wm-driven-threads-in-oep is created in the domain configuration file and limits the number of threads to a minimum of one and a maximum of two. The listing below shows how this work manager should be defined.

<?xml version="1.0" encoding="UTF-8"?>
<n1:config xsi:schemaLocation="http://www.bea.com/ns/wlevs/config/server wlevs_server_config.xsd"
xmlns:n1="http://www.bea.com/ns/wlevs/config/server"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">

   <netio>
      <name>NetIO</name>
      <port>9002</port>
   </netio>

   <netio>
      <name>sslNetIo</name>
      <ssl-config-bean-name>sslConfig</ssl-config-bean-name>
      <port>9003</port>
   </netio>

   <work-manager>
      <name>wm-driven-threads-in-oep</name>
      <min-threads-constraint>1</min-threads-constraint>
      <max-threads-constraint>2</max-threads-constraint>
   </work-manager>

   <work-manager>
      <name>JettyWorkManager</name>
      <min-threads-constraint>5</min-threads-constraint>
      <max-threads-constraint>10</max-threads-constraint>
   </work-manager>

   <jetty>
      <name>JettyServer</name>
      <network-io-name>NetIO</network-io-name>
      <work-manager-name>JettyWorkManager</work-manager-name>
      <scratch-directory>Jetty</scratch-directory>
      <secure-network-io-name>sslNetIo</secure-network-io-name>
   </jetty>

   <!-- the rest of the file was dropped for better clarity -->

</n1:config>

Work managers can be associated with one or multiple applications, and also can be associated with one particular component of the application. Having said that, keep in mind that sharing the same work manager across multiple applications can limit throughput since the max-threads-constraint value will be the same for all of them, forcing some applications to queue requests while they keep waiting for available threads in the pool. And that could mean keep waiting for ever. As a best practice, if you run more than one application per server, consider creating one work manager for each application to manage its threads boundaries individually.

Once created in the domain configuration file, the work manager can be further configured using the OEP Visualizer:

Now we are all set. We can easily restart the server and test the OEP application again. With a work manager in place limiting the maximum number of threads to only two, even if the code tries to allocate four threads there will be only two threads running. That’s the theory anyway. Unfortunately, the reality is slightly different. Back to the JRockit Mission Control we still see four threads allocated in the JVM:

So, what is really happening here? Since those threads were created using standard JDK techniques, the server has no control over the number of threads. That’s why even creating a work manager and associating it with the OEP application, the constraints were ignored. Also, the main adapter code runs on its own separate thread managed by the server, as a result of implementing the com.bea.wlevs.ede.api.RunnableBean interface. That means that the spawned threads have no relationship with the adapter thread, forcing them to behave like deamon threads: instead of having its life cycle associated with the application, they will have its life cycle associated with the JVM. The problem can become even worse if the spawned threads hold references to objects belonging to the adapter thread. If for some reason the adapter thread dies, its garbage will be retained in memory since the spawned threads will still have references of it’s objects, causing memory leak problems that could lead to out of memory errors.

With this problem in mind, there is a clearly a need for some technique that would allow threads created from an adapter implementation to: have the chance to clean up its garbage independent of the run() method logic; respect the constraints imposed by an associated work manager and have the flexibility to be marked as deamon or non-deamon depending of the need. The solution for this problem will be explored in the next section.

Solution: Implementing the WorkManagerAware Interface

When designing custom adapters, if you need to spawn threads to perform some work and need server control over those threads, you can use the com.bea.wlevs.ede.spi.WorkManagerAware interface. It is available through the OEP API and all you need is to make sure that your adapter class is implementing this interface. This allows your adapter class to receive a reference of the work manager associated with the application. From this work manager reference, you can request work scheduling for threads in a safe manner. The listing below shows the updated custom adapter implementation.

package com.oracle.fmw.ateam.soa;

import java.util.Random;

import com.bea.wlevs.ede.api.RunnableBean;
import com.bea.wlevs.ede.api.StreamSender;
import com.bea.wlevs.ede.api.StreamSource;
import com.bea.wlevs.ede.spi.WorkManagerAware;
import commonj.work.Work;
import commonj.work.WorkManager;

public class CustomInboundAdapter implements WorkManagerAware, RunnableBean,
		StreamSource {

	private static int NUMBER_OF_THREADS = 4;

	private StreamSender streamSender;
	private WorkManager workManager;
	private boolean suspended;

	@Override
	public void run() {

		String threadName = null;

		try {

			for (int i = 0; i < NUMBER_OF_THREADS; i++) {

				threadName = WorkManagerBasedThread.class.getSimpleName() + "-" + i;

				workManager.schedule(new WorkManagerBasedThread(threadName));

			}

		} catch (Exception ex) {

			ex.printStackTrace();

		}

	}

	@Override
	public synchronized void suspend() throws Exception {
		this.suspended = true;
	}

	@Override
	public void setEventSender(StreamSender streamSender) {
		this.streamSender = streamSender;
	}

	@Override
	public void setWorkManager(WorkManager workManager) {
		this.workManager = workManager;
	}

	private class WorkManagerBasedThread implements Work {

		private String threadName;

		public WorkManagerBasedThread(String threadName) {
			this.threadName = threadName;
		}

		@Override
		public void run() {

			final Random random = new Random(System.currentTimeMillis());
			int count = 0;

			try {

				while (!suspended) {

					count = random.nextInt(10);

					for (int i = 0; i < count; i++) {

						streamSender.sendInsertEvent(new Tick(threadName));

					}

					Thread.sleep(5000);

				}

			} catch (Exception ex) {

				ex.printStackTrace();

			}

		}

		@Override
		public boolean isDaemon() {
			
			// This way you can inform to the
			// OEP runtime that this thread is
			// deamon but without losing the
			// association with the WM...
			
			return false;
			
		}

		@Override
		public void release() {
			
			// Here you can put the logic to clean up
			// any resources allocated by the thread,
			// in a safe and guaranteed manner...
			
		}

	}

}

As you can see in the listing above, no changes were made in the thread logic. The main difference is that the threads are scheduled through the work manager reference. Also, the threads have the isDeamon() and release() callback methods, which can be used to change the way the thread behaves regarding its life cycle and how resources are cleaned up. After installing the new version of the OEP application, it is possible to see that the constraints imposed by the work manager are now being respected:

The console output listed below also shows that there are only two threads now performing the work:

<Aug 7, 2014 1:47:20 PM EDT> <Notice> <Deployment> <BEA-2045000> <The application bundle "wm-driven-threads-in-oep" was deployed successfully to file:/oracle/user_projects/domains/oep-development/defaultserver/applications/wm-driven-threads-in-oep/wm-driven-threads-in-oep.jar with version 1407433640951> 
<Aug 7, 2014 1:47:23 PM EDT> <Notice> <Spring> <BEA-2047000> <The application context for "wm-driven-threads-in-oep" was started successfully> 
Tick [uuid=a33b7f84-2489-4737-9e33-18cabd6cca9f, dateTime=Thu Aug 07 13:47:23 EDT 2014, threadName=WorkManagerBasedThread-0]
Tick [uuid=c84e2f0c-37fa-4dca-a897-4f75fa55a7b9, dateTime=Thu Aug 07 13:47:23 EDT 2014, threadName=WorkManagerBasedThread-0]
Tick [uuid=74023209-4964-4181-8f18-40eef8ef2476, dateTime=Thu Aug 07 13:47:23 EDT 2014, threadName=WorkManagerBasedThread-0]
Tick [uuid=f9ccce52-fee6-49ad-91cc-6d9023d2c967, dateTime=Thu Aug 07 13:47:23 EDT 2014, threadName=WorkManagerBasedThread-0]
Tick [uuid=a065ebf9-1320-4cc0-a9c3-b2e004398931, dateTime=Thu Aug 07 13:47:23 EDT 2014, threadName=WorkManagerBasedThread-0]
Tick [uuid=43e4db5b-72a1-4760-8205-b92a8a784cbb, dateTime=Thu Aug 07 13:47:23 EDT 2014, threadName=WorkManagerBasedThread-1]
Tick [uuid=3bf7cbc3-09b9-4f75-99e4-d223b1a400c3, dateTime=Thu Aug 07 13:47:23 EDT 2014, threadName=WorkManagerBasedThread-1]
Tick [uuid=48e2ab1f-4806-4d3b-8474-10fdb51d346b, dateTime=Thu Aug 07 13:47:23 EDT 2014, threadName=WorkManagerBasedThread-1]
Tick [uuid=413803e2-c5c2-4f49-879e-06c356bca0c0, dateTime=Thu Aug 07 13:47:23 EDT 2014, threadName=WorkManagerBasedThread-1]

You can download the final implementation of the project used in this article here.

All content listed on this page is the property of Oracle Corp. Redistribution not allowed without written permission