At UnboundID, we take performance seriously and are always trying to improve. This applies just as much for our client-side SDK as for our server products, since a server isn’t very useful without client applications to take advantage of it. There are a number of tools that can be used to measure various aspects of directory server performance, but it’s not as simple to measure the performance of client-side libraries.

To help address this problem, I’ve written a simple tool that can be used to perform a number of different kinds of LDAP operations using various Java-based LDAP SDKs. It’s not particularly elaborate and there’s only a command-line interface, but it provides a range of options, including which SDK to use, the type of operation to test, the number of concurrent threads to use, the length of time to run the test (and to warm-up before beginning to collect data), the type of entries to use, the number of entries to return per search, the result code that should be returned for each operation, and how frequently to close and re-establish connections.

It’s obviously the case that, as the primary developer for the UnboundID LDAP SDK for Java, I am more than a little biased about which SDK I prefer. However, to the best of my knowledge the way that the tool performs the tests is as fair as possible and uses the most efficient mechanism offered by each of the libraries. If anyone believes that there is a more efficient way to use any of the SDKs, then I’d be happy to hear about it and update the results accordingly.

At present, the tool provides at least some level of support for the following SDKs:

• Apache LDAP API, version 1.0.0-M3. Although I have written code in the hope of testing this SDK, it does not appear to be fully functional at the present time. For example, when trying to perform searches with multiple threads using a separate connection per thread (which is the only way I have used it to this point), it looks like only a single thread is actually able to perform searches and all the others throw timeout exceptions. If anyone knows how to work around that, I’d be happy to hear about it. Until this problem is resolved, this tool isn’t very useful for testing its performance.

• JNDI, as is included in Java SE. For my testing, I used Java SE 1.6.0_25. JNDI is a very abstract API that has the ability to communicate using a number of protocols, and as such was not designed specifically for LDAP. Unfortunately, this means that it’s not ideal for LDAP in a lot of ways. For example, it doesn’t appear that JNDI provides any way to get the actual numeric LDAP result code returned by the server in response to various operations, and it also looks like it does not support bind (for the purpose of authenticating clients) as a distinct type of operation but only in the course of establishing a connection or re-authenticating before performing some other kind of operation. As such, the performance testing tool does not support bind operations, and it does not support testing with operations that return non-successful responses because the result code cannot be verified.

• Netscape Directory SDK for Java, version 4.17 (compiled from source, as there does not appear to be a download for a pre-built version of the library). This SDK is fully supported by the performance testing tool.

• Novell LDAP Classes for Java, also known as JLDAP, version 2009.10.07-1. This SDK is fully supported by the performance testing tool.

• OpenDJ LDAP SDK, version 3.0.0 (snapshot build from May 28, 2011). This appears to be a fork of the OpenDS LDAP SDK that has had the package structure changed, and may have some number of additional changes as well. However, I was not able to successfully use this SDK to run any tests because the code that I used (despite identical code working for the OpenDS SDK, with the exception of changing the package names in import statements) threw an exception when trying to run, indicating that it was attempting to subclass a class that had previously been declared final. It also appeared to be missing an org.forgerock.i18n.LocalizedIllegalArgumentException class, although I worked around that problem by writing my own version of that class.

• OpenDS LDAP SDK for Java, 0.9.0 build from May 26, 2011. This SDK is fully supported by the performance testing tool. In addition, because the API provides options for both synchronous and asynchronous connections, the “–useSynchronousMode” option is supported to request using the synchronous version of the API that does not support the use of abandon or multiple concurrent operations on the same connection, while omitting this argument will use a version that does support this capability.

• UnboundID LDAP SDK for Java, version 2.2.0. This SDK is fully supported, including the use of the “–useSynchronousMode” option.

These tests obviously require communication with an LDAP directory server. Because the intention is not to measure the performance of the directory server but rather the SDK being used to communicate with that server, it is ideal to use something that is as fast as possible (so that the server is not a bottleneck) and that can be manipulated to give an arbitrary response for any operation. For this purpose, a custom server was created using the LDAP Listener API provided as part of the UnboundID LDAP SDK for Java. It is important to note, however, that even though this API is part of the UnboundID LDAP SDK, it can be used with any kind of client and all interaction with it was over the LDAP protocol using a socket connected over the test system’s loopback interface. The UnboundID LDAP SDK did not have any advantage over any other SDK when interacting with this server.

All of the tests that I ran used Java SE 1.6.0_25 (64-bit version) on a system with a 2.67GHz 8-core Intel Core i7 CPU with 12GB of memory, running a fully-patched version of Ubuntu Linux version 11.04. A detailed description of each of the tests that I ran is provided below, along with the results that I obtained. Each test was run with the JNDI, Netscape, Novell, OpenDS, and UnboundID SDKs, using 1, 2, 4, 8, 16, 32, 64, and 128 client threads. For the OpenDS and UnboundID SDKs, tests were run using both the asynchronous and synchronous modes of operation.

Add Operation Performance

When processing add operations, performance may vary based on the size of the entry being added to the server. As such, I ran two different add performance tests: a “normal-sized” entry (an inetOrgPerson entry with 15 attributes) and a “large” entry (a groupOfUniqueNames entry with 1000 uniqueMember values).

The results I measured when running these tests were:

API	Highest Normal-Sized Entry Add Throughput
JNDI	89,027.954 adds/sec
Netscape	98,916.582 adds/sec
Novell	86,766.964 adds/sec
OpenDS (asynchronous mode)	88,525.069 adds/sec
OpenDS (synchronous mode)	88,586.290 adds/sec
UnboundID (asynchronous mode)	142,105.659 adds/sec
UnboundID (synchronous mode)	174,665.853 adds/sec

 

SDK	Highest Large Entry Add Throughput
JNDI	6,472.209 adds/sec
Netscape	8,723.301 adds/sec
Novell	7,437.703 adds/sec
OpenDS (asynchronous mode)	9,454.340 adds/sec
OpenDS (synchronous mode)	9,747.643 adds/sec
UnboundID (asynchronous mode)	17,602.504 adds/sec
UnboundID (synchronous mode)	18,545.810 adds/sec

 

From these tests, it appears that the UnboundID LDAP SDK for Java is significantly faster than any of the other SDKs when processing add operations, and using the UnboundID LDAP SDK in synchronous mode provides a notable performance improvement over the default asynchronous mode. In contrast, the OpenDS LDAP SDK does not appear to exhibit a significant difference in add performance based on whether the asynchronous or synchronous version of the API is selected.

Search Operation Performance

As for adds, search operation performance can vary significantly based on the size of the entry being returned. As such, I ran tests search tests using the same “normal-sized” and “large” entries as for the add operation testing, and I also tested the case of returning only a single attribute in each entry. Further, because the server can return multiple entries for a single search operation, I ran tests with both operations returning a single entry and 100 identical entries. Results from those tests are provided below:

SDK	Highest Search Throughput for 1 Tiny Entry
JNDI	54,272.423 searches/sec
Netscape	74,601.755 searches/sec
Novell	69,686.323 searches/sec
OpenDS (asynchronous mode)	73,964.204 searches/sec
OpenDS (synchronous mode)	74,572.779 searches/sec
UnboundID (asynchronous mode)	109,159.192 searches/sec
UnboundID (synchronous mode)	168,315.209 searches/sec

 

SDK	Highest Search Throughput for 1 Normal-Sized Entry
JNDI	46,355.770 searches/sec
Netscape	49,668.681 searches/sec
Novell	55,988.055 searches/sec
OpenDS (asynchronous mode)	55,408.763 searches/sec
OpenDS (synchronous mode)	54,923.308 searches/sec
UnboundID (asynchronous mode)	83,846.853 searches/sec
UnboundID (synchronous mode)	115,738.348 searches/sec

 

SDK	Highest Search Throughput for 1 Large Entry
JNDI	11,045.600 searches/sec
Netscape	3,849.413 searches/sec
Novell	748.249 searches/sec
OpenDS (asynchronous mode)	10,449.903 searches/sec
OpenDS (synchronous mode)	10,374.687 searches/sec
UnboundID (asynchronous mode)	20,645.026 searches/sec
UnboundID (synchronous mode)	21,341.607 searches/sec

 

SDK	Highest Search Throughput for 100 Tiny Entries
JNDI	5,749.687 searches/sec
Netscape	2,768.797 searches/sec
Novell	2,739.363 searches/sec
OpenDS (asynchronous mode)	8,295.155 searches/sec
OpenDS (synchronous mode)	8,315.379 searches/sec
UnboundID (asynchronous mode)	5,566.711 searches/sec
UnboundID (synchronous mode)	7,265.108 searches/sec

 

SDK	Highest Search Throughput for 100 Normal-Sized Entries
JNDI	1,983.319 searches/sec
Netscape	875.249 searches/sec
Novell	1,681.767 searches/sec
OpenDS (asynchronous mode)	1,959.581 searches/sec
OpenDS (synchronous mode)	1,917.131 searches/sec
UnboundID (asynchronous mode)	2,308.414 searches/sec
UnboundID (synchronous mode)	3,278.463 searches/sec

 

SDK	Highest Search Throughput for 100 Large Entries
JNDI	127.716 searches/sec
Netscape	39.667 searches/sec
Novell	6.731 searches/sec
OpenDS (asynchronous mode)	117.633 searches/sec
OpenDS (synchronous mode)	117.233 searches/sec
UnboundID (asynchronous mode)	225.800 searches/sec
UnboundID (synchronous mode)	237.400 searches/sec

 

In this case, there is a significant variation in many of the SDKs based on the size and number of entries being returned. The UnboundID LDAP SDK is significantly faster than the other SDKs in most cases (with a notable improvement on top of that when using synchronous mode), but the OpenDS SDK is quite a bit faster than the UnboundID LDAP SDK in the case of a search returning 100 entries with only a single attribute per entry, but that is not the case for normal-sized or large entries. On the other hand, both the Netscape and Novell SDKs appear to be extremely slow when dealing with large search result entries, and the Netscape SDK is also much slower than the others for large entry sets. It is also important to note the significant drop in search performance when using JNDI for larger numbers of threads when returning a single tiny or normal-sized entry.

Modify Operation Performance

For a client SDK, modify performance has fewer variables than for either add or search operations. For a directory server, there are a number of factors, including the size of the target entry, the size of the modified attributes, and whether any of the target attributes is indexed, but none of these has an impact on the client. It is certainly the case that a modify request could update a large number of attributes and/or attribute values, but generally clients modify only one or two values at a time. As such, the only modify test run was for a modify operation replacing a single attribute value. Results for this test are:

SDK	Highest Modify Throughput
JNDI	139,593.362 searches/sec
Netscape	139,362.802 searches/sec
Novell	122,199.697 searches/sec
OpenDS (asynchronous mode)	109,556.189 searches/sec
OpenDS (synchronous mode)	109,832.363 searches/sec
UnboundID (asynchronous mode)	196,411.917 searches/sec
UnboundID (synchronous mode)	242,248.797 searches/sec

 

Again, the UnboundID LDAP SDK is significantly faster than the other SDKs, and again, there is a significant advantage to using synchronous mode. The Novell SDK’s modify performance seems to drop off significantly for higher numbers of threads.

Summary

Download a complete set of results for all tests run as an OpenDocument spreadsheet.

In most cases, the UnboundID LDAP SDK for Java is faster than all other SDKs by a wide margin, and in all cases using the UnboundID LDAP SDK for Java in synchronous mode was faster than using the SDK in the default asynchronous mode. As such, if you are using the UnboundID LDAP SDK for Java and don’t need to perform asynchronous operations, then it is highly recommended that you enable synchronous mode for connections used by that application.

The only case in which the UnboundID LDAP SDK for Java was not the fastest is for a search operation in which a large number of entries were returned with only a single attribute per entry. It was the fastest for both other tests involving a large number of entries, and it was also the fastest for returning only one entry with a single attribute. I will investigate the UnboundID LDAP SDK’s performance in this area to determine whether it can be improved.

The OpenDS LDAP SDK (which was started after I left OpenDS, and for which I have not participated in its development in any way) appears to be the second fastest. It was the only SDK to perform faster than the UnboundID LDAP SDK in any of the tests, and it was never the slowest of any of the tests. There does not appear to be any measurable difference in performance when using the synchronous mode versus asynchronous mode. Across all of the tests, the OpenDS LDAP SDK achieved about 56.0% of the overall performance of the UnboundID LDAP SDK in synchronous mode, and 67.8% of the performance of the UnboundID LDAP SDK in asynchronous mode.

JNDI offers middle-of-the-pack performance in most cases, but its very poor showing for searches returning a single entry with high numbers of threads may be a significant cause for concern, since this is a very common kind of operation.

The Novell SDK performance when dealing with large search result entries is very troublesome, and it is also significantly slower than all other SDKs for modify operations with a high degree of concurrency. The Netscape SDK also appears to have problems with large search result entries, and its search performance for searches returning multiple entries is a problem as well.