LDAP

SLAMD 2.0.0-20090712

/ Neil Wilson / 1 Comment

A new build of SLAMD is available for download from http://www.slamd.com/. The release notes provide a full list of the changes since the previous 2.0.0-20090227 release, but some of the most significant changes are listed below.

  • The process used to load classes has been updated so that if an attempt to load a class fails, it will re-try the attempt after substituting “com.sun.slamd” with just “com.slamd”. This makes it possible to view statistics collected from builds of SLAMD prior to the 2.0.0-20090227 release.
  • Two new jobs have been added which make it possible to perform LDAP searches using filters read from a file and LDAP modifications with target entry DNs read from a file. These jobs can be used if the target search filters or entry DNs do not follow a pattern that can be constructed like the other jobs provide.
  • A number of jobs have been updated to provide support for rate limiting. You can now specify the desired number of operations per second per client, and the clients will attempt to maintain that rate.
  • A number of shell scripts used to launch tools have been updated to fix problems that prevented them from working properly in some cases.
  • LDAP jobs have been updated to use the synchronous mode provided in the latest release of the LDAP SDK for Java. This allows them to be significantly more efficient and drive more load against LDAP directory servers.

UnboundID LDAP SDK for Java 0.9.10

/ Neil Wilson

The UnboundID LDAP SDK for Java version 0.9.10 has just been released. The release notes provide the complete set of changes, but this post will outline two of the most significant changes in this release.

Synchronous Mode

The LDAP SDK now includes an option to operate in synchronous mode. If an application using the LDAP SDK does not need to perform asynchronous operations, and if it does not attempt to process multiple operations on the same connection at the same time, then you can use the new synchronous mode to achieve a significant increase in throughput and reduction in response time. In my testing, I have generally seen a performance improvement of about 15-20% by enabling the option to use synchronous mode.

Prior to this release, the LDAP SDK only operated in an asynchronous manner, in which each connection had a dedicated reader thread that could be used to read responses from the server. Whenever a request was sent to the server, a blocking queue was used to provide communication between the reader thread and the thread that sent the request. After sending a request to the server, the thread that sent that request would block on this queue waiting for the response from the server to be provided by the reader thread. This design is necessary to be able to handle the possibility of asynchronous operations, in which multiple requests can be outstanding at the same time, but it does incur overhead as a result of thread synchronization and context switching. This asynchronous manner of communication is still the default, but if your application doesn’t need the ability to process multiple operations on the same connection at the same time, then you can choose to use the new synchronous mode for better performance.

To enable synchronous mode, all you need to do is to set the useSynchronousMode connection option on the connection before it is established. For example: 

LDAPConnectionOptions options = new LDAPConnectionOptions();
options.setUseSynchronousMode(true);
LDAPConnection connection =
new LDAPConnection(options, host, port, bindDN, password);

You shouldn’t need to make any other changes to your application, since everything else should be completely transparent. If you do attempt to invoke an asynchronous operation (or attempt to perform an abandon or invoke a cancel extended request, which rely on the ability to perform asynchronous operations), then the SDK will throw an LDAP exception with a “not supported” result.

The synchronous mode is completely compatible with the use of connection pools, and in fact if you are currently using connection pools in your application then it’s very likely that your application meets the requirements for using synchronous mode, since it’s not all that easy to invoke asynchronous operations using pooled connections or to use a pooled connection concurrently across multiple threads.

There has been some other restructuring in the LDAP SDK to provide for this capability, and those underlying changes may also provide some level of benefit even for applications using the default asynchronous mode.

Spurious “Server Down” Responses

A user reported periodically getting “server down” responses, especially when reading large result sets. In the UnboundID LDAP SDK for Java, the “server down” response indicates that a previously-established connection is no longer usable, and generally indicates one of two things:

  • The connection between the client and the server has been lost unexpectedly, most likely because of a network problem or because the server closed the connection.
  • The client encountered an unrecoverable error while decoding a response from the server that prevents it from being able to perform any more communication over that connection.

Upon closer investigation, I did find that there was a corner case in which the LDAP SDK could experience a problem if it was in the process of decoding a response from the server in which the entire LDAP message for the response was not sent all at once but instead came in multiple pieces and there was a delay of at least 50 milliseconds (which is the default socket timeout in asynchronous mode, needed to handle a corner case for StartTLS processing) when trying to read a subsequent piece of the response message. In this case, the LDAP SDK had a problem in which it would try to start reading a completely new response rather than finishing the old response, and it would encounter an unrecoverable decoding error when trying to read in the middle of the old message as if it were the beginning of a new one.

This problem has been corrected, and the LDAP SDK will now properly handle this condition and is more tolerant of delays encountered when reading a response in multiple segments.

UnboundID LDAP SDK for Java 0.9.9

/ Neil Wilson

The UnboundID LDAP SDK for Java version 0.9.9 has just been released. The release notes provide the complete set of changes, but some of the most significant changes since the 0.9.8 release are listed below.

Maximum Search Response Queue Size

The LDAP SDK now places a maximum size limit on the blocking queue used to hold search result entries and references while waiting for them to be processed. When using a search result listener to consume these types of responses, this can help prevent problems in which the client could potentially run out of memory if a large number of search results are returned faster than the client can process them. With this change, if the client falls behind then the LDAP SDK may eventually create back-pressure against the directory server so that it will only be able to return entries and references as fast as the client can consume them.

Referral Enhancements

It is now possible to control on a per-request basis whether the LDAP SDK should automatically try to follow any referrals returned by the server. This was previously only available on a per-connection basis. In addition, a bug has been fixed that could cause referral following to fail for search operations.

User-Friendly Names for Connections and Connection Pools

It is now possible to assign user-friendly names for individual LDAP connections or LDAP connection pools. These names can be helpful in applications which maintain multiple connections or connection pools for different purposes. These names can be queried from code, and will appear in thread stack traces for the threads used to read responses for those connections.

Debugging Enhancements

It is now possible to indicate that debug log messages should include a stack trace of the thread that generated the message. It is possible to capture a stack trace at the time that a connection is established. The debug log message format has been changed to make the messages more consistent and easier to parse. In the even that a connection is terminated because of an unrecoverable error while attempting to read data from the client, then the underlying cause for that is now more visible.

Additional Connection Pool Statistics

LDAP connection pools now expose additional statistics that can be used to keep track of how a connection was obtained when checking a connection out of the pool (e.g., whether it was obtained immediately without blocking, whether it had to wait for a connection to become available, or whether a new connection was created). This can be helpful when determining whether a connection pool is sized appropriately for the application that is using it.

A couple of announcements from UnboundID

/ Neil Wilson / 1 Comment

Today, UnboundID made a couple of announcements.

Sun DSEE to UnboundID Directory Server Migration

First, we’re announcing new pricing for our Directory Server for current customers of Sun’s DSEE product. If you’re currently using Sun DSEE, then you can migrate to the UnboundID Directory Server for less than what you’re paying now in maintenance costs.

In addition to saving money, you’ll also be getting:

  • Much better performance and scalability. I can’t post actual performance comparisons because of restrictions in the DSEE license agreement, but the UnboundID Directory Server leaves it in the dust.
  • Much smaller on-disk and in-memory footprint. We store data in a very compact form that takes much less space than DSEE requires (and in most cases, the fully-imported database including all the data and indexes requires notably less space than the LDIF file). You can fully cache a much larger data set on the same machine than DSEE, or you can use a smaller machine than DSEE requires for a given data set.
  • A much broader feature set. The UnboundID Directory Server includes virtually all features available in Sun DSEE, plus quite a bit more. Our server has lots of features like transactions, filtered logging, centralized logging, configuration auditing, multiple root users, automatic database priming, and incremental backup that DSEE just doesn’t have.
  • Significant improvements over DSEE in areas where both servers have overlapping features. This includes areas like server and environment management, monitoring and alerting, replication, password policy and security, standards compliance, and related tools.

Of course, we’re happy to work with anyone who wants to migrate from any other type of directory server to the UnboundID Directory Server. Contact sales@unboundid.com for more information.

UnboundID Synchronization Server

The UnboundID Synchronization Server provides the ability to synchronize data between directory servers or directory topologies. We currently support synchronization of data between any combination of UnboundID Directory Server and DSEE, with more data sources to be added in the future.

The Synchronization Server has a very powerful feature set allowing you to pick which entries to synchronize, which attributes to synchronize within those entries, and what kinds of transformations to apply (including renaming attributes, constructing or suppressing attribute values, transforming DNs, etc.). Synchronization occurs in real time, and can be configured in one or both directions (with the same or different data flowing in either direction).

The UnboundID Synchronization Server is free to use in the process of migrating your directory environment between Sun DSEE and the UnboundID Directory Server.

UnboundID LDAP SDK for Java 0.9.8

/ Neil Wilson / 1 Comment

The UnboundID LDAP SDK for Java version 0.9.8 has just been released. The release notes provide the complete set of changes, but the most significant changes since the 0.9.7 release are listed below. Because of the potential connection leak, anyone using connection pooling in the LDAP SDK is strongly encouraged to upgrade to this release.

StartTLS Bug Fix

Previous versions of the LDAP SDK had a bug in which attempting to process a StartTLS extended operation could fail to return if the server certificate was not trusted. The client connection was terminated, but the processExtendedOperation method would never return. This has been fixed.

LDAP Connection Pool Bug Fixes

Two bugs in the LDAPConnectionPool class have been corrected.

The first bug is that the connection pool could leak a connection when performing a background health check if that connection was examined multiple times in the same health check sequence.

The second bug is that when attempting to retrieve the root DSE from the directory server, two connections were obtained from the pool when only one was needed. Both connections were properly returned, but taking twice as many connections could have created an artificial connection shortage.

Operation Rate Limiting

A new FixedRateBarrier utility class has been added which can be used to help try to perform operations no faster than a specified rate. The authrate, modrate, and searchrate tools have been updated to use this and now provide a “--ratePerSecond” argument that can be used to attempt to perform authentications, modifies, and searches at a specified rate.

Transactions in the UnboundID Directory Server

/ Neil Wilson / 1 Comment

Not that long ago, directory servers didn’t really offer that much in the way of being able to perform atomic operations. The LDAP specification requires that individual updates be atomic, so you could be sure that if you updated multiple attributes in the same entry with a single modify operation then they would all succeed or all fail. In some directory servers, that’s the only type of atomicity you can get.

When the LDAPv3 specifications were updated a few years ago, some new RFCs were published that make it possible to perform some basic atomic operations within a single entry. They include:

  • LDAP Modify-Increment Extension (RFC 4525) — This makes it possible to atomically increment or decrement a single-valued integer attribute by a specified amount without needing to know the current value.
  • LDAP Read Entry Controls (RFC 4527) — This makes it possible to retrieve an entry immediately before a delete, modify, or modify DN operation, or immediately after an add, modify, or modify DN operation.
  • LDAP Assertion Control (RFC 4528) — This makes it possible to request that an operation be processed only if the target entry matches a specified filter.

These features can be very helpful when dealing with a single entry because they provide an atomic compare-and-set type of functionality. However, they do not provide any support for interacting with multiple entries as a single atomic unit. For that, you need transactions, and the UnboundID Directory Server supports two kinds of transactions.

Batched Transactions

The first type of transaction offered by the UnboundID Directory Server is called “batched transactions”, because it allows you to provide a number of add, delete, modify, modify DN, or password modify operations (targeting the same entry or different entries) which will get batched up and processed as a single atomic unit. This capability is based on the draft-zeilenga-ldap-txn Internet Draft titled “LDAP Transactions”, and the process for using them is basically as follows:

  1. Send a start transaction extended request to the Directory Server. In its response, the server will provide a transaction ID.
  2. Send a set of add, delete, modify, modify DN, and/or password modify operations. In each request, include a control that includes the transaction ID that you got in the start transaction response so that the server knows that operation is part of the transaction.
  3. Send an end transaction extended request to the Directory Server indicating that the transaction should be committed. The server will process all of the operations as a single atomic unit so that they will either all succeed or will all fail.

This can be very useful, but you are limited to only performing write operations, so you can’t read an entry and then attempt to update it with the guarantee that it hasn’t been modified between the read and the write. We do provide support for using the increment, read entry, and assertions capabilities with batched transactions, so you can read an entry outside of the transaction and then use the LDAP assertion control in an attempt to ensure that it hasn’t been altered since you last read it, but if something did happen to modify the entry then the entire transaction would fail.

Interactive Transactions

The second type of transactions offered by the UnboundID Directory Server is called “interactive transactions”, because operations which are part of the transaction are processed immediately as they are requested rather than being batched up and processed all at once. They provide all of the capabilities offered by batched transactions, but also make it possible to include search and compare operations within the transaction.

With interactive transactions, it is possible to read an entry and then modify it based on what you read with the guarantee that no other client would be allowed to change it while the transaction is still in progress. Once an entry has been read (via a search or compare operation) as part of a transaction, then any attempt to alter it outside of that transaction will be forced to wait until the transaction has been committed or aborted, although attempts to read it will still be allowed. Once an entry has been altered as part of a transaction (via an add, delete, modify, modify DN, or password modify operation), then any attempt to access it from outside the transaction will be forced to wait until the transaction has been committed or aborted. That is, any changes made as part of an interactive transaction will only be visible within the context of that transaction until it has been committed.

Using Transactions in Client Code

At present, the implementations for both batched transactions and interactive transactions are specific to the UnboundID Directory Server.

The “LDAP Transactions” Internet Draft hasn’t yet reached a sufficient level of maturity to allow interoperable implementations to be created (in particular, it doesn’t specify what OIDs should be used for the control, extended requests, and extended responses), nor is there a guarantee that this draft won’t change in incompatible ways before it is finalized and published as an RFC (if that ever happens). As a result, we guarantee that our implementation of batched transactions will remain backwards compatible, but we can’t make any claims of interoperability with any other type of client or server. If the “LDAP Transactions” draft ever is published as an RFC, then we will support the final specification in addition to what we currently offer.

The interactive transactions capability that we offer is specific to the UnboundID Directory Server and is not based on any public specification, since no draft or RFC covers anything like it.

When interacting with our Directory Server, the UnboundID LDAP SDK for Java may be used to perform both batched and interactive transactions. The version of the LDAP SDK which is available for public download doesn’t offer this capability, but if you get our Directory Server, then you also get a special “Commercial Edition” version of the LDAP SDK for Java that contains additional features that make it possible to use additional functionality specific to our Directory Server. This includes support for batched and interactive transactions, as well as other custom controls and extended operations we have added to our server, and also includes APIs for interacting with monitoring information and administrative tasks. The only differences between the Standard Edition and Commercial Edition versions of our LDAP SDK are around features that are specific to our Directory Server, so there really isn’t any benefit to having the Commercial Edition of our LDAP SDK if you don’t also have our Directory Server.

As with batched transactions, if that capability is ever published as an RFC, then the Standard Edition of our LDAP SDK (which may be used with any LDAPv3 directory server) will be updated to support that specification.

UnboundID LDAP SDK for Java 0.9.7

/ Neil Wilson

The UnboundID LDAP SDK for Java version 0.9.7 has just been released. The release notes provides information about the complete set of changes, but some of the changes since the 0.9.6 release are listed below. If you’d like to be notified by e-mail when a new release is available, send a message to ldapsdk-announce-subscribe@unboundid.com.

LDIF Reader Performance

A change introduced in the 0.9.6 release dramatically improved LDIF reading performance for large entries with attributes with a lot of values (e.g., large static groups). This change has been generalized so that it applies to entries of any size, which significantly benefits performance when reading smaller static groups.

Compact Entries

A new CompactEntry class has been added which can be used to represent a read-only entry in a form that requires much less memory than the existing Entry class, although with a potential performance penalty for some operations. This is primarily useful for the case in which it is necessary to store a large number of entries in memory, since it is possible to hold more entries in a given amount of memory. As an example, for a typical user entry, CompactEntry requires only 25% as much memory as Entry, and for a simple static group with 20 members, CompactEntry requires only 31% as much memory as Entry.

Android Demo Source

The source code for an application demonstrating the use of the UnboundID LDAP SDK for Java (mentioned in my last post) is now included.

Using LDAP on Android

/ Neil Wilson / 13 Comments

Mobile devices are quickly becoming indispensable tools for convenience and productivity wherever you may happen to be. Around the beginning of the year, I started to look at the Android platform for developing applications for mobile devices (and other low-end kinds of systems). I picked up a Google developer phone (basically an unlocked version of the T-Mobile G1) and was impressed by the pretty rich API that it offers. Quite a large portion of Java SE is available for use, although there are some notable exceptions. For example, it’s missing all of the GUI classes, but that makes sense since Android provides its own GUI library. But another significant omission is that of JNDI, which is the API included in Java SE for communicating with LDAP directories.

On one hand, this is a good thing since JNDI is a horrible API that is inconvenient to use and doesn’t provide very good access to the LDAP protocol, and the less opportunity there is to use it the better. But on the other hand, being able to communicate with an LDAP directory server on a mobile device can be extremely helpful. For example, it could be very convenient to have access to the entire corporate directory at your fingertips for phone numbers and e-mail addresses without the need to clutter your contacts list, or maybe to keep local contacts in sync with their corresponding entries from the corporate directory.

The good news, though, is that Android developers don’t need to worry about missing LDAP support for Android because the UnboundID LDAP SDK for Java works just fine on the Android platform. I’ve done quite a bit of testing and made sure that things just work. With a couple of exceptions (primarily CRAM-MD5, DIGEST-MD5, and GSSAPI, which require the Java SASL API that isn’t available on Android) pretty much all of the functionality that we offer in our LDAP SDK can be used on Android. This includes all of the core protocol, all of the controls and extended operations, SSL/StartTLS, connection pooling, and all the other stuff is there, too. Note that there is no special “Android” version of the LDAP SDK — the same SDK that you can use for writing desktop clients or server-side applications works without any changes on the Android platform.

To demonstrate this capability, I’ve put together a simple LDAP client for Android devices. You can see a video online at http://www.youtube.com/watch?v=1NlhwrcCeSQ.

UnboundID Directory Server

/ Neil Wilson / 2 Comments

I’ve already mentioned that one of the first things we started working on when we founded UnboundID was a Java-based LDAP SDK. It shouldn’t come as much surprise to anyone that we’ve also been working on an LDAP directory server, but we hadn’t actually announced that, at least not until today. The Directory Server product page is now online, and we also have a press release announcing the product, and also announcing a partnership with Alcatel-Lucent as an exclusive provider of our Directory Server in telco markets.

When we started working on our Directory Server, we did use OpenDS as the code base for the product (and portions of our server are covered by the CDDL license), but we’ve taken it far beyond what’s available in OpenDS. We’ve worked hard on increasing performance and scalability, reducing the on-disk and in-memory footprint, improving management and administration, and reducing operation response times. We’ve also packed in lots of new features, bug fixes, enhancements, and tools. I’ll talk more about some of the improvements we’ve made in upcoming posts.

UnboundID LDAP SDK for Java 0.9.6

/ Neil Wilson

The UnboundID LDAP SDK for Java version 0.9.6 has just been released. This post details some of the changes included since the previous 0.9.5 release. See the release notes for additional information. If you’d like to be notified by e-mail when a new release is available, send a message to ldapsdk-announce-subscribe@unboundid.com.

Some of the changes in this release were based on feedback we received regarding the LDAP SDK. If you have suggestions or other feedback, feel free to send your comments to ldapsdk-support@unboundid.com.

Intermediate Response Messages

The LDAP SDK now includes support for LDAP intermediate response messages as defined in RFC 3771. Intermediate responses make it possible for operations return multiple responses to a single request. A new IntermediateResponseListener interface has been added, and clients can use the LDAPRequest.setIntermediateResponseListener method to be notified of any intermediate responses returned for the operation.

AuthRate Tool

A new authrate tool has been added as an example program. It is similar to the existing SearchRate tool, except that when retrieving an entry it can also perform a bind to verify the user’s password. It supports simple authentication, as well as the CRAM-MD5, DIGEST-MD5, and PLAIN mechanisms.

CommandLineTool Example Usages

The CommandLineTool API has been updated so that it is now possible to provide examples in the usage output. All of the example tools provided with the LDAP SDK now provide example usages.

Connection Pool Connection Expiration

It is now possible to define a maximum age for connections included in a connection pool. If a maximum age is defined and a connection in the pool has been established for greater than this length of time, then the connection will be closed and replaced with a new connection.

Connection and Connection Pool Statistics

A new LDAPConnection.getConnectionStatistics method has been added which can be used to retrieve an LDAPConnectionStatistics object with information about how that connection has been used. This includes the number of times the connection has been established and closed, the number of times each type of request has been sent, the number of times each type of response has been received, and the total and average response time for operations.

A new LDAPConnectionPool.getConnectionPoolStatistics method has been added which can be used to retrieve an LDAPConnectionPoolStatistics object which can be used to retrieve information about how the connection pool has been used. This includes the number of connections currently available, the maximum number of connections that may be available, the number of successful and failed attempts to create new connections, the number of connections that have been closed as defunct, expired, or unneeded, the number of successful and failed attempts to retrieve a connection, and the number of valid connections released back to the pool.

This feature was suggested by Philip Peake.

Connection Pool Health Checking

A new LDAPConnectionPoolHealthCheck API has been added which makes it possible to perform health checking for connections in a connection pool. It can be used to perform checks to determine whether a connection is valid when a new connection is created, checked out, released back to the pool, or after an exception has been encountered while processing an operation in the pool. In addition, a background thread will be used to periodically invoke health checks against all available connections.

A health check implementation has been added which can be used to verify the validity of a connection by retrieving a specified entry from the directory server. It can assess the connection’s availability based on whether the entry can be retrieved and the length of time required to retrieve it.

This feature was suggested by Philip Peake.

Improved Interaction with DNs and RDNs as Strings

A number of convenience methods have been added to the DN and RDN classes to make it easier to deal with them using their string representations. This includes the ability to determine whether a string represents a valid DN or RDN, to compare DNs and RDNs based on their string representations, and to determine whether strings representing DNs have a hierarchical relationship.

This feature was suggested by Felix Gaehtgens.

Improved LDIFReader Performance

The LDIFReader class has been updated to improve its performance when reading entries containing an attribute with a large number of values (e.g., a static group). For a single-threaded test case involving reading group entries with 2000 members, the new implementation was measured at over 360 times faster.

This problem was reported by Felix Gaehtgens.