UnboundID

LDAP SDK now on SourceForge

/ Neil Wilson

Earlier today, I created a SourceForge project for the UnboundID LDAP SDK for Java. You can find it at http://sourceforge.net/projects/ldap-sdk/. This project contains a subversion repository containing the source code for the Standard Edition, as well as a set of mailing lists and a forum and a download of the current 1.0.0 release. You can now easily check out and build the LDAP SDK for yourself.

This makes it easy to see what we’re working on without having to wait for the next release (which currently includes things like OSGi support and easier migration from other Java-based LDAP APIs), and if you do run into a problem of some kind that requires a code fix, you can get access to that fix without needing to wait for a new release and without requiring us to provide you with a private build.

UnboundID LDAP SDK for Java 1.0.0

/ Neil Wilson / 4 Comments

UnboundID has released version 1.0.0 of the UnboundID LDAP SDK for Java. We have felt for a long time that it was more than stable enough for use in production applications (and in fact it is in use in production applications in several places), but had not yet declared it 1.0 just in case we received any feedback that might have caused us to consider altering some portion of the API in an incompatible way. Although we have gotten several comments and suggestions, and we have added a number of features in response to that feedback, there haven’t been any incompatible changes and we feel very confident that there won’t need to be any such changes in the future (and have an extensive test framework in place to ensure that doesn’t happen). We are therefore bumping the version number to 1.0.0 just in case the pre-1.0 version number caused anyone to shy away from using it.

A complete list of the changes since the 0.9.10 release can be found in the release notes, but some of the most significant changes include:

  • We have added support for the permissive modify control, which may be used to request that the directory server not reject attempts to add an attribute value that already exists or remove an attribute value that does not yet exist.
  • We have added a new LDAPThreadLocalConnectionPool class, which provides an alternate connection pool implementation that uses a separate connection per client thread that attempts to use the connection pool. This eliminates the need to explicitly configure the number of connections to maintain, and also eliminates contention between threads when trying to use the pool.
  • We have added the ability to create a Schema object by reading definitions from one or more LDIF files. It was previously only possible to create a Schema object by reading it from a directory server over LDAP.
  • We have updated the LDIF reader to provide the ability to reject entries that contain duplicate values for the same attribute in the same entry. We have also added the ability to use schema information when reading data from LDIF to help avoid incorrectly classifying values as duplicates (e.g., if two values differ only by capitalization, but are in an attribute type defined with a caseExactMatch equality matching rule).
  • We have updated the validate-ldif tool so that it has the ability to use schema definitions read from files (allowing it to be used in a completely offline manner), and to take advantage of the ability to reject entries with duplicate attribute values (although this can be disabled if desired).

UnboundID Directory Proxy Server

/ Neil Wilson

Today, UnboundID is announcing the UnboundID Directory Proxy Server, which is a fast, scalable, robust, and easy-to-use LDAPv3 proxy server. It is designed to work with any LDAPv3 directory server, although some advanced functionality is intended for use with the UnboundID Directory Server.

This post describes some of the features and advantages we offer in the Directory Proxy Server product. For more information, contact sales@unboundid.com.

Advanced Health Checking

The UnboundID Directory Proxy Server offers advanced health checking capabilities that can detect and react to problems as soon as they happen, and in some cases even before they happen, so that clients are not adversely impacted by (and often won’t even notice) unexpected hardware, software, or networking failures.

Some of the things that the Directory Proxy Server can consider when monitoring the health of the backend servers include:

  • The length of time required to process operations in the server.
  • The existence and content of specified entries in the server.
  • The replication backlog in the server (both in terms of the number of pending changes and the age of the oldest outstanding change).
  • The level of busyness for that server (both for operations requested through the Directory Proxy Server, as well as through other routes).

In the event that a problem is detected with a backend server, then the Directory Proxy Server can de-prioritize that server relative to other servers in the environment, or can take it out of the mix altogether. Further, if the Directory Proxy Server is used in front of the UnboundID Directory Server, then the Directory Server instances can proactively notify the Directory Proxy Server of any problems that arise. This can help avoid problems that the Directory Proxy Server may not be able to detect on its own (e.g., running low on disk space on the Directory Server system) until they become critical problems resulting in operation failures.

Flexible Load Balancing

The UnboundID Directory Proxy Server offers a number of options for load balancing requests across multiple backend servers, which can help provide both high performance and high availability. Some of the things that it can take into account when deciding where to send a request include:

  • The availability of the backend servers, so that servers which are experiencing a problem of some kind can be avoided.
  • The relative healths of the backend servers, so that servers with a higher health check score can be given a proportionally larger percent of the load.
  • The locations of the backend servers, so that servers in the same data center as the Directory Proxy Server will be preferred over those that are remote. If it is necessary to send requests to remote servers, then those locations can also be prioritized to prefer closer data centers over those that are farther away.
  • Statically-defined weights assigned to individual servers or sets of servers.
  • The type of operation being requested. If you want, you can configure different types of load balancing for each type of operation.
  • Information about the way that previous requests have been handled, so that related requests are consistently routed to the same backend server.
  • Information about the client issuing the request. You can define different policies for different kinds of clients, based on a wide range of criteria.

Scalability and Entry Balancing

The UnboundID Directory Proxy Server can help you improve the overall performance and scalability of your directory environment. The load balancing capabilities mentioned above allow you to spread requests across multiple servers with the same content to ensure high throughput and availability.

In order to achieve optimal performance, you need to be able to cache all of the data in memory. The very compact way in which the UnboundID Directory Server represents entry data both on disk and in memory ensures that any single instance can hold a large number of entries, but sometimes even that may not be enough and you may need to utilize more memory than is available in any single instance. For these cases, the UnboundID Directory Server offers entry balancing, which allows you to seamlessly spread data across multiple sets of servers to take advantage of the aggregate memory across all of those sets. You can do this without the need to alter the content or hierarchy of your data, and without restricting which attributes clients can use in their requests for optimal performance.

Simple Installation, Configuration, and Management

The process for installing and configuring the UnboundID Directory Proxy Server is faster easier than any other LDAP proxy server I have ever used, even for relatively complex deployments across multiple data centers or when using entry balancing. And because the Directory Proxy Server shares much of the same code base as the Directory Server, you can use the same graphical, interactive text-based, and non-interactive command-line interfaces to manage both products. If you’re already familiar with the UnboundID Directory Server, then you can be up and running with the Directory Proxy Server in no time. Even if you’re completely new to the products then you should find the administrative interfaces simple and intuitive, yet still very powerful.

Other Features

Some of the other capabilities offered by the UnboundID Directory Proxy Server include:

  • Easily track requests through the directory environment. Both the Directory Server and the Directory Proxy Server access logs contain information that can easily allow you to correlate information about operations across multiple instances.
  • Transform requests and responses passing through the Directory Proxy Server to match the way that clients expect to access the data, even if that isn’t how it actually appears in the backend servers.
  • Define limits on what clients are allowed to do in the directory environment, including what types of operations they may request and which portions of the DIT they can access. As with load balancing, you can create multiple policies for different kinds of clients based on a very wide range of criteria.
  • Allow administrators to easily access individual backend servers through the Directory Proxy Server (including access to things like the server configuration, schema, and monitor data), bypassing any load balancing and health checking processing that would otherwise be enforced for normal client requests.

SLAMD 2.0.0-20090712

/ Neil Wilson / 1 Comment

A new build of SLAMD is available for download from http://www.slamd.com/. The release notes provide a full list of the changes since the previous 2.0.0-20090227 release, but some of the most significant changes are listed below.

  • The process used to load classes has been updated so that if an attempt to load a class fails, it will re-try the attempt after substituting “com.sun.slamd” with just “com.slamd”. This makes it possible to view statistics collected from builds of SLAMD prior to the 2.0.0-20090227 release.
  • Two new jobs have been added which make it possible to perform LDAP searches using filters read from a file and LDAP modifications with target entry DNs read from a file. These jobs can be used if the target search filters or entry DNs do not follow a pattern that can be constructed like the other jobs provide.
  • A number of jobs have been updated to provide support for rate limiting. You can now specify the desired number of operations per second per client, and the clients will attempt to maintain that rate.
  • A number of shell scripts used to launch tools have been updated to fix problems that prevented them from working properly in some cases.
  • LDAP jobs have been updated to use the synchronous mode provided in the latest release of the LDAP SDK for Java. This allows them to be significantly more efficient and drive more load against LDAP directory servers.

UnboundID LDAP SDK for Java 0.9.10

/ Neil Wilson

The UnboundID LDAP SDK for Java version 0.9.10 has just been released. The release notes provide the complete set of changes, but this post will outline two of the most significant changes in this release.

Synchronous Mode

The LDAP SDK now includes an option to operate in synchronous mode. If an application using the LDAP SDK does not need to perform asynchronous operations, and if it does not attempt to process multiple operations on the same connection at the same time, then you can use the new synchronous mode to achieve a significant increase in throughput and reduction in response time. In my testing, I have generally seen a performance improvement of about 15-20% by enabling the option to use synchronous mode.

Prior to this release, the LDAP SDK only operated in an asynchronous manner, in which each connection had a dedicated reader thread that could be used to read responses from the server. Whenever a request was sent to the server, a blocking queue was used to provide communication between the reader thread and the thread that sent the request. After sending a request to the server, the thread that sent that request would block on this queue waiting for the response from the server to be provided by the reader thread. This design is necessary to be able to handle the possibility of asynchronous operations, in which multiple requests can be outstanding at the same time, but it does incur overhead as a result of thread synchronization and context switching. This asynchronous manner of communication is still the default, but if your application doesn’t need the ability to process multiple operations on the same connection at the same time, then you can choose to use the new synchronous mode for better performance.

To enable synchronous mode, all you need to do is to set the useSynchronousMode connection option on the connection before it is established. For example: 

LDAPConnectionOptions options = new LDAPConnectionOptions();
options.setUseSynchronousMode(true);
LDAPConnection connection =
new LDAPConnection(options, host, port, bindDN, password);

You shouldn’t need to make any other changes to your application, since everything else should be completely transparent. If you do attempt to invoke an asynchronous operation (or attempt to perform an abandon or invoke a cancel extended request, which rely on the ability to perform asynchronous operations), then the SDK will throw an LDAP exception with a “not supported” result.

The synchronous mode is completely compatible with the use of connection pools, and in fact if you are currently using connection pools in your application then it’s very likely that your application meets the requirements for using synchronous mode, since it’s not all that easy to invoke asynchronous operations using pooled connections or to use a pooled connection concurrently across multiple threads.

There has been some other restructuring in the LDAP SDK to provide for this capability, and those underlying changes may also provide some level of benefit even for applications using the default asynchronous mode.

Spurious “Server Down” Responses

A user reported periodically getting “server down” responses, especially when reading large result sets. In the UnboundID LDAP SDK for Java, the “server down” response indicates that a previously-established connection is no longer usable, and generally indicates one of two things:

  • The connection between the client and the server has been lost unexpectedly, most likely because of a network problem or because the server closed the connection.
  • The client encountered an unrecoverable error while decoding a response from the server that prevents it from being able to perform any more communication over that connection.

Upon closer investigation, I did find that there was a corner case in which the LDAP SDK could experience a problem if it was in the process of decoding a response from the server in which the entire LDAP message for the response was not sent all at once but instead came in multiple pieces and there was a delay of at least 50 milliseconds (which is the default socket timeout in asynchronous mode, needed to handle a corner case for StartTLS processing) when trying to read a subsequent piece of the response message. In this case, the LDAP SDK had a problem in which it would try to start reading a completely new response rather than finishing the old response, and it would encounter an unrecoverable decoding error when trying to read in the middle of the old message as if it were the beginning of a new one.

This problem has been corrected, and the LDAP SDK will now properly handle this condition and is more tolerant of delays encountered when reading a response in multiple segments.

UnboundID LDAP SDK for Java 0.9.9

/ Neil Wilson

The UnboundID LDAP SDK for Java version 0.9.9 has just been released. The release notes provide the complete set of changes, but some of the most significant changes since the 0.9.8 release are listed below.

Maximum Search Response Queue Size

The LDAP SDK now places a maximum size limit on the blocking queue used to hold search result entries and references while waiting for them to be processed. When using a search result listener to consume these types of responses, this can help prevent problems in which the client could potentially run out of memory if a large number of search results are returned faster than the client can process them. With this change, if the client falls behind then the LDAP SDK may eventually create back-pressure against the directory server so that it will only be able to return entries and references as fast as the client can consume them.

Referral Enhancements

It is now possible to control on a per-request basis whether the LDAP SDK should automatically try to follow any referrals returned by the server. This was previously only available on a per-connection basis. In addition, a bug has been fixed that could cause referral following to fail for search operations.

User-Friendly Names for Connections and Connection Pools

It is now possible to assign user-friendly names for individual LDAP connections or LDAP connection pools. These names can be helpful in applications which maintain multiple connections or connection pools for different purposes. These names can be queried from code, and will appear in thread stack traces for the threads used to read responses for those connections.

Debugging Enhancements

It is now possible to indicate that debug log messages should include a stack trace of the thread that generated the message. It is possible to capture a stack trace at the time that a connection is established. The debug log message format has been changed to make the messages more consistent and easier to parse. In the even that a connection is terminated because of an unrecoverable error while attempting to read data from the client, then the underlying cause for that is now more visible.

Additional Connection Pool Statistics

LDAP connection pools now expose additional statistics that can be used to keep track of how a connection was obtained when checking a connection out of the pool (e.g., whether it was obtained immediately without blocking, whether it had to wait for a connection to become available, or whether a new connection was created). This can be helpful when determining whether a connection pool is sized appropriately for the application that is using it.

A couple of announcements from UnboundID

/ Neil Wilson / 1 Comment

Today, UnboundID made a couple of announcements.

Sun DSEE to UnboundID Directory Server Migration

First, we’re announcing new pricing for our Directory Server for current customers of Sun’s DSEE product. If you’re currently using Sun DSEE, then you can migrate to the UnboundID Directory Server for less than what you’re paying now in maintenance costs.

In addition to saving money, you’ll also be getting:

  • Much better performance and scalability. I can’t post actual performance comparisons because of restrictions in the DSEE license agreement, but the UnboundID Directory Server leaves it in the dust.
  • Much smaller on-disk and in-memory footprint. We store data in a very compact form that takes much less space than DSEE requires (and in most cases, the fully-imported database including all the data and indexes requires notably less space than the LDIF file). You can fully cache a much larger data set on the same machine than DSEE, or you can use a smaller machine than DSEE requires for a given data set.
  • A much broader feature set. The UnboundID Directory Server includes virtually all features available in Sun DSEE, plus quite a bit more. Our server has lots of features like transactions, filtered logging, centralized logging, configuration auditing, multiple root users, automatic database priming, and incremental backup that DSEE just doesn’t have.
  • Significant improvements over DSEE in areas where both servers have overlapping features. This includes areas like server and environment management, monitoring and alerting, replication, password policy and security, standards compliance, and related tools.

Of course, we’re happy to work with anyone who wants to migrate from any other type of directory server to the UnboundID Directory Server. Contact sales@unboundid.com for more information.

UnboundID Synchronization Server

The UnboundID Synchronization Server provides the ability to synchronize data between directory servers or directory topologies. We currently support synchronization of data between any combination of UnboundID Directory Server and DSEE, with more data sources to be added in the future.

The Synchronization Server has a very powerful feature set allowing you to pick which entries to synchronize, which attributes to synchronize within those entries, and what kinds of transformations to apply (including renaming attributes, constructing or suppressing attribute values, transforming DNs, etc.). Synchronization occurs in real time, and can be configured in one or both directions (with the same or different data flowing in either direction).

The UnboundID Synchronization Server is free to use in the process of migrating your directory environment between Sun DSEE and the UnboundID Directory Server.

UnboundID LDAP SDK for Java 0.9.8

/ Neil Wilson / 1 Comment

The UnboundID LDAP SDK for Java version 0.9.8 has just been released. The release notes provide the complete set of changes, but the most significant changes since the 0.9.7 release are listed below. Because of the potential connection leak, anyone using connection pooling in the LDAP SDK is strongly encouraged to upgrade to this release.

StartTLS Bug Fix

Previous versions of the LDAP SDK had a bug in which attempting to process a StartTLS extended operation could fail to return if the server certificate was not trusted. The client connection was terminated, but the processExtendedOperation method would never return. This has been fixed.

LDAP Connection Pool Bug Fixes

Two bugs in the LDAPConnectionPool class have been corrected.

The first bug is that the connection pool could leak a connection when performing a background health check if that connection was examined multiple times in the same health check sequence.

The second bug is that when attempting to retrieve the root DSE from the directory server, two connections were obtained from the pool when only one was needed. Both connections were properly returned, but taking twice as many connections could have created an artificial connection shortage.

Operation Rate Limiting

A new FixedRateBarrier utility class has been added which can be used to help try to perform operations no faster than a specified rate. The authrate, modrate, and searchrate tools have been updated to use this and now provide a “--ratePerSecond” argument that can be used to attempt to perform authentications, modifies, and searches at a specified rate.

Transactions in the UnboundID Directory Server

/ Neil Wilson / 1 Comment

Not that long ago, directory servers didn’t really offer that much in the way of being able to perform atomic operations. The LDAP specification requires that individual updates be atomic, so you could be sure that if you updated multiple attributes in the same entry with a single modify operation then they would all succeed or all fail. In some directory servers, that’s the only type of atomicity you can get.

When the LDAPv3 specifications were updated a few years ago, some new RFCs were published that make it possible to perform some basic atomic operations within a single entry. They include:

  • LDAP Modify-Increment Extension (RFC 4525) — This makes it possible to atomically increment or decrement a single-valued integer attribute by a specified amount without needing to know the current value.
  • LDAP Read Entry Controls (RFC 4527) — This makes it possible to retrieve an entry immediately before a delete, modify, or modify DN operation, or immediately after an add, modify, or modify DN operation.
  • LDAP Assertion Control (RFC 4528) — This makes it possible to request that an operation be processed only if the target entry matches a specified filter.

These features can be very helpful when dealing with a single entry because they provide an atomic compare-and-set type of functionality. However, they do not provide any support for interacting with multiple entries as a single atomic unit. For that, you need transactions, and the UnboundID Directory Server supports two kinds of transactions.

Batched Transactions

The first type of transaction offered by the UnboundID Directory Server is called “batched transactions”, because it allows you to provide a number of add, delete, modify, modify DN, or password modify operations (targeting the same entry or different entries) which will get batched up and processed as a single atomic unit. This capability is based on the draft-zeilenga-ldap-txn Internet Draft titled “LDAP Transactions”, and the process for using them is basically as follows:

  1. Send a start transaction extended request to the Directory Server. In its response, the server will provide a transaction ID.
  2. Send a set of add, delete, modify, modify DN, and/or password modify operations. In each request, include a control that includes the transaction ID that you got in the start transaction response so that the server knows that operation is part of the transaction.
  3. Send an end transaction extended request to the Directory Server indicating that the transaction should be committed. The server will process all of the operations as a single atomic unit so that they will either all succeed or will all fail.

This can be very useful, but you are limited to only performing write operations, so you can’t read an entry and then attempt to update it with the guarantee that it hasn’t been modified between the read and the write. We do provide support for using the increment, read entry, and assertions capabilities with batched transactions, so you can read an entry outside of the transaction and then use the LDAP assertion control in an attempt to ensure that it hasn’t been altered since you last read it, but if something did happen to modify the entry then the entire transaction would fail.

Interactive Transactions

The second type of transactions offered by the UnboundID Directory Server is called “interactive transactions”, because operations which are part of the transaction are processed immediately as they are requested rather than being batched up and processed all at once. They provide all of the capabilities offered by batched transactions, but also make it possible to include search and compare operations within the transaction.

With interactive transactions, it is possible to read an entry and then modify it based on what you read with the guarantee that no other client would be allowed to change it while the transaction is still in progress. Once an entry has been read (via a search or compare operation) as part of a transaction, then any attempt to alter it outside of that transaction will be forced to wait until the transaction has been committed or aborted, although attempts to read it will still be allowed. Once an entry has been altered as part of a transaction (via an add, delete, modify, modify DN, or password modify operation), then any attempt to access it from outside the transaction will be forced to wait until the transaction has been committed or aborted. That is, any changes made as part of an interactive transaction will only be visible within the context of that transaction until it has been committed.

Using Transactions in Client Code

At present, the implementations for both batched transactions and interactive transactions are specific to the UnboundID Directory Server.

The “LDAP Transactions” Internet Draft hasn’t yet reached a sufficient level of maturity to allow interoperable implementations to be created (in particular, it doesn’t specify what OIDs should be used for the control, extended requests, and extended responses), nor is there a guarantee that this draft won’t change in incompatible ways before it is finalized and published as an RFC (if that ever happens). As a result, we guarantee that our implementation of batched transactions will remain backwards compatible, but we can’t make any claims of interoperability with any other type of client or server. If the “LDAP Transactions” draft ever is published as an RFC, then we will support the final specification in addition to what we currently offer.

The interactive transactions capability that we offer is specific to the UnboundID Directory Server and is not based on any public specification, since no draft or RFC covers anything like it.

When interacting with our Directory Server, the UnboundID LDAP SDK for Java may be used to perform both batched and interactive transactions. The version of the LDAP SDK which is available for public download doesn’t offer this capability, but if you get our Directory Server, then you also get a special “Commercial Edition” version of the LDAP SDK for Java that contains additional features that make it possible to use additional functionality specific to our Directory Server. This includes support for batched and interactive transactions, as well as other custom controls and extended operations we have added to our server, and also includes APIs for interacting with monitoring information and administrative tasks. The only differences between the Standard Edition and Commercial Edition versions of our LDAP SDK are around features that are specific to our Directory Server, so there really isn’t any benefit to having the Commercial Edition of our LDAP SDK if you don’t also have our Directory Server.

As with batched transactions, if that capability is ever published as an RFC, then the Standard Edition of our LDAP SDK (which may be used with any LDAPv3 directory server) will be updated to support that specification.

UnboundID LDAP SDK for Java 0.9.7

/ Neil Wilson

The UnboundID LDAP SDK for Java version 0.9.7 has just been released. The release notes provides information about the complete set of changes, but some of the changes since the 0.9.6 release are listed below. If you’d like to be notified by e-mail when a new release is available, send a message to ldapsdk-announce-subscribe@unboundid.com.

LDIF Reader Performance

A change introduced in the 0.9.6 release dramatically improved LDIF reading performance for large entries with attributes with a lot of values (e.g., large static groups). This change has been generalized so that it applies to entries of any size, which significantly benefits performance when reading smaller static groups.

Compact Entries

A new CompactEntry class has been added which can be used to represent a read-only entry in a form that requires much less memory than the existing Entry class, although with a potential performance penalty for some operations. This is primarily useful for the case in which it is necessary to store a large number of entries in memory, since it is possible to hold more entries in a given amount of memory. As an example, for a typical user entry, CompactEntry requires only 25% as much memory as Entry, and for a simple static group with 20 members, CompactEntry requires only 31% as much memory as Entry.

Android Demo Source

The source code for an application demonstrating the use of the UnboundID LDAP SDK for Java (mentioned in my last post) is now included.