UnboundID LDAP SDK for Java 0.9.9

The UnboundID LDAP SDK for Java version 0.9.9 has just been released. The release notes provide the complete set of changes, but some of the most significant changes since the 0.9.8 release are listed below.

Maximum Search Response Queue Size

The LDAP SDK now places a maximum size limit on the blocking queue used to hold search result entries and references while waiting for them to be processed. When using a search result listener to consume these types of responses, this can help prevent problems in which the client could potentially run out of memory if a large number of search results are returned faster than the client can process them. With this change, if the client falls behind then the LDAP SDK may eventually create back-pressure against the directory server so that it will only be able to return entries and references as fast as the client can consume them.

Referral Enhancements

It is now possible to control on a per-request basis whether the LDAP SDK should automatically try to follow any referrals returned by the server. This was previously only available on a per-connection basis. In addition, a bug has been fixed that could cause referral following to fail for search operations.

User-Friendly Names for Connections and Connection Pools

It is now possible to assign user-friendly names for individual LDAP connections or LDAP connection pools. These names can be helpful in applications which maintain multiple connections or connection pools for different purposes. These names can be queried from code, and will appear in thread stack traces for the threads used to read responses for those connections.

Debugging Enhancements

It is now possible to indicate that debug log messages should include a stack trace of the thread that generated the message. It is possible to capture a stack trace at the time that a connection is established. The debug log message format has been changed to make the messages more consistent and easier to parse. In the even that a connection is terminated because of an unrecoverable error while attempting to read data from the client, then the underlying cause for that is now more visible.

Additional Connection Pool Statistics

LDAP connection pools now expose additional statistics that can be used to keep track of how a connection was obtained when checking a connection out of the pool (e.g., whether it was obtained immediately without blocking, whether it had to wait for a connection to become available, or whether a new connection was created). This can be helpful when determining whether a connection pool is sized appropriately for the application that is using it.

UnboundID LDAP SDK for Java 0.9.8

The UnboundID LDAP SDK for Java version 0.9.8 has just been released. The release notes provide the complete set of changes, but the most significant changes since the 0.9.7 release are listed below. Because of the potential connection leak, anyone using connection pooling in the LDAP SDK is strongly encouraged to upgrade to this release.

StartTLS Bug Fix

Previous versions of the LDAP SDK had a bug in which attempting to process a StartTLS extended operation could fail to return if the server certificate was not trusted. The client connection was terminated, but the processExtendedOperation method would never return. This has been fixed.

LDAP Connection Pool Bug Fixes

Two bugs in the LDAPConnectionPool class have been corrected.

The first bug is that the connection pool could leak a connection when performing a background health check if that connection was examined multiple times in the same health check sequence.

The second bug is that when attempting to retrieve the root DSE from the directory server, two connections were obtained from the pool when only one was needed. Both connections were properly returned, but taking twice as many connections could have created an artificial connection shortage.

Operation Rate Limiting

A new FixedRateBarrier utility class has been added which can be used to help try to perform operations no faster than a specified rate. The authrate, modrate, and searchrate tools have been updated to use this and now provide a “--ratePerSecond” argument that can be used to attempt to perform authentications, modifies, and searches at a specified rate.

UnboundID LDAP SDK for Java 0.9.7

The UnboundID LDAP SDK for Java version 0.9.7 has just been released. The release notes provides information about the complete set of changes, but some of the changes since the 0.9.6 release are listed below. If you’d like to be notified by e-mail when a new release is available, send a message to ldapsdk-announce-subscribe@unboundid.com.

LDIF Reader Performance

A change introduced in the 0.9.6 release dramatically improved LDIF reading performance for large entries with attributes with a lot of values (e.g., large static groups). This change has been generalized so that it applies to entries of any size, which significantly benefits performance when reading smaller static groups.

Compact Entries

A new CompactEntry class has been added which can be used to represent a read-only entry in a form that requires much less memory than the existing Entry class, although with a potential performance penalty for some operations. This is primarily useful for the case in which it is necessary to store a large number of entries in memory, since it is possible to hold more entries in a given amount of memory. As an example, for a typical user entry, CompactEntry requires only 25% as much memory as Entry, and for a simple static group with 20 members, CompactEntry requires only 31% as much memory as Entry.

Android Demo Source

The source code for an application demonstrating the use of the UnboundID LDAP SDK for Java (mentioned in my last post) is now included.

Using LDAP on Android

Mobile devices are quickly becoming indispensable tools for convenience and productivity wherever you may happen to be. Around the beginning of the year, I started to look at the Android platform for developing applications for mobile devices (and other low-end kinds of systems). I picked up a Google developer phone (basically an unlocked version of the T-Mobile G1) and was impressed by the pretty rich API that it offers. Quite a large portion of Java SE is available for use, although there are some notable exceptions. For example, it’s missing all of the GUI classes, but that makes sense since Android provides its own GUI library. But another significant omission is that of JNDI, which is the API included in Java SE for communicating with LDAP directories.

On one hand, this is a good thing since JNDI is a horrible API that is inconvenient to use and doesn’t provide very good access to the LDAP protocol, and the less opportunity there is to use it the better. But on the other hand, being able to communicate with an LDAP directory server on a mobile device can be extremely helpful. For example, it could be very convenient to have access to the entire corporate directory at your fingertips for phone numbers and e-mail addresses without the need to clutter your contacts list, or maybe to keep local contacts in sync with their corresponding entries from the corporate directory.

The good news, though, is that Android developers don’t need to worry about missing LDAP support for Android because the UnboundID LDAP SDK for Java works just fine on the Android platform. I’ve done quite a bit of testing and made sure that things just work. With a couple of exceptions (primarily CRAM-MD5, DIGEST-MD5, and GSSAPI, which require the Java SASL API that isn’t available on Android) pretty much all of the functionality that we offer in our LDAP SDK can be used on Android. This includes all of the core protocol, all of the controls and extended operations, SSL/StartTLS, connection pooling, and all the other stuff is there, too. Note that there is no special “Android” version of the LDAP SDK — the same SDK that you can use for writing desktop clients or server-side applications works without any changes on the Android platform.

To demonstrate this capability, I’ve put together a simple LDAP client for Android devices. You can see a video online at http://www.youtube.com/watch?v=1NlhwrcCeSQ.

UnboundID LDAP SDK for Java 0.9.6

The UnboundID LDAP SDK for Java version 0.9.6 has just been released. This post details some of the changes included since the previous 0.9.5 release. See the release notes for additional information. If you’d like to be notified by e-mail when a new release is available, send a message to ldapsdk-announce-subscribe@unboundid.com.

Some of the changes in this release were based on feedback we received regarding the LDAP SDK. If you have suggestions or other feedback, feel free to send your comments to ldapsdk-support@unboundid.com.

Intermediate Response Messages

The LDAP SDK now includes support for LDAP intermediate response messages as defined in RFC 3771. Intermediate responses make it possible for operations return multiple responses to a single request. A new IntermediateResponseListener interface has been added, and clients can use the LDAPRequest.setIntermediateResponseListener method to be notified of any intermediate responses returned for the operation.

AuthRate Tool

A new authrate tool has been added as an example program. It is similar to the existing SearchRate tool, except that when retrieving an entry it can also perform a bind to verify the user’s password. It supports simple authentication, as well as the CRAM-MD5, DIGEST-MD5, and PLAIN mechanisms.

CommandLineTool Example Usages

The CommandLineTool API has been updated so that it is now possible to provide examples in the usage output. All of the example tools provided with the LDAP SDK now provide example usages.

Connection Pool Connection Expiration

It is now possible to define a maximum age for connections included in a connection pool. If a maximum age is defined and a connection in the pool has been established for greater than this length of time, then the connection will be closed and replaced with a new connection.

Connection and Connection Pool Statistics

A new LDAPConnection.getConnectionStatistics method has been added which can be used to retrieve an LDAPConnectionStatistics object with information about how that connection has been used. This includes the number of times the connection has been established and closed, the number of times each type of request has been sent, the number of times each type of response has been received, and the total and average response time for operations.

A new LDAPConnectionPool.getConnectionPoolStatistics method has been added which can be used to retrieve an LDAPConnectionPoolStatistics object which can be used to retrieve information about how the connection pool has been used. This includes the number of connections currently available, the maximum number of connections that may be available, the number of successful and failed attempts to create new connections, the number of connections that have been closed as defunct, expired, or unneeded, the number of successful and failed attempts to retrieve a connection, and the number of valid connections released back to the pool.

This feature was suggested by Philip Peake.

Connection Pool Health Checking

A new LDAPConnectionPoolHealthCheck API has been added which makes it possible to perform health checking for connections in a connection pool. It can be used to perform checks to determine whether a connection is valid when a new connection is created, checked out, released back to the pool, or after an exception has been encountered while processing an operation in the pool. In addition, a background thread will be used to periodically invoke health checks against all available connections.

A health check implementation has been added which can be used to verify the validity of a connection by retrieving a specified entry from the directory server. It can assess the connection’s availability based on whether the entry can be retrieved and the length of time required to retrieve it.

This feature was suggested by Philip Peake.

Improved Interaction with DNs and RDNs as Strings

A number of convenience methods have been added to the DN and RDN classes to make it easier to deal with them using their string representations. This includes the ability to determine whether a string represents a valid DN or RDN, to compare DNs and RDNs based on their string representations, and to determine whether strings representing DNs have a hierarchical relationship.

This feature was suggested by Felix Gaehtgens.

Improved LDIFReader Performance

The LDIFReader class has been updated to improve its performance when reading entries containing an attribute with a large number of values (e.g., a static group). For a single-threaded test case involving reading group entries with 2000 members, the new implementation was measured at over 360 times faster.

This problem was reported by Felix Gaehtgens.

LDAP SDK Features: Developer-Friendly Elements

One of the things that we’ve tried to do with the UnboundID LDAP SDK for Java is to make it as easy as possible to use. Some of this is kind of intangible (e.g., providing convenience methods for doing common things that might take multiple steps in other APIs), but there are things that I think are worth mentioning that we’ve done in this area to help developers write better applications. I’ll describe some of them here.

Extensive Javadoc Documentation

We provide a number of different types of documentation with the LDAP SDK (also available online at http://www.unboundid.com/products/ldapsdk/docs/), but I think that the most helpful documentation is the Javadoc that we generate from the source code. All public and protected methods are documented, including information about each argument and the value returned, and all public and protected fields are also documented.

Other than providing information about the available fields and methods, some of the most useful information included in the Javadoc is the class-level documentation that we provide. In many cases, this is quite extensive, and we include a number of code snippets that demonstrate how to use that class. For example, the class-level documentation for nearly all of the request types as well as the controls and extended operations provide examples for using them.

Another benefit of our Javadoc documentation is that we have generated it with the “-linksource” option, so that means that the entire source code for all of the documented classes is included as part of the documentation. This makes it possible to click on the name of a class or method in the Javadoc and be taken directly to the source code for it. In the event that you have a question about the API that can’t be answered by the Javadoc documentation itself, then using this to look at the source code itself can be very useful.

Annotations

The Javadoc documentation provides extensive information about how to use the LDAP SDK, but by itself there are some questions which a developer might have for which the answers aren’t immediately obvious. For many of these, we have defined annotations that may help make this clearer. Some of those annotations include:

  • @ThreadSafety — In many APIs, there isn’t much documentation available about the behavior of the associated code when accessed concurrently by multiple threads. To help make it clearer for our LDAP SDK, all public classes and interfaces have been marked with the @ThreadSafety annotation that indicates how those types of object may be treated in multithreaded contexts.
  • @Extensible / @NotExtensible — These annotations are used to mark interfaces and non-final classes to indicate whether they are intended to be extended by third-party code. There are some interfaces and non-final classes in the LDAP SDK which are primarily intended to be extended only by code within the LDAP SDK, and they have been marked with the @NotExtensible annotation. On the other hand, a number of interfaces and non-final classes are safe to extend by third-party code, and they are marked with the @Extensible annotation.
  • @Mutable / @NotMutable — These annotations are used to mark classes to indicate whether objects of that type are allowed to be altered by third party code. Most classes which are marked @NotMutable are fully immutable so there is no way to alter their contents at all, but there are a few cases in which it is necessary for an object to be modifiable by code within the LDAP SDK, and in those cases the associated class is marked @NotMutable to indicate that those objects shouldn’t be externally altered. For objects which may be safely altered by third-party code, the classes are marked @Mutable.
  • @InternalUseOnly — There are a few cases within the LDAP SDK where constraints enforced by Java require a class or method to be public but we don’t intend for it to be used by external code (e.g., a method required by an internal interface). In such cases, the associated class or method is marked with the @InternalUseOnly annotation to indicate that it’s not intended to be used by third-party code.

These annotations have a “runtime” retention, so they are accessible via reflection (and in fact, we have a number of unit tests to ensure that classes are marked correctly). They also appear in the generated Javadoc documentation, so that helps further improve the quality of the Javadoc.

Generics

Wherever appropriate, the LDAP SDK makes full use of generics. This is particularly true for all collections that are used anywhere in the code. This is very useful for developers because it helps them understand the types of element contained in that collection.

Debugging Support

If something goes wrong with a program using the LDAP SDK, then it may be helpful to debug the processing performed by the LDAP SDK. You can, of course, step through the code using a Java debugger, but you can also enable debugging support within the LDAP SDK itself. This uses the Java logging framework to provide messages about actions performed within the SDK, including:

  • Establishing and closing connections to directory servers
  • LDAP requests sent and responses received
  • ASN.1 elements encoded and decoded
  • LDIF entries and change records read and written
  • Exceptions caught within the LDAP SDK
  • Information about coding errors in which the LDAP SDK was used incorrectly (e.g., an argument that was inappropriately null or had a value outside of an acceptable range)

These types of debugging can be enabled or disabled on an individual basis, and you can control the minimum log level for messages that you want to see.

Debugging can be enabled or disabled programmatically by code using the LDAP SDK by calling methods in the com.unboundid.util.Debug class). However, debugging can also be controlled through the use of Java properties without the need to make any changes in the code using the LDAP SDK. The following properties are defined for use:

  • com.unboundid.ldap.sdk.debug.enabled — If this property is set to “true“, then debugging will be enabled.
  • com.unboundid.ldap.sdk.debug.level— If this property is set, then it may control the minimum debug level for messages to be written. Allowed debug levels include ALL, SEVERE, WARNING, INFO, CONFIG, FINE, FINER, FINEST, or OFF.
  • com.unboundid.ldap.sdk.debug.type — This may be set to a comma-delimited list of the types of debugging to be used within the LDAP SDK. Available debug types include ASN1, CODING_ERROR, CONNECT, EXCEPTION, LDAP, LDIF, and OTHER. If this property is not set, then all debug types will be used.

If debugging is enabled, then it will use a logger name of “com.unboundid.ldap.sdk“. By default, messages will be written to standard error, although the Java logging framework can be configured to use a custom logger to send the messages elsewhere.

Disconnect Notification

With many APIs that provide LDAP communication, if a connection to the directory server is lost, then the application may have no way of knowing until it tries to use that connection, and at that point it is often not possible to understand when the connection was closed or for what reason.

The UnboundID LDAP SDK for Java provides a com.unboundid.ldap.sdk.DisconnectHandler interface, which may be implemented by third-party code to provide a mechanism to notify an application whenever it detects that a connection has been closed. This interface defines the following method:

void handleDisconnect(LDAPConnection connection,
String host,
int port,
DisconnectType disconnectType,
String message,
Throwable cause)

Whenever a connection has been closed, the disconnect handler’s handleDisconnect method will be called to notify it of the disconnect and provide information about it. The DisconnectType element will provide information about the reason for the disconnect, and can be used to indicate whether the closure was expected (e.g., by calling connection.close() to perform an LDAP unbind). If desirable, the disconnect handler may attempt to re-establish the connection (perhaps to a different server), which may help avoid user-visible failures that may result from attempting to use the connection.

Updating SLAMD Jobs to the New API

Since the latest update to SLAMD broke backward compatibility with previous versions, then anyone who has written their own custom jobs will need to update them to work with the latest release. Here’s what you need to do in order to achieve that:

  • The overall package structure has changed from “com.sun.slamd.*” to be “com.slamd.*“. For example, if you had previously imported “com.sun.slamd.parameter.ParameterList“, then you will now need to import “com.slamd.parameter.ParameterList“.
  • The former “com.sun.slamd.example” package, which holds most of the jobs provided with SLAMD, has been renamed to “com.slamd.jobs“.
  • The code has been updated to use generics, so any collections used within SLAMD have been generified. In order to avoid build warnings, you should use generics in your code as well.
  • All cases in the code in which the StringBuffer class was previously used have been replaced with the new StringBuilder class. The StringBuilder class can help improve performance because it doesn’t perform any synchronization the way that StringBuffer does.
  • Previously, the JobClass class, which is the base class for all SLAMD jobs, had a single getJobDescription() method which could be used to obtain the description for the job. That has now been replaced with the following two methods:
    • public abstract String getShortDescription() — This should return a short (preferably a single sentence) overview of the purpose for the job. It will be used in a few places in the administrative interface, including as a pop-up hint when hovering over the name of the job on the page listing the jobs available to be scheduled. This method must be provided.
    • public String[] getLongDescription() — This may be used to return a more detailed description of the sentence. It should return a string array, and each element of the array will be treated as a separate paragraph in the administrative interface. If this method is not provided, then the short description will be used.
  • The JobClass class formerly provided a destroy() method that allowed you to provide code that would be used in an attempt to forcefully kill a job. Because the JobClass class extended Thread, it overrode the deprecated Thread.destroy() method, which could generate build warnings. I have renamed the JobClass.destroy() method to be destroyThread() to eliminate this conflict. I also made JobClass.destroy() final so that you can no longer override it in job classes.
  • The former com.sun.slamd.example.Base64Encoder class has been removed. The ability to perform base64 encoding and decoding is now provided by the com.unboundid.util.Base64 class in the UnboundID LDAP SDK for Java.
  • The former com.sun.slamd.example.ValuePattern class has been removed. This capability is now provided by the com.unboundid.util.ValuePattern class in the UnboundID LDAP SD for Java. The com.sun.slamd.example.NumericRange class, which had been used by the old ValuePattern implementation has also been removed.

SLAMD 2.0.0-20090227

After quite a significant hiatus, I have just released a new version of SLAMD, which I am calling 2.0.0-20090227. It isn’t the official 2.0.0 release, but it should be pretty solid, and there are significant improvements over the previous 2.0.0-alpha1 build. You can get it at http://www.slamd.com/, and the source is hosted in a subversion repository at http://slamd2.dev.java.net/.

This is the first public build of SLAMD since I left Sun, but the SLAMD codebase hasn’t exactly been collecting dust since then. One of the first things that I did when we founded UnboundID was to start working on the UnboundID LDAP SDK for Java, and within a few weeks of that I was already using it in SLAMD jobs. However, since until recently our LDAP SDK was not publicly available, I couldn’t really release a version of SLAMD that included it, and I didn’t want to maintain a separate private version of the code that made use of the SDK and a public version that didn’t. Now that our LDAP SDK is open source and publicly available, I’m able to use it in SLAMD without any concerns.

Since it has been over two years since the last public commit, and since I had problems with an older mail server, I have recreated all of the SLAMD mailing lists without any members. If you were previously subscribed to any of the mailing lists and want to keep it that way, then you’ll need to re-subscribe to the desired lists. Instructions for doing so are available at http://www.slamd.com/mailing-lists.shtml.

Note that a number of changes have been made in this release that make it incompatible with earlier releases, and you probably won’t be able to export data from an older server and import it into this new version and have it all work seamlessly. I think that the enhancements in this version were worth it, especially if you’re using it to test LDAP directory servers, and hopefully the incompatibilities introduced now will help avoid the need to introduce further incompatibilities in the future.

Changes Since SLAMD 2.0.0-alpha1

There have been a lot of changes made to SLAMD since the 2.0.0-alpha1 release. Some of the most significant changes are outlined below. There have also been a number of less significant bugfixes and enhancements that probably aren’t worth mentioning individually.

Source Code Refactoring

The source code has been completely refactored in several ways. The code now uses a simplified package structure of “com.slamd.*“. The “example” sub-package has been renamed to “jobs“. All of the source code is now in one single directory structure rather than having separate source locations for a number of the tools.

The source code has also been fully updated to use Java 5 functionality. As a result, SLAMD now requires that you use Java 5.0 or later (I recommend the latest Java 6 build). The code has been updated to use generics and builds cleanly with all lint options enabled. As a result of updating to Java 5, I have also been able to take advantage of the better concurrency support and more accurate timing that it provides.

The job API has been updated so that it is now possible to provide both long and short descriptions. Further, where it previously used the destroy() method as a means of trying to forcefully stop a job, I have changed that to destroyThread(), since the former method overrode the deprecated Thread.destroy() method which I shouldn’t have done in the first place.

Job Updates

As mentioned above, most of the LDAP-based jobs now use the UnboundID LDAP SDK for Java. This provides better performance and scalability than the Netscape Directory SDK that I had previously been using, and it’s also much more stable under load. All of the LDAP-based jobs using the UnboundID LDAP SDK now extend a common parent class which provides a number of benefits for these jobs, including:

  • It is now possible to perform client-side load balancing so that you can stress multiple servers at once with a single job.
  • All of these jobs support SSL and StartTLS for secure communication with the directory server, and I’ve simplified the interface so that it’s no longer necessary to worry about key or trust stores.
  • These jobs offer the ability to specify a response time threshold, which can be used to keep track of the number of times that an operation took longer than the specified length of time to complete.

I have consolidated many of the LDAP-based jobs where it made sense to do so, so that there are no longer as many jobs but you should still have the same basic set of capabilities, in addition to new features.

The LDAP Add and Delete Rate job now provides the ability to control how the operations are performed. You can choose to perform only adds, only deletes, or both adds and deletes. If you choose to perform both, then you can either have all of the adds done first and then all of the deletes, or you can choose to delete each entry immediately after adding it.

The LDAP AuthRate job now supports the ability to authenticate using the CRAM-MD5, DIGEST-MD5, or PLAIN SASL mechanisms in addition to using simple authentication.

The LDAP ModRate job allows you to specify the set of characters to use to create the modification, so it will work better with attributes that syntaxes that don’t accept arbitrary characters.

I’ve added a new LDAP Modify DN Rate job which can be used to measure modify DN performance. You can specify a range of entries to be renamed, and the job will actually rename them twice. The first time it will rename them to include the job ID in the RDN, and the second time it will rename them back to the original value.

I’ve added a new LDAP Multi-Connection SearchRate job which allows you to perform searches while maintaining large numbers of connections to the directory in order to test performance with lots of active connections.

I’ve added a new “Wait for Directory” job which allows you to wait for a directory server to become available and start accepting requests before continuing on with other jobs that need to communicate with the server.

I have gotten rid of some older jobs that hadn’t been updated in a while and that targeted outdated software or software that I wasn’t in a position to support. This primarily included the jobs targeting an ancient version of the Sun Calendar Server, old versions of the Access Manager product, and the Identity Synchronization for Windows. If this is a significant problem, then it shouldn’t be difficult to resurrect them, but I didn’t want to invest the time cleaning them up and I wasn’t in any position to test them to see if they still work properly.

Resource Monitor Updates

A new resource monitor has been provided that makes it possible to capture arbitrary information from an LDAP directory server that exposes monitoring information. This was written by Bertold Kolics and it is very useful because it can be used to collect all kinds of monitoring information about the state of the directory server while a job is running (if the directory that you’re targeting exposes the desired information).

I’ve also fixed some bugs in a couple of the resource monitors. In particular, the VMStat monitor has been updated to support newer versions of Linux, and the IOStat monitor has been updated to fix a problem where it didn’t always work properly on Solaris.

Other Updates

I have updated the TimeTracker so that it uses the high-resolution timer which can provide up to nanosecond-level accuracy. Previously the TimeTracker only supported millisecond-level accuracy, but this wasn’t fine-grained enough for timing things that took less than a few milliseconds to complete.

The SLAMD client and resource monitor clients have been updated so that it is now possible to explicitly specify the client ID and source IP address to use when communicating with the SLAMD server. This is particularly useful on multi-homed systems, especially when they might have multiple IP addresses on the same network.

I have updated the job scheduler so that it now uses a blocking queue that allows a new job to be seen immediately. If it’s eligible to start running at the time that it’s scheduled, then it will often be the case that the job will already be running before the administrative interface renders the page indicating that the job has been scheduled.

I have changed the layout of the tools directory so that instead of providing separate subdirectories for each of the major tools, the code for each of those tools is integrated into the main codebase and now there are just shell scripts and batch files for launching them. I have also added a new Tools Guide that summarizes the tools that are available and provides information about using them.

The UnboundID LDAP SDK for Java provides searchrate, modrate, and authrate command-line tools as example programs, and they are now exposed as tools available for use within SLAMD. They are also available through shell scripts or batch files in the tools directory.

I have made some changes to the way that statistics are displayed in the administrative interface so that they are less likely to require horizontal scrolling.

The descriptions for the available jobs have been improved, and it’s now easier to tell what a job does when you schedule it. On the page allowing you to pick which type of job to schedule, SLAMD now provides a pop-up hint that gives a short description of the job. The page allowing you to provide the parameters to use when scheduling a job has also been updated to provide a more detailed description of the job at the top of the page.

LDAP SDK Features: LDIF Processing

The LDAP Data Interchange Format (LDIF) is a standard way of representing directory data in plain text files, as defined in RFC 2849.  LDIF records can represent entire entires, or they can represent changes to be made to directory contents (add new entries, or remove, modify, or rename existing entries).  The UnboundID LDAP SDK for Java provides rich support for interacting with LDIF records, in many ways.  This post will describe the capabilities that it offers in this area.

Reading LDIF Records

The com.unboundid.ldif.LDIFReader class provides methods for reading data from LDIF files or from input streams.  Some of the methods it offers are:

  • readEntry() — This reads the next entry from the LDIF file or input stream.  This should be used when reading LDIF data known to contain only entries.
  • readChangeRecord() — This reads the next change record from the LDIF file or input stream.  This should only be used when reading LDIF data known to contain only change records.
  • readLDIFRecord() — This reads the next entry or change record from the LDIF file or input stream (both entries and change records implement the LDIFRecord interface).  This should be used when reading LDIF data that may contain a mix of entries and change records.
  • decodeEntry(String...) — This decodes the contents of the provided array as an LDIF entry.  The elements of the array should be the lines of LDIF entry to be parsed.  This is a static method and doesn’t require an LDIFReader object to be created to be able to use it.
  • decodeChangeRecord(String...) — This decodes the contents of the provided array as an LDIF change record.  The elements of the array should be the lines of the LDIF change record to be parsed.  This is a static method and doesn’t require an LDIFReader object to be created to be able to use it.

When reading and parsing LDIF records, if an invalid record is encountered then the LDIF reader will throw an LDIFException.  It will include a message about the problem encountered, the line number in the LDIF data on which the problem was encountered, and a flag indicating whether or not it is possible to continue reading data from the LDIF source.  Methods used to read LDIF data from a file or input stream may also throw an IOException.

Writing LDIF Records

The com.unboundid.ldif.LDIFWriter class provides methods for writing LDIF content to files or output streams.  Some of those methods include:

  • writeEntry(Entry) — This method writes the provided entry to the LDIF file or output stream.
  • writeEntry(Entry,String) — This method writes the provided entry to the LDIF file or output stream, immediately preceded by the specified comment.
  • writeChangeRecord(LDIFChangeRecord) — This method writes the provided change record to the LDIF file or output stream.
  • writeChangeRecord(LDIFChangeRecord,String) — This method writes the provided change record to the LDIF file or output stream, immediately preceded by the specified comment.
  • writeLDIFRecord(LDIFRecord) — This method writes the given LDIF record (which may either be an entry or change record) to the LDIF file or output stream.
  • writeLDIFRecord(LDIFRecord,String) — This method writes the given LDIF record (which may either be an entry or change record) to the LDIF file or output stream, immediately preceded by the specified comment.

Methods used for writing LDIF data may throw an IOException.

Parallel LDIF Processing

We’ve worked hard to make the process of reading and writing LDIF data as fast as possible.  While it’s pretty fast on its own as a serial process, portions of the processing may be parallelized for better performance on multi-CPU and/or multi-core systems.

When reading LDIF data, there are two phases of processing:  the process of reading raw data from the LDIF file or input stream, and the process of decoding that data as an entry or change record.  The first phase must be performed serially, but the second can be parallelized and multiple concurrent threads may be used to decode LDIF records.  As a result, when the LDIF reader is configured to use multiple concurrent threads, it may be the case that the limiting factor is the speed at which data can be read from the disk or input stream.

In order to use parallelism in the LDIF reader, it is only necessary to specify the number of additional threads to use when parsing entries or change records at the time the reader is created.  This parallelism will be provided completely behind the scenes, so that the caller may continue to use the readEntry(), readChangeRecord(), and readLDIFRecord() methods just as if all processing were performed serially.

Of course, introducing parallelism in the LDIF reader would be of limited usefulness if it introduced the possibility that entries or change records might be made available to the caller in a different order than they were contained in the original LDIF data.  As a result, the LDIF reader will ensure that the order of the data is preserved so that the only perceptible change between serial and parallel processing is the relative speed with which LDIF records may be made available to the caller.

The LDIF writer also provides support for parallel processing, in which case multiple entries or change records may be formatted into their LDIF representations in parallel, and then will be written serially to the LDIF file or output stream. As with the LDIF reader, parallelism may be enabled by specifying the number of threads to use to perform that processing at the time that the LDIF writer is created. However, parallel processing will only be performed in the LDIF writer if the entries or change records are provided to it using the writeLDIFRecords(List<? extends LDIFRecord>) method. The order that the records are provided in the list will be preserved when they are written to the LDIF file or output stream.

Transforming Entries Read from LDIF

When reading entries from an LDIF file or input stream, it may be useful to alter those entries in some way before they are made available to the caller, and in some cases it may be desirable to exclude entries altogether.  Either or both of these may be performed by providing a class which implements the com.unboundid.ldif.LDIFReaderEntryTranslator interface to the LDIF reader. If an entry translator is provided, then whenever an entry is read, the translate(Entry,long) method will be invoked to allow the translator to return a modified version of the provided entry, a completely new entry, or null to indicate that the entry should be omitted from the data made available to callers.

If the LDIF reader has been configured to perform parallel processing, then the entry translator will be invoked during the parallel portion of that processing, and as a result it may be faster to perform this transformation using the LDIFReaderEntryTranslator interface than performing it separately after the entry has been retrieved by the caller using a method like readEntry(). This does require the translator to be threadsafe, and it cannot depend on the order in which entries are processed.

Obtaining LDIF Representations of LDAP SDK Objects

There are a number of objects provided as part of the LDAP SDK which can be represented in LDIF form without the need for an LDIF writer. This includes the following types of objects:

  • com.unboundid.ldap.sdk.Entry — This can be represented as an LDIF entry.
  • com.unboundid.ldap.sdk.AddRequest — This can be represented as an LDIF add change record.
  • com.unboundid.ldap.sdk.DeleteRequest — This can be represented as an LDIF delete change record.
  • com.unboundid.ldap.sdk.ModifyRequest — This can be represented as an LDIF modify change record.
  • com.unboundid.ldap.sdk.ModifyDNRequest — This can be represented as an LDIF modify DN change record.

All of these objects provide the following methods that allow them to be represented in LDIF form:

  • toLDIF() — This returns a string array whose elements comprise the lines of the LDIF representation of the associated record.
  • toLDIFString() — This returns a string containing the LDIF representation of the associated record, including line breaks.

In addition, the above objects except Entry provide a toLDIFChangeRecord() method that allow them to be converted to the appropriate type of LDIF change record.

Using LDIF Records to Process Operations

The LDAP SDK also provides support for processing operations based on the LDIF representation.  There are two primary ways that this may be accomplished:

  • All types of the LDIF change records provide a processChange(LDAPConnection) method which can be used to directly process an operation from that change record.  Each change record type also provides a method for converting that change record object to an LDAP request (e.g., the LDIFAddChangeRecord class provides a toAddRequest() method to convert the change record to an equivalent add request).
  • Entries can be created from their LDIF representation using the Entry(String...) constructor, where the provided string array contains the lines that comprise the LDIF representation of that entry. Add requests can also be created from the LDIF representation of the entry to add using the AddRequest(String...) constructor. Modify requests can also be created from the LDIF representation of that modify request using the ModifyRequest(String...) constructor.

LDAP SDK Features: Client-Side Processing

When writing a directory-enabled application, you can let the directory server do a lot of the work for you, and in many cases that’s appropriate.  However, there are times when it might be better to do some of that processing on the client rather than the server (e.g., because of the performance hit of the additional requests, or because some target servers might not support all of the features you want to use).  In other cases it might be useful to perform client-side validation prior to sending a request to the server to reduce the likelihood of a failure or to provide better information about what needs to be fixed.  The UnboundID LDAP SDK for Java provides support for several types of client side processing, as described in this post.

Comparing Entries

The com.unboundid.ldap.sdk.Entry class includes a diff method that can be used to compare two entries. The complete definition for this method is:

public static List<Modification> diff(Entry sourceEntry,
Entry targetEntry, boolean ignoreRDN, String... attributes)

It can be used to compare the contents of the two provided entries and will return a list of the modifications that can be applied to the source entry in order to make it look like the target. You can optionally ignore differences in the RDN attribute, and you can optionally restrict the comparison to a specified set of attributes.

There are many uses for this capability. For example, if you have an application that allows a user to alter the contents of entries, then this method may be used to compare a local copy of the updated entry with the original entry in order to determine the modifications to apply to the server. Alternately, you could use this to compare an entry in two different servers to determine whether they are in sync.

Schema Validation

The LDAP SDK provides enhanced support for parsing server schema definitions, and can use that to perform a number of types of client-side processing. One such capability is exposed through the com.unboundid.ldap.sdk.schema.EntryValidator class. This class may be initialized with schema read from the server and provides the following method:

public boolean entryIsValid(Entry entry, List<String> invalidReasons)

This method may be used to determine whether the given entry conforms to the schema constraints defined in the server. The basic checks that it can perform include:

  • Ensure that the entry has a valid DN
  • Ensure that the entry has exactly one structural object class
  • Ensure that all of the entry’s object classes are defined in the schema
  • Ensure that all of the entry’s attributes are defined in the schema
  • Ensure that all of the attributes required by any of the entry’s object classes are present
  • Ensure that all of the attributes contained in the entry are allowed by at least one of that entry’s object classes
  • Ensure that all of the attribute values conform to the syntax defined for that attribute
  • Ensure that all of the attributes with multiple values are defined as multi-valued in the server schema

In addition, if there is a DIT content rule associated with the entry’s structural object class, then it will be used to ensure that all of the auxiliary object classes in the entry are allowed and that the entry doesn’t contain any prohibited attributes, and it may also allow additional attributes not allowed by any of the object classes. If there is a name form associated with the entry’s structural object class, then the entry validator will use it to ensure that the entry’s RDN includes all of the required attributes, and that all attributes included in the entry’s RDN are allowed.

All of these checks may be individually enabled or disabled, which makes it possible to tailor the types of validation to perform. In addition, the entryIsValid method is threadsafe, so it can be called concurrently by multiple threads. If the entry is invalid, then this method will return false, and the provided list will include one or more human-readable messages detailing the problems identified with the entry. It will also collect summary statistics about all of the entries examined so that if a number of entries are processed it is easy to understand the types of problems (if any) that were encountered.

One of the example programs provided with the LDAP SDK is a validate-ldif tool which uses the EntryValidator class to perform this validation for all entries in an LDIF file. This can provide a useful way to validate that the entries contained in an LDIF file are valid without needing to actually import the data into a server. If there are any problems, then the summary provided by the tool may be easier to interpret than the errors reported by the server.

Sorting

Many directory servers provide support for the server-side sort control as defined in RFC 2891. However, some servers may not support this control, and of the servers that do support it, some of them may require special configuration or indexing, and the client may require special permissions to be allowed to request it. In addition, for small result sets, it may be significantly more efficient (and less resource-intensive on the server) to perform sorting on the client rather than asking the server to do it.

The UnboundID LDAP SDK for Java provides a com.unboundid.ldap.sdk.EntrySorter class that can be used to accomplish this. It includes the following method that makes it possible to sort a collection of entries:

public SortedSet<Entry> sort(Collection<? extends Entry> entries)

This class also implements the Comparator<Entry> interface, so it can be used with any type of Java collection that can use comparators.

It is possible to define the sort criteria, including specifying which attributes should be used and indicating whether to sort in reverse order. It can also be configured to take the entry’s position in the DIT into consideration when sorting so that entries can be sorted by hierarchy instead of or in addition to performing attribute-based sorting.

For best results, the entry sorter is able to use schema read from the directory server in order to better understand which matching rules it should use for sorting, but if no schema is available it will simply assume that all sorting should be done using case-ignore ordering.

Filter Evaluation

There are a number of cases in which it might be useful to determine whether an entry matches a given set of criteria, and search filters provide a convenient way to express that criteria. If the entry exists in a directory server, then of course a search operation may be used to make the determination, but if the client has a local copy of the entry then it may be more convenient or efficient to perform the evaluation on the client. Further, there may be cases (e.g., for an entry read from an LDIF file) in which the entry isn’t available in a server. To provide this capability, the com.unboundid.ldap.sdk.Filter class includes the following methods:

public boolean matchesEntry(Entry entry)
public boolean matchesEntry(Entry entry, Schema schema)

If a schema object is provided, then it will be used to perform more accurate evaluation, but if no schema is available then it will work using case-ignore matching. This method currently supports all types of filter evaluation except approximate and extensible matching.