LDAP SDK Features: Developer-Friendly Elements

One of the things that we’ve tried to do with the UnboundID LDAP SDK for Java is to make it as easy as possible to use. Some of this is kind of intangible (e.g., providing convenience methods for doing common things that might take multiple steps in other APIs), but there are things that I think are worth mentioning that we’ve done in this area to help developers write better applications. I’ll describe some of them here.

Extensive Javadoc Documentation

We provide a number of different types of documentation with the LDAP SDK (also available online at http://www.unboundid.com/products/ldapsdk/docs/), but I think that the most helpful documentation is the Javadoc that we generate from the source code. All public and protected methods are documented, including information about each argument and the value returned, and all public and protected fields are also documented.

Other than providing information about the available fields and methods, some of the most useful information included in the Javadoc is the class-level documentation that we provide. In many cases, this is quite extensive, and we include a number of code snippets that demonstrate how to use that class. For example, the class-level documentation for nearly all of the request types as well as the controls and extended operations provide examples for using them.

Another benefit of our Javadoc documentation is that we have generated it with the “-linksource” option, so that means that the entire source code for all of the documented classes is included as part of the documentation. This makes it possible to click on the name of a class or method in the Javadoc and be taken directly to the source code for it. In the event that you have a question about the API that can’t be answered by the Javadoc documentation itself, then using this to look at the source code itself can be very useful.

Annotations

The Javadoc documentation provides extensive information about how to use the LDAP SDK, but by itself there are some questions which a developer might have for which the answers aren’t immediately obvious. For many of these, we have defined annotations that may help make this clearer. Some of those annotations include:

  • @ThreadSafety — In many APIs, there isn’t much documentation available about the behavior of the associated code when accessed concurrently by multiple threads. To help make it clearer for our LDAP SDK, all public classes and interfaces have been marked with the @ThreadSafety annotation that indicates how those types of object may be treated in multithreaded contexts.
  • @Extensible / @NotExtensible — These annotations are used to mark interfaces and non-final classes to indicate whether they are intended to be extended by third-party code. There are some interfaces and non-final classes in the LDAP SDK which are primarily intended to be extended only by code within the LDAP SDK, and they have been marked with the @NotExtensible annotation. On the other hand, a number of interfaces and non-final classes are safe to extend by third-party code, and they are marked with the @Extensible annotation.
  • @Mutable / @NotMutable — These annotations are used to mark classes to indicate whether objects of that type are allowed to be altered by third party code. Most classes which are marked @NotMutable are fully immutable so there is no way to alter their contents at all, but there are a few cases in which it is necessary for an object to be modifiable by code within the LDAP SDK, and in those cases the associated class is marked @NotMutable to indicate that those objects shouldn’t be externally altered. For objects which may be safely altered by third-party code, the classes are marked @Mutable.
  • @InternalUseOnly — There are a few cases within the LDAP SDK where constraints enforced by Java require a class or method to be public but we don’t intend for it to be used by external code (e.g., a method required by an internal interface). In such cases, the associated class or method is marked with the @InternalUseOnly annotation to indicate that it’s not intended to be used by third-party code.

These annotations have a “runtime” retention, so they are accessible via reflection (and in fact, we have a number of unit tests to ensure that classes are marked correctly). They also appear in the generated Javadoc documentation, so that helps further improve the quality of the Javadoc.

Generics

Wherever appropriate, the LDAP SDK makes full use of generics. This is particularly true for all collections that are used anywhere in the code. This is very useful for developers because it helps them understand the types of element contained in that collection.

Debugging Support

If something goes wrong with a program using the LDAP SDK, then it may be helpful to debug the processing performed by the LDAP SDK. You can, of course, step through the code using a Java debugger, but you can also enable debugging support within the LDAP SDK itself. This uses the Java logging framework to provide messages about actions performed within the SDK, including:

  • Establishing and closing connections to directory servers
  • LDAP requests sent and responses received
  • ASN.1 elements encoded and decoded
  • LDIF entries and change records read and written
  • Exceptions caught within the LDAP SDK
  • Information about coding errors in which the LDAP SDK was used incorrectly (e.g., an argument that was inappropriately null or had a value outside of an acceptable range)

These types of debugging can be enabled or disabled on an individual basis, and you can control the minimum log level for messages that you want to see.

Debugging can be enabled or disabled programmatically by code using the LDAP SDK by calling methods in the com.unboundid.util.Debug class). However, debugging can also be controlled through the use of Java properties without the need to make any changes in the code using the LDAP SDK. The following properties are defined for use:

  • com.unboundid.ldap.sdk.debug.enabled — If this property is set to “true“, then debugging will be enabled.
  • com.unboundid.ldap.sdk.debug.level— If this property is set, then it may control the minimum debug level for messages to be written. Allowed debug levels include ALL, SEVERE, WARNING, INFO, CONFIG, FINE, FINER, FINEST, or OFF.
  • com.unboundid.ldap.sdk.debug.type — This may be set to a comma-delimited list of the types of debugging to be used within the LDAP SDK. Available debug types include ASN1, CODING_ERROR, CONNECT, EXCEPTION, LDAP, LDIF, and OTHER. If this property is not set, then all debug types will be used.

If debugging is enabled, then it will use a logger name of “com.unboundid.ldap.sdk“. By default, messages will be written to standard error, although the Java logging framework can be configured to use a custom logger to send the messages elsewhere.

Disconnect Notification

With many APIs that provide LDAP communication, if a connection to the directory server is lost, then the application may have no way of knowing until it tries to use that connection, and at that point it is often not possible to understand when the connection was closed or for what reason.

The UnboundID LDAP SDK for Java provides a com.unboundid.ldap.sdk.DisconnectHandler interface, which may be implemented by third-party code to provide a mechanism to notify an application whenever it detects that a connection has been closed. This interface defines the following method:

void handleDisconnect(LDAPConnection connection,
String host,
int port,
DisconnectType disconnectType,
String message,
Throwable cause)

Whenever a connection has been closed, the disconnect handler’s handleDisconnect method will be called to notify it of the disconnect and provide information about it. The DisconnectType element will provide information about the reason for the disconnect, and can be used to indicate whether the closure was expected (e.g., by calling connection.close() to perform an LDAP unbind). If desirable, the disconnect handler may attempt to re-establish the connection (perhaps to a different server), which may help avoid user-visible failures that may result from attempting to use the connection.

Updating SLAMD Jobs to the New API

Since the latest update to SLAMD broke backward compatibility with previous versions, then anyone who has written their own custom jobs will need to update them to work with the latest release. Here’s what you need to do in order to achieve that:

  • The overall package structure has changed from “com.sun.slamd.*” to be “com.slamd.*“. For example, if you had previously imported “com.sun.slamd.parameter.ParameterList“, then you will now need to import “com.slamd.parameter.ParameterList“.
  • The former “com.sun.slamd.example” package, which holds most of the jobs provided with SLAMD, has been renamed to “com.slamd.jobs“.
  • The code has been updated to use generics, so any collections used within SLAMD have been generified. In order to avoid build warnings, you should use generics in your code as well.
  • All cases in the code in which the StringBuffer class was previously used have been replaced with the new StringBuilder class. The StringBuilder class can help improve performance because it doesn’t perform any synchronization the way that StringBuffer does.
  • Previously, the JobClass class, which is the base class for all SLAMD jobs, had a single getJobDescription() method which could be used to obtain the description for the job. That has now been replaced with the following two methods:
    • public abstract String getShortDescription() — This should return a short (preferably a single sentence) overview of the purpose for the job. It will be used in a few places in the administrative interface, including as a pop-up hint when hovering over the name of the job on the page listing the jobs available to be scheduled. This method must be provided.
    • public String[] getLongDescription() — This may be used to return a more detailed description of the sentence. It should return a string array, and each element of the array will be treated as a separate paragraph in the administrative interface. If this method is not provided, then the short description will be used.
  • The JobClass class formerly provided a destroy() method that allowed you to provide code that would be used in an attempt to forcefully kill a job. Because the JobClass class extended Thread, it overrode the deprecated Thread.destroy() method, which could generate build warnings. I have renamed the JobClass.destroy() method to be destroyThread() to eliminate this conflict. I also made JobClass.destroy() final so that you can no longer override it in job classes.
  • The former com.sun.slamd.example.Base64Encoder class has been removed. The ability to perform base64 encoding and decoding is now provided by the com.unboundid.util.Base64 class in the UnboundID LDAP SDK for Java.
  • The former com.sun.slamd.example.ValuePattern class has been removed. This capability is now provided by the com.unboundid.util.ValuePattern class in the UnboundID LDAP SD for Java. The com.sun.slamd.example.NumericRange class, which had been used by the old ValuePattern implementation has also been removed.

SLAMD 2.0.0-20090227

After quite a significant hiatus, I have just released a new version of SLAMD, which I am calling 2.0.0-20090227. It isn’t the official 2.0.0 release, but it should be pretty solid, and there are significant improvements over the previous 2.0.0-alpha1 build. You can get it at http://www.slamd.com/, and the source is hosted in a subversion repository at http://slamd2.dev.java.net/.

This is the first public build of SLAMD since I left Sun, but the SLAMD codebase hasn’t exactly been collecting dust since then. One of the first things that I did when we founded UnboundID was to start working on the UnboundID LDAP SDK for Java, and within a few weeks of that I was already using it in SLAMD jobs. However, since until recently our LDAP SDK was not publicly available, I couldn’t really release a version of SLAMD that included it, and I didn’t want to maintain a separate private version of the code that made use of the SDK and a public version that didn’t. Now that our LDAP SDK is open source and publicly available, I’m able to use it in SLAMD without any concerns.

Since it has been over two years since the last public commit, and since I had problems with an older mail server, I have recreated all of the SLAMD mailing lists without any members. If you were previously subscribed to any of the mailing lists and want to keep it that way, then you’ll need to re-subscribe to the desired lists. Instructions for doing so are available at http://www.slamd.com/mailing-lists.shtml.

Note that a number of changes have been made in this release that make it incompatible with earlier releases, and you probably won’t be able to export data from an older server and import it into this new version and have it all work seamlessly. I think that the enhancements in this version were worth it, especially if you’re using it to test LDAP directory servers, and hopefully the incompatibilities introduced now will help avoid the need to introduce further incompatibilities in the future.

Changes Since SLAMD 2.0.0-alpha1

There have been a lot of changes made to SLAMD since the 2.0.0-alpha1 release. Some of the most significant changes are outlined below. There have also been a number of less significant bugfixes and enhancements that probably aren’t worth mentioning individually.

Source Code Refactoring

The source code has been completely refactored in several ways. The code now uses a simplified package structure of “com.slamd.*“. The “example” sub-package has been renamed to “jobs“. All of the source code is now in one single directory structure rather than having separate source locations for a number of the tools.

The source code has also been fully updated to use Java 5 functionality. As a result, SLAMD now requires that you use Java 5.0 or later (I recommend the latest Java 6 build). The code has been updated to use generics and builds cleanly with all lint options enabled. As a result of updating to Java 5, I have also been able to take advantage of the better concurrency support and more accurate timing that it provides.

The job API has been updated so that it is now possible to provide both long and short descriptions. Further, where it previously used the destroy() method as a means of trying to forcefully stop a job, I have changed that to destroyThread(), since the former method overrode the deprecated Thread.destroy() method which I shouldn’t have done in the first place.

Job Updates

As mentioned above, most of the LDAP-based jobs now use the UnboundID LDAP SDK for Java. This provides better performance and scalability than the Netscape Directory SDK that I had previously been using, and it’s also much more stable under load. All of the LDAP-based jobs using the UnboundID LDAP SDK now extend a common parent class which provides a number of benefits for these jobs, including:

  • It is now possible to perform client-side load balancing so that you can stress multiple servers at once with a single job.
  • All of these jobs support SSL and StartTLS for secure communication with the directory server, and I’ve simplified the interface so that it’s no longer necessary to worry about key or trust stores.
  • These jobs offer the ability to specify a response time threshold, which can be used to keep track of the number of times that an operation took longer than the specified length of time to complete.

I have consolidated many of the LDAP-based jobs where it made sense to do so, so that there are no longer as many jobs but you should still have the same basic set of capabilities, in addition to new features.

The LDAP Add and Delete Rate job now provides the ability to control how the operations are performed. You can choose to perform only adds, only deletes, or both adds and deletes. If you choose to perform both, then you can either have all of the adds done first and then all of the deletes, or you can choose to delete each entry immediately after adding it.

The LDAP AuthRate job now supports the ability to authenticate using the CRAM-MD5, DIGEST-MD5, or PLAIN SASL mechanisms in addition to using simple authentication.

The LDAP ModRate job allows you to specify the set of characters to use to create the modification, so it will work better with attributes that syntaxes that don’t accept arbitrary characters.

I’ve added a new LDAP Modify DN Rate job which can be used to measure modify DN performance. You can specify a range of entries to be renamed, and the job will actually rename them twice. The first time it will rename them to include the job ID in the RDN, and the second time it will rename them back to the original value.

I’ve added a new LDAP Multi-Connection SearchRate job which allows you to perform searches while maintaining large numbers of connections to the directory in order to test performance with lots of active connections.

I’ve added a new “Wait for Directory” job which allows you to wait for a directory server to become available and start accepting requests before continuing on with other jobs that need to communicate with the server.

I have gotten rid of some older jobs that hadn’t been updated in a while and that targeted outdated software or software that I wasn’t in a position to support. This primarily included the jobs targeting an ancient version of the Sun Calendar Server, old versions of the Access Manager product, and the Identity Synchronization for Windows. If this is a significant problem, then it shouldn’t be difficult to resurrect them, but I didn’t want to invest the time cleaning them up and I wasn’t in any position to test them to see if they still work properly.

Resource Monitor Updates

A new resource monitor has been provided that makes it possible to capture arbitrary information from an LDAP directory server that exposes monitoring information. This was written by Bertold Kolics and it is very useful because it can be used to collect all kinds of monitoring information about the state of the directory server while a job is running (if the directory that you’re targeting exposes the desired information).

I’ve also fixed some bugs in a couple of the resource monitors. In particular, the VMStat monitor has been updated to support newer versions of Linux, and the IOStat monitor has been updated to fix a problem where it didn’t always work properly on Solaris.

Other Updates

I have updated the TimeTracker so that it uses the high-resolution timer which can provide up to nanosecond-level accuracy. Previously the TimeTracker only supported millisecond-level accuracy, but this wasn’t fine-grained enough for timing things that took less than a few milliseconds to complete.

The SLAMD client and resource monitor clients have been updated so that it is now possible to explicitly specify the client ID and source IP address to use when communicating with the SLAMD server. This is particularly useful on multi-homed systems, especially when they might have multiple IP addresses on the same network.

I have updated the job scheduler so that it now uses a blocking queue that allows a new job to be seen immediately. If it’s eligible to start running at the time that it’s scheduled, then it will often be the case that the job will already be running before the administrative interface renders the page indicating that the job has been scheduled.

I have changed the layout of the tools directory so that instead of providing separate subdirectories for each of the major tools, the code for each of those tools is integrated into the main codebase and now there are just shell scripts and batch files for launching them. I have also added a new Tools Guide that summarizes the tools that are available and provides information about using them.

The UnboundID LDAP SDK for Java provides searchrate, modrate, and authrate command-line tools as example programs, and they are now exposed as tools available for use within SLAMD. They are also available through shell scripts or batch files in the tools directory.

I have made some changes to the way that statistics are displayed in the administrative interface so that they are less likely to require horizontal scrolling.

The descriptions for the available jobs have been improved, and it’s now easier to tell what a job does when you schedule it. On the page allowing you to pick which type of job to schedule, SLAMD now provides a pop-up hint that gives a short description of the job. The page allowing you to provide the parameters to use when scheduling a job has also been updated to provide a more detailed description of the job at the top of the page.

LDAP SDK Features: LDIF Processing

The LDAP Data Interchange Format (LDIF) is a standard way of representing directory data in plain text files, as defined in RFC 2849.  LDIF records can represent entire entires, or they can represent changes to be made to directory contents (add new entries, or remove, modify, or rename existing entries).  The UnboundID LDAP SDK for Java provides rich support for interacting with LDIF records, in many ways.  This post will describe the capabilities that it offers in this area.

Reading LDIF Records

The com.unboundid.ldif.LDIFReader class provides methods for reading data from LDIF files or from input streams.  Some of the methods it offers are:

  • readEntry() — This reads the next entry from the LDIF file or input stream.  This should be used when reading LDIF data known to contain only entries.
  • readChangeRecord() — This reads the next change record from the LDIF file or input stream.  This should only be used when reading LDIF data known to contain only change records.
  • readLDIFRecord() — This reads the next entry or change record from the LDIF file or input stream (both entries and change records implement the LDIFRecord interface).  This should be used when reading LDIF data that may contain a mix of entries and change records.
  • decodeEntry(String...) — This decodes the contents of the provided array as an LDIF entry.  The elements of the array should be the lines of LDIF entry to be parsed.  This is a static method and doesn’t require an LDIFReader object to be created to be able to use it.
  • decodeChangeRecord(String...) — This decodes the contents of the provided array as an LDIF change record.  The elements of the array should be the lines of the LDIF change record to be parsed.  This is a static method and doesn’t require an LDIFReader object to be created to be able to use it.

When reading and parsing LDIF records, if an invalid record is encountered then the LDIF reader will throw an LDIFException.  It will include a message about the problem encountered, the line number in the LDIF data on which the problem was encountered, and a flag indicating whether or not it is possible to continue reading data from the LDIF source.  Methods used to read LDIF data from a file or input stream may also throw an IOException.

Writing LDIF Records

The com.unboundid.ldif.LDIFWriter class provides methods for writing LDIF content to files or output streams.  Some of those methods include:

  • writeEntry(Entry) — This method writes the provided entry to the LDIF file or output stream.
  • writeEntry(Entry,String) — This method writes the provided entry to the LDIF file or output stream, immediately preceded by the specified comment.
  • writeChangeRecord(LDIFChangeRecord) — This method writes the provided change record to the LDIF file or output stream.
  • writeChangeRecord(LDIFChangeRecord,String) — This method writes the provided change record to the LDIF file or output stream, immediately preceded by the specified comment.
  • writeLDIFRecord(LDIFRecord) — This method writes the given LDIF record (which may either be an entry or change record) to the LDIF file or output stream.
  • writeLDIFRecord(LDIFRecord,String) — This method writes the given LDIF record (which may either be an entry or change record) to the LDIF file or output stream, immediately preceded by the specified comment.

Methods used for writing LDIF data may throw an IOException.

Parallel LDIF Processing

We’ve worked hard to make the process of reading and writing LDIF data as fast as possible.  While it’s pretty fast on its own as a serial process, portions of the processing may be parallelized for better performance on multi-CPU and/or multi-core systems.

When reading LDIF data, there are two phases of processing:  the process of reading raw data from the LDIF file or input stream, and the process of decoding that data as an entry or change record.  The first phase must be performed serially, but the second can be parallelized and multiple concurrent threads may be used to decode LDIF records.  As a result, when the LDIF reader is configured to use multiple concurrent threads, it may be the case that the limiting factor is the speed at which data can be read from the disk or input stream.

In order to use parallelism in the LDIF reader, it is only necessary to specify the number of additional threads to use when parsing entries or change records at the time the reader is created.  This parallelism will be provided completely behind the scenes, so that the caller may continue to use the readEntry(), readChangeRecord(), and readLDIFRecord() methods just as if all processing were performed serially.

Of course, introducing parallelism in the LDIF reader would be of limited usefulness if it introduced the possibility that entries or change records might be made available to the caller in a different order than they were contained in the original LDIF data.  As a result, the LDIF reader will ensure that the order of the data is preserved so that the only perceptible change between serial and parallel processing is the relative speed with which LDIF records may be made available to the caller.

The LDIF writer also provides support for parallel processing, in which case multiple entries or change records may be formatted into their LDIF representations in parallel, and then will be written serially to the LDIF file or output stream. As with the LDIF reader, parallelism may be enabled by specifying the number of threads to use to perform that processing at the time that the LDIF writer is created. However, parallel processing will only be performed in the LDIF writer if the entries or change records are provided to it using the writeLDIFRecords(List<? extends LDIFRecord>) method. The order that the records are provided in the list will be preserved when they are written to the LDIF file or output stream.

Transforming Entries Read from LDIF

When reading entries from an LDIF file or input stream, it may be useful to alter those entries in some way before they are made available to the caller, and in some cases it may be desirable to exclude entries altogether.  Either or both of these may be performed by providing a class which implements the com.unboundid.ldif.LDIFReaderEntryTranslator interface to the LDIF reader. If an entry translator is provided, then whenever an entry is read, the translate(Entry,long) method will be invoked to allow the translator to return a modified version of the provided entry, a completely new entry, or null to indicate that the entry should be omitted from the data made available to callers.

If the LDIF reader has been configured to perform parallel processing, then the entry translator will be invoked during the parallel portion of that processing, and as a result it may be faster to perform this transformation using the LDIFReaderEntryTranslator interface than performing it separately after the entry has been retrieved by the caller using a method like readEntry(). This does require the translator to be threadsafe, and it cannot depend on the order in which entries are processed.

Obtaining LDIF Representations of LDAP SDK Objects

There are a number of objects provided as part of the LDAP SDK which can be represented in LDIF form without the need for an LDIF writer. This includes the following types of objects:

  • com.unboundid.ldap.sdk.Entry — This can be represented as an LDIF entry.
  • com.unboundid.ldap.sdk.AddRequest — This can be represented as an LDIF add change record.
  • com.unboundid.ldap.sdk.DeleteRequest — This can be represented as an LDIF delete change record.
  • com.unboundid.ldap.sdk.ModifyRequest — This can be represented as an LDIF modify change record.
  • com.unboundid.ldap.sdk.ModifyDNRequest — This can be represented as an LDIF modify DN change record.

All of these objects provide the following methods that allow them to be represented in LDIF form:

  • toLDIF() — This returns a string array whose elements comprise the lines of the LDIF representation of the associated record.
  • toLDIFString() — This returns a string containing the LDIF representation of the associated record, including line breaks.

In addition, the above objects except Entry provide a toLDIFChangeRecord() method that allow them to be converted to the appropriate type of LDIF change record.

Using LDIF Records to Process Operations

The LDAP SDK also provides support for processing operations based on the LDIF representation.  There are two primary ways that this may be accomplished:

  • All types of the LDIF change records provide a processChange(LDAPConnection) method which can be used to directly process an operation from that change record.  Each change record type also provides a method for converting that change record object to an LDAP request (e.g., the LDIFAddChangeRecord class provides a toAddRequest() method to convert the change record to an equivalent add request).
  • Entries can be created from their LDIF representation using the Entry(String...) constructor, where the provided string array contains the lines that comprise the LDIF representation of that entry. Add requests can also be created from the LDIF representation of the entry to add using the AddRequest(String...) constructor. Modify requests can also be created from the LDIF representation of that modify request using the ModifyRequest(String...) constructor.

LDAP SDK Features: Client-Side Processing

When writing a directory-enabled application, you can let the directory server do a lot of the work for you, and in many cases that’s appropriate.  However, there are times when it might be better to do some of that processing on the client rather than the server (e.g., because of the performance hit of the additional requests, or because some target servers might not support all of the features you want to use).  In other cases it might be useful to perform client-side validation prior to sending a request to the server to reduce the likelihood of a failure or to provide better information about what needs to be fixed.  The UnboundID LDAP SDK for Java provides support for several types of client side processing, as described in this post.

Comparing Entries

The com.unboundid.ldap.sdk.Entry class includes a diff method that can be used to compare two entries. The complete definition for this method is:

public static List<Modification> diff(Entry sourceEntry,
Entry targetEntry, boolean ignoreRDN, String... attributes)

It can be used to compare the contents of the two provided entries and will return a list of the modifications that can be applied to the source entry in order to make it look like the target. You can optionally ignore differences in the RDN attribute, and you can optionally restrict the comparison to a specified set of attributes.

There are many uses for this capability. For example, if you have an application that allows a user to alter the contents of entries, then this method may be used to compare a local copy of the updated entry with the original entry in order to determine the modifications to apply to the server. Alternately, you could use this to compare an entry in two different servers to determine whether they are in sync.

Schema Validation

The LDAP SDK provides enhanced support for parsing server schema definitions, and can use that to perform a number of types of client-side processing. One such capability is exposed through the com.unboundid.ldap.sdk.schema.EntryValidator class. This class may be initialized with schema read from the server and provides the following method:

public boolean entryIsValid(Entry entry, List<String> invalidReasons)

This method may be used to determine whether the given entry conforms to the schema constraints defined in the server. The basic checks that it can perform include:

  • Ensure that the entry has a valid DN
  • Ensure that the entry has exactly one structural object class
  • Ensure that all of the entry’s object classes are defined in the schema
  • Ensure that all of the entry’s attributes are defined in the schema
  • Ensure that all of the attributes required by any of the entry’s object classes are present
  • Ensure that all of the attributes contained in the entry are allowed by at least one of that entry’s object classes
  • Ensure that all of the attribute values conform to the syntax defined for that attribute
  • Ensure that all of the attributes with multiple values are defined as multi-valued in the server schema

In addition, if there is a DIT content rule associated with the entry’s structural object class, then it will be used to ensure that all of the auxiliary object classes in the entry are allowed and that the entry doesn’t contain any prohibited attributes, and it may also allow additional attributes not allowed by any of the object classes. If there is a name form associated with the entry’s structural object class, then the entry validator will use it to ensure that the entry’s RDN includes all of the required attributes, and that all attributes included in the entry’s RDN are allowed.

All of these checks may be individually enabled or disabled, which makes it possible to tailor the types of validation to perform. In addition, the entryIsValid method is threadsafe, so it can be called concurrently by multiple threads. If the entry is invalid, then this method will return false, and the provided list will include one or more human-readable messages detailing the problems identified with the entry. It will also collect summary statistics about all of the entries examined so that if a number of entries are processed it is easy to understand the types of problems (if any) that were encountered.

One of the example programs provided with the LDAP SDK is a validate-ldif tool which uses the EntryValidator class to perform this validation for all entries in an LDIF file. This can provide a useful way to validate that the entries contained in an LDIF file are valid without needing to actually import the data into a server. If there are any problems, then the summary provided by the tool may be easier to interpret than the errors reported by the server.

Sorting

Many directory servers provide support for the server-side sort control as defined in RFC 2891. However, some servers may not support this control, and of the servers that do support it, some of them may require special configuration or indexing, and the client may require special permissions to be allowed to request it. In addition, for small result sets, it may be significantly more efficient (and less resource-intensive on the server) to perform sorting on the client rather than asking the server to do it.

The UnboundID LDAP SDK for Java provides a com.unboundid.ldap.sdk.EntrySorter class that can be used to accomplish this. It includes the following method that makes it possible to sort a collection of entries:

public SortedSet<Entry> sort(Collection<? extends Entry> entries)

This class also implements the Comparator<Entry> interface, so it can be used with any type of Java collection that can use comparators.

It is possible to define the sort criteria, including specifying which attributes should be used and indicating whether to sort in reverse order. It can also be configured to take the entry’s position in the DIT into consideration when sorting so that entries can be sorted by hierarchy instead of or in addition to performing attribute-based sorting.

For best results, the entry sorter is able to use schema read from the directory server in order to better understand which matching rules it should use for sorting, but if no schema is available it will simply assume that all sorting should be done using case-ignore ordering.

Filter Evaluation

There are a number of cases in which it might be useful to determine whether an entry matches a given set of criteria, and search filters provide a convenient way to express that criteria. If the entry exists in a directory server, then of course a search operation may be used to make the determination, but if the client has a local copy of the entry then it may be more convenient or efficient to perform the evaluation on the client. Further, there may be cases (e.g., for an entry read from an LDIF file) in which the entry isn’t available in a server. To provide this capability, the com.unboundid.ldap.sdk.Filter class includes the following methods:

public boolean matchesEntry(Entry entry)
public boolean matchesEntry(Entry entry, Schema schema)

If a schema object is provided, then it will be used to perform more accurate evaluation, but if no schema is available then it will work using case-ignore matching. This method currently supports all types of filter evaluation except approximate and extensible matching.

LDAP SDK Features: Connection Pooling

In order to get the best performance and efficiency from a directory-enabled application, you will likely want to use some form of connection pooling.  Most LDAP APIs provide support for some level of connection pooling, but I’m not aware of any other API that provides anywhere near the power and convenience of the connection pooling capabilities of the UnboundID LDAP SDK for Java.  This post will examine many of the connection pooling features that it provides.

Connection Pool Implementations

The UnboundID LDAP SDK for Java actually provides two connection pool implementations.  The com.unboundid.ldap.sdk.LDAPConnectionPool class provides a single set of connections intended to be used for all types of operations. The com.unboundid.ldap.sdk.LDAPReadWriteConnectionPool class maintains two sets of connections, one set intended for use in processing write (add, delete, modify, and modify DN) operations, and the other for use in processing read (bind, compare, and search) operations. The former is the best choice for environments in which all of the directory servers have the same capabilities, while the latter is better for environments containing some a mix of read-only and read-write servers.  The read-write connection pool implementation may also be useful in environments in which it is recommended to try to send all writes to the same server, even if all of the servers technically support read and write operations.

Failover and Load Balancing

Connection pools can be created so that all connections in the pool are established to the same server, or they can be spread across multiple servers.  This can be accomplished through the com.unboundid.ldap.sdk.ServerSet API, which is used to determine which server to use at the time that a connection is created.  There are three server set implementations provided in the LDAP SDK, and you can create your own with whatever logic you choose. The included implementations are:

  • com.unboundid.ldap.sdk.SingleServerSet — This is a simple implementation in which connections will always be established to the same server.
  • com.unboundid.ldap.sdk.RoundRobinServerSet — This is an implementation that allows you to specify a list of servers. The first attempt to create a connection will use the first server, the second will choose the second server, and so on, and once it hits the end of the list it will wrap back around to the beginning.   If the selected server is unavailable, it will skip ahead to the next server in the list, and will only fail if none of the servers are available.
  • com.unboundid.ldap.sdk.FailoverServerSet — This is an implementation that allows you to specify an ordered list of servers, and when attempting to establish a new connection it will always try the first server, but if it isn’t available then it will try the second, then the third, and so on. This implementation also provides the ability to fail over between server sets rather than individual servers, which can provide a lot of flexibility (e.g., try round-robin across all servers in the local data center, but if none of them are available then try round-robin across all the servers in a remote data center).

When a connection pool is created with a server set, then that server set will be used to establish all connections in that pool.  For example, if you create a pool with ten connections and you use the round-robin server set with a list of two servers, then five of the connections in that pool will be established to the first server and five to the second.  This provides a simple mechanism for client-side load balancing, since operations performed using pooled connections will be spread across the two servers.  It also provides better availability, since if either server becomes unavailable then connections established to it will fail over to the other (and in many cases it will be transparent to the application).

If all of the servers defined for use in a connection pool are unavailable at the same time, then obviously any attempt to use that connection pool during that time will fail.  However, as soon as any of those servers is back online, then the pool will resume normal operation and operations will again be successful without the need to restart the application or re-create the pool.

Checking out and Releasing Connections

Traditionally, connection pools simply maintain a set of connections that the application can check out, use as needed, and return.  Although the implementations provided by the UnboundID LDAP SDK for Java go well beyond this, it still possible to check out and release connections as needed.

In the LDAPConnectionPool implementation, a connection can be obtained using the getConnection() method. The application can then do the appropriate processing, and when complete it can release the connection back to the pool with the releaseDefunctConnection(LDAPConnection) method. If an error occurs during processing that indicates the connection is no longer viable, then the releaseDefunctConnection(LDAPConnection) method should be used instead, which will cause the provided connection to be closed and discarded a new connection created in its place.

The LDAPReadWriteConnectionPool implementation provides similar methods for checking out and releasing connections, but there are separate methods for interacting with the read and write pools. The getReadConnection() method may be used to check out a connection from the read pool, and the getWriteConnection() method will check out a connection from the write pool. When returning connections, the releaseReadConnection(LDAPConnection) and releaseWriteConnection(LDAPConnection) methods may be used to return valid connections, and the releaseDefunctReadConnection(LDAPConnection) and releaseDefunctWriteConnection(LDAPConnection) methods may be used to return connections that are no longer valid.

Performing Operations in the Pool

One nice feature about the connection pools provided by the UnboundID LDAP SDK for Java is that they include convenience methods for performing operations using connections from that pool.  As a result, it is not necessary to check out and release connections in order to process an operation because the pool will do it on behalf of the requester.  It will also perform any necessary error handling, and evaluate the result code to indicate whether the connection should be destroyed and a new connection created in its place.

The provided connection pool implementations implement the com.unboundid.ldap.sdk.LDAPInterface interface, which is the same interface implemented by the LDAPConnection class.  You can process a search using a pooled connection by invoking one of the LDAPConnectionPool.search or LDAPReadWriteConnectionPool.search methods just like you can call LDAPConnection.search.  Because they all implement the same interface, it’s relatively simple to write an application that can be used to work with either single connections or connection pools.

The LDAPConnectionPool class also provides a processRequests(List<LDAPRequest>,boolean) method that can be used to process multiple requests in succession on a single connection from the pool. In this case, the results of those operations will be returned in a list, and in the event that an any of those requests does not complete successfully then it may either continue attempting to process subsequent requests in the lists or it may stop at that point.

Connection Availability

It is important to understand the behavior that a connection pool exhibits whenever a connection is needed but none are immediately available. The UnboundID LDAP SDK for Java provides two settings that can be used to control the connection pool’s behavior in this case. The first setting is whether the connection pool should be allowed to create a new connection if one is needed but none are available, and the second is how long to wait for a connection to become available before either creating a new connection or returning an error. These settings are controlled by the setCreateIfNecessary(boolean) and setMaxWaitTimeMillis(long) methods, respectively.

With the combination of these settings, any of the following behaviors may be selected if a connection is needed but none are available:

  • The connection pool should immediately throw an exception
  • The connection pool should immediately create a new connection
  • The connection pool should immediately create a new connection
  • The connection pool should wait for up to a specified length of time for a connection to become available, and if that time passes without a connection becoming available then it will throw an exception
  • The connection pool should wait for up to a specified length of time for a connection to become available, and if that time passes without a connection becoming available then it will throw an exception
  • The connection pool should wait as long as it takes for a connection to become available.

If the pool is configured to create new connections if none are available, then there may be periods of time in which there are more total connections associated with that connection pool than the maximum number of connections to maintain in the pool.  In this case, the behavior that the pool will exhibit when a connection is released will depend on the number of connections available in the pool at that time.  As long as the number of available connections in the pool is less than the maximum, then the connection will be released back to the pool and will be made available for use by subsequent requests.  If an attempt is made to release a connection when the pool already has the maximum number of available connections, then the released connection will be closed.  This provides a scalable and efficient way for the connection pool to grow in size as needed under heavy load while ensuring that it doesn’t hold onto too many connections during slower periods.

Making it Easier to Write Directory-Enabled Applications

I’ve been working with LDAP directory servers for about ten years, and for that entire time I’ve also been writing code.  I’ve written a lot of server-side code in the course of building directory servers, but I’ve written even more client-side code for interacting with them.  Unfortunately, I’ve always been a bit disappointed with the APIs that are available for LDAP communication, especially those for use in Java applications.

Java should be a great language for writing directory-enabled applications.  It’s fast, has a broad standard library, offers a number of frameworks for developing web-based and desktop applications, and it’s easy to write robust code in a short period of time.  Unfortunately, the APIs available for LDAP communication are pretty limited.  JNDI is a common choice because it’s part of the core Java runtime, but it’s very cumbersome and confusing and provides rather limited access to the elements of the LDAP protocol.  The Netscape Directory SDK for Java is more user friendly than JNDI, but it’s fairly buggy (especially under load), supports a pretty limited set of controls and extended operations, and is really showing its age after not having any new releases since 2002.  I’ve never actually used JLDAP, but it looks to expose pretty much the same API as the Netscape SDK and has also gone several years without a new release.

Today, UnboundID is correcting this problem with our release of the UnboundID LDAP SDK for Java.  It is a user-friendly, high-performance, feature-rich, and completely free Java API for communicating with LDAP directory servers and performing other directory-related tasks.  Some of the benefits that it provides include:

  • It is completely free to use and redistribute.  The LDAP SDK is available under either the GNU General Public License v2 (GPLv2) or the UnboundID LDAP SDK Free Use License.
  • It provides a broad feature set.  In addition to providing full support for the core LDAPv3 protocol, it also includes support for 17 standard controls, 4 standard extended operations, and 6 standard SASL mechanisms.  It also provides a number of related APIs for things like LDIF, base64 and ASN.1 parsing, working with root DSE and changelog entries, enhanced schema support, command line argument processing, and SSL/TLS communication.
  • It is much more convenient and easy to use than other LDAP APIs,  It is often possible to do what you want with quite a bit less code than the alternatives, and its use of Java features like generics, enums, annotations, and varargs can further enhance the development experience.
  • It provides support for connection pooling and client-side failover and load balancing.  Connections can be easily spread across multiple directory servers, and you can even have read and write operations sent to different sets of servers.  The connection pool classes implement the same interface as individual connections, so you can process operations using pooled connections without needing to deal with checking out and releasing connections and performing all of the necessary error handling (although that’s possible too if you need it).
  • It provides excellent performance and scalability.  My testing has shown it to be significantly faster than either JNDI or the Netscape SDK, and the searchrate tool that we include as an example can handily outperform the popular C-based version provided by another vendor.
  • It has no external dependencies.  Everything is included in a single jar file, and the only requirement is a Java SE 5.0 or higher runtime.
  • It is robust and reliable.  We have an extensive test suite for the SDK itself with over 26,000 test cases covering over 94% of the code.  The LDAP SDK is also used as an integral part of other UnboundID products, so it benefits from the testing we do for them as well.  It’s frequently subjected to very heavy and highly concurrent workloads so there shouldn’t be any surprises when moving your applications from testing into production (at least, not because of the LDAP SDK).
  • It includes generous documentation.  In addition to a more thorough overview of the benefits our LDAP SDK provides over other the alternatives, it also includes a getting started guide, Javadoc documentation with lots of examples, and a number of sample programs demonstrating the use of various SDK components.
  • Commercial support is available.  This can help ensure fast access to patches for any problems found in the LDAP SDK, and may also be used to request enhancements and additional functionality.  Developer support is also available to assist you in using the LDAP SDK to create directory-enabled applications.

You can find the UnboundID LDAP SDK for Java available for download at http://www.unboundid.com/products/ldapsdk/.  All of the documentation is available there as well (and also in the product download), including some frequently asked questions and a more detailed list of the advantages it offers over other LDAP APIs.

The UnboundID Website is Live

Since leaving Sun, I haven’t really said a whole lot about what I’ve been working on.  Shortly after our termination was finalized, the other OpenDS outcasts and I started a new company, named UnboundID, focusing on providing directory services solutions.

We have kind of been in “stealth mode” so far, and for a long time, we had only a simple “coming soon” page on our website, but as of last Friday we have an actual site (http://www.unboundid.com/).  There is still not a huge amount of content there, but we’ll be adding more in the upcoming weeks as we release more information about our products.

The Future of SLAMD

Even to the casual observer it’s not too hard to notice that not much has happened with SLAMD in quite a while.  In fact, it’s been almost two years since my last commit to the public repository.  There are several reasons for this, including:

  • It works pretty doggone well in its current state.  I still use it quite a bit and hear that several other people do, too.  There are definitely ways that it could be improved, but so far it has been able to meet my needs.  Most of the SLAMD code that I have written since then has been in the form of new jobs that were pretty task-specific and not something that are likely to be useful in other environments.
  • Most of the time, I have been really busy with other things.  I just haven’t had nearly as much time to invest in it as I would have liked.  Of course, I did have a forced two-month vacation near the end of last year, but the terms of my severance stated that I wasn’t allowed to do any work and I didn’t want to press my luck.  After that period ended I’ve been going full-steam-ahead on something else.
  • There are parts of it that could do with a redesign.  The code used to generate the administrative interface is currently all held in one large file and could stand to be broken up.  There are also many areas in which updating the code to require Java 5 would allow it to be much more efficient and scalable, and the introduction of features like generics and enums would make the code easier and safer to edit.

Ultimately, I think that it’s at the point where it would be better to invest the effort in a clean rewrite than to try to build upon what’s there now, but so far I haven’t had much opportunity to do either one of them.  It’s definitely something that I would like to do and I’m hopeful that I might have the time to do it at some point in the future.  I have a lot of ideas for interesting and powerful enhancements, so I don’t want to count it out just yet.

Clarifications on the Open Letter

It appears that there are some questions about the content in the open letter that I posted earlier this week. Simon Phipps (Sun’s chief open source officer) posted a comment on my blog that summarizes these questions, so I will use this post to reply to it. The original text from Simon’s post will be indented and italicized, and my responses will be in plain text.

Hi Neil,

Despite the fact you didn’t actually contact the Sun ombudsman service[1], I have had several referrals of your postings. I’ve done a little investigation and I have some questions about your story.

Actually, I did contact the Sun ombudsman service. The exact same text that was included in my blog post was also sent as an e-mail message. That message was sent from neil.a.wilson[at]directorymanager.org with a timestamp of “Wed, 28 Nov 2007 09:57:03 -0600” (9:57 AM US Central Time), and was addressed to neil.a.wilson[at]directorymanager.org. It was blind copied to the following recipients:

  • users[at]opends.dev.java.net
  • dev[at]opends.dev.java.net
  • jis[at]sun.com
  • ombudsman[at]sun.com

I did not receive any bounce messages in reply, and my mail server logs confirm that Sun’s mail server did in fact accept the message for delivery. If my message never made it into the ombudsman[at]sun.com inbox, then perhaps the problem is on your end (e.g., over-eager spam filtering, which happened to me on more than one occasion when I was a Sun employee).

It’s very regrettable that you were laid off, no question. That’s not a part of your narrative I can comment on for HR/legal reasons, but it’s always sad when business pressures force layoffs.

Thank you for the sentiment. While I wasn’t particularly happy about being laid off, I don’t hold a grudge against Sun because of it. Regardless of whether I think it was an intelligent move, Sun did have a justification for it (geographic consolidation). If the only thing that had happened was that I got laid off, then I fully expect that I would still be actively participating in the project. I believe I demonstrated that through my continued involvement in the project even after having received my layoff notification.

However, I do question how you characterize the requests to change the OpenDS governance. I note that the OpenDS governance was changed on April 28 by sshoaff[2] and that the original line reading:

“This Project Lead, who is appointed by Sun Microsystems, is responsible for managing the entire project”

was replaced by one reading

“This Project Lead, who is appointed and removed by a majority vote of the Project Owners, is responsible for managing the entire project”

I have not been able to find a discussion of this change anywhere, and I understand from your former managers that they were unaware of this change. While you characterize the request made of you as:

“demanded that the owners approve a governance change that would grant Sun full control of the OpenDS project”

it seems to me that what in fact happened was you were (collectively) asked to revert that change to its original state. On present data, it appears to me that far from Sun acting in bad faith over the governance, they were in fact making a reasonable request to correct an earlier error. Indeed, all that has happened to the governance document since then is to revert the change[3].

This is not the whole story.

First, the change to which you refer (committed in revision 1739 by Stephen Shoaff on April 28, 2007) was absolutely not unauthorized. Stephen Shoaff and Don Bowen both served as officers of the company (Stephen as the director of engineering for directory products, and Don as a director of product marketing for all identity products), and David Ely was the engineering manager and the Sun-appointed project lead for OpenDS under the original governance. This change was also discussed with Sun’s open source office, and while you (Simon) may not have been directly involved with those discussions, Don Bowen has informed me that there was a telephone conversation in which you told him that each project should make the decisions that are best for its respective community. We also involved the OpenDS and Identity Management communities in the process, although those conversations were on a personal basis with key members rather than at large on the public mailing lists. Unfortunately, none of us can currently produce any evidence to support these claims. When we received the layoff notification we were required to return or destroy any Sun property that we may have had, and since all of these discussions would be considered Sun-internal communication we no longer have access to any record of them in compliance with the notification requirement. However, full documentation to support all of these claims should exist within Sun should you feel the need to verify them.

Second, this was not the governance change to which I referred in my original post. In the meeting that the owners (including Ludovic) had on November 13, 2007, we were informed that it was Sun’s intention to replace the governance with something different and that the new governance would be chosen and managed by a Sun-selected committee. This change has not yet been applied, and as I am no longer involved with the project I cannot comment on whether there is still intent to make it. However, Eduardo referenced this future change on the OpenDS user mailing list today (https://opends.dev.java.net/servlets/ReadMsg?list=users&msgNo=627) when he said “We want to improve these governances, ideally in a consistent way.”

There was no discussion at all during the November 13 meeting of the change made in revision 1739, and it was not brought to our attention until the following evening. To the best of my knowledge the request to revert the change made in revision 1739 was never discussed with anyone other than Stephen Shoaff. I know that I personally never received any communication from anyone within Sun asking me to approve reverting this change.

Finally, I would ask Sun to justify their subsequent reversion of that change and how they believe that it was in the best interests of OpenDS, or how doing so was consistent with Sun’s public stance on the importance and value of community-led projects. Despite the fact that the change we made had more than sufficient authorization, I fail to see how reverting it is in any way an improvement. How is reverting to a Sun-appointed absolute authority better for the community than the consensus-driven model we thought Sun wanted?

I would be happy to continue to investigate this case, so if you would like to submit a complaint to ombudsman@sun.com with full data supporting your accusations I would be pleased to investigate further. I’m afraid I don’t usually read your blog so you’ll need to alert me (webmink@sun.com) to any postings here that need my attention.

Regards

Simon

[1] http://blogs.sun.com/webmink/entry/open_source_ombudsman
[2] http://tinyurl.com/ys5hf3
[3] http://tinyurl.com/yto9qs

I am afraid that there may not be any benefit to further investigation. It appears that you are using your position to attack my credibility and focus on damage control for Sun rather than acting impartially on my behalf as per your claim at http://blogs.sun.com/webmink/entry/open_source_ombudsman. Even if for some reason you did not receive the message that I originally sent to ombudsman[at]sun.com, I find it very discouraging and disappointing that Sun’s community advocate would choose to respond in such an inflammatory manner via e-mail messages and blog comments without even making an attempt to contact me for further clarification. You have accused me of launching an attack with partial facts but apparently have made no attempt to contact me to get the complete facts for yourself. I had gone out of my way to indicate that I felt that this was an isolated incident and not in-line with Sun’s true stance on open source, but it’s hard to continue to hold that position when Sun’s ombudsman and chief open source officer behaves in such a manner.