UnboundID LDAP SDK for Java 4.0.1

The UnboundID LDAP SDK for Java version 4.0.1 has been released. It is available for immediate download from the LDAP.com website, from our GitHub repository, from the SourceForge project, or from the Maven Central Repository.

This release fixes a number of issues and adds a few small features. Some of the most significant changes are:

  • Added a new JVMDefaultTrustManager class that can be used to automatically trust any certificate signed by an authority that the JVM considers trusted by default. The command-line tool framework has been updated so that if you don’t explicitly specify a trust behavior, it will now check the JVM-default trust manager before prompting about whether to trust the server certificate.
  • Updated the in-memory directory server to add support for encoding clear-text passwords using a pluggable mechanism. For example, you can automatically have clear-text passwords transformed so that they are stored as the base64-encoded representation of a salted message digest.
  • Updated the in-memory directory server to indicate which attributes will be treated as password attributes. Any password attribute can be used to provide credentials for a bind operation, and the values of password attributes will be encoded with the configured password encoder (if any). The server was formerly hard-coded to use userPassword as the password attribute, and this is still the default configuration, but it is now possible to configure the server to use one or more other attributes instead of or in addition to userPassword.
  • Added support for a new password update behavior request control. This control can be used in an upcoming release of the Ping Identity Directory Server to override the behavior the server would otherwise have used for a number of password-related properties (e.g., whether the password update is a self change or an administrative reset, whether to allow a pre-encoded password, which password storage scheme to use, etc.). The ldapmodify tool has been updated to make it easy to include this control in add and modify requests.
  • Updated the identify-unique-attribute-conflicts example tool to provide support for identifying conflicts between combinations of attributes. For example, you can use this feature to identify cases in which there may be duplicate uid values within the same organization, but ignore duplicate uid values for users in different organizations.
  • Fixed an OSGi problem in the jar file manifest. When the LDAP SDK supported Java 1.5 or later, the correct value for the Bundle-RequiredExecutionEnvironment property was “J2SE-1.5”. When we updated the LDAP SDK to require Java 7 or later, the value of this property was updated to be “J2SE-1.7” instead of the correct new value of “JavaSE-1.7”.
  • Fixed a problem that prevented the complete set of argument validation from being performed when running a tool in interactive mode. In particular, the interactive mode framework did not perform validation related to required, exclusive, and dependent argument sets.
  • Fixed an issue with the way that command-line tools handled trailing arguments in interactive mode. If the tool didn’t require any trailing arguments but allowed any number of them to be provided, then interactive mode did not allow trailing argument values to be provided.
  • Fixed an issue with the way that relative paths were handled in command-line tools run in interactive mode. When a Java File object is created from a relative path rather than an absolute path, the getParentFile() method may return null, and this could cause the LDAP SDK to incorrectly believe that the file’s parent didn’t exist. To avoid this, the LDAP SDK now uses getAbsoluteFile().getParentFile() in order to get the parent for any File that may have been created from a relative path.
  • Fixed an issue with command-line tools that default to interactive mode that could arise if the tool is invoked without any arguments, but if it tries to use a properties file referenced by an environment variable or JVM property. If the properties file contained some but not all of the arguments needed to invoke the tool, the command-line tool framework would still try to invoke the tool with just the arguments from the properties file, which could result in erratic behavior, unexpected errors, or uncaught exceptions. The tool will now launch in interactive mode to allow the missing arguments to be specified.
  • The ldapsearch tool has been updated so that the base DN argument is now optional in all circumstances. Previously, you had to explicitly provide either a base DN or an LDAP URL file, but this created a usability problem if you ran ldapsearch in interactive mode and wanted to search with a null base DN (that is, the DN with the empty string representation). Now, if you don’t provide either a base DN or an LDAP URL file, then ldapsearch will assume a null base DN.
  • Updated the class-level Javadoc documentation for a number of classes that implement controls and extended requests and responses. If it takes an encoded value, the Javadoc documentation now describes the encoding for that value.
  • Fixed a couple of problems with message format strings that had incorrect property references (for example, they referenced “{1}” when they should have referenced “{0}” as the first argument). The LDAP SDK build process has been updated to better catch these kinds of problems.
  • Improved the ByteStringBuffer.append(CharSequence) method so that it will be much more efficient for CharSequence implementations in which iterating through the characters using the charAt(int) method is expensive.

UnboundID LDAP SDK for Java 4.0.0

The UnboundID LDAP SDK for Java version 4.0.0 has been released. It is available for immediate download from the LDAP.com website, from our GitHub repository, from the SourceForge project, or from the Maven Central Repository.

Some of the most significant changes in this release are:

  • The LDAP SDK now requires Java SE 7 or later. Java SE 7 and 8 are officially supported. There are known issues when trying to build the LDAP SDK on Java SE 9 early access builds, but builds of the LDAP SDK should run without issues on Java SE 9. Java SE versions 1.5 and 1.6 are no longer supported.
  • We now provide only a single edition of the LDAP SDK. We used to provide Standard Edition, Commercial Edition, and Minimal Edition versions of the LDAP SDK, but they have been consolidated into a single edition that contains everything that was previously in the Commercial Edition (which was a superset of the Standard Edition, which was itself a superset of the Minimal Edition). That single edition is now called just “UnboundID LDAP SDK for Java” and is still available under the terms of the GNU General Public License version 2 (GPLv2), the GNU Lesser General Public License version 2.1 (LGPLv2.1), and the UnboundID LDAP SDK Free Use License.
  • The GitHub repository for the LDAP SDK has been moved into the Ping Identity organization. The URL to the repository has changed from https://github.com/unboundid/ldapsdk to https://github.com/pingidentity/ldapsdk, but a redirect is in place to ensure that links to the old URL will be automatically transferred to the new location.
  • All copyright notices have been updated to reference Ping Identity, and the LDAP SDK documentation now uses Ping Identity branding.
  • The open source repositories for the LDAP SDK have been updated to become a complete mirror of the internal repository used to create official builds. The biggest change to come from this is that the full set of LDAP SDK unit tests are now publicly available under the same licenses as the rest of the LDAP SDK.
  • This release fixes a bug in the logic for parsing DNs from a string in which one or more RDN values used a BER encoding by starting the value with the octothorpe (#) character. The LDAP SDK would incorrectly use the entire set of bytes (representing the BER type, length, and value) as the attribute value instead of just the BER element value.
  • This release fixes a bug in the LDAP connection pool’s connection handling. If the connection pool is configured with createIfNecessary set to false and the replaceDefunctConnection method is called but unable to create a new connection, then the defunct connection could be destroyed without allowing for a replacement. If this happened enough times, the pool could run out of connections and would refuse to create new connections.
  • This release fixes a bug in processing multi-stage SASL binds. Each bind request in a multi-stage bind should use a different LDAP message ID, but earlier versions of the LDAP SDK would use the same message ID for the later stages that it used for the first stage.
  • This release fixes a bug in the in-memory directory server’s LDIF import code that prevented it from applying the configured schema to the entries being imported.
  • This release fixes a bug in the in-memory directory server’s handling of LDAP subentries. The server could incorrectly return entries that are not LDAP subentries in response to a search request that included the subentries request control.
  • This release fixes bugs various bugs in the ldapsearch and ldapmodify command-line tools, and in the command-line argument parser.
  • The LDAP SDK documentation now includes a few new LDAP reference documents, including a result code reference guide, an OID reference guide, and an LDAPv3 wire protocol reference guide.
  • The set of LDAP-related specifications has been updated to include some additional RFCs (including 2926, 2985, 4226, and 6238), and updated versions of IETF drafts (including draft-kille-ldap-xmpp-schema, draft-seantek-ldap-pkcs9, and draft-wibrown-ldapssotoken).
  • When the LDAP SDK is checked out from a git repository, the build process can now capture information about the state of that repository, including the repository URL and the revision ID. This makes it easier to identify the precise source code revision used to create an LDAP SDK build for troubleshooting purposes. Previously, this information was only available if the LDAP SDK was checked out of a subversion repository.

Important updates about the upcoming 4.0.0 release of the UnboundID LDAP SDK for Java

TL;DR: The next release of the UnboundID LDAP SDK for Java will have a version number of 4.0.0, will require Java SE 7 or later, and there will be just one edition instead of the three editions that we currently maintain.

Although it’s still at least a month or two away, I wanted to make a couple of announcements about the next release of the UnboundID LDAP SDK for Java that might affect some of its users. These are some significant changes, so we’ll bump the version number to 4.0.0.

We’re going to start updating the LDAP SDK to reflect these changes immediately, but there’s still time before the release, so if you do have any concerns or questions about these changes, then now is the time to raise them. The best way to do that is to send us an email at ldapsdk-support@pingidentity.com. or use the SourceForge project discussion forum.

Requiring Java SE 7 or Later

The first change is that we’re going to require Java SE 7 or later to use the LDAP SDK.

All previous LDAP SDK releases have been compatible with Java SE 5.0 or later, but Java SE 5 is really old. According to http://www.oracle.com/technetwork/java/eol-135779.html, Oracle stopped providing public updates for it in 2009, and even extended support for it ended in 2015. There are a few things in the SDK that don’t work as well if you’re using Java SE 5, and for which we currently have to use reflection to access on newer VMs.

Dropping support for Java SE 5 allows us to simplify that code and potentially take advantage of new Java features that were added in Java SE 6 and 7. We’ll also be able to update some components that we use during the LDAP SDK build process, although this doesn’t have any impact on your ability to use the SDK. And as always, the LDAP SDK does not and will not depend on anything except Java SE, so you won’t need any third-party libraries to use it.

The older releases of the LDAP SDK aren’t going away, so if you really need to run on Java SE 5.0 or 6 for some reason, then you can continue to use one of the existing releases.

Only Releasing a Single Edition

Another notable change is that we’re only going to be providing a single edition of the LDAP SDK moving forward. Right now, we offer three editions:

  • Standard Edition (SE) — A fully-functional LDAP SDK for use with any type of LDAPv3 directory server.
  • Commercial Edition (CE) — Everything in the Standard Edition, plus additional features specifically intended for use in conjunction with the UnboundID/Ping Identity Directory Server, Directory Proxy Server, and other server products.
  • Minimal Edition (ME) — A very stripped-down version of the LDAP SDK that still provides core LDAPv3 support, but with a focus on keeping a very small jar file for space-constrained environments like Android or embedded systems.

Offering a separate Commercial Edition of the LDAP SDK was necessary in the past because it wasn’t open source and we only made it available to customers who had purchased our server software. But since then, we made it open source and publicly available. There were also concerns about a developer accidentally writing code that leveraged proprietary features that would prevent it from working against non-UnboundID/Ping Identity servers, but that’s easy enough to avoid by just staying away from classes in a package below com.unboundid.ldap.sdk.unboundidds.

Similarly, in the earlier days of Android and other Java-based embedded systems, you didn’t have as much room to work with as you do today, so having a Minimal Edition with a significantly smaller footprint was useful, but it did come at the cost of functionality and convenience. And even the Commercial Edition isn’t all that big (the jar file is around 3.5 megabytes, versus 650 kilobytes for the Minimal Edition, and two megabytes for the Standard Edition).

So from now on, we’re just going to have one edition, and we’ll just call it UnboundID LDAP SDK for Java. It’ll have everything in it that the Commercial Edition has, and it’ll continue to be open source under the terms of the GPLv2 and LGPLv2.1 (plus the UnboundID LDAP SDK Free Use License, which isn’t open source but lets you use and redistribute the LDAP SDK for just about any purpose as long as you don’t make any changes to it). The jar file will be named unboundid-ldapsdk.jar , and we’ll continue to publish it to Maven with a GroupId of com.unboundid and an ArtifactId of unboundid-ldapsdk.

A New LDAPv3 Wire Protocol Reference Guide

Have you ever wondered what exactly goes into LDAP requests and responses? Have you ever wanted to know what LDAP messages look like as they’re transferred between clients and servers? Have you had the opportunity to admire just how sleek and elegant the ASN.1 Basic Encoding Rules encoding can be? Then look no further.

I’ve created an LDAPv3 Wire Protocol Reference that provides an in-depth look at the encoding for all types of LDAP messages. It covers the complete encoding for all types of LDAPv3 requests and responses, including annotated examples and all the ASN.1 BER that you should need to understand those encodings.

This reference guide will be included in the documentation for the next release of the UnboundID LDAP SDK for Java (and you can get it now in that form by checking out and building the LDAP SDK from the GitHub project), but it’s also available online here.

Understanding and Defending Against LDAP Injection Attacks

Injection attacks are one of the most common sources of security holes because it’s so easy for an unsuspecting developer to leave the door open for them. But it’s also usually very easy to prevent them through some pretty simple means.

An injection attack happens when an application uses externally-obtained data in the course of its processing, but without making sure that data is acceptable and safe. It’s especially predominant in cases where an application plugs user input into some kind of a query or command that it sends to some kind of data repository, and doesn’t protect against the possibility that unexpected or malicious user input could cause the application to issue a different request than the one it expected. You’re probably most likely to hear about injection attacks when dealing with SQL (the structured query language, commonly used to interact with relational databases), but it can also affect interaction with other data repositories, like NoSQL databases and even LDAP directory servers.

LDAP directory servers actually have an inherent advantage over many other types of data stores when it comes to injection attacks because LDAP isn’t a text-based protocol and because LDAP APIs typically don’t make it possible to accidentally turn one type of operation into a different kind of operation. SQL injections are particularly dangerous because it’s possible for an SQL statement intended to just read some data from the database to be inadvertently converted into one that destroys, corrupts, or otherwise wreaks havoc on the data. This can’t happen in an LDAP injection, but there are still some very real threats that you need to protect against.

LDAP Filter Injections

By far, the most common type of LDAP injection attack is a filter injection. This can happen whenever you construct an LDAP search filter from its string representation and include user-provided data in the process.

For example, consider an application that offers an input field that makes it possible to look up a user by their username or their email address. Such an application might have the following code:

String filter = "(|(uid=" + userInput + ")(mail=" + userInput + "))";

If the user input is “jdoe”, then this will end up creating the filter “(|(uid=jdoe)(mail=jdoe))”. That seems safe enough, right? But instead, let’s consider what would happen if the user were to enter an asterisk instead of jdoe. That would cause the resulting filter to be “(|(uid=*)(mail=*))”, and that would match any entry within the scope of the search that has either at least one of the uid and mail attributes. And what if the user were to enter jdoe)(objectClass=*”? In that case, the code would create a filter of “(|(uid=jdoe)(objectClass=*)(mail=jdoe)(objectClass=*))”, and that filter would match any entry within the scope of the search, including those that don’t have either the uid attribute or the mail attribute.

The Risks of Filter Injection Attacks

As illustrated above, one of the key risks of a filter injection attack is that it could cause the application to expose more entries, or different kinds of entries, than the application intended to make available. But there are other dangers as well.

Leaking Sensitive Attribute Values

One risk that people don’t often think about is the possibility of using a filter injection attack to leak the values of attributes that contain sensitive information. For example, let’s say that we know that the directory server stores a user’s social security number in the ssn attribute in the user’s entry and that we want to find out what the social security number is for user jdoe. Let’s also assume that there aren’t any users in the directory that have a uid or mail value of noMatches. If the application constructs a search filter using the code listed above, then we might try entering the following into the input field:

noMatches)(&(uid=jdoe)(ssn=1*)

This would result in the application generating the following filter:

(|(uid=noMatches)(&(uid=jdoe)(ssn=1*))(mail=noMatches)(&(uid=jdoe)(ssn=1*)))

This filter would match the entry for user jdoe only if that entry has an ssn value whose first digit is one. If that filter doesn’t match, then we could replace the one with a two, then a three, and so on until we get a match, and then we know what the first digit of the social security number is. Then we can use the same technique to find the second digit, then the third, etc. until we know all nine digits.

Denial of Service Attacks

Filter injection attacks also open the door for very simple and very effective denial of service (DoS) attacks, whether against the application that interacts with the directory server, or against the directory server itself. An injection attack could turn what is expected to be a very efficient filter into one that is very time-consuming to process and takes up a lot of server resources. If you’re able to get enough of those going at the same time, it may eat up all of the available processing cycles in either the application or the directory server so that other requests can’t get through.

Further, if the application is designed to hold all of the entries returned from a search in memory at the same time, a search that returns a lot more entries than expected could cause the application to consume all available memory on the system, which could potentially make the application crash or spend all of its time in garbage collection, or could make the system start paging memory out to disk.

Invoking Operations Against Unintended Entries

We’ve already established that a filter injection attack has the potential to cause a search to match entries that the application didn’t expect to match. If the application makes those entries available to the end user in some way, then the application could be tricked into leaking information to the end user. But what if the application doesn’t simply make those entries available to the end user, but instead does something else with them? What if the application applies some update to each entry that matches the search filter? If you can trick the application into searching for the wrong entries, then that could lead to the application updating the wrong entries, which could cause data loss or corruption.

Defending Against Filter Injection Attacks

There are several ways that you can protect your LDAP-enabled application against filter injection attacks. Some of them are probably very easy to implement. Others may take additional effort. And you might want to implement more than one of these safeguards to take a kind of “belt and suspenders” approach.

Don’t Construct Filters by Concatenating Strings

You should never, never, never, never construct an LDAP search filter by concatenating strings, especially when that string contains any user input. Instead, you should leverage the features that your LDAP library offers to create filters programmatically.

For example, if you’re using the UnboundID LDAP SDK for Java, rather than:

String filter = "(|(uid=" + userInput + ")(mail=" + userInput + "))";

You should instead use:

Filter filter = Filter.createORFilter(
     Filter.createEqualityFilter("uid", userInput),
     Filter.createEqualityFilter("mail", userInput));

Much like using an SQL prepared statement, constructing an LDAP filter programmatically ensures that it isn’t possible for crafty input to result in a different kind of filter than the one that was intended.

If you’re using an LDAP library that doesn’t provide a way to programmatically generate search filters, then you should strongly consider selecting a new library to use for LDAP communication. If that’s not feasible, and you have to create filters from their string representations, then you absolutely must sanitize any user input included in the generated filter.

Sanitize User Input Included in Search Filters

There are two basic ways to sanitize user input.

The first is to reject any input that doesn’t appear to be valid. For example, if you have an input field in which you expect the user to provide a username or an email address, then you might only want to allow the input to contain letters, digits, periods, dashes, underscores, plus signs, and the at sign.

And if it’s a client-server application, then you should make sure to do the validation in server-side code rather than (or perhaps in addition to) client-side code. For example, if you have a web application, and it uses JavaScript to ensure that the input only contains valid characters, then an attacker might be able to get around that by disabling JavaScript support in their browser, or by not using a browser at all but some other kind of HTTP client that allows them to send exactly the request they want to send. If you have server-side validation, the attacker won’t be able to get around that with any amount of fancy client-side trickery. You can still use client-side validation if you want, since that can make for a better experience for legitimate end users, but just make sure to do the same validation in the server for actual security.

The second way to sanitize user input is by escaping any special characters that it may contain. RFC 4515 states that the following escaping must be applied to the assertion value in the string representation of an LDAP search filter:

  • The null character (U+0000) must be escaped as \00
  • The left parenthesis must be escaped as \28
  • The right parenthesis must be escaped as \29
  • The asterisk must be escaped as \2a
  • The backslash must be escaped as \5c

You can escape any other character by placing a backslash in front of the hexadecimal representation for each byte in the UTF-8 encoding for that character, but the above characters are the ones that absolutely have to be escaped.

The LDAP library you’re using may provide a mechanism to do this for you (for example, if you’re using the UnboundID LDAP SDK for Java, then you can use one of the the Filter.encodeValue(String) or Filter.encodeValue(byte[]) methods), but if it has that, then it’s probably got methods to help you programmatically construct the filter, and using that approach is definitely better than trying to perform your own escaping.

Use an AND Filter To Impose Restrictions on Matching Entries

To prevent a search from matching entries of a different type than you expect, you can wrap the filter inside of an AND that will only match the desired type of entry. For example, if you know that you only want to search for user entries, and you know that all user entries have the person object class, then rather than using a filter like:

(|(uid={input})(mail={input}))

You could instead use a filter like:

(&(objectClass=person)(|(uid={input})(mail={input})))

This will ensure that only user entries will be returned. If you have more specific criteria, then you can include that in the filter as well. Just note that this type of protection on its own isn’t enough to prevent all kinds of injection attacks since it doesn’t protect against wildcards or new components injected inside the OR. So you’ll still need to make use of one of the other types of protection listed above.

Restrict the Search Request in Other Ways

The search filter is just one of the elements of an LDAP search request. There are other elements that you may be able to adjust to reduce the effect of processing a search that isn’t exactly what you expected. Some of those are:

  • Set the base DN and scope to be as specific as possible to the type of search that you’re performing. For example, if you know that you’re searching for a user, and you know that all users are in a particular branch in the directory (for example, beneath “ou=People,dc=example,dc=com”), then base your search at that branch rather than at the root of the tree, so that the search won’t match any entries outside of that branch.
  • Use the size limit to prevent the server from returning more entries than expected. If you’re searching for a single user entry, then use a size limit of one, and the server will return an error result if it finds more than one matching entry.
  • Use the time limit to prevent the server from spending too much time processing the search. If you expect your search to be efficient, then setting the time limit to a second or two should be more than enough time to allow the server to process it under normal circumstances, but small enough to ensure that an unexpectedly inefficient search gets cut off before too long.

Leverage the Server’s Access Control Mechanism

Another great type of protection against attacks of all kinds is to ensure that requests are issued under an account that only has permission to do what it’s supposed to do. For example, if you only want the application to be able to search for entries by targeting the uid and mail attributes, then only give the application’s account permission to issues searches targeting those attributes. Similarly, give the application read access only to the attributes that it legitimately needs to get back in search result entries, and give it write access only to the attributes that it legitimately needs to be able to update.

If an application needs to process operations on behalf of another user, then you may want to use the proxied authorization request control (described in RFC 4370) to ensure that those operations are processed in accordance with that user’s access control rights.

LDAP DN Injections

Although filter injection attacks are by far the most prevalent, it is conceivably possible for an LDAP injection attack to target entry DNs. In particular, if an application constructs an LDAP DN from user input, then it may be possible for a malicious user to provide unexpected input that could end up targeting a different entry than was expected.

For example, let’s say that an application has the following code:

String userDN = "uid=" + userInput + ",ou=People,dc=example,dc=com";

If the user enters “jdoe”, then this would construct the following DN:

uid=jdoe,ou=People,dc=example,dc=com

But if the user enters “jdoe,ou=Secret Users”, then this would construct a DN that is one level below what the application intended:

uid=jdoe,ou=Secret Users,ou=People,dc=example,dc=com

While theoretically possible, DN injections aren’t all that common or practical. There are two key reasons for this.

First, it’s rare for applications to construct DNs based on user input. Or, at least, it’s rare for well-designed applications to construct DNs based on user input. It’s a very bad thing for an application to assume that DNs have a particular structure or pattern because not all of them do. And even if DNs have a known format when the application is being developed, it’s entirely possible that the format could change later (for example, to eliminate all personally-identifiable information from DNs), and the application would be broken. It’s far better for an application to search for an entry and learn its DN that way than to try to construct it.

Second, the only kind of injection that you can realistically achieve when constructing a DN is to target an entry deeper in the tree than you had initially expected. You can’t use an injection attack to target any arbitrary entry in the DIT. Fortunately, LDAP DNs don’t have any equivalent to the “..” in a filesystem path that allows you traverse up to its parent.

Nevertheless, if you do encounter a scenario in which you need to construct a DN from user input (for example, if you’re adding a new entry), then you should see if the LDAP library you’re using provides a way to safely construct the DN for you. For example, if you’re using the UnboundID LDAP SDK for Java, instead of using the code:

String userDN = "uid=" + userInput + ",ou=People,dc=example,dc=com";

You should use:

DN userDN = new DN(new RDN("uid", userInput), new RDN("ou", "People"),
     new RDN("dc", "example"), new RDN("dc", "com"));

This will ensure that all appropriate escaping is done, and will thwart any injection attempt.

If your LDAP library doesn’t have any kind of method like the above for constructing DNs safely, then you should either get a new LDAP library that does provide this support, or you should perform the escaping yourself. The rules for escaping special characters in DNs are a little different from the rules for escaping special characters in a search filter. All the necessary details for constructing the string representation of a DN is provided in RFC 4514, but the basics are:

  • You should escape the double quote character as \" or \22
  • You should escape the plus sign character as \+ or \2b
  • You should escape the comma character as \, or \2c
  • You should escape the semicolon character as \; or \3b
  • You should escape the less-than character as either \< or \3c
  • You should escape the greater-than character as either \> or \3e
  • You should escape the backslash character as either \\ or \5c
  • If a value has any leading or trailing spaces, then you should escape those spaces by prefixing them with a backslash or as \20. Spaces in the middle of a value don’t need to be escaped.
  • If the value starts with an octothorpe character (#), then you should escape it as either \# or \23. You only need to escape the octothorpe character at the beginning of a value, and not in the middle or at the end of the value.

LDAP Result Code and OID Reference Guides

I’ve added a couple of LDAP reference documents.

The LDAP Result Code Reference provides a fairly complete overview of many LDAP result codes, including the core LDAPv3 result codes defined in RFC 4511, server-side result codes from other specifications, and client-side result codes. It describes each of the result codes and some of the cases in which they may be used.

The LDAP OID Reference lists a number of object identifiers used throughout LDAP for things like schema elements, controls, and extended operations. It lists the OIDs, provides a brief explanation of its purpose, and, when applicable, provides a link to the relevant specification.

Both of these will also be included in the documentation for the next release of the UnboundID LDAP SDK for Java (and they’re already checked into the GitHub repository), but for now, you can find them on my website.

A Couple of Updates to the LDAP SDK Open Source Repository

Within the last weeks, there have been a couple of noteworthy changes to the open source repositories for the UnboundID LDAP SDK for Java.

Migrated the GitHub Repository to Ping Identity

The primary open source repository for the LDAP SDK is on GitHub. It used to be part of the UnboundID organization, but over the weekend, I migrated it to the Ping Identity organization (since Ping Identity acquired UnboundID last year). The new URL for the LDAP SDK project on GitHub is https://github.com/pingidentity/ldapsdk.

This should be a completely transparent migration. All of the content (revision history, releases, issues, forks, etc.) should have been preserved. The URLs used to access the repository have changed, but GitHub should redirect all of the old URLs to the new ones, so if you have any links or bookmarks that use the old URLs, you should still end up in the right place. This is also true for any clones of the repository, but if you have checked out the LDAP SDK, then you might want to update your local workspace to the new path. You can do that with the command:

git remote set-url origin git@github.com:pingidentity/ldapsdk.git

If you notice any problems with the repository as a result of the migration (or for any other reason), please open an issue, and we’ll look into it.

And for the record, we also migrated the other public UnboundID repositories to Ping Identity. That includes account-manager, auth-explorer, auth-ui, broker-groovy-sign-in-sample, broker-react-sign-in-sample, my-account, scim, scim2, server-sdk-maven, and status-servlet.

Published the LDAP SDK Unit Tests

When we originally made the LDAP SDK open source, we published everything that you need to build the LDAP SDK, but there was some content from our internal repository that wasn’t made public. The biggest omission from the open source repository was the set of unit tests.

The main reason for omitting the unit tests was that, at the time, most of them required the UnboundID Directory Server (now Ping Identity Directory Server), and that wasn’t publicly available. It was possible to run the tests without a Directory Server instance, but there would be large portions of the code that wouldn’t get covered. But then we updated the LDAP SDK to include the in-memory directory server, and we started using it for most unit tests created after that, so now you can get really good test coverage without an external Directory Server instance.

There are still some tests that will only get run if you have an external Directory Server instance, and if you want to run them, you can download the 6.0.1 release of the Ping Identity Directory Server from https://www.pingidentity.com/en/products/downloads/pingdirectory.html. You’ll need to create an account if you don’t already have one, but that’ll get you access to a free, fully-functional evaluation copy of the Directory Server. The license doesn’t allow you to use the Directory Server for any commercial purpose, but it’s certainly suitable for use in running the unit tests if you so choose.

And with this update, the GitHub repository for the LDAP SDK is now a complete mirror of the internal subversion repository, and we expect to always keep them in sync. At some point, we may make the GitHub the master repository for the LDAP SDK, but at present, there are some internal build processes that rely on our private repository (and rely on subversion rather than git).

UnboundID LDAP SDK for Java 3.2.1

We have just released the 3.2.1 version of the UnboundID LDAP SDK for Java. It is available for download from the LDAP.com website, as well as from GitHub, SourceForge, or the Maven Central Repository.

You can get a full list of changes included in this release from the release notes. The Commercial Edition release notes also provide information about additional changes only included in the Commercial Edition.

Some of the most significant changes in both the Standard Edition and the Commercial Edition include

  • Updated the documentation to indicate that, as a result of Ping Identity’s acquisition of UnboundID, all non-public feedback, feature enhancements, support requests, and other kinds of communication should now be sent to ldapsdk-support@pingidentity.com instead of ldapsdk-support@unboundid.com. We also now recommend using the GitHub issue tracker over the SourceForge mailing lists and discussion forums for bug reports and feature requests.
  • Fixed a bug in the RDN parsing code that could cause multiple consecutive spaces in the middle of an attribute value to be condensed down to a single space. The string representation of the RDN was preserved correctly, but the methods used to retrieve attribute values as a string or byte array could return values that were missing spaces.
  • Provided better handling for InterruptedException. A thread’s interrupted state will now be preserved for cases in which the LDAP SDK consumes an InterruptedException without doing something to handle it.
  • Fixed a bug in the support for the SASL ANONYMOUS mechanism that could cause the trace string to be omitted from the encoded bind request.
  • Updated the searchrate tool to provide support for generic controls, as well as specific support for the assertion, simple paged results, and server-side sort request controls.
  • Updated the authrate tool to add a new –bindOnly argument that allows you to indicate that the tool should only perform bind operations, rather than a search to find the entry and then a bind as that user. The base DN pattern will be used to construct the bind DN.
  • Updated the authrate tool to provided support for generic search and bind controls, as well as specific support for the authorization identity and password policy request controls.
  • Updated the search-and-modrate tool to provide support for generic search and modify controls, as well as specific support for the assertion, simple paged results, permissive modify, pre-read, and post-read request controls.
  • Added a Schema.getSchema method that can read schema information in LDIF form from an input stream.
  • Updated support for the GSSAPI SASL mechanism to make it possible to indicate in the generated configuration file whether the client should act as an initiator or an acceptor.
  • Updated the identify-unique-attribute-conflicts tool to include a time limit in search requests intended to determine whether a unique attribute value may also be in use in any other entries. This can help limit the effect of running the tool against a server that is not configured with the appropriate indexes needed to ensure that equality searches targeting the unique attributes can be processed efficiently.

Some of the additional changes only available in the Commercial Edition include:

  • Added a new version of the ldapsearch tool that provides a lot of additional functionality over the version provided in the Standard Edition. It includes much better output formatting (including support for alternate output formats like JSON, CSV, and tab-delimited text), support for a number of data transformations, more robust connection handling, support for referrals, support for a large number of search and bind controls, support for administrative sessions, support for unsolicited notifications, the ability to process multiple searches with search criteria provided in filter or LDAP URL files, rate limiting, and the ability to send results to a specified output file (or a separate output file per search).
  • Implemented caching for the matching rule instance used when requesting the jsonObjectExactMatch matching rule. This matching rule only exists in the Commercial Edition and needs to be loaded via reflection.
  • Updated the access and error log parsing APIs to include support for the triggeredByConn and triggeredByOp log fields used to indicate that the message is associated with the indicated operation.

The Story of UnboundID

unboundid-logo-blue-background-white-unbound-orange-id-600x139

At the end of 2016, I ceased to be an employee of UnboundID and became an employee of Ping Identity. Ping acquired UnboundID in early August of 2016, but most UnboundID staff remained employees of that company through the end of the year (mostly for bookkeeping convenience). As a co-founder, I was with UnboundID from the beginning. Here is our story.

Once upon a time, Sun Microsystems had the best LDAP directory server on the market. The 5.1 and 5.2 releases of the Directory Server Enterprise Edition (DSEE) product, which introduced multi-master replication and 64-bit support, respectively, were very exciting.

But then for some reason, Sun’s directory server engineering team seemed to contract a chronic case of inertia, in which everyone was content to rest on their laurels. It certainly wasn’t because the product had reached perfection, nor from a lack of ideas for improvement. After a while, I became frustrated with the lack of progress, and I wrote a hundred-plus-page document (which became known as the “directory manifesto”) that was full of things that we could do to improve the product. It had plenty of low-hanging fruit that would address pain points that customers were experiencing and lots of bigger ideas to help ensure our continued dominance of the high-end directory server market. But to no avail.

Eventually, I was able to nag the right people long enough to let me start working on a new directory server that was intended to be a DSEE replacement. And thus I became the first developer working on what would become OpenDS. Sun was committed to open sourcing all their software, from Solaris to Java to all of their enterprise software, and before long, we released OpenDS under the same CDDL license that they initially created for OpenSolaris.

At that time, Sun’s Directory Server engineering team was mostly split between Austin, Texas and Grenoble, France. In 2007, someone got the idea that it would be good to have it headquartered all in one place, and Grenoble was chosen as that one place. As a result, the five U.S. employees most closely connected with OpenDS were laid off: director of engineering Steve Shoaff, marketing lead Don Bowen, engineering manager David Ely, open source community manager Trey Drake, and myself as architect.

It didn’t take the five of us long to decide that we wanted to create a new company developing on top of the OpenDS codebase. We were still passionate about the product and excited about the opportunities that it could afford, and we planned to contribute back to the open source community. We even followed through on that with a handful of commits within a couple of weeks of the layoff. But then some nastiness arose between us and Sun’s management that I don’t want to get into here, and it became clear that we were no longer welcome participants.

So by the time we founded UnboundID (on December 17, 2007), we were to be competing against Sun. We’d comply with the terms of the open source license when altering existing code, but newly-created files would be our own private intellectual property, as allowed by the CDDL. We would, of course, also go head-to-head with other directory server vendors, like Oracle (who bought Sun within a few months of our departure), IBM, Microsoft, and CA. And we’d be up against open source offerings like OpenLDAP and the 389 Directory Server. We should’ve had no chance. And yet we were shockingly optimistic. We had a lot of ideas, a lot of drive, and dare I say a pretty good amount of talent.

Upon officially forming the company, David, Trey, and I worked furiously to make improvements to the codebase. We made dramatic improvements in performance, concurrency, and scalability. We added killer features, like support for transactions, filtered logging, change subscriptions, data transformations, data integrity checksums, and some new controls and extended operations. We made the server easier to manage by improving the out-of-the-box configuration, refined the command-line and interactive text-based interfaces, and added a web-based administration console.

We also created a new Java-based API for interacting with LDAP servers, because the existing options sucked. Before we released the UnboundID LDAP SDK for Java, you could basically choose between the horrible, confusing clunkiness of JNDI (where LDAP support is bolted on as an afterthought), or the buggy and no-longer-maintained Netscape Directory API. We wanted to create an API that made it easy to write applications that could take full advantage of any LDAP server, including the enhanced functionality we were building into our own software.

While David, Trey, and I were churning out the code, Steve and Don were working to sell it. And they did. Amazingly, we got our first customer within a matter of months: a large network equipment provider that supplied telephone companies with the equipment used to run their data centers. They loved our software, our enthusiasm, our ability to react quickly, and our willingness to put our source code in escrow so they wouldn’t be screwed if we went out of business. And before long, a number of big telcos were kicking the tires on our stuff and salivating at the idea of a modern, high-powered, feature-rich, and administrator-friendly directory service.

This first customer was a huge win for us. They resold our software and provided first-line support, which helped alleviate any downstream concerns about our viability. They declared our software to be carrier-grade, which served as validation to other potential customers in other industries. And it was also nice to be able to start getting a paycheck.

Plus, we were able to leverage this deal to get good terms on an initial round of funding from an investor. We were able to hire more people (many of whom were former colleagues from Sun who were all too happy to jump ship from their new Oracle overlords), and we started working on new products. David took on the Synchronization Server, and I started on the Directory Proxy Server.

The Synchronization Server provides a way to mirror the contents of two or more data repositories, so that changes made in one system appear in the other systems, usually in a matter of milliseconds. It can do one-way or bidirectional synchronization. It can synchronize all the data or just a configurable subset. And you can connect to a number of different types of repositories, including LDAP directory servers (both UnboundID and non-UnboundID), relational databases, NoSQL databases, and more (plus an API for developing your own support for additional types of data stores). It’s ideal for migrating data from your existing repository into the UnboundID Directory Server, and for keeping both infrastructures in sync for whatever length of time is necessary to complete the migration, or indefinitely if you want to keep both systems up and running.

As its name implies, the Directory Proxy Server is an LDAP proxy server. In the simplest deployments, it allows you to achieve better performance and higher availability through load balancing and advanced health checking techniques. In larger deployments, you can use entry balancing to transparently split data up into multiple sets (much like database sharding) for even greater scalability. It can also transform requests and responses as they pass between clients and backend directory servers, and you can do this on a per-application basis in case some clients have different expectations for how the data should look or how the server should behave.

We continued to grow. We sold more software. We gained more customers. We hired more employees. We wrote more code. But sadly, we also suffered some losses. Trey decided it was time for him to move on, so he left the company. But even more tragically, Don Bowen passed away in late 2009. He’d been diagnosed with brain cancer a mere three days after we founded the company, and somehow he continued to make incredible contributions toward our success for over a year and a half. I met Don in my first job out of college, at Caterpillar, where he introduced me to the world of LDAP. Within a few months, he’d gotten an offer to join the Baltimore-based startup (initially called B2B Communications, but later renamed TidePoint Corporation) and he took me with him. When that went south, we went our separate ways, only to meet up again when we both joined Sun at about the same time. We were friends as well as colleagues, and a tremendous amount of who I am today is because of Don.

And the work goes on. As more customers migrated from existing environments and deployed into new environments, we realized that we needed to provide public interfaces to allow our software to be customized, so we created the Server SDK. Our server products had always been very extensible and componentized, but before the Server SDK, we were the only ones who could take advantage of that. The Server SDK made it possible for customers to write their own extensions to customize the behavior of the server, from intercepting and altering operation requests and responses, to creating new loggers, password validators, extended operations, SASL mechanisms, sync sources and destinations, proxy transformations, and more. The initial intention was to only make the Server SDK available to customers who’d gone through at least some kind of training (it allows you to run custom code inside the server, so there’s a chance that a buggy extension could make the server unusable), and I really wish we’d stuck to that more than we did. But for the most part, it was a hit with customers, and an even bigger hit with the sales engineers helping them evaluate and then migrate to our software.

We also introduced a couple of additional server products: the Analytics Engine and the Data Broker. The Analytics Engine (formerly called the Metrics Engine) provides simple, graphical access to all kinds of historical information about the operation of the server broken down in all kinds of ways (e.g., the number of requests per second of each operation type, and a breakdown of their result codes and processing times), along with metrics from the underlying system like CPU, disk, and network utilization. The Data Broker provides several REST-based interfaces to interact with the environment, including support for OAuth 2 and OpenID Connect (for authentication, authorization, and federation), and SCIM (for data access). As much as I love LDAP and will continue to tout its superiority over HTTP, the kids these days are all about the web APIs, so we must oblige.

But even as we worked on new products, the Directory Server continued to grow and improve. I am particularly passionate about security, so that’s been a big focus of mine over the last several years. We added support for data encryption, several two-factor authentication mechanisms, more password storage schemes and password validators, sensitive attributes, retired passwords, password reset tokens, improved account lockout, signed logs, and more. But we also added a lot of non-security-related features, like JSON support, alarms and gauges, soft deletes, assured replication, and indexing improvements. It’s gotten faster, easier to use and administer, and just plain better.

On the whole, we’ve had an unbelievable run. We did have one down year with less-than-stellar sales, but for all other years, we reached or exceeded our goals. That continued through 2016, which was one of our best years ever (even if you ignore the whole “our company was bought” thing). We certainly weren’t floundering, and we weren’t really even looking to be acquired. We had partnered with Ping Identity on a number of deals in the past, as each company’s software complemented the other’s very well without too much overlap. And then Ping was acquired by Vista Equity Partners, who were looking for other opportunities to get into identity management, and then people started talking and it all sort of became a three-way deal, with the UnboundID acquisition by Ping Identity following Ping’s own acquisition within a couple of weeks.

So what’s next for the bigger, better Ping Identity? I can’t get into any specifics, but we’ve got a lot of great things in the works. Most of the UnboundID staff, and I believe all of the technical staff, are continuing on into Ping. Some people are changing roles (many moving up, others moving laterally), but my job isn’t really changing all that much. I’m still writing code, and hope that continues far into the future. Some of our product names have changed (for example, it’s now the Ping Identity Directory Server rather than the UnboundID Directory Server), and some haven’t (it’s still the UnboundID LDAP SDK for Java), but we’re still working to ensure that they remain the best products out there. So I need to get back to work.


P.S. I know that the only people I mentioned by name in this walk down memory lane are the UnboundID founders. I certainly don’t mean to imply that we’re the only ones responsible for or vital to the company’s success. There are so many other people that made big contributions to the company that I can’t list them all without fear of leaving someone out, without fear of mentioning someone who’d rather be left out, and without fear of making this long post even longer. So let me just say that if you worked for UnboundID in any capacity, you have my sincerest thanks.

UnboundID LDAP SDK for Java 3.2.0

We have just released the 3.2.0 version of the UnboundID LDAP SDK for Java. It is available for download via the LDAP.com website or from GitHub, as well as the Maven Central Repository.

You can get a full list of changes included in this release from the release notes (or the Commercial Edition release notes for changes specific to the Commercial Edition). Some of the most significant changes include:

  • Added a new transform-ldif tool that can be used to apply a number of transformations to data in an LDIF file. This includes the ability to scramble, replace, redact, or exclude a specified set of attributes; to replace existing values for a specified attribute; to use a sequential counter for values of a specified attribute; to add a given set of values to entries matching specified criteria; to exclude entries matching specified criteria; to rename attributes; to replace the base DN for entries in a specified subtree; and to flatten a DIT.
  • Updated all classes that offer a public void close() method that doesn’t throw any exceptions other than a possible IOException so that they implement the java.io.Closeable interface. This includes classes like LDAPConnection, LDAPConnectionPool, LDIFReader, LDIFWriter, and all EntrySource implementations. This allows code using these classes to take advantage of the try-with-resources facility introduced in Java SE 7.
  • Added support for parsing entries that contain information about the operations processed in the server for servers that support the syntax described in draft-chu-ldap-logschema-00.
  • Updated the modrate tool to make a number of improvements, including support for a number of controls, the ability to replace multiple values rather than just a single value, or the ability to perform an increment modification rather than a replace modification.
  • Added a new JSONBuffer class that can be used to efficiently construct the string representation of a JSON object, and a JSONObjectReader class that can be used to read JSON objects from an input stream. Added the ability to generate formatted, multi-line string representations of JSON objects with improved human readability.
  • Updated the LDIF reader to make it possible to specify the character set to use when reading data. Updated the LDIF writer to make it possible to automatically include a comment below any base64-encoded values that provides a non-base64-encoded representation (with special characters escaped) of the preceding value.
  • Updated the in-memory directory server to support the LDAP no-operation control as described in draft-zeilenga-ldap-noop-12.
  • Added a new base64 command-line tool that can be used to encode and decode data using the base64 format.
  • Dramatically improved the robustness of the identify-references-to-missing-entries and identify-unique-attribute-conflicts tools.
  • Updated the argument parser to add support for subcommands with their own distinct set of arguments.
  • Added support for timestamp arguments, which can be used to specify timestamps in either the generalized time syntax (including the time zone), or in a number of formats that indicate a time in the local time zone.
  • Updated the command-line tool API to provide the ability to default to interactively prompt for passwords that may be needed but not provided, and to send output to a specified file.
  • Updated the rate adjustor so that generated sample rate files include a number of additional examples for common patterns like square, stairstep, sine, sawtooth, triangle, and hockey stick.