Ping Identity Directory Server 8.2.0.0

We have just released version 8.2.0.0 of the Ping Identity Directory Server, with several big new features, as well as a number of other enhancements and fixes. The release notes provide a pretty comprehensive overview of the changes, but I’ll summarize the changes here. I’ll first provide a TL;DR version of the changes, followed by more detail about some of the more significant updates.

Summary of New Features and Enhancements

  • Added single sign-on support to the Administration Console (more information)
  • Added a new ds-pwp-modifiable-state-json operational attribute (more information)
  • Added support for password validation during bind (more information)
  • Added support for a recent login history (more information)
  • Added sample dsconfig batch files (more information)
  • Added JSON-formatted variants for the audit, HTTP operation, and synchronization log files (more information)
  • Added support for logging to standard output or standard error (more information)
  • Added support for rotating the logs/server.out log file (more information)
  • Improved support for logging to syslog servers (more information)
  • Added support for the OAUTHBEARER SASL mechanism (more information)
  • Added support for the ($attr.attrName) macro ACI (more information)
  • Added a remove-attribute-type-from-schema tool (more information)
  • Added a validate-ldap-schema tool (more information)
  • Added a number of security-related improvements to setup (more information)
  • Added a number of improvements to the manage-profile tool (more information)
  • Added a number of improvements to the parallel-update tool (more information)
  • Added various improvements to several other command-line tools, including ldappasswordmodify, ldapcompare, ldifsearch, ldifmodify, ldif-diff, ldap-diff, and collect-support-data (more information)
  • Added various usability improvements to command-line tools (more information)
  • Added several improvements to the dictionary password validator (more information)
  • Added a new AES256 password storage scheme (more information)
  • Added an export-reversible-passwords tool (more information)
  • Create a 256-bit AES encryption settings definition in addition to the 128-bit AES definition
  • Added password information about password quality requirements to the ds-pwp-state-json virtual attribute (more information)
  • Added the ability to augment the default set of crypto manager cipher suites (more information)
  • Improvements in delaying bind responses (more information)
  • Require a minimum ephemeral Diffie-Hellman key size of 2048-bits
  • Switched to using /dev/urandom as a source of secure random data (more information)
  • Improved the way we generate self-signed certificates and certificate signing requests (more information)
  • Added support for using elliptic curve keys in JWTs
  • Added new administrative alert types for account status notification events, privilege assignment, and rejection of insecure requests (more information)
  • Added improvements to monitor data and the way it is requested by the status tool (more information)
  • Added identity mapper improvements, including filter support, an aggregate identity mapper, and an out-of-the-box mapper for administrative users (more information)
  • Improved uniqueness control conflict prevention and detection (more information)
  • Added support for re-sending an internal replication message if there is no response to a dsreplication initialize request
  • Added a --force argument to dsreplication initialize that can be used to force initialization even if the source server is in lockdown mode
  • Added options to customize the response code and body in availability servlets
  • Increased the number of RDN components that a DN may have from 50 to 100
  • Updated the SCIM servlet to leverage a VLV index (if available) to support paging through search result sets larger than the lookthrough limit
  • Added an --adminPasswordFile argument to the manage-topology add-server command
  • Added a password policy configuration property to indicate whether the server should return the password expiring or password expired based on whether the client also provided the password policy request control
  • Updated support for the CRAM-MD5 and DIGEST-MD5 SASL mechanisms so they are no longer considered secure
  • Improved Directory Proxy Server support for several SASL mechanisms (more information)
  • Improved Directory Proxy Server support for the LDAP join control
  • Updated manage-topology add-server to add support for configuring failover between Synchronization Server instances
  • Added a Synchronization Server configuration property for customizing the sync pipe queue size
  • Added a Synchronization Server configuration property for processing changes with REPLACE modifications rather than ADD and DELETE modifications

Summary of Bug Fixes

  • Fixed an issue that could prevent the installer from removing information about the instance from the topology registry
  • Fixed an issue that could cause replication to miss changes if a backend was reverted to an earlier state without reverting the replication database
  • Fixed an issue in which a replica could enter lockdown mode after initialization
  • Fixed an issue that could allow some non-LDAP clients to inappropriately issue requests without the server in lockdown mode
  • Fixed an issue in which restoring an incremental backup could cause dependencies to be restored out of order, leading to an incomplete intermediate database file
  • Fixed a backup retention issue in which the process of purging old backups could cause old backups to be removed out of order
  • Fixed an issue in which the server could leak a small amount of memory upon closing a JMX connection
  • Fixed an issue that could cause the server.status file to be corrupted on Windows systems after an unplanned reboot if the server is configured to run as a Windows service
  • Fixed an issue that could cause the server to return a password expired response control in a bind response when the user’s account is expired but the client provided incorrect credentials
  • Fixed an issue in which a search that relied on a virtual attribute provider for efficient processing could omit object classes from search result entries
  • Fixed an issue in which the server did not properly handle the matched values control that used an extensible match filter with both an attribute type and a matching rule (for example, in conjunction with the jsonObjectFilterExtensibleMatch matching rule)
  • Fixed an issue in which the server could incorrectly log an error message at startup if it was configured with one or more ACIs that grant or deny permissions based on the use of SASL mechanisms
  • Fixed an issue in which the remove-defunct-server tool to fail to remove certain replication attributes when the tool was run with a topology JSON file
  • Fixed an issue in which manage-profile replace-profile could fail to apply changes if the profile included dsconfig batch files without a “.dsconfig” extension
  • Fixed an issue in which the server could raise an internal error and terminate the connection if a client attempted to undelete a non-soft-deleted entry
  • Fixed an issue that could cause the REST API to fail to decode certain types of credentials when using basic authentication
  • Fixed an issue in which the encryption-settings tool could leave the server without a preferred definition after importing a set of definitions with the --set-preferred argument but none of the imported definitions is marked preferred
  • Fixed an issue in which the manage-profile generate-profile command could run out of memory when trying to generate a profile containing large files
  • Fixed an issue in which the manage-profile generate-profile command could display a spurious message when generating the profile in an existing directory
  • Fixed an issue that could interfere with cursoring through paged search results when using the REST API and the results included entries with long DNs
  • Fixed an issue that could cause an exception in SCIM 1.1 processing as a result of inappropriate DN escaping
  • Fixed an issue that could cause the isMemberOf and isDirectMemberOf virtual attributes to miss updates if the same group is updated concurrently by multiple clients
  • Fixed an issue that could cause the server to return an objectClassViolation result code instead of the more appropriate attributeOrValueExists result code when attempting to add an object class value to an entry that already has that object class
  • Fixed an issue that could cause loggers to consume more CPU processing time than necessary in an idle server
  • Fixed an issue in which the stats collector plugin could generate unnecessary I/O when it is used exclusively for sending metrics to a StatsD endpoint
  • Fixed an issue in which the periodic stats logger could include duplicate column headers
  • Fixed an issue that could cause the server to periodically log an error message if certain internal backends are disabled
  • Fixed a typo in the default template that the multi-part email account status notification handler uses to warn about an upcoming password expiration
  • Fixed an issue in which the dsconfig list command could omit certain requested properties
  • Fixed an issue in which the dsreplication tool could incorrectly suppress LDAP SDK debug messages even if debugging was requested
  • Fixed an issue that could cause the Directory Proxy Server to log information about an internal exception if an entry-balanced search encountered a timeout when processing one or more backend sets
  • Fixed an issue in which the Synchronization Server could get stuck when attempting to retry failed operations when it already has too many other operations queued up for processing
  • Fixed an issue in which Synchronization Server loggers were not properly closed during the server shutdown process
  • Fixed an issue in which the synchronization server could fail to synchronize certain delete operations from an Oracle Unified Directory because of variations in the format of the targetUniqueID attribute

Single Sign-On for the Administration Console

The Administration Console now supports authenticating with an OpenID Connect token, and it can provide single sign-on across instances when using that authentication mechanism. At present, this is only supported when using a token minted by the PingOne service, and there must be corresponding records for the target user in both the PingOne service and the local server. General support for SSO with ID tokens from any OpenID Connect provider should be available soon in an upcoming release.

The new ds-pwp-modifiable-state-json Operational Attribute

We have added a new “Modifiable Password Policy State” plugin that provides support for the ds-pwp-modifiable-state-json operational attribute. The value of this attribute is a JSON object with fields that provide information about a limited set of password policy state properties, including:

  • password-changed-time — The time the user’s password was last changed
  • account-activation-time — The user’s account activation time
  • account-expiration-time — The user’s account expiration time
  • password-expiration-warned-time — The time the user was first warned about an upcoming password expiration
  • account-is-disabled — Whether the account is administratively disabled
  • account-is-failure-locked — Whether the account is locked as a result of too many failed authentication attempts
  • must-change-password — Whether the user will be required to choose a new password before they are permitted to request any operations

These elements of the password policy state can now be updated by modifying the value of the ds-pwp-modifiable-state-json attribute to set a new value that is a JSON object that holds the desired new values for the appropriate fields. Any fields not included in the provided object will remain unchanged.

This operational attribute is intended to work in conjunction with the existing ds-pwp-state-json virtual attribute, which provides read-only access to a wide range of information about the user’s password policy state and the configuration of the password policy that governs their account.

Both of these attributes can be accessed over either LDAP or the Directory REST API, and the UnboundID LDAP SDK for Java provides convenient support for both the ds-pwp-state-json and ds-pwp-modifiable-state-json attributes. The password policy state extended operation is still available and provides support for manipulating a broader set of password policy state properties, but it can only be used over LDAP (either programmatically or through the manage-account command-line tool).

Password Validation During Bind

The server has always provided a lot of options for validating passwords at the time that they’re set, whether in a self-change or an administrative reset. It now also provides the ability to perform validation in the course of processing bind operations in which the server has access to the user’s clear-text password.

This feature can help provide better support for detecting issues with passwords that were acceptable at the time they were set but have since become undesirable (for example, if someone uses the same password across multiple sites, and one of them suffers a data breach that exposes the password). As such, you may want to periodically check the password against services like Pwned Passwords or against a regularly updated dictionary of banned passwords.

Although you can configure the server to perform password validation for every bind attempt, the server also offers the ability to only check them periodically (for example, if it’s been at least a week since the last validation). This may be useful if it is necessary to interact with external services while performing the validation.

You can also configure the behavior that the server should exhibit when the password provided to a bind operation fails validation, including:

  • The server can allow the bind to proceed but place the user’s account in a “must change password” state so all other requests from that user will fail until they choose a new password.
  • The server can reject the bind attempt, requiring an administrative password reset before they can use their account.
  • The server can generate an account status notification that may be used to notify the user about the need to choose a new password (or to take some other action as determined by a custom account status notification handler).

Recent Login History

The server has always provided support for maintaining a last login time and IP address for each user, but that only reflects information about the most recent successful authentication attempt. And if account lockout is enabled, then the server can maintain a list of failed authentication attempts. However, that will only be updated until the account is locked, and it will be cleared by a successful login, so this is really only useful for the purpose of account lockout.

In the 8.2.0.0 release, we are adding support for maintaining a recent login history, stored in the password policy state for each account. For both successful and failed authentication attempts, the server will store the time, client IP address, and the type of authentication attempted. For a failed attempt, the server will also record a general reason for the failure.

The recent login history is available in a few different ways:

  • It is available through the ds-pwp-state-json operational attribute.
  • It is available through the password policy state extended operation and through the manage-account command-line tool that uses that operation behind the scenes.
  • Information about previous login attempts can be obtained on a successful bind by including the new “get recent login history” request control in the bind request. The ldapsearch and ldapmodify tools have been updated with a --getRecentLoginHistory argument that can be used to include this control in the bind request, and the UnboundID LDAP SDK for Java provides support for using this control programmatically.

The recent login history can be enabled in the password policy configuration, and you can configure it to keep up to a specified number of attempts or keep a record for up to a specified duration. This can be configured separately for successful and failed attempts. You can also configure the server to collapse information about similar authentication attempts (that is, attempts from the same IP address with the same result, authentication type, and failure reason) into a single record so, the history doesn’t fill up too quickly from clients that frequently try to authenticate to the server. When collapsing information about multiple similar attempts into a single record, the timestamp of that record will reflect the most recent successful attempt, and it will also include the number of additional similar attempts that have been collapsed into that record.

Sample dsconfig Batch Files

We have added a config/sample-dsconfig-batch-files directory with several well commented dsconfig batch files that demonstrate how to enable, disable, or configure various functionality in the server. Many of these batch files are for changes that can help improve server security, including configuring TLS protocols and cipher suites, managing password policy functionality, disabling insecure communication, enabling two-factor authentication options, and managing administrative users. They help serve as additional documentation and, in many cases, provide information about changes that we recommend but do not have in the out-of-the-box configuration for some reason.

In many cases, these batch files require a certain amount of editing before they can be used (for example, to indicate which TLS protocols to use, which password policy to update, etc.). In such cases, you shouldn’t edit the provided batch file directly but should instead make a copy and customize that as needed. This will help ensure that if we make changes to the existing batch files in the future, those changes will be applied when you update the server to that new version.

Additional JSON-Formatted Loggers

Our default access and error loggers use a legacy space-delimited text format that matches what is used by other types of directory servers. However, in the 6.0.0.0 release, we added support for formatting access and error log messages as JSON objects so that they can be more easily parsed by a wider variety of log management and analysis software.

In the 8.2.0.0 release, we are adding JSON support for additional types of loggers. These include:

  • A JSON-formatted audit log publisher, which provides a record of all changes to data in the server. In the past, audit logging has always used the standard LDIF format to provide the contents of the change and LDIF comments to capture information about other metadata (including the message timestamp, the connection and operation ID, the requester DN and IP address, etc.). The JSON-formatted audit logger still uses the standard LDIF format for representing information about changes, but all of the other metadata is now available through separate JSON fields to make it easier to access in a generic manner.
  • A JSON-formatted HTTP operation log publisher, which provides a record of HTTP requests received by and HTTP responses returned by the server. We already offered HTTP operation log publishers using the W3C common log format and a space-delimited format similar to our default access and audit loggers, but we now offer support for these messages formatted as JSON objects.
  • JSON-formatted Sync log publisher and Sync failed ops log publisher options, which provide information about processing performed by the Synchronization Server. In the past, these messages have used a less-structured format that is primarily intended for reading by a person. When attempting to parse these messages in an automated manner, the new JSON format is a much better option.

Each of these JSON-formatted logger types (including the existing JSON-formatted access and error loggers) includes a logType field in each log message that indicates the type of logger that generated the message. This makes it easier to identify the type of log message in cases where log messages from different sources are intermingled (for example, writing their log files to the same directory). While this is useful when writing these messages to files, it is even more important when logging to the console or to syslog, as will be described below.

We also fixed an issue in our existing support for JSON-formatted access and error log messages that caused the timestamps to be generated in a format that was not completely compliant with the ISO 8601 format described in RFC 3339. These timestamps incorrectly omitted the colon between the hour and minute components of the time zone offset.

Also, while the stats logger plugin has provided an option to write messages as JSON objects since the 8.0.0.0 release, you had to specifically create an instance of the plugin if you wanted to enable that support. In the 8.2.0.0 release, we have added a definition in the out-of-the-box configuration that can generate JSON-formatted output, so you merely need to enable it if you want to take advantage of this feature. This provides better parity with the CSV-formatted output that the plugin can also generate.

Logging to Standard Output and Standard Error

In container-based environments like Docker, it is relatively common to have applications write log messages to the container’s standard output or standard error stream, which will then be captured and funneled into log management software. We are now providing first-class support for that capability. In the 8.2.0.0 release, we are providing console-based variants of each of the following:

  • JSON-formatted access log messages
  • JSON-formatted error log messages
  • JSON-formatted audit log messages
  • JSON-formatted HTTP operation log messages
  • JSON-formatted Sync log messages
  • JSON-formatted Sync failed ops log messages

Each of these can be sent to either standard output or standard error, and you can mix messages from different types of loggers in the same stream. For example, if desired, you could send all log messages for all of these types of loggers to standard output and none of them to standard. Because these JSON objects include the logType field, you can easily identify the type of logger that generated each message.

You can also now configure the server to prevent it from logging messages in non-JSON format during startup. By default, the server will continue to write error log messages to standard error in the legacy space-delimited text format. It will also write information about administrative alerts to standard error, and you can either disable this entirely or have those messages formatted as JSON objects. With these changes, it is now possible to have all of the data written to standard output or standard error be formatted as JSON objects, and you don’t have to worry about JSON-formatted output intermingled with non-JSON output.

Note that we only recommend using console-based logging when running the server with the --nodetach argument, which prevents it from detaching from the console (and is the preferred mode for running in a container like Docker anyway). When running in the default daemon mode, any data written by these console-based loggers will be sent to the logs/server.out file, which is less efficient and less customizable than using the file-based JSON loggers. When running in --nodetach mode, the console-based loggers will only write to the JVM’s original standard output or standard error stream.

Rotation Support for server.out

With the exception of the new console-based loggers when the server is running in --nodetach mode, anything that would have normally gone to standard output or standard error is written into the logs/server.out file. Previously, the server would create a new server.out file when it started up and would continue writing to that same file for the entire life of the server. It would also only keep one prior copy of the server.out file (in server.out.previous).

The server generally doesn’t write much to standard output or standard error, so it’s really not much of a problem to have all data sent to the same file, but it can accumulate over time, and it’s possible that the file could grow large if the server runs for a very long time. To address that, we have added support for rotating this file. Rotation is only available based on the amount of data written, and retention is only available based on the number of files. By default, the server will rotate the file when it reaches 100 megabytes, and it will retain up to 10 rotated copies of the file. This behavior can be customized through the maximum-server-out-log-file-size and maximum-server-out-log-file-count properties in the global configuration.

Syslog Logging Improvements

The server has provided support for writing access and error log messages (in the legacy text-based formats) to syslog since the 2.1.0 release, but there were a number of limitations. Logging used an old version of the syslog protocol (as described in RFC 3164), and messages could only be sent as unencrypted UDP packets.

In the 8.2.0.0 release, we are significantly updating our support for logging over syslog. The former syslog-based access and error loggers are still available for the purpose of backward compatibility, but we have also added several new types of syslog-based loggers with new capabilities, including:

  • Access loggers using either JSON or the legacy text-based format
  • Error loggers using either JSON or the legacy text-based format
  • Audit loggers using the JSON format
  • HTTP operation loggers using the JSON format
  • Sync loggers using the JSON format
  • Sync failed ops loggers using the JSON format

These new loggers will use an updated version of the syslog protocol (as described in RFC 5424), and they also offer the option to communicate over UDP or TCP. When using TCP, you can optionally encrypt the data with TLS so that it can’t be observed over the network. When using TCP, you can also configure the logger with information about multiple syslog servers for the purpose of redundancy (this is not available when using the UDP-based transport, as the server has no way of knowing whether the messages were actually received).

The OAUTHBEARER SASL Mechanism

We have added support for the OAUTHBEARER SASL mechanism as described in RFC 7628. This makes it possible to authenticate to the server with an OAuth 2.0 bearer token, using an access token validator to verify its authenticity, map it to a local user in the server, and identify the associated set of OAuth scopes. The oauthscope ACI bind rule can be used to make access control decisions based on the scopes associated with the access token.

This ultimately makes it possible to authenticate to the server in a variety of ways that had not previously been available, especially when users are interacting with Web-based applications that use the Directory Server behind the scenes. For example, this could be used to authenticate clients with a FIDO security key.

In addition to the support for OAuth 2.0 bearer tokens as described in the official specification, our OAUTHBEARER implementation also supports authenticating with an OpenID Connect ID token, either instead of or in addition to an OAuth 2.0 bearer token. To assist with this, we have added support for ID token validators, including streamlined support for validating ID tokens issued by the PingOne service. If you want to authenticate with only an OpenID Connect token, you can do that with a standard OAUTHBEARER bind request by merely providing the OpenID Connect ID token in place of the OAuth access token. If you want to provide both an OAuth access token and an OpenID Connect ID token, you’ll need to include a proprietary pingidentityidtoken key-value pair in the encoded SASL credentials. The UnboundID LDAP SDK for Java provides an easy way to do this.

The ($attr.attrName) Macro ACI

We have updated our access control implementation to add support for the ($attr.attrName) macro ACI. This macro can be included in the value of the userdn and groupdn bind rules to indicate that the macro should be dynamically replaced with the value of the specified attribute in the authorized user’s entry.

This macro is particularly useful in multi-tenant deployments, in which a single server holds information about multiple different organizations. If a similar rule needs to be enforced across all tenants, you can create one rule for the entire server rather than a separate rule that is customized for each tenant. For example, if a user requesting an operation has an o attribute with a value of “XYZ Corp”, a bind rule of groupdn="ldap:///cn=Administrators,ou=Groups,o=($attr.o),dc=example,dc=com" would be interpreted as groupdn="ldap:///cn=Administrators,ou=Groups,o=XYZ Corp,dc=example,dc=com".

Our Delegated Administration application has been updated to support this macro ACI to provide an improved experience in multi-tenant environments.

Safely Removing Attribute Types From the Server Schema

In general, we don’t recommend removing attribute types (or other types of elements) from the server schema. It’s safe to do for elements that have never been used, but if an attribute has been used in the past (even if it’s not currently being used in any entries), there may still be references to it in the database (for example, in the entry compaction dictionary used to minimize the amount of space required to store it). In that case, the server will prevent you from removing the attribute type from the schema unless you first export and re-import the data to clear any references to it from the database. Further, if the database contains a reference to an attribute type that is not defined in the server schema, it will log a warning message on startup to notify administrators of the problem.

In the 8.2.0.0 release, we’re adding a new remove-attribute-type-from-schema tool that can be used to safely remove a previously used attribute type from the server schema without requiring an LDIF export and re-import. This tool will first ensure that the attribute type is not currently in use (including referenced by other schema elements or in the server configuration), and it will then clean up any lingering references to that attribute from the database before removing it from the schema.

This processing can also be automated through the new “remove attribute type” administrative task. The UnboundID LDAP SDK for Java provides convenient support for this task.

Validating Server Schema Definitions

We have added a new validate-ldap-schema tool that can be used to examine the server schema and report on any issues that it finds. This includes things like

  • Definitions that can’t be parsed
  • Elements that refer to other elements that aren’t defined in the schema
  • Multiple definitions with the same name or OID
  • Schema elements with an invalid name or OID
  • Schema elements without a name
  • Schema elements with an empty description
  • Attribute types without a syntax or an equality matching rule
  • Schema characteristics that some servers do not support (collective attributes, object classes with multiple superior classes)

The UnboundID LDAP SDK for Java also offers a SchemaValidator class to allow you to perform this validation programmatically.

setup Security Improvements

We’ve updated the server setup process to make it easier and more convenient to create a more secure installation.

It was already possible to use non-interactive setup to create an instance that only allowed TLS-encrypted connections (by simply not specifying a port to use for non-secure LDAP connections), but this was not an option for interactive setup. When running interactive setup, the resulting server would always accept unencrypted connections from LDAP clients. Now, the interactive setup process will allow you to create an instance that only accepts connections from LDAP clients that are encrypted with TLS. Alternatively, it is possible to enable support for accepting unencrypted LDAP connections but require those clients to use the StartTLS extended operation to secure communication before they are allowed to issue other requests. And we have redesigned the flow so that you’ll still get the same number of prompts when running setup.

We have also added a couple of new arguments to non-interactive setup (also available in manage-profile setup) to make it possible to impose additional security-related constraints for clients:

  • We have added a new --rejectInsecureRequests argument that can be used to ensure that the server will reject requests from clients communicating with the server over an unencrypted connection. This makes it possible to enable support for unencrypted LDAP connections but require that those clients use the StartTLS extended operation to secure those connections before any other requests are allowed.
  • We have added a new --rejecUnauthenticatedRequests argument to configure the server to reject any request received from unauthenticated clients.

We have also updated non-interactive setup (and manage-profile setup) so that you can provide the password for the initial root user in pre-encoded form (using the PBKDF2, SSHA256, SSHA384, or SSHA512 storage scheme). This eliminates the need to have access to the clear-text password when setting up the server. The pre-encoded password can be provided directly through the --rootUserPassword argument or in a file specified through the --rootUserPasswordFile argument.

manage-profile Tool Improvements

We have made several improvements to the manage-profile tool that can be used to set up or update a server instance. They include:

  • We updated manage-profile replace-profile to better support updating the server’s certificate key and trust store files. When the profile includes the --generateSelfSignedCertificate argument in the setup-arguments.txt file, the original key and trust store files will be retained. Otherwise, manage-profile will replace the server’s key and trust stores with those from the new profile.
  • When applying configuration changes from a dsconfig batch file with the server online, manage-profile replace-profile can now obtain connection arguments from the target server’s config/tools.properties file.
  • We updated manage-profile replace-profile so that if the new profile includes new encryption settings definitions, the new profile’s preferred definition will be used as the resulting instance’s preferred definition.
  • We fixed an issue in which manage-profile setup could complete with an exit code of zero even if the attempt to start the server failed.
  • We fixed an issue in which manage-profile replace-profile could fail when applying changes from a dsconfig batch file when using Boolean connection arguments like --useSSL.
  • We updated the tool to ensure that each instance gets a unique cluster name. We do not recommend having multiple servers in the same cluster for instances managed by manage-profile.
  • We made it possible to obtain debug information about how long each step of the setup or replace-profile process takes.

parallel-update Tool Improvements

The parallel-update tool can be used to apply changes read from an LDIF file in parallel with multiple concurrent threads. It behaves much like ldapmodify, but the parallelism can make it significantly faster to process large numbers of changes. The ldapmodify tool does offer a broader range of functionality than parallel-update, but we have added new capabilities to parallel-update to help narrow that gap. Some of these improvements include:

  • We have added support for several additional controls, including proxied authorization, manage DSA IT, ignore NO-USER-MODIFICATION, password update behavior, operation purpose, name with entryUUID, assured replication, replication repair, suppress operational attribute update, and suppress referential integrity updates. We have also added support for including arbitrary controls in add, bind, delete, modify, and modify DN requests.
  • We have made its communication more robust. The tool would previously only establish its connections when it was first started. If any of those connections became invalid over the course of processing requests, then subsequent operations attempted on that connection would fail. The tool will now attempt to detect when a connection is no longer valid and will attempt to establish a new connection if it becomes invalid, re-attempting any failed operation on that new connection.
  • We have added support for specifying multiple server addresses and ports. If this option is used, then the tool can automatically fail over to an alternative server if the first server becomes unavailable.
  • We have improved the ability to determine whether all operations were processed successfully. Previously, the tool would always use an exit code of zero if it was able to attempt all of the changes, even if some of them did not complete successfully. This is still the default behavior to preserve backward compatibility, but you can now use the --useFirstRejectResultCodeAsExitCode argument to indicates that if any operation fails, the result code from the first rejected operation will be used as the parallel-update exit code.
  • It is now possible to apply changes from an LDIF file that has been encrypted with a user-supplied passphrase or with a definition from the server’s encryption settings database.
  • Previously, if you didn’t provide a value for the --numThreads argument, the tool would use one thread by default. It now defaults to using eight threads.
  • The tool now offers a --followReferrals argument to allow it to automatically follow any referrals that are returned.
  • If you run the tool without any arguments, it will now start in an interactive mode instead of exiting with an error.

Other Command-Line Tool Improvements

We have made a variety of other command-line tool improvements.

We replaced the ldappasswordmodify tool with a new version that offers more functionality, including support for additional controls, support for multiple password change methods (the password modify extended operation, a regular LDAP modify operation, or an Active Directory-specific modify operation), and the ability to generate the new password on the client.

We replaced the ldapcompare tool with a new version that offers more functionality, including support for multiple compare assertions, following referrals, additional controls, and multiple output formats (including tab-delimited text, CSV, and JSON).

We replaced the ldifsearch, ldifmodify, and ldif-diff tools with more full-featured and robust implementations.

We updated the ldap-diff tool so that it no longer wraps long lines by default, as that interfered with certain types of text processing that clients may wish to perform on the output. However, the tool does offer a --wrapColumn argument that can be used to indicate that long lines should be wrapped at the specified column if that is desired.

We updated the collect-support-data tool to make it possible to specify how much data should be captured from the beginning and end of each log file to be included in the support data archive. This option is also available when invoking the tool through an administrative task or an extended operation.

Usability Improvements in Command-Line Tools

We have updated the server’s command-line framework to make it easier and more convenient to communicate over a secure connection when no trust-related arguments are provided. Most non-interactive tools will now see if the peer certificate can be trusted using the server’s default trust store or the topology registry before prompting the user about whether to trust it.

We have also updated interactive tools like dsconfig and dsreplication so that they will prefer establishing secure connections over insecure connections. The new default behavior will be to use the most secure option available, with a streamlined set of options that does not require any additional prompts than choosing to use an insecure connection.

We have also updated setup (whether operating interactive or non-interactive mode) to make it possible to generate a default tools.properties file that tools can use to provide default values for arguments that were not provided on the command line. If requested, this will include at least arguments needed to establish a connection (secure if possible), but it may also include properties for authentication if desired.

Dictionary Password Validator Improvements

The dictionary password validator can be used to reject passwords that are found in a dictionary file containing prohibited passwords. We have always offered this password validator in the server, and we include a couple of dictionaries, including one with words from various languages (including over 730,000 words in the latest release) and another with known commonly used passwords taken from data breaches and other sources (with over 640,000 passwords).

In the 8.2.0.0 release, we have made several improvements to the dictionary password validator. They include:

  • We now offer the ability to ignore non-alphabetic characters that appear at the beginning or end of the proposed password before checking the dictionary. This can help detect things like a simple dictionary word preceded or followed by digits or symbols. For example, if the dictionary contains the word “password”, then the validator could proposed passwords like “password123!” or “2020password”.
  • We now offer the ability to strip characters of diacritical marks (accents, cedillas, circumflexes, diaereses, tildes, umlauts, etc.) before checking the dictionary. For example, if a user proposes a password of “áçcèñt”, then the server would check to see if the provided dictionary contains the word “accent”.
  • We now offer the ability to define character substitutions that can be used when checking for alternative versions of the provided password. For example, you could indicate that the number “0” might map to the letter “o”, the number “1” or the symbol “!” might map to the letters “l” or “i”, that the number “7” might map to the letter “t”, and that the number “3” might map to the letter “e”. In that case, then the validator could reject a proposed password of “pr0h1b!73d” if the dictionary contains the word “prohibited”.
  • We now offer the ability to reject a proposed password if a word from the provided dictionary makes up more than a specified percentage of that password. For example, if you configure the validator to ban passwords in which over 70% of the value matches a word from the dictionary, then the server could reject “mypassword” if the dictionary contains “password” because the word “password” makes up 80% of “mypassword”.

The AES256 Password Storage Scheme

In general, we strongly recommend encoding passwords with non-reversible schemes, like those using a salted SHA-2 digest, or better yet, a more expensive algorithm like PBKDF2 or Argon2. However, in some environments, there may be a legitimate need to store passwords in a reversible form so the server can access the clear-text value (which may be needed for processing binds for some legacy SASL mechanisms like CRAM-MD5 or DIGEST-MD5), or so the clear-text password can be used for other purposes (e.g., for sending it to some other external service).

In the past, the best reversible storage scheme that we offered was AES, which used a 128-bit AES cipher with a key that was shared privately among instances of the replication topology, but that was otherwise not available outside the server (including to the Directory Proxy Server, which might need access to the password for processing certain types of SASL bind requests). This means that it could be used for cases in which the server itself needed the clear-text password, but not when it might be needed outside the server. It was also not a simple matter to rotate the encryption key if the need should arise.

To address these issues, and also to provide support for a stronger cipher, we are adding a new AES256 password storage scheme. As the name implies, it uses AES with a 256-bit key (rather than the 128-bit key for the older cipher). It also generates the encryption key from an encryption settings definition, and if a client knows the password used to create that encryption settings definition, and if it has access to the encoded password, then it can decrypt that encoded password to get the clear-text representation. The UnboundID LDAP SDK for Java provides a convenient way to decrypt an AES256-encoded password given the appropriate passphrase, and the class-level Javadoc provides enough information for clients using other languages or APIs to implement their own support for decrypting the password.

The export-reversible-passwords Tool

While the new AES256 password storage scheme gives you much better control over the key that is used when encrypting passwords, it does not provide direct support for rotating that encryption key should the need arise. You can reconfigure the storage scheme to change the encryption settings definition to use when encoding new passwords, but that does not affect existing passwords that have already been encoded with the earlier definition.

To help facilitate encryption key rotation for AES256-encoded passwords, or to facilitate migrating passwords that use a reversible encoding to use a different scheme, we have provided a new export-reversible-passwords command-line tool. It allows you to export data to LDIF with clear-text representations of any reversibly encoded passwords.

In most environments, you probably won’t actually need or want to use this tool, and you might be concerned about its potential for misuse. To help protect against that, there are a number of safeguards in place to help ensure that it cannot be used inappropriately or covertly. These include:

  • If you don’t encode passwords in a reversible form, then you don’t have anything to worry about. The server can’t get the clear-text representation of passwords encoded using non-reversible schemes.
  • The tool initiates the export by communicating with the server over a TLS-encrypted LDAP connection established over a loopback interface. It cannot be invoked with the server offline, and it cannot be invoked from a remote system.
  • The export can only be requested with someone who has the new permit-export-reversible-passwords privilege. This privilege is not granted to anyone (not even root users or topology administrators) by default, and an administrative alert will be generated whenever this privilege is assigned (as with any other privilege assignment) so that other administrators will be aware of it.
  • The export can only be requested if the server is configured with an instance of the “export reversible passwords” extended operation handler. This extended operation handler is not available as part of the out-of-the-box configuration, and an administrative alert will be generated whenever it is created and enabled (as with any other type of configuration change).
  • The output will be written to a file on the server file system, and that file must be placed beneath the server instance root. As such, it can only be accessed by someone with permission to access other files associated with the server instance.
  • The entire output file will itself be encrypted, either using a key generated from a user-supplied passphrase or from a server encryption settings definition. The encrypted file may be directly imported back into the server using the import-ldif tool, or it may be used as an input file for the ldapmodify or parallel-update tools. It can also be decrypted with the encrypt-file tool or accessed programmatically using the PassphraseEncryptedInputStream class in the UnboundID LDAP SDK for Java.
  • The server will generate an administrative alert whenever it begins the export process.

The output will be formatted as LDIF entries. By default, it will only include entries that contain reversibly encoded passwords, and the entries that are exported will only include the entry’s DN and the clear-text password. You can also use it to perform a full LDIF export, including entries that do not have passwords or that have passwords with a non-reversible encoding, and including other attributes in those entries.

Password Quality Requirements Information in the ds-pwp-state-json Virtual Attribute

We have updated the ds-pwp-state-json virtual attribute provider to add support for a new password-quality-requirement field whose value is a JSON object that provides information about the requirements that passwords will be required to satisfy. This information was already available through the “get password quality requirements” extended request or the “password validation details” request control, but including it in the ds-pwp-state-json attribute makes it easier to access by generic LDAP clients that aren’t written with the UnboundID LDAP SDK for Java, and it also makes it available to non-LDAP clients (for example, those using the Directory REST API).

The value of the password-quality-requirement field will be an array of JSON objects, each of which describes a requirement that passwords will be required to satisfy. These JSON objects may include the following fields:

  • description — A human-readable description for the requirement.
  • client-side-validation-type — A string that identifies the type of validation being performed. In some cases, clients may be able to use this field, along with information in the client-side-validation-properties field to pre-validate passwords to determine whether it is acceptable before sending it to the server, in order to give a better end-user experience. For example, if passwords may be required to meet certain length requirements, the client-side-validation-type value will be “length”, and it can use the min-password-length and max-password-length properties to convey the minimum and maximum (if defined) lengths for the password. Of course, some types of validation (for example, checking a server-side dictionary of prohibited passwords) can’t readily be performed by clients.
  • client-side-validation-properties — An array of properties that provide additional information about the requirement that may be useful in client-side validation. Each element of the array will be a JSON object with the following fields:

    • name — The name for the property.
    • value — The value for the property. Note that this will always be formatted as a JSON string, even if the value actually represents some other data type, like a number or Boolean value.
  • applies-to-add — Indicates whether this requirement will be imposed for new entries that are added with the same password policy as the entry to which the ds-pwp-state-json value is associated.
  • applies-to-self-change — Indicates whether this requirement will be imposed when the user is changing their own password.
  • applies-to-administrative-reset — Indicates whether this requirement will be imposed when the user’s password is being reset by an administrator.
  • applies-to-bind — Indicates whether this requirement may be checked during bind operations.

Augmenting Default Crypto Manager Cipher Suites

By default, the server automatically chooses an appropriate set of TLS cipher suites to enable. It tries to maintain a good balance between making it possible to use the strongest available security for clients that support it but still allowing reasonably secure alternatives for older clients that may not support the most recent options. However, it is also possible to override this default configuration in case you want to customize the set of enabled cipher suites (for example, if you know that you won’t need to support older clients, you can have only the strongest suites enabled).

The set of enabled cipher suites can be customized for each connection handler (for example, if you want to have different suites available for LDAP clients than those using HTTP). For any other communication that the server may use (for example, to secure replication communication between instances), that may be customized in the crypto manager configuration.

It has always been possible to explicitly state which TLS cipher suites should be enabled, using the ssl-cipher-suite property in either the connection handler or crypto manager configuration. In that case, you can simply list the names of the suites that should be enabled. For connection handlers, we have also offered the ability to augment the default set of suites to mostly use those values, but to also enable or disable certain suites by prefixing the name with a plus or minus sign. For example, specifying a value of “-TLS_RSA_WITH_AES_256_CBC_SHA256” indicates that the server should omit the “TLS_RSA_WITH_AES_256_CBC_SHA256” suite from the default set of cipher suites that would otherwise be available. However, we overlooked providing the ability to add or remove individual suites in the crypto manager configuration in previous releases. That has now been fixed.

Improvements in Delaying Bind Responses

In the 7.2.0.0 release, we made it possible to delay the response to any failed LDAP bind request as a means of rate-limiting password guessing attacks. This provides a great alternative to locking an account after too many failed bind attempts because it can dramatically impede how quickly clients can make guesses while also preventing the opportunity for malicious clients to intentionally lock out legitimate users by using up all of the allowed failures.

In the 8.1.0.0 release, we added the ability to configure the password policy to take some alternative action instead of locking the account after the specified number of failed attempts, and one of the available options is to delay the bind response. Although similar to the earlier feature, this is a little better for a couple of reasons:

  • It will only start delaying bind responses after a few failed attempts. This can help avoid any visible impact for users who simply mis-type their password one or two types. The delay will only kick in after the configured failure count threshold has been reached.
  • It will also delay the response to the first successful bind after the failure count threshold has been reached. This makes the delay even more effective for deterring malicious clients undertaking guessing attacks because it prevents them from detecting when the response has been delayed, terminating the connection, and establishing the new one to try again without waiting for the full configured delay period to elapse.

In the 8.1.0.0 release, this ability was limited to LDAP clients because the server can efficiently impose this delay for LDAP clients without holding up the worker thread being used to process the operation (thereby making it available to immediately start processing another request). This is unfortunately not possible for non-LDAP clients, like those using HTTP (for example, via SCIM or the Directory REST API).

In the 8.2.0.0 release, we are adding support for delaying the responses to non-LDAP clients. This provides better protection for servers in which clients may communicate through means other than LDAP, but we still can’t do it without tying up the thread processing the request, so you have to opt into it by setting the allow-blocking-delay property to true in the configuration for the delay bind response failure lockout action. As a means of mitigating the potential for tying up a worker thread, we recommend bumping up the number of HTTP request handler threads so that it’s less likely that they will all be consumed by delayed responses.

To improve security for administrative accounts, we are also updating the password policy that is used by default for root users and topology administrators so that it will impose a default delay of one second after five failed authentication attempts. This can help rate-limit password guessing attacks for these high-value accounts. This default will not be imposed for other users by default, but it is possible to enable this feature in other password policies if desired.

Using /dev/urandom for Secure Random Data

Being able to obtain secure random data (that is, data that is as unpredictable as possible) is critical to many types of cryptographic operations. By default, on Linux and other UNIX-based systems, the Java Virtual Machine relies on /dev/random to get this secure random data. This is indeed an excellent source of secure random data, and it relies on a pool of entropy that is managed by the underlying operating system and can be replenished by unpredictable inputs from a variety of sources.

Unfortunately, attempts to read from /dev/random will block if the operating system doesn’t have enough entropy available. This means that if the server experiences a surge of cryptographic processing (for example, if it needs to conduct TLS negotiation for a lot of clients all at once), it can run out of entropy, and attempts to retrieve additional secure random data can block until the pool is replenished, causing the associated cryptographic operations to stall for a noticeable period of time.

To address this, we are updating the server to use /dev/urandom instead of /dev/random as the source of secure random data. /dev/urandom behaves much like /dev/random in that it can be used to obtain secure random data, but attempts to read from /dev/urandom will never block if the pool of entropy runs low, and therefore cryptographic operations that rely on access to secure random data will never block.

While there is a common misconception that using /dev/urandom is notably less secure than /dev/random, this is really not true in any meaningful way. See https://www.2uo.de/myths-about-urandom/ for a pretty thorough refutation of this myth.

Improvements to Self-Signed Certificates and Certificate Signing Requests

If you enable secure communication when setting up the server, you’re given the option to use an existing certificate key store or to generate a self-signed certificate. When creating a self-signed certificate, we previously used a twenty-year lifetime so that administrators wouldn’t need to worry about it expiring. However, some TLS clients (and especially web browsers, as may be used to access the Administration Console, Delegated Administration, or other HTTP-based content) are starting to balk at certificates with long lifetimes and are making it harder to access content on servers that use them. As such, we are reducing the default lifetime for client-facing self-signed certificates from twenty years to one year.

When an installed certificate is nearing expiration, or if you need to replace it for some other reason, you can use the replace-certificate tool. If you choose to replace the certificate with a new self-signed certificate, that will also now have a default validity of one year. However, you can also generate a certificate signing request (CSR) so that you can get a certificate issued by a certification authority (CA), and we have updated that process, too.

When generating a CSR, you need to specify the subject DN for the desired certificate. The tool will list a number of attributes that are commonly used in subject DNs, but it previously only stated the subject DN should include at least the CN attribute. In accordance with CA/Browser Forum guidelines, we have updated this recommendation, and we now suggest including at least the CN, O, and C attributes in the requested subject DN. We have also updated the logic used to select host names and IP addresses for inclusion in the subject alternative name extension, including:

  • When obtaining the default set of DNS names to include in the subject alternative name extension, we no longer include unqualified names, and we will warn about the attempt to add any unqualified host names.
  • When obtaining the default set of DNS names to include in the subject alternative name extension, we no longer include names that are associated with a loopback address.
  • When obtaining the default set of IP addresses to include in the subject alternative name extension, we no longer include any IP addresses from IANA-reserved ranges (including loopback addresses or those reserved for private-use networks) and will warn about attempts to add any such addresses.

New Administrative Alert Types

We have defined new types of administrative alerts that can be used to notify administrators of certain events that occur within the server. These include:

  • We have added a new admin alert account status notification handler. If this is enabled and associated with a password policy, then the server can generate an administrative alert any time one of the selected account status notification events occurs. This is primarily intended for use with high-value accounts, like root users or topology administrators. For example, you can have the server generate an alert if a root user’s password is updated or if there are too many consecutive failed authentication attempts. This account status notification handler will use a different alert type for each type of account status notification (for example, it will use an alert type of password-reset-account-status-notification for the alert generated if a user’s password is reset by an administrator).
  • We have added a new privilege-assigned alert type that will be raised whenever any privileges are assigned. This includes creating a new entry with one or more explicitly defined privileges, updating an existing entry to add one or more explicitly defined privileges, creating a new root user or topology administrator that inherits a default set of root privileges, updating the default set of root privileges, or creating a new virtual attribute that assigns values for the ds-privilege-name attribute.
  • We added a new insecure-request-rejected alert type that will be raised if the server rejects a request received over an insecure connection as a result of the reject-insecure-requests global configuration property.

Monitor Improvements

We have updated the system information monitor entry to restrict the set of environment variables that it may contain. Previously, this monitor entry would include the names and values of all environment variables that are available to the server process, but this has the potential to leak sensitive information because some deployments use environment variables to hold sensitive information (for example, credentials or secret keys used by automated processes). To avoid leaking this information to clients with permission to access server monitor data, we now only include information about a predefined set of environment variables that are expected to be useful for troubleshooting purposes and is not expected to contain sensitive information.

We have also added a new isDocker attribute to the server information monitor entry to indicate whether the server is running in a Docker container.

The JVM memory usage monitor entry has been updated to fix an issue that could prevent it from reporting the total amount of memory held by all memory consumers (that is, components of the server that may be expected to consume a significant amount of data), and to fix an issue that could cause it to use an incomplete message for consumers without a defined maximum size. We have also added an additional memory-consumer-json attribute whose values are JSON objects with data that can be more easily be consumed by automated processes.

We have updated the status tool to optimize some of the searches that it uses to retrieve monitor data.

Identity Mapper Improvements

Identity mappers can be used to map a username or other identifier string to the entry for the user to which it refers. We have made the following identity mapper-related improvements in the server:

  • We have updated the exact match and regular expression identity mappers to make it possible to include only entries that match a given filter.
  • We have added an aggregate identity mapper that can be used to merge the results of other identity mappers with Boolean AND or OR combinations.
  • We have added new identity mapper instances in the out-of-the-box configuration for matching only root users, only topology administrators, and either root users or topology administrators.

Uniqueness Control Improvements

The uniqueness request control may be included in an add, modify, or modify DN request to request that the server should not allow the operation if it would result in multiple entries with the same value for a specified attribute (or a set of values for a set of attributes). It operates in two main phases:

  • In the pre-commit phase, the server can search for entries that already exist and conflict with the requested change. If a conflict is found in the pre-commit phase, then the operation will be rejected so that it is not allowed to create the conflict.
  • In the post-commit phase, the server can once again search for conflicts. This can help detect any conflicts that may have arisen as a result of multiple changes being processed concurrently on the same or different servers.

In the 8.2.0.0 release, we have added two enhancements to our support for the uniqueness request control:

  • It is now possible to indicate that the server should create a temporary conflict prevention details entry before beginning pre-commit processing. This is a hidden entry that won’t be visible to most clients, but that can make the server more likely to detect conflicts arising from concurrent operations so that they can be rejected before the conflict arises. While this dramatically increases the chance that concurrent conflicts will be prevented, it does incur an additional performance penalty, and it is disabled by default.
  • It is now possible to indicate that the server should raise an administrative alert if a uniqueness conflict is detected during post-commit processing. If a conflict is identified in the post-commit phase, then it’s too late to prevent it, but it is now at least possible to make it easier to be brought to the attention of server administrators so they can take any corrective action that may be necessary. This is enabled by default.

Improved Directory Proxy Server SASL Support

We have updated the Directory Proxy Server to provide better support for the PLAIN, UNBOUNDID-DELIVERED-OTP, UNBOUNDID-TOTP, and UNBOUNDID-YUBIKEY-OTP SASL mechanisms.

Previously, the Directory Proxy Server attempted to process these operations in the same way as the backend Directory Servers. It would retrieve the target user’s entry from the backend server, including the encoded password and any additional data needed in the course of processing the bind (for example, the user’s TOTP secret when processing an UNBOUNDID-TOTP bind). However, this would fail if the server backend server was configured to block external access to this information. Now, the Directory Proxy Server has been updated to simply forward the bind request to the appropriate backend server for processing, which eliminates the potential for problems in environments in which the Directory Proxy Server cannot get access to all necessary data in the target user’s entry.

UnboundID LDAP SDK for Java 5.1.3

We have just released version 5.1.3 of the UnboundID LDAP SDK for Java. It is available for download from GitHub and SourceForge, and it is available in the Maven Central Repository.

The biggest change in this release addresses an issue in the LDAP listener framework (including the in-memory directory server). The listener did not adequately protect against the case in which a malicious or errant client could send an LDAP request encoded as an ASN.1 BER sequence with a very large value length, which could result in the listener attempting to allocate up to two gigabytes of memory. The LDAP listener framework will now impose a maximum request size of 20 megabytes by default, which is the same as the default maximum size that the LDAP SDK will impose for responses read from a directory server. The maximum request size can be configured using the InMemoryDirectoryServerConfig.setMaxMessageSizeBytes method when using the in-memory directory server, or using the LDAPListenerConfig.setMaxMessageSizeBytes method when using the more general LDAP listener framework. If you’re using the LDAP listener framework (or the in-memory directory server) to accept requests from potentially untrusted clients, then we recommend upgrading to the 5.1.3 release.

Other changes since the 5.1.2 release include:

  • We have updated OID support to add methods for interacting with object identifiers in a hierarchical manner. It is now possible to create a new OID that is a child of a provided OID with a given subordinate component value. You can also get the parent for a provided OID and determine whether one OID is an ancestor or descendant of another.
  • We have updated the oid-lookup tool to add a new --exact-match argument that will cause it to only return items in which the OID, name, type, origin, or URL exactly matches the provided search string. The tool continues to use substring matching by default.
  • We have updated the ldap-result-code tool to add a new --output-format argument that allows you to customize whether the output should be formatted as a human-readable table, comma-separated values, tab-delimited text, or JSON objects. It will continue to format result codes in a table by default.

UnboundID LDAP SDK for Java 5.1.2

We have just released version 5.1.2 of the UnboundID LDAP SDK for Java. It is available for download from GitHub and SourceForge, and it is available in the Maven Central Repository. The release notes provide a pretty comprehensive overview of what’s changed since the 5.1.1 release, but here’s a summary:

Server-Agnostic Updates

  • We added a new parallel-update command-line tool that can be used to apply changes read from an LDIF file against an LDAP server using multiple concurrent threads.
  • We updated the ldapmodify and ldapdelete tools so that they will now default to retrying failed operations on a newly established connection if the failure suggests that the connection made for the initial attempt is no longer valid. This was previously available through the --retryFailedOperations argument, but it is now the default behavior, and a --neverRetry argument can be used if retry support is not wanted.
  • We added a new OIDRegistry class that provides a registry of object identifiers used in LDAP, including things like schema elements, controls, extended operations, and other sources. Each item in the registry has a name, an OID, and a type, and it may also have an origin string and a URL that may be used to retrieve additional information about the item.
  • We added a new oid-lookup command-line tool that can be used to search the OID registry to find items with a given OID, name, type, or other content.
  • We added a new ldap-result-code command-line tool that can be used to list all defined result codes and search for result codes with a given name or integer value.
  • We updated the LDAP listener framework and the in-memory directory server to support mutual TLS authentication, in which the server requests a certificate chain from the client and validates any chain that the client provides.
  • We updated the logic used to generate temporary self-signed certificates for use by the in-memory-directory-server and ldap-debugger command-line tools so that the certificates will be more acceptable to a wider range of clients. The certificate will now only be valid for one year, as some TLS clients balk at peer certificates with very long lifetimes. The subject alternative name extension will also default to only using canonical host names and IP addresses that are associated with non-loopback interfaces.
  • We added a new argument value validator that can be used to restrict command-line argument values to those that represent valid DNS host names. It can optionally accept or reject values that are numeric IP addresses, can accept or reject unqualified host names, and can accept or reject unresolvable host names.
  • We added a new SASLClientBindHandler class that can be used to invoke SASL bind operations that use a javax.security.sasl.SaslClient object to perform the actual SASL processing. We also added an LDAPConnection.applySASLSecurityLayer method that can be used to add a security layer that has been negotiated with a SaslClient to an established, clear-text connection.
  • We updated the HostNameSSLSocketVerifier to automatically trust any certificate when the client used a loopback IP address (not a host name, even if that name is associated with a loopback interface) as the address for the server. This is in accordance with the W3C Secure Contexts Candidate Recommendation, section 3.2.
  • We updated the HostNameSSLSocketVerifier and TrustAllSSLSocketVerifier classes to implement the javax.net.ssl.HostnameVerifier interface.
  • We updated the LDIF writer to improve human readability for the comments that it can automatically add for base64-encoded values.
  • We updated the manage-certificates command-line tool to add support for a new retrieve-server-certificate subcommand, which will retrieve the certificate chain from a specified server and optionally write a PEM-encoded or DER-encoded representation of the certificate(s) to a specified file.
  • We fixed an issue with the manage-certificates trust-server-certificate command that only caused it to wait for 60 milliseconds rather than the intended 60 seconds when trying to establish a connection to the target server.
  • We updated ldapsearch to add a new “dns-only” output format. If this format is selected, then the output will contain only the DNs of matching entries.
  • We updated support for the OAUTHBEARER SASL mechanism to make it possible to include arbitrary key-value pairs in the SASL credentials.
  • We fixed an issue in the command-line tool framework that affected tools that provide support for automatically writing the output to a file. The output file was never explicitly closed, which could cause problems in cases where the tool was invoked programmatically, and then an attempt was made to subsequently use the output from code in the same JVM.
  • We added a new StaticUtils.isIANAReservedIPAddress method that can be used to determine whether a provided IPv4 or IPv6 address is contained in an IANA-reserved range.
  • We updated the ChangeLogEntry class so that it will fall back to using the includedAttributes attribute if the deletedEntryAttrs attribute does not exist in changelog entries that represent a delete operation.
  • We updated the LDAP SDK’s support for passphrase-based encryption to make it possible to explicitly specify the type of cipher that should be used. Previously, you could only request either a baseline level of protection (which should be available in all supported Java versions) or the strongest supported level of protection (which might not be available in some JVMs)
  • We added a new X.509 trust manager that will never trust any certificate chain. This is primarily intended for testing purposes.

Updates Specific to the Ping Identity Directory Server

  • We added client-side support for a new ds-pwp-modifiable-state-json operational attribute that can be used to retrieve and manipulate a select set of password policy state information in a user’s entry.
  • We added support for a new “remove attribute type” administrative task that can be used to safely remove an attribute type definition from the server schema.
  • We added client-side support for interacting with passwords encoded with the new AES256 password storage scheme.
  • We updated the Filter.matchesEntry method to add limited support for extensible matching filters that contain both an attribute type and a matching rule ID, when the dnAttributes flag is not set, and when the matching rule ID targets the jsonObjectFilterExtensibleMatch matching rule.
  • We updated the uniqueness request control to make it possible to indicate whether the server should raise an administrative alert if it detects a conflict during the post-commit validation phase.
  • We updated the uniqueness request control to make it possible to indicate that the server should create a temporary conflict prevention details entry before beginning pre-commit processing as a means of better preventing conflicts that arise from concurrent changes in different servers.
  • We deprecated support for interactive transactions, whose use has been discouraged for many releases, and for which server-side support is being removed.

UnboundID LDAP SDK for Java 5.1.1

We have just released version 5.1.1 of the UnboundID LDAP SDK for Java. It is available for download from GitHub and SourceForge, and it is available in the Maven Central Repository. The release notes provide a pretty comprehensive overview of what’s changed since the 5.1.0 release, but here’s a summary:

Server-Agnostic Updates

  • We had added new @NotNull and @Nullable annotation types and updated the entire LDAP SDK codebase to mark all non-primitive fields, parameters, and method return values to indicate whether they may be null. These annotations will appear in the generated Javadoc documentation, and they will also be available at runtime for introspection by IDEs and other tools.
  • We have improved the logic used to validate certificate hostnames. The LDAP SDK now does a better job of handling hostnames with wildcards, and it does a better job of handling cases in which the connection was established with an IP address rather than a hostname. There is also an option to indicate whether to treat a certificate’s subject alternative name extension (if present) as the only authoritative source of allowed hostnames or to also allow looking at the CN attribute in the certificate subject DN even if the certificate contains a subject alternative name extension.
  • We fixed an issue that could prevent command-line tools that support subcommands from performing all appropriate validation when running in interactive mode. The command-line tool’s interactive mode framework neglected to perform required, dependent, and exclusive argument set validation for the selected subcommand, which could cause the tool to run with an inappropriate set of arguments.
  • We fixed issues in the code used to format strings in the comma-separated values (CSV) format. Previously, all ASCII control characters and all non-ASCII characters were silently dropped from the output. They are now included, but the value will be in quotes (and may span multiple lines if the value to format includes line breaks). Further, it had previously used the backslash character to escape any double quotes in the data (\"), but RFC 4180 indicates that each double-quote character should be escaped by preceding it with another double quote character (""). The LDAP SDK now uses the RFC-specified behavior as the default, but it is possible to fall back to the former backslash-based encoding if desired or needed for backward compatibility.
  • We updated ldapsearch to add multi-valued-csv and multi-valued-tab-delimited output formats. The existing csv and tab-delimited output formats only include the first value for multi-valued attributes, while the multi-valued variants use the vertical bar character (|) as a delimiter between values.
  • We updated the ldappasswordmodify command-line tool to default to using the password modify extended operation if it is unable to retrieve the server’s root DSE while attempting to determine which method to use to change the password. The tool would previously exit with an error if the --passwordChangeMethod argument was not provided and it couldn’t retrieve the root DSE to determine an appropriate method.
  • We updated the authrate command-line tool so that the --filter argument is not required if the --bindOnly argument is provided.
  • Updated the ldapcompare tool so that it always uses an exit code of zero (corresponding to the LDAP success result code) by default if all compare operations are processed successfully, regardless of whether the assertions matched or did not match the target entries. Previously, the tool would use an exit code of 5 or 6 (corresponding to the LDAP compareFalse and compareTrue result codes, respectively) if only a single compare assertion was processed and completed with the corresponding result code. However, returning a nonzero exit code by default can cause problems with scripts that invoke the tool and expect that a nonzero result code indicates that an error occurred. The new --useCompareResultCodeAsExitCode argument can be used to request the previous behavior.
  • We updated the ldapcompare tool to allow reading the raw assertion value from a file. If this option is used, then the attribute name or OID should be followed by a colon, a less-than sign, and the path to the file from which the value should be read (for example, “cn:</path/to/asserted-cn-value.txt”). If this option is used, then the exact bytes of the file (including line breaks) will be used as the assertion value for the compare operation.
  • We updated the ldifsearch. tool so that all non-LDIF output is written as LDIF comments (preceded by the octothorpe character, #) so that it will not interfere with the ability to parse the remaining output as LDIF.
  • We added support for the OAUTHBEARER SASL mechanism, as described in RFC 7628.
  • We updated the LDAP command-line tool framework to add support for authenticating with additional SASL mechanisms, including OAUTHBEARER, SCRAM-SHA-1, SCRAM-SHA-256, and SCRAM-SHA-512.
  • We fixed issues with the ldifsearch, ldifmodify., and ldif-diff tools that could arise if they were run in a manner that would cause the output to be both compressed and encrypted. The tool incorrectly attempted to compress the output after it was encrypted rather than before, making the compression ineffective and the output incompatible with tools that expect compression to be applied before encryption.
  • We fixed an issue with the ldifsearch tool that could prevent it from properly finalizing the output when using compression or encryption, potentially leaving buffered data unwritten.
  • We fixed an issue with the ldifmodify tool that caused it to use a nonzero exit code if it was only used to add new entries to a previously empty source LDIF file.
  • We updated the ldifmodify tool to use lenient mode by default when applying modifications. It would previously reject attempts to add attribute values that already existed or remove attribute values that do not exist, but this could cause problems with applications that did not expect this behavior, as a legacy version of the tool used lenient mode by default. A new --strictModifications argument has been added that can request the strict validation mode if desired.
  • We updated the LDAP SDK’s command-line tool framework so that when displaying an example command that is split across multiple lines, it will use an appropriate continuation character for the underlying platform. It previously always used the backslash character (\), which is correct for UNIX-based systems, but it will now use the caret character (^) when running on Windows systems.
  • We fixed an issue with the ldifsearch tool that caused its usage output to include example arguments and descriptions intended for use with the ldif-diff tool.
  • We fixed an issue in the manage-certificates tool usage output that caused the generate-certificate-signing-request subcommand’s --key-size-bits argument to use the wrong description.

Updates Specific to the Ping Identity Directory Server

  • We added support for a new “get recent login history” control that can be included in a bind request to indicate that the bind response (if authentication was successful) should include information about other recent successful and failed authentication attempts for that user. The ldapsearch and ldapmodify commands have been updated to provide support for this control. We also updated support for the password modify extended operation, the manage-account command-line tool, and the ds-pwp-state-json virtual attribute to provide support for retrieving a user’s recent login history.
  • We updated support for the password modify extended operation, the manage-account tool, and the ds-pwp-state-json virtual attribute to provide support for retrieving state information about password validation performed during bind operations, including the time that validation was last performed and whether the account is locked because the bind password failed validation.
  • We updated support for the ds-pwp-state-json virtual attribute to provide support for retrieving information about the quality requirements that the user’s password must satisfy.
  • We updated the set of potential authentication failure reasons to include an additional failure type for cases in which a password used in a bind request failed to satisfy one or more of the configured password validators.
  • We added a new password policy state account usability error that may be used if an account is locked because the user attempted to authenticate with a password that failed to satisfy one or more of the configured password validators.
  • We added a new password policy state account usability warning that may be used if an account contains a password that is encoded with a deprecated password storage scheme.
  • We updated the collect-support-data tool to add the ability to specify the amount of data to capture from each log file to be included in the support data archive. We have also updated client-side support for the administrative task and extended operation that may be used to invoke collect-support-data processing against a remote server to include support for the new arguments.

Ping Identity Directory Server 8.1.0.0

We have just released the Ping Identity Directory Server version 8.1.0.0. It’s a pretty big release with several new features, enhancements, and bug fixes. You can keep reading (or see the release notes) for a pretty comprehensive overview of what’s included, but here are the highlights:

  • Composed attributes, which are like constructed virtual attributes, except that their values are computed when the entry is created or updated, and they can be indexed for much faster searches.
  • A new JSON-formatted virtual attribute that exposes all kinds of password policy state information.
  • Alternative failure lockout actions, like delaying the bind response after too many failed authentication attempts instead of actually locking the account.
  • Support for SASL integrity and confidentiality when using GSSAPI.
  • Authentication support for the file servlet, and a default instance that allows administrators to access files below the server instance root over HTTPS.
  • It’s easier to run collect-support-data and manage-profile generate-profile remotely without needing command-line access to the server.
  • Improvements to the character set and attribute-value password validators, and significant updates to the banned password lists used by the dictionary password validator.
  • Several command-line tool improvements.
  • Performance improvements.

New Features

We added a new composed attribute plugin, which allows the server to generate values for an attribute from a combination of static text and the values of other attributes in the same entry. Composed attributes behave much like constructed virtual attributes, but the composed values are computed when the entry is created or updated, and they are stored in the database. This means that the values can be indexed, and it is possible to compose values for attributes that are required by one or more of the entry’s object classes. A new “populate composed attribute values” task allows you to generate composed values for entries that already existed at the time the composed attribute was enabled.

We added support for a new ds-pwp-state-json virtual attribute whose value is a JSON object with a comprehensive overview of the associated user’s password policy state and related configuration from their password policy. The UnboundID LDAP SDK provides enhanced support for extracting data from this virtual attribute, but any JSON API should be able to parse the value.

We updated the password policy to add support for alternative failure lockout actions. As an alternative to completely locking a user’s account after too many failed authentication attempts, it is now possible to delay the responses to subsequent bind operations as a way of imposing a rate limit on authentication attempts. It is also possible to use a “no-op” action that does not have any client-observable effect but can be used to make administrators aware of the issue through the server’s support for account status notifications.

We updated our support for the GSSAPI SASL mechanism to include support for the auth-int and auth-conf quality of protection (QoP) values. We previously supported GSSAPI for authentication only, but it is now possible to sign or encrypt all subsequent communication on the connection with a key negotiated during the authentication process.

We updated the file servlet to support HTTP basic authentication. If authentication support is enabled, you can optionally restrict access by group membership or with the new file-servlet-access privilege (which is included in the set of default privileges that can be automatically inherited by root users or topology administrators). The server is now configured with an instance of this file servlet at /instance-root that provides access to files within the server instance root to users with the file-servlet-access privilege. This servlet provides more convenient access to server files for instances running in containers or other environments where filesystem access is not readily available.

Enhancements

We made several improvements to the collect-support-data tool. They include:

  • The server now supports an extended operation that a remote client can use to invoke the collect-support-data tool on the server and stream the resulting archive back to the client. This can be especially useful if the server is running in a container or other kind of environment without convenient command-line access. This extended operation can be requested by adding the --useRemoteServer argument to the command line, or it can be used programmatically through the UnboundID LDAP SDK for Java.
  • The server now allows running collect-support-data as either a one-time task or a recurring task. The recurring task allows the server to automatically generate support data archives at regular intervals. The UnboundID LDAP SDK for Java has been updated to allow you to create an instance of the task programmatically.
  • The tool now offers an --outputPath argument that allows you to specify the path for the resulting support data archive. The path can target either a file or a directory, and if you specify a directory, then the file will be created in that directory with a name that the tool generates.
  • If the Delegated Admin application is configured in the server, collect-support-data will now capture its config.js configuration file, its version file, and any files used to create custom UI fields.

We made several improvements to the manage-profile tool, including:

  • The server now allows running manage-profile generate-profile as either a one-time task or a recurring task. The recurring task allows the server to generate an up-to-date profile at regular intervals. The UnboundID LDAP SDK for Java has been updated to allow you to create an instance of the task programmatically.
  • The manage-profile generate-profile subcommand now offers a --zip argument that will cause it to package the generated server profile in a zip file.
  • The manage-profile generate-profile subcommand now excludes the contents of the server’s bak and ldif directories by default, which can make the resulting server profiles much smaller.
  • We fixed an issue that could prevent the manage-profile replace-profile subcommand from creating new local DB backends through dsconfig batch files.
  • We fixed an issue that could prevent the manage-profile replace-profile subcommand from correctly exporting and re-importing data from a server with multiple backends.
  • We fixed an issue that could cause the server to warn about unexpected offline configuration changes the first time it was started after running manage-profile setup or manage-profile replace-profile.
  • We updated the manage-profile replace-profile subcommand so that it always requires a license file in the profile to apply. It is now easier to install a new license when updating the server to a new version.
  • We updated the manage-profile replace-profile subcommand so that it will check for any encryption-related arguments in the setup-arguments.txt file. If they are present, it will export the data to LDIF before applying the new profile, and it will re-import it after applying the profile.
  • We fixed an issue in which manage-profile replace-profile could fail to update recurring task chains.
  • We fixed an issue in which manage-profile replace-profile would not allow you to enable the LDAP changelog backend.

We made several updates to our SCIM support, including:

  • The server now allows you to join separate objects that will be returned as a single SCIM 2 resource object.
  • We updated our support for SCIM 2 PATCH operations so that they now require the schemas request attribute to be “urn:ietf:params:scim:api:messages:2.0:PatchOp” as per RFC 7644.
  • We updated SCIM 1.1 to use JSON-formatted responses by default when the request does not specify the expected content type.
  • We fixed a potential memory leak that could arise when processing SCIM requests.

The server now uses the 5.1.0 release of the UnboundID LDAP SDK for Java. In addition to several behind-the-scenes improvements, this also includes several improvements that are reflected in command-line tools, like:

  • Better default certificate trust settings for many of the tools that are provided with the server. In most cases, if you want to use a secure connection but don’t specify any trust-related arguments, the tool will automatically trust the certificates for any server in the topology.
  • Many of the tools that offer an interactive mode now use a more streamlined flow for obtaining the information needed to connect and authenticate to the server. It will prefer secure communication over insecure, and it can read the server configuration to determine the default port to suggest for the connection.
  • We updated ldapsearch to improve its usefulness in shell scripts. It is easier to extract attribute values to assign to script variables, and you can optionally have the tool exit with an error if the search completes successfully but does not return any entries.
  • We made several updates to the summarize-access-log tool so that it can now report on a lot of additional things like the use of TLS protocols and cipher suites, the most common authentication and authorization DNs, and information about the most expensive or biggest searches.

We improved performance for index-related processing when importing data from LDIF.

We improved the server’s performance when updating a composite index key that matches a very large number of entries.

We added support for caching password policies defined in user data rather than in the configuration, which can improve performance for password policy-related processing for entries making use of those policies.

We updated the character set password validator to make it possible to indicate that passwords must include characters from at least a specified number of character sets. For example, if you define sets that include lowercase letters, uppercase letters, digits, and symbols, you can require passwords to contain characters from at least three of those sets.

We updated the attribute-value password validator to make it possible to specify a minimum substring length when checking to see if the password contains the value for any other attribute in the user’s entry. This can help avoid problems with entries that have attributes with short values (especially values that are just one or two characters long).

We added a new “Replication Purge Delay” gauge that can help prevent administrators from setting a replication purge delay that is too low. If a server instance is offline for longer than the purge delay, it can be unable to re-join the replication topology when it is started because it missed changes that are no longer available in any of the other replicas.

We improved the logic that the server uses to automatically select an appropriate set of TLS cipher suites. It now does a better job of prioritizing the order for the cipher suites it selects, including preferring TLSv1.3-specific suites if they are available. This logic will also be used when creating LDAP client connections in command-line tools or server components that establish secure connections to other servers. This automatic selection can still be overridden by explicitly specifying the set of TLS cipher suites that should be used.

We updated the create-systemd-script tool so that it generates a forking service file, which is a better fit for the server because the process ultimately used to run the server is different from the start-server script used to launch it.

We updated the installer to require a minimum Java heap size of 768 megabytes when setting up the server.

We improved the logic used to automatically select the cache size for the replication database. The previously selected cache size could be too small under certain circumstances, which could cause the replication database to have an unnecessarily large on-disk footprint.

We updated the status tool to use more efficient search requests when retrieving replication state information.

We changed the behavior of the bypass-pw-policy privilege. It could previously allow a user to exempt themselves from certain password policy restrictions, but it will now only apply for operations against other users. A user with this privilege will be permitted to do the following:

  • Set a pre-encoded password for another user, even if that user’s password policy does not allow pre-encoded passwords.
  • Set a password for another user that does not satisfy all of the password validators in that user’s password policy.
  • Set a password for another user that is already in that user’s password history.

We updated the server to add an option to enter lockdown mode and report itself as unavailable if an error occurs while attempting to write to a log file.

We added a consent REST API that allows users to create and store consents, and to allow users to search for consents that have been granted to them.

We updated the commonly-used-passwords.txt file to include lots of additional values, especially from studies released at the end of 2019. We also updated wordlist.txt to add many additional English words, as well as words from several other languages. Both of these files can be used by the dictionary password validator to reject passwords that are likely to be guessed by attackers.

We added a global ACI that allows clients to use the pre-read and post-read controls by default. The server will only process these controls if the requester has permission to perform the associated write operation, and it will only include attributes in the pre-read or post-read entry that the client has permission to access.

We added an “--addBaseEntry” argument to dsreplication enable. If provided, the server will create the base entry in the target backend if it does not already exist. The base entry must be present in the backend when enabling replication for that backend.

We updated the general monitor entry to include locationName and locationDN attributes that can be used to determine the server’s location.

We updated the server so that it will log a warning if it is running on a Linux system with a memory control group that may allow portions of the process memory to be swapped out to disk.

We updated the HTTP connection handler to make it possible to require clients to present their own certificates to the server.

We added a new HTTP Processing (Percent) gauge that can be used to help monitor the server’s capacity for processing additional HTTP requests.

We updated the server to make information in the general monitor entry (that is, the “cn=monitor” entry itself) available over JMX. Previously, the server exposed information about monitor entries that exist below the general monitor entry, but not the general monitor entry itself.

We updated the LDAP external server configuration so that the use-administrative-operation-control property is only offered for specific types of server instances that support that control.

We updated the StatsD monitoring endpoint so that it no longer uses spaces, commas, or colons in metric names. Those characters are now replaced with underscores. We also remove single and double quotes from metric names.

We updated the Server SDK to provide methods for retrieving the name or DN of the client connection policy from a ClientContext or OperationContext.

We updated the server to provide support for additional debug logging when invoking Server SDK extensions.

We updated the server so that it will not allow changes to data below “cn=Cluster,cn=config” if the cluster contains servers with different versions.

We updated the server so that it will now warn if there are multiple different versions of the same library in the server classpath.

Bug Fixes

We fixed an issue in which retrieving the “cn=version,cn=monitor” entry could cause the underlying JVM to leak a small amount of memory.

We fixed a potential memory leak that could arise when processing SCIM requests.

We fixed an issue that could prevent changing the password for a topology administrator unless their password policy was configured to allow pre-encoded passwords.

We fixed an issue that could cause mirrored subtree polling to create unnecessary files in the server’s configuration archive.

We fixed an issue that could prevent the server from generating certificates for systems with hostnames containing non-ASCII characters.

We fixed an issue that could interfere with the ability to install custom extensions that require additional libraries.

We fixed a rare issue that could cause a delay during TLS handshake processing.

We fixed an issue that could cause the server to raise an administrative alert about an uncaught exception when the server tried to send data over a TLS-encrypted connection that is no longer valid.

We fixed an issue that could cause certain tools (including collect-support-data, dsreplication, and rebuild-index) from being able to use a tools.properties file if that file was encrypted.

We fixed an issue that could delay the shutdown process if the server was configured to communicate with an unresponsive StatsD endpoint over TCP.

We fixed an issue that caused exec recurring tasks to ignore the configured working directory.

We fixed an issue that could prevent the server from generating an “account updated” account status notification for a modify operation that matched the associated criteria but did not include a password update.

We fixed an issue that could cause the value of the load-balancing-algorithm-name property to be lost when adding using the manage-topology add-server command to add an instance to an existing topology.

We fixed a replication issue that could arise when re-initializing a replica with data containing changes that are older than the replica previously had.

We fixed a replication issue that could cause initialization to stall if an error occurred while trying to send an internal replication message.

We fixed an issue that could cause the server to consider obsolete replicas when attempting to determine the total replication backlog.

We fixed an issue that could cause the server to incorrectly report how much memory it is using after performing an explicitly requested garbage collection.

We fixed a rare replication issue that could arise when upgrading from a pre-7.3 release in an environment where servers had been removed from the topology. The server could incorrectly detect a backlog that it would never see as resolved.

We fixed an issue with the dsjavaproperties tool that allowed you to request both aggressive and semi-aggressive tuning options at the same time.

We fixed an issue that could cause the server to report a spurious error message when disabling the PingOne pass-through authentication plugin.

We fixed an issue that could cause the server to return an attribute with the name formatted in all lowercase characters if the attribute was present in an entry but not defined in the server schema, and if the client explicitly requested that attribute to be returned. The server will now format the attribute name using the same capitalization the client used when requesting that attribute.

We fixed an issue that could prevent certain command-line tools from reporting the correct error if a problem occurred and the server was configured with a custom result code map.

UnboundID LDAP SDK for Java 5.1.0

UnboundID LDAP SDK for Java version 5.1.0 has been released and is available for download from GitHub and SourceForge, and it is available in the Maven Central Repository. The release notes provide a pretty comprehensive overview of the changes since the previous 5.0.1 release, but here’s a summary:

  • We fixed an issue in which the JVM-default trust manager did not always correctly handle cross-signed issuer certificates when the presented chain included an expired issuer certificate. It will now check to see if it can build a valid path with an alternate trust anchor.
  • We added a new SchemaValidator class that can identify all kinds of problems with LDAP schema definitions. We also provide a new validate-ldap-schema command-line tool that will examine definitions contained in one or more LDIF files and report any problems that it finds.
  • We updated the in-memory-directory-server command-line tool to validate any schema definitions provided through the --useSchemaFile argument. Even if there are problems, the server will still try to use that schema to the best of its ability (as was previously the case). The --doNotValidateSchemaDefnitions argument can be used to disable the new validation if it is not desired.
  • We added a new ldappasswordmodify command-line tool that can be used to perform a self password change or an administrative password reset. It supports the password modify extended operation (as described in RFC 3062), and it can also change passwords using a regular LDAP modify operation or using an Active Directory-specific modification.
  • We added three new command-line tools for performing operations on data contained in LDIF files:

    • The ldifsearch tool can be used to identify entries that match a given set of search criteria.
    • The ldifmodify tool can be used to apply a set of add, delete, modify, and modify DN changes to LDIF data.
    • The ldif-diff tool can be used to identify differences between data in two provided LDIF files and report the differences in the form of LDIF change records.
  • We added a new version of the ldapcompare tool that can be used to perform LDAP compare operations in a directory server. The new version offers a lot of additional functionality like support for performing multiple compare assertions and using a variety of request controls, and it can generate parseable output in tab-delimited text, CSV, or JSON formats.
  • We updated the in-memory directory server to make it possible to add custom attributes to the root DSE. While it was already possible to replace the entire root DSE entry with a static entry, this new approach makes it possible to retain some dynamic content (for example, changelog-related attributes) while still customizing other attributes.
  • We made several changes in our support for entries with the ldapSubEntry object class:

    • We added a new RFC3672SubentriesRequestControl class with support for the LDAP subentries request control as described in RFC 3672.
    • The LDAP SDK already had support for an alternate version of the control described in draft-ietf-ldup-subentry through the SubentriesRequestControl class, but that class has been deprecated in favor of a new DraftLDUPSubentriesRequestControl class, which helps avoid confusion with the class that implements the RFC 3672 version of the control. The deprecated class is still fully functional and will be kept to preserve backward compatibility, but we recommend updating code that uses the old class for the sake of clarity.
    • The in-memory directory server has been updated with support for the RFC 3672 version of the control. It already had support for the draft-ietf-ldup-subentry version.
    • The in-memory directory server has been updated so that it will return entries with the ldapSubEntry object class if the filter includes an “(objectClass=ldapSubEntry)” component.
    • The ldapsearch command-line tool has been updated with support for the RFC 3672 version of the LDAP subentries control, using the new --rfc3672Subentries argument. It already had support for the draft-ietf-ldup-subentry version of the control through the --includeSubentries argument, and that argument is still available, but we now recommend using --draftLDUPSubentries instead for the sake of clarity.
  • We updated the ldapsearch tool to add a new “values-only” output format (as an alternative to the existing LDIF, tab-delimited text, CSV, and JSON output formats). If this output format is selected, then it will only output the values of the requested attributes without any entry DNs or attribute names. This can help extract raw attribute values from a directory server from a script without the need for any additional text processing.
  • We updated the ldapsearch tool to add a new --requireMatch argument. If this argument is provided and the search completes successfully but does not return any entries, then the tool will have an exit code of 94 (corresponding to the noResultsReturned result code) rather than zero. This argument does not have any visible effect on the output.
  • We updated the round-robin and fewest connections servers sets to expose the blacklist manager that they use to avoid trying to establish connections to servers that are believed to be unavailable.
  • We updated the manage-certificates tool to make it easier to list and export certificates from the JVM’s default trust store without needing to know the path to the appropriate file.
  • We improved the logic that the LDAP SDK uses when selecting ordering and substring matching rules for ordering operations involving attributes that are defined in the schema but whose definition does not specify an ordering matching rule. It will now try to infer an appropriate ordering matching rule from the equality matching rule before trying other alternatives like inferring a rule from the associated syntax or using a default rule.
  • We updated the LDAP command-line tool framework to make it easier and more convenient to communicate securely with the Ping Identity Directory Server (and other related server products). This includes:

    • We added a new TopologyRegistryTrustManager class that can use information in the server’s topology registry to determine whether to trust the certificates for instances in the topology.
    • If no trust-related arguments are specified when running the tool, it will now check the server’s default trust store and the topology registry to determine whether the presented certificate should be trusted. It will still also check the JVM’s default trust store, and it will still fall back to interactively prompting the user if the certificate cannot be trusted through other means.
  • We streamlined the process that LDAP command-line tools use to establish and authenticate connections when run in interactive mode. It will now recommend TLS encryption over unencrypted communication with a simplified set of arguments, and it will recommend simple authentication over unauthenticated connections. Further, when the tool is part of a Ping Identity Directory Server (or related server product) installation, it will read the configuration to determine the appropriate port to suggest when connecting to the server.
  • We made several improvements to the summarize-access-log tool that can be used to examine Ping Identity Directory Server access logs. These include:

    • You can now customize the maximum number of values to display for each item. It was previously hard-coded to use a limit of 20 values. If any values were omitted, then it will now tell you how many were left out.
    • You can now choose to de-anonymize the output to obtain the specific attribute values used in search filters and entry DNs (instead of displaying question marks as placeholders).
    • The output will now include information about the most common TLS protocols and cipher suites used for secure communication.
    • The output will now include the most common successful and failed bind DNs and the most common authentication mechanisms.
    • The output will now include the most common DNs used as alternate authorization identities (e.g., via the proxied authorization request control).
    • The output will now include the most common filters used for unindexed searches, the most common base DNs for searches with non-baseObject scopes, the filters for searches taking the longest to complete, and the most common filters for searches returning zero, one, or multiple entries.
    • When summarizing the most commonly invoked types of extended operations, the tool will now try to provide a human-readable name for the extended operation in addition to its OID.
  • We added client-side support for obtaining password policy state information from the Ping Identity Directory Server’s ds-pwp-state-json virtual attribute.
  • We added client-side support for the new populate composed attribute values and generate server profile administrative tasks in the Ping Identity Directory Server.
  • We added a new OID.parseNumericOID method that can be used to parse a provided string as a valid numeric object identifier, optionally performing strict validation. If the provided string does not represent a valid numeric OID, then the method will throw an exception with a message that explains the problem.
  • We improved the error messages generated for problems that may arise when parsing schema definitions.
  • We updated the schema parsing code so that it can now handle schema elements with a description value that is an empty string. Although empty descriptions (or other types of quoted strings) are not permitted in schema element definitions, some servers allow them. Empty descriptions are still not allowed by default, but that behavior can be overridden with a code change or a system property.
  • We added a new IA5 string argument value validator that can be used to require that the values of associated arguments are only permitted to contain ASCII characters. The manage-certificates tool has also been updated to provide better validation for certificate components that are required to be IA5 strings, including DNS names and email addresses in the subject alternative name extension.
  • We added support for encoding and decoding timestamps in the ISO 8601 format described in RFC 3339.
  • We updated the LDAP command-line tool framework so that if the --help-sasl argument is used in conjunction with a --saslOption argument that specifies the name of the SASL mechanism, the output will only include help information for that mechanism.
  • We fixed a bug in the StaticUtils.isASCIIString method that caused it to only look at the lowest byte for each character in the provided string.
  • We added new ByteStringBuffer utility methods, including getting individual bytes or sets of bytes at a specified position, for determining whether the buffer starts with or ends with a given set of bytes, and for reading the contents of a file or input stream into the buffer.
  • We added new StaticUtils convenience methods for reading and writing files as bytes, strings, or lists of lines.
  • We added support for new password policy state account usability warning and notice types for the Ping Identity Directory Server. The new types can be used to indicate that the account has too many outstanding authentication failures, but that the server will take some other action (for example, delaying the bind response) instead of completely preventing authentication.
  • We fixed an issue in the LDAP SDK’s JSON-formatted debug logging support for debug messages containing exceptions with another exception as the underlying cause.
  • We fixed an issue with the command-line tool framework that could prevent it from setting an argument value from a properties file even though that same value would have been permitted if it had been provided directly on the command line.
  • We updated the default standard schema provided with the LDAP SDK to include additional attribute syntaxes, matching rule, attribute type, and object class definitions.

UnboundID LDAP SDK for Java 5.0.1

The UnboundID LDAP SDK for Java is a fast, powerful, user-friendly, and completely free Java library for communicating with LDAP directory servers and performing other LDAP-related processing. We have just released version 5.0.1 of the LDAP SDK, and it is available for download from GitHub and SourceForge, as well as from the Maven Central Repository. The release notes are available online at https://docs.ldap.com/ldap-sdk/docs/release-notes.html.

This is a minor release that was primarily created in service of an upcoming release of the Ping Identity Directory Server, as it fixes an issue in a tool that only impacts that new release. Nevertheless, there are a couple of additional updates, so we’re making it publicly available.

The changes over the previous 5.0.0 release include:

  • We added a new LDAP connection logger API that can be used to keep a record of processing performed by the LDAP SDK, including successful and failed connection attempts, operation requests and responses (including non-final responses like search result entries, search result references, and intermediate responses), and disconnects. The LDAP SDK includes a connection logger instance that formats messages as JSON objects, but it’s an extensible API, so you’re free to create your own implementation using whatever format you want.
  • We have updated the LDAP command-line tool framework to make it possible to specify the address of the target directory server(s) using either –host or –address as an alternative to the existing –hostname argument.
  • We fixed an issue that prevented the collect-support-data tool from running properly in local mode when using a secure connection (either SSL or StartTLS). This functionality only applies to an upcoming release of the Ping Identity Directory Server, so existing installations should not have been affected, and new installations will have the fix.
  • We made minor updates to the usage output for several command-line tools to improve wording and fix typos. We also fixed typos in other messages used throughout the LDAP SDK.

Ping Identity Directory Server 8.0.0.1

The Ping Identity Directory Server version 8.0.0.1 has been released. It doesn’t look like the release notes on the website have been updated yet, but the changes over the previous 8.0.0.0 release include:

  • Fixed an issue that could cause the server to report an error when disabling an instance of the PingOne for Customers pass-through authentication plugin.
  • Improved the performance of password policy processing for operations targeting entries that reference password policies that are stored outside of the server configuration.
  • Updated file-based loggers, as well as the periodic stats logger and monitor history plugins, to add a logging-error-behavior property that can control the behavior the server exhibits if they encounter a write error. By default, the server will still write an error message to standard error, but the server can now be configured to enter lockdown mode to prevent normal clients from issuing requests that may not be properly logged.
  • Improved the performance of writing to temporary index files during LDIF import processing.
  • Fixed a potential memory leak that could occur when accessing the server via SCIM.
  • Fixed a potential off-heap memory leak that could occur when retrieving the version monitor entry.

UnboundID LDAP SDK for Java 5.0.0, now available under the Apache License

The UnboundID LDAP SDK for Java is a fast, powerful, user-friendly, and completely free Java library for communicating with LDAP directory servers and performing other LDAP-related processing. We have just released version 5.0.0 of the LDAP SDK, and it is available for download from GitHub and SourceForge, as well as from the Maven Central Repository. The release notes are available online at https://docs.ldap.com/ldap-sdk/docs/release-notes.html.

The most significant change in this new release is that the LDAP SDK is now available under the terms of the Apache License, Version 2.0, which is a very permissive OSI-approved open source license. Although it was already open source under the terms of the GNU GPLv2 and LGPLv2.1, the Apache License imposes fewer restrictions on how you can use the LDAP SDK. You are no longer required to offer to redistribute the source code (even if you want to use a modified version), and there’s no longer any concern about whether you need to keep the LDAP SDK jar file as a separate component. The Apache License is well respected and is often seen as more compatible and easier to use in non-open-source software than the GNU license, so we hope that this will make it easier to use in your applications, whether open source or proprietary. The LDAP SDK is still available for use under the terms of the GPLv2 and LGPLv2.1 (as well as the non-open-source UnboundID LDAP SDK Free Use License), but we recommend that new users consider using it under the Apache License.

Aside from adding the new license, we made several code changes in this release as well. They include:

  • The LDAP SDK offers an LDAPConnectionDetailsJSONSpecification class that allows you to define a JSON file with all of the settings needed to create and authenticate individual LDAP connections or connection pools. We’ve updated this class so that it’s now possible to indicate that when establishing a connection that is secured with SSL or StartTLS, the LDAP SDK should automatically trust any certificates signed by an authority in the JVM’s default set of trusted issuers. This was already the default behavior if you didn’t provide your own trust store (or choose to blindly trust all certificates, which isn’t recommended for production use), but it’s now possible to use this option in conjunction with a provided trust store so that it’s possible to trust a certificate either through that trust store or through the JVM’s default set of trusted issuers.
  • The KeyStoreKeyManager can be used to obtain a certificate from a key store file if one is needed during TLS negotiation. We have updated this class to provide an option to better validate that the key store can actually be used by this purpose with the settings that you provide. If you use this option and supply the alias of the certificate you wish to use, then the key manager will now verify that the alias exists in the key store, that it’s associated with a private key entry (as opposed to a trusted certificate entry, which only contains the public portion of a certificate and isn’t suitable for use if you need to present that certificate to the peer), and that all of the certificates in the chain are currently within their validity window. If you don’t specify a certificate alias, then the validation will make sure that the key store contains at least one private key entry in which all of the certificates in the chain are within their validity window.
  • The TrustStoreTrustManager can be used in the course of determining whether to trust a certificate presented by a peer during TLS negotiation. We have improved performance and concurrency for this trust manager by eliminating unnecessary synchronization that forced interaction with the trust store to be single-threaded.
  • We fixed an issue that could interfere with GSSAPI authentication if a JAAS login module configuration was loaded and cached by the JVM before the login attempt. In such cases, the cached configuration could be used instead of the one that was intended.
  • The LDAPDebuggerRequestHandler can be used to log detailed information about LDAP requests and responses that pass through an application using the LDAP SDK’s LDAPListener framework (including the in-memory directory server and the ldap-debugger command-line tool). We fixed an issue that could cause messages to be held up in an internal buffer rather than immediately written out as soon as they’re logged. In some cases, this could significantly delay the appearance of these messages or could prevent them from being written out at all if the amount of data to be logged was never enough to fill that internal buffer.
  • We added a new JSONAccessLogRequestHandler to the LDAPListener framework. This can log information about requests and responses as JSON objects, which are both human-readable and machine-parseable. While the existing AccessLogRequestHandler produces output that can be parsed programmatically to some extent, it is more optimized for human readability.
  • The LDAP SDK offers debugging logging support that can be helpful in diagnosing problems whose cause may not otherwise be readily apparent. Previously, the debug messages were logged in a form that was primarily intended to be human-readable rather than machine-parseable. They are now written in a JSON format that is both human-readable and machine-parseable.
  • The manage-certificates command-line tool provides a utility for interacting with certificate key and trust stores in the Java JKS format or the standard PKCS#12 format. When displaying detailed information about certificates in a key or trust store, the tool may not have been able to properly decode public key information for certificates with 384-bit elliptic curve public keys, and it also may not have been able to properly decode a subject alternative names extension that included one or more directoryName values. While it was still possible to display most of the information about the affected certificates, the updated version can now provide the full details about those elements.
  • The Ping Identity Directory Server includes a collect-support-data utility that can be used to gather a variety of information from a server installation that can be very useful for troubleshooting problems, tuning performance and scalability, and better understanding the environment in which the server is running. Previously, this utility could only be invoked by logging into the system on which the server instance is running and running the command-line tool. We have now added a couple of additional mechanisms for running the utility. It can now be invoked via an administrative task (either as an individual event that is requested by a remote client or as a recurring task that runs on a regular basis) that will create the resulting support data archive in a specified location on the system (which may be a shared filesystem for easier exfiltration). It can also be invoked via an extended operation that will run the tool and stream its output and the resulting support data archive back to the client in the form of intermediate response messages. Further, although the logic for actually collecting all of this support information remains in the server, we have added the collect-support-data command-line tool to the LDAP SDK so that it is easier to invoke the tool against a remote server without needing to install the server software on the client system.
  • The Ping Identity Directory Server provides a monitor backend that authorized clients can use to obtain a wealth of useful information about the state of the server, and the LDAP SDK includes support for retrieving and parsing the information in these monitor entries. We have updated the LDAP SDK’s support for the general monitor (that is, the top-level “cn=monitor” entry) to make it easier to obtain information about the cluster with which the server is associated, the location of the server instance, and a unique identifier that was generated for the server when the instance was initially configured.
  • The LDAP SDK offers a Version class that provides version information for the LDAP SDK, including the version number and information about the repository (e.g., the repository URL and revision ID) from which the LDAP SDK source code was obtained. This information was previously only offered as public static final constants, but referencing these constants from third-party applications could lead to unexpected behavior thanks to a “feature” of the Java compiler that will directly imbed the values of those constants (even if they come from a separate library) in the Java bytecode that it generates. This means that if your application references these LDAP SDK version constants and you compile it against one version of the LDAP SDK, then those version constants will be placed directly into the compiled bytecode. If you upgrade the LDAP SDK version that you use without recompiling your application (e.g., by just replacing the LDAP SDK jar file with a newer version), the code referencing the LDAP SDK version would still have the old values. To address this, we have updated the Version class to provide methods for obtaining the values of all the version constants. If you use these methods to obtain the values rather than referencing the constants directly, then you will always get the correct LDAP SDK version information even if you update the LDAP SDK without recompiling your application.

My Top 300 Films of the 2010s

Since the 2010s are coming to an end, everyone is posting their top movies of the decade. So I decided to give it a go.

According to my records, I saw 1,786 unique first-run movies in a theater between January 1, 2010 and December 31, 2019 (well, technically December 30, 2019, since I’m seeing the absolutely wonderful 1960 film The Apartment tonight rather than a new release). I went through that list and culled it down to my top 300 films. I’m not going to bother trying to pare it down any further, and I’m certainly not going to try to rank them so they’re presented alphabetically.

This list only includes films that I first saw in a theater, and only during their initial run (or at film festival screenings). It would be too much work for me to try to also include films that I first saw somewhere other than a theater, and it would also be too much work to include films that have been released since the beginning of 2010, but for which my first viewing was after its initial run. Also note that it includes films that IMDb lists with a release date before 2010, but for which it didn’t come to Austin, TX until after the beginning of 2010.

Anyway, here’s my list: