UnboundID LDAP SDK for Java 6.0.7

/ Neil Wilson

We have just released version 6.0.7 of the UnboundID LDAP SDK for Java. It is available for download from GitHub and SourceForge, and it is available in the Maven Central Repository. You can find the release notes at https://docs.ldap.com/ldap-sdk/docs/release-notes.html, but here’s a summary of the changes included in this version:

  • We fixed a bug in the SearchResultEntry.equals method that could prevent a SearchResultEntry from matching other types of Entry objects.
  • We fixed a bug in the Entry.applyModifications method that could cause it to fail with a NOT_ALLOWED_ON_RDN result if the provided entry was missing one or more of the attribute values used in its RDN.
  • We fixed a bug in the argument parser’s support for mutually dependent arguments with a set containing more than two arguments. Previously, the constraint would have been satisfied if at least two of the arguments were provided, rather than requiring all of them to be provided.
  • We added JSONObject methods for retrieving fields by name using case-insensitive matching (by default, JSON field names are treated in a case-sensitive manner). Because it is possible that a JSON object will have multiple fields with the same name when using case-insensitive matching, there are a few options for indicating how such conflicts should be handled, including only returning the first match, returning a map with all matching fields, or throwing an exception if there are multiple matches.
  • We updated the set of LDAP-related specifications to include the latest version of the draft-schmaus-kitten-sasl-ht proposal.

UnboundID LDAP SDK for Java 6.0.6

/ Neil Wilson

We have just released version 6.0.6 of the UnboundID LDAP SDK for Java. It is available for download from GitHub and SourceForge, and it is available in the Maven Central Repository. You can find the release notes at https://docs.ldap.com/ldap-sdk/docs/release-notes.html, but here’s a summary of the changes included in this version:

General Updates

  • We fixed an issue that could cause request failures when closing a connection operating in asynchronous mode with outstanding operations.
  • We fixed an issue that could interfere with the ability to get a default SSLContext on Java 17 when running in FIPS 140-2-compliant mode.
  • We updated LDAPConnectionOptions to add support for a new system property that can enable certificate hostname verification by default without any code changes.
  • We updated the LDAP command-line tool framework to add a new --verifyCertificateHostnames argument to enable hostname verification when performing TLS negotiation.
  • We improved the class-level Javadoc documentation for the SSLUtil class to provide a better overview of TLS protocol versions, TLS cipher suites, key managers, trust managers, and certificate hostname verification, and to provide better examples that illustrate best practices for establishing secure connections.
  • We fixed an issue in the JNDI compatibility support for controls, as well as extended requests and responses. Even though the implementation was based on the JNDI documentation, it appears that at least OpenJDK implementations do not abide by that documentation. The LDAP SDK is now compatible with the observed behavior rather than the documentation, although a system property can be used to revert to the former behavior.
  • We updated the SearchRequest class to add constructors that allow you to provide the search base DN with a DN object (as an alternative to existing constructors that allow you to specify it as a String).
  • We fixed an issue in the command-line tool framework in which an Error (for example, OutOfMemoryError) could cause the tool to report a NullPointerException rather than information about the underlying error.
  • We fixed an issue in the IA5 argument value validator that could allow it to accept argument values with non-ASCII characters.
  • We fixed an issue in the DNS hostname argument value validator that could prevent it from properly validating the last component of a fully qualified domain name, or the only component of an unqualified name.
  • We updated the identify-references-to-missing-entries tool to provide an option to generate an LDIF file with changes that can be used to remove identified references.
  • We updated the SelfSignedCertificateGenerator class to perform better validation for the subject alternative DNS names that it includes in a certificate.
  • We updated the manage-certificates generate-self-signed-certificate command to rename the --replace-existing-certificate argument to be --use-existing-key-pair. The former argument name still works, but it is hidden from the usage.
  • We included a native-image/resource-config.json file in the LDAP SDK jar file manifest, which can be used by the GraalVM native-image tool to ensure that appropriate resource files are included in the resulting image.

Updates Specific to Use With the Ping Identity Directory Server

  • * We updated the summarize-access-log tool to report on many more things, including the most common IP addresses for failed bind attempts, the most consecutive failed binds, information about work queue wait times, information about request and response controls, the number of components in search filters, and search filters that may indicate injection attempts.
  • We updated support for the audit data security administrative task to make it possible to specify the number and/or age of previous reports to retain.
  • We fixed issues that prevented specifying the criticality of the administrative operation and join request controls.

Ping Identity Directory Server 9.1.0.0

/ Neil Wilson

We have just released version 9.1.0.0 of the Ping Identity Directory Server and related products, including Directory Proxy Server, Synchronization Server, and Metrics Engine. See the release notes for a complete overview of changes, but here’s my summary:

Known Issues

  • When updating an 8.3 or 9.0 server to the 9.1 release, the Bouncy Castle library that the server uses for certain cryptographic processing will be updated. If it later becomes necessary to revert the update and return to the older version, the newer version of the Bouncy Castle libraries will remain in place. Although this will not cause any actual problems with the server, it may cause the server to log a message during startup about multiple versions of the library in the classpath.

Summary of New Features and Enhancements

  • Updated the Directory REST API to add support for a variety of request and response controls. [more information]
  • Added the ability to streamline the process for replacing a listener certificate if the new certificate is signed by the same issuer as the current certificate. [more information]
  • Updated the replace-certificate tool to make it possible to replace the server’s listener certificate even after it has expired. [more information]
  • Added support for sanitizing access log messages as they are logged. [more information]
  • Added support for generifying message strings in access and error log messages. [more information]
  • Updated the Synchronization Server to improve support for synchronizing changes to and from PingOne, including synchronizing using custom attributes, including multi-valued attributes and JSON-formatted attributes.
  • Improved the result that the server returns when using assured replication if it detects that the operation would ultimately be reverted as a result of a replication conflict.
  • Updated the sanitize-log tool so that its sanitization processing better aligns with the server’s new support for sanitized access logging. [more information]
  • Updated the summarize-access-log tool to support JSON-formatted access log files.
  • Added support for JSON-formatted controls in LDAP requests and responses. [more information]
  • Added a docker-pre-start-config command-line tool that can be used to speed up the startup process when running in a Docker container.
  • Updated manage-profile replace-profile to add a --skipValidation argument that can be used to skip the final validation process to reduce the time required for the update to complete.
  • Updated manage-profile generate-profile to add an --excludeSetupArguments argument that can generate a profile without the setup-arguments.txt file.
  • Improved the way that the Directory REST API processes PUT operations that alter an entry’s DN in conjunction with changes to other attributes in the entry.
  • Updated the active operation monitor provider to use millisecond precision in operation timestamps and to enable parsing the string representations of the operations using the LDAP SDK’s access log API.
  • Updated the status --fullVersion output to include the version of the collect-support-data tool.
  • Updated several libraries shipped with the server to improve functionality, address defects, and improve security.

Summary of Bug Fixes

  • Fixed an issue that could cause certain replication protocol messages to be dropped.
  • Fixed an issue that could cause a server to report missing changes and go into lockdown mode if it was restarted immediately after completing dsreplication initialize processing.
  • Fixed an issue that could prevent certain password policy functionality (including account status notification handlers) from being applied to an add operation in which an alternative policy should have been assigned using a virtual attribute rather than a real attribute.
  • Fixed an issue that could cause privileges assigned by virtual attribute to be overlooked in some cases (for example, when accessing topology-related functionality in the Admin Console).
  • Updated the server to create the esTokenizer.ping file if it does not exist but is needed. This file would not have been automatically created when upgrading a server with data encryption enabled from a pre-7.0 version to a later release with support for encrypted indexes.
  • Fixed an issue that could have incorrectly applied minimum and maximum password age constraints to users without a password.
  • Updated the JSON-formatted access logger to include the requester IP address field in disconnect, security negotiation, and client certificate log messages when appropriate.
  • Fixed an issue that prevented the server from refreshing monitor data used to detect and warn about upcoming certificate expiration. This could cause the server to continue to warn about an expiring certificate even after that certificate had been replaced. [more information]
  • Fixed issues that could prevent using the Amazon Secrets Manager, CyberArk Conjur, or HashiCorp Vault passphrase providers to obtain key and trust store PINs.
  • Fixed an issue that could cause the server to report a negative processing time in the access log for certain types of operations.
  • Updated the server to prevent inappropriate updates to the ds-pwp-modifiable-state-json operational attribute when the Modifiable Password Policy State plugin is not enabled.
  • Updated the server to prevent a user from updating their own password policy state using the ds-pwp-modifiable-state-json operational attribute.
  • Updated the server to prevent using the ds-pwp-modifiable-state-json attribute to alter a user’s password policy state in the same operation that also reset that user’s password.
  • Fixed an issue in which the dsreplication tool failed to properly normalize base DN values.
  • Fixed an issue that could prevent the Directory REST API from retrieving entries containing an attribute using the generalized time syntax whose value did not match an expected format.
  • Fixed an issue that could cause manage-profile replace-profile to fail with an error about merging configuration.
  • Updated manage-profile setup and manage-profile replace-profile to ensure that a pre-populated encryption settings database can only be provided in the post-setup files rather than in the pre-setup files, which fixes issues when using a customized cipher stream provider.
  • Updated the manage-topology add-server command to be more consistent when adding additional Synchronization Servers into a failover topology.
  • Fixed an issue in which the server could ignore certain indexes that it incorrectly believed to be redundant when evaluating search criteria.
  • Fixed an issue in which a SCIM request could return a less-appropriate error code in cases where an update violated a unique attribute constraint.
  • Fixed an issue that could cause the server to incorrectly reject a request containing a non-critical control that the requester was not allowed to use. The server will now process the operation as if that control had not been requested.
  • Fixed an issue that could allow the password policy state extended operation to create duplicate authentication failure time or grace login use time values in a user’s entry.
  • Fixed an issue that could adversely affect backward compatibility when attempting to use the legacy --useSSL or --useStartTLS arguments with the migrate-ldap-schema tool (as opposed to the newer arguments allowing you to independently specify security options for the source and target servers).
  • Fixed an issue that could prevent the server from generating an administrative alert to indicate that an outstanding alarm condition had been resolved.
  • Fixed an issue that could cause the server to report an internal error when attempting to obtain database statistics for a read-only backend.
  • Fixed an issue in the export-reversible-passwords tool that could cause it to report a timeout error when waiting for a response from the server.
  • Updated the export-reversible-passwords tool to automatically cancel an in-progress export if the tool used to invoke the export is terminated.
  • Fixed an issue that prevented the encode-password tool from working properly if the AES256 password storage scheme is enabled.
  • Updated the server to disable the index cursor entry limit by default, which is very unlikely to be needed and may interfere with the ability to effectively process certain requests.

Controls in the Directory REST API

It is now possible to use certain LDAP request and response controls through the Directory REST API. Controls should be provided as an array of JSON objects, and the following fields may be used in the JSON representation of the control:

  • oid — A mandatory string field that holds the object identifier for the control.
  • control-name — An optional string field that holds a user-friendly name for the control. Note that this is only intended for informational purposes and won’t be used in the course of actually decoding the control.
  • criticality — A mandatory Boolean field that indicates whether the control should be considered critical.
  • value-base64 — An optional string field whose values is the base64-encoded representation of the raw LDAP encoding for the value. This may be used for any type of control, including controls that the server supports but doesn’t offer a JSON-specific value encoding.
  • value-json — An optional JSON object field whose value contains zero or more fields that are specific to the type of control being represented.

At most one of the value-base64 and value-json fields may be present, and both may be omitted if the control doesn’t take a value.

For example, the following is a JSON representation of the Ping-proprietary join request control:

{ "oid":"1.3.6.1.4.1.30221.2.5.9",
  "control-name":"Join Request Control",
  "criticality":false,
  "value-json":{ "join-rule":{ "type":"dn",
                               "attribute":"manager" },
                 "base-dn-type":"use-custom-base-dn",
                 "base-dn-value":"ou=People,dc=example,dc=com",
                 "scope":"wholeSubtree",
                 "size-limit":10,
                 "attributes":[ "givenName",
                                "sn",
                                "cn",
                                "mail" ],
                 "require-match":false } }

For controls that take a value, the specific JSON encoding for that value varies from one control to another, and you should consult the documentation for a given control to understand how to properly encode and parse it.

Controls that may now be used in the Directory REST API include:

Improvements in Certificate Management

We have updated the replace-certificate tool and the corresponding support in the topology registry to add new options to make things easier when periodically replacing certificates, and especially listener certificates used for security communication with clients.

First, we have added a few new subcommands to the tool, including:

  • list-topology-registry-listener-certificates — Displays a list of the listener certificates associated with a specified instance in the topology registry.
  • list-topology-registry-inter-server-certificates — Displays a list of the inter-server certificates associated with a specific instance in the topology registry.
  • add-topology-registry-listener-certificate — Adds a new certificate to the set of listener certificates for a specified instance in the topology registry, but does not alter any certificate trust stores.

Next, we have updated the server’s use of topology registry listener certificates so that a server’s listener certificate may be trusted for inter-server communication if either the listener certificate itself or one of its issuers is found in the topology registry. Previously, the listener certificate itself was required to be present in the topology registry for that certificate to be trusted, but we have now added support for trusting issuer certificates. This makes it easier to an existing certificate with a new one signed by the same issuer because you can just update the key store without the need to update the topology registry, and there is less chance that a temporary communication problem between instances could arise if a server starts using a new listener certificate before the topology registry updates have propagated to all other instances in the topology.

If you decide to use this issuer-based trust, then there are two ways that you can get the issuer certificate into the topology registry:

  • Using the new replace-certificate add-topology-registry-listener-certificate command referenced above.
  • Using the replace-certificate replace-listener-certificate command with the new --topology-registry-update-type and --trust-store-update-type arguments, which make it possible to configure which certificate(s) in the trust chain will be added to the topology registry and the local trust store, respectively.

We also addressed an oversight in the replace-certificate replace-listener-certificate command that prevented it from being used to replace a listener certificate that had already expired, as the replace certificate tool would no longer trust that expired certificate and would therefore not establish a connection to the server to make the necessary updates. While we strongly recommend always replacing certificates before they actually expire, you can now use the --ignore-current-listener-certificate-validity-window argument to indicate that processing should proceed even if the presented listener certificate is outside its validity time frame. The listener certificate or one of its issuers must still be found in the topology registry.

Finally, we fixed an issue that prevented certificate information presented in the server’s monitor data from being updated. Since there is a gauge in place that uses this monitor information to warn about an upcoming certificate expiration, this bug prevented it from detecting when the certificate had been replaced, and the server would continue to warn about the upcoming expiration even after that certificate was no longer in use. That issue has been fixed, and certificate rotation should now be automatically detected.

Sanitizing Access Log Messages

The server’s access and error logs provide an invaluable resource for understanding how the server is used and for troubleshooting problems. However, they may also sometimes contain sensitive or personally identifiable information that you don’t want to have lying around on the filesystem. In the past, we have offered a sanitize-log tool that can be used to examine log files and produce a scrubbed copy that redacts or obscures potentially sensitive information, and that tool is still available (and improved). However, we have also added the ability to sanitize log messages as they are being written to help ensure that this information isn’t recorded in the logs in the first place.

For access logging, the new sanitization processing is based largely around log field syntaxes. Each log field is now associated with one of the following syntaxes:

  • String
  • Boolean
  • DN
  • Filter
  • Integer
  • Floating-point number
  • Generalized time
  • RFC 3339 timestamp
  • JSON object
  • Comma-delimited string list

We also offer a variety of sanitization types that may be used when determining how to handle a given log field, including:

  • preserve — The field will be included as-is, without any alteration or obfuscation.
  • omit — The field will be omitted from the log message entirely, without either the field name or its value present.
  • redact-entire-value — The field name will be included in the log message, but the entire value will be redacted such that all values for a given syntax will be the same. We actually try to preserve the original syntax when doing this when possible, using the following behavior:

    • String, Boolean, and string list fields will have their values replaced with the text “{REDACTED}”.
    • DN fields will have their values replaced with the text “redacted={REDACTED}”.
    • Filter fields will have their values replaced with the text “(redacted={REDACTED})”.
    • Integer fields will have their values replaced with the text “-999999999999999999”.
    • Floating-point fields will have their values replaced with the text “-999999.999999”.
    • Generalized time fields will have their values replaced with the text “99990101000000.000Z”.
    • RFC 3339 timestamp fields will have their values replaced with the text “9999-01-01T00:00:00.000Z”.
    • JSON object fields will have their values replaced with the text “{ "redacted":"{REDACTED}" }”.
  • redact-value-components — This behavior is similar to redact-entire-value, but it applies specifically to DN, filter, and JSON object values. In such cases, the redaction will be only applied to components within the value, rather than to the entire value itself. For DNs and filters, this means that only attribute values will be redacted while the rest of the text (including attribute names and punctuation) will be preserved, and for JSON objects it means that only field values will be redacted (and field names and punctuation will be preserved). For example, a DN of “uid=jdoe,ou=People,dc=example,dc=com” would become “uid={REDACTED},ou={REDACTED},dc={REDACTED},dc={REDACTED}”. You can optionally even configure this on a per-attribute or per-JSON-field basis, so that only certain attributes or fields (or all but a specified set of attributes or fields) will have their values redacted, and other attributes or fields will have their values preserved.
  • tokenize-entire-value — The field name will be included in the log message, but the value will be replaced with a token that is generated from the value but cannot be directly reversed back to its original value. This is similar to redaction, but different values will result in different tokens, and the same value will consistently result in the same token. This means that it’s possible to identify cases in which the same value appears across multiple log messages, even if you can’t tell what that value actually is. As with redaction, we attempt to preserve the original field syntax when possible, as follows:

    • String, Boolean, and string list fields will have their values tokenized with the text “{TOKENIZED:tokenValue}”, where tokenValue will be generated from the original value.
    • DN fields will have their values replaced with “tokenized={TOKENIZED:tokenValue}”.
    • Filter fields will have their values replaced with “(tokenized={TOKENIZED:tokenValue})”.
    • Integer fields will have their values replaced with a number that starts with “-999999999” and is followed by a sequence of nine additional digits generated from the original value.
    • Floating-point fields will have their values replaced with a number that starts with “-999999.” and is followed by a sequence of six additional digits generated from the original value.
    • Generalized time and RFC 3339 timestamp values will have their values replaced with a timestamp that uses the year 8888, but with other components of the timestamp generated from the original timestamp.
    • JSON object fields will have their values replaced with the text “{ "tokenized":"{TOKENIZED:tokenValue}" }”.
  • tokenize-value-components — This behavior is similar to redact-value-components, but the individual components will be tokenized rather than redacted. As with redact-value-components, this primarily applies to DNs, filters, and JSON objects.

The server now provides a log field syntax configuration object for each of the aforementioned syntaxes, and that object will include a default-behavior field that allows you to indicate that all values with that syntax should be treated in a given way. For example, if you configure the DN log field syntax with a default-behavior value of tokenize-value-components, then all DNs written to a text-formatted or JSON-formatted access log will automatically have their attribute values tokenized while the rest of the DN will remain intact.

For even more control over how individual log fields are treated, you can use log field behavior configuration objects. These objects allow you to choose the behavior to use on a field-by-field basis, and you can configure different behaviors for different loggers if desired.

If you’d rather still have access log messages written with all of the information intact, but still have the ability to sanitize those logs after the fact, then you can continue to use the sanitize-log tool to accomplish this, and we have updated this tool to provide improved behavior that is more closely aligned with the server’s support for sanitized logging. This includes:

  • It now knows about all predefined log fields that the server may use, and we have preselected default behaviors for each of those fields in a manner that we feel provides the best balance between privacy and usefulness.
  • It now uses the same syntax-aware redaction and tokenization logic that the server uses so that field values are more likely to conform to their original syntax.
  • You can customize the behavior that the tool exhibits on a per-syntax basis, or you can point it at a log field behavior configuration object.

Generified Log Message Strings

Sanitized logging and the sanitize-log tool provide very powerful mechanisms for removing or obscuring sensitive or identifiable information in log messages, but there are some access log fields that may have such a wide range of values that it’s not necessarily feasible to treat them in a “one size fits all” kind of way, especially while retaining the utility of that field. These kinds of fields include:

  • The diagnostic message that is returned to the client in an LDAP response.
  • An additional info message that is meant to be recorded in the access log for an operation but not returned to the client.
  • An authentication failure reason that is meant to explain why an authentication attempt failed.
  • A disconnect reason that is meant to provide the reason that a client connection was closed.

It’s possible that the values of these fields could occasionally include sensitive or identifiable information, but their content can also be very helpful in troubleshooting problems or understanding the server’s behavior in specific situations. As such, you would lose substantial benefit by configuring those fields to be redacted.

Further, while a field-based approach works well for sanitizing access log messages for the most part, it doesn’t really make any sense for error log messages, as the core content of an error log message appears in the same field for all messages.

To address these concerns, we have introduced the ability to log generified values of these strings, using the generify-message-strings-when-possible property in the configuration for each logger. If this property is set to true, then the server will log the generic format string for the associated message rather than the processed message that it would normally include.

For example, if a simple bind attempt fails because the target user doesn’t exist, the access log message for that operation might include an authentication failure reason like:

Unable to bind to the Directory Server as user uid=jdoe,ou=People,dc=example,dc=com because no such user exists in the server.

However, if the logger is configured to report the generic version of the message, then it would instead log:

Unable to bind to the Directory Server as user %s because no such user exists in the server.

The generic version of the message doesn’t specifically identify the user, thereby offering better privacy than the un-generified version, but at least it’s still useful enough to understand the reason that the authentication attempt failed.

JSON-Formatted LDAP Request and Response Controls

The Ping Identity Directory Server provides support for a very wide range of request and response controls. This includes not only standard controls defined in RFCs and Internet Drafts, but also many controls that we’ve defined ourselves for enhanced functionality. These controls can be easily accessed through the UnboundID LDAP SDK for Java, but they aren’t as readily available for use in applications written with other LDAP APIs, especially in languages that don’t run on the JVM. Even though the LDAP SDK documentation describes the format of the encoded values for these controls, generating or decoding those values may still be a substantial challenge, typically requiring a library for working with ASN.1 BER and knowing how to use it properly.

To help address that, we’ve introduced the ability to encode LDAP request controls as JSON objects, and to indicate that LDAP response controls should also be encoded as JSON objects. JSON is much easier to work with than ASN.1 BER across a variety of programming languages, and it doesn’t require as much expertise to understand how to properly generate or parse JSON objects.

The primary interface to this functionality is through the JSON-formatted request control, which has an OID of “1.3.6.1.4.1.30221.2.5.64” and a value that is the string representation of a JSON object that encapsulates the JSON representations of the request controls to send to the server (or you can omit the value if you don’t have any request controls to send but want all response controls to be returned using a JSON encoding). If the request includes a JSON-formatted request control, and if there are any response controls to be returned, then the server will return them in a JSON-formatted response control, which has an OID of “1.3.6.1.4.1.30221.2.5.65” and a value that uses the same format as the request control. Note that while you don’t need to use the UnboundID LDAP SDK for Java to generate or parse these controls, its Javadoc documentation (especially the documentation for the toJSONControl method for the controls in question) may be useful in understanding what fields may be used in these objects.

The value of a JSON-formatted request or response control is simply a JSON object that has a single controls field whose value is an array of the JSON objects that make up the individual request or response controls. For example, the following JSON object represents the encoded value for a JSON-formatted request control that embeds both a retain identity request control and a get authorization entry request control:

{ "controls":[
  { "oid":"1.3.6.1.4.1.30221.2.5.3",
    "control-name":"Retain Identity Request Control",
    "criticality":true },
  { "oid":"1.3.6.1.4.1.30221.2.5.6",
    "control-name":"Get Authorization Entry Request Control",
    "criticality":false,
    "value-json":{
      "include-authentication-entry":true,
      "include-authorization-entry":true,
      "attributes":[ "uid", "givenName", "sn", "cn", "mail" ] } } ] }

For controls that are supported in both LDAP and the Directory REST API, the JSON encoding that we use is the same in either case. However, the Directory Server supports a number of additional controls for LDAP requests and responses that it doesn’t currently support through the Directory REST API. In additional to all of the controls we currently support in the Directory REST API, we provide JSON encodings for each of the following additional controls in LDAP requests:

  • Account usable request and response controls
  • Administrative operation request control
  • Authorization identity request and response controls
  • Extended schema info request control
  • Get authorization entry request and response controls
  • Get backend set ID request and response controls
  • Get password policy state issues request and response controls
  • Get recent login history request and response controls
  • Get server ID request and response controls
  • Get user resource limits request and response controls
  • Hard delete request control
  • Override search limits request control
  • Password policy request and response controls
  • Replication repair request control
  • Retain identity request control
  • Return conflict entries request control
  • Route to backend set request control
  • Route to server request control
  • Server-side sort request and response controls
  • Simple paged results control
  • Soft delete request control
  • Soft-deleted entry access request control
  • Subentries request control
  • Subtree delete request control
  • Suppress operational attribute update request control
  • Undelete request control
  • Virtual list view request and response controls

UnboundID LDAP SDK for Java 6.0.5

/ Neil Wilson

We have just released version 6.0.5 of the UnboundID LDAP SDK for Java. It is available for download from GitHub and SourceForge, and it is available in the Maven Central Repository. You can find the release notes at https://docs.ldap.com/ldap-sdk/docs/release-notes.html, but here’s a summary of the changes included in this version:

General Updates:

  • We fixed an issue that could occasionally cause the LDAP SDK to hide the actual cause of a StartTLS failure by using information from a second, less useful exception.
  • We fixed an issue that could cause the ldifsearch tool to display a malformed message when the first unnamed trailing argument was expected to be a search filter but could not be parsed as a valid filter.
  • We improved support for validating and comparing values using the telephone number syntax. Previously, we used a loose interpretation of the specification, which would consider any printable string (including strings without any digits) to be valid, and would only ignore spaces and hyphens when comparing values. You can now configure varying levels of strictness (either programmatically or using system properties), including requiring at least one digit or strict conformance to the X.520 specification. You can also configure it to ignore all non-digit characters when comparing values, and this is now the default behavior.
  • We fixed a bug in which the ldapcompare tool did not properly close its output file if one was configured. The output file does get automatically closed when the tool exits so it’s not an issue when running ldapcompare from the command line, but this can cause problems if the tool is invoked programmatically from another application.
  • We fixed an issue with the tool properties file created using the --generatePropertiesFile argument in command-line tools that support it. The generated properties file did not properly escape backslash, carriage return, line feed, or form feed characters.

Updates Specific to Use With the Ping Identity Directory Server:

  • We added support for encoding controls to JSON objects, and for decoding JSON objects as controls. There is a generic JSON representation that will work for any type of control (in which the value is provided as the base64-encoded representation of the raw value used in the LDAP representation of the control), but most controls provided as part of the LDAP SDK also support a more user-friendly representation in which the components of the value are represented in a nested JSON object.
  • We added client-side support for a new JSON-formatted request control that can be used to send request controls to a Ping Identity Directory Server with the controls encoded as JSON objects rather than a raw LDAP representation. We also added support for a JSON-formatted response control that can be used to receive JSON-encoded response controls from the server.
  • We updated the ldapsearch and ldapmodify command-line tools to add a --useJSONFormattedRequestControls argument that will cause any request controls to be sent using a JSON-formatted request control, and it will cause any response controls returned by the server to be embedded in a JSON-formatted response control.
  • We fixed an issue with the way that the parallel-update tool created assured replication request controls when an explicit local or remote assurance level was specified. Previously, it would only specify a minimum assurance level without specifying a maximum level, which could cause the server to use a higher assurance level than requested by the client.
  • We updated the topology registry trust manager to allow trusting a certificate chain if either the peer certificate or any of its issuers is found in the server’s topology registry. Previously, it would only trust a certificate chain if the peer certificate itself was found in the topology registry, and having an issuer certificate was not sufficient. The former behavior is still available with a configuration option.
  • We updated the topology registry trust manager to make it possible to ignore the certificate validity window for peer and issuer certificates. The validity window is still respected by default, but if the trust manager is configured to ignore it, then a certificate chain may be trusted even if the peer or an issuer certificate is expired or not yet valid.

UnboundID LDAP SDK for Java 6.0.4

/ Neil Wilson

We have just released version 6.0.4 of the UnboundID LDAP SDK for Java. It is available for download from GitHub and SourceForge, and it is available in the Maven Central Repository. You can find the release notes at https://docs.ldap.com/ldap-sdk/docs/release-notes.html, but here’s a summary of the changes included in this version:

General Updates:

  • We fixed an issue with the Filter.matchesEntry method that could cause it to throw an exception rather than returning an appropriate Boolean result when evaluating an AND or an OR filter in which one of the nested elements used inappropriate matching (for example, if the assertion value did not conform to the syntax for the associated attribute type).
  • We fixed an issue with the way that decodeable controls are registered with the LDAP SDK. Under some circumstances, a thread could become blocked while attempting to create a new control.
  • We updated the JVM-default trust manager to properly check for the existence of a “jssecacerts” trust store file in accordance with the JSSE specification. It had previously only looked for a file named “cacerts”.
  • We updated the logic used to select the default set of supported cipher suites so that it will no longer exclude suites with names starting with “SSL_” by default on JVMs with a vendor string that includes “IBM”. IBM JVMs appear to use the “SSL_” prefix for some or all cipher suites, including those that are not associated with TLS protocols rather than a legacy SSL protocol. We also added a TLSCipherSuiteSelector.setAllowSSLPrefixedSuites method that can be used to override the default behavior.
  • We updated the LDIF reader to support reading modifications with attribute values read from a file referenced by URL. This was previously supported when reading LDIF entries or add change records, but it had been overlooked for LDIF modify change records.
  • We updated the LDIF reader so that it no longer generates comments attempting to clarify the contents of base64-encoded values if the value is longer than 1,000 bytes.
  • We updated the documentation to include the latest versions of the draft-behera-ldap-password-policy, draft-coretta-x660-ldap, and draft-ietf-kitten-scram-2fa specifications.

Updates Specific to Use With the Ping Identity Directory Server:

  • We added a new API for parsing access log messages generated by the server. The new API supports both text-formatted and JSON-formatted log messages, whereas the previous version only supported messages in the default text (“name=value”) format.
  • We updated the summarize-access-log tool (which can be used to perform basic analysis of server access log files) to add support for JSON-formatted log files.
  • We added support for retrieving and parsing X.509 certificate monitor entries.
  • We added client-side support for an administrative task that can cause the server to immediately refresh any cached certificate monitor data. The server will automatically refresh the cache every minute, but the task can be used to cause an immediate refresh.

Ping Identity Directory Server 9.0.0.0

/ Neil Wilson

We have just released version 9.0.0.0 of the Ping Identity Directory Server. The release notes provide a pretty comprehensive overview of what’s included, but here’s my summary.

Ping Identity Directory Server Products Do Not Use log4j

Recently, a very serious security issue (CVE-2021-44228) was identified in the Apache log4j library, which is used by many Java applications to provide logging support. None of the Ping Identity Directory Server, Directory Proxy Server, Synchronization Server, and Metrics Engine products make use of this library in any way, and it is not included as part of the server. Some of the libraries that we include with the server do have support for logging to log4j, but that functionality is not used, and the log4j library is not included as part of the server.

The standalone version of the Admin Console does include the log4j library, but it is included only as a transitive dependency of one of the other libraries used by the console, and the log4j library is not used in any way by the Admin Console. Because this vulnerability was disclosed very late in the release cycle for the Ping Identity server products, we have chosen to update to a non-vulnerable version of the library rather than remove it entirely, as that requires less testing. Again, even though the log4j library is included with the standalone Admin Console, it is not used in any way, so even if you are using an older version of the console with an older version of the log4j library, you are not vulnerable to the security issue.

The UnboundID LDAP SDK for Java does not include any third-party dependencies at all (other than a Java SE runtime environment at Java version 7 or later). It does not include or interact with log4j in any way.

Changes Affecting All Server Products

  • Added cipher stream providers for PKCS #11 tokens, Azure Key Vault, and CyberArk Conjur. [more information]
  • Added passphrase providers for Azure Key Vault and CyberArk Conjur. [more information]
  • Added password storage schemes for authenticating with passwords stored in external services, including AWS Secrets Manager, Azure Key Vault, CyberArk Conjur, and HashiCorp Vault. [more information]
  • Added extended operations for managing server certificates. [more information]
  • Added the ability to redact the values of sensitive configuration properties when constructing the dsconfig representation for a configuration change. [more information]
  • Included the original requester DN and client IP address in log messages for mirrored configuration changes. [more information]
  • Added TLS configuration properties for outbound connections. [more information]
  • Updated the Admin Console to support using PKCS #12 and BCFKS trust stores.
  • Updated the file servlet to support authenticating with OAuth 2.0 access tokens and OpenID Connect ID tokens, which makes it possible to download collect-support-data archives and server profiles generated through the Admin Console when authenticated with SSO.
  • Fixed an issue that could cause degraded performance and higher CPU utilization for some clients using TLSv1.3.
  • Fixed an issue that prevented the manage-profile replace-profile tool from working properly for servers running in FIPS 140-2-compliant mode.
  • Updated export-ldif to always base64-encode attribute values containing any ASCII control characters. Previously, only the null, line feed, and carriage return control characters caused values to be base64-encoded.
  • Fixed an issue in which some tools that operate on the server’s configuration did not use the correct matching rule for attribute types configured to use case-sensitive matching. If a config entry had an attribute with multiple values differing only in capitalization, all but one of the values could be lost.
  • Updated the Directory REST API to add support for attribute options.
  • Added the ability to recognize JVM builds from Eclipse Foundation, Eclipse Adoptium, and BellSoft.
  • Removed “-XX:RefDiscoveryPolicy=1” from the default set of options used to launch the JVM. In some cases, this option has been responsible for JVM crashes.

Changes Affecting the Directory Server

  • Added support for pluggable pass-through authentication. [more information]
  • Fixed an issue that could prevent authenticating with certain types of reversibly encrypted passwords that were encrypted on an instance that was subsequently removed from the topology. [more information]
  • Fixed an issue that prevented decoding the value of a proxied authorization v2 request control when the authorization identity had a specific length.
  • Fixed an issue that could cause sporadic failures when attempting to back up a backend with data encryption enabled. In such cases, the backup would likely succeed if re-attempted.
  • Added a replica-partial-backlog attribute to the replication summary monitor entry to provide information about how each replica contributes to the overall replication backlog.
  • Fixed an issue in which the server could use incorrect resource limit values (including size limit, time limit, lookthrough limit, and idle time limit) for users with custom limits who authenticated via pass-through authentication.
  • Fixed an issue in which the server did not properly update certain password policy state information for simple bind attempts targeting users without a password.
  • Fixed an issue in which the server may not handle other controls properly when processing an operation that includes the join request control. The server may have overlooked a control immediately following the join request control in the operation request, and it may have omitted appropriate non-join result controls from the response.
  • Fixed an issue in which a newly initialized server could go into lockdown mode with a warning about missing changes if it was restarted immediately after initialization completed.
  • Fixed an issue that could prevent changes applied to non-RDN attributes in the course of processing a modify DN operation from being replicated.
  • Fixed an issue that could prevent composed attribute values from being properly updated for operations that are part of a muti-update extended operation.
  • Improved performance for modify operations that need to update a composite index to add an entry ID to the middle of a very large ID set.
  • Added limits for the maximum number of attributes in an add request and the maximum number of modifications in a modify request. [more information]
  • Updated the dsreplication initialize-all command to support initializing multiple replicas in parallel.
  • Updated remove-defunct-server to add a --performLocalCleanup option that can be used to remove replication metadata from a server that is offline.
  • Added an option to the mirror virtual attribute provider to make it possible to bypass access control evaluation for the internal searches that it performs to retrieve data from other entries.
  • Fixed an issue in which an entry added with a createTimestamp attribute could lose the original formatting for that attribute when replicated to other servers.
  • Fixed an issue that could lead to long startup times in large topologies with data encryption enabled.
  • Updated the ldap-diff tool to add several new features. [more information]
  • Updated the migrate-ldap-schema tool to add several new features. [more information]

Changes Affecting the Directory Proxy Server

  • Fixed an issue that could cause certain internal operations initiated in the Directory Proxy Server to fail when forwarded to a backend Directory Server whose default password policy was configured in a way that interfered with the account used to authorize internal operations.
  • Improved the logic used to select the best error result to return to the client for operations broadcast to all backend sets. Previously, the server could have incorrectly returned a result indicating that the target entry did not exist when the operation failed for some other reason.
  • Updated the entry counter, hash DN, and round-robin placement algorithms to support excluding specific backend sets.

Changes Affecting the Synchronization Server

  • Added the ability to synchronize certain password policy state information from Active Directory to the Ping Identity Directory Server, including account disabled state and the password changed time.
  • Fixed an issue that could prevent synchronizing changes to entries that have multiple attributes with the same base attribute type but different sets of attribute options, particularly if any of the attributes have more values than the replace-all-attr-values limit defined in the associated Sync Class.
  • Added the ability to apply rate limiting when synchronizing changes to PingOne.
  • Fixed an issue in which the max-rate-per-second property was not properly applied when running the resync tool.

Changes Affecting the Metrics Engine

  • Fixed an issue that could prevent dashboard icons from being properly displayed.

New Cipher Stream Providers

The encryption settings database holds a set of definitions that include the keys used for data encryption. The encryption settings database is itself encrypted, and we use a component called a cipher stream provider for reading and writing that encrypted content. We already offered several cipher stream provider implementations, including:

  • Generate the encryption key with a passphrase read from a file.
  • Generate the encryption key with a passphrase provided interactively during server startup.
  • Protect the encryption key with AWS Key Management Service (KMS).
  • Generate the encryption key with a passphrase retrieved from AWS Secrets Manager.
  • Generate the encryption key with a passphrase retrieved from a HashiCorp Vault instance.
  • Use the Server SDK to develop your own custom cipher stream providers.

In the 9.0.0.0 release, we are introducing support for three new types of cipher stream providers:

  • Wrap the encryption key with a certificate read from a PKCS #11 token, like a Hardware Security Module (HSM). Note that because of the limitations in Java’s support for key wrapping, only certificates with RSA key pairs can be used for this purpose.
  • Generate the encryption key with a passphrase retrieved from Azure Key Vault.
  • Generate the encryption key with a passphrase retrieved from a CyberArk Conjur instance.

New Passphrase Providers

Passphrase providers offer a means of obtaining clear-text secrets that the server may need for things like accessing protected content in a certificate key store or authenticating to an external service. We already offered several passphrase provider implementations, including:

  • Read the secret from a file, which may optionally be encrypted with a key from the server’s encryption settings database.
  • Read the secret from an obscured value stored in the server’s configuration.
  • Read the secret from an environment variable.
  • Read the secret from AWS Secrets Manager.
  • Read the secret from a HashiCorp Vault instance.
  • Use the Server SDK to develop your own custom passphrase providers.

In the 9.0.0.0 release, we are introducing support for two new types of passphrase providers:

  • Read the secret from Azure Key Vault.
  • Read the secret from a CyberArk Conjur instance.

Password Storage Schemes for External Services

Password storage schemes are used to protect passwords held in the server. We already offered a variety of password storage schemes, including:

  • Schemes using salted 256-bit, 384-bit, and 512-bit SHA-2 digests. SHA-1 support is also available for legacy purposes, but is not recommended.
  • Schemes using more resource-intensive, brute-force-resistant algorithms like PBKDF2, bcrypt, scrypt, and Argon2.
  • A scheme that reversibly encrypts passwords with a 256-bit AES key obtained from the encryption settings database.
  • Schemes that reversibly encrypt passwords with legacy keys stored in the topology registry.

In the 9.0.0.0 release, we are introducing support for new password storage schemes that allow users to authenticate with passwords stored in external secret stores, including:

  • AWS Secrets Manager
  • Azure Key Vault
  • CyberArk Conjur
  • HashiCorp Vault

In these cases, the storage scheme is configured with the information needed to connect and authenticate to the external service, and the encoded representation of the password contains a JSON object with the information needed to identify the specific secret in that service to use as the password for the associated user.

These password storage schemes can be used to authenticate with both LDAP simple authentication and SASL mechanisms that use a password. However, these schemes are read-only: users can authenticate with a password stored in the associated external service, but password changes need to be made through that service rather than over LDAP.

Extended Operations for Certificate Management

We have added support for a set of extended operations that can be used to remotely manage certificates in server instances, including replacing listener and inter-server certificates and purging information about retired certificates from the topology registry. These operations are especially useful for managing certificates in instances running in Docker or in other cases where command-line access may not be readily available to run the replace-certificate tool.

When replacing certificates, the new key store can be obtained in several ways:

  • It can be read from a file that is already available to the server (for example, one that has been copied to the server or placed on a shared filesystem).
  • The raw bytes that make up the new key store file can be included directly in the extended request.
  • The individual certificates and private key can be provided in the extended request, in either PEM or DER form.

Many safeguards are in place to prevent these extended operations from being inappropriately used. These include:

  • The extended operation handler providing support for these operations is not enabled by default. It must be enabled before they can be used.
  • The extended operations will only be allowed over secure connections.
  • The extended operations can only be requested by a user with the permit-replace-certificate-request privilege. No users have this privilege by default (not even root users or topology administrators).
  • You can indicate which of the individual types of operations are allowed, and you can define connection and request criteria to further restrict the circumstances under which they may be used.
  • By default, it will only allow reading certificates from a file on the server filesystem. You have to specifically enable the option to allow providing the new certificate information from a remote client.
  • The server will generate administrative alerts for all successful and failed attempts to process these operations.

These extended operations can be invoked programmatically (support for them is included in the UnboundID LDAP SDK for Java). They can also be used through new subcommands in the replace-certificate command-line tool.

Redacting Sensitive Values in Configuration Changes

We have added a new redact-sensitive-values-in-config-logs global configuration property that can be used to indicate that the server should redact the values of sensitive configuration properties when generating the dsconfig representation for that configuration change, including the representation that is written to the config-audit.log file and included in alerts to notify administrators of the change.

By default, the values of sensitive configuration properties are obscured in a way that allows the server to obtain the clear-text value, but that is not readily apparent to an observer. This helps protect the values of these secrets while still allowing the config-audit.log file to be replayed. However, a determined user with access to this obfuscated representation may be able to determine the clear-text value that it represents.

If the redact-sensitive-values-in-config-logs property is set to true, then the values of sensitive configuration properties will be redacted rather than obscured. This prevents someone with access to the dsconfig representation of the change from being able to obtain the clear-text value of the secret, but it does mean that the config-audit.log file may no longer be replayable.

Original Requester Details for Mirrored Configuration Changes

When making configuration changes, log messages (including those written to the server’s access log and the config-audit.log file) include the DN of the user that requested the change and the IP address of the client system. However, for changes affecting mirrored configuration (including in the topology registry or cluster configuration), these values do not accurately reflect the DN and address of the original requester, but instead reflect either the details of an internal connection or of a connection from another server instance that has forwarded change to the topology master.

To address this, we have updated the server so that the DN and IP address of the original requester are included as part of changes to mirrored configuration. Records for these configuration changes that are written to config-audit.log and the server’s access log will now provide these values in the original-requester-dn and original-requester-ip fields.

New TLS Configuration Properties

We have updated the crypto manager configuration to add support for four new properties for configuring TLS communication:

  • outbound-ssl-protocol — Can be used to specify the set of TLS protocols that may be used for outbound connections (e.g., those used for pass-through authentication or for synchronization with remote servers).
  • outbound-ssl-cipher-suite — Can be used to specify the set of TLS cipher suites that may be used for outbound connections.
  • enable-sha-1-cipher-suites — Can be used to enable the use of TLS cipher suites that rely on the SHA-1 digest algorithm, which is no longer considered secure and is disabled by default.
  • enable-rsa-key-exchange-cipher-suites — Can be used to enable the use of TLS cipher suites that rely on the RSA key exchange algorithm, which does not provide support for forward secrecy and is disabled by default.

Pluggable Pass-Through Authentication

We have updated the Directory Server to add support for pluggable pass-through authentication. Previously, the server provided support for passing through simple bind attempts to another LDAP server or to PingOne. It is now possible to support pass-through authentication to other types of services, and the Server SDK has been updated to add support for creating custom pass-through authentication handlers.

This implementation includes an LDAP pass-through authentication handler that allows the new pluggable pass-through authentication plugin to be used as an alternative to the former LDAP-specific pass-through authentication plugin. The new implementation offers several advantages over the former one, including:

  • Better default configuration properties (especially for the override-local-password property).
  • The ability to indicate whether to attempt pass-through authentication for accounts in an usable password policy state (for example, those that are locked or that have expired passwords).
  • The ability to set timeouts for interaction with the external LDAP servers.
  • Improved diagnostic information about pass-through authentication attempts, including support for the password policy request control and password expired response control.
  • A new monitor entry with metrics about the processing performed by the plugin.

Preserving Secret Keys for Instances Removed From the Topology

Previously, when a server was removed from the topology (for example, by using the remove-defunct-server tool), secret keys associated with that instance could be lost. This is unlikely to cause any problems in most cases because these keys are no longer used for most purposes. However, it could be an issue if the server is configured to use a legacy password storage scheme that protects passwords with reversible encryption. These schemes encrypt passwords with keys from the topology registry, and if a server was removed from the topology, then keys specific to that instance were also removed. This could prevent remaining servers from being able to decrypt passwords that were initially encrypted by the instance that was removed. To address this, we now preserve any secret keys that are associated with an instance before removing that instance from the topology.

Affected password storage schemes include AES, Blowfish, RC4, and 3DES. The newer AES256 password storage scheme is not affected by this issue.

Size Limits for Add and Modify Requests

We have added new maximum-attributes-per-add-request and maximum-modifications-per-modify-request properties to the global configuration. The former can be used to limit the number of attributes that may be included in an add request, and the latter can be used to limit the number of modifications that may be included in a modify request. Neither of these properties affects the number of values that individual attributes may have.

These limits can help avoid potential denial-of-service attacks that use specially crafted add and modify requests. By default, add requests are limited to 1000 attributes, and modify requests are limited to 1000 modifications, which should be plenty for virtually all real-world use cases.

New ldap-diff Features

We have updated the ldap-diff tool to provide several new features. These include:

  • We have added an option to perform byte-for-byte comparisons when identifying differences. By default, the tool uses schema-aware matching, which may not flag differences in values that are logically equivalent but not identical (for example, values that differ only in capitalization for an attribute configured to use case-insensitive matching).
  • You can now use a properties file to provide default values for some or all of the command-line arguments.
  • We improved support for SASL authentication.

New migrate-ldap-schema Features

We have updated the migrate-ldap-schema tool to provide several new features. These include:

  • We have added more flexibility when securing communication with servers over TLS, including the ability to use different key and trust managers for the source and destination servers.
  • We have added support for SASL authentication.
  • We have added support for using a properties file to obtain default values for some or all of the command-line arguments.
  • We have added better validation for migrated attribute types and object classes.

UnboundID LDAP SDK for Java 6.0.3

/ Neil Wilson

We have just released version 6.0.3 of the UnboundID LDAP SDK for Java. It is available for download from GitHub and SourceForge, and it is available in the Maven Central Repository. You can find the release notes at https://docs.ldap.com/ldap-sdk/docs/release-notes.html, but here’s a summary of the changes included in this version:

General Updates:

  • We fixed an issue with the round-robin server set that could cause it to select some servers with a higher frequency than others in the event that one or more of the servers become unavailable.
  • We fixed a potential race condition that could leave a background Timer thread running if a connection was closed while waiting for a response to a request issued via the asynchronous API.
  • We fixed an issue in which the in-memory directory server could incorrectly include a matched DN in certain bind response messages.
  • We improved support for SSL/TLS debugging for SSLUtil operations when used in conjunction with the javax.net.debug system property.
  • We added methods to the JSONLDAPResultWriter class that can be used to encode entries, search result references, and LDAP results to JSON objects.
  • We added a CryptoHelper.inferKeyStoreType method that attempts to determine the key store type for a specified file that is expected to represent a JKS, PKCS #12, or BCFKS key store.
  • We added StaticUtils convenience methods for generating random bytes, integers, and strings, optionally using cryptographically secure random number generators.

Updates Specific to Use With the Ping Identity Directory Server:

  • We fixed an issue that could cause the manage-account tool to fail if it receives a response with an unrecognized password policy state operation type. This is most likely to occur when the tool is communicating with a version of the Ping Identity Directory Server that does not correspond to the version of the manage-account tool.
  • We added client-side support for new extended operations that can allow for remote management of certificates in Ping Identity Directory Server instances.
  • We updated the AuthenticationFailureReason class to add additional failure reason values related to pass-through authentication.
  • We updated the TaskManager class to add methods for using LDAPInterface objects (including connection pools or Server SDK internal connections) as an alternative to LDAPConnection objects when interacting with administrative tasks.

UnboundID LDAP SDK for Java 6.0.2

/ Neil Wilson

We have just released version 6.0.2 of the UnboundID LDAP SDK for Java. It is available for download from GitHub and SourceForge, and it is available in the Maven Central Repository. You can find the release notes at https://docs.ldap.com/ldap-sdk/docs/release-notes.html, but here’s a summary of the changes included in this version:

  • We fixed an issue in the JVM-default trust manager that could prevent it from properly trusting a certificate chain that should be considered valid through a cross-signed relationship, which may include certificates signed by the Let’s Encrypt service in some cases. Although the trust manager provided support for cross-signed certificates, that support would previously only be used if one or more of the certificates in the presented chain were outside of their current validity window.
  • We added the ability to use the tls-server-end-point channel binding type when authenticating with the GSSAPI SASL mechanism. This feature depends on the underlying JVM providing support for this channel binding type and will likely require Java 13 or later.
  • We fixed an issue in the in-memory directory server that could prevent it from returning search result references for smart referral entries within the scope of the search. It would previously only return references for smart referral entries that matched the search filter, but will now return references for any smart referral entry within the scope.
  • We updated the LDAP command-line tool framework to add a --defaultTrust argument that can be used to indicate that the tool should use a default set of non-interactive logic for determining whether to trust a presented certificate chain. This includes at least the JVM’s default trust store, but in tools that are part of a Ping Identity server installation, it may also include the server’s default trust store and the topology registry. This is the same logic that tools would previously use when invoked without any trust-related arguments, with the exception that it will not interactively prompt about whether to trust the presented chain if it cannot be trusted through any of the default mechanisms. As such, it is more suitable for use in scripts that are intended to run in non-interactive settings.

UnboundID LDAP SDK for Java 6.0.1

/ Neil Wilson

We have just released version 6.0.1 of the UnboundID LDAP SDK for Java. It is available for download from GitHub and SourceForge, and it is available in the Maven Central Repository. You can find the release notes at https://docs.ldap.com/ldap-sdk/docs/release-notes.html, but here’s a summary of the changes included in this version:

  • We added a new ldap-diff command-line tool that can be used to compare the contents of two LDAPv3 servers. Any differences identified will be written to a specified file as LDIF change records that may be used to update the source server to match the target server. The tool attempts to minimize the memory required to perform the comparison, and it can use multiple passes to avoid potential false positives that result from delays in replication or changes made while the tool is running.
  • We updated the ldifsearch tool to provide support for presenting the results in alternative output formats. It now supports the same formats as ldapsearch, including LDIF, JSON, CSV (single-valued and multi-valued), tab-delimited text (single-valued and multi-valued), DNs only, and values only.
  • We updated the LDIF reader to make it possible to customize the size limit that it imposes for values read from a file referenced by URL. This limit is a safety feature that prevents consuming too much memory when encountering a reference to a very large file, but the size limit was previously hard-coded to be 10 megabytes. That is still the default, but it’s now possible to use the com.unboundid.ldif.LDIFReader.maxURLFileSizeBytes system property to set an alternative limit.
  • We augmented the LDAP SDK’s debugging support to make it possible to automatically have debug messages written to a file specified by the com.unboundid.ldap.sdk.debug.file system property. This is intended to be used in conjunction with other debugging-related system properties to enable support for debugging in existing applications without the need for a code change.
  • We lowered the debug level for exception messages that may be logged as a result of a SocketTimeoutException that is caught internally in the course of trying to determine whether a connection in a connection pool is still valid. This is a completely normal condition that was previously inadvertently logged at a WARNING level. This could potentially mislead people into thinking that it’s a problem, or at the very least make it harder to find debug messages that are actually important. It is now logged at the FINEST level, so it will only be visible at the highest level of verbosity.
  • We updated support for the Ping Identity Directory Server’s matching entry count control to make it possible to request extended response data, including whether the search is fully indexed, whether identified candidate entries are known to be included in the scope of the search, and any remaining portion of the filter that was not used in the course of building the set of candidate entries.
  • We updated support for the Ping Identity Directory Server’s generate profile administrative task to clarify that all included paths must be relative rather than absolute. Those paths will be treated as relative to the server root, and relative paths that reference portions of the filesystem outside the server root will not be allowed.
  • We updated the documentation to include the latest revision of draft-coretta-x660-ldap in the set of LDAP-related specifications.

Ping Identity Directory Server 8.3.0.0

/ Neil Wilson

We have just released version 8.3.0.0 of the Ping Identity Directory Server. The release notes provide a pretty comprehensive overview of what’s included, but here’s my summary followed by more detailed explanations for many of the changes.

Summary of Deprecated Functionality

Summary of New Features and Enhancements

Summary of Bug Fixes

  • Fix an issue that could allow users in a “must change password” state to issue requests [more information]
  • Prevent warning messages for unrecognized JVM vendors [more information]
  • Fix an issue that could prevent ds-pwp-modifiable-state-json changes from being replicated right away [more information]
  • Improve the logic for maintaining the entry-balancing global index [more information]
  • Fix an issue that could prevent setting up the server on old JVMs without support for 256-bit AES
  • Fix an issue that could interfere with manage-profile replace-profile when using a StatsD monitoring endpoint [more information]
  • Avoid entering lockdown mode when incorrectly believing that there were missed replication changes [more information]
  • Improve replication for dependent changes that may be received out of order [more information]
  • Fix an issue with incorrectly reporting that certain filters were not indexed [more information]
  • Prevent dsreplication status from listing offline servers under incorrect domains [more information]
  • Allow configuring cipher stream providers in Directory Proxy Server, Synchronization Server, and Metrics Engine [more information]
  • Fix an issue preventing manage-profile replace-profile from updating mirrored configuration [more information]
  • Prevent offline config change warnings when using manage-profile replace-profile [more information]
  • Update manage-profile replace-profile to preserve setup logs [more information]
  • Improve validation and behavior when configuring an explicit set of TLS cipher suites [more information]
  • Improve manage-profile replace-profile detection of changes to files not included in the server profile [more information]
  • Fix an issue when trying to update a topology server group with a server that already exists in that group
  • Fix issues with import-ldif with --addMissingRDNAttributes [more information]
  • Fix an issue with dsjavaproperties with --initialize and --jvmTuningParameter [more information]
  • Fix an issue that could prevent Sync failed ops log publishers from being removed
  • Improve the result code when trying to add an entry through the Directory Proxy Server when no backend servers are available or when adding entries with missing parents [more information]
  • Fix a potentially incorrect warning about duplicate jar files detected during startup
  • Fix an issue that could prevent Server SDK plugins from seeing all content in an add operation [more information]
  • Avoid a potential reverse DNS warning message during setup
  • Fix an issue that could cause the server to provide an incorrect estimate for the number of entries matching a filter using a composite index [more information]
  • Improve prompts when using dsreplication in interactive mode [more information]

Deprecate TLSv1 and TLSv1.1

TLS version 1.0 became a standard over twenty years ago in 1999, and TLSv1.1 became a standard over fifteen years ago (in 2006). While they were undoubtedly improvements over the former SSL protocols, they are pretty ancient in the world of computer security and are no longer considered secure. RFC 8996 officially declared them to be historic protocols that should no longer be used. As such, these protocols will no longer be enabled by default in the Directory Server or related products.

Newer, more secure protocols have been around for many years (TLSv1.2 nearly 13 years ago in 2008, and TLSv1.3 almost 3 years ago in 2018), and these will be the only TLS protocol versions enabled by default. This shouldn’t cause any problems unless you have ancient clients that haven’t been updated in over a decade. You can look at SECURITY-NEGOTIATION messages in the server’s access log to identify which TLS protocol versions clients are using, and we strongly recommend updating any clients still using TLSv1 or TLSv1.1. However, if necessary, you can manually enable support for the legacy protocols in the connection handler configuration.

Changes to TLS Cipher Suite Selection

We have deprecated support for TLS cipher suites that use the SHA-1 message digest algorithm. SHA-1 is no longer considered secure, and we had previously only enabled support for these cipher suites to allow for their use in TLSv1 and TLSv1.1. As we have disabled support for those legacy TLS protocols by default, we have also disabled support for TLS cipher suites that rely on SHA-1.

We have also deprecated support for TLS cipher suites that rely on RSA key exchange. RSA key exchange does not support forward secrecy, which means that if the server certificate’s private key is compromised, then any data transferred over a TLS session negotiated with RSA key exchange using that certificate can be decrypted. Note that this does not prevent using certificates with RSA key pairs, as key agreement algorithms like ECDHE and DHE can still be used with RSA certificates.

Even though we no longer enable support for these cipher suites by default, you can manually enable them if necessary. You can also customize the set of enabled cipher suites on a per-connection-handler basis (for example, if you want to allow different cipher suites for HTTPS connections than LDAPS connections).

And in cases in which you do manually configure the set of TLS cipher suites, we have improved the validation that the server performs for that configuration. Previously, if you manually configured the set of cipher suites, but none of those suites were supported by the JVM, then the server would log a message for each of the unsupported suites and just fall back on a default set of suites. It will not reject an attempt to configure custom suites if none of those suites is available in the underlying JVM. For the sake of preserving compatibility (in cases where you update the server, or if you update the JVM to a version with a different set of supported cipher suites), the server will still allow you to include suites that the JVM doesn’t support, as long as at least one of the configured suites is supported, and it will continue to log a message about each unsupported suite.

Deprecate Incremental Backups

The server has always offered the ability to create incremental backups, which contained only database files that had changed since the previous backup. Unfortunately, there have been numerous problems with this over the years, and we have decided to deprecate this functionality rather than continuing to offer a potentially unreliable backup mechanism.

In the 8.3.0.0 release, the functionality will remain available and with all known issues addressed, but if you run the backup command with the --incremental argument, it will display a warning on the command line indicating that the functionality has been deprecated and will likely be removed in a future release. If you have the server itself initiate an incremental backup (through the tasks interface), then it will log a warning message and generate an administrative alert to provide a more visible warning.

As an alternative to performing incremental backups, we recommend LDIF exports. LDIF data compresses very well, and compressed and encrypted LDIF files are substantially smaller than full backups, allowing you to take them more frequently without consuming more disk space. LDIF exports are also more portable and more useful than backups, allowing you to do things like restore individual entries and perform offline analysis of the data.

Another option is to use the server’s data recovery log in addition to LDIF exports or full backups. The data recovery log has been available since the 7.2.0.0 release, and it is a compressed, encrypted audit log that provides a record of recent changes in reversible form so that they can be easily replayed or reverted if necessary. You can use the extract-data-recovery-log-changes tool to write changes matching desired criteria (for example, all changes recorded since the time of a backup or LDIF export) that can then be replayed using a tool like ldapmodify or parallel-update.

Add a FIPS 140-2-Compliant Mode

FIPS 140-2 is a U.S. government specification that defines requirements around the use of cryptography. U.S. government agencies, and organizations that work with those government agencies, may be required to abide by this specification. In the 8.3.0.0 release, we will allow you to set up the server to operate in a FIPS 140-2-compliant mode, in which only cryptographic operations allowed by the specification will be permitted.

Note that servers operating in FIPS 140-2-compliant mode are not directly compatible with servers not running in FIPS-compliant mode. It is not possible to have a mix of compliant and non-compliant servers in the same topology, nor can you directly replicate between them (although you can use the Synchronization Server to keep a FIPS-compliant topology in sync with a non-compliant topology). You will also not be allowed to update an existing non-FIPS-compliant instance to operate in FIPS 140-2-compliant mode.

When installed in this mode, the server uses the FIPS 140-2-certified Bouncy Castle BCFIPS provider in approved-only mode, along with the FIPS-compliant BCJSSE provider for TLS processing. The server will require secure communication, and certificates must be stored in either BCFKS key stores (which the manage-certificates tool now supports) or PKCS #11 tokens. Data encryption must also be enabled during setup, with at least one encryption settings definition. Because of requirements around the use of PBKDF2 in FIPS-compliant mode, passwords for root users and topology administrators will be required to be at least 14 characters long.

Also note that the Bouncy Castle FIPS 140-2-compliant SecureRandom implementation is very entropy-hungry. If the underlying system does not have enough entropy available, attempts to start the server or launch tools requiring the use of secure random numbers may block for extended periods of time. To avoid this, we strongly recommend installing a hardware random number generator or using an entropy-supplementing daemon like rngd.

Add Support for Passphrase Providers

In some cases, the server may require access to clear-text secrets for use in its processing. For example, it may need clear-text credentials for authenticating to external services or for accessing certificate key and trust stores. Historically, the server has allowed you to provide those secrets by either storing an obscured version of the secret directly in the configuration or by storing it in a file.

In the 8.3.0.0 release, we are introducing a new passphrase provider framework that provides an extensible framework for obtaining access to these kinds of secrets. Initially, we will allow obtaining secrets through the following mechanisms:

  • From an obscured representation of the secret stored directly in the configuration
  • From a file contained on the local filesystem, which may optionally be encrypted with a key from the server’s encryption settings database
  • From an environment variable
  • From a HashiCorp Vault server
  • From the Amazon AWS Secrets Manager service

In addition, the Server SDK provides support for creating custom passphrase provider implementations that use other services or methods for obtaining secrets.

Improved Auditability for SCIM2 Requests

By default, the server authorizes SCIM 2 requests through a combination of two mechanisms:

  • With an authorization identity of “cn=SCIM2 Servlet,cn=Root DNs,cn=config”. Although this is a root account, it does not inherit any of the default root privileges, and it is not assigned any privileges by default. Global ACIs grant that account a minimal level of access to use certain request controls (like server-side sort and virtual list view) and operational attribute (like createTimestamp and modifyTimestamp).
  • With any access control rights granted (via the oauthscope ACI bind rule) for scopes included in the OAuth bearer token used to authenticate to the server.

In the past, the server has not attempted to map the access token used to authorize SCIM2 requests to a local user in the server. As such, the access log would always report that the operations were requested by the “cn=SCIM2 Servlet,cn=Root DNs,cn=config” user, and that DN would also appear in the creatorsName and modifiersName attributes of created and updated entries. This is still the default behavior, but in the 8.3.0.0 release, we’re making it possible to map the access token to a local account and use that account as the authorization identity for any requested operations (along with rights granted to scopes included in the token).

Note that if you use this feature, then you will need to ensure that those mapped users have permission to the same request controls and operational attributes that are available to the SCIM2 Servlet user by default. The easiest way to do this is to ensure that the “scim2” scope is included in any access tokens used to authorize the requests, as this scope will be granted the same default access rights as the SCIM2 Servlet user. Otherwise, you will need to define additional access control rules to grant the necessary rights to the mapped users.

Join Virtual Attribute Types

The Directory Server has provided support for an LDAP join request control for several years. This control allows you to request that the server return “related” entries along with entries that match the search criteria. For example, if you use the manager attribute to hold the DN of a user’s manager, then you can use the join request control to return a user’s manager entry along with the user’s own entry. This is a powerful and useful feature, but because it requires an LDAP control, it’s not been available to non-LDAP clients (like those using SCIM or the Directory REST API). Further, because that control uses a proprietary encoding, it’s not all that convenient to use in LDAP clients unless you’re using the UnboundID LDAP SDK for Java.

In the 8.3.0.0 release, we’re adding support for three new virtual attribute types that make it possible to use some of the power afforded by the join request control but without the need to actually use a control. This makes the content accessible from any LDAP client regardless of the API used to implement it, and also to non-LDAP clients.

The new virtual attribute types include:

  • The DN join virtual attribute allows you to join an entry with zero or more other entries whose DNs are contained in the value of a specified attribute. For example, you could use this to join a user with their supervisor through the “manager” attribute in the user’s entry.
  • The reverse DN join virtual attribute allows you to join an entry with zero or more other entries that contain the DN of the source user’s entry in the value of a specified attribute. For example, you could use this to join a manager with their direct reports.
  • The equality join virtual attribute allows you to join an entry with zero or more other entries that are linked by a common attribute value (that is, the source user entry has a value for one attribute that matches the value of the same or a different attribute in other entries).

In each of these cases, values of the virtual attribute will be a JSON object containing a specified set of attributes from the joined entries.

Admin Console Improvements

In the 8.2.0.0 release, we introduced support for single sign-on authentication for the Admin Console, but this was only supported when authenticating with an OpenID Connect ID token from the PingOne service. In the 8.3.0.0 release, we have expanded that to make it possible to authenticate with tokens from any OpenID Connect provider that the server has been configured to accept.

We have also updated the Admin Console so that it’s possible to invoke either the collect-support-data tool or the manage-profile generate-profile tool against the target server through the web-based interface. The resulting support data archive or server profile will be sent to the client as a zip file download. Note that at present, this is only available when authenticating to the console through basic authentication; it is not yet allowed when authenticating via SSO.

Finally, we have updated the Admin Console so that if it’s running in a separate web application container (rather than as part of the Directory Server itself), it will write its log messages to standard output so that they will appear in the container’s console log by default. When the console is running in the same JVM as the Directory Server, its log output will appear in the logs/webapps/console.log file.

Fixes and Improvements for Accounts in a “Must Change Password” State

When the server receives a request to process an operation under an alternate authorization identity (for example, using the proxied authorization or intermediate client request control), it will make sure that the target account is in a usable state. If the account can’t be used for some reason (for example, because it’s been administratively disabled, the password has expired, or it’s been locked as a result of too many failed attempts), then the server will not allow it to be used as an alternate authorization identity. However, prior to the 8.3.0.0 release, the server did not check to see if an account was in a “must change password” state, which could have incorrectly allowed the server to process an operation as a user via an alternate authorization identity even though it would have been rejected when requested on a connection authenticated as that user. Unfortunately, this also affects requests passing through the Directory Proxy Server, as it uses the intermediate client request control to authorize requests as the end user in backend servers. This has been corrected in the 8.3.0.0 release, and we will be fixing it in updates to older releases in the near future.

We have also added support for a new must-change-password account status notification type. This can be used to trigger some action whenever a user successfully authenticates to the server with an account that is in a “must change password” state. For example, you can have the server send the user an email message indicating that they will need to choose a new password before they will be allowed to request any other operations. While it was already possible to do this in many cases through the existing password-reset account status notification type (which will be triggered if an administrator resets a user’s password), the new notification type allows you to take action even if that previous warning was not heeded, or if the “must change password” state was not triggered by an administrative reset.

In addition, we have updated the server so that when a user successfully binds with an account in a “must change password” state, the bind response will now include a diagnostic message that indicates that the user must choose a new password. This may be beneficial for some clients (although many clients don’t check for or use a diagnostic message in a successful response), but the primary benefit is that it makes it easier to spot this condition in the server’s access log.

Fixes and Improvements for the ds-pwp-modifiable-state-json Operational Attribute

In the 8.2.0.0 release, we introduced a new plugin that makes it possible for clients to change certain aspects of a user’s password policy state through a regular modify operation that provides a JSON-formatted value to the ds-pwp-modifiable-state-json operational attribute. While it was already possible to accomplish this using the password policy state extended operation, this operation uses a proprietary encoding that makes it less convenient for LDAP clients not written with the UnboundID LDAP SDK for Java, and non-LDAP clients (for example, those using SCIM or the Directory REST API) couldn’t request it at all. The server’s support for the ds-pwp-modifiable-state-json attribute addresses all of these issues and makes it easier for appropriately authorized clients to manipulate a user’s password policy state.

Unfortunately, there was a bug in the implementation of this plugin in the 8.2.0.0 release that prevented updates to this operational attribute from being replicated right away. The change would be applied locally, but it would only be replicated to other servers in the topology after another update to the entry. This has been fixed in the 8.3.0.0 release. It should also be addressed in an upcoming 8.2.x patch release, but for now, the best way to work around the problem in 8.2 servers is to ensure that you modify some other attribute in the entry after updating the ds-pwp-modifiable-state-json attribute.

There was also an issue with the out-of-the-box configuration for the plugin that prevented it from being invoked for operations that are part of a multi-update extended operation. This has been corrected in the 8.3.0.0 release. The best way to address this in the 8.2.0.0 release is to remove and re-add the Modifiable Password Policy State plugin, and it will be automatically created with the appropriate configuration.

Further, the implementation we used for the plugin in the 8.2.0.0 release included a couple of limitations that restricted the set of modifications in which it could be updated. Those restrictions include:

  • The server would not allow the ds-pwp-modifiable-state-json attribute to be updated in the same modify request as any other attributes.
  • The server would not allow the ds-pwp-state-json attribute to be updated in a modify operation that was part of an LDAP transaction or an atomic multi-update operation.

Both of those were intentional restrictions based on the way we chose to implement this functionality in the 8.2.0.0 release. But we redesigned the implementation in the 8.3.0.0 release so that these restrictions are no longer imposed.

Cipher Stream Provider Improvements

The server uses cipher stream providers to protect the contents of the encryption settings database. Previous versions of the server included several cipher stream provider implementations, including:

  • Read an encryption passphrase from a file contained on the server filesystem
  • Wait for an administrator to interactively supply the passphrase during the server startup process or when launching a tool that required access to the encryption settings database
  • Obtain an encryption passphrase from a HashiCorp Vault instance
  • Derive an encryption key through interaction with the Amazon Key Management Service
  • Use custom logic implemented in a Server SDK extension

In the 8.3.0.0 release, we have added support for an additional cipher stream provider implementation that obtains an encryption passphrase from the Amazon AWS Secrets Manager service.

We have also updated the Directory Proxy Server, Synchronization Server, and Metrics Engine to expose options for configuring cipher stream providers in those products. Previously, cipher stream provider configuration options were only provided for the Directory Server because they were initially only used for encrypting entries before storing them in the database. However, their usage has expanded to other products, and it is now possible to better protect the contents of the encryption settings database when using those products.

Improvements in PKCS #11 Support

The Directory Server has always provided support for a PKCS #11 key manager provider that makes it possible to access listener certificates in a PKCS #11 token, like an HSM. In the past, it was necessary to pre-configure the JVM with an appropriate security provider (typically by editing its java.security configuration file) with the information needed to access the token. While this worked just fine, it’s not ideal for cases in which it’s not convenient to alter the JVM configuration (for example, if the JVM is provided by the underlying operating system, or if the same installation is shared across multiple applications rather than being dedicated to the Directory Server).

In the 8.3.0.0 release, we’ve updated the server to make it possible to enable PKCS #11 support in a stock JVM by dynamically loading the provider. In such cases, you will likely need to provide a configuration file that provides the information the JVM needs to interact with the PKCS #11 token (for example, the path to the native library that implements support for interacting with the token, and possibly other configuration like the slot to use). You can provide this configuration file during setup if you want to enable PKCS #11 support out of the box, or you can specify it in the PKCS #11 key manager provider configuration. If you’ve already got a JVM that has been manually configured with a PKCS #11 security provider, then it will continue to work as before.

We have also updated the manage-certificates tool so that it can interact with certificates in PKCS #11 key stores. In such cases, the --keystore argument must be provided and must point to the provider configuration file needed to configure the JVM to interact with the PKCS #11 token, and the --keystore-type argument must be provided with a value of PKCS11. You will likely also need to use one of the --keystore-password, --keystore-password-file, and --prompt-for-keystore-password arguments to supply the user PIN for the PKCS #11 token.

Providing Initial TLS Certificates via PEM Files

When setting up the server, if you want to be able to accept secure client communication, then you will need to have a listener certificate to use during TLS negotiation. In the past, we have provided two options for this:

  • You can provide setup with information about existing key and trust stores to use.
  • You can have setup generate a self-signed certificate.

In the 8.3.0.0 release, we are adding support for an additional option: you can provide the listener certificate chain, the listener certificate’s private key, and any trusted certificates in PEM files using the --certificateChainPEMFile, --certificatePrivateKeyPEMFile, and --trustedCertificatePEMFile arguments. This can be useful when migrating from a server that uses PEM files rather than PKCS #12 files or Java key/trust stores. It can also be useful when setting up the server in FIPS 140-2-compliant mode, as you would otherwise need to have an existing BCFKS key store or PKCS #11 token with the desired certificate.

Fixes and Improvements for StatsD Monitoring

The server has provided support for a StatsD monitoring endpoint since the 8.0.0.0 release. In the 8.3.0.0 release, we’re adding the ability to include custom tags (as comma-separated key-value pairs) in each metric message. This can help better differentiate messages from separate instances that are going to the same StatsD endpoint.

We have also fixed an issue that could cause a failure to occur when using manage-profile replace-profile on an instance that is configured with a StatsD monitoring endpoint.

Command-Line Tool Fixes and Improvements

We have made many improvements to command-line tools in the 8.3.0.0 release. This includes:

  • We have updated the default JVM configuration for tools so that most of them will attempt to use less memory.
  • We have added a new oid-lookup tool that can be used to obtain information about a given object identifier that may be used by the server (for example, to identify a schema element, control, extended operation, SNMP trap, etc.), or to obtain the object identifier for something with a given name.
  • We have added a new remove-object-class-from-schema tool that can be used to safely remove an object class from the server schema, even if it has previously been (but is no longer) in use. This is similar to the remove-attribute-type-from-schema tool that was introduced in the 8.2.0.0 release, but for object classes rather than attribute types.
  • We made several improvements to manage-profile replace-profile, including:

    • It is now better able to detect changes in files referenced in the setup-arguments.txt file that reside outside the server profile.
    • The server will no longer warn about offline configuration changes the first time it is started after using manage-profile replace-profile to alter the configuration.
    • We fixed an issue that could cause certain setup log files, which might contain useful troubleshooting information about problems encountered during replace-profile processing, to be replaced by the versions in place before the attempt to run replace-profile.
    • We made it more efficient to apply changes that may require administrative actions.
    • We fixed an issue that could prevent replace-profile from updating information in mirrored configuration.
  • We fixed issues with the import-ldif tool when used in conjunction with the --addMissingRDNAttributes argument. In particular, it could fail to add missing RDN values to an entry if the attribute was required by any of the entry’s object classes, and it could have also resulted in entries with multiple values for single-valued attribute types.
  • We fixed an issue with the dsjavaproperties tool that could prevent it from changing the JVM tuning configuration when the --initialize and --jvmTuningParameter arguments were used together.
  • We have made several improvements to the manage-certificates tool, including:

    • We added a new copy-keystore subcommand to make it possible to copy some or all of the certificates in one key store to another key store (creating it if necessary) of the same or a different type.
    • We added support for BCFKS and PKCS11 key store types.
    • We updated the generate-self-signed-certificate subcommand to add support for optional --output-file and --output-format arguments that can be used to write a PEM-formatted or DER-formatted representation of the generated certificate to a specified file.
    • We updated the list-certificates subcommand display the key store type.
  • We have made several improvements to the dbtest tool, including:

    • We updated the dump-database-container subcommand to provide additional information about an entry’s encoding when dumping the id2entry database, including whether the entry is compressed, whether it is encrypted (and if so, with which encryption settings definition), whether there is a digest, and which attributes (if any) the entry may have. It is also possible to restrict the output to only include entries matching a specified filter.
    • We updated the dump-database-container subcommand to provide more useful information when dumping the state database, including the index name and a user-friendly representation of the trust state.
    • We have updated the dump-database-container subcommand to provide more useful information when dumping the contents of the recent-changes database, including an LDIF representation of the change, the change time, the replication CSN, and information about the original client request.
    • We added new dump-attribute-tokens, dump-object-class-tokens, and dump-dn-tokens subcommands that can display information that the server uses in the course of compacting entry data.
    • We added a new dump-metadata subcommand that can display information from the backend’s metadata database.
  • We updated the ldap-result-code tool to add support for additional output formats, including JSON, CSV, and tab-delimited text.

Index Fixes and Improvements

We also made a number of improvements in index management. These include:

  • We fixed an issue that could cause the server to incorrectly infer that a given search filter might not be indexed in a backend if it was covered by a composite index but not by any attribute indexes. The composite index would actually be used to process applicable searches, but this issue could potentially interfere with certain types of valid configuration changes.
  • We have updated import-ldif to increase the number of threads used to flush intermediate index files to disk, which can dramatically improve import performance. We have also made changes to help reduce the number of intermediate index files that the server generates during an LDIF import, which can help avoid running out of file descriptors when importing very large data sets in deployments with a large number of indexes.
  • We have dramatically improved the performance when removing entry IDs from very large composite indexes.
  • We have dramatically reduced the performance impact incurred when a large exploded index exceeds the index entry limit for a key.
  • We fixed an issue that could cause the server to return an incorrect estimate of the number of entries matching a composite index key with a large ID set after an unclean shutdown.

Performance Improvements

Besides those things already mentioned elsewhere above, we have made the following performance improvements to the server and associated tools:

  • We have updated the server’s support for the get user resource limits request control to make it possible to omit information about the target user’s group membership. The Directory Proxy Server now uses this option when forwarding bind requests to backend servers, which can dramatically improve bind performance in servers with a large number of dynamic groups.
  • We have updated the way the server processes searches that target the isMemberOf virtual attribute when the target entry is a dynamic group. This improvement is particularly significant when the client will be paging through the results rather than retrieving all entries in response to a single request.
  • We have updated setup to provide an --optionCacheDirectory argument that can be used to specify the path to cache files with pre-computed information about which JVM options can be used. This can help improve setup performance, as a significant amount of time during setup is spent automatically selecting an appropriate set of JVM options.
  • We have updated the server to make it possible to minimize the content of conflict prevention details entries that may be generated in response to the uniqueness request control. This can help improve write performance when updating large entries in a request that includes uniqueness constraints.
  • We have updated the purge expired data plugin to allow using multiple concurrent threads when deleting expired entries. The plugin is still single-threaded by default, and we recommend keeping that default configuration unless the plugin isn’t able to keep up with the rate at which entries should be purged.

Reduced DN Escaping

RFC 4514 specifies a minimum set of required escaping that must be used for attribute values used in DNs and RDNs. However, it also allows implementations to optionally escape any other characters as desired. In the Ping Identity Directory Server, we have historically also escaped all ASCII control characters and all non-ASCII characters.

As of the 8.3.0.0 release, we have reduced the amount of escaping that we use for non-ASCII characters. By default, we will no longer escape non-ASCII characters that we believe to be printable. This includes characters from the Unicode letter, number, space, punctuation, and symbol classes. We will continue to escape non-ASCII characters from other Unicode classes, as well as non-ASCII data that does not represent a valid UTF-8 encoding. And we will also continue to escape ASCII control characters.

For example, if an entry has a cn attribute value of “José Núñez 🇩🇴”, it would have previously been encoded as “cn=Jos\c3\a9 N\c3\ba\c3\b1ez \f0\9f\87\a9\f0\9f\87\b4” when returned in search result entries. It will now be returned as “cn=José Núñez 🇩🇴”. Note that LDIF requires all non-ASCII values to be base64-encoded, so if you’re looking at that entry in LDIF (like in the output from ldapsearch), then you’ll now see it as a base64-encoded value, but it will actually be returned by the server in unencoded form.

Improved Logging for Multi-Update Operations

The multi-update extended operation allows you to send multiple add, delete, modify, and modify DN operations in a single request. When you want to apply changes to multiple entries, this can help improve performance by reducing the number of round trips required in communicating with the server, and you can also optionally process the entire multi-update operation as a single atomic unit so that if any of the requests cannot be processed, then no changes will be applied.

In the past, the server’s access logging wasn’t as helpful as it should have been when recording information about operations processed as part of a multi-update extended operation. In particular, messages about the individual operations processed as part of the multi-update were only available if you enabled support for logging internal operations, and the extended result log message did not always include useful information about failures encountered during processing. We have addressed both of these issues in the 8.3.0.0 release so that the server’s processing for the individual operations that are part of a multi-update request will be logged without needing to enable internal operation logging, and if any failure is encountered during processing, then the extended result will provide the result code and diagnostic message from the first failure.

Include the Bouncy Castle Library by Default

When operating in non-FIPS-compliant mode, the Directory Server can use the Bouncy Castle cryptographic library for certain optional functionality, including support for the bcrypt, scrypt, and Argon2 password storage schemes. Previously, the server did not ship with the Bouncy Castle library out of an abundance of caution with regard to U.S. export control laws. If you wanted to use any of this functionality, you needed to download the appropriate jar file from the Bouncy Castle website and copy it into the server’s lib directory.

As of the 8.3.0.0 release, this is no longer necessary. The server now ships with both the FIPS 140-2-compliant and non-FIPS-compliant versions of the Bouncy Castle libraries. Note, however, that these libraries are not compatible with each other, so either one or the other will be activated when setting up the server. When running in FIPS 140-2-compliant mode, features that require the non-FIPS-compliant Bouncy Castle library (namely, the aforementioned bcrypt, scrypt, and Argon2 password storage schemes) will not be available.

Request Control Updates

The 8.3.0.0 includes the following updates in our support for request controls:

  • The assured replication request control (which can be used to delay the response to a change until the server has received confirmation that the change has been replicated) is now allowed through the Directory Proxy Server by default without requiring any special configuration. It was previously only possible to allow the control to be passed through the Directory Proxy Server by adding its OID to the supported-control-oid property in the proxying request processor configuration.
  • We have updated the server to allow the operation purpose request control (which the client can use to provide additional context about the request) to be included in a transaction or an atomic multi-update operation.
  • We have updated the default set of global ACIs to allow the LDAP assertion and permissive modify controls by default. These controls do not allow clients to do anything that they wouldn’t be able to do without them, but they can make processing safer and more efficient.

Unrecognized JVM Vendor Warnings

The server setup process validates the selected JVM to ensure that it is suitable for use. In particular, it examines the Java version (we currently support running on either Java 8 or Java 11) and the Java vendor. setup will refuse to run with an unsupported Java version, but it will merely display a warning message if it encounters a JVM with a supported version but from an unrecognized vendor.

Some Linux distributions, like Red Hat and Ubuntu, have overridden the vendor information for the JVMs that are included as part of their operating systems so that they use vendor strings that we did not previously recognize (and in at least one case, Ubuntu used a vendor of “Private Build”). We have updated the set of recognized vendors so that it will now recognize these vendors, and it will no longer display a warning when setting up the server when using one of those JVMs.

Entry Balancing Global Index Updates

When using the Directory Proxy Server with entry balancing (to shard the data across multiple replicated backend sets for improved performance with very large data sets), it maintains a global index that can be used to help it determine which backend set might have a given attribute value.

We have identified and corrected a number of cases in which the server might not correctly update the global index in response to some types of operations. These include:

  • In some cases, the server could have incorrectly populated the global index when processing an add operation. This could have led it to a scenario in which the Directory Proxy Server might only return a subset of entries containing a given attribute value in response to a search. Each global index configuration object now provides a guaranteed-unique configuration property that can be used to indicate that only a single entry is expected to have any one value for the associated attribute type. The global index will now be updated for add operations only for indexes in which guaranteed-unique is set to true.
  • We fixed an issue that could arise for add operations that used an indexed attribute value in the RDN but not in the content for the entry. In such cases, the attribute value from the RDN would not be populated in the global index for the associated attribute type. This has been corrected, and the global index will be populated for such add operations as long as that index is configured with guaranteed-unique set to true.
  • We fixed an issue that prevented the server from updating the global index for operations that were part of a successful transaction or an atomic multi-update operation.
  • We fixed an issue that could prevent global attribute indexes from being updated as a result of modifications used to remove specific attribute values (as opposed to the entire attribute) from an entry. This now will be properly handled for indexes that are configured with guaranteed-unique set to true.
  • Global attribute indexes were previously not updated for delete operations. We have updated the server so that if the server maintains a global index for any attribute used in a deleted entry’s RDN, and if that index is configured with guaranteed-unique set to true, then that global attribute index will be updated to reflect that the value has been removed.

Replication Improvements

The 8.3.0.0 release includes the following replication-related changes:

  • We fixed an issue that could cause the server to infer an incorrect replication enabled time if no value is provided in a message received from a remote replica. That would cause the server to believe that replication had been enabled in 1970 (the beginning of the epoch used for many OS timekeeping systems), and it would cause the server to enter lockdown mode because it believed it had missed changes from that replica.
  • We improved the processing that the server will perform if it receives interdependent changes in an out-of-order sequence. For example, if a client adds an entry and then immediately modifies it, and if a replica receives the add and modify operations from two different other replicas, then it’s possible that it could try to process the modify operation before the add. This could have previously caused replication of the modify operation to fail.
  • We fixed an issue that could cause dsreplication status to display information about offline servers below all replication domains, even if that server only participated in a subset of those domains. It will now only list them under domains in which they participate.
  • We improved the messages displayed when running dsreplication enable and dsreplication initialize in interactive mode to better highlight the servers and base DNs that will be affected by the operation.

Improved Result Codes through the Directory Proxy Server

We fixed a couple of issues that could cause the Directory Proxy Server to return less desirable result codes in certain cases when using an entry-balanced configuration. These include:

  • When attempting to add an entry when all servers in at least one of the backend sets were unavailable, the server could have returned a response with an inappropriate result code of 81 (server down). It will now use a result code of 52 (unavailable).
  • When attempting to add an entry more than one level below the balancing point, and when the ancestor immediately below the balancing point does not exist, then the server could have returned a response with an inappropriate result code of 80 (other) or 81 (server down). It will now use a result code of 32 (no such object).

Fix for Server SDK Plugins Targeting Add Operations

We fixed an issue that could prevent pre-operation add plugins written with the Server SDK from seeing changes made to the add operation by other plugins that are shipped with the server. For example, a Server SDK pre-operation add plugin was previously unable to see any attribute values generated by the composed attribute plugin.