Soft Deletes in the Ping Identity Directory Server

/ Neil Wilson

As its name implies, the LDAP delete operation removes an entry from the directory server. Typically, this completely removes the entry from the server, but there may be times when you would prefer for the entry to be hidden from LDAP clients, while still available in the server for at least a period of time.

The Ping Identity Directory Server offers this capability in the form of soft deletes. A soft-deleted entry still exists in the server, but it is renamed so that the DN includes the entry’s entryUUID value, and a special ds-soft-delete-entry object class is added that ensures the entry won’t be visible to most clients, and to provide additional metadata about the soft delete operation (including the entry’s original DN, the time the entry was soft-deleted, and the authorization DN and IP address of the client that requested it).

Using soft deletes can offer a number of benefits. Some of them may include:

  • It makes it easier to resurrect an entry if it is removed in error. We also provide a simple way to undelete a soft-deleted entry to restore it to “regular entry” status.
  • It provides LDAP-accessible auditing information about the delete operation. Even though soft-deleted entries aren’t visible to most clients, we do provide ways for authorized clients to see them if they’re specifically looking for them.
  • You can use this to prevent reuse of values, even after an entry has been deleted. For example, say that you’re an email provider and you don’t ever want to allow an email address to be reused, even if the former owner has removed their account. The unique attribute plugin has support for either permitting or rejecting conflicts with soft-deleted entries.

There are two ways that you can perform soft deletes in the Ping Identity Directory Server: you can configure the server to automatically turn regular deletes matching a given set of criteria into soft deletes, or you can explicitly request them with the soft delete request control. But before you can do either one, you need to set a soft delete policy.

Configuring the Server’s Soft Delete Policy

A soft delete policy can be used to specify the conditions under which the server should automatically turn regular delete operations into soft deletes, and can also be used to indicate the conditions under which the server should automatically clean up soft-deleted entries.

There are two properties that can be used to specify the conditions under which the server should automatically turn regular deletes into soft deletes:

  • auto-soft-delete-connection-criteria — A reference to a connection criteria object that specifies the clients whose delete requests should automatically be turned into soft deletes. This criteria can include anything the server knows about the requester, including their identity (where their entry is in the DIT, the contents of their entry, their group memberships, etc.), the address of the client, whether the communication is secure, and the protocol they are using to communicate with the server.
  • auto-soft-delete-request-criteria — A reference to a request criteria object that specifies which delete requests should automatically be turned into soft deletes. This criteria can include anything the server knows about the delete request, including the location of the target entry in the DIT, the content of that entry, the groups in which that entry is a member, the controls included in the request, and the origin of the request (e.g., directly requested by a client, replicated from another server, initiated by a component within the server, etc.).

If neither of these properties has a value, then only delete requests that include the soft delete request control will be treated as soft deletes. If only one of them has a value, then all delete requests that match that criteria object (or that include the soft delete request control) will be treated as soft deletes. If both of them have values, then only delete requests that match both sets of criteria (or that include the soft delete request control) will be treated as soft deletes.

By default, soft-deleted entries will remain in the server forever (or until someone explicitly deletes them), but you can also configure the server to automatically delete them under certain conditions. The soft delete policy offers two properties that can be used to control this:

  • soft-delete-retention-time — The maximum length of time that soft-deleted entries should be retained in the server before they are eligible to be automatically removed.
  • soft-delete-retain-number-of-entries — The maximum number of soft-deleted entries that should be retained in the server.

If either or both of these properties is configured, then soft-deleted entries that fall outside of either one of them will be eligible for removal. If neither is configured, then soft-deleted entries won’t be automatically removed by the server.

To enable soft delete functionality in the server, you need to create a soft delete policy, and you also need to update the global configuration to make it the active policy. With no active soft delete policy, the server will not automatically turn any deletes into soft deletes, nor will it allow clients to use the soft-delete request control.

Example 1: Only Explicit Soft Deletes Without Automatic Cleanup

If you don’t want the server automatically turning regular deletes into soft deletes, but you do want to allow clients to use the soft delete request control, and if you don’t want the server to automatically clean up any soft-deleted entries, then you can just create a soft delete policy with the default settings and make that the active policy. You can do that with the following configuration changes: 

dsconfig create-soft-delete-policy \
     --policy-name "Explicit Soft Delete Requests Without Cleanup"

dsconfig set-global-configuration-prop \
     --set "soft-delete-policy:Explicit Soft Delete Requests Without Cleanup"

Example 2: Automatic Soft Deletes With Automatic Cleanup

If you want the server to automatically turn all delete operations into soft deletes, and to keep soft-deleted entries around for 30 days, you can do that with the following changes: 

dsconfig create-request-criteria \
     --criteria-name "All Delete Requests" \
     --type simple \
     --set operation-type:delete

dsconfig create-soft-delete-policy \
     --policy-name "Automatic Soft Deletes" \
     --set "auto-soft-delete-request-criteria:All Delete Requests" \
     --set "soft-delete-retention-time:30 d"

dsconfig set-global-configuration-prop \
     --set "soft-delete-policy:Automatic Soft Deletes"

The Soft and Hard Delete Controls

The Soft Delete Request Control

If you want to explicitly control which delete requests get turned into soft deletes, then you can include the soft delete request control in the delete request. This request control has an OID of “1.3.6.1.4.1.30221.2.5.20”, and it can optionally have a value. If there is a value, then it should have the following ASN.1 encoding: 

SoftDeleteRequestValue ::= SEQUENCE {
     returnSoftDeleteResponse     [0] BOOLEAN DEFAULT TRUE,
     ... }

The Soft Delete Response Control

If the request control doesn’t have a value, or if it has a value with the returnSoftDeleteResponse flag set to true, then the delete result may include a soft delete response control with an OID of “1.3.6.1.4.1.30221.2.5.21” and whose value is simply the string representation of the DN for the soft-deleted entry. The soft-deleted entry DN will be the same as the original DN, but with the RDN updated to include the entry’s entryUUID attribute value. For example, if the entry “uid=jdoe,ou=People,dc=example,dc=com” has an entryUUID value of “53e84e32-4be9-4ed6-b489-88d8bea4bdcd”, then the resulting DN for the soft-deleted entry would be “entryUUID=53e84e32-4be9-4ed6-b489-88d8bea4bdcd+uid=jdoe,ou=People,dc=example,dc=com”. The soft-deleted entry will also include the ds-soft-delete-entry object class, and it will include a ds-soft-delete-from-dn attribute whose value was the DN of the original entry and a ds-soft-delete-timestamp attribute whose value reflects the time that the soft delete operation was performed.

The Hard Delete Request Control

We also offer a hard delete request control, which can be used to explicitly indicate that an entry should be completely removed, even if the server would have otherwise automatically turned the delete operation into a soft delete. The hard delete request control has an OID of “1.3.6.1.4.1.30221.2.5.22” and no value. There is no corresponding hard delete response control.

Using the Soft and Hard Delete Controls With ldapmodify

The ldapmodify command-line tool offers support for the soft delete request control via the “--softDelete” argument, and for the hard delete request control via the “--hardDelete” argument.

For example, the following can be used to remove the “uid=jdoe,ou=People,dc=example,dc=com” entry using a soft delete operation: 

$ bin/ldapmodify --hostname ldap.example.com \
     --port 636 \
     -useSSL \
     --trustStorePath config/truststore \
     --bindDN "uid=admin,dc=example,dc=com" \
     --softDelete
Enter the bind password:

# Successfully connected to ldap.example.com:636.

dn: uid=jdoe,ou=People,dc=example,dc=com
changetype: delete

# Deleting entry uid=jdoe,ou=People,dc=example,dc=com ...
# Result Code:  0 (success)
# Soft Delete Response Control:
#      OID:  1.3.6.1.4.1.30221.2.5.21
#      Soft-Deleted Entry DN:  entryUUID=53e84e32-4be9-4ed6-b489-88d8bea4bdcd+uid=jdoe,ou=People,dc=example,dc=com

Using the Soft and Hard Delete Controls With the UnboundID LDAP SDK for Java

The UnboundID LDAP SDK for Java supports the soft delete controls via the SoftDeleteRequestControl and the SoftDeleteResponseControl classes. It supports the hard delete request control via the HardDeleteRequestControl class. The class-level Javadoc documentation for the SoftDeleteRequestControl includes an example that demonstrates the use of these controls, and the related undelete and soft-deleted entry access request controls.

The Soft-Deleted Entry Access Request Control

There wouldn’t be much benefit to having soft-deleted entries if we didn’t provide a way to get access to them. By default, soft-deleted entries are hidden from clients, so they won’t be included in search results. However, there are two ways that you can access soft-deleted entries:

  • If you know the soft-deleted entry’s DN, then you can retrieve that entry with a search using the baseObject scope.
  • If you issue a search with the soft-deleted entry access request control, then soft-deleted entries can be included in the search results.

The latter option is much more useful than the former because it’s hard to know what a soft-deleted entry’s DN is unless you knew that entry’s UUID value before it was deleted, or you got the soft-deleted entry DN from the soft delete response control.

The soft-deleted entry access request control has an OID of “1.3.6.1.4.1.30221.2.5.24”. It may optionally have a value, and if it does, then that value must have the following ASN.1 encoding: 

SoftDeleteAccessRequestValue ::= SEQUENCE {
     includeNonSoftDeletedEntries     [0] BOOLEAN DEFAULT TRUE,
     returnEntriesInUndeletedForm     [1] BOOLEAN DEFAULT FALSE,
     ... }

The includeNonSoftDeletedEntries element of the request control indicates whether the server should include non-soft-deleted entries in the search results. If this is true (which is the default), then the set of entries returned may include both soft-deleted and non-soft-deleted entries. If this is false, then only soft-deleted entries will be returned.

The returnEntriesInUndeletedForm element of the request control indicates whether matching soft-deleted entries should be returned in their undeleted form (if true) rather than their soft-deleted form (if false). The main difference between these forms is that the undeleted form will have the entry’s original DN rather than the soft-deleted DN that includes the entryUUID attribute, and will not include the ds-soft-delete-entry object class or the ds-soft-delete-from-dn or ds-soft-delete-timestamp attributes.

If the request control doesn’t have a value, then it will behave as if you provided a value with includeNonSoftDeletedEntries set to true and returnEntriesInUndeletedForm set to false.

Using the Soft-Deleted Entry Access Request Control With ldapsearch

The ldapsearch command-line tool offers support for the soft-deleted entry access request control through the --includeSoftDeletedEntries argument. This argument must take a value, and that value should be one of the following:

  • with-non-deleted-entries — Indicates that both soft-deleted and non-soft-deleted entries should be included in the search results. Soft-deleted entries will be returned in their soft-deleted form.
  • without-non-deleted-entries — Indicates that only soft-deleted entries should be returned, in their soft-deleted form. Non-soft-deleted entries will not be returned.
  • deleted-entries-in-undeleted-form — Indicates that only soft-deleted entries should be returned, but they should be returned in their undeleted form.

For example, if you wanted to search for the soft-deleted entry with a uid value of jdoe, you could use a command like: 

$ bin/ldapsearch --hostname ldap.example.com \
     --port 636 \
     --useSSL \
     --trustStorePath config/truststore \
     --bindDN "uid=admin,dc=example,dc=com" \
     --baseDN "dc=example,dc=com" \
     --scope sub \
     --requestedAttribute "*" \
     --requestedAttribute "+" \
     --includeSoftDeletedEntries without-non-deleted-entries \
     "(uid=jdoe)"
Enter the bind password:

# Soft Delete Response Control:
#      OID:  1.3.6.1.4.1.30221.2.5.21
#      Soft-Deleted Entry DN:  entryUUID=53e84e32-4be9-4ed6-b489-88d8bea4bdcd+uid=jdoe,ou=People,dc=example,dc=com
dn: entryUUID=53e84e32-4be9-4ed6-b489-88d8bea4bdcd+uid=jdoe,ou=People,dc=example,dc=com
objectClass: top
objectClass: person
objectClass: organizationalPerson
objectClass: inetOrgPerson
objectClass: ds-soft-delete-entry
sn: Doe
cn: John Doe
givenName: John
uid: jdoe
createTimestamp: 20190227170715.814Z
creatorsName: cn=Directory Manager,cn=Root DNs,cn=config
modifyTimestamp: 20190227170715.814Z
modifiersName: cn=Directory Manager,cn=Root DNs,cn=config
entryUUID: 53e84e32-4be9-4ed6-b489-88d8bea4bdcd
ds-soft-delete-from-dn: uid=jdoe,ou=People,dc=example,dc=com
ds-soft-delete-timestamp: 20190227170734.870Z
ds-entry-checksum: 2440630426
subschemaSubentry: cn=schema

# Result Code:  0 (success)
# Number of Entries Returned:  1

Using the Soft-Deleted Entry Access Request Control With the UnboundID LDAP SDK for Java

The UnboundID LDAP SDK for Java provides support for the soft-deleted entry access request control through the SoftDeletedEntryAccessRequestControl class. The class-level Javadoc documentation for the SoftDeleteRequestControl class provides an example that demonstrates how to use this control (along with the soft delete, hard delete, and undelete request controls).

The Undelete Request Control

Support for soft deletes would also not be very useful if we didn’t provide a way to restore a soft-deleted entry back to being a regular, non-soft-deleted entry. And we do offer that ability through the undelete request control. If you include this control in a specially crafted add request, then the server will restore the target entry back to its former glory. The undelete request control has an OID of “1.3.6.1.4.1.30221.2.5.23”, and it does not need a value. There is no corresponding response control.

Note that I mentioned a “specially crafted add request” in that last paragraph. The server handles the undelete operation as an add operation, but if the undelete request control is present, then the contents of that add request will be a little different from when you’re adding an entry from scratch. Here’s what you need to include:

  • The DN included in the add request should be the DN that you want the undeleted to have. If you want this to be the entry’s original DN, then you could use the value of the soft-deleted entry’s ds-soft-delete-from-dn attribute, but you can choose something else if you want the restored entry to have a different DN.
  • The add request must include a ds-undelete-from-dn attribute whose value is the DN of the soft-deleted entry that you want to undelete.

Using the Undelete Request Control With ldapmodify

The ldapmodify tool supports the use of the undelete request control through the “--allowUndelete” argument. If you add this argument, then the undelete request control will automatically be included in any add requests that it sends. For example: 

$ bin/ldapmodify --hostname ldap.example.com \
     --port 636 \
     -useSSL \
     --trustStorePath config/truststore \
     --bindDN "uid=admin,dc=example,dc=com" \
     --allowUndelete
Enter the bind password:

# Successfully connected to ldap.example.com:636.

dn: uid=jdoe,ou=People,dc=example,dc=com
changetype: add
ds-undelete-from-dn: entryUUID=53e84e32-4be9-4ed6-b489-88d8bea4bdcd+uid=jdoe,ou=People,dc=example,dc=com

# Adding entry uid=jdoe,ou=People,dc=example,dc=com ...
# Result Code:  0 (success)

Using the Undelete Request Control with the UnboundID LDAP SDK for Java

The UnboundID LDAP SDK for Java supports the undelete request control via the UndeleteRequestControl class. This class even provides a helpful createUndeleteRequest convenience method that allows you to construct an appropriate add request when provided with the DN that you want the undeleted entry to have and the DN of the soft-deleted entry that you want to undelete. As noted above, the class-level Javadoc documentation for the SoftDeleteRequestControl class provides an example that demonstrates how to use all of the controls related to soft-delete processing.