So I guess SLAMD is a thing again…

The year was 2002. I had recently jumped ship from Netscape to Sun Microsystems after AOL bought Netscape and decided they wanted out of their iPlanet alliance. I was working as a sustaining engineer on whatever Sun’s brilliant marketeers decided to call their LDAP directory server at the time. One day, my boss, Steve Shoaff, came into my office with a couple of ideas. He said that he wanted me to build a tool that could measure the directory server performance with a lot of load by hitting it from multiple clients at the same time. And he said that he wanted to call it “SLAMD”, which is a play on “slapd”, which kind of stands for “standalone LDAP daemon” and is used in the process names of some directory server products.

So I built it, and I think that it’s fair to say that it turned into something substantially more impressive than either of us originally imagined. It had a Java-based API that you could use to define the types of workloads that you wanted to process, a web-based interface that you could use to schedule jobs and view the results (numerically and graphically), and a client that you could install on the systems that you wanted to used to drive load against the server. Over time, I added new types of jobs and lots of other features, like self-optimizing jobs (which repeatedly run the same job with different amounts of client load to find the optimal performance), job groups (which let you schedule several jobs to run in succession), and resource monitoring (which lets you monitor system statistics like CPU, disk, and network utilization on various systems).

SLAMD was pretty good at what it did, and it worked with all types of LDAP-compliant directory servers, so it became one of the preeminent directory server benchmarking tools. We convinced Sun to open source it, and lots of people started using it. It could be used for things other than directory servers, too (I did build some basic support for other protocols like HTTP, POP3, IMAP, and SMTP, and the ability to interact with relational databases), but LDAP performance and stress testing was always its big wheelhouse.

Fast forward several years, and SLAMD was still pretty great but was starting to show its age, at least under the covers. I started working on it in the Java 1.3 days, before nice features like generics, foreach, concurrency APIs, sub-millisecond timing, and so much more. The web interface was all hand-crafted HTML and mostly contained in one giant source file, and it was getting pretty unwieldy. I did make some attempts to try to modernize it, but I got really busy with other things, like creating OpenDS as a replacement for Sun’s stagnating C-based directory server, then moving on from Sun to launch UnboundID and working furiously to build up its directory server, directory proxy server, and LDAP SDK products.

Shortly after Sun and I parted ways, Oracle bought Sun and gradually started killing off most of the good things about it. This included shutting down the java.net site, which had been the open source repository for SLAMD, and I decided to take that opportunity to just let it kind of fade away. I figured it might be better to start something new from scratch, with a much more modern foundation, than to try to give SLAMD the kind of makeover I thought it needed. Of course, that was nearly a decade ago, and while I’ve done a lot since then, creating a new directory server benchmarking tool (other than a handful of command-line tools like searchrate and modrate that we ship with the LDAP SDK) hasn’t really been in the cards. Meanwhile, SLAMD is still getting a surprising amount of use. Even though it’s not so easy to get your hands on it anymore, people were still getting their hands on it and using it.

After having the topic come up several times in the last few weeks, I finally bit the bullet and dusted off the old code. I spent a couple of weekends doing some pretty extensive code cleanup. I fully generified everything, so there aren’t any more build warnings about raw types. I pulled in much more modern versions of dependencies like Apache Tomcat, the Berkeley DB Java Edition, and the UnboundID LDAP SDK for Java. I reorganized some of the jobs, including putting some legacy stuff out to pasture, and I wrote new versions of several of them. I split up some of the admin interface code into separate source files to make it more manageable, and I made some minor user interface enhancements.

So anyway, I went ahead and put the updated code on GitHub at https://github.com/dirmgr/slamd. Since no single entity owns the copyright on the code, it’s not possible to change the license, and it will therefore always will be licensed under the terms of the Sun Public License version 1.0. I’m not promising that I’ll add any major new features, but it’ll at least be more readily available than it has been, and with some more modern guts.

For now, if you want to use it, you’ll need to check it out and build it for yourself (there’s a README that tells you how to do that). Just know that it’s not backward-compatible with the version that I last touched in 2010, so don’t try to upgrade an existing instance (but if you do want the code for that old version, just check out revision 5777f3e5d78ff03985af4e68670e649127339c59, since I used it to seed the new repository).

Also note that there’s still a lot more work to do. There’s quite a bit more code cleanup that’s still on my to-do list (it builds cleanly with Java 8, but there are several deprecation warnings with Java 11). I plan on rewriting some more of the jobs (including making some potentially-incompatible changes). I know that some of the resource monitoring is broken (at least on Linux, which isn’t so concerned about maintaining consistent output in some of its commands). I haven’t touched any of the documentation. I’ve only done a very minimal amount of testing so far. So while it’s fine to play around with what’s there now, and please report issues if you find them, just know that I reserve the right to make even more non-backward-compatible changes as I continue to modernize the code.

SLAMD 2.0.1

I have just released a new version of SLAMD, version 2.0.1. It is available for download at http://www.slamd.com/. The release notes provide a pretty complete list of the changes included in this release, but some of the most notable updates include:

  • The code used to generate graphs has been improved in a number of ways. Various font sizes and styles are now used to make elements of graphs stand out, and antialiasing has been enabled for the text. The thickness of lines in line graphs has been increased and a light gray background is used for the plot area to help make the lines easier to see, especially for light colors like yellow. Also, the default graph size has been increased from 640×480 to 800×600.
  • When scheduling a job, the job duration and/or statistics collection interval can be specified in a more human-readable format. You can still use the number of seconds like in the past, but you can also use more user-friendly forms like “6 minutes”, “1 hour”, or “1 day 12 hours”.
  • A new version of the NetStat resource monitor is available that provides a number of bug fixes and increased functionality. It is now possible to configure the set of interfaces to monitor and/or to have statistics aggregated across the monitored interfaces.
  • The HTTP GetRate job has been updated so that it provides the ability to authenticate to a proxy server.
  • A bug has been fixed that could cause a problem with the use of the HTTP client if the response does not include a Content-Type header.
  • A bug has been fixed in a number of LDAP-related jobs that could cause them to behave incorrectly if a value pattern was specified using a sequential pattern. That pattern could be repeated by all threads on the client, instead of having the same range shared by all threads so that each value was used by only one of the threads.
  • A bug has been fixed in a number of jobs that could prevent statistics from being collected if a cool-down time was provided but no duration or stop time was defined.
  • Jobs providing rate-limiting capabilities have been updated so that you can control the period of time over which the rate limiting is enforced. This can be used to help make it more reliable over time, with the trade-off of occasionally exceeding the rate limit for a brief period of time to make up for periods of slower activity.
  • New jobs have been added which can be used to perform asynchronous LDAP search and modify operations with multiple concurrent outstanding requests on client connections.
  • The LDAPDecoder tool has been updated to provide the ability to decode LDAP traffic from snoop or tcpdump packets in which LDAP messages do not exactly align with packet boundaries (e.g., packets containing multiple messages, and/or messages spanning multiple packets). Also, when generating a SLAMD script to reproduce the LDAP communication, it is now possible to exclude information about server responses.

SLAMD 2.0.0-20090712

A new build of SLAMD is available for download from http://www.slamd.com/. The release notes provide a full list of the changes since the previous 2.0.0-20090227 release, but some of the most significant changes are listed below.

  • The process used to load classes has been updated so that if an attempt to load a class fails, it will re-try the attempt after substituting “com.sun.slamd” with just “com.slamd”. This makes it possible to view statistics collected from builds of SLAMD prior to the 2.0.0-20090227 release.
  • Two new jobs have been added which make it possible to perform LDAP searches using filters read from a file and LDAP modifications with target entry DNs read from a file. These jobs can be used if the target search filters or entry DNs do not follow a pattern that can be constructed like the other jobs provide.
  • A number of jobs have been updated to provide support for rate limiting. You can now specify the desired number of operations per second per client, and the clients will attempt to maintain that rate.
  • A number of shell scripts used to launch tools have been updated to fix problems that prevented them from working properly in some cases.
  • LDAP jobs have been updated to use the synchronous mode provided in the latest release of the LDAP SDK for Java. This allows them to be significantly more efficient and drive more load against LDAP directory servers.

Updating SLAMD Jobs to the New API

Since the latest update to SLAMD broke backward compatibility with previous versions, then anyone who has written their own custom jobs will need to update them to work with the latest release. Here’s what you need to do in order to achieve that:

  • The overall package structure has changed from “com.sun.slamd.*” to be “com.slamd.*“. For example, if you had previously imported “com.sun.slamd.parameter.ParameterList“, then you will now need to import “com.slamd.parameter.ParameterList“.
  • The former “com.sun.slamd.example” package, which holds most of the jobs provided with SLAMD, has been renamed to “com.slamd.jobs“.
  • The code has been updated to use generics, so any collections used within SLAMD have been generified. In order to avoid build warnings, you should use generics in your code as well.
  • All cases in the code in which the StringBuffer class was previously used have been replaced with the new StringBuilder class. The StringBuilder class can help improve performance because it doesn’t perform any synchronization the way that StringBuffer does.
  • Previously, the JobClass class, which is the base class for all SLAMD jobs, had a single getJobDescription() method which could be used to obtain the description for the job. That has now been replaced with the following two methods:
    • public abstract String getShortDescription() — This should return a short (preferably a single sentence) overview of the purpose for the job. It will be used in a few places in the administrative interface, including as a pop-up hint when hovering over the name of the job on the page listing the jobs available to be scheduled. This method must be provided.
    • public String[] getLongDescription() — This may be used to return a more detailed description of the sentence. It should return a string array, and each element of the array will be treated as a separate paragraph in the administrative interface. If this method is not provided, then the short description will be used.
  • The JobClass class formerly provided a destroy() method that allowed you to provide code that would be used in an attempt to forcefully kill a job. Because the JobClass class extended Thread, it overrode the deprecated Thread.destroy() method, which could generate build warnings. I have renamed the JobClass.destroy() method to be destroyThread() to eliminate this conflict. I also made JobClass.destroy() final so that you can no longer override it in job classes.
  • The former com.sun.slamd.example.Base64Encoder class has been removed. The ability to perform base64 encoding and decoding is now provided by the com.unboundid.util.Base64 class in the UnboundID LDAP SDK for Java.
  • The former com.sun.slamd.example.ValuePattern class has been removed. This capability is now provided by the com.unboundid.util.ValuePattern class in the UnboundID LDAP SD for Java. The com.sun.slamd.example.NumericRange class, which had been used by the old ValuePattern implementation has also been removed.

SLAMD 2.0.0-20090227

After quite a significant hiatus, I have just released a new version of SLAMD, which I am calling 2.0.0-20090227. It isn’t the official 2.0.0 release, but it should be pretty solid, and there are significant improvements over the previous 2.0.0-alpha1 build. You can get it at http://www.slamd.com/, and the source is hosted in a subversion repository at http://slamd2.dev.java.net/.

This is the first public build of SLAMD since I left Sun, but the SLAMD codebase hasn’t exactly been collecting dust since then. One of the first things that I did when we founded UnboundID was to start working on the UnboundID LDAP SDK for Java, and within a few weeks of that I was already using it in SLAMD jobs. However, since until recently our LDAP SDK was not publicly available, I couldn’t really release a version of SLAMD that included it, and I didn’t want to maintain a separate private version of the code that made use of the SDK and a public version that didn’t. Now that our LDAP SDK is open source and publicly available, I’m able to use it in SLAMD without any concerns.

Since it has been over two years since the last public commit, and since I had problems with an older mail server, I have recreated all of the SLAMD mailing lists without any members. If you were previously subscribed to any of the mailing lists and want to keep it that way, then you’ll need to re-subscribe to the desired lists. Instructions for doing so are available at http://www.slamd.com/mailing-lists.shtml.

Note that a number of changes have been made in this release that make it incompatible with earlier releases, and you probably won’t be able to export data from an older server and import it into this new version and have it all work seamlessly. I think that the enhancements in this version were worth it, especially if you’re using it to test LDAP directory servers, and hopefully the incompatibilities introduced now will help avoid the need to introduce further incompatibilities in the future.

Changes Since SLAMD 2.0.0-alpha1

There have been a lot of changes made to SLAMD since the 2.0.0-alpha1 release. Some of the most significant changes are outlined below. There have also been a number of less significant bugfixes and enhancements that probably aren’t worth mentioning individually.

Source Code Refactoring

The source code has been completely refactored in several ways. The code now uses a simplified package structure of “com.slamd.*“. The “example” sub-package has been renamed to “jobs“. All of the source code is now in one single directory structure rather than having separate source locations for a number of the tools.

The source code has also been fully updated to use Java 5 functionality. As a result, SLAMD now requires that you use Java 5.0 or later (I recommend the latest Java 6 build). The code has been updated to use generics and builds cleanly with all lint options enabled. As a result of updating to Java 5, I have also been able to take advantage of the better concurrency support and more accurate timing that it provides.

The job API has been updated so that it is now possible to provide both long and short descriptions. Further, where it previously used the destroy() method as a means of trying to forcefully stop a job, I have changed that to destroyThread(), since the former method overrode the deprecated Thread.destroy() method which I shouldn’t have done in the first place.

Job Updates

As mentioned above, most of the LDAP-based jobs now use the UnboundID LDAP SDK for Java. This provides better performance and scalability than the Netscape Directory SDK that I had previously been using, and it’s also much more stable under load. All of the LDAP-based jobs using the UnboundID LDAP SDK now extend a common parent class which provides a number of benefits for these jobs, including:

  • It is now possible to perform client-side load balancing so that you can stress multiple servers at once with a single job.
  • All of these jobs support SSL and StartTLS for secure communication with the directory server, and I’ve simplified the interface so that it’s no longer necessary to worry about key or trust stores.
  • These jobs offer the ability to specify a response time threshold, which can be used to keep track of the number of times that an operation took longer than the specified length of time to complete.

I have consolidated many of the LDAP-based jobs where it made sense to do so, so that there are no longer as many jobs but you should still have the same basic set of capabilities, in addition to new features.

The LDAP Add and Delete Rate job now provides the ability to control how the operations are performed. You can choose to perform only adds, only deletes, or both adds and deletes. If you choose to perform both, then you can either have all of the adds done first and then all of the deletes, or you can choose to delete each entry immediately after adding it.

The LDAP AuthRate job now supports the ability to authenticate using the CRAM-MD5, DIGEST-MD5, or PLAIN SASL mechanisms in addition to using simple authentication.

The LDAP ModRate job allows you to specify the set of characters to use to create the modification, so it will work better with attributes that syntaxes that don’t accept arbitrary characters.

I’ve added a new LDAP Modify DN Rate job which can be used to measure modify DN performance. You can specify a range of entries to be renamed, and the job will actually rename them twice. The first time it will rename them to include the job ID in the RDN, and the second time it will rename them back to the original value.

I’ve added a new LDAP Multi-Connection SearchRate job which allows you to perform searches while maintaining large numbers of connections to the directory in order to test performance with lots of active connections.

I’ve added a new “Wait for Directory” job which allows you to wait for a directory server to become available and start accepting requests before continuing on with other jobs that need to communicate with the server.

I have gotten rid of some older jobs that hadn’t been updated in a while and that targeted outdated software or software that I wasn’t in a position to support. This primarily included the jobs targeting an ancient version of the Sun Calendar Server, old versions of the Access Manager product, and the Identity Synchronization for Windows. If this is a significant problem, then it shouldn’t be difficult to resurrect them, but I didn’t want to invest the time cleaning them up and I wasn’t in any position to test them to see if they still work properly.

Resource Monitor Updates

A new resource monitor has been provided that makes it possible to capture arbitrary information from an LDAP directory server that exposes monitoring information. This was written by Bertold Kolics and it is very useful because it can be used to collect all kinds of monitoring information about the state of the directory server while a job is running (if the directory that you’re targeting exposes the desired information).

I’ve also fixed some bugs in a couple of the resource monitors. In particular, the VMStat monitor has been updated to support newer versions of Linux, and the IOStat monitor has been updated to fix a problem where it didn’t always work properly on Solaris.

Other Updates

I have updated the TimeTracker so that it uses the high-resolution timer which can provide up to nanosecond-level accuracy. Previously the TimeTracker only supported millisecond-level accuracy, but this wasn’t fine-grained enough for timing things that took less than a few milliseconds to complete.

The SLAMD client and resource monitor clients have been updated so that it is now possible to explicitly specify the client ID and source IP address to use when communicating with the SLAMD server. This is particularly useful on multi-homed systems, especially when they might have multiple IP addresses on the same network.

I have updated the job scheduler so that it now uses a blocking queue that allows a new job to be seen immediately. If it’s eligible to start running at the time that it’s scheduled, then it will often be the case that the job will already be running before the administrative interface renders the page indicating that the job has been scheduled.

I have changed the layout of the tools directory so that instead of providing separate subdirectories for each of the major tools, the code for each of those tools is integrated into the main codebase and now there are just shell scripts and batch files for launching them. I have also added a new Tools Guide that summarizes the tools that are available and provides information about using them.

The UnboundID LDAP SDK for Java provides searchrate, modrate, and authrate command-line tools as example programs, and they are now exposed as tools available for use within SLAMD. They are also available through shell scripts or batch files in the tools directory.

I have made some changes to the way that statistics are displayed in the administrative interface so that they are less likely to require horizontal scrolling.

The descriptions for the available jobs have been improved, and it’s now easier to tell what a job does when you schedule it. On the page allowing you to pick which type of job to schedule, SLAMD now provides a pop-up hint that gives a short description of the job. The page allowing you to provide the parameters to use when scheduling a job has also been updated to provide a more detailed description of the job at the top of the page.

The Future of SLAMD

Even to the casual observer it’s not too hard to notice that not much has happened with SLAMD in quite a while.  In fact, it’s been almost two years since my last commit to the public repository.  There are several reasons for this, including:

  • It works pretty doggone well in its current state.  I still use it quite a bit and hear that several other people do, too.  There are definitely ways that it could be improved, but so far it has been able to meet my needs.  Most of the SLAMD code that I have written since then has been in the form of new jobs that were pretty task-specific and not something that are likely to be useful in other environments.
  • Most of the time, I have been really busy with other things.  I just haven’t had nearly as much time to invest in it as I would have liked.  Of course, I did have a forced two-month vacation near the end of last year, but the terms of my severance stated that I wasn’t allowed to do any work and I didn’t want to press my luck.  After that period ended I’ve been going full-steam-ahead on something else.
  • There are parts of it that could do with a redesign.  The code used to generate the administrative interface is currently all held in one large file and could stand to be broken up.  There are also many areas in which updating the code to require Java 5 would allow it to be much more efficient and scalable, and the introduction of features like generics and enums would make the code easier and safer to edit.

Ultimately, I think that it’s at the point where it would be better to invest the effort in a clean rewrite than to try to build upon what’s there now, but so far I haven’t had much opportunity to do either one of them.  It’s definitely something that I would like to do and I’m hopeful that I might have the time to do it at some point in the future.  I have a lot of ideas for interesting and powerful enhancements, so I don’t want to count it out just yet.