Sun Microsystems

So I guess SLAMD is a thing again…

/ Neil Wilson

The year was 2002. I had recently jumped ship from Netscape to Sun Microsystems after AOL bought Netscape and decided they wanted out of their iPlanet alliance. I was working as a sustaining engineer on whatever Sun’s brilliant marketeers decided to call their LDAP directory server at the time. One day, my boss, Steve Shoaff, came into my office with a couple of ideas. He said that he wanted me to build a tool that could measure the directory server performance with a lot of load by hitting it from multiple clients at the same time. And he said that he wanted to call it “SLAMD”, which is a play on “slapd”, which kind of stands for “standalone LDAP daemon” and is used in the process names of some directory server products.

So I built it, and I think that it’s fair to say that it turned into something substantially more impressive than either of us originally imagined. It had a Java-based API that you could use to define the types of workloads that you wanted to process, a web-based interface that you could use to schedule jobs and view the results (numerically and graphically), and a client that you could install on the systems that you wanted to used to drive load against the server. Over time, I added new types of jobs and lots of other features, like self-optimizing jobs (which repeatedly run the same job with different amounts of client load to find the optimal performance), job groups (which let you schedule several jobs to run in succession), and resource monitoring (which lets you monitor system statistics like CPU, disk, and network utilization on various systems).

SLAMD was pretty good at what it did, and it worked with all types of LDAP-compliant directory servers, so it became one of the preeminent directory server benchmarking tools. We convinced Sun to open source it, and lots of people started using it. It could be used for things other than directory servers, too (I did build some basic support for other protocols like HTTP, POP3, IMAP, and SMTP, and the ability to interact with relational databases), but LDAP performance and stress testing was always its big wheelhouse.

Fast forward several years, and SLAMD was still pretty great but was starting to show its age, at least under the covers. I started working on it in the Java 1.3 days, before nice features like generics, foreach, concurrency APIs, sub-millisecond timing, and so much more. The web interface was all hand-crafted HTML and mostly contained in one giant source file, and it was getting pretty unwieldy. I did make some attempts to try to modernize it, but I got really busy with other things, like creating OpenDS as a replacement for Sun’s stagnating C-based directory server, then moving on from Sun to launch UnboundID and working furiously to build up its directory server, directory proxy server, and LDAP SDK products.

Shortly after Sun and I parted ways, Oracle bought Sun and gradually started killing off most of the good things about it. This included shutting down the java.net site, which had been the open source repository for SLAMD, and I decided to take that opportunity to just let it kind of fade away. I figured it might be better to start something new from scratch, with a much more modern foundation, than to try to give SLAMD the kind of makeover I thought it needed. Of course, that was nearly a decade ago, and while I’ve done a lot since then, creating a new directory server benchmarking tool (other than a handful of command-line tools like searchrate and modrate that we ship with the LDAP SDK) hasn’t really been in the cards. Meanwhile, SLAMD is still getting a surprising amount of use. Even though it’s not so easy to get your hands on it anymore, people were still getting their hands on it and using it.

After having the topic come up several times in the last few weeks, I finally bit the bullet and dusted off the old code. I spent a couple of weekends doing some pretty extensive code cleanup. I fully generified everything, so there aren’t any more build warnings about raw types. I pulled in much more modern versions of dependencies like Apache Tomcat, the Berkeley DB Java Edition, and the UnboundID LDAP SDK for Java. I reorganized some of the jobs, including putting some legacy stuff out to pasture, and I wrote new versions of several of them. I split up some of the admin interface code into separate source files to make it more manageable, and I made some minor user interface enhancements.

So anyway, I went ahead and put the updated code on GitHub at https://github.com/dirmgr/slamd. Since no single entity owns the copyright on the code, it’s not possible to change the license, and it will therefore always will be licensed under the terms of the Sun Public License version 1.0. I’m not promising that I’ll add any major new features, but it’ll at least be more readily available than it has been, and with some more modern guts.

For now, if you want to use it, you’ll need to check it out and build it for yourself (there’s a README that tells you how to do that). Just know that it’s not backward-compatible with the version that I last touched in 2010, so don’t try to upgrade an existing instance (but if you do want the code for that old version, just check out revision 5777f3e5d78ff03985af4e68670e649127339c59, since I used it to seed the new repository).

Also note that there’s still a lot more work to do. There’s quite a bit more code cleanup that’s still on my to-do list (it builds cleanly with Java 8, but there are several deprecation warnings with Java 11). I plan on rewriting some more of the jobs (including making some potentially-incompatible changes). I know that some of the resource monitoring is broken (at least on Linux, which isn’t so concerned about maintaining consistent output in some of its commands). I haven’t touched any of the documentation. I’ve only done a very minimal amount of testing so far. So while it’s fine to play around with what’s there now, and please report issues if you find them, just know that I reserve the right to make even more non-backward-compatible changes as I continue to modernize the code.

The Story of UnboundID

/ Neil Wilson

unboundid-logo-blue-background-white-unbound-orange-id-600x139

At the end of 2016, I ceased to be an employee of UnboundID and became an employee of Ping Identity. Ping acquired UnboundID in early August of 2016, but most UnboundID staff remained employees of that company through the end of the year (mostly for bookkeeping convenience). As a co-founder, I was with UnboundID from the beginning. Here is our story.

Once upon a time, Sun Microsystems had the best LDAP directory server on the market. The 5.1 and 5.2 releases of the Directory Server Enterprise Edition (DSEE) product, which introduced multi-master replication and 64-bit support, respectively, were very exciting.

But then for some reason, Sun’s directory server engineering team seemed to contract a chronic case of inertia, in which everyone was content to rest on their laurels. It certainly wasn’t because the product had reached perfection, nor from a lack of ideas for improvement. After a while, I became frustrated with the lack of progress, and I wrote a hundred-plus-page document (which became known as the “directory manifesto”) that was full of things that we could do to improve the product. It had plenty of low-hanging fruit that would address pain points that customers were experiencing and lots of bigger ideas to help ensure our continued dominance of the high-end directory server market. But to no avail.

Eventually, I was able to nag the right people long enough to let me start working on a new directory server that was intended to be a DSEE replacement. And thus I became the first developer working on what would become OpenDS. Sun was committed to open sourcing all their software, from Solaris to Java to all of their enterprise software, and before long, we released OpenDS under the same CDDL license that they initially created for OpenSolaris.

At that time, Sun’s Directory Server engineering team was mostly split between Austin, Texas and Grenoble, France. In 2007, someone got the idea that it would be good to have it headquartered all in one place, and Grenoble was chosen as that one place. As a result, the five U.S. employees most closely connected with OpenDS were laid off: director of engineering Steve Shoaff, marketing lead Don Bowen, engineering manager David Ely, open source community manager Trey Drake, and myself as architect.

It didn’t take the five of us long to decide that we wanted to create a new company developing on top of the OpenDS codebase. We were still passionate about the product and excited about the opportunities that it could afford, and we planned to contribute back to the open source community. We even followed through on that with a handful of commits within a couple of weeks of the layoff. But then some nastiness arose between us and Sun’s management that I don’t want to get into here, and it became clear that we were no longer welcome participants.

So by the time we founded UnboundID (on December 17, 2007), we were to be competing against Sun. We’d comply with the terms of the open source license when altering existing code, but newly-created files would be our own private intellectual property, as allowed by the CDDL. We would, of course, also go head-to-head with other directory server vendors, like Oracle (who bought Sun within a few months of our departure), IBM, Microsoft, and CA. And we’d be up against open source offerings like OpenLDAP and the 389 Directory Server. We should’ve had no chance. And yet we were shockingly optimistic. We had a lot of ideas, a lot of drive, and dare I say a pretty good amount of talent.

Upon officially forming the company, David, Trey, and I worked furiously to make improvements to the codebase. We made dramatic improvements in performance, concurrency, and scalability. We added killer features, like support for transactions, filtered logging, change subscriptions, data transformations, data integrity checksums, and some new controls and extended operations. We made the server easier to manage by improving the out-of-the-box configuration, refined the command-line and interactive text-based interfaces, and added a web-based administration console.

We also created a new Java-based API for interacting with LDAP servers, because the existing options sucked. Before we released the UnboundID LDAP SDK for Java, you could basically choose between the horrible, confusing clunkiness of JNDI (where LDAP support is bolted on as an afterthought), or the buggy and no-longer-maintained Netscape Directory API. We wanted to create an API that made it easy to write applications that could take full advantage of any LDAP server, including the enhanced functionality we were building into our own software.

While David, Trey, and I were churning out the code, Steve and Don were working to sell it. And they did. Amazingly, we got our first customer within a matter of months: a large network equipment provider that supplied telephone companies with the equipment used to run their data centers. They loved our software, our enthusiasm, our ability to react quickly, and our willingness to put our source code in escrow so they wouldn’t be screwed if we went out of business. And before long, a number of big telcos were kicking the tires on our stuff and salivating at the idea of a modern, high-powered, feature-rich, and administrator-friendly directory service.

This first customer was a huge win for us. They resold our software and provided first-line support, which helped alleviate any downstream concerns about our viability. They declared our software to be carrier-grade, which served as validation to other potential customers in other industries. And it was also nice to be able to start getting a paycheck.

Plus, we were able to leverage this deal to get good terms on an initial round of funding from an investor. We were able to hire more people (many of whom were former colleagues from Sun who were all too happy to jump ship from their new Oracle overlords), and we started working on new products. David took on the Synchronization Server, and I started on the Directory Proxy Server.

The Synchronization Server provides a way to mirror the contents of two or more data repositories, so that changes made in one system appear in the other systems, usually in a matter of milliseconds. It can do one-way or bidirectional synchronization. It can synchronize all the data or just a configurable subset. And you can connect to a number of different types of repositories, including LDAP directory servers (both UnboundID and non-UnboundID), relational databases, NoSQL databases, and more (plus an API for developing your own support for additional types of data stores). It’s ideal for migrating data from your existing repository into the UnboundID Directory Server, and for keeping both infrastructures in sync for whatever length of time is necessary to complete the migration, or indefinitely if you want to keep both systems up and running.

As its name implies, the Directory Proxy Server is an LDAP proxy server. In the simplest deployments, it allows you to achieve better performance and higher availability through load balancing and advanced health checking techniques. In larger deployments, you can use entry balancing to transparently split data up into multiple sets (much like database sharding) for even greater scalability. It can also transform requests and responses as they pass between clients and backend directory servers, and you can do this on a per-application basis in case some clients have different expectations for how the data should look or how the server should behave.

We continued to grow. We sold more software. We gained more customers. We hired more employees. We wrote more code. But sadly, we also suffered some losses. Trey decided it was time for him to move on, so he left the company. But even more tragically, Don Bowen passed away in late 2009. He’d been diagnosed with brain cancer a mere three days after we founded the company, and somehow he continued to make incredible contributions toward our success for over a year and a half. I met Don in my first job out of college, at Caterpillar, where he introduced me to the world of LDAP. Within a few months, he’d gotten an offer to join the Baltimore-based startup (initially called B2B Communications, but later renamed TidePoint Corporation) and he took me with him. When that went south, we went our separate ways, only to meet up again when we both joined Sun at about the same time. We were friends as well as colleagues, and a tremendous amount of who I am today is because of Don.

And the work goes on. As more customers migrated from existing environments and deployed into new environments, we realized that we needed to provide public interfaces to allow our software to be customized, so we created the Server SDK. Our server products had always been very extensible and componentized, but before the Server SDK, we were the only ones who could take advantage of that. The Server SDK made it possible for customers to write their own extensions to customize the behavior of the server, from intercepting and altering operation requests and responses, to creating new loggers, password validators, extended operations, SASL mechanisms, sync sources and destinations, proxy transformations, and more. The initial intention was to only make the Server SDK available to customers who’d gone through at least some kind of training (it allows you to run custom code inside the server, so there’s a chance that a buggy extension could make the server unusable), and I really wish we’d stuck to that more than we did. But for the most part, it was a hit with customers, and an even bigger hit with the sales engineers helping them evaluate and then migrate to our software.

We also introduced a couple of additional server products: the Analytics Engine and the Data Broker. The Analytics Engine (formerly called the Metrics Engine) provides simple, graphical access to all kinds of historical information about the operation of the server broken down in all kinds of ways (e.g., the number of requests per second of each operation type, and a breakdown of their result codes and processing times), along with metrics from the underlying system like CPU, disk, and network utilization. The Data Broker provides several REST-based interfaces to interact with the environment, including support for OAuth 2 and OpenID Connect (for authentication, authorization, and federation), and SCIM (for data access). As much as I love LDAP and will continue to tout its superiority over HTTP, the kids these days are all about the web APIs, so we must oblige.

But even as we worked on new products, the Directory Server continued to grow and improve. I am particularly passionate about security, so that’s been a big focus of mine over the last several years. We added support for data encryption, several two-factor authentication mechanisms, more password storage schemes and password validators, sensitive attributes, retired passwords, password reset tokens, improved account lockout, signed logs, and more. But we also added a lot of non-security-related features, like JSON support, alarms and gauges, soft deletes, assured replication, and indexing improvements. It’s gotten faster, easier to use and administer, and just plain better.

On the whole, we’ve had an unbelievable run. We did have one down year with less-than-stellar sales, but for all other years, we reached or exceeded our goals. That continued through 2016, which was one of our best years ever (even if you ignore the whole “our company was bought” thing). We certainly weren’t floundering, and we weren’t really even looking to be acquired. We had partnered with Ping Identity on a number of deals in the past, as each company’s software complemented the other’s very well without too much overlap. And then Ping was acquired by Vista Equity Partners, who were looking for other opportunities to get into identity management, and then people started talking and it all sort of became a three-way deal, with the UnboundID acquisition by Ping Identity following Ping’s own acquisition within a couple of weeks.

So what’s next for the bigger, better Ping Identity? I can’t get into any specifics, but we’ve got a lot of great things in the works. Most of the UnboundID staff, and I believe all of the technical staff, are continuing on into Ping. Some people are changing roles (many moving up, others moving laterally), but my job isn’t really changing all that much. I’m still writing code, and hope that continues far into the future. Some of our product names have changed (for example, it’s now the Ping Identity Directory Server rather than the UnboundID Directory Server), and some haven’t (it’s still the UnboundID LDAP SDK for Java), but we’re still working to ensure that they remain the best products out there. So I need to get back to work.

P.S. I know that the only people I mentioned by name in this walk down memory lane are the UnboundID founders. I certainly don’t mean to imply that we’re the only ones responsible for or vital to the company’s success. There are so many other people that made big contributions to the company that I can’t list them all without fear of leaving someone out, without fear of mentioning someone who’d rather be left out, and without fear of making this long post even longer. So let me just say that if you worked for UnboundID in any capacity, you have my sincerest thanks.

A couple of announcements from UnboundID

/ Neil Wilson / 1 Comment

Today, UnboundID made a couple of announcements.

Sun DSEE to UnboundID Directory Server Migration

First, we’re announcing new pricing for our Directory Server for current customers of Sun’s DSEE product. If you’re currently using Sun DSEE, then you can migrate to the UnboundID Directory Server for less than what you’re paying now in maintenance costs.

In addition to saving money, you’ll also be getting:

  • Much better performance and scalability. I can’t post actual performance comparisons because of restrictions in the DSEE license agreement, but the UnboundID Directory Server leaves it in the dust.
  • Much smaller on-disk and in-memory footprint. We store data in a very compact form that takes much less space than DSEE requires (and in most cases, the fully-imported database including all the data and indexes requires notably less space than the LDIF file). You can fully cache a much larger data set on the same machine than DSEE, or you can use a smaller machine than DSEE requires for a given data set.
  • A much broader feature set. The UnboundID Directory Server includes virtually all features available in Sun DSEE, plus quite a bit more. Our server has lots of features like transactions, filtered logging, centralized logging, configuration auditing, multiple root users, automatic database priming, and incremental backup that DSEE just doesn’t have.
  • Significant improvements over DSEE in areas where both servers have overlapping features. This includes areas like server and environment management, monitoring and alerting, replication, password policy and security, standards compliance, and related tools.

Of course, we’re happy to work with anyone who wants to migrate from any other type of directory server to the UnboundID Directory Server. Contact sales@unboundid.com for more information.

UnboundID Synchronization Server

The UnboundID Synchronization Server provides the ability to synchronize data between directory servers or directory topologies. We currently support synchronization of data between any combination of UnboundID Directory Server and DSEE, with more data sources to be added in the future.

The Synchronization Server has a very powerful feature set allowing you to pick which entries to synchronize, which attributes to synchronize within those entries, and what kinds of transformations to apply (including renaming attributes, constructing or suppressing attribute values, transforming DNs, etc.). Synchronization occurs in real time, and can be configured in one or both directions (with the same or different data flowing in either direction).

The UnboundID Synchronization Server is free to use in the process of migrating your directory environment between Sun DSEE and the UnboundID Directory Server.

Why I like Solaris

/ Neil Wilson / 10 Comments

A recent InfoWorld article asks whether Solaris is going to be able to continue to compete with Linux.  Apparently someone from the Linux community pointed Paul Krill, the author, to my earlier blog article about the earlier issue that I wrote about with OpenDS and he asked me about it.  While I didn’t really want to bring that up again, I did tell him that I felt Solaris is a better OS than Linux.  I was briefly quoted in the article, but I would like to expand on that a bit.

First, let me say that I’ve got a pretty decent amount of experience with both Solaris and Linux.  I started using Linux in 1995 and Solaris around 1998.  Linux was my primary desktop operating system from probably around 1997 through about 2004, when I switched to Solaris.  I still run Linux quite a bit, including on my laptop out of necessity (primarily because Solaris doesn’t support the Broadcom Ethernet interface), but for the work that I do, which is primarily development, I find that Solaris is just more convenient.  On servers, there is no question that I prefer Solaris over Linux.

My fondness for Solaris definitely did not come easy, nor was it always warranted.  My first experiences were pretty unpleasant compared with Linux.  For many years, Solaris was anything but user friendly, providing only relatively antiquated shells and shipping without critical utilities like gzip or a C compiler.  CDE remained the default desktop environment for far too long, and it wasn’t easy to come by a lot of software that was commonly included with Linux distributions.  And of course, I’m sure that Sun’s decision to stop shipping Solaris for x86 systems for a period of time seriously hindered its usability and probably steered a number of potential customers away.

However, my opinion started to sway as Solaris 10 started to materialize.  Being a Sun employee, I had access to internal builds and I started to check them out.  Months before it was released to the public, I had changed my tune.  It was a serious leap forward in usability and convenience, and the new features were very compelling.  Since then, I’ve followed the development of Solaris 10, Nevada (the code name for what may become Solaris 11), and OpenSolaris.

It’s a lot easier to see why Linux may be more appealing than Solaris on the desktop.  Linux has better hardware support, so it will run on a lot more systems than Solaris.  Availability of applications can also be an issue, but I think in many cases that’s somewhat due to lack of publicity.  Many people haven’t heard about Nexenta (which is very much like Ubuntu Linux but with a Solaris kernel) or Blastwave (a site providing a large amount of Solaris software using a relatively simple interface that behaves like apt-get), and if you’re running Sun’s OpenSolaris distribution then you can use IPS to get access to a number of applications in a similar method.  But even so, there is just more software that works on Linux (or works more easily on Linux) than on Solaris.

However, there are a lot of reasons that I prefer Solaris and am willing to overlook some of its shortcomings.  I’ve listed some of them below.  Note that I’m not trying to slam Linux.  I do like it and hope that it continues to improve.  Consider this a wish list.

Overall Feel and Consistency

Linux feels like it was written.  Solaris feels like it was designed.  While I think that Sun’s development processes can sometimes be a little heavyweight, and I think that Sun is trying to retain too much control over OpenSolaris, there is a lot to be said for having processes in place to guide development.

Unfortunately, this isn’t something that Linux can simply adopt.  Linux isn’t one thing and there isn’t any one organization controlling any significant part of it.  As Solaris integrates an increasing amount of third-party software, it will likely also begin to suffer from this somewhat although perhaps to a lesser extent because there are still some things that can be made a bit more common.

Interface Stability

This is an area where Solaris shines and Linux is just plain abysmal.  If an application runs on Solaris version X, then there’s an excellent chance it will work on version X+1.  This is far from true on Linux.  I’ve had countless experiences where an application that worked on one release of Red Hat or Ubuntu didn’t work on the next.  In some of those cases, recompiling the application was sufficient to get it running, but there are still a lot of cases where that’s not enough, and sometimes the source isn’t available to even try.

Seamless 32-bit and 64-bit Integration

Solaris has had 64-bit support since Solaris 7, and I know that I was testing 64-bit Solaris on x86-64 systems before the Linux kernel had 64-bit support for those systems.  Of course, Linux has included support for x86-64 systems for a while now, but the bigger issue with 32-bit versus 64-bit support in Linux is that it’s generally one or the other.  If you install a 64-bit Linux distribution, then it’s generally entirely 64-bit and may not even include the ability to run 32-bit applications at all, at least not without installing additional packages.

This leads to silliness like 64-bit web browsers and complaints about lack of 64-bit plugins.  I don’t see much need for a browser to support over 4GB of memory, which is the primary benefit for 64-bit applications.  There are a few other cases where 64-bit support may be beneficial (e.g., having access to more CPU registers), but they are primarily beneficial for applications that need to do a lot of calculation or cryptography.

It’s true that Sun should have provided a 64-bit Java plugin, and plugin providers are equally to blame for the same infraction.  However, if 32-bit and 64-bit support had been better integrated then only applications which truly benefit from 64-bit support would really need to be provided as such.

Process Rights Management

In both Linux and Solaris, root is an all-powerful user that can do pretty much anything.  Historically, there are a lot of things that only root can do.  Both Linux and Solaris provide mechanisms for granting a specified set of normal users the ability to run certain commands with root authority, but in such cases those applications have full root access and it is necessary to trust that they will behave properly and don’t have any security holes that could lead to unintended consequences.

Solaris improves upon this with process rights management, also known as least privilege.  Rather than granting an application full root access, it is possible to grant only those portions of root access that the application needs.  For example, a web server may need to listen on port 80, but there is no need to let it read or write any file on the system, load kernel modules, or halt or reboot the system.  Instead of granting it full root access (even if it gives it up later via setuid), you can just grant it the net_privaddr privilege so that can listen on ports below 1024 without having access to any of the other things that might be possible if it were truly running as root.

ZFS

ZFS is a phenomenal filesystem + volume manager.  Linux has good filesystems and a volume manager, but none of them have the combination of power, flexibility, and ease of use that ZFS offers.  It works well on a fraction of a drive, and it works well on a system like the Sun Fire x4500 with 48 drives.  It’s got good performance.  It’s got great data integrity features through redundancy, checksums, and multiple copies of data and metadata.  It’s got instantaneous atomic snapshots.  It never needs to be defragmented.  It never needs fsck.  The implementation of cryptographic support for encrypted filesystems is going to be integrated in the near future.

ZFS is great for servers, but it’s also very useful on desktop systems.  As a developer, I find the instant snapshot and rollback capability to be extremely useful for resetting an application to a known clean state when running tests.  Compression increases the amount of data that you can store and in many cases actualliy helps performance.  The ability to keep multiple copies can help prevent you from losing information even if you’ve only got a single disk.

Zones

Virtualization is a pretty hot topic right now, and there isn’t really a shortage of options.  VMware, VirtualBox, Xen, QEMU, and other solutions make it easy to run one operating system on top of another, or another operating system on top of itself.  They provide excellent separation of environments, but they are all pretty heavyweight because they require a significant amount of memory and disk space for the virtualized environment.

In some cases, e.g., for security or application testing, you want something that looks like a separate system but you don’t need a different operating system.  For those cases,  Solaris offers an excellent alternative in the form of zones.  A zone provides excellent separation and for the most part looks like a completely separate system, but requires virtually no memory and in many cases a relatively small amount of disk space because it’s able to share a large amount of the filesystem from the host system (also called the global zone).  Zones are trivial to configure, and when used in conjunction with ZFS snapshotting and cloning it can be possible to create new zones nearly instantly.

In the default case, a Solaris Zone runs exactly the same OS and kernel version as the host system, which makes it extremely cheap.  However, because of the strong compatibility that Solaris provides between versions, alternative implementations have been added which make it possible to create zones which appear to be running Solaris 8 or 9 on Solaris 10.

It’s also possible to run a Linux emulation layer in a zone, which is pretty impressive but unfortunately the support is pretty limited and it doesn’t really look like there’s much active development going on to improve it.  However, while I did make use of this capability when it was first released for a few specialty purposes (e.g., running Acrobat Reader), I haven’t used it in quite a while because I can run pretty much everything I need to natively in Solaris, and I do have Ubuntu installed in VirtualBox which provides a much better way to run Linux on Solaris.

Observability

Both Linux and Solaris provide pretty good ways of identifying what’s happening on the system, but Solaris tends to do a better job.  At a very high level, I’ve noticed that most Linux systems don’t ship with utilities like iostat and mpstat and require the installation of additional packages, whereas they are included by default on Solaris.

When looking into what a particular process is doing, both the Solaris truss and Linux strace are pretty similar, and they allow you to see the system calls that the process is making.  It does appear that at leasat some Linux distributions (like Ubuntu) don’t seem to provide pstack or an equivalent command for getting a stack trace of all threads in the process.

Of course, the big feature that Solaris has in this arena is DTrace.  Linux does have SystemTap, but it appears to be sat the present time it’s limited to only examining the kernel and doesn’t have any support for tracing user-space applications.  DTrace provides full support for debugging user-space applications, and there is also special support for tracing Java, Ruby, Perl, shell scripts, and other scripting languages.  And because Solaris provides stable interfaces, there’s a much better guarantee that DTrace scripts which use stable interfaces will continue to work in the future.

Sleeping with Solaris

/ Neil Wilson / 1 Comment

I recently came across an interesting issue on Solaris.  I was trying to do something pretty simple:

  1. Check to see if there’s work to do.
  2. If there is work to do, then do it and go back to step 1.
  3. Sleep for a short length of time and go back to step 1.

This worked great on Linux, but not so well on Solaris.  Things were getting done in fractions of a millisecond on Linux, but it was taking over ten milliseconds on Solaris.  I hypothesized that the sleep might be the problem, and a quick test confirmed it.  It turns out that on Solaris in the default configuration you can’t sleep for less than ten milliseconds at a time (at least in Java).  This is because on Solaris, the Java sleep method is dependent on system clock ticks, and the default configuration uses 100 ticks per second, or one tick every ten milliseconds.  Any attempt to sleep for less than that time will get rounded up to ten milliseconds, and attempts to sleep for a length of time greater than that will be rounded up to the nearest multiple of ten milliseconds.

You can change this behavior by adding the following to the /etc/system file and rebooting the system:

set hires_tick=1

This will change the default from 100 ticks per second to 1000 (which means you can sleep in increments of one millisecond instead of ten).  There is also a way to fine-tune how many ticks per second you want (via hires_hz), but changing that isn’t supported and every mention of it I can find indicates that you don’t want to increase the frequency anyway because it will probably degrade performance as it will become expensive to update the tick counter too often.

There are, of course, other ways to accomplish the task I had originally set out to do (e.g., blocking queues, semaphores, wait/notify, etc.), but the bigger takeaway here is that you shouldn’t depend on algorithms involving fine-grained sleeping, especially if your application needs to run on Solaris.

Why the Dislike for CDDL?

/ Neil Wilson / 4 Comments

Disclaimer: Although I was employed by Sun at the time the CDDL was created and chosen as the license for the OpenSolaris code base, I was not involved in either of those processes and was not privy to any of the related discussions. I am not making any attempt to speak for Sun, and all information provided in this post is either based on publicly-available information or my own personal opinion.

In reading discussions about what has happened with OpenDS, I’ve seen a wide range of reactions. This is to be expected, but one thing that I have found to be a bit surprising is that there have been some comments that are critical of the Common Development and Distribution License (CDDL). This isn’t the first time that such comments have been made, as I’ve heard them ever since the license was first created, but I am a little puzzled by them and the fact that they have persisted for so long. I think that the CDDL is an excellent open source license and that many of the negative comments stem from not really understanding it, while others may have something to do with the fact that the open source community in general has been and continues to be somewhat suspicious of Sun (who authored the CDDL).

The CDDL was originally created as a potential license for OpenSolaris. This drew a lot of criticism because many people, especially those in the Linux community, wanted Sun to use the GNU General Public License (GPL). Since GPLv3 was nowhere near complete at the time, if Sun did choose GPL then it would have to be GPLv2 but that would have been completely impossible for Sun to do in a reasonable way. While Sun certainly owns copyright on most of the code in Solaris, there are parts of the code that Sun licenses from third parties. Since GPLv2 doesn’t play well with non-GPLv2 code, if Sun had chosen to use GPLv2 for OpenSolaris, then they wouldn’t have been able to include some of those third-party components (especially those that interact directly with the kernel) which would have made it a lot less attractive for potential users. In that case, about the only people that would have been happy would be those in the Linux community because they would have been able to take the best parts of Solaris and pull them into Linux. OpenSolaris itself wouldn’t have been really useful until they had either re-written the third-party components or convinced their respective copyright owners to make them available under GPLv2. Other operating systems which use non-GPL licenses (like the BSD-based variants, which have gotten a lot of benefit from the OpenSolaris code) wouldn’t have been able to use it, and third-party vendors (especially those that need kernel-level interaction, like hardware device drivers) would have also found it much less attractive. It is possible that some of these concerns could have been addressed by creating GPL exceptions, much like they have done with Java, but even still there would have been significant deficiencies that GPLv2 doesn’t address like legal concerns about code which is covered by patents. Rather than try to pigeonhole OpenSolaris into GPLv2, Sun chose to look at other options, including the possibility of using their own license, which ultimately led to the creation of the CDDL.

Before I go any further, let me briefly describe the primary types of licenses that exist in the open source world. They fall into three basic categories:

  • Licenses which preserve open source at all costs, like the GPLv2. These licenses require that any software that uses code under such a license must always be open source. In other words, you can’t use code licensed in this manner in an application with closed-source components. This is very good for the community that releases the code under this license, since it ensures that they will always have access to any improvements made to it, but it’s less friendly to downstream developers since it creates significant restrictions on how they might be able to use that code.
  • Licenses which preserve freedom at all costs, like the BSD and Apache licenses. These licenses place very few restrictions on how other developers can use the code, and it’s entirely possible for someone to take code under such a license and make changes to it without making those changes available to anyone else, even the original developers.
  • Licenses which attempt to strike a balance between open source and freedom, like the Mozilla Public License, the CDDL, and GPLv3. These licenses generally require that any changes to the existing code be made available under the terms of the original license, but any completely new code that is created can be under a different license, including one that is closed source.

As someone who has done a significant amount of both open source and closed source development, I really like licenses in this third category. If I make code that I have written available under an open source license, then I like the guarantee that this code will remain open. On the other hand, I also like giving others the freedom to do what they want with their own code, even if some of their code happens to interact with some of my code, and I know that commercial users are much more likely to shy away from licenses in the “open source at all costs” camp than licenses in the other two categories.

So what are the specifics of the CDDL? It’s based on the Mozilla Public License, but clarifies some things that the MPL doesn’t cover. The basic principles of the CDDL are as follows:

  • CDDL has been approved by OSI as an official open source license, which means that it meets all of the minimum requirements defined at http://www.opensource.org/docs/osd.
  • CDDL is a file-based license. This means that if you make any changes to CDDL-licensed software, any existing files that you modify need to remain under CDDL, but any new files that you create can be under whatever license you want as long as that license isn’t incompatible with CDDL.
  • Similar to the above point, CDDL is very friendly when interacting with code under other licenses. This makes it easy to mix CDDL-licensed code with libraries under other licenses, or to use CDDL-licensed libraries in a project under a different license.
  • CDDL includes an explicit patent grant clause, which means that if any of the code is covered by patents then anyone using or extending that code is also granted the right to use those patents. It also includes a clause that terminates the usage rights of anyone who brings patent-based litigation against the code.
  • CDDL isn’t a Sun-specific license, and is suitable for software written by anyone. The only mention of Sun in the license is to indicate that Sun is the license steward and the only entity able to create new versions of the license

See http://www.sun.com/cddl/ and http://www.opensolaris.org/os/about/faq/licensing_faq/ for further information about CDDL license terms.

In my opinion, the CDDL is a very attractive license for open source software. It certainly doesn’t seem evil or unfair in any way, so I have a hard time understanding the bad reputation that it seems to have gotten. It is true that CDDL code can’t be mixed with GPLv2 code, but that’s not because CDDL is incompatible with GPLv2, but rather because GPLv2 is incompatible with CDDL. GPLv2 is incompatible with lots of other licenses, including other popular open source licenses like the Apache License, the BSD license, and the Mozilla Public License. In fact, the GPLv2 is even incompatible with the GPLv3 (as per http://www.gnu.org/philosophy/license-list.html#GNUGPL). It is unfortunate that the licenses used by OpenSolaris and Linux aren’t compatible with one another, but I think that it would have been a mistake to use GPLv2 for OpenSolaris and highly doubt that incompatibility with Linux was seen as a key benefit when CDDL was selected for OpenSolaris.

Our decision to use CDDL for OpenDS was made after careful consideration and was based on the merits of the license. We were certainly not pressured into using it by Sun, and in fact during discussions with Sun’s open source office they wanted to make sure that we weren’t choosing it just because we thought it was the company line but rather because it was the right license for the project. There are a number of other open source licenses out there, and they have their benefits as well, but if I were to be involved with the creation of a new open source software project, then I would imagine that CDDL would at least be in the running during the license selection process.

Clarifications on the Open Letter

/ Neil Wilson / 10 Comments

It appears that there are some questions about the content in the open letter that I posted earlier this week. Simon Phipps (Sun’s chief open source officer) posted a comment on my blog that summarizes these questions, so I will use this post to reply to it. The original text from Simon’s post will be indented and italicized, and my responses will be in plain text.

Hi Neil,

Despite the fact you didn’t actually contact the Sun ombudsman service[1], I have had several referrals of your postings. I’ve done a little investigation and I have some questions about your story.

Actually, I did contact the Sun ombudsman service. The exact same text that was included in my blog post was also sent as an e-mail message. That message was sent from neil.a.wilson[at]directorymanager.org with a timestamp of “Wed, 28 Nov 2007 09:57:03 -0600” (9:57 AM US Central Time), and was addressed to neil.a.wilson[at]directorymanager.org. It was blind copied to the following recipients:

  • users[at]opends.dev.java.net
  • dev[at]opends.dev.java.net
  • jis[at]sun.com
  • ombudsman[at]sun.com

I did not receive any bounce messages in reply, and my mail server logs confirm that Sun’s mail server did in fact accept the message for delivery. If my message never made it into the ombudsman[at]sun.com inbox, then perhaps the problem is on your end (e.g., over-eager spam filtering, which happened to me on more than one occasion when I was a Sun employee).

It’s very regrettable that you were laid off, no question. That’s not a part of your narrative I can comment on for HR/legal reasons, but it’s always sad when business pressures force layoffs.

Thank you for the sentiment. While I wasn’t particularly happy about being laid off, I don’t hold a grudge against Sun because of it. Regardless of whether I think it was an intelligent move, Sun did have a justification for it (geographic consolidation). If the only thing that had happened was that I got laid off, then I fully expect that I would still be actively participating in the project. I believe I demonstrated that through my continued involvement in the project even after having received my layoff notification.

However, I do question how you characterize the requests to change the OpenDS governance. I note that the OpenDS governance was changed on April 28 by sshoaff[2] and that the original line reading:

“This Project Lead, who is appointed by Sun Microsystems, is responsible for managing the entire project”

was replaced by one reading

“This Project Lead, who is appointed and removed by a majority vote of the Project Owners, is responsible for managing the entire project”

I have not been able to find a discussion of this change anywhere, and I understand from your former managers that they were unaware of this change. While you characterize the request made of you as:

“demanded that the owners approve a governance change that would grant Sun full control of the OpenDS project”

it seems to me that what in fact happened was you were (collectively) asked to revert that change to its original state. On present data, it appears to me that far from Sun acting in bad faith over the governance, they were in fact making a reasonable request to correct an earlier error. Indeed, all that has happened to the governance document since then is to revert the change[3].

This is not the whole story.

First, the change to which you refer (committed in revision 1739 by Stephen Shoaff on April 28, 2007) was absolutely not unauthorized. Stephen Shoaff and Don Bowen both served as officers of the company (Stephen as the director of engineering for directory products, and Don as a director of product marketing for all identity products), and David Ely was the engineering manager and the Sun-appointed project lead for OpenDS under the original governance. This change was also discussed with Sun’s open source office, and while you (Simon) may not have been directly involved with those discussions, Don Bowen has informed me that there was a telephone conversation in which you told him that each project should make the decisions that are best for its respective community. We also involved the OpenDS and Identity Management communities in the process, although those conversations were on a personal basis with key members rather than at large on the public mailing lists. Unfortunately, none of us can currently produce any evidence to support these claims. When we received the layoff notification we were required to return or destroy any Sun property that we may have had, and since all of these discussions would be considered Sun-internal communication we no longer have access to any record of them in compliance with the notification requirement. However, full documentation to support all of these claims should exist within Sun should you feel the need to verify them.

Second, this was not the governance change to which I referred in my original post. In the meeting that the owners (including Ludovic) had on November 13, 2007, we were informed that it was Sun’s intention to replace the governance with something different and that the new governance would be chosen and managed by a Sun-selected committee. This change has not yet been applied, and as I am no longer involved with the project I cannot comment on whether there is still intent to make it. However, Eduardo referenced this future change on the OpenDS user mailing list today (https://opends.dev.java.net/servlets/ReadMsg?list=users&msgNo=627) when he said “We want to improve these governances, ideally in a consistent way.”

There was no discussion at all during the November 13 meeting of the change made in revision 1739, and it was not brought to our attention until the following evening. To the best of my knowledge the request to revert the change made in revision 1739 was never discussed with anyone other than Stephen Shoaff. I know that I personally never received any communication from anyone within Sun asking me to approve reverting this change.

Finally, I would ask Sun to justify their subsequent reversion of that change and how they believe that it was in the best interests of OpenDS, or how doing so was consistent with Sun’s public stance on the importance and value of community-led projects. Despite the fact that the change we made had more than sufficient authorization, I fail to see how reverting it is in any way an improvement. How is reverting to a Sun-appointed absolute authority better for the community than the consensus-driven model we thought Sun wanted?

I would be happy to continue to investigate this case, so if you would like to submit a complaint to ombudsman@sun.com with full data supporting your accusations I would be pleased to investigate further. I’m afraid I don’t usually read your blog so you’ll need to alert me (webmink@sun.com) to any postings here that need my attention.

Regards

Simon

[1] http://blogs.sun.com/webmink/entry/open_source_ombudsman
[2] http://tinyurl.com/ys5hf3
[3] http://tinyurl.com/yto9qs

I am afraid that there may not be any benefit to further investigation. It appears that you are using your position to attack my credibility and focus on damage control for Sun rather than acting impartially on my behalf as per your claim at http://blogs.sun.com/webmink/entry/open_source_ombudsman. Even if for some reason you did not receive the message that I originally sent to ombudsman[at]sun.com, I find it very discouraging and disappointing that Sun’s community advocate would choose to respond in such an inflammatory manner via e-mail messages and blog comments without even making an attempt to contact me for further clarification. You have accused me of launching an attack with partial facts but apparently have made no attempt to contact me to get the complete facts for yourself. I had gone out of my way to indicate that I felt that this was an isolated incident and not in-line with Sun’s true stance on open source, but it’s hard to continue to hold that position when Sun’s ombudsman and chief open source officer behaves in such a manner.

An Open Letter to the OpenDS Community and to Sun Microsystems

/ Neil Wilson / 10 Comments

My name is Neil Wilson, and until recently I held the Owner and Committer roles in the open source OpenDS project. I helped found OpenDS, served as the project architect, and have contributed more code than anyone else. However, I must now regrettably inform you that I have been compelled to end all involvement with OpenDS. I have resigned all roles that I held in the project and have rescinded my Sun Contributor Agreement. I will no longer contribute code, documentation, bug reports, suggestions for improvement, or advice of any kind.

I joined Sun Microsystems in October of 2001, where I was directly involved with its proprietary directory products in addition to my later work with OpenDS. I wrote and analyzed code to provide new features, fix bugs, and improve performance, and I developed a number of tools to help improve the Directory Server experience. I had excellent working relationships with a number of customers, and I was instrumental in closing several deals worth many millions of dollars. I consistently received the top rating in annual performance reviews, and I worked with a number of other groups within Sun, as well as with Sun partners, to help ensure that the Directory Server products worked as well as possible with other Sun technologies, including Solaris, Java, and a number of other software products, as well as many different kinds of hardware.

On September 27, 2007, I was notified that Directory Server engineering, including OpenDS, was being consolidated in Grenoble, France, and that US-based positions were being eliminated. Some individuals were reassigned to work on other software products, but among those laid off were the four OpenDS project owners (myself, Stephen Shoaff, Don Bowen, and David Ely), as well as the OpenDS community manager (Trey Drake). We would technically remain Sun employees for the next two months, but were not able to access any Sun-internal resources and were not required to work in any way and were encouraged to use that time to seek employment elsewhere.

This was certainly a very surprising move, but the shock wore off and within a few days the OpenDS owners and community manager got together and decided that even if we were no longer working for Sun that we would like to continue our involvement with OpenDS and wished to ensure that the project was in the best possible position moving forward. To that end, we had face-to-face meetings, conference calls, and e-mail discussions with Sun employees still involved in the project to provide advice and knowledge transfers. I also continued participation on the project mailing lists, committed code changes, and updated the project issue tracker and documentation wiki.

The project owners also decided that as an act of good faith (and without any prompting from Sun) that we should elect a fifth owner who was a Sun employee, since Sun had certainly made a significant contribution to the project. We appointed Ludovic Poitou to this position, as he had served as the architect for Sun’s proprietary Directory Server product for several years, and further suggested that we should amend the project governance to ensure that Sun Microsystems was granted a permanent seat in the project ownership. On November 13, 2007, the OpenDS project owners (including Ludovic) met via conference call with the intention of discussing this governance change. However, during that meeting Ludovic informed us that Sun’s intention was to change the OpenDS governance policy so that the project was controlled entirely by a Sun-selected committee. This was a surprise to us, and we indicated that while we were willing to discuss this further to better understand what was involved, we were concerned that this was not necessarily in the best interests of the OpenDS project or its associated open source community. We noted that the current OpenDS governance policy stated that governance changes could only be made by a consensus of the project owners, and therefore we would be required to approve any potential change.

On November 14, 2007, a member of executive management within Sun’s software division contacted one of the recently-laid-off OpenDS project owners and demanded that the owners approve a governance change that would grant Sun full control of the OpenDS project. During this call, we were threatened that if we did not make this change we could face immediate termination and loss of all severance benefits. The four former-Sun owners discussed this and decided that we could not in good conscience approve the requested change as we did not believe that it would be in the best interests of the project, but we were also not willing to risk the considerable financial loss that could result if Sun decided to make good on that threat. After first trying to resolve the issue through more amicable avenues, we were ultimately compelled to resign our ownership and end our association with the project on November 19, 2007.

This was a very disappointing and hurtful turn of events. I believe that we acted only in good faith and in the best interests of the community, and we had clearly taken action to protect Sun’s position in the project even after our own jobs had been eliminated. OpenDS was founded as a community-focused “doacracy”, and no one has done more than I have to help ensure its success, or to ensure Sun’s success through OpenDS. However, Sun management has shown that at least in this case they are willing to resort to rather hostile tactics to preserve absolute control. This is most certainly not in the spirit of open source and open development that we tried to foster or that Sun claims to embody.

Please note that I don’t feel that this action was representative of Sun’s true open source strategy, but was a relatively isolated incident brought on by middle management acting of their own accord. I believe and certainly hope that the public statements made by individuals like CEO Jonathan Schwartz and Chief Open Source Officer Simon Phipps are honest and that Sun truly does want to be a genuine community-focused open source company, and I have no reason to believe that they were aware of or involved with any of what happened with OpenDS. Similarly, I sympathize with the remaining Sun-employed OpenDS engineers who may have been unwittingly drawn into this turmoil, and am disappointed that we will no longer be able to work together, but it was not my choice. Unfortunately, if Sun is unable to ensure that their middle management is on the same page as the senior management setting the open source strategy and the engineers making it happen, then it won’t take too many more incidents like this (or the Project Indiana / OpenSolaris Developer Preview naming fiasco) for people to start to question Sun’s true intentions.

In order to avoid potential retaliation from Sun, I have remained silent on this matter through the duration of the two-month period following the layoff notification during which I was still technically a Sun employee. Now that this time has elapsed, I am no longer at risk of losing severance benefits and I believe that it is important to clear the air. I have no desire to pursue this matter any further through legal or other channels, but simply wish to explain why I am no longer able to be involved with the OpenDS project.

I am passionate about the technology and hope to continue working in this area in the future, but I am not yet prepared to discuss where I’m going from here. You may watch my new blog at / for more information in the future.

Neil Wilson
neil.a.wilson[at]directorymanager.org