Tristan Juricek

… in PDX
July 2016!

I'm a software engineer, these are some things I've done.

(go home)

Helix Web Services

Helix Web Service, which I'll just call HWS, is a "middleware" web API platform for Helix Services. That being said, it's really a way of communicating with p4d, the main Perforce server, over HTTP. It's been several things over it's lifetime, but it's started out as a HTTP front-end for p4d, and that's probably where it'll end up for now.

HWS source is actually available over on the Perforce workshop. If you're not familiar with Perforce tools, well, it's probably easier to just unzip the project source.

HWS has actually been many things over it's lifetime:

  • A simple HTTP front-end for p4d, written in Sinatra, available as Gem for Linux.
  • A suite of web applications for managing a cluster of p4ds. This didn't go anywhere.
  • A broker that could be deployed in a SaaS solution or onsite.
  • A simple HTTP front-end for p4d, written Java 8, deployable to Linux, OS X, and Windows.

It's taken us about a year and a half to get where we are. And it's still not released, though I guess it will be sometime in 2016.

So, what can one do with HWS?

You can execute some, but not all, commands out of the Perforce command set without having to use one of the existing Perforce SDKs. Of course, at one point, we had used Swagger to generate 5 more APIs, but those APIs are actually kind of terrible to use, so it'll just be another Java SDK. So, in the end, we have a new API that can do a subset of the existing P4Java API.

I'm really trying to stay positive, and not burn bridges all over the place. However, when it comes to impact, HWS basically has very little to offer yet. Maybe that will change now that I'm leaving and the project is being given over to a team that has more bandwidth and the ear of upper management. We'll see.

How it all began

HWS began after a rather tumultuous period in the company history. You see, the first couple years at Perforce I worked on Commons, a web app aimed at simple front-office workflows. There were other web apps, one targeted for code review, and another for analytics. Many of these apps seemed to be complementary, but they essentially looked like they had been built by three different companies. They were completely different user experiences, and completely different product technologies. So one day, the teams of these different apps were all sat down in a room, and we were told, basically, we're halting work on everything. It wasn't our fault, but leadership was going to try to correct this. And Ruby on Rails is the future, where we'd have a bunch of Rails apps talking to a single consolidated "services layer".

Now, at this point, we hadn't exactly invested in the Ruby ecosystem. We did have an SDK, p4ruby, however, it didn't have a functional gem installation. You had to go to the site, download a couple of different .tgz files, and run some custom build steps to get our module working in your Ruby installation. You couldn't run gem install p4ruby and expect anything to work, at least it didn't work for any of the teams starting to try to get used to the Rails ecosystem.

P4ruby had been the effort of one of our technical managers. As I kind of got things limping along on with the gem installation, this manager offered up another vision for the future; I could work on the services layer that all our Rails apps should coordinate with each other.

Importantly, another major reset button got hit at this point; there was a massive restructuring, and basically a significant tier of management was let go. The manager I had been working with got promoted, to essentially run the company's engineering. Thus, I had hitched my horse to the right wagon.

In the meantime, I put together something as fast as possible to get a sense of how this services platform might work. It was really easy to expose a bunch of Perforce commands with a basic Sinatra app. But, it really wasn't interesting to the new Rails app efforts, because the commands I could expose just weren't the ones they needed. There's a lot of weaknesses to providing middleware to our APIs, and implementing the needed features were going to be very expensive. And I was the only developer on the project. And I was going to be the only developer. So, I had hitched my horse to a good wagon, it apparently was a very slow mover.

The first major redirection

To try to get the services platform integrated with… something… and out the door, the services project got moved over to another team, trying to create a synchronization platform for the company. You see, the original manager got promoted, so had a massive set of new responsibilities, and thus, wanted to see this services project continue, however, didn't have enough time to help foster it forward. So, this project was now going to start by extending it's basic set of commands with services needed by this synchronization tool.

This wasn't a terrible plan, but after several months, the sync project really hadn't achieved anything demonstrable. Part of the problem was that the product and technical managers kept attempting to boil the ocean. It was during this period that I was asked to start considering creating a series of web applications to basically implement cluster management. And it was still just going to be me doing everything, soup to nuts.

After some time, and a whole lot of little to show for it, the technical manager was let go, and the product manager left the company.

Next major redirection

The next manager who took over this sync project basically hit a reset button. Normally, you'd think this was a problem, but in this case, the manager was very technically savvy, and had a keen sense for what the term Minimum Viable Product actually means. He ruffled more than a few feathers, but after a couple of months, there was a sync tool starting to perform actual synchronization. And the back end I wrote actually had interesting services to help this happen.

We got to a point where we had this sync tool working with projects in a beta version of Helix Cloud, our nascent SaaS platform for smaller teams. This services platform (not yet called HWS) was actually deployed as part of the SaaS project. We could also deploy it outside, however, we lacked a concept of projects that were kind of central to the sync tool. So, it could work, it just wasn't very interesting for customers with existing repositories.

Let's rewrite it again!

Now at this point, it's been about a year since the web app teams got in a room and were told that the future was Ruby on Rails. So we got in a room and talked about a few issues that had come up. Notably, that these Rails apps tend to be kind of Linux centric, don't really work well on Windows, and hey, a ton of Perforce customers are game development and media firms that do very little with Linux.

After hashing out different possible directions, it was generally agreed that the JVM was the best platform; specifically, Java 8 with the Spark web application "framework". Given the nature of the projects, my project was the first candidate to hit the JVM. After all, there was only one of me writing code for this project, so there wasn't that much to rewrite.

Thus I converted the project, written in Ruby using Sinatra, to Java 8 using Spark. It was remarkably similar in approach, and, I was able to get things running on Windows and OS X quickly. Honestly, the hardest part was figuring out how to get Windows services integration running, then, I discovered install4j, and decided to give them money to solve that problem for me.

Now, during this conversion process, the company got bought by a private equity firm. During any kind of transition like this, change is bound to happen. In this case, the sync project that this services project supported (now finally called HWS), was put on hold. Also, the manager that had led the project left the company.

So, here I was, again under the umbrella of someone new. Amazingly, this person had more of an interest in seeing the project iterate towards something that was basically the first version of the project, only, this time it ran on 3 OSes easily. The last several months of this project has seen me add (and then remove) client SDKs, lots of documentation and testing variations, etc. Though no new major features that are noticeably different from the API created nearly 16 months earlier.

The same week we were told the company was to be bought out, I notified my manager that I was relocating to Portland, six months later, at the end of June. Originally my managers were optimistic that I would be kept on as a remote employee, and my work would continue on seeing this project to public release. It took them five months to decide that wasn't going to happen. Thus, the project is now being handed off to a team in a completely different part of the world, and I'm left just trying to support their interests the best I can.

Things I got right

First, by choosing a pet project of someone who is now at a fairly high level of the company, I retained my employment through a period where many others lost theirs. Being able to maintain employment until I and my partner were both ready to relocate can't be understated as a solid decision. Yes, it has nothing to do with the project per se, but it's significant to mention. The culture of an organization often creates projects of a particular type and quality, regardless of what your ideas really are. Sometimes it's best to go with the flow when there are other priorities on the line.

Second, I chose to implement utilizing a lot of Traits in Java 8. Even though I came across that article far after I did most of the work, it represents pretty much everything I did to implement the project quickly. Having used other frameworks, like Spring MVC, it's very nice to have the ability to compose features using interfaces. Implementation, without a doubt, was very fast (really the main implementation was done in a month). Time will tell if the team taking over maintains this design, though I'm optimistic.

What should have been done better

There has been so much churn in the leadership of the company it's hard to indicate if I should have pushed harder for a different direction or not. After all, I'm still employed. But in the end, a project that has had so many reincarnations really should be left behind until clear direction can be had. It's easy for me to say this now that I'm leaving, though. Without a doubt, had I drew a line in the sand and pushed for a direction, I'd probably be searching for employment while trying to pay for rent in the Bay Area for less than 6 months. Ugh.

On a more technical note, my main mistake was relying too heavily on fully integrated tests. You see, our APIs are rather undocumented when it comes to data, so I was reluctant to mock something I couldn't mock accurately. Ergo, unit tests were put on the back burner. I ended up creating a fairly complex set of integrated scenarios, that currently has about 25 different service configurations. Each configuration runs a set of functional tests.

It's been useful, but very tricky to debug, given that there is so many ways things can happen it can take some time to reproduce each environment. Developers need to be able to spin up 5 different OSes, and, when it comes to installers, issue commands with special user privileges.

Were I to do it over again, the best thing would be to rely heavily on unit tests for most regression testing, but with a way of capturing the commands being mocked, and run those in separate integration tests. I'm not 100% sure this is faster, but it does mean that developers should be able to experience all known variations of what we know, without having to spin up a bunch of different environments.

— June 11, 2016

(go home)