Java, Technology

Outsource the Scaffolding

Just today, I twittered a link to an insightful blog post, “Here’s the Deadliest Catch: Hiring an Agency to Build Your Startup”.  The title is largely self-explanatory, and I may blog my own riff on it at a later date.  Digg is often raised as a counter-example to outsourcing development of a startup, with version one developed for $200.  So how do you decide what the right balance is for you?

Ruby on Rails has this concept of “scaffolding“, which makes it trivial to get a simple web application running to perform create/read/update/delete operations on a simple business object.  So if you wanted to build a basic address book, Rails could basically create that application with a one line scaffold generate script.  It produces a simple web interface, and handles all of the database calls for you.  Java code to accomplish the same task is typically more verbose, but even if you wrote it from scratch in any language, it would be fairly intuitive for an experienced developer with no knowledge of your business to immediately understand the app and how to extend and maintain it.

At some point, almost any app becomes more complex.  The current version of Digg could not be developed for $200, and I dare say it would be foolish for someone like Digg to build an application of its current complexity completely contracted out.

I’ve been in startups all along the continuum, and made transitions between different mixes of internal versus external product development and infrastructure.  My current opinion is to outsource the scaffolding.  If you’re deploying a java/tomcat/MySQL app on a plain vanilla linux distribution, you can probably have a contract sys admin put that together for you – any other competent admin can look at how it’s configured and fix your problems.  When you start depending on all sorts of custom extensions, configurations, replication, huge scale, and/or DB sharding, it’s probably time to hire some admins that can own that custom knowledge you depend on.

The same thing goes for product development.  Are you building out a marketing web site?  An app that could be built with Rails scaffolding?  Contract it out – good contractors will realize how easy it is and crank it out for cheap (though maybe more than $200).  Most capable software developers you would hire or have hired don’t want to build or maintain that stuff anyhow.  Save internal development for the custom, proprietary stuff you need to spend time on, the meat and potatoes code that differentiates your application from everybody else’s.  The stuff that will take even a good developer more than a couple of hours to really comprehend.  Start throwing the switch to internal resources when it starts to get complicated – you’ll be much happier.  Trying to insource knowledge of a complex application that’s already been written by someone else is doable, but quite a challenge.

Atlanta, Java, Technology

The Next Episode

Back in March, I reached a decision to leave WeTheCitizens and spend some time in a contract role with Premiere. At the time, I expected that a contract gig would let me spend less after-hours time thinking about the day job, and more time on side projects and other interests. It turns out that even as a contractor, I cared about what I was working on. I consider that a good thing, we built up a good team, cranked out some pretty cool software, met some awesome people – I truly enjoyed my time at Premiere. The challenge for me on contract, was balancing life with billing more hours. In some ways, having a dollar amount directly attached to each hour of your time helps you better value your time. In other ways, it’s easy to imagine how working just a few more hours that week can earn you a shiny new gadget.

In spite of that, I was able to hatch FeatureFrame during my off-hours. It did occupy much of my creativity, and the process of building a product prototype, a pitch for Startup Riot, filing a provisional patent, designing a logo and business cards, and creating an actual business entity has been exciting, challenging, and in a twisted sense, fun. Many thanks to my Twitter posse, who often functioned as my co-founder, providing feedback and helping with decisions where I needed more than just my opinion. No I do not have stock options for any of you. Some interesting FeatureFrame news may yet be around the corner. It is still an interest and a passion of mine, and it will live on, but it’s not quite ready to be my main job.

The other thing I suspected when I left WeTheCitizens was that there were some interesting opportunities out there that weren’t crossing my inbox either because they were being filled by recruiters we’d used at WeTheCitizens, or because people generally seem to have the (often correct) impression that an executive in a startup isn’t really looking for another opportunity, at least until the startup’s been bought/folded and the earn-out is done. Being on contract actually did seem to send a clear signal, “Hey, I won’t be doing this new gig for too long, and I’ll need something interesting to do after that.”

A few moderately interesting things came and went, but one opportunity stood out, and I’m now a few weeks into my new role with Good Egg Studios. For all the times we jabbed at one of my Premiere colleagues for his work on, I never thought I would be working to build a virtual world for kids.  But here I am working on Elf Island.  A glance at the site and the blog should begin to make it obvious why.  This is an amazingly talented team, with the right marketing, the right funding, and the right product at the right time. The creativity and talent of the creative team is off the charts, and has combined with some talented technical resources to build a virtual world that a generation ahead of what kids are currently using.  The $2 billion valuation of Webkinz and Disney’s $700 million acquisition of Club Penguin are both examples of companies who have made this space extremely profitable.  The “secret” of Elf Island and our “play games, do good” mantra (which allows users to directly help non-profit causes) are going to be very sticky features that I think kids and adults will enjoy (even before launch, we’ve already had eager & hopeful users build us a fan site).  Our office space in west Atlanta is a win both for commute and for awesomeness – I need to take some photos, or come by and visit some time.

So that’s the news.  We are hiring a Java developer, a Flex developer, and a technical project manager (dev mgr?), so if you are somebody or know somebody who would enjoy such a role, do let me know.

Java, Technology

Running Eclipse on MacBooks with Java 6

The title of this post is a bit misleading in that you apparently cannot run Eclipse on a MacBook Pro with Java 6 set as the default JDK.  Never mind that it took Apple a year and a half after the release of Java 6 for Apple to support Java 6 on OS X in the first place.  I thought Apple was “developer friendly”?  My experience on a Mac has usually been slightly better than working on a PC, except that the failings of a PC can usually be dismissed as Microsoft’s incompetence.  Apple seems to act more like a highly-opinionated jerk.

After I upgraded to Java 6 (and then had to manually change my JDK symlink even after the upgrade), Eclipse refused to start.  The system log showed:

[0x0-0xa90a9].org.eclipse.eclipse[4265]: _NSJVMLoadLibrary: NSAddLibrary failed for /System/Library/Frameworks/JavaVM.framework/Versions/CurrentJDK/Libraries/libjvm.dylib
[0x0-0xa90a9].org.eclipse.eclipse[4265]: JavaVM FATAL: Failed to load the jvm library.

To resolve the issue, I edited /Applications/eclipse/, and uncommented this line:


Problem solved.  Apparently Eclipse uses 32-bit SWT-Cocoa bindings, and Apple just decided that they weren’t going to support 32-bit SWT any more in Java 6, breaking any app that uses them in the process.  So the fix is to just run Eclipse under Java 5 (Java 6 projects still work in this setup).  I’m starting to lose track of the consumer-unfriendly attitudes I’ve experienced from Apple.  Their version of Java 6 is late, incomplete, and lazy.

I will declare this now – Apple is every bit as evil as Microsoft.  If Apple EVER gets the market share that Microsoft once held, Microsoft’s anti-trust violations will seem like trivial misdemeanors compared to what Apple would do with such power.  I need to gather my thoughts on this soon and elaborate on this point.


Java5 Support in an App Server is news? Shame on IBM.

So IBM has released Websphere 6.1, apparently the first Websphere version to run on Java5, only 20 months after JDK 1.5.0 was released. Pathetic. I had to read the article twice just to make sure I wasn’t misreading it. Meanwhile, we run Tomcat 3.3 on Java5 with no patches from Jakarta (even though 3.3.2 was released prior to the Java5 release). A JBoss version released in January of 2005 had full Java5 support.

It’ll be tough to make many fans in the development community with support like that. I just plain don’t get it. You’re IBM, a huge company and one of the heavyweights in the J2EE App Server space. You have near unlimited resources at your fingertips. And you take over a year and a half to adjust to a JVM upgrade that did little more than syntactical upgrades. We migrated our app, over 2000 classes, and the biggest effort was unwiring all of the spots where someone used enum as a variable name. Did we upgrade every class to use the latest and greatest Java5 features? No, what’s the point of that? IBM should have pushed a compatibility release as quickly as possible, and split that effort off of whatever other features went into this release. Developer support matters.


[Atlanta JUG DevCon] Jim Waldo – Notes From the Java Frontier

Jim Waldo is one of Sun’s techie guys, a researcher, not a product or marketing guy. He’s sharing the “challenges” of open sourcing Java, citing interoperability as a potentially competing interest. The session is mostly a somewhat interesting rah-rah session for what we have accomplished in the developer industry, what Sun and Java have accomplished, and where we’re going. Not a bad presentation for a pep rally, but not much meat here either.


[Atlanta JUG] DevCon 2006

I’m sitting at the opening session of the Atlanta Java Users’ Group with pretty much all of our development team. The schedule came out late, and the number of breakout sessions is down from last year, so I’m not sure we’ll benefit from sending 7 people, but we’ll see.

They appear to have accidental wireless access, courtesy of some local cafe. Access to power outlets also looks limited, but our IT folks set me up with a new massive battery for my Thinkpad, so it may be alright.

I probably won’t live-blog each session, but I’ll try and cover some highlights – last year, Sun let slip that they were going to free Java Studio Creator, so there may be some interesting tidbits to come.


When Memory Leaks Attack

It seems Tomcat and Java are gushing memory on us. I’ve been kicking the tires on YourKit Java Profiler, which thus far has been running without incident on one of our production servers. Over the past 48 hours, the memory usage has grown by 34 MB, which is consistent with past evidence of memory leaks in our system. (Soundtrack for this post: “When Animals Attack” – Institute)

The results are fairly interesting – the number one culprit is org.xbill.DNS.*, which accounts for over half of the growth. Apparently Tomcat 3.3 (we’re upgrading to 5.5 soon) has DNS lookups and caching – that cache seems to be unlimited in size, with no expiration, and cannot be turned off [in 3.3]. So in an application that receives callbacks from high-traffic corporate web sites, with a broad range if inbound IP addresses, this becomes a large memory leak.

This is why JVMTI is such a leap over JVMPI – my first experience with a Java profiler was JProbe, and my main impression of it was how tediously slow it was. With JVMTI agents, our app runs pretty much at full speed, and can be profiled on demand. Finding this leak in a development or QA environment with a JVMPI profiler would be nearly impossible, because you just don’t tend to test with thousands of distinct IP addresses in QA.

The other interesting leak is a JVM Bug that will be fixed in Java 6. It seems that ObjectInputStream uses a SoftCache that results in retained references to classes that have been serialized. This explains the wealth of new HashMap$Entry and byte[] objects that are sitting in memory.

In a way, the results are encouraging – over the past 48 hours, running at high volume, the memory used by classes we’ve written has actually dropped by 150 kb. So the up side is that the memory growth isn’t our fault. The down side is that it’s much harder for us to resolve the issues since they’re not in our code.