Java, Technology

Outsource the Scaffolding

Just today, I twittered a link to an insightful blog post, “Here’s the Deadliest Catch: Hiring an Agency to Build Your Startup”.  The title is largely self-explanatory, and I may blog my own riff on it at a later date.  Digg is often raised as a counter-example to outsourcing development of a startup, with version one developed for $200.  So how do you decide what the right balance is for you?

Ruby on Rails has this concept of “scaffolding“, which makes it trivial to get a simple web application running to perform create/read/update/delete operations on a simple business object.  So if you wanted to build a basic address book, Rails could basically create that application with a one line scaffold generate script.  It produces a simple web interface, and handles all of the database calls for you.  Java code to accomplish the same task is typically more verbose, but even if you wrote it from scratch in any language, it would be fairly intuitive for an experienced developer with no knowledge of your business to immediately understand the app and how to extend and maintain it.

At some point, almost any app becomes more complex.  The current version of Digg could not be developed for $200, and I dare say it would be foolish for someone like Digg to build an application of its current complexity completely contracted out.

I’ve been in startups all along the continuum, and made transitions between different mixes of internal versus external product development and infrastructure.  My current opinion is to outsource the scaffolding.  If you’re deploying a java/tomcat/MySQL app on a plain vanilla linux distribution, you can probably have a contract sys admin put that together for you – any other competent admin can look at how it’s configured and fix your problems.  When you start depending on all sorts of custom extensions, configurations, replication, huge scale, and/or DB sharding, it’s probably time to hire some admins that can own that custom knowledge you depend on.

The same thing goes for product development.  Are you building out a marketing web site?  An app that could be built with Rails scaffolding?  Contract it out – good contractors will realize how easy it is and crank it out for cheap (though maybe more than $200).  Most capable software developers you would hire or have hired don’t want to build or maintain that stuff anyhow.  Save internal development for the custom, proprietary stuff you need to spend time on, the meat and potatoes code that differentiates your application from everybody else’s.  The stuff that will take even a good developer more than a couple of hours to really comprehend.  Start throwing the switch to internal resources when it starts to get complicated – you’ll be much happier.  Trying to insource knowledge of a complex application that’s already been written by someone else is doable, but quite a challenge.

Atlanta, Java, Technology

The Next Episode

Back in March, I reached a decision to leave WeTheCitizens and spend some time in a contract role with Premiere. At the time, I expected that a contract gig would let me spend less after-hours time thinking about the day job, and more time on side projects and other interests. It turns out that even as a contractor, I cared about what I was working on. I consider that a good thing, we built up a good team, cranked out some pretty cool software, met some awesome people – I truly enjoyed my time at Premiere. The challenge for me on contract, was balancing life with billing more hours. In some ways, having a dollar amount directly attached to each hour of your time helps you better value your time. In other ways, it’s easy to imagine how working just a few more hours that week can earn you a shiny new gadget.

In spite of that, I was able to hatch FeatureFrame during my off-hours. It did occupy much of my creativity, and the process of building a product prototype, a pitch for Startup Riot, filing a provisional patent, designing a logo and business cards, and creating an actual business entity has been exciting, challenging, and in a twisted sense, fun. Many thanks to my Twitter posse, who often functioned as my co-founder, providing feedback and helping with decisions where I needed more than just my opinion. No I do not have stock options for any of you. Some interesting FeatureFrame news may yet be around the corner. It is still an interest and a passion of mine, and it will live on, but it’s not quite ready to be my main job.

The other thing I suspected when I left WeTheCitizens was that there were some interesting opportunities out there that weren’t crossing my inbox either because they were being filled by recruiters we’d used at WeTheCitizens, or because people generally seem to have the (often correct) impression that an executive in a startup isn’t really looking for another opportunity, at least until the startup’s been bought/folded and the earn-out is done. Being on contract actually did seem to send a clear signal, “Hey, I won’t be doing this new gig for too long, and I’ll need something interesting to do after that.”

A few moderately interesting things came and went, but one opportunity stood out, and I’m now a few weeks into my new role with Good Egg Studios. For all the times we jabbed at one of my Premiere colleagues for his work on, I never thought I would be working to build a virtual world for kids.  But here I am working on Elf Island.  A glance at the site and the blog should begin to make it obvious why.  This is an amazingly talented team, with the right marketing, the right funding, and the right product at the right time. The creativity and talent of the creative team is off the charts, and has combined with some talented technical resources to build a virtual world that a generation ahead of what kids are currently using.  The $2 billion valuation of Webkinz and Disney’s $700 million acquisition of Club Penguin are both examples of companies who have made this space extremely profitable.  The “secret” of Elf Island and our “play games, do good” mantra (which allows users to directly help non-profit causes) are going to be very sticky features that I think kids and adults will enjoy (even before launch, we’ve already had eager & hopeful users build us a fan site).  Our office space in west Atlanta is a win both for commute and for awesomeness – I need to take some photos, or come by and visit some time.

So that’s the news.  We are hiring a Java developer, a Flex developer, and a technical project manager (dev mgr?), so if you are somebody or know somebody who would enjoy such a role, do let me know.

Java, Technology

Running Eclipse on MacBooks with Java 6

The title of this post is a bit misleading in that you apparently cannot run Eclipse on a MacBook Pro with Java 6 set as the default JDK.  Never mind that it took Apple a year and a half after the release of Java 6 for Apple to support Java 6 on OS X in the first place.  I thought Apple was “developer friendly”?  My experience on a Mac has usually been slightly better than working on a PC, except that the failings of a PC can usually be dismissed as Microsoft’s incompetence.  Apple seems to act more like a highly-opinionated jerk.

After I upgraded to Java 6 (and then had to manually change my JDK symlink even after the upgrade), Eclipse refused to start.  The system log showed:

[0x0-0xa90a9].org.eclipse.eclipse[4265]: _NSJVMLoadLibrary: NSAddLibrary failed for /System/Library/Frameworks/JavaVM.framework/Versions/CurrentJDK/Libraries/libjvm.dylib
[0x0-0xa90a9].org.eclipse.eclipse[4265]: JavaVM FATAL: Failed to load the jvm library.

To resolve the issue, I edited /Applications/eclipse/, and uncommented this line:


Problem solved.  Apparently Eclipse uses 32-bit SWT-Cocoa bindings, and Apple just decided that they weren’t going to support 32-bit SWT any more in Java 6, breaking any app that uses them in the process.  So the fix is to just run Eclipse under Java 5 (Java 6 projects still work in this setup).  I’m starting to lose track of the consumer-unfriendly attitudes I’ve experienced from Apple.  Their version of Java 6 is late, incomplete, and lazy.

I will declare this now – Apple is every bit as evil as Microsoft.  If Apple EVER gets the market share that Microsoft once held, Microsoft’s anti-trust violations will seem like trivial misdemeanors compared to what Apple would do with such power.  I need to gather my thoughts on this soon and elaborate on this point.


Java5 Support in an App Server is news? Shame on IBM.

So IBM has released Websphere 6.1, apparently the first Websphere version to run on Java5, only 20 months after JDK 1.5.0 was released. Pathetic. I had to read the article twice just to make sure I wasn’t misreading it. Meanwhile, we run Tomcat 3.3 on Java5 with no patches from Jakarta (even though 3.3.2 was released prior to the Java5 release). A JBoss version released in January of 2005 had full Java5 support.

It’ll be tough to make many fans in the development community with support like that. I just plain don’t get it. You’re IBM, a huge company and one of the heavyweights in the J2EE App Server space. You have near unlimited resources at your fingertips. And you take over a year and a half to adjust to a JVM upgrade that did little more than syntactical upgrades. We migrated our app, over 2000 classes, and the biggest effort was unwiring all of the spots where someone used enum as a variable name. Did we upgrade every class to use the latest and greatest Java5 features? No, what’s the point of that? IBM should have pushed a compatibility release as quickly as possible, and split that effort off of whatever other features went into this release. Developer support matters.


[Atlanta JUG DevCon] Jim Waldo – Notes From the Java Frontier

Jim Waldo is one of Sun’s techie guys, a researcher, not a product or marketing guy. He’s sharing the “challenges” of open sourcing Java, citing interoperability as a potentially competing interest. The session is mostly a somewhat interesting rah-rah session for what we have accomplished in the developer industry, what Sun and Java have accomplished, and where we’re going. Not a bad presentation for a pep rally, but not much meat here either.


[Atlanta JUG] DevCon 2006

I’m sitting at the opening session of the Atlanta Java Users’ Group with pretty much all of our development team. The schedule came out late, and the number of breakout sessions is down from last year, so I’m not sure we’ll benefit from sending 7 people, but we’ll see.

They appear to have accidental wireless access, courtesy of some local cafe. Access to power outlets also looks limited, but our IT folks set me up with a new massive battery for my Thinkpad, so it may be alright.

I probably won’t live-blog each session, but I’ll try and cover some highlights – last year, Sun let slip that they were going to free Java Studio Creator, so there may be some interesting tidbits to come.


When Memory Leaks Attack

It seems Tomcat and Java are gushing memory on us. I’ve been kicking the tires on YourKit Java Profiler, which thus far has been running without incident on one of our production servers. Over the past 48 hours, the memory usage has grown by 34 MB, which is consistent with past evidence of memory leaks in our system. (Soundtrack for this post: “When Animals Attack” – Institute)

The results are fairly interesting – the number one culprit is org.xbill.DNS.*, which accounts for over half of the growth. Apparently Tomcat 3.3 (we’re upgrading to 5.5 soon) has DNS lookups and caching – that cache seems to be unlimited in size, with no expiration, and cannot be turned off [in 3.3]. So in an application that receives callbacks from high-traffic corporate web sites, with a broad range if inbound IP addresses, this becomes a large memory leak.

This is why JVMTI is such a leap over JVMPI – my first experience with a Java profiler was JProbe, and my main impression of it was how tediously slow it was. With JVMTI agents, our app runs pretty much at full speed, and can be profiled on demand. Finding this leak in a development or QA environment with a JVMPI profiler would be nearly impossible, because you just don’t tend to test with thousands of distinct IP addresses in QA.

The other interesting leak is a JVM Bug that will be fixed in Java 6. It seems that ObjectInputStream uses a SoftCache that results in retained references to classes that have been serialized. This explains the wealth of new HashMap$Entry and byte[] objects that are sitting in memory.

In a way, the results are encouraging – over the past 48 hours, running at high volume, the memory used by classes we’ve written has actually dropped by 150 kb. So the up side is that the memory growth isn’t our fault. The down side is that it’s much harder for us to resolve the issues since they’re not in our code.


[NFJS Atlanta] Expert Panel Discussion

Panel members are:
Venkat Subramaniam – founder of Agile Developer
Keith Donald – one of the Spring junkies
Neal Ford – ThoughtWorks
Scott Davis – co-author of JBoss at work
Andy Glover – Vanward Technologies
Jared Richardson – author of “Ship It!”

Question on the Ruby phenomenon – the panel is a bit more sedate on this topic than I expected. No foaming at the mouth yet. Most of the panel members are fairly balanced, suggesting that Ruby is worth our attention to learn, but hasn’t wrested the crown from J2EE as the “it language”, as some people seem to believe lately.
Question on simplified web development, lowered development cycle time – lots of excitement over productivity on Ruby on Rails. Good mention of the Java alternative to RoR – Trails.
Question on goods and bads on Java 5 and Java 6 – one positive is JConsole with Java 5, and the level of instrumentation that comes with a standard JVM. Neal’s concerned about over-engineering in Java, including generics, the addition of Javascript into Java 6. He mentions Bruce Tate’s recent “Joe the Monkey” entry. Scott Davis jokes he won’t upgrade to the next version of Java until they deprecate something.
A decent discussion of agility in architecture. One interesting observation is that the lack of an analog for the DAO in MVC/web frameworks makes the choice of web framework a much bigger committment than a committment to a persistence solution.
Now we have some light references to test-driven development. It’s all fine and dandy, but I have yet to hear people provide a compelling approach to testing our sort of application. We have a heavily interactive web application that loads all of its pages based on user context, and has handshakes and signals to a back end server, sometimes out of sequence. We have pages and pages of Javascript includes. We have a matching engine that requires numerous active entries, and a non-deterministic outcome.

That’s about it. Interesting, but nothing unexpected.


Where am I blogging from? No Fluff Just Stuff’s Greater Atlanta Java Software Symposium.


[NFJS Atlanta] Managing Agile Projects

I neglected to write about last night’s Agile Estimating and Planning session by Mike Cohn, who will be presenting most of the sessions I’m attending today. It was a good session – the main takeaway was that in planning projects, and particularly in estimating features for software, relative estimation is more accurate than absolute estimation. We’re better at realizing that one feature (or “user story”) is twice as difficult as something we’ve done previously. The idea is that then, once we’ve defined an abstract measure of work, we can gain a pretty good idea of the “velocity” or our team – the number of units of work we can accomplish during an iteration. Re-examining the plan versus results after each iteration helps us find a good equilibrium for our team.

This morning, my first session is Managing Agile Projects. It has a good bit of overlap with yesterday’s session, but addresses a broader array of topics. The prime concern is dispelling myths about agile project management – that agile projects are impossible to plan, that they don’t need to be managed because they are self-organizing, and that the project manager’s role is to buy pizza and get out of the way.

The first new tool to me is the “burndown chart”, which can be used for both releases and iterations within a release. In essence, the y axis is the amount of work remaining, the x axis represents the amount of time for the release or iteration. Each iteration or day, you update the work remaining, and connecting the line of work remaining over time gives you the ability to extrapolate and see how well you are tracking to the project schedule. It gives you a basis to begin investigating sources of slowdown or unexpected efficiency, adjust the release (either date or content) for the actual progress, and improve your estimation for next time.

On to the necessity of managing agile projects, we’re back to my problem of him talking about this stuff in terms of teams. He’s assessing the readiness of multiple teams for agile development. With just one team, we don’t have the luxury of evaluating team by team. We have the team we’ve built, and they’re very good at what they do.

The idea of a task board to “assign” tasks and track progress is fairly compelling. Some of this comes down to better planning of releases, and getting out from under the weight of bug fixes is key to accomplishing this. Also, a lot of these ideas seem predicated on the assumption that tasks to be accomplished are all new work, and thus features can be dropped and you can still deliver a release/iteration. In reality, many of our tasks are wholesale replacement of existing application components. Cutting features from a replcement for an existing component is not an option – a replacement MUST be feature equivalent. I suppose we could cut COMPLETION of the replacement from a given iteration and advance the completion to the next iteration, and cut the scope of the progress on replacement for that iteration. I’m not sure here, and will need to think about it further.

Many of these sessions are useful not only for the content, but also in getting me thinking about our development process and procedures, and how they can be improved. I don’t necessarily have good time to think about this stuff during the rush of a given week.

Where am I blogging from? No Fluff Just Stuff’s Greater Atlanta Java Software Symposium.


[NFJS Atlanta] Intro to Portlets

Kito Mann is here to talk about Portlets (though I normally relate him to JSF). The nice thing about portlets is that they make portals not just a capital-P Portal that the vendors all try and sell, they help commoditize the idea of a portal into a lower-case-p, somewhat portable aggregation of componentized applications. Back at S.P. Richards, we found single sign-on, minor user personalization, and content aggregation to be the most valuable aspects of a portal to our business, and created a “portal” with a few webapps and Tomcat’s single sign on valve. The other bits – multiple device support, portlets, and complex personalization, weren’t really necessary for the needs of the business. At the time, the open source portals didn’t seem mature, and the Portlet spec had just been approved.

A question to ask shortly… Many Java APIs are fairly useless without the vendor-extensions. How much of the common portlet functionality depends on vendor-specific configuration?

Generally, portals are bite-sized applications that can be embedded in a portal view alongside other portlets. Google has an example of a non-hideous portal.
The “Top Stories”, “Weather”, and “Quote of the Day” would be portlets in Java land.

The rest of the session has been a fairly mundane tour through the Portlet API itself. Mostly stuff you could pick up from a magazine article. The actual demo of the portal using Liferay is fairly interesting. It looks like Liferay has come a long way in the past year or two, in terms of look and usability. Getting it working was also a beast at the time, and it didn’t respect J2EE security, but if I need a portal, I’d have to at least give Liferay a second look, based on what I’m seeing.

Where am I blogging from? No Fluff Just Stuff’s Greater Atlanta Java Software Symposium.