Agile. Now the corner-stone of any development organization, agile processes and development methodologies have taken over businesses, tech. articles, and the lives of the engineers tasked with playing their song. I’m not sure you could find a software engineer job listing that didn’t ask for “agile experience.” The entire idea behind agile processes is to enable us, as engineers, to respond quickly to changing business requirements – to be agile – right? If that’s truly the case, then why are they so painful?

For even a small team of 4 contributors and a dev manager (or program manager, product owner, etc), [sprint planning] is 2 days worth of salary for 5 people to just decide what we’re actually going to be working on. That’s really expensive.

Take for example Scrum, in its purest sense. In a nutshell and in contrast to “Waterfall Development” Scrum enables us to work in (typically) 2-week periods of development (known as “sprints”), so that we don’t spend ages up-front ironing all of the business requirements, design specs, user interactions, etc. just to watch them change in a month. To start every sprint, we typically spend a day or two selecting the stories that have written by our product owner, ensuring that they fit within our velocity, and then breaking those stories down in to actionable tasks. For even a small team of 4 contributors and a manager, that’s 2 days worth of salary for 5 people to just decide what we’re actually going to be working on. That’s really expensive. If you could eliminate that, you get another engineer on the team for “free.” Not only is this an expensive process, it’s absolutely soul crushing for those involved. During a sprint, we spend an alleged 10-15 minutes (which usually turns in to 30-45 minutes) for our “stand-up” every morning so that everybody is on the same page.

As another example, let’s look at Kanban. With this, we’ve managed to kill off the 1-2 day sprint planning meetings, but we still have the the stories on our Kanban board, and we still have our overly-long and grueling stand-up. Further, because we’ve removed the sprint planning and task break down, the level of detail that ends up going in the stories is far too high and the number of meetings to try and clarify that detail quickly gets out of hand. Not only that, because we’re constantly evolving a single story, the amount of work that ends up part of it grows, and that kills our velocity. What started out with good intentions has turned problematic.

That’s two different flavors of “Agile” processes, both still have weaknesses.

Saving Time And Money

One of the more recent changes I tried to implement on one of my teams to help mitigate some of this was to reduce the amount of time spent in our daily stand-up. A stand-up generally tasks each member of the team with answering three questions:

1. What did you do yesterday?
2. What are you doing today?
3. Are you blocked?

The first run at this goal got us down to:

1. What’s remaining on the story you’re working on?
2. Are you blocked?

The end result was:

1. Are you blocked?

In essence, the entire meeting was reduced to about 60-120 seconds of time where everybody working on the project would be in the same physical place at the same time to facilitate getting a hold of somebody if you needed to. That’s really it.

Anything outside of answering that single question has no real bearing; nearly all of the other information is available by looking at the Kanban board, and if more detail is needed about something, that’s a smaller conversation that doesn’t require the entire team and can be organized with “I need to talk to you and you after this.” In essence, the entire meeting was reduced to about 60-120 seconds of time where everybody working on the project would be in the same physical place at the same time to facilitate getting a hold of somebody if you needed to. That’s really it.

Expanding The Idea

What if we could throw out nearly all of the process? The “Processless Process”?

Most recently, two co-workers and I participated in our organization’s Hackathon. We took a slightly different approach than other teams did. While we did produce some great technical artifacts as well as some great metrics and statistics, we had a completely different focus; hack the process.

Our goal was to see just how much work we could crank out in two days using 3 people if the entire process was effectively thrown out. Bring two engineers and a creative guy together to just do their jobs without things getting in the way and see how successful could they be.

It turns out that if you let you talented and skilled people do what they’re good at and passionate about, a lot gets accomplished in a short amount of time.

It turns out that if you let you talented and skilled people do what they’re good at and passionate about, a lot gets accomplished in a very short amount of time. While we didn’t create production-ready code by any means, we did generate an entirely new site design, implement a fault-tolerant API abstraction and caching layer, migrate our assets to .webm and .webp saving between 60-80% of our file size, put together a new search results page with three different layout styles, and a new detail page. In two days.

The greatest outcome from that endeavor came from demo itself, garnering comments such as “Why aren’t we [working like this] today?”

Where Are We Now?

While it’s taken quite a while to grow roots, the fruits of that labor are beginning to bear. There a few experimental projects running through the organization now, and while the idea has been given a marketing-esque name, the concept itself is intact; dissolve the unnecessary process over-head, enable individuals to thrive and succeed without putting up roadblocks, and watch the speed at which development progresses.

These changes have enabled us to work very loosely and iteratively with the business. Our requirements are now akin to “this is kind of what we’re thinking” and our UX comps are more of a “this is kind of what we want it to look like”. We have a closer relationsihp with both UX and the business to refine and iterate on the design and the concept itself. We’re no longer risking throwing out large portions of work when we come across something that doesn’t work for one reason or another.

Our sprints, meetings, stories, story planning and task breakdown, and our daily standup have all been replaced with walking over to someone or shooting them an IM or an email and asking a question. We have one weekly meeting, for about 30 minutes, to go over customer and business feedback. That’s it.

Further, our time to market is drastically reduced. Pushing our in-progress code to production behind an A/B test that’s limited to a very small fraction of our users enables us to gather real-world metrics from our actual customers without affecting the bulk of the site.

Feedback across the board has been positive; the engineers like it, the UX/UI teams like it, the PMs, POs, and dev. managers like it, the directors like it. Thus far, there’s not been (to my knowledge) a single negative thing that’s been said with one exception – the concern that we’re moving too fast for the business. I’m not quite sure what that means yet, but I think it’s a good thing.

Finally, we’re being agile instead of doing agile. And it feels good.

I’ve started doing some Ruby development. The commonly used platform for the development team(s) tends to be one of OS X, Ubuntu, or Mint. I do not fall in to that category.

Getting RVM to install Ruby under Archlinux was fraught with issues. Most of them were trivial and just a matter of getting the right dependencies installed. There was, though, one issue that was particularly misleading. When trying to rvm install 2.1.1, I received this error:

Error running '__rvm_make -j2',
showing last 15 lines of /home/sbarker/.rvm/log/1394556125_ruby-2.1.1/make.log
make[2]: Leaving directory '/home/sbarker/.rvm/src/ruby-2.1.1/ext/readline'
exts.mk:199: recipe for target 'ext/readline/all' failed
make[1]: *** [ext/readline/all] Error 2
make[1]: *** Waiting for unfinished jobs....
compiling ossl_x509crl.c
compiling ossl_digest.c
compiling ossl_pkey_rsa.c
compiling ossl_engine.c
compiling ossl_ssl.c
installing default openssl libraries
linking shared-object openssl.so
make[2]: Leaving directory '/home/sbarker/.rvm/src/ruby-2.1.1/ext/openssl'
make[1]: Leaving directory '/home/sbarker/.rvm/src/ruby-2.1.1'
uncommon.mk:180: recipe for target 'build-ext' failed
make: *** [build-ext] Error 2
There has been an error while running make. Halting the installation.

A similar-ish error occurred when trying to install Ruby 2.0.0.

Initially this looked to be an error with OpenSSL dependencies. I spent longer than I would care to admit going down that road, trying several different options, including rvm autolibs enable, all to no avail. I finally gave up, and in admitting defeat, spun up an Ubuntu VM. And behold, the same error occurred. It wasn’t an OS issue.

Back to the drawing board, I dove in to ~/.rvm/src/ruby/2.1.1 to ./configure and make the package by hand to see if I could discover anything new. As it turns out, seeing the full context around the error pointed me in the right direction.

The first bit of the error reported by RVM is the key.

Error running '__rvm_make -j2',
showing last 15 lines of /home/sbarker/.rvm/log/1394556125_ruby-2.1.1/make.log
make[2]: Leaving directory '/home/sbarker/.rvm/src/ruby-2.1.1/ext/readline'
exts.mk:199: recipe for target 'ext/readline/all' failed
make[1]: *** [ext/readline/all] Error 2
make[1]: *** Waiting for unfinished jobs....

The issue lies with readline and not with openssl dependencies. RVM’s autolibs setting obviously didn’t trigger, since readline was present. I told RVM to install its own readline with rvm pkg install readline. When that finished building, I attempted to install Ruby 2.1.1 again with rvm install 2.1.1 -C --with-readline-dir=$HOME/.rvm/usr. The result, great success. RVM happily downloaded, built, and installed 2.1.1 for me. I repeated this for 2.0.0 and also had great success. 1.9.3 was never an issue, and that continued to be not an issue.

I recently ran in to this issue where Visual Studio 2013 didn’t see any of my MSTest unit tests, but only one machine. The repository for the code was identically cloned on three different machines and only the one was exhibiting the problem. I spent at least five days trying to solve this problem, trolling through StackOverflow, Google, Bing, and various forums, all to no available.

I finally found the solution.

Symptoms

• If I were to right-click on the solution or project level and tell Resharper to run all the tests in the solution or the project, I would get an error “No tests found in file” or “No tests found in solution.”

• If I opened a code file that contained a test class and test methods, the Resharper test runner would sometimes load the tests, but if I tried to run them they wouldn’t actually run, and Resharper would report them all as Inconclusive.

• If I were to tell Visual Studio to run all tests, the progress bar would complete, but the list of tests in the test runner would still be empty.

Solution

As it turns out, there was one difference with this single environment vs. the other two; the location of the code. In this one environment, the source code was located on a mapped network share. That was preventing Visual Studio from being able to work with the code properly. You may at some point have gotten a dialog from Visual Studio when opening the project or solution telling you that the code is an untrusted location. That’s the root of the issue. Visual Studio will still open the solution, compile the code, and run it, but the test runner won’t be able to do anything with it. The solution is simple, but stupid.

Open the Windows control panel, and open the Internet Options. From there, select the Security tab. Select the Trusted Sites group, and click the Site button. In the dialog that opens, add the IP address of the machine that the network share lives on. Save your back through the various dialogs, restart Visual Studio, clean and rebuild your project, and now the Visual Studio test runner or Resharper should have no problem seeing your tests.

Special Thanks to Microsoft

You know, for silently failing when not being able to work with the files that you asked it to work with.

Not so long ago, I had a post about Completely removing all traces of files and history from Git repositories. That title was somewhat misleading, since it was more about migrating from TFS to Git, and in the process pruning out files that you didn’t want to leave in your repository. So, if you’re looking to migrate from TFS to Git, and/or you’re looking to prune out history, have a look at that post.

Update: This is no longer necessary, check out the update here.

In my last post (PRO-TIP: Recursively updating all git repositories in a directory) I made mention of using Git’s new credential caching to improve your Git experience.

A couple things of note:

• The API for credential management in Git is fairly newish (as of 1.7.9) so you’ll a fairly newish Git to make use of it.

• This doesn’t work for Windows systems as git-credential-cache communicates through a Unix socket. Windows users can find joy here at https://github.com/anurse/git-credential-winstore (specifically for the download the joy is located at https://github.com/anurse/git-credential-winstore/downloads). Just make sure that the binary is in your path (which is likely C:\Program Files (x86)\Git\libexec\git-core if you’ve installed msysgit and didn’t mess with the defaults). The integration is with Windows Credential Manager which you can pull up via the Windows Control Panel.

Once you’ve got it installed, from a command prompt:

git config --global credential.helper winstore

Or you can edit your .gitconfig manually:

[credential]
     helper = winstore

I have a directory, let’s call it ./src (because it is). This directory has several other directories of which some subset are git repositories. Updating them all by hand is tedious at best and I am lazy. Here’s a one-liner that will do all of the work for you.

W=`pwd`;for i in $(find . -name .git);do D=$i;D=${D%/*};cd $W/$D;pwd;git pull;done

This has the potential to be a bit taxing as well though if you need to enter credentials for every pull. We can solve that one as well.

git config --global credential.helper 'cache --timeout=[S]'

The --timeout option has a parameter, S, which is the timeout in seconds. This can be omitted and a default timeout of 15 minutes will be used.

Being lazy + saving time + making things easy = Happy Panda!

At my company, we recently made the decision to migrate our source repositories from TFS to Git. While new projects have been going in to Git for quite some time, and some projects had already been migrated, the largest still remained. Let’s call this project Website.

The Website repository in TFS contained not only history for the last several years but also all of that history for our marketing content for the site, largely images. We determined that this shouldn’t be part of the repository at all (at least this one, it would continue to live in its own TFS repository until final arrangements could be made) and thus removing it would be part of our migration process. The process boiled down to this:

1. Branch out the Marketing content in to its own TFS repository using a build step for Website to perform the check out and get it where it needs to be.
2. Clone the TFS repository in to a new Git repository.
3. Clean out all of the old history of the Marketing content from Website.

I had no real involvement with the first step, so I spent some time with steps #2 and #3. I ran through this process on my development machine (an eight core Xeon 3.33ghz with 12gb of ram, Windows 7 Professional) a few times to make sure that I had the process down and to get an idea of timing so that we could better plan. Both steps took about 4 days each to complete. Woof. As it turns out, a co-worker spun up a large Ubuntu EC2 instance to do the heavy lifting after the process was down and brought the execution time down to about 2.5 hours total.

Here’s what we did to actually make this happen.

Clone the TFS repository in to a new Git repository.

This becomes a very straight forward process using git-tfs (http://git-tfs.com/, https://github.com/spraints/git-tfs). I won’t go in to the details of getting that pulled down and running. Once you have it though, you’re going to issue this command from the root of the directory that you want to clone the repository in to:

git tfs clone -d --username <username> http://server:8080/tfs/collection <repository> .

Note: You’ll have to do at least this initial clone on a Windows machine as the git-tfs binaries are Windows-only. Anything after that can run anywhere that Git can run.

Make sure to replace the placeholders (<username>, the URL, and <repository>) with values that are relevant for your system.

After that’s done spinning for a while, you’ll have a Git repository that mirrors your TFS repository.

Clean out all of the old history of the Marketing content from Website.

Now we want to get rid of all of the enormous Marketing content. This is where the Git voodoo comes in.

From the root directory of your Git repository, you’ll want to issue these commands:

git filter-branch --index-filter "git rm -rf --cached --ignore-unmatch Website/Marketing" --prune-empty HEAD
rm –rf ./.git/refs/original
git reflog expire --expire=0 --all
git gc --aggressive --prune=0

The filter-branch command will go through every commit in your repository removing any files and history in the Website/Marketing path and, if any commits would be left empty after that, removing the commit in its entirety. The next command removes the original references that were in the repository; these would be pointing to the bits of history that we removed earlier. After that, expire all of the references in the repository. Finally, garbage collect your repository.

The final result is going to be a commit-by-commit clone of your TFS repository, less any of the data you told it to remove and potentially less a few commits if they only contained files that you removed.

In the end, we had a TFS repository weighing in at about 1.2gb which become a Git repository sitting around 330mb. That’s a pretty substantial win.

Potential pitfall

You might find that the size of your repository didn’t decrease at all after you do this. My previously mentioned coworker had to actually clone the repository to another git repository to see the result and end up with a smaller repository.

cd ..
mkdir WebsiteClone
cd WebsiteClone
git clone ../Website

After seeing that the project that I’m working on had 48 branches hanging out, 40 of which were completely merged and abandoned, I threw together a quick bit of one-liner (it’s still one line even if it’s more than 80 columns, right?) to delete all of the fully merged branches.

Full disclosure; this works on my machine, which happens to be a Win7 machine with some release of Cygwin on it (which I loathe):

for k in $(git branch -a --merged) | grep -v "\->\|master" | sed s/remotes\\/origin\\///);\
do $(git branch -d $k; git push origin :$k); done

And when you’re done, make sure you alert others to git remote prune origin.

The breakdown

To elaborate on this a little bit, for each branch $k in every completely merged branch:

for k in $(git branch -a --merged)

Make sure that it isn’t pointing somewhere (like HEAD would be) and isn’t master:

 | grep -v "\->\|master"

Remove the remotes/origin/ part of the branch name:

| sed s/remotes\\/origin\\///);\

Delete the local branch (if it’s there) and push the delete to the repository:

do $(git branch -d $k; git push origin :$k); done

Inspiration taken from Graham King at http://www.darkcoding.net/software/cleaning-up-old-git-branches/

One of the great features of Github (and there are many) is the ability to fork a project to do your own development and experimentation on. I don’t, however, use Github, as I prefer to keep everything on my own server and not pay for something that I’m capable of hosting myself. I recently started a new project that had a base of the framework from an old project, so this was an obvious place to do some pseudo-refactoring (I’m leaving the old, original project intact) and giving me a good starting bed for future new projects. I had a few goals:

• Break the framework out of the Old Project into its own New Framework repository
• Fork New Framework and start development of New Project
• Easily fold changes to New Framework into New Project without tainting New Framework
• Easily fold changes to New Project that related explicitly to New Framework back in to New Framework

I started off by cloning Old Project in to New Framework and stripping out everything that had anything to do with Old Project.

git clone OldProject NewFramework
cd NewFramework
rm -rf .git/
git init
# *hack hack hack to remove all OldProject cruft from NewFramework*
git remote add origin [new-framework repository url]
git push origin master

At this point, we have our NewFramework directory that’s full of just NewFramework, and it’s all happy and pushed off to the NewFramework repository. This is the starting point that we want for future projects that will be using this framework which is perfect because that’s exactly the situation that we’re in. So, let’s start NewProject using NewFramework as a base.

git clone NewFramework NewProject
cd NewProject
git remote -v
git remote add upstream [url of the 'origin' remote from the git remote -v command]
git remote remove origin
git remote add origin [url of the repository for NewProject]
git push origin master
git branch upstream

And now we have our NewProject repository created with our bare copy of NewFramework committed and pushed up. When we have changes in NewFramework that we want to fold back in to NewProject, we can do this:

git checkout upstream
git pull upstream master
git checkout master
git merge upstream

And if we have some changes to the framework that we want to go back to NewFramework:

git log
# *find the sha of the commit we want*
git checkout upstream
git cherrypick [sha of the commit we want]
git push upstream upstream:master

Data backup is one of those things that everybody talks about, few people do, fewer people do well, and fewer still have actually tested.

What makes a good backup strategy?

To me, a backup strategy needs to have a few qualities.

• It has to be easy. If it’s not, you won’t keep up with it.
• It has to be reliable. Backing up your data won’t do you any good if your backups aren’t good.
• It has to be redundant. Backups can go bad too.
• It has to be recoverable from. If you’re encrypting your backups and you forget your key, they’re useless.

Now, what brought me here, and how did I attain those goals?

First, why so seriousparanoid?

I’ve been paranoid about data loss for a long time, and I spent a good deal of time and effort trying to figure out what the best strategy would be for me that met all of the requirements that I outlined above. But why was I so paranoid to begin with?

When I was a freshman in college I experienced my first hard drive failure. My Western Digital hard drive suddenly gave up the ghost, taking with it all of my software that I had written over the years (much of it written in x86 assembly) with it. Try as I may, I couldn’t recover anything. I would have paid anything then to get that data back, but being a poor college student professional data recovery wasn’t an option. With no real backups, my entire digital life to this point was wiped clean.

It was at this point that I learned the importance of backups. I didn’t, however, learn the importance of a good backup strategy. To that end, I would burn CDs and email myself copies of things that were REALLY important. Other important things were zipped up and stored on another hard drive. Sometimes I’d just copy and paste a folder somewhere else. I’d have multiple copies of things floating around, and no real way to tell which was the most recent, or most correct.

At the time, I thought that this worked. Mostly because I just didn’t know any better. Had I experienced a drive failure during that period, I’d have been sent on a wild goose chase through my old email, unlabeled physical media, and folders upon folders of copies of various files and zip archives. I’ve since seen the error my ways. In part because I’ve gotten smarter, and in part because technology has gotten smarter.

A new strategy is born

My new backup strategy is much, much more robust, easy to manage, and easy to recover from.

Technology

Hardware:

• OS Drive - OS, applications, basic configurations and settings (160gb OCZ Vertex 2 SSD)
• Secondary Drive - VMWare Virtual Machine (1tb Seagate)
• External Array - All other documents (2nd Gen Drobo w/ 4x 2tb Seagate)
• External Drive - Local backups (1.5tb Seagate)
• External Enclosure - Enclosure for the External Drive (SentrySafe QA0121 Fire-Safe Waterproof Data Storage Chest)
• Printer/Scanner - Document digitizing (Epson Artisan 835)

Software:

• Drobo Dashboard / DroboCopy
• CrashPlan with CrashPlan+ unlimited subscription
• TrueCrypt
• git

How Does This Work Together?

• Every 4 hours, CrashPlan backs up changes to the Virtual Machine on the Secondary Drive to the External Drive (encrypted with TrueCrypt), the Drobo, and to CrashPlan+.

• Once per week, DroboCopy copies the Virtual Machine to the Drobo. This is done to give me an instantly available copy-and-paste snapshot of the server to get back up while I recover the most recent version through a CrashPlan restore.

• In real-time, CrashPlan watches for changes to anything of high value on the Drobo and backs those changes up to the External Drive CrashPlan+.

• Those HVFs, in addition to source code, pictures, tax returns, and the like, include scans of important physical documents (product warranties, contracts, receipts, etc.) from the Artisan 835. The original physical documents are kept in a separate fire/water-proof safe. In addition, using the Artisan, I create hard copies of digital documents (receipts and the like) for physical storage.

• Source code is also stored in git repositories on the Virtual Machine so that I have full revision history for any project that I’m working on (my old CVS repositories have been deprecated and converted to git repositories).

What does all of this do for me? I have several points of recovery available, and aside from the OS and applications which have no irreplaceable value, I have no more than 4 hours of unrecoverable data. This all took quite a while to set up, but the peace of mind is worth it. The backups are pretty much out-of-site, out-of-mind, and I never have to worry about a manual step to protect my data.

Is it excessive? Perhaps, but I never worry about losing another piece of important data again.

There are a couple of things that I’d like to improve, but they’re not critical. One, I’d like to upgrade to a DroboFS to remove the dependency of having my Drobo physically attached to my primary machine. Second, I really wish CrashPlan would allow me to add machines to my account without buying a family subscription. I only have one more machine I’d like to add (the Virtual Machine), and the cost of a family subscription just isn’t worth it when I work around that limitation by backing up the entire machine (since it’s just a set of files). It’s just annoying.

Update

I’ve since upgraded from the Gen2 Drobo mentioned above to a DroboFS (2x 3Tb, 2x 2Tb, 1x 1Tb with dual-drive redundancy). In addition to the speed benefits and the obvious benefits of being a NAS, my paranoia during rebuilds for the array makes dual-drive redundancy a must have. Unfortunately, the DroboFS is currently having a lot of different issues (though none that seem to be putting my data at risk). I have a support ticket in with Data Robotics and hopefully they can address the issues.