24HourDiner

Find a solution. The payment system on my current project interacts with a third party tax vendor, and this vendor needs a unique id from our system for each transaction. This id needs to be unique for our entire organization - at least for the account created. At some point, a decision was made to just use the integer autoincrement id column from our order table as this unique id. This decision blatantly coupled the autoincrement id column of a per environment db table to an external third party system. I am sure there was a reason to do this at the time, although I don't know what it was. Without knowing the circumstances, I can't be upset with those that made the decision. I can however be upset that it was never fixed. Instead it was 'managed.' I think you can follow along here where this is going.

So, for all of our dev, test, and release environments (about 25 with dev machines) we need to have a unique order id when placing a test order - of which we do a lot!  Any time an order is submitted, if there is a duplicate order id, the request fails and we get a semi cryptic exception, the client says something bad happened, and you can try again. 

This ended up spawning the following:

• a wiki page so that devs can check out ranges of numbers so they can seed their dev environments
• separate ranges of numbers for each test and release environment
• someone to update this wiki page when these changed
• all of the database CI builds would keep track of the current id, save it to a text file, rebuild the db, and increment and reseed the order column
• having to relay this information to new (sets of!) devs and qa as the team changed (often!)
• having to listen to understandably annoyed new devs complain about wasting hours when we forgot to relay this information
• when we switched our synchronous build server to run multiple builds asynchronously, we got race conditions on the seed file transactions, wasting more time debugging what was going on.
• when we switched to azure, i had to write a separate process for updating db build scripts as you can't change the seed after deploy (you can hack by turning on identity insert, putting in your new number, and turning it off again though)

This is absolutely insane, and I had to FIGHT to get this fixed.  Oh, and telling someone using the system to "just push submit a couple more times to see if a new id works" should never be business as usual.

But I win my battles by attrition apparently.  There was only one person left that thinks this is somehow ok and was willing to fight me on it this time, but I won. No more ADD. His argument was "ok, we can fix it, but not by using guids. We can set up prefix and suffixes per environment in config and tie in build numbers, blah blah blah. This is just an INTERNAL problem." He continued to tell me that the financial department needs to look up these transactions and investigate them, etc, etc, and sometimes you have to just tell people to deal with crappy software and push the button again and ignore the errors in the log. He proceeds to tell me I am trying to solve a problem which is internal only and has nothing to do with the business, and by making the proposed change I am involving the business and will freak people out when they see a guid appended to the order id and separated by an underscore.

Again I am dumbfounded and ready to strangle and maim. While I agree that guids are ugly, this mentality is the reason teams and projects fail - the fear of collaboration and the fear of being wrong. This problem is indeed a business problem. It was always a business problem, as we are wasting resources and time managing a problem that could be fixed in a day. I can't even begin to calculate the time and money wasted on this poor implementation. With no other evidence I could easily say this stopped being an internal problem the moment we coupled the details of our internal systems to a third party vendor service.

Moreover, I actually walked down the hall and TALKED to the finance dept and people who do the work. They have never in fact investigated any single transaction, but would like to be able to search on the order id, rather than type or copy a guid into the vendor's application. I agree and this is a completely logical request, so I go back to my desk, log in to the vendor's system, and verify that the search is intelligent. When I type the order id into the search field and hit enter I am returned the correct records even with the appended Guid. The guid is saved into a separate ExternalReferenceId column, and hidden from every part of the application - it will never be seen except for in the vendor's system, and then it can be ignored. No receipts or summaries changed in any fashion. I verified everyone was ok with the change, and 6 hours later our system was rid of this dependency.

No more managing ranges of numbers, no need to manage environment based prefix/suffix combos in config files, no need for a new employee to spend 3 hours troubleshooting a known but not relayed issue, no need for anyone to have to remember this when prepping a new employee, no need to make unsupported claims about the needs of others without talking to them, no need for builds to get and reseed new databases, no need to ignore and not translate known cryptic messages, and most importanly, no need to lose faith and undermine the morale of all using the system by telling them to just 'hit the button again' and see if it works.

I am not purporting that this was the best or only solution, but it was a solution that didn't create more waste and in which everyone had a say. Software is built by people for people. Losing sight of this is a losing situation for all.  All problems are business problems. Solve them.

Yeah, so I haven't posted for over a month.  I have excuses though: I had a new release go out the door at work, studied for and bombed a midterm, discovered Dexter, mostly finished a bathroom remodel, and was sent to an Azure training workshop in Redmond. It has been a crazy month.  I guess I can always blame Willy, but that is just a copout. 

I flew into Seattle three weeks ago, and I was supposed to drive down to Vancouver, WA, and visit a friend of mine who had recently moved out there.  When I arrived, it was of course nasty rainy, and decided to not make the trip, as I had to study for a midterm and get some work done on the release we were working on for that following Thursday. I called him up to cancel, and turns out he hadn't actually arrived in town yet, and was still in fact driving the UHaul accross Idaho - this seemed great to me since although I wouldn't see my friend, I wouldn't feel that bad because he wasn't there yet.  Then I received a text a few minutes after the conversation - "Goonies never say release and school."

You see, the other reason I was visiting him was to drive to Astoria, Oregon - home to the Goonies and the only state in the lower 48 I haven't visited west of the Mississippi, (and the only others are in the opposite corner in New England).  I've always wanted to see where they filmed the Goonies, and it is Astoria, so I figured why not kill three birds with one stone and take a drive this trip. And then I decided to bail.  By the time I received this text, it was almost 1pm, and I was at the hotel in Redmond.  The likelihood of being there again any time soon was not good, so I said F*@# It! and headed out the door.  Three hours later I was enjoying a frosty beverage at the very cool Rogue Brewery pub on pier 39 in Astoria debating my strategy for plundering Willy's treasure. After I finished my tasty stout, I continued my tour around Astoria, climbed the column, checked out the town, and was headed over the bridge up the coast on my way back to Seattle.  The sun finally came out and I had a wonderful afternoon. 

And I bombed my final.  Not due to the road trip.  It was in fact all of about 8 hours start to finish and I was back at the hotel by 9pm.  No, I failed my test because I couldn't remember fairly basic algebra.  As my very bright and talented Iraqi professor tells us in during class, "Math is like a bag of tricks, and the more tricks you have, the more you can do."  Well, my bag is really quite empty these days. I haven't felt quite this dumb in a long time, and I was a Math minor! Granted I have not had a math class in nearly 12 years, but when I started the Master's program, I didn't think I would have a problem. Well, I have run through some new exercises trying to help my wife through her Math Accuplacer, (she is going back for her Master's in Education), and now that she has passed that,  I will be running through her "Teach Yourself Algebra Visually" book front to back to play remedial ed.  Talk about a humbling experience. Sheesh. Pulled out my calc books as well. It is going to be a fun summer...

- Don't tell them you can get five developers in India of the same quality or better for one of their salaries...

- Don't tell them that the recent exit of talent is a "net positive," especially if you have not had much direct contact with the team (two or more degrees of separation)...

- Don't remain silent. Don't ignore their concerns because you think you know everything - we know you are wise to be where you are and we know you have reasons. Please give feedback rather than say nothing. Please attempt to help your team understand your decisions - transparency is key to trust...

- Don't tell them "4 Weeks!$!%# I used to be a programmer and I bet I could do it in 2!"...

- Don't tell them object oriented programming is a bunch of hullabalhooey and is largely unnecessary. (If they are working in oo languages anyway)...

- Don't make ridiculously [false] general statements without qualification - i.e. "well, we cancelled retrospectives because no one thought they were valuable." Please take action when 15 people appear before you claiming the opposite of your statement...

- Don't tell your senior staff who the right candidates for new hires are without listening to the people doing the work. Don't tell your team to hire the most techically qualified senior devs available, even if they are not team players nor will ever share the team's principles. Don't tell them that this is ok as long as they will write their code as an api that you can just consume and never have to actually interact with the developer...

...Or don't pretend you are Agile. Or that you have the faintest clue.

</rant>

Working agreements are simply a set of protocols for how a development team will behave. I like when a team explicitly defines and posts working agreements in a common area.  Not everybody does - I've seen decent teams that think it is too fluffy, and gets into "personal issues" which tech people tend to ignore.  I have also seen once good teams descend into madness because they couldn't deal with personal issues or resolve even the simplest conflicts.

I think all teams should have a defined set as a backup, and I think they aid in many areas:

• Finding new candidates - if you have them defined and a new candidate refuses to agree, well than the decision is easy - no hire.
• Getting rid of the rogue developers - if there are people on the team that simply refuse to follow the rules, kick them off the team. Having these defined will help your cause.
• Team identity - a common set of rules and principles goes a long way in defining what your team wants to be and aids in developing trust.
• Conflict resolution - if the team says two people have to review the code as part of defining "done done" and Phil merges his source to the trunk without a review, it is much easier to point at the wall and say "what the hell Phil?" than it is to approach Phil out of the blue (at least for most people)

The first time I used working agreements was when I was on a team for a new project where the team was formed from the individuals of three different legacy subsystems. The systems were in different technologies, and the backgrounds of the developers was very different.  We could not come up with many things that everyone could agree on, but we started with 4 agreements, and we were off and running and almost immediately the team had an identity - albeit small - but better than non-existent. At each subsequent retrospective we would check in to make sure the set was still applicable, and update accordingly.  When I left we had about 10 agreements in place, which was definitely progress. 

I am currently in a situation where I am trying to get my team to establish a set, which  I think will help us for three reasons:

• We are integrating with another local team, and I think this will help for the same reasons mentioned above
• We have a team in India - a great, but siloed team, and I think this will help us start integrating more closely
• The team's background is widely varying - I'll talk about this in my 'Teamicide' post - and their is a pretty visible division between Agile supporters and the other half that think we are crazy (we think you are crazy too). We need to close the gap.
• We are rebuilding - trying to replace about 5 team members

Without further adieu, here is a list of where I like to start with possible agreements. Of course, the set of agreements is never set in stone, and should be revisited at every retrospective to make sure it is still in line with the team. These are neither comprehensive nor prescriptive and are highly contextual - also, they are in no particular order. One last note - from my experience if you don't post them where everyone can see them, they will be of limited value.

My preferred minimum set of agreements:

• No private vetoes
• Code coverage minimum for unit testing
• Work in progress limits -set so that there is at least one person always pairing if possible, and preferably everybody. (Although I have yet to work in a 100% pair environment, I always get more out of work when doing so.)
• Meeting guidelines (start on time, end on time, have agenda)
• Disciplinary actions - how do we hold each other accountable - donuts anyone?
• Code review guidelines - principles to follow, completion deadline, number of reviewers, etc
• A separate coding standard defined
• Definition of 'done done'
• builds - everyone owns the build system, you break it you fix it, and don't leave builds broken
• how and when to take on technical debt and how it will be re-payed

I also like to have the following:

• Process for resolving design conflicts - i.e. three people feel very strongly about how to do something, so how do we resolve peaceably
• Defect resolution guidelines - When a bug is found, always write a test that proves the existence of the bug. Think about the type of bug and ask whether the issue can appear somewhere else.  Always leave a new test or tests to make sure the bug won't pop back up.
• Unit test guidelines:
  
  • which mocking framework to use, if any
  • role and use of integration tests - i.e. guidelines for how to incorporate db tests - transaction scope, clean db per test, etc
  • favor state based over behavior - when not to mock (most of the time)
• always include QA in design discussion - always ask "how are we going to test this" while designing.

What agreements do you like to have in place?

Every so often I'll start a new position and upon arrival find that it is very difficult to get the entire system running locally on my machine.  I am not talking insane enterprise systems even - the last two positions have been fairly similar in architecture, consisting of 2-3 web sites, 2-3 databases, a set of services, and a couple of queues. This isn't terribly large. There is usually out of date documentation on how to get the system running locally, but some times not even that. This does not bode well for the first day warm fuzzy, and generally is foreshadowing the state of their deployment and CI practices.
 
Flat out - I think it is an essential practice for developers to use their own sandbox while developing. This is nothing new, but seems to be overlooked continually. This practice has many benefits, including minimal time between code and test, isolated testing, and easy debugging if required, not too mention all the developers on the team generally understand more of the sytem when this is easy to do and practiced. When I see that this is easy to do, I also get the feeling that someone actually thought about the architecture in terms of how they were going to test it.
 
To run a local sandbox, this would mean coding and testing against all components locally - all sites, services, and databases.  On my current team, everyone has done this at some point in time, but only about half do it consistently.  The reason for this, and at most shops, is that it can be a daunting task to keep a local environment in check.  To do this effectively, all code bases must be kept up to date, compiled and deployed locally at will, and all config settings must be set to run locally. All data repositories must be scripted and quick to deploy, and not necessarily required in my opinion, but nice to have, is a nice set of accurate seed data. (I'll post on that topic separately.)
 
This is how I choose to work, and is not how everyone needs to work, but I think if you try it you will not want to work any other way.  In the spirit of eliminating waste and to aid in making this always possible, I create local build scripts that will allow instant get latest and compile on all local code bases, a method of runnning the sites and services instantly, and a solution for the nightmare of config management (specifically .NET app and web config files when dealing with many environments).  The config management post will have to wait another couple of weeks, but here is the first entry in this series.
 
Local Build Scripts
I create a set of build scripts that anyone on the team can use - of course the dependency exists that all paths are local from a source root. Since I mostly work in .NET, mine are created using MSBuild and I  provide a set of batch files as runners (you can use powershell if you prefer).  You may kick them off from the command line using msbuild, with your favorite hotkey or command-line runner, or by clicking on the batch files.
 
I use SlickRun as my command line runner, so this is the method I choose, and around which I built what I currently consider my optimal solution (which of course changes twice a year or so).  For example, if I want to get latest on the WPF client for our current product and build it, all I have to do is hit Windows Key + W, then type "build client" into the window, and voila, done.  To start the client, all I have to do is hit Windows Key + W, and type "run client" into the window. Now I have a freshly compiled version of the latest source running on my desktop without ever opening Visual Studio.  Then I just continue testing my service code change, or whatever it was I was doing. 
 
About the build scripts
Each build script will get latest for its source, and its dependencies, i.e. Tools, Library, common, etc. where needed.  The default build is incremental, and the source is not cleaned - output folders remain for each subsequent build. I have the option to 'rebuild,' which simply means nuke everything and get a completely fresh copy and build from scratch. Each build is captured in its own log file created in a sub folder ".\Logs" should any debugging or errors be encountered. The build window will also remain open waiting for the user to 'press any key' to exit, so no quick close with errors.
 
As for the scripts, I include one each for each solution which represents a major component (i.e. website, service, db, etc.), and one for each database deployment, and one full script, which will get latest on everything and build the entire system.  I usually only run the latter as incremental, but can do a full rebuild - this is where you want to get a refill on the coffee.  In the spirit of CI I tend to run the latter at least twice a day as well. Also, in the latter, I tend to not deploy dbs by default, just get latest and run an incremental build for expediency's sake.
 
Using the build runners
There are at least four ways to run the builds:
1. MSBuild from command-line
2. Batch file from command-line
3. Batch files from explorer
4. Command Utilities

MSBuild from command-line
You can always run MSBuild from the command-line on the project file, which is just annoying but useful, and you will have to add your own logger if desired.  The default target and param set-up will work just fine by calling the project directly without any additional command line params.
 
Batch file from command-line
You may also just run any of the batch files from te command line.  In addition to a batch file for each project, I created a master build batch file which will run any of builds by accepting the project name as a parameter.  The master batch file is BuildLocal.bat. Call this file passing the name of the build file to run, excluding extensions. i.e.
">BuildLocal.bat client" will run client.build.proj with logging.
 
Batch files from explorer or hotkey app
Double click the individual runners for a build. You cannot use the master with this method, but must click on the build you want to run.  i.e. Clicking on Runners\BuildClient.bat will get latest and build the client.  If you are using a hotkey app or command-line runner app which does not support parameterization, you can assign these files as the target.  i.e. My keyboard has 5 hotkeys, and I could assign the hotkey "1" to the BuildClient.bat, and all I would have to do is hit the "1" key. There is only a target which will be run when the key is executed.
 
Command Runner Utilities
Use a command-line runner which allows passing params - these are like the built in Windows runner (Windows Key + R), but with more power.  I use SlickRun. SlickRun starts anytime I press "Windows Key + W." I use this to run just about everything on my pc - you can get more info at their site.  SlickRun supports "magic words," which are simply defined shortcuts for executing something. I set up a magic word, "build" which when run calls the bat file "Runners\BuildLocal.bat" which also takes a single parameter, which is the name of the build file up to the ".build.proj" extension.
 
So, typing "build client" will run the "build" magic word using "client" as its param, which in turn just executes the BuildLocal.bat file passing the value "client" as described in option 2 above. This is just faster. 
 
Caveats
I am not capturing the output from the tfs get operation to fail immediately if there are conflicts.  I plan on adding a feature that will detect this issue, then attempt to auto-resolve the conflicts, and then fail if they cannot be auto-resolved.  I generally only run into this with configs, which we'll deal with shortly, and when there is crazy refactoring involved - and then it is on check-in, so we are already in Visual Studio.  I also have a script I use that will blow away my local copy of source (full source clean) and then gets a fresh copy from source.  I will do this if I am getting conflicts on a project in which I have not worked for a while.
 
Next up -> Running the applications locally without Visual Studio.
Coming Soon -> Local config management (Actually, all config management)

Other reading:
- http://www.agiledata.org/essays/sandboxes.html
- http://codebetter.com/jeremymiller/2005/09/21/database-per-developer-and-environment/#

So we lost our dev manager back in December. We have been slowly trying to find a replacement, and have seen some good candidates.  One good thing about this is that how often is one in this glorious position where you have the ability to choose your direct superior? (Or at least provide insight into the process.)  In this event, what kind of questions do you ask someone to make sure you want them to be your boss for the next few years? What qualities do you want the candidate to have - what would you want to get out of the relationship in terms of coaching and guidance? What kind of manger do you *think* you would, or at least try, to be?

I'd like to know why she chose management. I'd like to know what she thinks about the industry and how she arrived at her current mindset.  I'd like to know what her management style is and why it is so?  Who was the worst boss she has ever had, why was the boss so terrible and how has she avoided the same pitfalls?  I'd like to know what she thought was absolutely the worst idea and then came to find out it wasn't such a bad idea having walked in their shoes for a while. At the same time I'd like to know who she admires and how she tries to emulate that person.

I'd like to know what kind of team she wants to manage? If she was a dev again, what kind of team would she want to be on? Are they the same? I'd want her to describe the worst person she had ever worked with, what she disliked about that person, and what she learned from their mistakes.

I'd like to know if she plans on developing at any time and how much knowledge she wants of the system. Will she veto design decisions, and if so, based on what? I'd want to know how she feels about slack. I'd like to know what she thinks about taking on technical debt, who has the responsibility for taking it on, and how she plans to pay it down once accepted.  I'd want to know her thoughts on the importance of project vision and how she would attempt to manage divergent visions when encountered.

What are her feelings on Agile and practices therein? What is her approach to testing? How about lean thinking - does she understand and apply lean principles (eliminate waste, create knowledge, build quaity in, defer commitment, optimize the whole, deliver fast, and respect people)? How does she describe the team roles (if using scrum), product owner and scrummaster in particular. What are her thoughts on a self organizing team? What working agreements would she subscribe to?

I'd like to know how she would describe the perfect sprint and release - what that even means to her - and some examples if it has ever happened.  I'd like to know the same about her worst failures - what could have been done differently and how they worked to make sure it didn't happen again. I'd like to know what format she prefers for retrospectives and the value she sees or does not see in them. I'd like to know what she is doing to actively improve herself and how she pushes her teams to do the same.

We also just shed the majority of our team - all through quitting - no layoffs.  I'll save that story for another post, but I guess I'd add the following to the interview: How would she stop people from wanting to leave their position? How would she attempt to motivate a team which is flailing? How would she deal with massive turnover?  How would she start trying to grow a new team? What is her idea of an ideal candidate? How would she attempt to grow trust amongst the team? And of course, the milllion dollar question - what is her approach for integrating an offshore team?

When she doesn't know the answer, what does she do?  What is she doing for her own personal development? Does she blog? Does she actively follow any blogs, journals, or newsletters, and if so - whose? What was the last book she read having anything to do with her professional career?

Of course, I think many of the above questions are not her responsibility at all. I think the onus lies on the team for most of the above, but the team needs guidance and coaching.  I think that hearing answers to the above should provide great insight into the candidate's thoughts and abilities.  My ideal candidate would probably not have answers to all of these questions, and if she did, I hope those answers were highly contextual. I would love her to say she doesn't know but would be glad to look for answers with the team.  After all, that is what it is all about right - inspect and adapt?

What would you look for?

So this weekend I was dropping off yet another round of stuff at the local ARC thrift store, and not getting much time to myself these days, I decided to stop in for a few minutes. Not only did I find a gently used Melissa and Doug double-sided easel for my girls, I scored what appears to be an unopened Domain Driven Design by Eric Evans in pristine condition for $6.99!

While I have read the distilled version available in pdf form from InfoQ, I have always wanted to dive into the full version and see what nuggets of wisdom I missed. And now I can. Lucky me!

I simply couldn't bring myself to title this blog entry "Hello World." You will have noticed however that after fighting the temptation I gave in to at least saying it in the first sentence. I wonder how many first blogs are titled hello world by developers. I suppose the reason I don't want to use the phrase is that it seems so cheap to just type it.  There is a certain sensuality seeing the words output to the screen or written to a file while trying out a new language or technology that typing 11 key strokes just does not deliver. Regardless, the spirit of the phrase is what I want to celebrate here: the anticipation of discovery and the joy of learning something new.

This blog is serving a few different purposes for me. I want to:

1. Have a regular outlet - both constructive and for an occasional rant (which I promise to limit even though I've been told my rants are rather enjoyable).
2. Share the knowledge I have.
3. Learn from everyone else, should you be inclined to start a dialogue.
4. Improve my communication skills and clarify my thoughts.

So before we start this journey, a few words about me.  I moved to Denver in 2004 from Chicago. Prior to the move I worked on mostly small web sites and software prototypes for small businesses. I also did quite a bit of tech support and sys admin work, not to mention bartending, catering, and managing coffee shops and outdoor retail stores. The move to Denver really was the start of my technical career, but all my previous work travelled with me in the form of human factors - usability, teamwork, and management practices. I have had the good (and bad) fortune to work at 8 companies [including 6 startups] and on about 20 different projects since I moved to Denver. The projects have operated under the guise of many different methodologies including RUP, good old waterfall, and various flavors of Agile. I have participated in both successes and failures, and some successes where the companies still failed.  I say good fortune regardless of the failures because I have met some extremely talented people and have learned more than I would have ever expected by seeing what not to do. While all of the startups (and a couple of the bigger players) are now gone, the relationships remain, as well as an abundance of material to write about.  Even setting up this blog today I outlined the first 16 entries.  

When it comes to creating software, I am all about getting things done - but in a clean, collaborative, and well tested manner. Philosophically and in practice I am definitely an agilist and consider myself a burgeoning software craftsman. When it gets down to it, I absolutely love what I do. There are not many careers where you can get paid to solve puzzles all day. In this blog I am going to write about agile practices, teams, and my idea of great software. I will explore issues we are facing every day and problems I am working through. I am hoping to place a great deal of emphasis on what it is to be a great team, as this is the most important thing I have arrived at thus far. Great teams deliver great software. This rings true every time - the only successes I have witnessed all started with a great team.

As of right now, I am committed to writing at least one entry a week. Hopefully I will deliver more right out of the gate, but I am a full time software engineer at a shrink wrapped multimedia publishing software company, I am pursuing my MS in Computer Science, and when not working I am usually chasing my amazing 23 month old twin daughters - so we shall see how it goes.  I hope you enjoy reading this, and I will try and deliver great content for discussion.

Cheers.