Agile Hurts
It was about 5 years ago that I first got slapped by agile. In that time, I’ve been in several purported agile environments, from pure high velocity environments to full blown faux-gile environments.
It takes me time to really internalize things such as agile. At this point in time I feel like I get it (don’t we always feel that way?). And here is my big lesson: agile is painful, and it’s supposed to be. A quick followup to that: agile isn’t a solution, it’s a mindset that will lead you to a solution.
This may be stating the obvious, but the processes falling under agile such as scrum and kanban drive out the pain points into the open. Extending that idea, you can’t homogenize a process across the board because each team’s problems are different. Perhaps more importantly, to successfully introduce agile into an organization requires everyone having the courage to face the problems that surface.
The biggest failure I see is ignoring these problems and not allowing the team to fix them. If you can’t trust and empower your team members, maybe you hired the wrong people.
Embracing Agile on a Fixed Price Project
Agile and Fixed Price
I’ll be honest, I was nervous. I’ve been developing in agile environments for several years now, and the thought of a fixed price project was frightening. It doesn’t seem like the principles of agile are conducive to a fixed price project. Customer collaboration? Continuous feedback? Responding to change? These concepts sound like they were designed to scuttle a fixed price project. But this is what the client wanted, and I was asked to lead the effort. Sure we had a change control process in place, but that was just going to break momentum in the long run.
Not only was it fixed price, but it was a new team, tackling a new-to-us codebase, and a new technology with a steep learning curve (WPF in this case). Did I mention that nobody on the team was involved in the time estimate? What we had was a list of stories to complete, and an incentive for finishing early. Likewise, there was a dis-incentive for finishing late.
The End Result
I take a lot of pride in the work I do, and I love delivering a product the customer loves just as much as I love a beautiful code solution. Delivering on time seemed impossible, much less delivering a product the customer loved.
It turns out we delivered a four month project three weeks early, and delivered a great solution that the customer loves. I was as surprised as everyone and have been reflecting on how we achieved it.
The Keys to Success
When it comes to agile, there are many principles to live by. The following items are what I tried to focus on.
- The simplest thing that could possibly work. Our UX guy probably lost several years off his life after all the great ideas that had to be left behind. Build the simplest thing, minimum viable product, maximize the work not done, whatever you want to call it.
- Continuous feedback from the customer is great. But you’ve got to realize you cannot deliver everything they want, or even everything you want. When you are under a fixed price, you must deliver what is spelled out. There is nothing wrong with telling the customer what is possible, and in fact we would do this all the time. But know that without change control you just can’t do it.
- Be ruthless. Hopefully your contract has spelled out stories and not detailed specs. With stories, you have more latitude to deliver the simplest thing possible, and ever better, deliver the right solution to the problem. But you must be ruthless. Anything out of scope is out of scope. It’s best if you do this as nicely as possible.
- We never had any change requests. To be honest I’m not sure if that is good or bad. Maybe we over-delivered. I think if we HAD gotten change requests, I would have liked to tackle them after the initial delivery. I think the momentum loss from the change request would outweigh tackling a change while things are fresh. That may seem a little backwards, but I feel that in a fixed price project you must stay focused on delivering.
Final Thoughts
I won’t lie. The key points above may have had nothing to do with our success. It may have had everything to do with having a great small team (3 devs and 1 UX) focused on a common goal. But if there is a take away, it’s that it is possible, and agile and fixed priced are not necessarily at odds.
The Beauty of the Open/Closed Principle
It’s good to be SOLID. Of the principles that make up SOLID, I’ve found the Open/Closed Principle (OCP for short) to be one of the more difficult. There are plenty of explanations around the web, but while identifying the presence or lack open/closed is easy, it’s tougher to achieve.
I also use OCP from the micro to macro scale. It can be applied to classes or it can be applied at a module/library level. Most material on the web covers the easier class level OCP. I’d like to cover the latter using some changes made recently to the Moe state machine library.
The Problem
On my current project, I encountered a great place to use a state machine to simplify a bunch of code. The usage of the state machine itself was interesting and I hope to post about it soon. From a Moe standpoint, I found that the way state actions were implemented didn’t suit my needs. First, I didn’t want to centralize enter/exit actions, particularly at the time of construction. Beforehand, you would add entry and exit actions when you built your state machine. This would mean they were all in a single chunk of code, or you would have to pass the builder around. Not ideal. Second, I also wanted to swap out these actions on demand. When the state machine wasn’t in use, I wanted to disconnect state actions so windows didn’t appear based on wayward events (ie, a late arriving error event from a webcam we were using).
The Solution
I revisited Moe to add the functionality I wanted. What I envisioned was a convention for state actions. Any class could have methods following the form “OnEnter” and “OnExit”. These methods would auto-magically be invoked upon entering and exiting states. These classes could also be added or removed as listeners at any time. I decided to call this functionality state delegates.
While working on Moe previously, I had done a lot of work to support the OCP and this can be seen in the Moe.StateMachine.Extensions library. In that library you can find several helpful additions for Moe, such as logging, state watchers, timers, and asynchronous state machine execution. I specifically wanted that functionality to exist outside of the core state machine. Would the hard work of that refactoring pay off and allow me to add state delegates?
The Changes
You can see the changes that were made in this commit. The solution was to add a traffic cop that would wire up to every state and intercept state enter/exits. When one of those happened, the StateDelegatePlugIn would run through the current listeners and delegate to any methods matching the state name.
This huge addition to functionality (in value, that is) was achieved without touching the state machine at all. This is the OCP on a larger scale. We’ve added valuable functionality to a library without modifying the library itself.
How to move towards OCP
It’s tough, if not impossible, to produce code supporting the OCP from the beginning. Before we can abstract, we need to SEE at least a few examples of usage first. If you were to go back and look at past commits, you’d find several that are focused on refactoring to the OCP after several functional additions were already in place. To begin with, the functional additions were intertwined with the state machine itself. By unwinding those dependencies, I was achieving OCP through actual usage. It would have been too much for my brain to try and see the resulting patterns and needs without already having multiple examples.
When I do something like this, I’m looking to make core classes as simple as possible, keeping as close to the Single Responsibility Principle as I can. Something that I should have done earlier was introduce the Extensions library. By forcing myself to move functional additions to a separate assembly, I would have no choice but to support OCP. When thinking of simplifying or SRP, think of the grand Unix commands. Grep, sed, sort, and so on. Have your components do one thing and do it well.
Is MVC the best we can do on the web?
Recently I’ve been thrust back into web development after a 5 year hiatus. Sadly I’ve discovered that not much has changed. We’re still forcing an inappropriate pattern onto web development.
Why not MVC?
The pattern we all know and love has been great for many years, and certainly in a thick client environment, it and its brethren seem appropriate. In these cases, the UI’s are often being developed by the same folks that are developing the M and the C. With web development, this is often not the case.
On my first significant web project back in 1999, the workflow was a designer delivering HTML files, and then a developer hacking that into a JSP/Servlet combo. This results in your controller and views being tightly coupled together. What happens if the designer needs to deliver completely new screens and workflows? My guess is you’d have to throw out the views and the controllers and start again. How diligent is your team at keeping the controllers thin and free of business logic? Most of us know that’s a good practice, but how often have you seen that violated?
Introducing a new TLA: SVF
Service, View, Flow. The last one is Workflow, but W just has too many syllables. Let’s partition a new way.
Service
A service takes data in, and returns data out. A service might fetch a customer or a customer’s orders, or it might create a new order for a customer. This has been common in software development for some time now, and it continues to make perfect sense in web development. Nice, simple, reusable, beautiful.
View
A view should only require some data, and be able to render that data. Imagine a view template that specifies the data it needs. A Customer. A list of orders for that customer. The template specifies it needs that info be specifying the services needed. An incoming web request/get doesn’t go to a controller, instead it goes straight for the view. The view will specify what it needs. It will work to render itself.
What about performing operations? Let the form or the link specify a target service and bind data to be passed. Now the designer can compose the views however they want, without impacting code much, if at all.
Flow
Flow, or workflow if you like, is the traffic cop. The designer can specify much of the workflow through the views, but occasionally some more complex logic may be necessary. In this case, your workflow can be a simple codified state machine that can determine the next view based on some logic.
What does it mean?
Developers can focus on building small simple services, designers can not only build the views, but actively maintain them over time (as opposed to painful markup merges or outright rewrites). We are minimizing the development of “throwaway” code, and in this case I’m referring to throwaway as code developed for a single purpose of gluing/coupling components together. In the realm of MVC, Controllers are essentially throwaway. To some extent the Views are too, since they are so tightly coupled to their controllers.
To visualize, imagine web services, with HTML documents that interact with those web services directly. All we need is a framework to tie them together.
An example
It seems that Twitter style micro-blogging is the Hello World of web apps, so let’s see what this might look like. We’ll take a look at a user’s home page, which will provide a form for a new micropost, as well as a list of our own posts. Such a page will need two primary services, one for the User, and one for the user’s Posts.
The View
@Fulfill User = User.Current
@Fulfill Posts = Posts.ByUser(User.Id)
<html>
<head>
<title>Hello Microblogger World</title>
</head>
<body>
Here, we see a couple template markups in our invented template language that are binding variables to services. First, the service identified by User.Current is bound to the variable User. Likewise, the service Posts.ByUser is passed a parameter (the current users ID), and the results are bound to the Posts variable. These are pre-requisites for displaying this view.
Next, we’ll have a nav bar that includes links to various other views (not controllers, not actions, but the actual views).
<nav class="round" >
<ul>
<li><a href="@View()" >Home</a></li>
<li><a href="@View('Users')" >Users</a></li>
<li><a href="@View('Users', User.Id)" >Profile</a></li>
<li><a href="@View('Users', User.Id, 'Edit')" >Settings</a></li>
<li><a href="@View('Help')" >Help</a></li>
<li><a href="@Service('User.Logout', 'Login')">Sign out</a></li>
</ul>
</nav>
Most of our links are pointing straight at other views. We are maintaining RESTy-ness with the links that would be generated by @View. The final link for signing out actually calls a User.Logout service, and specifies Login as the followup view. By doing this, the designer can maintain control of workflow instead of the developer coding this into a controller.
Now let’s see how we would handle posting a new microblog message.
<h1>Tell me what you're doing:</h1>
@ServiceForm('Posts.Create') {
<textarea cols="40" id="newpost" name="@Bind('message')" rows="20"/>
<input id="post_submit" name="commit" type="submit" value="Post" />
@}
We’re going to bind the HTML form to a service called Posts.Create. The textarea element is bound to the input parameter on the service, “message.”
Finally, we’ll iterate the user’s own messages
@ForEach('Posts') {
<tr>
<td class="gravatar">
<a href="@View('Users', User.Id)" >
<img class="gravatar" src="...."/></a>
</td>
<td class="post" >
<span class="content" >@this.Message</span>
<span class="timestamp" >@this.Timestamp</span>
</td>
</tr>
@}
We build a link much like before, and we iterate over the posts with the @ForEach tag.
The Services
We can cleanly build our services now. In this case, we need Users.Current (which would handle the authenticated user), User.Logout, Posts.ByUser, and Posts.Create. These services are very general, and once built the views can be endlessly changed and redesigned and reused without further code necessary.
class PostsService < Service
def by_user
user_id = request[:user_id]
... fetch return posts for user_id
end
def create
user_id = context[:user].user_id
message = request[:message]
... create new post for user
end
end
It’s interesting to note that these services are practically web services if the inputs and outputs were transformed from/to JSON or XML.
Getting with the Flow
For the sake of discussion, we’ll say that our Help views can be initiated from anywhere in the application. The help contains numerous individual views to assist the user, but when the user clicks a Done button, they should return to the screen they were on when they first clicked Help.
class HelpFlow < Flow
attr_accessor :view_stack
on_viewing(:help) do
view_stack.push(request[:source_view])
end
on_viewing(:done) do
redirect(view_stack.pop)
end
end
Changing roles
We are changing the roles and responsibilities now. Developers are focusing on developing code to process and return data. Designers are focusing on building the user experience. While offloading some responsibilities from developers, we are adding some for the designers. Communication will be necessary to work out what services will be needed, what data needs to go in and what data is coming out. In other words, the developers and designers will be coming up with an API between them.
Just today the value of this became apparent. The team lead and the designer were trying to figure out how to refactor the partitioning of the views. It turns out that partial views and layouts had been pulled out by some developer along the way to suit his purposes. Unfortunately, how he had partitioned them was damaging and resulted in considerable duplication. If the designer had been wholly responsible for views, they could have properly partitioned the different headers, footers, and so on in a logical manner that would minimize duplication.
Crunch time and Theory of Constraints
I’m continually amazed at the wastefulness and lack of focus I often see in software development. Worst of all, the problems seem to intensify as one gets closer to the deadline. It doesn’t have to be this way if you are focused on continual improvement.
A colleague of mine that I have tremendous respect for turned me on to Eli Goldratt and The Goal: A Process of Ongoing Improvement
Crunch Time
Anyone who has spent time in software development has experienced crunch time. You’re up against a tight deadline, and there is more work than time. The closer you get to the deadline, the more chaotic things get. You have X number of features that must go out, and they are all top priority. If a feature has to be done, it is of COURSE top priority. Along with all the other required functionality. How can we maximize chances at success?
Step One: Identify the constraint
This can be tricky, but very often it is development that is constraining the system. Coding is a time consuming process. For this post, we are going to assume it is the development that is the bottleneck.
Step Two: Exploit the constraint
No, this doesn’t mean treating your developers unfairly. It means utilizing your resource to the fullest. Remove any functionality not absolutely necessary. Re-evaluate targeted functionality for the simplest solution that could possibly work. Prioritize work, even if priorities are arbitrary. When the human brain encounters 10 different #1 priority tasks, nothing is going to get done. By prioritizing, you let development focus on individual tasks.
Step Three: Subordinate every other decision to the constraint
Have you ever been in crunch time and found yourself doing something that is unrelated to your goal and is wasting your time? As an example, if I’m a developer under the gun to get functionality developed, preparing release notes and responding to bug reports via some cumbersome bug tracking system is wasting my time. Find another resource that can handle these things for me so that I (the bottleneck) can focus on my goal. Don’t hand incomplete requirements to your developers that will require them to stop working to clarify.
Step Four: Elevate the constraint
Elevating the constraint is tough during crunch time. In this case, elevation could involve adding people or faster machines for the developers. Neither of these is a good idea when up against a deadline. But outside of that scenario, the above should be considered. Particularly computers. It absolutely baffles me the salaries organizations will pay developers and leave them with 2 or 3 year old machines. Spending $3k on a faster computer is a no-brainer.
Step Five: Repeat
Chances are that if you repeat the above steps, at some point in time your constraint is going to shift. Most likely, it will shift to your testing crew. Repeat the focusing steps to continually improve what you do.
Cross Functional Teams and Systems Thinking
Cross functional teams have been promoted by agile methods for years now, and it can be hard to fully appreciate their value without having experienced the tremendous value of such a team. The same could be said for much of agile practices, but I want to focus on cross functional teams.
What are cross functional teams?
A software development team that is cross functional might include developers, user experience designers, testers, production support, and subject matter experts. All of these individuals work together on a team towards a common goal. This differs from traditional methods where each specialty is in a silo, interacting with other specialties at specified gates or milestones.
Ok, so where is the value?
Having experienced a very successful, cross functional team led me to several conclusions. First, it improves communication among the people working towards the solution. When a developer has an unclear requirement or an alternative idea, it’s natural to turn to a team member who can clarify the requirement or refine the idea. If a tester encounters what might be a bug, it’s a simple matter of turning to the developer to work through the issue. This eliminates all the red tape (in the form of bug tracking) and waste and takes care of the problem quickly. And this arrangement allows for constant collaboration in solving problems, which not only solves problems more quickly, but typically results in a superior solution.
Recently I’ve been involved with a team that is not cross functional. Specifically, developers and testers are firmly segregated. In this organization, a build is handed off at the end of a sprint. Testing will only be conducted on functionality that is mostly complete. The intent is to minimize retesting, but this goal is misplaced in my opinion. Retesting shouldn’t be the worry, the testing inventory that builds up should.
In thinking about the problem, I realized a significant benefit cross functional teams, and it is that the approach forces systems thinking. Systems thinking, in its simplest form, is consideration of an entire system and how the components interact and affect one another. It’s simple, when divided into different groups, to focus on local optimization. That is, when I’m a tester thinking only about testing software, my apparent goal is to test software and report bugs. As a developer, my goal is to crank out the spec’d software.
It’s too easy to lose sight that our goal is to deliver solutions, or in agile terms, working software. Actually, that isn’t even quite correct. Our goal is to deliver business value. When we have a cross functional team, we are forced to think in terms of systems, whether we realize it or not. For someone altering requirements, the effects of those decisions are apparent when you are part of the team. When a developer is cranking out too much functionality (oh yes I said it), the effect on the tester sitting next to them will be evident.
We like to talk about constant feedback, and a cross functional team will give you feedback on how the team itself is performing, and how the system is working (or not).
Why the resistance?
Perhaps one of the greatest impediments is territoriality. Managers get queasy about relinquishing their minions to a team. This can even be extrapolated to the fear of empowering teams with decision making, but that is another post.
Mentioned earlier, local optimization is another roadblock. The idea of an employee’s not being 100% utilized in doing their job is heresy. Excess inventory- for example, testing inventory–can cost far more than time spent idle.
More Moe
In the last post, I introduced you to Moe the State Machine. In this post, we’ll explore some more of Moe’s basic functionality, and some of its extended functionality.
Builders and current context
When using the state builder, it is important to understand that the actual builder context will change as you add statements. For instance, consider the statechart at right. We have a superstate that has two children state, ChildA and ChildB. To create ParentState and ChildA, we can use the following code:
StateMachineBuilder stateBuilder = new StateMachineBuilder();
stateBuilder.DefaultTransition("ParentState");
stateBuilder.AddState("ParentState")
.DefaultTransition("ChildA")
.AddState("ChildA")
.TransitionOn("SWITCH").To("ChildB");
The AddState(“ParentState”) moves us into a builder for that state, where we set the default transition to ChildA. While still in this context, we call another AddState(“ChildA”) which makes it a substate of ParentState. The TransitionTo() of ChildA takes us to another context, that of the Transition context. Here we can specify where to transition, and guard clauses. Under certain circumstances, the builder context can shift back to the parent. For instance, from the Transition context, an AddState would automatically move up to the ChildA context (adding a child state to the child state, not what we intend for ChildB).
To get ChildB, we need to start again with ParentState and add another child state. We can get to any state’s context (created already or not) by using the StateMachineBuilder’s indexer like this:
stateBuilder["ParentState"]
.AddState("ChildB");
What if I don’t want to change state?
Transitions can have guard clauses. These are nothing more than conditional transitions. By combining multiple transitions together (and being very careful with your logic), you can use guarded transitions to make branching transitions (going to different states based on some other factor). Each of the three types of transitions supported (regular, default, and timed transitions) can have a guard clause specified with the .When() syntax of the fluent interface. All that is specified in the When method is a Func<bool>. It is up to you, the developer, to specify what the condition. A guarded transition might look like this:
stateBuilder["ChildB"]
.TransitionOn("SWITCH").To("ChildA").When(() => SystemInError());
In this case, we have a method returning a bool named SystemInError. By putting multiple transitions together with guard clauses, we can conditionally branch transitions.
There is no restriction on transitions to states, any state may be chosen, including the originating state. In the case of a loopback transition, the effect will be the OnExit and OnEnter actions will be executed.
You think you’re funny Mr. State? You’re going to Timeout!
A common feature of state machines, in particular HSMs, are the notion and use of timers. These timers are associated with a state, and upon entry to that state become active. After a specified amount of time, the state machine will fire an associated transition. In the Moe sample application, timers are used to signal a chance for the stop light. For longer running operations, a timer can initiate a timeout for an operation that has taken too long.
It is easiest to think of a timeout as a regular event transition that is fired automatically after a specified period of time. A timeout can be set up like this:
stateBuilder["ChildB"]
.Timeout(1000, "ErrorState");
Timeouts may also have guard clauses. In this case, the timeout will fire once, and if the guard is false, will not fire again.
Where have I been?
When Harel introduced statecharts, he included the idea of History states. As history state is a means of exiting some state, and when directed back to that state’s history, to traverse to the state upon last exit. There are two forms of history, deep and shallow. A deep history will go to the last absolute leaf state, whereas a shallow history will go only to the immediate child node. From there, any default transitions will occur leaving you in a leaf state.
Remember, you cannot be only in a superstate, you must always be in a leaf state.
History states are handled specially, here is how you would wire together a deep history, as opposed to the default shallow history.
stateBuilder["ParentState"].AddDeepHistory();
stateBuilder["CousinB"].TransitionOn("SWITCH").ToHistory("ParentState");
Other features
There are several other features of Moe.
- Asynchronous – Found in the Moe.StateMachine.Extensions.Asynchronous namespace, calling Asynchronous() on your state machine object prior to Start() will create a thread for event propagation in the state machine. When would you want synchronous and when asynchronous? Depends on what you’re doing. If you are tracking simple state transitions, workflow, parsing, or UI wizard type activities, synchronous is probably best. If you’re state machine is responding actively to external stimuli, asynchronous is probably the way to go.
- StateWatcher – Found in the Moe.StateMachine.Extensions.StateWatcher namespace, calling AddStateWatcher will take a callback and a list of states. As soon as one is entered, the callback is invoked. This is primarly just convenience.
- Logger – Building event driven systems can be challenging. Sometimes you just want to know the sequence of events and state transitions that occur, and by implementing a small ILogger interface, you can gain the information that might help with the tricky problems. You can attach a logger by importing the Moe.StateMachine.Extensions.Logger namespace, and calling Logger() on your state machine.
