Perspectives on Software Architecture

Software architecture is the foundation of Enterprise Software.  The underlying architecture should be well designed and thought through. Regardless of the technology selected for a Enterprise, certain underlying principles guide the initial development and evolution of the software. In this article I will be presenting the principles I have found most useful.

Collaborate

Collaboration applies across all major activities, from organizing and executing, to deployment and maintenance/operations. Collaborate with folks both inside and outside the known systems, and collaborate with folks that know the customer/business side.

Innovation

Creating an environment where innovation can thrive is an important aspect of collaboration.  Brainstorming sessions can be useful when conducted in a open, non-judgmental atmosphere. It is important to separate the communication of ideas from deep analysis of each idea. I recommend having an initial session to capture all ideas, and a separate session for diving deeply into each idea. During the capture session, all ideas are welcome, and are not immediately judged. Often this will lead to a chain of related ideas that ultimately lead to solution options. Always be open to improvements in the software architecture. In my experience, incorporating good ideas into the software architecture leads to a stronger architectural foundation.

Continual Improvement

Like Collaboration, Continual Improvement applies across all major activities. Utilize Effective Retrospectives at end of each sprint or activity to continually improve team performance.

Organize

Organizing is often a very challenging part of constructing software. Poorly organized initiatives often leads to poor architectural foundations. It can be quite complex to organize a software construction initiative. At a high level:

  • Research domains inside and outside system
  • Select appropariate tools/frameworks
  • Plan execution
  • Attract stakeholders and collaboration
  • Include retrospectives in planning and execution for a continual improvement feedback loop
  • Proactively identify unknowns and challenges and develop options to mitigate

Vision

Form a vision of where the architecture is evolving. Use the organizing phase as a tool for gathering the information needed to develop the appropriate vision for the organization. The vision should be continually refined. Start at a high level and work your way down, then work down and back up to verify the integrity of the design.

Execute

When breaking down the initiative into tasks, insure that each task takes no longer than 2 days max, if it does break the task down into smaller sub/dependent tasks.  Organize efforts into 1-2 week sprints. Conduct a demo and retrospective at the end of each sprint.

Sustained

Take an iterative approach to building out slices of functionality. Monitor progress and adherence to architectural principles. Utilize code reviews to train development team on best practices and preferred software patterns. My article Beyond Code outlines many useful activities related to sustained software development.

Transform

If you are working with a outdated legacy system, map a realistic path of transformation. A useful framework for working with legacy systems is based on an article written by John Boyd, titled “Destruction and Creation“. The basic concept is to break a system down into smaller pieces (Destruction), look for synthesis and integration, build up a new system outside existing domain concepts (Creation), test new creation by validating you can trace the concept back to it’s original domain. Rinse and repeat until successful validation.

Architectural Characteristics

Many folks use terms for software architecture goals, however to achieve architecture goals it is important to clearly understand what each term means at software level, and be able to communicate this to different levels of the organization from developers to CEO. And of course translate the goals into a solid architectural foundation of code.

Modular

Emphasizes separating the functionality of a program into independent, interchangeable modules, such that each contains everything necessary to execute only one aspect of the desired functionality.

Scalable

The capability of software to handle a growing amount of work, or its potential to be enlarged in order to accommodate that growth.

Extendable

Systems design principle where the implementation takes future growth into consideration. It is a systemic measure of the ability to extend a system and the level of effort required to implement the extension. Extensions can be through the addition of new functionality or through modification of existing functionality. The central theme is to provide for change – typically enhancements – while minimizing impact to existing system functions.

Maintainable

Maintainability is the ease with which a product can be maintained in order to:

  • Isolate and correct defects or their cause
  • Repair or replace faulty or worn-out components without having to replace still working parts
  • Prevent unexpected breakdowns
  • Maximize a product’s useful life
  • Maximize efficiency, reliability, and safety,
    meet new requirements
  • Make future maintenance easier, or
    cope with a changed environment

Layered Software Approach

Layered Software Approach is a very useful framework for organizing, categorizing (layer), and selecting appropriate software patterns for a modular, scalable, extensible, and maintainable architecture. Derek C. Ashmore’s does a excellent job detailing this approach in his book The Java EE Architect’s Handbook. His blog, The Java EE Architect’s Blog, is also an excellent resource. I have experimented with many of the concepts discussed in his writings, and I have found them to be very effective and useful.

Rest API

Front end systems can change at a very rapid rate. Decoupling UX layer from the rest of the system enables this layer to be extendable. A REST API helps achieve this. If you decide to utilize the REST architecture style it is import to understand and honor the architecture properties and constraints this style describes.

Automated Testing

The principles discussed thus far build a solid foundation for constructing software systems. Automated testing is critical for long term continual evolution of the architecture and underlying modules. It is important not to overlook this from the very beginning of a new system. For legacy systems there are many techniques to explore to introduce automated testing. In either case, it is important not to over look robust automated testing, as this supports ongoing quality, extension and evolution of the system.

Additional Resources

Scrum for Personal Productivity

Are you an agile veteran that now works at a company that doesn’t practice agile? Or maybe you work in an agile environment and have observed how effective it is. Embrace agile as a personal task and performance management system! Integrate core agile principles into managing your work load. Begin by adding existing tasks to a backlog. Going forward add any task that comes up to this backlog. Once a week, plan the next sprint by reviewing and prioritizing  items in the backlog. Make sure you align the backlog priorities with company, team or family priorities. Now determine how many items can you can make progress on in the next 1 week sprint. If an items is not completed in a single sprint and the item is still has priority, push it forward to the next sprint. During the sprint, start each day reviewing the items in the sprint. Check off the completed items completed and decide the focus for the day. When a sprint is completed, conduct a retrospective to review your performance and consider what you can improve on. Track these items and take action in the upcoming sprints to put the improvement in place. You can track all this in a simple spreadsheet with the following individual sheets:

  1. Current sprint – Your daily scrum is to review this first thing every morning.
  2. Backlog – Add any task that comes your way to the backlog.
  3. Retrospectives – Conduct effective retrospectives weekly to improve you personal performance.
  4. Sprint archive – At the end of a sprint, copy the current sprint to the sprint archive to a maintain a complete history of work accomplishments.

I have another sheet to capture mission statements, goals, strategies and tactics. If you’ve had positive experiences using agile at work, you understand how effective it can be. Consider organizing yourself using agile principles. Give it a try! I have found it to be the most effective personal management technique I have ever used, and I have tried quite a few.

Comments?

Effective Retrospectives

A key, often overlooked component of agile is effective retrospectives. I recently read Agile (micro)Management, and I realized how retrospectives are not only the key to continual improvement, but when conducted thoughtfully and transparently, they lead to improved engagement in the agile process.

Although I personally prefer the Stop, Start, Continue, More of Less of wheel method, there are many effective retrospectives techniques. To improve the effectiveness of agile, know the available techniques and test them out in your organization. And remember, retrospectives are the key to continual improvement.

Keys to Retrospective Success:

Consistency
For effective team engagement, as well as for continual improvement, retrospectives should be conducted regularly.

Inclusive
Retrospectives should include all agile team members.

Transparent
Items raised during retrospectives should be discussed in a positive, constructive manner. All team members should be aware of the ultimate conclusion. Items should not be swept under a rug.

Followup
Items that the team agrees should be addressed should have clear action items, that are followed through on. If the same item is raised at every retrospective, confidence in agile will slowly erode.

Retrospectives are one of the most critical aspects of effective agile implementations. Make sure you not only conduct regular retrospectives, also insure they are effectively conducted. It will give everyone greater confidence in agile, and it will insure that continual improvement is occurring.

At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly. – Principles behind the Agile Manifesto

Software Development Leadership

Leading software development teams present many different challenges. Over the years I have come to realize that there is a core set of principles related to successfully leading software developers, principles that I will share in this post. Many of these principles are not specifically related to managing a team of software developers, but are more about leadership in general. I encourage you to develop your own set of core principles, because to be truly guided by them you must own them. I have found that innovation flows from creating a positive software development environment, and so this article is ultimately about creating the type of environment where innovation, results and team development flourish.

Core Principles

  1. Optimism
  2. Enthusiasm
  3. Integrity
  4. Perseverance
  5. Organization
  6. Empowering
  7. Collaborative
  8. Communicative

Communication

Communicate clearly and directly; do not wait to correct undesirable habits, address these habits as soon as they are observed. Praise developers when they exhibit desirable habits, never focus only on the negative. If a developer has only bad habits, this needs to be addressed right away and if not rectified the developer may have to be dropped from the team.

Integrity

Be open, honest and transparent with your team and never deceive them. Take responsibility for your mistakes and learn from them, always demand the same of your team. Be honest and transparent with other groups you interact with as well. Remember that you often represent your team to other groups, and it is important that you represent the team well. Be transparent in all your dealings; always put the team above yourself. On a rare occasion, you may be faced with a situation where putting your team first conflicts with company plans. The team is depending on you to represent their best interests, so you must be willing to fight for them. However, never compromise your integrity. Check your gut and let your core principles guide you, if fighting for your team does the larger business serious harm, then direct your efforts more towards the proper treatment of your team. For example if the team is dependent on a certain product line that is slipping, and the company decides it can no longer afford to support the product, fight for the product and try to find ways to make it more profitable. However, if you realize that the company is correct to abandon the product, focus efforts on trying to find other products or groups within the company that your developers can move toward.

Optimism

Always maintain a positive attitude, even when faced with crisis and difficulties. Your team will take their cues from you, if you are negative this attitude will certainly spread and infect others. A challenge is an opportunity to succeed, this is how you should look at every difficulty that arises.

Guide, Mentor, Motivate and Develop team

Talk to developers about their goals, both long and short term goals. As a leader do all you can to help then realize their goals, even if their goal is to one day have your job. Build their personal career goals into your plans for that developer, and do what you can to help them progress towards these goals. If the developer has unrealistic goals, let them know. This of course can be very difficult, but it needs to be addressed. A developer with unrealistic goals will not be happy working on the team. But be fair, an unrealistic goal is not a goal that is too high to achieve, it is a goal that you believe could be harmful to your team or the company, and therefore you absolutely cannot support their goals. Identify future leaders and delegate more responsibility to them.

Clearly communicate expectations

Let folks know what is expected of them. Low performers on the team may not be clear on the team’s expectations. Often these individuals can improve by clearly communicating expectations, and giving prompt feedback when they are not performing to expectations. To develop a high performance team, it is important to address and improve low performers. It can be demotivating for the team to see others on the team that are not making a positive, constructive contribution.

Marketing your team

Make the rock stars on your team visible to the rest of the organization.  Put the team before yourself. When mistakes are made that affect other teams, take responsibility and communicate it as a team problem that you will remedy. When dealing with the issues within the team you can address individuals responsible for the mistake, however make every effort to avoid throwing the individual under the bus publicly.

Building Relationships

Work hard, but reserve some time for fun. It is important to keep a sense of humor, and talk about common interests unrelated to work. This helps you develop a rapport with those that you lead. Empower developers and give them a chance to design and be creative in their work. Often leaders that are particularly bright and talented coders want to micro-manage design and code. Resist this urge and let the developers be as creative as possible. If you think there are issues with a proposed design or coding approach, by asking questions you can often lead develops towards a better approach while still giving them the freedom to think on their own. Build up your developers; do not tear them down unless you sincerely feel that it is warranted. If it is warranted, be blunt and honest, make your point clearly and without fear, but be careful not to do permanent, destructive damage. Always keep in mind that your primary objective during these times is to avoid a repeat and teach a lesson that will stick. Ultimately your objective is to help the developer improve and to reach new professional heights.

Conclusion

I have outlined my philosophy about leading software development teams, but I’m always seeking to learn and improve. So please comment with your thoughts and feedback!

Beyond Code

Software development processes for successful software

Successful software. Yes, it exists today. And it is not an accident. It typically occurs because software development teams think beyond the writing code. They consider the wider software release world, including:

  • Project Management
  • Release and Configuration Management
  • Quality Assurance
  • Production Release
  • Production Performance, Maintenance and Operations
  • Customer Support
  • Continuous Improvement
  • Creative work environment

When software development teams seriously consider these items when designing, developing and releasing projects, time is not wasted, other groups develop an appreciation for the development team, and iterations/releases go much more smoothly. Much of what I outline here is common sense to many, yet time after time I see development teams and individuals neglect this. Additionally, I often see development teams that have good intentions, but lack the time. Often external groups fail to see the benefit of the processes discussed below. It is my hope that this article helps development teams as well as other groups within a company recognize the benefit of these practices.

Development process

Requirements review

First step is to understand as clearly as possible what it is you need to build. The process for gathering and documenting requirements varies greatly from company to company. Whatever process your company utilizes, there is some form of communication to the development team regarding what is needed. It is critical that the development team has a clear picture of what is needed. Sometimes this will take place though several release iterations, but the point I want to stress is that when there are uncertainties, a developer needs to do the leg work necessary to resolve the uncertainties. A developer must also have an audit trail preserved as to the resolution. Verbal resolution is usually not good enough. You want proof that this was the resolution as it could be forgotten. I’ve seen this happen hours before a production release.

Design review

Once the development team has a fairly clear understanding of what is expected, the task of designing the implementation begins. Again, this varies company to company, from detailed design specifications to informally explaining the planned coding approach to a lead developer. This is an important part of software development as often problems can be surfaced at this point, which is still early in the process. This saves critical time and money later in the process. Don’t waste the time of other functional groups by releasing substandard code. It is preferable that the individual(s) that review the design have system wide knowledge and experience.

Code review

Once the design is agreed upon, the actual coding and unit testing can begin. Once this nears completion, a code review should take place. A code review can often surface issues early in the process. Catching issues in before release to a QA team saves time and money.  It is important not to just review the code, but to look at the code in a wider context. The following areas should be covered:

  1. Content of code changes (new features, bug fixes)
  2. Review of source code
  3. Review of testing
  4. Review of any supporting documents, such as release notes and operational guides

I recommend utilizing a code review template to capture any action items from the above areas. The code review section of the document should capture all comments and suggestions from participants. This is important part of fostering an open, creative and innovative environment. Note every item needs to be addressed, however every item should be captured. The participants can must decide if the item must be included or if it can be deferred.

Documented test plan

During the design and coding phase, notes need to be kept that detail the testing of the feature(s). These notes will then be turned into a formal test plan that is submitted to QA on or before initial release to QA. This formalizes the development to QA communication, and is an important document that is often used by production support teams. In some organizations the QA group will create the formal test plans, however the development team should still create some form of development test documentation.

Automated build process

What ever build tool your team uses, the build scripts should be rock solid and easy to reproduce. Configuration data tied to local developer workstations need to be externalized. Scripts should be created to both build and package the release. The packaging of the release needs to take into account the production installation image, and should replicate this image as close as possible. Once the scripts are created, development teams should continue to seek out opportunities for improvement. Often you can build testing scripts into the build process which lead to greater and greater quality of the basic release artifacts. When you are ready to release to QA, it is important to go through the process two complete times. First, the development team should make sure all the code for the release is checked into source control. The designated developer should check out the code, build and package the release. Once this is complete the release artifacts should be closely inspected. If the developer is satisfied at this point, label the source, totally clean (delete) all project directories and files (make a back up first). Now you are ready to go through the process the second time, only this time you check out the source from the label you created the first time. Make sense? This ensures that no artifacts on the developers machine were not in the source control tool, and it also guarantees that the release can be recreated.

Source code management

Of course source control should be utilized to support new releases and recreate existing releases. I personally use a branch by task strategy. This is where every feature/task has its own branch created. When release time comes, tasks for the release are selected and merged into an intermediate integration branch. If the integration branch passes development testing, the integration branch is merged back into the main code line and the release is cut.

Release management

By having your build process replicate the production install image as closely as possible, you make the hand off to Release/Configuration Management much easier. By thinking of the production image of your software, you anticipate and account for the process that will lead to a successful and pain free production installation. If questions come up when you are considering this, talk to Release Management, they will certainly appreciate it and will usually be more then happy to help work out these issues.

Quality Assurance

Your development processes need to be solid enough to avoid unnecessary QA cycles. When a release is in QA, make yourself available to the QA staff to help clarify any questions. This will eliminate bugs being logged that are really only questions, and it will help isolate points of confusion to person to person communication. This is usually better then exposing the confusion to a wider group, which can make the development team look as though they have not thought everything out, which in turn leads to lower confidence in the development team by external groups. This also helps you to improve your process so as to avoid the same confusion in the future.

Project management

This is the area where software development team often have the biggest challenge. I have seen development teams present realistic estimates and that are immediately rejected. Often the team is forced to revise the original estimates to meet project management time lines. I recall a manager presenting a development schedule that estimated 8 weeks of development. The response from the project manager was a “no way” and “for these requirements” and “did you really read the requirements”. Eventually the software development manager agreed to revise the estimate to 6 weeks. The scope of the release was not changed. At this point, timelines were communicated a wider audience, including the end user community. When the project actually took 8 weeks to complete, the project manager was genuinely confused why it took 2 weeks longer then the estimate, questioning the team’s ability to estimate projects. Software development estimation is critical to software success. You must provide reasonable estimates and stick to them. Do not compromise unless something of equal value, like a dropped feature or added resource, is agreed to. Otherwise you must make it clear that the quality of the release may be compromised if the schedule is indeed pulled in. My motto is “Under promise and over deliver”. In the end your team will look much better being 1 week early then 2 weeks late. At the end of the day most companies will come to realize the benefit to both internal teams and external customers of delivering on time, and an important aspect of this is properly managing expectations and estimates.

Customer Support

This part of the organization is critical. Poor customer support leaves the impression of poor software, whereas excellent customer support will give your customer confidence and leave the impression of quality software, or at least a culture of customer focused continuous improvement. The software development team can help create a first class customer support group by careful attention to the items outlined in this article. Good documentation and prompt attention when customer support requires more expert guidance also will make a huge difference, so don’t ignore this aspect of the software process.

Managing the knowledge that is created by all the processes defined above is an ongoing and critical activity. Developers need to be knowledge management experts. Wiki’s are idea for managing this information. Note that it is critical to continually update whatever knowledge repository you use. Do not let the information get stale!

Continuous Improvement

Software development teams need to continually measure the results of their processes and improve wherever they can. The processes outlined within this article are not static. They need to be adjusted for your organization, measured and improved upon. Continually. It takes a real commitment to a holistic software design and development discipline.

Creative work environment

It is important to foster a creative and innovative environment. The team should always be open and encouraging to ideas and suggestions that improve processes, policies, and software. It is often helpful to have periodic idea storm sessions were everyone is comfortable making suggestions. It is often helpful to capture all suggestions, and not discuss the merits at an idea storm session. This will encourage folks to speak up without fear of having defend the idea. This can be a fine line. It is very helpful to discuss the actual idea, as others often speak up and help refine the idea, thereby making it better. However, all participants should refrain from judging any ideas.

Conclusion

As you can see, each aspect of the development process takes into account external functional groups. The processes defined above thus lead to quick iterations and confidence within external groups. This in the end is critical to the success of software. Quality Assurance, Release Management, Project Management, Operations, Executive Management and in the end the customer, all benefit from solid development processes. They appreciate the attention your team paid to quality, performance, release and operational issues. They are as busy as you are, so don’t waste their time!