VICI: A Software Development Project

[ocratato]

I thought I might create a thread to document a moderately complex program I have been working on.

The main aim of the thread is to show what is involved in creating a non-trivial program. We often get members that are just starting out on a programming career and get a bit carried away once they have done anything more complex than Hello World. I would like this thread to show that software development is far more than just writing some code, and that writing large programs need not be overwhelming.

The target program is one that I have been contemplating for many years but work commitments usually got in the way. I have often considered it to be a waste that the command line programs, like sort, are almost invisible from the GUI desktop environment and we occasionally get threads on these forums from users wanting a simple way of automating some actions. The project aims to remedy this by creating a GUI program that can be used to create simple scripts.

I think that the first step in any project is to create a vision statement. This should aim to describe what the project is about and hopefully enable some one to determine if the software would be of any use to them. Think of it as the first draft of the home page for the project.

Comments are, of course, very welcome, and I hope we can have some interesting discussions along the way.

[dobbi]

Good idea, it may prove to be instructive.

How do you intend to prototype that?

[ocratato]

Good idea, it may prove to be instructive.

How do you intend to prototype that?

Prototyping is an interesting issue. There are three different sorts of prototypes that could apply to this project.

The first is a prototype designed to elicit the user's requirements. It is hard for users to understand what is being discussed without something to look at so sometimes you need to develop something just to demonstrate the concepts. My view for this project is that such a development would be more trouble than it is worth. You would have to almost build the entire thing as part of the prototype. My intention is to develop something with sufficient flexibility to allow user's requests to be easily incorporated.

The second use for prototyping is to understand the components being used, such as how to do drag'n'drop using the Qt libraries. I find it quite difficult to do a detailed design without seeing code actually doing stuff, so there will be prototypes for the detailed design phase.

The third is the project itself. I intend for the project to attract other developers, but it is necessary for them to have something to work on - you cannot start a SourceForge project with just an idea and expect others to sign up. So my first working example will be the prototype for future development.

[ocratato]

Establishing a Project

On some of the projects I have been involved in over the years simply establishing the project was a major undertaking. A couple of them probably involved Cabinet level discussions by the national government. Often it will involve board meetings and such to get approval to proceed.

Large projects will start with the appointment of a Project Manager and some form of allocation of a budget. From there it will need to acquire some office space, IT infrastructure and support personnel. Most of the large projects I was involved with had finance officers and other managers to try to keep the project on track.

Fortunately for VICI we are only a relatively tiny project so we can avoid all of that overhead

My usual start is to create a directory structure:

vici
|-- bin
|-- docs
|   |-- images
|   `-- papers
|-- include
|   `-- vici
|-- lib
|   `-- vici
|-- logs
|-- share
|   `-- vici
`-- ws

I created a vici subdirectory under include, lib and share so that when the project is finally installed our libraries etc are less likely to collide with the installed stuff and we can worry less about what names to use, and also avoid having to put prefixes on the names.

The ws directory is where our source will go - ws is for Work Space which is what Eclipse calls its parent directory. We will have multiple Eclipse projects once we get to the coding phase.

The docs directory is where most of the focus will be for a while. The papers subdirectory is for downloaded articles and academic papers on the subject, plus any specifications etc from external sources. The images directory is just a convenient place to store the diagrams and screen shots that will be going into our design documents.

My development process tends to create quite a few documents so one of my first steps is to create a nice template for the Libre Office documents - see attached.

The next step is to do some preliminary analysis of the problem.

[ocratato]

Preliminary Analysis

In this step we find out as much as we can about the problem. An example I like is where you are asked to provide a display cabinet for the living room - what do you do ? I would check out the local furniture stores and any on-line catalogues I could find to see what sort of designs already exist. If one seems to fit the need then there is no need to build our own. Similarly with software development - there is no need to build something if it is already available in the repositories.

If there wasn't something that could be purchased then I would look into the sorts of timber, the fasteners, the surface finish and construction methods. I might also want to investigate appropriate wood working tools.

For this project I used Wikipedia to find a list of Visual Programming Languages and read about each one. If one was an exact fit to the Vision Statement then it would be "job done", otherwise I was interested in both what was good about the product, and what was bad. I also found some academic papers on the design of visual languages.

One of the tasks in this step is to define the boundaries of the project. In this case one edge is the interface to the command line programs, so a review of what programs might be involved and how they are invoked would be useful. Another edge is the user interface which in this case will be a flowchart GUI. I chose a flowchart as it is something that a novice user is more likely to be familiar with. We can make a stab at defining a visual scripting language so that we have something to base our ideas on. One of the advantages of the visual approach that is repeatedly mentioned is the ability to observe what the script is doing is a great benefit to the user.

Another interface that is important is to the desktop environment. The user needs to be able to install their script so that they can run it just like any other program.

The aim of the Preliminary Analysis is to avoid surprises - like "Why did you not just use XXX?" The more we can find out about the problem the less the chance of later embarrassment. Attached is my attempt at the analysis for Vici.

Next we will decide on a development model.

[ocratato]

Selecting the Development Model

The development model defines the overall structure of the development process. You have probably heard about terms like Agile, Test Driven Development or Waterfall – these are types of development models. I have a blog page that outlines my view of the different models: http://ocratato.blogspot.com.au/2011...t-process.html

When I was starting out on my career the Waterfall model was assumed to be the only possible approach. It had been seen to work for some small projects. Unfortunately it does not scale up well to the gigantic government projects attempted in the 80's and 90's. The idea is that you divide the process into four phases – Analysis, Design, Implementation and Testing. This fitted quite well with the simple project management tools of the day, and gave some inexperienced project managers unreasonable expectations. A lot of projects foundered in the first phase since it was a requirement that a full analysis should be undertaken before moving on. Of course there is no such thing as a “full analysis”, and thus the term “analysis paralysis” was coined, as projects spent huge amounts of time trying to develop a complete understanding of the problem domain.

The current hot topic is Agile development. This is basically my Incremental model with some management guidelines. This is appropriate where you have close collaboration with the customer. For Vici that is not really the case, at least not yet. One problem not obviously addressed by Agile is where a large amount of infrastructure needs to be built to support the functionality.

One of the Agile method's aims is to prefer working code over documentation. This is definitely a good idea where you have a small team working together. Unfortunately in a more distributed environment – over both space and time – it is not such a good idea. You need a bit of documentation to explain why things were done the way they were. The source code will tell you exactly what the program actually does – but not necessarily much guidance on what the programmer intended to happen.

For Vici a Test Driven Development model is probably the most appropriate. We have a fairly static set of requirements to start with, so the Incremental or Agile approach is not necessary. There is no large amount of infrastructure to build so the Spiral model is not applicable. The thing to be wary of in TDD is that it can devolve into a “debugging into existence” approach where the code is just a mess of functions that satisfy the test cases rather than having a well thought out design.

Next we will look at Requirements Gathering.

[ocratato]

Requirements Gathering

The aim of this task is to create a list of things that the system should provide once it has been completed. A lot of small projects neglect this step and suffer from an endless feature creep as new ideas are included. If you actually have a target that you are aiming for then it is considerably more likely that you will hit it.

For many projects there are a couple of other tasks that are preliminary steps to this one.

One common task is to build a prototype. Many users are unable to visualise what a system will be like unless they can see something solid. A prototype, even if it just screen mock-ups, will make an excellent basis for discussing the proposed system with its users. A danger is that the prototype can evolve into the final product – this should be avoided as you will not have a proper underlying design.

Another common task is to model the existing system. Often a software project will aim to replace some existing system. The existing system might be a manual paper based system, or it may be another software system that has reached the end of its life for some reason. The purpose of the model is to capture a detailed list of all the functions and data in the existing system – it is embarrassing when, after spending a lot of time and money to deliver something for the users to say, “That's very nice, but how do I do this important feature of the old system?”

The requirements for a system fall into three categories: Functional, Environmental, and Quality.

The functional requirements are simply those things that the users expect the system to do. This is usually the list that users will provide if they are asked to list the requirements for a system.

The environmental requirements documents those things that are often just assumed about where a system will run. It might be a statement that the system must run on a Red Hat Linux server, for example.

The quality requirements are actually the more important in determining the design for the system. These will guide us in creating the high level, or architectural, design. Attached is a document I created in another project that lists every quality attribute I could find – we can use this is a guide to select those that are important for Vici.

Next we will document the requirements for Vici.

[ocratato]

VICI Requirements

The requirements for Vici follow from the Vision Statement and the Preliminary Analysis. For other projects we would also include what we found from doing a prototype and from modelling the existing system. The aim is to try to capture everything that we will want the system to do. This can be very complicated and time consuming on a large project.

In real world projects, the requirements will often be written by the client who may have little understanding of what is or is not practical. For large projects they may be written by several different groups – I once had the misfortune of being involved in a project to create a unified human resources system for the Defence Department – many of the requirements were deliberately ambiguous so that everyone could agree on them. In another project the requirements were overly specific – including actual version numbers for some associated software, despite the fact that it would be well and truly obsolete by the time the project was to be delivered.

It is useful to give each requirement a unique identifier. We can then put them into a database and later be able to run queries to confirm that no requirements have been overlooked. It is also useful to give each requirement a priority so the design process can weight conflicting responsibilities.

The important trick with building a set of requirements is to say what you need the system to do without unnecessarily locking out design solutions.

The quality requirements cover not just the quality of the operational programs, but also the quality of the source code, design documents and development process. For example, a requirement is that the VICI system shall be constructed in a modular manner with the separate components capable of independent testing. This is not something that relates to the executing programs, but rather to the design of the system.

Next we look at the Requirements Analysis.

[ocratato]

Requirements Analysis

The usual purpose of this task is to ensure that the requirements make sense. Often they will be prepared by the client, and perhaps involve multiple departments of the organisation. The document can easily contain contradictory requirements or requirements that are too specific, or alternatively, ambiguous.

It is useful to get these problems resolved before you start design work.

In addition to highlighting any discrepancies, inconsistencies and omissions the task should involve the creation of a set of use-case diagrams. This gives everyone a solid description of what the system should do and how it should behave. It will also demonstrate to the client that the development team has got a good understanding of the proposal.

For Vici we can skip straight to the use-case diagrams, at least for this initial version.

[ocratato]

Project Management

Before we begin the design process it might be worth a few words about managing a project. Obviously, for this stage of Vici there is not much management necessary, but for most projects it is unfortunate fact of life.

Managing a project has four primary tasks:

Progress Control – most projects have some sort of budget based on time and cost. It is good management to try and keep the estimate of how much work remains on the planned track. To do this there needs to be some estimate of the time and effort required, and some means of measuring how much of the work has been accomplished.

Quality Control – typically this will involve setting up a development process that promotes the production of a high quality product. The test driven development process we are going to use for Vici is an example of such a process. It will also include reviews of documents and code, and tracking defects uncovered by testing.

Change Control – on large projects that take a significant amount of time to develop it is inevitable that the requirements will get changed due to either outside influences (e.g. changes to legislation) or changes to the way the business operates, perhaps as a result of the very software we are developing. Changes can also be necessary to the design or the code as problems are discovered during development. Keeping these changes under control is a significance task.

Knowledge Management – this task involves ensuring that everyone who needs access to information can locate what they need. Design documents are no use if they are locked away in some private directory structure. The task must also involve keeping the documentation up-to-date and ensuring that changes are appropriately distributed.

Next we will begin the design process.

[ocratato]

Architecture, Part 1

The first step in the design process is to decide the tactics we are going to use to try to satisfy the requirements. For the functional requirements this usually amounts to acquiring some software to perform the function. For the environmental requirements it usually becomes some constraint on the options we may select from. The quality requirements, however, are much more interesting at this point in the process.

For example, lets say we have a requirement to upgrade a system so that it can process 100,000 documents per day. We know that the existing system is taking about 10 seconds to process a document, and there are only 86400 seconds in a day. Unfortunately we cannot just add a “go faster” function to satisfy this requirement.

There are several strategies that can be used: we can get faster machines; we can do less processing; or we can distribute the processing over multiple machines. In this example, which is a real, the machines had already been just upgraded, so it would have needed some hard evidence to justify additional expenditure on new hardware. Most of the improvement came from doing the processing more efficiently by swapping out interpreted code for compiled code. We also split the processing into a chain of tasks which could be distributed across multiple computers. In the end we estimated that we could have handled twice the required number of documents.

When you are doing the design you will find that there are often several different tactics that can be applied to achieve a particular requirement. However, it will often be the case that tactics from different requirements are in conflict. The job of the designer is to select the best set from the alternatives.

The attached architecture document for Vici just lists the tactics that I have chosen. In a more real world project it would list the alternatives and provided some analysis that shows why the selection was chosen.

Once the tactics have been selected we can then list the responsibilities that must be implemented in order for the tactics to be accomplished. Many of the responsibilities will be implemented by the software we are going to construct, but there are quite a few others that are assigned to the development process itself. For example, part of Vici is a library of prepared commands – building this library will be part of the development process. Similarly, the requirements for an extensible and modular system lead to a tactic of building a set of modules that can be built in isolation, which in turn leads to a responsibility for creating a modular design. This responsibility is not something that can be implemented in code, it must be assigned to the architecture design process.

In the next part we will assign the responsibilities to a conceptual design model.

[ocratato]

Architecture, Part 2

The second step in the design process is to allocate the responsibilities to modules of the system, or to tasks of the development process.

Allocating responsibilities to the development process tasks ensures that they don't get over looked. In a real world project the project manager would take an outline of the development process tasks, (which may be a corporate standard) and tailors it to suit the specific project.

My blog pages give a slightly more detailed explanation of the tasks:
http://ocratato.blogspot.com.au/2011...ks-part-1.html
http://ocratato.blogspot.com.au/2011...ks-part-1.html
http://ocratato.blogspot.com.au/2011...ks-part-3.html
http://ocratato.blogspot.com.au/2011...ks-part-4.html

I have built this list of tasks from my experience on a wide range of projects. Some of the tasks may seem unfamiliar, but I hope this thread will go some way to explaining why I think they are important.

The important part of this step of the design process is to allocate responsibilities to modules of the proposed system. Getting this wrong will almost certainly spell doom for the project. It has been my experience that most software development failures can be traced back to a faulty architectural design – well, for those that got past the analysis phase.

My approach is to try to group the responsibilities such that they have some common function or purpose in the overall system.

[ocratato]

Architecture, Part 3

This is an update to the architecture document where we allocate the responsibilities to actual components. These may be components we intend to build, or third party components that we are going to incorporate into our system.

Selecting the most appropriate third party components is one of the more interesting tasks when building a system. You will have to balance the cost of commercial products, the openness of open source software, how well they fit your requirements, and how easy they are to integrate into your design. They can result in a system being delivered sooner. They can also result in a system that becomes increasingly difficult to maintain.

I was once given a substantial system to maintain that had about a dozen proprietary components included in its design. Over the years we dropped some because they were no longer needed, some because they were no longer available, and others as we found or built better alternatives. In the end it only had one that we could not replace. Coping with the “no longer available” components was the trickiest, and can easily result in your system having to be abandoned.

For the purists, the latest section of the architecture document is called the Logical Model, but in reality it is an amalgam of the logical model and the physical model. I could have broken the modules down into smaller components and then assigned them to libraries and programs but I chose to combine them directly into one model. I think this is OK for a relatively small project like VICI.

Next we will take a closer look at the interfaces.

[lsatenstein]

Hi Ocratato
Different industries proceed in different ways. I worked in Aerospace and banking.
In banking there was two forms of project launching
a) marketing came along and stipulated "Here is a project that we require". Please review and present an estimated time to implement (optimistic to pessimestic)
or
b) The government put up a new law. We must implement by a stipulated date. Please indicate hardware, software, support, training resources and date that protoype can be tested.

or take commercial (manufacturing)
a) We have a new product and it has some financial impacts to our ERP system
We need costing, service and spares setups, guarantees and a special division for tracking the $$$
Tell us what we absolutely require to get it done by a stupulated date, and provide some what-if's.

Is that what you had in mind?

[ocratato]

Hi Ocratato
Different industries proceed in different ways. I worked in Aerospace and banking.
In banking there was two forms of project launching
a) marketing came along and stipulated "Here is a project that we require". Please review and present an estimated time to implement (optimistic to pessimistic)
or
b) The government put up a new law. We must implement by a stipulated date. Please indicate hardware, software, support, training resources and date that prototype can be tested.

or take commercial (manufacturing)
a) We have a new product and it has some financial impacts to our ERP system
We need costing, service and spares setups, guarantees and a special division for tracking the $$$
Tell us what we absolutely require to get it done by a stipulated date, and provide some what-if's.

Is that what you had in mind?

Yes, for large projects it is the architecture documents that contain all that information, or the information a project manager can use to provide those numbers.

You have highlighted a serious problem with the software development process and how it meshes with the real world of project estimation. In order to make sensible estimates it is necessary to design the architecture of the solution, but, unfortunately there is never enough time to develop the architecture sufficiently - often the time frame for answering those questions is just a week or two - even for major projects.

The result is that the architecture used for the estimate is rushed and often very incomplete. This leads to under-estimates of the work involved and hence the huge project over-runs we all too familiar with. I have seen this happen over and over during my career.

Most procurement systems, especially in government, are geared toward buying something which already exists. The exception is for civil engineering construction, such as major highways for example. Software seems to be treated as a purchase of an existing item rather than as construction, hence the mismatch and consequent cost over-runs. Until procurement bodies start to work with software architects in the same way as the work with civil engineering architects I don't see the situation improving.