FedoraForum.org - Fedora Support Forums and Community
Page 3 of 8 FirstFirst 1 2 3 4 5 ... LastLast
Results 31 to 45 of 110
  1. #31
    Join Date
    Jul 2013
    Location
    San Diego, CA
    Posts
    10

    Re: VICI: A Software Development Project

    This thread should be turned into a textbook! Having just finished my CS degree, I can safely say that I've had more "Aha!" moments reading this thread than in any of the classes I've taken over the last three years. Thank you so much for this, ocratato!

  2. #32
    Join Date
    Oct 2010
    Location
    Canberra
    Posts
    2,682

    Re: VICI: A Software Development Project

    Interpreter, Part 3

    The main purpose of Vici is to start other programs. There are three classes that are used to manage these child processes:

    Process – this is a parent class that does the processing that is common to both internal commands and child processes. This mostly involves processing the parameters to the command to expand variables and expanding glob expressions into file lists.

    ChildProcess – this handles the creation of the new process. As is normal for Linux this is done by first forking the Vici process and then replacing the new process with the command to be run.

    ChildProcessMgr – this is used to detect when a child process has terminated. Once this has happened the corresponding Process object is updated with the exit status.

    The usual approach is to have the parent instance of ChildProcess record its pid in the ChildProcessMgr so that the latter can find the correct ChildProcess object. When testing this I found that on rare occasions the ChildProcessMgr would report that it had got a signal for a pid that it did not know about. It appears that sometimes the child process gets enough run time to complete before the parent process gets a turn. To prevent this I added a pipe between the parent and child and forced the child to wait until it got a byte of data from the parent. The parent records the pid of the child before sending the data thus preventing the race.

    The code to “exec” the child process might appear a bit odd as it takes some short cuts that would be inadvisable in normal circumstances. It should be remembered that this code is running in the forked child (not the main Vici process) and is about to be overwritten by the new process, hence there is little need to worry about things like memory leaks.

    The ChildProcessMgr uses a “singleton” pattern that has been modified for use in a multi-threaded environment where there is a non-zero chance that two threads might each try to create an instance. The main point of the class is to handle the SIGCHLD signals from the terminating child processes. When a signal is received it is important to loop over the waitpid call as it is possible for one SIGCHLD interrupt to mask another which would lead to missed calls.

    The unit of execution in Vici is not the process but the pipeline. All the processes that are connected via pipes need to be running simultaneously. The Pipeline class manages the creation of the Process objects and the connecting pipes. Since each process can have connections from both stdout and stderr the pipeline is actually more like a binary tree. The Pipeline starts at the first process and recursively adds the processes at its stdout and stderr to build the set of processes that must be run simultaneously.

  3. #33
    Join Date
    Oct 2010
    Location
    Canberra
    Posts
    2,682

    Re: VICI: A Software Development Project

    Interpreter, Part 4 - Threads

    Within the interpreter there is a lot of communication using pipes and other asynchronous communication channels. Linux provides two ways to process these – you can either use blocking read and write calls and threads, or non-blocking calls and the “select” system call. The former is easier to code since with event driven processing it can become quite difficult to understand the flow of the logic of the program (at least that has been my experience). The result is that Vici's interpreter makes extensive use of threads.

    There is a cost associated with using a thread based design in that it is all too easy to introduce race conditions that can cause faults that can be very difficult to resolve.

    There are now some choices for using threads in C++. I decided to use the new STL thread support (mostly out of curiosity). The interpreter uses std::thread, std::mutex, std::condition_variable and std::unique_lock to implement the threads and synchronise their actions.

    I tried several thread design patterns, but eventually decided that a thread pool would be the best approach. The following classes implement the design:

    ThreadPool – this class is responsible for starting and running the threads. It creates as many threads as are necessary at any instant. It does not queue up jobs waiting for a thread, but will always start a new thread if none are available. If we queued up jobs there is a danger that a pipeline would be stalled waiting for data from a queued up job.

    ThreadFunction – this is the base class for the thread function objects that are run by the thread pool. These objects are passed to the thread pool which then executes it within a thread. The thread pool takes ownership of the thread function object and deletes it once it has completed.

    ThreadFnOwner – this is an abstract interface class that is notified when the thread function starts and stops.

    One advantage of the thread based approach is that the interpreter can function independently of the GUI. The GUI can start the interpreter and immediately regain control so that the user interface remains responsive while the script is running.

    Each thread of control within the interpreter runs in a Thread object (derived from ThreadFunction) which is responsible for creating the Pipeline objects according to exit status of the preceding command.

    If a Pipeline is flagged to run as a background task it is wrapped in a Job object (also derived from ThreadFunction) that allows it to run independently.

    The normal commands run in separate processes so they do not need thread support, however the internal commands can be members of a pipeline and have their own read and write calls on the standard pipes. Hence each internal command is also run in its own thread.

    Finally there is the MasterThread which is derived from Qt's QThread class. This is used to interface the interpreter to the GUI as it is able to trigger events in the main application event loop.

  4. #34
    Join Date
    Oct 2010
    Location
    Canberra
    Posts
    2,682

    Re: VICI: A Software Development Project

    Interpreter, Part 5 – Events

    As the interpreter runs the script various events occur that we want the enclosing GUI to be notified about. When the script generates output and when it completes the user needs to be informed. During debugging the user will want to see changes to variables, the current execution point and be informed when a break-point is encountered.

    The GUI cannot simply wait for these events as this would cause the interface to freeze. Further, the events are occurring in threads that are different from the thread running the GUI event loop.

    The core classes that solve this problem are an EventMgr and the MasterThread.

    The EventMgr allows us to queue up Event objects as events of interest are performed. It uses a mutex to prevent objects from being queued or dequeued simultaneously. This allows us to collect the events from the disparate threads into a single queue.

    The MasterThread is a thread class that is derived from Qt's QThread class rather than use the STL's std::thread. The reason being that is also derived from QObject and can therefore use the Qt signal and slot mechanism. The MasterThread object therefore waits on the EventMgr's queue (using a condition variable) and when an event is received it signals the Interpreter which, in turn, calls the GUI code to notify the user of the event.

    One of the features of the visual language used by Vici is that a Display object can be attached to the end of a pipe. The idea is that the GUI will provide a window in which the text is displayed. Hence one of the events that we need to generate is when data is ready to be displayed.

    The solution is to include in the Display object a pipe that is in non-blocking mode. The text to be displayed is written to the pipe. The other end is monitored using the “select” system call. This all runs in a thread so that it can wait as necessary. When the select call returns with an indication that data is available an event is generated and passed to the GUI as described above. Code in the GUI can then read from the pipe and display the text in a window.

    When the GUI finds that there is no more text available it makes an acknowledgement call to the interpreter which is passed back to the Display object. The display object therefore alternates between waiting for data using the “select” call and waiting on a condition variable for the acknowledgement.

  5. #35
    Join Date
    Oct 2010
    Location
    Canberra
    Posts
    2,682

    Re: VICI: A Software Development Project

    Design Patterns

    A useful task to undertake once you have done a design for a complex component is to review the design against well known design patterns. Design patterns are solutions to common programming problems that have been tried and tested and shown to work reliably.

    The book “Design Patterns – Elements of Reusable Object-Oriented Software” by Gamma, Helm, Johnson and Vlissides is a good starting point for patterns, but there is now a large library of patterns – Wikipedia provides a useful starting point.

    While it would be good to design a component by making selections of various design patterns the reality is that you will be spending most of your mental energy on solving the problem. The available collection of design patterns is now too large for most people to have them all in mind while they are formulating the design. My approach is solve the problem first using some of the more well known patterns where appropriate but concentrating on the logic and structure demanded by the problem. It is much easier to recognise the patterns in your solution as you review against the library of patterns.

    Often your initial design will include a part of a well known pattern. You should either fill out the remainder of the pattern or provide some notes in your design document explaining why you think it is unnecessary. Otherwise some future maintainer will spot the pattern and possibly upset the balance of your design by completing the pattern without understanding all the implications.

    As an example, one of the patterns used in Vici's interpreter is the Singleton pattern. The problem is that we need to guarantee that there is only ever a single instance of the object. Things like the thread pool and the event manager must have just a single instance. While a global static instance of the object would suffice it does not provide a guarantee that a second instance will not be created. The solution is to have the object itself manage its instantiation. The usual declaration and implementation are
    Code:
    class Singleton
    {
    private:
    	Singleton();
    	static Singleton *me;
    public:
    	static Singleton & instance();
    };
    
    Singleton * Singleton::me = 0;
    
    Singleton & Singleton::instance()
    {
    	if ( me == 0 ) me = new Singleton;
    	return *me;
    }
    This solution works fine for most applications, but can fail when there are multiple threads involved. It will often be the case that the Singleton is responsible for some system access that takes a while to establish. It is therefore possible that while the first thread is attempting to construct the object that a second thread will find that me is still zero and also begin construction, which will probably lead to all sorts of problems.

    A solution to this that has been proposed is the “Double Checked Locking” pattern:
    Code:
    Singleton & Singleton::instance()
    {
    	if ( me == 0 ) 
    	{
    		acquire lock
    		if ( me == 0 ) 
    			me = new Singleton
    	}
    	return *me;
    }
    This is the design currently used in Vici for those cases where multiple threads may be involved.

    This solves the problem of a second thread attempting to simultaneously create an instance while the first one is busy initialising the object. However it does potentially introduce other problems. It is possible that the compiler will reorder things which could result in me being assigned a value as soon as the object has been allocated but prior to its initialisation. The second thread may then attempt to use the uninitialised object – more chaos will ensue. It would be advisable for the constructor to assign “me = this;” as its last statement to avoid this issue.

    The latest versions of C++ have addressed threading issues, in particular for a static object only one thread gets to initialise it – the others must wait until initialisation has completed. Hence for C++ all we need is

    Code:
    Singleton & Singleton::instance()
    {
    	static Singleton s;
    	return s;
    }
    The design of Vici will need to be updated to use the above pattern.

    One of the tasks of the development process is to conduct a review of the design. However it is often difficult to do a meaningful review of your own designs, which is a bit of a problem for a project with only one developer. Comparing the design to the published patterns may be enough to provide the fresh perspective that is needed.

  6. #36
    Join Date
    Oct 2010
    Location
    Canberra
    Posts
    2,682

    Re: VICI: A Software Development Project

    Object Design

    A common mistake in designing classes is to put too much knowledge of the external world into the object. Consider some objects for making tea – a teapot, a kettle, and a stove. We could model this so that the teapot has a “MakeACupOfTea” method which sends a message to the kettle to boil the water which in turn sends a message to the stove to supply some heat. The problem is that each object has to know about other objects. We could easily end up with the stove having methods for heating kettles, pots, saucepans, etc. Every time we want to warm something up we need to modify the stove class. Similarly the teapot knows about the kettle – what happens if we decide to heat the water in a saucepan?

    The underlying problem is that the knowledge of how to make tea is in the wrong place. A teapot needs to know how much hot water it contains and how much tea it has and perhaps how long it has been brewing. It should not have to know anything about kettles or other utensils. Similarly the stove should know how hot it is and that something has been placed on it to get warmed up, but it does not need to know exactly what.

    For this little story we can solve the problem by introducing abstract interfaces – heatable for anything that can be placed on the stove, and pour_into for anything that can have liquid poured into it. Objects can then interact via the interface without having to know exactly what the other object is. Since both the kettle and the teapot implement the pour_into interface we could even decide to pour the tea into the kettle without having to modify any of the objects.

    We now have a new problem. If the teapot does not know how to make tea – what does ? The answer is to introduce a new class that encapsulates the process – it knows about each of the objects involved in the making of a cup of tea and how to use them to get the desired results. For our story it would be a Maid class. Of course there is a danger that this object will end up with too much of the program logic, leaving the other classes as hollow entities that are little more than data structures.

    I find it interesting to compare the above with the early days of databases. It was discovered that over time the data of a system tended to be much more stable than the processes that used it. This resulted in the idea that when modelling a system it was best to start with a data model and then add the processes afterwards. We do the same when we split the business objects (teapot, kettle, stove) from the business process objects (Maid). The more dynamic nature of the business rules will often lead to a design that allows those rules to be easily changed, perhaps even at run time.

  7. #37
    Join Date
    Oct 2010
    Location
    Canberra
    Posts
    2,682

    Re: VICI: A Software Development Project

    Developers and the Development Model

    One of the subsidiary reasons for the VICI project is to demonstrate that the development process is valid. So far it seems to be holding up quite nicely, and I can see that it would work well for a software development organisation that was focussed on creating quality software. However I am having some doubts about how well it will work in the open source environment for projects that are much larger than VICI.

    The development process is designed to facilitate the creation of large projects. It uses a process of dividing the component modules into groups of smaller modules (recursively) until the modules can be easily developed by a small team (or even an individual). While the modules must be able to be tested in isolation and would be expected to have some functional cohesion it would be generally the case that a significant set of the modules will be needed in order for the overall project to be demonstrated.

    For a commercial development organisation that is committed to the overall project this is not a problem. They can simply assign designers and programmers to the components according to the needs of the development plan.

    For an open source project the developers are not likely to come on board until there is something that can be demonstrated. Very few open source developers seem to be interested in joining a project that has produced nothing tangible. Unfortunately a high level architectural design does not tend to evoke a lot of enthusiasm. We need some way of getting the core element of the project delivered as soon as possible so as to attract additional developers.

    The approach to use when there is a need to develop a lot of infrastructure for a project is the spiral development model. The basic idea is to do the project in several development cycles, starting with a few core functions and enough “quick-n-dirty” infrastructure components to demonstrate those functions, and then adding functionality and firming up the infrastructure in subsequent cycles – we spiral out from a small starting point.

    It seems that the same approach might solve the developer problem if the original team develops the first cycle so that there is something to show. There is the added benefit of having the development environment and techniques well defined before the additional developers come on board – they will have example designs, code and test harnesses.

    So far I have been working on VICI using a single development cycle approach. I have decided to change the approach to a spiral model. To speed things along to an initial working prototype I will drop further develop on some modules – the EBNF, Syntax and Search modules will be left at the design phase for now, and the Installer, Cron and Security modules will be left out of the first cycle altogether. The second cycle will complete the EBNF, Syntax and Search modules, and the following cycles will implement the Installer and Cron modules. I had already decided to leave the Security module for later, in part because the design used some encryption technology that may not be entirely compatible with SourceForge's terms and conditions.

    Time to get on with designing the symbols to use on the flow chart...

  8. #38
    Join Date
    Oct 2010
    Location
    Canberra
    Posts
    2,682

    Re: VICI: A Software Development Project

    Symbol Library

    The latest design for VICI is for the library responsible for drawing the symbols that the user places on the flow chart to represent command, files, variables etc.

    The user interface for the library consists of a toolbar that allows the user to set the font, line style and colours for the symbols, and an area showing an example of each symbol and line type. The idea is that the user selects a symbol from this palette and then clicks on a canvas to place a symbol. You can see an example of this user interface on the Source Forge page for VICI as the currently rightmost example screen grab. ( http://sourceforge.net/projects/ocratato-vici/ )

    There are several approaches possible for doing the symbols. I could have used pixmaps from PNG files for example, or selected the symbols from an SVG drawing. In the end I chose to simply draw them using Qt's drawing primitives.

    Unfortunately there is a bit of a problem with the library as it is currently designed. The original interface design overlooked the need for the user to interact with the symbols once they were placed onto the main canvas. The user needs to be able to select, open and move the symbols using click, double-click, and drag events which are delivered to the symbol object and need to be passed up to the owning chart item. The user may also need move symbols from one diagram to another or remove them altogether. Currently the interface definition does not include these interactions and will need to be amended.

  9. #39
    Join Date
    Oct 2010
    Location
    Canberra
    Posts
    2,682

    Re: VICI: A Software Development Project

    Derivative Programming

    I am currently working on the design for the canvas library that is responsible for allowing the user to enter and edit the flow charts. Progress could be better, but life sometimes gets in the way – I figure you only get one go at life so its best to make the most of it. However there was a feeling of being a bit overwhelmed by the potential complexity this module. A few dozen widgets each with their many properties can quickly multiply into a lot of interactions.

    It is all too easy when coding a GUI to just connect the event handlers directly to the widgets to display the changes of state. The result is that the application state gets encoded into the display widget properties in a rather uncontrolled and diffuse manner. You then find that you have to query other widgets in order to work out how to code the event handlers. The result is another form of spaghetti code where everything is connected to everything else.

    It is also an example of what I call derivative or differential programming where the changes to the inputs are used to directly control the outputs. Frequently the only way to determine the state of an output is to replay the history of all the changes that were made to it. Since widgets can be both inputs and outputs another danger is the “event storm” where changes from one event cause further events that ripple back and forth across the program. It has been my experience that programs written this way are far more complex and difficult to maintain even though they often start out as the simpler approach.

    Fortunately there is a well proven design pattern for this situation called Model View Controller (MVC). The incoming events (controls) are applied to an explicit model of the system which contains the application state. The display widgets then show a view of the current state. One of the advantages of the MVC solution is that it becomes almost trivial to add new views.

  10. #40
    pete_1967 Guest

    Re: VICI: A Software Development Project

    I know this is bit late to the party but something about this project has always been nagging me; mainly that I had a feeling this has already been done (or _very_ close to it). I just haven't been able to remember what, where and when until today I stumbled across it! http://onefinger.sourceforge.net/

  11. #41
    Join Date
    Oct 2010
    Location
    Canberra
    Posts
    2,682

    Re: VICI: A Software Development Project

    One Finger certainly has some interesting characteristics, but Vici will have a full flow chart interface for more complex scripts than this seems to support. I will have a look at the command entering function as it looks quite interesting.

    Thanks for the link.

  12. #42
    Join Date
    Oct 2010
    Location
    Canberra
    Posts
    2,682

    Re: VICI: A Software Development Project

    Diffuse Interfaces

    One of the things that I have noticed during the detailed design work for VICI is that it easy to introduce what might be called “diffuse interfaces”. If some aspect of the system occurs in more than one module then the designs for each module need to take into account the designs in other modules with respect to that aspect.

    The most common example is the GUI interface. If several modules have a part of the user interface and they are designed by different people (or at different times by the same person) then there can easily be inconsistencies introduced in the way the application looks and behaves. It is a common problem on large projects and the usual solution is to introduce a style guide for the designers and developers in the hope of getting a consistent interface.

    An alternative solution that I used on one project was to wrap the general purpose GUI library in our own project specific library. Within this wrapper there was a much reduced range of functionality and choice so that all of our applications had a consistent look and feel. This approach was justified since the underlying GUI library was Motif and putting a C++ wrapper around it gave us a huge boost to productivity (which you will understand if you have ever tried programming a Motif application).

    For most smallish projects, wrapping the GUI is a bit too expensive. However, it can have other benefits, such as allowing the possibility of switching between GUI toolkits depending on what is installed on the user's machine which will give your application the same look and feel as the other desktop applications.

    Within the design of VICI another diffuse interface has emerged – the built in commands. These are hardly mentioned in the architecture document, but feature in several of the modules. The interpreter has to implement them, but the canvas and symbol libraries need to provide some capabilities that depend on that implementation. The solution in this case will be to extend the architecture document to include a detailed description of the interfaces to the built in commands.

  13. #43
    Join Date
    Oct 2010
    Location
    Canberra
    Posts
    2,682

    Re: VICI: A Software Development Project

    Increment Plan

    As mentioned in a previous post I have decided to split the development into multiple development cycles, known as increments – we will incrementally approach the final solution.

    To manage this I have introduced an Increment Plan document that briefly describes what each development cycle will concentrate on. I have also modified the Requirements document to include a reference to the increment number as well as the priority for each requirement.

    The increment plan will be limited to the next few steps since it is disturbingly normal for projects to change direction and a detailed plan that goes far into the future would be a waste of time and may give misleading expectations.

    The architecture document will now include references to the increments for the tactics and responsibilities that are assigned to the components of the system. Hence we will be able to schedule the design and development in accordance with the increment plan. Each component will usually be either entirely within an increment, or completely skipped. Those that have responsibilities from more than one increment will have to be annotated to describe how the design will be changed.

    The detailed design documents are presenting a bit more of a challenge as I have not decided yet how to portray the evolving designs. I could have snapshots for each increment and even put them in the version control system, but this would make it difficult to describe the modifications from one design version to the next. While having a history of the evolving design is important, it will appear to be clutter for someone trying to understand the current design, so I am considering a structure where the main document contains the current design and an appendix holds the previous designs and the reasons for the changes.

  14. #44
    Join Date
    Oct 2010
    Location
    Canberra
    Posts
    2,682

    Re: VICI: A Software Development Project

    Doxygen Comments

    I was doing the detailed design for libcanvas which is the component that allows the user to edit a flowchart and was looking at the interface for the library that I had specified months ago. It seemed to include some methods that made no sense. Either I had got the interface specification wrong when I originally wrote it, or I have forgotten some aspect of the design.

    The real problem is that the interface specification does not include enough description. The name of a method and the names and types of its parameters and the return type are not always enough to make it clear what the method's purpose is.

    To try and fix this I have added comments using the Doxygen style to the interface specification (vici.h) and the common facilities infrastructure (libcfi) library. This hasn't solved the problem of the nonsensical interface specification but at least I now know where the problems are.

    I will add Doxygen comments to the code as I develop it. They will document what the code does, as opposed to the design documents which attempt to describe what the programs are supposed to do.

  15. #45
    Join Date
    Oct 2010
    Location
    Canberra
    Posts
    2,682

    Re: VICI: A Software Development Project

    The Canvas Library

    The canvas library is responsible for allowing the user to create and edit the flow charts that define the sequence of events in Vici's scripts. Eventually it will include support for testing the scripts as well, but for this first increment it will just allow the scripts to be created and modified and the running will be done from the run-time program.

    The symbol library is responsible for drawing the actual symbols on the chart, so for the development of libcanvas the test program is supplying dummy symbols in the form of blue boxes. However the canvas library is responsible for drawing the lines between the boxes. These lines represent the flow of control, data streams and signals passing between the processes and other objects such as variables and files.

    I have included a screen-shot of libcanvas on the SourceForge page. You can also have a look at the code for the libcanvas design.

    The code is very conventional, just using Qt's graphic scene to draw lines and text. Saving the structure to an XML file was easy enough, but reconstructing the objects from the XML proved a bit more complicated than I expected. It was rather tedious to write as each symbol type has its own peculiar requirements. I am glad I stopped at 16 different ones. At one point I considered writing a code generator to generate the classes and their dialog boxes.

Page 3 of 8 FirstFirst 1 2 3 4 5 ... LastLast

Similar Threads

  1. Web Development Software
    By Pepperonie in forum Using Fedora
    Replies: 2
    Last Post: 5th May 2007, 10:56 PM
  2. Development software for Linux
    By jo3 in forum Using Fedora
    Replies: 9
    Last Post: 15th June 2006, 03:28 PM
  3. Software Development
    By handshakeit in forum Using Fedora
    Replies: 0
    Last Post: 5th October 2005, 08:40 AM

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •