Pretty interesting infographic…
“A clever new infographic from gigaom.com shows how the App Store puts more than 100,000 apps at your fingertips — and generates millions of dollars for app developers worldwide.”
I just discovered Test::NoPlan (http://search.cpan.org/~duncs/Test-NoPlan-v0.0.4/lib/Test/NoPlan.pm)
It goes over you test suite to see if you have any no plan statements – very nice and useful.
So I have added the following test (file: t/test_plans.t) to my growing arsenal of distribution tests:
- use Test::NoPlan qw/ all_plans_ok /;
- all_plans_ok();
I once read an article on a new implementation of an autopilot system. My colleagues and me found it quite amusing that it used a dog-bites-man strategy, in the sense that the expert system would guide the pilot and warn him if he was doing something stupid or hazardous acting as a watchdog over human actions and interventions.
The above story is somewhat relevant to a pattern, which have seen at my workplace. To begin with we have increased the number of techies, also meaning more developers and I am one of the newest kids on the block (or the mill is perhaps a better term). After my start and much driven by me we have addressed our processes quite intensely.
We have relaunched the concept of version control since the existing system was not really put to use – it was just used by a single developer and not for everything. So we set up a new server and updated the version control system (Subversion) and migrated the existing projects from the old server.
In addition we made the same exercise with our ticketing system, we had an old Request Tracker installation. We set up a new one in parallel and started creating new issues (RTs) there – the old system had not been maintained and the number of tickets were quite low, so an update or migration seemed like a lot of work for almost nothing, the old version has been phased out now – and we don’t miss it.
So with the two corner stones in place, we have seen a lot of good practices being put to use. Our RT processing has been integrated with or Change Management process (based on an old version of Twiki – which is next in line to be updated IMHO). Our Change Management “system” is based on a list of Change Requests (CRs). We set some guidelines and to act as watchdog perimeters for changes to production environment components.
In addition to a high number of RTs (and lower number of CRs), our change process and work flow are both slowly falling into place. Control over what we do and what we need to do is also improving.
I have printed out the poster from ‘Ship It’ and put this up next to my desk. It is not the complete truth, but it is at such a high-level that it is difficult not to agree with it’s take on the good practices it lists.
So on the infrastructure side we now have the following established:
- Version control (Subversion)
- Feature tracking (RT)
- Issue tracking (RT)
We are a bit behind on the use of Subversion, but I guess we will get there eventually. For the techniques, we have the following:
- Technical lead(s)
- (working from) The list
- Code Reviews
The old dogs in our company are good technical leads, since they now the business by heart and have been employed for a long time.
We have had some basic code reviews, which have demonstrated the power of code review, we have basically used these as sort of daily meetings, just bi-weekly and focus has been more on security and general architecture, digging more into what should be the development process – I will get back to this later. I hope our code review process will be able to support QA more in the long run, but for now this is working great.
We have become really good at structuring our tasks in RT and communication are happening in the RT system instead of in diverse misc. media like email and jabber (well it does, but the essentials are propagated into RT), so we are all basically working from the list, which is perhaps one of the best practices. It gives us an idea of: who is working on what and how much we are actually doing, we are not into micromanagement, but it is a good idea to visualize the pressure under which you technical department is, workload wise
I myself is a heavy user on the version control system, working primarily as a developer and with source code. The page for our homepage (a Typo3/PHP setup) has also been put under version control. For that part we have seen a number of practices evolving out of thin air.
- Release management
- Scheduled (monthly) releases
This is great, we are much more in control of what features are released and they are actually put through an acceptance test cycle before final deployment, which has also proven quite good – slow, but good.
This all sounds incredible! – but we have problems. I have mentioned some of these already, but here is an accumulated list, with some new ones added.
Which leads me back to the story I began this block post with.
It is of outmost importance that we are very strict on our change management. I am all for ad hoc changes to the code as long as you do not break anything or introduce bugs, so regression test is quite important and you test the changes before releasing them to production.
We have on one occasion seen changes to a component, where it impacted an imminent release. I was working on a new release of a component, where some logging would be enable a minor thing, I was then notified of a change in production, I requested a diff, evaluated it and it looked all good so I applied the changes – missing a single aspect of the diff. It broke backwards compatibility.
So when I released, my non-intrusive change, it did not work. It took some time for me to fix the bug and get it right. This is of course my own fault and that is what happens if you apply changes in the 11th. hour. So I got my release up and running in another environment with both the change from production and my own change.
A few days after I was talking to another user of the component on another system, the system where the change had been made to production directly. He had not been notified of the change either so his use of the component was also broken, this had nothing to do with my release, since it had not been deployed to the system he was working on, but the issue fell back to me because he knew what I was working on, all I could do was enlighten him on the changes I had applied to make it work.
Apart from my little stray release, our main problem here is changes made for no apparent reason, but the good idea. The change was not in our list of changes to make so nobody could prepare themselves for the change. 2 people spent a lot of time working on correcting things, because somebody had an good idea about a change to an API.
So in retrospect I have gathered that we need to move on with some of the best practices we need to apply and a few other things, here is the short term list:
I had actually scripted the build and written tests, but making the change directly in production does not necessarily require these to practices to be executed and my test suite should have caught the API change – bummer.
So it is back in the saddle…
This is my first post written on the iPhone. I have had the WordPress app installed for a long time, but I have never really gotten around to use it.
So here goes, my first post written entirely on the iPhone. I normally use MacJournal, so I will have to synchronize these entries back to MacJournal, a feature which should work out of the box.
Next test must be to include an image, lets see, lets take a screenshot
Today is Ada Lovelace Day.
I am supposed to write a blog entry about women in tech. I have been giving this a lot of thought since I signed up for participation for Ada Lovelace Day.
When the kids left the house, I sat down at the computer. Going over the list of recent tweets, I fell over the first blog post on Ada Lovelace by Karen Pauley.
Somewhat still bewildered on what to write, I thought I might as well read it at least to get some inspiration.
After having written a short comment I went back to the list of tweets and the next tweet, which stood out was about the Perl Foundation getting a new president and a new treasurer.
And guess who…
So for Ada Lovelace day I am going to write a post on Karen Pauley, celebrating women in technology. This post will not do her justice, but I insist on choosing her as my subject for Ada Lovelace Day, even though she will not like it, so here goes.
The Perl community is a very complex system. Some people think that Perl as a language is hard to read and understand, it’s community being a very tightly knit and old community certainly has it’s dragons, trolls and a lot of magnificent individuals only a few people really understand the community as a whole or bears the merits to be able to maneuver and act in the Perl community without making a complete fool of themselves, making enemies out of individuals or groups of hackers or just grow tired and go somewhere else.
Karen is one of those persons.
I have met Karen on several occasions and I can only say that she is indeed fantastic. She is always positive and on of the kindest persons I have met. Nobody can make you feel that your contributions are worthwhile and special and that your participation and work is of the outmost importance.
Karen had decided to maneuver the Perl community on the public level. Karen is not a contributor in general to the Perl code base or CPAN, so you cannot judge her by her technical merits, I do not say that she is lacking technical merits, Karen does not make much fuss about her technical merits and it does not seem important to her to put these up for public display, this gives her a very special profile inside and outside the Perl community.
This profile is exactly what makes Karen so special and make her stand out. She is indeed the right choice for president for the Perl Community to take an example. Karen has not taken sides technically, she is not supporting the modern perl movement over some other way of doing things. Karen is supporting Perl on so many levels, her work in the community to get events running smoothly to get hackers to meet in the flesh is magnificent and it is hard to measure how this benefits the community in general, but I am of that belief that these contributions is what makes the community tick and glued together.
Karen, congratulations on your appointment as president for TPF, you deserve it and you are the right woman for the job.
This entry is dedicated to the women in tech, personally I would like to mention in alphabetical order:
- Allisson Randal, developer
- Hanne Vilmann, open source event organizer
- Karen Pauley, president of TPF
- Monica Lind, project manager
- Natalya Zhukouskaya, developer
- Nina Jansen, Astronomer, Singer, ruby/perl hacker
- Sanne Jørgensen, project manager
- Sidsel Jensen, developer
- Sine Grønfeldt, developer
- Stine Fie Brømsø Nielsen, teamleader
- Trine Gantriis Koch-Nielsen, UX/UI specialist
- Ulla Hald, manager
and all of those I forgot…
Sometime back I got a green grass project. The assignment was to implement a proprietary protocol over HTTP in Perl.
I chose to use Catalyst.
Having worked with mod_perl and Apache a lot in the past I thought it would be a good idea to use this too and the integration with Catalyst would be a walk in the park and expectedly working out of the box.
I started coding like crazy, doing semi-TDD and getting a lot done.
I was implementing a protocol, which was to be versioned and if the protocol was to have changes it would require a lot of work, not now what would come, I decided to let the controller reflect this so a new version of the protocol would only require a new controller – all of the used components would have to be revisited to the extent they would be influenced by the protocol changes – this seemed like the best approach at this time.
When wrapping things up examined all the code it struck me – the code base was a mix of OOP and procedural programming.
I had coded the application using what felt natural, I only implemented what was absolutely necessary. I isolated components for reusability where I found it useful – which is part of a long term plan to refactor a large part of the code base at work.
Which brings me to this list:
- TDD is good
- TIMTOWTDI is good (OOP and procedural mix?)
- Coding only what is necessary is good (LEAN?)
So having a large code base being a mix of OOP and procedural code, I refactored to OOP only keeping procedural code which was in external components regarding these as a hybrid sort of mixins.
The newer version new being dominantly OOP, using some common design patterns, looked more promising from a long term maintenance perspective.
I was much more satisfied with this new implementation, but I am still thinking much about the process and what actually happened.
A good architecture is of course important and the OOP refactoring did introduce a lot of code, which was only related to the actual OOP implementation and as such had nothing to do with solving the problem.
The evolution of a Python programmer comes to mind.
Am I doing enterprise programming when something more simple would suffice for solving the task or is the enterprise level completely okay, since the code is expected to live for a long time and maintenance is an important factor.
Back in school we were using a Danish book on OOAD and one of the things I took with me from that book was the authors thoughts on setting up some prime directives for your project.
Examples of prime directives could be:
Please feel free to make up your own.
So your project could have 1-3 prime directives, this would assist you in making decisions on:
- Architecture
- Implementation
- Testing
- Time/Projecting
I often think that project managers are a waste of resources and waste according to LEAN should be removed – okay, jokes aside.
Using the approach mentioned above developers are could be somewhat fitted to make their own decisions based on directives (most developers are quite clever, responsible and happy people), without a project manager and perhaps even without an architect – perhaps just a fellow developer or two to do code reviews with.
Looking at the project I was working on, I could defined the following 3 prime directives:
- It must be secure, since we are transporting sensitive data (SSL takes care of most of this part)
- It must be extensible, the protocol is the product and future changes should be able to be implemented without too much hassle, still predicting the future is hard and often wrong anyway, but be prepared at least)
- It must be atomic, the protocol handles requests, if these fail we should be able to roll-back and notify the client
So what prime directives are not important in this case:
- Ease of use, it is a machine to machine protocol
- Correctness, is touch directive, of course all computer programs should be correct, but please not that by correct in this sense I would use this directive if calculating prices or something else requiring calculation
- Must be fast, speed is not important in this case
So my tests should be implemented to ensure the following:
- We implement the specified protocol
- We are secure (I have decided just to assert that we are using SSL and not plain HTTP)
I could spend a lot of time testing extensibility, but I decided not to, I trust that my architecture and implementation supports this – since I have no idea of what extensibility requests could come.
The project is almost done, the project has most certainly given some food for thought, which will influence my next project. Hybrid implementations seem to come quite naturally, but I wonder if I am the only once coding this way.
The project later ran into some serious trouble, I will get back to that in a separate post, where I will also elaborate on the topic in this post.
I was home sick all day yesterday. In the afternoon I could not sleep anymore so I started reading Paul Grahams very promising ‘Hackers and Painters’.
http://www.amazon.co.uk/Hackers-Painters-Computer-Essays-Programming/dp/0596006624/ref=sr_1_3?ie=UTF8&s=books&qid=1266957762&sr=8-3
By now I have read the first 3 chapters and as I wrote the book is very promising, not to say good.
I am however very happy that I had my iPhone handy with the Dictionary.app installed (http://dictionary.reference.com/apps/iphone)
I really enjoy reading english books/blogs/articles etc, that push my boundaries for my English vocabulary – Paul has however proved to be quite the english tutor in my case.
After 3 chapters, my list of lookups look as follows:
I wonder if this pattern will continue…
A few days ago I read an article on Dr. Dobbs written by Peter Eastman. Eastman describes a concept of bug opportunities in bodies of code. He divides the bug opportunity into a set of levels, creating a simple taxonomy for bug opportunity classification:
After consuming the article it came to me that the levels could be extended even further, with the following:
Level 5. Misinterpretation of reference source or specification
Level 6. Error in reference source or specification
Let’s break down level 5.
This is the bug where your program runs perfectly unit-tests suite passes and everything. But the implementation is error prone and buggy in the sense that it does it is not correct an example could be adding taxes several times to a price (I have seen this one in production).
One of the problems that here is that the unit-tests are written by the same developer, who also wrote the implementation code for the application in question. So one could argue that the bug is first present in the test suite since it does not demonstrate the presence of the bug in the application code.
Level 6, is somewhat related to 5. But on a higher level, where the people who provided you with reference or specification got it wrong. An example could be… NASA (Lost in Space) – I am not sure of all the details in this case, but it does sound like a specification related bug.
Pair programing, dedicated testers and code review are of course all good remedies for many of the above examples. The taxonomy is however quite interesting in relation to bug hunting in your development process/method.
For more expensive bugs check:
- http://royal.pingdom.com/2009/03/19/10-historical-software-bugs-with-extreme-consequences/
- http://en.wikipedia.org/wiki/List_of_software_bugs
I have just recently released Workflow release 1.33.
I wanted to refer to a blog entry on the developer releases/branching strategy I have been using for the Worflow project, but I could not dig it up – then I found it in my blog backlog, dated 20.09.2009
So here goes,
The releases 1.33_4 and 1.33_5, both where released on the same day since they do not contain the same. Let me describe the strategy I am currently using.
The developer releases on CPAN, releases using the following format:
<major>.<minor>_<developer release>
These release are preliminary releases before a stable release. Lots of CPAN authors/contributors use this scheme and it is supported by CPAN. Many developer releases are however quite useful, at least this is my experience as a CPAN user. Workflows developer releases are however more like a snapshot, reflecting concluded work on a specific branch. They are therefor not recommended for use, unless they address a specific bug you have reported or patch you have sent.
To demonstrate. The last stable release was 1.32. Meaning that the next planned release is 1.33.
Since the release of 1.32, the following releases has been made:
All are based on branches and the releases are made from the relevant branch before merging into trunk. The actual merge will not be made until the preparation of the 1.33 release begins.
All branches are based on release 1.32 so the different releases do not contain each other. The reason for this is:
In regard to the above development releases, a merged release should probably be made in order to get the 1.33 release candidate tested.
I am not sure what strategy is made by other CPAN authors/contributors, so feedback is very much welcome, since I am not aware of what the best practice is or the de facto use of the developer release scheme.
During the weekend I finally found time to release Workflow 1.33. I have not mentioned the last developer releases in this blog, but since the release of 1.33_4 and 1.33_5, I made 3 more developer releases before the final release of Workflow 1.33
I was cleaning up in the Changes file and I found out that 1.32 was released last January (2009) – I find it somewhat embarrassing that it has takes so long to get 1.33 out of the door – perhaps a different strategy should be necessary in 2010 and onward.
The release contains a lot of contributions, each developer release have been based on branches implementing patches from a lot of different people. Quite a few are minor improvements, but among the bigger features I can mention:
- dynamic loading of config files, from Andrew O’Brien.
As I have mentioned there are a lot of contributors and all of their contributions matter, big and small.
The developer releases made since 1.33_5 was:
I have been using a strategy where each developer release have been based on a single patch so the different releases did not influence each other until late, I think I might need to reevaluate this practice and moderate it a bit, so bug fixes are releases faster and feature releases then can use the developer release scheme.
We are experiencing trouble with Perl 5.11 in version 1.33 (see: the test results for 1.33_8). I considered installing perl 5.11 to address these issues, but after some thinking, I decided not to let this postpone the upcoming release, since I gathered that nont of the Workflow users would probably be using 5.11 in production.
The plans for Workflow are as follows:
And hopefully all on a more regular schedule in 2010.
Oh and yes it is available on your local CPAN