Software Build Automation

Posted on by
Reply

I remember reading a book about 4 years ago on how to determine the maturity of software development companies. The thing that stood out at the time was the fact that mature development shops have a well managed and efficient automated build system with a sliding scale from that to companies that are starting and do not even have a source control system.

The reason why it stood out is that at the time we only had a very rudimentary build system and it was bugging me that we could not go from source to installers (including build # and documentation) without someone getting involved to hand crank it. Doing it that always ended up with inconsistency and we would end up testing software builds that had the wrong DLLs in it and simply just waste some testing time.

The decision was made to create a new build system and efforts started on getting it implemented. Unfortunately, along with the decision to implement a build system also came the decision to implement all of the functionality ourselves rather than use 3rd party OR open source tools. Due to this, the build system got half built before we realized that it was not maintainable and the project suspended.

I mention this in the hope that others do not suffer this growing pain as we did. It is also the case that at the time the tools we know use did not exist or certainly were not ready for reliable use so perhaps we just tried to it before the most optimum time.

Ok. So about a year later I started looking at this problem again and found some interesting tools out there that people were recommending.

  • Nant
  • CruiseControl.NET

Nant

Nant is a open source .NET build system which finds it’s home on sourceforge. At the time of looking we had started to move all of our code from VB/C++ over to C# so standardizing on a build system that is .NET based is fairly good one.

We still have to this day a set of components that are VB so it was important that it would support VB. Thankfully we are not the only company in the world that still have VB code and there is a vb6 task as part of the Nant Contrib project. This allowed us to have our VB6 projects compiled in the same build environment as our C# projects.

Nant allows for a great level of control on what is built and what is used to build it.

CruiseControl.NET

CruiseControl.NET (CC.NET) is a continuous integration server that watches your source control system and recognizes when changes have been made on a configured sub folder triggers a process to begin. In our case this process is a nant build task on the piece of functionality that has changed.

The great thing about this is that you get immediate feedback on all code checked in that it did not break anything. This can be anything from a new source file missing to breaking a build on a different version of ArcGIS (more on this later).

There are web-based and windows notification area (system tray) tools that allow anyone to see the current state of the build system.

To make the ‘test builds’ quick and painless we have separated the build files into product/project and even component level. This means that if we change component Y in project X then only that project will be built and not the whole source tree (which takes about 20 minutes and would be over the top for every check-in!). Due to this we have about 30 projects in each build server (up to 3 at any time — more later).

In the early hours of the morning we also have a batch build script triggered within CC.NET that builds all of the software in order, creates the installers and server folder structures for testing. This process takes anywhere up to about 30 minutes and again as it is within the build system we know whether it as built successfully when we log in every morning.

Build Servers

I mentioned that we have up to 3 build servers. Why on earth would we even contemplate having more than one you say? The simple and only reason that we have up to 3 is to support each version of ArcGIS. When using .NET the libraries referenced are version encoded so we made the decision to build the same source code in all of the versions that we currently support. There are ways of work within versioned assemblies (use of policy files for example) but we felt that it would be good to build on the actual version the user will be running the software on so the build servers were born.

What is the result of the build?

Well, that depends on what build we are talking about.

For a build that is triggered by a Subversion check-in just the component of software that has been changed is built. There *should* be a subsequent build of all components that depend on that particular assembly (or set of assemblies) but this is not supported in CC.NET and we have not encoded that logic in Nant as yet.

For a ‘batch build’ that has been triggered in the evening the Nant script:

  • Increments the build number is all of the AssemblyInfo.cs files
  • Builds all of the projects we want built
  • Reverts the AssemblyInfo.cs files so that they do not show up in SVN as edited.
  • Generates Installers for all of the products we have
  • Copies the installers to a well defined location

All of this is very important to a software team as it means that all of the tedious tasks that are prone to error are automated. This means we cannot get it wrong (well kind of) and we can be sleeping when it is being done (batch build is triggered at 12:30 AM).

Skype is back up!

Posted on by
1

Well after about 2 days suffering withdrawal symptoms it seems like Skype is back up and running again….at least for a limited set of people I have on my Skype list.You do not realize how much that you rely on something until it is not there and that was certainly the case with Skype. I use is VERY regularly throughout the day to communicate with co-workers and friends and when I could not get to them easily for the past few days it was strange.Not that it is as bad as some people will profess. I can still call, email, fax, knock on their door, use MSN Messenger or simply do without for a day or so but I agree that it still did not feel right.This is the first time that I have seen Skype go down and from the reaction to the event I bet it will be the last for a LONG time. Keep up the good word guys. You are still the easiest to use, cheap (free) and most reliable service of it’s kind out there.

WordPress plugin of note

Posted on by
Reply

I use WordPress for this blog (obvious from the URL I know!) and although I do not run my own version of it I do look at possible plugins from time to time.

These caught my eye this morning: http://wunder-ful.com/wordpress-plugins as it contains something that I wish I had on my blog .. date limits/ranges. There are scenarios where a blog is no longer relevant and should not be displayed in a feed or the blog page and WordPress has nothing to deal with this.

I tried the Posts Expire one on my own laptop  and it works like a charm. I hope that WordPress put this one in the core app at some point so I can use it on this blog.

Filebased Geodatabase Date Filtering

Posted on by
Reply

So I had to add some functionality to filter by date and normally this is really simple in ArcObjects as you use the IQueryFilter::WhereClause and enter a date formatted where clause in there.

One problem though. I am using Filebased Geodatabase and I did not know what the correct way to encode the date is! I thought about it for a little while and then decided to use the trusty Query Builder from within ArcMap. It worked a charm.

image

It is hard to see but they encode the date by pre-pending ‘date ‘ and encoding the actual date in ‘yyyy-mm-dd’ format.

Must archive this somewhere in my brain where it will not get lost!

Google Reader bound..

Posted on by
Reply

I have been using Office Outlook 2007 to read RSS feeds for the past 6 months or so  – perhaps even more but after Robert Scoble time and time again has said that Google Reader rocks I decided to give it a try.

So for the past week or so I have been using Google Reader rather than Outlook and I am now going to make the move permanent. For the amount of feeds I currently read, Outlook is just not cutting it. 

Google Reader is really fast and the feeds render nicely. The interface is typical Google – clean, crisp and simple to use. 

When reading entries I normally try process them all but there are frequently entries which I would like to read but that are too verbose to get through at that time. In this scenario I would find myself launching a Firefox of the blog entry window just so that I could read it later. I don’t need to do this anymore as reader has the ability to ”star’ or flag and item and has a neat filtering mechanism were I can go back and view all of the entries that I have starred.

I believe that Outlook has a similar capability but I have never liked the way they flag stuff. In this case I do not want my feeds flagged in with the email I have flagged.

The reason why I looked back at reader in the first place (after discounting it when it came out) was that it now supports Google Gears. This means that with the click of a button I can take all of the feeds offline and read them on a plane. I spend a bunch of my time on planes and this out of all of the needs is my highest priority and why in the past I have selected Outlook over Google Reader.

Given that I am on planes often and unless the stars align and I am not upgraded to those nice seats, reader has one more trick up it’s sleeve. Keyboard shortcuts. You can navigate and read all of the feeds and entries by solely using the keyboard and it is so quick and easy to use that it is the only way that I interact with reader even when sitting at my desk.

Thanks Google for developing 2 pieces of technology (Reader and Gears) that make my life easier!

Uninterruptible Power Supply

Posted on by
Reply

So I wake up this morning hoping to finish up development on a piece of work that I am doing to find that my day will be longer than expected. I try to spin up a Virtual PC (VPC) of the ArcGIS 9.1 development environment I have and Virtual PC tells me it cannot find it!

That would usually be OK. I have a clean VPC image of ArcGIS 9.1 and dev environment so normally I would just synchronize the source code and I am off and running. Not today.

I have been working on this new functionality for about 4 days now and have not bothered to add it to source control as it is not ‘ready’. So if I do not try and solve the problem I will lose out of 4 days of blood, sweat and most definitely tears.

It turns out that the storms that have been causing the amazing sound and light shows around here with the occasional power fluctuation were not liked by my external hard disk that is home to all of my VPCs :-(

I have had this trouble before and I was determined to go straight to BestBuy that morning and get a UPC. I didn’t and it was just a matter of time before it happened again I guess. The last time I had to spend about 4 hours persuading the hard disk it was OK and let me access the files so when it happened this morning I sighed, brewed some coffee and got ready for the fight.

It turns out that good ol’ chkdsk saved the day and I am back in. All the files *seem* file but time will tell I guess.

I am going to buy a UPS this time. Honest.

64-bit Windows and ArcGIS

Posted on by
9

So we predominantly use C# for development at TC Technology for many reasons which I will leave for a different post. During 64-bit testing though we found that it can be interesting when dealing with:

  • 64-bit Windows
  • ArcGIS 9.2
  • .NET 2.0

at the same time. When trying to initialize AO using the following code

IAoInitialize init = new AoInitializeClass();

it fails with an error like this:

System.Runtime.InteropServices.COMException
Retrieving the COM class factory for component with CLSID {2237A429-E4AC-40D6-96A5-5B6CCF40D56B} failed due to the following error: 80040154.

We did some digging and, thankfully, someone has already figured it out. It turns out that if you build any .NET code with the ‘Any CPU’ option which is the default then ArcObjects based applications fail. This makes sense as the c++ dlls that are built for AO are 32-bit so the application/assemblies that are consuming them also need to be 32-bit.

Fixing the problem is pretty easy (although if you have a bunch of assemblies like we do tedious!).

First thing to do is select ‘Configuration Manager’ from the  Solution Platforms drop down (the one with ‘Any CPU right now’):

image

It actually brings up the same configuration window as the Debug/Release configuration!

image

From the ‘Active solution platforms’ drop-down select ‘New’:

image

and from the resultant ‘New Solution Platform’ select x86. This is create a new configuration for x86 only. This way when it is run on a 64-bit environment the OS will know that it should be run as a 32-bit application.

S3 Backup – Part II

Posted on by
Reply

Ok. So I have been uploading my music to S3 for the past 3 days and it has eventually all loaded up there. I have not quite figured out why it does not use all of the bandwidth available but since I actually wanted to do some work this week it was not bad thing!S3 Backup is really nice. The wireless network flaked out on me a few times and I had to restart the backup. Thankfully S3 backup does an inventory of what is on the local machine and compares it to a continually updated DB on the S3 side. This means that you can restart backups from the point you list connectivity which is REALLY useful.My Music is around 7.5 GB which means that to get it into S3 cost me $0.97. Doing the math on the ongoing costs with the 7.5 GB it will cost me a whopping 80 cents a month to have it backed up. You just cannot beat that.Next on the list is 15GB of pictures and then 30GB of home video so I am guessing that the ongoing monthly cost to backup our family data will be less than $5. Well worth the money.The only thing that I would change is the speed of upload. I have not had time to figure out if it is S3 backup, S3 itself or simply just the upload speed that I have but I will!Thanks to Amazon and the guys at S3 backup for the good products/services. Having lost vital information in the past this is really a good thing for me…and I bet others.

Amazon S3 and backups

Posted on by
Reply

The fact that we have no backup system at home has been bugging me for a long time now. I have been waiting for a decent NAS to be available that has RAID or something similar in it. Seeing a webcast about Drobo started me thinking about this again.

I have came to the conclusion though that what we really need is backup that is not in the house and stored in a well managed location. Given this, Amazon S3 is the best bet in my opinion. You pay a very low ongoing cost and it is stored on huge servers with incredible redundancy.

There are many tools out there to perform the backup itself but I decided to go with S3 Backup as it was easy to setup and had incremental support. Backups seem to be slow (factor of the internet connection upload speed rather than the application/service) but the incremental support will stop this from being a problem. Once again time will tell if the right decision has been made!

From Perforce to Subversion

Posted on by
Reply

The company that I work for recently switched from Perforce at $800 a client to Subversion at $0 a client. The reasons for which are obvious but what we did not know until fully implementing the system is the pro’s and con’s of doing this for OUR company.

Perforce is a great product. It is fast, easy to use and has some neat tools for managing the editing cycle of source code etc. We have multiple sites for development and having a fast synchronization no matter where you are is very important and perforce does a great job of this. The differencing tools are great and check-in, check-out and revert of code is simple and intuitive.

That said, ‘free’ compared to $800 a client is not to be ignored and we decided to try out SVN. Configuration is a simple unless you want to use HTTPS (which everyone does) and then you have to trawl the knowledge base articles and use some magic to get it all to work. One the server is running though getting client online is a breeze.

TortoiseSVN for simple file management and AnkhSVN for Visual Studio integration are a must as far as I am concerned and once again they are free. TortoiseSVN in particular with it’s Windows Explorer integration and differencing tools is the best interface I have seen in a long time for source control. Perforce tries to have a Windows Explorer interface but it does not do well and certainly get’s no-where near TortoiseSVN.

One major factor that attracted me to SVN is the fact that you can do what you like to the source tree when offline. The concept of checking out for edit in a pain in my line of work where I spend the majority of my time on client sites. Having the ability to make code changes during the day and check them in at night is great.

I frequently do not have to do that though as more and more clients are providing WiFi connectivity outwith their firewall to the Internet and since we have the HTTPS version of SVN running I can simple check code in over the Web without even using VPN! We previously used secure tunnel for this and the client always flaked out at the worst times :(  

One of the benefits of SVN when using Apache is the WebDav. This allows us to include our documents, project plans etc. in the same versioned repository as the source code but with a fit-for-purpose way in Microsoft Office of getting access to and updating the files. This is something that we have not taken full advantage of as yet but will be starting in the very near future.

All in all I would say that SVN is well worth the effort it takes to setup the server and — touch wood — the database has been extremely stable and we have had not problems.

I know that this is a very generic conversation but I will get more in-depth about the branching capabilities (used all the time by us) and how easily it slots into CruiseControl.NET in a different post.