Category Archives: Programming

Techie programming type posts.

A Request

If anybody is writing .Net code that in the future I am going to have to find an error in, can they refrain from doing things like this:

catch (Exception e)
throw new ApplicationException("ConnectionException has occured: " + e.Message);

The particular application I was working on this morning used this particular gem pretty well everywhere. The problem was that we were getting an unhandled ApplicationException – the re-thrown error wasn’t being trapped anywhere else – and since the above bit of code throws away the stack trace from the original exception, we were left trying to guess where the error was actually occurring.

The irony is that the code was written by a contractor who apparently had a pretty stellar CV, and amongst other things quite regularly criticised other people on the team over their apparent bad programming. His contract wasn’t renewed, and since then we’ve been discovering quite how bad his programming really is – a definite lesson that however good a contractor appears on paper, you need to properly monitor what they are doing.

Other gems in this particular application include every SQL statement being built by concatenating strings together – and no, not a StringBuilder in sight. Using ToString() to do the conversions where needed, including quite a few points where he calls ToString() on a string – indeed a large amount of the data is just held as strings anyway, only being converted back to integers or dates in the stored procedures when SQLServer needs the right types. He had also written his own replace function, which he was using instead of String.Replace although it was functionally identical, and that wasn’t the only place he’d hand crafted code that replicated functionality in the framework. The code was littered with other common errors, for example it’s a good job we don’t have anybody with the surname O’Reilly…

Since we were fixing a specific problem, much as I would like to, there wasn’t the opportunity to fix all of these other problems, essentially we have to wait until something else breaks, and fix it then. And the source of the problem today? A stored procedure that was trying to convert one of the multitude of strings being passed through back to an integer in order to use it. That conversion was failing and throwing an error that was passed back to the client and helpfully chucked away by the error handling code I started with – the only clue we had as to where it was going wrong was that the error message seemed like it was coming from SQLServer.

How I Got Started in Software Development

Craig tagged me for this ages ago, so I guess I’d probably best put in my answers!

How old were you when you started programming?
I think I probably started aged around ten or eleven, with the arrival of a Sinclair Spectrum at home and a BBC Model B at school.

How did you get started in programming?
I’m not really sure of this one, but I guess it was just the interest in how the respective computers worked, and the challenge of getting them to do things.

What was your first language?
Definitely Spectrum Basic. Until I was actually doing a proper Computer Science course at school – something that seems to have fallen by the wayside for generic ICT classes now – most of the software development was at home, thanks in part to the listings that the computer magazines produced. Indeed back then there was even an entire magazine devoted to listings, Sincliar Programs, which I used to read and copy in the listings from.

What was the first real program that you wrote?
I’m not really sure, probably the old classic “Hello World” in Spectrum Basic. In terms of a real serious project, that was probably the programming project for my A-Level in Computer Science.

What languages have you used since you started programming?
Basic – Spectrum/BBC/VB6/VB.Net, C, C++, Modula-2, 68000 assembler, Pop-11, Occam, COBOL, SQLWindows, Java, C#.

What Was Your First Programming Gig?
This was a summer job as a result of a work experience week, working for the British Holstein Friesian Cattle Society in Rickmansworth. Their business was keeping pedigree records for several different breeds of cattle and sheep, and on request producing the lineage of an animal, usually when the animal was being bought or sold. I spent the summer there during a key period from the point of view of their computers as they were moving from an old system written in COBOL running on an ageing ICL ME-29, over to a system written in C on a UNIX box. The COBOL was probably the experience that left the biggest impression, as in COBOL the indentation of the lines of code is important, and the editor only went forwards through the file, and only allowed you to edit the line at the bottom of the screen!

If you knew then what you know now, would you have started programming?
Definitely yes, not least because if I hadn’t have gone into programming professionally I would have probably become one of the worryingly large group of amateur programmers I’ve come across hacking together key business applications, all of whom would much rather be doing a programming job than the one they ended up in.

If there is one thing you learned along the way that you would tell new developers what would that be?
One of the main tips I’d give is to give yourself a broad and general base – having done a couple of crash courses in VB doesn’t make you a programmer. Most of the best ones I’ve come across over the years have been flexible and adaptable, and usuall spent several years doing a broad based course – generally a degree of some sort – where they have been given the broad basis that allows them to cross-train quickly as technologies change. They’ve also got the broad IT knowledge to understand what else is going on around their job – you might not ever program professionally in assembly, or do any deep level AI research, but it is surprising how much of that sort of stuff is useful in ‘normal’ programming. Don’t get me wrong, I’ve met quite a few people who have learnt programming on-the-job and can cope, but equally I’ve met a lot of others who aren’t adaptable and end up stuck. For example one contractor we’ve had at work is self-taught and writes great ASP code – unfortunately he was hired to write ASP.Net, and he really doesn’t get the differences. With our entire suite of software being rewritten in a multi-tier C# design, using an object-oriented design – because he’d been taught how to write ASP pages in VB, rather than being taught software engineering he was effectively unable to do what we needed.

Along those lines, I’d also advise anybody to keep your skills current, and be especially wary of companies that don’t keep up to date, or program in something that isn’t mainstream. The biggest problem to get into is ending up in a very small niche programming market as I did with SQLWindows. Luckily I found a new employer who was happy to cross-train me into VB. You’re never going to totally escape from maintaining the creaky old VB6 systems, but make sure you’re in a company where you get a balance.

What’s the most fun you’ve ever had… programming?
I’ve had times when I’ve enjoyed programming professionally, but still one of the most fun bits of programming I’ve had was on the LPMud we had running at university. The game was all running in a variation of C, and one you reached wizard level you got to add to the game by writing code – indeed the wizards got programming level access to the guts of the game so you could actually manipulate the game environment on the fly. Programming for that there wasn’t really any pressure, and in terms of the game there weren’t really any limitations on what you could produce. Professional programming, the majority of the time you are working to a spec, and for a customer, and you don’t get nearly so much freedom in what you do.

A Problem with Fake MARS

The past couple of days I’ve had one of those real head-scratching problems you get as a software developer from time to time – the classic intermittent problem that apparently has no explanation.

It came up quite by chance – we have a customer who for one reason or another massively underestimated the volume of transactions they were going to put through their system, as a result some parts of their system have been throwing intermittent time-out errors when they’ve been querying big datasets. As a result I had to do a quick fix to one of their modules to increase the time-out on the ADO.Net command objects to allow the queries to successfully return.

Once I’d done that, I did what we would usually do in our team and did some quick informal testing to make sure nothing else was affected, and unfortunately in those tests another part of the application started throwing intermittent SQLServerconnection errors – even more strangely these weren’t coming from the point in the code where the connection was being made, but from deep down in the ExecuteReader method. As far as the code was concerned, it should be using an existing connection – why was it trying to make a new connection now?

At this point it is probably worth discussing what the code was actually doing. In essence it is two nested database queries – the code runs through the results set of the first, and based on the contents of each row does various different things, including running other database queries. All of this was being done using multiple command objects, all created using our core database access library. Digging in to what the library was doing, it was providing a single connection, and generating the commands from that single connection. It all worked fine, except when the size of the dataset started to get big, at which point the errors started to occur.

How the underlying database actually copes with this varies. In SQLServer 2005 parlance, the code is using Multiple Active Result Sets or MARS, Oracle has no problem doing it, and it is supported against all versions of Oracle in ADO.Net. But it was only supported from SQLServer 2005 – attempting to do the same in SQLServer 2000 using the SQLServer specific provider would result in an error being thrown. However even with a SQLServer 2000 back end, if you used the ADO.Net 2.0 OleDB provider it would work, with the OleDB working some magic in between to get around the problem.

All the code in this particular application was written using the generic ADO.Net objects, the idea being that it will work with any of the ADO.Net providers. Although the database being used is SQLServer 2005 – which supports MARS – the factory object that was being created was using the OleDB provider, to connect to the SQLServer 2005 back end. After a bit of digging around I came across this useful blog posting from 2004 with a question and answer over MARS – the relevant bits are towards the end – the OleDB magic bit is that provider fakes the MARS functionality by connecting and disconnecting multiple connection objects in the background – only in very specific situations does it use the SQLServer 2005 functionality – this was what was making the connection that was causing the error! Not surprisingly, this has a big impact on performance, and it was pretty obvious that something else was going awry here as well. Interestingly the article finishes off by advising only using the SqlClient library to talk to SQLServer – the warning is pretty stark:

IMHO it means that you should ONLY use SqlClient to talk to Sql Server, the risk of running into this fake MARS behavior is too great. I have seen this “feature� (fake MARS) cost hundreds of thousands of dollars an hour in lost sales as the server was inundated with unnecessary non-pooled connection open requests.

Understanding that, it looked like something was going wrong in the MARS support with in the OleDB provider. Quite apart from that, the algorithm it was using was going to perform badly with what we were trying to do – the ideal solution would just be to drop in the SqlClient instead, but unfortunately ditching the OleDB provider and going for the SqlClient instead isn’t really an option in this situation, as the application is slotting in with a whole suite of other applications that use OleDB, so the only option was coding around the restriction to ensure that the fake MARS functionality doesn’t operate. So out goes using our database library to manage connections, and it was back to a solution where I’m manually creating a connection object for each command and cleaning them up afterwards, and this seems to have got around the problem – in order to avoid this situation I just need the fine grain control over when things are being created.

Designing for an International Audience

One of the regular issues you often come across with software, is the old classic of the application or web site that goes wrong when faced with a foreign system. Common problems are applications that fall over when they find they’re running on a localised install of Windows (although I’ve come across applications that hit problems with having a British English install), and there are regular problems caused by especially numeric date formats.

Sometimes though you come across ideas where there is something fundamental at a design level that really doesn’t work when the idea is spread worldwide. Just one such bit of software is CommuterFeed that I came across today. It makes use of Twitter to provide a regularly updated feed of transport problems for large metropolitan areas. Essentially if you find a problem, you send a twitter to the commuter feed user, containing a code identifying the city, followed by a short description of the problem.

Looking up the UK cities, came up with a slightly odd looking list, including major cities like London and Manchester, but then also including weird choices such as Doncaster and Luton. A quick look at the list, and the city code for London (LHR) and you realise why such a weird list, the developers have opted for the IATA airport codes for the major airport for each city. When looking at it from a North American perspective that seems like a good idea – coming across the pond to the UK where many major areas that I might want to get transport information for aren’t listed because they don’t have an airport…

Developer Day Number Six

Today was the sixth of the Developer Developer Developer events at the Microsoft Campus in Reading, and as with the previous events, I spent the day enjoying the sessions. As in previous events what I actually attended didn’t quite match up with what I thought I would attend, but the beauty of these days is you can quite easily switch dependant on what takes your fancy on the day, or indeed which sessions have seats!

First off I sampled the first part of Oliver Sturm’s double presentation on producing business applications with Windows Presentation Foundation (WPF). Like most people, what I remember about WPF is the flashy eye candy filled demo applications – what Oliver aimed to do was show that alongside the eye candy was a strong platform that could produce the kind of ‘boring’ business applications that most people end up writing, something in which he very much succeeded.

I commented after DDD4 that the kind of material that Oliver covered needed an early slot, rather than the graveyard slot where everybody was tired, and it is great to see that the comment had been taken on board, as a result I thought I gained a lot from attending both the first and second parts of the presentation (although it would have been better in successive slots I have to say), and certainly have a better idea of what I can do with the new framework.

Another session that gave me a better idea of what I can do was the slightly mis-titled Cruise Control .Net session presented by Paul Lockwood. I say mis-titled as the titular piece of software only appeared at the end of the session, much of the earlier part was looking at all the other bits of software that Cruise Control .Net actually uses, and which provide much of the power of the continuous integration process. No matter as it was an interesting session, and certainly gave me some pointers towards what is needed in setting up a continuous build process. Having said that, at work we’ve been saying we’re going to set up a continuous build process for a long while, whether we’ll actually get round to doing it is another matter!

After that it was back to see Oliver for part two of his session, and then from there on to lunch.

The main lunchtime activity was the Grok talks, of which more in a moment, but first the one thing that really annoyed me about the day (well aside from the car service indicator, but that’s another story) which was the way lunch was handled. Now they’ve tried various ways around this from full scale hot food, through to the bagged lunches they have now. The bagged lunches seemed to have worked fairly well, but it is always slightly slow because they randomly pack the bags so whether you’re just like me and fussy, or more importantly have food allergies you sometimes need to look through to find a reasonable combination. Now obviously there have been comments about this, as this time they had taped all the bags shut and the only option was vegetarian or meat – but still with the same random selection. There were also Microsoft Events staff posted at each table handing the bags out and being downright rude if you tried to have a look at what was in the bags. Heaven help you if you actually had a legitimate reason to be careful what you got. Luckily my random selection was pretty good and I didn’t get a sandwich ruined by tomato and cucumber, and even struck lucky with the flavour of crisps. But seriously taping the bags shut may remove a symptom of the problem, but it’s not actually solving the problem, it’s just annoying!

Anyway, onto the lunchtime talks. In an improvement from last time they were actually held in Memphis rather than in the foyer. There was still a bit of a problem with noise as the doors were open and people were chatting outside – not helped by the lack of a microphone for the speakers in that room, but it was a definite improvement. Whilst on the subject of microphones, in answer to the organiser, who shall remain nameless, who introduced and closed the day in Chicago by saying both times “you can hear me, I don’t need a microphoneâ€?, “we can’t, and you doâ€?! I know it’s a pain to use a microphone, (and I know other people who don’t like it and think they can get away without) and it’s probably not comfortable, but you can’t be heard at the back if you don’t.

Anyway, back to the Grok Talks – there was a good mix of topics including tips on packaging up your custom controls, a demo of Windows Power Shell and some tips on how to speed up Reflection. Probably the two most memorable were firstly a senior programmer, whose name unfortunately I don’t remember, who did a primarily non-technical presentation about a recent project he led implementing a patient record system in the UAE. Basically by reviewing the project from a business perspective it highlighted all sorts of gotcha’s for other people developing software in foreign countries. In terms of software design, things like other countries having names that don’t fit neatly into the forename/surname structure used here are important, also the d’oh moment when they realised that having a picture of the patient was pretty useless when large numbers of the women wore a burqa was good to share. He also highlighted that the scheduling aspect of the system was complicated by Ramadan as the scheduling algorithm would be different in that period. He also highlighted issues of staff morale, and just getting things done – all useful stuff that some might consider common sense, but are easy to miss on a complex project.

The second most memorable was for totally different reasons. This one was Guy Smith-Ferrier talking about Extension Methods. It was memorable not because of the topic, but because Guy chose to do it as a Pecha Kucha where the presentation is limited to twenty slides, each shown for exactly twenty seconds. After those twenty seconds the slides automatically move on, whether the speaker is finished speaking to the slide or not! Even if you’re not really massively enthusiastic about the subject, the format itself does bring in a strong element of interest as you watch to see if the speaker succeeds or doesn’t manage to keep up. Although there were a few points were Guy fell behind, and even one occasion where he was waiting for the slide to move on, he largely succeeded in coping – maybe an idea to try for more speakers next time?

After lunch I stayed put in Memphis for a Question Time style session on recruitment, not because I was massively interested in the subject, but because the panel included Barry Dorrans on the panel alongside a recruitment consultant. To understand why, have a read of some of the posts on his blog… Anyway, it was a worthwhile session, as there was a good discussion of the pros and cons of going freelance – something I’ve considered before, but rejected – which was an eye opener, particularly the comments from the recruitment consultant about the issues with trying to swap back again. I also felt somewhat better about the lousy pass rate we got on the programming test we gave to potential developers on our most recent recruitment round – the manager on the panel said only one in twenty programmers pass his simple test which sounds much the same as ours. I also came away with a great little test for helpdesk operators too which I guess I’ll have to pass on. As to Barry he was entertaining and animated as always, and managed to not lay in to the recruitment consultant too much – and when he did, about the lack of technical knowledge they have, he largely agreed!

The final session was perhaps the one I had least idea before hand which I was going to attend. Eventually I resisted the temptation of Swaggily Fortunes, and went along to hear James Winters talk about how to write a Facebook application, mainly out of curiosity.

The first thing I learned from the session is that in order to make money from writing for Facebook you don’t need to do anything complicated, indeed James showed us an application that recently sold for about $25,000 that in reality took about three hours to write. To understand why, you have to go back and look at how the Facebook model actually makes money – advertising. Therefore the more users an application has, the more it is worth – so all the stupid little applications that some of the people wanting to use Facebook as a business tool tend to look down on are actually worth significantly more because they generally have many more users than the more serious applications.

The general impression I took away from the session is that a basic Facebook application is actually relatively simple to produce – the real skill is coming up with an idea that has the sort of viral penetration to spread through thousands of users, which is how you can make any sort of money as a Facebook developer. Aside from that the applications are really just web applications, albeit with some functional limitations imposed by Facebook.

Anyway, all in all it was a good day, and I picked up lots of useful bits and pieces – and maybe if I can think up a good idea I’ll make my millions writing a Facebook application… maybe not. Oh and if you’re wondering why I wasn’t micro-blogging along with some of the others on Twitter, blame the Twitter mobile service, as I tried to hook up but it wasn’t until I got home that I realised I wasn’t following the feed, so nothing had worked. Maybe next time…

DDD6 092 and DDD6 125 originally uploaded by blowdart2000.

A SQLServer 2005 Gotcha

We’ve just come up against a bit of a gotcha with Microsoft SQLServer 2005.

We’ve been working on a bespoke system for a customer for the past few months, and have been regularly testing against their test databases that they have been sending over, and they’ve been doing the same. Today the system was set up against their live database for the first time, at which point we got an “Ambiguous column name” error being generated whenever they loaded one of their custom query configurations into our system.

This not surprisingly had us flummoxed for a long while, and it wasn’t until I found this blog post that I nailed down what the problem actually is – SQL Server compatibility modes.

The test database that we have been using was originally created on SQL Server 2000, so when it is attached in SQL Server 2005 the system automatically sets the compatibility mode to 80 – the code for SQL Server 2000. Their live database however has recently been recreated in SQL Server 2005, so when that is attached the compatibility mode defaults to SQL Server 2005, and SQL Server 2005 is a lot more picky about certain bits of SQL syntax. As a result their custom query that works fine on the test system, that reuses a particular column twice and sorts by the same column falls foul of the new syntax rules.

Obviously you can get around the problem by switching the compatibility mode back on the new database, but longer term we’ll need to fix the problem.

It’s not the only change between SQL Server 2000 and SQL Server 2005 either – there is a pretty extensive list of all the changes with a probable impact on this blog post.