Mikester's Blog

Ruminations on Interfaced-Based Programming

One of the maxims of Object Oriented Design and use of Design Patterns is to Program to an interface, not an implementation.  This is one of those maxims that provides enormous benefit when applied, but it is not necessarily obvious the benefits to be had when you first encounter it.  I had a couple of recent experiences that makes it quite obvious to me that this principle is so vital to creating great products.  I'll explain.

Quite a few years ago, I took Organic Chemistry and the professor was horrible and should never have been teaching.  He was unkind and ineffective.  I was having difficulties with a certain type of problem.  At a particular point in Organic Chemistry, you are given a problem to work out the steps to synthesize (make from scratch) a particular compound.  Some of these problems were not too tough, but  more than a few seemed all but impossible.  The professor was absolutely no help.  Finally, I was talking to the Department Head (different professor) and he told me you need to work the problem backwards.  That is, start with the compound you are trying to synthesize, and then proceed backwards one step at a time.  This made a whole lot of sense.  For starters, when you start at the result, you have fewer permutations to look at and you are starting with one datum that has 100% certainty (the product).

Many computer problems can also be worked this way.  My software often has engineering applications and you may know a particular scenario (test data, for example)  that should provide the correct answer.  Well, you can start with the result and work back one step at a time.  The way I do this is to use interfaces for the various objects and then create mock objects, many of which will simply return a value.  So if the result is determined from an equation that is comprised of 10 other values, each of which are calculated by their own algorithm, I can substitute all 10 with 10 mock objects right off that bat.  By doing this, I am able to test my final result algorithm and get that working by focusing only on the final algorithm.  When I next replace a single mock object with a single concrete object,  any errors that appear are clearly in the new concrete object and only that code.  This reduces debugging time enormously.  Development continues by substituting each of the 10 mock objects with concrete ones, and always testing the final result after making one and only one change from a mock to concrete object.

The benefits of this approach are many.  The design stays very flexible, because it is interface-based.  The code produces correct results from the very start.  When the result suddenly is incorrect, there is so little code to check that the debugging exercise is trivial.  You gain a lot of sanity from this.

I also notice that when code gives me trouble, my productivity plummets.  I start thinking about other things.  I don't enjoy it.  It is not fun to figure out something that is a dog's breakfast.  I surf the Internet.  I play a game of Tetris.  I make some phone calls.  I take a walk.  I turn on the TV.  So while doing development by working backwards, or by using Test-drive Development principles may seem to take longer, it actually is much faster, because you get rid of your low productivity periods.

The other thing I like about this style of development, is you are very early on getting a result.  It lacks the specific mechanics because of all the mock objects, but it is always in a working condition.  It's analogous to doing Daily Builds.  The other approach is horrible.  You work days (or weeks) and you have yet to get a full working example to run.  When you are ready to try it and it doesn't run, well where do you look for the errors?  You may have dozens of objects, any of which could be in error.  And more than likely, you have multiple errors and debugging code with multiple errors can take a very long time.  And that is not much fun at all.  I hope this is of some use.

More on Productivity - thoughs on TDD and CodeGen

I took a class last Saturday on Test-Driven Development (TDD)- an all day affair sponsored by the local dotnet user group.  I enjoyed it very much and learned a few things, but perhaps not as much as I hoped.  It was supposed to be very hands on.  The room was full of geeks, perhaps 100 of us, nearly all with laptops, simultaneously typing away.  At least for the morning anyway.  I was not unschooled in test-driven development and in fact, I've read quite a bit, but I have not written tests before I start coding very often.  This is of course a central tenet of test-driven development.  You don't do design or much design.  The design evolves as a result of the development and testing you do.  The design in a sense is done during development of the tests.  You come up with the perfect API in a test for code that is unwritten. In other words, if you want to write an application that calculates the time until retirement, you would figure out the API you would like to use in a test to verify correctness.  So if you want days until retirement as a result, you might envision a very elegant API that gives you this results.  It might be something like:

Person p = new Person();
p.Birthday = "7/12/1942";
int daysToRetire = p.DaysToRetirement();

Your test would then do some sort of Assert to verify that this calculated result is correct.  I actually find this aspect of inventing the perfect API to be quite useful and one that I really believe has merit.  It helps drive clean code.

The other aspect of TDD that I like is the idea that you elevate the importance of testing to the beginning of a project.  This also has a lot going for it.  It also keeps your code in a working state.  You don't go off the rails for very long.  You test often and if there are problems, you fix them right away.  I do believe this will result in higher productivity in the long run.  It may not seem so when you are doing it, but it will shave off time at the end when you are trying to figure out why something doesn't work.

Now if you've been reading what I've had to say about code generation, you may end up wondering like I am, how this all fits together - or does it?  I do not believe TDD and code generation work that well together.  You derive your design with TDD.  In Code Generation, you start out with a working model that you then use to generate code.  You must have a debugged design in CodeGen before you generate code.  If you d0n't, you can generate a lot of buggy code, which is the wrong direction to go.   We want clean code (generated quickly) that works - that is our goal.

So perhaps you can should use TDD in the development of your templates.

I see code generation in being useful in generating tests, and in fact I've used it this way.  I've taken XML data sets that contain known inputs and outputs and created unit tests this way.  It's a great use for it, but it is not TDD by definition.  In TDD, you write a test, you make it fail, then you make it pass.  This shows that the test itself is bug-free.  Then you write the code implementation and now make the test pass.  At this point, you have something very useful.   You have a test that you can run very quickly to ensure code that you've now written still works - and continues to work as that test is run over and over and over again as development proceeds.

I am making an earnest effort to workout the best practices to create software, in particular software that solves engineering problems.   I am amazed at the varied approaches in development software and it sure looks to me like there is a lot of confusion out there.  I talk to a lot of developers and we all share some common goals.  I believe this is best distilled down to quality and productivity.  We want to develop great products quickly and many say you can't do that.  I disagree with that idea.  I believe it is possible to create quality products quickly and I also believe the best methodologies will get you there.  The think I like about Code Generation is you can get quality code quickly.  You can have an awesome, well-tested model and generate volumes of code with a single click.

In traditional Quality Engineering, there was an idea at one time that quality cost money.  It took a Japanese effort that reinvented the automobile industry to change that idea once and for all.  A man named Taguchi believe that cost and quality are related, except his viewpoint was that low quality was very costly.  I believe the same is true in software and six-sigma efforts are now infiltrating the software industry as well.

As you can tell from my ramblings I am not taking a huge stand here.  This is in part due to the fact that I really haven't used TDD as it was originally conceived.  However, I do not believe it is one or the other.  I think if you are working in a domain that you don't understand that well, it may have merit to derive the design through TDD.  But if you are a seasoned developer, familiar with architecture, design patterns, etc., TDD may not be the best approach.  You may already have a great head start and TDD would just slow you down.  If you know the design and have the code generation templates, well you not only have a great start, you are really well down the road.

But so far, the best things I've gotten from TDD so far are:

1. Write tests in the beginning
2. Test frequently
3. Invent the API upfront as part of your design

Points one and two can be used regardless of whether you are a big design upfront guy or a TDD guy.  I also think the idea of inventing your own ideal API can be used very effectively in design and it will help you get clean code that works.

Code Generation and FinalBuilder

I have been bitten by the code generation bug since last summer when I read the book Coder to Developer by Mike Gunderloy.  This is an excellent book that has one chapter on code generation.  It is a very rudimentary introduction to the topic, but once I realized I could have used it to create CRUD stored procedures in seconds (rather than hours), I became very interested.  Repeat:  very interested.  I have used a few code generators that are available including CodeSmith and MyGeneration.  More recently, I've been reading Kathleen Dollard's code generation book.  Kathleen describes a technique for using XSL to generate code.  I had only messed around a little with XSL a couple of years ago and found it a bit difficult and since I didn't have an urgent requirement to use it, I put it aside with a "that's useful and I'll keep it in mind for when I really do need it."

XSL is a wonderful text manipulation language and the bright idea that Kathleen had was to use it to generate code.  I doubt the XSL/XML creators ever had this use in mind, but it is a great technology to use.  Why?  Well, you develop templates for the code you want to generate from a code sample, so the template looks somewhat like the code you will be generating.  Second, it uses XML for the meta data.  Even though XML is a bit bloated in some ways, it has to have some verbosity as it must describe the data it contains.  Since database schema are in XML form, you can generate alls sorts of things as a result.  Third, it is a language that allows loops, variables, conditionals, etc.  So it has all of the things you might need.

Generated code comes in two flavors.  There is code that is generated and never edited.  It is used as is.  Further, Next we have code that is generated but that you may need to tweak.  If you use a code generator, it must be able to tell what is sacred or hand-crafted code from the non-sacred variety (which is safe to regenerate).   One of the better ways to do this is create an MD5 hash of the file and if it's been subsequently edited, you'll know it because of the differing hash.  Such a file is considered one of those sacred, handcrafted files.

There is another aspect to code generation that is important - a guiding principle of sorts.  And that is you must be able to regenerate your code with a single click.  Enter FinalBuilder.   Two key aspects of code generation are being able to do XML/XSLT transforms.  FinalBuilder has such an action.  Furthermore, you can mark a file with an MD5 hash (also a FinalBuilder action) so your scripts will know not to overwrite a changed file.  FinalBuilder also has actions to do basic file and directory manipulations, which are also part of a code generation scripting process.   The idea of writing my own engine did not appeal much to me so I was very happy when the idea came to me to use FinalBuilder for this.

This weekend I was in crunch mode on a project and I needed to work out what was occurring with my application.  I use Raize Software's CodeSite product.  It has a custom formatter feature, that allows you to create an output window pane of data values of your choosing (e.g., properties of a class).  I don't write these often and it's always been a trade-off of time.  Now I have a template so I create an XML file of the output I want and run a script to generate the code.  It works like a charm.

I will probably write an XML generator (when I have a few spare moments) for easily creating XML files that I might need for code generation.  This would revolve around my own business needs, where there isn't already an existing data source or XML schema.   Then I can use a FinalBuilder script to use the XML and XSL templates I develop.

Code Generation with FinalBuilder

There was an interesting post on another site about programmer productivity and how to measure it.  There was a lot of discussion on LOC (lines of code) or KLOC (thousands of lines of code) and there was pretty good agreement that it isn't a particularly good measure.  My own opinion is measuring productivity is fundamentally very simple.  You measure and reward what you want and you can also measure and penalize what you don't.  The key is figuring out carefully what it is you want because I can guarantee you will get what you reward.  I really believe fundamentally that productivity should be measured so you can make mid-course corrections.  If you know what you just did hurt your productivity, make sure you don't do that again.  But you may never know if it went up or down unless you do measure it.  I will talk at length on this topic another day as I have a lot to say about it but for now, suffice it to say that the measure I use personally is "bug free functions."  I will elaborate in another post on what constitutes a bug-free function but I bring this up because I will be comparing my own personal productivity in the coming months against data I collected in prior years.  Since I have changed a lot of things, I expect the productivity will be much higher.  However, I am most interested in the changes I've made recently because I think I am getting HUGE gains by using code generation.

So - I talked recently about an idea I had to use a build tool (FinalBuilder) to script a code generation process.  Well I used it today for a few minutes - yes only for a few minutes.  Why you ask?  Because I was done in a few minutes!  I had actually taken a database table, generated an ORM class by reading the schema and spitting out some XML, applied a template I wrote recently (XSL) to spit out a very nice looking Delphi class.  It was a good test because it is a table with a large number of fields so there was a little bit of everything there.  I then inherited from the class I generated, added some hand-crafted code, i.e., two method calls to do some custom processing.  I was done.  What I accomplished in minutes would have taken hours.  It is also not the most fun type of programming.  It would have involved typing and cut-and-paste and a lot of silly stuff that is very error-prone.  So instead of doing that the rest of the day, I get to write about it instead - which I might add is very, very therapeutic.  Writing about it also solidifies your own understanding.  I'm sure you know exactly what I mean.

The thing about Code Gen is it is a process.  Just like building an application is a process that merits a script to drive it, so is Code Gen and FinalBuilder is perfect for the job and up to it.  I do know I'll be adding some custom actions to FinalBuilder to do some things but even that doesn't look too difficult.  What I am really enjoying is knowing what I am doing is fruitful because the gains are so apparent.  The gains are not 5% or 10%.  They may well be >1000% type gains.  If you can take a task that takes five hours and reduce it 5 minutes, well you've saved a lot of time.  If you can generate 80% of your app that way, well think about the time to market that just went down A LOT.

No matter how you measure it, I am more productive.  Yes, the computer wrote the methods.  But I wrote the software to write the methods.  So I get credit for all the lines of code, or in my case for all those bug-free functions that I just "wrote."  I don't care how you measure it, with code gen vs. the old way, you are more productive - period.  Henry Ford understood automation, economies of scale, and how vital a factor is this thing called productivity.  It should always increase - it least that is my philosophy.  Every so often, breakthroughs are made and productivity moves ahead by quantum leaps, not the baby steps that are more normal.  Code generation is a quantum type leap of productivity.

I will be measuring this very closely with a couple of applications.  I should be starting application #1 in a week or two at the latest.

Code Generation Part II

I had an interesting idea yesterday.  I have only tried this in a very cursory manner so I'm not sure how workable this is going to be.  But I think it is going to work just fine.

Here is the situation:

Code generation is a process that involves a script of some sort to drive it.  Kathleen Dollard has developed her own script engine - she calls it a harness.  I have not played with it but it is very dotnet centric, which is fine.  I love dotnet.  I also love Delphi.  There is more to CodeGen than just doing XML/XSLT transforms.  There are tasks like checking to see if code is going to be overwritten that shouldn't be, creating directory structures, doing multiple transforms one at a time, etc.  It is a scripting process.  Doing code generation for a product involves several things, but the process includes:

1. Developing XML templates
2. Developing a script to drive the generation

Part two is very much like doing a software build.  So why not use a Build tool?  That was my great idea!  So I was playing with FinalBuilder yesterday and noticed it supports XML/XSLT transforms, among a lot of other actions.  So I played with one of my scripts and got it working pretty quickly.

There's a lot more to do here.  I'm not sure I have my arms around this beast completely, but I think this is a good way to go.  Scripts in FinalBuilder are easy to create.  It's very visual, it's easy to move things around.  It's easy to execute just the steps you want, etc.

I'm also thinking that CodeGen might be part of a build process.  For example, the FinalBuilder Code generation script might be put into source control and the FinalBuilder build script would check out the project which also includes the code generation script.  It runs it and files are generated and the project is built that way.  Sounds like a very cool process and I'm sure I'll have a ton to say on it at some future point.

On Code Generation

Quite a few years ago back in the days of DOS programming, I came across a tool that did some rudimentary code generation.  I was very, very excited about it!  To tell you the truth, I never ended up using it very much.  It really was a case of codus interruptus.  I was excited at first only to be disappointed at how little it did. This is not to impugn the author of the tool.  Quite the opposite.  Here was some daring individual that had the audacity to write a program to generate dialog boxes, complete with buttons.  Remember DOS buttons?  Hah!  They were text buttons made to look like a button.  Anyway, I've never forgotten that first case of code generation.  Back in those days, it was not called code generation but CASE, an acronymn that stands for computer-aided software development.

One of the exciting things about Delphi for me was it really did do some code generation.  It setup a project and if you put a button on a form and clicked on it, it put an empty event-handler on the form for you.  It put the method declaration in the class so all you had to do was write the code for the event handler.  This was very pivotal time because the IDEs became part of the development process.
Fastf forward a few years (perhaps 10 or so).  I read a book awhile back called Coder to Developer by Mike Gunderloy.  He mentions code generation as something we should be doi'ng.  He described a tool that enabled you to do, among other things, generate stored procedures for MS SQL tables.  All that CRUD stuff that is not hard, is boring, takes time, and is prone to error.  The technique I always used prior to reading this book was what I call cut-and-paste code generation.  Still way faster than writing it out completely by hand but any time you are using cut-and-paste, you will make errors and that's just a fact of life.  So add time to debug it all.  I had just read this chapter and a month or so prior to this, I had done a project that had legions of tables and I spent a day or so doing cut-and-paste code generation to write these stored procs.  So I did a simple test.  I took a tool I downloaded and generated these same stored procs in about 5 minutes - that included 4:30 to learn how to use the tool to do this.  I  became a convert right then and there.

Since then, I have played with various tools and have found a few that I use quite a bit for .net programming - Code Smith and MyGeneration.  Both are excellent and have their strengths and weaknesses.

I've been eagerly learning more and more about code generation and I started reading Code Generation in Microsoft.NET by Kathleen Dollard.  This book is another pivotal moment for me as a developer.  The method Kathleen advocates is using XML and XSLT to generate code.  Say what?  yes, using XML and XSLT to generate code.

I am going to leave it there because if I open up this can of worms, I'll be writing a book I'm sure and I have code to generate.  So stay tuned!

Getting More Done - Writing Code

I am fairly certain that in our business, getting more done in less time is one hot button.  Unlike most fields, software engineering methodologies and tools change very rapidly.  As an engineer, I can tell you most bread-and-butter engineering uses time-honored approaches.  It takes as long today to calculate the pressure drop in a pipe as it did 30 years ago.  Nothing has changed.  The methodology is the same. 

Computer science is utterly different.  I started writing programs in 1977 in Fortran 66, just before Fortran 77 came out.  There were no if-then-else constructs.  You had to use goto statements.  Well, maybe a Hello, World could be written without a goto, but anything else was spaghetti code.  It had to be.  We also had to use those old IBM key punch devices, along with a deck of cards, one card per line of code.  I then remember going to grad school and encountering the first terminal and was able to use the DEC vax to write programs, this time in Fortran 77.  This was a big step forward.  I also remember dialing up at 300 baud, which made 56K seem like the Jetons.

Fast forward to 1981 when the IBM PC emerged.  The compilers were barbaric.  Anyone remember edlin?  EDLIN is a very archiac, painful editor, which makes NOTEPAD look like the Ferrari of text editors.   It is still part of the OS.  Just drop down to the command prompt, type edlin, followed by the name of a text file and you can go back in time.

I use a product called CodeRush, which is an IDE add-on (there is a Delphi and a .NET version).  It makes editing go a lot faster, once you are up to speed on using the tool.  Now if you compare writing
code with a key punch device to writing code with CodeRush, you see true orders-of-magnitude increases in productivity.  I can imagine one line of code on a key punch machine would take several seconds minimum, not to mention it was impossible to make a correction, because the card had already been punched.  The key punch rooms had plenty of waste cannisters for the plethora of rejected cards.  Just for you youngsters:  after you key punched your cards, you walked over the the computer science building, submitted your deck of cards, and it would take as long as 24 hours for you to get it back.  Early in the semester, the programs would run with one hour turn around, but toward the end, 24 hours was not uncommon.  That is one edit, compile, run cycle.  Geez!

IDE tools are pretty darn good nowadays.  Borland and Microsoft have been good competitors in this regard.  But you can find productivity enhancements in well-written 3rd party tools.  CodeRush is merely one of these.

So what are we really trying to accomplish?  I've boiled it down to some fundamdentals:

1)  Writing code by hand will always be the slowest step, when evaluating as a lines of code per time metric.  It is also the one we probably like the least.  You type by hand the code you are creating.  It is also error-prone.

2)  Minimize the amount of code you need to write by hand by whatever means.  This can be tools like CodeRush, modeling tools, code-generators, and so on.  The less code you write by hand, the more productive you will be.

Let's look at the equation:

The time to write code is the sum of the time to create hand-written code and the time to generate code.  Each of these is a function of the amount of code to generate and the rate at which it is generated.  If we can get the rate up very high, the amount of code to generate does not have much of an effect.  With handwritten code, the rate is how fast you can type in code.  This is where good editing tools will help out a lot.  Increase the rate (the denominator), and the time decreases.  With generated code, the rate of generation is very high.  The time factor comes into developing the tools that enable you to reap the gains of code generation.  But you can see, there are two areas where we can get gains.

The other factor here is the split between hand-written and generated code.  The more code you can generate, the less time it will take - period.

On Developer Productivity

NOTE:  This is very much a work in progress.  I am pretty certain I will have quite a bit to say on this topic.  I have been acutely aware productivity is very important to our success as developers.  I hope to share what I learn to help your own productivity.

Developer Productivity

We probably have all seen the monster coders - the guys that use brute force and high velocity to crank out lots of code.  I'm not sure how good the code is in all cases, but I admire the ferocity.  In engineering school, there were at least two ways to describe a solution to a problem.  There was the brute force approach and then there was the elegant approach.  The brute force approach would be four pages of hand-written calculations with the last line being the correct answer.  It would often take hours and it would often be a landmine of potential errors.  If you were careful, you got the correct result.  Then in class, the professor would use some arcane mathematical identity and transform the problem into a problem that could be solved in two steps.  Our response was, okay so you're a PhD!  But the point is there is often an easy, elegant way to solve a problem and a multitude of ugly ways.  I think of coding as doing tax returns.  Give the same data to 10 CPAs and you'll get 10 different tax returns.  Developers given the same problem will have slightly different solutions in most cases.  In some cases, the solutions may be very different.  If you look at all the solutions, there may well be one solution that is better than the rest.

Anyway, we are interested in elegant solutions - not for the academic reasons, but for the simple reason that it will be less code to write and less code to maintain.  And hopefully easier to comprehend.

As an engineer, I like to model with mathematics or equations.  Why?  Because it is another way to look at something and you just might get a bright idea from it.  Ultimately, we want to shorten the time to market, i.e., the time from the idea of a product to the actual product.  So here is the basic equation:

Time to market = time to develop + time to test

The incorrect thing to do is to short change "time to test" because the product quality goes down, customers are dissatisfied, reputations get damaged, etc.  Testing is one of those things that "takes as long as it takes" - sort of.  There are of course testing methodologies, and I'll get into this later.  So we can shorten time to test, but we should do it by using correct testing methodologies, not by simply shortening the test.

So that leaves us "time to develop" as the area where we can get the gains.  In fact, a good job in this phase will make the testing phase go smoother.  Again, we don't shorten development time by executive decree.  We shorten it by implementing best practices in our software development process.  Training is very much a vital part of implementing best practices.  There is a lot to know in software development.  It's done one thing at a time.  As far as best practices goes, there is one test you can do in a few minutes that will give you some clue as to where you or your organization stands.  It is called the Joel Test.  These are 12 points that you should be living with daily and if you are not, you will struggle to some degree with productivity.  Now I don't agree 100% with Joel on the test, but that's for another post.  It still is a very good test and probably the easiest and best test you can do.  You can use it as a debugging checklist to find areas where you don't measure up.

Development Time

Another thing you can do with equations is "drill down" and find out what each term is really comprised of and so learn more about it.  Development time could be expressed as its own equation:

Development time = time to develop new code + time to reuse existing code

Well-tested, well-written code that can be reused will take much less time to implement than writing from scratch.  If it is robust, it will greatly reduce the amount of time needed in the testing phase.  Now reusing existing code can be done by "cut-and-paste inheritance" or by using code in libraries, class libraries, DLLs, etc.  If it's already written, it can save you a lot of time.

Let's drill down some more:

Time to develop new code = auto-generated code + hand-written code

Now we have new code being written that is generated by some code generation mechanism.  Then we have hand-written code.  There will always be some hand-written code in any project.  This can be reduced a great deal, especially with some bright ideas.  But hand-written code is the albatross - it is the one area that is going to take the longest to do.  It will be an area of likely bugs so will need the most testing.  The more you can reduce the amount of hand-written code, the faster your application will see the marketplace. 

So how much can hand-written code be reduced?  Well my feeling is there is a theoretical limit to the amount of new code that absolutely must be written.  And that is our goal.  This amount willdiffer from application to application.  But it should be a very small amount when the application is designed well and designed with productivity in mind.  Hand-written code should be those areas where the new application is appreciably different than other applications.  That is, the amount of hand-written code should be absolutely unique and not at all similar to anything written.

You see, something is either identical, similar or different.  If it is identical, it can be reused.  That's easy.  If it is similar, it can also be reused, though it is more difficult to do so, at least on the surface.  If it is truly different, then it's time to roll up your sleeves and get busy.