Mikester's Blog

Code Generation Part II

I had an interesting idea yesterday.  I have only tried this in a very cursory manner so I'm not sure how workable this is going to be.  But I think it is going to work just fine.

Here is the situation:

Code generation is a process that involves a script of some sort to drive it.  Kathleen Dollard has developed her own script engine - she calls it a harness.  I have not played with it but it is very dotnet centric, which is fine.  I love dotnet.  I also love Delphi.  There is more to CodeGen than just doing XML/XSLT transforms.  There are tasks like checking to see if code is going to be overwritten that shouldn't be, creating directory structures, doing multiple transforms one at a time, etc.  It is a scripting process.  Doing code generation for a product involves several things, but the process includes:

1. Developing XML templates
2. Developing a script to drive the generation

Part two is very much like doing a software build.  So why not use a Build tool?  That was my great idea!  So I was playing with FinalBuilder yesterday and noticed it supports XML/XSLT transforms, among a lot of other actions.  So I played with one of my scripts and got it working pretty quickly.

There's a lot more to do here.  I'm not sure I have my arms around this beast completely, but I think this is a good way to go.  Scripts in FinalBuilder are easy to create.  It's very visual, it's easy to move things around.  It's easy to execute just the steps you want, etc.

I'm also thinking that CodeGen might be part of a build process.  For example, the FinalBuilder Code generation script might be put into source control and the FinalBuilder build script would check out the project which also includes the code generation script.  It runs it and files are generated and the project is built that way.  Sounds like a very cool process and I'm sure I'll have a ton to say on it at some future point.

On Code Generation

Quite a few years ago back in the days of DOS programming, I came across a tool that did some rudimentary code generation.  I was very, very excited about it!  To tell you the truth, I never ended up using it very much.  It really was a case of codus interruptus.  I was excited at first only to be disappointed at how little it did. This is not to impugn the author of the tool.  Quite the opposite.  Here was some daring individual that had the audacity to write a program to generate dialog boxes, complete with buttons.  Remember DOS buttons?  Hah!  They were text buttons made to look like a button.  Anyway, I've never forgotten that first case of code generation.  Back in those days, it was not called code generation but CASE, an acronymn that stands for computer-aided software development.

One of the exciting things about Delphi for me was it really did do some code generation.  It setup a project and if you put a button on a form and clicked on it, it put an empty event-handler on the form for you.  It put the method declaration in the class so all you had to do was write the code for the event handler.  This was very pivotal time because the IDEs became part of the development process.
Fastf forward a few years (perhaps 10 or so).  I read a book awhile back called Coder to Developer by Mike Gunderloy.  He mentions code generation as something we should be doi'ng.  He described a tool that enabled you to do, among other things, generate stored procedures for MS SQL tables.  All that CRUD stuff that is not hard, is boring, takes time, and is prone to error.  The technique I always used prior to reading this book was what I call cut-and-paste code generation.  Still way faster than writing it out completely by hand but any time you are using cut-and-paste, you will make errors and that's just a fact of life.  So add time to debug it all.  I had just read this chapter and a month or so prior to this, I had done a project that had legions of tables and I spent a day or so doing cut-and-paste code generation to write these stored procs.  So I did a simple test.  I took a tool I downloaded and generated these same stored procs in about 5 minutes - that included 4:30 to learn how to use the tool to do this.  I  became a convert right then and there.

Since then, I have played with various tools and have found a few that I use quite a bit for .net programming - Code Smith and MyGeneration.  Both are excellent and have their strengths and weaknesses.

I've been eagerly learning more and more about code generation and I started reading Code Generation in Microsoft.NET by Kathleen Dollard.  This book is another pivotal moment for me as a developer.  The method Kathleen advocates is using XML and XSLT to generate code.  Say what?  yes, using XML and XSLT to generate code.

I am going to leave it there because if I open up this can of worms, I'll be writing a book I'm sure and I have code to generate.  So stay tuned!

Getting More Done - Writing Code

I am fairly certain that in our business, getting more done in less time is one hot button.  Unlike most fields, software engineering methodologies and tools change very rapidly.  As an engineer, I can tell you most bread-and-butter engineering uses time-honored approaches.  It takes as long today to calculate the pressure drop in a pipe as it did 30 years ago.  Nothing has changed.  The methodology is the same. 

Computer science is utterly different.  I started writing programs in 1977 in Fortran 66, just before Fortran 77 came out.  There were no if-then-else constructs.  You had to use goto statements.  Well, maybe a Hello, World could be written without a goto, but anything else was spaghetti code.  It had to be.  We also had to use those old IBM key punch devices, along with a deck of cards, one card per line of code.  I then remember going to grad school and encountering the first terminal and was able to use the DEC vax to write programs, this time in Fortran 77.  This was a big step forward.  I also remember dialing up at 300 baud, which made 56K seem like the Jetons.

Fast forward to 1981 when the IBM PC emerged.  The compilers were barbaric.  Anyone remember edlin?  EDLIN is a very archiac, painful editor, which makes NOTEPAD look like the Ferrari of text editors.   It is still part of the OS.  Just drop down to the command prompt, type edlin, followed by the name of a text file and you can go back in time.

I use a product called CodeRush, which is an IDE add-on (there is a Delphi and a .NET version).  It makes editing go a lot faster, once you are up to speed on using the tool.  Now if you compare writing
code with a key punch device to writing code with CodeRush, you see true orders-of-magnitude increases in productivity.  I can imagine one line of code on a key punch machine would take several seconds minimum, not to mention it was impossible to make a correction, because the card had already been punched.  The key punch rooms had plenty of waste cannisters for the plethora of rejected cards.  Just for you youngsters:  after you key punched your cards, you walked over the the computer science building, submitted your deck of cards, and it would take as long as 24 hours for you to get it back.  Early in the semester, the programs would run with one hour turn around, but toward the end, 24 hours was not uncommon.  That is one edit, compile, run cycle.  Geez!

IDE tools are pretty darn good nowadays.  Borland and Microsoft have been good competitors in this regard.  But you can find productivity enhancements in well-written 3rd party tools.  CodeRush is merely one of these.

So what are we really trying to accomplish?  I've boiled it down to some fundamdentals:

1)  Writing code by hand will always be the slowest step, when evaluating as a lines of code per time metric.  It is also the one we probably like the least.  You type by hand the code you are creating.  It is also error-prone.

2)  Minimize the amount of code you need to write by hand by whatever means.  This can be tools like CodeRush, modeling tools, code-generators, and so on.  The less code you write by hand, the more productive you will be.

Let's look at the equation:

The time to write code is the sum of the time to create hand-written code and the time to generate code.  Each of these is a function of the amount of code to generate and the rate at which it is generated.  If we can get the rate up very high, the amount of code to generate does not have much of an effect.  With handwritten code, the rate is how fast you can type in code.  This is where good editing tools will help out a lot.  Increase the rate (the denominator), and the time decreases.  With generated code, the rate of generation is very high.  The time factor comes into developing the tools that enable you to reap the gains of code generation.  But you can see, there are two areas where we can get gains.

The other factor here is the split between hand-written and generated code.  The more code you can generate, the less time it will take - period.

On Developer Productivity

NOTE:  This is very much a work in progress.  I am pretty certain I will have quite a bit to say on this topic.  I have been acutely aware productivity is very important to our success as developers.  I hope to share what I learn to help your own productivity.

Developer Productivity

We probably have all seen the monster coders - the guys that use brute force and high velocity to crank out lots of code.  I'm not sure how good the code is in all cases, but I admire the ferocity.  In engineering school, there were at least two ways to describe a solution to a problem.  There was the brute force approach and then there was the elegant approach.  The brute force approach would be four pages of hand-written calculations with the last line being the correct answer.  It would often take hours and it would often be a landmine of potential errors.  If you were careful, you got the correct result.  Then in class, the professor would use some arcane mathematical identity and transform the problem into a problem that could be solved in two steps.  Our response was, okay so you're a PhD!  But the point is there is often an easy, elegant way to solve a problem and a multitude of ugly ways.  I think of coding as doing tax returns.  Give the same data to 10 CPAs and you'll get 10 different tax returns.  Developers given the same problem will have slightly different solutions in most cases.  In some cases, the solutions may be very different.  If you look at all the solutions, there may well be one solution that is better than the rest.

Anyway, we are interested in elegant solutions - not for the academic reasons, but for the simple reason that it will be less code to write and less code to maintain.  And hopefully easier to comprehend.

As an engineer, I like to model with mathematics or equations.  Why?  Because it is another way to look at something and you just might get a bright idea from it.  Ultimately, we want to shorten the time to market, i.e., the time from the idea of a product to the actual product.  So here is the basic equation:

Time to market = time to develop + time to test

The incorrect thing to do is to short change "time to test" because the product quality goes down, customers are dissatisfied, reputations get damaged, etc.  Testing is one of those things that "takes as long as it takes" - sort of.  There are of course testing methodologies, and I'll get into this later.  So we can shorten time to test, but we should do it by using correct testing methodologies, not by simply shortening the test.

So that leaves us "time to develop" as the area where we can get the gains.  In fact, a good job in this phase will make the testing phase go smoother.  Again, we don't shorten development time by executive decree.  We shorten it by implementing best practices in our software development process.  Training is very much a vital part of implementing best practices.  There is a lot to know in software development.  It's done one thing at a time.  As far as best practices goes, there is one test you can do in a few minutes that will give you some clue as to where you or your organization stands.  It is called the Joel Test.  These are 12 points that you should be living with daily and if you are not, you will struggle to some degree with productivity.  Now I don't agree 100% with Joel on the test, but that's for another post.  It still is a very good test and probably the easiest and best test you can do.  You can use it as a debugging checklist to find areas where you don't measure up.

Development Time

Another thing you can do with equations is "drill down" and find out what each term is really comprised of and so learn more about it.  Development time could be expressed as its own equation:

Development time = time to develop new code + time to reuse existing code

Well-tested, well-written code that can be reused will take much less time to implement than writing from scratch.  If it is robust, it will greatly reduce the amount of time needed in the testing phase.  Now reusing existing code can be done by "cut-and-paste inheritance" or by using code in libraries, class libraries, DLLs, etc.  If it's already written, it can save you a lot of time.

Let's drill down some more:

Time to develop new code = auto-generated code + hand-written code

Now we have new code being written that is generated by some code generation mechanism.  Then we have hand-written code.  There will always be some hand-written code in any project.  This can be reduced a great deal, especially with some bright ideas.  But hand-written code is the albatross - it is the one area that is going to take the longest to do.  It will be an area of likely bugs so will need the most testing.  The more you can reduce the amount of hand-written code, the faster your application will see the marketplace. 

So how much can hand-written code be reduced?  Well my feeling is there is a theoretical limit to the amount of new code that absolutely must be written.  And that is our goal.  This amount willdiffer from application to application.  But it should be a very small amount when the application is designed well and designed with productivity in mind.  Hand-written code should be those areas where the new application is appreciably different than other applications.  That is, the amount of hand-written code should be absolutely unique and not at all similar to anything written.

You see, something is either identical, similar or different.  If it is identical, it can be reused.  That's easy.  If it is similar, it can also be reused, though it is more difficult to do so, at least on the surface.  If it is truly different, then it's time to roll up your sleeves and get busy.

 

Greetings

Greetings!

My name is Mike Stephenson and I am a self-employed engineer and software developer.  Naturally, my interest in software includes engineering applications.  I started the website engineering-software.com back in the early days of the Internet.  My original plan was to catalog and resell engineering software so engineers could go to the site and have a fair chance of finding something that will ease their lives.

I started doing freelance work as early as 1989, mostly in the form of engineering consulting.  I was working at a company called Solar Turbines (owned by Caterpillar) and I was given an assignment to develop a data acquisition software application for a test rig that we were developing.  I always enjoyed programming, so I dived into it and have never looked back.  Around that time, I decided I was going to do what I am now doing.  Decision is the first step.  I had very few skills.  I knew Fortran programming and I had just learned QuickBasic, but neither were going to allow my to develop real, saleable engineering applications.  So I decided to learn C.  It was C that was being used to produce most commercial applications though Turbo Pascal was also being used for this.   C allowed me the opportunity to directly access the video memory and you can make apps very responsive.  Ever since then, I look for tools that enable me to create responsive applications.  So I purchased Borland's C compiler (this was also in 1989) and I quickly learned enough to be dangerous and then decided object oriented programming was where it's at, so I dived into C++.  I then became a serious, hard-core C++ developer and started picking up freelance gigs.  I even wrote a few articles for the Borland Developer's Journal (now defunct).  I even dived into Pascal around the mid-1990s and this sequed quite nicely into Delphi, which debuted in Februrary 1995.  However, I waited a few months (I never much liked being a guinea pig).  So I picked up Delphi in October 1995 and had my first real project in Delphi before Christmas.

I instantly developed a love affair with Delphi.  It was elegant, fast and the IDE was slick and unimposing.  Anders Heilsberg really is a remarkable individual and here's a great big thank you for making my life easier.  I use Delphi to this day for any type of Win32 development, though I do have an occassional foray into C# WinForms.  Of course Anders is responsible for C# as well.  As a language, I like C# as much as Delphi.  I prefer the performance of Delphi apps for native 32-bit applications.  Delphi trumps C# (and VB) in this regard.

When it comes to the web, my chosen environment is C#, asp.net web applications.  Having gone through the growing pains of ASP development, I can assure you this is the right way to write web applications.  I think .net is a very well designed framework and I really believe this is one case where Microsoft hit a home run on its first try.

 

I decided to start a blog for a few reasons:

1)  Others have encouraged me to do so
2)  I really enjoy writing - it clarifies your thoughts
3)  I enjoy helping people and I hope this blog helps you even in some small way