Debug Whack-a-mole

 General, Projects, Software Engineering  Comments Off on Debug Whack-a-mole
Aug 122017
 

Have you ever been in this situation?  You’ve been on a project for months, you’ve poured your sweat and tears into code, design, documentation, and now it’s finally time to see something work.  You flip the switch, you run your program, and it has a bug.  No big deal, bugs happen!  You fix it, it takes a few minutes, maybe an hour, and you are ready to go again.  You flip the switch, you run your program, and it has a bug.  Well, at least it’s in a different spot this time, you made it slightly farther in the application this time.  You fix it, this one takes a couple hours, it’s nested down a few layers.  Now though, it’s fixed, the time is here, time to use your glorious creation.  You flip the switch, you run  your program, and it has a bug.  Hmm, you didn’t get quite as far as you did last time.  Was this where your first bug showed up?  You start to dig in, but this looks eerily similar to the bug you first fixed.  Oh well, fix in, only took a few minutes, and it’s that time again.  You flip the switch, you run your program, and it has a bug.  Ok,@#%!, seriously.  It got farther this time, something new, something to start digging into.  Fast forward a little bit, the next morning let’s say, and you’ve knocked out a couple other bugs but you are finally actually using your program.  You flip it on once more to use it, and bam bug….does that bug look like our very first one again?  How, in the name of all that is good in universe does this keep happening?


If this has ever sounded like you, or maybe sounds like your right now, let’s talk about why that might be happening and what there is to be done about it.  I know it’s something I will run across on occasion, especially in my earlier years of programming. It is especially bad when someone would give me a stack trace or I would see an error log, and I immediately jump in fix it, deploy it, turn it back to a QA for testing.  It’s what I like to call “Debug Whack-a-mole”

.

Can we find some common times or reasons that this tends to happen?  Personally, if I open code and make a fix based on an error report or log, without then running the program and checking the result, it happens probably more than 50% of the time.  If I am continually iterating over a large project that I am build from the ground up, and am just fixing bugs as I come across them…well, I’m not sure.  Why am I not sure? Well, often times I won’t write these bugs up.  I will see them as I am stepping through code, or I see something look off in the app, so I just put a fix in and move on.  So, how can I remember if I’ve already whacked this particular mole already?

From this, you probably are coming to the same conclusions I am.  There seem to be two big things that stick out, documentation and testing.  How can these be improved?  Will these actually help? What if these things aren’t required by your company, or this is just a personal project you are working on and formal documentation isn’t required?

image

I don’t want us to fight, I know there are some that hear ‘testing’ and ‘documentation’ and are ready to hurl things around their office, cubicle, nook, etc, but hear me out.  I won’t give you the normal party line for why it’s important and that it’s best practice, I just want to say one thing.

By not testing, by not providing ample documentation, you are wasting your own time.  There, shots fired.  I know, I really do, I get it.  You want to write code, we all do.  You write to sit down, listen to some music, crank out code, be a rock star.  Whether it’s work or personal projects (perhaps even more so for personal work), you need something to help you remember how things work, what your original intentions were, and to save yourself from, well…yourself.  You don’t future you to make an innocuous bug fix, breaking your baby!  Maybe, we can have a better discussion now.

Friends again? Great! I want to share what I have found has helped me, the most, to combat this insidious plague.

1 – Commit often

In the new age of git (and probably other code repositories), the mantra is “commit often”.  Why? What does this do for us, what good do we get out of it.  Aside from having very small unit of work that we can roll backward or forward, it also leave behind a great paper trail in your repo history.  It becomes like a stream of consciousness, change log.  Any time you are wondering, did I see this bug before, go look at your history.  Is there an entry that mentions something similar?  If so, it’s a great place to look. 

Image result for rockstar meme

Now, we are at the same cross roads as before.  This advice sounds great, you probably hear it a lot, but how does this help me Mr/Mrs RockStar Engineer?

First, small commits with comments, improve your overall documentation.  Even without comments or documentation in your code itself, you now have a small change set delta with a very pointed commented as to what you were doing, what you were fixing, or why you were making this commit.  Multiply that by dozens of small commits a day across weeks and months, and you have a very verbose and complete

Second, it makes sure you never loose work.  More than once, back in the SubVersion (SVN) days, I would have a days worth of work checked out on my machine.  This doesn’t seem too bad, until one harrowing day when I was walking around with my laptop and it fell (buttered toast style), landing flatly on the screen and keyboard.  When I picked it up, all seemed fine, the machine still ran, nothing to worry about.  Until, about 5 minutes later, when WindowsXP let me know that its delayed writes to disk were failing. Oh god, oh god no.  The disk was ruined, it later turned out that I could not even get an external hd case to help me recover.  I lost a full days worth of work.  It was devastating for me, mostly because it was just code for a personal project.  I didn’t HAVE to work on it, and never did re-write any of what I lost.  That side project died that day.

Lastly, it also makes it easy for you back out changes that cause problems. I can’t be the only one to go through a day of refactoring or a few days worth of cranking on code only to find that things have are imploding?  Maybe something less dramatic? Maybe, You’ve made a few bug fixes and run some tests on it, but somehow something big has broken deep in your code.  All of these and so many more can be fixed by rolling back some of these super small changes you’ve been making.

2 – Unit Tests

I am not a TDD evangelist, I probably haven’t given TDD a fair enough shot, but I do believe whole-heartedly about unit testing your code.  Whenever possible, break you code down into small parts that you can write some test around.  It can be hard to start doing and sometimes feels like you’re doing too much or not enough.  If it makes you feel any better, I often get the same concerns when I’m writing them too.

The guidelines I usually give my students  and mentees (and the bare bones that TDD would espouse), is that every function should be tested (having a hard time knowing what your functions should really be, try writing/drawing out what your program should do) .  Write your classes so that data can be ‘mocked’ up, even if the algorithms/functions to do that are hidden away from you main consumer (interfaces are great way to hide many of these).  When you have classes and functions that can be tested individually, write down checks for all the things you would be checking for manually.  I have a notorious bad habit (bad may be a strong word) of stepping through parts of my code that I am unsure of as I am running test scenarios on it.  Wouldn’t that be better as a codified test that is reproducible and helps keep the quality of your code up to par?

There are so many places to go, to find information about unit testing and how to add tests to your specific language choice, I won’t get into them.  I will say that in the C#, MS unit test is more than sufficient, especially when coupled with something like Rhino Mocks for mocking out external dependencies and making sure you have written code that relies on dependency injection for passing most if not all classes (see: coding to an interface, not an implementation design principle).

3 – Bug Tracker

Now, I don’t mean you should be using bugzilla or TFS (though, to be fair, I’m not saying that you shouldn’t) but do something.  If you run into a bug, write it down, make an item for it, do something.  Don’t just see it, fix it, and move on with your life.  If you are working on anything other than some throw away code or PoC, then you will most likely see some tale-tell sign of this bug in the future.  So, write it down, even if it is in the code somewhere, write it down.

4 – Comments

That brings us to my last bit of advice, comment the code you are changing to reflect why you did it.  I can’t count how many times I’ve been reading/stepping through code and come across something that looks odd.  I will think on it, maybe run some test stuff through it, then say “Well that seems like a dumb change, I think it should be this other way”. BAM, bug is back. 

Now, that exact statement can be changed a bit to be more general.  Comment anything that is out of the ordinary or isn’t immediately recognizable.  Not just to you, but even to the people who will come after you and maintain or enhance your program.  It is easy, in the moment, to know exactly what your code is doing and why, it won’t be tomorrow…or next week…or next month.  I have written too much code that even the next day I end up saying, “why did I do that?”.  Sometimes the answer is, I was tired, I was dumb, some rare times it’s I had a clever epiphany on how to solve a problem elegantly.  A clever, elegant solution isn’t worth much if you forget it the next day and take out, right?

Let’s stop debug whack-a-mole.  Let’s help others stop debug whack-a-mole. 

Keeping your code clean by picking up garbage and not allowing vandalism

 General, Software Engineering  Comments Off on Keeping your code clean by picking up garbage and not allowing vandalism
Dec 082015
 

One thing that I find to be the most important aspect of my job (jobs), is to ensure that code quality doesn’t end up at the bottom of the priority totem pole. My current company has the look and feel of a young start up, despite the fact they have been around for nearly two decades now. I know what you are thinking, “That’s great!”. And guess what, you’re right! Rock star engineers get perks that rank and file don’t. Engineers run the company and the business tends to play pick up. We make money, we make great things, we play with new toys and tech.

When could this scenario be bad? While this question may spawn multiple posts in the future, the main event for this, dear reader, is to pick on code quality. Start ups do one thing, code. They code fast, they fail fast, they fix fast. They also leave a lot of bad code and hacks in place all over gods creation. No need to clean up, no need to follow best practices, get it working and get it out the door.

Why would a start up be like this? A big reason is money. Day 1, the company is losing money. Get a product, get it working, get it to customers so money comes in. At some point though, you reach some sustainability. Money is coming in, take some time, pay some technical debt.

This sounds easy, so what’s the big deal? Aside from legacy systems that take years to convert or refactor, there is a strange phenomena that tends to occur among developers, managers, directors, and architects. There is garbage code there, a little bit more wont’ hurt. The system is already slow in this portion, a bad design choice to get the work done faster won’t hurt much more. After months or years of this thinking, you end up with a system on the verge of technical bankruptcy.

Think it sounds crazy? Well, I am not the first one to suggest such things, but I haven’t run much on the topic in a while. I also want to take the time to draw some similarities between a couple different sources.

First, Netflix had something interesting to say that was indirectly related to this topic. In one of the presentations they made about their company culture, one of the chief officers made a comment that ‘We see a piece of garbage in the hall, we stop and pick it up’. Ok, paraphrased, but look at that sentence. He continued to describe this simple proverb as, each employee should feel the devotion to the company and have such a high sense of worth with respect to the company, that they are not just willing but will automatically engage in behavior that will maintain, if not better, the company on a daily basis. If you see a piece of garbage in the hallway, you pick it up. Not because someone expects it, not because you get a promotion, not even because you want to save the planet. Simply, because you love the place you work and want it to be good.

But that isn’t code, or is it? You see garbage code when you are making a surgical change to fix a defect. What do you do? You pick it up. You refactor the code on the spot if possible and write it up, test it, and ship it out. Is the piece of garbage code bigger than just a crumpled up piece of paper or wrapper? Is it more on the size of a small car? Then make a plan to clean it up. No one expects a developer to be superman and fix 1,000 lines of code when they see it sucks, but make a note of it. Make a bug, send an email. What ever you do, don’t turn a blind eye to it!

Netflix not good enough? Fine, let’s pull from outside the tech community. The broken window theory offers an interesting look at social phenomena that can easily apply to code quality on large development teams. The theory comes from the following example [wikipedia]

Consider a building with a few broken windows. If the windows are not repaired, the tendency is for vandals to break a few more windows. Eventually, they may even break into the building, and if it's unoccupied, perhaps become squatters or light fires inside.

Or consider a pavement. Some litter accumulates. Soon, more litter accumulates. Eventually, people even start leaving bags of refuse from take-out restaurants there or even break into cars.

Let’s change that story slightly. Consider a code base with a few unused methods, a few poorly written classes, a few sloppy database stored procedures, some bad table designs. If these aren’t fixed, then the people living with the code, will make more bad decisions. They will break more windows, they will add more sloppy code, they will accept database contention, they will take 50 lines to do 2 lines worth of work. Why should they care? The code is already littered with garbage, one more piece is never the problem, and they are on deadlines or have lives to get back to.

My challenge to myself is simple. Don’t add garbage to the pile. Note where the piles are when I see them. If I can’t change them on the spot, if they are too big to clean up alone, or I am just too busy, research and document the issue. Talk to a manager, a lead, someone more senior. Your the senior most person? Guess what, the people living in that code are looking to you. Fix the windows, pick up the wrappers, and please, for the love all good things in life, don’t just pretend it’s not there.

Returning

 General, News  Comments Off on Returning
Nov 292015
 

The last post here is from Sep 2012, so needless to say, it has been a little while since I have put any time into this. However, that is going to change. I am making a commitment to document some faucet of my tech world on a weekly basis. In the time since I stopped writing, I stopped my studies toward my PhD and have taken two jobs. One took me to PNC bank in Dayton as a Sr Engineer and the second brought me back to Louisville as a Sr Engineer with a large software house in my home town.

The time at PNC provided me with some fantastic exposure to very talented engineers and strong leaders. I was able to take a lot from that experience, hopefully those experiences will translate well into some meaningful posts here. My time so far in this new company has given me a lot of opportunities to apply lessons learned and grow not only as a developer but also as a leader. I have also been delving deep into the guts of .Net and C#, focusing on performance and back end processing. This has caused me to become a source of knowledge for my team, as well as being a ‘Data Steward’ (MS-SQL source of knowledge).

In addition, I have started numerous projects, both of a technical nature and related to my newly purchased house. I will be documenting both, to some extent. Do I have curious as to what some of those may be? Well, here is a hint:

  • Roslyn and Compiled c# rules engine
  • 7 Days to Die server admin tool
  • BattleField4 automated server populator – chrome extension
  • More project Euler
  • lunch and learn presentations – some can be made public
  • built in book shelves
  • new deck
  • water cool gaming pc build

I will tell you more soon, don’t worry, no more 3 yr disappearances.

Hello World

 General, Projects, Web  Comments Off on Hello World
Jul 272012
 

That’s how these things are supposed to start right?

The first thing I need to decide is what all needs to be shared on here.  I would like to tell the world of my projects and my research.  Perhaps a bit of back story on what my current projects are and what I intend to research would be in order.  Currently, these web pages have been taking a good deal of my time.  Learning new platforms (such as wordpress, phpnuke) and delving into php and json.

Aside from that, I’ve also been putting a lot of time into researching for my phd work.  The main focus, so far, has been image processing and artificial intelligence.  I am working on image classification and how AI techniques (such as AdaBoost, Boosting/Bagging, etc) can enhance already established classifiers.

I also have small projects going with Arduino, Kinect, .Net, and Android.  Though, I’ve not found much time for anything during the semester.  The plan is for my findings and documentation to be put on here. I need better note taking and documentation and this should help out with that significantly.