Working on a script for extracting MS numbers for patches for work.

The following command allows 7zip to extract the needed files without extracting the hundreds of thousands of other items in a giant lump.

"c:\Program Files\7-Zip\7z.exe" x -ir!x/* cabs/package*.cab"

Bloody useful

Permalink | Leave a comment  »

Neat little way of thinking about logical vs syntax errors.

Ever had to hunt high and low for a reason why something is not working as intended? Ever had an if statement that always evaluates as true?

Yes?

So, all of you are probably aware that if the IF statement evaluates as true all the time chances are you’ve used an assignment operator instead of a comparison (aka = instead of ==). Thing is, in a mountain of code it can be a nightmare to find that out, so how do you prevent it from ever being a problem?

Think backwards.

Instead of:

IF ( favChocolate == “buttons”){

                Echo “He likes Cadbury buttons!”;

}

Use:

IF (“buttons” == favChocolate){

                Echo “he likes Cadbury buttons!”;

}

What does this do? Well in non-interpreted languages if you accidentally type:

If (“buttons” = favChocolate){... then it will result in a build error. You can’t assign a variable to a string.

If its interpreted it’ll result in a runtime error as again, you can’t assign a variable to a string J

Thus eliminating the guess work involved in finding logical errors in your tests.

Permalink | Leave a comment  »

Recently I had to fudge some data so that it would be imported into a database after an outage caused our "php" data loader to try and allocate a crazy amount of memory and die fantastically. Being a fan of automating everything I can I started out down the trail of "okay lets script this".

A few moments later I had a simple bash script looking somewhat like:

#!/bin/sh

filename=$1

while read line; do

#Read each line and grab the necessary fields, create the insert statements.

field1=`echo ${line} | awk {'print $1'}`

field2=`echo ${line} | awk {'print $7'}`

echo "INSERT INTO testtable VALUES ('${field1}',UNIX_TIMESTAMP($field2}));" > data.in

done < ${filename}

#Assume all is good and just feed the file to mysql for processing.

mysql -u root testdatabase < data.in

# EOF

Simplistic script - using my favourite awk statement for field delimitation (so much easier than relying on cut). Now im aware awk can do a lot more however everytime I pick up a book or decide to learn some I end up being told "ITS URGENT!" and having to drop everything to "just get it working". Its a common problem I feel when you're really just support staff trying to keep everything hunky-dory.

This time around while work finished at 17:30, it was now 19:00 and my script was still running having been set off at about 10am. A quick wc -l and some dodgy division told me that it still had about another 56 hours to run. I was processing 2,647,012 lines and wasn't even above 500,000 lines yet.

Although I had backgrounded the process I didn't want it to fail without me knowing. Anxious and having already done 1.5hrs of unpaid overtime I decided to see if there was a better way of doing the job. Sure enough AWK reared its head again its a turing-complete programming language specifically meant for text processing so why not have a decent look at it now that everyone else in the office has gone home and you're stuck here until its done.

So a poke around the internet for a bit of guidance in AWK and I come out with a solution of.

#!/bin/sh

filename=$1

awk BEGIN{

# Special characters represented by octal values to prevent any escaping issues.

q="\047" # single quotation mark

lb="\050" # left bracket

rb="\051" # right bracket

c="\054" # comma

sc="\073" # semi-colon

}

{

print "INSERT INTO testtable VALUES " lb q $1 q c "UNIX_TIMESTAMP(" $7 rb rb sc >> data.in

} ${filename}

mysql -u root testdatabase < data.in

#EOF

Time taken for processing: under 90 seconds.

Time taken for bash to process: Over 60 hours.

Just had to wait then for mysql to catch up and import the 2.6 million entries. Left work at 10:30pm with the solution in place.

There was a further issue with it. Every hour we collect new data and normally the php data loader works through those few thousand lines in a few minutes and no issues, however with the backlog we had mounted up it could no longer be trusted. Unfortunately the bash solution also took so long that while it processed entries the "queue" of entries got longer every hour as it was falling behind. Now with the awk solution in place it takes less than 30 seconds to get the entries into the database.

Lesson Learned: Never use bash while loops for iterating through large text files.

Permalink | Leave a comment  »

Decided to open this blog back up for business after a small hiatus and a rebranding. Here's hoping I get some decent posts up here along with some nice subscribers too!

Permalink | Leave a comment  »