Thursday, May 31, 2012

Sign of the Times

There's more and more incidents in the public domain where a security issue has seriously tarnished an organisations brand identity. Worst still is the simple fact that it could have been avoided in most cases.

Some of the most notable in the last 12 months that I remember were when the MySQL website was taken down using a SQL Injection attack vector, and the more recent PHP exploit on militarysingles.com (link takes you to the Imperva report PDF).

"Why are these such a big deal?", you may ask. Well from a distant perspective the former details an organisation (now owned by Oracle) who specialise in database systems. Even though the worst effect for customers was to have their details taken and published, it wasn't actually a vulnerability in their product. It was down to the website team not approaching the project with defensive coding in mind.

It was a political coup over brand identity.

The latter example (militarysingles.com) is both an exploit of a vulnerability as well as a lack of defensive coding for the particular scenario in question. The problem there is that gaining access to customer data in this instance means potentially sensitive information on military personal across the globe.

It's almost as if there's a lack of general awareness about key issues, and I can only draw from first-hand experience of contact with developers. For instance, I've done over 200 telephone and face-to-face interviews in the last few years for what is effectively around 30 or 40 actual roles. It's hard for me turning down applications from good developers when we find them but we're after a very specific skill set and way of thinking - especially with regards to security issues or vulnerabilities.

Hypothetical interview question: Array processing and sorting, with the ideal being some sort of pseudo-code or algorithm that details specific mechanisms or sorting. The answers to this vary but most people seem to respond with LINQ extension methods that do it all for you, without any real explanation. As an interviewer it's best to give as much chance as possible so you take the toys away and ask the question again (this time, only using .NET 2.0, or just C if it's familiar).

It's not really about the answer so much as how you come up with a solution but it's also surprising how few people think at this level. It's like all the available frameworks have dumbed-down general technical knowledge. Yes, they're useful in given situations and, yes, they make developers lives easier...But easier doesn't mean better*.

We used to ask a set of questions almost made up on the spot, but that's evolved to a question list - Some candidates recently made comments about some of them being straight from the interweb so I've added a few more different ones over time.

Some questions relate to defensive coding practises and are quite revealing in terms of the level of awareness of these attack vectors. Very few developers seem to know any more than how to prevent SQL Injection in .NET, and throw in a few answers about SSH, firewalls and DDoS mitigation. The last three are good to be aware of but only the third could be dealt with in any way by some development work - It's usually better to get something like RioRey or BlackLotus on the case.

Even the basics will only get you so far - The classic is hashing stored data. Great, but the other guys have read the same book and can break it. MD5 stopped being effective nearly a decade ago, even SHA-2 will have a shelf life defined only by "the other guys" intelligence. Maybe processing power too. CrackStation has a pretty good article actually explaining not only how but why.

At my current employer we have to take the topic very seriously - We're a global organisation who is a big target for black hats, hacktivists, competitors, money launderers and other such activity; so if there's a security problem it's got to be dealt with there and then. The sources of the majority of attacks are unsurprising, as are the sources of screen-scraping attempts. I supposed psychologically if it's not a physical attack, it's more difficult to easily quantify - which can make it more difficult to justify the additional expense for shoring up the defences to the board of directors in some places.

However, RSA were actually physically broken into and later some of their clients had their systems compromised as an indirect result. [RSA Labs has a pretty good collection of docs + discussions here]

There's always something out there that you don't know about - No system is impregnable when faced with a sufficiently well motivated and financed group of people - but you can do a lot to ward off and discourage the majority of infiltrators.

You / me / an individual can't know everything, that's why having a good team is important but here's a good place to start... OWASP (have a read of the reference area)

*(Comments about knowledge levels are generalisation only, there's a huge number of really good developers and specialists if you're lucky enough to find and hire them)

Monday, May 28, 2012

Altitude


Very relaxing weekend for me last weekend, how was yours?

Signed up at a gliding club nearby and they got me airborne on the day. I'd forgotten how much fun winch launches are :) The instructor was pretty nice about it - "Yeh, you can definitely fly a plane, you're just a bit rusty". I'll happily take that!

Pretty impressed with the new phone, too. Seems to cope with pretty much anything although I'm not sure I remember the pre-flight briefing from the stewardess mentioning turning off my phone so...

Please note: I was P2 in the aircraft and not in control at the time this photo was taken. Do not try and juggle text messaging with aviation.

Got a couple more ideas for phone apps and some sunburn out of the day. Sometimes you need to get up and away from the grind to really relax.

Good bunch at the gliding club and a far better way to spend a sunday than sitting in front of the TV.

Tuesday, May 22, 2012

Time Machine

It's been an interesting week - Got a replacement phone and decided that Nokia Lumia 800 is the way forward. I had to downgrade my HTC Desire to a Blackberry due to work, so going back up to a smart phone has been a real joy.

What I didn't know was that the Desire and Lumia share the same processor; surprised me as the WP is slick and has a great UX. The Desire ended up grinding a bit and was always running out of storage so we'll see how this goes. I've already lined up an idea for WP app, so as soon as Expression arrives I can get stuck in with some dev time.

Also feels like I've been set free just knowing there's 12 weeks left to go before starting work for myself. Paperwork's largely sorted so time for a beer in the garden I think.

Cheers!

Tuesday, May 15, 2012

Great scott, Marty!

I handed my notice in yesterday in what will be the biggest career change I've experienced in my short career.

When you take an objective view of any situation you're involved in and realise that nothing about that situation benefits any of your own goals or objectives, sometimes you come to the conclusion that it's just not worth doing.

The time I've had at this company has been a huge benefit - I've learnt so much about so many subjects and had the chance to work with the most highly skilled group of people I've ever met, but from this point I'm not going to learn anything new and there's virtually no opportunity to progress career or add to my achievements.

It's not as if there's any animosity or resentment either, if I can I aim to ensure my departure is a blip rather than a bump and that there's adequate continuity.

It boils down to time - I can't get to my long term goals without time spent with the latest technologies on my resume, and potential employers aren't interested in talking to someone with the length of notice clause I have in my contract. I need to apply my core experience to more specific areas in commonplace use. Some of the clauses in the contract prohibit particular blog topics too so it'll be akin to being set free in some respects (I won't miss being permanently on call either :) ).

When I was in Alberta earlier this year, the guy organising the NHL trip to Saddledome spoke about his motivations and experiences in setting up his own business. Struck me that it's (whatever "it" may be for each of us) a lot easier than we think or fear.

The Axiom Of Larry: "Make it happen".

I've also quit smoking with the help of the local quit clinic. So really, it's all about time and travel.

1.21 Gigawatts

Friday, May 11, 2012

Bogus Chroot

I saw the posts about the Google Chromium OS earlier this year and thought I'd take a look. An OS orientated around the old thin client principals perhaps?

From a personal interest level I thought I'd have a crack at compiling the source code myself and seeing what's involved so got the latest Ubuntu running on a VirtualBox VM and started having a play.

That was four weeks ago, and I've been distracted with tinkering on Mono and Java since. Linux has come a long way since I last used it back in around 2005 and I'm pretty rusty! Managed to follow the build instructions all the way to the point of actual build but then get a message that I have no idea about.

"Warning: Possible bogus chroot detected"

Maybe I should have just downloaded the redist and been content with that but for now, converting some of my own-time projects in .NET 4 to Java and Mono will keep me out of mischief. I quite like the Ubuntu environment: if you're a Microsoft-er with a familiarity with C# & PowerShell and have never used linux before you'll pick it up pretty quickly.

It's interesting to have a look over the technology fence and see how the Jones' have been doing lately, but I don't think I'd want to move (they're more than welcome to come over for dinner though).

Monday, May 07, 2012

Tales of the IUnexpected<Int32>

Being curious of the comparison of LINQ and classic actual-code (henceforth referred to as CAC) techniques, I wanted to know how big the gap was. You can tell how exciting my bank holiday Monday has been so far.

Wonder what would happen with strings? Reference types and immutable - good string operations can make average hardware work far better.

Generally I thought the major factor would be an ordinal comparison - or lack of in the LINQ mechanism, for example, the difference between:

There's about a 20% performance [speed] improvement on the optimised route so tried the LINQ version with and without this optimisation but it only seemed to have a very small benefit - probably around 3% tops.

The lhs / rhs values make a difference too - if the comparison finds a difference in earlier characters of the string it won't bother comparing the rest of the string [Assumption: you're doing a case-sensitive comparison]  and if the string length isn't equal it won't bother running the comparison at all, so tried a few combinations of differently cased characters in different positions in the two strings each with the same letters.

Hit the test five times each on debug versions of the exe outside of VS2010 to reduce complications and get an average to make it fairer.

So generation of the source for comparison done on a quasi-random character basis, but so that both method calls would use the same array:


Then use the result for the following comparison:



On the string comparison routines you can only really see the improvement with a string[] of around 1M elements but all the way from 100 elements up to the million the CAC version is faster (by about 100ms at the lM element range) for the same source string. That source may or may not contain the character it's checking for.

I would have expected a bigger difference with the predicate too (commented out in the above example) and both value and reference types to experience the same performance footprint. But would you expect numerical operations to have the same behaviour?

So if we play with performance for a very specific scenario where an array of integers is iterated through, and any element which is even is selected into a result set. So the CAC method is pretty much just a straight for loop (not a foreach):
I purposely created a variable scoped within the loop to hold the value from the particular slot rather than access the array element twice.

And the LINQ operation is a simple from object query:
The IL for the different mechanisms was pretty varied too. Will come back to that.

When I changed the size of the source array which both methods search on I found that below 450000 elements the CAC method was quicker than the LINQ version. Around 500000 elements they averaged about the same but when expanding the test source to over 600000 array elements the CAC version was actually marginally slower.

Mind you, we're talking of the result set examples as 1ms and 3ms for the smaller source arrays, 13ms and 14ms for that middle-size, around 20ms and 15ms for the large array measured across CAC and LINQ versions respectively.

Some LINQ mechanisms outperform CAC equivalents with much larger arrays? I suppose it's like the difference between Hashtable and Dictionary in that Dictionary performs better for smaller collections, but Hashtable far better with large collections. HybridDictionary is ok but there's a bit of a hit if the collection size moves over the threshold too often.

The IL seemed to show that the extensions and LINQ mechanisms use a specialised call set whereas the CAC was mostly flat stack instructions. More of it, just faster execution in the case of the Strings and smaller numerical arrays.

So, in summary, nothing really conclusive but something I didn't expect with the integer operations. The gap between the two is there but only if the software product you're working on has performance constraints.

Mind you, all software has performance constraints to one extent or another :)

Welcome

Hi there.

I'm going to use this as a platform for review and analysis of the various techniques and technologies I encounter....Basically this is just a new kind of notepad.

I'm starting with a fairly broad range from a background steeped in Microsoft technologies, but touching on areas such as Linux, CAD, web technologies, social media [e-sociology] and performance issues.

Catch you later.