But I really like my 27” iMac.  It really is an impressive piece of hardware.  I also recently acquired an Apple TV, perfect for streaming media directly to the TV.  Up until now I have been using Boot Camp and booting into Windows 7 for the majority of my day-to-day computer usage, but with the acquisition of the Apple TV, and for various other curiosity reasons, I’ve started having a bit of a play with OSX Lion.

It really is very different and my first dealings with it left me frustrated and vowing to return to Windows and never come back to OSX.  But I have persisted and gotten used to the niggling differences in the way you navigate around the OS.  There were still many things missing.  One feature in particular of Windows 7 I make frequent use of and missed when in OSX is some of the shortcuts, or Hotkeys, that make life easier, like the Win+Left or Win+Right to make the current window take up the left/right half of the screen, allowing you to work on multiple things at once.  And the deal-breaker for me when playing with OSX is the lack of MS TFS for development and Zune.  As I said I love my WP7, I need to sync it somehow.  This meant I had to spend more time switching between Operating Systems than anyone should have to.

Along comes the perfect solution to my problem.   Desktop^® 7 for Mac (http://www.parallels.com/au/products/desktop/).

Installation

When installing Parallels the first thing it asks you to do is to create a New Virtual PC.  What impressed me is that along side the options of “Install from DVD” and “Migrate Windows from a PC” there was the Option to “Use Windows from  Boot Camp”.  First sign of brilliance and first sigh of relief.  I wouldn’t have to re-install everything!!  A single click is enough for Parallels to install everything needed on both sides of the relationship.

The default settings of the new VM are to call it “My Boot Camp” and set the VM to use 1GB of memory.  At this point, before starting it up it is possible to re-configure the VM, eg: to assign more memory or to rename the VM to something more Useful.  I renamed it to “Win7 Boot Camp” and set it to use 2GB memory, then I simply clicked Start and before I knew it, I was up and running.

Using Parallels Desktop

So far, Parallels is pretty easy to use as it seamlessly integrates all of your favorite Windows functionality into the OSX OS. There are 3 modes that the Virtual Machine will run in, each with their own benefits:

Mode 1 – Full-Screen

Similar to any other OSX App, running in full screen mode allows you to “swipe” across to it.  In the case of the VM we are playing with, it allows you interact with it as if you were not in a VM, away from any OSX specific apps, while at the same time allowing you to “swipe” back to the OSX desktop or to any other apps you are running in full screen mode.

I’ll be in this mode when involved in more serious Windows App development or Service/Server configuration.

Mode 2 – Window mode

Just as the name implies, Windows 7 running in a Window.  Useful for keeping an eye on the progress of things, like installations, etc, but I’m not sure I’d use this mode all that much.

Mode 3 – Coherence Mode

By far one of the more impressive feats I’ve seen.  Coherence mode essentially merges Windows 7, both the virtual machine and the functionality of Windows, into one.  Say hello to my favorite shortcuts, the Cmd+Left and Cmd+Right keys to shift a window to the left or right side of the screen. Note: This only works on applications running within the Virtual Machine.  Bummer.

All your windows applications are available either by right-clicking (or Ctrl + Clicking for those who don’t know what right-click is) the Parallels icon in the taskbar to bring up the Windows Start Menu, or by clicking the Windows Applications folder in the right-hand side of the Dock.  Also, any Windows Applications you have open are visible in the OSX Dock with a Parallels Icon over-laid.

I’ll be in this mode for most of my casual computer use, browsing, etc.

Conclusion

So while I’m only beginning to scratch the surface and there is a lot more to be said, I’ll finish by saying that not only was Parallels insanely easy to set-up and install, it means I now have access to the best of both worlds with no problems … except one.  Am I still a Windows person?  A friend of mine commented, “Don’t tell me OSX is now your default OS!!?!?”.  How do I answer that without sounding like I’m in denial? If I’m in OSX for some functionality that is only available in OSX, but I’m using >80% Windows applications, am I really in OSX?

One deliverable that was expected of us (I feel) in the Waterfall model is the Gigantic Test Plan and a HUUGE Test Suite with ’000s of Test Cases.  It is, afterall, one way to guarantee confidence in a product.

But ’000s of test cases take a long time to plan and execute, which simply does not fit in the Agile world, or indeed in any RAD world.  Not only that, but it doesn’t really make sense.  As Anne-Marie Charrett said at STANZ 2011, “Why spend all your thinking time planning how not to think?“.  It makes us inflexible to change, it reduces the actual testing time (ie: testing, not just following a script) and it reduces the efficiency of the testing when the tester actually gets a chance to test something.

Here’s where you may expect me to go on to talk about Exploratory testing and how we are using it at Seek, but while it is definitely a big part of the way we do things, there is still a higher-level need to plan our testing and where Exploratory Testing would sit alongside Automation and Manual (Scripted) Testing.  After all. sometimes we need to script a test for something that can/should not be automated (eg: de-coupled systems, timing dependencies, DB Hacking dependencies, etc).  Hence JIT Test Planning.

The concept of JIT as it applies to Manufacturing and Development is nothing new.  As anyone who knows anything about the history of Agile development would know, it’s been around since the 60s/70s as one of Toyota’s techniques to meet fast changing consumer demands with minimum delays¹.

JIT Test Planning is a strategy of spending more time working with the BA in defining the requirements before taking those requirements and defining the Plan that will define how you test them.

One underlying goal of the way we are doing things is to avoid as much duplication of documentation as possible.  ie: if the User Story says Button A needs to do X, do we really need to right a separate document telling the tester to press Button A and to make sure X happens?  The manual test case works in collaboration with the User Story.  It allows the Tester to define the most efficient way to verify the Acceptance Criteria without repeating the Acceptance Criteria word for word.  The test case now focuses on providing instructions on how to test something, not just what to test.

Probably the best way of proving the beneficial effects of JIT Planning is to show an example graph from one of the projects it has been used on.  Note: The following graph covers manual testing only.

The New Way

The main thing to notice (besides the high quality) is the lack of ‘Grey’ area, where Grey is planned un-executed testing.  No grey means No wasting of time.  The lack of Grey is the result of two things:

1. We do not begin planning until all Acceptance Criteria have been detailed and the User Story has been accepted by Dev and QA.
2. Because Dev and QA begin at the same time, by the time QA is ready to test, there is already a product to be tested.

In Summary, JIT Planning gives us the clarity, relevance and accuracy of requirements as they are generated in collaboration without months/years between Business Analysis and Development as well as the confidence in Test Coverage.  It allows a much greater flexibility and acceptance of change (the Agile way) and it all works in collaboration with Exploratory and Automated testing.  I’ll talk more about the end to end process we follow in future posts.

More Reading:

1. http://www.businessdictionary.com/definition/just-in-time-JIT-inventory.html
2. http://www.investorglossary.com/just-in-time.htm
3. http://www.investorwords.com/2688/just_in_time.html
4. http://www.ambysoft.com/essays/agileProjectPlanning.html

It seems like in today’s world when a tester raises a bug it is assumed to be an issue with the developers code but the fact is a bug is simply product of an unexpected result.  It seems like such a simple thing until one ponders all the possible reasons for an unexpected result.

1. Developer/Code
1. It is a development/coding error
2. It is an unpredicted scenario/combination
2. Environment/Configuration
1. It is a deployment issue
2. It is an environment/data issue
3. Requirement/Analysis
1. It is an incorrect requirement
2. It is a missed requirement
3. It is an incorrectly managed change request
4. Non-functional requirements
4. Test
1. It is an incorrect test
2. It is a misunderstanding

(the list really does go on)

None of these examples negate the relevance of the defect at hand.  I can only assume that anyone reading this knows the story of the first bug ever found, erroneously attributed to Grace Hopper:

In 1946, when Hopper was released from active duty, she joined the Harvard Faculty at the Computation Laboratory where she continued her work on the Mark II and Mark III. Operators traced an error in the Mark II to a moth trapped in a relay, coining the term bug. This bug was carefully removed and taped to the log book. Stemming from the first bug, today we call errors or glitch’s [sic] in a program a bug.

Pasted from <http://en.wikipedia.org/wiki/Software_bug>

This is a perfect example of the fact that a bug is not necessarily the fault of anything or anyone in particular but may be the result of a particular set of circumstances that couldn’t have been predicted or managed.

Regardless each “issue” needs to be discussed, validated, prioritised and an appropriate plan for resolution needs to be defined, including assigning to the correct owner to ensure it is resolved.  This is the purpose of Triage^4,5,6.

After all, it’s all about a product/project being released with Quality and Quality is owned by everyone, right?

More Reading:

1. http://en.wikipedia.org/wiki/Software_testing#Defects_and_failures
2. http://www.sqatester.com/bugsfixes/bugdefecterror.htm
3. http://www.sqatester.com/bugsfixes/bugdefecterror.htm
4. http://msdn.microsoft.com/en-us/library/dd380707.aspx
5. http://geekswithblogs.net/srkprasad/archive/2004/08/20/9961.aspx
6. http://wiki.secondlife.com/wiki/Bug_triage

It did remind me of one thing, though.  It’s been quite a while since I blogged anything. Some the things we’ve been working on here at Seek (Tools, Processes, People, etc) I think have been really fantastic.  While talking with some people at STANZ about what we’ve been doing and seeing their responses (mostly good ) kinda points out that maybe it’s time to begin putting down on paper (on-screen), partially for my own benefit (you know, organizing my thoughts) but also, if there is anyone out there that may find the way we are doing things interesting or, more importantly, if anyone would have any input into how we can do things better.  We don’t have all the answers, after all.

Let’s see how I go in maintaining regular posts

Should ET be Session-based?

There is a big push towards Session-based testing, both in terms of small teams and large, extended teams, ie: getting people outside the team to get in and test in a single 90 minute session at the end of each iteration.  There is no doubt in my mind that this form of testing is extremely valuable, but the questions that keep coming to mind is, is this ET and is it always valuable or even necessary.  Sure there needs to be some sort of exploration involved in order for people to try to find any issues remaining in the system, but does it match the dictionary definition of ET.

Another question that comes to mind is, as mentioned in an earlier post, when can we begin this type of ET?  Correct me if I’m wrong but before we can enter this form of session-based testing the SUT needs to be of a certain level of quality.  How do we get there without scripted, if indeed we are aiming to replace Scripted testing with ET?

Where are we headed?

While re-reading one of James Bach‘s blog posts (yet again) I had a bit of an epiphany. Well I had a couple really.

1. What is the difference between a Scripted test and an ET test?  How do we write a scripted test?  We script a test by interrogating the requirements in order to “explore” the imagined SUT.  We rely on the Tester to be able to accurately translate the requirements into a test that will accurately predict what the finished product is.  Therefore the only real difference between Scripted and ET is the timing of when the script is written.  As mentioned in just about every web page/blog post on the internet regarding ET, one of the drawbacks of ET is the fact that we can not review the proposed tests before testing.  But this risk is offset by ensuring that the testing effort is now more focused on the most valuable parts of the site.
2. “Exploratory testing is especially useful in complex testing situations….” as James Bach says.  This leads me to the thought that Scripted and ET can and should live quite nicely together.  A lot of our time is wasted in trying to predict and script all the different combinations and paths possible through our SUT, but at the same time we need to prove that the system works as required.  It is in this environment that Scripted and ET can live together.Given we are operating of User Stories and Scenarios, we can easily define our Acceptance Criteria and therefore our critical area of Scripted Tests.  These tests usually tie directly into Automation.  The Tester’s role could now look something like:
   1. analyse the defined Stories/Scenarios/Acceptance Criteria,
   2. add any extra scenarios if they are missed (the Testers know the system best, after all )
   3. write the scripted tests directly off the scenarios/acceptance criteria
   4. determine the areas of functionality that will be ET’d based on a Risk and Coverage analysis (eg: Website Ajax transitions, Combinatorial, etc)
   5. define the Tours and Charters to be used in the ET sessions
   6. lead the execution of these ET sessions on each iteration.

Simple depiction of how User Stories can lead to both Scripted tests and ET

By doing it this we can both “Prove that it works” (via Scripted) and “Prove that it’s Broken” (via ET).  We can improve our efficiency by focusing both forms of testing on the parts of the SUT needed (based on Risk/Coverage analysis).  We are still getting the majority, if not all, the metrics we need out of the system in order to make informed decisions as to the Quality of the SUT.

So this is where we are at for now.  Sorry, it’s been a bit of a ramble.  There are still lots of challenges ahead and everything may change tomorrow, but with a common goal in mind we are on the right track.  I’ll let you know how we progress

Are you getting sick of constantly switching applications from one monitor to another manually, constantly minimizing, dragging and dropping, then maximising again (eg: browser windows, pvcs, etc)?  I know I am, and I am sick of it!!!

So I went looking for one of those multi-monitor apps they had on Win XP (most of which you had to pay for) and found that Windows 7 has it built in (who woulda thunk?).  Win+Shift+Left or Right will move the current window from one monitor to the other.  <sigh> So good.

Note: this is for Win7 and Win2K8 only (as far as I know)

Here’s a complete list of short-cut keys that may make your life easier:

http://www.mydigitallife.info/2009/07/31/windows-7-keyboard-shortcuts-accelerator-keys-or-hotkeys-full-listing/

Enjoy

One concept that has been talked about in recent years is ‘Exploratory Testing’ ; the concept of parallel learning, test planning and test execution. In my travels ET is often mixed up with Ad Hoc testing (or Bug Bashing). This an easy misunderstanding to fall into, especially when attempting to compare it the more traditional ‘Scripted Testing’. So what is the difference? James Bach has a pretty succinct briefing on what ET is but here is a brief comparison in my own words.

Key Points – Scripted vs Ad Hoc vs Exploratory

• Scripted
  • Requires large effort during the Analysis/Requirements gathering phase of the project. There can be little uncertainty in requirements.
  • New tests/Changes to requirements need to be planned for. As we all know, changes can affect the plan greatly. This often leads to a large amount of rigidness in the development/testing process, particular in time-sensitive projects. We need to be prepared to pay the cost of documenting and maintaining tests. Without adequate Change Management, things can be missed/misunderstood.
  • Depending on the AUT there may be large amount of duplication across Dev Nunits, QA Test Scripts and User/Business Acceptance tests.
  • Any bug/scenario investigation or any other deviation from the plan is deemed as “wasted time”, or more often than not, not tracked against the plan. This can result in a test phase that is deemed to be not running according to estimates and running out of time.

• Ad Hoc
  • Definition: “for the special purpose or end presently under consideration”.
  • Too often synonymous with sloppy careless work or improvised, impromptu “Bug Bashing”. While it can be quite valuable, it really doesn’t have a good reputation.
  • Lack of framework/management means results are unreliable and unclear. How can we determine whether something was tested or not?

• Exploratory
  • Definition: Simultaneous Learning, Test Planning and Test Execution.
  • Operates in Sessions at a scenario level, fits closely with Use Case/User stories techniques.
  • The term “exploratory” emphasizes the dominant thought process, The skill of the tester becomes a factor in terms of technical skills, Business/Domain knowledge and general curiosity.
  • The key is the cognitive engagement of the tester, and the tester’s responsibility for managing his or her time.
  • A good exploratory tester will write down test ideas and use them in later cycles. These notes often look like test scripts, although they aren’t.

Concerns with Exploratory Testing

So while ET really does seem like the way to go, there are a couple of open questions in my mind that trouble me:

1. At what stage can the testers get involved?
   I am a big fan of working closely with the Dev’s, maybe not quite paired testing, but with a focus on test early and test often. To me, ET seems to be dependent on an area of functionality being complete (or at least close to) or it would seem like a waste of time.
2. How can we be sure we tested everything?
   There appears to be a large dependency on the requirements being of a certain level of detail. What is the link between the defined Acceptance Criteria and Test Coverage? Having worked on some projects with a high level of complexity and a large number of combinations, how can we be sure that we have tested all of these combinations?

Which one to use?

I like to think of scripted testing being “Prove that it works” as opposed to exploratory being “Prove that it’s broken”. With this in mind, as the kid in the Tacos commercial says, ¿por qué no los dos? (Why don’t we have both?).

The more I think about implementing ET the more it seems the logical flow to follow is a blend of both, with scripted testing allowing us to test early on specific features/rules and to verify specific Acceptance Criteria and ET allowing us to increase test coverage, focusing on higher level areas of functionality (an attempt to visualise this can be seen in Fig. 1).

Fig. 1

I will post more on this subject, and the progress of implementation, as I proceed.  Wish me luck .

More Reading

Here’s some further reading for your pleasure. This is just a tiny selection of all the information out there.

• Wikipedia – http://en.wikipedia.org/wiki/Exploratory_testing
• Exploratory Testing Explained – http://www.satisfice.com/articles/et-article.pdf
• What is Exploratory Testing – http://www.satisfice.com/articles/what_is_et.shtml
• General Functionality and Stability Test Procedure for Certified for Microsoft Windows Logo – http://www.satisfice.com/tools/procedure.pdf
• Explaining Exploratory Testing – http://blogs.msdn.com/james_whittaker/archive/2009/01/08/explaining-exploratory-testing.aspx
• Exploratory Testing in Pairs – http://www.kaner.com/pdfs/exptest.pdf
• Exploratory Testing – Good and Bad sides – http://www.testandtry.com/2010/01/26/exploratory-testing-good-and-bad-sides/
• Example Exploratory Test – http://wiki.laptop.org/go/1_hour_smoke_test
• Context Driven School – http://en.wikipedia.org/wiki/Context-Driven_School

Firstly, who am I?  My name is Rob Manger and I am currently the QA Team Lead at http://www.seek.com.au.  I am passionate about process and toolset improvement when it comes to Quality Assurance.  I also love my gadgets and am just starting to get into photography.  So, you can expect topics to be covered in this blog are all near and dear to my heart.