Recently, Ministry of Testing asked about our favorite bugs ever on Twitter. So I thought this could be worth a blog post.

It all started about six years ago when I created my very first Selenium based test framework. I just learned about the concept of page objects - basically distinct classes that wrap each page of a web application and expose their functionality as methods.

Since all of our pages that we wanted to test had some functionality in common, I thought "Why not extend a BasePage class that has the shared functionality and let all of the isolated pages extend it". So I did.

Also, to be extra safe, I gave this BasePage class a constructor with a pageTitle parameter. The plan was that each page that was currently visible could verify its own headline before carrying out any other actions in the test scenario.

This was supposed to happen:

Whenever there is an action that triggers a switch to a new page

• construct this new Page (e.g. new LoginPage("Log In"))
• verify the current page's identity by comparing its headline against the passed constructor string ("Log In")
• continue with other actions and assertions on this page until the next page switch

This worked fine for weeks and we were happily seeing green tests.

One day, we triggered a new release of our web application. Our CI pipeline ran the UI tests, everything passed. QA did a final manual check and... notified us that there was a serious issue: one of the page menu links opened the wrong page!

I was confused. Very! "But the page headlines are automatically verified every time they are opened. And for the log in page, the test makes sure that the title is "Log In"! It's impossible! Are you sure we have a bug?!?"

Indeed, the test was wrong...

After some research I noticed that we did not compare the string parameter that was passed to the page to its headlines but to any element on the page that would match it. Since every page had the same menu to reach every other page via their title (e.g. a click on the "Log In" menu item sent you to the "Log In" page), this check was always true on every single page.

Essentially, this assertion passed every time, giving us a wrong feeling of security and - had it not been manually tested - would have caused a broken release.

So please, if you create any new tests, make sure they fail first!

Because a wrong test is worse than no tests...