The joys of Robotium

In this post, we'll start adding tests for the Settings activity, which is the only one which involves user input aside from selecting an answer or pushing buttons. This is ideal for using Robotium, which as we saw in the last post makes it amazingly simple to test the entire application from a functional standpoint.

In the last test, we used Robotium to uncover a bug ironically introduced for the purposes of automated testing. This time, we'll see if we can introduce some kind of edit function to keep an invalid value from being entered in the first place.

Here's a nice, simple example of a validation function from http://piyushnp.blogspot.com/2010/03/android-edittext-validation-example.html:

String invalidStr="AB10",validStr="0998257476";

Pattern pattern=Pattern.compile("[0-9]*");

//argument to matcher is string to validate
Matcher matcher=pattern.matcher(invalidStr);

if(!matcher.matches()) // on Success
{
//invalid
//pop up ERROR message
Toast.makeText(context," Invalid Input ",Toast.LENGTH_LONG).show();

}else
{
//validation successfull
//Go Ahead !!
}


Let's change it, Actually, it's pretty simple; the level must be 1 - 5, the start number needs to be greater than zero and less than the end number, and the end number needs to be less than the amount of words available for that level.

Let's code the first test.


if(!matcher.matches()) // on Success
{
//invalid
//pop up ERROR message
Toast.makeText(context," Invalid Input ",Toast.LENGTH_LONG).show();

}else
{
//validation successful
//Go Ahead !!
}

The question is, how does the program stop the user from pressing save - disable the save button?

Also, how does the Robotium check the toast message?

Robotium looks like it's well supported, too. I immediately found several hits which answered the above question, on the Robotium google groups blog.

First, let's code it to fail:

assertTrue(this.solo.waitForText("Level must be between 1 and x"));
assertTrue(this.solo.searchText("Level must be between 1 and x"));


That died on an illegal state, possibly because this message isn't triggered until the activity is exited.

Let try a pass, just in case:

assertTrue(this.solo.waitForText("Level must be between 1 and 5"));
assertTrue(this.solo.searchText("Level must be between 1 and 5"));

Not surprisingly, it's the same error. Let's comment out the finish and see if that still happens. Yep. What does logcat have to say.

Huh - database not open? Why didn't I get that before this test? Let's comment out the test and see if that happened. Yes. Hmm, maybe I just missed it before. Let's rerun the invalid level test, this time with a valid level.

Yes, it passed. In fact, I haven't accounted for the invalid level yet. What I need to do, is to have the validation not allow the database to be accessed unless a valid value is entered.

Actually, the if (valid) else (process) structure works perfectly for keeping it on the same page on an invalid entry; the "process" part will do the finish. But, for some reason it didn't work on the first test run; I still had the same exception. Then I ran it manually, and got the error message - it worked. Then, I ran the test a second time and it passed. It seem like sometimes you have to run it a couple of times.

Let's make sure it fails.

assertTrue(this.solo.waitForText("Level must be between 1 and x"));
assertTrue(this.solo.searchText("Level must be between 1 and x"))

Great - it worked; I got an assertion failed. It takes a while because the toast takes a while to run, however. Let's see if it succeeds now.

assertTrue(this.solo.waitForText("Level must be between 1 and 5"));
assertTrue(this.solo.searchText("Level must be between 1 and 5"))l

Awesome. It passes. It seems like it's much faster on a pass than a fail.

Ok, let's put in the validation for the start and end numbers. Let's test for a negative quiz start number. First the fail: "Quiz start number must be greater than x". Fail!

Then the pass: Quiz start number must be greater than zero". No. Hmmm...run it again? No, I forgot to clear the field:

solo.clearEditText(1);

Try it again. Pass :)

Next, let's check for the end quiz number for a value that's too high. For this, we need to retrive the number of records in the database for the level. We have that:

int count = dbHelper.getLevelCount(inputLevel);

Ok, first I'll make it fail by giving a valid value. Here's the whole test:

public void testStartQuizNumTooHigh() throws Exception {

solo.clickOnButton(0);
solo.clearEditText(0);
solo.enterText(0, "5");
solo.clearEditText(1);
solo.enterText(1, "100");
solo.clickOnButton(0);
assertTrue(this.solo.waitForText("Quiz start number must be less than or equal to 669"));
assertTrue(this.solo.searchText("Quiz start number must be less than or equal to 669"));

}

No good - It's crashing on database not open. Hmm. Let's back out the code changes and the test to see if it was accessing the db before. Hmm...it's still crashing. Why wasn't it doing this before?

There are a couple of accesses to the database showing in the log. It looks like it's happening on the second one. Which happens to be calling the same code that the initial crash was on. Let's comment that out, too. And it's still crashing, on the one remaining database call. Maybe this is a limitation of the test framework? Let me test it manually.

It doesn't crash manually. It won't be good if Robotium s incompatible with sqlite. Still, it's going down on the same routine; and I think there successful call before that.

Well, I commented out a couple of unnecessary database opens and closes, and it actually made things worse. Now the second test is giving me an invalid button index, while the third is still failing on an invalid state.

The good news is there seems to be a successful db call before the exception. The bad news is I unit tested all the database calls yesterday, and they all worked fine.

Ok, a rerun at least gets the second test working again.

I'll try moving the open and close back to the calling routine, although I don't see why this would make a difference. Hmmm. the call that does work is in a static method. Perhaps that has something to do with it.

Ah - an assert - fail. Right, that's what I wanted - it means it didn't crash on the db access. Somehow, pulling the open and close of the db out of the db helper and into the calling routine solves the problem. Ok. Well, those unit test I did yesterday weren't much help - they were supposed to catch something like that. Although, those results did agree with the production. So maybe the fault lays with robotium. Anyway, all it really means is moving the open and close back from the method to the calling method.

I've done it for one. Now the next 2.

Ok, I've done it for all of them. Let's run the robotium tests.

They are ok. Now, lets clear the log and run the Android unit tests. Great - everything is working.

I got jammed up for a bit on what I think was robotium being a bit picky about the state of SQLite, but now that's cleared up.

Cool, With Robotium's help, I've just implemented a great validation routine. Thanks Robotium - you guys rock. I can't be too harsh on Google - Robotium wraps their code.