The joys of Robotium

In this post, we'll start adding tests for the Settings activity, which is the only one which involves user input aside from selecting an answer or pushing buttons. This is ideal for using Robotium, which as we saw in the last post makes it amazingly simple to test the entire application from a functional standpoint.

In the last test, we used Robotium to uncover a bug ironically introduced for the purposes of automated testing. This time, we'll see if we can introduce some kind of edit function to keep an invalid value from being entered in the first place.

Here's a nice, simple example of a validation function from http://piyushnp.blogspot.com/2010/03/android-edittext-validation-example.html:

String invalidStr="AB10",validStr="0998257476";

Pattern pattern=Pattern.compile("[0-9]*");

//argument to matcher is string to validate
Matcher matcher=pattern.matcher(invalidStr);

if(!matcher.matches()) // on Success
{
//invalid
//pop up ERROR message
Toast.makeText(context," Invalid Input ",Toast.LENGTH_LONG).show();

}else
{
//validation successfull
//Go Ahead !!
}


Let's change it, Actually, it's pretty simple; the level must be 1 - 5, the start number needs to be greater than zero and less than the end number, and the end number needs to be less than the amount of words available for that level.

Let's code the first test.


if(!matcher.matches()) // on Success
{
//invalid
//pop up ERROR message
Toast.makeText(context," Invalid Input ",Toast.LENGTH_LONG).show();

}else
{
//validation successful
//Go Ahead !!
}

The question is, how does the program stop the user from pressing save - disable the save button?

Also, how does the Robotium check the toast message?

Robotium looks like it's well supported, too. I immediately found several hits which answered the above question, on the Robotium google groups blog.

First, let's code it to fail:

assertTrue(this.solo.waitForText("Level must be between 1 and x"));
assertTrue(this.solo.searchText("Level must be between 1 and x"));


That died on an illegal state, possibly because this message isn't triggered until the activity is exited.

Let try a pass, just in case:

assertTrue(this.solo.waitForText("Level must be between 1 and 5"));
assertTrue(this.solo.searchText("Level must be between 1 and 5"));

Not surprisingly, it's the same error. Let's comment out the finish and see if that still happens. Yep. What does logcat have to say.

Huh - database not open? Why didn't I get that before this test? Let's comment out the test and see if that happened. Yes. Hmm, maybe I just missed it before. Let's rerun the invalid level test, this time with a valid level.

Yes, it passed. In fact, I haven't accounted for the invalid level yet. What I need to do, is to have the validation not allow the database to be accessed unless a valid value is entered.

Actually, the if (valid) else (process) structure works perfectly for keeping it on the same page on an invalid entry; the "process" part will do the finish. But, for some reason it didn't work on the first test run; I still had the same exception. Then I ran it manually, and got the error message - it worked. Then, I ran the test a second time and it passed. It seem like sometimes you have to run it a couple of times.

Let's make sure it fails.

assertTrue(this.solo.waitForText("Level must be between 1 and x"));
assertTrue(this.solo.searchText("Level must be between 1 and x"))

Great - it worked; I got an assertion failed. It takes a while because the toast takes a while to run, however. Let's see if it succeeds now.

assertTrue(this.solo.waitForText("Level must be between 1 and 5"));
assertTrue(this.solo.searchText("Level must be between 1 and 5"))l

Awesome. It passes. It seems like it's much faster on a pass than a fail.

Ok, let's put in the validation for the start and end numbers. Let's test for a negative quiz start number. First the fail: "Quiz start number must be greater than x". Fail!

Then the pass: Quiz start number must be greater than zero". No. Hmmm...run it again? No, I forgot to clear the field:

solo.clearEditText(1);

Try it again. Pass :)

Next, let's check for the end quiz number for a value that's too high. For this, we need to retrive the number of records in the database for the level. We have that:

int count = dbHelper.getLevelCount(inputLevel);

Ok, first I'll make it fail by giving a valid value. Here's the whole test:

public void testStartQuizNumTooHigh() throws Exception {

solo.clickOnButton(0);
solo.clearEditText(0);
solo.enterText(0, "5");
solo.clearEditText(1);
solo.enterText(1, "100");
solo.clickOnButton(0);
assertTrue(this.solo.waitForText("Quiz start number must be less than or equal to 669"));
assertTrue(this.solo.searchText("Quiz start number must be less than or equal to 669"));

}

No good - It's crashing on database not open. Hmm. Let's back out the code changes and the test to see if it was accessing the db before. Hmm...it's still crashing. Why wasn't it doing this before?

There are a couple of accesses to the database showing in the log. It looks like it's happening on the second one. Which happens to be calling the same code that the initial crash was on. Let's comment that out, too. And it's still crashing, on the one remaining database call. Maybe this is a limitation of the test framework? Let me test it manually.

It doesn't crash manually. It won't be good if Robotium s incompatible with sqlite. Still, it's going down on the same routine; and I think there successful call before that.

Well, I commented out a couple of unnecessary database opens and closes, and it actually made things worse. Now the second test is giving me an invalid button index, while the third is still failing on an invalid state.

The good news is there seems to be a successful db call before the exception. The bad news is I unit tested all the database calls yesterday, and they all worked fine.

Ok, a rerun at least gets the second test working again.

I'll try moving the open and close back to the calling routine, although I don't see why this would make a difference. Hmmm. the call that does work is in a static method. Perhaps that has something to do with it.

Ah - an assert - fail. Right, that's what I wanted - it means it didn't crash on the db access. Somehow, pulling the open and close of the db out of the db helper and into the calling routine solves the problem. Ok. Well, those unit test I did yesterday weren't much help - they were supposed to catch something like that. Although, those results did agree with the production. So maybe the fault lays with robotium. Anyway, all it really means is moving the open and close back from the method to the calling method.

I've done it for one. Now the next 2.

Ok, I've done it for all of them. Let's run the robotium tests.

They are ok. Now, lets clear the log and run the Android unit tests. Great - everything is working.

I got jammed up for a bit on what I think was robotium being a bit picky about the state of SQLite, but now that's cleared up.

Cool, With Robotium's help, I've just implemented a great validation routine. Thanks Robotium - you guys rock. I can't be too harsh on Google - Robotium wraps their code.

Robotium rocks!

Today, we're going to add some functionality to test the entry of invalid values. Let's get busy.

First, I'll make a copy of an existing test using ActivityInstrumentationTestCase2. I know it's kind of slow, but I'm going to be accessing resources and I think I need it for that. Actually, I'm not sure. Let's try accessing a resource from a test that uses ActivityUnitTestCase. I'm having trouble with deciding. I think I just need to play it safe and follow along with the examples from Google. What were those? Oh, yeah, it was Spinner. Let's take a look at some of its code:


@Override
protected void setUp() throws Exception {
super.setUp();

setActivityInitialTouchMode(false);

mActivity = getActivity();

mSpinner = (Spinner) mActivity
.findViewById(com.android.example.spinner.R.id.Spinner01);

mPlanetData = mSpinner.getAdapter();

} // end of setUp() method definition



In the setup, it's grabbing the control it's intending to test, plus it's adapter.

// this is run only once at the start of the app,
// to make sure the project is initialized correctly

public void testPreConditions() {
assertTrue(mSpinner.getOnItemSelectedListener() != null);
assertTrue(mPlanetData != null);
assertEquals(mPlanetData.getCount(), ADAPTER_COUNT);
} // end of testPreConditions() method definition

Here it's checking to make sure everything's been initialized.


public void testSpinnerUI() {

mActivity.runOnUiThread(
new Runnable() {
public void run() {
mSpinner.requestFocus();
mSpinner.setSelection(INITIAL_POSITION);
} // end of run() method definition
} // end of anonymous Runnable object instantiation
); // end of invocation of runOnUiThread

This looks like it creates a new thread. It sets the focus on the control and sets the selection on it. I remember reading something about how you need to test UI on the UT thread - but it was done as an annotation.

Here's the rest of the method:

this.sendKeys(KeyEvent.KEYCODE_DPAD_CENTER);
for (int i = 1; i <= TEST_POSITION; i++) {
this.sendKeys(KeyEvent.KEYCODE_DPAD_DOWN);
} // end of for loop

this.sendKeys(KeyEvent.KEYCODE_DPAD_CENTER);

mPos = mSpinner.getSelectedItemPosition();
mSelection = (String)mSpinner.getItemAtPosition(mPos);
TextView resultView =
(TextView) mActivity.findViewById(
com.android.example.spinner.R.id.SpinnerResult
);

String resultText = (String) resultView.getText();

assertEquals(resultText,mSelection);

} // end of testSpinnerUI() method definition



Ok, let's model test activity based on spinner code. I also found another link,

http://mobile.tutsplus.com/tutorials/android/android-sdk-junit-testing/

which looks like a nice simple example.

We'll try something like this:

public void testSendKeys() {

mActivity.runOnUiThread(
new Runnable() {
public void run() {
mLevel.requestFocus();
//mLevel.setSelection(INITIAL_POSITION);
} // end of run() method definition
} // end of anonymous Runnable object instantiation
); // end of invocation of runOnUiThread

/// now on level
sendKeys(NUMBER_5);

// now on start quiz number
sendKeys(NUMBER_1);

// now on end quiz number
sendKeys(NUMBER_1000);

// save
sendKeys("ENTER");

Assert.assertEquals(true, true);
}

Ok, it crashed on an index out of bounds. I noticed it looked like it wasn't clearing the fields before it populated them. It was too fast. Let's unplug an run it on the emulator, which is much slower. Woah - way too slow. Back to the device.

Ok, after a little experimentation, this sequence gives me a passing test:

// / now on level
sendKeys(KeyEvent.KEYCODE_BACK);
sendKeys(KeyEvent.KEYCODE_DEL);

sendKeys(NUMBER_5);

// now on start quiz number
sendKeys(KeyEvent.KEYCODE_BACK);
sendKeys(KeyEvent.KEYCODE_DEL);
sendKeys(NUMBER_1);
// now on end quiz numbe
for (int i = 0; i < 5; i++) {
sendKeys(KeyEvent.KEYCODE_BACK);
}
for (int i = 0; i < 5; i++) {
sendKeys(KeyEvent.KEYCODE_DEL);
}

sendKeys(NUMBER_1000);

// save
sendKeys("ENTER");

Assert.assertEquals(true, true);


Now, let's see what happens when we put a crazy value in there somewhere.

sendKeys(NUMBER_9999);

9999 is a larger number then the available vocabulary limit.


I'm just having a lot of difficulty figuring out what's going on when the test runs; on the device, it goes by too fast, and sometimes the screen size changes. Also, it's not at clear it's entering the digits I want. Plus, I know it should be crashing, and it isn't. The emulator is incredibly slow.

So, I'm going to go to plan B and try Robotium. This is a frustrating thing. A morning's worth of work, pretty much trashed. This is the downside of TDD on Android. The tools are hard to work with.

Let's see, is there a tutorial or something?

Yeah. Ok, a youtube video. Ok, download the sample project. Ok, it needs notepad list. Ok, got that. Run the test. Ok, seems to work.

Let's make a copy of that project. Ok, don't forget to change the target package name in the manifest; and the package name. Add the target package to the projects tab in settings.

ok, let's try something like this:

solo.clickOnButton(0);

Wow, very cool. Not only did it press the correct button, it went to the next activity screen, without my having to specifically load it. Multi activities, something I hadn't even gotten close to with the Android testing framework.

I like the Robotium api - all the events are easy to pick out. It supports things like clear text.

Lets see if we can enter some text...

solo.clickOnButton(0);
solo.clearEditText(0);
solo.enterText(0, "17");

Ok, how to I save it?

solo.clickOnButton(0);
solo.clearEditText(0);
solo.enterText(0, "17");
solo.clickOnButton(0);

Index out of bounds - perfect! Amazing - this is so much easier than Android testing framework.

Ok, let's fix up the code that causes this problem. Suddenly, I'm back into production code. Actually - this should work for the lifecycle checks I had to comment out, which I went to such pains to create.

Let's fix index out of bounds.

Ouch. Again, adding test code to return a value is just a bad idea. Especially a hardcoded one. It was ok when I was building the function, but anything hardcoded has to be cleaned out pretty quickly. Let's comment out that code.

Pass.

The problem is, I now have tests to run from two separate projects. I wonder how many of the android junit test I could bring into this one. Either that, or just test everything functionally. I think there's some I couldn't really do that for, that really do test actual methods.

Well, that's good for now. Let's pick up some more of the great Robotium in the next post.