Thursday, August 4, 2011

Download media over the net or use resources?

When I initially received the sound files which play Jap anese/English at for levels 4 and 5 of the JLPT, they were named in a japanese [many spaces] kana [many spaces] english.mp3 format. The spaces plus the internal representation of the Japanese characters created problems for direct access to the files, so I renamed them to a more compact name, i.e. jlptlevel_wordId.mp3. This greatly improved my success in retrieving the files to the point where I no longer considered it a significant problem. However, there are a few outliers who still aren't found, so, I'm going to delve into that.

Let's open RazorSQL, and take a look an one of the missing names, 有名, or famous. It's at level 5. Ok, it's _id 629. So, the name would be 5_629.mp3. Let's see if it's there.

Yup - it is. Ok. Let's check the files on the device.

Ok, a couple of things - it looks like the zip file didn't get successfully deleted when downloaded Also, what is the level 4 folder doing in level 5? Finally, the files stop at 627. They should go up to 668. Also, there's a 5_124 without the .mpg ending. Let's fix that in the original file right now.

Now, do I check the zip file on the phone, or on the system? Let's check the one on the phone.

adb pull sdcard/data/.com.kanjisoft.jlpt5.full/aud5.zip .

Ok, that only goes to 626 files. Ok, so, lets take a look at the original downloaded from the web.

scp myuser@mysite.com:public_html/android/full_copy/audio5.zip .

It's about a 2.5 meg file. Download at around 80 KB a sec, or 640 kbps. So, it will take about 3 minutes.

Just as I suspected, it has all the files. The download was probably truncated.

I'm thinking, since I have this "prefer sdcard" option, can I perhaps include the mp3 files in the app? I originally set it up with the download because I didn't want to use a lot of space on the core memory. But, it's going to the SD card anyway, and then I wouldn't have to handle the downloads on my own.

Wait, there may be some kind of limitation on app sizes. I think they recently increased it on the Android Marketplace, but still.

Ok, here it is:

http://android-developers.blogspot.com/2010/12/android-market-client-update.html

We are also increasing the maximum size for .apk files on Market to 50MB, to better support richer games.

Cool. Ok, but the only problem is, it will take a lot longer to load the app for testing. But, who want's the headache of truncated downloads? Actually, 2.5 mb is the size of the zip file. Probably the best thing to do is unzip the file from resources, or wherever its kept, into the data.

Well, I include a database file in my resources, then copy it to the sdcard. What does the code for that look like?

There it is:

// Open your local db as the input stream
InputStream myInput = null;
try {
myInput = myContext.getAssets().open(DB_NAME);
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}

// Create the directory if it's not there
File dirs = new File(getRootPathName());
dirs.mkdirs();

// Create path to the just created empty db
String outFileName = DataBaseHelper.getRootPathName() + DB_NAME;

// create the db file
File file = new File(outFileName);

try {
file.createNewFile();
} catch (IOException e1) {
// TODO Auto-generated catch block
e1.printStackTrace();
}

// Open the empty db as the output stream
OutputStream myOutput = null;
try {
myOutput = new FileOutputStream(outFileName);

} catch (FileNotFoundException e) {
// TODO Auto-generated catch block
e.printStackTrace();

}

// transfer bytes from the inputfile to the outputfile
byte[] buffer = new byte[1024];
int length;

try {
while ((length = myInput.read(buffer)) > 0) {
myOutput.write(buffer, 0, length);
}
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}

// Close the streams
try {
myOutput.flush();

myOutput.close();
myInput.close();

} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}

// Log.d(TAG, "copyDatabase exit");

}

Ok, well, first of all, I should switch to getExternalCacheDir for the files. One benefit is that these will get uninstalled, if the app is uninstalled, according to google docs:

"Saving cache files

If you're using API Level 8 or greater, use getExternalCacheDir() to open a File that represents the external storage directory where you should save cache files. If the user uninstalls your application, these files will be automatically deleted. However, during the life of your application, you should manage these cache files and remove those that aren't needed in order to preserve file space.

If you're using API Level 7 or lower, use getExternalStorageDirectory() to open a File that represents the root of the external storage, then write your cache data in the following directory:

/Android/data//cache/

The is your Java-style package name, such as "com.example.android.app".

Maybe I ought to do the cache thing first. This should be really easy - just replace getExternalStorageDirectory() with getExternalCacheDir(); uninstall the app, then run it.

But first, let's upload the zip file with the corrected mp3.

Ok, it looks like the get cache command needs a context, in a static method. That's frustrating. Before, I could just use Environment. Ok. how do I get a context to this? Ok, I'm passing a context to the static createDb command. I'll store it as a static variable.

Ok, it's downloading the data. It would be nice if it displayed a better splash, it just goes dark for a second or two before displaying the download dialogue.

Ok, it's working! Let's figure out where the data ended up...

Ok, it puts it in sdcard/Android/data/my.package.name/cache/

So, I really don't need to specify the whole package name, but it doesn't hurt.

That does block me from 2.1 and below, unless I put code in to check for that.

Ok, let's check for the delete on that file. I also want to make the bar grow a bit more.


Yeah, it wasn't deleting it. That's easily remedied...its in an AsyncTask, so I'll just delete it on the download.

@Override
protected void onPostExecute(String result) {

zipFile.delete();

}

Ok, for some reason the progress dialog for the download reaches it's max before the file is finished downloading - I'd rather it be the other way around. The way I calculate the update is to increment a counter every time a chunk is downloaded, multiplied 1024, because thats the size of the chunk (actually a byte array of 1024{. The progress max is set to the file size; so it should be exact. But, it stays up for maybe a half minute after reaching completion. Actually, I wouldn't have to worry about this if I just included the media files in the app.

Ok, if you kill the extract, then the database doesn't get initialized. So, what I'll do is make the download extract the last thing to happen. That way, at least the db will be in place.

Is there a way I can get this stuff done faster? Each download takes a few minutes. The extract goes faster and is more dependable. If I were to do the copy, I would just need to dump the file into the assets folder. Hmmm..why is my database file kept in the "shell" project? I think there was a reason.

Well, even if I move the download to the very last line of the start activity, if I kill the app in the middle of the download/extract, the JLPT level shows up as zero. It should be pulling this from String resources. Actually, I had originally made it variable, and persisted it to shared preferences. It should really now just be taken from resources, since I have a string for each level in the corresponding shell. But, where does it finally get it from correctly?

Right, it's getting set in the shell project, it's no longer in strings in the core project.

Ok, I'm just tossing in a few debug statements. I can't be having a 0 jlpt level showing up.

Ok, while that's running, let's start working on the assets. Actually, let's move the database back to the core project.

Actually, what I should do is take a count of the files in the media directory, and if they aren't there, relaunch the extract. The check I have right now, for the existence of one file, is too exposed to the vagaries of a particular extract.

Ok, let's get this done. Copy the zip file into the core project.

Hmmmm...I just realized, if I want to minimize the space taken up by the app, I would need to put the zip files in the shell project, and move the extract logic into each shell, which is a definite loser. Ok, that's it I'm sticking with the download.

Instead, I'm going to change the check for the number of file.

Huh? The Jlpt level is going to zero, even on normal download extract. What's going on here?

Ok, let's finish this, and then move on to the jlpt level, or if blocked by that, solve it.

Heres' the code:

dirToCheck = new File(DataBaseHelper.getRootPathName() + "/aud"
+ appState.jlptLevel);

int numFiles = dirToCheck.listFiles().length;


if (appState.jlptLevel == 4) {
if (numFiles == DownloadAudioData.NUM_FILES_IN_LEVEL_4_FREE) {
return;
}
} else { // level 5
if (numFiles == DownloadAudioData.NUM_FILES_IN_LEVEL_5_FREE) {
return;
}
}

Ok, the asset folder definitely doesn't inherit from the core project. Put the database back in there.

Ok, crash on the install - NPE on the file create. Does it need a mkdirs? Lets try it.

dirToCheck = new File(DataBaseHelper.getRootPathName() + "/aud"
+ appState.jlptLevel);

dirToCheck.mkdirs();

Ok, now it's running the download.


It's showing the correct level.

Ok, it works off the download.

But, a kill and restart gives this:

E/AndroidRuntime( 3151): Caused by: java.lang.IndexOutOfBoundsException: Invalid index 0, size is 0
E/AndroidRuntime( 3151): at java.util.ArrayList.throwIndexOutOfBoundsException(ArrayList.java:257)
E/AndroidRuntime( 3151): at java.util.ArrayList.get(ArrayList.java:311)
E/AndroidRuntime( 3151): at com.jlptquiz.app.QuestionActivity.incrementQuestion(QuestionActivity.java:556)
E/AndroidRuntime( 3151): at com.jlptquiz.app.QuestionActivity.onResume(QuestionActivity.java:95)
E/AndroidRuntime( 3151): at android.app.Instrumentation.callActivityOnResume(Instrumentation.java:1150)
E/AndroidRuntime( 3151): at android.app.Activity.performResume(Activity.java:3832)
E/AndroidRuntime( 3151): at android.app.ActivityThread.performResumeActivity(ActivityThread.java:2110)
E/AndroidRuntime( 3151): ... 12 more

And furthermore, starting the app again reinitiates the download. Let's start with that one. Probably my new code is off or something.

The good news is if I cancel the download, the app runs normally, because the data is there.

How many files should be in there? It's only finding 653.

Google shows some unix wizardry to count the files:

echo "`ls -l | grep -v "^d" | wc -l` - 1" | bc

and it is 653. It would appear I'm missing 13 files. Ok, that might right. There were some I wasn't able to convert.

It's going to be 628 for level four.

98 for the free version of level 5 and 100 for the free version of level 4.

This will be a much better way to check.

Yes, it goes right in, without a problem - no kicking off an unnecessary download.

Ok, now, what happened to the index on the kill and restart in question activity?


E/AndroidRuntime( 3151): Caused by: java.lang.IndexOutOfBoundsException: Invalid index 0, size is 0
E/AndroidRuntime( 3151): at java.util.ArrayList.throwIndexOutOfBoundsException(ArrayList.java:257)
E/AndroidRuntime( 3151): at java.util.ArrayList.get(ArrayList.java:311)
E/AndroidRuntime( 3151): at com.jlptquiz.app.QuestionActivity.incrementQuestion(QuestionActivity.java:556)
E/AndroidRuntime( 3151): at com.jlptquiz.app.QuestionActivity.onResume(QuestionActivity.java:95)
E/AndroidRuntime( 3151): at android.app.Instrumentation.callActivityOnResume(Instrumentation.java:1150)
E/AndroidRuntime( 3151): at android.app.Activity.performResume(Activity.java:3832)
E/AndroidRuntime( 3151): at android.app.ActivityThread.performResumeActivity(ActivityThread.java:2110)
E/AndroidRuntime( 3151): ... 12 more



The question is, why isn't the array getting initialized? The onResume should be doing it.

Ok, what I did was to call finish if no questions were found. I'm not sure I'm hitting that code.

The next problem is that the vertical between rows is larger when you come back into it from being killed.

What would cause that? That's all layout. What is getting lost?

Ok, well, this is a medium level bug. The workaround is to go back to the start activity, which puts it all right. But the user might not know that.

Ok, well, I'll tackle that one tomorrow. We made some good improvements today. Basically, decided to stick with the download over the net, to avoid carrying multiple zip files in the core and not have to shift processing to the shell. We also set up a much more robust check for if the file download and extract went to completion. We fixed up the progress display to more closely match the files. We *might* have prevented a crash on a restart of question activity, but that remains to be better tested. We also uncovered an interesting bug - a change in the layout vertical and possibly the font size when we kill and restart the QuestionActivity.

Ah, wait - that problem might be this:

Typeface typeFace = Typeface.createFromAsset(getAssets(), CURRENT_FONT);

appState.setTypeFace(typeFace);

Let's move that to initialize, where it's called from either activity.

Yes. That's it. And, that's a wrap.

No comments:

Post a Comment