reducing size of MODs

[anallyst]

dear modders, i just finished my project to reduce the filesize of mods.

essentially, modders tend to pack their working copy of the data folder into a PAK and post a download link.
however, they will forget to clean up their stuff most of the time, thus leaving much unneeded files in the paks, unnecessary files which we have to download and store!
since i like it to get more than 2 mods on my SD-card, i've written a tool that can remove such data.
for example, this saved 19MB at Ikarugas Double Dragon Unlimited (exactly 50% of its size, after converting the music).

the second thing which can be done is converting .BOR music files to .OGG format.
for example, this saved around 300MB with alemaspers mod New Double Dragon 3 - The Dragon Stone.

after applying both techniques, DDU by Ikaruga was 19 MB instead of 48. 10 MB saved by ogg conversion, 19 MB by removing unreferenced files.

read below for the tools and instructions how that can be done.

[anallyst]

1) removing unnecessary files

NOTE: this procedure requires some basic command line skills. i will not make a 1-click tool out of it. also it requires to know how to pack and unpack a mod.

a) prelude: how to get the script to work

the script requires having a perl interpreter, and the module File::Slurp (or see NOTE2 at the bottom)

you can get perl for free by using linux or mac, on windows with the cygwin kit (strongly recommended).

after perl is installed, you can run the following command (in a unix style terminal, i.e. that from cygwin)

Code: [Select]

perl -e shell -MCPAN
this will open the perl package management. you have to answer some basic questions.
then you can type

Code: [Select]

install File::Slurp
this will install the required module.
afterwards you can just type

Code: [Select]

quitto exit the package manager.

b) finding out what is unused

method 1) let openbor tell us which files have not been accessed


- unpack the mod
- edit data/levels.txt so that you have enough credits to play through every scenario.
- pack the mod

run the packed mod with the openbor command line argument showfilesused=1

Code: [Select]

openbor.exe showfilesused=1

this should work from build 3307 upwards.
basically this will output any filename which is in the pak, but was not accessed in the current session.

- try to play every scenario in the game. use all available models in the progress.
- exit the game.
- open Logs/Openbor.log in a texteditor.
- find the section with the list of unused filenames

chances are high, that every file it reports after playing through every nuance of the game are indeed unnecessary. but maybe we didn't activate some hidden level or something.
thus we will use method 2 additionally, to be 100% sure.

method 2) give a list of filenames to the script to let it scan for references.

- copy this list into a plain text file. for example "unused.txt"
- now run the attached perl script.

Code: [Select]

./checkunneeded.pl unused.txt

this will print out a list which of those files is required, and which not.

it does so by recursively parsing every file that is loaded by default in openbor.
in theory, it should even work on a complete list of all files. (if you want to skip the playing the entire mod part) however that is untested. that list of filenames could be created with the following command:

Code: [Select]

find data/ -name '*' > completelist.txt
since the output will be multiple pages long, it may be best to redirect STDOUT to some file.

- check if the returned output is ok.
- now run

Code: [Select]

./checkunneeded.pl unused.txt -d
now it will delete all unused files.
- undo your changes to the credits settings in level.txt
- repack and have fun!

here is the code of the script:

Code: [Select]

#!/usr/bin/env perl
use strict;
use warnings;
use List::Util qw (first);

use File::Slurp;

my $syntax = "\nunnecessary file remover 1.0 by anallyst\n\n"
. "syntax: $0 list.txt [-d]\n\nwhere list.txt is the name of a file containing all the filenames which\n"
. "shall be checked if they are required by the game or not.\nif -d is used, the unnecesarry files will be deleted.\n"
. "if -d is ommited, only a dry-run is executed and a usage log of each file is printed.\n"
. "the list is typically created with find (shell command) or extracted from OpenBOR.log after playing a game while using the\n"
. "showfilesused=1 command line argument to OpenBORs binary\n\n";

my $in = $ARGV[0] or die ($syntax);
my $delete = defined($ARGV[1]) ? 1 : 0;

my @a = read_file($in);
my @textfiles = `find data/ -iname '*.txt'`;
my @scriptfiles = `find data/ -iname '*.c'`;
my @chkfiles = (@textfiles, @scriptfiles);
my @rootfiles = qw(data/scenes/howto.txt data/scenes/logo.txt data/scenes/intro.txt data/scenes/gameover.txt data/script.txt data/models.txt data/levels.txt data/lifebar.txt data/menu.txt);
my @cfiles = qw(data/scripts/update.c data/scripts/updated.c data/scripts/level.c data/scripts/endlevel.c data/scripts/keyall.c data/scripts/timetick.c data/scripts/score1.c data/scripts/score2.c data/scripts/score3.c data/scripts/score4.c data/scripts/key1.c data/scripts/key2.c data/scripts/key3.c data/scripts/key4.c data/scripts/join1.c data/scripts/join2.c data/scripts/join3.c data/scripts/join4.c data/scripts/respawn1.c data/scripts/respawn2.c data/scripts/respawn3.c data/scripts/respawn4.c data/scripts/die1.c data/scripts/die2.c data/scripts/die3.c data/scripts/die4.c);
my @neededfiles = qw(data/pal.act);
my @other = qw(data/bgs/logo data/bgs/title data/bgs/titleb data/bgs/loading data/bgs/loading2 data/bgs/hiscore data/bgs/complete data/bgs/unlockbg data/bgs/select);
my @sprites = qw(data/sprites/shadow1.gif data/sprites/shadow2.gif data/sprites/shadow3.gif data/sprites/shadow4.gif data/sprites/shadow5.gif data/sprites/shadow6.gif data/sprites/arrow.gif data/sprites/arrowl.gif data/sprites/hole);
my @fonts = qw(data/sprites/font data/sprites/font2 data/sprites/font3 data/sprites/font4 data/sprites/font5.gif data/sprites/font6.gif data/sprites/font7.gif data/sprites/font8.gif);
my @sounds = qw(data/sounds/go.wav data/sounds/beat1.wav data/sounds/block.wav data/sounds/fall.wav data/sounds/get.wav data/sounds/money.wav data/sounds/jump.wav data/sounds/indirect.wav data/sounds/punch.wav data/sounds/1up.wav data/sounds/timeover.wav data/sounds/beep.wav data/sounds/beep2.wav data/sounds/bike.wav);
my @video = qw(data/video.txt data/videopc.txt data/videowii.txt data/video169.txt data/video43.txt data/videopsp.txt data/video169.txt data/videodc.txt data/video43.txt data/videowiz.txt data/video169.txt data/videogp2x.txt data/video43.txt data/videodingoo.txt data/video43.txt data/videosymbian.txt);
my @music = qw(data/music/complete data/music/menu data/music/remix);
my @combinedroot = (@rootfiles, @video, @cfiles);
my @needed = (@neededfiles, @other, @sprites, @fonts, @sounds, @music, @combinedroot);

sub isInUncommentedLine {
my ($search, $file) = @_;
return 0 if(! -e $file);
my @fc = read_file $file;
my $ccomment = lc(substr($file, -4)) ne ".txt";
for my $l(@fc) {
if( ! (
($ccomment && $l =~ /^(\s*?)\/\//) ||
(!$ccomment && $l =~ /^(\s*?)#/)
)) {
return 1 if $l =~ /\Q$search\E/i;
}
}
return 0;
}

sub checkfile {
my ($search) = @_;
chomp($search);
for my $f(@chkfiles) {
chomp($f);
if (isInUncommentedLine($search, $f)) {
print "$search is referenced from $f\n" unless $delete;
if(first {/^\Q$f\E/i} @combinedroot) {
return $f;
} else {
my $ret = (checkfile($f));
return $ret if $ret ne "";
}
}
}
return "";
}

for my $b (@a) {
my $c = $b;
chomp($c);
my $req = first {(lc($c) eq $_) || ($_ !~ /\./ && $c =~ /\Q$_\E\./i)} @needed;
my $r = $req ? "default" : checkfile($b);
if($r ne "") {
print $c . " is required by $r!\n" unless $delete;
} else {
print $c . " is not needed\n" unless $delete;
if ($delete) {
unlink $c;
print "deleted $c\n";
}
}
}



NOTE: this will only work entirely with clean mods (most are, luckily), such that use full paths and don't omit the file extension when they reference a file. for "dirty" mods (like Angel-0) it is required to play through every stage, watch every sequence, play every character, available mode etc, so that the output of openbor itself is nearly complete.

after removing the junk from a dirty mod, it is best practice to play again through the whole mod and see if everything works. if not, OpenBOR's log will tell you which file it misses.

NOTE2: the dependency to File::Slurp could be replaced by using the following subroutine:

Code: [Select]

sub read_file{my $fn = shift; my @ret = `cat $fn`; return @ret;}this requires a unix environment like cygwin though.

[anallyst]

2) converting all music to OGG format

this requires a set of tools to automatically convert BOR to WAV, WAV to OGG, and then replace all links to those files automatically.

for the WAV to OGG conversion the program ffmpeg (availble for free on linux) is required.

i will upload the tools together with instructions here, after i cleaned it up a bit.

ok, here they are.
unpack them, and use them like
- whenever you want to work on a mod, cd into the dir with my tools,
- ./unpack name_of.PAK
- ./complete_convert.sh data
- after everything is done, there's an additional check you have to do since my tools doesnt account for music referenced in *.C files yet
- run

Code: [Select]

grep playmusic `find data/ -iname '*.c'`this will show you the .C files which reference music. those have to fixed by hand to point to the OGG files.
- ./pack.sh data mynewmodrepak.pak

done.

tools download link https://github.com/downloads/rofl0r/openbor/anallysts_tools.tar.bz2
i put them externally since i can't upload tar.bz2 files to this forum.

[MatMan]

If you use total commander(not free but supppppppper awesome) then you can download a plugin thats called "audioconverter 0.992 alpha 1(free)" http://www.totalcmd.net/directory/packer.html that pretty much converts almost any audio format to any audio format.

[Bloodbane]

 anallyst, I think you should add: removing thumbs.db from mod too.

[anallyst]

anallyst, I think you should add: removing thumbs.db from mod too.

that is done by my script above as well. since it is unneeded.

If you use total commander(not free but supppppppper awesome) then you can download a plugin thats called "audioconverter 0.992 alpha 1(free)" http://www.totalcmd.net/directory/packer.html that pretty much converts almost any audio format to any audio format.

yep, but that doesnt convert the links in the .txt and .c files.

[Roel]

2) converting all music to OGG format

this requires a set of tools to automatically convert BOR to WAV, WAV to OGG, and then replace all links to those files automatically.

for the WAV to OGG conversion the program ffmpeg (availble for free on linux) is required.

i will upload the tools together with instructions here, after i cleaned it up a bit.

I don't want to rain on your parade, but I can already predict what will happen now: modders find some cool game's soundtrack in MP3 format, convert it to WAV, convert it to BOR, which is then converted to WAV again and finally to OGG. WAV is (usually) lossless, but all other formats are lossy and will slightly distort the original signal. So what you'll end up with is an file in which the weaknesses of all compression schemes (MP3, ADPCM and VORBIS) have accumulated.

So a tip to modders: if you care about audio quality, keep the number of lossy conversions to a minimum!

[anallyst]

So what you'll end up with is an file in which the weaknesses of all compression schemes (MP3, ADPCM and VORBIS) have accumulated.

So a tip to modders: if you care about audio quality, keep the number of lossy conversions to a minimum!

thats totally true. however, IMHO overrated. i can't tell a difference from any of my OGG repaks.
there's the music of level 2.1 from ikaruga's double dragon unlimited, which sounds slightly bad.
however it does so in the .BOR version as well.

on the other side the benefits in terms of saved disk space and download time are huge.

edit: what i have to add here is that the quality of the used encoder is of big importance in this debate. most people use itunes or windows media player to convert their music.
and what is produced by these crappy programs is really a lousy copy.

[Roel]

So what you'll end up with is an file in which the weaknesses of all compression schemes (MP3, ADPCM and VORBIS) have accumulated.

So a tip to modders: if you care about audio quality, keep the number of lossy conversions to a minimum!

thats totally true. however, IMHO overrated. i can't tell a difference from any of my OGG repaks.
there's the music of level 2.1 from ikaruga's double dragon unlimited, which sounds slightly bad.
however it does so in the .BOR version as well.

Vorbis (the compression scheme employed by ogg files) is spectacular, but my point still stands. I think you're comparing the wrong files here - is there an audible difference between the ogg file and the original audio file? Remember, I didn't recommend not converting to ogg, I recommended keeping the number of conversions to a minimum. Dropping the intermediate conversions to BOR and back again would be best - if possible, of course.

[magggas]

You do a very good work man but i just want to say
please don't touch my MK Outworld Assassins,i mean i like to have it
how it is and don?t see in future an OGG version of it.

Maybe i will do this my self on my next mod

[MIKE MASTERS]

OK, let me see if I get this straight:

For a new Mod/Pak, of my own creation, I had to do all the conversion you said?
or I could just only convert my CD's to OGG and put'em in the Music folder instead of the .BOR music? (and of course, put the paths in every .txt/.c I want to work with, as the .BOR ones).
...
and one more question (or two), which version of OpenBor works better with OGG files?.
which settings are the best for those files (OGGs, Khz/bit rate)?

'cuz I use PSP Fat CF 5.50 Gen-B full to play all the games from here, and save some space will be good.

[anallyst]

For a new Mod/Pak, of my own creation, I had to do all the conversion you said?
or I could just only convert my CD's to OGG and put'em in the Music folder instead of the .BOR music? (and of course, put the paths in every .txt/.c I want to work with, as the .BOR ones).
...

well yeah, you are right but there are 2 things to point out

1) don't use copyrighted music. you may get into trouble, and also the ppl that distribute your mod. use music with CC license instead (there are huge archives).
or use the soundtrack from another game after you have been granted permission to use it.

2) use a good encoder to convert your music. on windows i believe foobar2000 does a good job.

and one more question (or two), which version of OpenBor works better with OGG files?.
which settings are the best for those files (OGGs, Khz/bit rate)?

on the repaks i have done i chose 48kbit mono, as this offers reasonable music quality for a pretty slim filesize.
the only exception is the repak of golden axe where i exchanged the music for the one from the amiga game. because amiga mods used different instruments on the different audio channels, it would not sound perfect without stereo.
the prefered openbor version to use is 3307.

[Damon Caskey]

the only exception is the repak of golden axe where i exchanged the music for the one from the amiga game. because amiga mods used different instruments on the different audio channels, it would not sound perfect without stereo.

You know OpenBOR supports stereo music yes? Stereo sound effects too if you know what you're doing.

DC

[MIKE MASTERS]

well yeah, you are right but there are 2 things to point out

1) don't use copyrighted music. you may get into trouble, and also the ppl that distribute your mod. use music with CC license instead (there are huge archives).
or use the soundtrack from another game after you have been granted permission to use it.

2) use a good encoder to convert your music. on windows i believe foobar2000 does a good job.

Yeah, of course, I didn't mean that, I was saying more like my Fan made MOD, plus all my CD's are Legal Copies, and I don't think on selling nothing, but yeah, I know what you mean.

on the repaks i have done i chose 48kbit mono, as this offers reasonable music quality for a pretty slim filesize...
the prefered openbor version to use is 3307.

I'll try it, Thanks.

[anallyst]

You know OpenBOR supports stereo music yes? Stereo sound effects too if you know what you're doing.

DC

sure, it's just that the average wave channel 2 looks exactly like channel1, so saving both of them would be just a waste of space. it's better to mix into one channel, and reduce the filesize by 50%, if you cant hear a difference.