The first task in implementing this new publishing process is to convert the existing static page into a dynamic page, e.g. one that reads data from an external source, in this case a database. I will attempt to chronicle my adventures. I’m learning things as I go, and I may find better ways of doing things, so later posts may reverse decisions made in earlier posts.

In this post, I will detail setting up a development environment.

The Requirements

The requirements for the new dynamic web page are as follows:

• Displays a list of all applications
• Displays various information on each application – name, description, current version, date released, license, links to artifacts (source/binary packages), etc
• Must maintain a history of old versions of each application
• Can display “other” artifacts for each application
• The same release notes should be used for both the main page and the blog news posts.

The Technologies

I would like to write my new front page with a basic page template, using inline tags to apply logic. My first instinct was to write something in Python, using a templating engine like Jinja2 (or even just Python’s built-in string.Template capabilities), but my web host does not have particularly good support for Python or adding extra site packages. The killer blow was the lack of built-in support for MySQL, my host’s default database system.

So, I decided to go with the usual PHP & MySQL combo, since this seems to be fully supported by my web host. I’m a newbie with both technologies, so I’m sure I’ll stumble over myself a few times.

In order to store richly formatted text in the database, for eventual presentation on an HTML page, I have decided to go with the Markdown markup language, reasons being:

• Concise syntax, so easier to read and write
• Can include normal HTML if required
• Processors available in many programming langauges (PHP and Python, specifically)

The Development Environment

Deploying my “work in progress” PHP code to my live server (www.jestarjokin.net) is a terrible idea, especially when I’m bumbling around trying to learn PHP. I need a development environment. I primarily use Windows as my operating system, and I don’t want to clutter things up installing PHP and MySQL just for a single project. A virtual machine sounds like the best approach – you can install whatever you want, set it up as a web server, muck around with configurations etc to your hearts content, and when you’re not using it you just turn it off and continue with your everyday computering.

There’s a few Virtual Machine host applications out there, I’m going with VirtualBox because it’s free (for now).

I stumbled across a full Ubuntu virtual machine image, Bitnami LAMPStack. This contains an installation of Ubuntu Linux, with Apache, MySQL, and PHP already installed. Too easy! (They also provide an OpenSUSE image, if that’s more your fancy.)

… Unfortunately, it doesn’t quite work how I expected out of the box, so there’s some tweaking required. I want to treat the VM the same as a remote host, so we need to open up some settings for remote connections.

Import Virtual Machine into VirtualBox

While the LAMPStack image contains an OVF file, which is a virtual machine configuration file that should be readable by VirtualBox, I received odd error messages. This isn’t a big deal, you just need to manually create your own virtual machine and import the LAMPStack hard drive image.

First, download and install VirtualBox. Then download and extract the Bitnami LAMPStack virtual machine images somewhere. I believe the hard drive images are configured to expand to 17 GB as required, but from a fresh download they should be 1 GB big (or less).

• Open VirtualBox
• Click “New”
• Name = “LAMPStack (BitNami)”
• OS = Linux / Ubuntu
• Next, choose how much memory to give the VM. Allocate at least 512 MB, I’m going with 1024 MB.
• Next, “use existing hard drive”, and point to the “VMDK” file in the Bitnami VM images. In my case, this was “bitnami-lampstack-5.3.8-2-ubuntu-10.10.vmdk”.
• Once done importing, select the new VM and click on “Settings”
• System->Processor, tick “Enable PAE/NX”
• Storage, click the “Add Hard Disk” icon on the IDE Controller, choose the existing VMDK image (same as before)
• Storage, delete the SATA Controller entry for that VMDK image
• (Optional) Network, choose “Bridged Adapter”. This depends on your network configuration and how you want to be able to access the virtual machine. I found going with NAT assigned an IP address on the wrong network (e.g. 10.0.0.1 instead of 192.168.1.1).

Start Virtual Machine & Login

Start up the virtual machine. It will list some information, such as the VM’s IP address. For my examples, I will be using IP address 192.168.1.2 to refer to the VM.

You will also be prompted to log in. The default login is:

• bitnami / bitnami

You’ll be prompted to change the password for the bitnami account.

Enable SSH/SFTP Access for File Transfers

We need a way to copy files onto the VM. The easiest option seems to be SFTP. You’ll need an SFTP client, I use WinSCP but Filezilla also works.

We need to change a configuration file on the server to enable SSH connections. Luckily, BitNami have already provided a configuration file, we just need to replace the existing file. In the VM, run these commands (without the “$”):

$ sudo mv /etc/init/ssh.conf.back /etc/init/ssh.conf
$ sudo start ssh

Using WinSCP, you can connect to your VM from its IP address (e.g. 192.168.1.2), using the “bitnami” user login.

Web pages are served from this directory, which is primarily where we want to put files:

/opt/bitnami/apache2/htdocs

Enable MySQL Remote Access

I want to manipulate the database from my real computer, not the VM, so we need to enable remote access to MySQL. There’s a bit of jiggery-pokery required to enable remote access to the MySQL database.

Note that if you add any MySQL users, you should grant access both from the “any host” wildcard (“%”), and from the various local host designations (“localhost”, “127.0.0.1″), otherwise you may have issues logging in to MySQL on the VM (which will affect your application).

Root access

By default there is only one MySQL user set up, with this login:

• root / bitnami

Grant root access from any host

$ /opt/bitnami/mysql/bin/mysql -u root -p -e "grant all privileges on *.* to 'root'@'%' identified by 'your_root_password'";

The “‘root’@'%’” part says that the user “root”, from any host, should be able to to anything on the server.

NOTE: this is probably very bad practice. You should only grant as much access as you want to non-root users. However, this is convenient for our development environment, as it allows us to fully manipulate the database from our main computer. Don’t do this sort of thing on your real server!

Disable bind address

Open up “/installdir/msyql/my.cnf” in vi. Comment out this line by adding a “#” to the start:

bind-address = 127.0.0.1

(Some vi hints: use “:w” to save and “:q” to quit. You may need to press the Insert key before you can enter the hash symbol.)

Poke a hole in the firewall

Run this command to open up port 3306 (the default MySQL port):

$ sudo ufw allow 3306

Enable phpMyAdmin Remote Access

phpMyAdmin is a useful web interface to administer MySQL. By default, it is restricted to “localhost” access, so we need to change this.

Using vi, edit the configuration file at “/opt/bitnami/apps/phpmyadmin/conf/phpmyadmin.conf”. Change the line highlighted in red (by default, it will say “Allow from 127.0.0.1″).

  Alias /phpmyadmin "/opt/bitnami/apps/phpmyadmin/htdocs"

  <Directory "/opt/bitnami/apps/phpmyadmin/htdocs">
  AuthType Basic
  AuthName phpMyAdmin
  AuthUserFile "/opt/bitnami/apache2/users"
  Require valid-user
  Order allow,deny

  Allow from all

  Satisfy all
  ErrorDocument 403 "For security reasons, this URL is only accesible
  using localhost (127.0.0.1) as the hostname"
  </Directory>

You can then access the phpMyAdmin page from this URL (changing the IP address to your VM’s):

http://192.168.1.108/phpmyadmin/

Log in using the MySQL “root” username & password.

Done

Finally, your development environment is ready to use! It’s a good idea to restart the VM after making the above changes, but afterwards you can just let it sleep/hibernate when you’re not using it, and it’ll be ready to rock next time.

Next Time

In the next part, I hope to go through a proposed database structure, followed by a naive implementation of the page in PHP.

• Fix “drawObject” in V3-4 (was missing 3rd argument and outputting the wrong opcode).
• Fix “getState” instruction’s syntax (was originally defined as “if (getState(Local[0] == 0))”, should have been “if (getState(Local[0]) == 0)”).
• “debug?” instruction renamed to “debug”.
• “PutCodeInString” – add support for a missing 2nd argument (as output by descumm).
• Restructure code path structure, separate out some tests.
• Added some tests based on my attempt to parse MI1 global scripts.
• Change license to MIT license.
• Update build scripts.
• Update to the manual – change info for “getState”, add support e-mail to contact list.

Binary:
http://www.jestarjokin.net/sw/scummbler_bin_20110712.zip
Source:
http://www.jestarjokin.net/sw/scummbler_src_20110712.zip

I’ve discovered there are some bugs in descumm’s output (introduced by yours truly). I discovered these after trying to parse all the global scripts from Monkey Island 1 VGA floppy.

Script 82 and 83 – missing 2nd argument for PutCodeInString

[0000] (27) PutCodeInString(24, );

When the string is empty, descumm outputs nothing for the second argument. It should output this:

[0000] (27) PutCodeInString(24, "");

Scummbler now supports either scenario.

Script 135 – unknown string function

Global script 135 contains this line:

[049C] (FA) VerbOps(Var[100],[SetXY(0,Var[228]),Text("How to deal with  frustration, disappointment, " + unknown8(8224) + " and irritating  cynicism."),On(),Key(Var[229]);]);

The “unknown8″ instruction is incorrect, and can’t be parsed by Scummbler. You must manually change the text like so:

[049C] (FA) VerbOps(Var[100],[SetXY(0,Var[228]),Text("How to deal with  frustration, disappointment,\xFE\x08  and irritating  cynicism."),On(),Key(Var[229]);]);

Also, there shouldn’t be a semi-colon after the “Key()” instruction, but Scummbler handles this anyway.

Script 152 – not joining string functions

[00BF] (14) print(255,[Color(Local[8]),Center(),Text(getString(VAR_HEAPSPACE)keepText())]);

This line should read:

[00BF] (14) print(255,[Color(Local[8]),Center(),Text(getString(VAR_HEAPSPACE) + keepText())]);

Again, you’ll have to manually alter the script.

Win32 Binaries

http://www.jestarjokin.net/sw/musicsort_bin_20110619.zip

Python Source

http://www.jestarjokin.net/sw/musicsort_src_20110619.zip

This script (and the binaries) is license under the MIT License.

An explanation follows.

My current tree structure looks something like this:

• Music
  • Albums
    • Genre1
    • Genre2
    • Genre3
  • Misc
    • Genre1
    • Genre2
    • Genre3

My naming conventions for the album directories is like this:
Artist – Year – Album [Aux] (Music Type & Bitrate)

e.g.

Slum Village – 2010 – Villa Manifesto (MP3 192)

The “Aux” label is optional, and can store information like whether the album comes from vinyl, or the label/pressing info, or the CD number in a multi-disc set.

While most tagging software can handle the tag-based information okay, it doesn’t satisfy my requirements for the Music Type & Bitrate. My logic is like this:
Music Type will roughly be the file extension, “MP3″ or “OGG” or “MP4″ or *shudder* “WMA”. (MP4 gets a little complicated, because there are a few different file formats/extensions, for what is basically the same compression technology, e.g. AAC/MP4/M4A)

Bitrate goes like this:

• If all files have the same bitrate, assume it’s a single constant bitrate (CBR).
  • e.g. (MP3 192)
• If there is a mixture of different bitrates, and they’re all divisble by 8000, assume it’s a mixture of CBR. Bitrate value should note that it’s mixed and provide the minimum and maxmium bitrates.
  • e.g. (MP3 mixed, 128-320)
• If all files have a mixture of different bitrates, and no bitraters are perfectly divisible by 8000, assume a variable bitrate (VBR). Also output the average bitrate across all files.

The script has been programmed so that if any value is missing (i.e. artist, year, or album), it will prompt the user to enter a new value for the missing value.

If there are multiple encountered values for any value (such as year), it will ask the user to choose one of the found values, or to enter their own value. The exception is the “artist” field – multiple values will instead default to a value of “VA” (for Various Artists).

Once the script has generated an output folder name, the user is prompted to confirm if it’s okay, or to enter “override” values for any of the information fields.

Because “auxilliary” information is not necessarily available from the tags, this info can be specified at runtime, seperately from other “override” values.

If the user enters any override values, the script will again prompt the user to confirm the output folder name. This will loop as long as the user keeps entering override values or auxilliary information. If the user does not enter “Y” or “Yes” in the confirmation prompt, the script will not attempt to move the folder/music files.

The script can be run across a single input folder, or across all sub-folders in the input folder. If the latter, you can skip processing folders by declining the prompt to confirm the output folder name.

If looking in a folder with no files and only one sub-folder, the script will automatically try to process that sub-folder.

Once the user confirms the output folder name, the script will prompt the user to choose a destination directory. The list of destinations is determined by all sub-folders in a previously defined destination root directory (specified in an external configuration file). For my purposes, I have a root “Album” directory, with a list of sub-folders named by genre. I also ignore any directories beginning with an underscore character, e.g. “_to_sort”. This lets me have an “incoming” directory of albums that require sorting.

The destination directory, and any other information that should not change at runtime, is specified in a seperate configuration file.

This was my first attempt at working with Unicode strings properly. I’m using a Windows (NTFS) system, which stores filenames as Unicode, and the album data could contain Unicode strings. The script will convert any input from the command line, such as initial arguments or any override values, to Unicode, based on the current system encoding.

Future work:

• Support analysis of files without any tags, trying to guess values based on the filename.
• Support for lossless compression (should be pretty simple really, just need to add the right calls to Mutagen).
• Make all MP4 files use the same music type (rather than AAC/MP4/M4A).
• Correct “bitrate” naming of albums that use a mixture of VBR and CBR.
• Support for cancelling the script – currently the only way to cancel is to press CTRL-C, which quits the whole thing.

The mailing list address:

support@lists.jestarjokin.net

You can set up subscriptions and other info here:

http://lists.jestarjokin.net/listinfo.cgi/support-jestarjokin.net

The archives will be located here:

http://lists.jestarjokin.net/private.cgi/support-jestarjokin.net

I will be discussing the format based on the “ERULES.PRG” file, which contains English text. The other rules files should be in exactly the same format, only the string values should be different. All bytes are in little-endian format (unless otherwise stated).

I’ll state up here at the start of this post, if anyone wants to join forces in exploring the intricacies of the game, feel free to send me an e-mail: jestarjokin@jestarjokin.net

View after the break, IF YOU DAAAAARE…!

ERULES.PRG

The rules file is split into 10 sections (based on the number of functions called to read in the data). The file begins with a small 2-byte header (containing a value of 0×0000); the 10 sections follow.

Section 1

Section 1 is pretty simple. The number of entries is given as 2 bytes, that number of bytes then follows.

Section 2

Section 2 starts with 2 bytes for the number of entries.

Each entry is made up of a number of components, starting with s2WordA, which is 2 bytes. If this value is not 0xFFFF, it is multiplied by 8 and added with 4 when read. This occurs again for s2WordB, another 2 bytes. s2WordC is read as 2 bytes, s2WordD is another 2 bytes.

After this, there are 10 individual bytes. Then follows two arrays, each 32 bytes long.

In total, one entry will be 82 bytes long. Robin Hood contains 39 entries.

Section 3

Section 3 contains string data! It begins with the number of entries, and the total size of the string data (in sub-section 2). Following are two sub-sections.

The first sub-section contains 2-byte values; these values represent the start of each string, given as an offset from the beginning of the string data chunk (in sub-section 2). For example, if the first string in sub-section 2 is “Wh\xCEps!\x00″, this is 7 characters long (the “\x” values are my way of denoting raw ASCII values). Therefore, the first entry in sub-section 1 is 0×00, and the second entry is 0×07.

The second sub-section contains all of the string values. The string values are all null-terminated. They are also compressed in an unusual manner; characters with an ASCII value over 127 (possibly higher) are treated specially, and converted based on a lookup table hardcoded into the executable. For example, the value 0xC8 is translate to “oo” (double O). This compresses the string “Whoops!” down to “Wh.ps!”, with “.” representing the value of 0xC8. Other common letter combinations are compressed, such as “me”, “or”, “I am”, etc.

Strings are also interesting in that they can contain multiple variations of one line of dialogue. These are represented by nesting one or more entries at the start of the string, enclosed by square brackets. e.g.

“[[Hello my sweet.]Hello darling.]Good morrow, my love.”

It seems the game will randomly (or perhaps linearly) choose which variation to display.

Section 4

Section 4 starts with 2 bytes for the number of entries, followed by that many entries. Each entry is a word (16-bit).

Section 5

Section 5 starts with 2 bytes for the number of entries, followed by that many entries. Each entry is a word (16-bit). This seems to be the same format as Section 4.

Section 6

Section 6 starts with 2 bytes for the number of entries, followed by that many entries. Each entry is a word (16-bit). It is then followed by another 2 bytes for the second lot of entries. Each of these second entries is only one byte long.

Section 6

Section 6 starts with 2 bytes for the number of entries, followed by that many entries. Each entry is a word (16-bit). It is then followed by another 2 bytes for the second lot of entries. Each of these second entries is only one byte long.

Section 7

Section 7 is hardcoded as being 60 (0x3C) bytes long. Each value in this section seems to be a single byte.

Section 8

Section 8 starts with 1 byte for the number of entries. The number of entries can be 0. There are two sub-sections here.

Section 8 Sub-Section 1 contains an array of bytes. Each byte represents the size of the data chunk in Sub-Section 2. This array is summed to get the total size of the data.

Section 8 Sub-Section 2 contains the variable-sized data. The size of this section is determined by the summed value generated when reading Sub-Section 1.

Section 9

Section 9 starts with 2 bytes for the number of entries. Each entry consists of four lots of 16-bit words.

In Robin Hood, there are 37 entries.

Section 10

Section 10 is an odd one. It contains with a 2-byte value, followed by an array of 20 bytes, then an array of 20 words, then another array of 20 words, then another array of 20 bytes.

The values of the last array are manipulated when each value is read in, like so:

if val == 0x20:
val = 0x39
else if val == 0x06:
val = 0x1C
else:
val = val - 0x41


That was a massive wall of text, right? I’ve also added it to the [http://rewiki.regengedanken.de/wiki/.PRG|REWiki] site.

It looks like there was a “Profidisk” expansion released for the game, but only for the German version. This expansion adds a few more scenarios for burglarising. I’d like to play this expansion in the English version, but lack the Deutsch to translate it myself, so if anyone out there could help me, contact me at my e-mail (listed later).

Turns out the text requires very little to change; all text data is stored in external files, which are XOR encrypted using a key ox 0×75. At this point I’m not sure if any graphics will require editing, for now I will concentrate on the text.

If anyone wants to help me out by translating the text, I’d be grateful. E-mail me at jestarjokin@jestarjokin.net.

Also, while the German game data was released to the public, the English version remains in limbo. Would it be worthwhile to translate the whole game into English again, to avoid any copyright issues?

It may also be worth editing the original English text anyway, since from memory it didn’t make much sense at times.

If you want to unencrypt the files yourself, here’s a small Python script to do so. If you have Python installed, you can run it like “xor.py 0×75 TEXT\*.txt”.

xor.zip

EDIT: 30/05/2011 – I’ve found my translator! Turns out Google Translate does a damn fine job. I’ll tweak the text a bit to match some of the existing English translations and maybe add some flavour… stay tuned.

Win32 Binaries
http://www.jestarjokin.net/sw/scummpacker_bin_20110521.zip

Python Source
http://www.jestarjokin.net/sw/scummpacker_src_20110521.zip

• Fix issue where importing sounds into a MONSTER speech map did not sort the sounds by ID (original offset), resulting in invalid game resources when exporting.

Win32 Binaries
http://www.jestarjokin.net/sw/scummspeaks_bin_20110512.zip

Python Source
http://www.jestarjokin.net/sw/scummspeaks_src_20110512.zip

• Complete rewrite. Breaks all compatability with old versions.
• Supports SCUMM v4, v5, and v6 games, as well some v3 games.
• Outputs some information (such as headers) to XML format.
• No longer need to explicitly state the order of each block.
• All object data are contained within their own sub-folder.
• Local scripts are contained within their own sub-folder.
• License changed from “public domain” to the MIT/X11 license.

Win32 Binaries
http://www.jestarjokin.net/sw/scummpacker_bin_20110425.zip
Python Source
http://www.jestarjokin.net/sw/scummpacker_src_20110425.zip

SCUMM Image Encoder v2 r1

• Updated to work with ScummPacker v3
• New option to “freeze” the palette, preventing the original palette from being updated.
• New option to specify the quantization (number of colours in the palette)
• License changed from “public domain” to the MIT/X11 license.

Win32 Binaries

http://www.jestarjokin.net/sw/scummimg_bin_20110425.zip

Python Source

http://www.jestarjokin.net/sw/scummimg_src_20110425.zip

And that’s it!

I’m afraid I have no intention on performing any further work on my SCUMM tools. I will still perform maintenance work, and will consider feature requests, but I will be focusing on other projects.

Summary

• Fix problem trying to export sound resources, where the sound map contains sounds that are not mapped to any line of text (and will effectively be unused in the game). This might exhibit itself in an error box popping up with a message like “973996″.

Win32 Binaries
http://www.jestarjokin.net/sw/scummspeaks_bin_20110330.zip
Python Source
http://www.jestarjokin.net/sw/scummspeaks_src_20110330.zip