Reading Unreal Tournament packages in the browser

I have been playing Unreal Tournament since about 2000, made maps under the name Sabre, and I still host a mirror of the Unreal Archive. A package inspector was the obvious thing for the UT corner of this site. Drop a .unr on the page, get the name table, exports, textures and sounds back, with nothing leaving the machine.

It is a port of the bunnytrack.net package explorer. The original is a single 2,200 line window.UTReader = function(arrayBuffer) constructor, and a copy of it still sits in public/UTReader.js on my site. I wanted the same logic in app/utils/UTReader.ts, typed, so the Vue page could import it and the compiler would catch a wrong offset.

The header

A package starts with a magic number. Four bytes, little endian:

const SIGNATURE_UT = 0x9E2A83C1;

Every .unr, .utx, .u, .uax and .umx file begins with it. The page reads that same uint32 before building a reader, so a file with the right extension and the wrong bytes is rejected early. After the signature the header is fixed width and boring, the nicest thing a header can be: two 16 bit versions, a flags word, then three pairs of count and offset:

header.version          = r.uint16();
header.licensee_version = r.uint16();
header.package_flags    = r.uint32();
header.name_count       = r.uint32();
header.name_offset      = r.uint32();
header.export_count     = r.uint32();
header.export_offset    = r.uint32();
header.import_count     = r.uint32();
header.import_offset    = r.uint32();

Then it forks on version. Below 68 you get heritage_count and heritage_offset. From 68 you get a 16 byte GUID, stitched from four uint32 reads, and a generation_count followed by that many { export_count, name_count } pairs. Retail UT99 content takes the GUID branch.

version is what everything downstream keys off. Checks for > 61, < 63, >= 63, < 64, === 65 and >= 66 are scattered through the struct readers, because the engine grew fields in place and never renumbered anything.

Compact indices

This is the part of the format that makes it interesting. Unreal stores most integers as a variable length "compact index", so small numbers cost one byte. It is not LEB128. The first byte is special:

compactIndex(): number {
  let value = this.uint8();

  const isNegative = value & 0b10000000;
  let readNextByte = value & 0b01000000;

  value = value & 0b00111111;

  for (let byteNum = 2, shiftAmt = 6; readNextByte; byteNum++, shiftAmt += 7) {
    const byte = this.uint8();

    const valueBitMask = byteNum < 5 ? 0b01111111 : 0b00011111;

    // JavaScript converts to signed integers when shifting left,
    // so use zero-fill right shift to convert back to unsigned.
    value = ((byte & valueBitMask) << shiftAmt | value) >>> 0;
    readNextByte = byte & 0b10000000;
  }

  return isNegative ? -value : value;
}

Bit 8 of the first byte is the sign, bit 7 the continuation flag, and the remaining six bits the low part of the value. Every byte after that uses bit 8 to continue and the lower seven as value bits, except the fifth, where only five bits are left before the value overflows 32 bits, hence the byteNum < 5 mask swap. The comment is not decoration either. The shifts run 6, 13, 20, 27, so (byte & 0b00011111) << 27 on that fifth byte can reach the sign bit and come out negative, because bitwise operators work on signed 32 bit integers. The >>> 0 puts it back.

The name table

Names are interned. Everything else refers to a string by its index here, and the table is a list of strings with a flags word each.

if (this.header.version < 64) {
  // null terminated
} else {
  nameTable.push({ name: r.getSizedText(), flags: r.uint32() });
}

getSizedText reads a length byte, slices size - 1 bytes so the trailing null is dropped, then advances by the full size. Decoding is windows-1252, not UTF-8, which matters the moment you open a map with an accented author name. The exception is string properties, where the sign of the length carries the encoding:

getStringProperty(): string {
  const strSize = this.compactIndex();
  const isUtf16 = strSize < 0;
  const charWidth = isUtf16 ? 2 : 1;
  ...
}

Positive means plain ANSI, negative means UTF-16LE. That comes from a note in the source credited to Anthrax, who maintains the OldUnreal UT99 patch, and I would not have guessed it from the bytes.

Exports and imports

Two tables, both read straight through from their header offsets. An export is something the package contains:

this.class_index       = r.compactIndex();
this.super_index       = r.compactIndex();
this.package_index     = r.int32();
this.object_name_index = r.compactIndex();
this.object_flags      = r.uint32();
this.serial_size       = r.compactIndex();
this.serial_offset     = this.hasData ? r.compactIndex() : 0;

package_index is a plain int32 while its neighbours are compact. Mixing the two in one struct is not something you can guess, and getting it wrong shifts every field after it into beautifully plausible nonsense. serial_offset is only present when serial_size > 0, which is what hasData checks, and it is where the object's bytes live.

An import is shorter, because the data is in some other package: class_package_index, class_name_index, package_index and object_name_index.

Locating an object

Both tables share one index space, and this is the trick that makes the whole thing hang together:

getObject(index: number): ExportTableObject | ImportTableObject | null {
  if (index === 0) return null;
  if (index < 0) return this.importTable[~index] ?? null;
  return this.exportTable[index - 1] ?? null;
}

Zero is null, positive a one-based index into the export table, negative an index into the import table decoded with a bitwise NOT, so -4 is the fourth import and 12 the twelfth export. Every reference in the file, a texture on a surface, a mover's brush, an actor in a level, is one of these numbers.

Each object also has a package_index, so packageObject resolves the group it lives in and uppermostPackageObject loops until it runs out of parents. That is how a texture reference in a map resolves to the .utx it came from.

Properties

An object's serial data starts with its property list. The loop reads a name and stops when that name is None. Each property has one info byte:

prop.type = PROPERTY_TYPES[infoByte & 0xF]!;
const propSizeInfo = (infoByte >> 4) & 0x7;
const arrayFlag = Boolean(infoByte >> 7);

Low nibble is the type index into PROPERTY_TYPES, which runs Unknown, Byte, Integer, Boolean, Float, Object, Name, String, Class, Array, Struct, Vector, Rotator, Str, Map, Fixed Array. Three bits give the size, and the codes are not lengths: 0 to 4 mean 1, 2, 4, 12 and 16 bytes, while 5, 6 and 7 mean "read a uint8, uint16 or uint32 next for the real size". The top bit is the array flag, except on a Boolean where it is the value itself.

Structs get a second name read for the subtype. I decode color, vector, rotator, scale and pointregion, and skip the rest by propSize. Objects with the RF_HasStack flag, 0x02000000, carry a StateFrame before their properties.

One wrinkle worth knowing. A single BinaryReader with one offset is shared by the whole package, and native object data is read from wherever the property loop stopped. So the cached properties getter seeks back to the saved #propertiesEndOffset when called again, otherwise a re-read leaves the cursor wrong and the mesh data comes out as garbage.

What the page shows

app/pages/random/ut/package-reader.vue is a drop zone and a set of tabs. The file is capped at 256MB, the extension has to be in the allowed set, then file.arrayBuffer() goes into new UTReader(buffer).readPackage().

Summary gives version, name count, export count and import count, plus level info from getLevelSummary, which looks up the export named LevelInfo0. By default it keeps only Author, IdealPlayerCount, LevelEnterText, Song and Title, but the page passes true, so the tab lists every property on the object. Song holds an object index rather than a string, so it goes through getObject and is reported as the uppermost package name, which is how you learn a map plays Foregone.umx.

Dependencies walks the import table for entries whose class is Package and which are not themselves inside a package. Each name is checked against a hardcoded table of stock UT99 packages, so dmeffects is known to be a .utx and botpack a .u, and anything missing is custom. That is the list you need when packaging a map for release.

Textures render on demand, one click each, because decoding every mip up front is slow. textureToCanvas takes mip 0, follows the texture's palette property through getObject, and writes one RGBA pixel per byte into an ImageData. Level screenshots use the same path, matched by export name against /^Screenshot([0-9]+)?$/i with a fallback to the Screenshot property on LevelInfo0.

Sounds list format, sample rate and bit depth. Those last two are not in the package structure: the reader sniffs the bytes at the audio offset for a PCM WAV header and pulls them from fixed positions. Playing one slices the buffer into a Blob.

What it does not do

Plenty. It only reads UT99 era packages. The original JavaScript declares signatures for UMOD installers and compressed .uz files but never uses them, so I did not port them and compressed content is rejected.

Only the classes in getObjectDataReader have native readers: Animation, Font, Level, LodMesh, Mesh, Model, Music, Palette, Polys, SkeletalMesh, SkelModel, Sound, TextBuffer and Texture. Anything else gives you its properties and nothing more.

Texture rendering is palette indexed only, so compressed formats come out wrong or fail. Meshes and BSP parse but the page shows none of it. Music parses but has no tab, so a .umx gets you the summary, the dependency list and the name list, and nothing to play. And there are no unit tests yet, which I am not proud of. The browser is the test harness.

Still, dropping a twenty five year old map on a web page and watching its screenshot appear is a good way to spend an evening.