Identifying file formats from headers

Question

Can the format of an archived file be identified from its leading bytes, without decompiling it — and is that worth storing as searchable metadata?

Summary

• Yes. 134,165 of 134,166 entries (99.999%) classify from their leading bytes. The single holdout is .vfont, which is deliberately obfuscated.
• 133,657 entries are Source 2 resources, all sharing one header shape with headerVersion == 12 and a table of four-character block tags.
• Block tags are the useful part: they say what a resource contains, and they map almost perfectly onto extensions — PHYS appears only on models, SrMa only on vcss_c.
• The header's leading u32 is the declared resource size, but for vtex_c, vsnd_c and vcs the archived entry is larger: bulk data is appended past it. For textures that appended data is 99.9% of the bytes.
• Reading all 134,166 headers takes ~12 seconds and is stored by vpkdb.py sniff.

Findings

Source 2 resource header

The first 16 bytes, little-endian:

Each block table entry is 12 bytes: a 4-character ASCII tag, u32 offset, u32 length.

Resource version is constant per extension, so it is a property of the type rather than of individual files: vcs 70, vrr_c 17, vjs_c 4, vcss_c/vxml_c 3, vmix_c/ vsvg_c/vts_c 2, vmat_c/vmdl_c 1, vdata_c/vnmclip_c 0.

What is in the archives

Block tags, and which extensions carry them

195 distinct block layouts exist. The four most common cover 91% of files: RED2,DATA,CTRL (79,362), RERL,RED2,DATA (26,110), RED2,DATA (16,383), RERL,RED2,DATA,INSG (4,276).

Observed, not inferred: the table above reports which extensions carry each tag. The meaning of most tags is not established here — see Open Questions.

Appended bulk data

For most resources the declared size equals the archived entry size. Three types append data past it:

Overall 93,524 of 133,657 resources carry appended data and 40,133 do not.

Observed deltas for textures include 5,592,400, 2,796,200 and 1,398,096 bytes — close to the 4/3 ratio of a full mip chain over its base level. Inferred: the appended region is mip/sample/shader-variant payload that the engine streams separately from the resource header. Not confirmed by parsing the payload itself.

Reproduce

python tools/vpkdb.py build
python tools/vpkdb.py sniff          # ~12s for 134,166 entries

python tools/vpkdb.py sql "SELECT kind, count(*) n FROM header GROUP BY kind ORDER BY n DESC"
python tools/vpkdb.py sql "
  SELECT f.ext, h.res_version, count(*) n FROM header h JOIN file f ON f.id=h.file_id
  WHERE h.kind='source2' GROUP BY f.ext, h.res_version ORDER BY f.ext"
python tools/vpkdb.py sql "
  SELECT f.ext, count(*) n, sum(h.payload) appended, sum(h.hdr_size) declared
  FROM header h JOIN file f ON f.id=h.file_id WHERE h.payload > 0 GROUP BY f.ext"

Block-tag frequency and the extensions carrying each tag:

python -c "
import sqlite3, collections
c = sqlite3.connect('file:db/deadlock.db?mode=ro', uri=True)
m = collections.defaultdict(collections.Counter)
for b, e in c.execute('SELECT h.blocks, f.ext FROM header h JOIN file f ON f.id=h.file_id WHERE h.blocks IS NOT NULL'):
    for t in set(b.split(',')): m[t][e] += 1
for t in sorted(m, key=lambda t: -sum(m[t].values())):
    print(f'{t:<7}{sum(m[t].values()):>8,}  ' + ', '.join(f'{e}:{n}' for e, n in m[t].most_common(3)))"

Browsable at /explorer/: the format, resource version and block chips appear in the detail pane, and the block dropdown filters the whole listing.

Gotchas

1. Block count is not small. Complex models carry up to 193 blocks (one set per mesh/LOD). An early cap of 32 silently misclassified 61 vmdl_c files as unknown. Validate arithmetically — the block table must fit inside the entry — not with a magic ceiling.
2. Read enough bytes to reach the end of the block table. A 192-byte window truncates it for 131 files. vpkdb.py sniff re-reads those with a computed window.
3. The leading u32 is not the archived size for vtex_c, vsnd_c and vcs. Using it as a file length will truncate 26 GB of texture data.
4. BOM-prefixed text fails a printable-ASCII check. Two .txt files use UTF-8 and UTF-16LE BOMs; detect the BOM first.
5. .vfont is deliberately obfuscated and has no usable leading magic.
6. Classification is by content, not extension — that is the point. Do not "fix" a disagreement by trusting the extension.

Open questions

• The meaning of DSTF, INSG, LaCo, FLCI, TBUF and STAT is unverified. SrMa (source map), MVTX/MIDX (vertex/index buffers) and MRPH (morph) are plausible expansions only.
• Whether the 10 vtex_c files still carrying REDI rather than RED2 are simply old assets was not investigated.
• The appended-region hypothesis was not confirmed by decoding a texture's mip chain.
• Block offsets and lengths are parsed but not stored; only the tags are. Storing the sizes would show how bytes are distributed inside a resource.
• Nothing here validates that a resource's declared version matches what the engine expects — only that it is constant per extension in this build.

Sources

Derived entirely from the local install at build 6679. Block tag names are read from the files themselves; no external format documentation was consulted, which is why most semantics are left open above.