Deadlock Research

AGENTS.md

Instructions for automated contributors. This file follows the AGENTS.md convention and is read by most coding agents (Codex, Cursor, Aider, Zed, Claude Code, and others). It is intentionally plain Markdown with no vendor-specific syntax — if your tool looks for a different filename, point that file at this one rather than duplicating the content.

What this project is

Reverse-engineering the data formats of Deadlock (Valve, Source 2; internal codename Citadel), with the goal of extracting game data — heroes, purchasable items, abilities — into usable structured form.

Nothing here is documented by Valve. Every fact is derived from a specific installed game build and goes stale when the game patches.

Non-negotiable rules

  1. Never state a fact about game data you have not run a command to confirm. Not from a wiki, not from a forum, not from training data. If an external source claims something, re-derive it locally before writing it down. Mark anything you could not verify as unverified, explicitly.
  2. Stamp everything with the game build. Run python tools/find_game.py and put the ClientVersion in the note's verified_against field. An unstamped claim is a rumour.
  3. Report what you did not check. Every research note has an Open Questions section. Leaving it empty when you skipped something is how wrong data spreads.
  4. Do not commit extracted game assets. They are large and copyrighted. Commit the scripts and the findings, never the output. .gitignore already excludes the usual output paths — keep it that way.
  5. Separate observation from inference. "277 entries carry EAbilityType_Item" is an observation. "There are 277 purchasable items" is an inference. Label the second one.

Doing research

The full loop is documented in docs/research/README.md. Short version:

  1. Frame one answerable question.
  2. Verify it against the real install using the tools below.
  3. Copy docs/research/TEMPLATE.md to docs/research/NNNN-kebab-title.md.
  4. Paste real commands and real (possibly trimmed) output. Never invent output.
  5. Add a row to the index table in docs/research/README.md.

Tools

Stdlib-only Python 3.10+. No install step, no virtualenv, no dependencies — this is deliberate so any contributor or agent can run them immediately.

python tools/find_game.py              # locate install, print build stamp
python tools/find_game.py --json       # same, machine readable
python tools/vpk_list.py               # summarize the VPK index
python tools/vpk_list.py --prefix scripts/ --ext vdata_c --list
python tools/vpk_list.py --manifest    # path/crc/size lines, for diffing across patches

If the install is not auto-detected, set DEADLOCK_GAME_DIR to the directory containing citadel/ and core/.

The file index (read this before grepping the game)

tools/vpkdb.py indexes every VPK into a SQLite database at db/deadlock.db. Build it once and query it rather than re-parsing archives or shelling out to a decompiler for questions the index can already answer.

python tools/vpkdb.py build                     # ~3s, indexes all 21 archives
python tools/vpkdb.py find hero --ext vdata_c   # substring; glob if it has * ? [
python tools/vpkdb.py show scripts/heroes.vdata_c
python tools/vpkdb.py cat scripts/items/items_game.txt   # raw bytes, no decompiler
python tools/vpkdb.py stats --by dir
python tools/vpkdb.py diff db/build-6600.db     # what a patch changed, by CRC32

Every command takes --json. For anything the subcommands do not cover, use the read-only SQL escape hatch instead of asking for a new flag:

python tools/vpkdb.py sql "SELECT ext, count(*) n, sum(size) b FROM file GROUP BY ext ORDER BY b DESC LIMIT 10"

Schema: file(path, dir, top, name, ext, size, length, preload, offset, archive_index, crc32, vpk_id), vpk(mount, name, rel_path, version, entry_count), meta(key, value). meta carries client_version — check it matches the note you are working from.

Header metadata. sniff reads the first bytes of every entry (~12s) and records the detected format, Source 2 resource version, block tags, and how many bytes are appended past the declared resource size. See note 0006.

python tools/vpkdb.py sniff
python tools/vpkdb.py sql "SELECT kind, count(*) n FROM header GROUP BY kind ORDER BY n DESC"
python tools/vpkdb.py sql "
  SELECT f.path FROM header h JOIN file f ON f.id=h.file_id WHERE h.blocks LIKE '%PHYS%' LIMIT 5"

Run it after build — rebuilding the index drops the header table. The explorer shows this metadata and gains a block-tag filter once it has been run.

Reading KeyValues v1 (.txt, .res, .gameevents, .vdf). Use tools/kv1.py. Loose text files are a mix of formats — 35 KV1, 26 KV3, 21 neither — so call sniff() rather than trusting the extension. Two rules from note 0008: resolve platform conditionals (they are expressions like [0 && $X360], and WIN32 wins because this is a Windows game), and expect duplicate keys, which collapse into lists.

Reading KV3. Use tools/kv3.py, never a regex. Note 0007 records why: keys are not identifiers (an identifier regex drops 8% of entries), keys can carry escapes, and booleans are spelled true / "true" / 1 within one document. Two hard rules:

Optional text layer. After decompiling (note 0003), index-text splits KV3 documents into top-level entries and indexes them with FTS5:

python tools/vpkdb.py index-text out/
python tools/vpkdb.py search "EItemSlotType_Tech EModTier_4"
python tools/vpkdb.py sql "SELECT key, lines FROM text_entry WHERE source='scripts/heroes.vdata'"

db/ is gitignored — it is derived data and rebuilds in seconds. Never commit it.

The same index is published as a browsable explorer at /explorer/ on the site, generated by tools/build_explorer.py. Its payload ships paths, sizes, extensions, archive indices and mounts — never file contents. Keep it that way: the site is public. crc32 is deliberately excluded because it is high-entropy and would nearly double the transfer for a field only the CLI needs.

Extraction via cat/extract yields compiled resources. Only genuinely loose files (.txt, .cfg, .res) are readable straight out; everything else still needs the decompiler in note 0003.

The gallery (site/gallery/, from tools/build_gallery.py) is a rendered view of the same data: heroes with their m_mapBoundAbilities signature slots, and named shop items — which is a narrower set than note 0002's item counts. The derivation is pinned in tools/test_gallery.py: 277 typed entries − 2 slotless bases − 80 unnamed (18 live tier-scaffolding + 62 disabled test items) = 195 shown, 173 live. Descriptions are sanitised at build time (escape everything, re-enable a fixed span whitelist); {s:}/{g:} templates are substituted there too. Keep game data out of innerHTML unless it went through that path.

Its third tab, item progression (#/progression), re-reads the same visible item set as build chains: connected components over m_vecComponentItems taken undirected, 35 chains over the 156 default-visible items with 57 items left standalone (note 0012). Its category axis is m_eShopFilters, the game's own 9-token item grouping. That field is where the shop files an item, not a list of what the item grants — Extra Health carries the Weapon Damage token — so do not relabel that axis as "what it gives you"; the page caption says what it is on purpose. The tokens are unlocalized at 6679, so the labels are derived by camel-splitting the token, and the ordering of the payload's shopFilters table is what the view's sections follow.

The ability-card kit (tools/assets/ability-card.{js,css}) is vendored and shipped verbatim; the gallery feeds it data and overrides its look from gallery.css rather than editing it. Kit renderers escape everything except the fields documented as HTML (description, note, tier bonus, and renderItemChain's { html } node escape hatch) — those must already have been through the build-time sanitiser.

The documentation site

docs/ holds Markdown sources only. The rendered site is built into site/, which is gitignored, and published to the gh-pages branch that GitHub Pages serves. Generated HTML must never be committed to main.

python tools/install_hooks.py       # one-time per clone: enables both hooks
python tools/decompile.py --fetch   # optional: populates out/ for the KV3 views
python tools/build_site.py          # render docs/ + explorer + content -> site/
python tools/build_site.py --check  # non-zero exit if site/ is stale
python tools/deploy_site.py         # build, commit to gh-pages, push
python -m unittest tools.test_kv3   # KV3 parser tests
node tools/test_explorer.js         # explorer regression tests (needs Node)

Every page footer carries the build time in a <time data-built datetime="..."> element whose visible text is the UTC fallback; a small inline script rewrites it to the reader's local timezone on load, and site/build.json records the same instant for the explorer. --check normalises both the readable and ISO timestamps away before comparing, so it answers "is the content stale?" rather than "has the clock moved?" — a rebuild always refreshes the stamp on disk but only reports staleness on real changes. Keep the build time out of index.json; putting it there would churn 7.8 MB per build.

build_site.py also runs build_content.py, which emits per-file artifacts under site/explorer/data/content/. They are keyed <mount>/<path> because 11 text files exist in both citadel and core with different contents. Content is only published for loose text, decompiled KV3 and English localization — never binary assets.

Images come from decompiling vtex_c (→ PNG) and vsvg_c (→ SVG), and are copied only when the data references them. Do not switch that to copying directories: the referenced set is 982 files / 36.6 MB, while panorama/images/heroes/backgrounds/ alone is 126 MB that nothing points at.

Images at or below 8 KB are inlined into the document that references them as data URIs (payload.images), because table rows re-render on every scroll frame and a per-row <img src> turns scrolling into a request storm. Larger art stays a URL and is fetched through a bounded queue (IMG_MAX_CONCURRENT) that retries with exponential backoff and renders a placeholder until ready. Do not emit a raw <img src> for a fetched URL — always go through imgTag.

Detection walks every value, not a field list — image references appear under at least 14 keys (m_strAbilityImage, m_strShopIconLarge, m_strWeaponImage, m_strLogoImageEnglish, …), so matching on field names misses most of them. Resolution: file://{images}/x.psd → panorama/images/x_psd.png, file://{images}/x.svg → panorama/images/x.svg. Editor tools/images/*.png and material *.vtex references are deliberately ignored.

tools/decompile.py writes out/<mount>/... and defaults to every mount with no path filter. Do not narrow it back to scripts/: 22 of the 99 vdata_c files live outside it, and narrowing silently removes their View button. Both test suites assert coverage against the index rather than a fixed count, so a gap fails rather than passing quietly.

build_site.py also invokes build_explorer.py, so --check covers the explorer too. Note that --check compares against db/deadlock.db: if you have not run vpkdb.py build, the explorer falls back to a placeholder page and --check will report a difference. That is expected, not a bug.

The explorer's JavaScript has no dependencies and no CDN, deliberately. Do not add a library without a concrete reason — a tree, a virtualised list, and substring search do not need one, and vendoring would break the repo's zero-setup property.

With hooks installed, pre-commit fails the commit if the Markdown stops rendering, and pre-push publishes on any push of main. Neither stages generated files.

New research notes are picked up automatically — the builder globs docs/research/NNNN-*.md. You still need to add the row to the index table in docs/research/README.md by hand.

Decompiling compiled resources (*_c files) requires the third-party Source2Viewer CLI. It is not vendored here. See note 0003 for how to obtain and invoke it.

Conventions

Before finishing a change