From ad57d055f2d2d549ab1988e9e8d9c87667f5f4e9 Mon Sep 17 00:00:00 2001 From: Benjamin Palko Date: Wed, 14 Oct 2026 17:04:03 -0400 Subject: [PATCH] add workflow --- AGENTS.md | 22 ++++- docs/WORKFLOW.md | 203 +++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 222 insertions(+), 3 deletions(-) create mode 100644 docs/WORKFLOW.md diff --git a/AGENTS.md b/AGENTS.md index b2a3cccc..9aebb16e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -62,19 +62,35 @@ other licence-restricted vendor files. ## Analysis workflow +The full reproducible procedure, with commands, is in +[`docs/WORKFLOW.md`](docs/WORKFLOW.md). Follow it for every binary. In summary: + 1. Route the target first: native executables/libraries go through REA `open_binary` (never pass a Ghidra project file as the target — pass the original binary path under `binaries/`). 2. Work summary-first: `binary_overview` / summary detail before drilling into `analyze_function` or pseudocode. Reuse retained Evidence IDs instead of re-running identical analysis. -3. Export raw decompilation artifacts (pseudocode, listings) into `decomp/` +3. **Third-party libraries: every time one is discovered** (an imported DLL, a + version string, RTTI class prefix, library error strings, or a PDB path), + do the following before continuing analysis: + - pin its version; + - **search online for its source or headers** (official releases, git tags, + vendor repos, clean-room equivalents); + - download it into `third-party/`, or into the gitignored + `third-party/_local/` if its licence forbids redistribution; + - record it in `third-party/README.md`. + + If no source exists, record that and the acquisition routes. See + `docs/WORKFLOW.md` §3. +4. Export raw decompilation artifacts (pseudocode, listings) into `decomp/` using per-binary subdirectories, e.g. `decomp/GameServer_Release_USA_x64/`. Keep filenames stable and function-address-based so results are reproducible. -4. Track targets, artifact SHA-256 identities, and per-binary status in + For full coverage, use the headless scripts in `decomp/scripts/`. +5. Track targets, artifact SHA-256 identities, and per-binary status in `decomp/INVENTORY.md`; keep a per-binary finding ledger in `decomp//evidence.md`. -5. Close the REA session (`close_binary`) when the investigation is complete. +6. Close the REA session (`close_binary`) when the investigation is complete. ## Evidence and claims diff --git a/docs/WORKFLOW.md b/docs/WORKFLOW.md new file mode 100644 index 00000000..1e9816b1 --- /dev/null +++ b/docs/WORKFLOW.md @@ -0,0 +1,203 @@ +# Workflow — reverse-engineering a server binary + +The repeatable process used for `Agent_Release_x64.exe`. Follow it for each +binary in `decomp/INVENTORY.md`. Ground rules, layout and tooling are in +`AGENTS.md`. What has been learned so far is in `docs/ARCHITECTURE.md`. + +Every finding is recorded as **observed**, **inference** or **unknown**, with an +Evidence ID or decomp file reference. Static analysis never establishes runtime +behaviour. + +--- + +## 0. Prerequisites (once per machine) + +- Ghidra is at `/opt/ghidra`, and Java 21 and Node come from `mise.toml`. + - REA MCP is registered project-locally in `opencode.json`. + - Don't reinstall any of these; repair the configuration instead. +- **Ghidra headless fails with `UnsupportedClassVersionError` (class file + 71.0):** its compiled-script cache was built by a newer JDK. + - Move `~/.config/ghidra/ghidra__DEV/osgi/compiled-bundles` aside. + - Ghidra rebuilds the cache on the next run. + +## 1. Identify the binary (cheap, before any decompilation) + +```sh +sha256sum binaries/.exe +objdump -p binaries/.exe | grep -E 'DLL Name|Subsystem|MajorLinker|pdb' +``` + +In REA: +1. Run `open_binary` with the original path under `binaries/`. +2. Run `binary_overview`. +3. Run `search_strings` (regex) for config paths, log text, protocol names and + class names. + +Record the SHA-256, linker version, PDB path, imports and procedure count in +`decomp//evidence.md`. + +## 2. Determine language and toolchain + +Check for these C++ markers: +- RTTI descriptors (`strings -a | grep '^\.?AV'`) +- `__CxxFrameHandler3` +- an `MSVCP*.dll` import + +All 16 servers are MSVC 2010 C++ (see `docs/ARCHITECTURE.md`). Re-check each +new binary and record the result. + +## 3. Discover third-party libraries and acquire their sources (**mandatory**) + +Any library you can match against upstream source saves you from +decompiling it, and gives names, struct layouts and intent that the binary +lacks. **Every time a library is found, search for its source online before +continuing.** + +### 3a. Find candidates + +| Signal | How | +|---|---| +| Imported DLLs | `objdump -p` DLL names, cross-checked against `binaries/*.dll` | +| DLL version resources | `strings -el \| grep -A1 -E '^(FileVersion\|ProductName\|CompanyName)$'` | +| Embedded version text | `strings -a \| grep -iE '/[0-9]\|part of OpenSSL\|libcurl/'` | +| Statically linked C++ libraries | RTTI class prefixes (`TiXml*`, `Proud::*`, `ATL::*`) | +| Library-specific strings | assert/error messages (`Proud::CNetServerImpl::…`, TinyXML error table) | +| PDB paths | e.g. `…\OpenLibrary\openssl-1.0.0d\…`, which names the exact version | + +### 3b. Pin the version + +Use, in this order of preference: +1. An explicit version string (VERSIONINFO, `"OpenSSL 1.0.0d 8 Feb 2011"`). +2. A version in the PDB path. +3. **Differential evidence:** download several candidate releases and compare a + distinctive artefact with the binary. Examples are error-string tables, + exported symbol lists and header constants. + - Example: the TinyXML error table contains `Memory allocation failed.`. + That entry exists in 2.5.x and was removed in 2.6, which gives 2.5.1–2.5.3. +4. If none of these work, choose the closest era-compatible release and mark it + **closest, not exact**. + +### 3c. Search for the source + +Search in this order: +1. Official release archives (project site or SourceForge; check `old/` and + `archeology/` paths). +2. Official git tags (`git ls-remote --tags `), or a GitHub tag archive + if the release tarball no longer exists. +3. Vendor package repositories. These often contain headers only. +4. Clean-room equivalents (for example mingw-w64 headers for Microsoft APIs) + or open reimplementations (for example Wine). +5. `gh search repos ` for forks or mirrors. Treat these as unverified and + label them that way. + +Verify each download against an official checksum where one exists. Always +record its SHA-256. + +### 3d. Check the licence and place the files + +| Licence | Location | Committed? | +|---|---|---| +| Open source (OpenSSL, curl/MIT, zlib, BSD, public domain) | `third-party/-/` | Yes | +| Proprietary or EULA-restricted (e.g. ProudNet) | `third-party/_local/-/` | **No.** It is gitignored. Ask the user before fetching. | +| No public source (XTrap, Apex, HASP runtime) | nothing downloaded | Record acquisition routes only | + +Other rules: +- Keep upstream files unmodified. If you must change one (for example curl's + version stamp), record exactly what changed. +- Read the licence before committing anything. If redistribution is + prohibited, it goes in `_local/` or isn't fetched. + +### 3e. Document it + +Add a row or section to `third-party/README.md` with: +- the observed identity in the binary, with where it was seen; +- the version and how it was pinned; +- the source URL, tag or commit, and SHA-256; +- match quality (exact / closest / headers only); +- the licence; +- any local edits. + +Also: +- Mention the library in `docs/ARCHITECTURE.md` if it is statically linked. +- Add it to the list in `AGENTS.md`. +- If no source exists, add it to the "No public source" and "Acquisition + routes" tables instead. + +### 3f. Use it + +- Label library functions in the decompiled output from known exports, class + method tables, error strings and the matched headers. +- Exclude labelled library code from application analysis. + +## 4. Decompile + +In REA, choose seeds from distinctive strings or imports and use +`trace_feature` to find the functions that reference them. Then run +`procedure_pseudo_code` / `batch_decompile` on those functions. + +For full coverage, run Ghidra headless with the repo scripts: + +```sh +analyzeHeadless /tmp/opencode/ghproj X -import binaries/.exe -overwrite \ + -scriptPath decomp/scripts -postScript ExportDecomp.java "$PWD/decomp//full" -deleteProject +analyzeHeadless /tmp/opencode/ghproj X -import binaries/.exe -overwrite \ + -scriptPath decomp/scripts -postScript VTableMap.java "$PWD/decomp//full/vtables.tsv" -deleteProject +``` + +This produces: +- `full/functions/.c` and `full/index.tsv` from `ExportDecomp.java`; +- `full/vtables.tsv` from `VTableMap.java`, which maps class vftable slots to + functions. + +Generate `cpp/classes.hpp` from `vtables.tsv`, then check that it compiles: +`g++ -fsyntax-only`. + +## 5. Record evidence + +In `decomp//`: +1. Run `export_evidence_bundle` to write `evidence-bundle.json`. +2. Extract the REA pseudocode to `.pseudo.c` and the overview to + `binary_overview.json`. +3. Write `evidence.md`: the findings table (conclusion → type → evidence), the + proposed function names, and the open questions. +4. Update the status and role of the binary in `decomp/INVENTORY.md`. + +## 6. Lift to readable C++ + +`decomp//cpp/*.cpp` holds hand-lifted code for the analysed functions: +- Cite the original address on each function. +- Mark every name as inferred. +- Record the original bugs you notice (leaks, races). + +## 7. Port to Linux + +`src//` builds the lifted code with Meson (see `src/README.md`). Keep a +`platform.{hpp,cpp}` layer that maps each Win32 call to its nearest POSIX +equivalent. + +The per-binary `README.md` must contain: +- a substitution table: original address → Win32 API → Linux replacement → + behaviour difference; +- what is stubbed or not ported, and why. + +Build with `ninja -C build` and smoke-test it. A passing test proves only that +the port works, not how the original behaves. + +## 8. Close + +Run `close_binary` when the investigation is complete. Before closing, export +the Evidence bundle, because session annotations are temporary. + +--- + +## Checklist per binary + +- [ ] SHA-256, imports, linker and PDB recorded +- [ ] Language confirmed +- [ ] **Every third-party library identified, its version pinned, its source + searched for and downloaded where licensing allows, and it documented in + `third-party/README.md`** +- [ ] Key functions decompiled; `full/` export done +- [ ] `evidence.md`, `INVENTORY.md`, `docs/ARCHITECTURE.md` updated +- [ ] Lift and Linux port updated, if in scope +- [ ] Evidence bundle exported; REA session closed