Custom Cipher Reverse — read the source, invert every primitive, run the pipeline backwards

A home-rolled cipher shipped as source (a Python/C "encrypt server" or chall.py) has no named attack in the literature — the win is to read the code, identify each primitive, and invert the pipeline step by step, last operation first.

The signal that gives it away

You're looking at a custom-cipher challenge, not a named-attack challenge, when:

- The challenge hands you encrypt() (or equivalent) source directly, rather than just a black-box binary or oracle. - The algorithm doesn't match anything from a standard library — no Crypto.Cipher.AES, no hashlib, no recognizable textbook name. - The prompt itself says "we made our own AES" / "we invented a new cipher" / similar framing. - The code is a loop of bit-twiddling: rounds, a key schedule, lookup tables, shifts/rotations, XORs, additions mod 2^k. - There is no known weakness to name-drop (this is NOT a case for differential/linear cryptanalysis, Coppersmith, or any of the classic algebraic attacks — those apply to *known* constructions with *known* weaknesses; here the construction itself is unknown and ad hoc).

If instead the source turns out to secretly reimplement a known primitive (a Feistel network, an SPN, a stream cipher) with one intentional flaw, that's actually a different case — see Cryptanalysis Methodology — classify the family before you reach for an attack to first classify the family, because a home-rolled *wrapper* around a real weakness (e.g. an ECB mode, a reused keystream) should be routed to the specific named attack instead of hand-inverted.

How to exploit it

1. Map the data flow first, don't start inverting yet. Read straight through encrypt() and write down, in order, every operation applied to the state: XOR with key, +k mod 2^n, rotate, S-box substitution, byte/bit permutation, mixing with a round counter, etc. Note anything that depends on the round index or a key schedule — you'll need the schedule reversed too (often just the same list of round keys in reverse order, sometimes literally the same key reused every round).

2. Invert each primitive individually. The recurring toolkit:

| Forward op | Inverse | |---|---| | x ^ k | x ^ k (XOR is its own inverse) | | (x + k) mod 2^n | (x - k) mod 2^n | | rotl(x, r) | rotr(x, r) | | permutation P | inverse permutation P^-1 (precompute by index-swap) | | S-box lookup SBOX[x] | inverse table: INV_SBOX[SBOX[i]] = i for all i | | byte/array reversal | reversal again (self-inverse) |

Concrete skeleton (parameters and constants are illustrative, not from any real challenge):

```python def rotr(x, r, bits=32): return ((x >> r) | (x << (bits - r))) & ((1 << bits) - 1)

INV_SBOX = [0] * 256 for i, v in enumerate(SBOX): INV_SBOX[v] = i

def decrypt_round(x, k): x = (x - k) & 0xffffffff # inverse of the forward add x = rotr(x, R) # inverse of the forward rotl x = INV_SBOX[x & 0xff] | (x & ~0xff) return x ^ k # xor is involutive, applied last since it was first ```

3. Apply the rounds in REVERSE order, with the reversed key schedule. This is the step people skip and then wonder why decryption produces garbage: if encryption does round(k0) -> round(k1) -> ... -> round(kN), decryption is inv_round(kN) -> ... -> inv_round(k1) -> inv_round(k0) — both the round order AND the key order flip, not just one of them.

4. Validate against a known plaintext/ciphertext pair before trusting the inverse on the real target. Almost every custom-cipher challenge gives you (or lets you generate) at least one known pair — use it to confirm every step round-trips exactly before running your decryptor on the actual flag ciphertext. This single habit catches the vast majority of off-by-one and order-of-operations bugs before they waste an hour.

5. If a step is genuinely non-invertible (truncation, a one-way hash mixed into the state, modular reduction that drops information), you can't algebraically undo it — treat that sub-step as its own small attack: brute-force the lost bits if the space is small, or look for a meet-in-the-middle split around it. Don't force an "inverse" formula onto a lossy operation; it will silently produce wrong output that still looks plausible.

Related

Encoding and Decoding Layers — peel base64/hex/URL/compression first or you attack the wrong plaintext Multi-Step Crypto Chain — recognizing and decomposing a pipeline challenge Cryptanalysis Methodology — classify the family before you reach for an attack Linear Algebra over GF(2) — solving bit-level unknowns as a linear system mod 2

What does NOT work

- Guessing a named attack because the cipher "looks like" AES/RSA/etc. A home-rolled cipher borrowing S-box-and-permutation *vocabulary* from AES does not inherit AES's proven security margin, but it also doesn't inherit any of the *named* attacks against real AES (related-key, biclique, ...) unless the challenge specifically reproduces the exact weakness those attacks target. Spend the time reading the actual code instead of pattern-matching to a textbook name. - Inverting operations in the same order they were applied. This is the single most common mistake: undo them last-to-first (LIFO). Getting the round order right but the key-schedule order wrong (or vice versa) is the classic silent failure mode — you get a decryptor that "runs" but outputs noise, and it's easy to misdiagnose that as a bug in the primitive inverses rather than the ordering. - Assuming every table lookup is invertible. An S-box only has a clean inverse table if it is a bijection (one-to-one on its domain) — this is a structural requirement of substitution-permutation network design specifically so that decryption is possible at all. If the challenge's "S-box" collapses multiple inputs to the same output (or maps to a smaller output space), building INV_SBOX[v]=i will silently overwrite earlier entries and give you a wrong, non-unique inverse. Check that the table is actually a permutation of its domain before trusting an inverse built this way. - Trusting the inverse without a known-plaintext check. Custom crypto code is exactly where off-by-one bit shifts, forgotten masks, and mod-arithmetic sign errors hide. An inverse that "looks right" by code inspection alone still needs to round-trip a real pair before you run it against the target. - Treating a non-invertible step as if it had a formula. Trying to algebraically "undo" a hash or a truncation instead of attacking it separately (brute force / meet-in-the-middle) burns time on an inverse that provably cannot exist.