flare-on 13: FlareOn13.doc
One file that is a PDF, a raw CD image, a UDF volume, a ZIP and a fat Mach-O at once, with six clues feeding one key.
Me: 10:09, 36.67M tokens, $9.73, didn't step in.
Truix: 5:31, 9.61M tokens, $4.13, didn't step in.
I mostly watched this one. A .doc that can't settle on a file type is pretty good spectator material. OpenAI's Sol 6 and its RE subagents pulled apart this FLARE-On challenge without any steering from me, although Sol did submit one of the clues as a flag before it found the checker.
A .doc with six different ideas about its file type
flareon13.doc is 1,855,887 bytes. The original SHA-256 is:
text codewrap
21632dd113a3b4d4678100fb4c37476a1aa77135b5e129e25fca92fbd6f4521f
The Python examples use Python 3.12 and write extracted files under re-analysis/reader-03/. Sol recovered everything statically on Windows x64, without running the DOS stub or either ARM64 executable.
The extension is pretty unhelpful. There are six format layers to account for: a DOS COM-style program, PDF, a raw CD-ROM image carrying UDF, a fat Mach-O inside that filesystem, ZIP, and a fixed VHD footer. Some overlap the same bytes; the Mach-O is a file inside the UDF layer.
Start with the boundaries:
shell codewrap
xxd -g1 -l 152 -s 0 flareon13.doc
xxd -g1 -l 64 -s 0x8D flareon13.doc
xxd -g1 -l 24 -s 0x1C4E39 flareon13.doc
xxd -g1 -l 8 -s 0x1C4F8F flareon13.doc
The first command shows 68 20 20 58 35 at offset zero, the start of a 16-bit DOS stub. The harmless EICAR antivirus-test text is visible at 0x60. Across 0x80 through 0x8B, the bytes are:
text codewrap
4c 45 21 24 25 50 44 46 2d 31 2e 37
L E ! $ % P D F - 1 . 7
That puts %PDF-1.7 at 0x84. PDF object 1 begins at 0x8D, declares /Length 1839086, and its stream data starts at 0xB2.
Near the end, the third command shows:
text codewrap
25 25 45 4f 46 0a 50 4b 03 04
% % E O F \n P K
The PDF's %%EOF starts at 0x1C4E39. Six bytes later, at 0x1C4E3F, there is a ZIP local-file header, PK\x03\x04. The final command finds conectix at 0x1C4F8F, the beginning of the last 512 bytes. That is the fixed-VHD footer, including a matching footer checksum.
A PDF viewer can happily finish at %%EOF while an archive tool finds a ZIP member after it.
Taking the sector headers out
The large PDF stream wraps a raw CD image. Its sectors are 2352 bytes long, with 2048 user-data bytes in each sector. Removing the sector overhead reconstructs a UDF 1.50 volume named FLAREON.
I'd be tempted to start counting at the PDF stream. That would be wrong: the raw-sector grid starts at document offset zero, not 0xB2. The PDF header and stream declaration already occupy part of sector zero.
For each of the first 781 sectors, take bytes [16:2064]. That removes the 16-byte front and the 288 bytes after the user data:
python codewrap
from hashlib import sha256
from pathlib import Path
raw = Path("flareon13.doc").read_bytes()
assert raw[0x84:0x8c] == b"%PDF-1.7"
assert len(raw) >= 781 * 2352
out = Path("re-analysis/reader-03")
out.mkdir(parents=True, exist_ok=True)
volume = b"".join(raw[s * 2352 + 16 : s * 2352 + 2064]
for s in range(781))
assert len(volume) == 1_599_488
print("UDF SHA-256:", sha256(volume).hexdigest())
(out / "flareon13.udf").write_bytes(volume)
# UDF sectors 278 onward are the contiguous click_me file's recorded extent.
click = volume[278 * 2048 : 278 * 2048 + 502_816]
print("click_me SHA-256:", sha256(click).hexdigest())
(out / "click_me").write_bytes(click)
The reconstructed volume should hash to:
text codewrap
6ff331ecba0148ec7082006cb33770a419f2d320846fb4e181ce617600c5a1d7
click_me should hash to:
text codewrap
c1df0f6bfdc367097a8fc2f904da0bdc1576fc0744e2fbbad27e205635cc92f9
A UDF-capable filesystem viewer can extract the files instead. There is also a README.TXT, whose entire useful contribution is benign corpus payload.
The recorded extent for click_me starts at UDF sector 278. Its first byte is at original document offset 0x9FA30, but the complete file is only contiguous after stripping the raw-sector overhead. If you anchor the extraction at the PDF stream, the click_me hash will disagree before you get anywhere near reversing its code.
The extracted file is a 502,816-byte fat Mach-O with two ARM64 slices:
| Slice | CPU subtype | Offset inside click_me |
Size | Code-signature identifier |
|---|---|---|---|---|
| 0 | 0 | 0x4000 |
0x10BD0 |
totally_fake_flag |
| 1 | 3 | 0x18000 |
0x62C20 |
true_honest_flag |
The small slice produces one clue. The larger slice was compiled with Crystal, with runtime type metadata such as LibC::Kevent at slice offset 0x41640, and checks all six clues before decrypting the answer. Despite the name, totally_fake_flag still has work to do.
Six formats does not mean six clues. The PDF gives two, the VHD gives none, and the Mach-O both supplies a clue and checks the whole set:
| Input | What to recover | Where it comes from |
|---|---|---|
| 1 | The modified antivirus-test string | DOS stub |
| 2 | White text encoded as EBCDIC | PDF object 5 |
| 3 | Text in an off-page JBIG2 image | PDF object 8 |
| 4 | A literal descriptor string | UDF |
| 5 | The contents of flag.txt |
ZIP |
| 6 | A masked custom-hash result | Small Mach-O slice |
The VHD footer and README.TXT are decoys, not extra inputs. I can live with a README that contributes nothing. The file named totally_fake_flag being necessary is more annoying.
Keeping the offsets straight
Sol's agents found six encoded digest records at original document offset 0xF9070. To inspect their references in the larger Mach-O, that offset has to pass through the sector conversion.
I don't trust mental arithmetic with this many coordinate systems. Just subtracting the document offset of click_me and the fat-file offset of slice 1 gives the wrong answer. Every raw sector includes 304 bytes that disappeared during extraction.
The conversion is:
text codewrap
Original document:
0xF9070 = 433 * 2352 + 0x640
Inside click_me:
(433 - 278) * 2048 + (0x640 - 16) = 0x4DE30
Inside Mach-O slice 1:
0x4DE30 - 0x18000 = 0x35E30
At image base 0x100000000:
0x100000000 + 0x35E30 = 0x100035E30
Sector 433 contains the record. Subtracting 278 removes the sectors before the file's extent; multiplying by 2048 uses the reconstructed sector size. Subtracting 16 removes the raw-sector header from the within-sector position.
One file, several coordinate systems. This diagram shows format order, not proportional byte lengths.
In IDA, load the larger slice, go to 0x100035E30, and follow its cross-reference to 0x1000017CC. The instructions load the first encoded record in two 16-byte pieces:
asm codewrap
1000017cc: LDR Q0, xmmword_100035E30
1000017d0: LDR Q1, xmmword_100035E40
1000017d4: MOV W1, #0x20
1000017d8: STP Q0, Q1, [X0]
1000017dc: MOV W0, #1
1000017e0: BL _calloc
The pattern continues for six records. Later, at 0x1000023D4 through 0x1000023F4, the code loads MD5's initialization words in reverse order: 0x10325476, 0x98BADCFE, 0xEFCDAB89, and 0x67452301.
Each stored record occupies 32 bytes, but the decoder only uses the low nibble of each byte. Two encoded bytes produce one decoded byte:
python codewrap
((hi & 15) << 4) | (lo & 15)
That turns each 32-byte record into a 16-byte MD5 digest. The table runs from document offset 0xF9070 through 0xF912F, giving Sol a way to check each clue's exact capitalization and encoding.
The DOS string changes before it is printed
The visible string at 0x60 begins with EICAR. The first 0x60 bytes are all printable ASCII, but disassembling them as 16-bit code at COM origin 0x100 shows a self-modifying DOS stub. It builds BX = 0x140, then changes the string one byte at a time:
asm codewrap
0100: push 0x2020
0103: pop ax
0104: xor ax, 0x2160 ; AX = 0x0140
0107: push ax
0108: pop bx
0109: push 0x4020
010c: pop ax
010d: xor ax, 0x6023 ; AL = 0x03
0110: xor byte ptr [bx + 0x20], al
Each following group reloads AX with 0x4020 before its XOR:
| File offset | In-memory address | XOR byte | Change |
|---|---|---|---|
0x60 |
BX + 0x20 |
0x03 |
E to F |
0x61 |
BX + 0x21 |
0x05 |
I to L |
0x62 |
BX + 0x22 |
0x02 |
C to A |
0x63 |
BX + 0x23 |
0x13 |
A to R |
0x64 |
BX + 0x24 |
0x17 |
R to E |
At 0x13B, the same push/pop/XOR pattern builds DX = 0x160, pointing to file offset 0x60. The last part sets AX = SI = 0x097B and subtracts SI from the words at 0x15C and 0x15E. That turns the printable bytes 48 2b 48 2a into cd 21 cd 20: int 21h, then int 20h. With AH = 09h, the first interrupt prints the string at DS:DX up to the $ at file offset 0x83; the second exits.
I like that even the interrupt instructions have to be assembled out of printable bytes. The first input is the string after those five XORs:
text codewrap
FLARE-STANDARD-ANTIVIRUS-TEST-FILE!
I like that the checker lets Sol settle this without running the stub. Compare both versions against the first encoded MD5 record:
python codewrap
from hashlib import md5
from pathlib import Path
d = Path("flareon13.doc").read_bytes()
disk = d[0x60:0x83]
assert disk.startswith(b"EICAR") and disk.endswith(b"FILE!")
in_memory_candidate = b"FLARE" + disk[5:]
encoded = d[0xF9070:0xF9070+32]
stored = bytes(((hi & 15) << 4) | (lo & 15)
for hi, lo in zip(encoded[::2], encoded[1::2]))
print("literal disk bytes match:", md5(disk).digest() == stored)
print("changed prefix matches:", md5(in_memory_candidate).digest() == stored)
The expected results are False, then True. A strings search gets you the wrong prefix; the checker wants FLARE.
White PDF text, encoded as EBCDIC
The PDF is encrypted, but an empty password decrypts it. With pypdf available, read the first page's decrypted content stream:
python codewrap
import re
from pypdf import PdfReader
pdf = PdfReader("flareon13.doc", strict=False)
print("PDF decrypt status:", pdf.decrypt(""))
content = pdf.pages[0].get_contents().get_data()
for encoded in re.findall(rb"<([0-9a-fA-F]{16,})>", content):
try:
text = bytes.fromhex(encoded.decode("ascii")).decode("cp037")
except (UnicodeError, ValueError):
continue
if "ghost" in text:
print("hidden cp037 text:", text)
Sol got decrypt status 2, the owner-password path. The parser can also warn:
text codewrap
invalid pdf header: b'h X5'
Those are the DOS bytes at the beginning of the polyglot. The PDF header is at 0x84, and the successful decryption still yields the page content.
Object 5 contains this hex text:
text codewrap
<9481899586998194856d8582838489836d878896a2a3>
Decode those bytes as EBCDIC code page 037 and you get mainframe_ebcdic_ghost. The page paints it white, which makes looking at the rendered page a poor way to find it.
Sol appended @flare-on.com and submitted mainframe_ebcdic_ghost@flare-on.com. CTFd rejected it.
I didn't step in there. The rejected submission sent Sol and the agents back to the program checking the clues, rather than appending the domain to another string. This one passed the second MD5 check.
Moving the JBIG2 image onto a page
The PDF also contains an 808-by-48 JBIG2 image. The original page draws it with:
text codewrap
q 12 0 0 1 -80 -80 cm /Im Do Q
That scales the image to 12 by 1 points and translates it to (-80, -80), off the page. I'm not going to find that by zooming.
Object 8's 381 encrypted bytes start at document offset 0x1C4A84. The empty-password PDF decryption yielded the revision-2, 40-bit RC4 file key 7f8fc27d1e. PDF encryption derives a separate key for each object. For object 8, generation 0, the derivation used here is:
python codewrap
md5(file_key + b"\x08\x00\x00\x00\x00").digest()[:10]
The five appended bytes encode the object and generation numbers. Decrypt the image stream with that key, then wrap the JBIG2 data in a new, unencrypted PDF whose page fits the image.
Rebuilding the page lets the image fill an 808-by-48 canvas instead. The complete wrapper includes RC4:
python codewrap
from hashlib import md5
from pathlib import Path
doc = Path("flareon13.doc").read_bytes()
file_key = bytes.fromhex("7f8fc27d1e")
image_key = md5(file_key + b"\x08\x00\x00\x00\x00").digest()[:10]
encrypted = doc[0x1C4A84:0x1C4A84 + 381]
assert len(encrypted) == 381
def rc4(key, data):
state = list(range(256))
j = 0
for i in range(256):
j = (j + state[i] + key[i % len(key)]) & 255
state[i], state[j] = state[j], state[i]
i = j = 0
out = bytearray()
for byte in data:
i = (i + 1) & 255
j = (j + state[i]) & 255
state[i], state[j] = state[j], state[i]
out.append(byte ^ state[(state[i] + state[j]) & 255])
return bytes(out)
image = rc4(image_key, encrypted)
draw = b"q 808 0 0 48 0 0 cm /Im Do Q"
objects = [
b"<< /Type /Catalog /Pages 2 0 R >>",
b"<< /Type /Pages /Kids [3 0 R] /Count 1 >>",
b"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 808 48] "
b"/Resources << /XObject << /Im 4 0 R >> >> /Contents 5 0 R >>",
b"<< /Type /XObject /Subtype /Image /Width 808 /Height 48 "
b"/BitsPerComponent 1 /ColorSpace /DeviceGray /Filter /JBIG2Decode "
b"/Length 381 >>\nstream\n" + image + b"\nendstream",
f"<< /Length {len(draw)} >>\nstream\n".encode() + draw + b"\nendstream",
]
parts, offsets = [b"%PDF-1.7\n"], []
for number, body in enumerate(objects, 1):
offsets.append(sum(map(len, parts)))
parts.append(f"{number} 0 obj\n".encode() + body + b"\nendobj\n")
xref = sum(map(len, parts))
parts.append(b"xref\n0 6\n0000000000 65535 f \n")
parts.extend(f"{offset:010} 00000 n \n".encode() for offset in offsets)
parts.append(
f"trailer\n<< /Root 1 0 R /Size 6 >>\nstartxref\n{xref}\n%%EOF\n".encode()
)
out = Path("re-analysis/reader-03/jbig2-preview.pdf")
out.parent.mkdir(parents=True, exist_ok=True)
out.write_bytes(b"".join(parts))
print(out)
The image dictionary keeps /Filter /JBIG2Decode, so a JBIG2-capable viewer does the image decoding. The new page has a [0 0 808 48] MediaBox and draws /Im across that page.
Sol used Edge's PDFium viewer. Open re-analysis/reader-03/jbig2-preview.pdf there, or in another viewer that supports JBIG2. If PyMuPDF is already available, this command produces a PNG at twice the image dimensions:
powershell codewrap
py -3.12 -c "import pymupdf; d=pymupdf.open('re-analysis/reader-03/jbig2-preview.pdf'); p=d[0].get_pixmap(matrix=pymupdf.Matrix(2,2)); p.save('re-analysis/reader-03/jbig2-preview.png'); print('image',p.width,p.height)"
That printed image 1616 96. The pixels spell jbig2_fax_geometry.

Much easier to read when it's on the page. Enlarged 2x.
The UDF clue and the ZIP called flag.txt
I appreciate that one of these clues is just sitting there. The UDF descriptor clue udf_tagged_descriptor appears literally in the original raw-sector bytes at 0xC1F5 and 0x154F5. The first occurrence can be checked without mounting or decoding anything:
powershell codewrap
py -3.12 -c "from pathlib import Path; raw=Path('flareon13.doc').read_bytes(); print(hex(raw.find(b'udf_tagged_descriptor')))"
The ZIP takes more work, despite containing a file helpfully named flag.txt.
shell codewrap
unzip -Z -v flareon13.doc
Its local header is at 0x1C4E3F. The member uses ZipCrypto encryption and compression method 2, the old Reduce-1 method.
The encrypted packed length is 222 bytes. Twelve of those belong to the ZipCrypto header, leaving a 210-byte compressed body. The uncompressed length is 18 bytes.
The compressed body starts with 192 zero bytes encoding an all-zero follower table: 256 follower sets, each with a six-bit count, take 256 * 6 / 8 = 192 bytes. That provides known plaintext for recovering ZipCrypto's internal state with bkcrack. The command used 32 known zero bytes at compressed-stream offset zero:
shell codewrap
./bkcrack-1.8.1-win64/bkcrack.exe -C flareon13.doc -c flag.txt -x 0 0000000000000000000000000000000000000000000000000000000000000000
The recovered internal keys were:
text codewrap
0917ed61 cfc8f696 5d3e9072
Those keys are enough to make a decrypted copy of the archive:
shell codewrap
./bkcrack-1.8.1-win64/bkcrack.exe -C flareon13.doc -k 0917ed61 cfc8f696 5d3e9072 -D re-analysis/reader-03/docpoly-decrypted.zip
The compressed bytes remain Reduce-1 data. For this member, the zero follower table is followed by the literal text, so the extraction doesn't need a general-purpose Reduce-1 implementation:
python codewrap
from pathlib import Path
from struct import unpack_from
p = Path("re-analysis/reader-03/docpoly-decrypted.zip").read_bytes()
off = p.index(b"PK\x03\x04")
method = unpack_from("<H", p, off + 8)[0]
packed = unpack_from("<I", p, off + 18)[0]
name_len, extra_len = unpack_from("<HH", p, off + 26)
body = p[
off + 30 + name_len + extra_len :
off + 30 + name_len + extra_len + packed
]
assert method == 2 and packed == 210 and body[:192] == bytes(192)
print(body[192:].decode("ascii"))
The remaining 18 bytes are reduce_not_deflate, the phrase supplied to the checker.
So flag.txt was another decoy. Its contents provide the fifth input.
The small Mach-O almost does SHA-1
The remaining clue comes from the small ARM64 slice at fat-file offset 0x4000. Its start function begins at VA 0x1000006B4.
The code processes totally_fake_flag with an 80-round SHA-1-like routine. Recognizing SHA-1's constants gets you close, but the message schedule differs in one instruction selection.
At 0x10000083C:
asm codewrap
10000083c: STRB WZR, [X22]
100000840: LDRB W8, [X22]
100000844: LDUR W11, [X9,#-0xC]
100000848: LDUR W12, [X9,#-0x20]
10000084c: CMP W8, #0
100000850: LDUR W13, [X9,#-0x38]
100000854: LDUR W14, [X9,#-0x40]
100000858: EOR W11, W12, W11
10000085c: EOR W12, W13, W14
100000860: EOR W11, W11, W12
100000864: EXTR W12, W11, W11, #0x1F
100000868: CSEL W11, W11, W12, EQ
10000086c: SUBS X10, X10, #1
100000870: STR W11, [X9],#4
100000874: B.NE loc_100000844
When X9 points to schedule word W[t], the four loads retrieve W[t-3], W[t-8], W[t-14], and W[t-16]. The displacements are in bytes, so -0xC means three 32-bit words back.
The three EOR instructions combine those words. EXTR W12,W11,W11,#0x1F prepares the usual one-bit left rotation. But the code has just stored zero through X22, loaded it into W8, and compared it to zero. CSEL W11,W11,W12,EQ selects the unrotated value.
The schedule loses the rotation:
text codewrap
SHA-1: W[t] = ROTL1(W[t-3] ^ W[t-8] ^ W[t-14] ^ W[t-16])
Here: W[t] = W[t-3] ^ W[t-8] ^ W[t-14] ^ W[t-16]
I'd recognize the SHA-1 constants and reach for hashlib.sha1(b"totally_fake_flag"). That gives the wrong digest. One missing rotation is enough.
Sol's reconstruction reads the initial message words and mask from the extracted binary, using the click_me extracted earlier:
python codewrap
from hashlib import md5, sha1, sha256
from pathlib import Path
from struct import unpack
EXPECTED_INPUT_SHA256 = (
"c1df0f6bfdc367097a8fc2f904da0bdc1576fc0744e2fbbad27e205635cc92f9"
)
EXPECTED_CLUE_MD5 = "9fc1de183e2761585dfce907afa249dc"
INITIAL = (0x67452301, 0xEFCDAB89, 0x98BADCFE, 0x10325476, 0xC3D2E1F0)
MASK32 = 0xFFFFFFFF
def derive(click: bytes) -> tuple[str, bytes]:
base = 0x4000
block = (
click[base + 0x1270 : base + 0x1280]
+ click[base + 0x1260 : base + 0x1270]
+ bytes(16)
+ click[base + 0x1250 : base + 0x1260]
)
words = list(unpack("<16I", block))
if (
b"".join(word.to_bytes(4, "big") for word in words[:4]) + b"g"
!= b"totally_fake_flag"
):
raise ValueError("unexpected slice-zero message layout")
for turn in range(16, 80):
words.append(
words[turn - 16] ^ words[turn - 14]
^ words[turn - 8] ^ words[turn - 3]
)
a, b, c, d, e = INITIAL
for turn in range(80):
if turn < 20:
f, k = (b & c) | (~b & d), 0x5A827999
elif turn < 40:
f, k = b ^ c ^ d, 0x6ED9EBA1
elif turn < 60:
f, k = (b & c) | (b & d) | (c & d), 0x8F1BBCDC
else:
f, k = b ^ c ^ d, 0xCA62C1D6
step = (((a << 5) | (a >> 27)) + f + e + k + words[turn]) & MASK32
a, b, c, d, e = step, a, ((b << 30) | (b >> 2)) & MASK32, c, d
digest_hex = "".join(
f"{(value + iv) & MASK32:08x}"
for value, iv in zip((a, b, c, d, e), INITIAL)
)
stored = click[base + 0x1280 : base + 0x1290]
clue = bytes(
ord(character) ^ mask
for character, mask in zip(digest_hex[:16], stored)
)
clue += bytes(
ord(digest_hex[16 + i]) ^ mask
for i, mask in enumerate((0x50, 0x4A, 0x07))
)
if md5(clue).hexdigest() != EXPECTED_CLUE_MD5:
raise ValueError("small-slice phrase does not pass the sixth MD5 gate")
return digest_hex, clue
content = Path("re-analysis/reader-03/click_me").read_bytes()
assert sha256(content).hexdigest() == EXPECTED_INPUT_SHA256
digest_hex, clue = derive(content)
print("small_slice_digest=" + digest_hex)
print(
"standard_sha1_is_different="
+ str(digest_hex != sha1(b"totally_fake_flag").hexdigest())
)
print("clue_utf8_hex=" + clue.hex())
print("clue_md5=" + md5(clue).hexdigest())
print(clue.decode("utf-8"))
It produces:
text codewrap
small_slice_digest=3e071c64234f1b9dd840e66a8bdf1608f1cf5be6
standard_sha1_is_different=True
clue_utf8_hex=c39864316e5f333733726e346c5f666c347233
clue_md5=9fc1de183e2761585dfce907afa249dc
Ød1n_373rn4l_fl4r3
The mask is XORed with the ASCII bytes of the digest's hexadecimal text, rather than the 20 binary digest bytes. The first 16 mask bytes live at small-slice offset 0x1280; three immediate values, 0x50, 0x4A, and 0x07, finish the output.
Then decode the result as UTF-8. The initial bytes c3 98 form Ø, giving Ød1n_373rn4l_fl4r3. Replacing that character with an ASCII O fails the sixth MD5 check.
Six MD5 checks, one RC4 key
The larger slice supplies the ordering and the final operation. It checks the six phrases against the nibble-encoded MD5 records, hashes each phrase with SHA-256, and XORs the six 32-byte digests together position by position.
After all that file-format archaeology, I'm happy to see a short XOR loop. Hash each phrase separately, though: concatenating them and hashing the result would produce a different key.
The ciphertext follows the digest records at document offsets 0xF9130 through 0xF917F. It uses the same low-nibble encoding, so 80 stored bytes become 40 ciphertext bytes. The first encoded pair is 86 84:
text codewrap
((0x86 & 0x0f) << 4) | (0x84 & 0x0f) = 0x64
Those bytes map to slice-1 offsets 0x35EF0 through 0x35F3F. The original code begins loading them at VA 0x10000191C.
The RC4 key-scheduling loop is at 0x100003530. It adds a permutation entry and key[i % key_len], wraps the index with AND #0xFF, swaps two entries, and repeats until i reaches 0x100. The following loop, at 0x100003598 through 0x1000035F0, generates the keystream and XORs it with the ciphertext.
The complete decoder can operate directly on the original document. It checks each recovered phrase against its stored MD5 before attempting the final decryption:
python codewrap
from hashlib import md5, sha256
from pathlib import Path
phrases = [
"FLARE-STANDARD-ANTIVIRUS-TEST-FILE!",
"mainframe_ebcdic_ghost",
"jbig2_fax_geometry",
"udf_tagged_descriptor",
"reduce_not_deflate",
"Ød1n_373rn4l_fl4r3",
]
raw = Path("flareon13.doc").read_bytes()
def low_nibble_pairs(encoded):
if len(encoded) % 2:
raise ValueError("odd nibble-encoded length")
return bytes(
((hi & 15) << 4) | (lo & 15)
for hi, lo in zip(encoded[::2], encoded[1::2])
)
for n, phrase in enumerate(phrases):
stored = low_nibble_pairs(
raw[0xF9070 + 32*n : 0xF9070 + 32*(n+1)]
)
actual = md5(phrase.encode("utf-8")).digest()
print(f"MD5 gate {n+1}:", actual.hex(), "match:", stored == actual)
assert stored == actual
digests = [sha256(s.encode("utf-8")).digest() for s in phrases]
key = bytes(
a ^ b ^ c ^ d ^ e ^ f
for a, b, c, d, e, f in zip(*digests)
)
print("RC4 key:", key.hex())
assert key.hex() == (
"5517bf5185094a81c64837a1387c8c2d755a2b26cbe342b34752ef62da43f473"
)
cipher = low_nibble_pairs(raw[0xF9130:0xF9180])
print("ciphertext:", cipher.hex())
assert cipher.hex() == (
"645128013769750e50451f917558f493e3aeed5a51d0cbfae98eed185e1b00b042a8f813a9d5551c"
)
s = list(range(256))
j = 0
for i in range(256):
j = (j + s[i] + key[i % len(key)]) & 255
s[i], s[j] = s[j], s[i]
i = j = 0
plain = bytearray()
for c in cipher:
i = (i + 1) & 255
j = (j + s[i]) & 255
s[i], s[j] = s[j], s[i]
plain.append(c ^ s[(s[i] + s[j]) & 255])
print("decoded:", plain.decode("ascii"))
The decoder prints six match: True lines and a 40-byte decrypted email. "Just a little treasure hunt." I'd call six overlapping formats a pretty generous definition of little.
CTFd accepted