Loading up the binary in IDA, we can see the program asks us to enter a key to continue, after which it calls a function verify_key on the input given. verify_key checks if the input is between 9 and 64 chars, otherwise it exits. We can see the function enc is called on the input string, and after the function runs, the returned value is compared to the string [OIonU2_<__nK<KsK. If they are equal, we’ll get the flag.

Looking into enc, we can see a 64 elements array initialised with malloc, the length of the input string is stored in a variable, and a counter variable is initialised to 0.

We can then see a loop until the counter reaches the same value as the input string length. The loop performs the following opperations:

arr[64][cnt] = ((s[cnt] + 12) * v72 + 17) % 70 + 48;
v72 = arr[64][cnt];

So, the value of arr has to be equal to [OIonU2_<__nK<KsK at the end of the enc function.

My solution was to write a script that would compute the value of the input string s and it dealt with the mod operation by computing 1000 (tweaked the program a bit while settling on this number) values the expression ((s[cnt] + 12) * v72 + 17) could have had. Not perfect, but quick and dirty.

def enc_re(arr_ord):
    def compute(arr, v72):
        _c = arr - 48
        ## candidates
        cand = []
        for _i in range(1000):
            c = _c + 70 * _i
            _x = c - 17
            _y = _x / v72
            a1 = _y - 12
            if _x % v72 == 0:
                ## are they printable ascii?
                if a1 >= 0 and a1 < 127:
                    if a1 not in cand:
                        cand.append(a1)
        return cand

    v72 = 72
    cand_dict = {}
    for _i in range(len(arr_ord)):
        cand = compute(arr_ord[_i], v72)
        if len(cand) == 0: 
            ## this helped tweak the number of repetitions in compute()
            print("did not compute")
        else:
            mstr = [chr(int(i)) for i in cand]
            cand_dict[_i] = mstr
        v72 = arr_ord[_i]
    for _key in cand_dict:
        print(_key, cand_dict[_key])

if __name__ == '__main__':
    target_string = "[OIonU2_<__nK<KsK"
    target_ord = [ ord(c) for c in target_string ]
    enc_re(target_ord)

What you get is a dictionary of possible values each character of the input could take:

0 ['\x01', '$', 'G', 'j']
1 ['\x02', '\x0c', '\x16', ' ', '*', '4', '>', 'H', 'R', '\\', 'f', 'p', 'z']
2 ['\x14', 'Z']
3 ['2', 'x']
4 ['\r', 'S']
5 ['\x06', '\r', '\x14', '\x1b', '"', ')', '0', '7', '>', 'E', 'L', 'S', 'Z', 'a', 'h', 'o', 'v', '}']
6 ['\x01', '\x0f', '\x1d', '+', '9', 'G', 'U', 'c', 'q']
7 ['\x04', '\x0b', '\x12', '\x19', ' ', "'", '.', '5', '<', 'C', 'J', 'Q', 'X', '_', 'f', 'm', 't', '{']
8 ['\r', '\x1b', ')', '7', 'E', 'S', 'a', 'o', '}']
9 ['\x06', '\r', '\x14', '\x1b', '"', ')', '0', '7', '>', 'E', 'L', 'S', 'Z', 'a', 'h', 'o', 'v', '}']
10 ['\x06', '\x14', '"', '0', '>', 'L', 'Z', 'h', 'v']
11 ['\x01', '\x0f', '\x1d', '+', '9', 'G', 'U', 'c', 'q']
12 ['\x04', '\x0b', '\x12', '\x19', ' ', "'", '.', '5', '<', 'C', 'J', 'Q', 'X', '_', 'f', 'm', 't', '{']
13 ['\x01', '\x0f', '\x1d', '+', '9', 'G', 'U', 'c', 'q']
14 ['\x01', '\x08', '\x0f', '\x16', '\x1d', '$', '+', '2', '9', '@', 'G', 'N', 'U', '\\', 'c', 'j', 'q', 'x']
15 ['\x0c', '\x1a', '(', '6', 'D', 'R', '`', 'n', '|']
16 ['\n', '\x18', '&', '4', 'B', 'P', '^', 'l', 'z']

And you can pick what your input should be. I chose G4ZxS09_7009_G26 after verifying a couple of options with GDB.

$> nc rev.tamuctf.com 7223

Please Enter a product key to continue: 
G4ZxS09_7009_G26 
gigem{k3y63n_m3?_k3y63n_y0u!}