pwning bin2json

So OJ posted this pwnable that had evidently survived the
BsidesCBR and Kiwicon CTFs and it got my interest.

I haven’t really been CTFing actively for years, but I still occasionally enjoy
pwning things, and I don’t remember if I’ve ever done one of OJ’s challenges.
So I grabbed a copy, and decided to see how lazy I could be while still shaking
a shell out of this thing.

His challenge talks some goofy binary protocol that I didn’t even attempt to
understand, but he helpfully included a sample script that emits a valid
message. I started building qemu for afl, and had a quick look at the
binary in binja. A quick patch to make it exit after a call to display_all
later to make fuzzing practical later, I started the fuzzer and got dinner.

Just to be clear, the process of setting up afl is really nothing special. All I did here was put a single testcase (The verbatim output of OJs sample) into an input directory and run it with:

afl-fuzz -Q -i input -o output -- ./bin2json

After about 20 mins I had a crash, and a quick inspection revealed this:

0x0000000000400ca3         mov        rax, qword [ds:rax+0x210]
0x0000000000400caa         test       rax, rax
0x0000000000400cad         je         0x400cc3

0x0000000000400caf         mov        rax, qword [ds:_IO_stdout]
0x0000000000400cb6         mov        rsi, rax                                  ; argument #2 for method fputc
0x0000000000400cb9         mov        edi, 0x2c                                 ; argument #1 for method fputc
0x0000000000400cbe         call       fputc

0x0000000000400cc3         mov        rax, qword [ss:rbp+var_338]               ; XREF=display_all+100
0x0000000000400cca         mov        eax, dword [ds:rax+0x200]
0x0000000000400cd0         test       eax, eax
0x0000000000400cd2         jne        0x400ce1

0x0000000000400cd4         mov        qword [ss:rbp+display_func], 0x400b9d
0x0000000000400cdf         jmp        0x400d1d

0x0000000000400ce1         mov        rax, qword [ss:rbp+var_338]               ; XREF=display_all+137
0x0000000000400ce8         mov        eax, dword [ds:rax+0x200]
0x0000000000400cee         cmp        eax, 0x2
0x0000000000400cf1         jne        0x400d00

0x0000000000400cf3         mov        qword [ss:rbp+display_func], 0x400bfa
0x0000000000400cfe         jmp        0x400d1d

0x0000000000400d00         mov        rax, qword [ss:rbp+var_338]               ; XREF=display_all+168
0x0000000000400d07         mov        eax, dword [ds:rax+0x200]
0x0000000000400d0d         cmp        eax, 0x1
0x0000000000400d10         jne        0x400d1d

0x0000000000400d12         mov        qword [ss:rbp+display_func], 0x400b4e

0x0000000000400d1d         mov        rax, qword [ss:rbp+display_func]          ; XREF=display_all+150, display_all+181, display_all+199
0x0000000000400d24         lea        rdx, qword [ss:rbp+var_320]
0x0000000000400d2b         mov        rdi, rdx
0x0000000000400d2e         call       rax

So, a classic uninitialized variable bug. var_320 is not initialized on
entry, so if var_338 is not 0, 1 or 2, then the value is left
unchanged, and subsequently called.

This is pretty simple to exploit, just find some other function called by main,
and corrupt that same slot on the stack, then arrange for display_all to be
called, and control pc.

As it happens, OJ thought of this and arranged the stack neatly in the only
other callable function (create_person) to avoid this. This is where I got
bored and wandered off, since this testcase gave me nothing to go on.

A few hours later, afl had found a second testcase that allowed PC control. I
never actually got around to reversing it to see how it worked, but quickly
looking at the crash I saw pc = 0xfcfcfcfcfcfcfcfc, and my testcase had a few
massive slabs of fc bytes in it.

A quick script to tamper with them in turn until I found which bytes were
corrupting that slot on the stack revealed that I had control of about 440
bytes of stack. I put a few quick gadgets together by hand to pop over the
garbage I needed on the stack, and then used ROPgadget to
generate a chain to get a shell. Their chain was gigantic, I dicked around for
a while trying to find a couple of neat gadgets to smuggle r10 into rax, which
was pretty close, and then remembered that I had full control just used a pop
rax
to populate the syscall number, which ROPgadget doesn’t do for reasons I
don’t understand.

Finally:

(yes, my dumb DO image I brought up quickly doesn’t have lldb in it’s repo for
some reason)

(gdb) r < /root/pwn/exploit_with_payload
Starting program: /root/pwn/bin2json < /root/pwn//exploit_with_payload
process 2099 is executing new program: /bin/dash

You can find the shitty script I used to patch my testcase here:

https://gist.github.com/richo/11c1539860f1c7e77e472de0a96c767f

In the spirit of really, really avoiding investing effort in this, I’ve left
all my janky experiments in as comments.

It’s worth shouting out that voltron saved my ass again, the stack view is the best when you’re trying to map between a binary testcase and where things end up in memory.

And of course, thanks OJ for nerdsniping me!

Posted in ctf | Leave a comment

IRC outage (2015-04-15)

Through a ridiculous series of events and incredibly poor luck (read: I hope it’s not incompetence) our VPS provider is experiencing power issues. You can follow along at TransIP’s , some of you may need to hunt down a babelfish.

Updates will be available through this post and through the psych0tik twitter.

Since I can’t make it go, I’m going to take a shower instead.

Update@2015-04-15 03:11 UTC:
Some systems are returning to service but we don’t have VPS control panel yet. I might recommend a good pair of jumper cables to them soon.

Update@2015-04-15 05:26 UTC:
Refresh; still nothing; repeat. I might sleep first and get everything running first thing in the morning. I’ll give them another 30 minutes.

Update@2015-04-15 06:19 UTC:
IRC is back!

Posted in psych0tik News | Leave a comment

QuickBlog: Trac, SVN, Python and threads

It’s been a long time since I’ve done any reasonable blogging.  Between work and life, my plate has be been full and the last thing on my list usually is “write a blog about stuff you’ve done.”  More times than not, the things I’ve done I cannot write about due to work contractual concerns with work.  In an attempt to revitalize some posting and encourage myself to push more content out, I’m going to try to start creating these QuickBlogs.  These will be mostly focused around specific, bite-size content and will likely get a lot less word-smithing than my typical articles.  So there’s the intro.  Quick. Easy.  Let’s move on to the juicy stuff.

The problem

As part of my job, I’m often forced to look at or work on obscure setups that probably shouldn’t be production solutions .. and then fix them.  In this case, we’re talking about a Trac instance supporting an SVN repository.  Trac is being run under Apache2 using mod_wsgi in it’s daemonized, workermode configuration.  The problem statement is simple:  Trac begins spitting out 500 response codes to certain requests and before long, all requests are returning 500’s after exceptionally long wait times.  To make matters a bit more interesting, these errors began occurring mere hours after a full system upgrade.

The first problem was logging.  Trac’s logs simply didn’t tell us what was happening.  Apache’s logs were even worse, random 500’s with no real indicator or commonality between the failures.  A peek at the resources on the box showed CPU and memory consumption would eventually reach obscene levels and the only obvious fix required processes to be killed.

After a number of eyes spent time pouring through the logs we identified a particular SVN revision that seemed to be giving Trac the business more than others.  A quick check of this changeset revealed a 1.2 Gigabyte diff of a large file with significant modifications.  This changeset also was committed just a few hours after our system upgrade, making sense with our problem timeline.  Now that we’ve got a potential root cause, it was time to sort out this logging issue.

Trac’s native logging leaves something to be desired, but the trac.log file did include enough log text to make finding the last logged action within the code trivial.  Looking around within the code, I noticed that Trac provides an object for doing debugging logging that I could use to dump to the trac.log file.  Most code within Trac has access to the self.log.debug() function, which I made heavy use of throughout to identify where execution was going, what variables looked like, and most importantly – where the code eventually would block and stop processing the diff.

A few hours of debugging later, I had my code location (or at least close enough to know what the issue was.)  After the diff() was run within Trac, Trac attempts to htmlify() the results for presentation to the user; an action that takes far too much time when your diff is 1.2 Gigabytes.  With the additional logging I added, I could see the 500’s and the completion of this long-running function coincide, telling me that I’ve found my bottleneck.  This is only half the answer however; as we have pages that have nothing to do with this, or any, changeset to the code that also will toss 500’s back at us.

The obvious question was “is this related to our massive changeset issue,” to which I thought an equally obvious answer was “probably, yes.”  During the troubleshooting of the changeset issue, I noticed that the logging and seemed to lag behind my actual work.  I’d make an update to the code, refresh the Trac web page, and my logs would not reflect my code changes.  A few requests later, I’d see those changes.  Based on this, I was fairly certain that something about these diffs was cascading into other execution.

With a problem definition in mind, I moved forward to test my problem hypothesis.  I modified my Trac logging so that I could clearly see when requests came in and completed.  I then opened up a two different pages in the Trac web interface.  One page was a diff-view of the problem changeset and the other was an unrelated, lightweight page that should have always returned content.  I issued a few fast paced requests to the diff-page (literally just spam clicking the refresh/cancel button in firefox), followed by a single request to our canary page (in this case a random Trac ticket.)  Watching the logs, I saw exactly what I expected – when each diff-page request got to the htmlify() function, it would block ALL THE OTHER REQUESTS from even being accepted (at the mod_wsgi level, Apache still saw them.)  As each request finished, the next in queue would be processed and would block again, and again and again.  This cascade of blocks caused the requests for our ticket to fail because even tho the Python/Trac code executed quickly, by the time the request was even processed by Trac, Apache had already given up on it and moved on with a 500.

So the question you may be asking here is (especially if you don’t know me, and haven’t heard a rant on python threading) – Why didn’t a worker-mode, daemonized version of mod_wsgi simply allow the various threads to handle the requests in parallel, completely side-stepping this problem?  The answer is within the Python interpreter itself and it’s GIL (as well as potentially the implementation on the Trac side.)

Python’s GIL acts as a safety mechanism, preventing you from doing “unsafe things” to volatile shared memory.  This works excellently for protecting your data, but also means that Python’s threads can only run one at a time, per core/processor.  Having only a single instance of Python means that, because Python can only use a single core/processor per instance, every request blocked the rest.  This resulted in a single bad page being able to push the entire request stack so far back in time that they all had to result in 500’s.

While Trac could be patched to detect large diffs to ignore or the Apache/mod_wsgi set-up could potentially be moved to a pre-fork/embedded mode (much more memory intensive, but less CPU blocking) – we determined the best fix was to simply purge our SVN of this bad revision (which really shouldn’t have been committed to begin with.)

Fixing the SVN was fairly trivial.  I identified the problem revisions, and used svnadmin’s dump and load, with a few flags, to rebuild the SVN repository minus the bad changes.  Note, this may cause problems if you’re removing anything that is modified by a future revision, in my case I knew these weren’t further modified and were safe to remove revisions of.   We’ll assume an SVN repository going up to revision 100, with problem revisions 89 and 92:

  
  mkdir /tmp/svn-workspace
  svnadmin create /tmp/svn-workspace/fixed-svn
  svnadmin dump /path/to/svn -r 1:88  > /tmp/svn-workspace/first_part.dump
  svnadmin dump /path/to/svn --incremental -r 89:91 > /tmp/svn-workspace/second_part.dump
  svnadmin dump /path/to/svn --incremental -r 93:HEAD > /tmp/svn-workspace/third_part.dump
  svnadmin load --force-uuid /tmp/svn-workspace/fixed-svn < /tmp/svn-workspace/first_part.dump
  svnadmin load --force-uuid /tmp/svn-workspace/fixed-svn < /tmp/svn-workspace/second_part.dump
  svnadmin load --force-uuid /tmp/svn-workspace/fixed-svn < /tmp/svn-workspace/third_part.dump
  trac-admin /path/to/tracdata/ repository resync YourSVNName

References:

http://blog.dscpl.com.au/2009/03/load-spikes-and-excessive-memory-usage.html

Posted in QuickBlogs | Tagged , , , , , | Leave a comment

Let’s write a compiler in rust: pt1

In the tried and true tradition of building esoteric things to deal with
brainfuck, nelhage nerdsniped me into writing a brainfuck compiler
in rust.

I’m gunna gloss over tons of stuff, and only write about the interesting parts.
I’m also not going to actually walk through the code as I wrote it because it
was and is incredibly sketchy, but it’s better seperated now so we’re going to
look at master (at time of writing).

Writing a parser

Rust makes this comically simple.

We define a recursive enum containing all the opcodes, and a Vec to hold the
contents of any loops:


#[deriving(Show)] pub enum OpCode {
Lshift,
Rshift,
Putc,
Getc,
Inc,
Dec,
Loop(Vec<OpCode>),
}

By deriving Show we’ll be able to use format!, println! and friends at
various points of debugging and get something plausible ish.

Our actual parser is just about as simple. I won’t inline the
code, but it should read pretty simply.

We create a mutable vec to hold our actual program, and a loop_stack to hold
the contents of each nested loop, inserting those loops into it’s parent each
time we enounter a closing bracket. This also makes it simple to test for
unbalanced brackets.

This then lets us return the Program. Using Option was left as an escape
hatch (unbalanced braces could be caught, as could the file errors, but for now
failing is reasonable since it’s not exactly “production ready”).

Next time, the evaluator.

Posted in Articles | Tagged , , | Leave a comment

Programming is literally the hardest thing

A conversation on the twitter really got me thinking, both about
programming as a profession, the human obsession with winning at everything (Up
to and including failing at life), and much though it kills me to say this,
privilege.

The blog post that started it all is interesting- it begins by
saying that the authors friends who have a more physical job than he does, and
he goes on to explain why programming is so awful.

It’s not that I don’t see the point of his post. Programming really is awful.
These past few weeks have been among the most harrowing and demoralising that I
can remember, both in terms of invested effort and in losing faith in the tools
that we trust day to day. I enjoy ranting about what a hilarious shitshow
software is well above the mean for people who do what I do.

That said though- it’s worth maintaining some modicum of perspective. Ignoring
everything else, programming is innately safe. No job is without its risks,
but comparing the outcome of RSI to the average workplace injury on a
construction site I would wager skews way in favour of the construction site
for both frequency and severity. (As an aside, where the hell would I find
stats for this?).

Michael raised a valid point about mental health, but I totally fail to see how
this relates to computer science. If you’re in a workplace that’s causing you
mental or emotional harm you should seek help, immediately. My intuition
suggests that you stand much better chances of actually receiving it if you’re
in the typically privileged shoes of your average software engineer.

My intent with this post isn’t to victim blame or to tell anyone miserable in
their job (or just needing a space online to vent) to ‘harden up’. I do really
stop to wonder though, when I read about the number of studies showing that
using a male name improves chances of success in the
workplace and academia whether taking to a privileged soapbox to complain how
hard it is after you get there without a handicap might be missing the bigger
picture.

Posted in psych0tik News | 3 Comments

Heartbleed, psych0tik edition

As per the rest of the internet, our collective jaws fell through the floor when Heartbleed was disclosed.

First and foremost- our IRC server was not vulnerable. At present, the only live IRC node is magikarp, owned and run my carbon, and running gnutls. OpenSSL based *clients* on the other hand, could plausibly have been compromised.

With that said, the first order of business once patching was in order was to develop a working exploit. Given that the internet has had a few days to patch, and that numerous other exploits are now public anyway, I’m releasing mine.

Given the time sensitive nature of the situation (We needed a working PoC to quickly enumerate internal services at $dayjob), I took the quick and dirty approach, rather than developing from scratch I produced a patched libssl that when linked against turns any SSL client into a working heartbleed vector.

You can find it at https://github.com/richo/openssl. For obvious reasons I would recommend building without shared object support, and not installing this anywhere.

Posted in psych0tik News | Leave a comment