A lesson on data structures, networking protocols, data sanitization and disclosure
Around 2 years ago, I was enthusiastically working on Spigot and Bukkit along with a couple of fairly popular plugins. During my poking around within the networking internals of Minecraft, I came across a fairly substantial problem that allowed anyone to send certain malformed packets and crash a server by running it out of memory.
Following the defacto standard procedure, I responsibly and privately disclosed the problem to Mojang on 10th July, 2013. That’s nearly 2 years ago. I asked for updates in one month intervals over the course of 3 months and was ignored or given highly unsatisfactory responses. I kept my hopes up that the problem would be patched and checked the source code on new releases whenever I could.
The version of the game when the vulnerability was reported was 1.6.2, the game is now on version 1.8.3. That’s right, 2 major versions and dozens of minor versions and a critical vulnerability that allows you to crash any server, and starve the actual machines of CPU and memory was allowed to exist.
The technical details
The minecraft protocol enables the exchange of information about what an inventory slot contains. This allows you to, for example, get the items in your hotbar when you log in. Items in minecraft also contain a feature that allows them to store arbitary metadata (used for enhancements, books etc). This metadata is stored in a format known as NBT. The NBT format is essentially like JSON but in binary form. This allows it to store complex data structures with nesting and the like.
For example, the NBT metadata (known as a tag) for a book might look something like this
The vulnerability stems from the fact that the client is allowed to send the server information about certain slots. This, coupled with the NBT format’s nesting allows us to craft a packet that is incredibly complex for the server to deserialize but trivial for us to generate.
In my case, I chose to create lists within lists, down to five levels. This is a json representation of what it looks like.
The root of the object,
rekt, contains 300 lists. Each list has a list with 10 sublists, and each of
those sublists has 10 of their own, up until 5 levels of recursion. That’s a total of
10^5 * 300 = 30,000,000 lists.
And this isn’t even the theoretical maximum for this attack. Just the nbt data for this payload is 26.6 megabytes. But luckily
minecraft implements a way to compress large packets, lucky us! zlib shrinks down our
evil data to a mere 39 kilobytes.
Note: in previous versions of minecraft, there was no protocol wide compression for big packets.
Previously, NBT was sent compressed with gzip and prefixed with a signed short of its length, which reduced our maximum payload
2^15 - 1. Now that the length is a varint capable of storing integers up to
2^28, our potential for attack has increased significantly. (for the more technically minded people, we fill the list up with TAG_ENDs, which aren’t accounted for by the accumulator Mojang attempted to implement to fix this)
When the server will decompress our data, it’ll have 27 megs in a buffer somewhere in memory, but that isn’t the bit that’ll kill it. When it attempts to parse it into NBT, it’ll create java representations of the objects meaning suddenly, the sever is having to create several million java objects including ArrayLists. This runs the server out of memory and causes tremendous cpu load.
This vulnerability exists on almost all previous and current minecraft versions as of 1.8.3, the packets used as attack vectors are the 0x08: Block Placement Packet and 0x10: Creative Inventory Action.
The fix for this vulnerability isn’t exactly that hard, the client should never really send a data structure as complex as NBT of arbitrary size and if it must, some form of recursion and size limits should be implemented. These were the fixes that I recommended to Mojang 2 years ago.
Proof of concept
A proof of concept of this exploit can be seen here on my Github repo.
The code to generate the posioned nbt can be seen in
start.py. The code has been tested under python 2.7, once
you have connected to a server simply enter
exploit in the command line and the packet will be sent to the server.
I thought a lot before writing this post, on the one hand I don’t want to expose thousands of servers to a major vulnerability, yet on the other hand Mojang has failed to act upon it. Mojang is no longer a small indie company making a little indie game, their software is used by thousands of servers, hundreds of thousands of people play on servers running their software at any given time. They have a responsibility to fix and properly work out problems like this. In addition, it should be noted that giving condescending responses to white hats who are responsibly disclosing vulnerabilities and trying to improve a product they enjoy is a sure fire way to get developers dis-interested the next time they come across a bug like this.
28th July, 2013: First contact with mojang employee about the issue, vulnerability disclosed and proof of concept provided.
19th August, 2013: Second time asking about fix, response given that its being worked on.
24th September, 2013: Third contact with employee, told that the problem has been delegated.
25th October, 2013: Fourth time I attempted to contact employee, ignored.
27th October, 2013: Another attempt to contact, ignored again. (at this point, I patiently waited for a fix)
In retrospect, yes, I should have given them a final warning sometime recently before this but I just expected to be shot down again
Update: With the release of this full disclosure I have actually made contact with mojang and they are working to fix the issue. Apparently the initial fix they tried failed which indicates a lack of proper testing.
Update 2: The exact problem that caused this bug to go unpatched has been identified. Mojang attempted to implement a fix for this problem, however they did not test their fix against the proof of concept I provided, which still crashed the server perfectly fine. This, in combination with ignoring me when I asked for status updates twice led me to believe that Mojang had attempted no fix. In retrospect, a final warning before this full disclosure more recently was propbably in order. A combination of mis-communication and lack of testing led to this situation today, hopefully it can be a good learning experience.
Update 3: This problem has been patched as of minecraft version 1.8.4
I’m happy to see that multiple other security issues have also been fixed. Once again, I feel better communication would have easily alleviated this problem. Keeping me in the loop and not ignoring me, in addition to proper testing would have easily led to this exploit being fixed long ago.
Edit: In regards to the statements that I never filed a report on their bug tracker, I’d just like to point out that nobody pointed me towards the tracker when I reported the issue, I wasn’t asked to make a ticket or told that I could follow the progress of the report on x-ticket.