Difference between revisions of "Toolchain"

From Forge Community Wiki
m (changed ref to right position)
(major update on deobfucsation information (WIP))
Line 1: Line 1:
−
This page will explain <code>decompile</code>-><code>deobfuscate</code>-><code>reobfuscate</code> process that forge uses when you install the <code>MDK</code>.
+
{{DISPLAYTITLE:Deobfuscation Process}}
  
−
== Why we need to do this ==
+
The toolchain of Forge automatically deobfuscates the base game code, from their obfuscated original names to deobfuscated and human-readable names. This process is done to allow mod developers to write their code based on readable and understandable class/method/field names, which greatly simplifies the modding process.
−
Minecraft is a commercial game, which means the source code is locked away, and not only is the source code unavailable to us, but they obfuscate the game code.
 
  
−
If you look into the client/server JAR, you'll see that all the class names have short, nonsense names like <code>aa</code>, <code>ab</code>, <code>xy</code>, etc. This is called Obfuscating.
+
== Obfuscation ==
−
The problem is, we don't<ref name="license">Mojang recently released this information in what we call <code>mojmappings</code>, but the licensing is a bit iffy [https://cpw.github.io/MinecraftMappingData For more info]</ref> have the information to convert those nonsensical names to readable, understandable names.
+
'''Obfuscation''' is the process of renaming all of the fields, methods, and classes of the source code into unreadable, machine-generated names (such as <code>aaa</code>, <code>bC</code>), and removing package structures. This is commonly used by companies to prevent external entities from decompiling their released binaries/executables and retrieving their source code/intellectual property.
−
So we need a toolchain to make it possible for us to read the source code so we can mod the game.
 
  
−
For more information about [https://en.wikipedia.org/wiki/Obfuscation_(software) Obfiscating].
+
Minecraft is a commercial game, which means the source code is unavailable to all except the developers of Mojang. To prevent piracy and the copying of their intellectual property, Mojang applies this obfuscation process to the game before they release it. They use a tool called [https://www.guardsquare.com/en/products/proguard ProGuard], which is both an optimizer<ref>An optimizer is a program that removes redundant/unused instructions and compacts the code to be faster and smaller.</ref> and an obfuscator.
  
 +
== Problematic Naming ==
 +
There are a few big problems with using these obfuscated names directly for modding.
 +
 +
# It is incredibly difficult to create mods using these obfuscated names. It requires immense patience to reverse-engineer the meanings behind each and every name, and to keep relating those names to what was already reverse-engineered.
 +
# Because the obfuscation process takes place after compilation (the obfuscator operates on the compiled classes), the obfuscated names are not handled by the compiler. Thus, obfuscated classes may contain member<ref>'''member''' refers to class fields and methods.</ref> names that are invalid in the Java source language, but valid in compiled bytecode; this means that the decompiled source of the game may not be recompilable.
 +
# These obfuscated names are automatically generated by the obfuscator for each independent release. This means that the obfuscated names may change signficantly between any two versions, making it harder for mod developers to update mods between releases.
  
 
== Deobfuscation ==
 
== Deobfuscation ==
−
First the obfuscated names need unique names. Since there could be some methods with the same name, the resulting code then would not compile.
+
The solution to these problems is a process known as '''deobfuscation''', where these obfuscated names are transformed into more readable names.
 +
 
 +
Normally, this process is done with the help of a '''deobfusation map''', a file generated by the obfuscator that contains a map of the obfuscated names to the original, non-obfuscated names in the source code. This is commonly used on debofuscating stack traces outputted by an obfuscated program, for debugging purposes.<ref name="retrace"/> However, we don't have this deobfuscation map file, as this is only accessible to the developers of Minecraft<ref name="mojmappings"/>.
 +
 
 +
To rectify this problem, Forge has it's own process to create deobfuscation mappings for the game, using community-sourced human-readable names. This process is split into two separate parts: the '''SRG renaming''', and the '''MCP mapping'''.
 +
 
 +
=== SRG Renaming ===
 +
First, the obfuscated names are renamed into '''SRG names'''.<ref>'''SRG''' stands for '''S'''ea'''RG'''e, co-author of the Mod Creator Pack, who created this process.</ref> Each obfuscated class, method, and field is assigned a unique number by the deobfuscator, through a sequential counter. This unique number is called the '''SRG ID''' of that class/method/field (henceforth called member).
 +
 
 +
The SRG name of the member is then derived from its SRG ID, its type (class, function, field, paramter), and (optionally) the obfuscated name of the object at the time it was given its SRG name<ref>The obfuscated name included with the SRG name should not be taken as the object's real obfuscated name, as it is only true for the version for which the object was first deobfuscated.</ref>. This inclusion of the SRG ID into the name guarantees that the SRG name for all members are unique.
 +
 
 +
* For classes -> <code>c_###_</code><ref name="classes_srg">Due to how the deobfuscation process is implemented in Forge, all classes are given their MCP names before being released to the public. Mod developers will never see the SRG names of classes outside of a deobfuscation workspace.</ref>
 +
* For functions/methods -> <code>func_###_</code>
 +
* For fields -> <code>field_###_</code>
 +
* For function/method parameters -> <code>p_###_#_</code> for normal methods, <code>p_i###_#</code> for constructors; the second number is the index<ref>The index of a parameter is determined by the preceeding parameters, where <code>double</code> and <code>long</code> increments it by 2, and all other primitives and reference types increments it by 1. </ref> of the parameter.
 +
 
 +
For example, <code>func_71410_x</code> refers to a function with SRG ID 71410 and original obfuscated name of <code>x</code>.<ref>For version 1.16.2, <code>func_71410_x</code> refers to <code>Minecraft.getInstance</code>, with real obfuscated name of <code>B</code>.</ref>
  
−
For this all of these obfuscated classes/methods/fields are assigned a number. This number increments sequentially and is called the <code>SRG ID</code>((SRG stands for Searge, which was one of the co-founders of the original set of scripts and programs which did this in the beginning)) of that class/method/field, this ensures all these classes/methods/fields (henceforth called members) have a unique name.\\
+
== MCP Mapping ==
−
These IDs are converted to <code>SRG</code> names, also called <code>Notch names</code>, simply by combining their ID and the type of member\\
+
WIP
−
<code>class</code> -> <code>c_XXX_</code> (you won't ever see this because of reasons, explained later)\\
 
−
<code>function</code> -> <code>func_XXX_</code>\\
 
−
<code>field</code> -> <code>field_XXX_</code>\\
 
−
<code>parameter</code> -> <code>p_XXX_X_</code> (there are two types of parameter names, also explained later)\\
 
−
These Ids will be gernerated automatically.
 
  
 
== Decompilation ==
 
== Decompilation ==
Line 53: Line 68:
 
TBD
 
TBD
  
−
<references/>
+
== See also ==
 +
* [https://en.wikipedia.org/wiki/Obfuscation_(software) Obfuscation]
 +
 
 +
<references>
 +
<ref name="retrace">For ProGuard users (such as Mojang), this is done using [https://www.guardsquare.com/en/products/proguard/manual/retrace ReTrace].</ref>
 +
<ref name="mojmappings">Mojang recently released their deobfuscation mappings for Minecraft (colloquially named <code>mojmappings</code>), but the licensing for its uses is a bit ambiguous. [https://cpw.github.io/MinecraftMappingData See this post by cpw] for more information.</ref>
 +
</references>

Revision as of 06:16, 28 October 2020


The toolchain of Forge automatically deobfuscates the base game code, from their obfuscated original names to deobfuscated and human-readable names. This process is done to allow mod developers to write their code based on readable and understandable class/method/field names, which greatly simplifies the modding process.

Obfuscation

Obfuscation is the process of renaming all of the fields, methods, and classes of the source code into unreadable, machine-generated names (such as aaa, bC), and removing package structures. This is commonly used by companies to prevent external entities from decompiling their released binaries/executables and retrieving their source code/intellectual property.

Minecraft is a commercial game, which means the source code is unavailable to all except the developers of Mojang. To prevent piracy and the copying of their intellectual property, Mojang applies this obfuscation process to the game before they release it. They use a tool called ProGuard, which is both an optimizer[1] and an obfuscator.

Problematic Naming

There are a few big problems with using these obfuscated names directly for modding.

  1. It is incredibly difficult to create mods using these obfuscated names. It requires immense patience to reverse-engineer the meanings behind each and every name, and to keep relating those names to what was already reverse-engineered.
  2. Because the obfuscation process takes place after compilation (the obfuscator operates on the compiled classes), the obfuscated names are not handled by the compiler. Thus, obfuscated classes may contain member[2] names that are invalid in the Java source language, but valid in compiled bytecode; this means that the decompiled source of the game may not be recompilable.
  3. These obfuscated names are automatically generated by the obfuscator for each independent release. This means that the obfuscated names may change signficantly between any two versions, making it harder for mod developers to update mods between releases.

Deobfuscation

The solution to these problems is a process known as deobfuscation, where these obfuscated names are transformed into more readable names.

Normally, this process is done with the help of a deobfusation map, a file generated by the obfuscator that contains a map of the obfuscated names to the original, non-obfuscated names in the source code. This is commonly used on debofuscating stack traces outputted by an obfuscated program, for debugging purposes.[3] However, we don't have this deobfuscation map file, as this is only accessible to the developers of Minecraft[4].

To rectify this problem, Forge has it's own process to create deobfuscation mappings for the game, using community-sourced human-readable names. This process is split into two separate parts: the SRG renaming, and the MCP mapping.

SRG Renaming

First, the obfuscated names are renamed into SRG names.[5] Each obfuscated class, method, and field is assigned a unique number by the deobfuscator, through a sequential counter. This unique number is called the SRG ID of that class/method/field (henceforth called member).

The SRG name of the member is then derived from its SRG ID, its type (class, function, field, paramter), and (optionally) the obfuscated name of the object at the time it was given its SRG name[6]. This inclusion of the SRG ID into the name guarantees that the SRG name for all members are unique.

  • For classes -> c_###_[7]
  • For functions/methods -> func_###_
  • For fields -> field_###_
  • For function/method parameters -> p_###_#_ for normal methods, p_i###_# for constructors; the second number is the index[8] of the parameter.

For example, func_71410_x refers to a function with SRG ID 71410 and original obfuscated name of x.[9]

MCP Mapping

WIP

Decompilation

More general info

FernFlower/ForgeFlower

Forge uses ForgeFlower, which is a Fork of FernFlower. FernFlower is a decompiler that takes the compiled Minecraft Jar and turn it in to semi readable source code.

Why semi readable?
Because the decompiler has no sense of formatting or indentation.

Mappings

We don't need this step but coding with the SRG names is hard and not really fun. So we need to take the SRG names and make them more user friendly, the user friendly names are community sourced names. Since SRG names are uniquely identifiable, due to the unique ID, it literally does a search-and-replace on all the source code text to do that SRG->MCP(( MCP stands for ModCoderPack (or previously, Minecraft Coder Pack) which was the toolchain that did all of this before the advent of ForgeGradle)) application.

Some additional Infos

  1. You will never see c_XXX_ for classes, because the class names are picked beforehand
    • this is still crowdsourced, but before a new version is ready for Forge, all classes are given an MCP name
    • this MCP name stays constant throughout the game version it was picked for; it can change between versions, but it usually wont for already-named classes (except for misspells, typos, and misnames)
  2. If you look into the JAR, you won't see any packages for the obfuscated classes, but the deobfuscated classes do have the packages, this is because the same process that names the classes, also decides what package they belong to
  3. parameters have special names
    • there are two types of parameter names: p_XXX_X_ and p_iXXX_X_
    • the one with the i means that it's a parameter for a constructor
    • the first set of numbers are the SRG ID of their parent method, and the second number denotes the index of the parameter
    • the index of the parameter is a bit more involved, but this will not be explained here
  4. If you look into the source, you'll see that parameters for lambdas don't have mapped names
    • This is because of a complication in how the lambdas are compiled/decompiled; it's a more advanced topic which involves how the compiler compiles lambdas which we will not explain here.

Reobfuscation

TBD

See also

  1. ↑ An optimizer is a program that removes redundant/unused instructions and compacts the code to be faster and smaller.
  2. ↑ member refers to class fields and methods.
  3. ↑ For ProGuard users (such as Mojang), this is done using ReTrace.
  4. ↑ Mojang recently released their deobfuscation mappings for Minecraft (colloquially named mojmappings), but the licensing for its uses is a bit ambiguous. See this post by cpw for more information.
  5. ↑ SRG stands for SeaRGe, co-author of the Mod Creator Pack, who created this process.
  6. ↑ The obfuscated name included with the SRG name should not be taken as the object's real obfuscated name, as it is only true for the version for which the object was first deobfuscated.
  7. ↑ Due to how the deobfuscation process is implemented in Forge, all classes are given their MCP names before being released to the public. Mod developers will never see the SRG names of classes outside of a deobfuscation workspace.
  8. ↑ The index of a parameter is determined by the preceeding parameters, where double and long increments it by 2, and all other primitives and reference types increments it by 1.
  9. ↑ For version 1.16.2, func_71410_x refers to Minecraft.getInstance, with real obfuscated name of B.
Retrieved from ‘https://forge.gemwire.uk/index.php?title=Toolchain&oldid=2184’