Difference between revisions of "Toolchain"

From Forge Community Wiki
(Forge side)
(Push changes to main page)
 
(4 intermediate revisions by 2 users not shown)
Line 7: Line 7:
  
 
== Overall process ==
 
== Overall process ==
When setting up the environment for the first time, a gradle refresh triggers three things:
+
Whenever a gradle refresh is triggered, a few things occur:
# ForgeGradle downloads the MCPConfig zip for the file you're using, and triggers the SetupMCP task.
+
# ForgeGradle downloads the jar and MCPConfig zip and triggers the extractSrg/createSrgToMcp task.
# After that, it processes the jar - applies access transformers, MCPCleanup and others
+
# After that, it processes the jar - applies access transformers from forge/userdev, SAS from forge, and decompiles.
# Finally, it patches and finalises the code, ready for modder consumption.
+
# Finally, it patches and finalizes the code, ready for modder consumption.
 +
 
 +
This is triggered whenever the <code>setup</code> task is ran.
  
 
== Needed knowledge ==
 
== Needed knowledge ==
 
=== Obfuscation ===
 
=== Obfuscation ===
'''Obfuscation''' is the process of renaming all of the fields, methods, and classes of the compiled code into unreadable, machine-generated names (such as <code>aaa</code>, <code>bC</code>), and removing package structures (this makes everything package-local and makes the obfuscated code smaller somewhat. This is commonly used by companies to prevent external entities from easily decompiling their released binaries/executables and retrieving their source code/intellectual property, though it does have size advantages.
+
'''Obfuscation''' is the process of renaming all of the fields, methods, and classes of the compiled code into unreadable, machine-generated names (such as <code>aaa</code>, <code>bC</code>), and removing package structures (this makes everything package-local and makes the obfuscated code smaller somewhat. This is commonly used by companies to prevent external entities from easily decompiling their released binaries/executables and retrieving their source code/intellectual property.
  
Additionally, due to the way the Local Variable Table (LVT) of Java bytecode is stored, every function-local variable name is turned to ☃ (that's right, a snowman) in the compiled files. This makes immediate recompilation of the game literally and physically impossible, as every Java compiler currently available requires that local variables have unique names.
+
Additionally, due to the way the Local Variable Table (LVT) of Java bytecode is stored, every function-local variable name is turned to <code></code> (that's right, a snowman) in the compiled files. This makes immediate recompilation of the game literally and physically impossible, as every Java compiler currently available requires that local variables have unique names.
  
 
Minecraft is a commercial game, which means the source code is unavailable to all except the developers of Mojang. To prevent piracy and the copying of their intellectual property, Mojang applies this obfuscation process to the game before they release it. They use a tool called [https://www.guardsquare.com/en/products/proguard ProGuard], which is both an optimizer<ref>An optimizer is a program that removes redundant/unused instructions and compacts the code to be faster and smaller.</ref> and an obfuscator.
 
Minecraft is a commercial game, which means the source code is unavailable to all except the developers of Mojang. To prevent piracy and the copying of their intellectual property, Mojang applies this obfuscation process to the game before they release it. They use a tool called [https://www.guardsquare.com/en/products/proguard ProGuard], which is both an optimizer<ref>An optimizer is a program that removes redundant/unused instructions and compacts the code to be faster and smaller.</ref> and an obfuscator.
Line 25: Line 27:
 
# It is incredibly difficult to create mods using these obfuscated names. It requires immense patience to reverse-engineer the meanings behind each and every name, and to keep relating those names to what was already reverse-engineered. Although, tools do exist to make this process easier, such as IntelliJ IDEA plugins that provide naming hints automatically.
 
# It is incredibly difficult to create mods using these obfuscated names. It requires immense patience to reverse-engineer the meanings behind each and every name, and to keep relating those names to what was already reverse-engineered. Although, tools do exist to make this process easier, such as IntelliJ IDEA plugins that provide naming hints automatically.
 
# Because the obfuscation process takes place after compilation (the obfuscator operates on the compiled classes), the obfuscated names are not handled by the compiler. Thus, obfuscated classes may contain member<ref>'''member''' refers to class fields and methods.</ref> names that are invalid in the Java source language, but valid in compiled bytecode (like ☃ discussed earlier); this means that the decompiled source of the game is not immediately recompilable.
 
# Because the obfuscation process takes place after compilation (the obfuscator operates on the compiled classes), the obfuscated names are not handled by the compiler. Thus, obfuscated classes may contain member<ref>'''member''' refers to class fields and methods.</ref> names that are invalid in the Java source language, but valid in compiled bytecode (like ☃ discussed earlier); this means that the decompiled source of the game is not immediately recompilable.
# These obfuscated names are automatically generated by the obfuscator for each independent release. This means that the obfuscated names may change signficantly between any two versions, making it harder for mod developers to update mods between releases.
+
# These obfuscated names are automatically generated by the obfuscator for each independent release. This means that the obfuscated names may change significantly between any two versions, making it harder for mod developers to update mods between releases.
  
 
=== SRGification ===
 
=== SRGification ===
 
Some background on what SRG is and how it works:
 
Some background on what SRG is and how it works:
  
'''SRG''' stands for '''S'''ea'''RG'''e, co-author of the Mod Creator Pack, who created this process.
+
'''SRG''' stands for '''S'''earge's '''R'''etro '''G'''uard; RetroGuard being an early attempt to reverse the ProGuard obfuscation, and Searge being co-author of the Mod Coder Pack or MCP<ref>This was previously called the ''Minecraft Coder Pack''</ref>, who created this process.
 
Each obfuscated class, method, and field is assigned a unique number by the [[Toolchain:MCPConfig|backend]], via a sequential counter. This unique number is called the '''SRG ID''' of that class/method/field (henceforth called member).
 
Each obfuscated class, method, and field is assigned a unique number by the [[Toolchain:MCPConfig|backend]], via a sequential counter. This unique number is called the '''SRG ID''' of that class/method/field (henceforth called member).
  
The SRG name of the member is then derived from its SRG ID, its type (function {given the prefix func_}, field {given the prefix field_}, parameter {given the prefix p_, or p_i if this is the parameter of a constructor}), and (optionally) the obfuscated name of the object at the time it was given its SRG name<ref>The SRG name for a given member is only created once, when it first appears in the code. Therefore, the SRG postfix may be different from the current obf name.</ref>. This inclusion of the SRG ID into the name guarantees that the SRG name for all members are unique, and is the reason the ID is generated.
+
The SRG name of the member is then derived from its SRG ID and its type (method {given the prefix <code>m_</code>}, field {given the prefix <code>f_</code>}, or parameter {given the prefix <code>p_</code>}). <ref>The SRG name for a given member is only created once, when it first appears in the code.</ref>. This inclusion of the SRG ID into the name guarantees that the SRG name for all members are unique, and is the reason the ID is generated.
  
The actual conversion of obf names to SRG names is done by a tool called [[Toolchain:SpecialSource|SpecialSource]]. More information on how it works can be found on that page.
+
The actual conversion of obf names to SRG names is done by a tool called [[Toolchain#Vignette|Vignette]]. More information on how it works can be found on that page.
  
 
== The Setup ==
 
== The Setup ==
 
The process can be broken up into 3 steps; MCPConfig, patch and provide.
 
The process can be broken up into 3 steps; MCPConfig, patch and provide.
 
The MCPConfig step is, understandably, the biggest and most prone to failure.
 
The MCPConfig step is, understandably, the biggest and most prone to failure.
An explanation of MCPConfig itself, how it works, what it's for (but NOT how to use it) can be found [[Toolchain:MCPConfig|here]]. For the purpose of this guide, you need only know that its' goal is to get the game decompiled, and into a state where it can immediately be recompiled. Due to certain flaws in the rest of the toolchain, this means it needs to fix and patch the source code before passing it onto Forge.
+
An explanation of MCPConfig itself, how it works, what it's for (but NOT how to use it) can be found [[Toolchain:MCPConfig|here]]. For the purpose of this guide, you need only know that its' goal is to get the game decompiled, and into a state where it can immediately be recompiled. This means it needs to fix and patch the source code before passing it onto Forge.
  
 
In this way, MCPConfig can be thought of the vanilla side of the setup. It does not modify the game.
 
In this way, MCPConfig can be thought of the vanilla side of the setup. It does not modify the game.
Line 47: Line 49:
  
 
=== Download and parsing MCPConfig ===
 
=== Download and parsing MCPConfig ===
The first thing that ForgeGradle does upon initialising a first-time setup, is starting the [https://github.com/MinecraftForge/ForgeGradle/blob/e2ed49546abced95650635f81be071441ec60995/src/mcp/java/net/minecraftforge/gradle/mcp/task/SetupMCPTask.java#L91 SetupMCP] task.  
+
The first thing that ForgeGradle does upon initializing a first-time setup, is starting the [https://github.com/MinecraftForge/ForgeGradle/blob/a0d2d1f488b3285ef7c75367feeebbe1a38aa19b/src/mcp/java/net/minecraftforge/gradle/mcp/tasks/SetupMCP.java#L74 SetupMCP] task.
  
This task then seeks to [https://github.com/MinecraftForge/ForgeGradle/blob/e2ed49546abced95650635f81be071441ec60995/src/mcp/java/net/minecraftforge/gradle/mcp/task/DownloadMCPConfigTask.java#L78 download the MCPConfig.zip jar] for the version you're setting up.
+
This task then seeks to [https://github.com/MinecraftForge/ForgeGradle/blob/a0d2d1f488b3285ef7c75367feeebbe1a38aa19b/src/mcp/java/net/minecraftforge/gradle/mcp/tasks/DownloadMCPConfig.java#L68 download the MCPConfig.zip jar] for the version you're setting up
 
Once it is acquired, it  
 
Once it is acquired, it  
[https://github.com/MinecraftForge/ForgeGradle/blob/e2ed49546abced95650635f81be071441ec60995/src/mcp/java/net/minecraftforge/gradle/mcp/util/MCPRuntime.java#L74 parses the steps contained within the config.json]. It does this by interpreting the file with the following rules:
+
[https://github.com/MinecraftForge/ForgeGradle/blob/a0d2d1f488b3285ef7c75367feeebbe1a38aa19b/src/mcp/java/net/minecraftforge/gradle/mcp/util/MCPRuntime.java#L75 parses the steps contained within the config.json]. It does this by interpreting the file with the following rules:
 
* Every key, except for libraries, is interpreted as the name of a step.
 
* Every key, except for libraries, is interpreted as the name of a step.
 
* Steps are executed in order.
 
* Steps are executed in order.
Line 60: Line 62:
 
** java -jar &lt;version&gt; &lt;args&gt; &lt;jvmArgs&gt;
 
** java -jar &lt;version&gt; &lt;args&gt; &lt;jvmArgs&gt;
  
An example config.json can be found [https://github.com/MinecraftForge/MCPConfig/blob/master/versions/release/1.16.4/config.json here].
+
{{Tip|content=A config.json can be found [https://github.com/MinecraftForge/MCPConfig/blob/master/versions/release/1.17.1/config.json here] which defines the following steps:
 
 
It defines the steps:
 
* [[Toolchain:MCinjector|mcinjector]]
 
** version: de.oceanlabs.mcp:mcinjector:3.8.0:fatjar
 
** args: --in {input} --out {output} --log {log} --level=INFO --lvt=LVT --exc {exceptions} --acc {access} --ctr {constructors}
 
 
* [[Toolchain:ForgeFlower|fernflower]]
 
* [[Toolchain:ForgeFlower|fernflower]]
** version: net.minecraftforge:forgeflower:1.5.478.16
+
** version: net.minecraftforge:forgeflower:1.5.498.12
 
** args: -din=1 -rbr=1 -dgs=1 -asc=1 -rsy=1 -iec=1 -jvn=1 -isl=0 -iib=1 -log=TRACE -cfg {libraries} {input} {output}
 
** args: -din=1 -rbr=1 -dgs=1 -asc=1 -rsy=1 -iec=1 -jvn=1 -isl=0 -iib=1 -log=TRACE -cfg {libraries} {input} {output}
 
** jvmargs: -Xmx4G
 
** jvmargs: -Xmx4G
 
* [[Toolchain:Mergetool|merge]]
 
* [[Toolchain:Mergetool|merge]]
** version: net.minecraftforge:mergetool:1.1.1:fatjar
+
** version: net.minecraftforge:mergetool:1.1.3:fatjar
** args: --client {client} --server {server} --ann {version} --output {output} --inject false"
+
** args: --client {client} --server {server} --ann {version} --output {output} --inject false
 
* [[Toolchain:MCInjector|rename]]
 
* [[Toolchain:MCInjector|rename]]
** version: net.md-5:SpecialSource:1.8.3:shaded
+
** version: net.minecraftforge.lex:vignette:0.2.0.10
** args: --in-jar {input} --out-jar {output} --srg-in {mappings}
+
** args: --jar-in {input} --jar-out {output} --mapping-format tsrg2 --mappings {mappings} --fernflower-meta --cfg {libraries} --create-inits --fix-param-annotations
** repo: https://repo1.maven.org/maven2/
 
  
More information about each of these tools can be found at the link provided, as well as what each of these arguments do. A brief description is provided.
+
More information about each of these tools can be found at the link provided, as well as what each of these arguments do. A brief description is provided.}}
  
=== MCInjector ===
+
=== ForgeFlower ===  
[https://github.com/ModCoderPack/MCInjector MCInjector] is the tool we use to apply various fixes to the code, while it is still in bytecode form. That meaning, it works on compiled code, not sourcecode. It does this because it's easier to rename LVT entries (from the snowman) to readable names while you can search for every other code path that references that specific entry; ergo renaming all accesses at once. This is impossible in sourcecode, where every name is identical and string matching is impossible.
+
The decompiler used by ForgeGradle is a custom fork of [https://github.com/JetBrains/intellij-community/tree/master/plugins/java-decompiler/engine Jetbrains' FernFlower], called [https://github.com/MinecraftForge/ForgeFlower ForgeFlower] which searches the jar for files, cleans up the bytecode, and then converts it into a reasonable best-guess interpretation.
  
It:
+
It also produces the following side effects:
 
* Removes synthetic parameters from constructors
 
* Removes synthetic parameters from constructors
 
** In bytecode, inner classes have the outer class as their first constructor parameter, but Java source code does not.
 
** In bytecode, inner classes have the outer class as their first constructor parameter, but Java source code does not.
* Handles adding annotations for parameters that have synthetic data
 
** In bytecode, these Nonnull (or whatever) annotations are attached to the parameters, not to the function that contains them.
 
 
* Adds constructors for inner classes
 
* Adds constructors for inner classes
 
** These are removed by Proguard sometimes, as they are not required in the bytecode if the parent has a default constructor.
 
** These are removed by Proguard sometimes, as they are not required in the bytecode if the parent has a default constructor.
* Adds synthetic (invisible) constructors for classes without them
+
* Hides bridge methods
 
+
* Decompiles generic signatures
 
+
* Encodes non-ASCII characters in string and character literals as Unicode escaped characters
It also applies fixes for things like EXC;
+
* Hides synthetic class members
* Renaming parameters inside constructors, such that subclasses retain the ID of their parent.
+
* Prevents simple lambdas from being inlined
* Fixing access for select functions, where it is required for the proper recompilation.
 
* Applying proper typing to constructor parameters
 
* Applying proper external (LWJGL) exception data to functions and classes.
 
 
 
Note that it does NOT rename to SRG. LVT (Local Variables) are renamed to lvt_&lt;index&gt;_&lt;version&gt;<ref>Index means: the place where this item was found. If it is the 4th entry in the table, it is index 3.</ref>
 
 
 
=== ForgeFlower ===
 
After the code has been cleaned up by MCInjector, to a state where it no longer conflicts with itself, it can be passed to the decompiler.
 
 
 
The decompiler used by ForgeGradle is a custom fork of [https://github.com/JetBrains/intellij-community/tree/master/plugins/java-decompiler/engine Jetbrains' FernFlower], called [https://github.com/MinecraftForge/ForgeFlower ForgeFlower].
 
 
 
It simply searches the jar for files, converts the bytecode into a reasonable best-guess interpretation.
 
  
 
As you can see by the repository, a lot of work has gone into tuning it for Minecraft's needs, but it is still a far way from perfect. This is why the patches are needed.
 
As you can see by the repository, a lot of work has gone into tuning it for Minecraft's needs, but it is still a far way from perfect. This is why the patches are needed.
  
If you [https://github.com/MinecraftForge/MCPConfig/blob/master/versions/release/1.16.4/patches/ look at the patches] for 1.16.4, they are mostly incredibly simple changes. Adding generics, making types more strict.  
+
If you [https://github.com/MinecraftForge/MCPConfig/tree/master/versions/release/1.17.1/patches look at the patches] for 1.17.1, they are mostly incredibly simple changes. Adding generics, making types more strict.  
  
 
This is all stuff that should be done by the decompiler, and PRs are always welcome at the ForgeFlower repository for changes and fixes that would reduce the amount of MCPConfig patches required to get the game to compile.
 
This is all stuff that should be done by the decompiler, and PRs are always welcome at the ForgeFlower repository for changes and fixes that would reduce the amount of MCPConfig patches required to get the game to compile.
Line 117: Line 99:
  
 
=== Mergetool ===
 
=== Mergetool ===
The game is split into two distributions; server and client.
+
The game is split into two distributions; server and client. Since the server is just a subset of the client, the client contains the server-only classes as well.
 
 
Because the server contains no rendering code, and the client contains none of the server-specific code (like the UI), this means there are differences between what can run on one side or the other.
 
  
To get around this, we have a tool called Mergetool, which can search for the differences between two files (down to the function level) and merge them into one large (referred to as joined) jar file.
+
To get around this, we have a tool called Mergetool, which can search for the differences between two files (down to the function level) and merge them into one large (referred to as joined) jar file. It also can also annotate those files as necessary.
  
 
It is a simple program, but it works.
 
It is a simple program, but it works.
  
=== SpecialSource ===
+
=== Vignette ===
SpecialSource is where SRG starts to come into play. It serves the role of our deobfuscator, performing deobfuscation.
+
Vignette is where SRG starts to come into play. It serves the role of our deobfuscator, performing deobfuscation.
  
This process is done with the help of a '''deobfusation map''', a file generated by the original obfuscator (in this case, ProGuard) that contains a map of the obfuscated names to original, non-obfuscated names. This is commonly used on debofuscating stack traces outputted by an obfuscated program, for debugging purposes.<ref name="retrace"/>
+
This process is done with the help of a '''deobfusation map''', a file generated by the original obfuscator (in this case, ProGuard) that contains a map of the obfuscated names to original, non-obfuscated names. This is commonly used on deobfuscating stack traces outputted by an obfuscated program, for debugging purposes.<ref name="retrace"/>
  
We have three sets of deobfuscation maps available to us; the obf->SRG mappings distributed with the MCPConfig system, the Yarn intermediary system, or the offical mappings.<ref name="mojmappings"/>.
+
We have three sets of deobfuscation maps available to us; the obf->SRG mappings distributed with the MCPConfig system, the Yarn intermediary system, or the official mappings.<ref name="mojmappings"/>.
  
To rectify this problem, Forge has it's own process to create deobfuscation mappings for the game, using community-sourced human-readable names. This process is split into two separate parts: the '''SRG renaming''', and the '''MCP mapping'''. During the SetupMCP task, only the SRG renaming is performed.
+
To rectify this problem, Forge has it's own process to create deobfuscation mappings for the game, using the official mappings. This process is split into two separate parts: the '''SRG renaming''', and the '''official mapping'''. During the SetupMCP task, only the SRG renaming is performed.
  
SpecialSource itself operates on the source jar, as can be gathered by the name, and takes in a .tsrg file, like that [https://github.com/MinecraftForge/MCPConfig/blob/master/versions/release/1.16.4/joined.tsrg contained in the MCPConfig zip].
+
Vignette itself operates on the compiled jar and takes in a .tsrg file, like that [https://github.com/MinecraftForge/MCPConfig/blob/master/versions/release/1.17.1/joined.tsrg contained in the MCPConfig zip]. It does this because it's easier to rename LVT entries (from the snowman) to readable names while you can search for every other code path that references that specific entry; ergo renaming all accesses at once. This is impossible in source code, where every name is identical and string matching is impossible.
  
It first renames classes, straight into MCP names.
+
It also:
Then, iterating the members of the class, it renames fields, methods, parameters and inner classes.
+
* Handles adding annotations for parameters that have synthetic data
 +
** In bytecode, these Nonnull (or whatever) annotations are attached to the parameters, not to the function that contains them.
 +
* Adds synthetic (invisible) constructors for classes without them
 +
 
 +
LVT (Local Variables) are initially renamed to lvt_&lt;index&gt;_&lt;version&gt;<ref>Index means: the place where this item was found. If it is the 4th entry in the table, it is index 3.</ref>
 +
 
 +
First, the class names are renamed from their obfuscated target to the specified mapping output.
 +
Then, iterating the members of the class, it renames fields, methods, parameters, and inner classes.
  
 
A recap from earlier:
 
A recap from earlier:
* For classes -> <code>c_###_</code> (but it is immediately changed without the c_ being written to disk)
+
* For classes -> <code>C_###_</code> (but it is transparently remapped without the C_ being written to disk)
* For functions/methods -> <code>func_&lt;ID&gt;_&lt;obf-name&gt;</code>
+
* For functions/methods -> <code>m_&lt;ID&gt;_</code>
* For fields -> <code>field_&lt;ID&gt;_&lt;obf-name&gt;</code>
+
* For fields -> <code>f_&lt;ID&gt;_</code>
* For function/method parameters -> <code>p_###_#_</code> for normal methods, <code>p_i###_#</code> for constructors; the second number is the index<ref>The index is the position of the argument. 0 on the left, 1 after that, 2 after that. Note that double and long arguments increase their index by two, rather than one.</ref> of the parameter.
+
* For function/method parameters -> <code>p_###_</code><ref>The index is the position of the argument. 0 on the left, 1 after that, 2 after that. Note that double and long arguments increase their index by two, rather than one of the parameter</ref>.
  
For example, <code>func_71410_x</code> refers to a function with SRG ID 71410 and original obfuscated name of <code>x</code>.<ref>For version 1.16.2, <code>func_71410_x</code> refers to <code>Minecraft.getInstance()</code>, with real obfuscated name of <code>B</code>, but when the getInstance() function was first discovered in code, it was called x.</ref>
+
For example, <code>m_91087_</code> refers to a function with SRG ID 91087.<ref>For version 1.17.1, <code>m_91087_</code> refers to <code>Minecraft#getInstance</code> with an obfuscated name of <code>dvp$C</code>.</ref>
  
At this point, combined with the MCInjector step from earlier, we have a completely SRGed-up source jar.
+
At this point, we have a completely remapped jar.
  
 
=== Patches ===
 
=== Patches ===
 
Once we have the source code ready to go, the final step in the setup is to apply patches.
 
Once we have the source code ready to go, the final step in the setup is to apply patches.
These are done trivially, using diffs and [https://github.com/CadixDev/gitpatcher gitpatcher].
+
These are done trivially using [https://github.com/MinecraftForge/DiffPatch DiffPatch].
 
 
=== Clean-up ===
 
Because of the way Proguard (or whatever obfuscator) and ForgeFlower mangle the source code, we need to take some steps to clean it up before we can proceed.
 
 
 
Assorted cleanup fixes are performed by the [https://github.com/MinecraftForge/MCPCleanup/ MCPCleanup] utility. These include:
 
* removing trailing whitespace at the end of lines
 
* removing extra newlines at the start and end of files
 
* removing extra newlines between every line of code (every set of concurrent newlines is replaced with a single)
 
* removing comments (// hello, /* hello */)
 
** the purpose of this step is unclear - it seems to be a preventative measure to ensure that Java changes do not interfere with the patch alignment in the future. Comments from the decompiler are still sometimes present in the source code.
 
* removing imports from the package a class is in
 
* removing comments that include the phrase <code>GL_[^*]+</code>
 
* replacing [[Toolchain:Magic Constants|magic constants]] with their code substitutions
 
* replacing <code>Character.valueOf(&lt;character&gt;)</code> with <code>&lt;character&gt;</code>
 
* replacing OpenGL integer constants with their code representation
 
* converting unicode character constants back into integer representation
 
* formatting the code with JAStyle
 
 
 
It also adjusts abstract functions in some way - after running the program on a jar, the parameters of a default abstract function get renamed, but it is unclear exactly what part of the source code does this.
 
  
 
== The Forge Side ==
 
== The Forge Side ==
  
After the MCPConfig/SetupMCP tasks are finished, ForgeGradle will print '''MCP Environment Setup is complete''' and wait for the user input.
+
When Forge is first loaded by gradle, it prepares the files for setting up the workspace.
  
The next step is to run the <code>gradlew setup</code> task, which does what it implies: Applying the Forge system to the processed vanilla code.
+
The initial setup is done through the <code>gradlew setup</code> task. It will run the SetupMCP tasks and then start applying the Forge system to the processed vanilla code. After the MCPConfig/SetupMCP tasks are finished, ForgeGradle will print '''MCP Environment Setup is complete''' and continue. The text '''pausing after requested step''' is also printed twice during this task such that some necessary data can be extracted.
  
First, it applies ATs. After that, patches. Finally, MCP/Crowdsourced mappings are applied.
+
First, it applies ATs. After that, patches. Finally, the official mappings, or some other mappings from a provider like Parchment, are applied.
  
 
All in all, compared to the MCPConfig setup, this is a string of extremely basic tasks - mostly just one-line commands.
 
All in all, compared to the MCPConfig setup, this is a string of extremely basic tasks - mostly just one-line commands.
Line 186: Line 154:
 
[[Access Transformers]] are a way of changing the visibility and finality of classes and class members. A full explanation of how it works, what the specification is, and what exactly they're used for, can be found at that page.
 
[[Access Transformers]] are a way of changing the visibility and finality of classes and class members. A full explanation of how it works, what the specification is, and what exactly they're used for, can be found at that page.
  
They are applied by passing the AT Config (nowadays called accesstransformer.cfg) into SpecialSource:
+
They are applied by passing the AT Config (nowadays called accesstransformer.cfg) into AccessTransformers:
* <code>SpecialSource.jar --in-jar {input} --out-jar {output} --access-transformer {at.cfg}</code>
+
* <code>AccessTransformers.jar --inJar {input} --outJar {output} --logFile accesstransform.log --atFile {at.cfg}</code>
  
[https://github.com/MinecraftForge/MinecraftForge/blob/c3e84646db70f518dd0b37a8fcfc42cb814d7ba8/src/main/resources/META-INF/accesstransformer.cfg The current AT cfg can be found here.]
+
[https://github.com/MinecraftForge/MinecraftForge/blob/1.17.x/src/main/resources/META-INF/accesstransformer.cfg The AT cfg for Forge can be found here.]
  
 
=== Applying Patches ===
 
=== Applying Patches ===
Line 195: Line 163:
 
The patches used for Forge itself are different from those used by MCPConfig, which means there are two separate patching stages performed.
 
The patches used for Forge itself are different from those used by MCPConfig, which means there are two separate patching stages performed.
  
As opposed to the minimal MCPConfig patching, with the goal to make the code recompileable, the Forge patching is done to apply the API and modloader to the code.
+
As opposed to the minimal MCPConfig patching, with the goal to make the code recompilable, the Forge patching is done to apply the API and mod loader to the code.
  
It does this in a very similar way, with the [https://github.com/CadixDev/gitpatcher gitpatcher] utility.
+
It does this in a very similar way, with the [https://github.com/MinecraftForge/BinaryPatcher BinaryPatcher] utility used at install time/during a CI build or [https://github.com/MinecraftForge/DiffPatch DiffPatch] in all other situations.
  
  
 
=== Applying Mappings ===
 
=== Applying Mappings ===
This step isn't strictly necessary, and it can be ommitted. However, working with exclusively SRG names is confusing for most people, so we have an extra step to apply human names to the SRG.
+
This step isn't strictly necessary, and it can be omitted. However, working with exclusively SRG names is confusing for most people, so we have an extra step to apply human names to the SRG.
  
These renames can come from any source; as mentioned earlier, the MCP/Crowdsourced naming, the Yarn naming, or the Mojang Obf-map naming. ForgeGradle does not care, as long as there is a valid SRG->names map.
+
These renames can come from any source; as mentioned earlier, the Yarn naming or the Mojang Obf-map naming. ForgeGradle does not care, as long as there is a valid SRG->names map.
  
How ForgeGradle retrieves these mappings is covered in the [[ForgeGradle/mappings appropriate article]].
+
How ForgeGradle retrieves these mappings is covered in the [[ForgeGradle/mappings|appropriate article]].
  
The process of renaming itself is a simple regex substitution, performed by [https://github.com/MinecraftForge/ForgeGradle/blob/e2ed49546abced95650635f81be071441ec60995/src/common/java/net/minecraftforge/gradle/common/util/McpNames.java#L104 ForgeGradle itself]. This is made possible by the assured uniqueness of SRG names.
+
The process of renaming itself is a simple regex substitution, performed by [https://github.com/MinecraftForge/ForgeGradle/blob/a0d2d1f488b3285ef7c75367feeebbe1a38aa19b/src/common/java/net/minecraftforge/gradle/common/util/McpNames.java#L105 ForgeGradle itself]. This is made possible by the assured uniqueness of SRG names.
  
 
=== Post-processing ===
 
=== Post-processing ===
Line 213: Line 181:
 
At this point, the files are ready to go. We have processed the Minecraft jar such that it can be recompiled, we have applied appropriate access transformers and the Forge patches, and optionally renamed every applicable SRG name to whatever chosen distribution of mappings.
 
At this point, the files are ready to go. We have processed the Minecraft jar such that it can be recompiled, we have applied appropriate access transformers and the Forge patches, and optionally renamed every applicable SRG name to whatever chosen distribution of mappings.
  
There is not much left to do but package the code into a jar file, and place it into the gradle cache (so that this process does not occur every single time the project is opened). It calculates this name based on many factors, and this is covered in the [[ForgeGradle#naming]] article.
+
There is not much left to do but package the code into a jar file, and place it into the gradle cache (so that this process does not occur every single time the project is opened). It calculates this name based on many factors, and this is covered in the [[ForgeGradle#naming|naming]] article.
  
== Some additional Infos ==
+
== Some Additional Info ==
# You will never see c_XXX_ for classes, because the class names are picked beforehand
+
# You will never see C_XXX_ for classes, because the class names are picked beforehand using the official mapping class names
#* this is still crowdsourced, but before a new version is ready for Forge, all classes are given an <code>MCP</code> name
 
#* this <code>MCP</code> name stays constant throughout the game version it was picked for; it can change between versions, but it usually wont for already-named classes (except for misspells, typos, and misnames)
 
 
# If you look into the JAR, you won't see any packages for the obfuscated classes, but the deobfuscated classes do have the packages, this is because the same process that names the classes, also decides what package they belong to
 
# If you look into the JAR, you won't see any packages for the obfuscated classes, but the deobfuscated classes do have the packages, this is because the same process that names the classes, also decides what package they belong to
# parameters have special names
 
#* there are two types of parameter names: <code>p_XXX_X_</code> and <code>p_iXXX_X_</code>
 
#* the one with the i means that it's a parameter for a constructor
 
#* the first set of numbers are the SRG ID of their parent method, and the second number denotes the index of the parameter
 
#* the index of the parameter is a bit more involved, but this will not be explained here
 
# If you look into the source, you'll see that parameters for lambdas don't have mapped names
 
#* This is because of a complication in how the lambdas are compiled/decompiled; it's a more advanced topic which involves how the compiler compiles lambdas which we will not explain here.
 
  
 
== See also ==
 
== See also ==

Latest revision as of 23:48, 19 August 2021

This page is under construction.

This page is incomplete, and needs more work. Feel free to edit and improve this page!

The Forge toolchain is designed to clean up, merge, deobfuscate, rename, patch and provide the Minecraft source code for the usage of modders, researchers, or just those curious about how the game works.

This serves as an explanation as to how the development environment sets up your code, and aims to help you troubleshoot in case something goes wrong.

Overall process

Whenever a gradle refresh is triggered, a few things occur:

  1. ForgeGradle downloads the jar and MCPConfig zip and triggers the extractSrg/createSrgToMcp task.
  2. After that, it processes the jar - applies access transformers from forge/userdev, SAS from forge, and decompiles.
  3. Finally, it patches and finalizes the code, ready for modder consumption.

This is triggered whenever the setup task is ran.

Needed knowledge

Obfuscation

Obfuscation is the process of renaming all of the fields, methods, and classes of the compiled code into unreadable, machine-generated names (such as aaa, bC), and removing package structures (this makes everything package-local and makes the obfuscated code smaller somewhat. This is commonly used by companies to prevent external entities from easily decompiling their released binaries/executables and retrieving their source code/intellectual property.

Additionally, due to the way the Local Variable Table (LVT) of Java bytecode is stored, every function-local variable name is turned to (that's right, a snowman) in the compiled files. This makes immediate recompilation of the game literally and physically impossible, as every Java compiler currently available requires that local variables have unique names.

Minecraft is a commercial game, which means the source code is unavailable to all except the developers of Mojang. To prevent piracy and the copying of their intellectual property, Mojang applies this obfuscation process to the game before they release it. They use a tool called ProGuard, which is both an optimizer[1] and an obfuscator.

Problematic Naming

There are a few big problems with using these obfuscated names directly for modding.

  1. It is incredibly difficult to create mods using these obfuscated names. It requires immense patience to reverse-engineer the meanings behind each and every name, and to keep relating those names to what was already reverse-engineered. Although, tools do exist to make this process easier, such as IntelliJ IDEA plugins that provide naming hints automatically.
  2. Because the obfuscation process takes place after compilation (the obfuscator operates on the compiled classes), the obfuscated names are not handled by the compiler. Thus, obfuscated classes may contain member[2] names that are invalid in the Java source language, but valid in compiled bytecode (like ☃ discussed earlier); this means that the decompiled source of the game is not immediately recompilable.
  3. These obfuscated names are automatically generated by the obfuscator for each independent release. This means that the obfuscated names may change significantly between any two versions, making it harder for mod developers to update mods between releases.

SRGification

Some background on what SRG is and how it works:

SRG stands for Searge's Retro Guard; RetroGuard being an early attempt to reverse the ProGuard obfuscation, and Searge being co-author of the Mod Coder Pack or MCP[3], who created this process. Each obfuscated class, method, and field is assigned a unique number by the backend, via a sequential counter. This unique number is called the SRG ID of that class/method/field (henceforth called member).

The SRG name of the member is then derived from its SRG ID and its type (method {given the prefix m_}, field {given the prefix f_}, or parameter {given the prefix p_}). [4]. This inclusion of the SRG ID into the name guarantees that the SRG name for all members are unique, and is the reason the ID is generated.

The actual conversion of obf names to SRG names is done by a tool called Vignette. More information on how it works can be found on that page.

The Setup

The process can be broken up into 3 steps; MCPConfig, patch and provide. The MCPConfig step is, understandably, the biggest and most prone to failure. An explanation of MCPConfig itself, how it works, what it's for (but NOT how to use it) can be found here. For the purpose of this guide, you need only know that its' goal is to get the game decompiled, and into a state where it can immediately be recompiled. This means it needs to fix and patch the source code before passing it onto Forge.

In this way, MCPConfig can be thought of the vanilla side of the setup. It does not modify the game.

The following steps are all executed in order of appearance.

Download and parsing MCPConfig

The first thing that ForgeGradle does upon initializing a first-time setup, is starting the SetupMCP task.

This task then seeks to download the MCPConfig.zip jar for the version you're setting up Once it is acquired, it parses the steps contained within the config.json. It does this by interpreting the file with the following rules:

  • Every key, except for libraries, is interpreted as the name of a step.
  • Steps are executed in order.
  • The version value is interpreted as a maven coordinate of a file to download.
  • If there is a repo value, it is used instead of the maven repositories defined in the buildscript, to retrieve the version.
  • The args array is parsed, and {values} like this are interpreted as inputs, which can be substituted accordingly.
  • Once the step is all parsed in, it is executed:
    • java -jar <version> <args> <jvmArgs>

A config.json can be found here which defines the following steps:

  • fernflower
    • version: net.minecraftforge:forgeflower:1.5.498.12
    • args: -din=1 -rbr=1 -dgs=1 -asc=1 -rsy=1 -iec=1 -jvn=1 -isl=0 -iib=1 -log=TRACE -cfg {libraries} {input} {output}
    • jvmargs: -Xmx4G
  • merge
    • version: net.minecraftforge:mergetool:1.1.3:fatjar
    • args: --client {client} --server {server} --ann {version} --output {output} --inject false
  • rename
    • version: net.minecraftforge.lex:vignette:0.2.0.10
    • args: --jar-in {input} --jar-out {output} --mapping-format tsrg2 --mappings {mappings} --fernflower-meta --cfg {libraries} --create-inits --fix-param-annotations
More information about each of these tools can be found at the link provided, as well as what each of these arguments do. A brief description is provided.

ForgeFlower

The decompiler used by ForgeGradle is a custom fork of Jetbrains' FernFlower, called ForgeFlower which searches the jar for files, cleans up the bytecode, and then converts it into a reasonable best-guess interpretation.

It also produces the following side effects:

  • Removes synthetic parameters from constructors
    • In bytecode, inner classes have the outer class as their first constructor parameter, but Java source code does not.
  • Adds constructors for inner classes
    • These are removed by Proguard sometimes, as they are not required in the bytecode if the parent has a default constructor.
  • Hides bridge methods
  • Decompiles generic signatures
  • Encodes non-ASCII characters in string and character literals as Unicode escaped characters
  • Hides synthetic class members
  • Prevents simple lambdas from being inlined

As you can see by the repository, a lot of work has gone into tuning it for Minecraft's needs, but it is still a far way from perfect. This is why the patches are needed.

If you look at the patches for 1.17.1, they are mostly incredibly simple changes. Adding generics, making types more strict.

This is all stuff that should be done by the decompiler, and PRs are always welcome at the ForgeFlower repository for changes and fixes that would reduce the amount of MCPConfig patches required to get the game to compile.

For now, it is a necessity.

Mergetool

The game is split into two distributions; server and client. Since the server is just a subset of the client, the client contains the server-only classes as well.

To get around this, we have a tool called Mergetool, which can search for the differences between two files (down to the function level) and merge them into one large (referred to as joined) jar file. It also can also annotate those files as necessary.

It is a simple program, but it works.

Vignette

Vignette is where SRG starts to come into play. It serves the role of our deobfuscator, performing deobfuscation.

This process is done with the help of a deobfusation map, a file generated by the original obfuscator (in this case, ProGuard) that contains a map of the obfuscated names to original, non-obfuscated names. This is commonly used on deobfuscating stack traces outputted by an obfuscated program, for debugging purposes.[5]

We have three sets of deobfuscation maps available to us; the obf->SRG mappings distributed with the MCPConfig system, the Yarn intermediary system, or the official mappings.[6].

To rectify this problem, Forge has it's own process to create deobfuscation mappings for the game, using the official mappings. This process is split into two separate parts: the SRG renaming, and the official mapping. During the SetupMCP task, only the SRG renaming is performed.

Vignette itself operates on the compiled jar and takes in a .tsrg file, like that contained in the MCPConfig zip. It does this because it's easier to rename LVT entries (from the snowman) to readable names while you can search for every other code path that references that specific entry; ergo renaming all accesses at once. This is impossible in source code, where every name is identical and string matching is impossible.

It also:

  • Handles adding annotations for parameters that have synthetic data
    • In bytecode, these Nonnull (or whatever) annotations are attached to the parameters, not to the function that contains them.
  • Adds synthetic (invisible) constructors for classes without them

LVT (Local Variables) are initially renamed to lvt_<index>_<version>[7]

First, the class names are renamed from their obfuscated target to the specified mapping output. Then, iterating the members of the class, it renames fields, methods, parameters, and inner classes.

A recap from earlier:

  • For classes -> C_###_ (but it is transparently remapped without the C_ being written to disk)
  • For functions/methods -> m_<ID>_
  • For fields -> f_<ID>_
  • For function/method parameters -> p_###_[8].

For example, m_91087_ refers to a function with SRG ID 91087.[9]

At this point, we have a completely remapped jar.

Patches

Once we have the source code ready to go, the final step in the setup is to apply patches. These are done trivially using DiffPatch.

The Forge Side

When Forge is first loaded by gradle, it prepares the files for setting up the workspace.

The initial setup is done through the gradlew setup task. It will run the SetupMCP tasks and then start applying the Forge system to the processed vanilla code. After the MCPConfig/SetupMCP tasks are finished, ForgeGradle will print MCP Environment Setup is complete and continue. The text pausing after requested step is also printed twice during this task such that some necessary data can be extracted.

First, it applies ATs. After that, patches. Finally, the official mappings, or some other mappings from a provider like Parchment, are applied.

All in all, compared to the MCPConfig setup, this is a string of extremely basic tasks - mostly just one-line commands.

Applying Access Transformers

Access Transformers are a way of changing the visibility and finality of classes and class members. A full explanation of how it works, what the specification is, and what exactly they're used for, can be found at that page.

They are applied by passing the AT Config (nowadays called accesstransformer.cfg) into AccessTransformers:

  • AccessTransformers.jar --inJar {input} --outJar {output} --logFile accesstransform.log --atFile {at.cfg}

The AT cfg for Forge can be found here.

Applying Patches

The patches used for Forge itself are different from those used by MCPConfig, which means there are two separate patching stages performed.

As opposed to the minimal MCPConfig patching, with the goal to make the code recompilable, the Forge patching is done to apply the API and mod loader to the code.

It does this in a very similar way, with the BinaryPatcher utility used at install time/during a CI build or DiffPatch in all other situations.


Applying Mappings

This step isn't strictly necessary, and it can be omitted. However, working with exclusively SRG names is confusing for most people, so we have an extra step to apply human names to the SRG.

These renames can come from any source; as mentioned earlier, the Yarn naming or the Mojang Obf-map naming. ForgeGradle does not care, as long as there is a valid SRG->names map.

How ForgeGradle retrieves these mappings is covered in the appropriate article.

The process of renaming itself is a simple regex substitution, performed by ForgeGradle itself. This is made possible by the assured uniqueness of SRG names.

Post-processing

At this point, the files are ready to go. We have processed the Minecraft jar such that it can be recompiled, we have applied appropriate access transformers and the Forge patches, and optionally renamed every applicable SRG name to whatever chosen distribution of mappings.

There is not much left to do but package the code into a jar file, and place it into the gradle cache (so that this process does not occur every single time the project is opened). It calculates this name based on many factors, and this is covered in the naming article.

Some Additional Info

  1. You will never see C_XXX_ for classes, because the class names are picked beforehand using the official mapping class names
  2. If you look into the JAR, you won't see any packages for the obfuscated classes, but the deobfuscated classes do have the packages, this is because the same process that names the classes, also decides what package they belong to

See also

  1. An optimizer is a program that removes redundant/unused instructions and compacts the code to be faster and smaller.
  2. member refers to class fields and methods.
  3. This was previously called the Minecraft Coder Pack
  4. The SRG name for a given member is only created once, when it first appears in the code.
  5. For ProGuard users (such as Mojang), this is done using ReTrace.
  6. Mojang recently released their deobfuscation mappings for Minecraft (colloquially named mojmappings), but the licensing for its uses is a bit ambiguous. See this post by cpw for more information.
  7. Index means: the place where this item was found. If it is the 4th entry in the table, it is index 3.
  8. The index is the position of the argument. 0 on the left, 1 after that, 2 after that. Note that double and long arguments increase their index by two, rather than one of the parameter
  9. For version 1.17.1, m_91087_ refers to Minecraft#getInstance with an obfuscated name of dvp$C.