The "next obfuscator" logic was split into three funcs for no reason;
it all fits into a single function in less than a dozen lines.
While here, avoid calling the obfRand variables the same as the
type itself, as that leads to significant confusion.
Beyond the usual updating of version strings, re-running `go generate`,
and updating the patches for the linker, we needed extra changes.
First, the linker started being stricter about which packages
are allowed to linkname runtime names by import path.
Stop obfuscating import paths in runtimeAndLinknamed.
Second, the compiler's intrinsics code adds an "add" function
much like the existing "addF", so tweak the regular expression.
Third, note that there is a new runtimeAndDeps windows-only package,
which we find thanks to the recent improvement to cover many platforms.
Fourth, the code around magic numbers in the runtime has changed,
so rewrite the code which patches it via go/ast.
Fixes#990.
Change getInt helper from panicking to returning an error when
encountering invalid or out-of-range control flow directive parameters.
This gives a clear error message to the user instead of a stack trace.
Check the return value of targetFile.Close() to catch delayed write
errors. The previous defer pattern silently discarded errors from
the final flush, which could hide data corruption on some filesystems.
Return a descriptive error instead of panicking when loadLinkerPatches
encounters an unsupported patch type (new/delete/copy/rename). This
gives a clear error message to the user instead of a stack trace.
- nextLinearTimeObfuscator was indexing linearTimeObfuscators instead
of linearTimePrimeObfuscators, making the prime variant unreachable.
- Fix copy-pasted docstring on byteLitWithExtKey (said "simple xor"
but the function uses an extended key).
- Add comment explaining the XOR self-inverse mapping.
While strictly speaking it would be okay to leave Go 1.24 support
in place for the time being, we are behind on a few tasks at the moment
so it's best to keep the setup at master simpler for the next release.
Go 1.25 already came out two weeks ago, and it seems to have been
a fairly smooth release, so I don't suspect any end users will have
trouble upgrading to it.
Note that two changes were necessary for garble to work on Go 1.25.0.
First, we stop deduplicating runtimeAndLinknamed with runtimeAndDeps.
Otherwise, for GOOS=windows, internal/runtime/cgroup would be missing
as it is a //go:linkname target from runtime on all platforms,
but it is not transitively imported from runtime on GOOS=windows.
Second, the testing/synctest package is now part of std,
and it is a //go:linkname target from the testing package
but not a transitive import from it. Teach appendListedPackages that,
when loading all packages for a `go test` run, it should load
the new testing/synctest package too.
Fixes#968.
Some programs which could automatically reverse string literals obfuscated with `-literals` exist.
They currently work by emulating the string literal decryption functions we insert.
We prevent this naive emulation from succeeding by making the decryption functions dependent on global state.
This can still be broken with enough effort, we are curious which approach reverse-engineers come up with next, we certainly still have some ideas to make this harder.
Fixes#926
---------
Co-authored-by: Paul Scheduikat <lu4p@pm.me>
When we build the patched cmd/link binary for use by garble,
we perform this build in a temporary directory so that the Go module
from the user does not get in the way.
When the user module made us upgrade the toolchain per GOTOOLCHAIN,
leaving that module's directory stops upgrading the toolchain,
so we patch a newer toolchain and build it with an older toolchain.
This is largely harmless, but it makes the newer toolchain think
it is actually an older toolchain, which leads to those pesky
"linker object header mismatch" version errors.
Updates #934.
x/exp/rand was being used for no apparent reason; use math/rand.
x/exp/maps and x/exp/slices can be replaced with maps and slices
respectively now that we require Go 1.23 or later.
Note that the APIs are slightly different due to iterators.
This lets us start taking advantage of featurs from Go 1.23,
particularly tracking aliases in go/types and iterators.
Note that we need to add code to properly handle or skip over the new
*types.Alias type which go/types produces for Go type aliases.
Also note that we actually turn this mode off entirely for now,
due to the bug reported at https://go.dev/issue/70394.
We don't yet remove our own alias tracking code yet due to the above.
We hope to be able to remove it very soon.
Panicking in small helpers or in funcs that don't return error
has proved useful to keep code easier to maintain,
particularly for cases that should typically never happen.
However, in these cases we can error just as easily.
In particular, I was getting a panic whenever I forgot
that I was running garble with Go master (1.23), which is over the top.
add trash block generator
For making static code analysis even more difficult, added feature for
generating trash blocks that will never be executed. In combination
with control flow flattening makes it hard to separate trash code from
the real one, plus it causes a large number of trash references to
different methods.
Trash blocks contain 2 types of statements:
1. Function/method call with writing the results into local variables
and passing them to other calls
2. Shuffling or assigning random values to local variables
In some cases, compiler could optimize the shuffle obfuscator,
causing exposing the obfuscated literal.
As a fix, added xor encryption of array indexes.
The Go 1.21 linker patches luckily rebased on master as of
de5b418bea70aaf27de1f47e9b5813940d1e15a4 just fine.
The addition of the strings import in the second patch was removed,
since the file in Go 1.22 now has this package import.
We can remove the Go 1.20 linker patches too, since we no longer support
that Go version in the upcoming release.
Start treating runtime/internal/startlinetest as part of the runtime,
since otherwise its test-only trickery breaks "garble build std":
# runtime/internal/startlinetest
[...]/XS7r7lPHkTG.s:23: ABI selector only permitted when compiling runtime, reference was to "HGoWHDsKwh.AlfA2or7Nnb"
asm: assembly of $WORK/.tmp/garble-shared1535203339/HGoWHDsKwh/XS7r7lPHkTG.s failed
While here, update actions/checkout and staticcheck in CI.
Without hardening, obfuscation is vulnerable to analysis via symbolic
execution because all keys are opened, and it is easy to trace their
connections. Added extendable (contribution-friendly) hardening
mechanism that makes it harder to determine relationship between key and
execution block through key obfuscation.
There are 2 hardeners implemented and both are compatible with literal
obfuscation, which can make analysis even more difficult.
This means we now have a unified cache directory for garble,
which is now documented in the README.
I considered using the same hash-based cache used for pkgCache,
but decided against it since that cache implementation only stores
regular files without any executable bits set.
We could force the cache package to do what we want here,
but I'm leaning against it for now given that single files work OK.
For each Go package we obfuscate, we need to store information about
how we obfuscated it, which is needed when obfuscating its dependents.
For example, if A depends on B to use the type B.Foo, A needs to know
whether or not B.Foo was obfuscated; it depends on B's use of reflect.
We record this information in a gob file, which is cached on disk.
To avoid rolling our own custom cache, and since garble is so closely
connected with cmd/go already, we piggybacked off of Go's GOCACHE.
In particular, for each build cache entry per `go list`'s Export field,
we would store a "garble" sibling file with that gob content.
However, this was brittle for two reasons:
1) We were doing this without cmd/go's permission or knowledge.
We were careful to use filename suffixes similar to Export files,
meaning that `go clean` and other commands would treat them the same.
However, this could confuse cmd/go at any point in the future.
2) cmd/go trims cache entries in GOCACHE regularly, to keep the size of
the build and test caches under control. Right now, this means that
every 24h, any file not accessed in the last five days is deleted.
However, that trimming heuristic is done per-file.
If the trimming removed Garble's sibling file but not the original
Export file, this could cause errors such as
"cannot load garble export file" which users already ran into.
Instead, start using github.com/rogpeppe/go-internal/cache,
an exported copy of cmd/go's own cache implementation for GOCACHE.
Since we need an entirely separate directory, we introduce GARBLE_CACHE,
defaulting to the "garble" directory inside the user's cache directory.
For example, on Linux this would be ~/.cache/garble.
Inside GARBLE_CACHE, our gob file cache will be under "build",
which helps clarify that this cache is used when obfuscating Go builds,
and allows placing other kinds of caches inside GARBLE_CACHE.
For example, we already have a need for storing linker binaries,
which for now still use their own caching mechanism.
This commit does not make our cache properly resistant to removed files.
The proof is that our seed.txtar testscript still fails the second case.
However, we do rewrite all of our caching logic away from Export files,
which in itself is a considerable refactor, and we add a few TODOs.
One notable change is how we load gob files from dependencies
when building the cache entry for the current package.
We used to load the gob files from all packages in the Deps field.
However, that is the list of all _transitive_ dependencies.
Since these gob files are already flat, meaning they contain information
about all of their transitive dependencies as well, we need only load
the gob files from the direct dependencies, the Imports field.
Performance is largely unchanged, since the behavior is similar.
However, the change from Deps to Imports saves us some work,
which can be seen in the reduced mallocs per obfuscated build.
It's unclear why the binary size isn't stable.
When reverting the Deps to Imports change, it then settles at 5.386Mi,
which is almost exactly in between the two measurements below.
I'm not sure why, but that metric appears to be slightly unstable.
goos: linux
goarch: amd64
pkg: mvdan.cc/garble
cpu: AMD Ryzen 7 PRO 5850U with Radeon Graphics
│ old │ new │
│ sec/op │ sec/op vs base │
Build-8 11.09 ± 1% 11.08 ± 1% ~ (p=0.796 n=10)
│ old │ new │
│ bin-B │ bin-B vs base │
Build-8 5.390Mi ± 0% 5.382Mi ± 0% -0.14% (p=0.000 n=10)
│ old │ new │
│ cached-sec/op │ cached-sec/op vs base │
Build-8 415.5m ± 4% 421.6m ± 1% ~ (p=0.190 n=10)
│ old │ new │
│ mallocs/op │ mallocs/op vs base │
Build-8 35.43M ± 0% 34.05M ± 0% -3.89% (p=0.000 n=10)
│ old │ new │
│ sys-sec/op │ sys-sec/op vs base │
Build-8 5.662 ± 1% 5.701 ± 2% ~ (p=0.280 n=10)
To inevstigate #721, I wrote this fuzzer to see if any particular
combination of string literals and literal obfuscators would result
in a broken program.
I didn't find anything, but I reckon this fuzzer can still be useful.
Go 1.21 already breaks one of the three patches.
The complexity of the patches will likely increase,
and we only ever need to actively maintain the highest set of patches,
so splitting by major version feels natural.
Changes literal obfuscation such that literals of any size will be obfuscated,
but beyond `maxSize` we only use the `simple` obfuscator.
This one seems to apply AND, OR, or XOR operators byte-wise and should be safe to use,
unlike some of the other obfuscators which are quadratic on the literal size or worse.
The test for literals is changed a bit to verify that obfuscation is applied.
The code written to the `extra_literals.go` file by the test helper now ensures
that Go does not optimize the literals away when we build the binary.
We also append a unique string to all literals so that we can test that
an unobfuscated build contains this string while an obfuscated build does not.
We're building the linker binary for the host GOOS,
not the target GOOS that we happen to be building for.
I noticed that, after running `go test`, my garble cache
would contain both link and link.exe, which made no sense
as I run linux and not windows.
`go env` has GOHOSTOS to mirror GOOS, but there is no
GOHOSTEXE to mirror GOEXE, so we reconstruct it from
runtime.GOOS, which is equivalent to GOHOSTOS.
Add a regression test as well.
When we use `go list` on the standard library, we need to be careful
about what flags are passed from the top-level build command,
because some flags are not going to be appropriate.
In particular, GOFLAGS=-modfile=... resulted in a failure,
reproduced via the GOFLAGS variable added to linker.txtar:
go: inconsistent vendoring in /home/mvdan/tip/src:
golang.org/x/crypto@v0.5.1-0.20230203195927-310bfa40f1e4: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
golang.org/x/net@v0.7.0: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
golang.org/x/sys@v0.5.1-0.20230208141308-4fee21c92339: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
golang.org/x/text@v0.7.1-0.20230207171107-30dadde3188b: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
To ignore the vendor directory, use -mod=readonly or -mod=mod.
To sync the vendor directory, run:
go mod vendor
To work around this problem, reset the -mod and -modfile flags when
calling "go list" on the standard library, as those are the only two
flags which alter how we load the main module in a build.
The code which builds a modified cmd/link has a similar problem;
it already reset GOOS and GOARCH, but it could similarly run into
problems if other env vars like GOFLAGS were set.
To be on the safe side, we also disable GOENV and GOEXPERIMENT,
which we borrow from Go's bootstrapping commands.
At linker stage, we now encrypt funcInfo.entryoff value with a simple algorithm (1 xor + 1 mul).
This makes it harder to relate function metadata (e.g. name) to function itself in binary, almost without affecting performance.
Daniel Martí