The main "linux" job is the slowest by far at over 20 minutes.
The CPU on the hosted Actions runners is very slow,
but luckily, we get 20 concurrent Linux runners.
Split the main Linux job further.
Also, now that generics in Go are widespread,
we no longer need a Go generics library for good coverage.
While here, time each of the third party project commands.
Given that the Linux test job is currently the slowest,
the added parallelism will help.
The full or "long" tests aren't needed to look for races,
as our short tests already cover almost all of our code.
This is a better compromise for CI speed.
Beyond the usual updating of version strings, re-running `go generate`,
and updating the patches for the linker, we needed extra changes.
First, the linker started being stricter about which packages
are allowed to linkname runtime names by import path.
Stop obfuscating import paths in runtimeAndLinknamed.
Second, the compiler's intrinsics code adds an "add" function
much like the existing "addF", so tweak the regular expression.
Third, note that there is a new runtimeAndDeps windows-only package,
which we find thanks to the recent improvement to cover many platforms.
Fourth, the code around magic numbers in the runtime has changed,
so rewrite the code which patches it via go/ast.
Fixes#990.
This makes the unobfuscated state easily accessible, for debugging.
Obfuscated code is output to: `./<debugdir>/garbled`
The original source code is output to:
`./<debugdir>/source`
Also enable caching for debugdir to allow for faster iteration.
While strictly speaking it would be okay to leave Go 1.24 support
in place for the time being, we are behind on a few tasks at the moment
so it's best to keep the setup at master simpler for the next release.
Go 1.25 already came out two weeks ago, and it seems to have been
a fairly smooth release, so I don't suspect any end users will have
trouble upgrading to it.
Note that two changes were necessary for garble to work on Go 1.25.0.
First, we stop deduplicating runtimeAndLinknamed with runtimeAndDeps.
Otherwise, for GOOS=windows, internal/runtime/cgroup would be missing
as it is a //go:linkname target from runtime on all platforms,
but it is not transitively imported from runtime on GOOS=windows.
Second, the testing/synctest package is now part of std,
and it is a //go:linkname target from the testing package
but not a transitive import from it. Teach appendListedPackages that,
when loading all packages for a `go test` run, it should load
the new testing/synctest package too.
Fixes#968.
I forgot to update the original Go template away from `go version`.
Note that `go env` already tells us what we need via e.g. GOVERSION,
so we can avoid asking for `go version` separately.
Our ISSUE_TEMPLATE.md helpfully stopped working without any warning.
I only noticed as we started getting low-quality bug reports.
This YAML borrows a bit from Go's own bug report template,
much like we had done previously with our markdown template.
This lets us start taking advantage of featurs from Go 1.23,
particularly tracking aliases in go/types and iterators.
Note that we need to add code to properly handle or skip over the new
*types.Alias type which go/types produces for Go type aliases.
Also note that we actually turn this mode off entirely for now,
due to the bug reported at https://go.dev/issue/70394.
We don't yet remove our own alias tracking code yet due to the above.
We hope to be able to remove it very soon.
And update some actions and staticcheck while here.
Drop the testing of Go master as well, as I haven't used or maintained
such a setup for a while now. We can simply add Go 1.24 RC versions
to the go-version matrix once they come out.
Fixes#859.
In early December, a new internal package linknamed from runtime
was introduced, internal/chacha8rand. Re-generate the tables.
Note that due to the same group of CLs and refactors,
math/rand and net are no longer linknamed from runtime in Go 1.22.
They are still in Go 1.21, so keep those entries around for now.
We can remove math/rand/v2, as it doesn't yet exist in 1.21.
Fixes#820.
The Go 1.21 linker patches luckily rebased on master as of
de5b418bea70aaf27de1f47e9b5813940d1e15a4 just fine.
The addition of the strings import in the second patch was removed,
since the file in Go 1.22 now has this package import.
We can remove the Go 1.20 linker patches too, since we no longer support
that Go version in the upcoming release.
Start treating runtime/internal/startlinetest as part of the runtime,
since otherwise its test-only trickery breaks "garble build std":
# runtime/internal/startlinetest
[...]/XS7r7lPHkTG.s:23: ABI selector only permitted when compiling runtime, reference was to "HGoWHDsKwh.AlfA2or7Nnb"
asm: assembly of $WORK/.tmp/garble-shared1535203339/HGoWHDsKwh/XS7r7lPHkTG.s failed
While here, update actions/checkout and staticcheck in CI.
Go 1.21.0 was released in August 2023, so our upcoming release
will no longer support the Go 1.20 release series.
The first Go 1.22 release candidate is also due in December 2023,
less than a month from now, so dropping 1.20 will simplify 1.22 work.
Update CI to use a newer version of Go master,
now that we're already getting release candidates.
Look at the diffs between Go 1.20 and master of `go help build`
and `go help testflag`, and add two flags that were recently added.
While here, bump a hopeful TODO for a feature request,
since that one definitely did not happen for 1.21.
Packages like os and sync have started using go:linknames pointing to
packages outside their dependency tree, much like runtime already did.
This started causing warnings to be printed while obfuscsating std:
> exec garble build -o=out_rebuild ./stdimporter
[stderr]
# sync
//go:linkname refers to syscall.hasWaitingReaders - add `import _ "syscall"` for garble to find the package
# os
//go:linkname refers to net.newUnixFile - add `import _ "net"` for garble to find the package
> bincmp out_rebuild out
PASS
Relax the restriction in listPackage so that any package in std
is now allowed to list packages in runtimeLinknamed,
which makes the warnings and any potential problems go away.
Also make these std test cases check that no warnings are printed,
since I only happened to notice this problem by chance.
And bump go-internal to its latest version, to include its fix
for those pesky "signal: killed" failures on macos.
While here, run the tests with -short on GOARCH=386,
and make our use of actions/setup-go a bit more consistent.
See the added comment.
An alternative would be to split up the work into more jobs,
but that doesn't feel necessary, and we still have a global limit
on how many free runners we can run jobs on at once.
While here, update the comments, and remove unnecessary step names.
A couple of new packages in runtimeAndDeps,
and go list's Package.DepsErrors may now include package build errors
which we want to ignore as we would print those as duplicates.
printOneCgoTraceback now returns a boolean rather than an int.
Since we need to have different logic based on the Go version,
and toolchainVersionSemver was only set for the main process,
move the string to the shared cache global.
This is a nice thing to do anyway, to reduce the number of globals.
While here, update actions/setup-go to v4, which starts caching
GOMODCACHE and GOCACHE by default now.
Disable it, because it still doesn't help in our case,
and GitHub's Actions caching is still really inefficient.
And update staticcheck too.
https://go.dev/cl/466095 lightly refactored the runtime
in a way that broke extractNameOff. In particular, the code
func cfuncname(f funcInfo) *byte {
if !f.valid() || f.nameOff == 0 {
return nil
}
return &f.datap.funcnametab[f.nameOff]
}
func funcname(f funcInfo) string {
return gostringnocopy(cfuncname(f))
}
is now simply
func funcname(f funcInfo) string {
if !f.valid() {
return ""
}
return f.datap.funcName(f.nameOff)
}
Since extractNameOff looked for the func named cfuncname,
and looked for the nameOff selector inside an index expression,
all of that code no longer worked properly.
It all existed to find the name of the field, nameOff,
so that we would automatically adapt if upstream renames it.
Unsurprisingly, the code using the field got refactored first.
It doesn't seem like the extra code on our part is helping us,
and assuming the name of the field works for all Go versions,
so do that instead.
If upstream does rename the field in the future,
the obfuscated Go builds will start failing in an obvious way.
If or when that comes to pass, we can change our constant string.
Go master, the upcoming Go 1.21, has had its merge window open for over
two weeks at this point, and it seems calmer at this point.
ALso update staticcheck to its latest release, which supports Go 1.20.
tip is now the start of Go 1.21 as of a couple of days ago.
However, the first week or two is when the biggest changes land,
which means that Go tip is far more prone to bugs and changes that might
break garble.
We'll start tracking tip again in a few weeks, once the dust has settled
and we can look at what changes might have broken garble.
Even if diffoscope is installed, because further investigation might be
needed, and some failures are rare or hard to reproduce.
Make GitHub Actions upload those artifacts,
so that a failed CI run on Windows or Mac due to bincmp
allows us to download and inspect those binaries locally.
Some big changes landed in Go for the upcoming 1.20.
While here, remove the use of GOGC=off with make.bash,
as https://go.dev/cl/436235 makes that unnecessary now.
While here, start the changelog for the upcoming release,
which will likely be a bugfix release as it's a bit early to drop 1.18.
We also bump staticcheck to get a version that supports 1.19.
I also noticed the "Go version X or newer" messages were slightly weird
and inconsistent. Our policy, per the README, is "Go version X or newer",
so the errors given to the user were unnecessarily confusing.
For example, now that Go 1.19 is out, we shouldn't simply recommend that
they upgrade to 1.18; we should recommend 1.18 or later.
When obfuscating the following piece of code:
func issue_573(s struct{ f int }) {
var _ *int = &s.f
/*x*/
}
the function body would roughly end up printed as:
we would roughly end up with:
var _ *int = &dZ4xYx3N
/*x*/.rbg1IM3V
Note that the /*x*/ comment got moved earlier in the source code.
This happens because the new identifiers are longer, so the printer
thinks that the selector now ends past the comment.
That would be fine - we don't really mind where comments end up,
because these non-directive comments end up being removed anyway.
However, the resulting syntax is wrong, as the period for the selector
must be on the first line rather than the second.
This is a go/printer bug that we should fix upstream,
but until then, we must work around it in Go 1.18.x and 1.19.x.
The fix is somewhat obvious in hindsight. To reduce the chances that
go/printer will trip over comments and produce invalid syntax,
get rid of most comments before we use the printer.
We still keep the removal of comments after printing,
since go/printer consumes some comments in ast.Node Doc fields.
Add the minimized unit test case above, and add the upstream project
that found this bug to check-third-party.
andybalholm/brotli helps cover a compression algorithm and ccgo code
generation from C to Go, and it's also a fairly popular module,
particular with HTTP implementations which want pure-Go brotli.
While here, fix the check-third-party script: it was setting GOFLAGS
a bit too late, so it may run `go get` on the wrong mod file.
Fixes#573.
A chunk from crypto/internal/boring has been split away as a separate
package very recently, shortly before 1.19rc1 is due for release.
See https://go.dev/cl/407135 for more information.
Makes garble work on the latest Go tip again.
Our tests should already be pretty extensive,
and any bug fixes should result in more regression test cases,
but testing against a few diverse and popular third party modules
will help prevent unintended regressions while developing garble.
The list is short for now. More can be added later.
This adds protobuf and wireguard from the original issue,
but not cobra and logrus, as they aren't particularly complex nor add
significant variety on top of protobuf and wireguard.
While here, we remove the job that only runs crlf-test.sh,
as we don't really need a separate job for a tiny script.
Fixes#240.
It added packages which are only built with the boringcrypto build tag,
so trying to `go list` them will fail even though it doesn't matter.
While here, a few more minor cleanups:
1) Hide GarbleActionID and ToObfuscate from encoding/json, so that they
can't possibly collide with the fields consumed from `go list -json`.
2) Add test cases for `garble build` with packages that fail to load.
Note that this requires GOGARBLE=* to avoid its "does not match any
package to be built" error.
3) Remove the last use of interface{}, in a testdata file.
Fixes#531.
Daniel Martí