'Add import' action fails in module mode with "-mod=vendor" flag

[nezorflame]

Description

While using Go modules in vendor mode (GOFLAGS="-mod=vendor") in a project with some indirect dependencies, action Go: Add imports fails to execute with error:

Could not find packages. Ensure 'gopkgs -format {{.Name}};{{.ImportPath}}' runs successfully.

go env output

GO111MODULE="on"
GOARCH="amd64"
GOOS="darwin"
GOFLAGS="-mod=vendor"

Steps to reproduce

Let's assume that we have a project /my/home/Go/src/my-service with a dependency in go.mod:

...
github.com/some/indirect/dependency v1.2.3 // indirect

go mod tidy and go mod vendor were run successfully, go build -mod=vendor runs fine.

If you enable Developer Tools and run Go: Add imports again, you can see a more elaborate error:

Running gopkgs failed with "stat /my/home/Go/src/my-service/vendor/github.com/some/indirect/dependency: no such file or directory"
Check if you can run 'gopkgs -format {{.Name}};{{.ImportPath}}' in a terminal successfully.

It seems that go mod vendor ignores indirect dependencies, thus they're not present in the vendor folder, but gopkgs tries to get them anyway.

[nezorflame]

Guess I'll also ping @uudashr since he's the author of gopkgs which is being used by vscode-go.

[nezorflame]

From what I can see, this happens:

1. gopkgs runs go list -m -f={{.Path}};{{.Dir}} (notice that -mod=vendor is not in the code, but since we have GOFLAGS="-mod=vendor" it'll be acting if it was): https://github.com/uudashr/gopkgs/blob/master/gopkgs.go#L366
2. The output shows ALL of the packages, including the indirect ones
3. While walking through the list of packages in the loop https://github.com/uudashr/gopkgs/blob/master/gopkgs.go#L94 we eventually stumble upon an indirect dependency which wasn't pulled by go mod vendor, which in turn results in a folder which doesn't exist, hence causing the stat error.

I guess, there're two possible solutions:

1. Exclude indirect dependencies by using the {{if not .Indirect}}{{.Path}};{{.Dir}}{{end}} flag value for go list -f
2. Somehow make go mod vendor download the indirect dependencies as well (doubt that).

I've created an issue uudashr/gopkgs#15 and made a PR uudashr/gopkgs#16 with a possible fix. WDYT?

[Vstural]

same issue but without -mod=vendor

set GO111MODULE=on
set GOARCH=amd64
set GOFLAGS=

reload vscode and try to run Go: Add import,got a warning

Could not find packages. Ensure `gopkgs -format {{.Name}};{{.ImportPath}}` runs successfully.

[nezorflame]

@Vstural if you press Toogle Developer Tools in VSCode's Help section and check Console, what error does it show for you on that action?

[Vstural]

@nezorflame
here is result after try to Go: Add import

[Extension Host] debugger listening on port 60326
workbench.desktop.main.js:sourcemap:230 [Extension Host] vscode-vibrancy is active!
workbench.desktop.main.js:sourcemap:230 [Extension Host] Running gopkgs failed with "EOF"Check if you can run `gopkgs -format {{.Name}};{{.ImportPath}}` in a terminal successfully.

and run gopkgs in bash

$ gopkgs -format "{{.Name}};{{.ImportPath}}"
EOF

in powershell

PS C:\Users\Nzeli> gopkgs -format "{{.Name}};{{.ImportPath}}"
EOF

[nezorflame]

@Vstural did you also set up vendor by go mod vendor? Or do you face it on a plain module project?

[Vstural]

@nezorflame
this is files tree, wondering is there a different between a module project or a standard project to gopkgs?

├─.vscode
├─client
├─cmd
│  ├─client
│  └─server
├─protocol
├─server
├─tools
└─vendor
    └─github.com
        └─Vstural
            └─socksgo

[ramya-rao-a]

Hey @nezorflame

Thats some great investigations!
Since I am in no way an expert in how packages should be dealt with in module or vendor mode, I would defer to @stamblerre here on what the next steps should be.

@stamblerre Can you provide your thoughts on the scenario here or the issue uudashr/gopkgs#15?

@Vstural Your case looks very different to me where gopkgs fails with a different error. Since you can see the same when running from the terminal, can you please log an issue at https://github.com/uudashr/gopkgs/ as well so that @uudashr can take a look?

[nezorflame]

I should also mention that even though my fix works, when you add new import and some func from it, if you use action Go to Definition on that func gopls redirects you to the system module instead of the vendor one.
But I guess that could be some other issue.

[nezorflame]

@stamblerre @ramya-rao-a @uudashr the more I look into this, the more confused I get.

Let's take some test project, e.g. github.com/nezorflame/test.
Assume it has one external dependency github.com/pkg/errors :

Now we download some package to be used in our project later, e.g. to refactor our code and move to Go1.13-like errors with golang.org/x/xerrors:

As expected, it downloads the latest version of that module and inserts it into our go.mod as // indirect since we didn't actually use it in the package yet.

Now we do go mod vendor to go into the vendor mode, and the only thing we have in vendor is our direct dependency github.com/pkg/errors (as expected):

Now we want to list the packages with go list and here's where things get interesting.

If we were in a plain modules mode without vendor, we'd do this (I've shortened the output to exclude my $GOPATH):

$ go list -m -f='{{.Path}};{{.Dir}}' all
github.com/nezorflame/test;.../src/github.com/nezorflame/test
github.com/pkg/errors;.../pkg/mod/github.com/pkg/[email protected]
golang.org/x/xerrors;.../pkg/mod/golang.org/x/[email protected]

All seems well, we did download these modules so the output is as expected.
But we're using vendor, so we need to add the -mod=vendor flag:

$ go list -mod=vendor -m -f='{{.Path}};{{.Dir}}' all
github.com/nezorflame/test;.../src/github.com/nezorflame/test
github.com/pkg/errors;.../src/github.com/nezorflame/test/vendor/github.com/pkg/errors
golang.org/x/xerrors;.../src/github.com/nezorflame/test/vendor/golang.org/x/xerrors

And here we are - go list tells us that we should expect the package golang.org/x/xerrors inside of the vendor folder, while we didn't use it yet and it only exists in our $GOPATH/pkg!

Is that working as expected? I think that the main issue is here, not in the gopkgs per se which is just parsing the output of the go list.

[nezorflame]

From what I've been told over Gophers Slack, using go list with both -mod=vendor and -m flags is kinda broken, "because it doesn't have enough information to produce the same output as go list -m".
I guess that in case of detecting GOFLAGS='-mod=vendor' gopkgs still needs to alter the call to go list, but I'm not sure to which one yet.

[nezorflame]

To sum it up, I think that gopkgs should have the following behaviour:

• we check if GOFLAGS contains -mod=vendor, if not - continue as before
• if it does contain it, we:

1. Make a second call to the go list with explicit empty flag -mod=.
   This is required since the first call has given us vendor paths instead of the module paths, and since go mod vendor ignores some indirect dependencies, these folders won't exist and we need to grab a module from $GOPATH/pkg instead
2. We save both outputs as pkgDir and vendorDir to pick one of them later
3. When we're inside of the collectModPkgs(), we choose the directory which exists, preferring vendorDir over pkgDir

[ramya-rao-a]

This makes me wonder if we should disable the feature Go: Add Import feature altogether when in module mode until gopkgs can fully support modules

[nezorflame]

@ramya-rao-a gopkgs itself supports modules, the problem appears only when we enter the -mod=vendor territory which is kinda quirky even for go list.
Although I imagine that there would be some people like me who have to enable vendor mode due to some CI restrictions and will face the same issue.

I'm also against disabling this feature since it's the only way to actually add new import in module mode right now. Otherwise you have to do it manually since gopls doesn't suggest modules which are not present in the project while you type.

[nezorflame]

In any case, if everyone agrees to #2800 (comment), I've already changed the PR to reflect this behaviour: uudashr/gopkgs#16

[nezorflame]

@uudashr and I agreed to use the -mod= flag directly while querying go list to avoid the problem, this seems to do the trick.
New PR: uudashr/gopkgs#18

[uudashr]

The gopkgs already fixed. Can someone clarify, do the issues still exist? cc @nezorflame

[nezorflame]

I've tested this solution more thoroughly.

While gopkgs doesn't fail anymore, it actually doesn't return the imports of the module anymore. The only imports it returns are the Go ones. I guess the solution with -mod="" didn't work after all, we need to only use -mod="" when we're using the vendor folder as the base.

[nezorflame]

@ramya-rao-a after another discussion with @uudashr it seems that the current master has this fixed:

The last issue being supporting v2 import is being resolved in uudashr/gopkgs#20

I think that this issue can be closed now, if the current behavior suits everyone.

[ramya-rao-a]

Thanks for all the work here and follow up @nezorflame and @uudashr
It is very much appreciated :)

Happy Coding!