option for protoc-gen-connect-go to output to same package as protoc-gen-go

[frantjc]

I had a use-case where I wanted .connect.go files generated with protoc-gen-connect-go to output to the same directory as .pb.go files generated with protoc-gen-go.

The reason I needed this was to preserve import paths when switching from Go structs that I'd written myself with ones generated from .proto files. The same package previously referenced Service Clients generated by protoc-gen-grpc-go which, when extracted to the ___connect package that protoc-gen-connect-go outputs to, caused an import cycle.

I forked and modified proton-gen-connect-go to never append the ___connect part to the import path and I would like to contribute it back upstream. However, I was hoping for some direction on what flag I should check to disable appending ___connect to the import path: an environment variable? A flag passed to protoc-gen-connect-go? Something else entirely?

Hope that was clearly articulated. Thanks!

[akshayjshah]

Thanks for proposing this, @frantjc! We've gone back and forth on this, most recently in #70.

We'd rather not do this because it makes it hard to move away from local code generation. With the current approach, you can go get go.buf.build/bufbuild/connect-go/your-org/your-schema - no local buf or protoc invocation needed, because the Buf Schema Registry knows that it only needs to run protoc-gen-connect-go. At Buf, we think that making it easier to consume generated code is critical to 10x-ing protobuf usage. As soon as multiple plugins are generating code into the same package, this becomes impossible.

That said, we're willing to add an option if there's significant interest. I'm going to convert this issue to a discussion and wait a bit to see if it accumulates a bunch of upvotes or comments.

[acookin]

I have a fork that adds a force-same-package option (defaulted to false) that enables generating the package in the same package as specified by the proto file. I'm willing to contribute it back and do most of the leg work here.
Our use case is that we are almost always import the generated connect and the package generated by the protoc-gen-go plugin, and we are using bazel in a private repo. We are only using the go and connect gen plugins, so we are not worried about bloat in the generated package, and are much more concerned with the developer experience for our engineers; they expect the code to be generated in the package specified in the proto file.
I understand and totally buy in to your reasoning for the default behavior here, but I think that making this tool also play nicely with bazel and what the average engineer expects would make this much more approachable for teams like ours. This tool is great, and I would hate for folks to shy away from it because of this issue.

[lbensaad]

@frantjc , I did not know you asked the same thing I need (#389).
In my opinion, connect should follow the flag --connect-go_out passed to protoc and just put the generated file in the folder specified by that flag. For the package name it should also take it from the .proto file. In my opinion, this behavior of adding a suffix to the package and putting it in other folder is breaking protobuf specs.

@akshayjshah I hope you put this behavior behind a flag, for example usePkgSuffix, and give us the possibility to disable or enable this behavior.

I wanted to switch from grpc-go but the way how connect structures code made several breaking changes in my code structure. Because I structure my code trying to flow clear architecture, where each feature has the folders: app, data, domain, and port. The folder port contains all my grpc and pb code, but when I use connect it adds another folder named portconnect that will contain only one file. When I import the portconnect package in the port package it causes an import cycle error, because the port package contains the .pb.go file which is imported by portconnect

[bufdev]

If we can come up with clear design, I'm not 100% against adding an option for this. We may want to consider what the shape of the option is exactly, i.e. do we want something like package={suffix,same,whatever} or same_package=true, or something else.

One note:

In my opinion, this behavior of adding a suffix to the package and putting it in other folder is breaking protobuf specs.

Just to be sure we're on the same page: is no Protobuf spec here, Protobuf does not define anything with regards to the outputted stubs. The choice to output i.e. grpc-go code to the same package as golang/protobuf code was a choice (and in our opinion, the wrong one), just as generating to a separate package is a choice. In our view, you should not tie a specific RPC framework's generated stubs to the same package as the base types. In theory, you should be able to have i.e. Twirp, gRPC, and Connect generated code all happy generate within the same repository and use the same interface names for Client and Server stubs, which you cannot do if they all generate to the same package. Similarly, I should not need to import i.e. google.golang.org/grpc if I am using github.com/bufbuild/connect-go, which would be the case if both grpc-go and connect-go stubs were generated to the same package as the base types.

[frantjc]

I agree with connect's design choice to generate a new package, it can just lead to problems when migrating from gRPC to connect due to gRPC's differing design choice.

I am, however, still a proponent for an option to modify this behavior, as more functionality can't really hurt, whatever shape it may take! :)

[lbensaad]

Thank you all. I have successfully replaced grpc-go and I am using connect-go, and now I understand you. I like its simplicity and the concept of directly using net/http 😍.
So no need for this feature and I am down-voting it.
I am also ditching grpc-go and grpc-web and down-voting them as well 😌

[mickeyh]

I understand (and agree) that it's cleaner to generate a separate package.

However, not exposing the option to generate in the same output package and directory (as most plugins do) means that protoc-gen-connect-go is less likely to "just work" with other tools and libraries.

A specific example is that rules_go (generally used to build Go code in Bazel repos, and more specifically used to transparently do protobuf codegen) doesn't work out of box with Connect. In principle rules_go could implement support for proto plugins generating code in subpackages but this is (substantially) more complex than implementing the optional flag in protoc-gen-connect-go and just feeding an extra option in to the Connect go_proto_compiler rule.

[singh-hovr]

Agree, There should be a option for this.

[benmcginnis]

would really like to have this, we're going to have to maintain a fork otherwise if we want to use connect

[acookin]

we are also going to maintain a fork of connect until this option to the plugin is provided

[singh-hovr]

I am open to create the PR for this feature

[XSAM]

I am not familiar with bazel, but I wonder if it is possible to generate a subfolder inside the importpath of go_proto_library . So, the generated folder path in bazel keep consistent with the one from buf.

Having different output path is a terrible user experience when switching local buf to bazel, as it changed the import path of ConnectRPC.

[emcfarlane]

A new option package_suffix has been added to protoc-gen-connect-go in #803 and released in v1.18.0. This can be used to change the sub-directory of the generated code and has the default value connect. An empty value denotes no sub-package and generates to the same folder as the imported go code.

For example to generate output into the same directory, update to the latest protoc-gen-connect-go:

 go install connectrpc.com/connect/cmd/protoc-gen-connect-go@latest

And include the new option in your buf.gen.yaml configuration file:

 version: v2
 plugins:
   - local: protoc-gen-go
     out: .
     opt: paths=source_relative
   - local: protoc-gen-connect-go
     out: .
-    opt: paths=source_relative
+    opt:
+      - paths=source_relative
+      - package_suffix

See the docs for protoc-gen-connect-go in pkg.go.dev.