What's so confusing from this post, is the complete lack of motivations for why anyone would want this over a shell script at the root of their project. I'd rather it even blindly rerun the command every `go build` than this half hearted attempt.
This looks much like my team's `make update` rule (that merely runs ./mk/update.sh). Well, except now the dependencies on external tools are scattered throughout your project. Oh and build failures wouldn't even tell you that you need to run go generate.
What a mis-step. Then again, maybe I'm somehow mistaking a build system with a compiler[0]?
Worse yet, a shell script would work whereas go generate does not.
Go generate is probably the most half-assed thing the go team has done yet.
Now, I know the above line is harsh, but I don't say that without proof. Run the following and bear witness to its terrible implementation.
curl http://pastebin.com/raw.php?i=fhu14iwk > main.go
cat main.go
go run main.go
go generate -v main.go
go generate -help
go generate -x -v -run ".*(echo|one).*"
Yeah...
If you don't feel like looking at the above, the implementation of go generate is a hack of substring matching, not language parsing, and it also doesn't implement, in its production release, the "-run" flag documented in "-help".
Here's a corresponding C program in which I use a #define that is not processed by the pre-processor since it's not a dumb string matcher. http://pastebin.com/raw.php?i=dE7QsTgS
Define might be. But actual directives are not. In go, the directives are.
.PHONY targets and multi-line Make rule bodies are a pet peeve of mine too. I'm not sure why people are so afraid to create extra scripts. When you're in the business of running processes, the programming language is shell and the unit of abstraction is a script file: You shouldn't be afraid to make a .sh file at nearly the level of granularity you'd make a top-level function in your general purpose environment.
I find that in recent versions of GNU make, the ONESHELL and define directives alleviate some of the pain of multi-line rules. There is also this emacs package [1] I wrote while maintaining an inline-shell heavy makefile framework that had to run on make 3.81 (the version that comes with CentOS 6).
I definitely agree with you on extracting rules into separate scripts, but I've yet to see a make-based build system that embraces this idea.
Because having one function per file is a huge hassle when editing them. I'm working on a system which is built out of >50 batch files in that manner; I had to write a tool to show me the callgraph in order to properly understand it.
> What's so confusing from this post, is the complete lack of motivations for why anyone would want this over a shell script at the root of their project.
I'm not very familiar with go but I think the motivation might be inspired by Python's "one way to do it" maxim. https://wiki.python.org/moin/TOOWTDI
This seems to be the philosophy adopted by Go as well (e.g. the gofmt tool, the integrated build system, only one loop construct, slices, etc.).
Personally, I think this philosophy is extremely beneficial in making code more readable and I would say this feature is consistent with it.
> I'm not very familiar with go but I think the motivation
The motivation is pretty transparent: since the Go team is categorically refusing to implement generics, they are trying to find workarounds to avoid having developers copy/paste code to simulate genericity.
Their solution is to have a tool generate that code instead of developers doing the copy/paste.
Yes it's under-motivated, but you can see it if you read in between the lines. Start out with the goal that authors of open source Go packages should check in generated Go source and nothing more is needed. Even this much support for code generation isn't strictly necessary.
The article only gives one reason why generated code should be checked in ("if only for the reason that the program it invokes might not be available on the target machine") but I suspect there might be other unstated reasons: it's useful to have a permanent record of generated code in source control for auditing and debugging, and it ensures that package authors can't make downstream builds slower by choosing a slow build tool.
In the open source world, your dependencies are maintained by people in other organizations. Avoiding dependencies on other teams' crufty build files is a good thing.
> The article only gives one reason why generated code should be checked in ("if only for the reason that the program it invokes might not be available on the target machine") but I suspect there might be other unstated reasons: it's useful to have a permanent record of generated code in source control for auditing and debugging, and it ensures that package authors can't make downstream builds slower by choosing a slow build tool.
I've definitely found diffing the generated code useful for confirming refactorings of the generation code don't change anything unexpected - and I imagine those doing my code reviews do as well. I might not make much use of the actual VCS history, but committing to VCS gives me sane diffs to look at for minimal effort.
> In the open source world, your dependencies are maintained by people in other organizations. Avoiding dependencies on other teams' crufty build files is a good thing.
Agreed (although sometimes the build files are good.) I'm pretty sure that moving crufty build rules into such a primitive build system isn't help on that front however. Even as a Microsoft coolaid drinking, Visual Studio using, C# loving windows dev... I think I'd much rather hack up or rewrite someone's Makefile than go diving through the source to find all the similar-but-not-quite-the-same build rules scattered throughout the codebase.
The norm in Go is that project info lives, as much as possible, in the source files themselves--that includes dependencies, architecture-specific build tags, canonical import paths, etc. Also, as many steps as reasonably possible are accessible through the 'go' tool--testing, building, fmt, vet, get/install, etc.--even if the go tool has to invoke other programs on the local machine to do it (like how go get invokes git/hg, or go build can invoke a C compiler for cgo).
This applies the "project info in the source" and "make stuff doable through the go tool" ideas to codegen. I don't think it's life-changing (I hope it's not too life-changing, i.e., that people don't go too crazy building hard-to-maintain Go-generating hacks) but it seems consistent.
> An explicit goal for Go from the beginning was to be able to build Go code using only the information found in the source itself, not needing to write a makefile or one of the many modern replacements for makefiles. If Go needed a configuration file to explain how to build your program, then Go would have failed.
"cross platform compatible with no other dependencies".
No, not at all. Not even close.
Your go generate comment takes the form of
//go:generate someCommand arg1 arg2
That's as far from cross-platform as possible. For example, their "yacc" example will not run on a platform without yacc. Any command you put there will have to be present on each platform and any command you put there is a dependency. Worse yet, it's a dependency that you can't explicitly declare and manage. You can't say "runtime dependencies: github.com/tools/godep, generate dependencies: yacc".
This is a terrible mess if you want to write a cross-platform generator and it's an even bigger mess if your generator isn't something present on every platform already because otherwise generate will just fail with "command not found".
Nitpick: Go's yacc is part of the standard Go distribution, every Go installation has it. But it doesn't matter, go generate can indeed use programs a user might not have.
But you know what, that doesn't matter either, it's true for shell scripts and it's true for makefiles as well. Go generate changes nothing in that respect, however, it does improve portability significantly in other ways. For example, in the Go tree we replaced shell scripts with go generate and perl scripts with go code called by go generate. This is important because Windows and Plan 9 don't have bourne shells, Plan 9 doesn't have perl at all and in Windows perl has to be installed. Note that there are no makefiles, which don't exist natively on Windows and Plan 9 either.
Indeed. Why mix two grammars in one file? Now, that "go generate" call has to parse go (at least up to comments; I think it should ignore "//go:generate" inside multi-line strings), and then parse those comments.
I would think it would be way simpler to only allow those special comments in the input file for "go generate".
If I were snarky, I would add that, if this gets used a lot, it might be handy to add dependency information to that file, so that "go generate" can skip repeated work. Once it does that, people can call it every time they compile their code, gaining convenience without much loss of time.
I also wonder what the distribution of code will look like. The goal seems to be to shield ordinary programmers from code generation. That is somewhat laudable (I would like to see a 'layered' language where upper layers are highly similar, but simpler to use and a bit less powerful, but haven't seen a good one yet), but at odds with the GPL. I think the GPL will require you to distribute the master code.
When used as an alternative for generics, distributing only the generated code also will lead to diverging of the various generated versions. People will fork and then modify only eight of those ten generated files.
While I agree with you I think it's just the cross compatibility the main benefit. So our friends on windows don't have to install cygwin to run a Makefile.
However I'd admit that I don't know windows and real motivations behind it.
TVTropes (to a certain, ludic, extent) and especially c2 convinced me that the conventional wiki + opinions formula occasionally works really, really well. In the pretty eerie context where everybody's passionate about the subject, that a mature level of discussion and a reasonable level of mind-share ensure that conflicting POVs are expressed, discussed without ever stuff falling down into wedge politics and flame warfare.
I can't read the wikipedia these days, there aren't enough opinions.
I have a lot of faith in the Go folks, as I've found the language to be a pleasure to work with in practice -- basically very well-thought-out.
But man, I don't know...having to manually run a generate command when eg changing string constants seems pretty weird. And if you change the constants without rerunning go generate (manually!) you may get nonsense values. It seems pretty creaky!
Maybe it will end up working out well in practice. (As Go tends to do.)
It doesn't really happen all that often in practice, and when you're hacking on a package that relies on this it's easy enough to have `go generate && go test` sitting in your shell history for that iteration cycle.
Well, I think the jury's still out, since there hasn't been a lot of use in the wild yet. I realize it can be quite frustrating to get negative feedback from people who haven't even used a feature yet, so I hope I'm not coming across as overly negative. I am overall a huge fan of Go :)
And I do appreciate the minimalistic nature of the approach...
I expect we'll start to see people do a lot of generics-type stuff with this -- e.g., add an go:generate annotation to get a customized Foo collection or whatever.
I imagine most of the complaining is by people that haven't even spent much time in the language, let alone this new tool.
I'm would wager that generics were a driving force behind this. They don't want to add them to the language (fair), but are willing to offer a way for us to add them at precompile time.
How would that work? AFAIU the idea is that library authors check in the generated code -- so a consumer of your code wouldn't be able to (or at least would be actively discouraged from) generate new code for a library. And thus wouldn't be able to generate "instances" for generic code.
I haven't spent much time looking into this, and haven't used it yet, but from the Stringer example, they're providing us with a tool to generate strings for enum values. That same approach could be used to generate support for typed collections for your types. I could make the collection-generating code, and you could generate the collections for your types.
Yeah, for your types, but if I'm a provider of a library of generic data structures (which is what the people asking for generics are probably thinking about), you're out of luck since I have no way of letting you generate appropriate instances.
`go generate` is deliberately not integrated with the other commands because it should not be used for things that are necessary when you just want to use a package. We don't want to force people to have to install random development tools just to be able to compile against someone's Go package.
I love that goal, and certainly one of the pleasures of working w/ Go is how often you don't have to hunt down a world of external dependencies.
That said, if I'm reading that code right, if I, say, decide to reorder a constant definition (say swap Aspirin and Ibuprofen), the Go code will still compile, but the string constants will be silently wrong until I realize I need to rerun go generate. No?
It's a problem in general: if I modify code that requires rerunning go generate, and I forget or don't realize I need to do it, I'm going to get at best a confusing compiler error, and at worst silently wrong code... Right?
Again, maybe this all works fine in practice, but to just brush off these concerns seems a bit glib...
One of the really great things about go generate is that the command to run (and thus the reminder that you need to run it) can and should exist very near to the code that might change. If you see
I'm not trying to brush off concerns. They're fair, but I don't think it's particularly fatal. If you're using stringer, you've opted into it by adding //go:generate lines to your source, and so if you're messing with your code you just have to remember to run `go generate` (or stick a comment near the constant definition to remind yourself). In practice you won't be messing with the constants a huge amount, and if you're using something like stringer then it's easy to keep that in mind.
What you said makes sense for a single-author codebase. For a shared codebase with many authors, I could envision it being an occasional problem, especially if misused/abused.
They are legitimate concerns, but your given example (re-ordering constants) could break other things silently in any case. There's always a line between risk and convenience. For some people Go is too conservative, and for others it is too liberal.
It is just extremely frustrating to see them using contradictory arguments for various parts of the language. This is a very liberal way to do this, which contradicts most of the rest of the language (library dependencies, language itself, features).
For instance in this case, the very existence of the code generation libraries contradicts the reflection package.
Compile-time function execution would have been so much more consistent it's horrifying :
But like every other decision in Go, you can either assume the Go developers actually believe the arguments they're putting up ... or you can ask yourself "would the Go compiler/toolchain become easier or harder to implement ?" And use that argument as almost the sole argument for any decision. One of these ways of thinking explains every decision very consistently, the other leaves one listing the exceptions.
And the right solution, mixins + CTFE is legendary hard to implement. As in, someone who's written 3 commercial C++ compilers from scratch has serious trouble getting it to work.
> And use that argument as almost the sole argument for any decision. One of these ways of thinking explains every decision very consistently, the other leaves one listing the exceptions
That's not accurate. There are a lot of things about Go that are very difficult to implement. The concurrency system and garbage collector are obvious examples. Some less obvious ones are discussed here: http://talks.golang.org/2014/compiling.slide
Conclusion
Go is simpler to compile than most languages
There are still complexities for the compiler
Most complexities stem from making Go easier to write
I would also argue that if you take a compiler book, and you look up these problems, they're all solved. That is certainly not true for most other languages such as Prolog, Haskell, Java, Rust or D, especially not at the time they were written. Most of those added a chapter or two to our knowledge of compilers. Or ten, in the case of Haskell.
Something which using Make would've fixed, incidentally. One of the neat things about Make is that you can define arbitrary dependencies (e.g. pill_strings.go depends on pill.go) and Make knows to rerun the tool when needed.
What upsets me about go generate is that it is orthogonal to the language, yet it comes baked in. It unnecessarily increases the footprint of the language, while essentially killing any alternative approaches. Feature creep comes one feature at a time, I guess.
[ps. yet thank you big time for being able to program in golang]
Why does it upset you that it comes baked in? If you don't care for it, don't use it. It's for package authors, so you won't be exposed to it unless you want to.
Unfortunately it is not a preprocessor, has nothing to do with that. Of course go comes with go/{ast,parser,scanner} which will facilitate that, but go generate is just not that.
Lisp macros are runtime code generation. This is not runtime code generation, it's code generated at some arbitrary time by the author, it's not even build-time code generation. This feature has a completely different use-case than Lisp macros, the comparison is completely unwarranted.
Lisp macros are not really 'runtime' code generation. There is a reason one might say that it is, but that's because a full Lisp is available during compilation.
In a typical Common Lisp project, Lisp macros generate the code at compile time. Since a full Common Lisp is available at compile time, this looks similar to runtime, but it isn't. For that the macro itself and the code it needs to run needs to be available during compilation.
During development the compile-time and runtime code often run inside the same Lisp system. But that's a certain style of development and not necessary. It's also possible that the generated code will form an application, which then runs outside of the compile environment.
To take a typical data structure declaration and to generate compile-time code for it is quite typical. What Lisp typically not do is to dump the generate code as source to a file, but to feed the generated code to the compiler and then create compiled code.
I'm a bit surprised that the Go team did this. They already have a Go code generator for Google protocol buffer marshaling code. (https://github.com/golang/protobuf) This new generator scheme doesn't seem to be suitable for integration with that.
(Why does every language now have to come with its very own build/packaging environment, with its very own directory structure conventions? Go has one, Rust has a different one, Java has several, and a new one for Python was discussed here yesterday.)
The protobuf code is a concrete instance of a code generator. `go generate` is a general mechanism for triggering a code generator. They are complementary, and you could imagine the former using the latter.
I'd guess Go has one to avoid ending up with several like Java, to do Go-specific things like extract the dependency graph from the source code, and to have one fewer dependency on the platform--I don't know much about how you build on plan9 or Solaris, I know go build is the same program on each.
That said, go tools are canonical but not mandatory. If switching to this doesn't work for the protobuf foks, it's simple enough not to switch.
Please re-read the article and grep the Go source code to see how go generate is being used. It's not for building anything, it's not involved in building at all. It's a tool authors use when they see fit, and it's completely orthogonal to building.
You can always emulate all autotools' shortcomings, you know, for backwards compatibility. And then add a bunch of defects of your own making on the top.
This is a pretty simple little thing, not even sure what it really brings new to the Go table except saying to the community "this is how we think it should be done, and how we now do it"
Does go support meta programming? I feel that this seems more like Macros replacement or build tasks that generate files which are included compile time.
You should definitely commit the output of yacc to git. This way not only you don't require your users to have yacc, but you make sure they are using the correct yacc version; you know, the one you tested against. And when you change the yacc version, you'll know because it's in your history; otherwise it's hard to remember at which particular time you decided to upgrade the yacc version used by the project.
Committing cpp output makes no sense because cpp output is specific to the machine which ran cpp.
Well, it's more than just dependencies. `go generate` isn't meant to be involved at all if you're just building a package or command for using it, which is quite a bit different to `make`.
Now put those two together, and you can understand what the idea here is: It doesn't do dependencies and it isn't meant for end users, only package authors.
The Go philosophy is that if you truly need a make file or other such system, use it. It's always been that way, actually; if you poke hard enough you can still find old Go tutorials which require you to use make (or some other build tool). But if you need it for just this simple little thing, you shouldn't have to pull in make, just as the developing Go philosophy is that it is silly to pull in entire packages for one little function. The purpose of "go generate" is not to replace make; it is to close out another 80% of the uses of make (my made-up number) by offering a simple, high bang-for-the-buck addition that can be depended on in the base language.
Some projects will still need make. This expands the set that don't. There's no interest in trying to make this grow to include all things that use make. It fits the pragmatic philosophy of Go.
tl;dr Go has its own version of the make tool set that doesn't allow dependencies outside of Go source files, and doesn't rebuild dependencies on file updates.
This looks much like my team's `make update` rule (that merely runs ./mk/update.sh). Well, except now the dependencies on external tools are scattered throughout your project. Oh and build failures wouldn't even tell you that you need to run go generate.
What a mis-step. Then again, maybe I'm somehow mistaking a build system with a compiler[0]?
[0]: https://news.ycombinator.com/item?id=8733493