He's making the wrong assumption that you don't need to setup a build environment when building with make, but to have gcc you will still also need to install g++ and build-tools. Also, he refers to building on Windows using yet another specialist tool, but have you recently tried building anything C++ on Windows when you don't have visual studio, or even worse, CYGWIN installed?
When you make such a statement, then please show me a makefile that's not 10.000 lines long, that will do the same as this, but without NPM and 'downloading half of the internet'.
gulp.task('scripts', function() {
// Minify and copy all JavaScript (except vendor scripts)
return gulp.src(paths.scripts)
.pipe(coffee())
.pipe(uglify())
.pipe(concat('all.min.js'))
.pipe(gulp.dest('build/js'));
});
%.coffee : %.js
js2coffee $< > $@
%.ugly : %.coffee
uglifyjs $< > $@
build/js/all.min.js : $(UGLY)
cat $^ > $@
PS. I'll happily agree that collecting the list of script files to operate on -- and the list of script files after the transformation (eg $(UGLY)) -- is slightly annoying in make.
Also, what your example looks like voodoo. Could you explain what the parameters do?
On top of that, tools like uglifyjs still run on NPM/Node afaik.
GMake was first released in 1977: http://en.wikipedia.org/wiki/Make_(software)
They've worked on this thing for decades
There are 42,000 issues filed for make. if you could resolve each of these issues in 10 minutes, you'd spend 291 days of frustration. http://savannah.gnu.org/bugs/?group=make
kids these days.
You mean like this?
scripts: ${SCRIPTS}
cat $^ | coffee -sc | uglifyjs -cm > build/js/all.min.js
Your example proves the opposite of the point you're trying to make. Starting from your example and trying to compress it with a DSL, you literally couldn't do better than Make syntax: the "gulp.task('" part is implicit, the "', function() { return gulp.src(" part is a single character (':'), every ").pipe(" is a single character ('|'), and ").pipe(gulp.dest('...'))" is a single character ('>').
I submit that's impossible, simply because this stuff took 2 years to evolve...
Before and during the entirety of those 2 years Make has been a better tool, for those of us JS coders who didn't dismiss it offhand as being always thousands of lines and only for dinosaur C coders.
Make has many problems, but taking thousands of lines to simply pipe together commands has never been one of them. Having to write .pipe() where in shell you could just do '|' has never been one of them, either.
The single most valuable thing Make has going for it is that the primitive is a Unix pipeline. Anything that can be built with tools you can invoke from your shell can be built with Make, and the language of action is about as universal as it gets for Unix-like systems.
Yes, the dependency syntax is somewhat confusing, and it's not obvious how to understand and debug Makefiles at first glance, but the GNU make documentation is decent, and time spent learning the language and the tools (e.g., "make -d") is much better than trying to reinvent the system, especially without understanding it. Every reinvention I've had the displeasure of working with missed some important (if not widespread) use case.
Not that I agree with him. Being able to do some printf debugging (or even use a real debugger!) to troubleshoot issues is a big plus that he doesn't mention.
$ cat Makefile
production:
brunch build --production
test:
brunch build
karma startcomputers are fast enough that this doesn't often bother me anymore, but i've run across some huge Rakefiles that could benefit from a rewrite in Make.
You might like tup[1]. Its killer feature is that it automatically determines file-based dependencies by tracking reads and writes (using a FUSE filesystem). It has an extreme emphasis on correct, repeatable builds, and is very fast. Other stuff:
- does work in parallel, and will let you know if your build isn't parallel safe. (note it is NOT relying on your specification of dependencies: even if you manually specify dependencies, it will tell you if something's wrong based on what it actually observes your dependencies to be)
- tracks changes to the build script and reruns if the commands change.
- cleans up old output files automatically if build rules are removed.
- lets you maintain multiple build variants (say for different architectures, configurations, etc)
- autogenerates .gitignore files for your build output
- very easy to get started, and "Just Works".
- for advanced usage, it is scriptable in Lua.
I've tried every build system out there. For Unix-y file-based build tasks, tup is, by far, the best. I don't know why it isn't more well known.
"This page compares make to tup. This page is a little biased because tup is so fast. How fast? This one time a beam of light was flying through the vacuum of space at the speed of light and then tup went by and was like "Yo beam of light, you need a lift?" cuz tup was going so fast it thought the beam of light had a flat tire and was stuck. True story. Anyway, feel free to run your own comparisons if you don't believe me and my (true) story."
Similar to what lsof, procmon (windows) do.
The huge Rakefiles you've seen could possibly have simply benefited from a rewrite in Rake. Rake has 'file' tasks which implement the file dependencies of 'make' but for some reason most users of Rake seem to ignore them completely.
I've given up trying to educate folks, I just make a note to check in with them, 6 months to a year later, to see if they are still building everything.
CXX=g++
hello: main.o factorial.o hello.o
clean:
rm -rf *o hello
cat <<EOF > @all.do
redo hello
EOF
cat <<EOF > hello.do
o='main.o factorial.o hello.o'
redo-ifchange $o
g++ $o -o $3
EOF
cat <<EOF > default.o.do
redo-ifchange $2.cpp
g++ -c $1 -o $3
EOF
cat <<EOF > @clean.do
rm -rf *o hello
EOF
These are just shell scripts and can be extended as much as necesary. For instance, one can create a dependency on the compiler flags with these changes:
cat <<EOF | install -m 0755 /dev/stdin cc
#!/bin/sh
g++ -c "\$@"
EOF
# sed -i 's/^\(redo-ifchange.\+\)/\1 cc/' *.do
# sed -i 's}g++ -c}./cc}' *.do
And compared to that Makefile, the redo scripts you list don't seem simpler at all. I've seen reasonably compelling arguments for redo, but that wasn't one.
You're right, of course.
> Also, I think you wanted .o, not o.
I would, yes, but I copied the Makefile ;)
Should have been clearer; I meant that redo is simpler (and more reliable) than make.
For simple projects, redo scripts are a bit longer. However, as the projects grow, the redo scripts reach an asymptote whereas Makefiles don't. The only way to reduce the growth in make is to add functions and implicit rules which get ugly real fast.
In general, I found CMake quite useable for my needs, and quite clean. It also required less build system code than redo. CMake fits quite nicely into a (C or C++) project which consists of many binaries and libraries which can depend on each other.
Not quite sure what you mean here. The scripts don't do anything complicated and redo catches errors that could occur.
As for readability, etc, I suppose it's relative. Simple makefiles do read very nicely. Unfortunately, they aren't always simple and hairy makefiles are just horrible to write, read and maintain. I've had no such problems with do scripts.
With make it was easier for me to grasp the idea (or maybe I was simply 20 years younger then).
1) I have cookies enabled. 2) The Eurpoean law is daft, but since you feel you must comply do it in a more user friendly way.
Thanks.
Note that this doesn't exist. Only local law could force sites to implement this, and afaik only the UK had a defunct specification of something like that they did'nt even follow for government pages. So, just the usual law craze.
> Is there a nice JS lib to detect and put up an unobtrusive hover-over popup in the bottom corner or something?
Well, I don't know if this is nice enough as I didn't use it, but it looks ok: http://sitebeam.net/cookieconsent/
I've always liked how rockpapershotgun.com does it...it also uses a sticky-slim footer, but the text reads: "Rock Paper Shotgun uses cookies. For some reason we are now obliged to notify you of this fact. Not that you care"
In your make file example the .o files are just depending on the .cpp files, not the header files they include, the header files those included header files include and the files they include etc etc. This means nothing will be recompiled/relinked if a constant in a header file changes for example! Changed function signatures will give you cryptic linker errors with the standard solution "just try make clean first".
To solve this you can either manually update the make file every time any file changes the files it includes, which almost defeats the purpose of having an automatic build system. Or you can use automatic dependency generation by invoking your compiler with a special flag (-MMD for GCC), and suddenly make isn't as simple anymore as you laid it out to be. In conclusion your build tool must be aware of ALL inclusion rules as your compiler(preprocessor) has, or be given the information somehow. Maybe it's better to just use something designed for your particular toolchain that can come bundled with this knowledge?
It's so bad (specifically due to the way file preprocessing works), that you need to have large parts of a compiler to accurately determine what the dependencies of a source file are.
This is why a decent module system should be the top priority for C++17, though it doesn't look likely so far.
.depend :
gcc -M *.c > .depend
source .depend
Just a nitpick: did you mean `include .depend`?
Your build system is an integral part of your whole program and you want to treat it just like any other code. This means refactoring, this means modularity, this means libraries, this means no copying and pasting... All this is far easier with a system embedded in your main language than in Make. You can use your existing tooling, debuggers and frameworks to support your build system. If you're using a typed language, you can use the types to both constrain and guide your build files, making everything safer.
Using an embedded DSL integrates far better with the rest of your ecosystem than relying on Make.
Apart from making the logic of your build system easier to describe and maintain, an embedded DSL also makes arbitrary meta-tasks easier. You might want to monitor random parts of your build process, report to different services, connect to different front-ends (an IRC bot, a CI system...) and maybe even intelligently plug into the features of your main language. Wouldn't it be great to have a make system that's deeply aware of how your server is configured, how your type system works, what your compile-time metaprogramming is doing an so on?
You could just glue together a bunch of disparate scripts with a Make file. Or you could use a DSL and call these services through well-defined, maybe even typed interfaces! No need for serializing and deserializing: you can keep everything inside your system.
Sure, if you're just going to use your DSL as a different syntax for Make, you're not gaining much. But it allows you to do far more in a far better way, while fitting in more naturally with the rest of your code. I'm definitely all for it!
nix tried to solve much of this, but I agree it can't compete with the bazillion other options.
Such a tool should be: - Zero (or few) dependencies. Likely written in plain C (or C++, D, Rust) and compiled to distribute in binary form. - Cross-platform - Support any mix of project languages and build tasks. - Recognizes standard folder hierarchies for popular projects. - Easy enough to learn. Not overly verbose (looking at you, XML). Similar to Make if possible.
Examples of the auto-discovery: It can find "src", "inc", and "lib" directories then look inside and see .h files then make some educated guesses to build the dependency tree of header and source files (even with mix of C and C++). Or it could see a Rails app and figure out to invoke the right Rake commands, perhaps checking for the presence of an asset pipeline etc. Or a Node.js project. It could check for GIT or SVN and make sure any sub-modules have been checked out.
Almost every new project I download starts with a sad, manual, and demoralizing installation of a bunch of third-party stuff that you have to google to find out what's missing. And it's not educational at all, because in a few years all these tools will now be obsolete.
(The best project I ever encountered was the Stripe CTF, which almost always used just one command to install a complete working copy of everything you needed and didn't have. I'm still impressed with that.)
For instance, redux [https://github.com/gyepisam/redux] is written in Go (not compiled for binary distribution, but I could add that), is cross platform, supports any mix of languages and tasks, is very easy to learn.
It uses shell scripts to create targets so everything is scriptable. Stuff like recognizing standard folder hierarchies and auto-discovery can be added with small scripts or tools. It can be as simple as you want or as complex as you need.
Try this next time (only the pertinent lines are included):
SOURCES=$(wildcard $(SRCDIR)/*.erl)
OBJECTS=$(addprefix $(OBJDIR)/, $(notdir $(SOURCES:.erl=.beam)))
DEPS = $(addprefix $(DEPDIR)/, $(notdir $(SOURCES:.erl=.Pbeam))) $(addprefix $(DEPDIR)/, $(notdir $(TEMPLATES:.dtl=.Pbeam)))
-include $(DEPS)
# define a suffix rule for .erl -> .beam
$(OBJDIR)/%.beam: $(SRCDIR)/%.erl | $(OBJDIR)
$(ERLC) $(ERLCFLAGS) -o $(OBJDIR) $<
#see this: http://www.gnu.org/software/make/manual/html_node/Pattern-Match.html
$(DEPDIR)/%.Pbeam: $(SRCDIR)/%.erl | $(DEPDIR)
$(ERLC) -MF $@ -MT $(OBJDIR)/$*.beam $(ERLCFLAGS) $<
#the | pipe operator, defining an order only prerequisite. Meaning
#that the $(OBJDIR) target should be existent (instead of more recent)
#in order to build the current target
$(OBJECTS): | $(OBJDIR)
$(OBJDIR):
test -d $(OBJDIR) || mkdir $(OBJDIR)
$(DEPDIR):
test -d $(DEPDIR) || mkdir $(DEPDIR)
> Generating all the targets to get around this is a nightmare that results in unreadable debug messages and horribly unpredictable call paths.
If you think of Makefiles as a series of call paths, you're going to have a bad time. It's a dependency graph. You define rules for going from one node to the next and let Make figure out how to walk the graph.
You can do implicit rules which requires a very verbose makefile, which is what automake and other make generation tools do. God help you figure out what went wrong.
If you make people go to a directory approach you've now imposed a new structure on their code. One reason for the multitude of packages is each one matches their target community better.
Sadly it's not ready for prime-time yet (early designing stage), so I won't link my highly unfinished project.
If what you are doing in your flavor-of-the-month build tool translates to a roughly equivalent number of lines in Make, then yes, you should probably look at using Make. But the thing is, Make is stupid, it doesn't know a lot. Sometimes that is a good thing, sometimes it is not.
I've written about this before on HN: I mostly program in C++ and when I build my stuff I want a build tool that understands things like header dependencies, optimization levels, shared libraries etc. It's a bonus if my build files are portable.
My point is that these alternative tools often strive to raise the abstraction level and the reason people use them isn't necessarily because they haven't discovered Make.
It reminds me of the jQuery cycle: use jQuery for everything -> decide that depending on frameworks is lame -> use "vanilla JS" for everything -> realize this requires polyfills and various other inconvenient, inelegant things -> either go back to using jQuery, or gain a much deeper understanding as to why everyone uses it.
Make is not an amazing (abit slightly bloated) meta tool that solves all your problems on all platforms (abit slowly).
Make is vanilla javascript, along with all the bumps and hassles of not working correctly on multiple platforms, having odd obscure syntactic oddities and only kind of supporting various operations in newer versions (which may or may not be available on various platforms).
The newer build tools are trying to do exactly what jquery does, and abstracting away those rough edges for a consistent build behavior with better syntax.
Going back to make is the 'use "vanilla JS" for everything' step in your list above, not the final step.
––
[1] http://en.wikipedia.org/wiki/Reactive_programming
[2] http://en.wikipedia.org/wiki/Functional_reactive_programming
For example:
clean:
rm -rf *o hello
Now, are you sure it'll handle files with spaces in the name? What about dashes, brackets and asterisks? Accents? What if a directory ends in .o? Hidden files?
This specific case may support all of the above. But if it doesn't, what will happen? How long until you notice, and how long still to diagnose the problem?
Just like I prefer static strong typing when developing non-trivial projects, the build system should be more structured. I agree endless reinventing is tiring, but it may have some credit in this case.
It's an easy rule.
Just like I prefer static strong typing...
Maybe you downloaded something and it came with a bracket because that was in the page title. Or you copied a duplicated file and your system helpfully appended " (2)" at the end. Or there was an excel file updated by someone not so technical and this person didn't know they have to strip accents from the words in their native language (possibly losing meaning). Or someone saved their "Untitled Document.txt". Or you needed to include a date in the directory name. Or you are just human and didn't mean to break the build by pressing the biggest button on your keyboard when saving a file.
And remember "break the build" here is not "a red light flashes and you get an email". It means you get unknown behavior throughout the process, including security features and file removal.
Strict rules for source code file names are good because names usually bleed into the language itself. Python file names become identifiers when you import them. Identifiers in turn are strict because parsing is strict, and there are many good reasons for strict parsing in general purpose languages.
Lacking accent support in file names, as some very popular software do, is terrible. Lacking support for spaces is just atrocious.
I love shell, I use it daily for one-off tasks, but I don't think it's a good fit to manage the build system of a project.
For those that are curious which build tools exist for Python, here's an (incomplete) list:
* pyinvoke (https://github.com/pyinvoke) - claims to be the successor of fabric, pretty solid and well-maintained
* fabric (http://www.fabfile.org/) - not actually a build tool but often used as one
* paver (http://paver.github.io/paver/) - no longer actively maintained
* doit (http://pydoit.org/) - one of the few tools that actually support monitoring file states (like the original make)
* disttools (https://docs.python.org/2/distutils/) - not actually a "universal" build tool but intended to distribute Python packages
Documentation can be challenging to find, and it isn't the most actively developed project in the world, but what it does, it does pretty well (including supporting more than python dependencies).
- Started using hand-written Makefiles and autoconf. Then someone wants to build on Windows, in Visual Studio nonetheless. Add manually created VStudio project files to the project. Then someone wants to use Xcode, so add manually created Xcode project files. Now you add files, or even need to change a compiler option. Fix the options in the Makefile, open the VisualStudio project, fix the options there, open the project in Xcode, fix the options there. Depending on the project complexity, this can take hours. The next guy needs to build the project in an older VisualStudio version, but the project files are not backward compatible...
- Next step was to create my own "meta-build-system" in TCL (this was around 1999), which takes a simple descriptions of the project (what files to compile into what targets, and the dependencies between target), and creates Makefiles, VStudio-files and Xcode-files, this worked fine until the target project file formats change (happens with every new VisualStudio version).
- Someone then pointed me to cmake which does exactly that but much better (creates Makefiles, VStudio-, Xcode-projects, etc... from a generic description of the sources, targets and their dependencies), and I'm a fairly happy cmake user since then.
- Recently I started to wrap different cmake configuration (combinations of target platforms, build tools/IDE to use, and compile config (Release, Debug, etc...)) under a single python frontend script, since there can be dozens of those cmake configs for one project (target platforms: iOS, Android, OSX, Linux, Windows, emscripten, Google Native Client; build tools: make, ninja, Xcode, VStudio, Eclipse; compile configs: Debug, Release). But the frontend python script only calls cmake with the right options, nothing complicated.
Of course now I'm sorta locked-in to cmake, and setting up a cmake-based build-system can be complex and challenging as well, but the result is an easy to maintain cross-platform build system which also supports IDEs.
I general I'm having a lot less problems compiling cmake-based projects on my OSX and Windows machines then autoconf+Makefile-based projects.
[edit: formatting]
My own experience is with gyp and ninja which is used by the Chromium team (http://martine.github.io/ninja/) which they use to build Windows, OSX, Linux, Android (and maybe iOS?)
Of course for personal projects I'll probably never notice the speed difference but for bigger ones Ninja is FAST.
Nice to see it mentioned in a context other than "oh god what a mess"... even though, in fairness, many aspects of it are a complete dog's dinner.
That is horrifying.
It is bloated, difficult to read, tends towards duplication. It also doesn't do dependency management all that well, doesn't cache build results (so it does a complete rebuild every time), and is difficult to extend.
Not an Ant fan. I have used both Rake and Gradle successfully, and have been much happier with each. Their scripts tend to be (much) more compact, easier to read, and less prone to duplication.
I also note that google are working on a successor to GYP, GN which targets Ninja http://code.google.com/p/chromium/wiki/gn.
Hindered by the fact I can't add an arbitrary github repo through Gradle? Yes. That seems like it should be solvable though...
I need cookies to read a blog post? Don't think so. Probably not worth the read
Eventually, you grow out of it. There's a lot of build tools, each are better at some things than others. It's not that much grunt work to convert things from one to another (even very large projects). If your build tool is working for you, leave it alone. If it's getting in your way or slowing things down, try another one. Move on.
All I know is for years, decades, I carried around the source to some simplistic make. I hate GNU make, I hate some of the unix makes. I loved the simple make.
The beauty of make is it just spelled out what you needed to do. Every darn time make tried to get clever it just made it worse. It seemed like it would be better and then it was not.
Make is the ultimate less is more. Just use it and be happy.
If you are interested in a comparison of a few interesting build tools, please check out Neil Mitchell's "build system shootout" : https://github.com/ndmitchell/build-shootout . Neil is the author of the `Shake` build system. The shootout compares `Make`, `Ninja`, `Shake`, `tup` and `fabricate`.
Another possibly interesting build tool is `buck`, although it is primarily aimed at java / android development. See http://facebook.github.io/buck/ . There's a little discussion about `gerrit`'s move to `buck` here: http://www.infoq.com/news/2013/10/gerrit-buck .
Here's some questions I'd ask of a build system:
- is it mature?
- which platforms does it support?
- which language ecosystems does it support? (language-agnostic? C/C++? ruby? python? java?)
- does it support parallel builds?
- does it support incremental builds?
- are incremental builds accurate?
- is it primarily file-based?
- how does it decide when build targets are up-to-date, if at all? (e.g. timestamps, md5 hash of content, notification from the operating system)
- does it allow build scripts for different components to be defined across multiple files and handled during the same build?
- does it enforce a particular structure upon your build scripts that makes them more maintainable?
- how does it automatically discover dependencies, if at all? (e.g. parsing source files, asking the compiler, builds instrumented via FUSE/strace)
- how easy is it to debug?
- is it possible to extend in a full-featured programming language?
- does it let you augment the build dependency graph mid-way through execution of a build?
- how simply can it be used with other tools such as your chosen continuous integration server, test framework(s), build artifact caches, etc?
Many of these criteria are completely overkill for trivial build tasks, where you don't really need anything fancy.
I'd like to think people have decided that it's easier to replicate the task part of Makefiles onto their environment as the simpler alternative to making dependency management and various other language-specific tasks available to make.
I have since repented. I find autotools (with occasionally a script of [ruby|python|perl] to handle something that would otherwise be tricky to do in make or m4, which is then called by the makefile) works a treat. Just don't try to do anything tricky in the auto tool files - as I said, boot anything exotic out to a separate tool.
Also, any discussion of build tools without also discussing package management is but half a discussion.
So, yeah, there are too many build-tools. Whatever.
Here's a Makefile example that utilizes fswatch: http://blogs.mpr.org/developer/2014/02/makefile/