> There is no other way. BPipe is based on the idea of a DAG. You just don't see it.

So if you think Bpipe uses a DAG, then I wonder how you would think it deals with:

  run { fix_names + fix_names + fix_names }

In terms of the pipeline stages that run this is cyclic, so it cannot be a DAG. On the other hand the files created do usually form a DAG dependency relationship, but even there, in the most general case, it's not at all impossible in an imperative pipeline to read a file in and write the same file out again in modified form (or more likely, to modify it in place), so the file depends on itself - another non-DAG relationship. I'm sure you'll object to this in a purist sense, and tell me it is a horribly broken idea, but as a practising bioinformatician, when I have a 10TB file and modifying it in place will save me hours and huge amounts of space, I'm much more interested in getting my job done than being pure about things.

I think you're right that we're at diminishing returns here, and I'm sorry I've frustrated you. We're trying to bite off more than we can chew in a forum like this.

I wish you all the best with Drake and I'll definitely check it out down the track (when it supports parallelism, since that's too important to me right now). For now, though, I don't intend to read / respond to any more replies in this thread.