Code generation is great. If you have a well understood domain, Lisp macros allows you to construct an effective DSL so that your programmers no longer have to write "lisp code" but to program in the new language. We do this today using functions/methods, but the issue tends to be that besides understanding these functions, we need to learn how to orchestrate them and write boiler code all around them. A good DSL hides all that. The issue as I've observed is that when most people create a DSL, they really forget that they have indeed created a new language and don't document it enough. The consumers of the DSL because they can read lisp code, don't focus and live in the DSL layer but wish to peer below then get terrified when they see the macros. Most developers get terrified when they peer one level below. Have you seen the bytecode generated by python or asm code by C? Do you think those are bad because those are code generation that don't look the way you think they ought to?

Whatever programming language you are using, at some point or at some representation the compiler will start generating code. The difference is that the default code representation of lisp allows it at source level, making writing (and debugging) them a lot easier. An if statement in python generates a shitload of bytecode that the optimizer then does what it pleases to. With lisp you can watch that process by watching the source->source transformations of macros and the optimizer.

Macros allow us to extend the language. It is not just C-like code generation. The loop constructs of common lisp, or the for loops of racket (or my own reimplementation for guile scheme) are macros.

With lisp macros we can express zero cost abstractions that in other languages would have a lot of overhead, and make them feel like a regular language construct (CLOS started like that)

My own racket-like loops for guile generate code that the optimizer can then turn into optimal code: https://bitbucket.org/bjoli/guile-for-loops/overview

The downside being that macro expansion slows down compilation. My workstation can expand about 1000 loop macros/s with guile and a couple of orders of magnitude more using my chez scheme prototype (somewhere above 100k)

Why do you need the meta-capabilities of lisp? Well, language designers are human and can't predict all the features that you might find useful. Take, for example, list comprehensions.

I added some basic ones to common lisp in a couple of minutes.

  (defun read-comprehension (stream char)
    (declare (ignore char))
    (destructuring-bind (expr for var in list &optional if filter-exp)
        (read-delimited-list #\] stream)
      `(loop ,for ,var ,in ,list
          ,@(when filter-exp `(,if ,filter-exp))
          collect ,expr)))

  (set-macro-character #\[ #'read-comprehension)

  (eval (read-from-string "[(+ x 1) for x in '(1 2 3) if (> x 2)]"))
  ;; => (4)

> I much more need a syntax that does not hide what happens

Yes sir!

> (procedure call,

   80febf0:	08 
   80febf1:	c7 44 24 04 01 00 00 	movl   $0x1,0x4(%esp)
   80febf8:	00 
   80febf9:	89 3c 24             	mov    %edi,(%esp)
   80febfc:	89 44 24 0c          	mov    %eax,0xc(%esp)
   80fec00:	e8 cb c8 f4 ff       	call   804b4d0 <__fprintf_chk@plt>

> list indexing or at least "indexing").

Gotcha covered there too!

    80fd7eb:	8d 7c 0e ff          	lea    -0x1(%esi,%ecx,1),%edi

> I much more need a syntax that does not hide what happens (procedure call, list indexing or at least "indexing").

Each of those is actually hiding what really happens. E.g. for a procedure call the runtime compares the arity of the call and the called function, inserts each argument into a data structure of some sort, perhaps grafts together environments or does other things on a language-specific basis, performs a jump of some sort, then the called function takes over, and finally stashes its return value(s) in a data structure of some sort and performs another jump.

Even list indexing in a language like Python is relatively complex, involving a lookup of the list length, bounds-checking &c.

> Code generation is mostly bad, it typically means the problem wasn't thought through, and that there is a lack of clearly defined building blocks.

Proper syntactic abstraction is all about building the right blocks for the problem at hand. At the end of the day, all we have are bits of silicon transferring electron: everything above that is an abstraction. A sufficiently-powerful programming language enables the programmer to develop the abstractions he needs for the problem he has.

Code generation is brilliant. Remember that a compiler is a code generator. Using your own code generator allows you to declare what you want at a high level and then generate bug free code that implements it. It is like having a super fast automatic programmer at your side making you super productive.

>Code generation is mostly bad(...)C structs and enums,

C macros (or "C code generation" if you prefer) is very, very different from Lisp macros.

Lisp macros are built into the language, the whole language itself is designed around this capability.

Macros/Code Generation used correctly is exactly the opposite of what you are saying. Macros/Code Generation used correctly forces you to figure out at a much higher level what it is you want to achieve. And declare it in a formal computer readable way.

Looking at a page about list comprehensions in Python.

  S = {x² : x in {0 ... 9}}
  V = (1, 2, 4, 8, ..., 2¹²)
  M = {x | x in S and x even}

Python (from that page)

  S = [x**2 for x in range(10)]
  V = [2**i for i in range(13)]
  M = [x for x in S if x % 2 == 0]

I did not see 10 or 13 in the original definitions

Perl 6 closest syntax to original

  my \S = ($_² for 0 ... 9);
  my \V = (1, 2, 4, 8 ... 2¹²); # almost identical
  my \M = ($_ if $_ %% 2 for S);

Perl 6 closest syntax to Python

  my \S = [-> \x { x**2 } for ^10];
  my \V = [-> \i { 2**i } for ^13];
  my \M = [-> \x { x if x % 2 == 0 } for S];

Perl 6 more idiomatic

  my \S = (0..9)»²;
  my \V = 1, 2, 4 ... 2¹²;
  my \M = S.grep: * %% 2;

Perl 6 with infinite sequences

  my \S = (0..*).map: *²;
  my \V = 1, 2, 4 ... *;
  my \M = S.grep: * %% 2;

---

Python

  string = 'The quick brown fox jumps over the lazy dog'
  words = string.split()
  stuff = [[w.upper(), w.lower(), len(w)] for w in words]

Perl 6

  my \string = 'The quick brown fox jumps over the lazy dog';
  my \words = string.words;
  my \stuff = [[.uc, .lc, .chars] for words]

I wouldn't say Python's list comprehensions are really that big of a selling point. I mean based on what I've seen to understand it the biggest difference is that feature in Python runs from the inside out.

  noprimes = [j for i in range(2, 8) for j in range(i*2, 50, i)]
                         range(2, 8)
                for i in
                                              range(i*2, 50, i)
                                     for j in
              j
             [                                                 ]

Whereas Perl 6 is either left-to-right

  my \noprimes = (2..^8).map: { |($^i*2, $i*3 ...^ 50).map: { $^j }}
                 (2..^8)
                        .map:
                              { |($^i*2, $i*3 ...^ 50)             }
                                                      .map:
                                                            { $^j }

or right-to-left

  my \noprimes = ({ $^j } for ({ |($^i*2, $i*3 ...^ 50) } for 2..^8))
                                                              2..^8
                                                          for
                               { |($^i*2, $i*3 ...^ 50) }
                              (                                    )
                          for
                  { $^j }
                 (                                                  )

Here are a couple other translations using Set operators

  primes = [x for x in range(2, 50) if x not in noprimes]

  my \primes = (2..^50).grep: * ∉ noprimes;

  my \prime-set = 2..^50 (-) noprimes; # a Set object
  my \primes = prime-set.keys.sort;

Also I'm fairly certain Python doesn't have a way of using them on a Supply concurrently.

  # create a prime supply and act on it in the background
  Supply.interval(1).grep(*.is-prime).act: &say;

  say 'main thread still running';
  # says 'main thread still running' immediately

  # deadlock main thread,
  # otherwise the program would terminate
  await Promise.new;

  # waits 2 seconds from the .act call, says 2
  # waits 1 second , says 3
  # waits 2 seconds, says 5
  # waits 2 seconds, says 7
  # waits 4 seconds, says 11
  # and so on until it is terminated

Basically the only selling point of that feature in Python over say Perl6 is that is at times a bit closer to what a mathematician would normally write.

I'm not a mathematician.

To me it seems a bit verbose and clunky. It probably also only works as a single line, which seems odd in a line oriented programming language. Basically it seems like a little DSL for list generation, sort of like Perl's regexes are a DSL for string operations.

Perhaps you may want to reread the last paragraph from the original article. Maybe in a few years the world will have passed you by. Then again maybe not.