Schemesh: Fusion between Unix shell and Lisp REPL (opens in new tab)

The docs are terrible, and I've always meant to go back and improve them... but it's never quite been a priority and I've never actually summoned the motivation to do it...

But it's always nice to hear when someone uses it and likes it despite that!

> 3. from what I understand, shell syntax is available only at REPL top level. Once you switch to Lisp syntax with `(`, you can return to shell syntax only with `)`. Thus means you cannot embed shell syntax inside Lisp syntax, i.e. you cannot do `(define j {find -type f | less})`

It's possible I misunderstand what you mean because I'm not sure what piping to less is supposed to accomplish here, but this is not true. The following program works just fine:

    #lang rash

    (require racket/port
             racket/string)

    (define (echo!!! message)
      (define output {echo $message |> port->string |> string-trim})
      (string-append output "!!!"))

    (echo!!! "Hello")

Yes, Rash has a variety of limitations. Let me give some more context to these:

>1. no job control

Racket is missing a feature in its rktio library needed to do job control with its process API, which Rash uses. At one point I added one or two other minor features needed for job control, but I ran out of steam and never finished the final one. It's a small feature, even, though now I don't remember much of the context. I hope I wrote enough notes to go back and finish this some day.

>2. multi-line editing is limited

I always intended to write a nice line editor that would do this properly. But, again, I never got around to it. I would still like to, and I will probably take a serious look at your line editor some time.

The design was intended as something to use interactively as well as for scripting. But since I never improved the line editing situation, even I only use it for scripting. After documentation issues, this is the most pressing thing that I would fix.

>3. from what I understand, shell syntax is available only at REPL top level. Once you switch to Lisp syntax with `(`, you can return to shell syntax only with `)`. Thus means you cannot embed shell syntax inside Lisp syntax, i.e. you cannot do `(define j {find -type f | less})`

As mentioned is not correct, you can recursively switch between shell and lisp.

>4. shell commands are Lisp functions, not Lisp objects. Inspecting and redirecting them after they have been created is difficult

This one is a design flaw. I've meant to go back and fix it (eg. just retrofitting a new pipe operator that returns the subprocess pipeline segment as an object rather than its ports or outputs), but, of course, haven't gotten around to it.

>5. Rash is written in Racket, which has larger RAM footprint than schemesh running on vanilla Chez Scheme: at startup, ~160MB vs. ~32MB

Yep.

>6. Racket/Rash support for multi-language at REPL is limited: once you do `#lang racket`, you cannot go back to `#lang rash`

So actually `#lang` is not supported at all in the REPL. It's neither supported in the Racket REPL nor the rash repl. In practice, what `#lang` does is (1) set the reader for a module, and (2) set the base import for the module, IE what symbol definitions are available. With the REPL you have to do this more manually. The repl in Racket is sort of second class in various ways, in part due to the “the top level is hopeless” problems for macros. (Search for that phrase and you can find many issues with repls and macros discussed over the years in the Racket mailing list.) Defining a new `#lang` in Racket includes various pieces about setting up modules specifically, and since the top level repl is not a module, it would need some different support that is currently not there, and would need to be retrofitted for various `#lang`s. But you can start a repl with an arbitrary reader, and use `eval` with arbitrary modules loaded or symbols defined. My intention with a rash line editor would also have been to make some infrastructure for better language-specific repls in racket generally. But, well, obviously I never actually did it. If I do make time for rash repl improvements in the near future, it will just as likely be ways for using it more nicely with emacs rather than actually writing a new line editor... we'll see.

I'm always sad when I think about how I've left Rash to languish. In grad school I was always stressed about publication (which I ultimately did poorly at), which sapped a lot of my desire and energy to actually get into the code and make improvements. Since graduating and going into industry, and with kids, I've rarely felt like I have the time or energy after all of my other responsibilities to spend time on hobby projects. Some day I would like to get back into it, fix its issues, polish it up, document it properly, etc. Alas, maybe some day.

Unfortunately, Eshell does not have job control. Quoting from <https://www.gnu.org/software/emacs/manual/html_mono/eshell.h...>:

> A command invocation followed by an ampersand (&) will be run in the background. Eshell has no job control, so you can not suspend or background the current process, or bring a background process into the foreground. That said, background processes invoked from Eshell can be controlled the same way as any other background process in Emacs.

For things like this, we would have to switch to something like M-x shell or even M-x ansi-term. In Emacs, we have an assortment of shells and terminal implementations. As a long time Emacs user, I know when to use which, so it does not bother me. However, I can imagine how this might feel cumbersome for newer Emacs users.

In fact, this is one of the reasons I think your project is fantastic. It offers some of the Eshell-like experience, and more, to non-Emacs users, which is very compelling!

I really like how you don’t sacrifice complete command-line first shell feel, and escaping into a sane language with real datastructures is literally one character away.

Rather than the tclsh way of saying “we’ll just make the Lisp seem really shelly” which is a dud to anyone who is not a lisper.

Now, it’d be really cool if schemesh had a TUI library at the maturity level of Ratatui.

So... it sacrifices sub-shell syntax with parentheses being hijacked for Scheme. Have you also lost $(...) shell interpolation as the saner alternative to `...`?

> This is a common question :)

It'd be really great if you could put your answer in the readme. It was the first question that came to my mind when looking at your project.

I'm looking forward to trying out schemesh!

I found another, possibly simpler solution.

The functions (sh-fd-stdin) (sh-fd-stdout) and (sh-fd-stderr) return the integer file descriptors that a schemesh builtin should use to perform I/O.

With them, you can do

  true (lambda () (lisp-expr-writes-to-fd (sh-fd-stdout))) | grep foo | true (lambda () (lisp-expr-reads-from-fd (sh-fd-stdin)))

It should work :)

I was thinking of a Lisp/Scheme-like frankenshell for a while. A REPL language (and especially a shell) should focus on ergonomics first - we're commanding the computer to do stuff here and now, not writing elaborate programs (usually).

In my opinion, the outmost parens (even when invoking Lisp functions), as well as all the elaborate glue function names, kinda kill it for interactive use. If you think about it, it's leaking the implementation details into the syntax, and makes for poor idioms. Not very Lispy.

My idea is something like:

    >>> + 1 2 3
    6

(And you would never know if it's /bin/+ or (define (+ ...)))

    >>> seq 1 10 | sum

Let's assume seq is an executable, and sum is a Scheme function. Each "token" seq produces (by default delimited by all whitespace, maybe you could override the rules for a local context, parameterize?) is buffered by the shell, and at the end the whole thing turned into a list of strings. The result is passed to sum as a parameter. (Of course this would break if sum expects a list of integers, but it could also parse the strings as it goes.)

The other way around would also work. If seq produces a list of integers, it's turned into a list of strings and fed into sum as input lines.

The shell could scan $PATH and create a simple function wrapper for each executable.

Now to avoid unnecessary buffering or type conversion, a typed variant of Scheme could be used, possibly with multiple dispatch (per argument/return type). E.g. if the next function in the pipeline accepts an input port or a lazy string iterator, the preceding shell command wrapper could return an output port.

The tricky case with syntax is what to do with tokens like "-9", "3.14", etc. The lexer could store both the parsed value (if it is valid), and the original string. Depending on the context, it could be resolved to either, but retain strong (dynamic) typing when interacting with a Scheme function, so "3.14.15" wouldn't work if a typed function only accepts numbers.

Reminds me of Tcl a bit.