Tutorial

Here we explain in greater details how to use sh.js, and to a lesser extent, how it works.

Before you start

The following tutorial aims people who are at least novice at shell scripting and know their way in node.js. In particular, you should be familiar with:

require

Depending on your installation, there are two ways you can require sh.js.

1. On npm installs

If you installed with npm, simply require sh:

 
 
var sh = require('sh');
sh('echo hello');

2. On git installs

If you installed with git, you need to require a path:

 
 
var sh = require('/path/to/shjs/sh');
sh('echo hello');

Introduction

Examples in this tutorial may have a file name indicating you can run it in the examples directory.

In a terminal, head to the examples directory, and run ./00-hello.example.js

At its core, sh.js launches programs:

 
 
 
// examples/01-ls.example.js

sh('ls');

This spawns a new process that executes the ls program.

That returns a function that you may call:

 
 
 
 
// examples/02-ls_grep.example.js

var f = sh('ls');
f('grep \\.example\\.js$');

By calling it, you tell sh.js that you want ls' output to be used as grep's input. You can pipe like that as many times as you want:

 
 
 
// examples/03-multi_pipe.example.js

sh('echo hello')('cat')('cat')('cat')('cat');

That is the equivalent to the following Bash command:

 
echo hello | cat | cat | cat | cat

Now here's how to redirect standard output to a file:

 
 
 
// examples/04-find_files.example.js

sh('find . -size -100c').file('100c_files');

Bash equivalent:

 
find . -size -100c > 100c_files

Alternatively, there is an .append() method that appends to a file instead of (over)writting it (like >>).

You can also cache the output of a command, and receive it in an callback:

1
2
3
4
5
// examples/05-count_find_results.example.js

sh('cat 100c_files')('wc -l').result(function(count) {
  console.log('found %d file(s) smaller than 100 bytes', Number(count));
});

Bash equivalent:

 
echo found `cat 100c_files | wc -l` file\(s\) smaller than 100 bytes

I hope you can see the pattern here: anywhere piping is syntactically correct, .file(), .append() and .result() are correct too:

1
2
3
4
5
6
7
8
9
10
11
sh('echo hello')('cat');
//              ^^^^^^^ piping is correct

sh('echo hello').file('hello.dump');
//              ^^^^^^ therefore .file() is correct too

sh('echo hello').append('hello.dump');
//              ^^^^^^^^ and so is .append()

sh('echo hello').result(function(output) { console.log(output); });
//              ^^^^^^^^ as well as .result()

Piping stderr is quite similar to stdout, you need to interleave .err:

 
 
 
// examples/06-pipe_err_to_null.example.js

sh('find /var -type s').err.file('/dev/null');

Bash equivalent:

 
find /var -type s 2> /dev/null

That will find socket files and discard errors messages.

Note: see the section on advanced piping to redirect both stdout and stderr.

Quoting and escaping arguments

Typing commands with sh.js is meant to feel almost like a standard shell.

You can quote arguments:

1
2
3
4
5
6
// examples/07-quoting.example.js

sh('echo "hello        world"');

// Output:
// hello        world

You can escape characters:

1
2
3
4
5
6
// examples/08-escaping.example.js

sh('echo hello\\ \\ world \\" \\\' \\\\');

// Output:
// hello  world " ' \

In case quoting or escaping are not satisfying, you may pass an array of arguments instead:

 
sh(['echo', 'hello  world " \' \\']);

Finally, to avoid getting stuck because of a bug, you can access the parser like so:

1
2
3
4
5
6
7
8
// examples/09-parser.example.js

var parser = require('../sh.js')._internal.parser;

console.log(parser.parse('echo hello\\ \\ world \\" \\\' \\\\'));

// Output:
// [ 'echo', 'hello  world', '"', '\'', '\' ]

The .parse() method returns an array with the arguments as it interprets them. If you spot a bug, please report it.

Concurrent vs. sequential commands

sh.js is non-blocking. Every call returns immediately after some minor computations. For instance, consider:

1
2
3
4
5
6
7
8
9
// examples/10-concurrent.example.js

sh('date');
sh('sleep 2');
sh('date');

// Output:
// jeudi 18 novembre 2010, 20:30:54 (UTC+0100)
// jeudi 18 novembre 2010, 20:30:54 (UTC+0100)

Unlike in Bash, both dates are the same because calls to sh.js do not block. Both date commands are run concurrently. This is useful to run tasks in parallel.

Bash equivalent:

 
 
 
date &
sleep 2 &
date &

If you want to run commands sequentially, use .then():

1
2
3
4
5
6
7
8
9
// examples/11-sequential.example.js

sh('date')
.then('sleep 2')
.then('date');

// Output:
// jeudi 18 novembre 2010, 20:33:07 (UTC+0100)
// jeudi 18 novembre 2010, 20:33:09 (UTC+0100)

This does not block the execution of your program, but it schedules the next command only after the previous one exits. Notice dates are not the same, unlike before.

Bash equivalent:

 
 
 
date
sleep 2
date

Concurrent and sequential commands are not mutually exclusive, you can use both:

1
2
3
4
5
6
7
8
9
10
// examples/12-concurrent_sequential.example.js

var s = sh('sleep 1');

s.and('sleep 2').and('date');
s.and('sleep 2').and('date');

// Output:
// jeudi 18 novembre 2010, 20:35:46 (UTC+0100)
// jeudi 18 novembre 2010, 20:35:46 (UTC+0100)

sleep 2 commands are run after the sleep 1 command, so that's sequential. But notice the dates are the same because sleep 2 commands are run concurrently.

Bash equivalent:

 
sleep 1 && ( ( sleep 2 && date ) & ( sleep 2 && date ) & )

Reacting to an exit status with .and() and .or()

Using .and() and .or() methods, you can handle the exit of the previous process depending on its exit status. Take the following example:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
// examples/13-gzip_zeros.example.js

var s = sh('dd if=/dev/zero count=10000 bs=50K')('gzip').file('zeros.gz');

s.then('echo compression stopped');
s.or('echo compression failed');
s.and('echo compression succeeded');

// Output:
// 10000+0 enregistrements lus
// 10000+0 enregistrements écrits
// 512000000 octets (512 MB) copiés, 4,00177 s, 128 MB/s
// compression stopped
// compression succeeded

That copies 500 million null bytes from /dev/zero and gzips them to zeros.gz.

Bash equivalent:

1
2
3
4
5
6
7
dd if=/dev/zero count=10000 bs=50K | gzip > zeros.gz

status=$?

echo compression stopped
test $status || echo compression failed
test $status && echo compression succeeded

.and(), .or() and .then() all accept functions as arguments, so you can run JavaScript instead of a program:

1
2
3
4
5
// examples/14-callback.example.js

sh('sleep 2').and(function() {
  console.log('woke up!');
});

Setting the environment: .cd() and .define()

If you want to run commands in a particular directory, call .cd() before them in a sequence:

1
2
3
4
5
// examples/15-ls_cd_ls.example.js

sh('ls -l')
.and.cd('..')
.and('ls -l');

Pay attention on how to call .cd(). Here we call it as a method of .and.

You may also use it as a method of .or, .then or sh like below:

 
 
 
sh('ls -l').or.cd('..').and('ls -l');
sh('ls -l').then.cd('..').and('ls -l');
sh.cd('..').and('ls -l');

Now if you want to set an environment variable, here's how:

 
 
 
// examples/16-grep_my_var.example.js

sh.define('MY_VAR', 123).and('env')('grep MY_VAR');

You may also set several variables in one call:

1
2
3
4
5
6
7
// examples/17-grep_my_vars.example.js

sh.define({
  'MY_VAR1': 123,
  'MY_VAR2': 'abc'
})
.and('env')('grep MY_VAR');

Finally, to unset a variable, just set it to sh.UNSET:

1
2
3
4
5
// examples/18-unset_my_var.example.js

sh.define('MY_VAR', 123)
.and.define('MY_VAR', sh.UNSET)
.and('env')('grep MY_VAR');

By the way, once you cd to a directory or set environment variable, you can store the shell in a JavaScript variable, and reuse it later:

1
2
3
4
5
6
7
// examples/19-store_shell.example.js

var sh1 = sh.cd('/').and;
var sh2 = sh.cd('/var').and;

sh1('ls');
sh2('ls');

Don't forget the .and however.

There is a restriction in that .cd() or .define() cannot be in a pipeline like so:

 
 
// this doesn't work for now
sh.cd('abcd').file('/dev/null').and('ls -l');

I hope that will work soon.

Advanced piping and redirecting

Note: the main challenge of making sh.js was to find the right syntax. Suggestions are welcome.

On the one hand, a process has three standard streams and you may want to start several processes upon its exit. On the other hand, JavaScript doesn't have lots of idioms to embed such syntax.

We're trying to reproduce the following Bash command (this uses process substitution which is not part of the standard):

1
2
3
4
5
6
7
8
9
10
11
12
13
ls / non_existent_file 
  2> >( sed s/non_existent/NON_EXISTENT/ ) \
  > >( grep etc )

# Output
# etc
# ls: ne peut accéder NON_EXISTENT_file: Aucun fichier ou dossier de ce type

# Note: everyone is more familiar with the following, but for the purpose of
# writing shell scripts in JavaScript, the syntax poses the same problems
# as above:

ls / non_existent_file > out_stream 2> err_stream

So I've found four ways that I believe do make sense, three of which are implemented, the other one being removed from the code:

1. Chaining

This syntax was removed. It looked like:

 
 
 
 
// This was removed, it won't work
sh('ls / non_existent_file')
  .pipe('grep etc')
  .err('sed s/non_existent/NON_EXISTENT/');

The reason this was removed is I found it too complicated when there is a lot of recursion. It felt awkward because .pipe() and .err() break easily if they are not properly ordered.

2. Variables

By declaring a variable, you can make several method calls on the first command:

1
2
3
4
5
6
// examples/20-pipes_with_variables.example.js

var l = sh('ls / non_existent_file');

l('grep etc');
l.err('sed s/non_existent/NON_EXISTENT/');

Pros:

Cons:

3. Closures

Passing a closure after the command string will run it and identify the piping:

1
2
3
4
5
6
// examples/21-pipes_in_closures.example.js

sh('ls / non_existent_file', function() {
  this.out('grep etc');
  this.err('sed s/non_existent/NON_EXISTENT/');
});

Pros:

Cons:

4. Arguments

In this syntax, you use one argument for each stream to tell the command how you want to pipe it

1
2
3
4
5
6
// examples/22-branch_arguments.example.js

sh('ls / non_existent_file',
  sh.out('grep etc'),
  sh.err('sed s/non_existent/NON_EXISTENT/')
);

Pros:

Cons:

Putting everything together

Of course, the idioms above are not mutally exclusive.

1
2
3
4
5
6
7
8
9
10
11
12
13
// examples/23-advanced.example.js

var l = sh('ls / non_existent_file',
  sh.out('grep etc'),
  sh.and('echo ls succeeded'),
  function() {
    this.err.file('/dev/null');
    this.or('echo ls failed');
    this.then('echo ls finished (1)');
  }
);

l.then('echo ls finished (2)');

Here we combine:

Random tricks

Output Only

If you want to redirect stderr to stdout, like with &> or |& in traditional shells, pass sh.OO after the command:

 
sh('./configure', sh.OO).file('output+errors');

That's double-capital-O, as in "Output Only".

sh.OO must come before a closure if you use one:

 
 
 
 
sh('./configure', sh.OO, function(c) {
  c.and('echo configure succeeded');
  c.or('echo configure failed');
}).file('output+errors');

Define environment

If you want a clean environment, use sh.ENV:

1
2
3
4
5
6
7
8
9
// examples/24-clean-env.example.js

sh.define(sh.ENV, {
  'MY_VAR': 123
})
.and('env');

// Output: the environment is empty except for the variable we just defined
// MY_VAR=123

Conclusion

That tutorial covered mostly everything. This is a work in progress. Every piece of feedback will be greatly appreciated (use my email in git log).