Using ack and regex to convert unstructured NASA GISS temperature data to CSV

by dann 7 years ago

macOS ◆ xterm-256color ◆ bash 2817 views

Simple exercise in text processing/wrangling with a grep-like (ack) and regular expressions.

Context: http://2017.compciv.org/syllabus/assignments/homework/contrived-cli-data-crunching.html#nasa-annual-global-temperature-averages

Original data URL: https://data.giss.nasa.gov/gistemp/graphs_v3/Fig.A.txt

Test data URL: http://stash.compciv.org/2017/nasa-gistemp-sample.txt

Ack: beyondgrep.com/documentation/ack-2.14-man.html

Regex basics:

optionality with ? http://www.regular-expressions.info/alternation.html
limited repetition with {n} http://www.regular-expressions.info/repeat.html
one-or-more matching with + http://www.regular-expressions.info/repeat.html
Capturing groups with ( ) http://www.regular-expressions.info/brackets.html

Note that I throw in an extra ack for clarity sake, to separate the captured-group matching from the pattern matching. To do everything with one ack:

   curl -s http://stash.compciv.org/2017/nasa-gistemp-sample.txt  \
          |  ack '(\d{4}) +(-?\d+\.\d+)' --output '$1,$2'

More by dann

untitled 00:05

by dann 7 years ago

Downloading and analyzing Alexa's top 1,000,000 site rankings as bulk data 00:35

by dann 8 years ago

ignore this blah blah 00:30

by dann 7 years ago

Using csvkit (csvgrep, csvcut, csvsort, csvstat) to quickly search and analyze 260K (60MB) Chicago crime records 03:21

by dann 5 years ago

See all

You can download this recording in asciicast v1 format, as a .json file.

Download

Replay in terminal

You can replay the downloaded recording in your terminal using the asciinema play command:

asciinema play 100135.json

If you don't have asciinema CLI installed then see installation instructions.

Use with stand-alone player on your website

Download asciinema player from the releases page (you only need .js and .css file), then use it like this:

<!DOCTYPE html>
<html>
<head>
  <link rel="stylesheet" type="text/css" href="asciinema-player.css" />
</head>
<body>
  <div id="player"></div>
  <script src="asciinema-player.min.js"></script>
  <script>
    AsciinemaPlayer.create(
      '/assets/100135.json',
      document.getElementById('player'),
      { cols: 116, rows: 22 }
    );
  </script>
</body>
</html>

See asciinema player quick-start guide for full usage instructions.

While this site doesn't offer GIF conversion at the moment, you can still do it yourself with the help of asciinema GIF generator utility - agg.

Once you have it installed run the following command to create GIF file:

agg https://asciinema.org/a/100135 100135.gif

Or, if you already downloaded the recording file:

agg 100135.json 100135.gif

Check agg --help for all available options. You can change font family and size, select color theme, adjust speed and more.

See agg manual for full usage instructions.