Fluent Bit Stream Processor: CREATE STREAM

by edsiper 5 years ago

GNU/Linux ◆ xterm-256color ◆ bash 2985 views

The following demo shows how to create streams of data based on results from previous queries.

The recently introduced CREATE STREAM statement allows to create a stream of data based on the results of a query from a previous stream, the example takes an input file with apache logs in JSON format and then 3 queries do stream processing:

[STREAM_TASK]
     Name  res1
     Exec  CREATE STREAM res1 AS SELECT host, code AS http_status, size FROM STREAM:apache;

[STREAM_TASK]
     Name  res2
     Exec  CREATE STREAM res2 AS SELECT * FROM STREAM:res1;

[STREAM_TASK]
     Name  res3
     Exec  CREATE STREAM res3 AS SELECT COUNT(http_status), avg(size)  FROM STREAM:res2;

Note that in the last stream task (res3) we are using the aggregation functions COUNT() and AVG().

More by edsiper

Fluent Bit Stream Processor: conditionals and quering sub-maps / nested keys 01:17

by edsiper 4 years ago

Fluent Bit + HTTP input PoC 01:02

by edsiper 3 years ago

node_exporter_metrics + stdout 00:31

by edsiper 2 years ago

Fluent Bit + Golang plugin 01:01

by edsiper 7 years ago

See all

You can download this recording in asciicast v2 format, as a .cast file.

Download

Replay in terminal

You can replay the downloaded recording in your terminal using the asciinema play command:

asciinema play 229260.cast

If you don't have asciinema CLI installed then see installation instructions.

Use with stand-alone player on your website

Download asciinema player from the releases page (you only need .js and .css file), then use it like this:

<!DOCTYPE html>
<html>
<head>
  <link rel="stylesheet" type="text/css" href="asciinema-player.css" />
</head>
<body>
  <div id="player"></div>
  <script src="asciinema-player.min.js"></script>
  <script>
    AsciinemaPlayer.create(
      '/assets/229260.cast',
      document.getElementById('player'),
      { cols: 154, rows: 39 }
    );
  </script>
</body>
</html>

See asciinema player quick-start guide for full usage instructions.

While this site doesn't offer GIF conversion at the moment, you can still do it yourself with the help of asciinema GIF generator utility - agg.

Once you have it installed run the following command to create GIF file:

agg https://asciinema.org/a/229260 229260.gif

Or, if you already downloaded the recording file:

agg 229260.cast 229260.gif

Check agg --help for all available options. You can change font family and size, select color theme, adjust speed and more.

See agg manual for full usage instructions.