Vincent Bernathttp://www.luffy.cx/en/blog/atom.xml/2024-01-04T11:18:21ZNon-interactive SSH password authenticationVincent Bernat2023-11-05T17:25:26Zhttp://www.luffy.cx/en/blog/2023-sshpass-without-sshpass.html
<p>SSH offers several forms of authentication, such as <strong>passwords</strong> and
<strong>public keys</strong>. The latter are considered more secure. However, password
authentication remains prevalent, particularly with network equipments.<sup id="fnref-why"><a class="footnote-ref" href="#fn-why">1</a></sup></p>
<p>A classic solution to avoid typing a password for each connection is
<a href="https://sourceforge.net/projects/sshpass/" title="sshpass: Non-interactive ssh password auth">sshpass</a>, or its more correct variant <a href="https://github.com/clarkwang/passh">passh</a>. Here is a wrapper for <em>Zsh</em>,
getting the password from <a href="https://www.passwordstore.org/" title="The standard UNIX password manager">pass</a>, a simple password manager:<sup id="fnref-security"><a class="footnote-ref" href="#fn-security">2</a></sup></p>
<div class="language-bash codehilite"><pre><span/>pssh<span class="o">()</span><span class="w"> </span><span class="o">{</span>
<span class="w"> </span>passh<span class="w"> </span>-p<span class="w"> </span><<span class="o">(</span>pass<span class="w"> </span>show<span class="w"> </span>network/ssh/password<span class="w"> </span><span class="p">|</span><span class="w"> </span>head<span class="w"> </span>-1<span class="o">)</span><span class="w"> </span>ssh<span class="w"> </span><span class="s2">"</span><span class="nv">$@</span><span class="s2">"</span>
<span class="o">}</span>
compdef<span class="w"> </span><span class="nv">pssh</span><span class="o">=</span>ssh
</pre></div>
<p>This approach is a bit brittle as it requires to parse the output of the <code>ssh</code>
command to look for a password prompt. Moreover, if no password is required, the
password manager is still invoked. Since <a href="https://www.openssh.com/txt/release-8.4" title="OpenSSH 8.4 release notes">OpenSSH 8.4</a>, we can use
<code>SSH_ASKPASS</code> and <code>SSH_ASKPASS_REQUIRE</code> instead:</p>
<div class="language-bash codehilite"><pre><span/>ssh<span class="o">()</span><span class="w"> </span><span class="o">{</span>
<span class="w"> </span><span class="nb">set</span><span class="w"> </span>-o<span class="w"> </span>localoptions<span class="w"> </span>-o<span class="w"> </span>localtraps
<span class="w"> </span><span class="nb">local</span><span class="w"> </span><span class="nv">passname</span><span class="o">=</span>network/ssh/password
<span class="w"> </span><span class="nb">local</span><span class="w"> </span><span class="nv">helper</span><span class="o">=</span><span class="k">$(</span>mktemp<span class="k">)</span>
<span class="w"> </span><span class="nb">trap</span><span class="w"> </span><span class="s2">"command rm -f </span><span class="nv">$helper</span><span class="s2">"</span><span class="w"> </span>EXIT<span class="w"> </span>INT
<span class="w"> </span>><span class="w"> </span><span class="nv">$helper</span><span class="w"> </span><span class="s"><<EOF</span>
<span class="s">#!$SHELL</span>
<span class="s">pass show $passname | head -1</span>
<span class="s">EOF</span>
<span class="w"> </span>chmod<span class="w"> </span>u+x<span class="w"> </span><span class="nv">$helper</span>
<span class="w"> </span><span class="nv">SSH_ASKPASS</span><span class="o">=</span><span class="nv">$helper</span><span class="w"> </span><span class="nv">SSH_ASKPASS_REQUIRE</span><span class="o">=</span>force<span class="w"> </span><span class="nb">command</span><span class="w"> </span>ssh<span class="w"> </span><span class="s2">"</span><span class="nv">$@</span><span class="s2">"</span>
<span class="o">}</span>
</pre></div>
<p>If the password is incorrect, we can display a prompt on the second
tentative:</p>
<div class="language-bash codehilite"><pre><span/>ssh<span class="o">()</span><span class="w"> </span><span class="o">{</span>
<span class="w"> </span><span class="nb">set</span><span class="w"> </span>-o<span class="w"> </span>localoptions<span class="w"> </span>-o<span class="w"> </span>localtraps
<span class="w"> </span><span class="nb">local</span><span class="w"> </span><span class="nv">passname</span><span class="o">=</span>network/ssh/password
<span class="w"> </span><span class="nb">local</span><span class="w"> </span><span class="nv">helper</span><span class="o">=</span><span class="k">$(</span>mktemp<span class="k">)</span>
<span class="w"> </span><span class="nb">trap</span><span class="w"> </span><span class="s2">"command rm -f </span><span class="nv">$helper</span><span class="s2">"</span><span class="w"> </span>EXIT<span class="w"> </span>INT
<span class="w"> </span>><span class="w"> </span><span class="nv">$helper</span><span class="w"> </span><span class="s"><<EOF</span>
<span class="s">#!$SHELL</span>
<span class="s">if [ -k $helper ]; then</span>
<span class="s"> {</span>
<span class="s"> oldtty=\$(stty -g)</span>
<span class="s"> trap 'stty \$oldtty < /dev/tty 2> /dev/null' EXIT INT TERM HUP</span>
<span class="s"> stty -echo</span>
<span class="s"> print "\rpassword: "</span>
<span class="s"> read password</span>
<span class="s"> printf "\n"</span>
<span class="s"> } > /dev/tty < /dev/tty</span>
<span class="s"> printf "%s" "\$password"</span>
<span class="s">else</span>
<span class="s"> pass show $passname | head -1</span>
<span class="s"> chmod +t $helper</span>
<span class="s">fi</span>
<span class="s">EOF</span>
<span class="w"> </span>chmod<span class="w"> </span>u+x<span class="w"> </span><span class="nv">$helper</span>
<span class="w"> </span><span class="nv">SSH_ASKPASS</span><span class="o">=</span><span class="nv">$helper</span><span class="w"> </span><span class="nv">SSH_ASKPASS_REQUIRE</span><span class="o">=</span>force<span class="w"> </span><span class="nb">command</span><span class="w"> </span>ssh<span class="w"> </span><span class="s2">"</span><span class="nv">$@</span><span class="s2">"</span>
<span class="o">}</span>
</pre></div>
<p>A possible improvement is to use a different password entry depending on the
remote host:<sup id="fnref-zsh"><a class="footnote-ref" href="#fn-zsh">3</a></sup></p>
<div class="language-bash codehilite"><pre><span/>ssh<span class="o">()</span><span class="w"> </span><span class="o">{</span>
<span class="w"> </span><span class="c1"># Grab login information</span>
<span class="w"> </span><span class="nb">local</span><span class="w"> </span>-A<span class="w"> </span>details
<span class="w"> </span><span class="nv">details</span><span class="o">=(</span><span class="si">${</span><span class="p">=</span><span class="si">${</span><span class="p">(M)</span><span class="si">${</span><span class="k">:-</span><span class="s2">"</span><span class="si">${</span><span class="p">(@f)</span><span class="k">$(</span><span class="nb">command</span><span class="w"> </span>ssh<span class="w"> </span>-G<span class="w"> </span><span class="s2">"</span><span class="nv">$@</span><span class="s2">"</span><span class="w"> </span><span class="m">2</span>>/dev/null<span class="k">)</span><span class="si">}</span><span class="s2">"</span><span class="si">}</span><span class="p">:#(host|hostname|user) *</span><span class="si">}}</span><span class="o">)</span>
<span class="w"> </span><span class="nb">local</span><span class="w"> </span><span class="nv">remote</span><span class="o">=</span><span class="si">${</span><span class="nv">details</span><span class="p">[host]</span><span class="k">:-</span><span class="nv">details</span><span class="p">[hostname]</span><span class="si">}</span>
<span class="w"> </span><span class="nb">local</span><span class="w"> </span><span class="nv">login</span><span class="o">=</span><span class="si">${</span><span class="nv">details</span><span class="p">[user]</span><span class="si">}</span>@<span class="si">${</span><span class="nv">remote</span><span class="si">}</span>
<span class="w"> </span><span class="c1"># Get password name</span>
<span class="w"> </span><span class="nb">local</span><span class="w"> </span>passname
<span class="w"> </span><span class="k">case</span><span class="w"> </span><span class="s2">"</span><span class="nv">$login</span><span class="s2">"</span><span class="w"> </span><span class="k">in</span>
<span class="w"> </span>admin@*.example.net<span class="o">)</span><span class="w"> </span><span class="nv">passname</span><span class="o">=</span>company1/ssh/admin<span class="w"> </span><span class="p">;;</span>
<span class="w"> </span>bernat@*.example.net<span class="o">)</span><span class="w"> </span><span class="nv">passname</span><span class="o">=</span>company1/ssh/bernat<span class="w"> </span><span class="p">;;</span>
<span class="w"> </span>backup@*.example.net<span class="o">)</span><span class="w"> </span><span class="nv">passname</span><span class="o">=</span>company1/ssh/backup<span class="w"> </span><span class="p">;;</span>
<span class="w"> </span><span class="k">esac</span>
<span class="w"> </span><span class="c1"># No password name? Just use regular SSH</span>
<span class="w"> </span><span class="o">[[</span><span class="w"> </span>-z<span class="w"> </span><span class="nv">$passname</span><span class="w"> </span><span class="o">]]</span><span class="w"> </span><span class="o">&&</span><span class="w"> </span><span class="o">{</span>
<span class="w"> </span><span class="nb">command</span><span class="w"> </span>ssh<span class="w"> </span><span class="s2">"</span><span class="nv">$@</span><span class="s2">"</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nv">$?</span>
<span class="w"> </span><span class="o">}</span>
<span class="w"> </span><span class="c1"># Invoke SSH with the helper for SSH_ASKPASS</span>
<span class="w"> </span><span class="c1"># […]</span>
<span class="o">}</span>
</pre></div>
<p>It is also possible to make <code>scp</code> invoke our custom <code>ssh</code> function:</p>
<div class="language-bash codehilite"><pre><span/>scp<span class="o">()</span><span class="w"> </span><span class="o">{</span>
<span class="w"> </span><span class="nb">set</span><span class="w"> </span>-o<span class="w"> </span>localoptions<span class="w"> </span>-o<span class="w"> </span>localtraps
<span class="w"> </span><span class="nb">local</span><span class="w"> </span><span class="nv">helper</span><span class="o">=</span><span class="k">$(</span>mktemp<span class="k">)</span>
<span class="w"> </span><span class="nb">trap</span><span class="w"> </span><span class="s2">"command rm -f </span><span class="nv">$helper</span><span class="s2">"</span><span class="w"> </span>EXIT<span class="w"> </span>INT
<span class="w"> </span>><span class="w"> </span><span class="nv">$helper</span><span class="w"> </span><span class="s"><<EOF </span>
<span class="s">#!$SHELL</span>
<span class="s">source ${(%):-%x}</span>
<span class="s">ssh "\$@"</span>
<span class="s">EOF</span>
<span class="w"> </span><span class="nb">command</span><span class="w"> </span>scp<span class="w"> </span>-S<span class="w"> </span><span class="nv">$helper</span><span class="w"> </span><span class="s2">"</span><span class="nv">$@</span><span class="s2">"</span>
<span class="o">}</span>
</pre></div>
<p>For the complete code, have a look at my <a href="https://github.com/vincentbernat/zshrc/blob/master/rc/ssh.zsh">zshrc</a>. As an alternative, you can
put the <code>ssh()</code> function body into its own script file and replace <code>command ssh</code>
with <code>/usr/bin/ssh</code> to avoid an unwanted recursive call. In this case, the
<code>scp()</code> function is not needed anymore.</p>
<div class="admonition">
<p class="admonition-title">Update (2023-12)</p>
<p>This post was heavily discussed on <a href="https://news.ycombinator.com/item?id=38762214">Hacker News</a>.</p>
</div>
<div class="footnote">
<hr/>
<ol>
<li id="fn-why">
<p>First, some vendors make it <a href="/en/blog/2020-syncing-ssh-keys-iosxr-ansible" title="Syncing SSH keys on Cisco IOS-XR with a custom Ansible module">difficult to associate an SSH key with a
user</a>. Then, many vendors do not support certificate-based
authentication, making it difficult to scale. Finally, interactions between
public-key authentication and finer-grained authorization methods like
TACACS+ and Radius are still uncharted territory. <a class="footnote-backref" href="#fnref-why" title="Jump back to footnote 1 in the text">↩︎</a></p>
</li>
<li id="fn-security">
<p>The clear-text password never appears on the command line, in the
environment, or on the disk, making it difficult for a third party without
elevated privileges to capture it. On Linux, <em>Zsh</em> provides the password
through a file descriptor. <a class="footnote-backref" href="#fnref-security" title="Jump back to footnote 2 in the text">↩︎</a></p>
</li>
<li id="fn-zsh">
<p>To decipher the fourth line, you may get help from <code>print -l</code> and the
<a href="https://manpages.debian.org/bookworm/zsh-common/zshexpn.1.en.html" title="zshexpn(1) manual page">zshexpn(1)</a> manual page. <code>details</code> is an <a href="https://scriptingosx.com/2019/11/associative-arrays-in-zsh/" title="Associative arrays in zsh">associative array</a> defined
from an array alternating keys and values. <a class="footnote-backref" href="#fnref-zsh" title="Jump back to footnote 3 in the text">↩︎</a></p>
</li>
</ol>
</div>
DDoS detection and remediation with Akvorado and FlowspecVincent Bernat2023-03-06T07:34:03Zhttp://www.luffy.cx/en/blog/2023-akvorado-ddos-flowspec.html
<p><a href="/en/blog/2022-akvorado-flow-collector" title="Akvorado: a flow collector, enricher, and visualizer">Akvorado</a> collects <a href="https://www.rfc-editor.org/rfc/rfc3176" title="RFC 3176: Specification of the IP Flow Information Export (IPFIX) Protocol for the Exchange of Flow Information">sFlow</a> and <a href="https://www.rfc-editor.org/rfc/rfc7011" title="RFC 7011: Specification of the IP Flow Information Export (IPFIX) Protocol for the Exchange of Flow Information">IPFIX</a> flows, stores them in a
<a href="https://clickhouse.com/" title="ClickHouse: OLAP DBMS">ClickHouse</a> database, and presents them in a web console. Although it lacks
built-in <abbr title="Distributed Denial of Service">DDoS</abbr> detection, it’s possible to create one by crafting custom
<em>ClickHouse</em> queries.</p>
<h1 id="ddos-detection"><abbr title="Distributed Denial of Service">DDoS</abbr> detection<a class="headerlink" href="#ddos-detection" title="Permanent link"></a></h1>
<p>Let’s assume we want to detect <abbr title="Distributed Denial of Service">DDoS</abbr> targeting our customers. As an example, we
consider a <abbr title="Distributed Denial of Service">DDoS</abbr> attack as a collection of flows over <strong>one minute</strong> targeting a
<strong>single customer IP address</strong>, from a <strong>single source port</strong> and matching one
of these conditions:</p>
<ul>
<li>an average bandwidth of 1 Gbps,</li>
<li>an average bandwidth of 200 Mbps when the protocol is UDP,</li>
<li>more than 20 source IP addresses and an average bandwidth of 100 Mbps, or</li>
<li>more than 10 source countries and an average bandwidth of 100 Mbps.</li>
</ul>
<p>Here is the SQL query to detect such attacks over the last 5 minutes:</p>
<div class="language-sql codehilite"><pre><span/><span class="k">SELECT</span><span class="w"> </span><span class="o">*</span>
<span class="k">FROM</span><span class="w"> </span><span class="p">(</span>
<span class="w"> </span><span class="k">SELECT</span>
<span class="w"> </span><span class="n">toStartOfMinute</span><span class="p">(</span><span class="n">TimeReceived</span><span class="p">)</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">TimeReceived</span><span class="p">,</span>
<span class="w"> </span><span class="n">DstAddr</span><span class="p">,</span>
<span class="w"> </span><span class="n">SrcPort</span><span class="p">,</span>
<span class="w"> </span><span class="n">dictGetOrDefault</span><span class="p">(</span><span class="s1">'protocols'</span><span class="p">,</span><span class="w"> </span><span class="s1">'name'</span><span class="p">,</span><span class="w"> </span><span class="n">Proto</span><span class="p">,</span><span class="w"> </span><span class="s1">'???'</span><span class="p">)</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">Proto</span><span class="p">,</span>
<span class="w"> </span><span class="k">SUM</span><span class="p">(((((</span><span class="n">Bytes</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="n">SamplingRate</span><span class="p">)</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="mi">8</span><span class="p">)</span><span class="w"> </span><span class="o">/</span><span class="w"> </span><span class="mi">1000</span><span class="p">)</span><span class="w"> </span><span class="o">/</span><span class="w"> </span><span class="mi">1000</span><span class="p">)</span><span class="w"> </span><span class="o">/</span><span class="w"> </span><span class="mi">1000</span><span class="p">)</span><span class="w"> </span><span class="o">/</span><span class="w"> </span><span class="mi">60</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">Gbps</span><span class="p">,</span>
<span class="w"> </span><span class="n">uniq</span><span class="p">(</span><span class="n">SrcAddr</span><span class="p">)</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">sources</span><span class="p">,</span>
<span class="w"> </span><span class="n">uniq</span><span class="p">(</span><span class="n">SrcCountry</span><span class="p">)</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">countries</span>
<span class="w"> </span><span class="k">FROM</span><span class="w"> </span><span class="n">flows</span>
<span class="w"> </span><span class="k">WHERE</span><span class="w"> </span><span class="n">TimeReceived</span><span class="w"> </span><span class="o">></span><span class="w"> </span><span class="n">now</span><span class="p">()</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="nb">INTERVAL</span><span class="w"> </span><span class="mi">5</span><span class="w"> </span><span class="k">MINUTE</span>
<span class="w"> </span><span class="k">AND</span><span class="w"> </span><span class="n">DstNetRole</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'customers'</span>
<span class="w"> </span><span class="k">GROUP</span><span class="w"> </span><span class="k">BY</span>
<span class="w"> </span><span class="n">TimeReceived</span><span class="p">,</span>
<span class="w"> </span><span class="n">DstAddr</span><span class="p">,</span>
<span class="w"> </span><span class="n">SrcPort</span><span class="p">,</span>
<span class="w"> </span><span class="n">Proto</span>
<span class="p">)</span>
<span class="k">WHERE</span><span class="w"> </span><span class="p">(</span><span class="n">Gbps</span><span class="w"> </span><span class="o">></span><span class="w"> </span><span class="mi">1</span><span class="p">)</span>
<span class="w"> </span><span class="k">OR</span><span class="w"> </span><span class="p">((</span><span class="n">Proto</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'UDP'</span><span class="p">)</span><span class="w"> </span><span class="k">AND</span><span class="w"> </span><span class="p">(</span><span class="n">Gbps</span><span class="w"> </span><span class="o">></span><span class="w"> </span><span class="mi">0</span><span class="p">.</span><span class="mi">2</span><span class="p">))</span><span class="w"> </span>
<span class="w"> </span><span class="k">OR</span><span class="w"> </span><span class="p">((</span><span class="n">sources</span><span class="w"> </span><span class="o">></span><span class="w"> </span><span class="mi">20</span><span class="p">)</span><span class="w"> </span><span class="k">AND</span><span class="w"> </span><span class="p">(</span><span class="n">Gbps</span><span class="w"> </span><span class="o">></span><span class="w"> </span><span class="mi">0</span><span class="p">.</span><span class="mi">1</span><span class="p">))</span><span class="w"> </span>
<span class="w"> </span><span class="k">OR</span><span class="w"> </span><span class="p">((</span><span class="n">countries</span><span class="w"> </span><span class="o">></span><span class="w"> </span><span class="mi">10</span><span class="p">)</span><span class="w"> </span><span class="k">AND</span><span class="w"> </span><span class="p">(</span><span class="n">Gbps</span><span class="w"> </span><span class="o">></span><span class="w"> </span><span class="mi">0</span><span class="p">.</span><span class="mi">1</span><span class="p">))</span>
<span class="k">ORDER</span><span class="w"> </span><span class="k">BY</span>
<span class="w"> </span><span class="n">TimeReceived</span><span class="w"> </span><span class="k">DESC</span><span class="p">,</span>
<span class="w"> </span><span class="n">Gbps</span><span class="w"> </span><span class="k">DESC</span>
</pre></div>
<p>Here is an example output<sup id="fnref-format"><a class="footnote-ref" href="#fn-format">1</a></sup> where two of our users are under attack. One
from what looks like an <a href="https://www.cloudflare.com/learning/ddos/ntp-amplification-ddos-attack/" title="NTP amplification DDoS attack">NTP amplification attack</a>, the other from a <a href="https://www.cloudflare.com/learning/ddos/dns-amplification-ddos-attack/" title="DNS amplification attack">DNS
amplification attack</a>:</p>
<div class="lf-table"><table>
<thead>
<tr>
<th align="right">TimeReceived</th>
<th align="left">DstAddr</th>
<th align="right">SrcPort</th>
<th align="left">Proto</th>
<th align="right">Gbps</th>
<th align="right">sources</th>
<th align="right">countries</th>
</tr>
</thead>
<tbody>
<tr>
<td align="right">2023-02-26 17:44:00</td>
<td align="left"><code>::ffff:203.0.113.206</code></td>
<td align="right">123</td>
<td align="left">UDP</td>
<td align="right">0.102</td>
<td align="right">109</td>
<td align="right">13</td>
</tr>
<tr>
<td align="right">2023-02-26 17:43:00</td>
<td align="left"><code>::ffff:203.0.113.206</code></td>
<td align="right">123</td>
<td align="left">UDP</td>
<td align="right">0.130</td>
<td align="right">133</td>
<td align="right">17</td>
</tr>
<tr>
<td align="right">2023-02-26 17:43:00</td>
<td align="left"><code>::ffff:203.0.113.68</code></td>
<td align="right">53</td>
<td align="left">UDP</td>
<td align="right">0.129</td>
<td align="right">364</td>
<td align="right">63</td>
</tr>
<tr>
<td align="right">2023-02-26 17:43:00</td>
<td align="left"><code>::ffff:203.0.113.206</code></td>
<td align="right">123</td>
<td align="left">UDP</td>
<td align="right">0.113</td>
<td align="right">129</td>
<td align="right">21</td>
</tr>
<tr>
<td align="right">2023-02-26 17:42:00</td>
<td align="left"><code>::ffff:203.0.113.206</code></td>
<td align="right">123</td>
<td align="left">UDP</td>
<td align="right">0.139</td>
<td align="right">50</td>
<td align="right">14</td>
</tr>
<tr>
<td align="right">2023-02-26 17:42:00</td>
<td align="left"><code>::ffff:203.0.113.206</code></td>
<td align="right">123</td>
<td align="left">UDP</td>
<td align="right">0.105</td>
<td align="right">42</td>
<td align="right">14</td>
</tr>
<tr>
<td align="right">2023-02-26 17:40:00</td>
<td align="left"><code>::ffff:203.0.113.68</code></td>
<td align="right">53</td>
<td align="left">UDP</td>
<td align="right">0.121</td>
<td align="right">340</td>
<td align="right">65</td>
</tr>
</tbody>
</table>
</div><h1 id="ddos-remediation"><abbr title="Distributed Denial of Service">DDoS</abbr> remediation<a class="headerlink" href="#ddos-remediation" title="Permanent link"></a></h1>
<p>Once detected, there are at least two ways to stop the attack at the network
level:</p>
<ul>
<li>blackhole the traffic to the targeted user (<em><abbr title="Remote-Triggered Blackhole">RTBH</abbr></em>), or</li>
<li>selectively drop packets matching the attack patterns (<em>Flowspec</em>).</li>
</ul>
<h2 id="traffic-blackhole">Traffic blackhole<a class="headerlink" href="#traffic-blackhole" title="Permanent link"></a></h2>
<p>The easiest method is to sacrifice the attacked user. While this helps the
attacker, this protects your network. It is a method supported by all routers.
You can also offload this protection to many transit providers. This is useful
if the attack volume exceeds your internet capacity.</p>
<p>This works by advertising with <abbr title="Border Gateway Protocol">BGP</abbr> a route to the attacked user with a specific
community. The border router modifies the next hop address of these routes to a
specific IP address configured to forward the traffic to a null interface. <a href="https://www.rfc-editor.org/rfc/rfc7999" title="BLACKHOLE Community">RFC 7999</a> defines <code>65535:666</code> for this purpose. This is known as a
“remote-triggered blackhole” (<abbr title="Remote-Triggered Blackhole">RTBH</abbr>) and is explained in more detail in <a href="https://www.rfc-editor.org/rfc/rfc3882" title="Configuring BGP to Block Denial-of-Service Attacks">RFC 3882</a>.</p>
<p>It is also possible to blackhole the source of the attacks by leveraging
<em>unicast Reverse Path Forwarding</em> (<abbr title="Unicast Reverse Path Forwarding">uRPF</abbr>) from <a href="https://www.rfc-editor.org/rfc/rfc3704" title="Ingress Filtering for Multihomed Networks">RFC 3704</a>, as explained in <a href="https://www.rfc-editor.org/rfc/rfc5635" title="Remote Triggered Black Hole Filtering with Unicast Reverse Path Forwarding (uRPF)">RFC 5635</a>. However, <abbr title="Unicast Reverse Path Forwarding">uRPF</abbr> can be a serious tax on your router resources. See
“<a href="https://xrdocs.io/ncs5500/tutorials/ncs5500-urpf/" title="NCS5500 uRPF: Configuration and Impact on Scale">NCS5500 <abbr title="Unicast Reverse Path Forwarding">uRPF</abbr>: Configuration and Impact on Scale</a>” for an example of the kind
of restrictions you have to expect when enabling <abbr title="Unicast Reverse Path Forwarding">uRPF</abbr>.</p>
<p>On the advertising side, we can use <a href="https://bird.network.cz/" title="The BIRD Internet Routing Daemon">BIRD</a>. Here is a complete configuration
file to allow any router to collect them:</p>
<div class="language-junos codehilite"><pre><span/><span class="k">log</span> stderr all;
<span class="k">router</span> id <span class="m">192.0.2.1</span><span class="p">;</span>
<span class="k">protocol</span> device <span class="p">{</span>
scan time <span class="m">10</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">protocol</span> bgp exporter <span class="p">{</span>
ipv4 <span class="p">{</span>
import none;
export where proto = <span class="s2">"blackhole4"</span><span class="p">;</span>
<span class="p">};</span>
ipv6 <span class="p">{</span>
import none;
export where proto = <span class="s2">"blackhole6"</span><span class="p">;</span>
<span class="p">};</span>
local as <span class="m">64666</span><span class="p">;</span>
<span class="hll"> neighbor range <span class="m">192.0.2.0/24</span> external;
</span> multihop;
dynamic name <span class="s2">"exporter"</span><span class="p">;</span>
dynamic name digits <span class="m">2</span><span class="p">;</span>
graceful restart yes;
graceful restart time <span class="m">0</span><span class="p">;</span>
<span class="hll"> long lived graceful restart yes;
</span><span class="hll"> long lived stale time <span class="m">3600</span><span class="p">;</span> <span class="c c-Singleline"># keep routes for 1 hour!</span>
</span><span class="p">}</span>
<span class="k">protocol</span> static blackhole4 <span class="p">{</span>
ipv4;
<span class="hll"> route <span class="m">203.0.113.206/32</span> blackhole <span class="p">{</span>
</span><span class="hll"> bgp_community.add((65535, <span class="m">666</span>));
</span><span class="hll"> <span class="p">};</span>
</span><span class="hll"> route <span class="m">203.0.113.68/32</span> blackhole <span class="p">{</span>
</span><span class="hll"> bgp_community.add((65535, <span class="m">666</span>));
</span><span class="hll"> <span class="p">};</span>
</span><span class="p">}</span>
<span class="k">protocol</span> static blackhole6 <span class="p">{</span>
ipv6;
<span class="p">}</span>
</pre></div>
<p>We use <a href="/en/blog/2018-bgp-llgr" title="BGP LLGR: robust and reactive BGP sessions"><abbr title="Border Gateway Protocol">BGP</abbr> long-lived graceful restart</a> to ensure routes are kept for
one hour, even if the <abbr title="Border Gateway Protocol">BGP</abbr> connection goes down, notably during maintenance.</p>
<p>On the receiver side, if you have a Cisco router running IOS XR, you can use the
following configuration to blackhole traffic received on the <abbr title="Border Gateway Protocol">BGP</abbr> session. As the
<abbr title="Border Gateway Protocol">BGP</abbr> session is dedicated to this usage, The community is not used, but you can
also forward these routes to your transit providers.</p>
<div class="language-ios codehilite"><pre><span/><span class="nf">router</span> static
<span class="nf">vrf</span> public
<span class="nf">address-family</span> ipv4 unicast
<span class="m">192.0.2.1/32</span> Null0 description <span class="s2">"BGP blackhole"</span>
<span class="c c-Singleline"> !</span>
<span class="nf">address-family</span> ipv6 unicast
<span class="m">2001:db8::1/128</span> Null0 description <span class="s2">"BGP blackhole"</span>
<span class="c c-Singleline"> !</span>
<span class="c c-Singleline"> !</span>
<span class="c c-Singleline">!</span>
<span class="nf">route-policy</span> blackhole_ipv4_in_public
<span class="nf">if</span> destination in (0.0.0.0/0 le <span class="m">31</span>) then
<span class="nf">drop</span>
<span class="nf">endif</span>
<span class="nf">set</span> next-hop <span class="m">192.0.2.1</span>
<span class="nf">done</span>
<span class="nf">end-policy</span>
<span class="c c-Singleline">!</span>
<span class="nf">route-policy</span> blackhole_ipv6_in_public
<span class="nf">if</span> destination in (::/0 le <span class="m">127</span>) then
<span class="nf">drop</span>
<span class="nf">endif</span>
<span class="nf">set</span> next-hop <span class="m">2001:db8::1</span>
<span class="nf">done</span>
<span class="nf">end-policy</span>
<span class="c c-Singleline">!</span>
<span class="nf">router</span> bgp <span class="m">12322</span>
<span class="nf">neighbor-group</span> BLACKHOLE_IPV4_PUBLIC
<span class="nf">remote-as</span> <span class="m">64666</span>
<span class="nf">ebgp-multihop</span> <span class="m">255</span>
<span class="nf">update-source</span> Loopback10
<span class="nf">address-family</span> ipv4 unicast
<span class="nf">maximum-prefix</span> <span class="m">100</span> <span class="m">90</span>
<span class="nf">route-policy</span> blackhole_ipv4_in_public in
<span class="nf">route-policy</span> drop out
<span class="nf">long-lived-graceful-restart</span> stale-time send <span class="m">86400</span> accept <span class="m">86400</span>
<span class="c c-Singleline"> !</span>
<span class="nf">address-family</span> ipv6 unicast
<span class="nf">maximum-prefix</span> <span class="m">100</span> <span class="m">90</span>
<span class="nf">route-policy</span> blackhole_ipv6_in_public in
<span class="nf">route-policy</span> drop out
<span class="nf">long-lived-graceful-restart</span> stale-time send <span class="m">86400</span> accept <span class="m">86400</span>
<span class="c c-Singleline"> !</span>
<span class="c c-Singleline"> !</span>
<span class="nf">vrf</span> public
<span class="nf">neighbor</span> <span class="m">192.0.2.1</span>
<span class="nf">use</span> neighbor-group BLACKHOLE_IPV4_PUBLIC
<span class="nf">description</span> akvorado-1
</pre></div>
<p>When the traffic is blackholed, it is still reported by <em>IPFIX</em> and <em>sFlow</em>.
In <em>Akvorado</em>, use <code>ForwardingStatus >= 128</code> as a filter.</p>
<p>While this method is compatible with all routers, it makes the attack successful
as the target is completely unreachable. If your router supports it, <em>Flowspec</em>
can selectively filter flows to <strong>stop the attack without impacting the
customer</strong>.</p>
<h2 id="flowspec">Flowspec<a class="headerlink" href="#flowspec" title="Permanent link"></a></h2>
<p><em>Flowspec</em> is defined in <a href="https://www.rfc-editor.org/rfc/rfc8955" title="Dissemination of Flow Specification Rules">RFC 8955</a> and enables the transmission of flow
specifications in <abbr title="Border Gateway Protocol">BGP</abbr> sessions. A flow specification is a set of matching
criteria to apply to IP traffic. These criteria include the source and
destination prefix, the IP protocol, the source and destination port, and the
packet length. Each flow specification is associated with an action, encoded as an
extended community: traffic shaping, traffic marking, or redirection.</p>
<p>To announce flow specifications with <em>BIRD</em>, we extend our configuration. The
extended community used shapes the matching traffic to 0 byte per second.</p>
<div class="language-junos codehilite"><pre><span/><span class="k">flow4</span> table flowtab4;
<span class="k">flow6</span> table flowtab6;
<span class="k">protocol</span> bgp exporter <span class="p">{</span>
flow4 <span class="p">{</span>
import none;
export where proto = <span class="s2">"flowspec4"</span><span class="p">;</span>
<span class="p">};</span>
flow6 <span class="p">{</span>
import none;
export where proto = <span class="s2">"flowspec6"</span><span class="p">;</span>
<span class="p">};</span>
<span class="c c-Singleline"># […]</span>
<span class="p">}</span>
<span class="k">protocol</span> static flowspec4 <span class="p">{</span>
flow4;
route flow4 <span class="p">{</span>
dst <span class="m">203.0.113.68/32</span><span class="p">;</span>
sport = <span class="m">53</span><span class="p">;</span>
length >= <span class="m">1476</span> && <= <span class="m">1500</span><span class="p">;</span>
proto = <span class="m">17</span><span class="p">;</span>
<span class="p">}{</span>
bgp_ext_community.add((generic, <span class="m">0</span>x80060000, <span class="m">0</span>x00000000));
<span class="p">};</span>
route flow4 <span class="p">{</span>
dst <span class="m">203.0.113.206/32</span><span class="p">;</span>
sport = <span class="m">123</span><span class="p">;</span>
length = <span class="m">468</span><span class="p">;</span>
proto = <span class="m">17</span><span class="p">;</span>
<span class="p">}{</span>
bgp_ext_community.add((generic, <span class="m">0</span>x80060000, <span class="m">0</span>x00000000));
<span class="p">};</span>
<span class="p">}</span>
<span class="k">protocol</span> static flowspec6 <span class="p">{</span>
flow6;
<span class="p">}</span>
</pre></div>
<p>If you have a Cisco router running IOS XR, the configuration may look like
this:</p>
<div class="language-ios codehilite"><pre><span/><span class="nf">vrf</span> public
<span class="nf">address-family</span> ipv4 flowspec
<span class="nf">address-family</span> ipv6 flowspec
<span class="c c-Singleline">!</span>
<span class="nf">router</span> bgp <span class="m">12322</span>
<span class="nf">address-family</span> vpnv4 flowspec
<span class="nf">address-family</span> vpnv6 flowspec
<span class="nf">neighbor-group</span> FLOWSPEC_IPV4_PUBLIC
<span class="nf">remote-as</span> <span class="m">64666</span>
<span class="nf">ebgp-multihop</span> <span class="m">255</span>
<span class="nf">update-source</span> Loopback10
<span class="nf">address-family</span> ipv4 flowspec
<span class="nf">long-lived-graceful-restart</span> stale-time send <span class="m">86400</span> accept <span class="m">86400</span>
<span class="nf">route-policy</span> accept in
<span class="nf">route-policy</span> drop out
<span class="nf">maximum-prefix</span> <span class="m">100</span> <span class="m">90</span>
<span class="nf">validation</span> disable
<span class="c c-Singleline"> !</span>
<span class="nf">address-family</span> ipv6 flowspec
<span class="nf">long-lived-graceful-restart</span> stale-time send <span class="m">86400</span> accept <span class="m">86400</span>
<span class="nf">route-policy</span> accept in
<span class="nf">route-policy</span> drop out
<span class="nf">maximum-prefix</span> <span class="m">100</span> <span class="m">90</span>
<span class="nf">validation</span> disable
<span class="c c-Singleline"> !</span>
<span class="c c-Singleline"> !</span>
<span class="nf">vrf</span> public
<span class="nf">address-family</span> ipv4 flowspec
<span class="nf">address-family</span> ipv6 flowspec
<span class="nf">neighbor</span> <span class="m">192.0.2.1</span>
<span class="nf">use</span> neighbor-group FLOWSPEC_IPV4_PUBLIC
<span class="nf">description</span> akvorado-1
</pre></div>
<p>Then, you need to enable <em>Flowspec</em> on all interfaces with:</p>
<div class="language-ios codehilite"><pre><span/><span class="nf">flowspec</span>
<span class="nf">vrf</span> public
<span class="nf">address-family</span> ipv4
<span class="nf">local-install</span> interface-all
<span class="c c-Singleline"> !</span>
<span class="nf">address-family</span> ipv6
<span class="nf">local-install</span> interface-all
<span class="c c-Singleline"> !</span>
<span class="c c-Singleline"> !</span>
<span class="c c-Singleline">!</span>
</pre></div>
<p>As with the <abbr title="Remote-Triggered Blackhole">RTBH</abbr> setup, you can filter dropped flows with <code>ForwardingStatus >=
128</code>.</p>
<h1 id="ddos-detection-continued"><abbr title="Distributed Denial of Service">DDoS</abbr> detection (continued)<a class="headerlink" href="#ddos-detection-continued" title="Permanent link"></a></h1>
<p>In the example using <em>Flowspec</em>, the flows were also filtered on the length of the packet:</p>
<div class="language-junos codehilite"><pre><span/><span class="k">route</span> flow4 <span class="p">{</span>
dst <span class="m">203.0.113.68/32</span><span class="p">;</span>
sport = <span class="m">53</span><span class="p">;</span>
length >= <span class="m">1476</span> && <= <span class="m">1500</span><span class="p">;</span>
proto = <span class="m">17</span><span class="p">;</span>
<span class="p">}{</span>
bgp_ext_community.add((generic, <span class="m">0</span>x80060000, <span class="m">0</span>x00000000));
<span class="p">};</span>
</pre></div>
<p>This is an important addition: legitimate DNS requests are smaller than this and
therefore not filtered.<sup id="fnref-dns"><a class="footnote-ref" href="#fn-dns">2</a></sup> With <em>ClickHouse</em>, you can get the 10<sup>th</sup>
and 90<sup>th</sup> percentiles of the packet sizes with <code>quantiles(0.1,
0.9)(Bytes/Packets)</code>.</p>
<p>The last issue we need to tackle is how to optimize the request: it may need
several seconds to collect the data and it is likely to consume substantial
resources from your <em>ClickHouse</em> database. One solution is to create a
materialized view to pre-aggregate results:</p>
<div class="language-sql codehilite"><pre><span/><span class="k">CREATE</span><span class="w"> </span><span class="k">TABLE</span><span class="w"> </span><span class="n">ddos_logs</span><span class="w"> </span><span class="p">(</span>
<span class="w"> </span><span class="n">TimeReceived</span><span class="w"> </span><span class="n">DateTime</span><span class="p">,</span>
<span class="w"> </span><span class="n">DstAddr</span><span class="w"> </span><span class="n">IPv6</span><span class="p">,</span>
<span class="w"> </span><span class="n">Proto</span><span class="w"> </span><span class="n">UInt32</span><span class="p">,</span>
<span class="w"> </span><span class="n">SrcPort</span><span class="w"> </span><span class="n">UInt16</span><span class="p">,</span>
<span class="w"> </span><span class="n">Gbps</span><span class="w"> </span><span class="n">SimpleAggregateFunction</span><span class="p">(</span><span class="k">sum</span><span class="p">,</span><span class="w"> </span><span class="n">Float64</span><span class="p">),</span>
<span class="w"> </span><span class="n">Mpps</span><span class="w"> </span><span class="n">SimpleAggregateFunction</span><span class="p">(</span><span class="k">sum</span><span class="p">,</span><span class="w"> </span><span class="n">Float64</span><span class="p">),</span>
<span class="w"> </span><span class="n">sources</span><span class="w"> </span><span class="n">AggregateFunction</span><span class="p">(</span><span class="n">uniqCombined</span><span class="p">(</span><span class="mi">12</span><span class="p">),</span><span class="w"> </span><span class="n">IPv6</span><span class="p">),</span>
<span class="w"> </span><span class="n">countries</span><span class="w"> </span><span class="n">AggregateFunction</span><span class="p">(</span><span class="n">uniqCombined</span><span class="p">(</span><span class="mi">12</span><span class="p">),</span><span class="w"> </span><span class="n">FixedString</span><span class="p">(</span><span class="mi">2</span><span class="p">)),</span>
<span class="w"> </span><span class="k">size</span><span class="w"> </span><span class="n">AggregateFunction</span><span class="p">(</span><span class="n">quantiles</span><span class="p">(</span><span class="mi">0</span><span class="p">.</span><span class="mi">1</span><span class="p">,</span><span class="w"> </span><span class="mi">0</span><span class="p">.</span><span class="mi">9</span><span class="p">),</span><span class="w"> </span><span class="n">UInt64</span><span class="p">)</span>
<span class="hll"><span class="p">)</span><span class="w"> </span><span class="n">ENGINE</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">SummingMergeTree</span>
</span><span class="n">PARTITION</span><span class="w"> </span><span class="k">BY</span><span class="w"> </span><span class="n">toStartOfHour</span><span class="p">(</span><span class="n">TimeReceived</span><span class="p">)</span>
<span class="k">ORDER</span><span class="w"> </span><span class="k">BY</span><span class="w"> </span><span class="p">(</span><span class="n">TimeReceived</span><span class="p">,</span><span class="w"> </span><span class="n">DstAddr</span><span class="p">,</span><span class="w"> </span><span class="n">Proto</span><span class="p">,</span><span class="w"> </span><span class="n">SrcPort</span><span class="p">)</span>
<span class="n">TTL</span><span class="w"> </span><span class="n">toStartOfHour</span><span class="p">(</span><span class="n">TimeReceived</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="nb">INTERVAL</span><span class="w"> </span><span class="mi">6</span><span class="w"> </span><span class="n">HOUR</span><span class="w"> </span><span class="k">DELETE</span><span class="w"> </span><span class="p">;</span>
<span class="k">CREATE</span><span class="w"> </span><span class="n">MATERIALIZED</span><span class="w"> </span><span class="k">VIEW</span><span class="w"> </span><span class="n">ddos_logs_view</span><span class="w"> </span><span class="k">TO</span><span class="w"> </span><span class="n">ddos_logs</span><span class="w"> </span><span class="k">AS</span>
<span class="w"> </span><span class="k">SELECT</span>
<span class="w"> </span><span class="n">toStartOfMinute</span><span class="p">(</span><span class="n">TimeReceived</span><span class="p">)</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">TimeReceived</span><span class="p">,</span>
<span class="w"> </span><span class="n">DstAddr</span><span class="p">,</span>
<span class="w"> </span><span class="n">Proto</span><span class="p">,</span>
<span class="w"> </span><span class="n">SrcPort</span><span class="p">,</span>
<span class="w"> </span><span class="k">sum</span><span class="p">(((((</span><span class="n">Bytes</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="n">SamplingRate</span><span class="p">)</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="mi">8</span><span class="p">)</span><span class="w"> </span><span class="o">/</span><span class="w"> </span><span class="mi">1000</span><span class="p">)</span><span class="w"> </span><span class="o">/</span><span class="w"> </span><span class="mi">1000</span><span class="p">)</span><span class="w"> </span><span class="o">/</span><span class="w"> </span><span class="mi">1000</span><span class="p">)</span><span class="w"> </span><span class="o">/</span><span class="w"> </span><span class="mi">60</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">Gbps</span><span class="p">,</span>
<span class="w"> </span><span class="k">sum</span><span class="p">(((</span><span class="n">Packets</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="n">SamplingRate</span><span class="p">)</span><span class="w"> </span><span class="o">/</span><span class="w"> </span><span class="mi">1000</span><span class="p">)</span><span class="w"> </span><span class="o">/</span><span class="w"> </span><span class="mi">1000</span><span class="p">)</span><span class="w"> </span><span class="o">/</span><span class="w"> </span><span class="mi">60</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">Mpps</span><span class="p">,</span>
<span class="w"> </span><span class="n">uniqCombinedState</span><span class="p">(</span><span class="mi">12</span><span class="p">)(</span><span class="n">SrcAddr</span><span class="p">)</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">sources</span><span class="p">,</span>
<span class="w"> </span><span class="n">uniqCombinedState</span><span class="p">(</span><span class="mi">12</span><span class="p">)(</span><span class="n">SrcCountry</span><span class="p">)</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">countries</span><span class="p">,</span>
<span class="w"> </span><span class="n">quantilesState</span><span class="p">(</span><span class="mi">0</span><span class="p">.</span><span class="mi">1</span><span class="p">,</span><span class="w"> </span><span class="mi">0</span><span class="p">.</span><span class="mi">9</span><span class="p">)(</span><span class="n">toUInt64</span><span class="p">(</span><span class="n">Bytes</span><span class="o">/</span><span class="n">Packets</span><span class="p">))</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="k">size</span>
<span class="w"> </span><span class="k">FROM</span><span class="w"> </span><span class="n">flows</span>
<span class="w"> </span><span class="k">WHERE</span><span class="w"> </span><span class="n">DstNetRole</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'customers'</span>
<span class="w"> </span><span class="k">GROUP</span><span class="w"> </span><span class="k">BY</span>
<span class="w"> </span><span class="n">TimeReceived</span><span class="p">,</span>
<span class="w"> </span><span class="n">DstAddr</span><span class="p">,</span>
<span class="w"> </span><span class="n">Proto</span><span class="p">,</span>
<span class="w"> </span><span class="n">SrcPort</span>
</pre></div>
<p>The <code>ddos_logs</code> table is using the <code>SummingMergeTree</code> engine. When the table
receives new data, <em>ClickHouse</em> replaces all the rows with the same sorting key,
as defined by the <code>ORDER BY</code> directive, with one row which contains summarized
values using either the <code>sum()</code> function or the explicitly specified aggregate
function (<code>uniqCombined</code> and <code>quantiles</code> in our example).<sup id="fnref-materialized"><a class="footnote-ref" href="#fn-materialized">3</a></sup></p>
<p>Finally, we can modify our initial query with the following one:</p>
<div class="language-sql codehilite"><pre><span/><span class="k">SELECT</span><span class="w"> </span><span class="o">*</span>
<span class="k">FROM</span><span class="w"> </span><span class="p">(</span>
<span class="w"> </span><span class="k">SELECT</span>
<span class="w"> </span><span class="n">TimeReceived</span><span class="p">,</span>
<span class="w"> </span><span class="n">DstAddr</span><span class="p">,</span>
<span class="w"> </span><span class="n">dictGetOrDefault</span><span class="p">(</span><span class="s1">'protocols'</span><span class="p">,</span><span class="w"> </span><span class="s1">'name'</span><span class="p">,</span><span class="w"> </span><span class="n">Proto</span><span class="p">,</span><span class="w"> </span><span class="s1">'???'</span><span class="p">)</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">Proto</span><span class="p">,</span>
<span class="w"> </span><span class="n">SrcPort</span><span class="p">,</span>
<span class="w"> </span><span class="k">sum</span><span class="p">(</span><span class="n">Gbps</span><span class="p">)</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">Gbps</span><span class="p">,</span>
<span class="w"> </span><span class="k">sum</span><span class="p">(</span><span class="n">Mpps</span><span class="p">)</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">Mpps</span><span class="p">,</span>
<span class="w"> </span><span class="n">uniqCombinedMerge</span><span class="p">(</span><span class="mi">12</span><span class="p">)(</span><span class="n">sources</span><span class="p">)</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">sources</span><span class="p">,</span>
<span class="w"> </span><span class="n">uniqCombinedMerge</span><span class="p">(</span><span class="mi">12</span><span class="p">)(</span><span class="n">countries</span><span class="p">)</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">countries</span><span class="p">,</span>
<span class="w"> </span><span class="n">quantilesMerge</span><span class="p">(</span><span class="mi">0</span><span class="p">.</span><span class="mi">1</span><span class="p">,</span><span class="w"> </span><span class="mi">0</span><span class="p">.</span><span class="mi">9</span><span class="p">)(</span><span class="k">size</span><span class="p">)</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="k">size</span>
<span class="w"> </span><span class="k">FROM</span><span class="w"> </span><span class="n">ddos_logs</span>
<span class="w"> </span><span class="k">WHERE</span><span class="w"> </span><span class="n">TimeReceived</span><span class="w"> </span><span class="o">></span><span class="w"> </span><span class="n">now</span><span class="p">()</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="nb">INTERVAL</span><span class="w"> </span><span class="mi">60</span><span class="w"> </span><span class="k">MINUTE</span>
<span class="w"> </span><span class="k">GROUP</span><span class="w"> </span><span class="k">BY</span>
<span class="w"> </span><span class="n">TimeReceived</span><span class="p">,</span>
<span class="w"> </span><span class="n">DstAddr</span><span class="p">,</span>
<span class="w"> </span><span class="n">Proto</span><span class="p">,</span>
<span class="w"> </span><span class="n">SrcPort</span>
<span class="p">)</span>
<span class="k">WHERE</span><span class="w"> </span><span class="p">(</span><span class="n">Gbps</span><span class="w"> </span><span class="o">></span><span class="w"> </span><span class="mi">1</span><span class="p">)</span>
<span class="w"> </span><span class="k">OR</span><span class="w"> </span><span class="p">((</span><span class="n">Proto</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'UDP'</span><span class="p">)</span><span class="w"> </span><span class="k">AND</span><span class="w"> </span><span class="p">(</span><span class="n">Gbps</span><span class="w"> </span><span class="o">></span><span class="w"> </span><span class="mi">0</span><span class="p">.</span><span class="mi">2</span><span class="p">))</span><span class="w"> </span>
<span class="w"> </span><span class="k">OR</span><span class="w"> </span><span class="p">((</span><span class="n">sources</span><span class="w"> </span><span class="o">></span><span class="w"> </span><span class="mi">20</span><span class="p">)</span><span class="w"> </span><span class="k">AND</span><span class="w"> </span><span class="p">(</span><span class="n">Gbps</span><span class="w"> </span><span class="o">></span><span class="w"> </span><span class="mi">0</span><span class="p">.</span><span class="mi">1</span><span class="p">))</span><span class="w"> </span>
<span class="w"> </span><span class="k">OR</span><span class="w"> </span><span class="p">((</span><span class="n">countries</span><span class="w"> </span><span class="o">></span><span class="w"> </span><span class="mi">10</span><span class="p">)</span><span class="w"> </span><span class="k">AND</span><span class="w"> </span><span class="p">(</span><span class="n">Gbps</span><span class="w"> </span><span class="o">></span><span class="w"> </span><span class="mi">0</span><span class="p">.</span><span class="mi">1</span><span class="p">))</span>
<span class="k">ORDER</span><span class="w"> </span><span class="k">BY</span>
<span class="w"> </span><span class="n">TimeReceived</span><span class="w"> </span><span class="k">DESC</span><span class="p">,</span>
<span class="w"> </span><span class="n">Gbps</span><span class="w"> </span><span class="k">DESC</span>
</pre></div>
<h1 id="gluing-everything-together">Gluing everything together<a class="headerlink" href="#gluing-everything-together" title="Permanent link"></a></h1>
<p>To sum up, building an anti-<abbr title="Distributed Denial of Service">DDoS</abbr> system requires to following these steps:</p>
<ol>
<li>define a set of criteria to detect a <abbr title="Distributed Denial of Service">DDoS</abbr> attack,</li>
<li>translate these criteria into SQL requests,</li>
<li>pre-aggregate flows into <code>SummingMergeTree</code> tables,</li>
<li>query and transform the results to a <em>BIRD</em> configuration file, and</li>
<li>configure your routers to pull the routes from <em>BIRD</em>.</li>
</ol>
<p>A Python script like the following one can handle the fourth step. For each
attacked target, it generates both a <em>Flowspec</em> rule and a blackhole route.</p>
<div class="language-python codehilite"><pre><span/><span class="kn">import</span> <span class="nn">socket</span>
<span class="kn">import</span> <span class="nn">types</span>
<span class="kn">from</span> <span class="nn">clickhouse_driver</span> <span class="kn">import</span> <span class="n">Client</span> <span class="k">as</span> <span class="n">CHClient</span>
<span class="c1"># Put your SQL query here!</span>
<span class="n">SQL_QUERY</span> <span class="o">=</span> <span class="s2">"…"</span>
<span class="c1"># How many anti-DDoS rules we want at the same time?</span>
<span class="n">MAX_DDOS_RULES</span> <span class="o">=</span> <span class="mi">20</span>
<span class="k">def</span> <span class="nf">empty_ruleset</span><span class="p">():</span>
<span class="n">ruleset</span> <span class="o">=</span> <span class="n">types</span><span class="o">.</span><span class="n">SimpleNamespace</span><span class="p">()</span>
<span class="n">ruleset</span><span class="o">.</span><span class="n">flowspec</span> <span class="o">=</span> <span class="n">types</span><span class="o">.</span><span class="n">SimpleNamespace</span><span class="p">()</span>
<span class="n">ruleset</span><span class="o">.</span><span class="n">blackhole</span> <span class="o">=</span> <span class="n">types</span><span class="o">.</span><span class="n">SimpleNamespace</span><span class="p">()</span>
<span class="n">ruleset</span><span class="o">.</span><span class="n">flowspec</span><span class="o">.</span><span class="n">v4</span> <span class="o">=</span> <span class="p">[]</span>
<span class="n">ruleset</span><span class="o">.</span><span class="n">flowspec</span><span class="o">.</span><span class="n">v6</span> <span class="o">=</span> <span class="p">[]</span>
<span class="n">ruleset</span><span class="o">.</span><span class="n">blackhole</span><span class="o">.</span><span class="n">v4</span> <span class="o">=</span> <span class="p">[]</span>
<span class="n">ruleset</span><span class="o">.</span><span class="n">blackhole</span><span class="o">.</span><span class="n">v6</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">return</span> <span class="n">ruleset</span>
<span class="n">current_ruleset</span> <span class="o">=</span> <span class="n">empty_ruleset</span><span class="p">()</span>
<span class="n">client</span> <span class="o">=</span> <span class="n">CHClient</span><span class="p">(</span><span class="n">host</span><span class="o">=</span><span class="s2">"clickhouse.akvorado.net"</span><span class="p">)</span>
<span class="k">while</span> <span class="kc">True</span><span class="p">:</span>
<span class="n">results</span> <span class="o">=</span> <span class="n">client</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">SQL_QUERY</span><span class="p">)</span>
<span class="n">seen</span> <span class="o">=</span> <span class="p">{}</span>
<span class="n">new_ruleset</span> <span class="o">=</span> <span class="n">empty_ruleset</span><span class="p">()</span>
<span class="k">for</span> <span class="p">(</span><span class="n">t</span><span class="p">,</span> <span class="n">addr</span><span class="p">,</span> <span class="n">proto</span><span class="p">,</span> <span class="n">port</span><span class="p">,</span> <span class="n">gbps</span><span class="p">,</span> <span class="n">mpps</span><span class="p">,</span> <span class="n">sources</span><span class="p">,</span> <span class="n">countries</span><span class="p">,</span> <span class="n">size</span><span class="p">)</span> <span class="ow">in</span> <span class="n">results</span><span class="p">:</span>
<span class="k">if</span> <span class="p">(</span><span class="n">addr</span><span class="p">,</span> <span class="n">proto</span><span class="p">,</span> <span class="n">port</span><span class="p">)</span> <span class="ow">in</span> <span class="n">seen</span><span class="p">:</span>
<span class="k">continue</span>
<span class="n">seen</span><span class="p">[(</span><span class="n">addr</span><span class="p">,</span> <span class="n">proto</span><span class="p">,</span> <span class="n">port</span><span class="p">)]</span> <span class="o">=</span> <span class="kc">True</span>
<span class="c1"># Flowspec</span>
<span class="k">if</span> <span class="n">addr</span><span class="o">.</span><span class="n">ipv4_mapped</span><span class="p">:</span>
<span class="n">address</span> <span class="o">=</span> <span class="n">addr</span><span class="o">.</span><span class="n">ipv4_mapped</span>
<span class="n">rules</span> <span class="o">=</span> <span class="n">new_ruleset</span><span class="o">.</span><span class="n">flowspec</span><span class="o">.</span><span class="n">v4</span>
<span class="n">table</span> <span class="o">=</span> <span class="s2">"flow4"</span>
<span class="n">mask</span> <span class="o">=</span> <span class="mi">32</span>
<span class="n">nh</span> <span class="o">=</span> <span class="s2">"proto"</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">address</span> <span class="o">=</span> <span class="n">addr</span>
<span class="n">rules</span> <span class="o">=</span> <span class="n">new_ruleset</span><span class="o">.</span><span class="n">flowspec</span><span class="o">.</span><span class="n">v6</span>
<span class="n">table</span> <span class="o">=</span> <span class="s2">"flow6"</span>
<span class="n">mask</span> <span class="o">=</span> <span class="mi">128</span>
<span class="n">nh</span> <span class="o">=</span> <span class="s2">"next header"</span>
<span class="k">if</span> <span class="n">size</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">==</span> <span class="n">size</span><span class="p">[</span><span class="mi">1</span><span class="p">]:</span>
<span class="n">length</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">"length = </span><span class="si">{</span><span class="nb">int</span><span class="p">(</span><span class="n">size</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span><span class="si">}</span><span class="s2">"</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">length</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">"length >= </span><span class="si">{</span><span class="nb">int</span><span class="p">(</span><span class="n">size</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span><span class="si">}</span><span class="s2"> && <= </span><span class="si">{</span><span class="nb">int</span><span class="p">(</span><span class="n">size</span><span class="p">[</span><span class="mi">1</span><span class="p">])</span><span class="si">}</span><span class="s2">"</span>
<span class="n">header</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">"""</span>
<span class="s2"># Time: </span><span class="si">{</span><span class="n">t</span><span class="si">}</span>
<span class="s2"># Source: </span><span class="si">{</span><span class="n">address</span><span class="si">}</span><span class="s2">, protocol: </span><span class="si">{</span><span class="n">proto</span><span class="si">}</span><span class="s2">, port: </span><span class="si">{</span><span class="n">port</span><span class="si">}</span>
<span class="s2"># Gbps/Mpps: </span><span class="si">{</span><span class="n">gbps</span><span class="si">:</span><span class="s2">.3</span><span class="si">}</span><span class="s2">/</span><span class="si">{</span><span class="n">mpps</span><span class="si">:</span><span class="s2">.3</span><span class="si">}</span><span class="s2">, packet size: </span><span class="si">{</span><span class="nb">int</span><span class="p">(</span><span class="n">size</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span><span class="si">}</span><span class="s2"><=X<=</span><span class="si">{</span><span class="nb">int</span><span class="p">(</span><span class="n">size</span><span class="p">[</span><span class="mi">1</span><span class="p">])</span><span class="si">}</span>
<span class="s2"># Flows: </span><span class="si">{</span><span class="n">flows</span><span class="si">}</span><span class="s2">, sources: </span><span class="si">{</span><span class="n">sources</span><span class="si">}</span><span class="s2">, countries: </span><span class="si">{</span><span class="n">countries</span><span class="si">}</span>
<span class="s2">"""</span>
<span class="n">rules</span><span class="o">.</span><span class="n">append</span><span class="p">(</span>
<span class="sa">f</span><span class="s2">"""</span><span class="si">{</span><span class="n">header</span><span class="si">}</span>
<span class="s2">route </span><span class="si">{</span><span class="n">table</span><span class="si">}</span><span class="s2"> </span><span class="se">{{</span>
<span class="s2"> dst </span><span class="si">{</span><span class="n">address</span><span class="si">}</span><span class="s2">/</span><span class="si">{</span><span class="n">mask</span><span class="si">}</span><span class="s2">;</span>
<span class="s2"> sport = </span><span class="si">{</span><span class="n">port</span><span class="si">}</span><span class="s2">;</span>
<span class="s2"> </span><span class="si">{</span><span class="n">length</span><span class="si">}</span><span class="s2">;</span>
<span class="s2"> </span><span class="si">{</span><span class="n">nh</span><span class="si">}</span><span class="s2"> = </span><span class="si">{</span><span class="n">socket</span><span class="o">.</span><span class="n">getprotobyname</span><span class="p">(</span><span class="n">proto</span><span class="p">)</span><span class="si">}</span><span class="s2">;</span>
<span class="se">}}{{</span>
<span class="s2"> bgp_ext_community.add((generic, 0x80060000, 0x00000000));</span>
<span class="se">}}</span><span class="s2">;</span>
<span class="s2">"""</span>
<span class="p">)</span>
<span class="c1"># Blackhole</span>
<span class="k">if</span> <span class="n">addr</span><span class="o">.</span><span class="n">ipv4_mapped</span><span class="p">:</span>
<span class="n">rules</span> <span class="o">=</span> <span class="n">new_ruleset</span><span class="o">.</span><span class="n">blackhole</span><span class="o">.</span><span class="n">v4</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">rules</span> <span class="o">=</span> <span class="n">new_ruleset</span><span class="o">.</span><span class="n">blackhole</span><span class="o">.</span><span class="n">v6</span>
<span class="n">rules</span><span class="o">.</span><span class="n">append</span><span class="p">(</span>
<span class="sa">f</span><span class="s2">"""</span><span class="si">{</span><span class="n">header</span><span class="si">}</span>
<span class="s2">route </span><span class="si">{</span><span class="n">address</span><span class="si">}</span><span class="s2">/</span><span class="si">{</span><span class="n">mask</span><span class="si">}</span><span class="s2"> blackhole </span><span class="se">{{</span>
<span class="s2"> bgp_community.add((65535, 666));</span>
<span class="se">}}</span><span class="s2">;</span>
<span class="s2">"""</span>
<span class="p">)</span>
<span class="n">new_ruleset</span><span class="o">.</span><span class="n">flowspec</span><span class="o">.</span><span class="n">v4</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span>
<span class="nb">set</span><span class="p">(</span><span class="n">new_ruleset</span><span class="o">.</span><span class="n">flowspec</span><span class="o">.</span><span class="n">v4</span><span class="p">[:</span><span class="n">MAX_DDOS_RULES</span><span class="p">])</span>
<span class="p">)</span>
<span class="n">new_ruleset</span><span class="o">.</span><span class="n">flowspec</span><span class="o">.</span><span class="n">v6</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span>
<span class="nb">set</span><span class="p">(</span><span class="n">new_ruleset</span><span class="o">.</span><span class="n">flowspec</span><span class="o">.</span><span class="n">v6</span><span class="p">[:</span><span class="n">MAX_DDOS_RULES</span><span class="p">])</span>
<span class="p">)</span>
<span class="c1"># TODO: advertise changes by mail, chat, ...</span>
<span class="n">current_ruleset</span> <span class="o">=</span> <span class="n">new_ruleset</span>
<span class="n">changes</span> <span class="o">=</span> <span class="kc">False</span>
<span class="k">for</span> <span class="n">rules</span><span class="p">,</span> <span class="n">path</span> <span class="ow">in</span> <span class="p">(</span>
<span class="p">(</span><span class="n">current_ruleset</span><span class="o">.</span><span class="n">flowspec</span><span class="o">.</span><span class="n">v4</span><span class="p">,</span> <span class="s2">"v4-flowspec"</span><span class="p">),</span>
<span class="p">(</span><span class="n">current_ruleset</span><span class="o">.</span><span class="n">flowspec</span><span class="o">.</span><span class="n">v6</span><span class="p">,</span> <span class="s2">"v6-flowspec"</span><span class="p">),</span>
<span class="p">(</span><span class="n">current_ruleset</span><span class="o">.</span><span class="n">blackhole</span><span class="o">.</span><span class="n">v4</span><span class="p">,</span> <span class="s2">"v4-blackhole"</span><span class="p">),</span>
<span class="p">(</span><span class="n">current_ruleset</span><span class="o">.</span><span class="n">blackhole</span><span class="o">.</span><span class="n">v6</span><span class="p">,</span> <span class="s2">"v6-blackhole"</span><span class="p">),</span>
<span class="p">):</span>
<span class="n">path</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="s2">"/etc/bird/"</span><span class="p">,</span> <span class="sa">f</span><span class="s2">"</span><span class="si">{</span><span class="n">path</span><span class="si">}</span><span class="s2">.conf"</span><span class="p">)</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="sa">f</span><span class="s2">"</span><span class="si">{</span><span class="n">path</span><span class="si">}</span><span class="s2">.tmp"</span><span class="p">,</span> <span class="s2">"w"</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
<span class="k">for</span> <span class="n">r</span> <span class="ow">in</span> <span class="n">rules</span><span class="p">:</span>
<span class="n">f</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="n">r</span><span class="p">)</span>
<span class="n">changes</span> <span class="o">=</span> <span class="p">(</span>
<span class="n">changes</span> <span class="ow">or</span> <span class="ow">not</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">exists</span><span class="p">(</span><span class="n">path</span><span class="p">)</span> <span class="ow">or</span> <span class="ow">not</span> <span class="n">samefile</span><span class="p">(</span><span class="n">path</span><span class="p">,</span> <span class="sa">f</span><span class="s2">"</span><span class="si">{</span><span class="n">path</span><span class="si">}</span><span class="s2">.tmp"</span><span class="p">)</span>
<span class="p">)</span>
<span class="n">os</span><span class="o">.</span><span class="n">rename</span><span class="p">(</span><span class="sa">f</span><span class="s2">"</span><span class="si">{</span><span class="n">path</span><span class="si">}</span><span class="s2">.tmp"</span><span class="p">,</span> <span class="n">path</span><span class="p">)</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">changes</span><span class="p">:</span>
<span class="k">continue</span>
<span class="n">proc</span> <span class="o">=</span> <span class="n">subprocess</span><span class="o">.</span><span class="n">Popen</span><span class="p">(</span>
<span class="p">[</span><span class="s2">"birdc"</span><span class="p">,</span> <span class="s2">"configure"</span><span class="p">],</span>
<span class="n">stdin</span><span class="o">=</span><span class="n">subprocess</span><span class="o">.</span><span class="n">DEVNULL</span><span class="p">,</span>
<span class="n">stdout</span><span class="o">=</span><span class="n">subprocess</span><span class="o">.</span><span class="n">PIPE</span><span class="p">,</span>
<span class="n">stderr</span><span class="o">=</span><span class="n">subprocess</span><span class="o">.</span><span class="n">PIPE</span><span class="p">,</span>
<span class="p">)</span>
<span class="n">stdout</span><span class="p">,</span> <span class="n">stderr</span> <span class="o">=</span> <span class="n">proc</span><span class="o">.</span><span class="n">communicate</span><span class="p">(</span><span class="kc">None</span><span class="p">)</span>
<span class="n">stdout</span> <span class="o">=</span> <span class="n">stdout</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s2">"utf-8"</span><span class="p">,</span> <span class="s2">"replace"</span><span class="p">)</span>
<span class="n">stderr</span> <span class="o">=</span> <span class="n">stderr</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s2">"utf-8"</span><span class="p">,</span> <span class="s2">"replace"</span><span class="p">)</span>
<span class="k">if</span> <span class="n">proc</span><span class="o">.</span><span class="n">returncode</span> <span class="o">!=</span> <span class="mi">0</span><span class="p">:</span>
<span class="n">logger</span><span class="o">.</span><span class="n">error</span><span class="p">(</span>
<span class="s2">"</span><span class="si">{}</span><span class="s2"> error:</span><span class="se">\n</span><span class="si">{}</span><span class="se">\n</span><span class="si">{}</span><span class="s2">"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span>
<span class="s2">"birdc reconfigure"</span><span class="p">,</span>
<span class="s2">"</span><span class="se">\n</span><span class="s2">"</span><span class="o">.</span><span class="n">join</span><span class="p">(</span>
<span class="p">[</span><span class="s2">" O: </span><span class="si">{}</span><span class="s2">"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">line</span><span class="p">)</span> <span class="k">for</span> <span class="n">line</span> <span class="ow">in</span> <span class="n">stdout</span><span class="o">.</span><span class="n">rstrip</span><span class="p">()</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s2">"</span><span class="se">\n</span><span class="s2">"</span><span class="p">)]</span>
<span class="p">),</span>
<span class="s2">"</span><span class="se">\n</span><span class="s2">"</span><span class="o">.</span><span class="n">join</span><span class="p">(</span>
<span class="p">[</span><span class="s2">" E: </span><span class="si">{}</span><span class="s2">"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">line</span><span class="p">)</span> <span class="k">for</span> <span class="n">line</span> <span class="ow">in</span> <span class="n">stderr</span><span class="o">.</span><span class="n">rstrip</span><span class="p">()</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s2">"</span><span class="se">\n</span><span class="s2">"</span><span class="p">)]</span>
<span class="p">),</span>
<span class="p">)</span>
<span class="p">)</span>
</pre></div>
<hr/>
<p>Until <em>Akvorado</em> integrates <abbr title="Distributed Denial of Service">DDoS</abbr> detection and mitigation, the ideas presented
in this blog post provide a solid foundation to get started with your own
anti-<abbr title="Distributed Denial of Service">DDoS</abbr> system. 🛡️</p>
<div class="footnote">
<hr/>
<ol>
<li id="fn-format">
<p><em>ClickHouse</em> can export results using <em>Markdown</em> format when
appending <code>FORMAT Markdown</code> to the query. <a class="footnote-backref" href="#fnref-format" title="Jump back to footnote 1 in the text">↩︎</a></p>
</li>
<li id="fn-dns">
<p>While most DNS clients should retry with TCP on failures, this is not
always the case: until <a href="https://git.musl-libc.org/cgit/musl/commit/?id=51d4669fb97782f6a66606da852b5afd49a08001">recently</a>, <a href="https://musl.libc.org/">musl libc</a> did not implement this. <a class="footnote-backref" href="#fnref-dns" title="Jump back to footnote 2 in the text">↩︎</a></p>
</li>
<li id="fn-materialized">
<p>The materialized view also aggregates the data at hand, both
for efficiency and to ensure we work with the right data types. <a class="footnote-backref" href="#fnref-materialized" title="Jump back to footnote 3 in the text">↩︎</a></p>
</li>
</ol>
</div>
Building a SQL-like language to filter flowsVincent Bernat2023-02-13T08:06:06Zhttp://www.luffy.cx/en/blog/2023-sql-like-language-filter.html
<p><a href="/en/blog/2022-akvorado-flow-collector" title="Akvorado: a flow collector, enricher, and visualizer">Akvorado</a> collects network flows using <a href="https://www.rfc-editor.org/rfc/rfc7011" title="RFC 7011: Specification of the IP Flow Information Export (IPFIX) Protocol for the Exchange of Flow Information">IPFIX</a> or <a href="https://www.rfc-editor.org/rfc/rfc3176" title="RFC 3176: Specification of the IP Flow Information Export (IPFIX) Protocol for the Exchange of Flow Information">sFlow</a>. It stores them
in a <a href="https://clickhouse.com/" title="ClickHouse: OLAP DBMS">ClickHouse</a> database. A web console allows a user to query the data and
plot some graphs. A nice aspect of this console is how we can filter flows with
a SQL-like language:</p>
<figure><div class="lf-media-outer"><span class="lf-media-inner"><video src="https://d2pzklc15kok91.cloudfront.net/images/akvorado-filter.d8bf4a41cd7883.mp4" width="700" height="420" muted="" loop="" autoplay="" playsinline="" controls="" class="lf-media lf-opaque"/></span></div><figcaption>Filter editor in Akvorado console</figcaption></figure>
<!--
InIfBoundary = external AND
ExporterRegion = "france" AND
InIfConnectivity = "transit" AND
SrcAS = AS15169 AND
DstAddr << 2a01:e0f:ffff::/48
-->
<p>Often, web interfaces expose a <a href="https://dabernathy89.github.io/vue-query-builder/" title="Vue Query Builder">query</a>
<a href="https://react-querybuilder.js.org/" title="React Query Builder">builder</a> to build such filters. I think combining a
SQL-like language with an editor supporting <strong>completion</strong>, <strong>syntax
highlighting</strong>, and <strong>linting</strong> is a better approach.<sup id="fnref-lazy"><a class="footnote-ref" href="#fn-lazy">1</a></sup></p>
<p>The language parser is built with <a href="https://github.com/mna/pigeon" title="Command pigeon generates parsers in Go from a PEG grammar">pigeon</a> (<em>Go</em>) from a <a href="https://en.wikipedia.org/wiki/Parsing_expression_grammar" title="Parsing expression grammar on Wikipedia">parsing expression
grammar</a>—or <abbr title="Parsing Expression Grammar">PEG</abbr>. The editor component is <a href="https://codemirror.net/" title="CodeMirror: Extensible Code Editor">CodeMirror</a> (<em>TypeScript</em>).</p>
<h1 id="language-parser">Language parser<a class="headerlink" href="#language-parser" title="Permanent link"></a></h1>
<p><abbr title="Parsing Expression Grammar">PEG</abbr> grammars are relatively recent<sup id="fnref-recent"><a class="footnote-ref" href="#fn-recent">2</a></sup> and are an alternative to
context-free grammars. They are easier to write and they can generate better
error messages. Python <a href="https://peps.python.org/pep-0617/" title="PEP 617 – New PEG parser for CPython">switched from an <abbr title="left-to-right, leftmost derivation">LL</abbr>(1)-based parser to a <abbr title="Parsing Expression Grammar">PEG</abbr>-based
parser</a> in Python 3.9.</p>
<p><a href="https://github.com/mna/pigeon" title="Command pigeon generates parsers in Go from a PEG grammar">pigeon</a> generates a parser for Go. A grammar is a set of rules. Each rule is
an identifier, with an optional user-friendly label for error messages, an
expression, and an action in Go to be executed on match. You can find the
complete grammar in <a href="https://github.com/akvorado/akvorado/blob/main/console/filter/parser.peg"><code>parser.peg</code></a>. Here is a simplified rule:</p>
<div class="language-pigeon codehilite"><pre><span/>ConditionIPExpr <span class="s2">"condition on IP"</span> <span class="o">←</span>
<span class="nv">column</span>:<span class="p">(</span><span class="s2">"ExporterAddress"</span><span class="o">i</span> <span class="p">{</span><span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="s">"ExporterAddress"</span><span class="p">,</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">}</span>
/ <span class="s2">"SrcAddr"</span><span class="o">i</span> <span class="p">{</span><span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="s">"SrcAddr"</span><span class="p">,</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">}</span>
/ <span class="s2">"DstAddr"</span><span class="o">i</span> <span class="p">{</span><span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="s">"DstAddr"</span><span class="p">,</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">})</span> _
<span class="nv">operator</span>:<span class="p">(</span><span class="s2">"="</span> / <span class="s2">"!="</span><span class="p">)</span> _
<span class="nv">ip</span>:IP <span class="p">{</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nx">fmt</span><span class="p">.</span><span class="nx">Sprintf</span><span class="p">(</span><span class="s">"%s %s IPv6StringToNum(%s)"</span><span class="p">,</span>
<span class="w"> </span><span class="nx">toString</span><span class="p">(</span><span class="nx">column</span><span class="p">),</span><span class="w"> </span><span class="nx">toString</span><span class="p">(</span><span class="nx">operator</span><span class="p">),</span><span class="w"> </span><span class="nx">quote</span><span class="p">(</span><span class="nx">ip</span><span class="p">)),</span><span class="w"> </span><span class="kc">nil</span>
<span class="w"> </span><span class="p">}</span>
</pre></div>
<p>The rule identifier is <code>ConditionIPExpr</code>. It case-insensitively matches
<code>ExporterAddress</code>, <code>SrcAddr</code>, or <code>DstAddr</code>. The action for each case returns the
proper case for the column name. That’s what is stored in the <code>column</code> variable.
Then, it matches one of the possible operators. As there is no code block, it
stores the matched string directly in the <code>operator</code> variable. Then, it tries to
match the <code>IP</code> rule, which is defined elsewhere in the grammar. If it succeeds,
it stores the result of the match in the <code>ip</code> variable and executes the final
action. The action turns the column, operator, and IP into a proper expression
for <em>ClickHouse</em>. For example, if we have <code>ExporterAddress = 203.0.113.15</code>, we
get <code>ExporterAddress = IPv6StringToNum('203.0.113.15')</code>.</p>
<p>The <code>IP</code> rule uses a rudimentary regular expression but checks if the matched
address is correct in the action block, thanks to <code>netip.ParseAddr()</code>:</p>
<div class="language-pigeon codehilite"><pre><span/>IP <span class="s2">"IP address"</span> <span class="o">←</span> <span class="p">[</span><span class="s">0-9A-Fa-f:.</span><span class="p">]</span>+ <span class="p">{</span>
<span class="w"> </span><span class="nx">ip</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">netip</span><span class="p">.</span><span class="nx">ParseAddr</span><span class="p">(</span><span class="nb">string</span><span class="p">(</span><span class="nx">c</span><span class="p">.</span><span class="nx">text</span><span class="p">))</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="s">""</span><span class="p">,</span><span class="w"> </span><span class="nx">errors</span><span class="p">.</span><span class="nx">New</span><span class="p">(</span><span class="s">"expecting an IP address"</span><span class="p">)</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nx">ip</span><span class="p">.</span><span class="nx">String</span><span class="p">(),</span><span class="w"> </span><span class="kc">nil</span>
<span class="p">}</span>
</pre></div>
<p>Our parser safely turns the filter into a <code>WHERE</code> clause accepted by
<em>ClickHouse</em>:<sup id="fnref-ast"><a class="footnote-ref" href="#fn-ast">3</a></sup></p>
<div class="language-sql codehilite"><pre><span/><span class="k">WHERE</span><span class="w"> </span><span class="n">InIfBoundary</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'external'</span><span class="w"> </span>
<span class="k">AND</span><span class="w"> </span><span class="n">ExporterRegion</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'france'</span><span class="w"> </span>
<span class="k">AND</span><span class="w"> </span><span class="n">InIfConnectivity</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'transit'</span><span class="w"> </span>
<span class="k">AND</span><span class="w"> </span><span class="n">SrcAS</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">15169</span><span class="w"> </span>
<span class="k">AND</span><span class="w"> </span><span class="n">DstAddr</span><span class="w"> </span><span class="k">BETWEEN</span><span class="w"> </span><span class="n">toIPv6</span><span class="p">(</span><span class="s1">'2a01:e0f:ffff::'</span><span class="p">)</span><span class="w"> </span>
<span class="w"> </span><span class="k">AND</span><span class="w"> </span><span class="n">toIPv6</span><span class="p">(</span><span class="s1">'2a01:e0f:ffff:ffff:ffff:ffff:ffff:ffff'</span><span class="p">)</span>
</pre></div>
<h1 id="integration-in-codemirror">Integration in CodeMirror<a class="headerlink" href="#integration-in-codemirror" title="Permanent link"></a></h1>
<p><a href="https://codemirror.net/" title="CodeMirror: Extensible Code Editor">CodeMirror</a> is a versatile code editor that can be easily integrated into
JavaScript projects. In <em>Akvorado</em>, the <a href="https://vuejs.org/" title="Vue.js: the Progressive JavaScript Framework">Vue.js</a> component,
<a href="https://github.com/akvorado/akvorado/blob/main/console/frontend/src/components/InputFilter.vue"><code>InputFilter</code></a>, uses <em>CodeMirror</em> as its foundation and
leverages features such as syntax highlighting, linting, and completion. The
source code for these capabilities can be found in the
<a href="https://github.com/akvorado/akvorado/tree/main/console/frontend/src/codemirror/lang-filter"><code>codemirror/lang-filter/</code></a> directory.</p>
<h2 id="syntax-highlighting">Syntax highlighting<a class="headerlink" href="#syntax-highlighting" title="Permanent link"></a></h2>
<p>The <abbr title="Parsing Expression Grammar">PEG</abbr> grammar for Go cannot be utilized directly<sup id="fnref-pegjs"><a class="footnote-ref" href="#fn-pegjs">4</a></sup> and the requirements
for parsers for editors are distinct: they should be error-tolerant and operate
incrementally, as code is typically updated character by character. <em>CodeMirror</em>
offers a solution through its own parser generator, <a href="https://lezer.codemirror.net/" title="The Lezer Parser System">Lezer</a>.</p>
<p>We don’t need this additional parser to fully understand the filter language.
Only the basic structure is needed: column names, comparison and logic
operators, quoted and unquoted values. The <a href="https://github.com/akvorado/akvorado/blob/main/console/frontend/src/codemirror/lang-filter/syntax.grammar">grammar</a> is therefore quite short
and does not need to be updated often:</p>
<div class="language-lezer codehilite"><pre><span/><span class="kt">@top</span> <span class="ss">Filter</span> <span class="p">{</span>
expression
<span class="p">}</span>
<span class="ss">expression</span> <span class="p">{</span>
Not expression |
<span class="s2">"("</span> expression <span class="s2">")"</span> |
<span class="s2">"("</span> expression <span class="s2">")"</span> And expression |
<span class="s2">"("</span> expression <span class="s2">")"</span> Or expression |
comparisonExpression And expression |
comparisonExpression Or expression |
comparisonExpression
<span class="p">}</span>
<span class="ss">comparisonExpression</span> <span class="p">{</span>
Column Operator Value
<span class="p">}</span>
<span class="ss">Value</span> <span class="p">{</span>
String | Literal | ValueLParen ListOfValues ValueRParen
<span class="p">}</span>
<span class="ss">ListOfValues</span> <span class="p">{</span>
ListOfValues ValueComma <span class="p">(</span>String | Literal<span class="p">)</span> |
String | Literal
<span class="p">}</span>
<span class="c1">// […]</span>
<span class="kt">@tokens</span> <span class="p">{</span>
<span class="c1">// […]</span>
<span class="ss">Column</span> <span class="p">{</span> std.asciiLetter <span class="p">(</span>std.asciiLetter|std.digit<span class="p">)</span>* <span class="p">}</span>
<span class="ss">Operator</span> <span class="p">{</span> $<span class="p">[</span><span class="s">a-zA-Z!=><</span><span class="p">]</span>+ <span class="p">}</span>
<span class="ss">String</span> <span class="p">{</span>
<span class="s1">'"'</span> <span class="p">(</span>!<span class="p">[</span><span class="s">\\\n"</span><span class="p">]</span> | <span class="s2">"\\"</span> _<span class="p">)</span>* <span class="s1">'"'</span>? |
<span class="s2">"'"</span> <span class="p">(</span>!<span class="p">[</span><span class="s">\\\n'</span><span class="p">]</span> | <span class="s2">"\\"</span> _<span class="p">)</span>* <span class="s2">"'"</span>?
<span class="p">}</span>
<span class="ss">Literal</span> <span class="p">{</span> <span class="p">(</span>std.digit | std.asciiLetter | $<span class="p">[</span><span class="s">.:/</span><span class="p">])</span>+ <span class="p">}</span>
<span class="c1">// […]</span>
<span class="p">}</span>
</pre></div>
<p>The expression <code>SrcAS = 12322 AND (DstAS = 1299 OR SrcAS = 29447)</code> is parsed to:</p>
<div class="language-text-only codehilite"><pre><span/>Filter(Column, Operator, Value(Literal),
And, Column, Operator, Value(Literal),
Or, Column, Operator, Value(Literal))
</pre></div>
<p>The last step is to teach <em>CodeMirror</em> how to map each token to a highlighting
tag:</p>
<div class="language-typescript codehilite"><pre><span/><span class="k">export</span><span class="w"> </span><span class="kd">const</span><span class="w"> </span><span class="nx">FilterLanguage</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">LRLanguage</span><span class="p">.</span><span class="nx">define</span><span class="p">({</span>
<span class="w"> </span><span class="nx">parser</span><span class="o">:</span><span class="w"> </span><span class="kt">parser.configure</span><span class="p">({</span>
<span class="w"> </span><span class="nx">props</span><span class="o">:</span><span class="w"> </span><span class="p">[</span>
<span class="w"> </span><span class="nx">styleTags</span><span class="p">({</span>
<span class="w"> </span><span class="nx">Column</span><span class="o">:</span><span class="w"> </span><span class="kt">t.propertyName</span><span class="p">,</span>
<span class="w"> </span><span class="nx">String</span><span class="o">:</span><span class="w"> </span><span class="kt">t.string</span><span class="p">,</span>
<span class="w"> </span><span class="nx">Literal</span><span class="o">:</span><span class="w"> </span><span class="kt">t.literal</span><span class="p">,</span>
<span class="w"> </span><span class="nx">LineComment</span><span class="o">:</span><span class="w"> </span><span class="kt">t.lineComment</span><span class="p">,</span>
<span class="w"> </span><span class="nx">BlockComment</span><span class="o">:</span><span class="w"> </span><span class="kt">t.blockComment</span><span class="p">,</span>
<span class="w"> </span><span class="nx">Or</span><span class="o">:</span><span class="w"> </span><span class="kt">t.logicOperator</span><span class="p">,</span>
<span class="w"> </span><span class="nx">And</span><span class="o">:</span><span class="w"> </span><span class="kt">t.logicOperator</span><span class="p">,</span>
<span class="w"> </span><span class="nx">Not</span><span class="o">:</span><span class="w"> </span><span class="kt">t.logicOperator</span><span class="p">,</span>
<span class="w"> </span><span class="nx">Operator</span><span class="o">:</span><span class="w"> </span><span class="kt">t.compareOperator</span><span class="p">,</span>
<span class="w"> </span><span class="s2">"( )"</span><span class="o">:</span><span class="w"> </span><span class="nx">t</span><span class="p">.</span><span class="nx">paren</span><span class="p">,</span>
<span class="w"> </span><span class="p">}),</span>
<span class="w"> </span><span class="p">],</span>
<span class="w"> </span><span class="p">}),</span>
<span class="p">});</span>
</pre></div>
<h2 id="linting">Linting<a class="headerlink" href="#linting" title="Permanent link"></a></h2>
<p>We offload linting to the original parser in Go. The
<code>/api/v0/console/filter/validate</code> endpoint accepts a filter and returns a JSON
structure with the errors that were found:</p>
<div class="language-json codehilite"><pre><span/><span class="p">{</span>
<span class="w"> </span><span class="nt">"message"</span><span class="p">:</span><span class="w"> </span><span class="s2">"at line 1, position 12: string literal not terminated"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"errors"</span><span class="p">:</span><span class="w"> </span><span class="p">[{</span>
<span class="w"> </span><span class="nt">"line"</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"column"</span><span class="p">:</span><span class="w"> </span><span class="mi">12</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"offset"</span><span class="p">:</span><span class="w"> </span><span class="mi">11</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"message"</span><span class="p">:</span><span class="w"> </span><span class="s2">"string literal not terminated"</span><span class="p">,</span>
<span class="w"> </span><span class="p">}]</span>
<span class="p">}</span>
</pre></div>
<p>The <a href="https://github.com/akvorado/akvorado/blob/main/console/frontend/src/codemirror/lang-filter/linter.ts">linter source</a> for <em>CodeMirror</em> queries the API and turns each error into
a <a href="https://codemirror.net/docs/ref/#lint.Diagnostic">diagnostic</a>.</p>
<h2 id="completion">Completion<a class="headerlink" href="#completion" title="Permanent link"></a></h2>
<p>The completion system takes a hybrid approach. It splits the work between the
frontend and the backend to offer useful suggestions for completing filters.</p>
<p>The <a href="https://github.com/akvorado/akvorado/blob/main/console/frontend/src/codemirror/lang-filter/complete.ts">frontend</a> uses the parser built with <em>Lezer</em> to determine the context of
the completion: do we complete a column name, an operator, or a value? It also
extracts the column name if we are completing something else. It forwards the
result to the backend through the <code>/api/v0/console/filter/complete</code> endpoint.
Walking the syntax tree was not as easy as I thought, but <a href="https://github.com/akvorado/akvorado/blob/main/console/frontend/src/codemirror/lang-filter/complete.test.ts">unit tests</a> helped
a lot.</p>
<p>The <a href="https://github.com/akvorado/akvorado/blob/9eee46caded6fa6cc2dfb90d1941b2ef05d15e9f/console/filter.go#L76">backend</a> uses the parser generated by <em>pigeon</em> to complete a column name
or a comparison operator. For values, the completions are either static or
extracted from the <em>ClickHouse</em> database. A user can complete an <abbr title="Autonomous System">AS</abbr> number from
an organization name thanks to the following snippet:</p>
<div class="language-go codehilite"><pre><span/><span class="nx">results</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="p">[]</span><span class="kd">struct</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nx">Label</span><span class="w"> </span><span class="kt">string</span><span class="w"> </span><span class="s">`ch:"label"`</span>
<span class="w"> </span><span class="nx">Detail</span><span class="w"> </span><span class="kt">string</span><span class="w"> </span><span class="s">`ch:"detail"`</span>
<span class="p">}{}</span>
<span class="nx">columnName</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="s">"DstAS"</span>
<span class="nx">sqlQuery</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">fmt</span><span class="p">.</span><span class="nx">Sprintf</span><span class="p">(</span><span class="s">`</span>
<span class="s"> SELECT concat('AS', toString(%s)) AS label, dictGet('asns', 'name', %s) AS detail</span>
<span class="s"> FROM flows</span>
<span class="s"> WHERE TimeReceived > date_sub(minute, 1, now())</span>
<span class="s"> AND detail != ''</span>
<span class="s"> AND positionCaseInsensitive(detail, $1) >= 1</span>
<span class="s"> GROUP BY label, detail</span>
<span class="s"> ORDER BY COUNT(*) DESC</span>
<span class="s"> LIMIT 20</span>
<span class="s">`</span><span class="p">,</span><span class="w"> </span><span class="nx">columnName</span><span class="p">,</span><span class="w"> </span><span class="nx">columnName</span><span class="p">)</span>
<span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">conn</span><span class="p">.</span><span class="nx">Select</span><span class="p">(</span><span class="nx">ctx</span><span class="p">,</span><span class="w"> </span><span class="o">&</span><span class="nx">results</span><span class="p">,</span><span class="w"> </span><span class="nx">sqlQuery</span><span class="p">,</span><span class="w"> </span><span class="nx">input</span><span class="p">.</span><span class="nx">Prefix</span><span class="p">);</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nx">c</span><span class="p">.</span><span class="nx">r</span><span class="p">.</span><span class="nx">Err</span><span class="p">(</span><span class="nx">err</span><span class="p">).</span><span class="nx">Msg</span><span class="p">(</span><span class="s">"unable to query database"</span><span class="p">)</span>
<span class="w"> </span><span class="k">break</span>
<span class="p">}</span>
<span class="k">for</span><span class="w"> </span><span class="nx">_</span><span class="p">,</span><span class="w"> </span><span class="nx">result</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="k">range</span><span class="w"> </span><span class="nx">results</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nx">completions</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nb">append</span><span class="p">(</span><span class="nx">completions</span><span class="p">,</span><span class="w"> </span><span class="nx">filterCompletion</span><span class="p">{</span>
<span class="w"> </span><span class="nx">Label</span><span class="p">:</span><span class="w"> </span><span class="nx">result</span><span class="p">.</span><span class="nx">Label</span><span class="p">,</span>
<span class="w"> </span><span class="nx">Detail</span><span class="p">:</span><span class="w"> </span><span class="nx">result</span><span class="p">.</span><span class="nx">Detail</span><span class="p">,</span>
<span class="w"> </span><span class="nx">Quoted</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span>
<span class="w"> </span><span class="p">})</span>
<span class="p">}</span>
</pre></div>
<p>In my opinion, the completion system is a major factor in making the field
editor an efficient way to select flows. While a query builder may have been
more beginner-friendly, the completion system’s ease of use and functionality
make it more enjoyable to use once you become familiar.</p>
<div class="footnote">
<hr/>
<ol>
<li id="fn-lazy">
<p>Moreover, building a query builder did not seem like a fun task for me. <a class="footnote-backref" href="#fnref-lazy" title="Jump back to footnote 1 in the text">↩︎</a></p>
</li>
<li id="fn-recent">
<p>They were introduced in 2004 in “<a href="https://bford.info/pub/lang/peg.pdf">Parsing Expression Grammars: A
Recognition-Based Syntactic Foundation</a>.” <abbr title="left-to-right">LR</abbr> parsers were
introduced in <a href="https://web.archive.org/web/20120315152151/http://www.cs.dartmouth.edu/~mckeeman/cs48/mxcom/doc/knuth65.pdf" title="On the Translation of Languages from Left to Right">1965</a>, <abbr title="lookahead left-to-right">LALR</abbr> parsers in 1969, and <abbr title="left-to-right, leftmost derivation">LL</abbr> parsers in the
1970s. <a href="https://en.wikipedia.org/wiki/Yacc" title="Yacc article on Wikipedia">Yacc</a>, a popular parser generator, was written in 1975. <a class="footnote-backref" href="#fnref-recent" title="Jump back to footnote 2 in the text">↩︎</a></p>
</li>
<li id="fn-ast">
<p>The parser returns a string. It does not generate an intermediate <abbr title="Abstract Syntax Tree">AST</abbr>.
This makes it simpler and it currently fits our needs. <a class="footnote-backref" href="#fnref-ast" title="Jump back to footnote 3 in the text">↩︎</a></p>
</li>
<li id="fn-pegjs">
<p>It could be manually translated to JavaScript with <a href="https://peggyjs.org/" title="Peggy: Parser Generator for JavaScript">Peggy</a>. <a class="footnote-backref" href="#fnref-pegjs" title="Jump back to footnote 4 in the text">↩︎</a></p>
</li>
</ol>
</div>
Hacking the Geberit Sigma 70 flush plateVincent Bernat2023-02-11T21:22:56Zhttp://www.luffy.cx/en/blog/2023-geberit-sigma-70.html
<p>My toilet is equipped with a <a href="https://www.geberitnorthamerica.com/products/actuator-plates-and-flush-controls/sigma-series-flush-plates/sigma70/">Geberit Sigma 70</a> flush plate. The sales pitch
for this <a href="https://youtu.be/yqN2B0qxxFM?t=38">hydraulic-assisted device</a> praises the
“ingenious mount that acts like a rocker switch.” In practice, the flush is very
capricious and has a very high failure rate. <strong>Avoid this type of
mechanism!</strong> Prefer a fully mechanical version like the <a href="https://www.geberitnorthamerica.com/products/actuator-plates-and-flush-controls/sigma-series-flush-plates/sigma20/">Geberit Sigma 20</a>.</p>
<p>After several plumbers, exchanges with Geberit’s technical department, and the
expensive replacement of the entire mechanism, I was still getting a failure rate
of over 50% for the small flush. I finally managed to decrease this rate to 5%
by applying two <a href="https://www.amazon.com/s?k=8mm+silicone+bumpers">8 mm silicone bumpers</a> on the back of the plate. Their
locations are indicated by red circles on the picture below:</p>
<figure><div class="lf-media-outer"><span class="lf-media-inner"><img alt="Geberit Sigma 70 flush plate. Top: the mechanism that converts the mechanical press into a hydraulic impulse. Bottom: the back of the plate with the two places where to apply the bumpers." src="https://d2pzklc15kok91.cloudfront.net/images/geberit-sigma-70.365084ed1e16c2.jpg" width="700" height="905" class="lf-media lf-opaque" style="background-image: url(data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAIwAAAC1AQMAAAC3e4UYAAAABlBMVEXTx7YAAABS4fLDAAAAGklEQVR42u3BMQEAAADCoPVPbQo/oAAAADgYDW8AAcJLTF4AAAAASUVORK5CYII=)"/></span></div><figcaption>Geberit Sigma 70 flush plate. Above: the mechanism installed on the wall. Below, the back of the glass plate. In red, the two places where to apply the silicone bumpers.</figcaption></figure>
<p>Expect to pay about 5 € and as many minutes for this operation.</p>
Fast and dynamic encoding of Protocol Buffers in GoVincent Bernat2023-02-06T08:58:12Zhttp://www.luffy.cx/en/blog/2023-dynamic-protobuf-golang.html
<p><a href="https://developers.google.com/protocol-buffers/" title="Protocol Buffers">Protocol Buffers</a> are a popular choice for <strong>serializing structured data</strong>
due to their compact size, fast processing speed, language independence, and
compatibility. There exist other alternatives, including <a href="https://capnproto.org/" title="Cap'n Proto">Cap’n Proto</a>,
<a href="https://cbor.io/" title="RFC 8949 Concise Binary Object Representation">CBOR</a>, and <a href="https://avro.apache.org/" title="Apache Avro">Avro</a>.</p>
<p>Usually, data structures are described in a <strong>proto definition file</strong>
(<code>.proto</code>). The <code>protoc</code> compiler and a language-specific plugin convert it into
code:</p>
<div class="language-bash-session codehilite"><pre><span/><span class="gp">$ </span>head<span class="w"> </span>flow-4.proto
<span class="go">syntax = "proto3";</span>
<span class="go">package decoder;</span>
<span class="go">option go_package = "akvorado/inlet/flow/decoder";</span>
<span class="go">message FlowMessagev4 {</span>
<span class="go"> uint64 TimeReceived = 2;</span>
<span class="go"> uint32 SequenceNum = 3;</span>
<span class="go"> uint64 SamplingRate = 4;</span>
<span class="go"> uint32 FlowDirection = 5;</span>
<span class="gp">$ </span>protoc<span class="w"> </span>-I<span class="o">=</span>.<span class="w"> </span>--plugin<span class="o">=</span>protoc-gen-go<span class="w"> </span>--go_out<span class="o">=</span><span class="nv">module</span><span class="o">=</span>akvorado:.<span class="w"> </span>flow-4.proto
<span class="gp">$ </span>head<span class="w"> </span>inlet/flow/decoder/flow-4.pb.go
<span class="go">// Code generated by protoc-gen-go. DO NOT EDIT.</span>
<span class="go">// versions:</span>
<span class="go">// protoc-gen-go v1.28.0</span>
<span class="go">// protoc v3.21.12</span>
<span class="go">// source: inlet/flow/data/schemas/flow-4.proto</span>
<span class="go">package decoder</span>
<span class="go">import (</span>
<span class="go"> protoreflect "google.golang.org/protobuf/reflect/protoreflect"</span>
</pre></div>
<p><a href="/en/blog/2022-akvorado-flow-collector" title="Akvorado: a flow collector, enricher, and visualizer">Akvorado</a> collects network flows using <a href="https://www.rfc-editor.org/rfc/rfc7011" title="RFC 7011: Specification of the IP Flow Information Export (IPFIX) Protocol for the Exchange of Flow Information">IPFIX</a> or <a href="https://www.rfc-editor.org/rfc/rfc3176" title="RFC 3176: Specification of the IP Flow Information Export (IPFIX) Protocol for the Exchange of Flow Information">sFlow</a>, decodes them
with <a href="https://github.com/NetSampler/GoFlow2">GoFlow2</a>, encodes them to <em>Protocol Buffers</em>, and sends them to
<a href="https://kafka.apache.org/" title="Apache Kafka">Kafka</a> to be stored in a <a href="https://clickhouse.com/" title="ClickHouse: OLAP DBMS">ClickHouse</a> database. Collecting a new field,
such as source and destination MAC addresses, requires modifications in multiple
places, including the proto definition file and the ClickHouse migration code.
Moreover, the cost is paid by all users.<sup id="fnref-cost"><a class="footnote-ref" href="#fn-cost">1</a></sup> It would be nice to have an
<strong>application-wide schema</strong> and let users enable or disable the fields they
need.</p>
<p>While the main goal is flexibility, we do not want to sacrifice performance. On
this front, this is quite a success: when upgrading from 1.6.4 to 1.7.1, the
decoding and encoding performance almost doubled! 🤗</p>
<div class="language-text-only codehilite"><pre><span/>goos: linux
goarch: amd64
pkg: akvorado/inlet/flow
cpu: AMD Ryzen 5 5600X 6-Core Processor
│ initial.txt │ final.txt │
│ sec/op │ sec/op vs base │
Netflow/with_encoding-12 12.963µ ± 2% 7.836µ ± 1% -39.55% (p=0.000 n=10)
Sflow/with_encoding-12 19.37µ ± 1% 10.15µ ± 2% -47.63% (p=0.000 n=10)
</pre></div>
<h1 id="faster-protocol-buffers-encoding">Faster Protocol Buffers encoding<a class="headerlink" href="#faster-protocol-buffers-encoding" title="Permanent link"></a></h1>
<p>I use the <a href="https://github.com/akvorado/akvorado/blob/protobuf-bench-initial/inlet/flow/decoder_test.go">following code</a> to benchmark both the decoding and
encoding process. Initially, the <code>Decode()</code> method is a thin layer above
<em>GoFlow2</em> producer and stores the decoded data into the in-memory structure
generated by <code>protoc</code>. Later, some of the data will be encoded directly during
flow decoding. This is why we measure both the decoding and the
encoding.<sup id="fnref-netflow"><a class="footnote-ref" href="#fn-netflow">2</a></sup></p>
<div class="language-go codehilite"><pre><span/><span class="kd">func</span><span class="w"> </span><span class="nx">BenchmarkDecodeEncodeSflow</span><span class="p">(</span><span class="nx">b</span><span class="w"> </span><span class="o">*</span><span class="nx">testing</span><span class="p">.</span><span class="nx">B</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nx">r</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">reporter</span><span class="p">.</span><span class="nx">NewMock</span><span class="p">(</span><span class="nx">b</span><span class="p">)</span>
<span class="w"> </span><span class="nx">sdecoder</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">sflow</span><span class="p">.</span><span class="nx">New</span><span class="p">(</span><span class="nx">r</span><span class="p">)</span>
<span class="w"> </span><span class="nx">data</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">helpers</span><span class="p">.</span><span class="nx">ReadPcapPayload</span><span class="p">(</span><span class="nx">b</span><span class="p">,</span>
<span class="w"> </span><span class="nx">filepath</span><span class="p">.</span><span class="nx">Join</span><span class="p">(</span><span class="s">"decoder"</span><span class="p">,</span><span class="w"> </span><span class="s">"sflow"</span><span class="p">,</span><span class="w"> </span><span class="s">"testdata"</span><span class="p">,</span><span class="w"> </span><span class="s">"data-1140.pcap"</span><span class="p">))</span>
<span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="nx">_</span><span class="p">,</span><span class="w"> </span><span class="nx">withEncoding</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="k">range</span><span class="w"> </span><span class="p">[]</span><span class="kt">bool</span><span class="p">{</span><span class="kc">true</span><span class="p">,</span><span class="w"> </span><span class="kc">false</span><span class="p">}</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nx">title</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="kd">map</span><span class="p">[</span><span class="kt">bool</span><span class="p">]</span><span class="kt">string</span><span class="p">{</span>
<span class="w"> </span><span class="kc">true</span><span class="p">:</span><span class="w"> </span><span class="s">"with encoding"</span><span class="p">,</span>
<span class="w"> </span><span class="kc">false</span><span class="p">:</span><span class="w"> </span><span class="s">"without encoding"</span><span class="p">,</span>
<span class="w"> </span><span class="p">}[</span><span class="nx">withEncoding</span><span class="p">]</span>
<span class="w"> </span><span class="kd">var</span><span class="w"> </span><span class="nx">got</span><span class="w"> </span><span class="p">[]</span><span class="o">*</span><span class="nx">decoder</span><span class="p">.</span><span class="nx">FlowMessage</span>
<span class="w"> </span><span class="nx">b</span><span class="p">.</span><span class="nx">Run</span><span class="p">(</span><span class="nx">title</span><span class="p">,</span><span class="w"> </span><span class="kd">func</span><span class="p">(</span><span class="nx">b</span><span class="w"> </span><span class="o">*</span><span class="nx">testing</span><span class="p">.</span><span class="nx">B</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="nx">i</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="mi">0</span><span class="p">;</span><span class="w"> </span><span class="nx">i</span><span class="w"> </span><span class="p"><</span><span class="w"> </span><span class="nx">b</span><span class="p">.</span><span class="nx">N</span><span class="p">;</span><span class="w"> </span><span class="nx">i</span><span class="o">++</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nx">got</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">sdecoder</span><span class="p">.</span><span class="nx">Decode</span><span class="p">(</span><span class="nx">decoder</span><span class="p">.</span><span class="nx">RawFlow</span><span class="p">{</span>
<span class="w"> </span><span class="nx">Payload</span><span class="p">:</span><span class="w"> </span><span class="nx">data</span><span class="p">,</span>
<span class="w"> </span><span class="nx">Source</span><span class="p">:</span><span class="w"> </span><span class="nx">net</span><span class="p">.</span><span class="nx">ParseIP</span><span class="p">(</span><span class="s">"127.0.0.1"</span><span class="p">),</span>
<span class="w"> </span><span class="p">})</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">withEncoding</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="nx">_</span><span class="p">,</span><span class="w"> </span><span class="nx">flow</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="k">range</span><span class="w"> </span><span class="nx">got</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nx">buf</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="p">[]</span><span class="kt">byte</span><span class="p">{}</span>
<span class="w"> </span><span class="nx">buf</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">protowire</span><span class="p">.</span><span class="nx">AppendVarint</span><span class="p">(</span><span class="nx">buf</span><span class="p">,</span><span class="w"> </span><span class="nb">uint64</span><span class="p">(</span><span class="nx">proto</span><span class="p">.</span><span class="nx">Size</span><span class="p">(</span><span class="nx">flow</span><span class="p">)))</span>
<span class="w"> </span><span class="nx">proto</span><span class="p">.</span><span class="nx">MarshalOptions</span><span class="p">{}.</span><span class="nx">MarshalAppend</span><span class="p">(</span><span class="nx">buf</span><span class="p">,</span><span class="w"> </span><span class="nx">flow</span><span class="p">)</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">})</span>
<span class="w"> </span><span class="p">}</span>
<span class="p">}</span>
</pre></div>
<p>The canonical Go implementation for <em>Protocol Buffers</em>,
<a href="https://pkg.go.dev/google.golang.org/protobuf" title="Go support for Protocol Buffers"><code>google.golang.org/protobuf</code></a> is not the most
efficient one. For a long time, people were relying on <a href="https://github.com/gogo/protobuf" title="Protocol Buffers for Go with Gadgets">gogoprotobuf</a>.
However, the project is now <a href="https://github.com/gogo/protobuf/issues/691" title="#691: GoGo Protobuf looking for new ownership">deprecated</a>. A good replacement is
<a href="https://github.com/planetscale/vtprotobuf" title="Protocol Buffers compiler that generates optimized marshaling & unmarshaling Go code">vtprotobuf</a>.<sup id="fnref-vtprotobuf"><a class="footnote-ref" href="#fn-vtprotobuf">3</a></sup></p>
<div class="language-text-only codehilite"><pre><span/>goos: linux
goarch: amd64
pkg: akvorado/inlet/flow
cpu: AMD Ryzen 5 5600X 6-Core Processor
│ initial.txt │ bench-2.txt │
│ sec/op │ sec/op vs base │
Netflow/with_encoding-12 12.96µ ± 2% 10.28µ ± 2% -20.67% (p=0.000 n=10)
Netflow/without_encoding-12 8.935µ ± 2% 8.975µ ± 2% ~ (p=0.143 n=10)
Sflow/with_encoding-12 19.37µ ± 1% 16.67µ ± 2% -13.93% (p=0.000 n=10)
Sflow/without_encoding-12 14.62µ ± 3% 14.87µ ± 1% +1.66% (p=0.007 n=10)
</pre></div>
<h1 id="dynamic-protocol-buffers-encoding">Dynamic Protocol Buffers encoding<a class="headerlink" href="#dynamic-protocol-buffers-encoding" title="Permanent link"></a></h1>
<p>We have our baseline. Let’s see how to encode our <em>Protocol Buffers</em> without a
<code>.proto</code> file. The wire format is simple and rely a lot on variable-width
integers.</p>
<p>Variable-width integers, or <em>varints</em>, are an efficient way of encoding unsigned
integers using a variable number of bytes, from one to ten, with small values
using fewer bytes. They work by splitting integers into 7-bit payloads and using
the 8<sup>th</sup> bit as a continuation indicator, set to 1 for all payloads
except the last.</p>
<figure><div class="lf-media-outer"><span class="lf-media-inner"><img alt="Variable-width integers encoding in Protocol Buffers: conversion of 150 to a varint" src="https://d2pzklc15kok91.cloudfront.net/images/protobuf-varint.8f8b7e23fe94cc.svg" width="272" height="151" class="lf-media"/></span></div><figcaption>Variable-width integers encoding in Protocol Buffers</figcaption></figure>
<p>For our usage, we only need two types: variable-width
integers and byte sequences. A byte sequence is encoded by prefixing it by its
length as a varint. When a message is encoded, each key-value pair is turned
into a record consisting of a field number, a wire type, and a payload. The
field number and the wire type are encoded as a single variable-width integer
called a tag.</p>
<figure><div class="lf-media-outer"><span class="lf-media-inner"><img alt="Message encoded with Protocol Buffers: three varints, two sequences of bytes" src="https://d2pzklc15kok91.cloudfront.net/images/protobuf-message.f05804988087fb.svg" width="512" height="208" class="lf-media"/></span></div><figcaption>Message encoded with Protocol Buffers</figcaption></figure>
<p>We use the following low-level functions to build the output buffer:</p>
<ul>
<li><a href="https://pkg.go.dev/google.golang.org/protobuf@v1.28.1/encoding/protowire#AppendTag"><code>protowire.AppendTag()</code></a> encodes a tag,</li>
<li><a href="https://pkg.go.dev/google.golang.org/protobuf@v1.28.1/encoding/protowire#AppendVarint"><code>protowire.AppendVarint()</code></a> encodes a variable-width integer, and</li>
<li><a href="https://pkg.go.dev/google.golang.org/protobuf@v1.28.1/encoding/protowire#AppendBytes"><code>protowire.AppendBytes()</code></a> append bytes as is.</li>
</ul>
<p>Our schema abstraction contains the appropriate information to encode a message
(<code>ProtobufIndex</code>) and to generate a proto definition file (fields starting with
<code>Protobuf</code>):</p>
<div class="language-go codehilite"><pre><span/><span class="kd">type</span><span class="w"> </span><span class="nx">Column</span><span class="w"> </span><span class="kd">struct</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nx">Key</span><span class="w"> </span><span class="nx">ColumnKey</span>
<span class="w"> </span><span class="nx">Name</span><span class="w"> </span><span class="kt">string</span>
<span class="w"> </span><span class="nx">Disabled</span><span class="w"> </span><span class="kt">bool</span>
<span class="w"> </span><span class="c1">// […]</span>
<span class="w"> </span><span class="c1">// For protobuf.</span>
<span class="w"> </span><span class="nx">ProtobufIndex</span><span class="w"> </span><span class="nx">protowire</span><span class="p">.</span><span class="nx">Number</span>
<span class="w"> </span><span class="nx">ProtobufType</span><span class="w"> </span><span class="nx">protoreflect</span><span class="p">.</span><span class="nx">Kind</span><span class="w"> </span><span class="c1">// Uint64Kind, Uint32Kind, …</span>
<span class="w"> </span><span class="nx">ProtobufEnum</span><span class="w"> </span><span class="kd">map</span><span class="p">[</span><span class="kt">int</span><span class="p">]</span><span class="kt">string</span>
<span class="w"> </span><span class="nx">ProtobufEnumName</span><span class="w"> </span><span class="kt">string</span>
<span class="w"> </span><span class="nx">ProtobufRepeated</span><span class="w"> </span><span class="kt">bool</span>
<span class="p">}</span>
</pre></div>
<p>We have a few <a href="https://github.com/akvorado/akvorado/blob/main/common/schema/protobuf.go">helper methods</a> around the <code>protowire</code> functions to directly
encode the fields while decoding the flows. They skip disabled fields or
non-repeated fields already encoded. Here is an excerpt of the <a href="https://github.com/akvorado/akvorado/blob/main/inlet/flow/decoder/sflow/decode.go">sFlow
decoder</a>:</p>
<div class="language-go codehilite"><pre><span/><span class="nx">sch</span><span class="p">.</span><span class="nx">ProtobufAppendVarint</span><span class="p">(</span><span class="nx">bf</span><span class="p">,</span><span class="w"> </span><span class="nx">schema</span><span class="p">.</span><span class="nx">ColumnBytes</span><span class="p">,</span><span class="w"> </span><span class="nb">uint64</span><span class="p">(</span><span class="nx">recordData</span><span class="p">.</span><span class="nx">Base</span><span class="p">.</span><span class="nx">Length</span><span class="p">))</span>
<span class="nx">sch</span><span class="p">.</span><span class="nx">ProtobufAppendVarint</span><span class="p">(</span><span class="nx">bf</span><span class="p">,</span><span class="w"> </span><span class="nx">schema</span><span class="p">.</span><span class="nx">ColumnProto</span><span class="p">,</span><span class="w"> </span><span class="nb">uint64</span><span class="p">(</span><span class="nx">recordData</span><span class="p">.</span><span class="nx">Base</span><span class="p">.</span><span class="nx">Protocol</span><span class="p">))</span>
<span class="nx">sch</span><span class="p">.</span><span class="nx">ProtobufAppendVarint</span><span class="p">(</span><span class="nx">bf</span><span class="p">,</span><span class="w"> </span><span class="nx">schema</span><span class="p">.</span><span class="nx">ColumnSrcPort</span><span class="p">,</span><span class="w"> </span><span class="nb">uint64</span><span class="p">(</span><span class="nx">recordData</span><span class="p">.</span><span class="nx">Base</span><span class="p">.</span><span class="nx">SrcPort</span><span class="p">))</span>
<span class="nx">sch</span><span class="p">.</span><span class="nx">ProtobufAppendVarint</span><span class="p">(</span><span class="nx">bf</span><span class="p">,</span><span class="w"> </span><span class="nx">schema</span><span class="p">.</span><span class="nx">ColumnDstPort</span><span class="p">,</span><span class="w"> </span><span class="nb">uint64</span><span class="p">(</span><span class="nx">recordData</span><span class="p">.</span><span class="nx">Base</span><span class="p">.</span><span class="nx">DstPort</span><span class="p">))</span>
<span class="nx">sch</span><span class="p">.</span><span class="nx">ProtobufAppendVarint</span><span class="p">(</span><span class="nx">bf</span><span class="p">,</span><span class="w"> </span><span class="nx">schema</span><span class="p">.</span><span class="nx">ColumnEType</span><span class="p">,</span><span class="w"> </span><span class="nx">helpers</span><span class="p">.</span><span class="nx">ETypeIPv4</span><span class="p">)</span>
</pre></div>
<p>For fields that are required later in the pipeline, like source and destination
addresses, they are stored unencoded in a separate structure:</p>
<div class="language-go codehilite"><pre><span/><span class="kd">type</span><span class="w"> </span><span class="nx">FlowMessage</span><span class="w"> </span><span class="kd">struct</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nx">TimeReceived</span><span class="w"> </span><span class="kt">uint64</span>
<span class="w"> </span><span class="nx">SamplingRate</span><span class="w"> </span><span class="kt">uint32</span>
<span class="w"> </span><span class="c1">// For exporter classifier</span>
<span class="w"> </span><span class="nx">ExporterAddress</span><span class="w"> </span><span class="nx">netip</span><span class="p">.</span><span class="nx">Addr</span>
<span class="w"> </span><span class="c1">// For interface classifier</span>
<span class="w"> </span><span class="nx">InIf</span><span class="w"> </span><span class="kt">uint32</span>
<span class="w"> </span><span class="nx">OutIf</span><span class="w"> </span><span class="kt">uint32</span>
<span class="w"> </span><span class="c1">// For geolocation or BMP</span>
<span class="w"> </span><span class="nx">SrcAddr</span><span class="w"> </span><span class="nx">netip</span><span class="p">.</span><span class="nx">Addr</span>
<span class="w"> </span><span class="nx">DstAddr</span><span class="w"> </span><span class="nx">netip</span><span class="p">.</span><span class="nx">Addr</span>
<span class="w"> </span><span class="nx">NextHop</span><span class="w"> </span><span class="nx">netip</span><span class="p">.</span><span class="nx">Addr</span>
<span class="w"> </span><span class="c1">// Core component may override them</span>
<span class="w"> </span><span class="nx">SrcAS</span><span class="w"> </span><span class="kt">uint32</span>
<span class="w"> </span><span class="nx">DstAS</span><span class="w"> </span><span class="kt">uint32</span>
<span class="w"> </span><span class="nx">GotASPath</span><span class="w"> </span><span class="kt">bool</span>
<span class="w"> </span><span class="c1">// protobuf is the protobuf representation for the information not contained above.</span>
<span class="w"> </span><span class="nx">protobuf</span><span class="w"> </span><span class="p">[]</span><span class="kt">byte</span>
<span class="w"> </span><span class="nx">protobufSet</span><span class="w"> </span><span class="nx">bitset</span><span class="p">.</span><span class="nx">BitSet</span>
<span class="p">}</span>
</pre></div>
<p>The <code>protobuf</code> slice holds encoded data. It is initialized with a capacity of
500 bytes to avoid resizing during encoding. There is also some reserved room at
the beginning to be able to encode the total size as a variable-width integer.
Upon finalizing encoding, the remaining fields are added and the message length
is prefixed:</p>
<div class="language-go codehilite"><pre><span/><span class="kd">func</span><span class="w"> </span><span class="p">(</span><span class="nx">schema</span><span class="w"> </span><span class="o">*</span><span class="nx">Schema</span><span class="p">)</span><span class="w"> </span><span class="nx">ProtobufMarshal</span><span class="p">(</span><span class="nx">bf</span><span class="w"> </span><span class="o">*</span><span class="nx">FlowMessage</span><span class="p">)</span><span class="w"> </span><span class="p">[]</span><span class="kt">byte</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nx">schema</span><span class="p">.</span><span class="nx">ProtobufAppendVarint</span><span class="p">(</span><span class="nx">bf</span><span class="p">,</span><span class="w"> </span><span class="nx">ColumnTimeReceived</span><span class="p">,</span><span class="w"> </span><span class="nx">bf</span><span class="p">.</span><span class="nx">TimeReceived</span><span class="p">)</span>
<span class="w"> </span><span class="nx">schema</span><span class="p">.</span><span class="nx">ProtobufAppendVarint</span><span class="p">(</span><span class="nx">bf</span><span class="p">,</span><span class="w"> </span><span class="nx">ColumnSamplingRate</span><span class="p">,</span><span class="w"> </span><span class="nb">uint64</span><span class="p">(</span><span class="nx">bf</span><span class="p">.</span><span class="nx">SamplingRate</span><span class="p">))</span>
<span class="w"> </span><span class="nx">schema</span><span class="p">.</span><span class="nx">ProtobufAppendIP</span><span class="p">(</span><span class="nx">bf</span><span class="p">,</span><span class="w"> </span><span class="nx">ColumnExporterAddress</span><span class="p">,</span><span class="w"> </span><span class="nx">bf</span><span class="p">.</span><span class="nx">ExporterAddress</span><span class="p">)</span>
<span class="w"> </span><span class="nx">schema</span><span class="p">.</span><span class="nx">ProtobufAppendVarint</span><span class="p">(</span><span class="nx">bf</span><span class="p">,</span><span class="w"> </span><span class="nx">ColumnSrcAS</span><span class="p">,</span><span class="w"> </span><span class="nb">uint64</span><span class="p">(</span><span class="nx">bf</span><span class="p">.</span><span class="nx">SrcAS</span><span class="p">))</span>
<span class="w"> </span><span class="nx">schema</span><span class="p">.</span><span class="nx">ProtobufAppendVarint</span><span class="p">(</span><span class="nx">bf</span><span class="p">,</span><span class="w"> </span><span class="nx">ColumnDstAS</span><span class="p">,</span><span class="w"> </span><span class="nb">uint64</span><span class="p">(</span><span class="nx">bf</span><span class="p">.</span><span class="nx">DstAS</span><span class="p">))</span>
<span class="w"> </span><span class="nx">schema</span><span class="p">.</span><span class="nx">ProtobufAppendIP</span><span class="p">(</span><span class="nx">bf</span><span class="p">,</span><span class="w"> </span><span class="nx">ColumnSrcAddr</span><span class="p">,</span><span class="w"> </span><span class="nx">bf</span><span class="p">.</span><span class="nx">SrcAddr</span><span class="p">)</span>
<span class="w"> </span><span class="nx">schema</span><span class="p">.</span><span class="nx">ProtobufAppendIP</span><span class="p">(</span><span class="nx">bf</span><span class="p">,</span><span class="w"> </span><span class="nx">ColumnDstAddr</span><span class="p">,</span><span class="w"> </span><span class="nx">bf</span><span class="p">.</span><span class="nx">DstAddr</span><span class="p">)</span>
<span class="w"> </span><span class="c1">// Add length and move it as a prefix</span>
<span class="w"> </span><span class="nx">end</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nb">len</span><span class="p">(</span><span class="nx">bf</span><span class="p">.</span><span class="nx">protobuf</span><span class="p">)</span>
<span class="w"> </span><span class="nx">payloadLen</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">end</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="nx">maxSizeVarint</span>
<span class="w"> </span><span class="nx">bf</span><span class="p">.</span><span class="nx">protobuf</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">protowire</span><span class="p">.</span><span class="nx">AppendVarint</span><span class="p">(</span><span class="nx">bf</span><span class="p">.</span><span class="nx">protobuf</span><span class="p">,</span><span class="w"> </span><span class="nb">uint64</span><span class="p">(</span><span class="nx">payloadLen</span><span class="p">))</span>
<span class="w"> </span><span class="nx">sizeLen</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nb">len</span><span class="p">(</span><span class="nx">bf</span><span class="p">.</span><span class="nx">protobuf</span><span class="p">)</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="nx">end</span>
<span class="w"> </span><span class="nx">result</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">bf</span><span class="p">.</span><span class="nx">protobuf</span><span class="p">[</span><span class="nx">maxSizeVarint</span><span class="o">-</span><span class="nx">sizeLen</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="nx">end</span><span class="p">]</span>
<span class="w"> </span><span class="nb">copy</span><span class="p">(</span><span class="nx">result</span><span class="p">,</span><span class="w"> </span><span class="nx">bf</span><span class="p">.</span><span class="nx">protobuf</span><span class="p">[</span><span class="nx">end</span><span class="p">:</span><span class="nx">end</span><span class="o">+</span><span class="nx">sizeLen</span><span class="p">])</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nx">result</span>
<span class="p">}</span>
</pre></div>
<p>Minimizing allocations is critical for maintaining encoding performance. The
benchmark tests should be run with the <code>-benchmem</code> flag to monitor allocation
numbers. Each allocation incurs an additional cost to the garbage collector. The
<a href="https://go.dev/blog/pprof" title="Profiling Go Programs">Go profiler</a> is a valuable tool for identifying areas of code that can be
optimized:</p>
<div class="language-bash-session codehilite"><pre><span/><span class="gp">$ </span>go<span class="w"> </span><span class="nb">test</span><span class="w"> </span>-run<span class="o">=</span>__nothing__<span class="w"> </span>-bench<span class="o">=</span>Netflow/with_encoding<span class="w"> </span><span class="se">\</span>
<span class="gp">> </span><span class="w"> </span>-benchmem<span class="w"> </span>-cpuprofile<span class="w"> </span>profile.out<span class="w"> </span><span class="se">\</span>
<span class="gp">> </span><span class="w"> </span>akvorado/inlet/flow
<span class="go">goos: linux</span>
<span class="go">goarch: amd64</span>
<span class="go">pkg: akvorado/inlet/flow</span>
<span class="go">cpu: AMD Ryzen 5 5600X 6-Core Processor</span>
<span class="go">Netflow/with_encoding-12 143953 7955 ns/op 8256 B/op 134 allocs/op</span>
<span class="go">PASS</span>
<span class="go">ok akvorado/inlet/flow 1.418s</span>
<span class="gp">$ </span>go<span class="w"> </span>tool<span class="w"> </span>pprof<span class="w"> </span>profile.out
<span class="go">File: flow.test</span>
<span class="go">Type: cpu</span>
<span class="go">Time: Feb 4, 2023 at 8:12pm (CET)</span>
<span class="go">Duration: 1.41s, Total samples = 2.08s (147.96%)</span>
<span class="go">Entering interactive mode (type "help" for commands, "o" for options)</span>
<span class="gp gp-VirtualEnv">(pprof)</span> <span class="go">web</span>
</pre></div>
<p>After <a href="https://github.com/akvorado/akvorado/commit/e352202631a898947925337232647fdce50aa0f1">using the internal schema</a> instead of code generated from the
proto definition file, the performance improved. However, this comparison is not
entirely fair as less information is being decoded and previously <em>GoFlow2</em> was
decoding to its own structure, which was then copied to our own version.</p>
<div class="language-text-only codehilite"><pre><span/>goos: linux
goarch: amd64
pkg: akvorado/inlet/flow
cpu: AMD Ryzen 5 5600X 6-Core Processor
│ bench-2.txt │ bench-3.txt │
│ sec/op │ sec/op vs base │
Netflow/with_encoding-12 10.284µ ± 2% 7.758µ ± 3% -24.56% (p=0.000 n=10)
Netflow/without_encoding-12 8.975µ ± 2% 7.304µ ± 2% -18.61% (p=0.000 n=10)
Sflow/with_encoding-12 16.67µ ± 2% 14.26µ ± 1% -14.50% (p=0.000 n=10)
Sflow/without_encoding-12 14.87µ ± 1% 13.56µ ± 2% -8.80% (p=0.000 n=10)
</pre></div>
<p>As for testing, we use <a href="https://pkg.go.dev/github.com/jhump/protoreflect"><code>github.com/jhump/protoreflect</code></a>: the
<code>protoparse</code> package parses the proto definition file we generate and the
<code>dynamic</code> package decodes the messages. Check the <a href="https://github.com/akvorado/akvorado/blob/9c51b2284513526f6491a9138953e0bd00f680a8/common/schema/tests.go#L44-L117"><code>ProtobufDecode()</code>
method</a> for more details.<sup id="fnref-protoprint"><a class="footnote-ref" href="#fn-protoprint">4</a></sup></p>
<p>To get the final figures, I have also optimized the decoding in <em>GoFlow2</em>. It
was relying heavily on <a href="https://pkg.go.dev/encoding/binary#Read"><code>binary.Read()</code></a>. This function may use
reflection in certain cases and each call allocates a byte array to read data.
<a href="https://github.com/netsampler/goflow2/pull/141" title="#141: decoders: replace binary.Read with a version without reflection and allocations">Replacing it with a more efficient version</a> provides the following
improvement:</p>
<div class="language-text-only codehilite"><pre><span/>goos: linux
goarch: amd64
pkg: akvorado/inlet/flow
cpu: AMD Ryzen 5 5600X 6-Core Processor
│ bench-3.txt │ bench-4.txt │
│ sec/op │ sec/op vs base │
Netflow/with_encoding-12 7.758µ ± 3% 7.365µ ± 2% -5.07% (p=0.000 n=10)
Netflow/without_encoding-12 7.304µ ± 2% 6.931µ ± 3% -5.11% (p=0.000 n=10)
Sflow/with_encoding-12 14.256µ ± 1% 9.834µ ± 2% -31.02% (p=0.000 n=10)
Sflow/without_encoding-12 13.559µ ± 2% 9.353µ ± 2% -31.02% (p=0.000 n=10)
</pre></div>
<p>It is now easier to collect <a href="https://demo.akvorado.net/docs/internals#schema">new data</a> and the inlet component is faster! 🚅</p>
<div class="admonition">
<p class="admonition-title">Notice</p>
<p>Some paragraphs were editorialized by <a href="https://chat.openai.com/chat">ChatGPT</a>, using
“editorialize and keep it short” as a prompt. The result was proofread by a
human for correctness. The main idea is that <em>ChatGPT</em> should be better at
English than me.</p>
</div>
<div class="footnote">
<hr/>
<ol>
<li id="fn-cost">
<p>While empty fields are not serialized to <em>Protocol Buffers</em>, empty
columns in <em>ClickHouse</em> take some space, even if they compress well.
Moreover, unused fields are still decoded and they may clutter the
interface. <a class="footnote-backref" href="#fnref-cost" title="Jump back to footnote 1 in the text">↩︎</a></p>
</li>
<li id="fn-netflow">
<p>There is a similar function using <em>NetFlow</em>. NetFlow and IPFIX
protocols are less complex to decode than sFlow as they are using a simpler
<abbr title="Type-Length-Value">TLV</abbr> structure. <a class="footnote-backref" href="#fnref-netflow" title="Jump back to footnote 2 in the text">↩︎</a></p>
</li>
<li id="fn-vtprotobuf">
<p><code>vtprotobuf</code> generates more optimized Go code by removing an
abstraction layer. It directly generates the code encoding each field to
bytes:</p>
<div class="language-go codehilite"><pre><span/><span class="k">if</span><span class="w"> </span><span class="nx">m</span><span class="p">.</span><span class="nx">OutIfSpeed</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nx">i</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">encodeVarint</span><span class="p">(</span><span class="nx">dAtA</span><span class="p">,</span><span class="w"> </span><span class="nx">i</span><span class="p">,</span><span class="w"> </span><span class="nb">uint64</span><span class="p">(</span><span class="nx">m</span><span class="p">.</span><span class="nx">OutIfSpeed</span><span class="p">))</span>
<span class="w"> </span><span class="nx">i</span><span class="o">--</span>
<span class="w"> </span><span class="nx">dAtA</span><span class="p">[</span><span class="nx">i</span><span class="p">]</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="mh">0x6</span>
<span class="w"> </span><span class="nx">i</span><span class="o">--</span>
<span class="w"> </span><span class="nx">dAtA</span><span class="p">[</span><span class="nx">i</span><span class="p">]</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="mh">0xd8</span>
<span class="p">}</span>
</pre></div>
<p><a class="footnote-backref" href="#fnref-vtprotobuf" title="Jump back to footnote 3 in the text">↩︎</a></p>
</li>
<li id="fn-protoprint">
<p>There is also a <code>protoprint</code> package to generate proto definition
file. I did not use it. <a class="footnote-backref" href="#fnref-protoprint" title="Jump back to footnote 4 in the text">↩︎</a></p>
</li>
</ol>
</div>
Managing infrastructure with Terraform, CDKTF, and NixOSVincent Bernat2022-12-26T15:18:38Zhttp://www.luffy.cx/en/blog/2022-cdktf-nixos.html
<p>A few years ago, I downsized my personal infrastructure. Until 2018, there were
a dozen containers running on a single <a href="https://www.hetzner.com/" title="Hetzner: dedicated servers, cloud, storage & hosting">Hetzner</a> server.<sup id="fnref-hetzner"><a class="footnote-ref" href="#fn-hetzner">1</a></sup> I migrated
my emails to <a href="https://www.fastmail.com/">Fastmail</a> and my DNS zones to <a href="https://www.gandi.net/en" title="Gandi: Domain Names, Web Hosting, Mail">Gandi</a>. It left me with only my
blog to self-host. As of today, my low-scale infrastructure is composed of 4
virtual machines running <a href="https://nixos.org/" title="Nix: reproducible builds and deployments">NixOS</a> on <em>Hetzner Cloud</em> and <a href="https://www.vultr.com/" title="Vultr.com: SSD VPS Servers, Cloud Servers and Cloud Hosting">Vultr</a>, a handful
of DNS zones on <em>Gandi</em> and <a href="https://aws.amazon.com/route53/" title="Amazon AWS: Route 53">Route 53</a>, and a couple of <a href="https://aws.amazon.com/cloudfront/" title="Amazon AWS: CloudFront">Cloudfront</a>
distributions. It is managed by <a href="https://developer.hashicorp.com/terraform/cdktf" title="Cloud Development Kit for Terraform">CDK for Terraform</a> (<abbr title="Cloud Development Kit for Terraform">CDKTF</abbr>), while <em>NixOS</em>
deployments are handled by <a href="https://github.com/NixOS/nixops" title="NixOps: tool for deploying to NixOS machines in a network or cloud">NixOps</a>.</p>
<p>In this article, I provide a brief introduction to <em>Terraform</em>, <em><abbr title="Cloud Development Kit for Terraform">CDKTF</abbr></em>, and the
<em>Nix</em> ecosystem. I also explain how to use <em>Nix</em> to access these tools within
your shell, so you can quickly start using them.</p>
<div class="admonition">
<p class="admonition-title">Update (2023-11)</p>
<p><em>HashiCorp</em> <a href="https://www.hashicorp.com/blog/hashicorp-adopts-business-source-license">switched to the Business Source License</a>
for all its software. It is quite a disappointment, especially for <em>Terraform</em>,
a key component in automation as it has benefited from a large community. There
is a community-driven fork, <a href="https://opentofu.org/" title="The open source infrastructure as code tool">OpenTofu</a>. However, it does not cover <em><abbr title="Cloud Development Kit for Terraform">CDKTF</abbr></em>.</p>
</div>
<div class="toc">
<ul>
<li><a href="#cdktf-infrastructure-as-code">CDKTF: infrastructure as code</a><ul>
<li><a href="#managing-servers">Managing servers</a></li>
<li><a href="#managing-dns-records">Managing DNS records</a></li>
<li><a href="#about-pulumi">About Pulumi</a></li>
</ul>
</li>
<li><a href="#nixos-nixops">NixOS & NixOps</a><ul>
<li><a href="#nixos-declarative-linux-distribution">NixOS: declarative Linux distribution</a></li>
<li><a href="#nixops-nixos-deployment-tool">NixOps: NixOS deployment tool</a></li>
</ul>
</li>
<li><a href="#tying-everything-together-with-nix">Tying everything together with Nix</a><ul>
<li><a href="#brief-introduction-to-nix-flakes">Brief introduction to Nix flakes</a></li>
<li><a href="#nix-and-cdktf">Nix and CDKTF</a></li>
<li><a href="#nixops">NixOps</a></li>
</ul>
</li>
</ul>
</div>
<h1 id="cdktf-infrastructure-as-code"><abbr title="Cloud Development Kit for Terraform">CDKTF</abbr>: infrastructure as code<a class="headerlink" href="#cdktf-infrastructure-as-code" title="Permanent link"></a></h1>
<p><a href="https://www.terraform.io/">Terraform</a> is an “infrastructure-as-code” tool. You can define your
infrastructure by declaring resources with the <a href="https://developer.hashicorp.com/terraform/language/modules"><abbr title="HashiCorp Configuration Language">HCL</abbr> language</a>. This language
has some additional features like loops to declare several resources from a
list, built-in functions you can call in expressions, and string templates.
<em>Terraform</em> relies on a <a href="https://registry.terraform.io/browse/providers">large set of providers</a> to manage resources.</p>
<h2 id="managing-servers">Managing servers<a class="headerlink" href="#managing-servers" title="Permanent link"></a></h2>
<p>Here is a short example using the <a href="https://registry.terraform.io/providers/hetznercloud/hcloud/latest/docs">Hetzner Cloud provider</a> to spawn a virtual
machine:</p>
<div class="language-terraform codehilite"><pre><span/><span class="kr">variable</span><span class="w"> </span><span class="nv">"hcloud_token"</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="na">sensitive</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="no">true</span>
<span class="p">}</span>
<span class="kr">provider</span><span class="w"> </span><span class="nv">"hcloud"</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="na">token</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nv">var.hcloud_token</span>
<span class="p">}</span>
<span class="kr">resource</span><span class="w"> </span><span class="nc">"hcloud_server"</span><span class="w"> </span><span class="nv">"web03"</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="na">name</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">"web03"</span>
<span class="w"> </span><span class="na">server_type</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">"cpx11"</span>
<span class="w"> </span><span class="na">image</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">"debian-11"</span>
<span class="w"> </span><span class="na">datacenter</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">"nbg1-dc3"</span>
<span class="p">}</span>
<span class="kr">resource</span><span class="w"> </span><span class="nc">"hcloud_rdns"</span><span class="w"> </span><span class="nv">"rdns4-web03"</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="na">server_id</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nv">hcloud_server.web03.id</span>
<span class="w"> </span><span class="na">ip_address</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nv">hcloud_server.web03.ipv4_address</span>
<span class="w"> </span><span class="na">dns_ptr</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">"web03.luffy.cx"</span>
<span class="p">}</span>
<span class="kr">resource</span><span class="w"> </span><span class="nc">"hcloud_rdns"</span><span class="w"> </span><span class="nv">"rdns6-web03"</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="na">server_id</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nv">hcloud_server.web03.id</span>
<span class="w"> </span><span class="na">ip_address</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nv">hcloud_server.web03.ipv6_address</span>
<span class="w"> </span><span class="na">dns_ptr</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">"web03.luffy.cx"</span>
<span class="p">}</span>
</pre></div>
<p><em><abbr title="HashiCorp Configuration Language">HCL</abbr></em> expressiveness is quite limited and I find a general-purpose language more
convenient to describe all the resources. This is where <em>CDK for Terraform</em>
comes in: you can manage your infrastructure using your preferred programming
language, including TypeScript, Go, and Python. Here is the previous example
using <em><abbr title="Cloud Development Kit for Terraform">CDKTF</abbr></em> and TypeScript:</p>
<div class="language-typescript codehilite"><pre><span/><span class="k">import</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nx">App</span><span class="p">,</span><span class="w"> </span><span class="nx">TerraformStack</span><span class="p">,</span><span class="w"> </span><span class="nx">Fn</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="kr">from</span><span class="w"> </span><span class="s2">"cdktf"</span><span class="p">;</span>
<span class="k">import</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nx">HcloudProvider</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="kr">from</span><span class="w"> </span><span class="s2">"./.gen/providers/hcloud/provider"</span><span class="p">;</span>
<span class="k">import</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="kr">as</span><span class="w"> </span><span class="nx">hcloud</span><span class="w"> </span><span class="kr">from</span><span class="w"> </span><span class="s2">"./.gen/providers/hcloud"</span><span class="p">;</span>
<span class="kd">class</span><span class="w"> </span><span class="nx">MyStack</span><span class="w"> </span><span class="k">extends</span><span class="w"> </span><span class="nx">TerraformStack</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="kr">constructor</span><span class="p">(</span><span class="nx">scope</span><span class="o">:</span><span class="w"> </span><span class="kt">Construct</span><span class="p">,</span><span class="w"> </span><span class="nx">name</span><span class="o">:</span><span class="w"> </span><span class="kt">string</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">super</span><span class="p">(</span><span class="nx">scope</span><span class="p">,</span><span class="w"> </span><span class="nx">name</span><span class="p">);</span>
<span class="w"> </span><span class="kd">const</span><span class="w"> </span><span class="nx">hcloudToken</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ow">new</span><span class="w"> </span><span class="nx">TerraformVariable</span><span class="p">(</span><span class="k">this</span><span class="p">,</span><span class="w"> </span><span class="s2">"hcloudToken"</span><span class="p">,</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="kr">type</span><span class="o">:</span><span class="w"> </span><span class="s2">"string"</span><span class="p">,</span>
<span class="w"> </span><span class="nx">sensitive</span><span class="o">:</span><span class="w"> </span><span class="kt">true</span><span class="p">,</span>
<span class="w"> </span><span class="p">});</span>
<span class="w"> </span><span class="kd">const</span><span class="w"> </span><span class="nx">hcloudProvider</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ow">new</span><span class="w"> </span><span class="nx">HcloudProvider</span><span class="p">(</span><span class="k">this</span><span class="p">,</span><span class="w"> </span><span class="s2">"hcloud"</span><span class="p">,</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nx">token</span><span class="o">:</span><span class="w"> </span><span class="kt">hcloudToken.value</span><span class="p">,</span>
<span class="w"> </span><span class="p">});</span>
<span class="w"> </span><span class="kd">const</span><span class="w"> </span><span class="nx">web03</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ow">new</span><span class="w"> </span><span class="nx">hcloud</span><span class="p">.</span><span class="nx">server</span><span class="p">.</span><span class="nx">Server</span><span class="p">(</span><span class="k">this</span><span class="p">,</span><span class="w"> </span><span class="s2">"web03"</span><span class="p">,</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nx">name</span><span class="o">:</span><span class="w"> </span><span class="s2">"web03"</span><span class="p">,</span>
<span class="w"> </span><span class="nx">serverType</span><span class="o">:</span><span class="w"> </span><span class="s2">"cpx11"</span><span class="p">,</span>
<span class="w"> </span><span class="nx">image</span><span class="o">:</span><span class="w"> </span><span class="s2">"debian-11"</span><span class="p">,</span>
<span class="w"> </span><span class="nx">datacenter</span><span class="o">:</span><span class="w"> </span><span class="s2">"nbg1-dc3"</span><span class="p">,</span>
<span class="w"> </span><span class="nx">provider</span><span class="o">:</span><span class="w"> </span><span class="kt">hcloudProvider</span><span class="p">,</span>
<span class="w"> </span><span class="p">});</span>
<span class="w"> </span><span class="ow">new</span><span class="w"> </span><span class="nx">hcloud</span><span class="p">.</span><span class="nx">rdns</span><span class="p">.</span><span class="nx">Rdns</span><span class="p">(</span><span class="k">this</span><span class="p">,</span><span class="w"> </span><span class="s2">"rdns4-web03"</span><span class="p">,</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nx">serverId</span><span class="o">:</span><span class="w"> </span><span class="kt">Fn.tonumber</span><span class="p">(</span><span class="nx">web03</span><span class="p">.</span><span class="nx">id</span><span class="p">),</span>
<span class="w"> </span><span class="nx">ipAddress</span><span class="o">:</span><span class="w"> </span><span class="kt">web03.ipv4Address</span><span class="p">,</span>
<span class="w"> </span><span class="nx">dnsPtr</span><span class="o">:</span><span class="w"> </span><span class="s2">"web03.luffy.cx"</span><span class="p">,</span>
<span class="w"> </span><span class="nx">provider</span><span class="o">:</span><span class="w"> </span><span class="kt">hcloudProvider</span><span class="p">,</span>
<span class="w"> </span><span class="p">});</span>
<span class="w"> </span><span class="ow">new</span><span class="w"> </span><span class="nx">hcloud</span><span class="p">.</span><span class="nx">rdns</span><span class="p">.</span><span class="nx">Rdns</span><span class="p">(</span><span class="k">this</span><span class="p">,</span><span class="w"> </span><span class="s2">"rdns6-web03"</span><span class="p">,</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nx">serverId</span><span class="o">:</span><span class="w"> </span><span class="kt">Fn.tonumber</span><span class="p">(</span><span class="nx">web03</span><span class="p">.</span><span class="nx">id</span><span class="p">),</span>
<span class="w"> </span><span class="nx">ipAddress</span><span class="o">:</span><span class="w"> </span><span class="kt">web03.ipv6Address</span><span class="p">,</span>
<span class="w"> </span><span class="nx">dnsPtr</span><span class="o">:</span><span class="w"> </span><span class="s2">"web03.luffy.cx"</span><span class="p">,</span>
<span class="w"> </span><span class="nx">provider</span><span class="o">:</span><span class="w"> </span><span class="kt">hcloudProvider</span><span class="p">,</span>
<span class="w"> </span><span class="p">});</span>
<span class="w"> </span><span class="p">}</span>
<span class="p">}</span>
<span class="kd">const</span><span class="w"> </span><span class="nx">app</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ow">new</span><span class="w"> </span><span class="nx">App</span><span class="p">();</span>
<span class="ow">new</span><span class="w"> </span><span class="nx">MyStack</span><span class="p">(</span><span class="nx">app</span><span class="p">,</span><span class="w"> </span><span class="s2">"cdktf-take1"</span><span class="p">);</span>
<span class="nx">app</span><span class="p">.</span><span class="nx">synth</span><span class="p">();</span>
</pre></div>
<p>Running <code>cdktf synth</code> generates a configuration file for <em>Terraform</em>, <code>terraform
plan</code> previews the changes, and <code>terraform apply</code> applies them. Now that you
have a general-purpose language, you can use functions.</p>
<h2 id="managing-dns-records">Managing DNS records<a class="headerlink" href="#managing-dns-records" title="Permanent link"></a></h2>
<p>While using <em><abbr title="Cloud Development Kit for Terraform">CDKTF</abbr></em> for 4 web servers may seem a tad overkill, this is quite
different when it comes to managing a few DNS zones. With <a href="https://stackexchange.github.io/dnscontrol/" title="DNSControl: Synchronize your DNS to multiple providers from a simple DSL">DNSControl</a>, which
is using JavaScript as a domain-specific language, I was able to define the
<code>bernat.ch</code> zone with this snippet of code:</p>
<div class="language-javascript codehilite"><pre><span/><span class="nx">D</span><span class="p">(</span><span class="s2">"bernat.ch"</span><span class="p">,</span><span class="w"> </span><span class="nx">REG_NONE</span><span class="p">,</span><span class="w"> </span><span class="nx">DnsProvider</span><span class="p">(</span><span class="nx">DNS_BIND</span><span class="p">,</span><span class="w"> </span><span class="mf">0</span><span class="p">),</span><span class="w"> </span><span class="nx">DnsProvider</span><span class="p">(</span><span class="nx">DNS_GANDI</span><span class="p">),</span>
<span class="w"> </span><span class="nx">DefaultTTL</span><span class="p">(</span><span class="s1">'2h'</span><span class="p">),</span>
<span class="w"> </span><span class="nx">FastMailMX</span><span class="p">(</span><span class="s1">'bernat.ch'</span><span class="p">,</span><span class="w"> </span><span class="p">{</span><span class="nx">subdomains</span><span class="o">:</span><span class="w"> </span><span class="p">[</span><span class="s1">'vincent'</span><span class="p">]}),</span>
<span class="w"> </span><span class="nx">WebServers</span><span class="p">(</span><span class="s1">'@'</span><span class="p">),</span>
<span class="w"> </span><span class="nx">WebServers</span><span class="p">(</span><span class="s1">'vincent'</span><span class="p">);</span>
</pre></div>
<p>This generated 38 records. With <em><abbr title="Cloud Development Kit for Terraform">CDKTF</abbr></em>, I use:</p>
<div class="language-typescript codehilite"><pre><span/><span class="ow">new</span><span class="w"> </span><span class="nx">Route53Zone</span><span class="p">(</span><span class="k">this</span><span class="p">,</span><span class="w"> </span><span class="s2">"bernat.ch"</span><span class="p">,</span><span class="w"> </span><span class="nx">providers</span><span class="p">.</span><span class="nx">aws</span><span class="p">)</span>
<span class="w"> </span><span class="p">.</span><span class="nx">sign</span><span class="p">(</span><span class="nx">dnsCMK</span><span class="p">)</span>
<span class="w"> </span><span class="p">.</span><span class="nx">registrar</span><span class="p">(</span><span class="nx">providers</span><span class="p">.</span><span class="nx">gandiVB</span><span class="p">)</span>
<span class="w"> </span><span class="p">.</span><span class="nx">www</span><span class="p">(</span><span class="s2">"@"</span><span class="p">,</span><span class="w"> </span><span class="nx">servers</span><span class="p">)</span>
<span class="w"> </span><span class="p">.</span><span class="nx">www</span><span class="p">(</span><span class="s2">"vincent"</span><span class="p">,</span><span class="w"> </span><span class="nx">servers</span><span class="p">)</span>
<span class="w"> </span><span class="p">.</span><span class="nx">www</span><span class="p">(</span><span class="s2">"media"</span><span class="p">,</span><span class="w"> </span><span class="nx">servers</span><span class="p">)</span>
<span class="w"> </span><span class="p">.</span><span class="nx">fastmailMX</span><span class="p">([</span><span class="s2">"vincent"</span><span class="p">]);</span>
</pre></div>
<p>All the magic is in the code that I did not show you. You can check the
<a href="https://github.com/vincentbernat/cdktf-take1/blob/main/luffy/dns.ts">dns.ts</a> file in the <a href="https://github.com/vincentbernat/cdktf-take1">cdktf-take1</a> repository to see how it works. Here is a
quick explanation:</p>
<ul>
<li><code>Route53Zone()</code> creates a new zone hosted by <em>Route 53</em>,</li>
<li><code>sign()</code> signs the zone with the provided master key,</li>
<li><code>registrar()</code> registers the zone to the registrar of the domain and sets up DNSSEC,</li>
<li><code>www()</code> creates <code>A</code> and <code>AAAA</code> records for the provided name pointing to the web servers,</li>
<li><code>fastmailMX()</code> creates the <code>MX</code> records and other support records to direct emails to <em>Fastmail</em>.</li>
</ul>
<p>Here is the content of the <code>fastmailMX()</code> function. It generates a few records
and returns the current zone for chaining:</p>
<div class="language-typescript codehilite"><pre><span/><span class="nx">fastmailMX</span><span class="p">(</span><span class="nx">subdomains?</span><span class="o">:</span><span class="w"> </span><span class="kt">string</span><span class="p">[])</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="p">(</span><span class="nx">subdomains</span><span class="w"> </span><span class="o">??</span><span class="w"> </span><span class="p">[])</span>
<span class="w"> </span><span class="p">.</span><span class="nx">concat</span><span class="p">([</span><span class="s2">"@"</span><span class="p">,</span><span class="w"> </span><span class="s2">"*"</span><span class="p">])</span>
<span class="w"> </span><span class="p">.</span><span class="nx">forEach</span><span class="p">((</span><span class="nx">subdomain</span><span class="p">)</span><span class="w"> </span><span class="p">=></span>
<span class="w"> </span><span class="k">this</span><span class="p">.</span><span class="nx">MX</span><span class="p">(</span><span class="nx">subdomain</span><span class="p">,</span><span class="w"> </span><span class="p">[</span>
<span class="w"> </span><span class="s2">"10 in1-smtp.messagingengine.com."</span><span class="p">,</span>
<span class="w"> </span><span class="s2">"20 in2-smtp.messagingengine.com."</span><span class="p">,</span>
<span class="w"> </span><span class="p">])</span>
<span class="w"> </span><span class="p">);</span>
<span class="w"> </span><span class="k">this</span><span class="p">.</span><span class="nx">TXT</span><span class="p">(</span><span class="s2">"@"</span><span class="p">,</span><span class="w"> </span><span class="s2">"v=spf1 include:spf.messagingengine.com ~all"</span><span class="p">);</span>
<span class="w"> </span><span class="p">[</span><span class="s2">"mesmtp"</span><span class="p">,</span><span class="w"> </span><span class="s2">"fm1"</span><span class="p">,</span><span class="w"> </span><span class="s2">"fm2"</span><span class="p">,</span><span class="w"> </span><span class="s2">"fm3"</span><span class="p">].</span><span class="nx">forEach</span><span class="p">((</span><span class="nx">dk</span><span class="p">)</span><span class="w"> </span><span class="p">=></span>
<span class="w"> </span><span class="k">this</span><span class="p">.</span><span class="nx">CNAME</span><span class="p">(</span><span class="sb">`</span><span class="si">${</span><span class="nx">dk</span><span class="si">}</span><span class="sb">._domainkey`</span><span class="p">,</span><span class="w"> </span><span class="sb">`</span><span class="si">${</span><span class="nx">dk</span><span class="si">}</span><span class="sb">.</span><span class="si">${</span><span class="k">this</span><span class="p">.</span><span class="nx">name</span><span class="si">}</span><span class="sb">.dkim.fmhosted.com.`</span><span class="p">)</span>
<span class="w"> </span><span class="p">);</span>
<span class="w"> </span><span class="k">this</span><span class="p">.</span><span class="nx">TXT</span><span class="p">(</span><span class="s2">"_dmarc"</span><span class="p">,</span><span class="w"> </span><span class="s2">"v=DMARC1; p=none; sp=none"</span><span class="p">);</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="k">this</span><span class="p">;</span>
<span class="p">}</span>
</pre></div>
<p>I encourage you to browse the <a href="https://github.com/vincentbernat/cdktf-take1">repository</a> if you need more
information.</p>
<h2 id="about-pulumi">About Pulumi<a class="headerlink" href="#about-pulumi" title="Permanent link"></a></h2>
<p>My first tentative around <em>Terraform</em> was to use <a href="https://www.pulumi.com/" title="Pulumi: universal infrastructure as code">Pulumi</a>. You can find this
attempt on <a href="https://github.com/vincentbernat/pulumi-take1">GitHub</a>. This is quite similar to what I currently do
with <em><abbr title="Cloud Development Kit for Terraform">CDKTF</abbr></em>. The main difference is that I am using Python instead of TypeScript
because I was not familiar with TypeScript at the time.<sup id="fnref-python"><a class="footnote-ref" href="#fn-python">2</a></sup></p>
<p><em>Pulumi</em> predates <em><abbr title="Cloud Development Kit for Terraform">CDKTF</abbr></em> and it uses a slightly different approach. <em><abbr title="Cloud Development Kit for Terraform">CDKTF</abbr></em>
generates a <em>Terraform</em> configuration (in JSON format instead of <abbr title="HashiCorp Configuration Language">HCL</abbr>), delegating
planning, state management, and deployment to <em>Terraform</em>. It is therefore bound
to the limitations of what can be expressed by <em>Terraform</em>, notably when you
need to transform data obtained from one resource to another.<sup id="fnref-transform"><a class="footnote-ref" href="#fn-transform">3</a></sup>
<em>Pulumi</em> needs specific providers for each resource. Many <em>Pulumi</em> providers are
thin wrappers encapsulating <em>Terraform</em> providers.</p>
<p>While <em>Pulumi</em> provides a good user experience, I switched to <abbr title="Cloud Development Kit for Terraform">CDKTF</abbr> because
writing providers for <em>Pulumi</em> is a chore. <abbr title="Cloud Development Kit for Terraform">CDKTF</abbr> does not require such a step.
Outside the big players (AWS, Azure and Google Cloud), the existence, quality,
and freshness of the <em>Pulumi</em> providers are inconsistent. Most providers rely on
a <em>Terraform</em> provider and they may lag a few versions behind, miss a few
resources, or have a few bugs of their own.</p>
<p>When a provider does not exist, you can write one with the help of the
<a href="https://github.com/pulumi/pulumi-terraform-bridge">pulumi-terraform-bridge</a> library. The <em>Pulumi</em> project provides a
<a href="https://github.com/pulumi/pulumi-tf-provider-boilerplate">boilerplate</a> for this purpose. I had a bad experience with it when writing
providers for <a href="https://github.com/vincentbernat/pulumi-gandi-old" title="Deprecated Pulumi provider for Gandi">Gandi</a> and <a href="https://github.com/vincentbernat/pulumi-vultr" title="Pulumi provider for Vultr">Vultr</a>: the
<code>Makefile</code> <a href="https://github.com/pulumi/pulumi-tf-provider-boilerplate/pull/51" title="PR #51: Do not install pulumi automatically">automatically installs <em>Pulumi</em></a> using a <code>curl | sh</code>
pattern and <a href="https://github.com/pulumi/pulumi-tf-provider-boilerplate/pull/52" title="PR #52: Fix Makefile to work with a POSIX shell">does not work with <code>/bin/sh</code></a>. There is a lack of
interest for community-based contributions<sup id="fnref-maint"><a class="footnote-ref" href="#fn-maint">4</a></sup> or even for <a href="https://twitter.com/briggsl/status/1473009153983623176">providers for
smaller players</a>.</p>
<h1 id="nixos-nixops">NixOS & NixOps<a class="headerlink" href="#nixos-nixops" title="Permanent link"></a></h1>
<p><a href="https://nixos.org/manual/nix/stable/language/index.html" title="Nix, the language">Nix</a> is a functional, purely-functional programming language.
<a href="https://nixos.org/manual/nix/stable/introduction.html" title="Nix, the package manager">Nix</a> is also the name of the package manager that is built on top of
the <em>Nix</em> language. It allows users to declaratively install packages.
<a href="https://github.com/NixOS/nixpkgs">nixpkgs</a> is a repository of packages. You can <a href="https://nixos.org/manual/nix/stable/installation/multi-user.html">install <em>Nix</em></a> on
top of a regular Linux distribution. If you want more details, a good resource
is the <a href="https://nixos.org/">official website</a>, and notably the <a href="https://nixos.org/learn.html">“learn” section</a>. There is a steep learning curve, but the reward is tremendous.</p>
<h2 id="nixos-declarative-linux-distribution">NixOS: declarative Linux distribution<a class="headerlink" href="#nixos-declarative-linux-distribution" title="Permanent link"></a></h2>
<p><a href="https://nixos.org/" title="Nix: reproducible builds and deployments">NixOS</a> is a Linux distribution built on top of the <em>Nix</em> package manager.
Here is a configuration snippet to add some packages:</p>
<div class="language-nix codehilite"><pre><span/>environment<span class="o">.</span><span class="ss">systemPackages</span> <span class="o">=</span> <span class="k">with</span> pkgs<span class="p">;</span>
<span class="p">[</span>
bat
htop
liboping
mg
mtr
ncdu
tmux
<span class="p">];</span>
</pre></div>
<p>It is possible to alter an existing derivation<sup id="fnref-derivation"><a class="footnote-ref" href="#fn-derivation">5</a></sup> to use a different version, enable a
specific feature, or apply a patch. Here is how I enable and configure <em>Nginx</em>
to disable the stream module, add the <a href="https://github.com/google/ngx_brotli">Brotli compression module</a>, and add the
<a href="https://github.com/masonicboom/ipscrub">IP address anonymizer module</a>. Moreover, instead of using <em>OpenSSL 3</em>, I keep
using <em>OpenSSL 1.1</em>.<sup id="fnref-openssl"><a class="footnote-ref" href="#fn-openssl">6</a></sup></p>
<div class="language-nix codehilite"><pre><span/>services<span class="o">.</span><span class="ss">nginx</span> <span class="o">=</span> <span class="p">{</span>
<span class="ss">enable</span> <span class="o">=</span> <span class="no">true</span><span class="p">;</span>
<span class="ss">package</span> <span class="o">=</span> <span class="p">(</span>pkgs<span class="o">.</span>nginxStable<span class="o">.</span>override <span class="p">{</span>
<span class="ss">withStream</span> <span class="o">=</span> <span class="no">false</span><span class="p">;</span>
<span class="ss">modules</span> <span class="o">=</span> <span class="k">with</span> pkgs<span class="o">.</span>nginxModules<span class="p">;</span> <span class="p">[</span>
brotli
ipscrub
<span class="p">];</span>
<span class="ss">openssl</span> <span class="o">=</span> pkgs<span class="o">.</span>openssl_1_1<span class="p">;</span>
<span class="p">});</span>
</pre></div>
<p>If you need to add some patches, it is also possible. Here are the patches I
added in 2019 to circumvent the <a href="https://github.com/Netflix/security-bulletins/blob/master/advisories/third-party/2019-002.md">DoS vulnerabilities in <em>Nginx</em></a>
until they were fixed in <em>NixOS</em>:<sup id="fnref-security"><a class="footnote-ref" href="#fn-security">7</a></sup></p>
<div class="language-nix codehilite"><pre><span/>services<span class="o">.</span>nginx<span class="o">.</span><span class="ss">package</span> <span class="o">=</span> pkgs<span class="o">.</span>nginxStable<span class="o">.</span>overrideAttrs <span class="p">(</span>old<span class="p">:</span> <span class="p">{</span>
<span class="ss">patches</span> <span class="o">=</span> old<span class="o">.</span>patches <span class="o">++</span> <span class="p">[</span>
<span class="c1"># HTTP/2: reject zero length headers with PROTOCOL_ERROR.</span>
<span class="p">(</span>pkgs<span class="o">.</span>fetchpatch <span class="p">{</span>
<span class="ss">url</span> <span class="o">=</span> <span class="l">https://github.com/nginx/nginx/commit/dbdd</span><span class="p">[</span><span class="err">…</span><span class="p">]</span><span class="o">.</span>patch<span class="p">;</span>
<span class="ss">sha256</span> <span class="o">=</span> <span class="s2">"a48190[…]"</span><span class="p">;</span>
<span class="p">})</span>
<span class="c1"># HTTP/2: limited number of DATA frames.</span>
<span class="p">(</span>pkgs<span class="o">.</span>fetchpatch <span class="p">{</span>
<span class="ss">url</span> <span class="o">=</span> <span class="l">https://github.com/nginx/nginx/commit/94c5</span><span class="p">[</span><span class="err">…</span><span class="p">]</span><span class="o">.</span>patch<span class="p">;</span>
<span class="ss">sha256</span> <span class="o">=</span> <span class="s2">"af591a[…]"</span><span class="p">;</span>
<span class="p">})</span>
<span class="c1"># HTTP/2: limited number of PRIORITY frames.</span>
<span class="p">(</span>pkgs<span class="o">.</span>fetchpatch <span class="p">{</span>
<span class="ss">url</span> <span class="o">=</span> <span class="l">https://github.com/nginx/nginx/commit/39bb</span><span class="p">[</span><span class="err">…</span><span class="p">]</span><span class="o">.</span>patch<span class="p">;</span>
<span class="ss">sha256</span> <span class="o">=</span> <span class="s2">"1ad8fe[…]"</span><span class="p">;</span>
<span class="p">})</span>
<span class="p">];</span>
<span class="p">});</span>
</pre></div>
<p>If you are interested, have a look at my relatively small configuration:
<a href="https://github.com/vincentbernat/nixops-take1/blob/master/tags/common.nix"><code>common.nix</code></a> contains the configuration to be applied to any host
(SSH, users, common software packages), <a href="https://github.com/vincentbernat/nixops-take1/blob/master/tags/web.nix"><code>web.nix</code></a> contains the
configuration for the web servers, <a href="https://github.com/vincentbernat/nixops-take1/blob/master/tags/isso.nix"><code>isso.nix</code></a> runs <a href="/en/blog/2020-docker-nixos-isso" title="Running Isso on NixOS in a Docker container">Isso</a> into a
<em>systemd</em> container.</p>
<h2 id="nixops-nixos-deployment-tool">NixOps: NixOS deployment tool<a class="headerlink" href="#nixops-nixos-deployment-tool" title="Permanent link"></a></h2>
<p>On a single node, <em>NixOS</em> configuration is in the <code>/etc/nixos/configuration.nix</code>
file. After modifying it, you have to run <code>nixos-rebuild switch</code>. <em>Nix</em> fetches
all possible dependencies from the binary cache and builds the remaining
packages. It creates a new entry in the boot loader menu and activates the new
configuration.</p>
<p>To manage several nodes, there exists several options, including <a href="https://github.com/NixOS/nixops" title="NixOps: tool for deploying to NixOS machines in a network or cloud">NixOps</a>,
<a href="https://github.com/serokell/deploy-rs" title="simple multi-profile Nix flake deploy tool">deploy-rs</a>, <a href="https://github.com/zhaofengli/colmena" title="Simple, stateless NixOS deployment tool">Colmena</a>, and <a href="https://github.com/DBCDK/morph/" title="NixOS deployment tool">morph</a>. I do not know all of them, but from
my point of view, the differences are not that important. It is also possible to
build such a tool yourself as <em>Nix</em> provides the most important building blocks:
<code>nix build</code> and <code>nix copy</code>. <em>NixOps</em> is one of the first tools available but I
encourage you to explore the alternatives.</p>
<p><em>NixOps</em> configuration is written in <em>Nix</em>. Here is a simplified configuration
to deploy <code>znc01.luffy.cx</code>, <code>web01.luffy.cx</code>, and <code>web02.luffy.cx</code>, with the
help of the <code>server</code> and <code>web</code> functions:</p>
<div class="language-nix codehilite"><pre><span/><span class="k">let</span>
<span class="ss">server</span> <span class="o">=</span> hardware<span class="p">:</span> name<span class="p">:</span> imports<span class="p">:</span> <span class="p">{</span>
deployment<span class="o">.</span><span class="ss">targetHost</span> <span class="o">=</span> <span class="s2">"</span><span class="si">${</span>name<span class="si">}</span><span class="s2">.luffy.cx"</span><span class="p">;</span>
networking<span class="o">.</span><span class="ss">hostName</span> <span class="o">=</span> name<span class="p">;</span>
networking<span class="o">.</span><span class="ss">domain</span> <span class="o">=</span> <span class="s2">"luffy.cx"</span><span class="p">;</span>
<span class="ss">imports</span> <span class="o">=</span> <span class="p">[</span> <span class="p">(</span><span class="l">./hardware/.</span> <span class="o">+</span> <span class="s2">"/</span><span class="si">${</span>hardware<span class="si">}</span><span class="s2">.nix"</span><span class="p">)</span> <span class="p">]</span> <span class="o">++</span> imports<span class="p">;</span>
<span class="p">};</span>
<span class="ss">web</span> <span class="o">=</span> hardware<span class="p">:</span> idx<span class="p">:</span> imports<span class="p">:</span>
server hardware <span class="s2">"web</span><span class="si">${</span>lib<span class="o">.</span>fixedWidthNumber <span class="mi">2</span> idx<span class="si">}</span><span class="s2">"</span> <span class="p">([</span> <span class="l">./web.nix</span> <span class="p">]</span> <span class="o">++</span> imports<span class="p">);</span>
<span class="k">in</span> <span class="p">{</span>
network<span class="o">.</span><span class="ss">description</span> <span class="o">=</span> <span class="s2">"Luffy infrastructure"</span><span class="p">;</span>
network<span class="o">.</span><span class="ss">enableRollback</span> <span class="o">=</span> <span class="no">true</span><span class="p">;</span>
<span class="ss">defaults</span> <span class="o">=</span> <span class="nb">import</span> <span class="l">./common.nix</span><span class="p">;</span>
<span class="ss">znc01</span> <span class="o">=</span> server <span class="s2">"exoscale"</span> <span class="p">[</span> <span class="l">./znc.nix</span> <span class="p">];</span>
<span class="ss">web01</span> <span class="o">=</span> web <span class="s2">"hetzner"</span> <span class="mi">1</span> <span class="p">[</span> <span class="l">./isso.nix</span> <span class="p">];</span>
<span class="ss">web02</span> <span class="o">=</span> web <span class="s2">"hetzner"</span> <span class="mi">2</span> <span class="p">[];</span>
<span class="p">}</span>
</pre></div>
<h1 id="tying-everything-together-with-nix">Tying everything together with Nix<a class="headerlink" href="#tying-everything-together-with-nix" title="Permanent link"></a></h1>
<p>The <em>Nix</em> ecosystem is a unified solution to the various problems around
software and configuration management. A very interesting feature is the
<a href="https://web.archive.org/web/2022/https://nixos.org/guides/declarative-and-reproducible-developer-environments.html">declarative and reproducible developer environments</a>. This is similar to
Python virtual environments, except it is not language-specific.</p>
<h2 id="brief-introduction-to-nix-flakes">Brief introduction to Nix flakes<a class="headerlink" href="#brief-introduction-to-nix-flakes" title="Permanent link"></a></h2>
<p>I am using <a href="https://nixos.wiki/wiki/Flakes" title="Flakes on NixOS wiki">flakes</a>, a new <em>Nix</em> feature improving reproducibility by pinning
all dependencies and making the build hermetic. While the feature is marked as
experimental,<sup id="fnref-experimental"><a class="footnote-ref" href="#fn-experimental">8</a></sup> it is widely used and you may see <code>flake.nix</code> and
<code>flake.lock</code> at the root of some repositories.</p>
<p>As a short example, here is the <code>flake.nix</code> content shipped with <a href="/en/blog/2013-snimpy" title="Snimpy: SNMP & Python">Snimpy</a>, an
interactive SNMP tool for Python relying on <a href="https://www.ibr.cs.tu-bs.de/projects/libsmi/" title="libsmi: library to Access SMI MIB Information">libsmi</a>, a C library:</p>
<div class="language-nix codehilite"><pre><span/><span class="p">{</span>
<span class="ss">inputs</span> <span class="o">=</span> <span class="p">{</span>
nixpkgs<span class="o">.</span><span class="ss">url</span> <span class="o">=</span> <span class="s2">"nixpkgs"</span><span class="p">;</span>
flake-utils<span class="o">.</span><span class="ss">url</span> <span class="o">=</span> <span class="s2">"github:numtide/flake-utils"</span><span class="p">;</span>
<span class="p">};</span>
<span class="ss">outputs</span> <span class="o">=</span> <span class="p">{</span> self<span class="p">,</span> <span class="o">...</span> <span class="p">}@</span>inputs<span class="p">:</span>
inputs<span class="o">.</span>flake-utils<span class="o">.</span>lib<span class="o">.</span>eachDefaultSystem <span class="p">(</span>system<span class="p">:</span>
<span class="k">let</span>
<span class="ss">pkgs</span> <span class="o">=</span> inputs<span class="o">.</span>nixpkgs<span class="o">.</span>legacyPackages<span class="o">.</span><span class="s2">"</span><span class="si">${</span>system<span class="si">}</span><span class="s2">"</span><span class="p">;</span>
<span class="k">in</span>
<span class="p">{</span>
<span class="c1"># nix build</span>
packages<span class="o">.</span><span class="ss">default</span> <span class="o">=</span> pkgs<span class="o">.</span>python3Packages<span class="o">.</span>buildPythonPackage <span class="p">{</span>
<span class="ss">name</span> <span class="o">=</span> <span class="s2">"snimpy"</span><span class="p">;</span>
<span class="ss">src</span> <span class="o">=</span> self<span class="p">;</span>
<span class="ss">preConfigure</span> <span class="o">=</span> <span class="s s-Multiline">''echo "1.0.0-0-000000000000" > version.txt''</span><span class="p">;</span>
<span class="ss">checkPhase</span> <span class="o">=</span> <span class="s2">"pytest"</span><span class="p">;</span>
<span class="ss">checkInputs</span> <span class="o">=</span> <span class="k">with</span> pkgs<span class="o">.</span>python3Packages<span class="p">;</span> <span class="p">[</span> pytest mock coverage <span class="p">];</span>
<span class="ss">propagatedBuildInputs</span> <span class="o">=</span> <span class="k">with</span> pkgs<span class="o">.</span>python3Packages<span class="p">;</span> <span class="p">[</span> cffi pysnmp ipython <span class="p">];</span>
<span class="ss">buildInputs</span> <span class="o">=</span> <span class="p">[</span> pkgs<span class="o">.</span>libsmi <span class="p">];</span>
<span class="p">};</span>
<span class="c1"># nix run + nix shell</span>
apps<span class="o">.</span><span class="ss">default</span> <span class="o">=</span> <span class="p">{</span>
<span class="ss">type</span> <span class="o">=</span> <span class="s2">"app"</span><span class="p">;</span>
<span class="ss">program</span> <span class="o">=</span> <span class="s2">"</span><span class="si">${</span>self<span class="o">.</span>packages<span class="o">.</span><span class="s2">"</span><span class="si">${</span>system<span class="si">}</span><span class="s2">"</span><span class="o">.</span>default<span class="si">}</span><span class="s2">/bin/snimpy"</span><span class="p">;</span>
<span class="p">};</span>
<span class="c1"># nix develop</span>
devShells<span class="o">.</span><span class="ss">default</span> <span class="o">=</span> pkgs<span class="o">.</span>mkShell <span class="p">{</span>
<span class="ss">name</span> <span class="o">=</span> <span class="s2">"snimpy-dev"</span><span class="p">;</span>
<span class="ss">buildInputs</span> <span class="o">=</span> <span class="p">[</span>
self<span class="o">.</span>packages<span class="o">.</span><span class="s2">"</span><span class="si">${</span>system<span class="si">}</span><span class="s2">"</span><span class="o">.</span>default<span class="o">.</span>inputDerivation
pkgs<span class="o">.</span>python3Packages<span class="o">.</span>ipython
<span class="p">];</span>
<span class="p">};</span>
<span class="p">});</span>
<span class="p">}</span>
</pre></div>
<p>If you have <em>Nix</em> installed on your system:</p>
<ul>
<li><code>nix run github:vincentbernat/snimpy</code> runs <em>Snimpy</em>,</li>
<li><code>nix shell github:vincentbernat/snimpy</code> provides a shell with <em>Snimpy</em> ready-to-use,</li>
<li><code>nix build github:vincentbernat/snimpy</code> builds the Python package, tests included, and</li>
<li><code>nix develop .</code> provides a shell to hack around <em>Snimpy</em>—when run from a fresh checkout.<sup id="fnref-checkout"><a class="footnote-ref" href="#fn-checkout">9</a></sup></li>
</ul>
<p>For more information about <em>Nix flakes</em>, have a look at <a href="https://www.tweag.io/blog/2020-05-25-flakes/" title="Nix Flakes, Part 1: An introduction and tutorial">the tutorial from
Tweag</a>.</p>
<h2 id="nix-and-cdktf">Nix and <abbr title="Cloud Development Kit for Terraform">CDKTF</abbr><a class="headerlink" href="#nix-and-cdktf" title="Permanent link"></a></h2>
<p>At the root of the <a href="https://github.com/vincentbernat/cdktf-take1">repository I use for <abbr title="Cloud Development Kit for Terraform">CDKTF</abbr></a>, there is a
<a href="https://github.com/vincentbernat/cdktf-take1/blob/main/flake.nix"><code>flake.nix</code></a> file to set up a shell with <em>Terraform</em> and
<em><abbr title="Cloud Development Kit for Terraform">CDKTF</abbr></em> installed and with the appropriate environment variables to automate my
infrastructure.</p>
<p><em>Terraform</em> is already packaged in <em>nixpkgs</em>. However, I need to apply a patch
on top of the <em>Gandi</em> provider. Not a problem with <em>Nix</em>!</p>
<div class="language-nix codehilite"><pre><span/><span class="ss">terraform</span> <span class="o">=</span> pkgs<span class="o">.</span>terraform<span class="o">.</span>withPlugins <span class="p">(</span>p<span class="p">:</span> <span class="p">[</span>
p<span class="o">.</span>aws
p<span class="o">.</span>hcloud
p<span class="o">.</span>vultr
<span class="p">(</span>p<span class="o">.</span>gandi<span class="o">.</span>overrideAttrs
<span class="p">(</span>old<span class="p">:</span> <span class="p">{</span>
<span class="ss">src</span> <span class="o">=</span> pkgs<span class="o">.</span>fetchFromGitHub <span class="p">{</span>
<span class="ss">owner</span> <span class="o">=</span> <span class="s2">"vincentbernat"</span><span class="p">;</span>
<span class="ss">repo</span> <span class="o">=</span> <span class="s2">"terraform-provider-gandi"</span><span class="p">;</span>
<span class="ss">rev</span> <span class="o">=</span> <span class="s2">"feature/livedns-key"</span><span class="p">;</span>
<span class="ss">hash</span> <span class="o">=</span> <span class="s2">"sha256-V16BIjo5/rloQ1xTQrdd0snoq1OPuDh3fQNW7kiv/kQ="</span><span class="p">;</span>
<span class="p">};</span>
<span class="p">}))</span>
<span class="p">]);</span>
</pre></div>
<p><em><abbr title="Cloud Development Kit for Terraform">CDKTF</abbr></em> is written in TypeScript. I have a
<a href="https://github.com/vincentbernat/cdktf-take1/blob/main/package.json"><code>package.json</code></a> file with all the dependencies
needed, including the ones to use TypeScript as the language to define
infrastructure:</p>
<div class="language-json codehilite"><pre><span/><span class="p">{</span>
<span class="w"> </span><span class="nt">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"cdktf-take1"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"version"</span><span class="p">:</span><span class="w"> </span><span class="s2">"1.0.0"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"main"</span><span class="p">:</span><span class="w"> </span><span class="s2">"main.js"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"types"</span><span class="p">:</span><span class="w"> </span><span class="s2">"main.ts"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"private"</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"dependencies"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"@types/node"</span><span class="p">:</span><span class="w"> </span><span class="s2">"^14.18.30"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"cdktf"</span><span class="p">:</span><span class="w"> </span><span class="s2">"^0.13.3"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"cdktf-cli"</span><span class="p">:</span><span class="w"> </span><span class="s2">"^0.13.3"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"constructs"</span><span class="p">:</span><span class="w"> </span><span class="s2">"^10.1.151"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"eslint"</span><span class="p">:</span><span class="w"> </span><span class="s2">"^8.27.0"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"prettier"</span><span class="p">:</span><span class="w"> </span><span class="s2">"^2.7.1"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"ts-node"</span><span class="p">:</span><span class="w"> </span><span class="s2">"^10.9.1"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"typescript"</span><span class="p">:</span><span class="w"> </span><span class="s2">"^3.9.10"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"typescript-language-server"</span><span class="p">:</span><span class="w"> </span><span class="s2">"^2.1.0"</span>
<span class="w"> </span><span class="p">}</span>
<span class="p">}</span>
</pre></div>
<p>I use <a href="https://yarnpkg.com/" title="Yarn package manager">Yarn</a> to get a <a href="https://github.com/vincentbernat/cdktf-take1/blob/main/yarn.lock"><code>yarn.lock</code></a> file that can be
used directly to declare a derivation containing all the dependencies:</p>
<div class="language-nix codehilite"><pre><span/><span class="ss">nodeEnv</span> <span class="o">=</span> pkgs<span class="o">.</span>mkYarnModules <span class="p">{</span>
<span class="ss">pname</span> <span class="o">=</span> <span class="s2">"cdktf-take1-js-modules"</span><span class="p">;</span>
<span class="ss">version</span> <span class="o">=</span> <span class="s2">"1.0.0"</span><span class="p">;</span>
<span class="ss">packageJSON</span> <span class="o">=</span> <span class="l">./package.json</span><span class="p">;</span>
<span class="ss">yarnLock</span> <span class="o">=</span> <span class="l">./yarn.lock</span><span class="p">;</span>
<span class="p">};</span>
</pre></div>
<p>The next step is to generate the <em><abbr title="Cloud Development Kit for Terraform">CDKTF</abbr></em> providers from the <em>Terraform</em> providers
and turn them into a derivation:</p>
<div class="language-nix codehilite"><pre><span/><span class="ss">cdktfProviders</span> <span class="o">=</span> pkgs<span class="o">.</span>stdenvNoCC<span class="o">.</span>mkDerivation <span class="p">{</span>
<span class="ss">name</span> <span class="o">=</span> <span class="s2">"cdktf-providers"</span><span class="p">;</span>
<span class="ss">nativeBuildInputs</span> <span class="o">=</span> <span class="p">[</span>
pkgs<span class="o">.</span>nodejs
terraform
<span class="p">];</span>
<span class="ss">src</span> <span class="o">=</span> nix-filter <span class="p">{</span>
<span class="ss">root</span> <span class="o">=</span> <span class="l">./.</span><span class="p">;</span>
<span class="ss">include</span> <span class="o">=</span> <span class="p">[</span> <span class="l">./cdktf.json</span> <span class="l">./tsconfig.json</span> <span class="p">];</span>
<span class="p">};</span>
<span class="ss">buildPhase</span> <span class="o">=</span> <span class="s s-Multiline">''</span>
<span class="s s-Multiline"> export HOME=$(mktemp -d)</span>
<span class="s s-Multiline"> export CHECKPOINT_DISABLE=1</span>
<span class="s s-Multiline"> export DISABLE_VERSION_CHECK=1</span>
<span class="s s-Multiline"> export PATH=</span><span class="si">${</span>nodeEnv<span class="si">}</span><span class="s s-Multiline">/node_modules/.bin:$PATH</span>
<span class="s s-Multiline"> ln -nsf </span><span class="si">${</span>nodeEnv<span class="si">}</span><span class="s s-Multiline">/node_modules node_modules</span>
<span class="s s-Multiline"> # Build all providers we have in terraform</span>
<span class="s s-Multiline"> for provider in $(cd </span><span class="si">${</span>terraform<span class="si">}</span><span class="s s-Multiline">/libexec/terraform-providers; echo */*/*/*); do</span>
<span class="s s-Multiline"> version=</span><span class="se">''$</span><span class="s s-Multiline">{provider##*/}</span>
<span class="s s-Multiline"> provider=</span><span class="se">''$</span><span class="s s-Multiline">{provider%/*}</span>
<span class="s s-Multiline"> echo "Build $provider@$version"</span>
<span class="s s-Multiline"> cdktf provider add --force-local $provider@$version | cat</span>
<span class="s s-Multiline"> done</span>
<span class="s s-Multiline"> echo "Compile TS → JS"</span>
<span class="s s-Multiline"> tsc</span>
<span class="s s-Multiline"> ''</span><span class="p">;</span>
<span class="ss">installPhase</span> <span class="o">=</span> <span class="s s-Multiline">''</span>
<span class="s s-Multiline"> mv .gen $out</span>
<span class="s s-Multiline"> ln -nsf </span><span class="si">${</span>nodeEnv<span class="si">}</span><span class="s s-Multiline">/node_modules $out/node_modules</span>
<span class="s s-Multiline"> ''</span><span class="p">;</span>
<span class="p">};</span>
</pre></div>
<p>Finally, we can define the development environment:</p>
<div class="language-nix codehilite"><pre><span/>devShells<span class="o">.</span><span class="ss">default</span> <span class="o">=</span> pkgs<span class="o">.</span>mkShell <span class="p">{</span>
<span class="ss">name</span> <span class="o">=</span> <span class="s2">"cdktf-take1"</span><span class="p">;</span>
<span class="ss">buildInputs</span> <span class="o">=</span> <span class="p">[</span>
pkgs<span class="o">.</span>nodejs
pkgs<span class="o">.</span>yarn
terraform
<span class="p">];</span>
<span class="ss">shellHook</span> <span class="o">=</span> <span class="s s-Multiline">''</span>
<span class="s s-Multiline"> # No telemetry</span>
<span class="s s-Multiline"> export CHECKPOINT_DISABLE=1</span>
<span class="s s-Multiline"> # No autoinstall of plugins</span>
<span class="s s-Multiline"> export CDKTF_DISABLE_PLUGIN_CACHE_ENV=1</span>
<span class="s s-Multiline"> # Do not check version</span>
<span class="s s-Multiline"> export DISABLE_VERSION_CHECK=1</span>
<span class="s s-Multiline"> # Access to node modules</span>
<span class="s s-Multiline"> export PATH=$PWD/node_modules/.bin:$PATH</span>
<span class="s s-Multiline"> ln -nsf </span><span class="si">${</span>nodeEnv<span class="si">}</span><span class="s s-Multiline">/node_modules node_modules</span>
<span class="s s-Multiline"> ln -nsf </span><span class="si">${</span>cdktfProviders<span class="si">}</span><span class="s s-Multiline"> .gen</span>
<span class="s s-Multiline"> # Credentials</span>
<span class="s s-Multiline"> for p in \</span>
<span class="s s-Multiline"> njf.nznmba.pbz/Nqzvavfgengbe \</span>
<span class="s s-Multiline"> urgmare.pbz/ivaprag@oreang.pu \</span>
<span class="s s-Multiline"> ihyge.pbz/ihyge@ivaprag.oreang.pu; do</span>
<span class="s s-Multiline"> eval $(pass show $(echo $p | tr 'A-Za-z' 'N-ZA-Mn-za-m') | grep '^export')</span>
<span class="s s-Multiline"> done</span>
<span class="s s-Multiline"> eval $(pass show personal/cdktf/secrets | grep '^export')</span>
<span class="s s-Multiline"> export TF_VAR_hcloudToken="$HCLOUD_TOKEN"</span>
<span class="s s-Multiline"> export TF_VAR_vultrApiKey="$VULTR_API_KEY"</span>
<span class="s s-Multiline"> unset VULTR_API_KEY HCLOUD_TOKEN</span>
<span class="s s-Multiline"> ''</span><span class="p">;</span>
<span class="p">};</span>
</pre></div>
<p>The derivations listed in <code>buildInputs</code> are available in the provided shell.
The content of <code>shellHook</code> is sourced when starting the shell. It sets up some
symbolic links to make the JavaScript environment built at an earlier step
available, as well as the generated <em><abbr title="Cloud Development Kit for Terraform">CDKTF</abbr></em> providers. It also exports all the
credentials.<sup id="fnref-obfuscated"><a class="footnote-ref" href="#fn-obfuscated">10</a></sup></p>
<p>I am also using <a href="https://github.com/direnv/direnv/" title="direnv: unclutter your .profile">direnv</a> with an <a href="https://github.com/vincentbernat/cdktf-take1/blob/main/.envrc"><code>.envrc</code></a> to
automatically load the development environment. This also enables the
environment to be available from inside <em>Emacs</em>, notably when using <a href="https://github.com/emacs-lsp/lsp-mode/" title="Language Server Protocol Support for Emacs">lsp-mode</a>
to get TypeScript completions. Without <em>direnv</em>, <code>nix develop .</code> can activate
the environment.</p>
<p>I use the following commands to deploy the infrastructure:<sup id="fnref-cdktf-cli"><a class="footnote-ref" href="#fn-cdktf-cli">11</a></sup></p>
<div class="language-bash-session codehilite"><pre><span/><span class="gp">$ </span>cdktf<span class="w"> </span>synth
<span class="gp">$ </span><span class="nb">cd</span><span class="w"> </span>cdktf.out/stacks/cdktf-take1
<span class="gp">$ </span>terraform<span class="w"> </span>plan<span class="w"> </span>--out<span class="w"> </span>plan
<span class="gp">$ </span>terraform<span class="w"> </span>apply<span class="w"> </span>plan
<span class="gp">$ </span>terraform<span class="w"> </span>output<span class="w"> </span>-json<span class="w"> </span>><span class="w"> </span>~-automation/nixops-take1/cdktf.json
</pre></div>
<p>The last command generates a JSON file containing various data to complete the
deployment with <em>NixOps</em>.</p>
<h2 id="nixops">NixOps<a class="headerlink" href="#nixops" title="Permanent link"></a></h2>
<p>The <a href="https://github.com/vincentbernat/nixops-take1/blob/master/cdktf.json">JSON file</a> exported by <em>Terraform</em> contains the list of servers with various
attributes:</p>
<div class="language-json codehilite"><pre><span/><span class="p">{</span>
<span class="w"> </span><span class="nt">"hardware"</span><span class="p">:</span><span class="w"> </span><span class="s2">"hetzner"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"ipv4Address"</span><span class="p">:</span><span class="w"> </span><span class="s2">"5.161.44.145"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"ipv6Address"</span><span class="p">:</span><span class="w"> </span><span class="s2">"2a01:4ff:f0:b91::1"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"web05.luffy.cx"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"tags"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span>
<span class="w"> </span><span class="s2">"web"</span><span class="p">,</span>
<span class="w"> </span><span class="s2">"continent:NA"</span><span class="p">,</span>
<span class="w"> </span><span class="s2">"continent:SA"</span>
<span class="w"> </span><span class="p">]</span>
<span class="p">}</span>
</pre></div>
<p>In <a href="https://github.com/vincentbernat/nixops-take1/blob/master/network.nix"><code>network.nix</code></a>, this list is imported and
transformed into an attribute set describing the servers. A simplified version
looks like this:</p>
<div class="language-nix codehilite"><pre><span/><span class="k">let</span>
<span class="ss">lib</span> <span class="o">=</span> inputs<span class="o">.</span>nixpkgs<span class="o">.</span>lib<span class="p">;</span>
<span class="ss">shortName</span> <span class="o">=</span> name<span class="p">:</span> <span class="nb">builtins</span><span class="o">.</span>elemAt <span class="p">(</span>lib<span class="o">.</span>splitString <span class="s2">"."</span> name<span class="p">)</span> <span class="mi">0</span><span class="p">;</span>
<span class="ss">domainName</span> <span class="o">=</span> name<span class="p">:</span> lib<span class="o">.</span>concatStringsSep <span class="s2">"."</span> <span class="p">(</span><span class="nb">builtins</span><span class="o">.</span>tail <span class="p">(</span>lib<span class="o">.</span>splitString <span class="s2">"."</span> name<span class="p">));</span>
<span class="ss">server</span> <span class="o">=</span> hardware<span class="p">:</span> name<span class="p">:</span> imports<span class="p">:</span> <span class="p">{</span>
<span class="ss">networking</span> <span class="o">=</span> <span class="p">{</span>
<span class="ss">hostName</span> <span class="o">=</span> shortName name<span class="p">;</span>
<span class="ss">domain</span> <span class="o">=</span> domainName name<span class="p">;</span>
<span class="p">};</span>
deployment<span class="o">.</span><span class="ss">targetHost</span> <span class="o">=</span> name<span class="p">;</span>
<span class="ss">imports</span> <span class="o">=</span> <span class="p">[</span> <span class="p">(</span><span class="l">./hardware/.</span> <span class="o">+</span> <span class="s2">"/</span><span class="si">${</span>hardware<span class="si">}</span><span class="s2">.nix"</span><span class="p">)</span> <span class="p">]</span> <span class="o">++</span> imports<span class="p">;</span>
<span class="p">};</span>
<span class="ss">cdktf-servers-json</span> <span class="o">=</span> <span class="p">(</span>lib<span class="o">.</span>importJSON <span class="l">./cdktf.json</span><span class="p">)</span><span class="o">.</span>servers<span class="o">.</span>value<span class="p">;</span>
<span class="ss">cdktf-servers</span> <span class="o">=</span> <span class="nb">map</span>
<span class="p">(</span>s<span class="p">:</span>
<span class="k">let</span>
<span class="ss">tags-maybe-import</span> <span class="o">=</span> <span class="nb">map</span> <span class="p">(</span>t<span class="p">:</span> <span class="l">./.</span> <span class="o">+</span> <span class="s2">"/</span><span class="si">${</span>t<span class="si">}</span><span class="s2">.nix"</span><span class="p">)</span> s<span class="o">.</span>tags<span class="p">;</span>
<span class="ss">tags-import</span> <span class="o">=</span> <span class="nb">builtins</span><span class="o">.</span>filter <span class="p">(</span>t<span class="p">:</span> <span class="nb">builtins</span><span class="o">.</span>pathExists t<span class="p">)</span> tags-maybe-import<span class="p">;</span>
<span class="k">in</span>
<span class="p">{</span>
<span class="ss">name</span> <span class="o">=</span> shortName s<span class="o">.</span>name<span class="p">;</span>
<span class="ss">value</span> <span class="o">=</span> server s<span class="o">.</span>hardware s<span class="o">.</span>name tags-import<span class="p">;</span>
<span class="p">})</span>
cdktf-servers-json<span class="p">;</span>
<span class="k">in</span>
<span class="p">{</span>
<span class="o">//</span> <span class="p">[</span><span class="err">…</span><span class="p">]</span>
<span class="p">}</span> <span class="o">//</span> <span class="nb">builtins</span><span class="o">.</span>listToAttrs cdktf-servers
</pre></div>
<p>For <code>web05</code>, this expands to:</p>
<div class="language-nix codehilite"><pre><span/><span class="ss">web05</span> <span class="o">=</span> <span class="p">{</span>
<span class="ss">networking</span> <span class="o">=</span> <span class="p">{</span>
<span class="ss">hostName</span> <span class="o">=</span> <span class="s2">"web05"</span><span class="p">;</span>
<span class="ss">domainName</span> <span class="o">=</span> <span class="s2">"luffy.cx"</span><span class="p">;</span>
<span class="p">};</span>
deployment<span class="o">.</span><span class="ss">targetHost</span> <span class="o">=</span> <span class="s2">"web05.luffy.cx"</span><span class="p">;</span>
<span class="ss">imports</span> <span class="o">=</span> <span class="p">[</span> <span class="l">./hardware/hetzner.nix</span> <span class="l">./web.nix</span> <span class="p">];</span>
<span class="p">};</span>
</pre></div>
<p>As for <em><abbr title="Cloud Development Kit for Terraform">CDKTF</abbr></em>, at the root of the <a href="https://github.com/vincentbernat/nixops-take1">repository I use for NixOps</a>,
there is a <a href="https://github.com/vincentbernat/nixops-take1/blob/master/flake.nix"><code>flake.nix</code></a> file to set up a shell with
<em>NixOps</em> configured. Because <em>NixOps</em> do not support rollouts, I usually use the
following commands to deploy on a single server:<sup id="fnref-disable"><a class="footnote-ref" href="#fn-disable">12</a></sup></p>
<div class="language-bash-session codehilite"><pre><span/><span class="gp">$ </span>nix<span class="w"> </span>flake<span class="w"> </span>update
<span class="gp">$ </span>nixops<span class="w"> </span>deploy<span class="w"> </span>--include<span class="o">=</span>web04
<span class="gp">$ </span>./tests<span class="w"> </span>web04.luffy.cx
</pre></div>
<p>If the tests are OK, I deploy the remaining nodes gradually with the following
command:</p>
<div class="language-bash-session codehilite"><pre><span/><span class="gp">$ </span><span class="o">(</span><span class="nb">set</span><span class="w"> </span>-e<span class="p">;</span><span class="w"> </span><span class="k">for</span><span class="w"> </span>h<span class="w"> </span><span class="k">in</span><span class="w"> </span>web<span class="o">{</span><span class="m">03</span>..06<span class="o">}</span><span class="p">;</span><span class="w"> </span><span class="k">do</span><span class="w"> </span>nixops<span class="w"> </span>deploy<span class="w"> </span>--include<span class="o">=</span><span class="nv">$h</span><span class="p">;</span><span class="w"> </span><span class="k">done</span><span class="o">)</span>
</pre></div>
<p><code>nixops deploy</code> rolls out all servers in parallel and therefore could cause a
short outage where all <em>Nginx</em> are down at the same time.</p>
<hr/>
<p>This post has been a work-in-progress for the past three years, with the content
being updated and refined as I experimented with different solutions. There is
still much to explore<sup id="fnref-todo"><a class="footnote-ref" href="#fn-todo">13</a></sup> but I feel there is enough content to publish now.
🎄</p>
<div class="footnote">
<hr/>
<ol>
<li id="fn-hetzner">
<p>It was an AMD Athlon 64 X2 5600+ with 2 GB of RAM and 2×400 GB disks
with software RAID. I was paying something around 59 € per month for it.
While it was a good deal in 2008, by 2018 it was no longer cost-effective.
It was running on Debian Wheezy with <a href="http://linux-vserver.org/Welcome_to_Linux-VServer.org" title="Linux-VServer">Linux-VServer</a> for isolation, both
of which were outdated in 2018. <a class="footnote-backref" href="#fnref-hetzner" title="Jump back to footnote 1 in the text">↩︎</a></p>
</li>
<li id="fn-python">
<p>I also did not use Python because <em>Poetry</em> support in <em>Nix</em> was a bit
<a href="https://github.com/nix-community/poetry2nix/issues/750" title="Issue #750: infinite recursion">broken</a> around the time I started hacking around <em><abbr title="Cloud Development Kit for Terraform">CDKTF</abbr></em>. <a class="footnote-backref" href="#fnref-python" title="Jump back to footnote 2 in the text">↩︎</a></p>
</li>
<li id="fn-transform">
<p><em>Pulumi</em> can <a href="https://www.pulumi.com/docs/intro/concepts/inputs-outputs/#apply">apply arbitrary functions</a> with the <code>apply()</code>
method on an output. It makes it easy to transform data that are not known
during the planning stage. <em>Terraform</em> has <a href="https://developer.hashicorp.com/terraform/language/functions">functions</a> to
serve a similar purpose, but they are more limited. <a class="footnote-backref" href="#fnref-transform" title="Jump back to footnote 3 in the text">↩︎</a></p>
</li>
<li id="fn-maint">
<p>The two mentioned pull requests are not merged yet. The second one is
superseded by <a href="https://github.com/pulumi/pulumi-tf-provider-boilerplate/pull/61" title="PR #61: Explicitly specify bash as shell in Makefile">PR #61</a>, submitted two months later, which
enforces the use of <code>/bin/bash</code>. I also submitted <a href="https://github.com/pulumi/pulumi-tf-provider-boilerplate/pull/56" title="PR #56: Use go:embed instead of go generate for schema">PR #56</a>,
which was merged 4 months later and quickly <a href="https://github.com/pulumi/pulumi-tf-provider-boilerplate/pull/66" title="PR #66: Revert “Use go:embed instead of go generate for schema”">reverted</a> without
an explanation. <a class="footnote-backref" href="#fnref-maint" title="Jump back to footnote 4 in the text">↩︎</a></p>
</li>
<li id="fn-derivation">
<p>You may consider packages and derivations to be synonyms in the
<em>Nix</em> ecosystem. <a class="footnote-backref" href="#fnref-derivation" title="Jump back to footnote 5 in the text">↩︎</a></p>
</li>
<li id="fn-openssl">
<p><em>OpenSSL 3</em> has outstanding <a href="https://github.com/openssl/openssl/issues/17627#issuecomment-1060123659">performance regressions</a>. <a class="footnote-backref" href="#fnref-openssl" title="Jump back to footnote 6 in the text">↩︎</a></p>
</li>
<li id="fn-security">
<p><em>NixOS</em> can be a bit slow to integrate patches since they need to
rebuild parts of the binary cache before releasing the fixes. In this
specific case, they were fast: the vulnerability and patches were released
on August 13<sup>th</sup> 2019 and available in <em>NixOS</em> on August 15th. As a
comparison, Debian only released the fixed version on August 22<sup>nd</sup>, which is unusually late. <a class="footnote-backref" href="#fnref-security" title="Jump back to footnote 7 in the text">↩︎</a></p>
</li>
<li id="fn-experimental">
<p>Because <em>flakes</em> are experimental, many documentations do not
use them and it is an additional aspect to learn. <a class="footnote-backref" href="#fnref-experimental" title="Jump back to footnote 8 in the text">↩︎</a></p>
</li>
<li id="fn-checkout">
<p>It is possible to replace <code>.</code> with <code>github:vincentbernat/snimpy</code>,
like in the other commands, but having <em>Snimpy</em> dependencies without
<em>Snimpy</em> source code is less interesting. <a class="footnote-backref" href="#fnref-checkout" title="Jump back to footnote 9 in the text">↩︎</a></p>
</li>
<li id="fn-obfuscated">
<p>I am using <a href="https://www.passwordstore.org/" title="The standard Unix password manager">pass</a> as a password manager. The password names are
only obfuscated to avoid spam. <a class="footnote-backref" href="#fnref-obfuscated" title="Jump back to footnote 10 in the text">↩︎</a></p>
</li>
<li id="fn-cdktf-cli">
<p>The <code>cdktf</code> command can wrap the <code>terraform</code> commands, but I
prefer to use them directly as they are more flexible. <a class="footnote-backref" href="#fnref-cdktf-cli" title="Jump back to footnote 11 in the text">↩︎</a></p>
</li>
<li id="fn-disable">
<p>If the change is risky, I <a href="https://github.com/vincentbernat/cdktf-take1/commit/606e6fb657179408b163664dd74ff5ab38b28246">disable the server</a> with <abbr title="Cloud Development Kit for Terraform">CDKTF</abbr>. This
removes it from the web service DNS records. <a class="footnote-backref" href="#fnref-disable" title="Jump back to footnote 12 in the text">↩︎</a></p>
</li>
<li id="fn-todo">
<p>I would like to replace <em>NixOps</em> with an alternative handling
progressive rollouts and checks. I am also considering switching to
<a href="https://www.nomadproject.io/" title="Nomad: Orchestration Made Easy">Nomad</a> or <em>Kubernetes</em> to deploy workloads. <a class="footnote-backref" href="#fnref-todo" title="Jump back to footnote 13 in the text">↩︎</a></p>
</li>
</ol>
</div>