Vincent Bernathttp://www.luffy.cx/fr/blog/atom.xml/2024-01-04T22:18:24ZAuthentification SSH non interactive par mot de passeVincent Bernat2023-11-05T17:25:26Zhttp://www.luffy.cx/fr/blog/2023-sshpass-sans-sshpass.html
<p>SSH offre plusieurs formes d’authentification, telles que par <strong>mot de passe</strong>
et par <strong>clé publique</strong>. Cette dernière est considérée comme plus sécurisée.
Cependant, l’authentification par mot de passe reste répandue, notamment avec
les équipements réseau<sup id="fnref-pourquoi"><a class="footnote-ref" href="#fn-pourquoi">1</a></sup>.</p>
<p>Une solution classique pour éviter de taper un mot de passe à chaque connexion
est <a href="https://sourceforge.net/projects/sshpass/" title="sshpass: Non-interactive ssh password auth">sshpass</a>, ou sa variante plus correcte <a href="https://github.com/clarkwang/passh">passh</a>. Voici une fonction pour
<em>Zsh</em>, récupérant le mot de passe depuis <a href="https://www.passwordstore.org/" title="The standard UNIX password manager">pass</a>, un gestionnaire de mots de
passe<sup id="fnref-security"><a class="footnote-ref" href="#fn-security">2</a></sup> :</p>
<div class="language-bash codehilite"><pre><span/>pssh<span class="o">()</span><span class="w"> </span><span class="o">{</span>
<span class="w"> </span>passh<span class="w"> </span>-p<span class="w"> </span><<span class="o">(</span>pass<span class="w"> </span>show<span class="w"> </span>network/ssh/password<span class="w"> </span><span class="p">|</span><span class="w"> </span>head<span class="w"> </span>-1<span class="o">)</span><span class="w"> </span>ssh<span class="w"> </span><span class="s2">"</span><span class="nv">$@</span><span class="s2">"</span>
<span class="o">}</span>
compdef<span class="w"> </span><span class="nv">pssh</span><span class="o">=</span>ssh
</pre></div>
<p>Cette approche est un peu fragile car elle nécessite d’examiner la sortie de la
commande <code>ssh</code> pour y chercher une invite de mot de passe. De plus, si aucun mot
de passe n’est requis, le gestionnaire de mots de passe est tout de même
invoqué. Depuis <a href="https://www.openssh.com/txt/release-8.4" title="OpenSSH 8.4 release notes">OpenSSH 8.4</a>, nous pouvons à la place utiliser <code>SSH_ASKPASS</code>
et <code>SSH_ASKPASS_REQUIRE</code> :</p>
<div class="language-bash codehilite"><pre><span/>ssh<span class="o">()</span><span class="w"> </span><span class="o">{</span>
<span class="w"> </span><span class="nb">set</span><span class="w"> </span>-o<span class="w"> </span>localoptions<span class="w"> </span>-o<span class="w"> </span>localtraps
<span class="w"> </span><span class="nb">local</span><span class="w"> </span><span class="nv">passname</span><span class="o">=</span>network/ssh/password
<span class="w"> </span><span class="nb">local</span><span class="w"> </span><span class="nv">helper</span><span class="o">=</span><span class="k">$(</span>mktemp<span class="k">)</span>
<span class="w"> </span><span class="nb">trap</span><span class="w"> </span><span class="s2">"command rm -f </span><span class="nv">$helper</span><span class="s2">"</span><span class="w"> </span>EXIT<span class="w"> </span>INT
<span class="w"> </span>><span class="w"> </span><span class="nv">$helper</span><span class="w"> </span><span class="s"><<EOF</span>
<span class="s">#!$SHELL</span>
<span class="s">pass show $passname | head -1</span>
<span class="s">EOF</span>
<span class="w"> </span>chmod<span class="w"> </span>u+x<span class="w"> </span><span class="nv">$helper</span>
<span class="w"> </span><span class="nv">SSH_ASKPASS</span><span class="o">=</span><span class="nv">$helper</span><span class="w"> </span><span class="nv">SSH_ASKPASS_REQUIRE</span><span class="o">=</span>force<span class="w"> </span><span class="nb">command</span><span class="w"> </span>ssh<span class="w"> </span><span class="s2">"</span><span class="nv">$@</span><span class="s2">"</span>
<span class="o">}</span>
</pre></div>
<p>Si le mot de passe est incorrect, nous pouvons afficher une invite de mot de
passe à la seconde tentative :</p>
<div class="language-bash codehilite"><pre><span/>ssh<span class="o">()</span><span class="w"> </span><span class="o">{</span>
<span class="w"> </span><span class="nb">set</span><span class="w"> </span>-o<span class="w"> </span>localoptions<span class="w"> </span>-o<span class="w"> </span>localtraps
<span class="w"> </span><span class="nb">local</span><span class="w"> </span><span class="nv">passname</span><span class="o">=</span>network/ssh/password
<span class="w"> </span><span class="nb">local</span><span class="w"> </span><span class="nv">helper</span><span class="o">=</span><span class="k">$(</span>mktemp<span class="k">)</span>
<span class="w"> </span><span class="nb">trap</span><span class="w"> </span><span class="s2">"command rm -f </span><span class="nv">$helper</span><span class="s2">"</span><span class="w"> </span>EXIT<span class="w"> </span>INT
<span class="w"> </span>><span class="w"> </span><span class="nv">$helper</span><span class="w"> </span><span class="s"><<EOF</span>
<span class="s">#!$SHELL</span>
<span class="s">if [ -k $helper ]; then</span>
<span class="s"> {</span>
<span class="s"> oldtty=\$(stty -g)</span>
<span class="s"> trap 'stty \$oldtty < /dev/tty 2> /dev/null' EXIT INT TERM HUP</span>
<span class="s"> stty -echo</span>
<span class="s"> print "\rpassword: "</span>
<span class="s"> read password</span>
<span class="s"> printf "\n"</span>
<span class="s"> } > /dev/tty < /dev/tty</span>
<span class="s"> printf "%s" "\$password"</span>
<span class="s">else</span>
<span class="s"> pass show $passname | head -1</span>
<span class="s"> chmod +t $helper</span>
<span class="s">fi</span>
<span class="s">EOF</span>
<span class="w"> </span>chmod<span class="w"> </span>u+x<span class="w"> </span><span class="nv">$helper</span>
<span class="w"> </span><span class="nv">SSH_ASKPASS</span><span class="o">=</span><span class="nv">$helper</span><span class="w"> </span><span class="nv">SSH_ASKPASS_REQUIRE</span><span class="o">=</span>force<span class="w"> </span><span class="nb">command</span><span class="w"> </span>ssh<span class="w"> </span><span class="s2">"</span><span class="nv">$@</span><span class="s2">"</span>
<span class="o">}</span>
</pre></div>
<p>Une amélioration possible est d’utiliser une entrée de mot de passe différente
en fonction de l’hôte distant<sup id="fnref-zsh"><a class="footnote-ref" href="#fn-zsh">3</a></sup> :</p>
<div class="language-bash codehilite"><pre><span/>ssh<span class="o">()</span><span class="w"> </span><span class="o">{</span>
<span class="w"> </span><span class="c1"># Informations de connexion</span>
<span class="w"> </span><span class="nb">local</span><span class="w"> </span>-A<span class="w"> </span>details
<span class="w"> </span><span class="nv">details</span><span class="o">=(</span><span class="si">${</span><span class="p">=</span><span class="si">${</span><span class="p">(M)</span><span class="si">${</span><span class="k">:-</span><span class="s2">"</span><span class="si">${</span><span class="p">(@f)</span><span class="k">$(</span><span class="nb">command</span><span class="w"> </span>ssh<span class="w"> </span>-G<span class="w"> </span><span class="s2">"</span><span class="nv">$@</span><span class="s2">"</span><span class="w"> </span><span class="m">2</span>>/dev/null<span class="k">)</span><span class="si">}</span><span class="s2">"</span><span class="si">}</span><span class="p">:#(host|hostname|user) *</span><span class="si">}}</span><span class="o">)</span>
<span class="w"> </span><span class="nb">local</span><span class="w"> </span><span class="nv">remote</span><span class="o">=</span><span class="si">${</span><span class="nv">details</span><span class="p">[host]</span><span class="k">:-</span><span class="nv">details</span><span class="p">[hostname]</span><span class="si">}</span>
<span class="w"> </span><span class="nb">local</span><span class="w"> </span><span class="nv">login</span><span class="o">=</span><span class="si">${</span><span class="nv">details</span><span class="p">[user]</span><span class="si">}</span>@<span class="si">${</span><span class="nv">remote</span><span class="si">}</span>
<span class="w"> </span><span class="c1"># Nom du mot de passe</span>
<span class="w"> </span><span class="nb">local</span><span class="w"> </span>passname
<span class="w"> </span><span class="k">case</span><span class="w"> </span><span class="s2">"</span><span class="nv">$login</span><span class="s2">"</span><span class="w"> </span><span class="k">in</span>
<span class="w"> </span>admin@*.example.net<span class="o">)</span><span class="w"> </span><span class="nv">passname</span><span class="o">=</span>company1/ssh/admin<span class="w"> </span><span class="p">;;</span>
<span class="w"> </span>bernat@*.example.net<span class="o">)</span><span class="w"> </span><span class="nv">passname</span><span class="o">=</span>company1/ssh/bernat<span class="w"> </span><span class="p">;;</span>
<span class="w"> </span>backup@*.example.net<span class="o">)</span><span class="w"> </span><span class="nv">passname</span><span class="o">=</span>company1/ssh/backup<span class="w"> </span><span class="p">;;</span>
<span class="w"> </span><span class="k">esac</span>
<span class="w"> </span><span class="c1"># Sans nom de mot de passe, on se rabat sur ssh</span>
<span class="w"> </span><span class="o">[[</span><span class="w"> </span>-z<span class="w"> </span><span class="nv">$passname</span><span class="w"> </span><span class="o">]]</span><span class="w"> </span><span class="o">&&</span><span class="w"> </span><span class="o">{</span>
<span class="w"> </span><span class="c1"># Just regular ssh...</span>
<span class="w"> </span><span class="nb">command</span><span class="w"> </span>ssh<span class="w"> </span><span class="s2">"</span><span class="nv">$@</span><span class="s2">"</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nv">$?</span>
<span class="w"> </span><span class="o">}</span>
<span class="w"> </span><span class="c1"># Invocation de SSH avec SSH_ASKPASS</span>
<span class="w"> </span><span class="c1"># […]</span>
<span class="o">}</span>
</pre></div>
<p>Il est également possible d’adapter <code>scp</code> pour qu’il utilise notre fonction
<code>ssh</code> :</p>
<div class="language-bash codehilite"><pre><span/>scp<span class="o">()</span><span class="w"> </span><span class="o">{</span>
<span class="w"> </span><span class="nb">set</span><span class="w"> </span>-o<span class="w"> </span>localoptions<span class="w"> </span>-o<span class="w"> </span>localtraps
<span class="w"> </span><span class="nb">local</span><span class="w"> </span><span class="nv">helper</span><span class="o">=</span><span class="k">$(</span>mktemp<span class="k">)</span>
<span class="w"> </span><span class="nb">trap</span><span class="w"> </span><span class="s2">"command rm -f </span><span class="nv">$helper</span><span class="s2">"</span><span class="w"> </span>EXIT<span class="w"> </span>INT
<span class="w"> </span>><span class="w"> </span><span class="nv">$helper</span><span class="w"> </span><span class="s"><<EOF </span>
<span class="s">#!$SHELL</span>
<span class="s">source ${(%):-%x}</span>
<span class="s">ssh "\$@"</span>
<span class="s">EOF</span>
<span class="w"> </span><span class="nb">command</span><span class="w"> </span>scp<span class="w"> </span>-S<span class="w"> </span><span class="nv">$helper</span><span class="w"> </span><span class="s2">"</span><span class="nv">$@</span><span class="s2">"</span>
<span class="o">}</span>
</pre></div>
<p>Le code complet est disponible dans mon <a href="https://github.com/vincentbernat/zshrc/blob/master/rc/ssh.zsh">zshrc</a>. Il est également possible de
placer le contenu de la fonction <code>ssh()</code> dans un script séparé et de remplacer
<code>command ssh</code> par <code>/usr/bin/ssh</code> pour éviter un appel récursif. Dans ce cas, la
fonction <code>scp()</code> n’est plus nécessaire.</p>
<div class="admonition">
<p class="admonition-title">Mise à jour (12.2023)</p>
<p>Cet article a été longuement débattu sur <a href="https://news.ycombinator.com/item?id=38762214">Hacker News</a>.</p>
</div>
<div class="footnote">
<hr/>
<ol>
<li id="fn-pourquoi">
<p>Tout d’abord, certains constructeurs rendent <a href="/fr/blog/2020-synchro-clefs-ssh-iosxr-ansible" title="Synchroniser les clefs SSH sur Cisco IOS-XR avec un module Ansible sur mesure">difficile
l’association d’une clé SSH à un utilisateur</a>. Ensuite, de nombreux
constructeurs ne prennent pas en charge l’authentification basée sur un
certificat, ce qui complique la mise en place à grande échelle. Enfin, les
interactions entre l’authentification par clé publique et des méthodes
d’autorisation plus fines telles que TACACS+ et Radius restent un domaine
inexploré. <a class="footnote-backref" href="#fnref-pourquoi" title="Jump back to footnote 1 in the text">↩︎</a></p>
</li>
<li id="fn-security">
<p>Le mot de passe en clair n’apparaît jamais sur la ligne de
commande, dans l’environnement ou sur le disque, rendant difficile sa
capture par un tiers non privilégié. Sur Linux, <em>Zsh</em> fournit le mot de
passe via un descripteur de fichier. <a class="footnote-backref" href="#fnref-security" title="Jump back to footnote 2 in the text">↩︎</a></p>
</li>
<li id="fn-zsh">
<p>Pour comprendre la magie noire derrière la quatrième ligne, vous pouvez
vous aider de <code>print -l</code> et de la page de manuel <a href="https://manpages.debian.org/bookworm/zsh-common/zshexpn.1.en.html" title="zshexpn(1) manual page">zshexpn(1)</a>. <code>details</code>
est un <a href="https://scriptingosx.com/2019/11/associative-arrays-in-zsh/" title="Associative arrays in zsh">tableau associatif</a> défini à partir d’un tableau
alternant clefs et valeurs. <a class="footnote-backref" href="#fnref-zsh" title="Jump back to footnote 3 in the text">↩︎</a></p>
</li>
</ol>
</div>
Détection et suppression des DDoS avec Akvorado et FlowspecVincent Bernat2023-03-06T07:34:03Zhttp://www.luffy.cx/fr/blog/2023-akvorado-ddos-flowspec.html
<p><a href="/fr/blog/2022-akvorado-collecteur-flux" title="Akvorado : collecteur et visualisateur de flux réseau">Akvorado</a> collecte des flux <a href="https://www.rfc-editor.org/rfc/rfc3176" title="RFC 3176: Specification of the IP Flow Information Export (IPFIX) Protocol for the Exchange of Flow Information">sFlow</a> et <a href="https://www.rfc-editor.org/rfc/rfc7011" title="RFC 7011: Specification of the IP Flow Information Export (IPFIX) Protocol for the Exchange of Flow Information">IPFIX</a>, les stocke dans une base
de données <a href="https://clickhouse.com/" title="ClickHouse: OLAP DBMS">ClickHouse</a> et les présente dans une console web. Bien qu’il n’y
ait pas de détection de <abbr title="Distributed Denial of Service">DDoS</abbr> intégrée, il est possible d’en créer une à
partir de requêtes pour <em>ClickHouse</em>.</p>
<h1 id="detection-des-attaques-ddos">Détection des attaques <abbr title="Distributed Denial of Service">DDoS</abbr><a class="headerlink" href="#detection-des-attaques-ddos" title="Permanent link"></a></h1>
<p>Supposons que nous voulions détecter des attaques <abbr title="Distributed Denial of Service">DDoS</abbr> ciblant nos clients. À
titre d’exemple, nous considérons une attaque <abbr title="Distributed Denial of Service">DDoS</abbr> comme une collection de flux
sur <strong>une minute</strong> ciblant une <strong>adresse IP client unique</strong>, à partir d’un
<strong>port source unique</strong> et correspondant à l’une de ces conditions :</p>
<ul>
<li>une bande passante moyenne de 1 Gbit/s,</li>
<li>une bande passante moyenne de 200 Mbit/s lorsque le protocole est UDP,</li>
<li>plus de 20 adresses IP sources et une bande passante moyenne de 100 Mbit/s, ou</li>
<li>plus de 10 pays sources et une bande passante moyenne de 100 Mbit/s.</li>
</ul>
<p>Voici la requête SQL pour détecter de telles attaques au cours des 5 dernières
minutes :</p>
<div class="language-sql codehilite"><pre><span/><span class="k">SELECT</span><span class="w"> </span><span class="o">*</span>
<span class="k">FROM</span><span class="w"> </span><span class="p">(</span>
<span class="w"> </span><span class="k">SELECT</span>
<span class="w"> </span><span class="n">toStartOfMinute</span><span class="p">(</span><span class="n">TimeReceived</span><span class="p">)</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">TimeReceived</span><span class="p">,</span>
<span class="w"> </span><span class="n">DstAddr</span><span class="p">,</span>
<span class="w"> </span><span class="n">SrcPort</span><span class="p">,</span>
<span class="w"> </span><span class="n">dictGetOrDefault</span><span class="p">(</span><span class="s1">'protocols'</span><span class="p">,</span><span class="w"> </span><span class="s1">'name'</span><span class="p">,</span><span class="w"> </span><span class="n">Proto</span><span class="p">,</span><span class="w"> </span><span class="s1">'???'</span><span class="p">)</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">Proto</span><span class="p">,</span>
<span class="w"> </span><span class="k">SUM</span><span class="p">(((((</span><span class="n">Bytes</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="n">SamplingRate</span><span class="p">)</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="mi">8</span><span class="p">)</span><span class="w"> </span><span class="o">/</span><span class="w"> </span><span class="mi">1000</span><span class="p">)</span><span class="w"> </span><span class="o">/</span><span class="w"> </span><span class="mi">1000</span><span class="p">)</span><span class="w"> </span><span class="o">/</span><span class="w"> </span><span class="mi">1000</span><span class="p">)</span><span class="w"> </span><span class="o">/</span><span class="w"> </span><span class="mi">60</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">Gbps</span><span class="p">,</span>
<span class="w"> </span><span class="n">uniq</span><span class="p">(</span><span class="n">SrcAddr</span><span class="p">)</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">sources</span><span class="p">,</span>
<span class="w"> </span><span class="n">uniq</span><span class="p">(</span><span class="n">SrcCountry</span><span class="p">)</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">countries</span>
<span class="w"> </span><span class="k">FROM</span><span class="w"> </span><span class="n">flows</span>
<span class="w"> </span><span class="k">WHERE</span><span class="w"> </span><span class="n">TimeReceived</span><span class="w"> </span><span class="o">></span><span class="w"> </span><span class="n">now</span><span class="p">()</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="nb">INTERVAL</span><span class="w"> </span><span class="mi">5</span><span class="w"> </span><span class="k">MINUTE</span>
<span class="w"> </span><span class="k">AND</span><span class="w"> </span><span class="n">DstNetRole</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'customers'</span>
<span class="w"> </span><span class="k">GROUP</span><span class="w"> </span><span class="k">BY</span>
<span class="w"> </span><span class="n">TimeReceived</span><span class="p">,</span>
<span class="w"> </span><span class="n">DstAddr</span><span class="p">,</span>
<span class="w"> </span><span class="n">SrcPort</span><span class="p">,</span>
<span class="w"> </span><span class="n">Proto</span>
<span class="p">)</span>
<span class="k">WHERE</span><span class="w"> </span><span class="p">(</span><span class="n">Gbps</span><span class="w"> </span><span class="o">></span><span class="w"> </span><span class="mi">1</span><span class="p">)</span>
<span class="w"> </span><span class="k">OR</span><span class="w"> </span><span class="p">((</span><span class="n">Proto</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'UDP'</span><span class="p">)</span><span class="w"> </span><span class="k">AND</span><span class="w"> </span><span class="p">(</span><span class="n">Gbps</span><span class="w"> </span><span class="o">></span><span class="w"> </span><span class="mi">0</span><span class="p">.</span><span class="mi">2</span><span class="p">))</span><span class="w"> </span>
<span class="w"> </span><span class="k">OR</span><span class="w"> </span><span class="p">((</span><span class="n">sources</span><span class="w"> </span><span class="o">></span><span class="w"> </span><span class="mi">20</span><span class="p">)</span><span class="w"> </span><span class="k">AND</span><span class="w"> </span><span class="p">(</span><span class="n">Gbps</span><span class="w"> </span><span class="o">></span><span class="w"> </span><span class="mi">0</span><span class="p">.</span><span class="mi">1</span><span class="p">))</span><span class="w"> </span>
<span class="w"> </span><span class="k">OR</span><span class="w"> </span><span class="p">((</span><span class="n">countries</span><span class="w"> </span><span class="o">></span><span class="w"> </span><span class="mi">10</span><span class="p">)</span><span class="w"> </span><span class="k">AND</span><span class="w"> </span><span class="p">(</span><span class="n">Gbps</span><span class="w"> </span><span class="o">></span><span class="w"> </span><span class="mi">0</span><span class="p">.</span><span class="mi">1</span><span class="p">))</span>
<span class="k">ORDER</span><span class="w"> </span><span class="k">BY</span>
<span class="w"> </span><span class="n">TimeReceived</span><span class="w"> </span><span class="k">DESC</span><span class="p">,</span>
<span class="w"> </span><span class="n">Gbps</span><span class="w"> </span><span class="k">DESC</span>
</pre></div>
<p>Voici un exemple de sortie<sup id="fnref-format"><a class="footnote-ref" href="#fn-format">1</a></sup> où deux de nos utilisateurs sont attaqués.
L’un d’eux subit apparemment une <a href="https://www.cloudflare.com/learning/ddos/ntp-amplification-ddos-attack/" title="NTP amplification DDoS attack">attaque d’amplification NTP</a>, l’autre une <a href="https://www.cloudflare.com/learning/ddos/dns-amplification-ddos-attack/" title="DNS amplification attack">attaque d’amplification DNS</a> :</p>
<div class="lf-table"><table>
<thead>
<tr>
<th align="right">TimeReceived</th>
<th align="left">DstAddr</th>
<th align="right">SrcPort</th>
<th align="left">Proto</th>
<th align="right">Gbps</th>
<th align="right">sources</th>
<th align="right">countries</th>
</tr>
</thead>
<tbody>
<tr>
<td align="right">2023-02-26 17:44:00</td>
<td align="left"><code>::ffff:203.0.113.206</code></td>
<td align="right">123</td>
<td align="left">UDP</td>
<td align="right">0.102</td>
<td align="right">109</td>
<td align="right">13</td>
</tr>
<tr>
<td align="right">2023-02-26 17:43:00</td>
<td align="left"><code>::ffff:203.0.113.206</code></td>
<td align="right">123</td>
<td align="left">UDP</td>
<td align="right">0.130</td>
<td align="right">133</td>
<td align="right">17</td>
</tr>
<tr>
<td align="right">2023-02-26 17:43:00</td>
<td align="left"><code>::ffff:203.0.113.68</code></td>
<td align="right">53</td>
<td align="left">UDP</td>
<td align="right">0.129</td>
<td align="right">364</td>
<td align="right">63</td>
</tr>
<tr>
<td align="right">2023-02-26 17:43:00</td>
<td align="left"><code>::ffff:203.0.113.206</code></td>
<td align="right">123</td>
<td align="left">UDP</td>
<td align="right">0.113</td>
<td align="right">129</td>
<td align="right">21</td>
</tr>
<tr>
<td align="right">2023-02-26 17:42:00</td>
<td align="left"><code>::ffff:203.0.113.206</code></td>
<td align="right">123</td>
<td align="left">UDP</td>
<td align="right">0.139</td>
<td align="right">50</td>
<td align="right">14</td>
</tr>
<tr>
<td align="right">2023-02-26 17:42:00</td>
<td align="left"><code>::ffff:203.0.113.206</code></td>
<td align="right">123</td>
<td align="left">UDP</td>
<td align="right">0.105</td>
<td align="right">42</td>
<td align="right">14</td>
</tr>
<tr>
<td align="right">2023-02-26 17:40:00</td>
<td align="left"><code>::ffff:203.0.113.68</code></td>
<td align="right">53</td>
<td align="left">UDP</td>
<td align="right">0.121</td>
<td align="right">340</td>
<td align="right">65</td>
</tr>
</tbody>
</table>
</div><h1 id="suppression-des-attaques-ddos">Suppression des attaques <abbr title="Distributed Denial of Service">DDoS</abbr><a class="headerlink" href="#suppression-des-attaques-ddos" title="Permanent link"></a></h1>
<p>Une fois détectée, il existe au moins deux façons d’arrêter l’attaque au niveau
du réseau :</p>
<ul>
<li>supprimer tout le trafic vers l’utilisateur ciblé (<em><abbr title="Remote Triggered Blackhole">RTBH</abbr></em>)</li>
<li>supprimer sélectivement les paquets correspondant aux motifs de l’attaque
(<em>Flowspec</em>)</li>
</ul>
<h2 id="suppression-du-trafic-avec-rtbh">Suppression du trafic avec <abbr title="Remote Triggered Blackhole">RTBH</abbr><a class="headerlink" href="#suppression-du-trafic-avec-rtbh" title="Permanent link"></a></h2>
<p>La méthode la plus simple consiste à sacrifier l’utilisateur attaqué. Bien que
cela aide l’attaquant, cela protège avant tout votre réseau. C’est une méthode
prise en charge par tous les routeurs. Vous pouvez également déléguer cette
protection à la plupart des fournisseurs de transit. Cela est utile si le volume
des attaques dépasse la capacité de votre connexion Internet.</p>
<p>Cela fonctionne en annonçant avec <abbr title="Border Gateway Protocol">BGP</abbr> une route vers l’utilisateur attaqué avec
une communauté spécifique. Le routeur de bordure modifie l’adresse du prochain
routeur de ces routes vers une adresse IP spécifique configurée pour transférer
le trafic vers l’interface « nulle ». La <a href="https://www.rfc-editor.org/rfc/rfc7999" title="BLACKHOLE Community">RFC 7999</a> définit <code>65535:666</code> à
cette fin. C’est ce qu’on appelle un trou noir déclenché à distance (“<em>remote
triggered blackhole</em>”, <abbr title="Remote Triggered Blackhole">RTBH</abbr>) et expliqué en détail dans la <a href="https://www.rfc-editor.org/rfc/rfc3882" title="Configuring BGP to Block Denial-of-Service Attacks">RFC 3882</a>.</p>
<p>Il est également possible de bloquer la source des attaques en utilisant <abbr title="Unicast Reverse Path Forwarding">uRPF</abbr>
défini dans la <a href="https://www.rfc-editor.org/rfc/rfc3704" title="Ingress Filtering for Multihomed Networks">RFC 3704</a>. C’est expliqué dans la <a href="https://www.rfc-editor.org/rfc/rfc5635" title="Remote Triggered Black Hole Filtering with Unicast Reverse Path Forwarding (uRPF)">RFC 5635</a>. Cependant,
<abbr title="Unicast Reverse Path Forwarding">uRPF</abbr> peut être une charge importante pour les ressources de votre routeur.
Consultez “<a href="https://xrdocs.io/ncs5500/tutorials/ncs5500-urpf/" title="NCS5500 uRPF: Configuration and Impact on Scale">NCS5500 <abbr title="Unicast Reverse Path Forwarding">uRPF</abbr>: Configuration et Impact sur l’échelle</a>” pour un exemple des types de restrictions
auxquelles vous devez vous attendre lorsque vous activez <abbr title="Unicast Reverse Path Forwarding">uRPF</abbr>.</p>
<p>Pour annoncer les routes, nous pouvons utiliser <a href="https://bird.network.cz/" title="The BIRD Internet Routing Daemon">BIRD</a>. Voici un fichier de
configuration complet pour permettre à tout routeur de les collecter :</p>
<div class="language-junos codehilite"><pre><span/><span class="k">log</span> stderr all;
<span class="k">router</span> id <span class="m">192.0.2.1</span><span class="p">;</span>
<span class="k">protocol</span> device <span class="p">{</span>
scan time <span class="m">10</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">protocol</span> bgp exporter <span class="p">{</span>
ipv4 <span class="p">{</span>
import none;
export where proto = <span class="s2">"blackhole4"</span><span class="p">;</span>
<span class="p">};</span>
ipv6 <span class="p">{</span>
import none;
export where proto = <span class="s2">"blackhole6"</span><span class="p">;</span>
<span class="p">};</span>
local as <span class="m">64666</span><span class="p">;</span>
<span class="hll"> neighbor range <span class="m">192.0.2.0/24</span> external;
</span> multihop;
dynamic name <span class="s2">"exporter"</span><span class="p">;</span>
dynamic name digits <span class="m">2</span><span class="p">;</span>
graceful restart yes;
graceful restart time <span class="m">0</span><span class="p">;</span>
<span class="hll"> long lived graceful restart yes;
</span><span class="hll"> long lived stale time <span class="m">3600</span><span class="p">;</span> <span class="c c-Singleline"># keep routes for 1 hour!</span>
</span><span class="p">}</span>
<span class="k">protocol</span> static blackhole4 <span class="p">{</span>
ipv4;
<span class="hll"> route <span class="m">203.0.113.206/32</span> blackhole <span class="p">{</span>
</span><span class="hll"> bgp_community.add((65535, <span class="m">666</span>));
</span><span class="hll"> <span class="p">};</span>
</span><span class="hll"> route <span class="m">203.0.113.68/32</span> blackhole <span class="p">{</span>
</span><span class="hll"> bgp_community.add((65535, <span class="m">666</span>));
</span><span class="hll"> <span class="p">};</span>
</span><span class="p">}</span>
<span class="k">protocol</span> static blackhole6 <span class="p">{</span>
ipv6;
<span class="p">}</span>
</pre></div>
<p>Nous utilisons <a href="/fr/blog/2018-bgp-llgr" title="BGP LLGR : sessions BGP fiables et réactives"><abbr title="Border Gateway Protocol">BGP</abbr> LLGR</a> pour garantir que les routes sont conservées pendant
une heure, même si la connexion <abbr title="Border Gateway Protocol">BGP</abbr> est interrompue, notamment pendant une
maintenance.</p>
<p>Du côté du récepteur, si vous disposez d’un routeur Cisco fonctionnant sous IOS
XR, vous pouvez utiliser la configuration suivante pour éliminer le trafic reçu
sur la session <abbr title="Border Gateway Protocol">BGP</abbr>. Comme la session <abbr title="Border Gateway Protocol">BGP</abbr> est dédiée à cette utilisation, la
communauté <code>65535:666</code> n’est pas utilisée, mais vous pouvez aussi transférer ces
routes à vos fournisseurs de transit.</p>
<div class="language-ios codehilite"><pre><span/><span class="nf">router</span> static
<span class="nf">vrf</span> public
<span class="nf">address-family</span> ipv4 unicast
<span class="m">192.0.2.1/32</span> Null0 description <span class="s2">"BGP blackhole"</span>
<span class="c c-Singleline"> !</span>
<span class="nf">address-family</span> ipv6 unicast
<span class="m">2001:db8::1/128</span> Null0 description <span class="s2">"BGP blackhole"</span>
<span class="c c-Singleline"> !</span>
<span class="c c-Singleline"> !</span>
<span class="c c-Singleline">!</span>
<span class="nf">route-policy</span> blackhole_ipv4_in_public
<span class="nf">if</span> destination in (0.0.0.0/0 le <span class="m">31</span>) then
<span class="nf">drop</span>
<span class="nf">endif</span>
<span class="nf">set</span> next-hop <span class="m">192.0.2.1</span>
<span class="nf">done</span>
<span class="nf">end-policy</span>
<span class="c c-Singleline">!</span>
<span class="nf">route-policy</span> blackhole_ipv6_in_public
<span class="nf">if</span> destination in (::/0 le <span class="m">127</span>) then
<span class="nf">drop</span>
<span class="nf">endif</span>
<span class="nf">set</span> next-hop <span class="m">2001:db8::1</span>
<span class="nf">done</span>
<span class="nf">end-policy</span>
<span class="c c-Singleline">!</span>
<span class="nf">router</span> bgp <span class="m">12322</span>
<span class="nf">neighbor-group</span> BLACKHOLE_IPV4_PUBLIC
<span class="nf">remote-as</span> <span class="m">64666</span>
<span class="nf">ebgp-multihop</span> <span class="m">255</span>
<span class="nf">update-source</span> Loopback10
<span class="nf">address-family</span> ipv4 unicast
<span class="nf">maximum-prefix</span> <span class="m">100</span> <span class="m">90</span>
<span class="nf">route-policy</span> blackhole_ipv4_in_public in
<span class="nf">route-policy</span> drop out
<span class="nf">long-lived-graceful-restart</span> stale-time send <span class="m">86400</span> accept <span class="m">86400</span>
<span class="c c-Singleline"> !</span>
<span class="nf">address-family</span> ipv6 unicast
<span class="nf">maximum-prefix</span> <span class="m">100</span> <span class="m">90</span>
<span class="nf">route-policy</span> blackhole_ipv6_in_public in
<span class="nf">route-policy</span> drop out
<span class="nf">long-lived-graceful-restart</span> stale-time send <span class="m">86400</span> accept <span class="m">86400</span>
<span class="c c-Singleline"> !</span>
<span class="c c-Singleline"> !</span>
<span class="nf">vrf</span> public
<span class="nf">neighbor</span> <span class="m">192.0.2.1</span>
<span class="nf">use</span> neighbor-group BLACKHOLE_IPV4_PUBLIC
<span class="nf">description</span> akvorado-1
</pre></div>
<p>Lorsque le trafic est éliminé, il est toujours remonté par <em>IPFIX</em> et <em>sFlow</em>.
Dans <em>Akvorado</em>, utilisez <code>ForwardingStatus >= 128</code> comme filtre pour
sélectionner celui-ci.</p>
<p>Bien que cette méthode soit compatible avec tous les routeurs, elle permet à
l’attaque de réussir car la cible est complètement inaccessible. Si votre
routeur le prend en charge, <em>Flowspec</em> peut filtrer sélectivement les flux pour
<strong>arrêter l’attaque sans affecter le client</strong>.</p>
<h2 id="suppression-du-trafic-avec-flowspec">Suppression du trafic avec Flowspec<a class="headerlink" href="#suppression-du-trafic-avec-flowspec" title="Permanent link"></a></h2>
<p><em>Flowspec</em> est défini dans la <a href="https://www.rfc-editor.org/rfc/rfc8955" title="Dissemination of Flow Specification Rules">RFC 8955</a> et permet la transmission de
spécifications de flux dans des sessions <abbr title="Border Gateway Protocol">BGP</abbr>. Une spécification de flux est un
ensemble de critères à appliquer au trafic IP. Ceux-ci incluent le préfixe
source et destination, le protocole IP, le port source et destination et la
longueur des paquets. Chaque spécification de flux est associée à une action,
encodée sous forme d’une communauté étendue : limitation du trafic, marquage ou
redirection.</p>
<p>Pour annoncer des spécifications de flux avec <em>BIRD</em>, nous complétons notre
configuration. La communauté étendue utilisée limite le trafic correspondant à 0
octet par seconde.</p>
<div class="language-junos codehilite"><pre><span/><span class="k">flow4</span> table flowtab4;
<span class="k">flow6</span> table flowtab6;
<span class="k">protocol</span> bgp exporter <span class="p">{</span>
flow4 <span class="p">{</span>
import none;
export where proto = <span class="s2">"flowspec4"</span><span class="p">;</span>
<span class="p">};</span>
flow6 <span class="p">{</span>
import none;
export where proto = <span class="s2">"flowspec6"</span><span class="p">;</span>
<span class="p">};</span>
<span class="c c-Singleline"># […]</span>
<span class="p">}</span>
<span class="k">protocol</span> static flowspec4 <span class="p">{</span>
flow4;
route flow4 <span class="p">{</span>
dst <span class="m">203.0.113.68/32</span><span class="p">;</span>
sport = <span class="m">53</span><span class="p">;</span>
length >= <span class="m">1476</span> && <= <span class="m">1500</span><span class="p">;</span>
proto = <span class="m">17</span><span class="p">;</span>
<span class="p">}{</span>
bgp_ext_community.add((generic, <span class="m">0</span>x80060000, <span class="m">0</span>x00000000));
<span class="p">};</span>
route flow4 <span class="p">{</span>
dst <span class="m">203.0.113.206/32</span><span class="p">;</span>
sport = <span class="m">123</span><span class="p">;</span>
length = <span class="m">468</span><span class="p">;</span>
proto = <span class="m">17</span><span class="p">;</span>
<span class="p">}{</span>
bgp_ext_community.add((generic, <span class="m">0</span>x80060000, <span class="m">0</span>x00000000));
<span class="p">};</span>
<span class="p">}</span>
<span class="k">protocol</span> static flowspec6 <span class="p">{</span>
flow6;
<span class="p">}</span>
</pre></div>
<p>Si vous avez un routeur ASR tournant sous IOS XR, la configuration ressemble à ceci :</p>
<div class="language-ios codehilite"><pre><span/><span class="nf">vrf</span> public
<span class="nf">address-family</span> ipv4 flowspec
<span class="nf">address-family</span> ipv6 flowspec
<span class="c c-Singleline">!</span>
<span class="nf">router</span> bgp <span class="m">12322</span>
<span class="nf">address-family</span> vpnv4 flowspec
<span class="nf">address-family</span> vpnv6 flowspec
<span class="nf">neighbor-group</span> FLOWSPEC_IPV4_PUBLIC
<span class="nf">remote-as</span> <span class="m">64666</span>
<span class="nf">ebgp-multihop</span> <span class="m">255</span>
<span class="nf">update-source</span> Loopback10
<span class="nf">address-family</span> ipv4 flowspec
<span class="nf">long-lived-graceful-restart</span> stale-time send <span class="m">86400</span> accept <span class="m">86400</span>
<span class="nf">route-policy</span> accept in
<span class="nf">route-policy</span> drop out
<span class="nf">maximum-prefix</span> <span class="m">100</span> <span class="m">90</span>
<span class="nf">validation</span> disable
<span class="c c-Singleline"> !</span>
<span class="nf">address-family</span> ipv6 flowspec
<span class="nf">long-lived-graceful-restart</span> stale-time send <span class="m">86400</span> accept <span class="m">86400</span>
<span class="nf">route-policy</span> accept in
<span class="nf">route-policy</span> drop out
<span class="nf">maximum-prefix</span> <span class="m">100</span> <span class="m">90</span>
<span class="nf">validation</span> disable
<span class="c c-Singleline"> !</span>
<span class="c c-Singleline"> !</span>
<span class="nf">vrf</span> public
<span class="nf">address-family</span> ipv4 flowspec
<span class="nf">address-family</span> ipv6 flowspec
<span class="nf">neighbor</span> <span class="m">192.0.2.1</span>
<span class="nf">use</span> neighbor-group FLOWSPEC_IPV4_PUBLIC
<span class="nf">description</span> akvorado-1
</pre></div>
<p>Ensuite, il convient d’activer <em>Flowspec</em> sur toutes les interfaces :</p>
<div class="language-ios codehilite"><pre><span/><span class="nf">flowspec</span>
<span class="nf">vrf</span> public
<span class="nf">address-family</span> ipv4
<span class="nf">local-install</span> interface-all
<span class="c c-Singleline"> !</span>
<span class="nf">address-family</span> ipv6
<span class="nf">local-install</span> interface-all
<span class="c c-Singleline"> !</span>
<span class="c c-Singleline"> !</span>
<span class="c c-Singleline">!</span>
</pre></div>
<p>Comme pour la configuration à base de <abbr title="Remote Triggered Blackhole">RTBH</abbr>, vous pouvez afficher les flux
éliminés avec le filtre <code>ForwardingStatus >= 128</code>.</p>
<h1 id="detection-des-attaques-ddos-suite">Détection des attaques <abbr title="Distributed Denial of Service">DDoS</abbr> (suite)<a class="headerlink" href="#detection-des-attaques-ddos-suite" title="Permanent link"></a></h1>
<p>Dans l’exemple utilisant <em>Flowspec</em>, les flux sont aussi filtrés selon la
longueur des paquets :</p>
<div class="language-junos codehilite"><pre><span/><span class="k">route</span> flow4 <span class="p">{</span>
dst <span class="m">203.0.113.68/32</span><span class="p">;</span>
sport = <span class="m">53</span><span class="p">;</span>
length >= <span class="m">1476</span> && <= <span class="m">1500</span><span class="p">;</span>
proto = <span class="m">17</span><span class="p">;</span>
<span class="p">}{</span>
bgp_ext_community.add((generic, <span class="m">0</span>x80060000, <span class="m">0</span>x00000000));
<span class="p">};</span>
</pre></div>
<p>C’est une addition importante : les requêtes DNS légitimes ont une taille
inférieure à cela et ne sont donc pas filtrées<sup id="fnref-dns"><a class="footnote-ref" href="#fn-dns">2</a></sup>. Avec <em>ClickHouse</em>, vous pouvez
obtenir les 10<sup>ème</sup> et 90<sup>ème</sup> centiles des tailles de paquets
avec <code>quantiles(0.1, 0.9)(Bytes/Packets)</code>.</p>
<p>Le dernier problème que nous devons résoudre est de rendre les requêtes plus
rapides : elles peuvent avoir besoin de plusieurs secondes pour collecter les
données et sont susceptibles de consommer des ressources substantielles de votre
base de données <em>ClickHouse</em>. Une solution consiste à créer une vue matérialisée
pour pré-agréger les résultats :</p>
<div class="language-sql codehilite"><pre><span/><span class="k">CREATE</span><span class="w"> </span><span class="k">TABLE</span><span class="w"> </span><span class="n">ddos_logs</span><span class="w"> </span><span class="p">(</span>
<span class="w"> </span><span class="n">TimeReceived</span><span class="w"> </span><span class="n">DateTime</span><span class="p">,</span>
<span class="w"> </span><span class="n">DstAddr</span><span class="w"> </span><span class="n">IPv6</span><span class="p">,</span>
<span class="w"> </span><span class="n">Proto</span><span class="w"> </span><span class="n">UInt32</span><span class="p">,</span>
<span class="w"> </span><span class="n">SrcPort</span><span class="w"> </span><span class="n">UInt16</span><span class="p">,</span>
<span class="w"> </span><span class="n">Gbps</span><span class="w"> </span><span class="n">SimpleAggregateFunction</span><span class="p">(</span><span class="k">sum</span><span class="p">,</span><span class="w"> </span><span class="n">Float64</span><span class="p">),</span>
<span class="w"> </span><span class="n">Mpps</span><span class="w"> </span><span class="n">SimpleAggregateFunction</span><span class="p">(</span><span class="k">sum</span><span class="p">,</span><span class="w"> </span><span class="n">Float64</span><span class="p">),</span>
<span class="w"> </span><span class="n">sources</span><span class="w"> </span><span class="n">AggregateFunction</span><span class="p">(</span><span class="n">uniqCombined</span><span class="p">(</span><span class="mi">12</span><span class="p">),</span><span class="w"> </span><span class="n">IPv6</span><span class="p">),</span>
<span class="w"> </span><span class="n">countries</span><span class="w"> </span><span class="n">AggregateFunction</span><span class="p">(</span><span class="n">uniqCombined</span><span class="p">(</span><span class="mi">12</span><span class="p">),</span><span class="w"> </span><span class="n">FixedString</span><span class="p">(</span><span class="mi">2</span><span class="p">)),</span>
<span class="w"> </span><span class="k">size</span><span class="w"> </span><span class="n">AggregateFunction</span><span class="p">(</span><span class="n">quantiles</span><span class="p">(</span><span class="mi">0</span><span class="p">.</span><span class="mi">1</span><span class="p">,</span><span class="w"> </span><span class="mi">0</span><span class="p">.</span><span class="mi">9</span><span class="p">),</span><span class="w"> </span><span class="n">UInt64</span><span class="p">)</span>
<span class="hll"><span class="p">)</span><span class="w"> </span><span class="n">ENGINE</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">SummingMergeTree</span>
</span><span class="n">PARTITION</span><span class="w"> </span><span class="k">BY</span><span class="w"> </span><span class="n">toStartOfHour</span><span class="p">(</span><span class="n">TimeReceived</span><span class="p">)</span>
<span class="k">ORDER</span><span class="w"> </span><span class="k">BY</span><span class="w"> </span><span class="p">(</span><span class="n">TimeReceived</span><span class="p">,</span><span class="w"> </span><span class="n">DstAddr</span><span class="p">,</span><span class="w"> </span><span class="n">Proto</span><span class="p">,</span><span class="w"> </span><span class="n">SrcPort</span><span class="p">)</span>
<span class="n">TTL</span><span class="w"> </span><span class="n">toStartOfHour</span><span class="p">(</span><span class="n">TimeReceived</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="nb">INTERVAL</span><span class="w"> </span><span class="mi">6</span><span class="w"> </span><span class="n">HOUR</span><span class="w"> </span><span class="k">DELETE</span><span class="w"> </span><span class="p">;</span>
<span class="k">CREATE</span><span class="w"> </span><span class="n">MATERIALIZED</span><span class="w"> </span><span class="k">VIEW</span><span class="w"> </span><span class="n">ddos_logs_view</span><span class="w"> </span><span class="k">TO</span><span class="w"> </span><span class="n">ddos_logs</span><span class="w"> </span><span class="k">AS</span>
<span class="w"> </span><span class="k">SELECT</span>
<span class="w"> </span><span class="n">toStartOfMinute</span><span class="p">(</span><span class="n">TimeReceived</span><span class="p">)</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">TimeReceived</span><span class="p">,</span>
<span class="w"> </span><span class="n">DstAddr</span><span class="p">,</span>
<span class="w"> </span><span class="n">Proto</span><span class="p">,</span>
<span class="w"> </span><span class="n">SrcPort</span><span class="p">,</span>
<span class="w"> </span><span class="k">sum</span><span class="p">(((((</span><span class="n">Bytes</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="n">SamplingRate</span><span class="p">)</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="mi">8</span><span class="p">)</span><span class="w"> </span><span class="o">/</span><span class="w"> </span><span class="mi">1000</span><span class="p">)</span><span class="w"> </span><span class="o">/</span><span class="w"> </span><span class="mi">1000</span><span class="p">)</span><span class="w"> </span><span class="o">/</span><span class="w"> </span><span class="mi">1000</span><span class="p">)</span><span class="w"> </span><span class="o">/</span><span class="w"> </span><span class="mi">60</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">Gbps</span><span class="p">,</span>
<span class="w"> </span><span class="k">sum</span><span class="p">(((</span><span class="n">Packets</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="n">SamplingRate</span><span class="p">)</span><span class="w"> </span><span class="o">/</span><span class="w"> </span><span class="mi">1000</span><span class="p">)</span><span class="w"> </span><span class="o">/</span><span class="w"> </span><span class="mi">1000</span><span class="p">)</span><span class="w"> </span><span class="o">/</span><span class="w"> </span><span class="mi">60</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">Mpps</span><span class="p">,</span>
<span class="w"> </span><span class="n">uniqCombinedState</span><span class="p">(</span><span class="mi">12</span><span class="p">)(</span><span class="n">SrcAddr</span><span class="p">)</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">sources</span><span class="p">,</span>
<span class="w"> </span><span class="n">uniqCombinedState</span><span class="p">(</span><span class="mi">12</span><span class="p">)(</span><span class="n">SrcCountry</span><span class="p">)</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">countries</span><span class="p">,</span>
<span class="w"> </span><span class="n">quantilesState</span><span class="p">(</span><span class="mi">0</span><span class="p">.</span><span class="mi">1</span><span class="p">,</span><span class="w"> </span><span class="mi">0</span><span class="p">.</span><span class="mi">9</span><span class="p">)(</span><span class="n">toUInt64</span><span class="p">(</span><span class="n">Bytes</span><span class="o">/</span><span class="n">Packets</span><span class="p">))</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="k">size</span>
<span class="w"> </span><span class="k">FROM</span><span class="w"> </span><span class="n">flows</span>
<span class="w"> </span><span class="k">WHERE</span><span class="w"> </span><span class="n">DstNetRole</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'customers'</span>
<span class="w"> </span><span class="k">GROUP</span><span class="w"> </span><span class="k">BY</span>
<span class="w"> </span><span class="n">TimeReceived</span><span class="p">,</span>
<span class="w"> </span><span class="n">DstAddr</span><span class="p">,</span>
<span class="w"> </span><span class="n">Proto</span><span class="p">,</span>
<span class="w"> </span><span class="n">SrcPort</span>
</pre></div>
<p>La table <code>ddos_logs</code> utilise le moteur <code>SummingMergeTree</code>. Lorsque la table
reçoit de nouvelles données, <em>ClickHouse</em> remplace toutes les lignes ayant la
même clé de tri, telle que définie par la directive <code>ORDER BY</code>, par une seule
ligne qui contient des valeurs résumées<sup id="fnref-materialized"><a class="footnote-ref" href="#fn-materialized">3</a></sup> en utilisant soit la
fonction <code>sum()</code> soit la fonction d’agrégation explicitement spécifiée
(<code>uniqCombined</code> et <code>quantiles</code> dans notre exemple).</p>
<p>Enfin, nous modifions notre requête initiale avec la requête suivante :</p>
<div class="language-sql codehilite"><pre><span/><span class="k">SELECT</span><span class="w"> </span><span class="o">*</span>
<span class="k">FROM</span><span class="w"> </span><span class="p">(</span>
<span class="w"> </span><span class="k">SELECT</span>
<span class="w"> </span><span class="n">TimeReceived</span><span class="p">,</span>
<span class="w"> </span><span class="n">DstAddr</span><span class="p">,</span>
<span class="w"> </span><span class="n">dictGetOrDefault</span><span class="p">(</span><span class="s1">'protocols'</span><span class="p">,</span><span class="w"> </span><span class="s1">'name'</span><span class="p">,</span><span class="w"> </span><span class="n">Proto</span><span class="p">,</span><span class="w"> </span><span class="s1">'???'</span><span class="p">)</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">Proto</span><span class="p">,</span>
<span class="w"> </span><span class="n">SrcPort</span><span class="p">,</span>
<span class="w"> </span><span class="k">sum</span><span class="p">(</span><span class="n">Gbps</span><span class="p">)</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">Gbps</span><span class="p">,</span>
<span class="w"> </span><span class="k">sum</span><span class="p">(</span><span class="n">Mpps</span><span class="p">)</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">Mpps</span><span class="p">,</span>
<span class="w"> </span><span class="n">uniqCombinedMerge</span><span class="p">(</span><span class="mi">12</span><span class="p">)(</span><span class="n">sources</span><span class="p">)</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">sources</span><span class="p">,</span>
<span class="w"> </span><span class="n">uniqCombinedMerge</span><span class="p">(</span><span class="mi">12</span><span class="p">)(</span><span class="n">countries</span><span class="p">)</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">countries</span><span class="p">,</span>
<span class="w"> </span><span class="n">quantilesMerge</span><span class="p">(</span><span class="mi">0</span><span class="p">.</span><span class="mi">1</span><span class="p">,</span><span class="w"> </span><span class="mi">0</span><span class="p">.</span><span class="mi">9</span><span class="p">)(</span><span class="k">size</span><span class="p">)</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="k">size</span>
<span class="w"> </span><span class="k">FROM</span><span class="w"> </span><span class="n">ddos_logs</span>
<span class="w"> </span><span class="k">WHERE</span><span class="w"> </span><span class="n">TimeReceived</span><span class="w"> </span><span class="o">></span><span class="w"> </span><span class="n">now</span><span class="p">()</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="nb">INTERVAL</span><span class="w"> </span><span class="mi">60</span><span class="w"> </span><span class="k">MINUTE</span>
<span class="w"> </span><span class="k">GROUP</span><span class="w"> </span><span class="k">BY</span>
<span class="w"> </span><span class="n">TimeReceived</span><span class="p">,</span>
<span class="w"> </span><span class="n">DstAddr</span><span class="p">,</span>
<span class="w"> </span><span class="n">Proto</span><span class="p">,</span>
<span class="w"> </span><span class="n">SrcPort</span>
<span class="p">)</span>
<span class="k">WHERE</span><span class="w"> </span><span class="p">(</span><span class="n">Gbps</span><span class="w"> </span><span class="o">></span><span class="w"> </span><span class="mi">1</span><span class="p">)</span>
<span class="w"> </span><span class="k">OR</span><span class="w"> </span><span class="p">((</span><span class="n">Proto</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'UDP'</span><span class="p">)</span><span class="w"> </span><span class="k">AND</span><span class="w"> </span><span class="p">(</span><span class="n">Gbps</span><span class="w"> </span><span class="o">></span><span class="w"> </span><span class="mi">0</span><span class="p">.</span><span class="mi">2</span><span class="p">))</span><span class="w"> </span>
<span class="w"> </span><span class="k">OR</span><span class="w"> </span><span class="p">((</span><span class="n">sources</span><span class="w"> </span><span class="o">></span><span class="w"> </span><span class="mi">20</span><span class="p">)</span><span class="w"> </span><span class="k">AND</span><span class="w"> </span><span class="p">(</span><span class="n">Gbps</span><span class="w"> </span><span class="o">></span><span class="w"> </span><span class="mi">0</span><span class="p">.</span><span class="mi">1</span><span class="p">))</span><span class="w"> </span>
<span class="w"> </span><span class="k">OR</span><span class="w"> </span><span class="p">((</span><span class="n">countries</span><span class="w"> </span><span class="o">></span><span class="w"> </span><span class="mi">10</span><span class="p">)</span><span class="w"> </span><span class="k">AND</span><span class="w"> </span><span class="p">(</span><span class="n">Gbps</span><span class="w"> </span><span class="o">></span><span class="w"> </span><span class="mi">0</span><span class="p">.</span><span class="mi">1</span><span class="p">))</span>
<span class="k">ORDER</span><span class="w"> </span><span class="k">BY</span>
<span class="w"> </span><span class="n">TimeReceived</span><span class="w"> </span><span class="k">DESC</span><span class="p">,</span>
<span class="w"> </span><span class="n">Gbps</span><span class="w"> </span><span class="k">DESC</span>
</pre></div>
<h1 id="assemblage-final">Assemblage final<a class="headerlink" href="#assemblage-final" title="Permanent link"></a></h1>
<p>Pour résumer, la construction d’un système de suppression des attaques <abbr title="Distributed Denial of Service">DDoS</abbr>
nécessite de suivre les étapes suivantes :</p>
<ol>
<li>définir un ensemble de critères pour détecter une attaque <abbr title="Distributed Denial of Service">DDoS</abbr></li>
<li>traduire ces critères en requêtes SQL</li>
<li>pré-agréger les flux dans des tables <code>SummingMergeTree</code></li>
<li>interroger et transformer les résultats en un fichier de configuration pour <em>BIRD</em></li>
<li>configurer vos routeurs pour extraire les routes de <em>BIRD</em></li>
</ol>
<p>Un script Python comme celui qui suit permet de gérer la quatrième étape. Pour
chaque cible attaquée, il génère à la fois une règle <em>Flowspec</em> et une route de
type « trou noir ».</p>
<div class="language-python codehilite"><pre><span/><span class="kn">import</span> <span class="nn">socket</span>
<span class="kn">import</span> <span class="nn">types</span>
<span class="kn">from</span> <span class="nn">clickhouse_driver</span> <span class="kn">import</span> <span class="n">Client</span> <span class="k">as</span> <span class="n">CHClient</span>
<span class="c1"># Insérez ici votre requête SQL</span>
<span class="n">SQL_QUERY</span> <span class="o">=</span> <span class="s2">"…"</span>
<span class="c1"># Combien de règles anti-DDoS doit-on publier au plus ?</span>
<span class="n">MAX_DDOS_RULES</span> <span class="o">=</span> <span class="mi">20</span>
<span class="k">def</span> <span class="nf">empty_ruleset</span><span class="p">():</span>
<span class="n">ruleset</span> <span class="o">=</span> <span class="n">types</span><span class="o">.</span><span class="n">SimpleNamespace</span><span class="p">()</span>
<span class="n">ruleset</span><span class="o">.</span><span class="n">flowspec</span> <span class="o">=</span> <span class="n">types</span><span class="o">.</span><span class="n">SimpleNamespace</span><span class="p">()</span>
<span class="n">ruleset</span><span class="o">.</span><span class="n">blackhole</span> <span class="o">=</span> <span class="n">types</span><span class="o">.</span><span class="n">SimpleNamespace</span><span class="p">()</span>
<span class="n">ruleset</span><span class="o">.</span><span class="n">flowspec</span><span class="o">.</span><span class="n">v4</span> <span class="o">=</span> <span class="p">[]</span>
<span class="n">ruleset</span><span class="o">.</span><span class="n">flowspec</span><span class="o">.</span><span class="n">v6</span> <span class="o">=</span> <span class="p">[]</span>
<span class="n">ruleset</span><span class="o">.</span><span class="n">blackhole</span><span class="o">.</span><span class="n">v4</span> <span class="o">=</span> <span class="p">[]</span>
<span class="n">ruleset</span><span class="o">.</span><span class="n">blackhole</span><span class="o">.</span><span class="n">v6</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">return</span> <span class="n">ruleset</span>
<span class="n">current_ruleset</span> <span class="o">=</span> <span class="n">empty_ruleset</span><span class="p">()</span>
<span class="n">client</span> <span class="o">=</span> <span class="n">CHClient</span><span class="p">(</span><span class="n">host</span><span class="o">=</span><span class="s2">"clickhouse.akvorado.net"</span><span class="p">)</span>
<span class="k">while</span> <span class="kc">True</span><span class="p">:</span>
<span class="n">results</span> <span class="o">=</span> <span class="n">client</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">SQL_QUERY</span><span class="p">)</span>
<span class="n">seen</span> <span class="o">=</span> <span class="p">{}</span>
<span class="n">new_ruleset</span> <span class="o">=</span> <span class="n">empty_ruleset</span><span class="p">()</span>
<span class="k">for</span> <span class="p">(</span><span class="n">t</span><span class="p">,</span> <span class="n">addr</span><span class="p">,</span> <span class="n">proto</span><span class="p">,</span> <span class="n">port</span><span class="p">,</span> <span class="n">gbps</span><span class="p">,</span> <span class="n">mpps</span><span class="p">,</span> <span class="n">sources</span><span class="p">,</span> <span class="n">countries</span><span class="p">,</span> <span class="n">size</span><span class="p">)</span> <span class="ow">in</span> <span class="n">results</span><span class="p">:</span>
<span class="k">if</span> <span class="p">(</span><span class="n">addr</span><span class="p">,</span> <span class="n">proto</span><span class="p">,</span> <span class="n">port</span><span class="p">)</span> <span class="ow">in</span> <span class="n">seen</span><span class="p">:</span>
<span class="k">continue</span>
<span class="n">seen</span><span class="p">[(</span><span class="n">addr</span><span class="p">,</span> <span class="n">proto</span><span class="p">,</span> <span class="n">port</span><span class="p">)]</span> <span class="o">=</span> <span class="kc">True</span>
<span class="c1"># Flowspec</span>
<span class="k">if</span> <span class="n">addr</span><span class="o">.</span><span class="n">ipv4_mapped</span><span class="p">:</span>
<span class="n">address</span> <span class="o">=</span> <span class="n">addr</span><span class="o">.</span><span class="n">ipv4_mapped</span>
<span class="n">rules</span> <span class="o">=</span> <span class="n">new_ruleset</span><span class="o">.</span><span class="n">flowspec</span><span class="o">.</span><span class="n">v4</span>
<span class="n">table</span> <span class="o">=</span> <span class="s2">"flow4"</span>
<span class="n">mask</span> <span class="o">=</span> <span class="mi">32</span>
<span class="n">nh</span> <span class="o">=</span> <span class="s2">"proto"</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">address</span> <span class="o">=</span> <span class="n">addr</span>
<span class="n">rules</span> <span class="o">=</span> <span class="n">new_ruleset</span><span class="o">.</span><span class="n">flowspec</span><span class="o">.</span><span class="n">v6</span>
<span class="n">table</span> <span class="o">=</span> <span class="s2">"flow6"</span>
<span class="n">mask</span> <span class="o">=</span> <span class="mi">128</span>
<span class="n">nh</span> <span class="o">=</span> <span class="s2">"next header"</span>
<span class="k">if</span> <span class="n">size</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">==</span> <span class="n">size</span><span class="p">[</span><span class="mi">1</span><span class="p">]:</span>
<span class="n">length</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">"length = </span><span class="si">{</span><span class="nb">int</span><span class="p">(</span><span class="n">size</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span><span class="si">}</span><span class="s2">"</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">length</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">"length >= </span><span class="si">{</span><span class="nb">int</span><span class="p">(</span><span class="n">size</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span><span class="si">}</span><span class="s2"> && <= </span><span class="si">{</span><span class="nb">int</span><span class="p">(</span><span class="n">size</span><span class="p">[</span><span class="mi">1</span><span class="p">])</span><span class="si">}</span><span class="s2">"</span>
<span class="n">header</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">"""</span>
<span class="s2"># Time: </span><span class="si">{</span><span class="n">t</span><span class="si">}</span>
<span class="s2"># Source: </span><span class="si">{</span><span class="n">address</span><span class="si">}</span><span class="s2">, protocol: </span><span class="si">{</span><span class="n">proto</span><span class="si">}</span><span class="s2">, port: </span><span class="si">{</span><span class="n">port</span><span class="si">}</span>
<span class="s2"># Gbps/Mpps: </span><span class="si">{</span><span class="n">gbps</span><span class="si">:</span><span class="s2">.3</span><span class="si">}</span><span class="s2">/</span><span class="si">{</span><span class="n">mpps</span><span class="si">:</span><span class="s2">.3</span><span class="si">}</span><span class="s2">, packet size: </span><span class="si">{</span><span class="nb">int</span><span class="p">(</span><span class="n">size</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span><span class="si">}</span><span class="s2"><=X<=</span><span class="si">{</span><span class="nb">int</span><span class="p">(</span><span class="n">size</span><span class="p">[</span><span class="mi">1</span><span class="p">])</span><span class="si">}</span>
<span class="s2"># Flows: </span><span class="si">{</span><span class="n">flows</span><span class="si">}</span><span class="s2">, sources: </span><span class="si">{</span><span class="n">sources</span><span class="si">}</span><span class="s2">, countries: </span><span class="si">{</span><span class="n">countries</span><span class="si">}</span>
<span class="s2">"""</span>
<span class="n">rules</span><span class="o">.</span><span class="n">append</span><span class="p">(</span>
<span class="sa">f</span><span class="s2">"""</span><span class="si">{</span><span class="n">header</span><span class="si">}</span>
<span class="s2">route </span><span class="si">{</span><span class="n">table</span><span class="si">}</span><span class="s2"> </span><span class="se">{{</span>
<span class="s2"> dst </span><span class="si">{</span><span class="n">address</span><span class="si">}</span><span class="s2">/</span><span class="si">{</span><span class="n">mask</span><span class="si">}</span><span class="s2">;</span>
<span class="s2"> sport = </span><span class="si">{</span><span class="n">port</span><span class="si">}</span><span class="s2">;</span>
<span class="s2"> </span><span class="si">{</span><span class="n">length</span><span class="si">}</span><span class="s2">;</span>
<span class="s2"> </span><span class="si">{</span><span class="n">nh</span><span class="si">}</span><span class="s2"> = </span><span class="si">{</span><span class="n">socket</span><span class="o">.</span><span class="n">getprotobyname</span><span class="p">(</span><span class="n">proto</span><span class="p">)</span><span class="si">}</span><span class="s2">;</span>
<span class="se">}}{{</span>
<span class="s2"> bgp_ext_community.add((generic, 0x80060000, 0x00000000));</span>
<span class="se">}}</span><span class="s2">;</span>
<span class="s2">"""</span>
<span class="p">)</span>
<span class="c1"># Blackhole</span>
<span class="k">if</span> <span class="n">addr</span><span class="o">.</span><span class="n">ipv4_mapped</span><span class="p">:</span>
<span class="n">rules</span> <span class="o">=</span> <span class="n">new_ruleset</span><span class="o">.</span><span class="n">blackhole</span><span class="o">.</span><span class="n">v4</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">rules</span> <span class="o">=</span> <span class="n">new_ruleset</span><span class="o">.</span><span class="n">blackhole</span><span class="o">.</span><span class="n">v6</span>
<span class="n">rules</span><span class="o">.</span><span class="n">append</span><span class="p">(</span>
<span class="sa">f</span><span class="s2">"""</span><span class="si">{</span><span class="n">header</span><span class="si">}</span>
<span class="s2">route </span><span class="si">{</span><span class="n">address</span><span class="si">}</span><span class="s2">/</span><span class="si">{</span><span class="n">mask</span><span class="si">}</span><span class="s2"> blackhole </span><span class="se">{{</span>
<span class="s2"> bgp_community.add((65535, 666));</span>
<span class="se">}}</span><span class="s2">;</span>
<span class="s2">"""</span>
<span class="p">)</span>
<span class="n">new_ruleset</span><span class="o">.</span><span class="n">flowspec</span><span class="o">.</span><span class="n">v4</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span>
<span class="nb">set</span><span class="p">(</span><span class="n">new_ruleset</span><span class="o">.</span><span class="n">flowspec</span><span class="o">.</span><span class="n">v4</span><span class="p">[:</span><span class="n">MAX_DDOS_RULES</span><span class="p">])</span>
<span class="p">)</span>
<span class="n">new_ruleset</span><span class="o">.</span><span class="n">flowspec</span><span class="o">.</span><span class="n">v6</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span>
<span class="nb">set</span><span class="p">(</span><span class="n">new_ruleset</span><span class="o">.</span><span class="n">flowspec</span><span class="o">.</span><span class="n">v6</span><span class="p">[:</span><span class="n">MAX_DDOS_RULES</span><span class="p">])</span>
<span class="p">)</span>
<span class="c1"># TODO: publier les changements par courriel ou chat</span>
<span class="n">current_ruleset</span> <span class="o">=</span> <span class="n">new_ruleset</span>
<span class="n">changes</span> <span class="o">=</span> <span class="kc">False</span>
<span class="k">for</span> <span class="n">rules</span><span class="p">,</span> <span class="n">path</span> <span class="ow">in</span> <span class="p">(</span>
<span class="p">(</span><span class="n">current_ruleset</span><span class="o">.</span><span class="n">flowspec</span><span class="o">.</span><span class="n">v4</span><span class="p">,</span> <span class="s2">"v4-flowspec"</span><span class="p">),</span>
<span class="p">(</span><span class="n">current_ruleset</span><span class="o">.</span><span class="n">flowspec</span><span class="o">.</span><span class="n">v6</span><span class="p">,</span> <span class="s2">"v6-flowspec"</span><span class="p">),</span>
<span class="p">(</span><span class="n">current_ruleset</span><span class="o">.</span><span class="n">blackhole</span><span class="o">.</span><span class="n">v4</span><span class="p">,</span> <span class="s2">"v4-blackhole"</span><span class="p">),</span>
<span class="p">(</span><span class="n">current_ruleset</span><span class="o">.</span><span class="n">blackhole</span><span class="o">.</span><span class="n">v6</span><span class="p">,</span> <span class="s2">"v6-blackhole"</span><span class="p">),</span>
<span class="p">):</span>
<span class="n">path</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="s2">"/etc/bird/"</span><span class="p">,</span> <span class="sa">f</span><span class="s2">"</span><span class="si">{</span><span class="n">path</span><span class="si">}</span><span class="s2">.conf"</span><span class="p">)</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="sa">f</span><span class="s2">"</span><span class="si">{</span><span class="n">path</span><span class="si">}</span><span class="s2">.tmp"</span><span class="p">,</span> <span class="s2">"w"</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
<span class="k">for</span> <span class="n">r</span> <span class="ow">in</span> <span class="n">rules</span><span class="p">:</span>
<span class="n">f</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="n">r</span><span class="p">)</span>
<span class="n">changes</span> <span class="o">=</span> <span class="p">(</span>
<span class="n">changes</span> <span class="ow">or</span> <span class="ow">not</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">exists</span><span class="p">(</span><span class="n">path</span><span class="p">)</span> <span class="ow">or</span> <span class="ow">not</span> <span class="n">samefile</span><span class="p">(</span><span class="n">path</span><span class="p">,</span> <span class="sa">f</span><span class="s2">"</span><span class="si">{</span><span class="n">path</span><span class="si">}</span><span class="s2">.tmp"</span><span class="p">)</span>
<span class="p">)</span>
<span class="n">os</span><span class="o">.</span><span class="n">rename</span><span class="p">(</span><span class="sa">f</span><span class="s2">"</span><span class="si">{</span><span class="n">path</span><span class="si">}</span><span class="s2">.tmp"</span><span class="p">,</span> <span class="n">path</span><span class="p">)</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">changes</span><span class="p">:</span>
<span class="k">continue</span>
<span class="n">proc</span> <span class="o">=</span> <span class="n">subprocess</span><span class="o">.</span><span class="n">Popen</span><span class="p">(</span>
<span class="p">[</span><span class="s2">"birdc"</span><span class="p">,</span> <span class="s2">"configure"</span><span class="p">],</span>
<span class="n">stdin</span><span class="o">=</span><span class="n">subprocess</span><span class="o">.</span><span class="n">DEVNULL</span><span class="p">,</span>
<span class="n">stdout</span><span class="o">=</span><span class="n">subprocess</span><span class="o">.</span><span class="n">PIPE</span><span class="p">,</span>
<span class="n">stderr</span><span class="o">=</span><span class="n">subprocess</span><span class="o">.</span><span class="n">PIPE</span><span class="p">,</span>
<span class="p">)</span>
<span class="n">stdout</span><span class="p">,</span> <span class="n">stderr</span> <span class="o">=</span> <span class="n">proc</span><span class="o">.</span><span class="n">communicate</span><span class="p">(</span><span class="kc">None</span><span class="p">)</span>
<span class="n">stdout</span> <span class="o">=</span> <span class="n">stdout</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s2">"utf-8"</span><span class="p">,</span> <span class="s2">"replace"</span><span class="p">)</span>
<span class="n">stderr</span> <span class="o">=</span> <span class="n">stderr</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s2">"utf-8"</span><span class="p">,</span> <span class="s2">"replace"</span><span class="p">)</span>
<span class="k">if</span> <span class="n">proc</span><span class="o">.</span><span class="n">returncode</span> <span class="o">!=</span> <span class="mi">0</span><span class="p">:</span>
<span class="n">logger</span><span class="o">.</span><span class="n">error</span><span class="p">(</span>
<span class="s2">"</span><span class="si">{}</span><span class="s2"> error:</span><span class="se">\n</span><span class="si">{}</span><span class="se">\n</span><span class="si">{}</span><span class="s2">"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span>
<span class="s2">"birdc reconfigure"</span><span class="p">,</span>
<span class="s2">"</span><span class="se">\n</span><span class="s2">"</span><span class="o">.</span><span class="n">join</span><span class="p">(</span>
<span class="p">[</span><span class="s2">" O: </span><span class="si">{}</span><span class="s2">"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">line</span><span class="p">)</span> <span class="k">for</span> <span class="n">line</span> <span class="ow">in</span> <span class="n">stdout</span><span class="o">.</span><span class="n">rstrip</span><span class="p">()</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s2">"</span><span class="se">\n</span><span class="s2">"</span><span class="p">)]</span>
<span class="p">),</span>
<span class="s2">"</span><span class="se">\n</span><span class="s2">"</span><span class="o">.</span><span class="n">join</span><span class="p">(</span>
<span class="p">[</span><span class="s2">" E: </span><span class="si">{}</span><span class="s2">"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">line</span><span class="p">)</span> <span class="k">for</span> <span class="n">line</span> <span class="ow">in</span> <span class="n">stderr</span><span class="o">.</span><span class="n">rstrip</span><span class="p">()</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s2">"</span><span class="se">\n</span><span class="s2">"</span><span class="p">)]</span>
<span class="p">),</span>
<span class="p">)</span>
<span class="p">)</span>
</pre></div>
<hr/>
<p>Jusqu’à ce qu’<em>Akvorado</em> intègre la détection et la lutte contre les attaques
<abbr title="Distributed Denial of Service">DDoS</abbr>, les idées présentées dans ce billet fournissent une base solide pour
commencer à construire votre propre système anti-<abbr title="Distributed Denial of Service">DDoS</abbr>. 🛡️</p>
<div class="footnote">
<hr/>
<ol>
<li id="fn-format">
<p><em>ClickHouse</em> peut exporter les résultats au format <em>Markdown</em> en
ajoutant <code>FORMAT Markdown</code> à la requête. <a class="footnote-backref" href="#fnref-format" title="Jump back to footnote 1 in the text">↩︎</a></p>
</li>
<li id="fn-dns">
<p>Bien que la plupart des clients DNS réessaient avec TCP en cas d’échec,
ce n’est pas toujours le cas : jusqu’à <a href="https://git.musl-libc.org/cgit/musl/commit/?id=51d4669fb97782f6a66606da852b5afd49a08001">récemment</a>, la
<a href="https://musl.libc.org/">bibliothèque Musl</a> ne le faisait pas. <a class="footnote-backref" href="#fnref-dns" title="Jump back to footnote 2 in the text">↩︎</a></p>
</li>
<li id="fn-materialized">
<p>La vue matérialisée agrège également les données qu’elle a sous
la main, à la fois pour l’efficacité et pour s’assurer que nous travaillons
avec les bons types de données. <a class="footnote-backref" href="#fnref-materialized" title="Jump back to footnote 3 in the text">↩︎</a></p>
</li>
</ol>
</div>
Langage inspiré de SQL pour filtrer les fluxVincent Bernat2023-02-13T08:06:06Zhttp://www.luffy.cx/fr/blog/2023-langage-sql-filtrage.html
<p><a href="/fr/blog/2022-akvorado-collecteur-flux" title="Akvorado : collecteur et visualisateur de flux réseau">Akvorado</a> collecte les flux réseau à l’aide d’<a href="https://www.rfc-editor.org/rfc/rfc7011" title="RFC 7011: Specification of the IP Flow Information Export (IPFIX) Protocol for the Exchange of Flow Information">IPFIX</a> ou de <a href="https://www.rfc-editor.org/rfc/rfc3176" title="RFC 3176: Specification of the IP Flow Information Export (IPFIX) Protocol for the Exchange of Flow Information">sFlow</a>. Il
les stocke dans une base de données <a href="https://clickhouse.com/" title="ClickHouse: OLAP DBMS">ClickHouse</a>. Une console web permet à
l’utilisateur de faire des requêtes sur les données pour obtenir des graphiques.
Un aspect intéressant de cette console est la possibilité de filtrer les flux
avec un langage inspiré de SQL :</p>
<figure><div class="lf-media-outer"><span class="lf-media-inner"><video src="https://d2pzklc15kok91.cloudfront.net/images/akvorado-filter.d8bf4a41cd7883.mp4" width="700" height="420" muted="" loop="" autoplay="" playsinline="" controls="" class="lf-media lf-opaque"/></span></div><figcaption>Filter editor in Akvorado console</figcaption></figure>
<!--
InIfBoundary = external AND
ExporterRegion = "france" AND
InIfConnectivity = "transit" AND
SrcAS = AS15169 AND
DstAddr << 2a01:e0f:ffff::/48
-->
<p>Souvent, les interfaces web exposent un <a href="https://dabernathy89.github.io/vue-query-builder/" title="Vue Query Builder">constructeur</a> <a href="https://react-querybuilder.js.org/" title="React Query Builder">de
requêtes</a> pour concevoir de tels filtres. À la place, j’ai
choisi de combiner un langage similaire à SQL avec un éditeur prenant en charge
la <strong>complétion</strong>, la <strong>coloration syntaxique</strong> et la <strong>vérification
syntaxique</strong><sup id="fnref-lazy"><a class="footnote-ref" href="#fn-lazy">1</a></sup>.</p>
<p>L’analyseur syntaxique du langage est construit avec <a href="https://github.com/mna/pigeon" title="Command pigeon generates parsers in Go from a PEG grammar">pigeon</a> (<em>Go</em>) à partir
d’une <a href="https://en.wikipedia.org/wiki/Parsing_expression_grammar" title="Parsing expression grammar on Wikipedia">grammaire d’expression d’analyse syntaxique</a>
(<em>parsing expression grammar</em> ou <abbr title="Parsing Expression Grammar">PEG</abbr>). Le composant à la base de l’éditeur est
<a href="https://codemirror.net/" title="CodeMirror: Extensible Code Editor">CodeMirror</a> (<em>TypeScript</em>).</p>
<h1 id="analyseur-syntaxique">Analyseur syntaxique<a class="headerlink" href="#analyseur-syntaxique" title="Permanent link"></a></h1>
<p>Les grammaires <abbr title="Parsing Expression Grammar">PEG</abbr> sont relativement récentes<sup id="fnref-recent"><a class="footnote-ref" href="#fn-recent">2</a></sup> et sont une alternative
aux grammaires contextuelles. Elles sont plus faciles à écrire et peuvent
générer de meilleurs messages d’erreur. Par exemple, Python a <a href="https://peps.python.org/pep-0617/" title="PEP 617 – New PEG parser for CPython">basculé d’un
analyseur de type <abbr title="left-to-right, leftmost derivation">LL</abbr>(1) à un analyseur basé sur une grammaire <abbr title="Parsing Expression Grammar">PEG</abbr></a>
depuis Python 3.9.</p>
<p><a href="https://github.com/mna/pigeon" title="Command pigeon generates parsers in Go from a PEG grammar">pigeon</a> génère un analyseur pour Go. Une grammaire est un ensemble de règles.
Chaque règle est un identifiant, avec optionnellement une étiquette utilisée pour les messages
d’erreur, une expression et une action en Go à exécuter. Vous pouvez trouver la
grammaire complète dans <a href="https://github.com/akvorado/akvorado/blob/main/console/filter/parser.peg"><code>parser.peg</code></a>. Voici une règle simplifiée :</p>
<div class="language-pigeon codehilite"><pre><span/>ConditionIPExpr <span class="s2">"condition on IP"</span> <span class="o">←</span>
<span class="nv">column</span>:<span class="p">(</span><span class="s2">"ExporterAddress"</span><span class="o">i</span> <span class="p">{</span><span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="s">"ExporterAddress"</span><span class="p">,</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">}</span>
/ <span class="s2">"SrcAddr"</span><span class="o">i</span> <span class="p">{</span><span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="s">"SrcAddr"</span><span class="p">,</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">}</span>
/ <span class="s2">"DstAddr"</span><span class="o">i</span> <span class="p">{</span><span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="s">"DstAddr"</span><span class="p">,</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">})</span> _
<span class="nv">operator</span>:<span class="p">(</span><span class="s2">"="</span> / <span class="s2">"!="</span><span class="p">)</span> _
<span class="nv">ip</span>:IP <span class="p">{</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nx">fmt</span><span class="p">.</span><span class="nx">Sprintf</span><span class="p">(</span><span class="s">"%s %s IPv6StringToNum(%s)"</span><span class="p">,</span>
<span class="w"> </span><span class="nx">toString</span><span class="p">(</span><span class="nx">column</span><span class="p">),</span><span class="w"> </span><span class="nx">toString</span><span class="p">(</span><span class="nx">operator</span><span class="p">),</span><span class="w"> </span><span class="nx">quote</span><span class="p">(</span><span class="nx">ip</span><span class="p">)),</span><span class="w"> </span><span class="kc">nil</span>
<span class="w"> </span><span class="p">}</span>
</pre></div>
<p>L’identifiant de la règle est <code>ConditionIPExpr</code>. Elle attend soit
<code>ExporterAddress</code>, soit <code>SrcAddr</code>, soit <code>DstAddr</code>, sans distinction de la case.
L’action pour chaque cas renvoie le nom de la colonne correspondante. C’est ce
qui est stocké dans la variable <code>column</code>. Ensuite, elle attend un des deux
opérateurs possibles. Comme il n’y a pas de bloc de code, l’opérateur est stocké
dans la variable <code>operator</code>. Ensuite, elle attend une chaîne validée par la
règle <code>IP</code> qui est définie ailleurs dans la grammaire. Si c’est le cas, elle
stocke le résultat dans la variable <code>ip</code> et exécute l’action finale. L’action
transforme la colonne, l’opérateur et l’adresse IP en une expression SQL pour
<em>ClickHouse</em>. Par exemple, si nous avons <code>ExporterAddress = 203.0.113.15</code>, nous
obtenons <code>ExporterAddress = IPv6StringToNum('203.0.113.15')</code>.</p>
<p>La règle <code>IP</code> utilise une expression régulière rudimentaire mais vérifie si
l’adresse correspondante est correcte dans le bloc d’action, grâce à
<code>netip.ParseAddr()</code>:</p>
<div class="language-pigeon codehilite"><pre><span/>IP <span class="s2">"IP address"</span> <span class="o">←</span> <span class="p">[</span><span class="s">0-9A-Fa-f:.</span><span class="p">]</span>+ <span class="p">{</span>
<span class="w"> </span><span class="nx">ip</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">netip</span><span class="p">.</span><span class="nx">ParseAddr</span><span class="p">(</span><span class="nb">string</span><span class="p">(</span><span class="nx">c</span><span class="p">.</span><span class="nx">text</span><span class="p">))</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="s">""</span><span class="p">,</span><span class="w"> </span><span class="nx">errors</span><span class="p">.</span><span class="nx">New</span><span class="p">(</span><span class="s">"expecting an IP address"</span><span class="p">)</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nx">ip</span><span class="p">.</span><span class="nx">String</span><span class="p">(),</span><span class="w"> </span><span class="kc">nil</span>
<span class="p">}</span>
</pre></div>
<p>Cet analyseur transforme de manière sécurisée un filtre en une clause <code>WHERE</code>
acceptée par <em>ClickHouse</em><sup id="fnref-ast"><a class="footnote-ref" href="#fn-ast">3</a></sup> :</p>
<div class="language-sql codehilite"><pre><span/><span class="k">WHERE</span><span class="w"> </span><span class="n">InIfBoundary</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'external'</span><span class="w"> </span>
<span class="k">AND</span><span class="w"> </span><span class="n">ExporterRegion</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'france'</span><span class="w"> </span>
<span class="k">AND</span><span class="w"> </span><span class="n">InIfConnectivity</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'transit'</span><span class="w"> </span>
<span class="k">AND</span><span class="w"> </span><span class="n">SrcAS</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">15169</span><span class="w"> </span>
<span class="k">AND</span><span class="w"> </span><span class="n">DstAddr</span><span class="w"> </span><span class="k">BETWEEN</span><span class="w"> </span><span class="n">toIPv6</span><span class="p">(</span><span class="s1">'2a01:e0f:ffff::'</span><span class="p">)</span><span class="w"> </span>
<span class="w"> </span><span class="k">AND</span><span class="w"> </span><span class="n">toIPv6</span><span class="p">(</span><span class="s1">'2a01:e0f:ffff:ffff:ffff:ffff:ffff:ffff'</span><span class="p">)</span>
</pre></div>
<h1 id="integration-dans-codemirror">Intégration dans CodeMirror<a class="headerlink" href="#integration-dans-codemirror" title="Permanent link"></a></h1>
<p><a href="https://codemirror.net/" title="CodeMirror: Extensible Code Editor">CodeMirror</a> est un éditeur de code polyvalent qui peut être facilement
intégré dans les projets JavaScript. Dans <em>Akvorado</em>, le composant
<a href="https://vuejs.org/" title="Vue.js: the Progressive JavaScript Framework">Vue.js</a> <a href="https://github.com/akvorado/akvorado/blob/main/console/frontend/src/components/InputFilter.vue"><code>InputFilter</code></a> utilise <em>CodeMirror</em> et tire
parti de fonctionnalités telles que la coloration syntaxique, la vérification
syntaxique et la complétion. Le code source de ces fonctionnalités se trouve
dans le répertoire <a href="https://github.com/akvorado/akvorado/tree/main/console/frontend/src/codemirror/lang-filter"><code>codemirror/lang-filter/</code></a>.</p>
<h2 id="coloration-syntaxique">Coloration syntaxique<a class="headerlink" href="#coloration-syntaxique" title="Permanent link"></a></h2>
<p>La grammaire <abbr title="Parsing Expression Grammar">PEG</abbr> pour Go ne peut pas être utilisé directement<sup id="fnref-pegjs"><a class="footnote-ref" href="#fn-pegjs">4</a></sup> et les
exigences pour les analyseurs syntaxiques utilisés dans les éditeurs sont
différentes : ils doivent être tolérants aux erreurs et fonctionner de manière
incrémentielle, car le code est généralement mis à jour caractère par caractère.
<em>CodeMirror</em> propose une solution via son propre générateur d’analyseur, <a href="https://lezer.codemirror.net/" title="The Lezer Parser System">Lezer</a>.</p>
<p>Nous n’avons pas besoin que cet analyseur supplémentaire comprenne pleinement le
langage des filtres. Seule la structure est nécessaire : les noms de colonnes, les
opérateurs de comparaison et de logique, les valeurs entre guillemets ou non. La
<a href="https://github.com/akvorado/akvorado/blob/main/console/frontend/src/codemirror/lang-filter/syntax.grammar">grammaire</a> est donc assez courte et n’a pas besoin d’être mise à jour
souvent :</p>
<div class="language-lezer codehilite"><pre><span/><span class="kt">@top</span> <span class="ss">Filter</span> <span class="p">{</span>
expression
<span class="p">}</span>
<span class="ss">expression</span> <span class="p">{</span>
Not expression |
<span class="s2">"("</span> expression <span class="s2">")"</span> |
<span class="s2">"("</span> expression <span class="s2">")"</span> And expression |
<span class="s2">"("</span> expression <span class="s2">")"</span> Or expression |
comparisonExpression And expression |
comparisonExpression Or expression |
comparisonExpression
<span class="p">}</span>
<span class="ss">comparisonExpression</span> <span class="p">{</span>
Column Operator Value
<span class="p">}</span>
<span class="ss">Value</span> <span class="p">{</span>
String | Literal | ValueLParen ListOfValues ValueRParen
<span class="p">}</span>
<span class="ss">ListOfValues</span> <span class="p">{</span>
ListOfValues ValueComma <span class="p">(</span>String | Literal<span class="p">)</span> |
String | Literal
<span class="p">}</span>
<span class="c1">// […]</span>
<span class="kt">@tokens</span> <span class="p">{</span>
<span class="c1">// […]</span>
<span class="ss">Column</span> <span class="p">{</span> std.asciiLetter <span class="p">(</span>std.asciiLetter|std.digit<span class="p">)</span>* <span class="p">}</span>
<span class="ss">Operator</span> <span class="p">{</span> $<span class="p">[</span><span class="s">a-zA-Z!=><</span><span class="p">]</span>+ <span class="p">}</span>
<span class="ss">String</span> <span class="p">{</span>
<span class="s1">'"'</span> <span class="p">(</span>!<span class="p">[</span><span class="s">\\\n"</span><span class="p">]</span> | <span class="s2">"\\"</span> _<span class="p">)</span>* <span class="s1">'"'</span>? |
<span class="s2">"'"</span> <span class="p">(</span>!<span class="p">[</span><span class="s">\\\n'</span><span class="p">]</span> | <span class="s2">"\\"</span> _<span class="p">)</span>* <span class="s2">"'"</span>?
<span class="p">}</span>
<span class="ss">Literal</span> <span class="p">{</span> <span class="p">(</span>std.digit | std.asciiLetter | $<span class="p">[</span><span class="s">.:/</span><span class="p">])</span>+ <span class="p">}</span>
<span class="c1">// […]</span>
<span class="p">}</span>
</pre></div>
<p>L’expression <code>SrcAS = 12322 AND (DstAS = 1299 OR SrcAS = 29447)</code> est analysée ainsi :</p>
<div class="language-text-only codehilite"><pre><span/>Filter(Column, Operator, Value(Literal),
And, Column, Operator, Value(Literal),
Or, Column, Operator, Value(Literal))
</pre></div>
<p>La dernière étape est d’indiquer à <em>CodeMirror</em> la correspondance entre chaque
symbole et sa catégorie pour la coloration syntaxique :</p>
<div class="language-typescript codehilite"><pre><span/><span class="k">export</span><span class="w"> </span><span class="kd">const</span><span class="w"> </span><span class="nx">FilterLanguage</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">LRLanguage</span><span class="p">.</span><span class="nx">define</span><span class="p">({</span>
<span class="w"> </span><span class="nx">parser</span><span class="o">:</span><span class="w"> </span><span class="kt">parser.configure</span><span class="p">({</span>
<span class="w"> </span><span class="nx">props</span><span class="o">:</span><span class="w"> </span><span class="p">[</span>
<span class="w"> </span><span class="nx">styleTags</span><span class="p">({</span>
<span class="w"> </span><span class="nx">Column</span><span class="o">:</span><span class="w"> </span><span class="kt">t.propertyName</span><span class="p">,</span>
<span class="w"> </span><span class="nx">String</span><span class="o">:</span><span class="w"> </span><span class="kt">t.string</span><span class="p">,</span>
<span class="w"> </span><span class="nx">Literal</span><span class="o">:</span><span class="w"> </span><span class="kt">t.literal</span><span class="p">,</span>
<span class="w"> </span><span class="nx">LineComment</span><span class="o">:</span><span class="w"> </span><span class="kt">t.lineComment</span><span class="p">,</span>
<span class="w"> </span><span class="nx">BlockComment</span><span class="o">:</span><span class="w"> </span><span class="kt">t.blockComment</span><span class="p">,</span>
<span class="w"> </span><span class="nx">Or</span><span class="o">:</span><span class="w"> </span><span class="kt">t.logicOperator</span><span class="p">,</span>
<span class="w"> </span><span class="nx">And</span><span class="o">:</span><span class="w"> </span><span class="kt">t.logicOperator</span><span class="p">,</span>
<span class="w"> </span><span class="nx">Not</span><span class="o">:</span><span class="w"> </span><span class="kt">t.logicOperator</span><span class="p">,</span>
<span class="w"> </span><span class="nx">Operator</span><span class="o">:</span><span class="w"> </span><span class="kt">t.compareOperator</span><span class="p">,</span>
<span class="w"> </span><span class="s2">"( )"</span><span class="o">:</span><span class="w"> </span><span class="nx">t</span><span class="p">.</span><span class="nx">paren</span><span class="p">,</span>
<span class="w"> </span><span class="p">}),</span>
<span class="w"> </span><span class="p">],</span>
<span class="w"> </span><span class="p">}),</span>
<span class="p">});</span>
</pre></div>
<h2 id="verification-syntaxique">Vérification syntaxique<a class="headerlink" href="#verification-syntaxique" title="Permanent link"></a></h2>
<p>La vérification syntaxique est déléguée à l’analyseur syntaxique en Go. Le point
d’accès <code>/api/v0/console/filter/validate</code> accepte un filtre et retourne une
structure JSON avec les éventuelles erreurs:</p>
<div class="language-json codehilite"><pre><span/><span class="p">{</span>
<span class="w"> </span><span class="nt">"message"</span><span class="p">:</span><span class="w"> </span><span class="s2">"at line 1, position 12: string literal not terminated"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"errors"</span><span class="p">:</span><span class="w"> </span><span class="p">[{</span>
<span class="w"> </span><span class="nt">"line"</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"column"</span><span class="p">:</span><span class="w"> </span><span class="mi">12</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"offset"</span><span class="p">:</span><span class="w"> </span><span class="mi">11</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"message"</span><span class="p">:</span><span class="w"> </span><span class="s2">"string literal not terminated"</span><span class="p">,</span>
<span class="w"> </span><span class="p">}]</span>
<span class="p">}</span>
</pre></div>
<p>Le <a href="https://github.com/akvorado/akvorado/blob/main/console/frontend/src/codemirror/lang-filter/linter.ts">greffon</a> pour <em>CodeMirror</em> interroge cette API et transforme
chaque erreur en un <a href="https://codemirror.net/docs/ref/#lint.Diagnostic">diagnostic</a>.</p>
<h2 id="completion">Complétion<a class="headerlink" href="#completion" title="Permanent link"></a></h2>
<p>Le système de complétion adopte une approche hybride. Il répartit le travail
entre le frontend et le backend pour offrir ses suggestions.</p>
<p>Le <a href="https://github.com/akvorado/akvorado/blob/main/console/frontend/src/codemirror/lang-filter/complete.ts">frontend</a> utilise l’analyseur construit avec <em>Lezer</em> pour déterminer le
contexte de la complétion : s’agit-il compléter un nom de colonne, un opérateur
ou une valeur ? Il extrait également le nom de la colonne si nous complétons
autre chose. Il transfère le résultat au backend via le point de terminaison
<code>/api/v0/console/filter/complete</code>. Parcourir l’arbre syntaxique n’a pas été
aussi facile que je le pensais, mais les <a href="https://github.com/akvorado/akvorado/blob/main/console/frontend/src/codemirror/lang-filter/complete.test.ts">tests unitaires</a> ont
beaucoup aidé.</p>
<p>Le <a href="https://github.com/akvorado/akvorado/blob/9eee46caded6fa6cc2dfb90d1941b2ef05d15e9f/console/filter.go#L76">backend</a> utilise l’analyseur généré par <em>pigeon</em> pour compléter les noms
de colonnes et les opérateurs de comparaison. Pour les valeurs, les complétions
sont statiques out extraites de la base de données <em>ClickHouse</em>. Un utilisateur
peut compléter un numéro <abbr title="Autonomous System">AS</abbr> à partir d’un nom d’organisation grâce au code
suivant :</p>
<div class="language-go codehilite"><pre><span/><span class="nx">results</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="p">[]</span><span class="kd">struct</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nx">Label</span><span class="w"> </span><span class="kt">string</span><span class="w"> </span><span class="s">`ch:"label"`</span>
<span class="w"> </span><span class="nx">Detail</span><span class="w"> </span><span class="kt">string</span><span class="w"> </span><span class="s">`ch:"detail"`</span>
<span class="p">}{}</span>
<span class="nx">columnName</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="s">"DstAS"</span>
<span class="nx">sqlQuery</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">fmt</span><span class="p">.</span><span class="nx">Sprintf</span><span class="p">(</span><span class="s">`</span>
<span class="s"> SELECT concat('AS', toString(%s)) AS label, dictGet('asns', 'name', %s) AS detail</span>
<span class="s"> FROM flows</span>
<span class="s"> WHERE TimeReceived > date_sub(minute, 1, now())</span>
<span class="s"> AND detail != ''</span>
<span class="s"> AND positionCaseInsensitive(detail, $1) >= 1</span>
<span class="s"> GROUP BY label, detail</span>
<span class="s"> ORDER BY COUNT(*) DESC</span>
<span class="s"> LIMIT 20</span>
<span class="s">`</span><span class="p">,</span><span class="w"> </span><span class="nx">columnName</span><span class="p">,</span><span class="w"> </span><span class="nx">columnName</span><span class="p">)</span>
<span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">conn</span><span class="p">.</span><span class="nx">Select</span><span class="p">(</span><span class="nx">ctx</span><span class="p">,</span><span class="w"> </span><span class="o">&</span><span class="nx">results</span><span class="p">,</span><span class="w"> </span><span class="nx">sqlQuery</span><span class="p">,</span><span class="w"> </span><span class="nx">input</span><span class="p">.</span><span class="nx">Prefix</span><span class="p">);</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nx">c</span><span class="p">.</span><span class="nx">r</span><span class="p">.</span><span class="nx">Err</span><span class="p">(</span><span class="nx">err</span><span class="p">).</span><span class="nx">Msg</span><span class="p">(</span><span class="s">"unable to query database"</span><span class="p">)</span>
<span class="w"> </span><span class="k">break</span>
<span class="p">}</span>
<span class="k">for</span><span class="w"> </span><span class="nx">_</span><span class="p">,</span><span class="w"> </span><span class="nx">result</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="k">range</span><span class="w"> </span><span class="nx">results</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nx">completions</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nb">append</span><span class="p">(</span><span class="nx">completions</span><span class="p">,</span><span class="w"> </span><span class="nx">filterCompletion</span><span class="p">{</span>
<span class="w"> </span><span class="nx">Label</span><span class="p">:</span><span class="w"> </span><span class="nx">result</span><span class="p">.</span><span class="nx">Label</span><span class="p">,</span>
<span class="w"> </span><span class="nx">Detail</span><span class="p">:</span><span class="w"> </span><span class="nx">result</span><span class="p">.</span><span class="nx">Detail</span><span class="p">,</span>
<span class="w"> </span><span class="nx">Quoted</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span>
<span class="w"> </span><span class="p">})</span>
<span class="p">}</span>
</pre></div>
<p>À mon avis, ce système de complétion est un élément important qui fait de
l’éditeur un moyen efficace de sélectionner des flux. Alors qu’un constructeur
de requêtes aurait pu être plus convivial pour les débutants, la facilité
d’utilisation et les fonctionnalités du système de complétion le rendent plus
agréable à utiliser une fois que l’utilisateur familiarisé.</p>
<div class="footnote">
<hr/>
<ol>
<li id="fn-lazy">
<p>De plus, créer un constructeur de requêtes me semblait une tâche assez
rébarbative. <a class="footnote-backref" href="#fnref-lazy" title="Jump back to footnote 1 in the text">↩︎</a></p>
</li>
<li id="fn-recent">
<p>Elles ont été introduites en 2004 dans « <a href="https://bford.info/pub/lang/peg.pdf">Parsing Expression
Grammars: A Recognition-Based Syntactic Foundation</a> ». Les
analyseurs <abbr title="left-to-right">LR</abbr> ont été introduits en <a href="https://web.archive.org/web/20120315152151/http://www.cs.dartmouth.edu/~mckeeman/cs48/mxcom/doc/knuth65.pdf" title="On the Translation of Languages from Left to Right">1965</a>, les analyseurs <abbr title="lookahead left-to-right">LALR</abbr>
en 1969 et les analyseurs <abbr title="left-to-right, leftmost derivation">LL</abbr> dans les années 1970. <a href="https://fr.wikipedia.org/wiki/Yacc_(logiciel)" title="Article sur Yacc sur Wikipedia">Yacc</a>, un générateur
d’analyseurs populaire, a été écrit en 1975. <a class="footnote-backref" href="#fnref-recent" title="Jump back to footnote 2 in the text">↩︎</a></p>
</li>
<li id="fn-ast">
<p>L’analyseur retourne une chaîne. Il ne génère pas un arbre syntaxique
intermédiaire. Cela le rend plus simple et suffit à nos besoins. <a class="footnote-backref" href="#fnref-ast" title="Jump back to footnote 3 in the text">↩︎</a></p>
</li>
<li id="fn-pegjs">
<p>Elle pourrait être manuellement traduite en JavaScript avec <a href="https://peggyjs.org/" title="Peggy: Parser Generator for JavaScript">Peggy</a>. <a class="footnote-backref" href="#fnref-pegjs" title="Jump back to footnote 4 in the text">↩︎</a></p>
</li>
</ol>
</div>
Fiabiliser la plaque de déclenchement Geberit Sigma 70Vincent Bernat2023-02-11T21:22:56Zhttp://www.luffy.cx/fr/blog/2023-geberit-sigma-70.html
<p>Mes WC sont équipés d’une chasse d’eau avec plaque de déclenchement <a href="https://catalog.geberit.fr/fr-FR/Plaque-de-d%C3%A9clenchement-Geberit-Sigma70/PRO_2693806.html">Geberit
Sigma 70</a>. Le discours commercial autour de ce dispositif à <a href="https://youtu.be/yqN2B0qxxFM?t=38">asservissement
hydraulique</a> vante le « mécanisme révolutionnaire qui n’utilise pas
d’électronique embarquée ». En pratique, le déclenchement est très capricieux et
a un taux d’échec particulièrement élevé. <strong>Évitez ce type de mécanisme !</strong>
Préférez lui une version intégralement mécanique comme le <a href="https://catalog.geberit.fr/fr-FR/Plaque-de-d%C3%A9clenchement-Geberit-Sigma20-pour-rin%C3%A7age-double-touche/PRO_219937.html">Geberit Sigma 20</a>.</p>
<p>Après le passage de plusieurs plombiers, des échanges avec le service technique
de Geberit et le remplacement à grands frais de l’intégralité du mécanisme,
j’observais toujours un taux d’échec de plus de 50% pour la petite chasse. J’ai
finalement réussi à réduire ce taux à 5% en appliquant <a href="https://www.amazon.fr/s?k=patins+silicones+transparents+rond+8mm">deux patins ronds en
silicone transparent de 8 mm de diamètre</a> sur le dos de la plaque de
déclenchement. Leurs emplacements sont indiqués par des ronds rouges sur la
photo ci-dessous :</p>
<figure><div class="lf-media-outer"><span class="lf-media-inner"><img alt="Plaque de déclenchement Geberit Sigma 70. En haut le mécanisme posé en applique et convertissant l'appui mécanique en une impulsion hydraulique. En bas, le dos de la plaque de déclenchement avec les deux emplacements où appliquer les patins." src="https://d2pzklc15kok91.cloudfront.net/images/geberit-sigma-70.365084ed1e16c2.jpg" width="700" height="905" class="lf-media lf-opaque" style="background-image: url(data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAIwAAAC1AQMAAAC3e4UYAAAABlBMVEXTx7YAAABS4fLDAAAAGklEQVR42u3BMQEAAADCoPVPbQo/oAAAADgYDW8AAcJLTF4AAAAASUVORK5CYII=)"/></span></div><figcaption>Mécanisme Geberit Sigma 70. En haut: le mécanisme posé en applique. En bas, le dos de la plaque de déclenchement en verre. En rouge, les deux emplacements où appliquer les patins.</figcaption></figure>
<p>Comptez environ 5 € et autant de minutes pour cette opération.</p>
Encodage rapide et dynamique des Protocol Buffers en GoVincent Bernat2023-02-06T08:58:12Zhttp://www.luffy.cx/fr/blog/2023-protobuf-dynamique-golang.html
<p>Les <a href="https://developers.google.com/protocol-buffers/" title="Protocol Buffers">Protocol Buffers</a> sont un choix populaire pour la <strong>sérialisation de données
structurées</strong> en raison de leur taille compacte, de leur rapidité de traitement,
de leur indépendance du langage cible et de leur compatibilité. D’autres
alternatives existent, notamment <a href="https://capnproto.org/" title="Cap'n Proto">Cap’n Proto</a>, <a href="https://cbor.io/" title="RFC 8949 Concise Binary Object Representation">CBOR</a> et <a href="https://avro.apache.org/" title="Apache Avro">Avro</a>.</p>
<p>Les structures de données sont habituellement décrites dans un <strong>fichier de
définition de protocole</strong> (<code>.proto</code>). Le compilateur <code>protoc</code> et un greffon
spécifique à un langage les convertissent en code:</p>
<div class="language-bash-session codehilite"><pre><span/><span class="gp">$ </span>head<span class="w"> </span>flow-4.proto
<span class="go">syntax = "proto3";</span>
<span class="go">package decoder;</span>
<span class="go">option go_package = "akvorado/inlet/flow/decoder";</span>
<span class="go">message FlowMessagev4 {</span>
<span class="go"> uint64 TimeReceived = 2;</span>
<span class="go"> uint32 SequenceNum = 3;</span>
<span class="go"> uint64 SamplingRate = 4;</span>
<span class="go"> uint32 FlowDirection = 5;</span>
<span class="gp">$ </span>protoc<span class="w"> </span>-I<span class="o">=</span>.<span class="w"> </span>--plugin<span class="o">=</span>protoc-gen-go<span class="w"> </span>--go_out<span class="o">=</span><span class="nv">module</span><span class="o">=</span>akvorado:.<span class="w"> </span>flow-4.proto
<span class="gp">$ </span>head<span class="w"> </span>inlet/flow/decoder/flow-4.pb.go
<span class="go">// Code generated by protoc-gen-go. DO NOT EDIT.</span>
<span class="go">// versions:</span>
<span class="go">// protoc-gen-go v1.28.0</span>
<span class="go">// protoc v3.21.12</span>
<span class="go">// source: inlet/flow/data/schemas/flow-4.proto</span>
<span class="go">package decoder</span>
<span class="go">import (</span>
<span class="go"> protoreflect "google.golang.org/protobuf/reflect/protoreflect"</span>
</pre></div>
<p><a href="/fr/blog/2022-akvorado-collecteur-flux" title="Akvorado : collecteur et visualisateur de flux réseau">Akvorado</a> collecte les flux réseau à l’aide de <a href="https://www.rfc-editor.org/rfc/rfc7011" title="RFC 7011: Specification of the IP Flow Information Export (IPFIX) Protocol for the Exchange of Flow Information">IPFIX</a> ou <a href="https://www.rfc-editor.org/rfc/rfc3176" title="RFC 3176: Specification of the IP Flow Information Export (IPFIX) Protocol for the Exchange of Flow Information">sFlow</a>, les
décode à l’aide de <a href="https://github.com/NetSampler/GoFlow2">GoFlow2</a>, les encode en <em>Protocol Buffers</em> et les envoie à
<a href="https://kafka.apache.org/" title="Apache Kafka">Kafka</a> pour les stocker dans une base de données <a href="https://clickhouse.com/" title="ClickHouse: OLAP DBMS">ClickHouse</a>. La collecte
d’un nouveau champ, comme les adresses MAC source et destination, nécessite des
modifications à plusieurs endroits, y compris le fichier de définition de
protocole et le code de migration pour ClickHouse. De plus, le coût est supporté
par tous les utilisateurs<sup id="fnref-cost"><a class="footnote-ref" href="#fn-cost">1</a></sup>. Il serait agréable d’avoir un <strong>schéma commun
à l’application</strong> et de permettre aux utilisateurs d’activer ou de désactiver
les champs dont ils ont besoin.</p>
<p>Bien que le principal objectif est la flexibilité, nous ne voulons pas sacrifier
les performances. Sur ce front, c’est un véritable succès: lors de la mise à
niveau de 1.6.4 à 1.7.1, les performances de décodage et d’encodage ont presque
doublé ! 🤗</p>
<div class="language-text-only codehilite"><pre><span/>goos: linux
goarch: amd64
pkg: akvorado/inlet/flow
cpu: AMD Ryzen 5 5600X 6-Core Processor
│ initial.txt │ final.txt │
│ sec/op │ sec/op vs base │
Netflow/with_encoding-12 12.963µ ± 2% 7.836µ ± 1% -39.55% (p=0.000 n=10)
Sflow/with_encoding-12 19.37µ ± 1% 10.15µ ± 2% -47.63% (p=0.000 n=10)
</pre></div>
<h1 id="encodage-plus-rapide-des-protocol-buffers">Encodage plus rapide des Protocol Buffers<a class="headerlink" href="#encodage-plus-rapide-des-protocol-buffers" title="Permanent link"></a></h1>
<p>J’utilise le <a href="https://github.com/akvorado/akvorado/blob/protobuf-bench-initial/inlet/flow/decoder_test.go">code suivant</a> pour mesurer les performances du
processus de décodage et d’encodage. Initialement, la méthode <code>Decode()</code> est une
simple façade au-dessus du producteur <em>GoFlow2</em>. Elle stocke les données
décodées dans la structure en mémoire générée par <code>protoc</code>. Par la suite,
certaines données seront encodées directement pendant le décodage des flux.
C’est pourquoi nous mesurons à la fois le décodage et l’encodage<sup id="fnref-netflow"><a class="footnote-ref" href="#fn-netflow">2</a></sup>.</p>
<div class="language-go codehilite"><pre><span/><span class="kd">func</span><span class="w"> </span><span class="nx">BenchmarkDecodeEncodeSflow</span><span class="p">(</span><span class="nx">b</span><span class="w"> </span><span class="o">*</span><span class="nx">testing</span><span class="p">.</span><span class="nx">B</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nx">r</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">reporter</span><span class="p">.</span><span class="nx">NewMock</span><span class="p">(</span><span class="nx">b</span><span class="p">)</span>
<span class="w"> </span><span class="nx">sdecoder</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">sflow</span><span class="p">.</span><span class="nx">New</span><span class="p">(</span><span class="nx">r</span><span class="p">)</span>
<span class="w"> </span><span class="nx">data</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">helpers</span><span class="p">.</span><span class="nx">ReadPcapPayload</span><span class="p">(</span><span class="nx">b</span><span class="p">,</span>
<span class="w"> </span><span class="nx">filepath</span><span class="p">.</span><span class="nx">Join</span><span class="p">(</span><span class="s">"decoder"</span><span class="p">,</span><span class="w"> </span><span class="s">"sflow"</span><span class="p">,</span><span class="w"> </span><span class="s">"testdata"</span><span class="p">,</span><span class="w"> </span><span class="s">"data-1140.pcap"</span><span class="p">))</span>
<span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="nx">_</span><span class="p">,</span><span class="w"> </span><span class="nx">withEncoding</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="k">range</span><span class="w"> </span><span class="p">[]</span><span class="kt">bool</span><span class="p">{</span><span class="kc">true</span><span class="p">,</span><span class="w"> </span><span class="kc">false</span><span class="p">}</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nx">title</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="kd">map</span><span class="p">[</span><span class="kt">bool</span><span class="p">]</span><span class="kt">string</span><span class="p">{</span>
<span class="w"> </span><span class="kc">true</span><span class="p">:</span><span class="w"> </span><span class="s">"with encoding"</span><span class="p">,</span>
<span class="w"> </span><span class="kc">false</span><span class="p">:</span><span class="w"> </span><span class="s">"without encoding"</span><span class="p">,</span>
<span class="w"> </span><span class="p">}[</span><span class="nx">withEncoding</span><span class="p">]</span>
<span class="w"> </span><span class="kd">var</span><span class="w"> </span><span class="nx">got</span><span class="w"> </span><span class="p">[]</span><span class="o">*</span><span class="nx">decoder</span><span class="p">.</span><span class="nx">FlowMessage</span>
<span class="w"> </span><span class="nx">b</span><span class="p">.</span><span class="nx">Run</span><span class="p">(</span><span class="nx">title</span><span class="p">,</span><span class="w"> </span><span class="kd">func</span><span class="p">(</span><span class="nx">b</span><span class="w"> </span><span class="o">*</span><span class="nx">testing</span><span class="p">.</span><span class="nx">B</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="nx">i</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="mi">0</span><span class="p">;</span><span class="w"> </span><span class="nx">i</span><span class="w"> </span><span class="p"><</span><span class="w"> </span><span class="nx">b</span><span class="p">.</span><span class="nx">N</span><span class="p">;</span><span class="w"> </span><span class="nx">i</span><span class="o">++</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nx">got</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">sdecoder</span><span class="p">.</span><span class="nx">Decode</span><span class="p">(</span><span class="nx">decoder</span><span class="p">.</span><span class="nx">RawFlow</span><span class="p">{</span>
<span class="w"> </span><span class="nx">Payload</span><span class="p">:</span><span class="w"> </span><span class="nx">data</span><span class="p">,</span>
<span class="w"> </span><span class="nx">Source</span><span class="p">:</span><span class="w"> </span><span class="nx">net</span><span class="p">.</span><span class="nx">ParseIP</span><span class="p">(</span><span class="s">"127.0.0.1"</span><span class="p">),</span>
<span class="w"> </span><span class="p">})</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">withEncoding</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="nx">_</span><span class="p">,</span><span class="w"> </span><span class="nx">flow</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="k">range</span><span class="w"> </span><span class="nx">got</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nx">buf</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="p">[]</span><span class="kt">byte</span><span class="p">{}</span>
<span class="w"> </span><span class="nx">buf</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">protowire</span><span class="p">.</span><span class="nx">AppendVarint</span><span class="p">(</span><span class="nx">buf</span><span class="p">,</span><span class="w"> </span><span class="nb">uint64</span><span class="p">(</span><span class="nx">proto</span><span class="p">.</span><span class="nx">Size</span><span class="p">(</span><span class="nx">flow</span><span class="p">)))</span>
<span class="w"> </span><span class="nx">proto</span><span class="p">.</span><span class="nx">MarshalOptions</span><span class="p">{}.</span><span class="nx">MarshalAppend</span><span class="p">(</span><span class="nx">buf</span><span class="p">,</span><span class="w"> </span><span class="nx">flow</span><span class="p">)</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">})</span>
<span class="w"> </span><span class="p">}</span>
<span class="p">}</span>
</pre></div>
<p>L’implémentation Go de référence pour les <em>Protocol Buffers</em>,
<a href="https://pkg.go.dev/google.golang.org/protobuf" title="Go support for Protocol Buffers"><code>google.golang.org/protobuf</code></a> n’est pas la plus
efficace. Pendant longtemps, l’alternative la plus courante était
<a href="https://github.com/gogo/protobuf" title="Protocol Buffers for Go with Gadgets">gogoprotobuf</a>. Cependant, le projet est maintenant <a href="https://github.com/gogo/protobuf/issues/691" title="#691: GoGo Protobuf looking for new ownership">obsolète</a>.
<a href="https://github.com/planetscale/vtprotobuf" title="Protocol Buffers compiler that generates optimized marshaling & unmarshaling Go code">vtprotobuf</a> est un bon remplacement<sup id="fnref-vtprotobuf"><a class="footnote-ref" href="#fn-vtprotobuf">3</a></sup>.</p>
<div class="language-text-only codehilite"><pre><span/>goos: linux
goarch: amd64
pkg: akvorado/inlet/flow
cpu: AMD Ryzen 5 5600X 6-Core Processor
│ initial.txt │ bench-2.txt │
│ sec/op │ sec/op vs base │
Netflow/with_encoding-12 12.96µ ± 2% 10.28µ ± 2% -20.67% (p=0.000 n=10)
Netflow/without_encoding-12 8.935µ ± 2% 8.975µ ± 2% ~ (p=0.143 n=10)
Sflow/with_encoding-12 19.37µ ± 1% 16.67µ ± 2% -13.93% (p=0.000 n=10)
Sflow/without_encoding-12 14.62µ ± 3% 14.87µ ± 1% +1.66% (p=0.007 n=10)
</pre></div>
<h1 id="encodage-dynamique-des-protocol-buffers">Encodage dynamique des Protocol Buffers<a class="headerlink" href="#encodage-dynamique-des-protocol-buffers" title="Permanent link"></a></h1>
<p>Nous avons désormais une bonne référence départ. Voyons comment encoder nos
<em>Protocol Buffers</em> sans un fichier <code>.proto</code>. Le format utilisé est relativement
simple et repose beaucoup sur les entiers à longueur variable.</p>
<p>Les entiers à longueur variable sont un moyen efficace d’encoder des entiers non
signés en utilisant un nombre variable d’octets, de un à dix, les petites
valeurs utilisant moins d’octets. Ils fonctionnent en scindant les entiers par
groupe de 7 bits et en utilisant le 8<sup>ème</sup> bit comme signal de
continuation : il est mis à 1 pour tous les groupes sauf le dernier.</p>
<figure><div class="lf-media-outer"><span class="lf-media-inner"><img alt="Encodage des entiers à longueur variable dans les Protocol Buffers : conversion de 150" src="https://d2pzklc15kok91.cloudfront.net/images/protobuf-varint.8f8b7e23fe94cc.svg" width="272" height="151" class="lf-media"/></span></div><figcaption>Encodage des entiers à longueur variable</figcaption></figure>
<p>Pour notre utilisation, nous avons besoin de deux types seulement : les entiers
à longueur variable et les séquences d’octets. Une séquence d’octets est codée en
la préfixant par sa longueur sous forme d’entier à longueur variable. Lorsqu’un
message est codé, chaque couple clé-valeur est transformé en un enregistrement
composé d’un numéro de champ, d’un type et de la valeur. Le numéro de champ et
le type sont codés en un seul entier de longueur variable appelé « <em>tag</em> ».</p>
<figure><div class="lf-media-outer"><span class="lf-media-inner"><img alt="Message codé avec les Protocol Buffers : 3 entiers et 2 séquences d'octets" src="https://d2pzklc15kok91.cloudfront.net/images/protobuf-message.f05804988087fb.svg" width="512" height="208" class="lf-media"/></span></div><figcaption>Message codé avec les Protocol Buffers</figcaption></figure>
<p>Nous utilisons les fonctions bas niveau suivantes pour construire le message
codé :</p>
<ul>
<li><a href="https://pkg.go.dev/google.golang.org/protobuf@v1.28.1/encoding/protowire#AppendTag"><code>protowire.AppendTag()</code></a> code un « <em>tag</em> »,</li>
<li><a href="https://pkg.go.dev/google.golang.org/protobuf@v1.28.1/encoding/protowire#AppendVarint"><code>protowire.AppendVarint()</code></a> code un entier,</li>
<li><a href="https://pkg.go.dev/google.golang.org/protobuf@v1.28.1/encoding/protowire#AppendBytes"><code>protowire.AppendBytes()</code></a> ajoute des octets sans modification.</li>
</ul>
<p>Notre abstraction pour le schéma contient les informations appropriées pour
coder un message (<code>ProtobufIndex</code>) et pour générer un fichier de définition de
protocole (les champs commençant par <code>Protobuf</code>) :</p>
<div class="language-go codehilite"><pre><span/><span class="kd">type</span><span class="w"> </span><span class="nx">Column</span><span class="w"> </span><span class="kd">struct</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nx">Key</span><span class="w"> </span><span class="nx">ColumnKey</span>
<span class="w"> </span><span class="nx">Name</span><span class="w"> </span><span class="kt">string</span>
<span class="w"> </span><span class="nx">Disabled</span><span class="w"> </span><span class="kt">bool</span>
<span class="w"> </span><span class="c1">// […]</span>
<span class="w"> </span><span class="c1">// For protobuf.</span>
<span class="w"> </span><span class="nx">ProtobufIndex</span><span class="w"> </span><span class="nx">protowire</span><span class="p">.</span><span class="nx">Number</span>
<span class="w"> </span><span class="nx">ProtobufType</span><span class="w"> </span><span class="nx">protoreflect</span><span class="p">.</span><span class="nx">Kind</span><span class="w"> </span><span class="c1">// Uint64Kind, Uint32Kind, …</span>
<span class="w"> </span><span class="nx">ProtobufEnum</span><span class="w"> </span><span class="kd">map</span><span class="p">[</span><span class="kt">int</span><span class="p">]</span><span class="kt">string</span>
<span class="w"> </span><span class="nx">ProtobufEnumName</span><span class="w"> </span><span class="kt">string</span>
<span class="w"> </span><span class="nx">ProtobufRepeated</span><span class="w"> </span><span class="kt">bool</span>
<span class="p">}</span>
</pre></div>
<p>Nous avons quelques <a href="https://github.com/akvorado/akvorado/blob/main/common/schema/protobuf.go">méthodes simples</a> autour des fonctions
<code>protowire</code> pour encoder directement les champs lors du décodage des flux. Ils
sautent les champs désactivés ou ceux déjà encodés mais non répétables. Voici un
extrait du <a href="https://github.com/akvorado/akvorado/blob/main/inlet/flow/decoder/sflow/decode.go">décodeur sFlow</a>:</p>
<div class="language-go codehilite"><pre><span/><span class="nx">sch</span><span class="p">.</span><span class="nx">ProtobufAppendVarint</span><span class="p">(</span><span class="nx">bf</span><span class="p">,</span><span class="w"> </span><span class="nx">schema</span><span class="p">.</span><span class="nx">ColumnBytes</span><span class="p">,</span><span class="w"> </span><span class="nb">uint64</span><span class="p">(</span><span class="nx">recordData</span><span class="p">.</span><span class="nx">Base</span><span class="p">.</span><span class="nx">Length</span><span class="p">))</span>
<span class="nx">sch</span><span class="p">.</span><span class="nx">ProtobufAppendVarint</span><span class="p">(</span><span class="nx">bf</span><span class="p">,</span><span class="w"> </span><span class="nx">schema</span><span class="p">.</span><span class="nx">ColumnProto</span><span class="p">,</span><span class="w"> </span><span class="nb">uint64</span><span class="p">(</span><span class="nx">recordData</span><span class="p">.</span><span class="nx">Base</span><span class="p">.</span><span class="nx">Protocol</span><span class="p">))</span>
<span class="nx">sch</span><span class="p">.</span><span class="nx">ProtobufAppendVarint</span><span class="p">(</span><span class="nx">bf</span><span class="p">,</span><span class="w"> </span><span class="nx">schema</span><span class="p">.</span><span class="nx">ColumnSrcPort</span><span class="p">,</span><span class="w"> </span><span class="nb">uint64</span><span class="p">(</span><span class="nx">recordData</span><span class="p">.</span><span class="nx">Base</span><span class="p">.</span><span class="nx">SrcPort</span><span class="p">))</span>
<span class="nx">sch</span><span class="p">.</span><span class="nx">ProtobufAppendVarint</span><span class="p">(</span><span class="nx">bf</span><span class="p">,</span><span class="w"> </span><span class="nx">schema</span><span class="p">.</span><span class="nx">ColumnDstPort</span><span class="p">,</span><span class="w"> </span><span class="nb">uint64</span><span class="p">(</span><span class="nx">recordData</span><span class="p">.</span><span class="nx">Base</span><span class="p">.</span><span class="nx">DstPort</span><span class="p">))</span>
<span class="nx">sch</span><span class="p">.</span><span class="nx">ProtobufAppendVarint</span><span class="p">(</span><span class="nx">bf</span><span class="p">,</span><span class="w"> </span><span class="nx">schema</span><span class="p">.</span><span class="nx">ColumnEType</span><span class="p">,</span><span class="w"> </span><span class="nx">helpers</span><span class="p">.</span><span class="nx">ETypeIPv4</span><span class="p">)</span>
</pre></div>
<p>Les champs nécessaires dans la suite du traitement, comme les adresses source et
destination, sont stockés non codés dans une structure séparée :</p>
<div class="language-go codehilite"><pre><span/><span class="kd">type</span><span class="w"> </span><span class="nx">FlowMessage</span><span class="w"> </span><span class="kd">struct</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nx">TimeReceived</span><span class="w"> </span><span class="kt">uint64</span>
<span class="w"> </span><span class="nx">SamplingRate</span><span class="w"> </span><span class="kt">uint32</span>
<span class="w"> </span><span class="c1">// For exporter classifier</span>
<span class="w"> </span><span class="nx">ExporterAddress</span><span class="w"> </span><span class="nx">netip</span><span class="p">.</span><span class="nx">Addr</span>
<span class="w"> </span><span class="c1">// For interface classifier</span>
<span class="w"> </span><span class="nx">InIf</span><span class="w"> </span><span class="kt">uint32</span>
<span class="w"> </span><span class="nx">OutIf</span><span class="w"> </span><span class="kt">uint32</span>
<span class="w"> </span><span class="c1">// For geolocation or BMP</span>
<span class="w"> </span><span class="nx">SrcAddr</span><span class="w"> </span><span class="nx">netip</span><span class="p">.</span><span class="nx">Addr</span>
<span class="w"> </span><span class="nx">DstAddr</span><span class="w"> </span><span class="nx">netip</span><span class="p">.</span><span class="nx">Addr</span>
<span class="w"> </span><span class="nx">NextHop</span><span class="w"> </span><span class="nx">netip</span><span class="p">.</span><span class="nx">Addr</span>
<span class="w"> </span><span class="c1">// Core component may override them</span>
<span class="w"> </span><span class="nx">SrcAS</span><span class="w"> </span><span class="kt">uint32</span>
<span class="w"> </span><span class="nx">DstAS</span><span class="w"> </span><span class="kt">uint32</span>
<span class="w"> </span><span class="nx">GotASPath</span><span class="w"> </span><span class="kt">bool</span>
<span class="w"> </span><span class="c1">// protobuf is the protobuf representation for the information not contained above.</span>
<span class="w"> </span><span class="nx">protobuf</span><span class="w"> </span><span class="p">[]</span><span class="kt">byte</span>
<span class="w"> </span><span class="nx">protobufSet</span><span class="w"> </span><span class="nx">bitset</span><span class="p">.</span><span class="nx">BitSet</span>
<span class="p">}</span>
</pre></div>
<p>Le tableau <code>protobuf</code> contient les données encodées. Il est initialisé avec une
capacité de 500 octets pour éviter les redimensionnements pendant l’encodage. Il
y a également quelques octets réservés au début pour pouvoir encoder la taille
totale en tant qu’entier de longueur variable. Lors de la finalisation de
l’encodage, les champs restants sont ajoutés et la longueur du message est
insérée dans l’espace libre au début :</p>
<div class="language-go codehilite"><pre><span/><span class="kd">func</span><span class="w"> </span><span class="p">(</span><span class="nx">schema</span><span class="w"> </span><span class="o">*</span><span class="nx">Schema</span><span class="p">)</span><span class="w"> </span><span class="nx">ProtobufMarshal</span><span class="p">(</span><span class="nx">bf</span><span class="w"> </span><span class="o">*</span><span class="nx">FlowMessage</span><span class="p">)</span><span class="w"> </span><span class="p">[]</span><span class="kt">byte</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nx">schema</span><span class="p">.</span><span class="nx">ProtobufAppendVarint</span><span class="p">(</span><span class="nx">bf</span><span class="p">,</span><span class="w"> </span><span class="nx">ColumnTimeReceived</span><span class="p">,</span><span class="w"> </span><span class="nx">bf</span><span class="p">.</span><span class="nx">TimeReceived</span><span class="p">)</span>
<span class="w"> </span><span class="nx">schema</span><span class="p">.</span><span class="nx">ProtobufAppendVarint</span><span class="p">(</span><span class="nx">bf</span><span class="p">,</span><span class="w"> </span><span class="nx">ColumnSamplingRate</span><span class="p">,</span><span class="w"> </span><span class="nb">uint64</span><span class="p">(</span><span class="nx">bf</span><span class="p">.</span><span class="nx">SamplingRate</span><span class="p">))</span>
<span class="w"> </span><span class="nx">schema</span><span class="p">.</span><span class="nx">ProtobufAppendIP</span><span class="p">(</span><span class="nx">bf</span><span class="p">,</span><span class="w"> </span><span class="nx">ColumnExporterAddress</span><span class="p">,</span><span class="w"> </span><span class="nx">bf</span><span class="p">.</span><span class="nx">ExporterAddress</span><span class="p">)</span>
<span class="w"> </span><span class="nx">schema</span><span class="p">.</span><span class="nx">ProtobufAppendVarint</span><span class="p">(</span><span class="nx">bf</span><span class="p">,</span><span class="w"> </span><span class="nx">ColumnSrcAS</span><span class="p">,</span><span class="w"> </span><span class="nb">uint64</span><span class="p">(</span><span class="nx">bf</span><span class="p">.</span><span class="nx">SrcAS</span><span class="p">))</span>
<span class="w"> </span><span class="nx">schema</span><span class="p">.</span><span class="nx">ProtobufAppendVarint</span><span class="p">(</span><span class="nx">bf</span><span class="p">,</span><span class="w"> </span><span class="nx">ColumnDstAS</span><span class="p">,</span><span class="w"> </span><span class="nb">uint64</span><span class="p">(</span><span class="nx">bf</span><span class="p">.</span><span class="nx">DstAS</span><span class="p">))</span>
<span class="w"> </span><span class="nx">schema</span><span class="p">.</span><span class="nx">ProtobufAppendIP</span><span class="p">(</span><span class="nx">bf</span><span class="p">,</span><span class="w"> </span><span class="nx">ColumnSrcAddr</span><span class="p">,</span><span class="w"> </span><span class="nx">bf</span><span class="p">.</span><span class="nx">SrcAddr</span><span class="p">)</span>
<span class="w"> </span><span class="nx">schema</span><span class="p">.</span><span class="nx">ProtobufAppendIP</span><span class="p">(</span><span class="nx">bf</span><span class="p">,</span><span class="w"> </span><span class="nx">ColumnDstAddr</span><span class="p">,</span><span class="w"> </span><span class="nx">bf</span><span class="p">.</span><span class="nx">DstAddr</span><span class="p">)</span>
<span class="w"> </span><span class="c1">// Add length and move it as a prefix</span>
<span class="w"> </span><span class="nx">end</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nb">len</span><span class="p">(</span><span class="nx">bf</span><span class="p">.</span><span class="nx">protobuf</span><span class="p">)</span>
<span class="w"> </span><span class="nx">payloadLen</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">end</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="nx">maxSizeVarint</span>
<span class="w"> </span><span class="nx">bf</span><span class="p">.</span><span class="nx">protobuf</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">protowire</span><span class="p">.</span><span class="nx">AppendVarint</span><span class="p">(</span><span class="nx">bf</span><span class="p">.</span><span class="nx">protobuf</span><span class="p">,</span><span class="w"> </span><span class="nb">uint64</span><span class="p">(</span><span class="nx">payloadLen</span><span class="p">))</span>
<span class="w"> </span><span class="nx">sizeLen</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nb">len</span><span class="p">(</span><span class="nx">bf</span><span class="p">.</span><span class="nx">protobuf</span><span class="p">)</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="nx">end</span>
<span class="w"> </span><span class="nx">result</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">bf</span><span class="p">.</span><span class="nx">protobuf</span><span class="p">[</span><span class="nx">maxSizeVarint</span><span class="o">-</span><span class="nx">sizeLen</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="nx">end</span><span class="p">]</span>
<span class="w"> </span><span class="nb">copy</span><span class="p">(</span><span class="nx">result</span><span class="p">,</span><span class="w"> </span><span class="nx">bf</span><span class="p">.</span><span class="nx">protobuf</span><span class="p">[</span><span class="nx">end</span><span class="p">:</span><span class="nx">end</span><span class="o">+</span><span class="nx">sizeLen</span><span class="p">])</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nx">result</span>
<span class="p">}</span>
</pre></div>
<p>Minimiser les allocations est essentiel pour maintenir de bonnes performances.
Les tests doivent être exécutés avec le drapeau <code>-benchmem</code> pour surveiller le
nombre d’allocations : chacune entraîne un coût indirect pour le ramasse-miette.
Le <a href="https://go.dev/blog/pprof" title="Profiling Go Programs">profileur Go</a> est un outil précieux pour identifier les zones
du code qui peuvent être optimisées :</p>
<div class="language-bash-session codehilite"><pre><span/><span class="gp">$ </span>go<span class="w"> </span><span class="nb">test</span><span class="w"> </span>-run<span class="o">=</span>__nothing__<span class="w"> </span>-bench<span class="o">=</span>Netflow/with_encoding<span class="w"> </span><span class="se">\</span>
<span class="gp">> </span><span class="w"> </span>-benchmem<span class="w"> </span>-cpuprofile<span class="w"> </span>profile.out<span class="w"> </span><span class="se">\</span>
<span class="gp">> </span><span class="w"> </span>akvorado/inlet/flow
<span class="go">goos: linux</span>
<span class="go">goarch: amd64</span>
<span class="go">pkg: akvorado/inlet/flow</span>
<span class="go">cpu: AMD Ryzen 5 5600X 6-Core Processor</span>
<span class="go">Netflow/with_encoding-12 143953 7955 ns/op 8256 B/op 134 allocs/op</span>
<span class="go">PASS</span>
<span class="go">ok akvorado/inlet/flow 1.418s</span>
<span class="gp">$ </span>go<span class="w"> </span>tool<span class="w"> </span>pprof<span class="w"> </span>profile.out
<span class="go">File: flow.test</span>
<span class="go">Type: cpu</span>
<span class="go">Time: Feb 4, 2023 at 8:12pm (CET)</span>
<span class="go">Duration: 1.41s, Total samples = 2.08s (147.96%)</span>
<span class="go">Entering interactive mode (type "help" for commands, "o" for options)</span>
<span class="gp gp-VirtualEnv">(pprof)</span> <span class="go">web</span>
</pre></div>
<p>Après avoir <a href="https://github.com/akvorado/akvorado/commit/e352202631a898947925337232647fdce50aa0f1">utilisé le schéma interne</a> au lieu du code généré à
partir du fichier de définition, les performances se sont améliorées. Cependant,
cette comparaison n’est pas tout à fait équitable car moins d’informations sont
décodées et, auparavant, <em>GoFlow2</em> décodait les flux vers sa propre structure
qui était ensuite copiée dans notre version.</p>
<div class="language-text-only codehilite"><pre><span/>goos: linux
goarch: amd64
pkg: akvorado/inlet/flow
cpu: AMD Ryzen 5 5600X 6-Core Processor
│ bench-2.txt │ bench-3.txt │
│ sec/op │ sec/op vs base │
Netflow/with_encoding-12 10.284µ ± 2% 7.758µ ± 3% -24.56% (p=0.000 n=10)
Netflow/without_encoding-12 8.975µ ± 2% 7.304µ ± 2% -18.61% (p=0.000 n=10)
Sflow/with_encoding-12 16.67µ ± 2% 14.26µ ± 1% -14.50% (p=0.000 n=10)
Sflow/without_encoding-12 14.87µ ± 1% 13.56µ ± 2% -8.80% (p=0.000 n=10)
</pre></div>
<p>Concernant les tests, nous utilisons
<a href="https://pkg.go.dev/github.com/jhump/protoreflect"><code>github.com/jhump/protoreflect</code></a> : le paquet <code>protoparse</code> analyse
le fichier de définition que nous avons construit dynamiquement et le paquet
<code>dynamic</code> décode les messages. Jetez un œil à la méthode <a href="https://github.com/akvorado/akvorado/blob/9c51b2284513526f6491a9138953e0bd00f680a8/common/schema/tests.go#L44-L117"><code>ProtobufDecode()</code>
method</a> pour plus de détails<sup id="fnref-protoprint"><a class="footnote-ref" href="#fn-protoprint">4</a></sup>.</p>
<p>Pour obtenir les chiffres finaux, j’ai aussi optimisé le décodage dans
<em>GoFlow2</em>. Il s’appuyait fortement sur <a href="https://pkg.go.dev/encoding/binary#Read"><code>binary.Read()</code></a>. Cette
fonction peut utiliser la réflexion dans certains cas et chaque appel alloue un
tableau d’octets pour lire les données. <a href="https://github.com/netsampler/goflow2/pull/141" title="#141: decoders: replace binary.Read with a version without reflection and allocations">En la remplaçant par une version plus
efficace</a>, on obtient encore une amélioration notable des performances :</p>
<div class="language-text-only codehilite"><pre><span/>goos: linux
goarch: amd64
pkg: akvorado/inlet/flow
cpu: AMD Ryzen 5 5600X 6-Core Processor
│ bench-3.txt │ bench-4.txt │
│ sec/op │ sec/op vs base │
Netflow/with_encoding-12 7.758µ ± 3% 7.365µ ± 2% -5.07% (p=0.000 n=10)
Netflow/without_encoding-12 7.304µ ± 2% 6.931µ ± 3% -5.11% (p=0.000 n=10)
Sflow/with_encoding-12 14.256µ ± 1% 9.834µ ± 2% -31.02% (p=0.000 n=10)
Sflow/without_encoding-12 13.559µ ± 2% 9.353µ ± 2% -31.02% (p=0.000 n=10)
</pre></div>
<p>Il est maintenant plus facile de collecter de <a href="https://demo.akvorado.net/docs/internals#schema">nouvelles données</a> et
le composant recevant les flux est désormais plus rapide ! 🚅</p>
<div class="admonition">
<p class="admonition-title">Note</p>
<p>La plupart des paragraphes ont été traduits de l’anglais par
<a href="https://chat.openai.com/chat">ChatGPT</a> en utilisant les instructions suivantes : <span lang="en">“From now
on, I will paste Markdown code in English and I would like you to translate it
to French. Keep the markdown markup and enclose the result into a code block.
Thanks.”</span> Le résultat a été légèrement édité si nécessaire. Comparé à
<a href="https://www.deepl.com/translator">DeepL</a>, <em>ChatGPT</em> est capable de conserver le formatage, les anglicismes,
mais son français est moins bon et il est nécessaire de lui rappeler
régulièrement les instructions.</p>
</div>
<div class="footnote">
<hr/>
<ol>
<li id="fn-cost">
<p>Bien que les champs vides ne sont pas sérialisés en <em>Protocol Buffers</em>,
les colonnes vides dans <em>ClickHouse</em> occupent de la place, même si
elles se compressent bien. De plus, les champs inutilisés sont toujours
décodés et peuvent encombrer l’interface. <a class="footnote-backref" href="#fnref-cost" title="Jump back to footnote 1 in the text">↩︎</a></p>
</li>
<li id="fn-netflow">
<p>Il existe une fonction similaire pour <em>NetFlow</em>. Les protocoles
NetFlow et IPFIX sont moins complexes à décoder que sFlow car ils utilisent
une structure <abbr title="Type-Length-Value">TLV</abbr> plus simple. <a class="footnote-backref" href="#fnref-netflow" title="Jump back to footnote 2 in the text">↩︎</a></p>
</li>
<li id="fn-vtprotobuf">
<p><code>vtprotobuf</code> génère un code mieux optimisé en supprimant un
niveau d’indirection. Il produit du code codant chaque champ en octets :</p>
<div class="language-go codehilite"><pre><span/><span class="k">if</span><span class="w"> </span><span class="nx">m</span><span class="p">.</span><span class="nx">OutIfSpeed</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nx">i</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">encodeVarint</span><span class="p">(</span><span class="nx">dAtA</span><span class="p">,</span><span class="w"> </span><span class="nx">i</span><span class="p">,</span><span class="w"> </span><span class="nb">uint64</span><span class="p">(</span><span class="nx">m</span><span class="p">.</span><span class="nx">OutIfSpeed</span><span class="p">))</span>
<span class="w"> </span><span class="nx">i</span><span class="o">--</span>
<span class="w"> </span><span class="nx">dAtA</span><span class="p">[</span><span class="nx">i</span><span class="p">]</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="mh">0x6</span>
<span class="w"> </span><span class="nx">i</span><span class="o">--</span>
<span class="w"> </span><span class="nx">dAtA</span><span class="p">[</span><span class="nx">i</span><span class="p">]</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="mh">0xd8</span>
<span class="p">}</span>
</pre></div>
<p><a class="footnote-backref" href="#fnref-vtprotobuf" title="Jump back to footnote 3 in the text">↩︎</a></p>
</li>
<li id="fn-protoprint">
<p>Il existe également un paquet <code>protoprint</code> pour générer le
fichier de définition. Je ne l’ai pas utilisé. <a class="footnote-backref" href="#fnref-protoprint" title="Jump back to footnote 4 in the text">↩︎</a></p>
</li>
</ol>
</div>
Gérer une infrastructure avec Terraform, CDKTF et NixOSVincent Bernat2022-12-26T15:18:38Zhttp://www.luffy.cx/fr/blog/2022-cdktf-nixos.html
<p>Il y a quelques années, j’ai réduit mon infrastructure personnelle au strict
minimum. Jusqu’en 2018, elle représentait une douzaine de conteneurs tournant
sur un seul serveur <a href="https://www.hetzner.com/" title="Hetzner: dedicated servers, cloud, storage & hosting">Hetzner</a><sup id="fnref-hetzner"><a class="footnote-ref" href="#fn-hetzner">1</a></sup>. J’ai migré mon courriel vers
<a href="https://www.fastmail.com/">Fastmail</a> et mes zones DNS vers <a href="https://www.gandi.net/en" title="Gandi: Domain Names, Web Hosting, Mail">Gandi</a>. Il ne me restait plus que mon blog
à héberger. À ce jour, ma petite infrastructure est composée de 4 machines
virtuelles exécutant <a href="https://nixos.org/" title="Nix: reproducible builds and deployments">NixOS</a> sur <em>Hetzner Cloud</em> et <a href="https://www.vultr.com/" title="Vultr.com: SSD VPS Servers, Cloud Servers and Cloud Hosting">Vultr</a>, d’une poignée
de zones DNS sur <em>Gandi</em> et <a href="https://aws.amazon.com/route53/" title="Amazon AWS: Route 53">Route 53</a>, et de quelques distributions
<a href="https://aws.amazon.com/cloudfront/" title="Amazon AWS: CloudFront">Cloudfront</a>. Elle est gérée par <a href="https://developer.hashicorp.com/terraform/cdktf" title="Cloud Development Kit for Terraform">CDK pour Terraform</a>
(<abbr title="Cloud Development Kit for Terraform">CDKTF</abbr>), tandis que les déploiements de <em>NixOS</em> sont gérés par <a href="https://github.com/NixOS/nixops" title="NixOps: tool for deploying to NixOS machines in a network or cloud">NixOps</a>.</p>
<p>Dans cet article, je présente brièvement <em>Terraform</em>, <em><abbr title="Cloud Development Kit for Terraform">CDKTF</abbr></em> et l’écosystème
<em>Nix</em>. J’explique également comment utiliser <em>Nix</em> pour accéder à ces outils
dans votre shell afin de les utiliser rapidement.</p>
<div class="admonition">
<p class="admonition-title">Mise à jour (11.2023)</p>
<p>Récemment, <em>HashiCorp</em> <a href="https://www.hashicorp.com/blog/hashicorp-adopts-business-source-license">a opté pour la Business
Source License</a> pour tous ces
logiciels. C’est une déception, notamment pour <em>Terraform</em> qui est un composant
clé pour automatiser les infrastructures et qui a bénéficié d’une large
communauté. Il existe désormais un fork communautaire, <a href="https://opentofu.org/" title="The open source infrastructure as code tool">OpenTofu</a>, mais
celui-ci ne couvre pas <em><abbr title="Cloud Development Kit for Terraform">CDKTF</abbr></em>.</p>
</div>
<div class="toc">
<ul>
<li><a href="#cdktf-infrastructure-en-tant-que-code">CDKTF : infrastructure en tant que code</a><ul>
<li><a href="#gerer-des-serveurs">Gérer des serveurs</a></li>
<li><a href="#gerer-des-enregistrements-dns">Gérer des enregistrements DNS</a></li>
<li><a href="#a-propos-de-pulumi">À propos de Pulumi</a></li>
</ul>
</li>
<li><a href="#nixos-nixops">NixOS & NixOps</a><ul>
<li><a href="#nixos-distribution-linux-declarative">NixOS : distribution Linux déclarative</a></li>
<li><a href="#nixops-outil-de-deploiement-pour-nixos">NixOps : outil de déploiement pour NixOS</a></li>
</ul>
</li>
<li><a href="#connecter-le-tout-avec-nix">Connecter le tout avec Nix</a><ul>
<li><a href="#courte-introduction-aux-flakes-nix">Courte introduction aux « flakes » Nix</a></li>
<li><a href="#nix-et-cdktf">Nix et CDKTF</a></li>
<li><a href="#nixops">NixOps</a></li>
</ul>
</li>
</ul>
</div>
<h1 id="cdktf-infrastructure-en-tant-que-code"><abbr title="Cloud Development Kit for Terraform">CDKTF</abbr> : infrastructure en tant que code<a class="headerlink" href="#cdktf-infrastructure-en-tant-que-code" title="Permanent link"></a></h1>
<p><a href="https://www.terraform.io/">Terraform</a> est un outil d’« infrastructure en tant que code ». Vous pouvez
définir votre infrastructure en déclarant des ressources avec le <a href="https://developer.hashicorp.com/terraform/language/modules">langage
<abbr title="HashiCorp Configuration Language">HCL</abbr></a>. Ce dernier possède quelques fonctionnalités supplémentaires,
comme des boucles permettant de déclarer plusieurs ressources à partir d’une
liste, des fonctions intégrées que vous pouvez appeler dans les expressions et
l’expansion de variables dans les chaînes de caractères. <em>Terraform</em> s’appuie
sur un large <a href="https://registry.terraform.io/browse/providers">ensemble de fournisseurs</a> pour gérer les
ressources.</p>
<h2 id="gerer-des-serveurs">Gérer des serveurs<a class="headerlink" href="#gerer-des-serveurs" title="Permanent link"></a></h2>
<p>Voici un court exemple utilisant le <a href="https://registry.terraform.io/providers/hetznercloud/hcloud/latest/docs">fournisseur pour Hetzner Cloud</a> pour créer une machine virtuelle :</p>
<div class="language-terraform codehilite"><pre><span/><span class="kr">variable</span><span class="w"> </span><span class="nv">"hcloud_token"</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="na">sensitive</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="no">true</span>
<span class="p">}</span>
<span class="kr">provider</span><span class="w"> </span><span class="nv">"hcloud"</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="na">token</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nv">var.hcloud_token</span>
<span class="p">}</span>
<span class="kr">resource</span><span class="w"> </span><span class="nc">"hcloud_server"</span><span class="w"> </span><span class="nv">"web03"</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="na">name</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">"web03"</span>
<span class="w"> </span><span class="na">server_type</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">"cpx11"</span>
<span class="w"> </span><span class="na">image</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">"debian-11"</span>
<span class="w"> </span><span class="na">datacenter</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">"nbg1-dc3"</span>
<span class="p">}</span>
<span class="kr">resource</span><span class="w"> </span><span class="nc">"hcloud_rdns"</span><span class="w"> </span><span class="nv">"rdns4-web03"</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="na">server_id</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nv">hcloud_server.web03.id</span>
<span class="w"> </span><span class="na">ip_address</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nv">hcloud_server.web03.ipv4_address</span>
<span class="w"> </span><span class="na">dns_ptr</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">"web03.luffy.cx"</span>
<span class="p">}</span>
<span class="kr">resource</span><span class="w"> </span><span class="nc">"hcloud_rdns"</span><span class="w"> </span><span class="nv">"rdns6-web03"</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="na">server_id</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nv">hcloud_server.web03.id</span>
<span class="w"> </span><span class="na">ip_address</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nv">hcloud_server.web03.ipv6_address</span>
<span class="w"> </span><span class="na">dns_ptr</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">"web03.luffy.cx"</span>
<span class="p">}</span>
</pre></div>
<p>L’expressivité de <em><abbr title="HashiCorp Configuration Language">HCL</abbr></em> est assez limitée et je trouve qu’un langage généraliste
est plus pratique pour décrire les ressources. C’est là qu’intervient <em>CDK pour
Terraform</em> : vous pouvez gérer votre infrastructure à l’aide de votre langage de
programmation préféré, notamment TypeScript, Go et Python. Voici l’exemple
précédent utilisant <em><abbr title="Cloud Development Kit for Terraform">CDKTF</abbr></em> et TypeScript :</p>
<div class="language-typescript codehilite"><pre><span/><span class="k">import</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nx">App</span><span class="p">,</span><span class="w"> </span><span class="nx">TerraformStack</span><span class="p">,</span><span class="w"> </span><span class="nx">Fn</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="kr">from</span><span class="w"> </span><span class="s2">"cdktf"</span><span class="p">;</span>
<span class="k">import</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nx">HcloudProvider</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="kr">from</span><span class="w"> </span><span class="s2">"./.gen/providers/hcloud/provider"</span><span class="p">;</span>
<span class="k">import</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="kr">as</span><span class="w"> </span><span class="nx">hcloud</span><span class="w"> </span><span class="kr">from</span><span class="w"> </span><span class="s2">"./.gen/providers/hcloud"</span><span class="p">;</span>
<span class="kd">class</span><span class="w"> </span><span class="nx">MyStack</span><span class="w"> </span><span class="k">extends</span><span class="w"> </span><span class="nx">TerraformStack</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="kr">constructor</span><span class="p">(</span><span class="nx">scope</span><span class="o">:</span><span class="w"> </span><span class="kt">Construct</span><span class="p">,</span><span class="w"> </span><span class="nx">name</span><span class="o">:</span><span class="w"> </span><span class="kt">string</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">super</span><span class="p">(</span><span class="nx">scope</span><span class="p">,</span><span class="w"> </span><span class="nx">name</span><span class="p">);</span>
<span class="w"> </span><span class="kd">const</span><span class="w"> </span><span class="nx">hcloudToken</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ow">new</span><span class="w"> </span><span class="nx">TerraformVariable</span><span class="p">(</span><span class="k">this</span><span class="p">,</span><span class="w"> </span><span class="s2">"hcloudToken"</span><span class="p">,</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="kr">type</span><span class="o">:</span><span class="w"> </span><span class="s2">"string"</span><span class="p">,</span>
<span class="w"> </span><span class="nx">sensitive</span><span class="o">:</span><span class="w"> </span><span class="kt">true</span><span class="p">,</span>
<span class="w"> </span><span class="p">});</span>
<span class="w"> </span><span class="kd">const</span><span class="w"> </span><span class="nx">hcloudProvider</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ow">new</span><span class="w"> </span><span class="nx">HcloudProvider</span><span class="p">(</span><span class="k">this</span><span class="p">,</span><span class="w"> </span><span class="s2">"hcloud"</span><span class="p">,</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nx">token</span><span class="o">:</span><span class="w"> </span><span class="kt">hcloudToken.value</span><span class="p">,</span>
<span class="w"> </span><span class="p">});</span>
<span class="w"> </span><span class="kd">const</span><span class="w"> </span><span class="nx">web03</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ow">new</span><span class="w"> </span><span class="nx">hcloud</span><span class="p">.</span><span class="nx">server</span><span class="p">.</span><span class="nx">Server</span><span class="p">(</span><span class="k">this</span><span class="p">,</span><span class="w"> </span><span class="s2">"web03"</span><span class="p">,</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nx">name</span><span class="o">:</span><span class="w"> </span><span class="s2">"web03"</span><span class="p">,</span>
<span class="w"> </span><span class="nx">serverType</span><span class="o">:</span><span class="w"> </span><span class="s2">"cpx11"</span><span class="p">,</span>
<span class="w"> </span><span class="nx">image</span><span class="o">:</span><span class="w"> </span><span class="s2">"debian-11"</span><span class="p">,</span>
<span class="w"> </span><span class="nx">datacenter</span><span class="o">:</span><span class="w"> </span><span class="s2">"nbg1-dc3"</span><span class="p">,</span>
<span class="w"> </span><span class="nx">provider</span><span class="o">:</span><span class="w"> </span><span class="kt">hcloudProvider</span><span class="p">,</span>
<span class="w"> </span><span class="p">});</span>
<span class="w"> </span><span class="ow">new</span><span class="w"> </span><span class="nx">hcloud</span><span class="p">.</span><span class="nx">rdns</span><span class="p">.</span><span class="nx">Rdns</span><span class="p">(</span><span class="k">this</span><span class="p">,</span><span class="w"> </span><span class="s2">"rdns4-web03"</span><span class="p">,</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nx">serverId</span><span class="o">:</span><span class="w"> </span><span class="kt">Fn.tonumber</span><span class="p">(</span><span class="nx">web03</span><span class="p">.</span><span class="nx">id</span><span class="p">),</span>
<span class="w"> </span><span class="nx">ipAddress</span><span class="o">:</span><span class="w"> </span><span class="kt">web03.ipv4Address</span><span class="p">,</span>
<span class="w"> </span><span class="nx">dnsPtr</span><span class="o">:</span><span class="w"> </span><span class="s2">"web03.luffy.cx"</span><span class="p">,</span>
<span class="w"> </span><span class="nx">provider</span><span class="o">:</span><span class="w"> </span><span class="kt">hcloudProvider</span><span class="p">,</span>
<span class="w"> </span><span class="p">});</span>
<span class="w"> </span><span class="ow">new</span><span class="w"> </span><span class="nx">hcloud</span><span class="p">.</span><span class="nx">rdns</span><span class="p">.</span><span class="nx">Rdns</span><span class="p">(</span><span class="k">this</span><span class="p">,</span><span class="w"> </span><span class="s2">"rdns6-web03"</span><span class="p">,</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nx">serverId</span><span class="o">:</span><span class="w"> </span><span class="kt">Fn.tonumber</span><span class="p">(</span><span class="nx">web03</span><span class="p">.</span><span class="nx">id</span><span class="p">),</span>
<span class="w"> </span><span class="nx">ipAddress</span><span class="o">:</span><span class="w"> </span><span class="kt">web03.ipv6Address</span><span class="p">,</span>
<span class="w"> </span><span class="nx">dnsPtr</span><span class="o">:</span><span class="w"> </span><span class="s2">"web03.luffy.cx"</span><span class="p">,</span>
<span class="w"> </span><span class="nx">provider</span><span class="o">:</span><span class="w"> </span><span class="kt">hcloudProvider</span><span class="p">,</span>
<span class="w"> </span><span class="p">});</span>
<span class="w"> </span><span class="p">}</span>
<span class="p">}</span>
<span class="kd">const</span><span class="w"> </span><span class="nx">app</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ow">new</span><span class="w"> </span><span class="nx">App</span><span class="p">();</span>
<span class="ow">new</span><span class="w"> </span><span class="nx">MyStack</span><span class="p">(</span><span class="nx">app</span><span class="p">,</span><span class="w"> </span><span class="s2">"cdktf-take1"</span><span class="p">);</span>
<span class="nx">app</span><span class="p">.</span><span class="nx">synth</span><span class="p">();</span>
</pre></div>
<p>La commande de <code>cdktf synth</code> génère un fichier de configuration pour
<em>Terraform</em>, <code>terraform plan</code> prévisualise les changements et <code>terraform apply</code>
les applique. Maintenant que vous disposez d’un langage généraliste, vous pouvez
utiliser des fonctions.</p>
<h2 id="gerer-des-enregistrements-dns">Gérer des enregistrements DNS<a class="headerlink" href="#gerer-des-enregistrements-dns" title="Permanent link"></a></h2>
<p>Si l’utilisation de <em><abbr title="Cloud Development Kit for Terraform">CDKTF</abbr></em> pour 4 serveurs web peut sembler un peu exagérée, il
en va tout autrement lorsqu’il s’agit de gérer quelques zones DNS. Avec
<a href="https://stackexchange.github.io/dnscontrol/" title="DNSControl: Synchronize your DNS to multiple providers from a simple DSL">DNSControl</a>, qui utilise JavaScript comme langage, j’ai pu définir la zone
<code>bernat.ch</code> avec ce bout de code :</p>
<div class="language-javascript codehilite"><pre><span/><span class="nx">D</span><span class="p">(</span><span class="s2">"bernat.ch"</span><span class="p">,</span><span class="w"> </span><span class="nx">REG_NONE</span><span class="p">,</span><span class="w"> </span><span class="nx">DnsProvider</span><span class="p">(</span><span class="nx">DNS_BIND</span><span class="p">,</span><span class="w"> </span><span class="mf">0</span><span class="p">),</span><span class="w"> </span><span class="nx">DnsProvider</span><span class="p">(</span><span class="nx">DNS_GANDI</span><span class="p">),</span>
<span class="w"> </span><span class="nx">DefaultTTL</span><span class="p">(</span><span class="s1">'2h'</span><span class="p">),</span>
<span class="w"> </span><span class="nx">FastMailMX</span><span class="p">(</span><span class="s1">'bernat.ch'</span><span class="p">,</span><span class="w"> </span><span class="p">{</span><span class="nx">subdomains</span><span class="o">:</span><span class="w"> </span><span class="p">[</span><span class="s1">'vincent'</span><span class="p">]}),</span>
<span class="w"> </span><span class="nx">WebServers</span><span class="p">(</span><span class="s1">'@'</span><span class="p">),</span>
<span class="w"> </span><span class="nx">WebServers</span><span class="p">(</span><span class="s1">'vincent'</span><span class="p">);</span>
</pre></div>
<p>Cela produit 38 enregistrements. Avec <em><abbr title="Cloud Development Kit for Terraform">CDKTF</abbr></em>, j’écris :</p>
<div class="language-typescript codehilite"><pre><span/><span class="ow">new</span><span class="w"> </span><span class="nx">Route53Zone</span><span class="p">(</span><span class="k">this</span><span class="p">,</span><span class="w"> </span><span class="s2">"bernat.ch"</span><span class="p">,</span><span class="w"> </span><span class="nx">providers</span><span class="p">.</span><span class="nx">aws</span><span class="p">)</span>
<span class="w"> </span><span class="p">.</span><span class="nx">sign</span><span class="p">(</span><span class="nx">dnsCMK</span><span class="p">)</span>
<span class="w"> </span><span class="p">.</span><span class="nx">registrar</span><span class="p">(</span><span class="nx">providers</span><span class="p">.</span><span class="nx">gandiVB</span><span class="p">)</span>
<span class="w"> </span><span class="p">.</span><span class="nx">www</span><span class="p">(</span><span class="s2">"@"</span><span class="p">,</span><span class="w"> </span><span class="nx">servers</span><span class="p">)</span>
<span class="w"> </span><span class="p">.</span><span class="nx">www</span><span class="p">(</span><span class="s2">"vincent"</span><span class="p">,</span><span class="w"> </span><span class="nx">servers</span><span class="p">)</span>
<span class="w"> </span><span class="p">.</span><span class="nx">www</span><span class="p">(</span><span class="s2">"media"</span><span class="p">,</span><span class="w"> </span><span class="nx">servers</span><span class="p">)</span>
<span class="w"> </span><span class="p">.</span><span class="nx">fastmailMX</span><span class="p">([</span><span class="s2">"vincent"</span><span class="p">]);</span>
</pre></div>
<p>Toute la magie est située dans les fonctions appelées. Vous pouvez regarder le
fichier <a href="https://github.com/vincentbernat/cdktf-take1/blob/main/luffy/dns.ts">dns.ts</a> dans le dépôt <a href="https://github.com/vincentbernat/cdktf-take1">cdktf-take1</a> pour comprendre comment cela
fonctionne. Rapidement :</p>
<ul>
<li><code>Route53Zone()</code> crée une zone sur <em>Route 53</em>,</li>
<li><code>sign()</code> signe la zone avec une clé maître,</li>
<li><code>registrar()</code> inscrit la zone auprès du registre de domaines et configure DNSSEC,</li>
<li><code>www()</code> crée les enregistrements <code>A</code> et <code>AAAA</code> pour les serveurs web,</li>
<li><code>fastmailMX()</code> crée les enregistrements <code>MX</code> et d’autres enregistrements liés
pour configurer <em>Fastmail</em> en tant que fournisseur de courriel.</li>
</ul>
<p>Voici le contenu de la fonction <code>fastmailMX()</code>. Elle génère quelques
enregistrements et retourne la zone en cours pour faciliter le chaînage :</p>
<div class="language-typescript codehilite"><pre><span/><span class="nx">fastmailMX</span><span class="p">(</span><span class="nx">subdomains?</span><span class="o">:</span><span class="w"> </span><span class="kt">string</span><span class="p">[])</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="p">(</span><span class="nx">subdomains</span><span class="w"> </span><span class="o">??</span><span class="w"> </span><span class="p">[])</span>
<span class="w"> </span><span class="p">.</span><span class="nx">concat</span><span class="p">([</span><span class="s2">"@"</span><span class="p">,</span><span class="w"> </span><span class="s2">"*"</span><span class="p">])</span>
<span class="w"> </span><span class="p">.</span><span class="nx">forEach</span><span class="p">((</span><span class="nx">subdomain</span><span class="p">)</span><span class="w"> </span><span class="p">=></span>
<span class="w"> </span><span class="k">this</span><span class="p">.</span><span class="nx">MX</span><span class="p">(</span><span class="nx">subdomain</span><span class="p">,</span><span class="w"> </span><span class="p">[</span>
<span class="w"> </span><span class="s2">"10 in1-smtp.messagingengine.com."</span><span class="p">,</span>
<span class="w"> </span><span class="s2">"20 in2-smtp.messagingengine.com."</span><span class="p">,</span>
<span class="w"> </span><span class="p">])</span>
<span class="w"> </span><span class="p">);</span>
<span class="w"> </span><span class="k">this</span><span class="p">.</span><span class="nx">TXT</span><span class="p">(</span><span class="s2">"@"</span><span class="p">,</span><span class="w"> </span><span class="s2">"v=spf1 include:spf.messagingengine.com ~all"</span><span class="p">);</span>
<span class="w"> </span><span class="p">[</span><span class="s2">"mesmtp"</span><span class="p">,</span><span class="w"> </span><span class="s2">"fm1"</span><span class="p">,</span><span class="w"> </span><span class="s2">"fm2"</span><span class="p">,</span><span class="w"> </span><span class="s2">"fm3"</span><span class="p">].</span><span class="nx">forEach</span><span class="p">((</span><span class="nx">dk</span><span class="p">)</span><span class="w"> </span><span class="p">=></span>
<span class="w"> </span><span class="k">this</span><span class="p">.</span><span class="nx">CNAME</span><span class="p">(</span><span class="sb">`</span><span class="si">${</span><span class="nx">dk</span><span class="si">}</span><span class="sb">._domainkey`</span><span class="p">,</span><span class="w"> </span><span class="sb">`</span><span class="si">${</span><span class="nx">dk</span><span class="si">}</span><span class="sb">.</span><span class="si">${</span><span class="k">this</span><span class="p">.</span><span class="nx">name</span><span class="si">}</span><span class="sb">.dkim.fmhosted.com.`</span><span class="p">)</span>
<span class="w"> </span><span class="p">);</span>
<span class="w"> </span><span class="k">this</span><span class="p">.</span><span class="nx">TXT</span><span class="p">(</span><span class="s2">"_dmarc"</span><span class="p">,</span><span class="w"> </span><span class="s2">"v=DMARC1; p=none; sp=none"</span><span class="p">);</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="k">this</span><span class="p">;</span>
<span class="p">}</span>
</pre></div>
<p>Je vous encourage à parcourir le <a href="https://github.com/vincentbernat/cdktf-take1">dépôt</a> pour plus de détails !</p>
<h2 id="a-propos-de-pulumi">À propos de Pulumi<a class="headerlink" href="#a-propos-de-pulumi" title="Permanent link"></a></h2>
<p>Ma première tentative autour de <em>Terraform</em> a été d’utiliser <a href="https://www.pulumi.com/" title="Pulumi: universal infrastructure as code">Pulumi</a>. Vous
pouvez trouver cette tentative sur <a href="https://github.com/vincentbernat/pulumi-take1">GitHub</a>. C’est assez
similaire à ce que je fais actuellement avec <em><abbr title="Cloud Development Kit for Terraform">CDKTF</abbr></em>. La principale différence
est que j’utilise Python au lieu de TypeScript<sup id="fnref-python"><a class="footnote-ref" href="#fn-python">2</a></sup> car ce dernier ne
m’était pas familier à l’époque.</p>
<p><em>Pulumi</em> est antérieur à <em><abbr title="Cloud Development Kit for Terraform">CDKTF</abbr></em> et il utilise une approche légèrement
différente. <em><abbr title="Cloud Development Kit for Terraform">CDKTF</abbr></em> génère une configuration pour <em>Terraform</em> (au format JSON au
lieu de <abbr title="HashiCorp Configuration Language">HCL</abbr>), laissant la planification, la gestion des états et le déploiement
à ce dernier. Il est donc lié aux limites de ce qui peut être exprimé par
<em>Terraform</em>, notamment lorsque vous devez transformer des données obtenues d’une
ressource à une autre<sup id="fnref-transform"><a class="footnote-ref" href="#fn-transform">3</a></sup>. <em>Pulumi</em> a besoin de fournisseurs spécifiques
pour chaque ressource. De nombreux fournisseurs encapsulent des fournisseurs
<em>Terraform</em>.</p>
<p>Bien que <em>Pulumi</em> offre une bonne expérience utilisateur, je suis passé à
<em><abbr title="Cloud Development Kit for Terraform">CDKTF</abbr></em> car écrire des fournisseurs pour <em>Pulumi</em> est une corvée. <em><abbr title="Cloud Development Kit for Terraform">CDKTF</abbr></em> ne
nécessite pas de passer par cette étape. En dehors des grands acteurs (AWS,
Azure et Google Cloud), l’existence, la qualité et la fraîcheur des fournisseurs
<em>Pulumi</em> sont inégales. La plupart des fournisseurs s’appuient sur un
fournisseur <em>Terraform</em> et il se peut qu’ils soient en retard de quelques
versions, qu’il leur manque quelques ressources ou qu’ils présentent des bogues
qui leur sont propres.</p>
<p>Lorsqu’un fournisseur n’existe pas, vous pouvez en écrire un à l’aide de la
bibliothèque <a href="https://github.com/pulumi/pulumi-terraform-bridge">pulumi-terraform-bridge</a>. Le projet Pulumi fournit un
<a href="https://github.com/pulumi/pulumi-tf-provider-boilerplate">modèle</a> à cet effet. J’ai eu une mauvaise expérience avec celui-ci
lors de l’écriture de fournisseurs pour <a href="https://github.com/vincentbernat/pulumi-gandi-old" title="Deprecated Pulumi provider for Gandi">Gandi</a> et
<a href="https://github.com/vincentbernat/pulumi-vultr" title="Pulumi provider for Vultr">Vultr</a> : le <code>Makefile</code> <a href="https://github.com/pulumi/pulumi-tf-provider-boilerplate/pull/51" title="PR #51: Do not install pulumi automatically">installe automatiquement
Pulumi</a> en utilisant <code>curl | sh</code> et <a href="https://github.com/pulumi/pulumi-tf-provider-boilerplate/pull/52" title="PR #52: Fix Makefile to work with a POSIX shell">ne fonctionne pas avec
<code>/bin/sh</code></a>. Il y a un manque d’intérêt pour les contributions
communautaires<sup id="fnref-maint"><a class="footnote-ref" href="#fn-maint">4</a></sup> ou même pour <a href="https://twitter.com/briggsl/status/1473009153983623176">les fournisseurs pour les acteurs
tiers</a>.</p>
<h1 id="nixos-nixops">NixOS & NixOps<a class="headerlink" href="#nixos-nixops" title="Permanent link"></a></h1>
<p><a href="https://nixos.org/manual/nix/stable/language/index.html" title="Nix, the language">Nix</a> est un langage de programmation purement fonctionnel.
<a href="https://nixos.org/manual/nix/stable/introduction.html" title="Nix, the package manager">Nix</a> est aussi le nom du gestionnaire de paquets qui est construit
au-dessus du langage <em>Nix</em>. Il permet aux utilisateurs d’installer des paquets
de manière déclarative. <a href="https://github.com/NixOS/nixpkgs">nixpkgs</a> est un dépôt de paquets. Vous pouvez
<a href="https://nixos.org/manual/nix/stable/installation/multi-user.html">installer <em>Nix</em></a> au-dessus d’une distribution Linux ordinaire. Si
vous voulez plus de détails, une bonne ressource est le <a href="https://nixos.org/">site
officiel</a>, notamment la <a href="https://nixos.org/learn.html">section « Apprendre »</a>. La
courbe d’apprentissage est rude, mais la récompense est grande.</p>
<h2 id="nixos-distribution-linux-declarative">NixOS : distribution Linux déclarative<a class="headerlink" href="#nixos-distribution-linux-declarative" title="Permanent link"></a></h2>
<p><a href="https://nixos.org/" title="Nix: reproducible builds and deployments">NixOS</a> est une distribution Linux construite au-dessus du gestionnaire de
paquets <em>Nix</em>. Voici un bout de configuration pour ajouter quelques paquets :</p>
<div class="language-nix codehilite"><pre><span/>environment<span class="o">.</span><span class="ss">systemPackages</span> <span class="o">=</span> <span class="k">with</span> pkgs<span class="p">;</span>
<span class="p">[</span>
bat
htop
liboping
mg
mtr
ncdu
tmux
<span class="p">];</span>
</pre></div>
<p>Il est possible de modifier une dérivation<sup id="fnref-derivation"><a class="footnote-ref" href="#fn-derivation">5</a></sup> existante pour utiliser une version
différente, activer une fonctionnalité spécifique ou appliquer un correctif.
Voici comment j’active et configure <em>Nginx</em> pour désactiver le module <code>stream</code>,
ajouter le module de <a href="https://github.com/google/ngx_brotli">compression Brotli</a> et ajouter
le module d’<a href="https://github.com/masonicboom/ipscrub">anonymisation des adresses IP</a>. De
plus, au lieu d’utiliser <em>OpenSSL 3</em>, je continue à utiliser <em>OpenSSL 1.1</em><sup id="fnref-openssl"><a class="footnote-ref" href="#fn-openssl">6</a></sup>.</p>
<div class="language-nix codehilite"><pre><span/>services<span class="o">.</span><span class="ss">nginx</span> <span class="o">=</span> <span class="p">{</span>
<span class="ss">enable</span> <span class="o">=</span> <span class="no">true</span><span class="p">;</span>
<span class="ss">package</span> <span class="o">=</span> <span class="p">(</span>pkgs<span class="o">.</span>nginxStable<span class="o">.</span>override <span class="p">{</span>
<span class="ss">withStream</span> <span class="o">=</span> <span class="no">false</span><span class="p">;</span>
<span class="ss">modules</span> <span class="o">=</span> <span class="k">with</span> pkgs<span class="o">.</span>nginxModules<span class="p">;</span> <span class="p">[</span>
brotli
ipscrub
<span class="p">];</span>
<span class="ss">openssl</span> <span class="o">=</span> pkgs<span class="o">.</span>openssl_1_1<span class="p">;</span>
<span class="p">});</span>
</pre></div>
<p>Si vous avez besoin d’ajouter certaines modifications, c’est également possible.
À titre d’exemple, voici comment j’ai corrigé en avance les <a href="https://github.com/Netflix/security-bulletins/blob/master/advisories/third-party/2019-002.md">failles de sécurité
découvertes en 2019 dans <em>Nginx</em></a> en attendant que cela soit corrigé
dans <em>NixOS</em><sup id="fnref-security"><a class="footnote-ref" href="#fn-security">7</a></sup> :</p>
<div class="language-nix codehilite"><pre><span/>services<span class="o">.</span>nginx<span class="o">.</span><span class="ss">package</span> <span class="o">=</span> pkgs<span class="o">.</span>nginxStable<span class="o">.</span>overrideAttrs <span class="p">(</span>old<span class="p">:</span> <span class="p">{</span>
<span class="ss">patches</span> <span class="o">=</span> old<span class="o">.</span>patches <span class="o">++</span> <span class="p">[</span>
<span class="c1"># HTTP/2: reject zero length headers with PROTOCOL_ERROR.</span>
<span class="p">(</span>pkgs<span class="o">.</span>fetchpatch <span class="p">{</span>
<span class="ss">url</span> <span class="o">=</span> <span class="l">https://github.com/nginx/nginx/commit/dbdd</span><span class="p">[</span><span class="err">…</span><span class="p">]</span><span class="o">.</span>patch<span class="p">;</span>
<span class="ss">sha256</span> <span class="o">=</span> <span class="s2">"a48190[…]"</span><span class="p">;</span>
<span class="p">})</span>
<span class="c1"># HTTP/2: limited number of DATA frames.</span>
<span class="p">(</span>pkgs<span class="o">.</span>fetchpatch <span class="p">{</span>
<span class="ss">url</span> <span class="o">=</span> <span class="l">https://github.com/nginx/nginx/commit/94c5</span><span class="p">[</span><span class="err">…</span><span class="p">]</span><span class="o">.</span>patch<span class="p">;</span>
<span class="ss">sha256</span> <span class="o">=</span> <span class="s2">"af591a[…]"</span><span class="p">;</span>
<span class="p">})</span>
<span class="c1"># HTTP/2: limited number of PRIORITY frames.</span>
<span class="p">(</span>pkgs<span class="o">.</span>fetchpatch <span class="p">{</span>
<span class="ss">url</span> <span class="o">=</span> <span class="l">https://github.com/nginx/nginx/commit/39bb</span><span class="p">[</span><span class="err">…</span><span class="p">]</span><span class="o">.</span>patch<span class="p">;</span>
<span class="ss">sha256</span> <span class="o">=</span> <span class="s2">"1ad8fe[…]"</span><span class="p">;</span>
<span class="p">})</span>
<span class="p">];</span>
<span class="p">});</span>
</pre></div>
<p>Si cela vous intéresse, jetez un coup d’œil à ma configuration relativement
réduite : <a href="https://github.com/vincentbernat/nixops-take1/blob/master/tags/common.nix"><code>common.nix</code></a> contient la configuration à appliquer sur
tous les serveurs (SSH, utilisateurs, paquets communs), <a href="https://github.com/vincentbernat/nixops-take1/blob/master/tags/web.nix"><code>web.nix</code></a>
contient la configuration pour les serveurs web et <a href="https://github.com/vincentbernat/nixops-take1/blob/master/tags/isso.nix"><code>isso.nix</code></a>
exécute <a href="/en/blog/2020-docker-nixos-isso" title="Running Isso on NixOS in a Docker container">Isso</a> dans un conteneur <em>systemd</em>.</p>
<h2 id="nixops-outil-de-deploiement-pour-nixos">NixOps : outil de déploiement pour NixOS<a class="headerlink" href="#nixops-outil-de-deploiement-pour-nixos" title="Permanent link"></a></h2>
<p>Sur un seul nœud, la configuration de <em>NixOS</em> se trouve dans le fichier
<code>/etc/nixos/configuration.nix</code>. Après l’avoir modifiée, vous devez exécuter la
commande <code>nixos-rebuild switch</code>. <em>Nix</em> va chercher toutes les dépendances
possibles dans le cache binaire et construit le reste. Il crée une nouvelle
entrée dans le menu du chargeur de démarrage et active la nouvelle
configuration.</p>
<p>Pour gérer plusieurs nœuds, il existe plusieurs options, dont <a href="https://github.com/NixOS/nixops" title="NixOps: tool for deploying to NixOS machines in a network or cloud">NixOps</a>,
<a href="https://github.com/serokell/deploy-rs" title="simple multi-profile Nix flake deploy tool">deploy-rs</a>, <a href="https://github.com/zhaofengli/colmena" title="Simple, stateless NixOS deployment tool">Colmena</a> et <a href="https://github.com/DBCDK/morph/" title="NixOS deployment tool">morph</a>. Je ne les connais pas toutes, mais de
mon point de vue, les différences ne sont pas si importantes. Il est également
possible de construire un tel outil soi-même car <em>Nix</em> fournit les blocs de
construction les plus importants : <code>nix build</code> et <code>nix copy</code>. <em>NixOps</em> est l’un
des premiers outils publiés mais je vous encourage à explorer les alternatives.</p>
<p>La configuration de <em>NixOps</em> est écrite avec la langage <em>Nix</em>. Voici une
configuration simplifiée pour déployer <code>znc01.luffy.cx</code>, <code>web01.luffy.cx</code> et
<code>web02.luffy.cx</code>, à l’aide des fonctions <code>server</code> et <code>web</code> :</p>
<div class="language-nix codehilite"><pre><span/><span class="k">let</span>
<span class="ss">server</span> <span class="o">=</span> hardware<span class="p">:</span> name<span class="p">:</span> imports<span class="p">:</span> <span class="p">{</span>
deployment<span class="o">.</span><span class="ss">targetHost</span> <span class="o">=</span> <span class="s2">"</span><span class="si">${</span>name<span class="si">}</span><span class="s2">.luffy.cx"</span><span class="p">;</span>
networking<span class="o">.</span><span class="ss">hostName</span> <span class="o">=</span> name<span class="p">;</span>
networking<span class="o">.</span><span class="ss">domain</span> <span class="o">=</span> <span class="s2">"luffy.cx"</span><span class="p">;</span>
<span class="ss">imports</span> <span class="o">=</span> <span class="p">[</span> <span class="p">(</span><span class="l">./hardware/.</span> <span class="o">+</span> <span class="s2">"/</span><span class="si">${</span>hardware<span class="si">}</span><span class="s2">.nix"</span><span class="p">)</span> <span class="p">]</span> <span class="o">++</span> imports<span class="p">;</span>
<span class="p">};</span>
<span class="ss">web</span> <span class="o">=</span> hardware<span class="p">:</span> idx<span class="p">:</span> imports<span class="p">:</span>
server hardware <span class="s2">"web</span><span class="si">${</span>lib<span class="o">.</span>fixedWidthNumber <span class="mi">2</span> idx<span class="si">}</span><span class="s2">"</span> <span class="p">([</span> <span class="l">./web.nix</span> <span class="p">]</span> <span class="o">++</span> imports<span class="p">);</span>
<span class="k">in</span> <span class="p">{</span>
network<span class="o">.</span><span class="ss">description</span> <span class="o">=</span> <span class="s2">"Luffy infrastructure"</span><span class="p">;</span>
network<span class="o">.</span><span class="ss">enableRollback</span> <span class="o">=</span> <span class="no">true</span><span class="p">;</span>
<span class="ss">defaults</span> <span class="o">=</span> <span class="nb">import</span> <span class="l">./common.nix</span><span class="p">;</span>
<span class="ss">znc01</span> <span class="o">=</span> server <span class="s2">"exoscale"</span> <span class="p">[</span> <span class="l">./znc.nix</span> <span class="p">];</span>
<span class="ss">web01</span> <span class="o">=</span> web <span class="s2">"hetzner"</span> <span class="mi">1</span> <span class="p">[</span> <span class="l">./isso.nix</span> <span class="p">];</span>
<span class="ss">web02</span> <span class="o">=</span> web <span class="s2">"hetzner"</span> <span class="mi">2</span> <span class="p">[];</span>
<span class="p">}</span>
</pre></div>
<h1 id="connecter-le-tout-avec-nix">Connecter le tout avec Nix<a class="headerlink" href="#connecter-le-tout-avec-nix" title="Permanent link"></a></h1>
<p>L’écosystème <em>Nix</em> est une solution unifiée aux différents problèmes liés à la
gestion des logiciels et des configurations. Les <a href="https://web.archive.org/web/2022/https://nixos.org/guides/declarative-and-reproducible-developer-environments.html">environnements de
développement déclaratifs et reproductibles</a> en constituent une caractéristique très intéressante.
Cela ressemble aux environnements virtuels de Python, mais ils ne sont pas
spécifiques à un langage.</p>
<h2 id="courte-introduction-aux-flakes-nix">Courte introduction aux « flakes » Nix<a class="headerlink" href="#courte-introduction-aux-flakes-nix" title="Permanent link"></a></h2>
<p>J’utilise les « <a href="https://nixos.wiki/wiki/Flakes" title="Flakes on NixOS wiki">flakes</a> », une nouvelle fonctionnalité de <em>Nix</em> qui améliore
la reproductibilité en fixant toutes les dépendances et en isolant le processus
de construction. Bien que cette fonctionnalité soit marquée comme
expérimentale<sup id="fnref-experimental"><a class="footnote-ref" href="#fn-experimental">8</a></sup>, elle est de plus en plus populaire et vous pouvez
trouver <code>flake.nix</code> et <code>flake.lock</code> à la racine de certains dépôts.</p>
<p>A titre d’exemple, voici le contenu du <code>flake.nix</code> livré avec <a href="/en/blog/2013-snimpy" title="Snimpy: SNMP & Python">Snimpy</a>, un
outil SNMP interactif pour Python reposant sur <a href="https://www.ibr.cs.tu-bs.de/projects/libsmi/" title="libsmi: library to Access SMI MIB Information">libsmi</a>, une bibliothèque C :</p>
<div class="language-nix codehilite"><pre><span/><span class="p">{</span>
<span class="ss">inputs</span> <span class="o">=</span> <span class="p">{</span>
nixpkgs<span class="o">.</span><span class="ss">url</span> <span class="o">=</span> <span class="s2">"nixpkgs"</span><span class="p">;</span>
flake-utils<span class="o">.</span><span class="ss">url</span> <span class="o">=</span> <span class="s2">"github:numtide/flake-utils"</span><span class="p">;</span>
<span class="p">};</span>
<span class="ss">outputs</span> <span class="o">=</span> <span class="p">{</span> self<span class="p">,</span> <span class="o">...</span> <span class="p">}@</span>inputs<span class="p">:</span>
inputs<span class="o">.</span>flake-utils<span class="o">.</span>lib<span class="o">.</span>eachDefaultSystem <span class="p">(</span>system<span class="p">:</span>
<span class="k">let</span>
<span class="ss">pkgs</span> <span class="o">=</span> inputs<span class="o">.</span>nixpkgs<span class="o">.</span>legacyPackages<span class="o">.</span><span class="s2">"</span><span class="si">${</span>system<span class="si">}</span><span class="s2">"</span><span class="p">;</span>
<span class="k">in</span>
<span class="p">{</span>
<span class="c1"># nix build</span>
packages<span class="o">.</span><span class="ss">default</span> <span class="o">=</span> pkgs<span class="o">.</span>python3Packages<span class="o">.</span>buildPythonPackage <span class="p">{</span>
<span class="ss">name</span> <span class="o">=</span> <span class="s2">"snimpy"</span><span class="p">;</span>
<span class="ss">src</span> <span class="o">=</span> self<span class="p">;</span>
<span class="ss">preConfigure</span> <span class="o">=</span> <span class="s s-Multiline">''echo "1.0.0-0-000000000000" > version.txt''</span><span class="p">;</span>
<span class="ss">checkPhase</span> <span class="o">=</span> <span class="s2">"pytest"</span><span class="p">;</span>
<span class="ss">checkInputs</span> <span class="o">=</span> <span class="k">with</span> pkgs<span class="o">.</span>python3Packages<span class="p">;</span> <span class="p">[</span> pytest mock coverage <span class="p">];</span>
<span class="ss">propagatedBuildInputs</span> <span class="o">=</span> <span class="k">with</span> pkgs<span class="o">.</span>python3Packages<span class="p">;</span> <span class="p">[</span> cffi pysnmp ipython <span class="p">];</span>
<span class="ss">buildInputs</span> <span class="o">=</span> <span class="p">[</span> pkgs<span class="o">.</span>libsmi <span class="p">];</span>
<span class="p">};</span>
<span class="c1"># nix run + nix shell</span>
apps<span class="o">.</span><span class="ss">default</span> <span class="o">=</span> <span class="p">{</span>
<span class="ss">type</span> <span class="o">=</span> <span class="s2">"app"</span><span class="p">;</span>
<span class="ss">program</span> <span class="o">=</span> <span class="s2">"</span><span class="si">${</span>self<span class="o">.</span>packages<span class="o">.</span><span class="s2">"</span><span class="si">${</span>system<span class="si">}</span><span class="s2">"</span><span class="o">.</span>default<span class="si">}</span><span class="s2">/bin/snimpy"</span><span class="p">;</span>
<span class="p">};</span>
<span class="c1"># nix develop</span>
devShells<span class="o">.</span><span class="ss">default</span> <span class="o">=</span> pkgs<span class="o">.</span>mkShell <span class="p">{</span>
<span class="ss">name</span> <span class="o">=</span> <span class="s2">"snimpy-dev"</span><span class="p">;</span>
<span class="ss">buildInputs</span> <span class="o">=</span> <span class="p">[</span>
self<span class="o">.</span>packages<span class="o">.</span><span class="s2">"</span><span class="si">${</span>system<span class="si">}</span><span class="s2">"</span><span class="o">.</span>default<span class="o">.</span>inputDerivation
pkgs<span class="o">.</span>python3Packages<span class="o">.</span>ipython
<span class="p">];</span>
<span class="p">};</span>
<span class="p">});</span>
<span class="p">}</span>
</pre></div>
<p>Si <em>Nix</em> est installé sur votre système :</p>
<ul>
<li><code>nix run github:vincentbernat/snimpy</code> exécute <em>Snimpy</em>,</li>
<li><code>nix shell github:vincentbernat/snimpy</code> fournit un shell avec <em>Snimpy</em> prêt à être utilisé,</li>
<li><code>nix build github:vincentbernat/snimpy</code> construit le paquet Python,</li>
<li><code>nix develop .</code> fournit un shell pour développer autour de <em>Snimpy</em> depuis un
clône du dépôt<sup id="fnref-checkout"><a class="footnote-ref" href="#fn-checkout">9</a></sup>.</li>
</ul>
<p>Pour plus d’informations sur les <em>flakes</em>, regardez le <a href="https://www.tweag.io/blog/2020-05-25-flakes/" title="Nix Flakes, Part 1: An introduction and tutorial">tutoriel de
Tweag</a>.</p>
<h2 id="nix-et-cdktf">Nix et <abbr title="Cloud Development Kit for Terraform">CDKTF</abbr><a class="headerlink" href="#nix-et-cdktf" title="Permanent link"></a></h2>
<p>A la racine du <a href="https://github.com/vincentbernat/cdktf-take1">dépôt que j’utilise pour <abbr title="Cloud Development Kit for Terraform">CDKTF</abbr></a>, il y a un fichier
<a href="https://github.com/vincentbernat/cdktf-take1/blob/main/flake.nix"><code>flake.nix</code></a> pour configurer un shell avec <em>Terraform</em>
et <em><abbr title="Cloud Development Kit for Terraform">CDKTF</abbr></em> installés et avec les variables d’environnement nécessaires pour
automatiser mon infrastructure.</p>
<p>Terraform est déjà présent dans <em>nixpkgs</em>, mais je dois appliquer une rustine sur
le fournisseur <em>Gandi</em>. Ce n’est pas un problème avec Nix !</p>
<div class="language-nix codehilite"><pre><span/><span class="ss">terraform</span> <span class="o">=</span> pkgs<span class="o">.</span>terraform<span class="o">.</span>withPlugins <span class="p">(</span>p<span class="p">:</span> <span class="p">[</span>
p<span class="o">.</span>aws
p<span class="o">.</span>hcloud
p<span class="o">.</span>vultr
<span class="p">(</span>p<span class="o">.</span>gandi<span class="o">.</span>overrideAttrs
<span class="p">(</span>old<span class="p">:</span> <span class="p">{</span>
<span class="ss">src</span> <span class="o">=</span> pkgs<span class="o">.</span>fetchFromGitHub <span class="p">{</span>
<span class="ss">owner</span> <span class="o">=</span> <span class="s2">"vincentbernat"</span><span class="p">;</span>
<span class="ss">repo</span> <span class="o">=</span> <span class="s2">"terraform-provider-gandi"</span><span class="p">;</span>
<span class="ss">rev</span> <span class="o">=</span> <span class="s2">"feature/livedns-key"</span><span class="p">;</span>
<span class="ss">hash</span> <span class="o">=</span> <span class="s2">"sha256-V16BIjo5/rloQ1xTQrdd0snoq1OPuDh3fQNW7kiv/kQ="</span><span class="p">;</span>
<span class="p">};</span>
<span class="p">}))</span>
<span class="p">]);</span>
</pre></div>
<p><em><abbr title="Cloud Development Kit for Terraform">CDKTF</abbr></em> est écrit en TypeScript. J’ai un fichier
<a href="https://github.com/vincentbernat/cdktf-take1/blob/main/package.json"><code>package.json</code></a> avec toutes les dépendances
nécessaires, y compris celles pour utiliser TypeScript comme langage cible :</p>
<div class="language-json codehilite"><pre><span/><span class="p">{</span>
<span class="w"> </span><span class="nt">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"cdktf-take1"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"version"</span><span class="p">:</span><span class="w"> </span><span class="s2">"1.0.0"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"main"</span><span class="p">:</span><span class="w"> </span><span class="s2">"main.js"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"types"</span><span class="p">:</span><span class="w"> </span><span class="s2">"main.ts"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"private"</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"dependencies"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"@types/node"</span><span class="p">:</span><span class="w"> </span><span class="s2">"^14.18.30"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"cdktf"</span><span class="p">:</span><span class="w"> </span><span class="s2">"^0.13.3"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"cdktf-cli"</span><span class="p">:</span><span class="w"> </span><span class="s2">"^0.13.3"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"constructs"</span><span class="p">:</span><span class="w"> </span><span class="s2">"^10.1.151"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"eslint"</span><span class="p">:</span><span class="w"> </span><span class="s2">"^8.27.0"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"prettier"</span><span class="p">:</span><span class="w"> </span><span class="s2">"^2.7.1"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"ts-node"</span><span class="p">:</span><span class="w"> </span><span class="s2">"^10.9.1"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"typescript"</span><span class="p">:</span><span class="w"> </span><span class="s2">"^3.9.10"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"typescript-language-server"</span><span class="p">:</span><span class="w"> </span><span class="s2">"^2.1.0"</span>
<span class="w"> </span><span class="p">}</span>
<span class="p">}</span>
</pre></div>
<p>J’utilise <a href="https://yarnpkg.com/" title="Yarn package manager">Yarn</a> pour obtenir un fichier <a href="https://github.com/vincentbernat/cdktf-take1/blob/main/yarn.lock"><code>yarn.lock</code></a>
qui peut ensuite être utilisé directement pour construire la dérivation
contenant toutes les dépendances :</p>
<div class="language-nix codehilite"><pre><span/><span class="ss">nodeEnv</span> <span class="o">=</span> pkgs<span class="o">.</span>mkYarnModules <span class="p">{</span>
<span class="ss">pname</span> <span class="o">=</span> <span class="s2">"cdktf-take1-js-modules"</span><span class="p">;</span>
<span class="ss">version</span> <span class="o">=</span> <span class="s2">"1.0.0"</span><span class="p">;</span>
<span class="ss">packageJSON</span> <span class="o">=</span> <span class="l">./package.json</span><span class="p">;</span>
<span class="ss">yarnLock</span> <span class="o">=</span> <span class="l">./yarn.lock</span><span class="p">;</span>
<span class="p">};</span>
</pre></div>
<p>L’étape suivant est de générer les fournisseurs <em><abbr title="Cloud Development Kit for Terraform">CDKTF</abbr></em> à partir des
fournisseurs <em>Terraform</em> et de les inclure dans une dérivation :</p>
<div class="language-nix codehilite"><pre><span/><span class="ss">cdktfProviders</span> <span class="o">=</span> pkgs<span class="o">.</span>stdenvNoCC<span class="o">.</span>mkDerivation <span class="p">{</span>
<span class="ss">name</span> <span class="o">=</span> <span class="s2">"cdktf-providers"</span><span class="p">;</span>
<span class="ss">nativeBuildInputs</span> <span class="o">=</span> <span class="p">[</span>
pkgs<span class="o">.</span>nodejs
terraform
<span class="p">];</span>
<span class="ss">src</span> <span class="o">=</span> nix-filter <span class="p">{</span>
<span class="ss">root</span> <span class="o">=</span> <span class="l">./.</span><span class="p">;</span>
<span class="ss">include</span> <span class="o">=</span> <span class="p">[</span> <span class="l">./cdktf.json</span> <span class="l">./tsconfig.json</span> <span class="p">];</span>
<span class="p">};</span>
<span class="ss">buildPhase</span> <span class="o">=</span> <span class="s s-Multiline">''</span>
<span class="s s-Multiline"> export HOME=$(mktemp -d)</span>
<span class="s s-Multiline"> export CHECKPOINT_DISABLE=1</span>
<span class="s s-Multiline"> export DISABLE_VERSION_CHECK=1</span>
<span class="s s-Multiline"> export PATH=</span><span class="si">${</span>nodeEnv<span class="si">}</span><span class="s s-Multiline">/node_modules/.bin:$PATH</span>
<span class="s s-Multiline"> ln -nsf </span><span class="si">${</span>nodeEnv<span class="si">}</span><span class="s s-Multiline">/node_modules node_modules</span>
<span class="s s-Multiline"> # Build all providers we have in terraform</span>
<span class="s s-Multiline"> for provider in $(cd </span><span class="si">${</span>terraform<span class="si">}</span><span class="s s-Multiline">/libexec/terraform-providers; echo */*/*/*); do</span>
<span class="s s-Multiline"> version=</span><span class="se">''$</span><span class="s s-Multiline">{provider##*/}</span>
<span class="s s-Multiline"> provider=</span><span class="se">''$</span><span class="s s-Multiline">{provider%/*}</span>
<span class="s s-Multiline"> echo "Build $provider@$version"</span>
<span class="s s-Multiline"> cdktf provider add --force-local $provider@$version | cat</span>
<span class="s s-Multiline"> done</span>
<span class="s s-Multiline"> echo "Compile TS → JS"</span>
<span class="s s-Multiline"> tsc</span>
<span class="s s-Multiline"> ''</span><span class="p">;</span>
<span class="ss">installPhase</span> <span class="o">=</span> <span class="s s-Multiline">''</span>
<span class="s s-Multiline"> mv .gen $out</span>
<span class="s s-Multiline"> ln -nsf </span><span class="si">${</span>nodeEnv<span class="si">}</span><span class="s s-Multiline">/node_modules $out/node_modules</span>
<span class="s s-Multiline"> ''</span><span class="p">;</span>
<span class="p">};</span>
</pre></div>
<p>Enfin, nous définissons l’environnement de développement :</p>
<div class="language-nix codehilite"><pre><span/>devShells<span class="o">.</span><span class="ss">default</span> <span class="o">=</span> pkgs<span class="o">.</span>mkShell <span class="p">{</span>
<span class="ss">name</span> <span class="o">=</span> <span class="s2">"cdktf-take1"</span><span class="p">;</span>
<span class="ss">buildInputs</span> <span class="o">=</span> <span class="p">[</span>
pkgs<span class="o">.</span>nodejs
pkgs<span class="o">.</span>yarn
terraform
<span class="p">];</span>
<span class="ss">shellHook</span> <span class="o">=</span> <span class="s s-Multiline">''</span>
<span class="s s-Multiline"> # No telemetry</span>
<span class="s s-Multiline"> export CHECKPOINT_DISABLE=1</span>
<span class="s s-Multiline"> # No autoinstall of plugins</span>
<span class="s s-Multiline"> export CDKTF_DISABLE_PLUGIN_CACHE_ENV=1</span>
<span class="s s-Multiline"> # Do not check version</span>
<span class="s s-Multiline"> export DISABLE_VERSION_CHECK=1</span>
<span class="s s-Multiline"> # Access to node modules</span>
<span class="s s-Multiline"> export PATH=$PWD/node_modules/.bin:$PATH</span>
<span class="s s-Multiline"> ln -nsf </span><span class="si">${</span>nodeEnv<span class="si">}</span><span class="s s-Multiline">/node_modules node_modules</span>
<span class="s s-Multiline"> ln -nsf </span><span class="si">${</span>cdktfProviders<span class="si">}</span><span class="s s-Multiline"> .gen</span>
<span class="s s-Multiline"> # Credentials</span>
<span class="s s-Multiline"> for p in \</span>
<span class="s s-Multiline"> njf.nznmba.pbz/Nqzvavfgengbe \</span>
<span class="s s-Multiline"> urgmare.pbz/ivaprag@oreang.pu \</span>
<span class="s s-Multiline"> ihyge.pbz/ihyge@ivaprag.oreang.pu; do</span>
<span class="s s-Multiline"> eval $(pass show $(echo $p | tr 'A-Za-z' 'N-ZA-Mn-za-m') | grep '^export')</span>
<span class="s s-Multiline"> done</span>
<span class="s s-Multiline"> eval $(pass show personal/cdktf/secrets | grep '^export')</span>
<span class="s s-Multiline"> export TF_VAR_hcloudToken="$HCLOUD_TOKEN"</span>
<span class="s s-Multiline"> export TF_VAR_vultrApiKey="$VULTR_API_KEY"</span>
<span class="s s-Multiline"> unset VULTR_API_KEY HCLOUD_TOKEN</span>
<span class="s s-Multiline"> ''</span><span class="p">;</span>
<span class="p">};</span>
</pre></div>
<p>Les dérivations listées dans <code>buildInputs</code> sont disponibles dans le shell
fourni. Le contenu de <code>shellHook</code> est exécuté lors du démarrage du shell. Il
établit des liens symboliques pour rendre disponible l’environnement JavaScript
construit à une étape précédente, ainsi que les fournisseurs <em><abbr title="Cloud Development Kit for Terraform">CDKTF</abbr></em> générés. Il
exporte également toutes les informations d’identification<sup id="fnref-obfuscated"><a class="footnote-ref" href="#fn-obfuscated">10</a></sup>.</p>
<p>J’utilise également <a href="https://github.com/direnv/direnv/" title="direnv: unclutter your .profile">direnv</a> avec un fichier <a href="https://github.com/vincentbernat/cdktf-take1/blob/main/.envrc"><code>.envrc</code></a>
pour passer automatiquement dans l’environnement de développement. Cela permet
également à ce dernier d’être disponible depuis <em>Emacs</em>, notamment lors de
l’utilisation de <a href="https://github.com/emacs-lsp/lsp-mode/" title="Language Server Protocol Support for Emacs">lsp-mode</a> pour obtenir les complétions. <code>nix develop .</code>
permet aussi d’activer manuellement l’environnement.</p>
<p>J’utilise les commandes suivantes pour déployer<sup id="fnref-cdktf-cli"><a class="footnote-ref" href="#fn-cdktf-cli">11</a></sup> :</p>
<div class="language-bash-session codehilite"><pre><span/><span class="gp">$ </span>cdktf<span class="w"> </span>synth
<span class="gp">$ </span><span class="nb">cd</span><span class="w"> </span>cdktf.out/stacks/cdktf-take1
<span class="gp">$ </span>terraform<span class="w"> </span>plan<span class="w"> </span>--out<span class="w"> </span>plan
<span class="gp">$ </span>terraform<span class="w"> </span>apply<span class="w"> </span>plan
<span class="gp">$ </span>terraform<span class="w"> </span>output<span class="w"> </span>-json<span class="w"> </span>><span class="w"> </span>~-automation/nixops-take1/cdktf.json
</pre></div>
<p>La dernière commande produit un fichier JSON contenant les données nécessaires
pour finir le déploiement avec <em>NixOps</em>.</p>
<h2 id="nixops">NixOps<a class="headerlink" href="#nixops" title="Permanent link"></a></h2>
<p>Le <a href="https://github.com/vincentbernat/nixops-take1/blob/master/cdktf.json">fichier JSON</a> exporté par <em>Terraform</em> contient la
liste des serveurs avec quelques attributs :</p>
<div class="language-json codehilite"><pre><span/><span class="p">{</span>
<span class="w"> </span><span class="nt">"hardware"</span><span class="p">:</span><span class="w"> </span><span class="s2">"hetzner"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"ipv4Address"</span><span class="p">:</span><span class="w"> </span><span class="s2">"5.161.44.145"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"ipv6Address"</span><span class="p">:</span><span class="w"> </span><span class="s2">"2a01:4ff:f0:b91::1"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"web05.luffy.cx"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"tags"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span>
<span class="w"> </span><span class="s2">"web"</span><span class="p">,</span>
<span class="w"> </span><span class="s2">"continent:NA"</span><span class="p">,</span>
<span class="w"> </span><span class="s2">"continent:SA"</span>
<span class="w"> </span><span class="p">]</span>
<span class="p">}</span>
</pre></div>
<p>Dans le fichier <a href="https://github.com/vincentbernat/nixops-take1/blob/master/network.nix"><code>network.nix</code></a>, cette liste est
importée et transformée en un ensemble d’attributs décrivant les serveurs. Une
version simplifiée ressemble à cela :</p>
<div class="language-nix codehilite"><pre><span/><span class="k">let</span>
<span class="ss">lib</span> <span class="o">=</span> inputs<span class="o">.</span>nixpkgs<span class="o">.</span>lib<span class="p">;</span>
<span class="ss">shortName</span> <span class="o">=</span> name<span class="p">:</span> <span class="nb">builtins</span><span class="o">.</span>elemAt <span class="p">(</span>lib<span class="o">.</span>splitString <span class="s2">"."</span> name<span class="p">)</span> <span class="mi">0</span><span class="p">;</span>
<span class="ss">domainName</span> <span class="o">=</span> name<span class="p">:</span> lib<span class="o">.</span>concatStringsSep <span class="s2">"."</span> <span class="p">(</span><span class="nb">builtins</span><span class="o">.</span>tail <span class="p">(</span>lib<span class="o">.</span>splitString <span class="s2">"."</span> name<span class="p">));</span>
<span class="ss">server</span> <span class="o">=</span> hardware<span class="p">:</span> name<span class="p">:</span> imports<span class="p">:</span> <span class="p">{</span>
<span class="ss">networking</span> <span class="o">=</span> <span class="p">{</span>
<span class="ss">hostName</span> <span class="o">=</span> shortName name<span class="p">;</span>
<span class="ss">domain</span> <span class="o">=</span> domainName name<span class="p">;</span>
<span class="p">};</span>
deployment<span class="o">.</span><span class="ss">targetHost</span> <span class="o">=</span> name<span class="p">;</span>
<span class="ss">imports</span> <span class="o">=</span> <span class="p">[</span> <span class="p">(</span><span class="l">./hardware/.</span> <span class="o">+</span> <span class="s2">"/</span><span class="si">${</span>hardware<span class="si">}</span><span class="s2">.nix"</span><span class="p">)</span> <span class="p">]</span> <span class="o">++</span> imports<span class="p">;</span>
<span class="p">};</span>
<span class="ss">cdktf-servers-json</span> <span class="o">=</span> <span class="p">(</span>lib<span class="o">.</span>importJSON <span class="l">./cdktf.json</span><span class="p">)</span><span class="o">.</span>servers<span class="o">.</span>value<span class="p">;</span>
<span class="ss">cdktf-servers</span> <span class="o">=</span> <span class="nb">map</span>
<span class="p">(</span>s<span class="p">:</span>
<span class="k">let</span>
<span class="ss">tags-maybe-import</span> <span class="o">=</span> <span class="nb">map</span> <span class="p">(</span>t<span class="p">:</span> <span class="l">./.</span> <span class="o">+</span> <span class="s2">"/</span><span class="si">${</span>t<span class="si">}</span><span class="s2">.nix"</span><span class="p">)</span> s<span class="o">.</span>tags<span class="p">;</span>
<span class="ss">tags-import</span> <span class="o">=</span> <span class="nb">builtins</span><span class="o">.</span>filter <span class="p">(</span>t<span class="p">:</span> <span class="nb">builtins</span><span class="o">.</span>pathExists t<span class="p">)</span> tags-maybe-import<span class="p">;</span>
<span class="k">in</span>
<span class="p">{</span>
<span class="ss">name</span> <span class="o">=</span> shortName s<span class="o">.</span>name<span class="p">;</span>
<span class="ss">value</span> <span class="o">=</span> server s<span class="o">.</span>hardware s<span class="o">.</span>name tags-import<span class="p">;</span>
<span class="p">})</span>
cdktf-servers-json<span class="p">;</span>
<span class="k">in</span>
<span class="p">{</span>
<span class="o">//</span> <span class="p">[</span><span class="err">…</span><span class="p">]</span>
<span class="p">}</span> <span class="o">//</span> <span class="nb">builtins</span><span class="o">.</span>listToAttrs cdktf-servers
</pre></div>
<p>Pour <code>web05</code>, on obtient ceci :</p>
<div class="language-nix codehilite"><pre><span/><span class="ss">web05</span> <span class="o">=</span> <span class="p">{</span>
<span class="ss">networking</span> <span class="o">=</span> <span class="p">{</span>
<span class="ss">hostName</span> <span class="o">=</span> <span class="s2">"web05"</span><span class="p">;</span>
<span class="ss">domainName</span> <span class="o">=</span> <span class="s2">"luffy.cx"</span><span class="p">;</span>
<span class="p">};</span>
deployment<span class="o">.</span><span class="ss">targetHost</span> <span class="o">=</span> <span class="s2">"web05.luffy.cx"</span><span class="p">;</span>
<span class="ss">imports</span> <span class="o">=</span> <span class="p">[</span> <span class="l">./hardware/hetzner.nix</span> <span class="l">./web.nix</span> <span class="p">];</span>
<span class="p">};</span>
</pre></div>
<p>Comme pour <em><abbr title="Cloud Development Kit for Terraform">CDKTF</abbr></em>, à la racine du <a href="https://github.com/vincentbernat/nixops-take1">dépôt que j’utilise pour
NixOps</a>, il y a un fichier <a href="https://github.com/vincentbernat/nixops-take1/blob/master/flake.nix"><code>flake.nix</code></a>
pour fournir un shell avec <em>NixOps</em> configuré. Comme <em>NixOps</em> ne supporte pas
les déploiements progessifs, j’utilise généralement ces commandes pour déployer
sur un unique serveur<sup id="fnref-disable"><a class="footnote-ref" href="#fn-disable">12</a></sup> :</p>
<div class="language-bash-session codehilite"><pre><span/><span class="gp">$ </span>nix<span class="w"> </span>flake<span class="w"> </span>update
<span class="gp">$ </span>nixops<span class="w"> </span>deploy<span class="w"> </span>--include<span class="o">=</span>web04
<span class="gp">$ </span>./tests<span class="w"> </span>web04.luffy.cx
</pre></div>
<p>Si les tests se déroulent sans soucis, je déploie les autres nœuds un par un
avec la commande suivante :</p>
<div class="language-bash-session codehilite"><pre><span/><span class="gp">$ </span><span class="o">(</span><span class="nb">set</span><span class="w"> </span>-e<span class="p">;</span><span class="w"> </span><span class="k">for</span><span class="w"> </span>h<span class="w"> </span><span class="k">in</span><span class="w"> </span>web<span class="o">{</span><span class="m">03</span>..06<span class="o">}</span><span class="p">;</span><span class="w"> </span><span class="k">do</span><span class="w"> </span>nixops<span class="w"> </span>deploy<span class="w"> </span>--include<span class="o">=</span><span class="nv">$h</span><span class="p">;</span><span class="w"> </span><span class="k">done</span><span class="o">)</span>
</pre></div>
<p>La commande <code>nixops deploy</code> déploie tous les serveurs en parallèle et peut donc
provoquer une panne si tous les serveurs <em>Nginx</em> sont indisponibles au même
moment.</p>
<hr/>
<p>Cet article est en chantier depuis trois ans. Le contenu a été mis à jour et
affiné au fur et à mesure de mes expérimentations. Il y a encore beaucoup à
explorer<sup id="fnref-todo"><a class="footnote-ref" href="#fn-todo">13</a></sup>, mais j’estime que le contenu est désormais suffisant pour être
publié ! 🎄</p>
<div class="footnote">
<hr/>
<ol>
<li id="fn-hetzner">
<p>C’était un AMD Athlon 64 X2 5600+ avec 2 Go de RAM et 2 disques de
400 Go en RAID logiciel. Je payais quelque chose autour de 59 € par mois
pour cela. Si c’était une bonne affaire en 2008, en 2018, ce n’était plus
rentable. Il fonctionnait sous Debian Wheezy avec <a href="http://linux-vserver.org/Welcome_to_Linux-VServer.org" title="Linux-VServer">Linux-VServer</a> pour
l’isolation, tous deux dépassés en 2018. <a class="footnote-backref" href="#fnref-hetzner" title="Jump back to footnote 1 in the text">↩︎</a></p>
</li>
<li id="fn-python">
<p>Je n’ai pas non plus utilisé Python car le support de <em>Poetry</em> dans
<em>Nix</em> était <a href="https://github.com/nix-community/poetry2nix/issues/750" title="Issue #750: infinite recursion">cassé</a> quand j’ai commencé à essayer d’utiliser
<em><abbr title="Cloud Development Kit for Terraform">CDKTF</abbr></em>. <a class="footnote-backref" href="#fnref-python" title="Jump back to footnote 2 in the text">↩︎</a></p>
</li>
<li id="fn-transform">
<p><em>Pulumi</em> peut <a href="https://www.pulumi.com/docs/intro/concepts/inputs-outputs/#apply">appliquer des fonctions arbitraires</a> à l’aide de <code>apply()</code>. Cela permet de transformer les
données qui ne sont pas connues lors de l’étape de planification.
<em>Terraform</em> a des <a href="https://developer.hashicorp.com/terraform/language/functions">fonctions</a> pour un usage similaire mais
elles sont plus limitées. <a class="footnote-backref" href="#fnref-transform" title="Jump back to footnote 3 in the text">↩︎</a></p>
</li>
<li id="fn-maint">
<p>Les deux modifications mentionnées ne sont pas encore fusionnées. La
seconde est remplacée par la <a href="https://github.com/pulumi/pulumi-tf-provider-boilerplate/pull/61" title="PR #61: Explicitly specify bash as shell in Makefile">PR #61</a>, soumise deux mois plus
tard, qui impose l’utilisation de <code>/bin/bash</code>. J’ai également soumis la
<a href="https://github.com/pulumi/pulumi-tf-provider-boilerplate/pull/56" title="PR #56: Use go:embed instead of go generate for schema">PR #56</a>, qui a été fusionnée 4 mois plus tard et rapidement
annulée sans explication. <a class="footnote-backref" href="#fnref-maint" title="Jump back to footnote 4 in the text">↩︎</a></p>
</li>
<li id="fn-derivation">
<p>Grosso modo, une dérivation est un synonyme pour paquet dans
l’écosystème <em>Nix</em>. <a class="footnote-backref" href="#fnref-derivation" title="Jump back to footnote 5 in the text">↩︎</a></p>
</li>
<li id="fn-openssl">
<p><em>OpenSSL 3</em> a de nombreux <a href="https://github.com/openssl/openssl/issues/17627#issuecomment-1060123659">problèmes de performance</a>. <a class="footnote-backref" href="#fnref-openssl" title="Jump back to footnote 6 in the text">↩︎</a></p>
</li>
<li id="fn-security">
<p><em>NixOS</em> peut être un peu lent à intégrer les correctifs car il est
nécessaire de reconstruire une partie du cache binaire. Dans ce cas précis,
cela a été rapide : la vulnérabilité et les correctifs ont été publiés le 13
août 2019 et disponibles dans NixOS le 15 août. À titre de comparaison,
Debian n’a publié la version corrigée que le 22 août, ce qui est
inhabituellement tardif. <a class="footnote-backref" href="#fnref-security" title="Jump back to footnote 7 in the text">↩︎</a></p>
</li>
<li id="fn-experimental">
<p>Comme les <em>flakes</em> sont expérimentaux, de nombreuses
documentations ne les utilisent pas et c’est un aspect supplémentaire à
apprendre. <a class="footnote-backref" href="#fnref-experimental" title="Jump back to footnote 8 in the text">↩︎</a></p>
</li>
<li id="fn-checkout">
<p>Comme dans les autres exemples, il est possible de remplacer <code>.</code>
par <code>github:vincentbernat/snimpy</code>. Cependant, obtenir les dépendances de
<em>Snimpy</em> sans son code source a peu d’intérêt. <a class="footnote-backref" href="#fnref-checkout" title="Jump back to footnote 9 in the text">↩︎</a></p>
</li>
<li id="fn-obfuscated">
<p>J’utilise le gestionnaire de mots de passe <a href="https://www.passwordstore.org/" title="The standard Unix password manager">pass</a>. Le nom des
mots de passe sont brouillés uniquement pour éviter les courriers
indésirables. <a class="footnote-backref" href="#fnref-obfuscated" title="Jump back to footnote 10 in the text">↩︎</a></p>
</li>
<li id="fn-cdktf-cli">
<p>La commande <code>cdktf</code> sait appeler les commandes <code>terraform</code>, mais
je préfère les utiliser directement car elles sont plus flexibles. <a class="footnote-backref" href="#fnref-cdktf-cli" title="Jump back to footnote 11 in the text">↩︎</a></p>
</li>
<li id="fn-disable">
<p>Si le changement est risqué, je <a href="https://github.com/vincentbernat/cdktf-take1/commit/606e6fb657179408b163664dd74ff5ab38b28246">désactive le serveur</a> avec <em><abbr title="Cloud Development Kit for Terraform">CDKTF</abbr></em>. Cela le retire des enregistrements DNS. <a class="footnote-backref" href="#fnref-disable" title="Jump back to footnote 12 in the text">↩︎</a></p>
</li>
<li id="fn-todo">
<p>Je voudrais remplacer <em>NixOps</em> avec une alternative gérant les
déploiements progressifs et les tests. Je voudrais aussi passer à <a href="https://www.nomadproject.io/" title="Nomad: Orchestration Made Easy">Nomad</a>
ou <em>Kubernetes</em> pour déployer les applications. <a class="footnote-backref" href="#fnref-todo" title="Jump back to footnote 13 in the text">↩︎</a></p>
</li>
</ol>
</div>