Jekyll2024-01-03T14:58:01+00:00https://www.tech-notes.net/feed.xmlTech NotesUseful notes about DevOps and system engineeringYaroslav YarmoshykUsing git-crypt to encrypt sensitive data in your git repository2024-01-03T00:00:00+00:002024-01-03T00:00:00+00:00https://www.tech-notes.net/git-crypt<p>This is another note of the kind: “I don’t want to google for it next time I need it in few years”</p>
<p>Sometimes we need to store sensitive data in our git repository and it definitely should not be stored in plaintext. This is where <a href="https://github.com/AGWA/git-crypt">git-crypt</a> becomes very usefull. It allows to encrypt plaintext with the help of gpg keys. The enabledment is pretty simple.</p>
<h2 id="gpg-part">GPG part</h2>
<p>First make sure that you have gpg installed of the latest version:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>gpg <span class="nt">--version</span>
</code></pre></div></div>
<p>Create a new key:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>gpg <span class="nt">--full-generate-key</span>
</code></pre></div></div>
<p>Options to be defined:</p>
<ol>
<li>RSA and RSA</li>
<li>4096 bit</li>
<li>defined whatever expiry you want. Normal practice is 1 year.</li>
</ol>
<h2 id="git-repository-part">GIT repository part</h2>
<p>List keys:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>gpg <span class="nt">-k</span>
</code></pre></div></div>
<p>Inside your git repository run the following:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>git-crypt init
git-crypt add-gpg-user USER_ID
</code></pre></div></div>
<p>Specify files to encrypt by creating a <code class="language-plaintext highlighter-rouge">.gitattributes</code> file:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>secret.txt <span class="nv">filter</span><span class="o">=</span>git-crypt <span class="nv">diff</span><span class="o">=</span>git-crypt
<span class="k">*</span>.key <span class="nv">filter</span><span class="o">=</span>git-crypt <span class="nv">diff</span><span class="o">=</span>git-crypt
</code></pre></div></div>
<p>You’ll need to git-add and git-commit the newly created files</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>git add .gitattributes
git add .git-crypt
git commit .git-crypt .gitattributes <span class="nt">-m</span> <span class="s2">"Added git crypt"</span>
git push
</code></pre></div></div>
<p>That is it. The file will showup encrypted in your git repository
<img src="/wp-content/uploads/2024/gpg-encrypted-file.png" alt="gpg-encrypted-file" /></p>
<p>After cloning a repository with encrypted files, unlock with GPG:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>git-crypt unlock
</code></pre></div></div>
<h3 id="adding-new-key-to-repository">Adding new key to repository</h3>
<p>For example you need to add one more contributor to your repository.</p>
<p>New contributor got to share his GPG key by exporting it:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>gpg <span class="nt">--armor</span> <span class="nt">--export</span> USER_ID <span class="o">></span> USER_ID.gpg
</code></pre></div></div>
<p>You’ll need to import it, trust it and add user:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>gpg <span class="nt">--import</span> USER_ID.gpg
</code></pre></div></div>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>gpg ––edit–key D2B3EAAF9A8D5DB93CC30B26CCA243599CC80727B
<span class="o">></span> trust
<span class="o">></span> save
<span class="o">></span> quit
</code></pre></div></div>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>git-crypt add-gpg-user D2B3EAAF9A8D5DB93CC30B26CCA243599CC80727B
</code></pre></div></div>
<h2 id="links">Links:</h2>
<ol>
<li><a href="https://medium.com/@sumitkum/securing-your-secret-keys-with-git-crypt-b2fa6ffed1a6">Securing your secret keys with git-crypt</a></li>
<li><a href="https://github.com/AGWA/git-crypt">AGWA/git-crypt</a></li>
</ol>adminThis is another note of the kind: “I don’t want to google for it next time I need it in few years”GIT cheatsheet2023-09-18T00:00:00+00:002023-09-18T00:00:00+00:00https://www.tech-notes.net/git-cheatsheet<p>This is one of the cheetsheets I have at this website. The primary purpose of this one is to collect all usefull commands you might need in your daily work.
It is worth to note that current changes are being applied to local git repo and need to be published to remote.</p>
<p>I will publish it as is and will extend with additional notes as I have more examples.</p>
<h2 id="reset-not-commited-changes">Reset not commited changes</h2>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>git reset --hard
</code></pre></div></div>
<h2 id="reset-changes-in-the-selected-file">Reset changes in the selected file:</h2>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>git checkout <span class="nt">--</span> file
</code></pre></div></div>
<h2 id="get-commit-history">Get commit history</h2>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>git log <span class="nt">--oneline</span>
</code></pre></div></div>
<p>Get commit history for the specified file:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>git log <span class="nt">--follow</span> <span class="nt">--</span> <span class="k">**</span>filename<span class="k">**</span>
</code></pre></div></div>
<h2 id="revert-to-the-previous-sate-commit-sha">Revert to the previous sate (commit sha)</h2>
<p>find the commit sha from the previous command:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>git checkout <span class="k">**</span>COMMIT_SHA<span class="k">**</span>
</code></pre></div></div>
<p>alternative way to roll back to the previous commit:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>git reset <span class="k">**</span>COMMIT_SHA<span class="k">**</span>
</code></pre></div></div>
<p>In this case all the files will remain modified. So you nedd to reset the changes and force push to remote:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>git reset <span class="nt">--hard</span>
git push <span class="nt">-ff</span>
</code></pre></div></div>adminThis is one of the cheetsheets I have at this website. The primary purpose of this one is to collect all usefull commands you might need in your daily work. It is worth to note that current changes are being applied to local git repo and need to be published to remote.List items should be accessed using square brackets2023-02-17T00:00:00+00:002023-02-17T00:00:00+00:00https://www.tech-notes.net/list-items-square-brackets<p>This is a note about the following error that is produced by <code class="language-plaintext highlighter-rouge">tflint</code></p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Warning: List items should be accessed using square brackets <span class="o">(</span>terraform_deprecated_index<span class="o">)</span>
on ec2.tf line 182:
182: target_id <span class="o">=</span> aws_instance.ec2.<span class="k">*</span>.id[count.index]
Reference: https://github.com/terraform-linters/tflint-ruleset-terraform/blob/v0.2.2/docs/rules/terraform_deprecated_index.md
</code></pre></div></div>
<p>I decided to make this not becuase the explanation at <a href="https://github.com/terraform-linters/tflint-ruleset-terraform/blob/v0.2.2/docs/rules/terraform_deprecated_index.md">github.com/terraform-linters</a> is not very clear.
<br />
The fix is the to replace all occurances of <code class="language-plaintext highlighter-rouge">aws_instance.ec2.*.id[count.index]</code> with <code class="language-plaintext highlighter-rouge">aws_instance.ec2.[count.index].id</code></p>
<p>Another optio is to switch to for loop. Here is an example in optputs:</p>
<div class="language-tcl highlighter-rouge"><div class="highlight"><pre class="highlight"><code>output <span class="s2">"ec2_private_ips"</span> <span class="p">{</span>
value = <span class="p">[</span>for instances in aws_instance.ec2 : instances.private_ip<span class="p">]</span>
<span class="p">}</span>
</code></pre></div></div>Yaroslav YarmoshykThis is a note about the following error that is produced by tflint Warning: List items should be accessed using square brackets (terraform_deprecated_index) on ec2.tf line 182: 182: target_id = aws_instance.ec2.*.id[count.index] Reference: https://github.com/terraform-linters/tflint-ruleset-terraform/blob/v0.2.2/docs/rules/terraform_deprecated_index.mdUse gitlab to store terraform state file2023-01-10T00:00:00+00:002023-01-10T00:00:00+00:00https://www.tech-notes.net/use-gitlab-as-terraform-backend<p>This is a short note about the way to configure your GitlabCI to store the Terraform state file.
Terraform backend configuration remains pretty simple:</p>
<div class="language-terraform highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">terraform</span> <span class="p">{</span>
<span class="nx">backend</span> <span class="s2">"http"</span> <span class="p">{</span>
<span class="nx">skip_cert_verification</span> <span class="p">=</span> <span class="kc">true</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>The following variables need to be defined in your <code class="language-plaintext highlighter-rouge">.gitlab-ci.yml</code> file:</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">variables</span><span class="pi">:</span>
<span class="na">TF_HTTP_ADDRESS</span><span class="pi">:</span> <span class="s2">"</span><span class="s">${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/terraform/state/statefilename"</span>
<span class="na">TF_HTTP_LOCK_ADDRESS</span><span class="pi">:</span> <span class="s2">"</span><span class="s">${TF_HTTP_ADDRESS}/lock"</span>
<span class="na">TF_HTTP_UNLOCK_ADDRESS</span><span class="pi">:</span> <span class="s2">"</span><span class="s">${TF_HTTP_ADDRESS}/lock"</span>
<span class="na">TF_HTTP_USERNAME</span><span class="pi">:</span> <span class="s2">"</span><span class="s">gitlab-ci-token"</span>
<span class="na">TF_HTTP_PASSWORD</span><span class="pi">:</span> <span class="s2">"</span><span class="s">${CI_JOB_TOKEN}"</span>
<span class="na">TF_HTTP_LOCK_METHOD</span><span class="pi">:</span> <span class="s2">"</span><span class="s">POST"</span>
<span class="na">TF_HTTP_UNLOCK_METHOD</span><span class="pi">:</span> <span class="s2">"</span><span class="s">DELETE"</span>
</code></pre></div></div>
<p>Basically this will work. You don’t need to modify anything.</p>
<p>However you might want to use different state files for every environment. In this case you can replace the <code class="language-plaintext highlighter-rouge">statefilename</code> in the <code class="language-plaintext highlighter-rouge">TF_HTTP_ADDRESS</code> with the <code class="language-plaintext highlighter-rouge">$CI_ENVIRONMENT_SLUG</code> which is the pre-defined variable that contains the name of your environment. Please note that this variable will eist only if you have thefollowing block in you per-environment jobs:</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">environment</span><span class="pi">:</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">dev</span>
</code></pre></div></div>Yaroslav YarmoshykThis is a short note about the way to configure your GitlabCI to store the Terraform state file. Terraform backend configuration remains pretty simple: terraform { backend "http" { skip_cert_verification = true } }Start and Stop Ec2 Instances with CloudWatch Event rule and Lambda2022-11-04T00:00:00+00:002022-11-04T00:00:00+00:00https://www.tech-notes.net/start-stop-ec2-with-lambda-cloudwatch-rule<p>This note can be usefull for those of you who needs to hav EC2 instance running during the certain period of time during the day and awoid spending money when the instance is not needed.
I use the <a href="https://bmwitcher.medium.com/using-lambda-cloudwatch-events-to-start-and-stop-ec2-instances-48e31ff0daf2">bmwitcher.medium.com</a> article but with a small changes:</p>
<ol>
<li>There is only one lambda function</li>
<li>The cloudwatch event rules have custom contraint in json format to be forwarded as event to the lambda</li>
</ol>
<h2 id="steps-are-the-following">Steps are the following</h2>
<ol>
<li>Create lambda function with the code specified below
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">boto3</span>
<span class="n">region</span> <span class="o">=</span> <span class="s">'us-east-1'</span>
<span class="n">ec2</span> <span class="o">=</span> <span class="n">boto3</span><span class="p">.</span><span class="n">client</span><span class="p">(</span><span class="s">'ec2'</span><span class="p">,</span> <span class="n">region_name</span><span class="o">=</span><span class="n">region</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">lambda_handler</span><span class="p">(</span><span class="n">event</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
<span class="k">if</span> <span class="n">event</span><span class="p">[</span><span class="s">"action"</span><span class="p">]</span> <span class="o">==</span> <span class="s">"stop"</span><span class="p">:</span>
<span class="n">ec2</span><span class="p">.</span><span class="n">stop_instances</span><span class="p">(</span><span class="n">InstanceIds</span><span class="o">=</span><span class="p">[</span><span class="n">event</span><span class="p">[</span><span class="s">"instance_id"</span><span class="p">]])</span>
<span class="k">if</span> <span class="n">event</span><span class="p">[</span><span class="s">"action"</span><span class="p">]</span> <span class="o">==</span> <span class="s">"start"</span><span class="p">:</span>
<span class="n">ec2</span><span class="p">.</span><span class="n">start_instances</span><span class="p">(</span><span class="n">InstanceIds</span><span class="o">=</span><span class="p">[</span><span class="n">event</span><span class="p">[</span><span class="s">"instance_id"</span><span class="p">]])</span>
</code></pre></div> </div>
</li>
<li>Create the following IAM policy
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="nl">"Version"</span><span class="p">:</span><span class="w"> </span><span class="s2">"2012-10-17"</span><span class="p">,</span><span class="w">
</span><span class="nl">"Statement"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="nl">"Action"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="s2">"ec2:DescribeInstances"</span><span class="w">
</span><span class="p">],</span><span class="w">
</span><span class="nl">"Resource"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="s2">"*"</span><span class="w">
</span><span class="p">],</span><span class="w">
</span><span class="nl">"Effect"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Allow"</span><span class="p">,</span><span class="w">
</span><span class="nl">"Condition"</span><span class="p">:</span><span class="w"> </span><span class="p">{}</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="nl">"Action"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="s2">"ec2:StartInstances"</span><span class="p">,</span><span class="w">
</span><span class="s2">"ec2:StopInstances"</span><span class="w">
</span><span class="p">],</span><span class="w">
</span><span class="nl">"Resource"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="s2">"arn:aws:ec2:::instance/*"</span><span class="w">
</span><span class="p">],</span><span class="w">
</span><span class="nl">"Effect"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Allow"</span><span class="p">,</span><span class="w">
</span><span class="nl">"Condition"</span><span class="p">:</span><span class="w"> </span><span class="p">{}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">]</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div> </div>
</li>
<li>
<p>Attach this policy to the lambda execution IAM role. The role name can be found on the lambda configuration tab
<img src="/wp-content/uploads/2022/lambda_execution_role.png" alt="lambda execution role" title="lambda execution role" /></p>
</li>
<li>Create 2 separarte CloudWatch rules to stop and start instances at the spcified schedule
<img src="/wp-content/uploads/2022/cw_create_rule.png" alt="CloudWatch create event rule" title="CloudWatch create event rule" /></li>
</ol>
<h2 id="links">Links</h2>
<ul>
<li><a href="https://bmwitcher.medium.com/using-lambda-cloudwatch-events-to-start-and-stop-ec2-instances-48e31ff0daf2">bmwitcher.medium.com</a></li>
</ul>Yaroslav YarmoshykThis note can be usefull for those of you who needs to hav EC2 instance running during the certain period of time during the day and awoid spending money when the instance is not needed. I use the bmwitcher.medium.com article but with a small changes: There is only one lambda function The cloudwatch event rules have custom contraint in json format to be forwarded as event to the lambdaFind neighbors of the matrix element at the given position2022-08-16T00:00:00+00:002022-08-16T00:00:00+00:00https://www.tech-notes.net/find-neighbours-in-matrix<p>I force myself to learn something but before learning something new I need to memorize what I knew before. So I take python labs at <a href="https://exercism.org/">exercism.org</a>.</p>
<p>In one of the labs I had a chllange to find the neighour elements of the matrix element at the given position so I decided to save the solution for further refference.</p>
<p>For example given the matrix:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>matrix = [
"123",
"456",
"789"
]
</code></pre></div></div>
<p>I need to build the function that takes the the following arguments:</p>
<ul>
<li>matrix</li>
<li>element position (x, y as a separate arguments)</li>
</ul>
<p>The function should return all the elements arround the specified coordinates.</p>
<p>For example:</p>
<ul>
<li>if I specify <code class="language-plaintext highlighter-rouge">x=0</code>, <code class="language-plaintext highlighter-rouge">y=1</code> that corrensponds with the “2” the function should return 1,3,4,5,6</li>
<li>if I specify <code class="language-plaintext highlighter-rouge">x=1</code>, <code class="language-plaintext highlighter-rouge">y=1</code> that corrensponds with the “5” the function should return 1,3,4,6,7,8,9</li>
</ul>
<p>The logics is fairy simple:</p>
<ul>
<li>loop over rows in range x-1, x+2</li>
<li>loop over columns in range y-1, y+2</li>
<li>row index should be <code class="language-plaintext highlighter-rouge">>0</code> and <code class="language-plaintext highlighter-rouge"><len(row) </code></li>
<li>column index should be <code class="language-plaintext highlighter-rouge">>0</code> and <code class="language-plaintext highlighter-rouge"><num(columns)</code> or <code class="language-plaintext highlighter-rouge"><len(row[0])</code></li>
<li>exclude the element at specified position</li>
</ul>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">get_neighbors</span><span class="p">(</span><span class="n">matrix</span><span class="p">,</span> <span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">):</span>
<span class="n">num_rows</span><span class="p">,</span> <span class="n">num_cols</span> <span class="o">=</span> <span class="nb">len</span><span class="p">(</span><span class="n">matrix</span><span class="p">),</span> <span class="nb">len</span><span class="p">(</span><span class="n">matrix</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span> <span class="p">(</span><span class="mi">0</span> <span class="k">if</span> <span class="n">x</span><span class="o">-</span><span class="mi">1</span> <span class="o"><</span> <span class="mi">0</span> <span class="k">else</span> <span class="n">x</span><span class="o">-</span><span class="mi">1</span><span class="p">),</span> <span class="p">(</span><span class="n">num_rows</span> <span class="k">if</span> <span class="n">x</span><span class="o">+</span><span class="mi">2</span> <span class="o">></span> <span class="n">num_rows</span> <span class="k">else</span> <span class="n">x</span><span class="o">+</span><span class="mi">2</span><span class="p">),</span> <span class="mi">1</span> <span class="p">):</span>
<span class="k">for</span> <span class="n">j</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span> <span class="p">(</span><span class="mi">0</span> <span class="k">if</span> <span class="n">y</span><span class="o">-</span><span class="mi">1</span> <span class="o"><</span> <span class="mi">0</span> <span class="k">else</span> <span class="n">y</span><span class="o">-</span><span class="mi">1</span><span class="p">),</span> <span class="p">(</span><span class="n">num_cols</span> <span class="k">if</span> <span class="n">y</span><span class="o">+</span><span class="mi">2</span> <span class="o">></span> <span class="n">num_cols</span> <span class="k">else</span> <span class="n">y</span><span class="o">+</span><span class="mi">2</span><span class="p">),</span> <span class="mi">1</span> <span class="p">):</span>
<span class="k">if</span> <span class="n">matrix</span><span class="p">[</span><span class="n">x</span><span class="p">][</span><span class="n">y</span><span class="p">]</span> <span class="o">!=</span> <span class="n">matrix</span><span class="p">[</span><span class="n">i</span><span class="p">][</span><span class="n">j</span><span class="p">]:</span>
<span class="k">print</span><span class="p">(</span><span class="s">" • matrix["</span><span class="o">+</span><span class="nb">str</span><span class="p">(</span><span class="n">i</span><span class="p">)</span><span class="o">+</span><span class="s">"]["</span><span class="o">+</span><span class="nb">str</span><span class="p">(</span><span class="n">j</span><span class="p">)</span><span class="o">+</span><span class="s">"] = "</span><span class="o">+</span><span class="nb">str</span><span class="p">(</span><span class="n">matrix</span><span class="p">[</span><span class="n">i</span><span class="p">][</span><span class="n">j</span><span class="p">]))</span>
</code></pre></div></div>
<p>This example prints the values but it is possible to build the <code class="language-plaintext highlighter-rouge">list()</code> with the discovered values.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">get_neighbors</span><span class="p">(</span><span class="n">matrix</span><span class="p">,</span> <span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">):</span>
<span class="n">num_rows</span><span class="p">,</span> <span class="n">num_cols</span> <span class="o">=</span> <span class="nb">len</span><span class="p">(</span><span class="n">matrix</span><span class="p">),</span> <span class="nb">len</span><span class="p">(</span><span class="n">matrix</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span>
<span class="n">result</span> <span class="o">=</span><span class="p">[]</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span> <span class="p">(</span><span class="mi">0</span> <span class="k">if</span> <span class="n">x</span><span class="o">-</span><span class="mi">1</span> <span class="o"><</span> <span class="mi">0</span> <span class="k">else</span> <span class="n">x</span><span class="o">-</span><span class="mi">1</span><span class="p">),</span> <span class="p">(</span><span class="n">num_rows</span> <span class="k">if</span> <span class="n">x</span><span class="o">+</span><span class="mi">2</span> <span class="o">></span> <span class="n">num_rows</span> <span class="k">else</span> <span class="n">x</span><span class="o">+</span><span class="mi">2</span><span class="p">),</span> <span class="mi">1</span> <span class="p">):</span>
<span class="k">for</span> <span class="n">j</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span> <span class="p">(</span><span class="mi">0</span> <span class="k">if</span> <span class="n">y</span><span class="o">-</span><span class="mi">1</span> <span class="o"><</span> <span class="mi">0</span> <span class="k">else</span> <span class="n">y</span><span class="o">-</span><span class="mi">1</span><span class="p">),</span> <span class="p">(</span><span class="n">num_cols</span> <span class="k">if</span> <span class="n">y</span><span class="o">+</span><span class="mi">2</span> <span class="o">></span> <span class="n">num_cols</span> <span class="k">else</span> <span class="n">y</span><span class="o">+</span><span class="mi">2</span><span class="p">),</span> <span class="mi">1</span> <span class="p">):</span>
<span class="k">if</span> <span class="n">matrix</span><span class="p">[</span><span class="n">x</span><span class="p">][</span><span class="n">y</span><span class="p">]</span> <span class="o">!=</span> <span class="n">matrix</span><span class="p">[</span><span class="n">i</span><span class="p">][</span><span class="n">j</span><span class="p">]:</span>
<span class="n">result</span><span class="p">.</span><span class="n">append</span><span class="p">(</span><span class="n">matrix</span><span class="p">[</span><span class="n">i</span><span class="p">][</span><span class="n">j</span><span class="p">])</span>
<span class="k">return</span> <span class="n">result</span>
</code></pre></div></div>
<p>Another option is to build the <code class="language-plaintext highlighter-rouge">dict()</code> object using the discovered value as a key and cooridinates as a values.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">get_neighbors</span><span class="p">(</span><span class="n">matrix</span><span class="p">,</span> <span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">):</span>
<span class="n">num_rows</span><span class="p">,</span> <span class="n">num_cols</span> <span class="o">=</span> <span class="nb">len</span><span class="p">(</span><span class="n">matrix</span><span class="p">),</span> <span class="nb">len</span><span class="p">(</span><span class="n">matrix</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span>
<span class="n">result</span> <span class="o">=</span> <span class="nb">dict</span><span class="p">()</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span> <span class="p">(</span><span class="mi">0</span> <span class="k">if</span> <span class="n">x</span><span class="o">-</span><span class="mi">1</span> <span class="o"><</span> <span class="mi">0</span> <span class="k">else</span> <span class="n">x</span><span class="o">-</span><span class="mi">1</span><span class="p">),</span> <span class="p">(</span><span class="n">num_rows</span> <span class="k">if</span> <span class="n">x</span><span class="o">+</span><span class="mi">2</span> <span class="o">></span> <span class="n">num_rows</span> <span class="k">else</span> <span class="n">x</span><span class="o">+</span><span class="mi">2</span><span class="p">),</span> <span class="mi">1</span> <span class="p">):</span>
<span class="k">for</span> <span class="n">j</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span> <span class="p">(</span><span class="mi">0</span> <span class="k">if</span> <span class="n">y</span><span class="o">-</span><span class="mi">1</span> <span class="o"><</span> <span class="mi">0</span> <span class="k">else</span> <span class="n">y</span><span class="o">-</span><span class="mi">1</span><span class="p">),</span> <span class="p">(</span><span class="n">num_cols</span> <span class="k">if</span> <span class="n">y</span><span class="o">+</span><span class="mi">2</span> <span class="o">></span> <span class="n">num_cols</span> <span class="k">else</span> <span class="n">y</span><span class="o">+</span><span class="mi">2</span><span class="p">),</span> <span class="mi">1</span> <span class="p">):</span>
<span class="k">if</span> <span class="n">matrix</span><span class="p">[</span><span class="n">x</span><span class="p">][</span><span class="n">y</span><span class="p">]</span> <span class="o">!=</span> <span class="n">matrix</span><span class="p">[</span><span class="n">i</span><span class="p">][</span><span class="n">j</span><span class="p">]:</span>
<span class="n">result</span><span class="p">[</span><span class="n">matrix</span><span class="p">[</span><span class="n">i</span><span class="p">][</span><span class="n">j</span><span class="p">]]</span> <span class="o">=</span> <span class="p">[</span><span class="n">i</span><span class="p">,</span><span class="n">j</span><span class="p">]</span>
<span class="k">return</span> <span class="n">result</span>
</code></pre></div></div>
<p>Or you can do whatever modifications with the original mattrix you’d like to inside this function and return it as a result.</p>adminI force myself to learn something but before learning something new I need to memorize what I knew before. So I take python labs at exercism.org.Refresh Jenkins job properties2022-07-18T00:00:00+00:002022-07-18T00:00:00+00:00https://www.tech-notes.net/refresh-jenkins-job-parameters<p>When you are dialing with groovy pipelines in Jenkins sometimes you might need to add new parameters. The tricky thing is in fact that job parameters are being updated only after the execution. With this in mind you can add a dummy stage to be used to run in jenkins after you add new job parameters into the jenkins file.</p>
<p>The step with condition looks like the following:</p>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">if</span> <span class="o">(</span><span class="n">Refresh</span><span class="o">)</span> <span class="o">{</span>
<span class="n">currentBuild</span><span class="o">.</span><span class="na">result</span> <span class="o">=</span> <span class="s1">'ABORTED'</span>
<span class="n">error</span><span class="o">(</span><span class="s1">'Stopping early…'</span><span class="o">)</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Here is the example of pipeline that has it. There is a bolean parameter <code class="language-plaintext highlighter-rouge">refresh</code> that is going to be represented as a checkbox on the pipeline build screen.
The stage above will be executed when the checkbox is selected (<code class="language-plaintext highlighter-rouge">refresh == True</code>). The stage will not do enything except loading jenkins file and exiting the pipeline with the status = ABORTED. In the jenkins UI the build will be colored in grey.</p>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">pipeline</span> <span class="o">{</span>
<span class="n">agent</span> <span class="n">any</span>
<span class="n">parameters</span> <span class="o">{</span>
<span class="n">booleanParam</span><span class="o">(</span><span class="nl">name:</span> <span class="s1">'refresh'</span><span class="o">,</span> <span class="nl">defaultValue:</span> <span class="kc">false</span><span class="o">,</span> <span class="nl">description:</span> <span class="s1">'Refresh pipeline properties'</span><span class="o">)</span>
<span class="o">}</span>
<span class="n">stages</span> <span class="o">{</span>
<span class="n">stage</span><span class="o">(</span><span class="s1">'Refresh properties'</span><span class="o">)</span> <span class="o">{</span>
<span class="k">if</span> <span class="o">(</span><span class="n">Refresh</span><span class="o">)</span> <span class="o">{</span>
<span class="n">currentBuild</span><span class="o">.</span><span class="na">result</span> <span class="o">=</span> <span class="s1">'ABORTED'</span>
<span class="n">error</span><span class="o">(</span><span class="s1">'Stopping early…'</span><span class="o">)</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>Yaroslav YarmoshykRefresh Jenkins job propertiesCheatsheet for Groovy to be used in Jenkins pipelines2022-07-16T00:00:00+00:002022-07-15T00:00:00+00:00https://www.tech-notes.net/groovy-cheatsheet<p>Groovy is an object-oriented programming language used for JVM platform. This dynamic language has a lot of features like processing lists and arrays, loops and parralel execution. It is very usefull when you are working with Jenkins to automate your CI/CD pipelines.</p>
<p>I will not go deap into details. Obviously you are looking for some examples here.</p>
<h2 id="variables-definition">Variables definition</h2>
<p>Define global variable for your pipeline script:</p>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">foo</span> <span class="o">=</span> <span class="s2">"bar"</span>
</code></pre></div></div>
<p>Define local variable for the current scope (stage, step, script)</p>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kt">def</span> <span class="n">qwe</span> <span class="o">=</span> <span class="s2">"rty"</span>
</code></pre></div></div>
<p>Define environment variables</p>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">env</span><span class="o">.</span><span class="na">AWS_DEFAULT_REGION</span> <span class="o">=</span> <span class="s2">"us-west-1"</span>
</code></pre></div></div>
<p>another option is to use environment block:</p>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">environmnt</span> <span class="o">{</span>
<span class="n">AWS_DEFAULT_REGION</span> <span class="o">=</span> <span class="s2">"us-west-1"</span>
<span class="n">profile</span> <span class="o">=</span> <span class="s2">"dev"</span>
<span class="o">}</span>
</code></pre></div></div>
<h2 id="modify-the-value-of-the-variable">Modify the value of the variable</h2>
<p>Let’s consider the following example:</p>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kt">def</span> <span class="n">myvar</span> <span class="o">=</span> <span class="s2">"my-custom-value"</span>
</code></pre></div></div>
<p>replace all <code class="language-plaintext highlighter-rouge">-</code> with <code class="language-plaintext highlighter-rouge">+</code>. The following will result into <code class="language-plaintext highlighter-rouge">my+custom+value</code></p>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">myvar</span><span class="o">.</span><span class="na">replaceAll</span><span class="o">(</span><span class="s2">"-"</span><span class="o">,</span> <span class="s2">"+"</span><span class="o">)</span>
</code></pre></div></div>
<h2 id="split-value">Split value</h2>
<p>In order to split the string we have the following options:</p>
<ul>
<li>tokenize() returns a list, will ignore empty string</li>
<li>split() returns multiple strings</li>
</ul>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">myvar</span><span class="o">.</span><span class="na">split</span><span class="o">(</span><span class="s2">"-"</span><span class="o">)</span>
</code></pre></div></div>
<p>Will result into the following:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>my
custom
value
</code></pre></div></div>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">myvar</span><span class="o">.</span><span class="na">tokenize</span><span class="o">(</span><span class="s2">"-"</span><span class="o">)</span>
</code></pre></div></div>
<p>Will result into the following:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">[</span><span class="s1">'my'</span>, <span class="s1">'custom'</span>, <span class="s1">'value'</span><span class="o">]</span>
</code></pre></div></div>
<h2 id="collections">Collections</h2>
<p>Let’s create a list value from myvar string:</p>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kt">def</span> <span class="n">myList</span> <span class="o">=</span> <span class="n">myvar</span><span class="o">.</span><span class="na">tokenize</span><span class="o">(</span><span class="s2">"-"</span><span class="o">)</span>
</code></pre></div></div>
<p>Check if the list in not empty:</p>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">if</span><span class="o">(</span><span class="n">myList</span><span class="o">)</span> <span class="o">{</span>
<span class="o">...</span> <span class="k">do</span> <span class="n">something</span> <span class="o">...</span>
<span class="o">}</span>
</code></pre></div></div>
<p>get the size of array:</p>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">myList</span><span class="o">.</span><span class="na">size</span><span class="o">()</span>
</code></pre></div></div>
<p>Access elements in array</p>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">myList</span><span class="o">[</span><span class="mi">0</span><span class="o">]</span> <span class="c1">// first element</span>
<span class="n">myList</span><span class="o">[</span><span class="mi">1</span><span class="o">,</span><span class="mi">2</span><span class="o">,-</span><span class="mi">1</span><span class="o">]</span> <span class="c1">//second, third and first element</span>
</code></pre></div></div>
<p>loop over array and check if one array contains elements from other array</p>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// define new array</span>
<span class="n">newList</span> <span class="o">=</span> <span class="o">[</span><span class="s1">'my'</span><span class="o">,</span> <span class="s1">'list'</span> <span class="o">]</span>
<span class="c1">// loop over new array</span>
<span class="n">newList</span><span class="o">.</span><span class="na">each</span> <span class="o">{</span>
<span class="c1">// check if element exists in first array</span>
<span class="k">if</span> <span class="n">myList</span><span class="o">.</span><span class="na">contains</span><span class="o">(</span><span class="n">it</span><span class="o">)</span> <span class="o">{</span>
<span class="c1">// do something</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="c1">// If you need indexes</span>
<span class="n">newList</span><span class="o">.</span><span class="na">eachWithIndex</span><span class="o">{</span>
<span class="n">element</span><span class="o">,</span> <span class="n">index</span> <span class="o">-></span> <span class="k">assert</span> <span class="n">myList</span><span class="o">[</span><span class="n">index</span><span class="o">]</span> <span class="o">==</span> <span class="n">element</span>
<span class="o">}</span>
</code></pre></div></div>
<p>More <a href="https://www.tutorialspoint.com/groovy/groovy_lists.htm">operations with collections</a></p>
<h2 id="searching-over-array">Searching over array</h2>
<ul>
<li><code class="language-plaintext highlighter-rouge">find()</code> - returns the first appearance of a search request. Returns <code class="language-plaintext highlighter-rouge">Str</code></li>
<li><code class="language-plaintext highlighter-rouge">findAll()</code> - returns all appearances of a search request. Returns <code class="language-plaintext highlighter-rouge">List</code></li>
</ul>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">myList</span><span class="o">.</span><span class="na">findAll</span><span class="o">(</span><span class="s2">"my"</span><span class="o">)</span>
</code></pre></div></div>
<p>in case of numeric search:</p>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kt">def</span> <span class="n">intArray</span> <span class="o">=</span> <span class="o">[</span><span class="s2">"1"</span><span class="o">,</span> <span class="s2">"2"</span><span class="o">,</span> <span class="s2">"3"</span><span class="o">,</span> <span class="s2">"4"</span><span class="o">]</span>
<span class="n">intArray</span><span class="o">.</span><span class="na">findAll</span> <span class="o">{</span> <span class="n">it</span> <span class="o">></span> <span class="s2">"2"</span> <span class="o">}</span> <span class="c1">// returns ["3", "4"]</span>
</code></pre></div></div>
<p>add element to array:</p>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kt">def</span> <span class="n">intArray</span> <span class="o">=</span> <span class="o">[</span><span class="s2">"1"</span><span class="o">,</span> <span class="s2">"2"</span><span class="o">,</span> <span class="s2">"3"</span><span class="o">,</span> <span class="s2">"4"</span><span class="o">]</span>
<span class="n">intArray</span><span class="o">.</span><span class="na">add</span><span class="o">(</span><span class="s2">"5"</span><span class="o">)</span>
</code></pre></div></div>
<p><a href="http://docs.groovy-lang.org/next/html/documentation/working-with-collections.html#Collections-Lists">List methods</a> available in Groovy are another benefit.</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">Add()</code> Append the new value to the end of this list.</li>
<li><code class="language-plaintext highlighter-rouge">Get()</code> Returns the element at the specified position in this list.</li>
<li><code class="language-plaintext highlighter-rouge">Contains()</code> Returns true if this list contains the specified value.</li>
<li><code class="language-plaintext highlighter-rouge">Minus()</code> Create a new list of original elements that removes the specified element</li>
<li><code class="language-plaintext highlighter-rouge">Plus()</code> Create a new list of the original list elements and the specified elements.</li>
<li><code class="language-plaintext highlighter-rouge">Pop()</code> Remove the last item from this list</li>
<li><code class="language-plaintext highlighter-rouge">Remove()</code> Remove elements from the specified position in the list</li>
<li><code class="language-plaintext highlighter-rouge">Reverse()</code> Create a new list that is the opposite of the original list’s elements</li>
<li><code class="language-plaintext highlighter-rouge">Size()</code> Get the number of elements in this list.</li>
<li><code class="language-plaintext highlighter-rouge">Sort()</code> Returns a sorted copy of the original list</li>
</ul>
<h2 id="working-with-maps">Working with maps</h2>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kt">def</span> <span class="n">myMap</span> <span class="o">=</span> <span class="o">[</span>
<span class="s2">"name"</span><span class="o">:</span> <span class="s2">"Elon Mask"</span><span class="o">,</span>
<span class="s2">"gender"</span><span class="o">:</span> <span class="s2">"male"</span><span class="o">,</span>
<span class="s2">"occupation"</span><span class="o">:</span> <span class="s2">"business magnate"</span>
<span class="o">]</span>
<span class="k">if</span><span class="o">(!</span><span class="n">myMap</span><span class="o">.</span><span class="na">car</span><span class="o">)</span> <span class="o">{</span>
<span class="n">println</span> <span class="s2">"Returns true true if cat key is in the list, otherwise return false"</span>
<span class="o">}</span>
</code></pre></div></div>
<p>another example with bit more nested map:</p>
<pre><code class="language-grovy">def myMap = [
"a": [
"id": "1",
"name": "foo"
],
"b": [
"id": "2",
"name": "bar"
],
"c": [
"id": "3",
"name": "foo"
]
]
</code></pre>
<p>Find first appearance of name==foo.
‘?’ ensures that the following argument won’t be executed if the result is null</p>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">myMap</span><span class="o">.</span><span class="na">find</span> <span class="o">{</span> <span class="n">it</span><span class="o">.</span><span class="na">value</span><span class="o">.</span><span class="na">name</span> <span class="o">==</span> <span class="s2">"foo"</span> <span class="o">}?.</span><span class="na">key</span> <span class="c1">//returns "a"</span>
</code></pre></div></div>
<p>Find all appearances of name==foo, list and grabs only specific components of nested elements, list</p>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">myMap</span><span class="o">.</span><span class="na">findAll</span> <span class="o">{</span> <span class="n">it</span><span class="o">.</span><span class="na">value</span><span class="o">.</span><span class="na">name</span> <span class="o">==</span> <span class="s2">"foo"</span> <span class="o">}.</span><span class="na">collect</span> <span class="o">{</span> <span class="n">it</span><span class="o">.</span><span class="na">value</span><span class="o">.</span><span class="na">id</span><span class="o">}</span> <span class="c1">//returns ["1", "3"]</span>
</code></pre></div></div>
<p>Access map elements</p>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">myMap</span><span class="o">.</span><span class="na">a</span><span class="o">.</span><span class="na">id</span> <span class="c1">//returns "1"</span>
<span class="n">myMap</span><span class="o">[</span><span class="s2">"a"</span><span class="o">][</span><span class="s2">"id"</span><span class="o">]</span> <span class="c1">//returns "1"</span>
</code></pre></div></div>
<p><a href="http://docs.groovy-lang.org/next/html/documentation/working-with-collections.html#Collections-Lists">Map methods</a> available in Groovy are another benefit.</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">containsKey()</code> Does this Map contain this key. <a href="https://www.tutorialspoint.com/groovy/groovy_containskey.htm#">Example</a></li>
<li><code class="language-plaintext highlighter-rouge">get()</code> Look up the key in this Map and return the corresponding value. If there is no entry in this Map for the key, then return null.</li>
<li><code class="language-plaintext highlighter-rouge">keySet()</code> Obtain a Set of the keys in this Map <a href="https://www.tutorialspoint.com/groovy/groovy_keyset.htm">Example</a></li>
<li><code class="language-plaintext highlighter-rouge">put()</code> Associates the specified value with the specified key in this Map. <a href="https://www.tutorialspoint.com/groovy/groovy_put.htm">Example</a></li>
<li><code class="language-plaintext highlighter-rouge">size()</code> Returns the number of key-value mappings in this Map.</li>
<li><code class="language-plaintext highlighter-rouge">values()</code> Returns a collection view of the values contained in this Map. <a href="https://www.tutorialspoint.com/groovy/groovy_values.htm">Example</a></li>
</ul>
<h2 id="conditions">Conditions</h2>
<h3 id="switchcase-is-faster-and-are-much-easier-to-read">Switch/case is faster and are much easier to read</h3>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">weekDays</span> <span class="o">=</span> <span class="o">[</span>
<span class="s2">"Monday"</span><span class="o">,</span>
<span class="s2">"Tuesday"</span><span class="o">,</span>
<span class="s2">"Wednesday"</span><span class="o">,</span>
<span class="s2">"Thursday"</span><span class="o">,</span>
<span class="s2">"Friday"</span><span class="o">,</span>
<span class="s2">"Saturday"</span><span class="o">,</span>
<span class="s2">"Sunday"</span>
<span class="o">]</span>
<span class="n">weekDays</span><span class="o">.</span><span class="na">each</span> <span class="o">{</span> <span class="n">day</span> <span class="o">-></span>
<span class="k">switch</span><span class="o">(</span><span class="n">day</span><span class="o">)</span> <span class="o">{</span>
<span class="k">case</span> <span class="s2">"Monday"</span><span class="o">:</span> <span class="c1">// if no break, will execute the next action</span>
<span class="k">case</span> <span class="s2">"Tuesday"</span><span class="o">:</span>
<span class="k">case</span> <span class="s2">"Wednesday"</span><span class="o">:</span>
<span class="n">println</span> <span class="s2">"${day} is just another business day"</span>
<span class="k">break</span> <span class="c1">// 'break' breaks the loop, no other action will be executed</span>
<span class="k">case</span> <span class="s2">"Friday"</span><span class="o">:</span>
<span class="n">println</span> <span class="s2">"TGIF!"</span>
<span class="k">break</span>
<span class="k">case</span> <span class="s2">"Thursday"</span><span class="o">:</span>
<span class="n">println</span> <span class="s2">"On ${day} we praise Thor"</span>
<span class="k">break</span>
<span class="k">case</span> <span class="s2">"Saturday"</span><span class="o">:</span>
<span class="k">case</span> <span class="s2">"Sunday"</span><span class="o">:</span>
<span class="n">println</span> <span class="s2">"It's ${day} :)"</span>
<span class="nl">default:</span> <span class="c1">// default condition, when other didn't match</span>
<span class="k">break</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<h3 id="if-statements">If statements</h3>
<p>If statements are less flexible than case. Groovy doesn’t support elseif/elif. So they are applicable only in <code class="language-plaintext highlighter-rouge">if ... else</code> way:</p>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kt">int</span> <span class="n">a</span> <span class="o">=</span> <span class="mi">2</span>
<span class="c1">//Check for the boolean condition </span>
<span class="k">if</span> <span class="o">(</span><span class="n">a</span><span class="o"><</span><span class="mi">100</span><span class="o">)</span> <span class="o">{</span>
<span class="c1">//If the condition is true print the following statement </span>
<span class="n">println</span><span class="o">(</span><span class="s2">"The value is less than 100"</span><span class="o">);</span>
<span class="o">}</span> <span class="k">else</span> <span class="o">{</span>
<span class="c1">//If the condition is false print the following statement </span>
<span class="n">println</span><span class="o">(</span><span class="s2">"The value is greater than 100"</span><span class="o">);</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Do something with list if it is not empty:</p>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">if</span><span class="o">(</span><span class="n">myList</span><span class="o">)</span> <span class="o">{</span>
<span class="o">...</span> <span class="k">do</span> <span class="n">something</span> <span class="o">...</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Do something if varible has some value</p>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">if</span><span class="o">(</span> <span class="n">foo</span> <span class="o">==</span> <span class="s2">"bar"</span> <span class="o">)</span> <span class="o">{</span>
<span class="o">...</span> <span class="k">do</span> <span class="n">something</span> <span class="o">...</span>
<span class="o">}</span>
</code></pre></div></div>
<h2 id="variables-with-shell-execution">Variables with shell execution</h2>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kt">def</span> <span class="n">myVar</span> <span class="o">=</span> <span class="s2">"Im with Groovy"</span>
<span class="n">sh</span> <span class="s2">"""
myVar="Im with Shell"
# this one is groovy
echo ${myVar}
# but this one is shell
echo \${myVar}
"""</span>
</code></pre></div></div>
<p>Create groovy variable from shell output</p>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kt">def</span> <span class="n">myWhoAmI</span> <span class="o">=</span> <span class="n">sh</span><span class="o">(</span><span class="nl">script:</span> <span class="s2">"whoami"</span><span class="o">,</span> <span class="nl">returnStdout:</span> <span class="kc">true</span><span class="o">).</span><span class="na">toString</span><span class="o">().</span><span class="na">trim</span><span class="o">()</span>
</code></pre></div></div>
<p><code class="language-plaintext highlighter-rouge">trim()</code> will remove empty space at the end.
<code class="language-plaintext highlighter-rouge">toString()</code> will convert the output to the str type</p>
<p>Read file and save it into variable</p>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kt">def</span> <span class="n">varFromFile</span> <span class="o">=</span> <span class="n">readFile</span> <span class="s1">'filename.txt'</span>
<span class="n">echo</span> <span class="s2">"The contexts of the file is ${varFromFile}"</span>
</code></pre></div></div>
<p>Another trick is to capture the status of the sh command and do something is status is not error:</p>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kt">def</span> <span class="n">commandStatus</span> <span class="o">=</span> <span class="n">sh</span><span class="o">(</span><span class="nl">script:</span> <span class="s2">"whoami 2>&1> whoami.txt"</span><span class="o">,</span> <span class="nl">returnStatus:</span><span class="kc">true</span><span class="o">).</span><span class="na">toString</span><span class="o">().</span><span class="na">trim</span><span class="o">()</span>
<span class="k">if</span> <span class="o">(</span><span class="n">commandStatus</span> <span class="o">==</span> <span class="mi">0</span><span class="o">)</span> <span class="o">{</span>
<span class="n">myWhoAmI</span> <span class="o">=</span> <span class="n">readFile</span> <span class="s1">'whoami.txt'</span>
<span class="o">}</span> <span class="k">else</span> <span class="o">{</span>
<span class="n">println</span><span class="o">(</span><span class="s1">'Something went wrong'</span><span class="o">)</span>
<span class="o">}</span>
</code></pre></div></div>
<h2 id="exiting-pipeline">Exiting pipeline</h2>
<p>If something goes wrong in your pipeline than obviously you’d like to make it stop at that point and change the job status. The best way to do this is is the following:</p>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="n">currentBuild</span><span class="o">.</span><span class="na">result</span> <span class="o">=</span> <span class="s1">'UNSTABLE'</span>
<span class="k">return</span>
</code></pre></div></div>
<p><code class="language-plaintext highlighter-rouge">currentBuild.result</code> is the system variable that can be defined in the pipeline. Possible values are “SUCCESS”, “UNSTABLE”, “FAILURE”, “NOT_BUILT”, “ABORTED”. May be null for an ongoing build.</p>
<p>It can be used in exception handling:</p>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">try</span> <span class="o">{</span>
<span class="n">stage</span> <span class="o">{</span>
<span class="o">....</span>
<span class="o">}</span>
<span class="o">}</span> <span class="k">catch</span> <span class="o">(</span><span class="n">e</span><span class="o">)</span> <span class="o">{</span>
<span class="k">if</span> <span class="o">(</span><span class="n">something</span><span class="o">)</span> <span class="o">{</span>
<span class="n">currentBuild</span><span class="o">.</span><span class="na">result</span> <span class="o">=</span> <span class="s1">'SUCCESS'</span>
<span class="k">return</span>
<span class="o">}</span> <span class="n">elif</span> <span class="o">(</span><span class="n">something</span><span class="o">)</span> <span class="o">{</span>
<span class="n">currentBuild</span><span class="o">.</span><span class="na">result</span> <span class="o">=</span> <span class="s1">'UNSTABLE'</span>
<span class="k">return</span>
<span class="o">}</span> <span class="n">elif</span> <span class="o">(</span><span class="n">something</span><span class="o">)</span> <span class="o">{</span>
<span class="n">currentBuild</span><span class="o">.</span><span class="na">result</span> <span class="o">=</span> <span class="s1">'NOT_BUILT'</span>
<span class="k">return</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Alternatively you can interrupt pipeline execution inside the stage if something is not working. For example:</p>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">stage</span><span class="o">(</span><span class="s1">'Stage1'</span><span class="o">){</span>
<span class="o">...</span>
<span class="k">if</span> <span class="o">(</span> <span class="n">somevar</span> <span class="o">==</span> <span class="kc">false</span> <span class="o">)</span> <span class="o">{</span>
<span class="n">currentBuild</span><span class="o">.</span><span class="na">result</span> <span class="o">=</span> <span class="s1">'ABORTED'</span>
<span class="k">return</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<h2 id="parsing-xml">Parsing XML</h2>
<p>The Groovy <code class="language-plaintext highlighter-rouge">XmlParser class</code> employs a simple model for parsing an XML document into a tree of <code class="language-plaintext highlighter-rouge">Node instances</code>.
Lets’ consider we have an xml like the following <code class="language-plaintext highlighter-rouge">Movies.xml</code>:</p>
<div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt"><collection</span> <span class="na">shelf =</span> <span class="s">"New Arrivals"</span><span class="nt">></span>
<span class="nt"><movie</span> <span class="na">title =</span> <span class="s">"Enemy Behind"</span><span class="nt">></span>
<span class="nt"><type></span>War, Thriller<span class="nt"></type></span>
<span class="nt"><format></span>DVD<span class="nt"></format></span>
<span class="nt"><year></span>2003<span class="nt"></year></span>
<span class="nt"><rating></span>PG<span class="nt"></rating></span>
<span class="nt"><stars></span>10<span class="nt"></stars></span>
<span class="nt"><description></span>Talk about a US-Japan war<span class="nt"></description></span>
<span class="nt"></movie></span>
<span class="nt"><movie</span> <span class="na">title =</span> <span class="s">"Transformers"</span><span class="nt">></span>
<span class="nt"><type></span>Anime, Science Fiction<span class="nt"></type></span>
<span class="nt"><format></span>DVD<span class="nt"></format></span>
<span class="nt"><year></span>1989<span class="nt"></year></span>
<span class="nt"><rating></span>R<span class="nt"></rating></span>
<span class="nt"><stars></span>8<span class="nt"></stars></span>
<span class="nt"><description></span>A schientific fiction<span class="nt"></description></span>
<span class="nt"></movie></span>
<span class="nt"></collection></span>
</code></pre></div></div>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">groovy.xml.MarkupBuilder</span>
<span class="kn">import</span> <span class="nn">groovy.util.*</span>
<span class="kt">def</span> <span class="n">parser</span> <span class="o">=</span> <span class="k">new</span> <span class="n">XmlParser</span><span class="o">()</span>
<span class="kt">def</span> <span class="n">doc</span> <span class="o">=</span> <span class="n">parser</span><span class="o">.</span><span class="na">parse</span><span class="o">(</span><span class="s2">"Movies.xml"</span><span class="o">);</span>
<span class="n">doc</span><span class="o">.</span><span class="na">movie</span><span class="o">.</span><span class="na">each</span><span class="o">{</span>
<span class="n">bk</span><span class="o">-></span>
<span class="n">print</span><span class="o">(</span><span class="s2">"Movie Name:"</span><span class="o">)</span>
<span class="n">println</span> <span class="s2">"${bk['@title']}"</span>
<span class="n">print</span><span class="o">(</span><span class="s2">"Movie Type:"</span><span class="o">)</span>
<span class="n">println</span> <span class="s2">"${bk.type[0].text()}"</span>
<span class="n">print</span><span class="o">(</span><span class="s2">"Movie Format:"</span><span class="o">)</span>
<span class="n">println</span> <span class="s2">"${bk.format[0].text()}"</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Another flow is to loop over all keys and values inside xml file using <code class="language-plaintext highlighter-rouge">XmlSlurper</code>:</p>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">doc</span> <span class="o">=</span> <span class="k">new</span> <span class="n">XmlSlurper</span><span class="o">().</span><span class="na">parse</span><span class="o">(</span><span class="s2">"Movies.xml"</span><span class="o">)</span>
<span class="n">doc</span><span class="o">.</span><span class="na">collection</span><span class="o">.</span><span class="na">each</span> <span class="o">{</span> <span class="n">item</span> <span class="o">-></span>
<span class="n">println</span> <span class="s2">"item index: ${item.@indexNum}"</span>
<span class="n">item</span><span class="o">.</span><span class="na">children</span><span class="o">().</span><span class="na">each</span> <span class="o">{</span> <span class="n">tag</span> <span class="o">-></span>
<span class="n">println</span> <span class="s2">" ${tag.name()}: ${tag.text()}"</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<h2 id="parsing-json">Parsing Json</h2>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">groovy.json.JsonSlurper</span>
<span class="kt">def</span> <span class="n">jsonSlurper</span> <span class="o">=</span> <span class="k">new</span> <span class="n">JsonSlurper</span><span class="o">()</span>
<span class="kt">def</span> <span class="n">object</span> <span class="o">=</span> <span class="n">jsonSlurper</span><span class="o">.</span><span class="na">parseText</span><span class="o">(</span><span class="s1">'{ "name": "John", "ID" : "1"}'</span><span class="o">)</span>
<span class="n">println</span><span class="o">(</span><span class="n">object</span><span class="o">.</span><span class="na">name</span><span class="o">);</span>
<span class="n">println</span><span class="o">(</span><span class="n">object</span><span class="o">.</span><span class="na">ID</span><span class="o">);</span>
</code></pre></div></div>
<p>Loop over all key/values in the given json file</p>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kt">def</span> <span class="n">jsonFile</span> <span class="o">=</span> <span class="n">parse</span><span class="o">(</span><span class="k">new</span> <span class="n">File</span><span class="o">(</span><span class="s1">'filename.json'</span><span class="o">))</span>
<span class="kt">def</span> <span class="n">jsonObject</span> <span class="o">=</span> <span class="k">new</span> <span class="n">groovy</span><span class="o">.</span><span class="na">json</span><span class="o">.</span><span class="na">JsonSlurper</span><span class="o">().</span><span class="na">jsonFile</span>
<span class="n">jsonObject</span><span class="o">.</span><span class="na">each</span> <span class="o">{</span> <span class="n">key</span><span class="o">,</span> <span class="n">value</span> <span class="o">-></span>
<span class="n">println</span> <span class="s2">"$key : $value"</span>
<span class="o">}</span>
</code></pre></div></div>
<h2 id="templating">Templating</h2>
<p>Groovy’s template engine is very similar to bash <code class="language-plaintext highlighter-rouge">envsubst</code>.</p>
<h3 id="simpletemplateengine">SimpleTemplateEngine()</h3>
<p>Let’s assume we have the following <code class="language-plaintext highlighter-rouge">Student.template</code> file:</p>
<div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt"><Student></span>
<span class="nt"><name></span>${name}<span class="nt"></name></span>
<span class="nt"><ID></span>${id}<span class="nt"></ID></span>
<span class="nt"><subject></span>${subject}<span class="nt"></subject></span>
<span class="nt"></Student></span>
</code></pre></div></div>
<p>The following will update the variables with the give values</p>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">groovy.text.*</span>
<span class="kn">import</span> <span class="nn">java.io.*</span>
<span class="kt">def</span> <span class="n">file</span> <span class="o">=</span> <span class="k">new</span> <span class="n">File</span><span class="o">(</span><span class="s2">"D:/Student.template"</span><span class="o">)</span>
<span class="kt">def</span> <span class="n">binding</span> <span class="o">=</span> <span class="o">[</span>
<span class="s1">'name'</span> <span class="o">:</span> <span class="s1">'Joe'</span><span class="o">,</span>
<span class="s1">'id'</span> <span class="o">:</span> <span class="mi">1</span><span class="o">,</span>
<span class="s1">'subject'</span> <span class="o">:</span> <span class="s1">'Physics'</span>
<span class="o">]</span>
<span class="kt">def</span> <span class="n">engine</span> <span class="o">=</span> <span class="k">new</span> <span class="n">SimpleTemplateEngine</span><span class="o">()</span>
<span class="kt">def</span> <span class="n">template</span> <span class="o">=</span> <span class="n">engine</span><span class="o">.</span><span class="na">createTemplate</span><span class="o">(</span><span class="n">file</span><span class="o">)</span>
<span class="kt">def</span> <span class="n">writable</span> <span class="o">=</span> <span class="n">template</span><span class="o">.</span><span class="na">make</span><span class="o">(</span><span class="n">binding</span><span class="o">)</span>
<span class="n">println</span> <span class="n">writable</span>
</code></pre></div></div>
<h3 id="streamingtemplateengine">StreamingTemplateEngine()</h3>
<p>The StreamingTemplateEngine engine is another templating engine available in Groovy. This is kind of equivalent to the SimpleTemplateEngine, but creates the template using writeable closures making it more scalable for large templates.</p>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kt">def</span> <span class="n">text</span> <span class="o">=</span> <span class="s1">'''This Tutorial is <% out.print TutorialName %> The Topic name
is ${TopicName}'''</span>
<span class="kt">def</span> <span class="n">template</span> <span class="o">=</span> <span class="k">new</span> <span class="n">groovy</span><span class="o">.</span><span class="na">text</span><span class="o">.</span><span class="na">StreamingTemplateEngine</span><span class="o">().</span><span class="na">createTemplate</span><span class="o">(</span><span class="n">text</span><span class="o">)</span>
<span class="kt">def</span> <span class="n">binding</span> <span class="o">=</span> <span class="o">[</span>
<span class="n">TutorialName</span> <span class="o">:</span> <span class="s2">"Groovy"</span><span class="o">,</span>
<span class="n">TopicName</span> <span class="o">:</span> <span class="s2">"Templates"</span><span class="o">,</span>
<span class="o">]</span>
<span class="n">String</span> <span class="n">response</span> <span class="o">=</span> <span class="n">template</span><span class="o">.</span><span class="na">make</span><span class="o">(</span><span class="n">binding</span><span class="o">)</span>
<span class="n">println</span><span class="o">(</span><span class="n">response</span><span class="o">)</span>
</code></pre></div></div>
<h3 id="xmltemplateengine">XmlTemplateEngine()</h3>
<p>The XmlTemplateEngine is used in templating scenarios where both the template source and the expected output are intended to be XML.</p>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kt">def</span> <span class="n">binding</span> <span class="o">=</span> <span class="o">[</span><span class="nl">StudentName:</span> <span class="s1">'Joe'</span><span class="o">,</span> <span class="nl">id:</span> <span class="mi">1</span><span class="o">,</span> <span class="nl">subject:</span> <span class="s1">'Physics'</span><span class="o">]</span>
<span class="kt">def</span> <span class="n">engine</span> <span class="o">=</span> <span class="k">new</span> <span class="n">groovy</span><span class="o">.</span><span class="na">text</span><span class="o">.</span><span class="na">XmlTemplateEngine</span><span class="o">()</span>
<span class="kt">def</span> <span class="n">text</span> <span class="o">=</span> <span class="s1">'''\
<document xmlns:gsp='http://groovy.codehaus.org/2005/gsp'>
<Student>
<name>${StudentName}</name>
<ID>${id}</ID>
<subject>${subject}</subject>
</Student>
</document>
'''</span>
<span class="kt">def</span> <span class="n">template</span> <span class="o">=</span> <span class="n">engine</span><span class="o">.</span><span class="na">createTemplate</span><span class="o">(</span><span class="n">text</span><span class="o">).</span><span class="na">make</span><span class="o">(</span><span class="n">binding</span><span class="o">)</span>
<span class="n">println</span> <span class="n">template</span><span class="o">.</span><span class="na">toString</span><span class="o">()</span>
</code></pre></div></div>
<h3 id="exteral-links">Exteral links:</h3>
<ul>
<li><a href="https://www.tutorialspoint.com/groovy/groovy_switch_statement.htm">Switch cheat-sheet</a></li>
<li><a href="https://www.tothenew.com/blog/groovy-tokenize-vs-split/">Groovy : tokenize() vs split()</a></li>
<li><a href="http://docs.groovy-lang.org/next/html/documentation/working-with-collections.html">Working with collections in Groovy</a></li>
<li><a href="https://www.tutorialspoint.com/groovy/groovy_xml.htm">Groovy - XML</a></li>
</ul>Yaroslav YarmoshykCheatsheet for Groovy to be used in Jenkins pipelinesCloudflare: restrict access to your website for a country2022-06-06T00:00:00+00:002022-06-06T00:00:00+00:00https://www.tech-notes.net/cloudflare-country-restrictions<p>Today I woke up and suddenly realised that most of the articles in this blog were written in russian language and the major visitors come from russian-speeking countries.
Almost 80% of traffic came to my website from russia. Here is the data for last Friday:<br /></p>
<center><img src="/wp-content/uploads/2022/tech-notes-traffic-stats.png" alt="drawing" width="30%" height="30%" /></center>
<p>I need to prevent these guys from using my background in their work so no more traffic from russia (I wrote from small letter for purpose). I don’t care about the numbers becuase this is non-profitable website.</p>
<p>I use <a href="https://jekyllrb.com/">jekyll</a> to organize the website content and <a href="https://docs.aws.amazon.com/AmazonS3/latest/userguide/website-hosting-custom-domain-walkthrough.html">AWS S3</a> to store the files so I don’t much options how to restrict access to it. But I also use <a href="https://www.cloudflare.com/">CloudFlare</a> as a CDN for it and I found a cool thing in it that can help me to acheive what I need.</p>
<p>Restricting access to your website with CloudFlare for a particular country is fairy simple. You need WAF section that is available even in the free tier account.
You got to create new firewall rule as shown at the screenshot below
<img src="/wp-content/uploads/2022/cloudflare_country_block.png" alt="cloudflare_country_block!" title="cloudflare_country_block" /></p>
<p>An now you can hapily observe how they hit the wall or their expectations:
<img src="/wp-content/uploads/2022/blocked-access-waf.png" alt="blocked-access-waf!" title="blocked-access-waf" /></p>Yaroslav YarmoshykToday I woke up and suddenly realised that most of the articles in this blog were written in russian language and the major visitors come from russian-speeking countries. Almost 80% of traffic came to my website from russia. Here is the data for last Friday:Use bash script as data source in terrafform2022-01-06T00:00:00+00:002022-01-06T00:00:00+00:00https://www.tech-notes.net/terraform-bash-script-external-data-source<p>I was for a multiple times when I had to extract some data with bash script and next re-use the result in terraform resource provisioning.
I’d like to share this for future refference.</p>
<p>Terraform has the <a href="https://registry.terraform.io/providers/hashicorp/external/latest/docs/data-sources/data_source">external data source</a> that allows to run external program and handle it’s outputs in further infrastructure code.
Next you can find the example of the bash script. The most important part in it is crafting the proper result to be returned into terraform.
It consists of three functions:</p>
<ol>
<li><code class="language-plaintext highlighter-rouge">error_exit</code> to be used to return proper error message.
<img src="/wp-content/uploads/2022/jq-not-found.png" alt="error message example!" title="jq-not-found" /></li>
<li><code class="language-plaintext highlighter-rouge">check_deps</code> - to be used to check whether all required binaries are present. In this particular example I’m checking whether jq and curl binaries are available. JQ will always remain in the list becuase it is needed to craft the responce.</li>
<li><code class="language-plaintext highlighter-rouge">extract_data</code> - to actually get the required data and return responce. In this example I’m reading the public IP address of the machine where terraform is being executed using the external website ifconfig.me</li>
</ol>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">#!/usr/bin/env bash</span>
<span class="k">function </span>error_exit<span class="o">()</span> <span class="o">{</span>
<span class="nb">echo</span> <span class="s2">"</span><span class="nv">$1</span><span class="s2">"</span> 1>&2
<span class="nb">exit </span>1
<span class="o">}</span>
<span class="k">function </span>check_deps<span class="o">()</span> <span class="o">{</span>
<span class="nv">jq_test</span><span class="o">=</span><span class="si">$(</span>which jq<span class="si">)</span>
<span class="nv">curl_test</span><span class="o">=</span><span class="si">$(</span>which curl<span class="si">)</span>
<span class="k">if</span> <span class="o">[[</span> <span class="nt">-z</span> <span class="nv">$jq_test</span> <span class="o">]]</span><span class="p">;</span> <span class="k">then </span>error_exit <span class="s2">"JQ binary not found"</span><span class="p">;</span> <span class="k">fi
if</span> <span class="o">[[</span> <span class="nt">-z</span> <span class="nv">$curl_test</span> <span class="o">]]</span><span class="p">;</span> <span class="k">then </span>error_exit <span class="s2">"curl binary not found"</span><span class="p">;</span> <span class="k">fi</span>
<span class="o">}</span>
<span class="k">function </span>extract_data<span class="o">()</span> <span class="o">{</span>
<span class="nv">my_ip</span><span class="o">=</span><span class="si">$(</span>curl <span class="nt">-q</span> https://ifconfig.me/ip<span class="si">)</span>
jq <span class="nt">-n</span> <span class="nt">--arg</span> my_ip <span class="s2">"</span><span class="nv">$my_ip</span><span class="s2">"</span> <span class="s1">'{"my_ip":"'</span><span class="nv">$my_ip</span><span class="s1">'"}'</span>
<span class="o">}</span>
check_deps
extract_data
</code></pre></div></div>
<p>This script needs to be save in the folder where your module code is located. I use separate subfolders for scripts. Also this file needs to be executable</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">chmod</span> +x extract_ip.sh
</code></pre></div></div>
<p>The definition of the external data source looks like the following:</p>
<div class="language-tcl highlighter-rouge"><div class="highlight"><pre class="highlight"><code>data <span class="s2">"external"</span> <span class="s2">"external_ip"</span> <span class="p">{</span>
program = <span class="p">[</span>
<span class="s2">"/bin/bash"</span>, <span class="s2">"</span><span class="nv">${path.module}</span><span class="s2">/scripts/extract_ip.sh"</span>
<span class="p">]</span>
<span class="p">}</span>
</code></pre></div></div>
<p>The resutls can be refferenced in your code as the following:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>module "some-name" {
...
my_ip = data.external.external_ip.result.my_ip
...
}
</code></pre></div></div>
<p>I prepared the separate <a href="https://github.com/yyarmoshyk/terraform-bash-external-datasource">github repo</a> with this example.</p>Yaroslav YarmoshykI was for a multiple times when I had to extract some data with bash script and next re-use the result in terraform resource provisioning. I’d like to share this for future refference.