urn:augustl-com:feed2023-12-07T00:00:00.000+01:00August Lilleaas' blogBlogging with Word in your Jamstack2023-12-07T00:00:00.000+01:00August Lilleaasurn:augustl-com:feed:post::blog:2023:blogging_with_word<p>I <a href="https://augustl.com/blog/2023/pro_kotlin_web_apps_from_scratch/">wrote </a><a href="https://augustl.com/blog/2023/pro_kotlin_web_apps_from_scratch/">a</a><a href="https://augustl.com/blog/2023/pro_kotlin_web_apps_from_scratch/"> book</a> in Word. That’s 72 000 words in Word. </p><p>But I’ve stopped blogging, partly because writing markdown in my code editor just doesn’t vibe with me anymore. These days, I ask myself the question: how can I just get a Word document online as fast as possible?</p><p>I like Word! It has nice tooling for grammar and spell checking, and it’s a nice environment for editing prose.</p><p>So, I wrote a parser for Word files that slurps them into my blog engine and emits clean HTML – like the post you’re seeing now! Yes, my friends, I now have support for .<code>docx</code> files in my personal blog.</p><p>Tldr: <a href="https://github.com/augustl/augustl.com/tree/master/posts/blog/2023">Here’s the Word document</a> (called blogging_with_word.docx), and <a href="https://github.com/augustl/augustl.com/blob/b6a3382e6ad49a4bd710332618cb58140c2cf4bd/src/augustl_com/word_post_parser.clj">here’s the parser code</a>.</p><h2>A quick word on tech</h2><p>Programmers should build their own <span style="text-decoration: line-through">lightsaber</span> blog.</p><p>This blog is a Jamstack style Clojure app (it used to be a Ruby/Middleman app, and before that, Rails) that reads a bunch of files and outputs static HTML, that I host on CloudFront in AWS. <a href="https://github.com/augustl/augustl.com">Source code here</a>.</p><p>This setup turns out to be the best of both worlds. I get to write some fun code, once, and hopefully when it’s all up and running, I can just paste another docx file into the correct folder and have a fresh blog post online in no time.</p><p>Back in the old days I enjoyed hand-typing markdown in my code editor. In fact, the first years of this blog was hand-typing HTML. Clearly, I have changed.</p><h2>Parsing Word docs</h2><p>I’m glad I use <a href="https://news.ycombinator.com/item?id=4310723">a robust technology like the JVM</a>, which is the platform Clojure runs on. Clojure has great Java interop, so I’m using the <a href="https://poi.apache.org/components/document/quick-guide-xwpf.html">good old Apache POI library</a>. It has full support for everything Microsoft Office, and convenient APIs for extracting text from a Word document.</p><p>Let’s get into it!</p><pre><code>(defn get-body-from-docx [^File file]
(with-open [doc (XWPFDocument. (io/input-stream file))]
(->> (iterator-seq (.getParagraphsIterator doc))
(drop-while #(re-find header-line-re (.getText %)))
(map (fn [para]
(if-let [runs (seq (.getRuns para))]
{::runs runs
::style (.getStyleID para)
::para para
::doc doc}
::blank)))
(word-paragraphs-hiccup-seq))))</code></pre><p><code>iterator-seq</code> and the <code>getParagraphsIterator()</code> method are the juicy bits. It yields the Word document paragraphs as a plain lazy Clojure sequence, which lets me pretend I’m writing code against a neat Clojure library, instead of a noun heavy Java library.</p><p>The <code>drop-while</code> gets rid of the initial header lines. My word document starts with some metadata directly in the text, such as <code>date: 2023-12-07</code>. It also has title of the post, and so on.</p><p>Then I map each paragraph to <code>::blank</code> for empty lines, and create a map with all the data I need for further processing for lines with contents.</p><h2>Chunking paragraphs</h2><p>Then, the fun process of converting Word paragraphs to HTML begins. Paragraphs in Word documents are not like paragraphs on the web. A paragraph in Word is more like a sequence of <code><div></code>s, or lines of text separated by a <code><br></code> tag. </p><p>Let’s say the word document looks like this:</p><pre><code>Here is my line
Here is my other line
with no space between
Here is my last line</code></pre><p>That should result in HTML that looks roughly like this:</p><pre><code><p>Here is my line</p>
<p>Here is my other line<br>with no space between</p>
<p>Here is my last line</p></code></pre><p>In other words, the list of paragraphs from Word needs to be chunked properly.</p><p>This is what <code>word-paragraphs-hiccup-seq</code> does. That function is just some looping mechanics, the meat of that implementation is a function, <code>get-next-hiccup-tag</code>. This function returns the new HTML for the paragraphs it “consumed”, and the remaining paragraphs to be processed. </p><pre><code>(defn get-next-hiccup-tag [paragraphs]
(let [paragraphs (->> paragraphs
(drop-while #(= % ::blank))
{style ::style runs ::runs doc ::doc} (first paragraphs)]
(case style
"Code" (let [[code-paras rest] (->> paragraphs
(consume-paragraph-chunk #(= "Code" (::style %))))]
[(get-code-block-tag code-paras) rest])
"Heading2" [(into [:h2] (runs-to-hiccup-seq runs doc)) (rest paragraphs)]
"Heading3" [(into [:h3] (runs-to-hiccup-seq runs doc)) (rest paragraphs)]
"Heading4" [(into [:h4] (runs-to-hiccup-seq runs doc)) (rest paragraphs)]
(get-next-hiccup-paragraph paragraphs))))</code></pre><p>The first step is to remove any preceding blank paragraphs. If I insert 10 blank spaces in a row, I don’t want word to insert to paragraphs separated by 9 (?) <code><br></code> tags. This makes my Word documents behave similar to Markdown, where you also have to do some work to get a bunch of whitespace between paragraphs.</p><p>The <code>::style</code> property of the first paragraph determines what to do. If it’s a code block or a header, I immediately create an element and then return the remaining paragraphs. There’s some logic in code blocks I’ll return to later. But that’s about it.</p><p>If the paragraph does not have a style associated with it (meaning it’s just a normal block of text), I invoke the main bulk of the chunking logic.</p><pre><code>(defn get-next-hiccup-paragraph [paragraphs]
(let [[text-paragraphs rest] (->> paragraphs
(split-with
#(and (not= % ::blank)
(nil? (::style %)))))]
[(into [:p]
(->> text-paragraphs
(map #(runs-to-hiccup-seq (::runs %) (::doc %)))
(interpose [:br])))
rest]))</code></pre><p>First, I take all the blocks of non-styled paragraphs that also aren’t blank lines. Then, I build a <code><p></code> tag that contains each paragraph converted into HTML (it reads out bold, italics, links, etc from the text runs), and I insert a <code><br></code> between each line in the paragraph.</p><p>The <code>rest</code> is what is returned to the looping code, meaning that for the next iteration of the processing loop, the remaining characters will be processed.</p><h2>Processing code blocks</h2><p>I invoke a function called <code>consume-paragraph-chunk</code> to build the code blocks. Here’s that function.</p><pre><code>(defn consume-paragraph-chunk [pred xs]
(let [[chunk rest] (split-with #(or (pred %) (= ::blank %)) xs)
whitespace-tail (->> chunk (reverse) (take-while #(= ::blank %)))]
[(->> chunk
(drop-last (count whitespace-tail)))
(concat whitespace-tail rest)]))</code></pre><p>Maybe there’s a smarter way to do this, but at this point I just wanted to get this blog post published. The function takes all lines that are blank or matches the predicate (in this case, <code>::style</code> is <code>"Code"</code>). I don’t want it to include any tails of whitespace after the last chunk of code, so I yank those out and add them to the <code>rest</code> for further processing in the loop.</p><h2>Great success</h2><p>So that’s about it! I wrote this very blog post in Word. This has increased the likelihood of increased blog output by at least 1%.</p><p>My number one fear is that this will turn out to be some kind of churnfest due to incompatibilities with my old blog posts in new versions of Word, or something like that. The <code>docx</code> format has been stable for a while now though, so I sure hope not. At any rate, it was a fun little project to get up and running.</p><h2>Test area</h2><p>Because I’m lazy, I don’t want to set up a separate test environment. So here’s a blurb that tests some random stuff.</p><p>Here, we should see a continuous block of code, with newlines inside it but not after.</p><pre><code>(defn here-is-some-code []
;; Test newline stuff
;; Ok here we go
(prn "Hello, World!"))</code></pre><p>Here, we should see a set of words separated by <code><br></code> tags. And, we should not see an actual line break here, but a single paragraph, and some in-line escaped HTML.</p><p>Can we have in-line code? <code>(+ 1 2)</code>. This should use a monospace font.</p><p>Here<br />We<br />Go</p><p>We’ve already tested links. But let’s <a href="https://crud.business/">test it again</a>.</p>Pro Kotlin Web Apps from Scratch2023-02-02T00:00:00.000+01:00August Lilleaasurn:augustl-com:feed:post::blog:2023:pro_kotlin_web_apps_from_scratch<p>I wrote a book!</p>
<p><img src="/static/posts/pro-kotlin-web-apps-from-scratch/kotlin-book-pro-kotlin-web-apps-from-scratch-cover.jpg" alt="Pro Kotlin Web Apps from Scratch book cover"></p>
<p><a href="https://link.springer.com/book/10.1007/978-1-4842-9057-6">Buy the book from the publisher</a></p>
<p><a href="https://www.amazon.com/Pro-Kotlin-Apps-Scratch-Production-ready/dp/1484290569">Buy the book from Amazon</a></p>
<h2>Book source code</h2>
<p><a href="https://github.com/Apress/Pro-Kotlin-Web-Apps-from-Scratch">All the code from the book is available on Apress' GitHub</a>.</p>
<p>You can also get <a href="https://github.com/augustl/pro-kotlin-web-apps-from-scratch">the code from the book mirrored to my personal GitHub</a>.</p>
It's christmas, and time for failure!2019-12-24T00:00:00.000+01:00August Lilleaasurn:augustl-com:feed:post::blog:2019:its_christmas_and_time_for_failure<p>Well, that's it, then. I failed.</p>
<p>I was supposed to publish a blog post every day in December until the 24th.</p>
<p>I haven't blogged since the 18th, so that means I failed to write the 5 final blog posts except this one.</p>
<p>In case you didn't know, I <a href="https://augustl.com/blog/2019/cultivating_my_failures/">fail a lot</a>.</p>
<p>I sort of expected that I'd fail to write 24 blog posts.</p>
<p>But failing is better than not trying.</p>
<p>The <a href="https://augustl.com/blog/2019/linus_and_linux_happy_accident/">blog post about Linus Torvalds</a> ended up <a href="https://www.reddit.com/r/linux/comments/e6fvj0/linux_exists_only_because_of_a_happy_accident/">on the front page of /r/Linux</a> and got shared on Twitter a whole lot. I guess many people haven't heard that story before, and it's a pretty cool story!</p>
<p>THe <a href="https://augustl.com/blog/2019/best_bug_predictor_is_organizational_complexity/">blog post about the Microsoft Research study after Vista</a> generated a gazillion Tweets, around 90k page views, and great discussion on <a href="https://news.ycombinator.com/item?id=21823199">HN</a> and <a href="https://www.reddit.com/r/programming/comments/ec9s2o/the_1_bug_predictor_is_not_technical_its/">Reddit</a>. It even generated some leads towards a path as a professional writer.</p>
<p>In other words, doors are now open, that weren't open before I started the advent calendar blog project.</p>
<p>And I generated a bunch of other blog posts that didn't necessarily get a whole lot of attention, but that I'm very proud of, and that some people got a lot out of.</p>
<p>I so very almost decided not to start this blogging project, because I sort of knew that I probably wouldn't finish it properly.</p>
<p>So I failed.</p>
<p>But I'm glad I did!</p>
<p>Happy holidays!</p>
The #1 bug predictor is not technical, it's organizational complexity2019-12-18T00:00:00.000+01:00August Lilleaasurn:augustl-com:feed:post::blog:2019:best_bug_predictor_is_organizational_complexity<p>EDIT: Discussion on <a href="https://news.ycombinator.com/item?id=21823199">Hacker News</a> and <a href="https://www.reddit.com/r/programming/comments/ec9s2o/the_1_bug_predictor_is_not_technical_its/">/r/programming</a>.</p>
<p>In this post, I'll explore the findings made by Microsoft Research, after the unsuccessful launch of Windows Vista in 2007. Microsoft decided to dig deep and figure out what went wrong.</p>
<p>The marketing department busied themselves with pranking people into saying that they like Vista.</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/ihorvo2tEuA" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
<p>The research department, on the other hand, decided to do some research.</p>
<p>This is what they found.</p>
<h2>Organizational complexity is the best predictor of bugs in a software module</h2>
<p>This first section will summarize the findings. Later on, I'll describe what the findings mean, and more details about the methods used.</p>
<p>Microsoft Research <a href="https://www.microsoft.com/en-us/research/publication/the-influence-of-organizational-structure-on-software-quality-an-empirical-case-study/">published a paper in which they developed a new statistical model</a> for predicting whether or not a software module was at risk of having bugs, based on a statistical analysis of the module itself.</p>
<p>The new model they developed was compared to these pre-existing well known models for predicting software bugs:</p>
<ul>
<li><strong>Code Churn</strong>. Measures the number of changes in version control of a module.</li>
<li><strong>Code Complexity</strong>. Measures the number of code paths, the number of functions that call each other internally in the module, depth of inheritance hierarchies, coupling between objects, number of sub classes, etc.</li>
<li><strong>Dependencies</strong>. Measures the number of external modules calling you, how many external modules you're calling, how many layers a module is away from the hardware, etc.</li>
<li><strong>Code Coverage</strong>. Traditional test coverage that most developers are probably already familiar with.</li>
<li><strong>Pre-Release Bugs</strong>. Number of issues in the issue tracker.</li>
</ul>
<p>In addition to these pre-existing models, the new model they
developed was:</p>
<ul>
<li><strong>Organizational Complexity</strong>. Measures number of developers working on the module, number of ex-developers that <em>used</em> to work on the module but no longer do, how big a fraction of the organization as a whole that works or has worked on the module, the distance in the organization between the developer and the decision maker, etc.</li>
</ul>
<p>The results really do speak for themselves.</p>
<table>
<thead>
<tr><th>Model</th><th>Precision</th><th>Recall</th></tr>
</thead>
<tbody>
<tr><td>Organizational Structure</td><td>86.2%</td><td>84.0%</td></tr>
<tr><td>Code Churn</td><td>78.6%</td><td>79.9%</td></tr>
<tr><td>Code Complexity</td><td>79.3%</td><td>66.0%</td></tr>
<tr><td>Dependencies</td><td>74.4%</td><td>69.9%</td></tr>
<tr><td>Code Coverage</td><td>83.8%</td><td>54.4%</td></tr>
<tr><td>Pre-Release Bugs</td><td>73.8%</td><td>62.9%</td></tr>
</tbody>
</table>
<p>(Higher numbers are better. A more detailed description follows.)</p>
<p>That's pretty interesting!</p>
<p>Organizational structure has the highest precision, <em>and</em> the highest recall. (Again, more on the details later).</p>
<p>That's pretty interesting, isn't it? The distance to decision makers and the number of developers working on a project is clearly and unambiguously the issue that is the best predictor of future problems with a code base.</p>
<p>Another shocking discovery for me personally, is that the only one that I've actually used myself - code coverage - has the lowest recall. It has a high precision, so bad code coverage does mean a high chance of bugs, i.e. low amount of false flags. But with a low recall, there are lots of bug that code coverage doesn't actually catch.</p>
<h2>Predicting bugs?</h2>
<p>What do these numbers actually mean? And what does it mean to predict bugs in a software module?</p>
<p>Again, <a href="https://www.microsoft.com/en-us/research/publication/the-influence-of-organizational-structure-on-software-quality-an-empirical-case-study/">the original paper</a> has the full description. What follows is my layman summary in blog form.</p>
<p>The "precision" and "recall" values are the results of a method for evaluating the actual predictive powers of the analytical models.</p>
<p>The gist of it is that you compare two things: The predictions made by the prediction method, without any knowledge of real-world bugs. And actual real-world bug information, gathered some time after release.</p>
<p>The very general summary of the statistical process is as follows:</p>
<ul>
<li>You analyze a software module using the prediction model, and return a p value (a number between 0 and 1, where 0 means zero chance of bug, and 1 means it's completely confident that there's bugs).</li>
<li>You use that p value to define a binary "yes" or "no" for whether or not you think a module has a bug. If p > 0.5, the module is flagged as buggy.</li>
<li>A module, in the context of Windows Vista, is an individual DLL. So a device driver, a kernel module, etc.</li>
<li>You take the 3000 (ish) modules in Vista and, divide them into 3 random parts. One part gets the prediction method run on them, and that result is compared with the remaining two parts based on whether or not it actually had a bug in the real world</li>
<li>You extract precision and recall values out of that (more on that later)</li>
<li>Repeat 100 or so times, and check that you get an roughly even distribution of precision and recall values across all the random selections.</li>
</ul>
<p>And voila! You have your numbers.</p>
<p>So what exactly is precision and recall?</p>
<h2>Precision and recall</h2>
<p><em>EDIT: This section had precision and recall swapped. The article has been updated. Thanks to <a href="https://www.reddit.com/r/programming/comments/ec9s2o/the_1_bug_predictor_is_not_technical_its/fba2g0w/">/u/programmingfriend on Reddit</a> for pointing that out!</em></p>
<p>When you run your comparison of "here's the result my prediction method got" and compare it to "here's how many bugs the module <em>actually</em> had", you get precision and recall values.</p>
<p>Remember that we checked 1/3rd of the modules with the prediction method, and compared it to the actual real world results of the remaining 2/3rds. This comparison yields the numbers in the table above.</p>
<p>So what does it mean?</p>
<ul>
<li><strong>Precision</strong> - of the modules that the prediction thought had bugs, how many did actually have bugs?</li>
<li><strong>Recall</strong> - how many modules that had bugs, did the prediction model detect?</li>
</ul>
<p>Having a low recall is not the end of the world. It just means that you could have detected more buggy modules, and that some buggy modules went under the radar.</p>
<p>Having a low precision is worse. That means that the prediction model tagged a module as buggy - but it turned out that the module actually wasn't.</p>
<h2>Doesn't that mean that I need actual bug data to predict bugs?</h2>
<p>No.</p>
<p>The Microsoft Research team compared their prediction models to real world bug information 6 months prior to release.</p>
<p>But they only used this to get some numbers of how valid the methods are.</p>
<p>The actual prediction method in and of itself only requires access to the source code, and in the case of the organizational complexity analysis, data about committers in the HR system so they can run the organizational metrics as well.</p>
<p>But no real world bug data is required to run these models.</p>
<p>So now that the model has been verified, you can run it on an unreleased module and get predictions on whether or not that module will have bugs in it.</p>
<h2>The study has been successfully replicated</h2>
<p>Science is going through somewhat of a <a href="https://en.wikipedia.org/wiki/Replication_crisis">replication crisis</a>, where fundamental and highly cited studies has turned out to not be reproducible.</p>
<p>Thankfully, the predictive value of organizational complexity to software has been replicated!</p>
<p><a href="http://www.scs.ryerson.ca/~avm/dat/manuscripts/ICSE_2015.pdf">In the replicated study</a> the predictive value of organizational structure is not as high. Out of 4 measured models, it gets the 2nd highest precision and the 3rd highest recall. The study itself does conclude that organizational complexity as a bug prediction method is worth investigating further. The study is also based on individual functions in C/C++, and not entire modules like Microsoft Research did, which can be a reason for at least a part of the discrepancy.</p>
<p>Also, the model for measuring organizational complexity doesn't measure the organization compared to other organizations. I wouldn't be surprised if Microsoft is one of the most complex software organizations in the world. So maybe that's why organizational complexity was at the top of the list at Microsoft.</p>
<p>Another thing to consider is the difference between formal and informal organization structure, which none of the studies takes into account. The Microsoft study just automated its model against data from the HR system. Watch Ed Catmull of Pixar talk about this here (10:06)</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/k2h2lvhzMDc?start=606" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
<p>At any rate, this discovery coupled with the findings from <a href="https://www.amazon.com/Accelerate-Software-Performing-Technology-Organizations-ebook/dp/B07B9F83WM/ref=sr_1_1?keywords=accellerate&qid=1576617532&s=books&sr=1-1">Accelerate</a> leads me to at the very least believe that social elements is probably under-measured in software projects, and should be taken more seriously.</p>
The green padlock2019-12-17T00:00:00.000+01:00August Lilleaasurn:augustl-com:feed:post::blog:2019:the_green_padlock<p>We just love that green padlock, don't we. The green padlock means that nobody can read your communications online.</p>
<p>Yeah, right.</p>
<h2>Whose CA is it anyway?</h2>
<p>You probably know about CAs already. Certificate Authorities. There has to be some trust somewhere in HTTPS, and that's where it's at.</p>
<p>So which CAs do you have on your system?</p>
<p>Who do you decide to trust?</p>
<p>In my case, I trust two parties: Mozilla and Microsoft. Firefox (Mozilla) has their own list of CAs and don't piggyback on the OS at all. But I also have to trust the CA list in Mocrosoft when I download dependencies on Maven or use Chrome because the tracking protection in Firefox is making a website I use not work properly (sites that use Facebook for commenting, for example).</p>
<p>So it's not really your decision, is it?</p>
<p>But surely, we can trust Mozilla and Microsoft (and Apple and Canonical and ...)?</p>
<h2>Yeah, but no</h2>
<p><a href="https://ccadb-public.secure.force.com/mozilla/IncludedCACertificateReport">Here's the list from Mozilla</a>. I use Mozilla as an example, because they're the ultimate good guy on the internet.</p>
<p>I mean, "CFCA EV ROOT" is something we can trust, right? China Financial Certification Authority?</p>
<p>And "E-Tugra Certification Authority" from Turkey?</p>
<p>And probably 50% of the CAs that are from USA?</p>
<p>There's absolutely no way that the governments in those countries can issue fake certificates for surveillance purposes?</p>
<p>We trust them completely?</p>
<p>Sure, you can do that. And most of us do!</p>
<p>But is your data actually safe?</p>
<h2>The dog that doesn't bark</h2>
<p>Have you noticed how there's a battle against encryption going on? Many goverments really don't like WhatsApp and Telegram all that much. They use end-to-end encryption, which means that there's no 3rd party to trust (loosely speaking). The govermnemt can't just issue a MITM certificate that looks real, you actually have to get access to the end user device (loosely speaking) to read the data.</p>
<p>So the governments don't like that, because they're locked out of WhatsApp.</p>
<p>What about encryption on iOS and modern Android devices? The government sure would like a back door into that system, and complain loudly about their inability to access locked mobile phones.</p>
<p>How often have you heard the governments complain about https?</p>
<p>Never?</p>
<p>I've never heard them complain about that.</p>
<p>That's the dog that doesn't bark.</p>
<p>Of course the government can read our HTTPS traffic if they want to.</p>
<h2>Conspiracy?</h2>
<p>Yeah I guess. I don't know this to be true.</p>
<p>And I totally get that being locked out of communications sucks, when your job is to monitor communications. <a href="https://en.wikipedia.org/wiki/Enigma_machine">We've spent lots of resources in the past</a> attempting to read encrypted/scrambled comms, so there's no reason to assume that will stop now.</p>
<p>I'm honestly not sure if I think it's that bad if the government can read HTTPS, which I'm sure they can, or they would have complained about it.</p>
<p>But can we trust the CA system?</p>
<p>Nah.</p>
Validating data after storing it2019-12-16T00:00:00.000+01:00August Lilleaasurn:augustl-com:feed:post::blog:2019:validating_data_after_storing_it<p>This is me trying to be some kind of thought leader or something. Someone told me they do this in their system, and it stayed with me. So I want to share what they told me.</p>
<p>Keeping it short and sweet.</p>
<h2>Mission critical</h2>
<p>Typically, we validate our data on the way in. Validating the shape of our data is a big part of modelling a domain, so this is commonplace and normal, and it makes complete sense.</p>
<p>But what if data is mission critical? Even if it's not 100% valid yet?</p>
<p>Let's say it's a medical system, for ordering important and vital equipment. Maybe social security number is a required part of the form.</p>
<p>Should you be blocked from ordering vital equipment just because you don't have access to or don't have the time to find the social security number of the person in question?</p>
<p>In that case, it's better to allow "invalid" input. So you want to have a place to store the social security number, but you should allow blank values there.</p>
<p>Maybe you have a separate part of the system that lists all the "invalid" items that you are required to fix at some time. And if you don't, maybe the owners of the system will fine you or punish you in some way. The social security number is <em>definitely</em> required, after all - just not right away.</p>
<h2>Mundane</h2>
<p>This happened to me once:</p>
<p>A electrician was going to replace something in my breaker box. He had a PDA thing with him, where he took some pictures and filled out a form. He did this while he was in my house.</p>
<p>Then the app on his PDA crashed, and all the data was lost.</p>
<p>So he had to do it all over again, while still being in my house. And being visibly annoyed. While billing me, probably.</p>
<p>Bugs can happen, of course. There's no known process for making crash-free systems.</p>
<p>But what if the default for this system was: "always store all data, validate it after it's stored?"</p>
<p>Then, at least the pictures he took and the form he filled out could be constantly saved while he worked. Without adding any extra layer of "draft" storage etc. It could just run the normal operation of saving the form.</p>
<p>Then, in his car later, he could see that all is saved in the database, but he needs to fill in a few more fields to make the form completely "valid".</p>
<p>So maybe this is a good idea?</p>
<p>To separate the idea of "storing" and "validating" data?</p>
<p>Probably not in all cases. But I bet that a whole lot of systems could be made that way without much hassle. And it seems to me that we default to validating data before storing it without really thinking about it on a case-by-case basis.</p>
According to Microsoft, the number one predictor of software bugs is...2019-12-15T00:00:00.000+01:00August Lilleaasurn:augustl-com:feed:post::blog:2019:microsoft_predicts_software_bugs<p>UPDATE: This article was too clickbait-y. I've <a href="https://augustl.com/blog/2019/best_bug_predictor_is_organizational_complexity/">made a proper version of it here</a>.</p>
<p>Microsoft launched Windows Vista in 2007, with varying degrees of success.</p>
<p>Microsoft Research made a substantial effort to figure out what went wrong.</p>
<p>They learned something surprising (or maybe not?) about the number one predictor of software bugs.</p>
<h2>Microsoft Research, and Vista</h2>
<p>Windows Vista was a problematic release, to say the least. Vista is often placed alongside ME (Millennium Edition) as the two versions of Windows that were, put simply, bad releases.</p>
<p>I'm not going to go into great detail here. But they struggled, and Microsoft knew it. For example, they had this amazing commercial running in 2008: The Mojave Experiment, where Microsoft basically pranks people into saying positive things about Vista.</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/ihorvo2tEuA" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
<p>Microsoft Research was tasked with discovering and developing methods Microsoft could use in future projects to avoid repeating the same kind of mistakes.</p>
<h2>What did they find out?</h2>
<p>Microsoft Research <a href="https://www.microsoft.com/en-us/research/publication/the-influence-of-organizational-structure-on-software-quality-an-empirical-case-study/">developed a new statistical model</a> for predicting whether or not a module risked having bugs. To evaluate the quality of the new model that they made, they compared it to other well known models used in other studies before them.</p>
<ul>
<li><strong>Code Churn</strong>. Measures the number of changes in version control of a module.</li>
<li><strong>Code Complexity</strong>. Measures the number of code paths, the number of functions that call each other internally in the module, depth of inheritance hierarchies, coupling between objects, number of sub classes, etc.</li>
<li><strong>Dependencies</strong>. Measures the number of external modules calling you, how many external modules you're calling, how many layers a module is away from the hardware, etc.</li>
<li><strong>Code Coverage</strong>. Traditional test coverage that most developers are probably already familiar with.</li>
<li><strong>Pre-Release Bugs</strong>. Number of issues in the issue tracker.</li>
</ul>
<p>In addition to these pre-existing models, the new model they made was:</p>
<ul>
<li><strong>Organizational Complexity</strong>. Measures number of developers workong on the module, number of ex-developers that <em>used</em> to work on the module but no longer does, how big a fraction of the organization as a whole that works or has worked on the module, the distance in the organization between the developer and the decision maker, etc.</li>
</ul>
<p>Microsoft published a study, where they used statistical methods to measure these different methods up against one another.</p>
<p>If you want details, and happen to understand Norwegian, you can listen to <a href="https://utviklingslandet.no/ep/2019-05-08/">this podcast episode</a> that I made. The short summary is that the two columns in the table below shows the following:</p>
<ul>
<li><strong>Precision</strong> - of the modules you thought had bugs, how many did actually have bugs?</li>
<li><strong>Recall</strong> - how many modules that had bugs, did we detect?</li>
</ul>
<p>The results below are Microsoft running their models against the various modules in Vista, and compared that with how many bugs the module actually had 6 months after Vista was released. They found the following:</p>
<table>
<thead>
<tr><th>Model</th><th>Precision</th><th>Recall</th></tr>
</thead>
<tbody>
<tr><td>Organizational Structure</td><td>86.2%</td><td>84.0%</td></tr>
<tr><td>Code Churn</td><td>78.6%</td><td>79.9%</td></tr>
<tr><td>Code Complexity</td><td>79.3%</td><td>66.0%</td></tr>
<tr><td>Dependencies</td><td>74.4%</td><td>69.9%</td></tr>
<tr><td>Code Coverage</td><td>83.8%</td><td>54.4%</td></tr>
<tr><td>Pre-Release Bugs</td><td>73.8%</td><td>62.9%</td></tr>
</tbody>
</table>
<p>Yowza!</p>
<p>The organizational structure has the highest precision - so when it predicted a module would have a bug, it has the highest success rate for doing so. <em>And</em> it has the highest recall - so it's the method that catched the highest amount of bugs.</p>
<p>That's pretty interesting, isn't it? The distance to decision makers and the number of developers working on a project is clearly and unambiguously the issue that is the best predictor of future problems with a code base.</p>
<p>Another shocking discovery for me personally, is that the only one that I've actually used myself - code coverage - has the lowest recall. It has a high precision, so bad code coverage does mean a high chance of bugs, i.e. low amount of false flags. But with a low recall, there are lots of bug that code coverage doesn't actually catch.</p>
<h2>And the study is replicated!</h2>
<p>Science is going through somewhat of a <a href="https://en.wikipedia.org/wiki/Replication_crisis">replication crisis</a>, where fundamental and highly cited studies has turned out to not be reproducible.</p>
<p>Thankfully, the predictive value of organizational complexity to software has been replicated!</p>
<p><a href="http://www.scs.ryerson.ca/~avm/dat/manuscripts/ICSE_2015.pdf">In the replicated study</a> the predictive value of organizational structure is not as high. Out of 4 measured models, it gets the 2nd highest precision and the 3rd highest recall. The study itself concludes that organizational complexity as a bug prediction method is worth investigating further. The study is also based on individual functions in C/C++, and not entire modules like Microsoft Research did, which can be a reason for at least a part of the discrepancy.</p>
<p>Also, the model for measuring organizational complexity doesn't measure the organization itself, compared to other organizations. I wouldn't be surprised if Microsoft is one of the most complex software organizations in the world. So maybe that's why organizational complexity was at the top of the list at Microsoft.</p>
<p>Another thing to consider is the difference between formal and informal organization structure, which none of the studies takes into account. The Microsoft study just automated its model against data from the HR system. Watch Ed Catmull of Pixar talk about this here (10:06)</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/k2h2lvhzMDc?start=606" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
The amazingly amazing amaze of lazy data structures2019-12-14T00:00:00.000+01:00August Lilleaasurn:augustl-com:feed:post::blog:2019:amazing_lazy_data_structures<p>Two things makes me keep coming back to Clojure: <a href="https://augustl.com/blog/2019/you_have_to_know_about_persistent_data_structures/">Persistent data structures</a> and lazy sequences.</p>
<p>This post is about lazy sequences.</p>
<h2>Lazy fibonacci</h2>
<p>To demonstrate lazy sequences, let's implement the Fibonacci sequence.</p>
<p>Here's a JavaScript version of the Clojure code to follow, so you have something to compare it to.</p>
<pre><code class="highlight"><span class="c1">// JavaScript version, for good measure</span>
<span class="kd">function</span> <span class="nx">fib</span><span class="p">(</span><span class="nx">n</span><span class="p">,</span> <span class="nx">xs</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">xs</span> <span class="o">=</span> <span class="nx">xs</span> <span class="o">||</span> <span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="mi">1</span><span class="p">]</span>
<span class="kr">const</span> <span class="nx">c</span> <span class="o">=</span> <span class="nx">xs</span><span class="p">.</span><span class="nx">length</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">c</span> <span class="o"><</span> <span class="nx">n</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="nx">fib</span><span class="p">(</span><span class="nx">n</span><span class="p">,</span> <span class="nx">xs</span><span class="p">.</span><span class="nx">concat</span><span class="p">([</span>
<span class="nx">xs</span><span class="p">[</span><span class="nx">xs</span><span class="p">.</span><span class="nx">length</span> <span class="o">-</span> <span class="mi">1</span><span class="p">]</span> <span class="o">+</span> <span class="nx">xs</span><span class="p">[</span><span class="nx">xs</span><span class="p">.</span><span class="nx">length</span> <span class="o">-</span> <span class="mi">2</span><span class="p">]</span>
<span class="p">]))</span>
<span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
<span class="k">return</span> <span class="nx">xs</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="nx">fib</span><span class="p">(</span><span class="mi">10</span><span class="p">)</span>
<span class="c1">// [0, 1, 1, 2, 3, 5, 8, 13, 21, 34]</span></code></pre>
<p>Here's the Clojure version of the same code.</p>
<pre><code class="highlight"><span class="c1">; "Plain" version</span>
<span class="p">(</span><span class="kd">defn </span><span class="nv">fib</span>
<span class="p">([</span><span class="nv">n</span><span class="p">]</span>
<span class="p">(</span><span class="nf">fib</span> <span class="nv">n</span> <span class="p">[</span><span class="mi">0</span> <span class="mi">1</span><span class="p">]))</span>
<span class="p">([</span><span class="nv">n</span> <span class="nv">xs</span><span class="p">]</span>
<span class="p">(</span><span class="k">let </span><span class="p">[</span><span class="nv">c</span> <span class="p">(</span><span class="nb">count </span><span class="nv">xs</span><span class="p">)]</span>
<span class="p">(</span><span class="k">if </span><span class="p">(</span><span class="nb">< </span><span class="nv">c</span> <span class="nv">n</span><span class="p">)</span>
<span class="p">(</span><span class="nf">fib</span> <span class="nv">n</span>
<span class="p">(</span><span class="nb">conj </span><span class="nv">xs</span>
<span class="p">(</span><span class="nb">+ </span><span class="p">(</span><span class="nb">nth </span><span class="nv">xs</span> <span class="p">(</span><span class="nb">- </span><span class="nv">c</span> <span class="mi">1</span><span class="p">))</span>
<span class="p">(</span><span class="nb">nth </span><span class="nv">xs</span> <span class="p">(</span><span class="nb">- </span><span class="nv">c</span> <span class="mi">2</span><span class="p">)))))</span>
<span class="nv">xs</span><span class="p">))))</span>
<span class="p">(</span><span class="nf">fib</span> <span class="mi">10</span><span class="p">)</span>
<span class="c1">; [0 1 1 2 3 5 8 13 21 34]</span></code></pre>
<p>All good, right?</p>
<p>Well, it's not the prettiest code in the world.</p>
<p>Let's fix that, by making literally the prettiest code in the world.</p>
<pre><code class="highlight"><span class="c1">; Lazy version - literally the prettiest </span>
<span class="c1">; code in the world!</span>
<span class="p">(</span><span class="kd">defn </span><span class="nv">fib</span>
<span class="p">([]</span>
<span class="p">(</span><span class="nf">fib</span> <span class="mi">0</span> <span class="mi">1</span><span class="p">))</span>
<span class="p">([</span><span class="nv">a</span> <span class="nv">b</span><span class="p">]</span>
<span class="p">(</span><span class="nf">lazy-seq</span> <span class="p">(</span><span class="nb">cons </span><span class="nv">a</span> <span class="p">(</span><span class="nf">fib</span> <span class="nv">b</span> <span class="p">(</span><span class="nb">+ </span><span class="nv">a</span> <span class="nv">b</span><span class="p">))))))</span>
<span class="p">(</span><span class="nb">take </span><span class="mi">10</span> <span class="p">(</span><span class="nf">fib</span><span class="p">))</span>
<span class="c1">; (0 1 1 2 3 5 8 13 21 34)</span></code></pre>
<p>Oh yass.</p>
<p>First of all, that's <em>much</em> less code.</p>
<p>Second of all, you can make it use BigDecimals and get <code>(fib 100000)</code> super fast, with good memory usage and no stack overflow and all that jazz.</p>
<pre><code class="highlight"><span class="c1">; Clojure uses long by default, so make it use </span>
<span class="c1">; bigdec for arbitrarily sized numbers</span>
<span class="p">(</span><span class="kd">defn </span><span class="nv">fib</span>
<span class="p">([]</span>
<span class="p">(</span><span class="nf">fib</span> <span class="p">(</span><span class="nf">bigdec</span> <span class="mi">0</span><span class="p">)</span> <span class="p">(</span><span class="nf">bigdec</span> <span class="mi">1</span><span class="p">)))</span>
<span class="p">([</span><span class="nv">a</span> <span class="nv">b</span><span class="p">]</span>
<span class="p">(</span><span class="nf">lazy-seq</span> <span class="p">(</span><span class="nb">cons </span><span class="nv">a</span> <span class="p">(</span><span class="nf">fib</span> <span class="nv">b</span> <span class="p">(</span><span class="nb">+ </span><span class="nv">a</span> <span class="nv">b</span><span class="p">))))))</span>
<span class="p">(</span><span class="nb">time </span><span class="p">(</span><span class="nb">last </span><span class="p">(</span><span class="nb">take </span><span class="mi">100000</span> <span class="p">(</span><span class="nf">fib</span><span class="p">))))</span>
<span class="c1">; "Elapsed time: 299.1678 msecs"</span>
<span class="c1">; 1605285768272[...20880 digits (yes,</span>
<span class="c1">; really) cut for brevity]790626M</span></code></pre>
<p>That's pretty cool.</p>
<p>By the way, the threading macro is awesome. You use it to write statements in the order that they are called.</p>
<pre><code class="highlight"><span class="c1">; Plain version</span>
<span class="p">(</span><span class="nb">time </span><span class="p">(</span><span class="nb">last </span><span class="p">(</span><span class="nb">take </span><span class="mi">100000</span> <span class="p">(</span><span class="nf">fib</span><span class="p">))))</span>
<span class="c1">; Awesome threading macro version</span>
<span class="p">(</span><span class="nf">->></span> <span class="p">(</span><span class="nf">fib</span><span class="p">)</span>
<span class="p">(</span><span class="nb">take </span><span class="mi">100000</span><span class="p">)</span>
<span class="p">(</span><span class="nf">last</span><span class="p">)</span>
<span class="p">(</span><span class="nf">time</span><span class="p">))</span></code></pre>
<p>So what actually <em>is</em> this laziness?</p>
<p>Well, the whole idea is that you separate two concepts: generating data, and extracting/using data.</p>
<p>See how the first version had both iteration, data generation, checking if our max <code>n</code> was reached etc in the same code?</p>
<p>We don't need to do that with laziness.</p>
<p>With the lazy version, we just define what the data structure looks like. You get a "lazy sequence" back, and you can think of it as a view of an infinitely sized Fibonacci sequence, that is not actually computed until you actually ask for some data from the sequence.</p>
<p>So it's the call to <code>(take)</code> that actually makes something happen.</p>
<pre><code class="highlight"><span class="c1">; "iterate" is also lay</span>
<span class="p">(</span><span class="nb">take </span><span class="mi">10</span> <span class="p">(</span><span class="nb">iterate inc </span><span class="mi">0</span><span class="p">))</span>
<span class="c1">; (0 1 2 3 4 5 6 7 8 9)</span>
<span class="c1">; So is "range"</span>
<span class="p">(</span><span class="nb">take </span><span class="mi">10</span> <span class="p">(</span><span class="nb">range </span><span class="mi">-50</span> <span class="mi">9000</span> <span class="mi">3</span><span class="p">))</span>
<span class="c1">; (-50 -47 -44 -41 -38 -35 -32 -29 -26 -23)</span></code></pre>
<h2>Lazy map</h2>
<p>This code prints absolutely nothing.</p>
<pre><code class="highlight"><span class="p">(</span><span class="nb">map </span><span class="p">(</span><span class="k">fn </span><span class="p">[</span><span class="nv">n</span><span class="p">]</span> <span class="p">(</span><span class="nb">print </span><span class="nv">n</span> <span class="s">""</span><span class="p">))</span>
<span class="p">[</span><span class="mi">1</span> <span class="mi">2</span> <span class="mi">3</span> <span class="mi">4</span> <span class="mi">5</span> <span class="mi">6</span> <span class="mi">7</span> <span class="mi">8</span> <span class="mi">9</span> <span class="mi">10</span><span class="p">])</span>
<span class="c1">; Nothing.. Absolutely nothing</span></code></pre>
<p>That's because <code>(map)</code> is also lazy. We have to actually get some items from it for the map function to be called.</p>
<pre><code class="highlight"><span class="c1">; Wait what, now?</span>
<span class="p">(</span><span class="nf">->></span> <span class="p">(</span><span class="nb">map </span><span class="p">(</span><span class="k">fn </span><span class="p">[</span><span class="nv">n</span><span class="p">]</span> <span class="p">(</span><span class="nb">print </span><span class="nv">n</span> <span class="s">""</span><span class="p">)</span> <span class="nv">n</span><span class="p">)</span>
<span class="p">[</span><span class="mi">1</span> <span class="mi">2</span> <span class="mi">3</span> <span class="mi">4</span> <span class="mi">5</span> <span class="mi">6</span> <span class="mi">7</span> <span class="mi">8</span> <span class="mi">9</span> <span class="mi">10</span><span class="p">])</span>
<span class="p">(</span><span class="nb">take </span><span class="mi">3</span><span class="p">))</span>
<span class="c1">; prints</span>
<span class="c1">; 1 2 3 4 5 6 7 8 9 10</span>
<span class="c1">; returns</span>
<span class="c1">; (1 2 3)</span></code></pre>
<p>Surprised? We ask for 3 items, but all 10 items seems to be printed! Is it not lazy after all?</p>
<p>It turns out that under the hood, Clojure will some times do a performance optimization and realize lazy sequences in batches of 32.</p>
<pre><code class="highlight"><span class="c1">; Batching</span>
<span class="p">(</span><span class="nf">->></span> <span class="p">(</span><span class="nb">map </span><span class="p">(</span><span class="k">fn </span><span class="p">[</span><span class="nv">n</span><span class="p">]</span> <span class="p">(</span><span class="nb">print </span><span class="nv">n</span> <span class="s">""</span><span class="p">)</span> <span class="nv">n</span><span class="p">)</span>
<span class="p">[</span><span class="mi">1</span> <span class="mi">2</span> <span class="mi">3</span> <span class="mi">4</span> <span class="mi">5</span> <span class="mi">6</span> <span class="mi">7</span> <span class="mi">8</span> <span class="mi">9</span> <span class="mi">10</span> <span class="mi">11</span> <span class="nv">...</span> <span class="mi">34</span> <span class="mi">35</span> <span class="mi">36</span><span class="p">])</span>
<span class="p">(</span><span class="nb">take </span><span class="mi">3</span><span class="p">))</span>
<span class="c1">; prints</span>
<span class="c1">; 1 2 3 4 5 6 7 8 9 10 11 ... 30 31 32 </span>
<span class="c1">; returns</span>
<span class="c1">; (1 2 3)</span></code></pre>
<p>Note: it will run the mapping function 32 times and "pre-cache" the lazy results - but you'll still get a sequence with 3 elements in it returned from <code>(take)</code>.</p>
<p>But, if your source is a lazy sequence and not a vector, it won't do that.</p>
<pre><code class="highlight"><span class="c1">; No batching when source is lazy</span>
<span class="p">(</span><span class="nf">->></span> <span class="p">(</span><span class="nb">map </span><span class="p">(</span><span class="k">fn </span><span class="p">[</span><span class="nv">n</span><span class="p">]</span> <span class="p">(</span><span class="nb">print </span><span class="nv">n</span> <span class="s">""</span><span class="p">)</span> <span class="nv">n</span><span class="p">)</span>
<span class="p">(</span><span class="nb">iterate inc </span><span class="mi">1</span><span class="p">))</span>
<span class="p">(</span><span class="nb">take </span><span class="mi">3</span><span class="p">))</span>
<span class="c1">; prints</span>
<span class="c1">; 1 2 3</span>
<span class="c1">; returns</span>
<span class="c1">; (1 2 3)</span></code></pre>
<h2>Generating PIN codes</h2>
<p>This one is pretty cool. Let's say you want to generate N random 4 digit pin codes.</p>
<p>You can generate an infinite lazy sequence of random numbers:</p>
<pre><code class="highlight"><span class="c1">; Possible duplicates!</span>
<span class="p">(</span><span class="nf">->></span> <span class="p">(</span><span class="nf">repeatedly</span> <span class="o">#</span><span class="p">(</span><span class="nb">rand-int </span><span class="mi">9999</span><span class="p">))</span>
<span class="p">(</span><span class="nb">take </span><span class="mi">5</span><span class="p">))</span>
<span class="c1">; (643 6000 483 8668 7493)</span></code></pre>
<p>But this list can contain duplicates! The chances are low when you just get 5 items obviously, but that doesn't matter - our requirement was that the list should be unique.</p>
<p>Enter the awesomeness of <code>(distinct)</code>. That function will take a list, and return a list with no duplicates - <em>and it is lazy!</em></p>
<pre><code class="highlight"><span class="c1">; No duplicates!</span>
<span class="p">(</span><span class="nf">->></span> <span class="p">(</span><span class="nf">repeatedly</span> <span class="o">#</span><span class="p">(</span><span class="nb">rand-int </span><span class="mi">9999</span><span class="p">))</span>
<span class="p">(</span><span class="nf">distinct</span><span class="p">)</span>
<span class="p">(</span><span class="nb">take </span><span class="mi">5</span><span class="p">)</span>
<span class="c1">; Also format it, for good measure</span>
<span class="p">(</span><span class="nb">map </span><span class="o">#</span><span class="p">(</span><span class="nf">format</span> <span class="s">"%04d"</span> <span class="nv">%</span><span class="p">)))</span>
<span class="c1">; ("6856" "3461" "8833" "4884" "0004")</span></code></pre>
<p>So good. So, so very good.</p>
What is WSL, and how can it possibly even?2019-12-13T00:00:00.000+01:00August Lilleaasurn:augustl-com:feed:post::blog:2019:what_is_wsl<p>Some of the other posts in my 2019 blog advent calendar has been about Windows. They all mention WSL. In this post, I'll focus only on WSL, and what it's all about, and how it works, and all that jazz.</p>
<h2>What is WSL?</h2>
<p>WSL is short for Windows Subsystem for Linux.</p>
<p>It's a Linux, made for your Windows.</p>
<p>Think of it like WINE, but the other way around. It's a thing that lets you run Linux software on Windows.</p>
<p>And it's <em>so good</em>.</p>
<h2>WSL is what enables Windows for me</h2>
<p>As I've <a href="https://augustl.com/blog/2019/choosing_windows_over_macos_linux/">blogged about before</a>, I've switched to Windows for my main development environment. The presence of WSL is the main enabler of this swap.</p>
<p>The issue for me is that every now and then, you'll come across software that just doesn't run well on Windows. This is almost never because of Windows itself, but because the developer community of which I'm a part tend to use use macOS or Linux. So they just haven't bothered running their software Windows.</p>
<p>I can't blame them.</p>
<p>So what do you do? Raise your hands, look at a nearby cloud, and yell at it?</p>
<p>No, you just run it in WSL.</p>
<h2>So what is WSL exactly</h2>
<p>WSL is a <em>full re-implementation of the Linux kernel</em> against the NT kernel.</p>
<p>In case you didn't know, NT used to be a version of windows, but it's actually the name of the kernel itself. It's powering Windows, X-Box, Windows Mobile, Windows Server, and Hololens.</p>
<p>Why is that cool?</p>
<p>One of the main problems with porting unix software to Windows, is that forking is incredibly slow. <code>fork()</code> is a very common system call do to on Linux and macOS. It creates a copy of the process that called <code>fork()</code>, and starts running it immediately. It uses fancy copy-on-write semantics on the RAM they share, etc. Windows <em>has</em> forking, but it has plenty of overhead. Windows was <em>not</em> designed for fast process creation, so it's pretty resource intensive.</p>
<p>But the NT kernel itself does not have this problem. It's solely a problem in the Windows API layer. A process in NT is super light weight.</p>
<p>So since WSL implements all the Linux system calls (like <code>fork()</code>) directly against NT, it's able to bypass the Windows stuff and make it super fast and lightweight!</p>
<p>This is one of the things that makes WSL <em>much</em> better than Cygwin. Cygwin provided the same sort of API that WSL does - ish. And that's a pretty big "ish". You had to re-compile the software against Cygwin, you couldn't just run native Linux binaries. And Cygwin implemented their calls against Win32, not the NT kernel.</p>
<p>WSL runs the existing Linux binaries that are already compiled for Linux. And runs them much faster.</p>
<h2>Do you want some <code>apt-get</code> with that?</h2>
<p>The other really cool thing, is that WSL is not just a kernel. It's actually a full distro.</p>
<p><em>Multiple</em> distros, in fact.</p>
<p>At the time of writing, you can choose Ubuntu 16.04 LTS, Ubuntu 18.04 LTS, Fedora, OpenSUSE, Debian, Alpine and a few others.</p>
<p>So you can install all the extra software you need with <code>apt-get</code> (or yum, or whatever, depending on your distro of choice).</p>
<p>For example, the other day I needed to convert an Apple iOS camera file in the HEIF format to jpg files. ImageMagic in apt doesn't have that, so I used apt to get the build-deps for ImageMagic, downloaded the sources, and compiled ImageMagic myself with just a handful of commands, with HEIF support enabled.</p>
<p>Good luck doing that easily under Windows (without WSL) or Cygwin.</p>
<h2>I gotta admit something</h2>
<p>There's one thing I never checked: How easy it is to compile ImageMagic on Windows. Maybe my point above is moot?</p>
<p>But who cares!</p>
<p>I don't <em>have</em> to figure that out, I already know how to do it on Linux, which means I know how to do it on WSL.</p>
<p>And I'm sure that there are cases when doing something with <code>apt</code> is much easier than doing it directly on Windows itself.</p>
<p><code>curl</code> is a good example. I usually need <code>curl</code> when there's some kind of emergenzy. "The thing is down, go check the thing!!". At that point I don't have the mental resources needed to be able to figure out how to do that in PowerShell. I do actually use PowerShell for most things, but not all things. I can just open a WSL terminal, with zero startup time, and do a good old <code>curl</code>.</p>
<p>Because why not?</p>
<h2>And it feels really <em>really</em> native</h2>
<p>If I start a server in Linux and bind it to port 3001, I can go to localhost:3001 in a browser in Windows, and it just works.</p>
<p>This is why it's amazing to not have any virtualization layer between me and Linux. It's all running against the same kernel, so when WSL calls NT that binds port 3001, it binds port 3001 for both Windows and WSL, just like that.</p>
<p>If you're so inclined, you can even get an X-Org server and run that on Windows, and set <code>DISPLAY</code> in ~/.bashrc in WSL, and you're good to go. It's a full Linux, isn't it? So that stuff just works as well. I don't really do this, because I haven't needed it. But it's nice to have the option, in case you need to run some GUI stuff that's easier to launch from Linux than from Windows.</p>
<h2>Enter... WSL2</h2>
<p>But hey. Apparently, none of this was good enough.</p>
<p>Because coming up is a remake of WSL, based on a real (thankfully upstream) Linux kernel, instead of the re-implementation against NT that is WSL 1.</p>
<p>Microsoft has <a href="https://devblogs.microsoft.com/commandline/announcing-wsl-2/">a blog post about this</a>. They use some fancy new virtualization technology to make sure there's no startup times, the environment is not isolated (so binding ports etc is just like WSL 1), and it has no resource usage overhead compared to WSL 1 (which has no overhead).</p>
<p>Apparently, one of the upsides of this is that with a real kernel running, things like Docker will start working under WSL. So you won't need to virtualize (manually) to use Docker, and tell it how many cores it should allocate and how much memory it should have available. Because apparently that's the sort of problem that WSL2 is not supposed to have.</p>
<p>I'm looking forward to it! I'm super happy with WSL the way it is now, but if it's gonna get even better, then that's good too.</p>
<p>They're also gonna try to fix the biggest gripe with WSL1 which is file system performance. NTFS just isn't as fast as ext4, and WSL is on top of NTFS. WSL2 uses some fancy something something which is gonna make everything much better, apparently.</p>
<p>You can probably tell I don't know a whole lot about WSL2. That's because it isn't released yet, and I haven't actually used it.</p>
<p>But the way I see it, all the changes Microsoft has made the last.. What.. 5 years? 10 years? Has all been steps in the right direction. So I expect WSL2 to continue that trend.</p>
Field manual - choosing React Native2019-12-12T00:00:00.000+01:00August Lilleaasurn:augustl-com:feed:post::blog:2019:field_manual_react_native<p>Here's my opinionated summarized thoughts of things to know about and take into consideration, when deciding whether or not to choose React Native for your next mobile app.</p>
<h2>What is React Native?</h2>
<p><a href="https://facebook.github.io/react-native/">React Native</a> is a cross platform native mobile app framework.</p>
<p>React Native is developed and maintained by Facebook.</p>
<p>React itself is primarily thought of as web technology. But React Native is <em>not</em> a web view wrapped in a native app.</p>
<p>React Native will render pure native GUIs, using the built-in native GUI widgets on iOS and Android.</p>
<p>A separate thread in your app will execute and evaluate JavaScript. This thread is where all your React code lives, which is written in JavaScript (or any language that compiles to JavaScript, like TypeScript, PureScript, etc).</p>
<p>With React, you return <code>div</code> and <code>p</code> elements when you render. In React Native, you either return specific named instances of native elements, like <code>ImagePickerIOS</code> and <code>ToastAndroid</code>. Or you return abstract React Native elements like <code>View</code>, <code>Text</code> and <code>Touchable</code>, for the elements where it makes sense to have the exact same render logic on iOS and Android.</p>
<p>Just like on the web, React Native maintains the previous and current versions of a virtual "DOM" (except it's not a "DOM", but native views), diffs them, and figures out what updates that needs to be made on the native side. Then, the web thread sends messages to the native component of React Native that runs inside UIKit or the Android GUI stack, and the native GUI is updated according to the changes that the JS thread determined needed to happen.</p>
<p>Architecturally speaking, the simplest way (in my opinion) to set things up is to create a plain iOS native app and a plain Android native app, include React Native in both of them, point it to your JS build server in dev or the compiled JS in release, and tell your native app to render its view using React Native. This sets up the bridge mentioned above, where a JS thread talks to the native side of things that you set up, and updates <em>real native views</em> as your app runs.</p>
<p>It seems to me that Facebook probably has some in-house tooling that it hasn't released yet. For example, navigation is not solved <em>at all</em>. Navigation requires its own section.</p>
<h2>It's nice that it's "just a native app"</h2>
<p>Need to e.g. send push notifications? Just do it.</p>
<p>I personally like this aspect of React Native a lot. I don't have to look up "how to send push notifications with React Native". My setup is that I have two completely normal native apps that I created myself, and they both pull in React Native and use that to render views.</p>
<p>So when I added support for push notifications, I just updated my plain native apps as you would do if you were actually making a plain native app. It's easy to send messages back and forth between your JS thread and the native world, so it's completely up to you how much you want to handle in native and how much you want to handle using JS.</p>
<p>Same with image pickers. I ended up writing 50-ish lines of native code in iOS and Android to listen to an event, open the native image picker, and emit an event back with "here's the image". No need to pull in a dependency to do this, no need to rewrite anything, etc.</p>
<h2>Navigation in React Native</h2>
<p>Getting navigation right is in my opinion the biggest caveat with React Native.</p>
<p>In a big part because you're completely on your own. React Native itself has nothing here to help you.</p>
<p>Writing it yourself is not recommended. The bridge between your JS code, which would define/declare the navigation structure, and your native code, is asynchronous. So it then falls on you to create a reliable asynchronous state replication system that calls asynchronous APIs on the JS side and asynchronous APIs on the native side, and hope there are no race conditions. I've been there, done that. Never again.</p>
<p>There are a few options that are dependencies you can drag in. <a href="https://reactnavigation.org">react-navigation</a> is a non-native implementation of navigation. In other words, if you use react-navigation on iOS to push and pop UIViewControllers, it won't actually use a <em>real</em> <code>UINavigationController</code>, but a re-implementation that sort of kind of <em>looks</em> like <code>UINavigationController</code>. Then there's <a href="https://wix.github.io/react-native-navigation/#/">react-native-navigation</a>, which implements the async native bridge I talked about in the previous paragraph. It turns out that this is not easy to do, and we had problems with the occasional strange race condition and crash. It also changes a lot, in a non-backwards compatible way, which rubs me the wrong way. Your experience may differ from mine, of course.</p>
<p>The last option, which is what I probably would have done if I were to start a new React Native app today, is to re-implement navigation twice, once for each platform. Android has some great new constructs for navigation using fragments and jetpack. iOS has storyboards, which among other things solves the problem of double navigation - if you double tap a button that calls "push" to go to a new screen, you risk navigating twice. Storyboards solves that.</p>
<p>For the types of apps I write, the navigation typically isn't <em>that</em> massive, it's the rendering of the views and the interaction and the data synchronization and stuff like that that's the biggest part of the codebase by far. So having a bunch of storyboard pages in iOS where all they do is to render their view using React Native, seems like a good choice to me.</p>
<h2>Performance</h2>
<p>There's no way to get around the fact that JavaScript has to be loaded and parsed as part of the startup procedures.</p>
<p>On iOS, you at least have the benefit of super fast JavaScript execution. But your app startup time <em>will</em> be slower. You can always split up your app in many parts, so that your login screen is a separate JS file, and when that's loaded you start loading the JS for the next screen in the background, etc. But in my experience, React Native is chosen because the team is small and fast delivery lead times is important, so this is not the sort of thing you typically spend much time fixing. Your mileage may vary.</p>
<p>On Android, however, things are looking up. Since Android is a more flexible runtime, technically speaking, than iOS, Facebook has created its own JS engine, specifically for React Native, called <a href="https://github.com/facebook/hermes">Hermes</a>. Essentially, all of the JS is compiled ahead of time, so it's just a matter of loading the compiled JS into memory and start executing it, giving load times similar to or as good as that of a pure native app.</p>
<p>Other than that, performance is good. Modern devices are good at executing JavaScript. Memory usage is higher of course, due to having all the JavaScript loaded (unless you set up some fancy partial loading, as mentioned), so you might have some issues with memory management on older devices.</p>
<h2>Sophisticated animations: write them as pure native</h2>
<p>In my experience, if you need sophisticated animations, you will end up hitting walls when implementing those in React Native. Write them as pure native instead.</p>
<p>First, writing something as pure native is easy. You can render a <code>MyFancyView</code> in your React code, and write that view yourself, in normal native, so you have full control of the specific GUI APIs you want to use to render your view just like you want to.</p>
<p>Secondly, the async bridge between your JavaScript and your native code is the biggest actual limitation here. Some animation APIs on the native side of things require immediate synchronous answers, and the native code simply can't do that, it has to wait for an asynchronous event loop on the JavaScript thread to get answers.</p>
<h2>Graphics: use bitmaps</h2>
<p>My instinct was to use vector graphics, because it's 2019. Don't do that, use bitmaps.</p>
<p>There are packages for rendering SVGs etc in React Native, but those are full re-implementations of SVGs and your mileage may vary, to say the least.</p>
<p>Android has some vector support, but iOS really prefers bitmaps (in different resolutions) so your life will be much easier if you just use bitmaps.</p>
<h2>Adopt?</h2>
<p>Personally, I would have tried Flutter the next time I were to create a native mobile app. But that's mostly because I have now actually tried React Native on a large app, and only used Flutter for smaller apps in my spare time. Maybe the grass is just greener on the other side :)</p>
<p>But, you should make up your own mind.</p>
Why I ditched macOS, Linux, and chose Windows for development work2019-12-11T00:00:00.000+01:00August Lilleaasurn:augustl-com:feed:post::blog:2019:choosing_windows_over_macos_linux<p>I’ve already told you the <a href="https://augustl.com/blog/2019/set_up_win10_dev_environment_for_macos_linux_users/">"how"</a>. Now it’s time for the "why".</p>
<h2>Actually, I don’t care what you use</h2>
<p>I just wanted to get this one out of the way. I have absolutely nothing against the use of macOS or Linux. Personal preferences are both <em>normal</em> and <em>OK</em>.</p>
<p>I’ve used all of them, extensively. My gaming PC is Windows. I've had many Macs. I've ran Linux on a Mac. I've had Thinkpads and Dells with Linux. I have switched to Windows before, and ended up switching back. Maybe that will happen again in the future. Those that know me, laugh at the idea that I started that sentence with "maybe".</p>
<p>What has changed?</p>
<h2>The big thing that is different now is WSL.</h2>
<p>I live in a unix world. All the servers I deploy to are unix. The software platforms I choose - Clojure, Kotlin, and other JVM languages - have one thing in common:</p>
<p>They're mostly used by people that use Linux or macOS.</p>
<p>So you're bound to run into problems on Windows. It's getting pretty rare, but it does happen. For example, a couple of years ago, I tried to compile ClojureScript on Windows, and it failed because of the way it shelled out to call npm for installing npm module dependencies. Almost nobody combines Windows and ClojureScript, at least two years ago. So my guess is that it simply never occurred to anyone to test it on Windows. It's fixed now. And you can sort of blame the JVM APIs that shelling out isn't made cross platform compatible. But the point remains the same. It'll happen.</p>
<p>But because of WSL, this is now a thing of the past.</p>
<h2>WSL seals the deal</h2>
<p>WSL is short for Windows Subsystem for Linux. It's a complete re-implementation of the Linux kernel system calls, against the NT kernel.</p>
<p>Think of it as the business version of WINE, that lets you run Windows programs on Linux. Just much, much better.</p>
<p>There has always been Cygwin. And I don't think I know anyone that uses Windows that didn't use to use Cygwin before WSL came along. But WSL is so much better than Cygwin.</p>
<p>I won't go into detail about how WSL works here, I'll save that for a separate blog post.</p>
<p>But suffice to say, WSL rocks. It's a proper Linux run-time, with no boot time, embedded right into Windows. And you can choose between a bunch of distros, so you have the Ubuntu <code>apt-get</code> repos right at your fingertips if you choose Ubuntu, for example. That's a world of difference from cygwin.</p>
<p>For example, last week I needed imagemagick with the extensions for reading the Apple camera file format HEIF. I just used apt to get the build-deps for imagemagick and built it myself with the necessary flags for HEIF support. Good luck doing that on cygwin.</p>
<p>I try to do most things directly on Windows, to get the hang of it. There are some things, though, such as curl, that I have absolutely no idea how to do on Windows. I just open a WSL shell and use curl. Real curl. On Linux. On Windows.</p>
<h2>Windows has a good terminal now</h2>
<p>This is a half truth. For ages, there's been <a href="https://cmder.net">cmdr</a>, and it works great. The option menus looks like someone used an AI to convert config files and option flags to a GUI, and it does require some setup.</p>
<p>So now, I'm using the <a href="https://www.microsoft.com/en-us/p/windows-terminal-preview/9n0dx20hk701">new official terminal from Microsoft</a>, which is in early access and available for anyone to try. I've used it for more than two months now and haven't had a single issue, so it's as good as stable in my book.
It's nice to be able to use ctrl+shift+c to copy, instead of having to do the weird right click insert enter stuff that I never quite got the hang of in the old built-in terminal in
Windows. It's <a href="https://github.com/Microsoft/Terminal">even open source</a> - you can build it yourself!</p>
<h2>I don't like Apple hardware</h2>
<p>The new 16" Macbook Pro is a step in the right direction.
But since 2015, Macbook Pro has been pretty much useless as far as I'm concerned. I don't like that lock-in.</p>
<p>I mean, I guess someone sees my Dell XPS 15 laptop and thinks "wow, poor guy, he doesn't have a touch bar" before they think anything else. But that person sure aint me. And I like that it has a thunderbolt 3 port for connecting my dock so I can just attach one single cable when I arrive at work. And that it also has USB-A ports and a HDMI port.</p>
<p>And I have a gamer gene. I like to have a wide variety of hardrware available, and I read news about the latest offerings from AMD and Intel. Have you seen the <a href="https://www.youtube.com/watch?v=WxocVricANg">latest episodes from LinusTechTips</a> about this? Do you want to rely on Apple and hope they make laptops with AMD CPUs in them? I mean maybe it will happen. But I like to not have to rely on one single company for my hardware.</p>
<h2>I'm not a huge fan of the window management in macOS</h2>
<p>Have you noticed that we're purely in the domain of taste now? Don't blame me! I said it in the intro. Your taste can be different from mine.</p>
<p>I really like that I can just hold the Windows key and press arrow up to maximize a window. And maximize means "fill the screen", not "whatever the developers of the app want".</p>
<p>One of my main gripes with macOS is that Command+tab switches between apps and not Windows. I have multiple windows of IntelliJ open at the same time, and I might have an incognito window of Firefox open to test some login stuff, and I want to switch betwen those two windows. And some times I work with two IntelliJ windows and one Firefox window sort of kind of at the same time. Then I want to have those in the glorious and predictable LRU cache that is alt+tab on Windows.</p>
<p>This is a pretty minor gripe. But it is actually quite annoying when I prefer the way Windows does it, and I'm constantly reminded that apparently macOS isn't made for people like me.
There are some plugins I found that makes Command+tab switch between windows, not apps. The best one I found shows a preview of the window contents that can be many minutes old, so I think I have an unread message in Slack, but then I saw that it was just an old preview I had in Command+tab. And every now and then, it just didn't show all the windows I had open.</p>
<p>Meh.</p>
<h2>I don't particularly like the state of software on Linux</h2>
<p>Heck, even Linus Torvalds agrees (6:27).</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/5PmHRSeA2c8?start=387" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
<p>I make a podcast, and i use Ableton Live to edit that. Ableton Live exists for macOS and Windows. That's it.</p>
<p>I somehow managed to set up Ubuntu wrong once, and set up an encrypted home folder instead of a fully encrypted disk. Because Linux, this is two completely different things, and Dropbox suddenly changed to support the latter, and not the former.</p>
<p>The issue, as Linus mentions above, is that there's no such thing as "linux". If you make software, you have to distribute it for Ubuntu, Debian, Fedora 23, Fedora 24, CentOS, Arch, ... The list goes on and on.</p>
<h2>This post is not meant to persuade you</h2>
<p>A long list of reasons that are mostly subjective is not very persuasive. And that's OK. I just wanted to tell you about why I chose Windows, that's all.</p>
<p>You can check out my article on <a href="https://augustl.com/blog/2019/set_up_win10_dev_environment_for_macos_linux_users/">how to set up a Windows development environment</a> if you want to!</p>
A short recipe for setting up the Microsoft SQL Server docker image2019-12-10T00:00:00.000+01:00August Lilleaasurn:augustl-com:feed:post::blog:2019:sql_server_docker_image_recipe<p>More Microsoft stuff.</p>
<p>At my current client, we're in the process of moving from Oracle to SQL Server. So far so good! In fact, there's a lot to like about SQL Server. The driver code is open source, for example, so when Weird Stuff happens in our app, we can look at the source code of the driver itself in the stack traces and see what it's up to.</p>
<p>Anyway. The docker image.</p>
<p>The Dockerfile itself is pretty plain.</p>
<pre><code class="highlight"><span class="k">FROM</span><span class="s"> mcr.microsoft.com/mssql/server:2019-GA-ubuntu-16.04</span>
<span class="k">ENV</span> ACCEPT_EULA y
<span class="k">ENV</span> SA_PASSWORD MyFancyPassword123
<span class="k">COPY</span> ./init.sql .
<span class="k">COPY</span> ./entrypoint.sh .
<span class="k">EXPOSE</span><span class="s"> 1433</span>
<span class="k">CMD</span> /bin/bash ./entrypoint.sh</code></pre>
<p>Obviously, you have to make sure you actually <a href="https://go.microsoft.com/fwlink/?LinkId=746388">accept the EULA</a>.</p>
<p>The <code>init.sql</code> file can have whatever SQL you want in it, to set it up. Here's a good starting point.</p>
<pre><code class="highlight"><span class="k">CREATE</span> <span class="k">DATABASE</span> <span class="n">myapp</span><span class="p">;</span>
<span class="k">go</span>
<span class="n">USE</span> <span class="n">myapp</span><span class="p">;</span>
<span class="k">go</span>
<span class="k">CREATE</span> <span class="n">LOGIN</span> <span class="n">myapp_login</span> <span class="k">WITH</span> <span class="n">PASSWORD</span><span class="o">=</span><span class="s1">'My_App_1234'</span><span class="p">,</span> <span class="n">DEFAULT_DATABASE</span><span class="o">=</span><span class="n">myapp</span><span class="p">;</span>
<span class="k">go</span>
<span class="k">CREATE</span> <span class="k">USER</span> <span class="n">myapp</span> <span class="k">FOR</span> <span class="n">LOGIN</span> <span class="n">myapp_login</span> <span class="k">WITH</span> <span class="n">DEFAULT_SCHEMA</span><span class="o">=</span><span class="n">dbo</span><span class="p">;</span>
<span class="k">go</span>
<span class="k">ALTER</span> <span class="k">ROLE</span> <span class="n">db_owner</span> <span class="k">ADD</span> <span class="n">MEMBER</span> <span class="n">myapp</span><span class="p">;</span>
<span class="k">go</span></code></pre>
<p>The interesting part is the entrypoint.</p>
<p>The main difficulty here is to set up the docker image so that the init script runs at startup. The PostgreSQL docker image has a neat API where you can put sql scripts or shell scripts in <code>/docker-entrypoint-initdb.d/</code> (PostgreSQL only). SQL Server has nothing like that, so we have to do something else.</p>
<p>Thankfully, Microsoft has documentet a stable API for how to start the actual SQL Server instance. So we can use that to start it manually, and override the built-in entrypoint.</p>
<p>Here's <code>entrypoint.sh</code>:</p>
<pre><code class="highlight"><span class="ch">#!/bin/bash</span>
<span class="c1"># Run init-script with long timeout - and make it run in the background</span>
/opt/mssql-tools/bin/sqlcmd -S localhost -l <span class="m">60</span> -U SA
<span class="se">\ </span>-P <span class="s2">"MyFancyPassword123"</span> -i init.sql <span class="p">&</span>
<span class="c1"># Start SQL server</span>
/opt/mssql/bin/sqlservr</code></pre>
<p>The last part is the default entry point. By running sqlcmd with a 60 second timeout and pointing it to <code>init.sql</code>, and immediately backgrounding it with that <code>&</code> in there, we're making sure that the <code>init.sql</code> script runs on startup, and that the docker file is properly started up.</p>
<p>That's about it!</p>
My all-time favorite talks2019-12-09T00:00:00.000+01:00August Lilleaasurn:augustl-com:feed:post::blog:2019:my_all_time_favorite_talks<p>It's sharing time!</p>
<h2>Rich Hickey: <em>That</em> talk</h2>
<p><strong>Simple Made Easy</strong> is the most mind bending talk I've ever seen. I think most Clojure programmers will tell you that. I'm not really a <em>Clojure programmer</em>, I like lots of things, but I do have an extra special relationship with Clojure, Datomic and the thoughts that lies underneath the surface.</p>
<p>This talk is about Clojure, without really mentioning Clojure.</p>
<a href="https://www.infoq.com/presentations/Simple-Made-Easy/">
<p><img src="/static/posts/all_time_favorite_talks/simple_made_easy.jpg">
<p><a href="https://www.infoq.com/presentations/Simple-Made-Easy/">https://www.infoq.com/presentations/Simple-Made-Easy/</a></p>
<h2>Kathryn McKinley: How Google makes search fast</h2>
<p>Google clocks down CPUs to save power, but if a search is taking too long they'll clock up a core and re-run the search there. The talk goes in detail about how they figured out where to set the thresholds here, and what they do to achieve their goal of having 98% of searches complete in under 200 milliseconds. (Or was it 99% and 300 milliseconds? Don't remember the exact details there.)</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/_Zoa3xkzgFk" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
<p><a href="https://www.thestrangeloop.com/2017/measuring-and-optimizing-tail-latency.html">https://www.thestrangeloop.com/2017/measuring-and-optimizing-tail-latency.html</a></p>
<h2>Byrd & Friedman: Writing programs that write themselves as output - and more shenaningans</h2>
<p>For example, if you fast forward to 0:45:00, you'll see how they generate all possible programs that return the number 6.</p>
<p>Pretty mind blowing that this stuff is even possible</p>
<iframe title="vimeo-player" src="https://player.vimeo.com/video/66548859" width="640" height="360" frameborder="0" allowfullscreen></iframe>
<p><a href="http://2013.flatmap.no/danwill.html">http://2013.flatmap.no/danwill.html</a></p>
<h2>Guy Steele: the worst program ever written</h2>
<p>It's pretty cool to see how horrible software used to be in the 70s. He talks about cool auto-parallelism stuff in his own language, Fortress, with the context of all the things we used to have to think about when writing software in the (not so good) old days.</p>
<a href="https://www.infoq.com/presentations/Thinking-Parallel-Programming/">
<p><img src="/static/posts/all_time_favorite_talks/parallell_programming.jpg">
<p><a href="https://www.infoq.com/presentations/Thinking-Parallel-Programming/">https://www.infoq.com/presentations/Thinking-Parallel-Programming/</a></p>
<h2>Strange Loop</h2>
<p>A bit of meta at the end. 3 of the 4 talks above are from <a href="https://thestrangeloop.com/">Strange Loop</a>. And the last one <em>could</em> have been from Strange Loop - they've had a few talks there as well, it's just that when I saw them, it was at <a href="https://flatmap.no">flatMap</a>.</p>
<p>You should <em>totally</em> go to Strange Loop.</p>
You have to know about persistent data structures2019-12-08T00:00:00.000+01:00August Lilleaasurn:augustl-com:feed:post::blog:2019:you_have_to_know_about_persistent_data_structures<p>The first thing I look for when checking out a new programming language is immutable persistent data structures. This is an injury I've gotten from Clojure, which has them out of the box. I tend to structure my whole app around these kinds of data structures. They are smart, fast, efficient, immutable and amazing.</p>
<p>In this post, I'll try to make you feel the same way about them.</p>
<h2>Axiom: immutability is good</h2>
<p>I'm not going to try to sell this one. Because it's a tough sell.</p>
<p>As in, I know a few people that don't like immutable values, and I have no idea how to sell it to them.</p>
<p>Personally, I like to use something that fundamentally eliminates a whole category of bugs from my code.</p>
<p>Sorry. Couldn't resist being snarky. I don't want to trash talk those who don't like immutability. But I just don't get it.</p>
<h2>The problem with immutable values is that they're immutable</h2>
<p>Let's say you have an immutable hash map or something. It's implemented using some kind of tree structure under the hood, like a lot of data structures are.</p>
<p><img src="/static/posts/you_have_to_Know_about_persistent_data_structures/the_basic.png">
<p>The problem happens when we want to add a key to this map. The data structure is immutable, so we can't change it!</p>
<p>So, we have to create a whole new copy of everything, an add our new stuff to it.</p>
<p><img src="/static/posts/you_have_to_Know_about_persistent_data_structures/complete_clone.png">
<p>(Red = new stuff)</p>
<p>A new node was added at the bottom right of the image. And we had to copy <em>all the data</em> in the tree structure, since we can't change the immutable original.</p>
<p>So that's the end of the story, then? Immutability sucks, because you have to copy everything all the time you want to change something?</p>
<p>Well, yes.</p>
<p>But, also, no.</p>
<h2>Copy on write - nah</h2>
<p>I mean, I'm not gonna diss copy on write completely. There does exist some mildly useful software that leverages it, such as the Linux Kernel.</p>
<p>The whole central and important concept of forking a process in Linux/Unix, relies on copy on write (CoW). When you have a process that uses 11GB of RAM, and you fork it, you have a new process, identical to the original one, with its own copy of the 11GB of RAM. To make the process of forking not horribly tear-inducingly slow, the kernel relies on CoW. The action of forking doesn't actually do anything to those 11GBs of RAM. It's only when you start writing to memory from the fork, that the kernel starts copying and then writing - copy on write - the memory of the original process.</p>
<p>CoW is technically speaking a type of persistent data structure.</p>
<p>However, CoW is at its most useful when the data is mutable. When your data is immutable to the core, you have another option.</p>
<p>That other option is...</p>
<h2>Structural sharing - yass</h2>
<p>Now <em>this</em> is my kind of persistent data structure.</p>
<p>So.</p>
<p>We have this huge data structure. We know that it is immutable. We want to change one little piece of it. What optimization can we run on this process?</p>
<p>Do you see it yet?</p>
<p>The old data structure <em>is immutable</em>. We can just use that!</p>
<p><img src="/static/posts/you_have_to_Know_about_persistent_data_structures/structural_sharing.png">
<p>(Red = new stuff)</p>
<p>See what we did? Why copy immutable data?</p>
<p>We just need to create a new root node, a new 2nd node, and a new data node, and then just point to the old stuff! The old stuff is immutable, after all. The whole idea is that it will never change.</p>
<p>You just understood the Clojure state model. The values themselves are immutable. You can have functions that takes an immutable value, that return an updated immutable value. Creating new immutable values is cheap - because you can share almost all of the data with the old immutable value.</p>
<p>Yay!</p>
<h2>Clojure does another really cool thing</h2>
<p>In the real world, it turns out that if you just have a small map with just a handful of keys, operating on a fancy tree structure can be kind of expensive.</p>
<p>Under the hood, Clojure will do a very cool optimization: if your map has fewer than 8 (I think) items in it, it will just be represented as a normal mutable list under the hood.</p>
<p>When you add an item, Clojure will make a full copy of that list, containing the old items and the newly added item.</p>
<p>Why is that cool?</p>
<p>Well, it turns out that it's actually faster to do it that way, when you only have a small amount of data. Checking a list of 8 items for whether or not your key exists in it, is much faster than having to maintain and balance a binary tree or a b-tree.</p>
<p>Think of it like garbage collection generations. Under the hood, your stuff will be in the "eden" space or "tenured" space. Which is fancy speak for: stuff that is just created and immediately discarded is garbage collected differently than long lived stuff. This happens completely under the hood, as a performance optimization, without you having to do anything special to make it happen.</p>
<h2>An other another cool thing!</h2>
<p>Let's say you have this function in a tight loop that adds 100s of items to a map, based on a data stream of some kind. Maybe a CSV or a database or something like that.</p>
<p>By default, all maps are immutable, so that means you have to do 100s of potentially expensive operations on the persistent data structures.</p>
<p>However! Clojure has transients.</p>
<p>Transients are a special version of immutable data structures. You first "unlock" it and create a "transient" map (or list or set or ...).</p>
<p>Here's a normal immutable example:</p>
<pre><code class="highlight"><span class="p">(</span><span class="nf">->></span> <span class="p">[</span><span class="mi">1</span> <span class="mi">2</span> <span class="mi">3</span> <span class="mi">4</span><span class="p">]</span>
<span class="p">(</span><span class="nf">reduce</span>
<span class="p">(</span><span class="k">fn </span><span class="p">[</span><span class="nv">res</span> <span class="nv">curr</span><span class="p">]</span> <span class="p">(</span><span class="nb">conj </span><span class="nv">res</span> <span class="p">(</span><span class="nb">+ </span><span class="mi">1</span> <span class="nv">curr</span><span class="p">)))</span>
<span class="p">[]))</span>
<span class="c1">;; [2 3 4 5]</span></code></pre>
<p>Here it is with transients. Notice how we have to use special versions of the functions, like <code>conj!</code> instead of <code>conj</code>.</p>
<pre><code class="highlight"><span class="p">(</span><span class="nf">->></span> <span class="p">[</span><span class="mi">1</span> <span class="mi">2</span> <span class="mi">3</span> <span class="mi">4</span><span class="p">]</span>
<span class="p">(</span><span class="nf">reduce</span>
<span class="p">(</span><span class="k">fn </span><span class="p">[</span><span class="nv">res</span> <span class="nv">curr</span><span class="p">]</span> <span class="p">(</span><span class="nf">conj!</span> <span class="nv">res</span> <span class="p">(</span><span class="nb">+ </span><span class="mi">1</span> <span class="nv">curr</span><span class="p">)))</span>
<span class="p">(</span><span class="nf">transient</span> <span class="p">[])))</span>
<span class="c1">;; clojure.lang.PersistentVector$TransientVector@139e6e1d</span></code></pre>
<p>Whoops! We forgot to make it persistent:</p>
<pre><code class="highlight"><span class="p">(</span><span class="nf">->></span> <span class="p">[</span><span class="mi">1</span> <span class="mi">2</span> <span class="mi">3</span> <span class="mi">4</span><span class="p">]</span>
<span class="p">(</span><span class="nf">reduce</span>
<span class="p">(</span><span class="k">fn </span><span class="p">[</span><span class="nv">res</span> <span class="nv">curr</span><span class="p">]</span> <span class="p">(</span><span class="nf">conj!</span> <span class="nv">res</span> <span class="p">(</span><span class="nb">+ </span><span class="mi">1</span> <span class="nv">curr</span><span class="p">)))</span>
<span class="p">(</span><span class="nf">transient</span> <span class="p">[]))</span>
<span class="p">(</span><span class="nf">persistent!</span><span class="p">))</span>
<span class="c1">;; [2 3 4 5]</span></code></pre>
<p>This optimization allows Clojure to know that in your tight loop, nobody else will actually use the data structure, so it can be "less fancy" in how it builds it up. And much faster, skipping the entire potential overhead of managing a persistent data structure.</p>
<p>Clojure has pretty cool data structures.</p>
Field manual: choosing Flutter2019-12-07T00:00:00.000+01:00August Lilleaasurn:augustl-com:feed:post::blog:2019:field_manual_flutter<p>Here's my opinionated summarized thoughts of things to take into consideration when deciding whether or not to choose Flutter.</p>
<h2>What is Flutter?</h2>
<p><a href="https://flutter.dev">Flutter</a> is a cross platform native mobile app framework.</p>
<p>Flutter is developed and maintained by Google.</p>
<p>It's heavily tied into the <a href="https://dart.dev">Dart (programming language)</a> ecosystem. So you're gonna be writing all of your code in Dart. The Dart code is compiled into native iOS or native Android code.</p>
<p>Flutter does not use web technology for anything.</p>
<p>Flutter also does not use existing native GUI frameworks for anything - Flutter <em>completely re-implements the GUI stack, using their own rendering engine</em>.</p>
<p>You can almost think of flutter like a 3D gaming engine, except that it's for 2D GUIs. Your app will show a Flutter view, and the Flutter view will do everything itself. Flutter renders <em>absolutely everything</em>.</p>
<p>So when you see a table view and scroll it, you're not seeing <em>anything</em> from the Android or iOS native code, you're seeing Flutter.</p>
<p>You can choose between a material (Android-like) look, a "cupertino" look (iOS-like), or take complete control of the look and feel yourself. If you choose "cupertino", you app will look exactly like an iOS app on iOS <em>and</em> Android. Vice versa for material. This means that Google has put a great effort into making the respective themes look and feel completely like their normal native counterparts. And from what I can tell, they've succeeded. There are no obvious tells that something is "a Flutter app".</p>
<p>Flutter is a well put together system. Some systems gives me the impression that it's only partly open sourced, and that the vendor has internal utils that are vital but not released. That is <em>not</em> the case with Flutter. Flutter has solutions and APIs for all the problems you typically encounter.</p>
<h2>Why is Flutter a good idea?</h2>
<p>Flutter is a good fit for small teams that needs to make apps that works on both iOS and Android.</p>
<p>It's an even better fit if you don't want your app to look and feel like a native app.</p>
<p>That's actually quite common these days in my experience. A lot of the apps on my home screen don't look even a little bit native (Notion, Spotify, Snapchat, Vipps, IKEA Home Smart, Duolingo, Economist Espresso). For that kind of app, there's no need to choose a framework that makes your app look native.</p>
<p>You do have that option, of course. There are constructs in Flutter to have your app dynamically switch between themes, and use the material theme on Android and the Cupertino theme on iOS.</p>
<h2>Mixing Flutter and "normal" native</h2>
<p><em>Note: in the apps I've made with Flutter, I'm yet to use this functionality myself, so this is not first hand knowledge or experience.</em></p>
<p>When you create a new Flutter app, it generates an iOS app and an Android app for you. These are just normal iOS apps, that are set up so that the first thing that renders after the app launches, is the single Flutter view, that runs your Dart code to render the GUI.</p>
<p>This means that for rendering "normal" native views, you have two options.</p>
<p>One is to simply navigate away from the Flutter view, and open a normal native view. Since Flutter runs under a normal native app that you can essentially do what you want with, just modify the generated apps to support navigation into another view controller or fragment or activity. The Flutter project has <a href="https://github.com/flutter/flutter/tree/master/examples/platform_view">provided examples and best practices</a> for how to do that.</p>
<p>The other is to use a new mechanism in Flutter to embed views from native inside your Flutter app. This is called "platform views" and relies on <a href="https://api.flutter.dev/flutter/widgets/AndroidView-class.html">AndroidView</a> or <a href="https://api.flutter.dev/flutter/widgets/UiKitView-class.html">UIKitView</a>. This is early access stuff and work in progress, but you have the option.</p>
<p>There are also packages available for embedding some of the more common views, relying on the "platform view" mechanisms. One such exapmle is the <a href="https://pub.dev/packages/google_maps_flutter">Google Maps package</a>.</p>
<h2>Does a completely new rendering engine make sense?</h2>
<p>It does.</p>
<p>First and foremost, you get the exact same rendering engine on both iOS and Android. So you can make fast, smooth - and intricate - native animations, and have them work just as you expect on both platforms. You almost don't have to do any cross platform testing of that sort of thing. Flutter does not piggyback on any native GUI rendering capabilities, so a difference between platforms is a bug in Flutter, not a bug in your code. And Flutter is widely used and well tested at this point.</p>
<p>Secondly, you get a <em>different</em> rendering engine than other rendering engines. It's made to be declarative to the core. So Flutter can completely skip the reconsiliation step between the declared GUI and the actual rendered platform GUI, which is what React has to do. And personally, I like that it exposes a whole lot of useful constructs to you, that you can play around with. There are defined APIs for almost everything, and very little is hidden from you. In my experience, Flutter is the most powerful and flexible rendering engine I've ever worked with.</p>
<h2>Flutter Web</h2>
<p>Flutter for native mobile apps doesn't use web technology for anything.</p>
<p>But you can <a href="https://flutter.dev/web">use Flutter to make web apps</a>.</p>
<p>More specifically: the Flutter rendering engine has been ported, and implemented using web technology.</p>
<p>If you prefer to work with the Flutter rendering API instead of the CSS/DOM APIs directly, this can be a nice productivity boost. Interaction and animation heavy GUIs can be easier to implement using Flutter, especially if you have some prior knowledge of Flutter and the various APIs it exposes.</p>
<p>One of the last remaining problems of web GUIs, despite all the progress that is being made these days, is that it's difficult to display huge lists/tables of information and maintain 60 fps scrolling speed, fast initial render, and low memory usage.</p>
<p>With Flutter web, this is a problem of the past.</p>
<p>This means, obviously, that it <em>is</em> possible to make performant table views using CSS/DOM. Because Flutter web renders to the DOM, it doesn't render to a canvas or WebGL or anything like that. Flutter makes those rather abstract APIs much more accessible, though, through a familiar lazy list view API.</p>
<p>I might end up blogging more about the details of this at a later time. It's a very interesting subject.</p>
<h2>Adopt?</h2>
<p>I'm not a tech radar. Make up your own mind. :)</p>
The important difference between layout and rendering2019-12-06T00:00:00.000+01:00August Lilleaasurn:augustl-com:feed:post::blog:2019:the_important_difference_between_layout_and_rendering<p>Layout and rendering reminds me of variable names in math. What the heck is "c"? Well, that's the speed of light, obviously! And what about "L"? Angular momentum, silly! There's a <a href="https://en.wikipedia.org/wiki/List_of_letters_used_in_mathematics_and_science">whole list of these</a>.</p>
<p>Different rendering engines might have different names for these. But the concept of separating layout and rendering exists in every rendering engine I'm aware of.</p>
<p>Let's see how this works, using the web and HTML as an example.</p>
<h2>How to cause "layout" in plain old HTML</h2>
<p>Allright, we have a div, with plain old HTML. No JavaScript, no React.</p>
<pre><code class="highlight"><span class="p"><</span><span class="nt">div</span><span class="p">></span>hai<span class="p"></</span><span class="nt">div</span><span class="p">></span></code></pre>
<p>Nothing special about it.</p>
<p>Let's give it a margin.</p>
<pre><code class="highlight"><span class="p"><</span><span class="nt">div</span> <span class="na">style</span><span class="o">=</span><span class="s">"margin-left: 20px;"</span><span class="p">></span>hai<span class="p"></</span><span class="nt">div</span><span class="p">></span></code></pre>
<p>And BOOM, the browser has to do a "layout" pass.</p>
<p>Why?</p>
<p>Because another completely unrelated element might <em>also</em> have to change <em>its</em> layout when you change a margin.</p>
<p>For example, let's say your actual full HTML looks like this:</p>
<pre><code class="highlight"><span class="p"><</span><span class="nt">div</span> <span class="na">style</span><span class="o">=</span><span class="s">"width: 200px; background: #c1b4ae;"</span><span class="p">></span>
<span class="p"><</span><span class="nt">div</span> <span class="na">style</span><span class="o">=</span><span class="s">"background: #be5a38; display: inline-block; width: 160px;"</span><span class="p">></span>
foo
<span class="p"></</span><span class="nt">div</span><span class="p">></span>
<span class="p"><</span><span class="nt">div</span> <span class="na">style</span><span class="o">=</span><span class="s">"background: #92140c; display: inline-block;"</span><span class="p">></span>
hai
<span class="p"></</span><span class="nt">div</span><span class="p">></span>
<span class="p"></</span><span class="nt">div</span><span class="p">></span></code></pre>
<p>It looks like this:</p>
<div style="width: 200px; background: #c1b4ae;">
<div style="background: #be5a38; display: inline-block; width: 160px;">
foo
</div>
<div style="background: #92140c; display: inline-block;">
hai
</div>
</div>
<p></p>
<p>Then we add a margin.</p>
<pre><code class="highlight"><span class="c"><!-- This causes new layout! --></span>
<span class="p"><</span><span class="nt">div</span> <span class="na">style</span><span class="o">=</span><span class="s">"width: 200px; background: #c1b4ae;"</span><span class="p">></span>
<span class="p"><</span><span class="nt">div</span> <span class="na">style</span><span class="o">=</span><span class="s">"background: #be5a38; display: inline-block; width: 160px;"</span><span class="p">></span>
foo
<span class="p"></</span><span class="nt">div</span><span class="p">></span>
<span class="p"><</span><span class="nt">div</span> <span class="na">style</span><span class="o">=</span><span class="s">"background: #92140c; display: inline-block; margin-left: 20px;"</span><span class="p">></span>
hai
<span class="p"></</span><span class="nt">div</span><span class="p">></span>
<span class="p"></</span><span class="nt">div</span><span class="p">></span></code></pre>
<p>Which then looks like this:</p>
<div style="width: 200px; background: #c1b4ae;">
<div style="background: #be5a38; display: inline-block; width: 160px;">foo</div>
<div style="background: #92140c; display: inline-block; margin-left: 20px;">hai</div>
</div>
<p></p>
<p>See?</p>
<p>A different element on the page had to change! The surrounding grey box in the background had to be made larger.</p>
<p>And we didn't change anything on that box.</p>
<p>What if we had another box around that?</p>
<p>And then another box around that?</p>
<p>And then maybe a sidebar somewhere, that now moved a little bit?</p>
<p>See where this is going?</p>
<p>If you try to animate an element smoothly using "margin", you're out of luck. Every time you change the margin, the browser has to check a lot of extra stuff to see what else has to be rendered as well as the element that you changed.</p>
<p>This is the "layout" phase. It's the expensive part.</p>
<h2>How to cause "rendering" on the web</h2>
<p>Short answer: you don't cause it. It happens all the time.</p>
<p>It's true!</p>
<p>Under the hood, your entire UI is rendered, 60 times every second, from scratch. <em>All</em> of it.</p>
<p>The browser uses the cached layout structure when rendering, so the browser knows the size of all the boxes, where they should be on the page, and all that jazz.</p>
<p>Typically, each box or section of your GUI has ended up as a texture in OpenGL or Metal or something like that. Your GPU will then paint the textures in the correct position on the screen.</p>
<p>The GPU is <em>really super ultra fast</em> at doing this.</p>
<p>That's why "rendering" is fast.</p>
<h2>Why does that matter?</h2>
<p>Good question.</p>
<p>If you want rendering to be fast, you got to ask yourself:</p>
<ul>
<li>Does this cause full layout?</li>
<li>Or does it just cause a new render with the existing laout?</li>
</ul>
<p>This is super important. Your buttery smooth 60 fps animations does <em>not</em> want to cause a new layout. You just want to tweak the rendering.</p>
<p>There are two main ways of doing that.</p>
<h2>Creating a new texture</h2>
<p>Let's say you have the divs from above:</p>
<pre><code class="highlight"><span class="p"><</span><span class="nt">div</span> <span class="na">style</span><span class="o">=</span><span class="s">"width: 200px; background: #c1b4ae;"</span><span class="p">></span>
<span class="p"><</span><span class="nt">div</span> <span class="na">style</span><span class="o">=</span><span class="s">"background: #be5a38; display: inline-block; width: 160px;"</span><span class="p">></span>
foo
<span class="p"></</span><span class="nt">div</span><span class="p">></span>
<span class="p"><</span><span class="nt">div</span> <span class="na">style</span><span class="o">=</span><span class="s">"background: #92140c; display: inline-block; margin-left: 20px;"</span><span class="p">></span>
hai
<span class="p"></</span><span class="nt">div</span><span class="p">></span>
<span class="p"></</span><span class="nt">div</span><span class="p">></span></code></pre>
<p>Which then looks like this:</p>
<div style="width: 200px; background: #c1b4ae;">
<div style="background: #be5a38; display: inline-block; width: 160px;">foo</div>
<div style="background: #92140c; display: inline-block; margin-left: 20px;">hai</div>
</div>
<p></p>
<p>All we want to do, is to change the color on one of them, from <span style="color: #fff; background-color: #92140c">Pretty Smooth Red</span> to <span style="color: #fff; background-color: blue">Ugly Web Blue</span>.</p>
<pre><code class="highlight"><span class="c"><!-- This does _not_ cause new layout, just new render --></span>
<span class="p"><</span><span class="nt">div</span> <span class="na">style</span><span class="o">=</span><span class="s">"width: 200px; background: #c1b4ae;"</span><span class="p">></span>
<span class="p"><</span><span class="nt">div</span> <span class="na">style</span><span class="o">=</span><span class="s">"background: #be5a38; display: inline-block; width: 160px;"</span><span class="p">></span>
foo
<span class="p"></</span><span class="nt">div</span><span class="p">></span>
<span class="p"><</span><span class="nt">div</span> <span class="na">style</span><span class="o">=</span><span class="s">"background: blue; display: inline-block; margin-left: 20px;"</span><span class="p">></span>
hai
<span class="p"></</span><span class="nt">div</span><span class="p">></span>
<span class="p"></</span><span class="nt">div</span><span class="p">></span></code></pre>
<div style="width: 200px; background: #c1b4ae;">
<div style="background: #be5a38; display: inline-block; width: 160px;">foo</div>
<div style="background: blue; display: inline-block; margin-left: 20px;">hai</div>
</div>
<p></p>
<p>The browser knows that there is absolutely no way that changing the background-color of an element needs a new layout. So all it has to do, is to create a new texture for that specific div, and then just do a new rendering. Now, generating the texture has the potential of being somewhat expensive. But it's <em>nothing</em> compared to a full layout, which we promptly avoided just now.</p>
<h2>Tweaking the texture</h2>
<p>Now, CSS has a separate property for just affecting the rendering stage. Let's move a box, without causing layout!</p>
<pre><code class="highlight"><span class="c"><!-- This also doesn't cause new layout, just new render --></span>
<span class="p"><</span><span class="nt">div</span> <span class="na">style</span><span class="o">=</span><span class="s">"width: 200px; background: #c1b4ae;"</span><span class="p">></span>
<span class="p"><</span><span class="nt">div</span> <span class="na">style</span><span class="o">=</span><span class="s">"background: #be5a38; display: inline-block; width: 160px;"</span><span class="p">></span>
foo
<span class="p"></</span><span class="nt">div</span><span class="p">></span>
<span class="p"><</span><span class="nt">div</span> <span class="na">style</span><span class="o">=</span><span class="s">"transform: translate(20px, -5px); background: blue; display: inline-block; margin-left: 20px;"</span><span class="p">></span>
hai
<span class="p"></</span><span class="nt">div</span><span class="p">></span>
<span class="p"></</span><span class="nt">div</span><span class="p">></span></code></pre>
<div style="width: 200px; background: #c1b4ae;">
<div style="background: #be5a38; display: inline-block; width: 160px;">foo</div>
<div style="background: blue; display: inline-block; margin-left: 20px; transform: translate(20px, -5px)">hai</div>
</div>
<p></p>
<p>We only changed the "transform" property. Notice how it moves without affecting the layout at all? That's because it doesn't affect the layout - by design.</p>
<p>It's super fast, because it's only the GPU that has to care.</p>
<p>When everything is rendered, it gets the result of the layout that was performed earlier, to see where everything should be positioned.</p>
<p>Then, it applies the transform <em>at the GPU level</em> to the texture. If it's something a GPU can do fast, it's applying transforms. It's the specialty of a GPU.</p>
<p>So that's almost like not doing any extra work at all as far as the GPU is concerned. It just takes the existing texture, positions it based on the existing layout information, and performs a super cheap transform to figure out where to draw it.</p>
<h2>Not just the web</h2>
<p>I use web and HTML as an example here, but all rendering engines that I'm familiar with employs techniques like this.</p>
<p>The terminology might not be the same in all of them, but the principle is there. Figuring out where to put stuff is much different than just rendering it after that has been figured out.</p>
<p>Now you know!</p>
Linux exists only because of a happy accident2019-12-05T00:00:00.000+01:00August Lilleaasurn:augustl-com:feed:post::blog:2019:linus_and_linux_happy_accident<p>EDIT: Discussion <a href="https://www.reddit.com/r/linux/comments/e6fvj0/linux_exists_only_because_of_a_happy_accident/">here</a>, <a href="https://www.reddit.com/r/programming/comments/e6kn97/linux_exists_only_because_of_a_happy_accident/">here</a> and <a href="https://news.ycombinator.com/item?id=21711572">here</a>.</p>
<p>You should read the book <a href="https://www.amazon.com/Just-Fun-Story-Accidental-Revolutionary/dp/0066620732">Just For Fun</a>, Linus Torvalds' autobiography.</p>
<p>I also make a podcast, in Norwegian. This post is based on an episode of that podcast, so if you understand Norwegian, or just want to pretend you do, you should also <a href="https://utviklingslandet.no/ep/2019-06-26/">listen to the episode we made</a> about the same subject as this blog post.</p>
<p>It's story time.</p>
<h2>How Linus Torvalds accidentally created Linux</h2>
<p>It's 1990, and Linus Torvalds was an unhappy Minix user.</p>
<p>After owning a few interesting computers (a VIC-20 and a Sinclair QL), Linus saved up some money and bought a very grey and uninteresting IBM compatible PC. With it was bundled a version of DOS, which he immediately replaced with Minix.</p>
<p>By "immediately", I mean "spent the better part of a week installing it".</p>
<p>Minix was reasonably popular at the time, and also free, which was a killer combo for a poor university student in Helsinki, Finland. Linus ordered Minix in advance. A month later, it arrived - in the form of 16 floppy disks.</p>
<p>Designed by Andrew Tanenbaum, Minix was meant as a teaching aide at mr. Tanenbaum's professorate at an Amsterdam university. It was gimped by design, and not really a full blown OS. There were patches floating around to improve its usefulness, and one of the more popular and useful were made by Bruce Evans, an Australian.</p>
<p>16 floppy disks, manual patching - no wonder it took Linus almost a week to set it up.</p>
<p>Linus was reasonably happy with Minix, but it lacked some of the things he needed. Linus being Linus, he wrote his own.</p>
<p>His main project was a terminal emulator, for calling up the university computers to go online, or just use the more powerful computers available to students. He wanted to learn the inns and outs of his IBM PC, so he wrote the terminal emulator on the hardware level.</p>
<p>No OS.</p>
<p>This terminal emulator was the beginning of Linux.</p>
<p>He wrote a disk driver, so he could save his work while he was outside Minix land. Eventually, his terminal emulator was able to launch BASH. (Yes, BASH is that old.) Linus called it his "gnu-emacs of terminal emulation programs".</p>
<p>It's fall, 1991.</p>
<p>Linus now considers shelving his terminal emulator. He's reasonably happy with it, and it does what he needs it to do.</p>
<p>In other words: Linus became bored with a project he was working on. Boy, do I recognize that feeling.</p>
<p>Then, Linus <strong>accidentally deleted his Minix partition</strong></p>
<p>In Linus' own words:</p>
<blockquote>
<p>I probably would have stopped by the end of 1991. I had done a lot of things I thought were interesting. Everything didn’t really work perfectly, but in a software kind of world I find that once you solve the fundamental problems of a project, it’s easy to lose interest. And that’s what was happening to me. Trying to debug software is not very engaging.</p>
<p>[...]</p>
<p>Back then, I was booting into Linux but used Minix as the main development environment. Most of what I was doing under Linux was reading email and news from the university’s computer via the terminal emulator I had written. The university computer was constantly busy, so I had written a program that auto-dialed into it. But in December, I mistakenly auto-dialed my hard disk instead of my modem. I was trying to auto-dial /dev/tty1 [...]. But by mistake I auto-dialed /dev/hda1, which is the hard disk device. The end result was that I inadvertently overwrote some of the most critical parts of the partition where I had Minix. Yes, that meant I couldn’t boot Minix anymore.</p>
<p>That was the point where I had a decision to make: I could reinstall Minix, or I could bite the bullet and acknowledge that Linux was good enough that I didn’t need Minix. I would write the programs to compile Linux, under itself, and whenever I felt I needed Minix, I would just add the desired feature to Linux.</p>
</blockquote>
<p>So there you have it.</p>
<p>Who knows, if Linus didn't accidentally ruin his Minix partition, Linux might never have seen the light of day.</p>
<p>Here's to happy accidents!</p>
DevOps: The magic of going back in time with Datomic2019-12-04T00:00:00.000+01:00August Lilleaasurn:augustl-com:feed:post::blog:2019:devops_going_back_in_time_with_datomic<p>I'm a bit of a <a href="https://www.datomic.com/">Datomic</a> fan. That's a bit of an understatement.</p>
<p>I'm going to tell you a DevOps story today, about Marzia and Felix. They are both DevOps experts, delivering quality software at breakneck speeds. Marzia uses Datomic, and Felix uses PostgreSQL. Let's see how their DevOps acts out in the world.</p>
<h2>It's 11:30 pm, almost midnight</h2>
<p>Felix is sleeping.</p>
<p><strong>The System</strong> is operational. Everything is running A-OK. No alerts are triggered, no warnings are active.</p>
<p>Earlier today, Felix deployed to production. Four times, actually! DevOps, yay!</p>
<p><strong>The System</strong> receives orders from many different systems in the organization. Felix is experienced and knows that stuff can go wrong, so he has created an audit log table in postgres that stores all the incoming orders. It's great to be able to replay orders that goes wrong.</p>
<p>It's <em>possible</em> that <strong>The System</strong> has a complete meltdown and that orders are missed - but that doesn't matter. What matters, is that when everything is up and running, and some validation error or other software bug fails to process the order, it can simply be reprocessed by finding it in the orders audit log.</p>
<p>It's 11:30pm. An order fails.</p>
<p>Nobody is alerted. This is just normal business. The different system that posted the order to us got an error message back. It's their responsibility to retry until it works.</p>
<h2>It's 08:41 am, the next day</h2>
<p>Felix just arrived at the office.</p>
<p>The Boss asks Felix why the order failed. It was an important order, and the system that sent it to us hasn't retried it.</p>
<p>Cool! We have an audit log, so let's try! We just send it again. We don't have to spend time digging in the logs to find the first failure.</p>
<p>It succeeds.</p>
<p>(Have you ever told yourself: "damn, I wish it didn't work?". That's normal. It happens to engineers all the time.)</p>
<p>Felix is told that The Boss changed some of the rules in the admin setup this morning, before Felix arrived. The admin setup is very powerful and flexible. Lots of grids and configs to tweak and adjust how orders are processed.</p>
<p>Oh well. It worked. We're not sure why it failed.</p>
<p>Moving on to other things!</p>
<h2>It's 11:30 pm, almost midnight.</h2>
<p>Marzia is sleeping.</p>
<p><strong>The Other System</strong> is operational. Everything is running A-OK. No alerts are triggered, no warnings are active.</p>
<p>Earlier today, Marzia deployed to production. Four times, actually! DevOps, yay!</p>
<p><strong>The Other System</strong> receives orders from many different systems in the organization. Marzia is a fan of Datomic, so she stores all the rules about how to process orders, and all the data about the orders themselves, as well as the raw input data that <strong>The Other System</strong> receives, in Datomic.</p>
<p>They use redis for some other things, and Riak as well. It's nice to be able to use different databases with different semantics for different parts of <strong>The Other System</strong>.</p>
<p>It's 11:30pm. An order fails.</p>
<p>Nobody is alerted. This is just normal business. The different system that posted the order to us got an error mesage back. It's their responsibility to retry until it works.</p>
<h2>It's 08:41 am, the next day</h2>
<p>Marzia just arrived at the office.</p>
<p>The Boss asks Marzia why the order failed. It was an important order, and the system that sent it to us hasn't retried it.</p>
<p>Cool! Marzia stores all order related information in Datomic, so we have full access to the historical information about all the data in our system.</p>
<p>(Say what?)</p>
<p>Marzia is told that The Boss changed some of the rules in the admin setup this morning, before Marzia arrived.</p>
<p>Marzia finds the transaction where the boss changed the rules.</p>
<p>(Say <em>what??</em>)</p>
<pre><code class="highlight"><span class="c1">;; Finds all transactions by boss-user</span>
<span class="p">(</span><span class="nf">->></span> <span class="p">(</span><span class="nf">d/q</span> <span class="o">'</span><span class="p">[</span><span class="ss">:find</span> <span class="p">[</span><span class="nv">?tx</span> <span class="nv">...</span><span class="p">]</span>
<span class="ss">:in</span> <span class="nv">$</span> <span class="nv">?boss-id</span>
<span class="ss">:where</span>
<span class="p">[</span><span class="nv">?tx</span> <span class="ss">:tx/changed-by</span> <span class="nv">?boss-id</span><span class="p">]]</span>
<span class="p">(</span><span class="nf">d/history</span> <span class="nv">db-3</span><span class="p">)</span>
<span class="p">(</span><span class="ss">:db/id</span> <span class="nv">boss-user</span><span class="p">))</span>
<span class="c1">;; Get the last 3 ones</span>
<span class="p">(</span><span class="nf">sort</span><span class="p">)</span>
<span class="p">(</span><span class="nf">reverse</span><span class="p">)</span>
<span class="p">(</span><span class="nb">take </span><span class="mi">3</span><span class="p">)</span>
<span class="p">(</span><span class="nf">map</span>
<span class="p">(</span><span class="k">fn </span><span class="p">[</span><span class="nv">tx</span><span class="p">]</span>
<span class="c1">;; Get raw data from transaction log for this tx</span>
<span class="p">(</span><span class="nf">->></span> <span class="p">(</span><span class="nf">d/tx-range</span> <span class="nv">db-log</span> <span class="nv">tx</span> <span class="p">(</span><span class="nb">inc </span><span class="nv">tx</span><span class="p">))</span>
<span class="p">(</span><span class="nf">first</span><span class="p">)</span>
<span class="ss">:data</span>
<span class="c1">;; Remove noise we don't want</span>
<span class="p">(</span><span class="nb">filter </span><span class="p">(</span><span class="k">fn </span><span class="p">[[</span><span class="nv">e</span> <span class="nv">a</span> <span class="nv">v</span> <span class="nv">t</span> <span class="nv">added?</span><span class="p">]]</span> <span class="p">(</span><span class="nb">and </span><span class="nv">added?</span> <span class="p">(</span><span class="nb">not= </span><span class="nv">e</span> <span class="nv">tx</span><span class="p">))))</span>
<span class="c1">;; Give us the good stuff</span>
<span class="p">(</span><span class="nb">map </span><span class="p">(</span><span class="k">fn </span><span class="p">[[</span><span class="nv">e</span> <span class="nv">a</span> <span class="nv">v</span><span class="p">]]</span>
<span class="p">{</span><span class="ss">:id</span> <span class="nv">e</span>
<span class="ss">:attr</span> <span class="p">(</span><span class="nf">d/ident</span> <span class="nv">db-3</span> <span class="nv">a</span><span class="p">)</span>
<span class="ss">:val</span> <span class="nv">v</span><span class="p">}))))))</span></code></pre>
<p>Marzia could have asked for more than the 3 latest ones, but decided that 3 was a good start. Here's what Marzia gets back from that query.</p>
<pre><code class="highlight"><span class="p">({</span><span class="ss">:timestamp</span> <span class="o">#</span><span class="nv">inst</span><span class="s">"2019-11-28T07:54:28.102-00:00"</span>,
<span class="ss">:changes</span>
<span class="p">({</span><span class="ss">:id</span> <span class="mi">17592186045421</span>,
<span class="ss">:attr</span> <span class="ss">:order-rule/zip-code</span>, <span class="ss">:val</span> <span class="s">"0061"</span>, <span class="ss">:added?</span> <span class="nv">true</span><span class="p">}</span>
<span class="p">{</span><span class="ss">:id</span> <span class="mi">17592186045421</span>,
<span class="ss">:attr</span> <span class="ss">:order-rule/zip-code</span>, <span class="ss">:val</span> <span class="s">"0056"</span>, <span class="ss">:added?</span> <span class="nv">false</span><span class="p">})}</span>
<span class="p">{</span><span class="ss">:timestamp</span> <span class="o">#</span><span class="nv">inst</span><span class="s">"2019-11-28T07:53:58.031-00:00"</span>,
<span class="ss">:changes</span>
<span class="p">({</span><span class="ss">:id</span> <span class="mi">17592186092838</span>,
<span class="ss">:attr</span> <span class="ss">:customer/name</span>, <span class="ss">:val</span> <span class="s">"Some Flower Shop"</span>, <span class="ss">:added?</span> <span class="nv">true</span><span class="p">}</span>
<span class="p">{</span><span class="ss">:id</span> <span class="mi">17592186092838</span>,
<span class="ss">:attr</span> <span class="ss">:customer/department</span>, <span class="ss">:val</span> <span class="mi">17592186860812</span>, <span class="ss">:added?</span> <span class="nv">true</span><span class="p">})}</span>
<span class="p">{</span><span class="ss">:timestamp</span> <span class="o">#</span><span class="nv">inst</span><span class="s">"2019-11-28T07:52:01.511-00:00"</span>,
<span class="ss">:changes</span>
<span class="p">({</span><span class="ss">:id</span> <span class="mi">17592186089123</span>,
<span class="ss">:attr</span> <span class="ss">:customer/name</span>, <span class="ss">:val</span> <span class="s">"Some Office"</span>, <span class="ss">:added?</span> <span class="nv">true</span><span class="p">}</span>
<span class="p">{</span><span class="ss">:id</span> <span class="mi">17592186089123</span>,
<span class="ss">:attr</span> <span class="ss">:customer/name</span>, <span class="ss">:val</span> <span class="s">"Some Offcie"</span>, <span class="ss">:added?</span> <span class="nv">false</span><span class="p">})})</span></code></pre>
<p>Marzia found it! The query yielded three transactions, performed around 07:55. The oldest one fixed a spelling error in a customer name, the second one added a new customer and set its name and assigned it to a department, and the most recent one changed an <code>:order-rule/zip-code</code> from "0056" to "0061".</p>
<p>Marzia knows that the zip code is important to the routing, and the change of zip codes triggered some rules that were changed by a code change the day before.</p>
<p>And the order that failed was related to the department that order rule is assigned to.</p>
<pre><code class="highlight"><span class="p">(</span><span class="nf">d/q</span>
<span class="o">'</span><span class="p">[</span><span class="ss">:find</span> <span class="p">[</span><span class="nv">?rule-id</span> <span class="nv">...</span><span class="p">]</span>
<span class="ss">:in</span> <span class="nv">$</span> <span class="nv">?order-id</span>
<span class="ss">:where</span>
<span class="p">[</span><span class="nv">?order-id</span> <span class="ss">:order/zip-code</span> <span class="nv">?zip-code</span><span class="p">]</span>
<span class="p">[</span><span class="nv">?rule-id</span> <span class="ss">:order-rule/zip-code</span> <span class="nv">?zip-code</span><span class="p">]]</span>
<span class="nv">db</span>
<span class="nv">failed-order-id</span><span class="p">)</span>
<span class="c1">;; => #{17592186045421}</span></code></pre>
<p>Jut to be on the safe side, Marzia checks that the order rule in question was actually associated with the same department around the time the order failed. This can be done easily, by running the same query again, but running it with the database <code>as-of</code> a provided timestamp.</p>
<pre><code class="highlight"><span class="p">(</span><span class="nf">d/q</span>
<span class="o">'</span><span class="p">[</span><span class="ss">:find</span> <span class="p">[</span><span class="nv">?rule-id</span> <span class="nv">...</span><span class="p">]</span>
<span class="ss">:in</span> <span class="nv">$</span> <span class="nv">?order-id</span>
<span class="ss">:where</span>
<span class="p">[</span><span class="nv">?order-id</span> <span class="ss">:order/zip-code</span> <span class="nv">?zip-code</span><span class="p">]</span>
<span class="p">[</span><span class="nv">?rule-id</span> <span class="ss">:order-rule/zip-code</span> <span class="nv">?zip-code</span><span class="p">]]</span>
<span class="p">(</span><span class="nf">d/as-of</span> <span class="nv">db</span> <span class="o">#</span><span class="nv">inst</span> <span class="s">"2019-11-27T23:30:00.000"</span><span class="p">)</span>
<span class="nv">failed-order-id</span><span class="p">)</span>
<span class="c1">;; => #{17592186045421}</span></code></pre>
<p>Marzia could also have checked the full history, in case it changed departments rapidly back and forth for some reason, but decides to leave it at that.</p>
<h2>Retrospective</h2>
<p>Felix had an audit log, but not much else.</p>
<p>Marzia queried the Datomic database, and had all the info readily available.</p>
<p>Datomic keeps the history for <em>all</em> the changes across the database over time. Datomic also treats transactions as first class entities, so the only "clever" thing that was done here was to make sure to tag transactions with the user that performed the change, so that it's easy to find that later on.</p>
<pre><code class="highlight"><span class="p">(</span><span class="nf">d/transact</span> <span class="nv">conn</span>
<span class="p">[[</span><span class="ss">:db/add</span> <span class="p">(</span><span class="ss">:db/id</span> <span class="nv">my-order-rule</span><span class="p">)</span> <span class="ss">:order-rule/zip-code</span> <span class="s">"0061"</span><span class="p">]</span>
<span class="p">[</span><span class="ss">:db/add</span> <span class="s">"datomic.tx"</span> <span class="ss">:tx/changed-by</span> <span class="p">(</span><span class="ss">:db/id</span> <span class="nv">boss-user</span><span class="p">)]])</span></code></pre>
<p>This is a DevOps superpower. You can track your data as it changes and churns, across deploys and code changes, and all the way back to the beginning of time. And it's just there, out of the box, you don't have to think up the use cases for time tracking up front.</p>
<p>You can and should create audit logs. But it's super nice to be able to have <em>all</em> of the data that governs your system with change tracking.</p>
<p>You do it with your source code. You do it with your log files. You should also do it with critical business data.</p>
You should use rebase/fixup in IntelliJ IDEA more often2019-12-03T00:00:00.000+01:00August Lilleaasurn:augustl-com:feed:post::blog:2019:using_rebase_fixup_in_intellij_idea<p>This one applies to all JetBrains products with git integration.</p>
<p>You can also be inspired by this post and use rebase/fixup with just plain old git. Just do <code>git rebase --interactive</code> to do essentially the same stuff, just without all the visual aid.</p>
<h2>The problem with dirty histories</h2>
<p>You commit something.</p>
<ul>
<li><code>[123abc]</code> Fix issue with article image CSS</li>
</ul>
<p>Then you commit something else.</p>
<ul>
<li><code>[123abc]</code> Fix issue with article image CSS</li>
<li><code>[456def]</code> Always run cloudfront invalidation in deploy script</li>
</ul>
<p>Whoops! You forgot something when fixing the article image CSS.</p>
<ul>
<li><code>[123abc]</code> Fix issue with article image CSS</li>
<li><code>[456def]</code> Always run cloudfront invalidation in deploy script</li>
<li><code>[789ghi]</code> Whoops! Add missing semi colon.</li>
</ul>
<p>We're all professionals here. We don't want a dirty git history like that! Thankfully, you didn't push yet, so you can just go ahead and fix that right up!</p>
<h2>Re-arranging and merging history</h2>
<p>What we want, is to combine the "woops" commit and the "fix CSS" commit into one, as if we never even did our oopsie. If the oopsie was actually deployed to production or somehing, having it in the git log is valuable. But when it's not even pushed yet, you should remove it.</p>
<p>Something like this:</p>
<ul>
<li><code>[123abc]</code> Fix issue with article image CSS + Whoops! Add missing semi colon</li>
<li><code>[456def]</code> Always run cloudfront invalidation in deploy script</li>
</ul>
<h2>Using rebase and fixup</h2>
<p>We can do this! As mentioned above, you can do it in plain git with <code>git rebase --interactive <sha here></code>. Here's how to do it visually in IntelliJ</p>
<p>First, select the earliest commit in the history that you want to operate on. In our case, we want to merge the 3rd most recent and the most recent commit. So we select the 3rd most recent, and choose "Interactively Rebase from Here..."</p>
<p><img src="/static/posts/intellij_rebase_fixup/step_1.jpg">
<p>You get a popup showing your commits, in reverse order.</p>
<p style="margin-right: 20%"><img src="/static/posts/intellij_rebase_fixup/step_2.jpg">
<p>What we want to do at this point, is to merge the last commit on that list into the first commit on that list. To do that, we first move it one step up by selecting it and clicking those small arrows on the side there.</p>
<p style="margin-right: 20%"><img src="/static/posts/intellij_rebase_fixup/step_3.jpg">
<p>The reason we do that, is that this list of commit is what Git will apply various operations on, in the order they appear (from top to bottom). That is why the commits come in reverse order.</p>
<p>The default operation is "pick", which just means "use this commit". I.e. don't change anything, just leave it as is.</p>
<p>The one we want to choose, is "fixup". Click the little arrow for the "whoops" commit, which is now the 2nd commit in the list.</p>
<p style="margin-right: 20%"><img src="/static/posts/intellij_rebase_fixup/step_4.jpg">
<p>That means that Git will take the commit marked as "fixup", amend it into the commit before it, and discard the commit message. Click the button do make that happen.</p>
<p style="margin-right: 20%"><img src="/static/posts/intellij_rebase_fixup/step_5.jpg">
<p>And voila, you're done! Your history will now look like this:</p>
<p><img src="/static/posts/intellij_rebase_fixup/step_6.jpg">
<p>The commits have been merged, and it will be as if you never made your mistake!</p>
<p>It's nice to not have to admit to making silly mistakes. And it's even nicer that your small (or big) mistakes doesn't clutter up the git history.</p>
<h2>One more thing</h2>
<p>I actually used this feature for real when writing this article! The three commits I made above were made to this project. But they were just faux commits that I wanted to get rid of.</p>
<p>So first, I created a commit for this article (which I'm now adding to). Then, I chose the first faux commit and started a rebase. I made the popup look like this:</p>
<p style="margin-right: 20%"><img src="/static/posts/intellij_rebase_fixup/real_step.jpg">
<p>I moved the commit for this article to the top, and let it stay on "pick". Then I picked "skip" - i.e. get rid of - my two faux commits.</p>
<p>And just like that, they are gone from the project.</p>
<p>Pretty nice!</p>
How to set up a Windows 10 development environment - for macOS and Linux users2019-12-02T00:00:00.000+01:00August Lilleaasurn:augustl-com:feed:post::blog:2019:set_up_win10_dev_environment_for_macos_linux_users<p>So yeah, after years of back and forth between macOS and Linux, and numerous <a href="https://augustl.com/blog/2016/notes_on_windows/">failed attempts</a> of switching to Windows before, I'm back again. I used Bootcamp on a Macbook Pro for a month, and got me a Dell XPS which I have been using for two months now.</p>
<p>Maybe I'll write more about why I did that later. For now, I'll go through what I've done to set it up to get it the way I want it.</p>
<h1>A good terminal - finally!</h1>
<p>Opening up the terminal on a Windows box has felt eerily like opening a DOS virtual machine or something. Old and clunky.</p>
<p>Microsoft has made a new one, which is in early access at the time of writing. I've been using it for a few months and I've had zero problems with it. Works great!</p>
<p>You use <code>ctrl+shift+{1,2,3,...}</code> to open new terminals (the numbers correspond to the different types of terminals you can open), and <code>ctrl+alt+{1,2,3,...}</code> to switch between tabs.</p>
<p>You can install it <a href="https://www.microsoft.com/en-us/p/windows-terminal-preview/9n0dx20hk701">from the Store</a> - or build it yourself <a href="https://github.com/microsoft/terminal">from source</a> if you fancy that sort of thing!</p>
<p><img src="/static/posts/win10_dev_env/windows_terminal_1.jpg">
<p><img src="/static/posts/win10_dev_env/windows_terminal_2.jpg">
<h1>You are going to need WSL</h1>
<p>This is going to be the most important step.</p>
<p>WSL is a full Linux environment, natively on Windows. Yes, you read that right.</p>
<p>You've also seen WSL already. Scroll up! It's in the terminal section, the part with <code>git status</code>. Looks great, doesn't it?</p>
<p>I'm writing this blog post using WSL. I'm currently working on making <a href="https://github.com/magnars/optimus">optimus</a> run on Windows. But no problem, it already runs on Windows! Well, on WSL, that is. Which feels just like running it on Windows.</p>
<p>Follow <a href="https://docs.microsoft.com/en-us/windows/wsl/install-win10">this guide to set up WSL</a> on your machine. As you can see, you can even choose between a number of distros. I, and most people, use Ubuntu, but it's nice to have the option.</p>
<p>I use PowerShell for most things. But I still haven't learned what the alternative for "curl" is, because I use WSL to do that.</p>
<p>WSL is your savior when you run into software that doesn't exist or won't run easily on Windows. In the world where I live, that's commonplace, since everyone uses macOS or Linux. And so do I :)</p>
<p>WSL is actually a full re-implemenation of the Linux system calls against the NT kernel. NT was designed to be able to do this sort of thing. On win32, starting a new process is slow and heavy, which can make things like git slow, since it's designed to fork a lot. But creating a new process is super fast on NT, which WSL takes full advantage of. Pretty neat stuff. In WSL2, they're improving things even further, and WSL2 can even run Docker images.</p>
<h1>How I set up Git</h1>
<p>I use <a href="https://gitforwindows.org/">gitforwindows.org</a>. Here's my <code>.gitconfig</code>.</p>
<pre><code class="highlight"><span class="err">[user]</span>
<span class="na"> name</span> <span class="o">=</span> <span class="s">August Lilleaas</span>
<span class="na"> email</span> <span class="o">=</span> <span class="s">august@augustl.com</span>
<span class="na">[includeIf "gitdir</span><span class="o">:</span><span class="s">C:/Users/August/code/animalia/"]</span>
<span class="na"> path</span> <span class="o">=</span> <span class="s">C:\\Users\\August\\.gitconfig-work</span>
<span class="err">[core]</span>
<span class="na"> excludesfile</span> <span class="o">=</span> <span class="s">C:\\Users\\August\\.gitignore</span>
<span class="na"> autocrlf</span> <span class="o">=</span> <span class="s">input</span>
<span class="na"> sshCommand</span> <span class="o">=</span> <span class="s">"C:/Windows/System32/OpenSSH/ssh.exe"</span>
<span class="err">[pull]</span>
<span class="na"> rebase</span> <span class="o">=</span> <span class="s">true</span></code></pre>
<p><a href="https://animalia.no">Animalia</a> is my current customer, and I have a separate gitconfig set up for them. This gitconfig sets my user.email to my work email, for example.</p>
<p>I told git to use the built-in OpenSSH command. Yes, Win10 has OpenSSH built in, including a service that runs the ssh agent automatically to cahce your ssh keys etc. But I guess this is pretty recent, since Git for Windows still comes with its own OpenSSH implementation bundled with it.</p>
<p>Setting <code>core.autocrlf</code> is super ridiculously important. On Windows, the default line separator is <code>\r\n</code>. I looked this up, and that's actually the "proper" way to do it. The BIOS/UEFI works like that, for example. But Linux and macOS expects just <code>\n</code> for newlines, so something has to be done. I've configured all my editors to just use <code>\n</code> as Windows can handle that just fine too. But to be on the safe side, git is also configured to actually check for <code>\r\n</code> and replace it with <code>\n</code> before it is committed. A nice safety net to have.</p>
<p>I actually have some more stuff in my real config - such as GPG signing of commits (to get <a href="https://github.com/augustl/augustl.com/commit/c27d49ae59272cf2e1764339e5e913d9d033268f">that nifty verified badge</a> on Github). I use <a href="https://www.gpg4win.org/">gpg4win.org</a> for that, which works beautifully out of the box.</p>
<pre><code class="highlight"><span class="err">[user]</span>
<span class="c"># Replace with your actual fingerprint obviously</span>
<span class="na"> signingkey</span> <span class="o">=</span> <span class="s">0B50B734F49ACA83497E64E14FC2CE07E6E2A9DF</span>
<span class="err">[commit]</span>
<span class="na"> gpgsign</span> <span class="o">=</span> <span class="s">true</span></code></pre>
<p>I also configured GPG to cache my keys for longer, so I don't have to enter the GPG key passphrase almost every time I commit. The file to edit is C:\Users\MyUser\AppData\Roaming\gnupg\gpg-agent.conf (or %appdata%\gnupg\gpg-agent.conf for short).</p>
<pre><code class="highlight"><span class="na">default-cache-ttl 34560000</span>
<span class="na">max-cache-ttl 34560000</span></code></pre>
<h1>You probably want VSCode</h1>
<p>VSCode - or Visual Studio Code - is a lightweight editor (at least compared to Visual Studio itself) that's gaining a lot of traction, and is a pretty nice editor in my estimation. I'm an IDE guy and I use IntelliJ for my main projects. But for editing gitconfigs, opening log files, and so on, I use VSCode.</p>
<p><img src="/static/posts/win10_dev_env/vscode.jpg">
<p>You can <a href="https://code.visualstudio.com/">download it for free</a> (it's open source and everything), and it's a much better alternative to the buil-in notepad - obviously.</p>
<h1>Switching between Java versions</h1>
<p>I'm not completely happy with this - I liked <a href="https://sdkman.io/">sdkman</a> on macOS/Linux better. But it's the best one I've foudn so far.</p>
<p>Jabba is a PowerShell thingie to download, install and switch between JVM versions. At work we still have some apps that require Java 8, and others use Java 11, so it's nice to be able to switch.</p>
<p>You can <a href="https://sdkman.io/">get jabba here</a>. The installation instructions there are good, so I won't bore you with the details.</p>
<p>One thing you need to fix is some permission stuff I haven't yet understood completely. Seems like PowerShell wants this sort of thing to be signed, and jabba is not signed (it seems). Instead of changing execution permissions system wide, I've configured Windows Terminal with a separate PowerShell setup, aside from the default one, that allows jabba to run.</p>
<p>Click on "settings" in Windows Terminal, and your editor (VSCode for me) will open with the config file. Mine looks sort of like this:</p>
<pre><code class="highlight"><span class="p">{</span>
<span class="s2">"$schema"</span><span class="o">:</span> <span class="s2">"https://aka.ms/terminal-profiles-schema"</span><span class="p">,</span>
<span class="s2">"defaultProfile"</span><span class="o">:</span> <span class="s2">"{61c54bbd-c2c6-5271-96e7-009a87ff44bf}"</span><span class="p">,</span>
<span class="s2">"profiles"</span><span class="o">:</span>
<span class="p">[</span>
<span class="p">{</span>
<span class="s2">"guid"</span><span class="o">:</span> <span class="s2">"{61c54bbd-c2c6-5271-96e7-009a87ff44bf}"</span><span class="p">,</span>
<span class="s2">"name"</span><span class="o">:</span> <span class="s2">"Windows PowerShell"</span><span class="p">,</span>
<span class="s2">"commandline"</span><span class="o">:</span> <span class="s2">"powershell.exe -NonInteractive -ExecutionPolicy Bypass"</span><span class="p">,</span>
<span class="s2">"hidden"</span><span class="o">:</span> <span class="kc">false</span>
<span class="p">},</span>
<span class="p">{</span>
<span class="s2">"guid"</span><span class="o">:</span> <span class="s2">"{71c54bbd-c2c6-5271-96e7-009a87ff44bf}"</span><span class="p">,</span>
<span class="s2">"name"</span><span class="o">:</span> <span class="s2">"Windows PowerShell (plain)"</span><span class="p">,</span>
<span class="s2">"commandline"</span><span class="o">:</span> <span class="s2">"powershell.exe"</span><span class="p">,</span>
<span class="s2">"hidden"</span><span class="o">:</span> <span class="kc">false</span>
<span class="p">},</span>
<span class="c1">// ... the rest of my stuff here...</span>
<span class="p">],</span>
<span class="p">}</span></code></pre>
<p>I also have <code>.jabbarc</code> in my system gitignore file (see gitconfig above) and put that in various projects. For example, one of the projects at work has this <code>.jabbarc</code> in the root of the project:</p>
<pre><code class="highlight">zulu@1.8.232</code></pre>
<p>Then I can just type <code>jabba use</code> when opening a new PowerShell terminal to my project folder, to use that version.</p>