Refactoring MDN macros with async, wait for, and Object. freeze()

A frozen cleaning soap bubble

In 03 of last year, the MDN Anatomist team began the experiment associated with publishing a monthly changelog upon Mozilla Hacks. After nine several weeks of the particular changelog format , we’ ve decided it’ s time to try out something that we hope will be of interest towards the web development community more generally, and more fun for us to write. These types of posts may not be monthly, and they won’ t contain the kind of granular fine detail that you would expect from a changelog. They will cover some of the more fascinating engineering work we do to handle and grow the MDN Internet Docs site. And if you want to understand exactly what has changed and who has led to MDN, you can always check the repos on GitHub.

In January, we landed a significant refactoring of the KumaScript codebase which is going to be the topic of this post since the work included some techniques appealing to JavaScript programmers.

Modern JavaScript

Among the pleasures of undertaking a big refactor like this is the opportunity to modernize the particular codebase. JavaScript has matured a lot since KumaScript was first written, and am was able to take advantage of this, using let and const , classes , arrow features , for... of spiral , the particular spread (… ) operator , and destructuring assignment in the refactored code. Because KumaScript runs like a Node-based server, I didn’ capital t have to worry about browser compatibility or transpilation: I was free (like a kid within a candy store! ) to use all of the most recent JavaScript features supported by Client 10 .

KumaScript and macros

Upgrading to modern JavaScript was a large amount of fun, but it wasn’ t cause enough to justify the time used on the refactor. To understand why the team allowed me to work about this project, you need to understand what KumaScript really does and how it works. So bear beside me while I explain this framework, and then we’ ll get back to one of the most interesting parts of the refactor.

First, you should know that Kuma is the Python-based wiki that power MDN, and KumaScript is a machine that renders macros in MDN documents. If you look at the raw type of an MDN document (such since the HTML < body > component ) you’ ll find lines like this:

 It must be the second element of an HTMLElement("html") element.
 
 

The information within the double curly braces is really a macro invocation. In this case, the macro is defined to render the cross-reference link to the MDN paperwork for the html element. Using macros such as this keeps our links and angle-bracket formatting consistent across the site plus makes things simpler for authors.

MDN has been making use of macros like this since before the Kuma server existed. Before Kuma, all of us used a commercial wiki item which allowed macros to be described in a language they called DekiScript. DekiScript was a JavaScript-based templating vocabulary with a special API for getting together with the wiki. So when we relocated to the Kuma server, our files were full of macros defined within DekiScript, and we needed to implement our very own compatible version, which we known as KumaScript.

Since the macros were defined using JavaScript, we couldn’ t implement all of them directly in our Python-based Kuma machine, so KumaScript became a separate services, written in Node. This was seven years ago in early 2012, when Client itself was only on edition 0. 6. Fortunately, a JavaScript-based templating system known as EJS currently existed at that time, so the basic equipment for creating KumaScript were all in position.

But there was the catch: some of our macros required to make HTTP requests to retrieve data they needed. Consider the HTMLElement macro shown above for instance. That macro renders a link to the MDN documents for a specified HTML tag. However it also includes a tooltip (via the particular title attribute) on the link that includes a fast summary of the element:

A rendered link to documentation for an CODE element, displaying a tooltip that contains a summary of the linked documentation. inch width=

That summary has to come from the particular document being linked to. This means that the particular implementation of the KumaScript macro must fetch the page it is backlinking to in order to extract some of the content. Furthermore, macros like this are usually written by technical writers, not software program engineers, and so the decision was produced (I assume by whoever developed the DekiScript macro system) that will things like HTTP fetches would be carried out with blocking functions that returned synchronously, so that technical writers would not need to deal with nested callbacks.

It was a good design decision, but it produced things tricky for KumaScript. Client does not naturally support blocking system operations, and even if it did, the particular KumaScript server could not just cease responding to incoming requests while it fetched documents for pending requests. The particular upshot was that KumaScript used the node-fibers binary extension to Node to be able to define methods that blocked whilst network requests were pending. And moreover, KumaScript adopted the node-hirelings collection to manage a pool of kid processes. (It was written by the initial author of KumaScript for this purpose). This enabled the KumaScript machine to continue to handle incoming requests within parallel because it could farm away the possibly-blocking macro rendering phone calls to a pool of hireling child procedures.

Async and watch for

This fibers+hirelings remedy rendered MDN macros for seven years, but by 2018 this had become obsolete. The original style decision that macro authors must not have to understand asynchronous programming along with callbacks (or Promises) is still an excellent decision. But when Node 8 additional support for the new async and await key phrases, the fibers extension and hirelings library were no longer necessary.

You can read about async functions and await expressions on MDN, but the gist is this:

  • If you declare a functionality async , you might be indicating that it returns a Guarantee. And if you return a worth that is not a Promise, that worth will be wrapped in a resolved Guarantee before it is returned.
  • The await operator makes asynchronous Claims appear to behave synchronously. It enables you to write asynchronous code that is as effortless to read and reason about because synchronous code.

As an example, consider this line of code:

 let response sama dengan await fetch(url);
 
 

In web browsers, the fetch() function begins an HTTP request and earnings a Promise object that will solve to a response object once the HTTP response begins to arrive from the machine. Without await , you’ d have to call the particular . then() method of the returned Promise, plus pass a callback function to get the response object. But the miracle of await lets us pretend that fetch() really blocks until the HTTP response will be received. There is only one catch:

  • You can only make use of await within functions that are themselves announced async . Interim, await doesn’ t actually make anything prevent: the underlying operation is still fundamentally asynchronous, and even if we pretend that it is not really, we can only do that within a few larger asynchronous operation.

What this all indicates is that the design goal of safeguarding KumaScript macro authors from the difficulty of callbacks can now be done with Claims and the await keyword. And this is the understanding with which I undertook our KumaScript refactor.

As I mentioned previously, each of our KumaScript macros is applied as an EJS template. The EJS library compiles templates to JavaScript functions. And to my delight, the most recent version of the library has already been up-to-date with an alternative to compile web templates to async functions, which means that await is now backed in EJS.

With this particular new library in place, the refactor was relatively simple. I had to find all of the blocking functions available to our macros and convert them to use Claims instead of the node-fibers extension. Then, I had been able to do a search-and-replace on our macro files to insert the await key phrase before all invocations of these features. Some of our more complicated macros determine their own internal functions, and when individuals internal functions used await , I had to take the extra step of changing those functions to become async . I did so get tripped up by 1 piece of syntax, however , when I transformed an old line of blocking code such as this:

 var name = wiki. getPage(slug). title;
 
 

To this:

 let title = wait for wiki. getPage(slug). title;
 
 

I didn’ t capture the error on that range until I started seeing problems from the macro. In the old KumaScript, wiki. getPage() would block and come back the requested data synchronously. Within the new KumaScript, wiki. getPage() is announced async which means it returns a Guarantee. And the code above is trying to reach a non-existent name property on that will Promise object.

By mechanical means inserting an watch for in front of the invocation will not change that fact because the await operator provides lower precedence than the . property access owner. In this case, I needed to add some additional parentheses to wait for the Promise to solve before accessing the title property:

 let title sama dengan (await wiki. getPage(slug)). title;
 
 

This relatively little change in our KumaScript code implies that we no longer need the fibres extension compiled into our Client binary; it means we don’ big t need the hirelings package anymore; and it means that I was able to get rid of a bunch of code that handled the particular complicated details of communication between the primary process and the hireling worker procedures that were actually rendering macros.

And here’ s the particular kicker: when rendering macros that not make HTTP requests (or when the HTTP results are cached) I saw rendering speeds raise by a factor of 25 (ofcourse not 25% faster– 25 times quicker! ). And at the same time CPU fill dropped in half. Within production, the new KumaScript server is definitely measurably faster, but not nearly 25x faster, because, of course , the time needed to make asynchronous HTTP requests rules the time required to synchronously render the particular template. But achieving a 25x speedup, even if only under managed conditions, made this refactor an extremely satisfying experience!

Object. create() plus Object. freeze()

There is one other part of this KumaScript refactor that I wish to talk about because it highlights some JavaScript techniques that deserve to be much better known. As I’ ve created above, KumaScript uses EJS themes. When you render an EJS design template, you pass in an object that will defines the bindings available to the particular JavaScript code in the template. Over, I described a KumaScript macro that called a function called wiki. getPage() . In order for it to do that, KumaScript needs to pass an object to the EJS design template rendering function that binds title wiki to an object that includes a getPage property in whose value is the relevant function.

For KumaScript, there are 3 layers of this global environment that people make available to EJS templates. The majority of fundamentally, there is the macro API, including wiki. getPage() and a number of related features. All macros rendered by KumaScript share this same API. Over this API layer is an env item that gives macros access to page-specific beliefs such as the language and title from the page within which they appear. Once the Kuma server submits an MDN page to the KumaScript server just for rendering, there are typically multiple macros to be rendered within the page. Yet all macros will see the same beliefs for per-page variables like env. title and env. location . Finally, each individual macro invocation on a page can include arguments, that are exposed by binding these to variables $0 , $1 , and so forth

So , in order to provide macros, KumaScript has to prepare a subject that includes bindings for a relatively complicated API, a set of page-specific variables, plus a set of invocation-specific arguments. When refactoring this code, I had two objectives:

  • I didn’ t want to have to rebuild the whole object for each macro to be made.
  • I wanted to ensure that macro code could not alter the environment plus thereby affect the output of upcoming macros.

We achieved the first goal by using the JavaScript prototype string and Object. create() . Rather than defining all 3 layers of the environment on a single item, I first created an object that will defined the fixed macro API and the per-page variables. I used again this object for all macros in just a page. When it was time to provide an individual macro, I used Object. create() to create a new object that passed down the API and per-page bindings, and I then added the macro argument bindings to that new item. This meant that there was a lot less setup work to do for each person macro to be rendered.

But if I was going to reuse the thing that defined the API plus per-page variables, I had to be extremely sure that a macro could not get a new environment, because that would mean that the bug in one macro could get a new output of a subsequent macro. Making use of Object. create() helped a lot with this: in case a macro runs a line of program code like wiki sama dengan null; , that will only impact the environment object created for that one provide, not the prototype object it inherits from, and so the wiki. getPage() functionality will still be available to the next macro to become rendered. (I should point out that will using Object. create() like this can cause a few confusion when debugging because a subject created this way will look like it is vacant even though it has inherited properties. )

This Object. create() method was not enough, however , because a macro that included the code wiki. getPage = null; would still be in a position to alter its execution environment plus affect the output of subsequent macros. So , I took the extra phase of calling Object. freeze() on the prototype object (and recursively on the objects it references) before I created objects that will inherited from it.

Object. freeze() continues to be part of JavaScript since 2009, however, you may not have ever used it or else a library author. It hair down an object, making all of the properties read-only. Additionally it “ seals” the object, which means that new properties can not be added and existing properties cannot be deleted or configured to make all of them writable again.

I’ ve always found it comforting to know that Object. freeze() is there if I require it, but I’ ve rarely in fact needed it. So it was fascinating to have a legitimate use for this function. There was clearly one hitch worth mentioning, nevertheless: after triumphantly using Object. freeze() , I found that will my attempts to stub away macro API methods like wiki. getPage() were failing silently. By fastening down the macro execution environment therefore tightly, I’ d locked away my own ability to write tests! The answer was to set a flag whenever testing and then omit the Object. freeze() step when the flag was established.

If this all noises intriguing, you can take a look at the Environment class in the KumaScript source program code.

David is a software professional on the MDN team at Mozilla, and the author of the book Javascript: The Definitive Manual .

More articles by Jesse Flanagan…

If you liked Refactoring MDN macros with async, wait for, and Object. freeze() by David Flanagan Then you'll love Web Design Agency Miami

Add a Comment

Your email address will not be published. Required fields are marked *

Shares