Bridging the Rust FFI frontier along with Protocol Buffers

My team, the application services team at Mozilla, works on Opera Sync, Firefox Accounts and WebPush .

These features are currently shipped upon Firefox Desktop, Android and iOS browsers. They will soon be available within our new products such as our upcoming Google android browser, our password manager Lockbox, and Firefox for Fire TELEVISION.

Solving for one a lot of targets

Until now, regarding historical reasons, these functionalities are implemented in widely different styles. They’ ve been tightly combined to the product they are supporting, making them hard to reuse. For example , you can find currently three Firefox Sync customers: one in JavaScript, one within Java and another one in Fast.

Considering the size in our team, we quickly realized that the current approach to shipping products may not scale across more products and would certainly lead to quality issues such as insects or uneven feature-completeness across platform-specific implementations. About a year ago, all of us decided to plan for the future. As you may know, Mozilla is making a pretty big wager on the new Rust programming vocabulary, so it was natural for us to follow along with suit.

A cross-platform strategy using Rust

Our new strategy is as comes after: We will build cross-platforms components, applying our core business logic making use of Rust and wrapping it in the thin platform-native layer, such as Kotlin for Android and Swift intended for iOS.

The biggest and many obvious advantage is that we have one canonical code base, written inside a safe and statically typed vocabulary, deployable on every platform. Each upstream business logic change receives to all our products with an edition bump.

How we resolved the FFI challenge safely

However , one of the challenges for this new approach is safely transferring richly-structured data across the API border in a way that’ s memory-safe plus works well with Rust’ s ownership program. We wrote the ffi-support cage to help with this, if you write program code that does FFI ( Foreign perform interface ) tasks you need to strongly consider using it. Our initial variations were implemented by serializing our own data structures as JSON guitar strings and in a few cases returning the C-shaped struct by pointer. The process for returning, say, a save from the user’ s synced information looked like this:

Returning Save data from Rust to Kotlin using JSON (simplified)

So what’ s the problem with this particular approach?

• Performance: JSON serializing and de-serializing is notoriously gradual, because performance was not a primary style goal for the format. On top of that, an additional string copy happens on the Coffee layer since Rust strings are usually UTF-8 and Java strings are usually UTF-16-ish. At scale, it can expose significant overhead.
• Complexity and safety: every data structure is by hand parsed and deserialized from JSON strings. A data structure industry modification on the Rust side must become reflected on the Kotlin side, or even an exception will most likely occur.
• Even worse, in some cases we were not coming back JSON strings but C-shaped Corrosion structs by pointer: forget to revise the Structure Kotlin subclass or maybe the Objective-C struct and you have a serious memory space corruption on your hands.

We quickly realized there is probably a better and faster method that could be safer than our present solution.

Data serialization with Protocol Buffers v. two

Thankfully, there are many data serialization formats out there that will aim to be fast. The ones using a schema language will even auto-generate information structures for you!

Right after some exploration, we ended up purchasing Protocol Buffers version 2 .

The — relative— basic safety comes from the automated generation associated with data structures in the languages we all care about. There is only one source of truth— the. proto schema file— that all our data classes are created at build time, both at the Rust and consumer side.

protoc (the Protocol Buffers code generator) can emit code in more compared to 20 languages! On the Rust part, we use the prost crate which usually outputs really clean-looking structs simply by leveraging Rust derive macros.

Returning Bookmark information from Rust to Kotlin making use of Protocol Buffers 2 (simplified)

And of course, on top of that, Protocol Buffers are faster than JSON.

There are a few downsides to this approach: it really is more work to convert our own internal types to the generated protobuf structs – e. g the url:: Url has to be converted to a Thread first– whereas back when we were making use of serde-json serialization, any struct implementing serde:: Serialize was a line away from being delivered over the FFI barrier. It also provides one more step during our create process, although it was pretty simple to integrate.

One thing We ought to mention: Since we deliver both the producer and consumer of the binary streams as an unit, we now have the freedom to change our information exchange format transparently without influencing our Android and iOS customers at all.

A look forward

Looking forward, there’ ersus probably a high-level system that might be used to exchange data over the FFI, maybe based on Rust macros. There’ s also been talk about using FlatBuffers to squeeze out even more functionality. In our case Protobufs provided the correct trade-off between ease-of-use, performance, plus relative safety.

Up to now, our components are present both upon iOS, on Firefox and Lockbox, and on Lockbox Android, and will shortly be in our new upcoming Google android browser.

Firefox iOS has started to oxidize by changing their password syncing engine using the one we built in Rust. The master plan is to eventually do the same upon Firefox Desktop as well.

If you are interested in helping us develop the future of Firefox Sync and more, or simply just following our progress, head to the particular application solutions Github repository.

If you liked Bridging the Rust FFI frontier along with Protocol Buffers by Edouard Oger Then you'll love Web Design Agency Miami