Delivery a security update of Firefox in under a day
One of Mozilla’ s top focal points is to keep our users secure; this commitment is written directly into our objective . As soon as we discover an essential issue in Firefox, we strategy a rapid mitigation. This post will explain how we fixed a Pwn2Own take advantage of discovery in less than 22 hours, with the collaborative and well-coordinated efforts of the global cross-functional team of launch and QA engineers, security specialists, and other stakeholders.
Pwn2Own is definitely an annual computer hacking contest. The aim of this event is to find security vulnerabilities in major software such as web browsers. Last week, this event took place in Vancouver. Without getting into technical details from the exploit here, this blog post can describe how Mozilla responded rapidly to ship updated builds associated with Firefox once an exploit had been found during Pwn2Own.
We will share some of the processes that enable us to update plus release a new version of the Opera browser to hundreds of millions of customers on a regular recurring basis.
This browser is a huge computer software: 18 million+ lines of program code, 6 platforms (Windows 32 & 64bit, GNU/Linux 32 & 64bit, Mac OS X and Android), 90 languages, plus installers, updaters, etc . Releasing such a beast requires coordination among many people from various cross-functional teams spanning locations for example San Francisco, Philadelphia, Paris, Cluj within Romania, and Rangiora in Brand new Zealand.
The time of the Pwn2Own event is known days beforehand, and so Mozilla is ready! The Opera train release calendar takes into consideration the timing associated with Pwn2Own. We try not to ship a brand new version of Firefox to end customers on the release channel on the same time as Pwn2Own.
The Firefox Chemspill
The chemspill is a “ security-driven dot release of our product . ” It’ s an indoor name for the Mozilla machinery that will produces updated builds of Opera on all channels (Nightly, Beta, Release, ESR) in response to an event that will negatively impacts browser stability or even user security.
The rapid response model is similar to the way in which emergency personnel organize and mobilize to deal with a chemical spill and its particular hazards. All key people cease working on their current tasks and concentrate only on the cleanup itself. Mainly because our focus is our owners, we need to ensure that they are using the most dependable and fastest version of Opera!
This year, we a new private Slack channel prior to Pwn2Own to coordinate all the activity associated with the event. The initial Slack group comprised only of security experts, company directors of engineering, senior engineers, discharge managers and release engineers – essential staff.
All of us prepared a release checklist beforehand with added items and a particular focus on the potential for a chemspill brought on by Pwn2Own. This document assisted track the cross-functional tasks, their particular owners, status and due date, which usually helped track individual tasks as well as the necessary coordination. It also helped stakeholders view and report chemspill position down to the minute.
One of the members in our security team was attending the particular Pwn2Own event. After it was announced that one of the participants, Rich Zhu, found the security issue within Firefox, this Mozilla representative obtained the exploit directly from Richard Zhu as part of the regular Pwn2Own disclosure procedure for affected vendors. The pester was added to our bug monitoring system at ten: 59AM PDT upon March 15th with the necessary personal privacy settings. Soon after, the chemspill group reviewed the issue and made a decision in order to ship updated builds ASAP.
In parallel, there was an analysis happening on the private Slack funnel. When we saw the tweet from cybersecurity reporter @howelloneill that made this news public, we knew it was time for you to identify the developer who’ g be getting to work on fixing the particular bug…
And so, rapidly, the developer got to work.
The fix: planning, danger analysis, go-live timelines
While engineers were investigating the particular exploit and coming up with a fix, the particular cross-functional coordination needed to ship up-to-date builds had already begun. The particular chemspill team met within two hours of the event. We discussed the following steps in terms of fix preparedness, test plans, go-to-build, QA sign-offs, and determined the sequence associated with steps along with rough timelines. All of us needed to ensure a smooth hand-off through folks in North America to people in Europe (France, Romania, UK) and then back to California by early morning.
From the moment we had information regarding the exploit, two discussions started in parallel: a technical debate on the bug tracking system; along with a release-oriented discussion, driven by the launch and security managers, on the Slack channel.
12 a few minutes later, at eleven: 11AM , a relevant developer can be contacted.
eleven: 17AM : The bug is definitely updated to confirm that our long-term assistance release (ESR) has also been impacted by the problem.
12: 32PM : Less than 3 hours after the disclosure, the developer provides a first area addressing the issue.
fourteen: 21PM : An improved version from the fix is pushed.
15: 23PM : This area is pushed to the development department. Then, in the next 70 minutes, all of us go through the process of getting the patch arrived into the other release and pre-release repositories.
seventeen: 16PM : Little more than six hours after the publication of the take advantage of, the Beta and Release develops (desktop and Android) are in improvement.
During the build stage
Let’ s have a step back to describe the regular workflow that occurs every time a new build of Opera is released. Building the Opera browser with our complete test package for all platforms takes about five hours. While the builds are in improvement, many teams are working in seite an seite.
Test plan
The QA team styles a test plan with the help of engineering. Whenever fixing security issues, we have always two goals in mind:
- Verify that the fix tackles the security issue,
- Capture any other potential regressions due to the repair.
With these 2 goals, the QA team aspires to cover a wide range of cases using various inputs.
For example , these test case #3 has been performed on the various impacted versions plus platforms:
Check Case 3 (ogg enabled fake – Real. ogg File)
- Pick a channel
- Navigate to about: config
- Set pref “ media. ogg. enabled” in order to false
- Download an. ogg file
- Drag the particular. ogg file into the Mozilla develop
- Notice an error message/prompt “ You have chosen to spread out [name of file]. ogg
- Try and open the particular file with Firefox as the software
- Notice that Firefox does not play the chosen. ogg file (or any sound)
- Replicate step 1 for all builds (ESR, REMOTE CONTROL, Beta/DevEdition, Fennec)
Exploit analysis
In parallel, our security professionals jumped on the exploit to analyze this.
They look carefully at several things:
- How the exploit works technically
- How we could have detected the problem ourselves
- The in progress initiatives: How to mitigate this kind of attack
- The stalled efforts: What we began but didn’ t finish
- The long term efforts: Scoping the long run work to eliminate or mitigate this particular category of attacks
Outreach
The weeknesses was found to be in a collection that did not originate with the Mozilla project, and is used by other software program. Because we didn’ t wish to 0-day the vulnerable software collection and make the vulnerability more widely recognized, we reached out to the maintainer from the library directly. Then, we researched which other applications use this program code and we tried to notify them plus make them aware of the issue.
In parallel, we worked with the particular library maintainers to prepare a new edition of the standalone library code.
Last but not least, as GNU/Linux distributions provide packages of this library, we all also informed these distributions in regards to the issue.
Once the creates are ready
After approximately 5 hours, the builds had been ready. This is when the QA group starts executing the test plans.
They verify all the situations on a bunch of different platforms/operating techniques.
In a matter of 22 hours, less than a day time from when the exploit was discovered, Mozilla was ready to push up-to-date builds of Firefox for Desktop computer and Android on our Nightly, Beta, ESR and release update station.
For the release go live , the safety team wrote the security advisories plus created an entry for the CVE (Common Vulnerabilities and Exposures), the public reference that lists openly known cybersecurity vulnerabilities.
And then, at the last moment, all of us discovered a second variant of the impacted code and had to rebuild the particular Android version. This was also affecting Firefox ESR on ARM products. We shipped this fix too at 23: 10PM .
Nobody wants to see their product get pwned , but as with so much within software development, preparation and dexterity can make the difference between a chemspill where no damage is done, and also a potentially endangering situation.
Through the combined work of many distributed teams, and good preparing and communication, Mozilla was able to ensure that you release a fix for the vulnerability as quickly as possible, ensuring the security of users all over the world. That’ s a story we think will be worth sharing.
Related Assets
If you’ lso are interested in learning more about Mozilla’ h security initiatives or Firefox safety, here are some resources to help you get started:
Mozilla Protection
Mozilla Security Blog
Bug Bounty Program
Mozilla Protection playlist on YouTube
If you liked Delivery a security update of Firefox in under a day by Sylvestre Ledru Then you'll love Web Design Agency Miami
Related Posts
About The Author
