Oct 03, 2017

WebExtensions Migration Story of Tree Style Tab


I started to develop WebExtensions-based version of the Tree Style Tab at late August 2017, and released as the version 2.0 at 26th November.
(A screenshot of Firefox Nightly 58 with Tree Style Tab's sidebar panel)

The largest reason why I did it is: many numbers of new WebExteisons APIs I required are landed to Firefox 57. Thank you developers for their great effort.

There is no technical novelty topics, but I wrote this as a historical document: a migration story of a very legacy addon.

What is the "Tree Style Tab"?

Tree Style Tab aka TST is a Firefox addon which helps your web browsing, providing ability to show tabs as tree-view and making histories and relations of them clearly visible.

An Old Tale / Past History of TST

TST is originated from an ancient Firefox addon TabBrowser Extensions aka TBE and a very unique web browser named "iRider".

The TBE was an all-in-one sytle addon which empower poor tabber browsing features of Firefox 1.5 around 2004. I mindlessly merged many good-looking features to TBE, including iRider's features: "tree-view, tab like UI of histories with thumbnails" and "ability to close multiple tabs by one action: dragging on closeboxes".

However, because it became too chaosful, I couldn't update TBE for Firefox 2 and it died as an abandoned addon for Firefox 1.5. I gave up to selvage TBE, instead I decided to redevelop new tiny feature-oriented addons from features of TBE I really required. TST was born as one of such feature-oriented addons, to provide tree-view tabs based on indentation, at 2007.

Many numbers of addons gave up to be updated for frequently updated Firefox, but TST is still alive until 2017. And now there is TST 2.0, an WebExtensions-based version for Firefox 57 and later - it is the largest milestone of TST since it was born.

Phase 1: Planning for Migration

Before I started to migration, I knew that well: it is impossible to completely-migrate TST to WebExtensions. Thus I decided the starting point of this migration project as: make the concept clear and triage many features.

Not "Supporting to WebExtensions APIs", but "Development of WebExtensions-based version TST"

By the way, I sometimes see a phrase like "supporting to WebExtensions" on some blog posts in Japan. However I mainly use another phrase "WebExtensions-based version". What's the difference?

Basically Firefox is an application like webapp, built from XML + CSS + JavaScript running on the Gecko rendering engine. Inherently a legacy addon is just a "monkey patch", injecting arbitrarily scripts into the namespece of Firefox's scripts.
(Architecture of Legacy Addons)

On the other hand, an WebExtensions-based addon is a small software executed in a sandboxed namespace, and it communicates with Firefox both way via WebExtensions APIs.
(Architecture of WebExtensions-based Addons)

Because they are completely different, a legacy addon cannot be migrated to WebExtensions-based one only with small changes, and we need to create a new WebExtensions-based addon which provides similar experience to users. As a matter of fact, TST 2.0 is also a new addon created from scratch. This is the reason why I use the phrase "WebExtensions-based version".

Spec Determined from Technical Limitations

By limitations of WebExtensions APIs, some legacy TST's features are definitely migratable but some others are unmigratable.

  • The sidebar feature is the only one choice to implement vertical tab bar. Sidebar on WebExtensions is impossible to be opened by arbitrarily trigger (including mouseover), and impossible to be changed its position to top or bottom. Thus "auto hide tab bar" and "tree in horizontal tab bar" are impossible to be implemented.
  • By the policy of WebExtensions APIs, addons cannot block Firefox's default action (or do something) around tabs before it is actually done. Because addons can listen after-action events for tabs, any feature require hooking before tab open cannot be implemented. Because addons only can listen after-action events for tabs, any feature require hooking before tab open cannot be implemented.
  • By the reason told at the previous topic, there is only one available source to detect the "parent" tab of a new tab: tabs.Tab.openerTabId. openerTabId was available on Google Chrome but wasn't on Firefox's WebExtensions for a long time. To be honest, it was the largest trigger of starting this migration project that the feature becomes available on Firefox 57.
  • Legacy TST stores tree structure of tabs as a part of session information of tabs and windows, but when I started this migration project, WebExtensions' sessions API didn't provide features to do that. However, the bug requesting the feature seemed to become fixed soon (and actually landed on Firefox 57).

After these researches, I started this migration without concrete assurance of success.

Intentional Decisions without Technical Reasons

When we migrate a legacy addon to WebExtensions-based version, we need to judge all features: "migratable, and migrate", "unmigratable and drop", and "migratable but drop from too large cost". This is very hard and stressful. Why I successfully did it on TST 2.0 is: there is a clear concept, and I did all decisions based on it.

Since I initially developed TST as a spin-out of TBE, I have two concepts for TST:

  • Concentrate to the single feature "tree of tabs". Even if it is technically possible, don't include features not related to "tree" (and not included in Firefox itself).
  • Instead, design it as easy-to-coraborate with other addons, easy-to-combinate features users want.

Feature-oriented addon is easily kept its simple architecture. Just my private opinion; because not only it has less features, but its clear concept introduces stability and consistency to its design also.
(Pros-Cons of All-in-One Style Addon)
(Pros-Cons of Feature-Oriented Addon)

Clear criterion reduces costs to judge. It is the best way: "drop non-important things and concentrate to only important things", but to do that actually, we need to make it clear: "what is really important for the project?"

Less Synergy between WebExtensions-based addons

And, I really wanted to keep the concept: high interoperability with other addons.

You will think that a monkey-patch legacy addon seems hard to be work with other addons, and addons based on stable WebExtensions APIs seems do it easily. But, on another viewpoint, that is false. Because WebExtensions-based addons are clearly separated to sandboxed namespaces, they cannot collaborate with others easily.

For example, in old addons world, addons could collaborate with others implicitely without special hard work, like: Tree Style Tab provides a vertical tab bar, another addon embeds thumbnail to tabs, more another addon handles double-click on tabs, and some more another shows a sidebar panel to the bottom blank space of the tab bar.
(There is Synergy Between Legacy Addons)

However, that is impossible on WebExtensions. TST 2.0's vertical tab bar is just a sidebar panel and it is isolated, so features of other addons never affect to it. Moreover, you cannot show both TST's tab bar and bookmarks sidebar together, because sidebar panels are exclusive. You can use features of each addon separately, but you cannot combine them - there is only few synergy.
(Less Synergy Between WebExtensions-based Addons)

Even if an addon provides very useful feature, it won't be really useful if it is exclusive with other addons. Possibly the user will abandon one side, possibly both sides. This is the largest my worrying about WebExtensions.

From the conclusion: TST 2.0 lost its implicit interoperability with other addons. Other addons can collaborate with TST only around few points. For intentional combinations, TST 2.0 still provides APIs for other addons. I wrote more about this later.

Phase 2: Early Development

As I told, migration of TST to WebExtensions means creation of a new WebExtensions-based addon. This entry doesn't describe about generic information about development of WebExtensions addons, instead describing about development of TST on WebExtensions.

Experimental Development of Basic Features

I had two choices to develop TST on WebExtensions: as an improved version of other existing addon, or an independent addon created from scratch.

Because there are some existing vertical-tab addons and TST 2.0 is (maybe) last player, it is reasonable that I develop it as a folked version of one of those preceding addons. However, I worried about it is very hard to read and compare codes of them, and I needed to migrate myself from legacy addon way to WebExtensions way, so I decided to create TST 2.0 from scratch to learn WebExtensions way. (But to be honest, I started to develop a simple experimental version anyway and I realized that it is very heavy project. Because I have to know details of my addon TST 2.0 completely, I decided to start development from very small implementation and improve it step by step.)

To learn about sidebar APIs and verify a new feature tabs.Tab.openerTabId, I started to develop a simple sidebar addon providing vertical tabs which observes tab events:

  • Do nothing before opening tab, instead attach an already opened tab to existing tree based on the tab's information especially tabs.Tab.openerTabId - it means a child tab is intentionally opened from the parent.
  • When a tab is moved by dragging on the tab bar of Firefox itself or done by other addons via WebExtensions API, embed it into existing tree from its position.
  • When a parent tab is closed, maintain the tree to keep its structure by promoting one of children to new parent.
  • When a parent tab is moved inside of descendants of itself, it is pushed back to the previous position, to prevent broken tree structure.
  • When a parent tab is closed and it has collapsed descendants, then close all them also together.

While development I needed to verify behaviors around closing of a tab in a tree again and again, thus I implemented more features: collapse/expand tree, drag and drop of tabs, moving tabs across windows, auto-scrolling of the tab bar, and more. By some reasons - they didn't depend on Firefox's implementation, and it is very hard that completely reimplement them keeping old behavior - I copied many codes for these features from legacy TST.

I thought that the feature "auto-grouping of newly opened tabs" was unmigratable by a technical reason that addons cannot hook before new tabs are actually opened, but finally I implemented it in different form: when multiple tabs are opened with no "opener" information, they are grouped as a folder tab because maybe they are opened from a bookmark folder. This is a last-ditch measure, but it seems to work as expected in most cases.

And more, it was out of the plan but I implemented restoration of tree structure after restart, based on storage.local. Basically I don't like to implement such a temporary feature - definitely removed in future versions, however it was really required to verify various tree-related features with restarting. (But finally I removed this temporary mechanism and completely replaced with the better one based on browser.sessions.setWindowValue() and browser.sessions.setTabValue().)

Changing to Centralized Architecture with the Background Page

Legacy TST worked as a part of each Firefox window and there was no central management system.
(Architecture of Legacy TST)

TST 2.0 was also started based on similar policy - implemented only with sidebar in each window - but I decided to change the architecture totally: the background page manages everything as the master process and each sidebar simply render tree based on messages from the master.
(Architecture of TST 2.0)

There are major two reasons:

  • It was very hard to manage various flags without the master process.

    When observing tab related events, we need to detect which is triggered by TST itself and which is done by other addons or user's action, because we need to fixup broken tree structure automatically if the change is done from outside of TST.

    WebExtensions API doesn't provide any feature to give something extra information to opened tabs, so we need to use a flag like "TST is now trying to open a new tab" at first, then call an WebExtensions API to open a new tab, and finally unflag it. However, inter-window messages are asynchronous on WebExtensions and it is very hard to manage such flags for complicated cases like a moving of tabs across windows, like: "Which window should flag it?", "When should we unflag it?", "Which message is actually delivered at first: flagging, or requesting of a new tab?", and more. To manage such flags synchronously, we need to do it on a single master process.

  • We need to track changes around tabs while the sidebar is closed/unloaded.

    The sidebar feature of WebExtensions just provides an exclusive sidebar panel, so TST's vertical tab bar may be unloaded by some reasons. Then, tree structure of tabs are easily broken by opening/closing/moving of tabs. Because we never know the reason why the tab is opened/closed/moved later, it is very hard to maintain completely broken tree when the sidebar is reloaded. Instead we need to listen tab events constantly and maintain tree on the time. Only the master process living while the sidebar is closed can do that.

Because new APIs to store/read extra information to windows and tabs are landed to Firefox while I doing this changing, I started issuing unique IDs for tabs and store them. (Already there is a request on Bugzilla but it doesn't fixed yet for a long time, so we have to do it by self.) This is also one of important tasks of the master process.

On actual implementation, the centralizing strategy is partially applied.
(Centralized Architecture which is Actually Not-Implemented)
Actually both background page and sidebars track tab events and sidebars autonomously maintain themselves for simple cases, and they keep tree information by self.
(Semi-Autonomous Architecture which is Actually Implemented)
As the result, event handlers for drag-and-drop events can do complicated decisions intelligently without asynchronous messaging, based on complete information of tabs stored in the sidebar itself.

Edit at 2021-08-12: Finally the semi-autonomous strategy described above was migrated to more simpler fully-centralized strategy, to avoid various troubles from broken sychronization. The sidebar page still has complete information of tree structure, but the sidebar just notifies events, and always the background page operates tabs and the tree structure based on the notifications, then the sidebar is fully updated based on messages from the background page.

(By the way, there is an API browser.runtime.getBackgroundPage() to access the namespace of the background script from sidebar scripts, but TST doesn't use it and using browser.runtime.sendMessage() for all massaging between background and sidebar scripts, because browser.runtime.getBackgroundPage() doesn't work on private-browsing windows.)

Styling and Animation Effects based on CSS

I knew that styling of tabs is definitely possible but takes time, so I did experimental implementations at first. After successful experiments, I started to do those visual tasks.

Both legacy TST and TST 2.0 control appearance of tabs with CSS, but TST 2.0's styling is very improved.

  • Legacy TST needed to cancel Firefox's built-in styles of tabs and applied tree styles after that. Because is very difficult, there were very large numbers of style definitions to cancel default styles for each Firefox version and each platform.
    (Style rules of legacy TST: there are too many lines to cancel Firefox's builtin styles)

    However, sidebar of WebExtensions is completely isolated document and we need to define all styles for sidebar by self, so I only needed to write very simple CSS.
    (Style rules of TST 2.0: very simple and there are only truly required style definitions)

  • Legacy TST applied animation effects around tabs with complexed combination of JavaScript and CSS, but now TST 2.0's animation are done with pure CSS.

    When we define appearance of UIs with CSS, it is important that defining basic size of UI elements. The size of tab icon is consistently 16px, but text size can be 12px, 16px or others. Thus tabs can have odd appearance in some environments, if I define them with fixed size - sometimes too small, sometimes too large. Relative units like "em" don't solve this problem.

    In TST 2.0, this problem is solved by custom properties feature of CSS. Custom properties are similar to variables, and you can refer the value of them via var(variable name) and easily overridden with dynamic definitions. Combinations of this and calc() can define appearance of tabs flexibly, like "a half of --tab-height", "100% height minus --favicon-size", and more. On the startup TST measures actual size of UIs with getBoundingClientRect() and defines basic UI size as a custom property overriding the old one - it seems more cross-platform friendly.
    (Codes to generate style sheet dynamically to define custom properties based on measured tab size)

    Dummy elements to measure their size have special style definitions: position: fixed; visibility: hidden; opacity: 0; pointer-events: none;, to make them invisible and unclickable. Elements with pointer-events: none are very useful to show arbitrarily visual effect on existing UIs without blocking of user interaction, without complex background images, deeply nested elements, and so on.

  • Dynamically calculated sizes and custom properties are also applied to animation effects around indentation and collapse/expand tree. Legacy TST applies animation effects to each tab via its style attribute directly, however TST 2.0 doesn't. There is only one dynamically-generated style sheet including animation definitions, and each tab element just gets/loses classes to trigger animation effects.
    (Generated CSS rules to define animated indentations)

    Not only custom properties (var()) and calc(), but the template syntax also made it possible. Template syntax allows us to define quite long string literal with embedded variables very easily. (This improvement can be backported to legacy TST, but I won't do it by self because I hope to concentrate my resources to development of future versions.)

Phase 3: Reintroduce Interoperability with Other Addons

Because TST 2.0 was becoming to usable as a single addon, I started to try reintroducing interoperability with other addons - it is the important concept of TST.

Implicit Collaboration of Addons

Some basic features of TST are designed to affect to other addons implicitly.

  • browser.tabs.create() called with openerTabId produces a new child tab. (Any new tab with openerTabId is handled by TST automatically.)
  • When a tab is closed and there are collapsed descendants, they are also closed automatically.

So, other WebExtensions addons like mouse gesture, custom keybindings, etc will work with TST naturally.

Context Menu on Tabs

However, WebExtensions-based addons cannot work together on their own area including sidebar implicitly. Additional context menu items seem reasonable alternative of such combinations, but currently it is impossible to provide custom tab bar UI in the sidebar quoting native tab context menu with added items. (There are some requests like bug 1376251 and bug 1396031 for this purpose.)

To be honest I hoped to wait until those bugs become fixed, but context-menu-less tab bar is too unusable. Thus I decided to implement fake context menu inside the sidebar reluctantly, like other vertical tab addons.

As my last stand, I designed TST's fake context menu to mimic Firefox's native one as possible as I can, instead of unique one.
(Screenshot of fake context menu in the sidebar)
Because it is just a temporary/disposable implementation until native context menu become available on TST's sidebar. I don't want to maintain it as "more useful/unuseful than Firefox's one", except some technically impossible features.

So TST's APIs mimic native menus APIs of WebExtensions. Because they are designed as subsets of WebExtensions menus APIs, there are some merits for other addons:

  • They can support TST's fake context menu with just few changes, if they already provide context menu items via WebExtensions menu API.
  • If any new WebExtensions API become available to provide native context menu on sidebar in future versions of Firefox, then you simply only have to remove codes for TST's fake context menu.

For dogfooding, TST's custom context menu items are also implemented via this fake APIs.
(Screenshot of extra menu items in fake context menu in the sidebar)

Edit at 2021-08-12: After that the menus API was improved to override Firefox's native context menu at Firefox 64, thus TST was also updated to use the feature. As the result context menu commands added by other addons are automatically enabled at the context menu on TST's sidebar, and they collaborate with each other implicitly.

Other APIs

On the other side, TST now has two groups of API types except fake context menu: aggressive APIs to send commands to TST, and passive APIs to receive notification messages from TST.

  • For aggressive APIs, your addon needs to send messages via browser.runtime.sendMessage(), to collapse/expand tree, to attach/detach tabs, and so on. They returns results as Promise, like generic WebExtensions API.
  • For passive APIs, your addon needs to register a listener for browser.runtime.onMessageExternal, to receive notifications about events on the vertical tab bar like click, dragging, etc. One more, you need to register your addon itself to TST. This is due to a limitation of browser.runtime.onMessageExternal - an addon cannot broadcast any message to others without indicating receiver's ID.

For passive APIs, now please remind about importance of the order of addons' initialization. When TST is initialized before your addon does, then yours just have to send a message to TST. However, sometimes other addons (including yours) can be initialized before TST starts to listen messages from other addons. In this case, other addons cannot know when they should send messages to register themselves to TST. Of course TST cannot broadcast any message to notify it is ready.
(Sequence Graph of Two Addons Combined with Messaging)

Eventually addons cannot solve this problem fundamentally, TST now has a strategy:

  • TST basically doesn't send any message to others aggressively.
  • If there is any addon previously registered itself, TST notifies a "ready" event to it.
  • If an addon which can work with TST is activated before TST is installed, TST don't work with it until the addon is reloaded (disabled and enabled) by the user or it is restarted from auto-update.

(Sequence Graph of Successful Initialization Based on Cached Information)

At beginning I designed passive APIs to notify only minimum information like custom DOM events on legacy TST, but they didn't work as I expected, because WebExtensions APIs are quite asynchronous. By continuous events like mouseover, tab's state is changed again and again while you are requesting to get "current" state of tabs, so listeners cannot know what they should do correctly.
(Sequence Graph of Combination Based on Notifications with Poor Information)

So I changed TST's API strategy like native WebExtensions APIs: put rich information to notification messages. For example, a notification message for click on a tab will have a complete tabs.Tab object with more extra information: descendant tabs and states of the tab.
(Sequence Graph of Combination Based on Notifications with Rich Information)
If you are planning to provide something notification APIs for other addons, please note that such rich informations are important to make your API actually usable for developers.


Never Surrender, then You'll Find Out New Way... Maybe

As above, the project to migrate TST to WebExtensions is reached at a milestone; TST 2.0.

Someone may say the project just failed, because there are some dropped fatures, because there are different (lesser) user experience. However, if I aimed to such goals, I couldn't release TST 2.0. It has been successfully done, because I gave up to struggle around technically impossible things, and because I decided to concentrate to features very important for myself.

I think other authors of legacy addons are also have such migration problems. You may be despair because it is impossible to completely migrate everything. But I recommend you to try migration - actually some TST features were successfully migrated regardless I gave up to migrate it before. I don't hope disappearing of such pioneers with legacy addons. Anyway please think more deeply and find out a new solution to reproduce the unique core value of your addon for future versions of Firefox.

Future of WebExtensions, Future of My Addons Except TST

Legacy addon system was quite flexible and various unique addons were born on it. Such addons were not there if Firefox's addon system was started with few limtied APIs like the WebExtensions.

But on the other side, because legacy addons were actually just "monkey patch"s, they had negative side effects - strong dependency on specific Firefox version, breaking of Firefox's native features, and so on. As the result, only few experts could develop safe addons.

Non-WebExtensions addons definitely die, on a new product beyond Firefox, or on a version of Firefox. Actually it happens on the version 57 of "Firefox" - it seems better than any other future timing. (Of course it seems possibly too late and we should did that on more earlier versions - but we couldn't did.)

Now WebExtensions API provides some unique features not included in Google Chrome's spec. browser.sessions.setTabValue(), browser.sessions.getTabValue(), browser.sessions.setWindowValue(), and browser.sessions.getWindowValue() are also, and TST couldn't be migrated to WebExtensions if they are not available. Most my proposals about new WebExtensions APIs are actually rejected, but they finally made the policy of WebExtensions APIs more clear, so I belive that my work is not meaningless. Because I'm a Firefox user and I still depend on many other my own addons, I still take effort for migration of them.

As the last process of TST's migration project, I wrote this long article.

TST 2.0 is now public, and I got many many feedbacks day by day. I thought to develop the WebExtensions-based version Multiple Tab Handler at the next step, but it will take more time...

By the way, I'm ordinally working as an employee at ClearCode Inc., for technical support around company use of Firefox and Thunderbird, based on knowledge from experience of addon development.

Moreover, like as figures in this article, I'm drawing comic-style articles describing Linux command-line knowledge: System Administrator Girl, on an magazine "Nikkei Linux".

Thus this article became one of my all-out effort unintentionally. I'm happy if this helps or amuses you. Thanks!


wikieditish message: Ready to edit this entry.