UPDATE: If you have been granted access to Mozilla’s jump host, you can create an SSH tunnel to the internal symbolication server:

ssh -L 8000:breakpad-symbolapi1.dmz.phx1.mozilla.com:80 mpt-vpn.mozilla.com -N

You will then be able to use Benoit’s SPS extension and the custom about:telemetry extension out of the box on Windows profiling builds.

UPDATE #2: Refreshed the links to point to a more recent build of Firefox, the old version was incompatible with the latest SPS extensions


In my previous blog post, I gave an overview of the new Hang Detector feature and the Snappy Symbolication Server. Since then, more than one person has asked me for info on deploying the server locally so I’ll reproduce a quick and dirty walkthrough here for easy reference. The instructions below are the minimum required to see a demo of the functionality.

Setting up a profiling Nightly:

  1. Download and install the profiling Nightly from March 28th, 2012: https://people.mozilla.com/~vdjeric/firefox-15.0a1.en-US.win32.installer.exe
  2. Disable updates in the profiling Nightly from Firefox -> Options -> Options -> Advanced -> Update -> “Never check for updates”.
    This will prevent this test version of Nightly from getting out of sync with the debug symbols you will download in the next step.
  3. Opt into Telemetry from Firefox -> Options -> Options -> Advanced -> General -> “Submit performance data”. Restart the browser.
  4. Install a custom version of the about:telemetry extension: https://people.mozilla.com/~vdjeric/ping.telemetry.xpi

Setting up a Symbolication Server locally:

  1. Download some of the symbols for this Nightly profiling build from https://people.mozilla.com/~vdjeric/symbols_ffx.zip. Extract the zip file locally.
  2. From Windows command line: “git clone git://github.com/vdjeric/Snappy-Symbolication-Server.git”. You can get “git” for Windows from http://code.google.com/p/msysgit/ or you can install Cygwin.
  3. Edit the Snappy Symbolication Server’s “sample.conf” file so that firefoxSymbolsPath and osSymbolsPath point to the “symbols_ffx” directory extracted in step #5. You might also want to set enableTracing = 1 to help with identifying issues.
  4. Run the symbolication server locally with “python symbolicationWebService.py sample.conf”. You can get python for Windows from http://www.activestate.com/activepython/downloads or you can install Cygwin.

Testing:

  1. Click on “Force Hang” from the “about:telemetry” URL in the profiling Nightly, acknowledge all alerts. Reload the page and click “Symbolicate Stacks”. You should now see a stack that looks something like this:

Hang report #1 (18 seconds):

??? (in ntdll.dll)
PRMJ_Now (in mozjs.dll)
js_Date(JSContext *,unsigned int,JS::Value *) (in mozjs.dll)
???
???

The question marks above are for OS libraries (ntdll.dll) for which you don’t have local symbols and JIT-ed code on the bottom of stack which is not possible to stackwalk reliably.

Note: To generate your own Firefox symbols instead of downloading a sample as above, you’ll first have to make a local debug build of Firefox with an additional “ac_add_options –enable-profiling” line in your .mozconfig. After you’ve built this profiling version of Firefox, you can generate symbols for all your Firefox libraries by running “make buildsymbols” from your object directory.