987406 - Add ssltunnel to xre.zip bundle used for b2g tests

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Martin Thomson [:mt:]]

Running tests that use https resources means that ssltunnel needs to be run.  It is not present in the current build of the xre.zip, so cannot run.

Bug 907770 includes the gecko-side changes for this.  That requires the xre bundle, currently at http://runtime-binaries.pvt.build.mozilla.org/tooltool/sha512/cba263cef46b57585334f4b71fbbd15ce740fa4b7260571a9f7a76f8f0d6b492b93b01523cb01ee54697cc9b1de1ccc8e89ad64da95a0ea31e0f119fe744c09f to be updated.

Bug 742597 caused a bundle to be created that would probably suffice, I just want to make sure that there aren't other things here that need to be considered.

Comment 1

[Jonathan Griffin (:jgriffin)]

The tegra bundle won't work for B2G, because it's too old, and its httpd.js won't match in-tree server.js which is currently bundled in tests.zip.

I think we should add ssltunnel from a linux32 desktop build to the existing xre.zip from tooltool.

Comment 2

[Jonathan Griffin (:jgriffin)]

Actually, we'll need linux32, linux64, and osx versions of this.

Comment 3

[Jonathan Griffin (:jgriffin)]

The most recent copy of xre.zip comes from xulrunner-sdk-26.  I'll take this and add ssltunnel from here:  http://ftp.mozilla.org/pub/mozilla.org/firefox/releases/26.0/

Comment 4

[Jonathan Griffin (:jgriffin)]

For the record, the xre.zip in bug 987957 is the combination of http://runtime-binaries.pvt.build.mozilla.org/tooltool/sha512/d4a0da54e75c27cd2f535e66b586f119ef08b3bde4a9eee03662d296b3434189c542c0a7e7a75954030c04396a9823e22e1f884f5d87c0f4017944cd50ff38de (which we're currently using on linux64 gaia unit tests) with ssltunnel from http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/26.0-candidates/build2/linux-x86_64/en-US/firefox-26.0.tests.zip.

If this works, I'll create linux32 and osx packages as well.

Comment 5

[(not currently active) Ted Mielczarek]

Should be fine, ssltunnel hasn't changed in a while. (I wish we just had implemented this all in Python.)

Comment 6

[Jonathan Griffin (:jgriffin)]

Attachment: Use updated xre.zip in emulator unittests,

I'll update desktop configs in a separate patch.

Comment 7

[Andrew Halberstadt [:ahal]]

Comment on attachment 8398074 [details] [diff] [review]
Use updated xre.zip in emulator unittests,

Review of attachment 8398074 [details] [diff] [review]:
-----------------------------------------------------------------

Lgtm.. I guess :)

Comment 8

[Jonathan Griffin (:jgriffin)]

Comment on attachment 8398074 [details] [diff] [review]
Use updated xre.zip in emulator unittests,

https://hg.mozilla.org/build/mozharness/rev/bce52213ed6b

Will watch cedar for bustage.

Comment 9

[Jonathan Griffin (:jgriffin)]

That definitely didn't work:  

15:34:32     INFO -  MochitestServer : launching [u'/builds/slave/test/build/xre/bin/xpcshell', '-g', '/builds/slave/test/build/xre/bin', '-v', '170', '-f', '/builds/slave/test/build/xre/bin/components/httpd.js', '-e', "const _PROFILE_PATH = '/tmp/tmp60hZKU'; const _SERVER_PORT = '8888'; const _SERVER_ADDR = '10.0.2.2'; const _TEST_PREFIX = undefined; const _DISPLAY_RESULTS = false;", '-f', './server.js']
15:34:32     INFO -  runtests.py | Server pid: 2392
15:36:02  WARNING -  TEST-UNEXPECTED-FAIL | runtests.py | Timed out while waiting for server startup.
15:37:06     INFO -  Traceback (most recent call last):
15:37:06     INFO -    File "runtestsb2g.py", line 339, in run_remote_mochitests
15:37:06     INFO -      retVal = mochitest.run_tests(options)
15:37:06     INFO -    File "runtestsb2g.py", line 117, in run_tests
15:37:06     INFO -      self.startWebServer(options)
15:37:06     INFO -    File "runtestsb2g.py", line 217, in startWebServer
15:37:06     INFO -      self.server.ensureReady(90)
15:37:06     INFO -    File "/builds/slave/test/build/tests/mochitest/runtests.py", line 190, in ensureReady
15:37:06     INFO -      sys.exit(1)
15:37:06     INFO -  SystemExit: 1
15:38:09     INFO -  Automation Error: Exception caught while running tests

I've backed out and will investigate.

Comment 10

[Jonathan Griffin (:jgriffin)]

This failed because I failed to add httpd.js to the new xre.zip.  :(  I've fixed this, but it will require another tooltool upload.

Comment 11

[John Hopkins (:jhopkins)]

in production.

Comment 12

[Jonathan Griffin (:jgriffin)]

ok, trying again: https://hg.mozilla.org/build/mozharness/rev/e2da84d8c986

Comment 13

[Martin Thomson [:mt:]]

No screams yet?  Trying this out: https://tbpl.mozilla.org/?tree=Try&rev=e2aa932bebb7

Comment 14

[Jonathan Griffin (:jgriffin)]

You won't see it in a try job yet; mozharness changes have to be explicitly moved into production before they take effect globally.  I'm watching some jobs on cedar (which works on pre-production changes), and if they go green, I'll put this patch into production.

Comment 15

[Jonathan Griffin (:jgriffin)]

This patch works 80% of the time!

Seriously.

It works for most emulator tests, but unfortunately it fails on reftests running on the old r3 mac minis.  Apparently those machines are 32-bit, since this version of xre.zip fails with:

OSError: [Errno 8] Exec format error

We're phasing tests out on r3 mac minis, but we're not quite there, so I'll have to back this out and try again with a 32-bit version.

Comment 16

[Jonathan Griffin (:jgriffin)]

(The good news is that the 64-bit xre.zip works fine and I'll use that for linux64 b2g desktop builds.)

Comment 17

[Jonathan Griffin (:jgriffin)]

Landed the 32-bit xre.zip change:  https://hg.mozilla.org/build/mozharness/rev/52636b92cbfd

Watching cypress...

Comment 18

[Jonathan Griffin (:jgriffin)]

(In reply to Jonathan Griffin (:jgriffin) from comment #17)
> Landed the 32-bit xre.zip change: 
> https://hg.mozilla.org/build/mozharness/rev/52636b92cbfd
> 
> Watching cypress...

Great success!  I'll merge to production on Monday a.m.

Comment 19

[John Hopkins (:jhopkins)]

in production.

Comment 20

[Jonathan Griffin (:jgriffin)]

So, the emulators are done, I now need to fix the B2G desktop build tests.

Comment 21

[Martin Thomson [:mt:]]

The desktop builds should be OK, since they are compiling xpcshell and ssltunnel in a compatible form.

Comment 22

[Jonathan Griffin (:jgriffin)]

Ah, right.  In that case, I'll close this; re-open if needed.

Comment 23

[Martin Thomson [:mt:]]

Encountering issues with the b2g emulator: https://tbpl.mozilla.org/?tree=Try&rev=2b5bb0340054

The actual problem seems consistent:

10:35:59     INFO -  INFO | runtests.py | SSL tunnel pid: 2413
10:35:59     INFO -  Failed to init NSS:

This suggests that the xre bundle (http://runtime-binaries.pvt.build.mozilla.org/tooltool/sha512/ce1ad86b21d534bd706cd70f1d763b9ca1cfd0a1ea4d519864c961e202c36ccf48bcfa9c572113076ee8b3d0770f60b8967b7620b4adaad531f47b993eb57912) doesn't contain a version of ssltunnel that runs on the emulator machines.  Given that it's a 64bit Ubuntu machine, I'm at a loss to explain this.

Comment 24

[Jonathan Griffin (:jgriffin)]

We're off the mac minis now, so will try bumping our xre.zip to 64-bit.

Comment 25

[Jonathan Griffin (:jgriffin)]

Attachment: Use a 64-bit version of xre.zip on emulator unittests,

Comment 26

[Jonathan Griffin (:jgriffin)]

Comment on attachment 8407778 [details] [diff] [review]
Use a 64-bit version of xre.zip on emulator unittests,

https://hg.mozilla.org/build/mozharness/rev/2a89305f4a67

Comment 27

[Justin Wood (:Callek)]

(In reply to Jonathan Griffin (:jgriffin) from comment #26)
> Comment on attachment 8407778 [details] [diff] [review]
> Use a 64-bit version of xre.zip on emulator unittests,
> 
> https://hg.mozilla.org/build/mozharness/rev/2a89305f4a67

In production

Comment 28

[Jonathan Griffin (:jgriffin)]

Martin, can you try again?  The 64-bit version is now live.

Comment 29

[Martin Thomson [:mt:]]

Bug 907770 needs a rebase.  It seems like these files see a lot of churn.  Let me see if I can't get a try run overnight.

Comment 30

[Martin Thomson [:mt:]]

See https://tbpl.mozilla.org/?tree=Try&rev=ec37aba56d96

Comment 31

[Martin Thomson [:mt:]]

Looks like our old friend:  https://tbpl.mozilla.org/php/getParsedLog.php?id=37968603&tree=Try&full=1

17:39:41     INFO -  INFO | runtests.py | SSL tunnel pid: 2420
17:39:41     INFO -  Failed to init NSS: None

A few builds seem to be burning here, but it doesn't look like anything else is broken.

Comment 32

[Jonathan Griffin (:jgriffin)]

I can reproduce this locally; it doesn't seem to be a problem with ssltunnel itself, but how we're invoking.  Digging more...

Comment 33

[Martin Thomson [:mt:]]

I don't know if this helps, but there's a possibility that the emulator needs to use a different type of certificate DB.  Android insists on having the DB in the "new" form, the desktop tests in the "old" form.  Maybe my intelligence on this was wrong:

    if self.certdbNew:
 android and b2g use the new DB formats exclusively
		certdbPath = "sql:" + options.profilePath
		else:
		# desktop seems to use the old
		certdbPath = options.profilePath

Comment 34

[Martin Thomson [:mt:]]

I hope that the missing comment in the above is obvious - I have to remember that tab thing.

Comment 35

[Jonathan Griffin (:jgriffin)]

At least one of the problems (hopefully the only one) is that we're producing an invalid ssltunnel.cfg:

httpproxy:1
certdbdir:None
forward:127.0.0.1:8888
websocketserver:10.0.2.2:9988
listen:*:4443:pgo server certificate
websocketserver:10.0.2.2:9988
listen:*:4443:pgo server certificate
listen:self-signed.example.com:443:4443:selfsigned
...

We don't include a real certdbdir because we're not passing --certificate-path to runtestsb2g.py.  This will need a mozharness fix, and then an in-tree config fix.

With these things fixed locally, everything works for me on my machine, so we'll hope the same is true for buildbot.

Comment 36

[Jonathan Griffin (:jgriffin)]

Attachment: Set up certificate_path processing,

This mozharness part of the fix.

Comment 37

[Jonathan Griffin (:jgriffin)]

Attachment: Pass certificate_path to B2G mochitests,

The in-tree part of the fix.

Comment 38

[Jonathan Griffin (:jgriffin)]

Comment on attachment 8408735 [details] [diff] [review]
Set up certificate_path processing,

Review of attachment 8408735 [details] [diff] [review]:
-----------------------------------------------------------------

ahal is on Easter break today

Comment 39

[Jonathan Griffin (:jgriffin)]

Comment on attachment 8408735 [details] [diff] [review]
Set up certificate_path processing,

https://hg.mozilla.org/build/mozharness/rev/d92a223fe008

Comment 40

[Martin Thomson [:mt:]]

Should a respin of the try run work at this point?

Comment 41

[Jonathan Griffin (:jgriffin)]

(In reply to Martin Thomson [:mt] from comment #40)
> Should a respin of the try run work at this point?

No.  I need to verify the above landing in staging on cypress, push it to production, then land the gecko bits.  I'll ping you when it's ready to re-test.

Comment 42

[Jonathan Griffin (:jgriffin)]

(In reply to Jonathan Griffin (:jgriffin) from comment #39)
> Comment on attachment 8408735 [details] [diff] [review]
> Set up certificate_path processing,
> 
> https://hg.mozilla.org/build/mozharness/rev/d92a223fe008

in production

Comment 43

[Jonathan Griffin (:jgriffin)]

Martin, the mozharness piece is in production now.  I haven't been able to land the gecko piece, because m-i is closed, but if you add this patch to your patch queue, you could try it out with a try job anyway:

https://bug987406.bugzilla.mozilla.org/attachment.cgi?id=8408736

Comment 44

[Jonathan Griffin (:jgriffin)]

Ah, m-i just opened and I landed it:  https://hg.mozilla.org/integration/mozilla-inbound/rev/5d9fb05d9d12

You'll still need to include this patch in your queue, or rebase on top of m-i before you push to try.

Comment 45

[Martin Thomson [:mt:]]

Rebase, rebase, rebase...

https://tbpl.mozilla.org/?tree=Try&rev=f7c3e60b8541

Comment 46

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/5d9fb05d9d12

Comment 47

[Martin Thomson [:mt:]]

Good news, it looks like ssltunnel runs.  Bad news, the test still times out. But it does seem to at least make progress, so it might be a test issue. Signs are good for this bug.

Comment 48

[Jonathan Griffin (:jgriffin)]

I also saw that we're producing an ssltunnel.cfg in which a couple of lines are repeated:

websocketserver:10.0.2.2:9988
listen:*:4443:pgo server certificate
websocketserver:10.0.2.2:9988
listen:*:4443:pgo server certificate

Not sure if this would cause any problem, though.

Comment 49

[Jordan Lund (:jlund)]

mozharness patch is in production: http://hg.mozilla.org/build/mozharness/rev/c9615cf1645a :)

Comment 50

[Martin Thomson [:mt:]]

(In reply to Jonathan Griffin (:jgriffin) from comment #48)
> I also saw that we're producing an ssltunnel.cfg in which a couple of lines
> are repeated:
> 
> websocketserver:10.0.2.2:9988
> listen:*:4443:pgo server certificate
> websocketserver:10.0.2.2:9988
> listen:*:4443:pgo server certificate
> 
> Not sure if this would cause any problem, though.

I wouldn't have thought so.  I don't see how that would happen using either trunk or my modified code unless there is a python bug or something equally perverse.  Especially if those are the only lines duplicated, see: https://hg.mozilla.org/try/rev/5328e2ef6ad2#l2.197 or for trunk: https://hg.mozilla.org/try/rev/5328e2ef6ad2#l2.740

Comment 51

[Jonathan Griffin (:jgriffin)]

You're right; this was an artifact of how I was retrieving a copy of the file during testing.

Comment 52

[Martin Thomson [:mt:]]

Dammit, I should have noticed this more quickly.  The test that fails fails because it doesn't run on e10s.  Running on desktop with --e10s causes it to time out in exactly the same way.  I'll need to disable that test on b2g for that reason.  ssltunnel is working perfectly.

Comment 53

[Martin Thomson [:mt:]]

https://tbpl.mozilla.org/?tree=Try&rev=698102343745

Comment 54

[Martin Thomson [:mt:]]

https://tbpl.mozilla.org/?tree=Try&rev=4f9132738b08

Yes, it works.

Comment 55

[Jonathan Griffin (:jgriffin)]

\o/