Daniel R.: HTTPS testing with jWebUnit

A common problem with testing HTTPS is having to deal with self-signed certificates. While Java has made it pretty easy to add certificates to your local keystore, for development purposes it always seemed like more trouble than it was worth. While some of the standard solutions worked well in the past, trying to use jWebUnit which builds upon HttpUnit resurfaced many of the same issues. It seems HttpUnit knows best and forces using the old school Sun SSL provider in com.meterware.httpunit.HttpsProtocolSupport. As a result the typical Trust All Certificates hack doesn't work. To complicate matters connecting to localhost, 127.0.0.1, and whatever other name your machine might have, can cause validation issues. This can be worked around with a custom HostnameVerifier. Alas, again HttpUnit makes it hard to get around that problem too.

A simple jWebUnit test making an HTTPS connection to your local machine produces an error like:
java.lang.RuntimeException: java.io.IOException at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:602) at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:272) at com.sun.net.ssl.internal.www.protocol.https.HttpsURLConnectionOldImpl.getResponseCode(DashoA12275) at com.meterware.httpunit.HttpWebResponse.readResponseHeader(HttpWebResponse.java:162) at com.meterware.httpunit.HttpWebResponse.readHeaders(HttpWebResponse.java:200) at com.meterware.httpunit.HttpWebResponse.(HttpWebResponse.java:56) at com.meterware.httpunit.HttpWebResponse.(HttpWebResponse.java:67) at com.meterware.httpunit.WebConversation.newResponse(WebConversation.java:76) at com.meterware.httpunit.WebWindow.getResource(WebWindow.java:164) at com.meterware.httpunit.WebWindow.getSubframeResponse(WebWindow.java:128) at com.meterware.httpunit.WebWindow.getResponse(WebWindow.java:121) at com.meterware.httpunit.WebWindow.updateWindow(WebWindow.java:144) at com.meterware.httpunit.WebWindow.getSubframeResponse(WebWindow.java:130) at com.meterware.httpunit.WebWindow.getResponse(WebWindow.java:121) at com.meterware.httpunit.WebClient.getResponse(WebClient.java:113) at net.sourceforge.jwebunit.HttpUnitDialog.(HttpUnitDialog.java:48) at net.sourceforge.jwebunit.WebTester.beginAt(WebTester.java:55) at net.sourceforge.jwebunit.WebTestCase.beginAt(WebTestCase.java:46) (... additional lines deleted ...) Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: No trusted certificate found at com.sun.net.ssl.internal.ssl.BaseSSLSocketImpl.a(DashoA12275) at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA12275) at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA12275) at com.sun.net.ssl.internal.ssl.SunJSSE_az.a(DashoA12275) at com.sun.net.ssl.internal.ssl.SunJSSE_az.a(DashoA12275) at com.sun.net.ssl.internal.ssl.SunJSSE_ax.a(DashoA12275) at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA12275) at com.sun.net.ssl.internal.ssl.SSLSocketImpl.j(DashoA12275) at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(DashoA12275) at sun.net.www.protocol.https.HttpsClient.afterConnect(DashoA12275) at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(DashoA12275) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:626) at sun.net.www.protocol.http.HttpURLConnection.getHeaderFieldKey(HttpURLConnection.java:1504) at com.sun.net.ssl.internal.www.protocol.https.HttpsURLConnectionOldImpl.getHeaderFieldKey(DashoA12275) at com.meterware.httpunit.HttpWebResponse.loadHeaders(HttpWebResponse.java:216) at com.meterware.httpunit.HttpWebResponse.readHeaders(HttpWebResponse.java:198) ... 31 more Caused by: sun.security.validator.ValidatorException: No trusted certificate found at sun.security.validator.SimpleValidator.buildTrustedChain(SimpleValidator.java:304) at sun.security.validator.SimpleValidator.engineValidate(SimpleValidator.java:107) at sun.security.validator.Validator.validate(Validator.java:202) at com.sun.net.ssl.internal.ssl.X509TrustManagerImpl.checkServerTrusted(DashoA12275) at com.sun.net.ssl.internal.ssl.JsseX509TrustManager.checkServerTrusted(DashoA12275) ... 44 more at net.sourceforge.jwebunit.HttpUnitDialog.(HttpUnitDialog.java:50) at net.sourceforge.jwebunit.WebTester.beginAt(WebTester.java:55) at net.sourceforge.jwebunit.WebTestCase.beginAt(WebTestCase.java:46) (... additional lines deleted ...)
Plugging in the standard trust all solution from above does nothing. A co-worker pointed me at a different solution to the problem. That gets around the SSL certificate check issue. However, problems still arise if you try to connect to 127.0.0.1 when your certificate is signed as localhost:
java.lang.RuntimeException: java.io.IOException at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:602) at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:272) at com.sun.net.ssl.internal.www.protocol.https.HttpsURLConnectionOldImpl.getResponseCode(DashoA12275) at com.meterware.httpunit.HttpWebResponse.readResponseHeader(HttpWebResponse.java:162) at com.meterware.httpunit.HttpWebResponse.readHeaders(HttpWebResponse.java:200) at com.meterware.httpunit.HttpWebResponse.(HttpWebResponse.java:56) at com.meterware.httpunit.HttpWebResponse.(HttpWebResponse.java:67) at com.meterware.httpunit.WebConversation.newResponse(WebConversation.java:76) at com.meterware.httpunit.WebWindow.getResource(WebWindow.java:164) at com.meterware.httpunit.WebWindow.getSubframeResponse(WebWindow.java:128) at com.meterware.httpunit.WebWindow.getResponse(WebWindow.java:121) at com.meterware.httpunit.WebWindow.updateWindow(WebWindow.java:144) at com.meterware.httpunit.WebWindow.getSubframeResponse(WebWindow.java:130) at com.meterware.httpunit.WebWindow.getResponse(WebWindow.java:121) at com.meterware.httpunit.WebClient.getResponse(WebClient.java:113) at net.sourceforge.jwebunit.HttpUnitDialog.(HttpUnitDialog.java:48) at net.sourceforge.jwebunit.WebTester.beginAt(WebTester.java:55) at net.sourceforge.jwebunit.WebTestCase.beginAt(WebTestCase.java:46) (... additional lines deleted ...) Caused by: java.io.IOException: HTTPS hostname wrong: should be <127.0.0.1> at sun.net.www.protocol.https.HttpsClient.b(DashoA12275) at sun.net.www.protocol.https.HttpsClient.afterConnect(DashoA12275) at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(DashoA12275) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:626) at sun.net.www.protocol.http.HttpURLConnection.getHeaderFieldKey(HttpURLConnection.java:1504) at com.sun.net.ssl.internal.www.protocol.https.HttpsURLConnectionOldImpl.getHeaderFieldKey(DashoA12275) at com.meterware.httpunit.HttpWebResponse.loadHeaders(HttpWebResponse.java:216) at com.meterware.httpunit.HttpWebResponse.readHeaders(HttpWebResponse.java:198) ... 31 more at net.sourceforge.jwebunit.HttpUnitDialog.(HttpUnitDialog.java:50) at net.sourceforge.jwebunit.WebTester.beginAt(WebTester.java:55) at net.sourceforge.jwebunit.WebTestCase.beginAt(WebTestCase.java:46) at net.ruckus.test.jwebunit.AbstractWebTestCase.login(AbstractWebTestCase.java:154) at net.ruckus.test.jwebunit.AbstractWebTestCase.login(AbstractWebTestCase.java:143) at net.ruckus.ui.web.SiteWebTest.testGet(SiteWebTest.java:20) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:324) at junit.framework.TestCase.runTest(TestCase.java:154) at junit.framework.TestCase.runBare(TestCase.java:127) at junit.framework.TestResult$1.protect(TestResult.java:106) at junit.framework.TestResult.runProtected(TestResult.java:124) at junit.framework.TestResult.run(TestResult.java:109) at junit.framework.TestCase.run(TestCase.java:118) at junit.framework.TestSuite.runTest(TestSuite.java:208) at junit.framework.TestSuite.run(TestSuite.java:203) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:478) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:344) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:196) (... additional lines deleted ...)
Because HttpUnit is forcing the old Sun protocols you need to use the Sun specific classes when creating a custom HostnameVerifier instead of the more generic javax.net.ssl classes. You'll have to live with the deprecated warnings:
public class DeprecatedNullHostnameVerifier implements com.sun.net.ssl.HostnameVerifier { private static boolean INITIALIZED = false; public boolean verify(String arg0, String arg1) { return true; } public static void install() { if (!INITIALIZED) { com.sun.net.ssl.HttpsURLConnection.setDefaultHostnameVerifier(new DeprecatedNullHostnameVerifier()); INITIALIZED = true; } } }
Combiing the XTrustProvider and the DeprecatedNullHostnameVerifier into a run-once JUnit setup method will let you easily test any HTTPS site.

Comments

Thanks for this info, I've stopped swearing at httpunit now. But HttpUnit does seem to be poorly implemented on a number of levels. Firstly there's the use of old SSL libraries as you've pointed out, but some other problems also crop up because of the Http client they're using which doesn't support arbitrary HTTP methods and doesn't support timeouts. I see you've also picked up on some JavaScript issues. Is HttpUnit really the best library available for performing these sorts of tests? Are there any alternatives?

Posted by: stephendv | October 22, 2005 3:15 AM

Hello! Could you please post here some an example how to work with HTTPS using httpunit? I can't find any tutorial about it, and cannot understand your example :( How do I use the DeprecatedNullHostnameVerifier? For example, if I execute the following code, I always get errors! Code: import com.meterware.httpunit.WebConversation; import com.meterware.httpunit.WebResponse; import com.meterware.httpunit.WebForm; import com.sun.net.ssl.HttpsURLConnection; import java.io.IOException; import org.xml.sax.SAXException; public class SSLClient { public static void main(String[] args) { try { try { HttpsURLConnection.setDefaultHostnameVerifier(new DeprecatedNullHostnameVerifier()); } catch (Exception e) { e.printStackTrace(); } WebConversation conversation = new WebConversation(); WebResponse response = conversation.getResponse("https://mysite/c"); WebForm login = response.getForms()[0]; login.setParameter("login", "admin"); login.setParameter("password", "123456"); response = login.submit(); System.out.println(response.getText()); } catch (IOException e) { e.printStackTrace(); } catch (SAXException e) { e.printStackTrace(); } } } The errors are as following: java.io.IOException at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:591) at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:272) at com.sun.net.ssl.internal.www.protocol.https.HttpsURLConnectionOldImpl.getResponseCode(DashoA6275) at com.meterware.httpunit.HttpWebResponse.readResponseHeader(HttpWebResponse.java:162) at com.meterware.httpunit.HttpWebResponse.readHeaders(HttpWebResponse.java:200) at com.meterware.httpunit.HttpWebResponse.(HttpWebResponse.java:56) at com.meterware.httpunit.HttpWebResponse.(HttpWebResponse.java:67) at com.meterware.httpunit.WebConversation.newResponse(WebConversation.java:76) at com.meterware.httpunit.WebWindow.getResource(WebWindow.java:164) at com.meterware.httpunit.WebWindow.getSubframeResponse(WebWindow.java:128) at com.meterware.httpunit.WebWindow.getResponse(WebWindow.java:121) at com.meterware.httpunit.WebWindow.getResponse(WebWindow.java:102) at com.meterware.httpunit.WebClient.getResponse(WebClient.java:87) at SSLClient.main(SSLClient.java:30) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:324) at com.intellij.rt.execution.application.AppMain.main(AppMain.java:78) Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: No trusted certificate found at com.sun.net.ssl.internal.ssl.BaseSSLSocketImpl.a(DashoA6275) at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA6275) at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA6275) at com.sun.net.ssl.internal.ssl.SunJSSE_az.a(DashoA6275) at com.sun.net.ssl.internal.ssl.SunJSSE_az.a(DashoA6275) at com.sun.net.ssl.internal.ssl.SunJSSE_ax.a(DashoA6275) at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA6275) at com.sun.net.ssl.internal.ssl.SSLSocketImpl.j(DashoA6275) at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(DashoA6275) at sun.net.www.protocol.https.HttpsClient.afterConnect(DashoA6275) at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(DashoA6275) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:615) at sun.net.www.protocol.http.HttpURLConnection.getHeaderFieldKey(HttpURLConnection.java:1485) at com.sun.net.ssl.internal.www.protocol.https.HttpsURLConnectionOldImpl.getHeaderFieldKey(DashoA6275) at com.meterware.httpunit.HttpWebResponse.loadHeaders(HttpWebResponse.java:216) at com.meterware.httpunit.HttpWebResponse.readHeaders(HttpWebResponse.java:198) ... 14 more Caused by: sun.security.validator.ValidatorException: No trusted certificate found at sun.security.validator.SimpleValidator.buildTrustedChain(SimpleValidator.java:304) at sun.security.validator.SimpleValidator.engineValidate(SimpleValidator.java:107) at sun.security.validator.Validator.validate(Validator.java:202) at com.sun.net.ssl.internal.ssl.X509TrustManagerImpl.checkServerTrusted(DashoA6275) at com.sun.net.ssl.internal.ssl.JsseX509TrustManager.checkServerTrusted(DashoA6275) ... 27 more Please help!

Posted by: J-webbian | November 9, 2005 9:43 AM

You are a star. This is **Exactly** what I was looking for and it works perfectly too. Well Done.

Posted by: Jude | March 12, 2008 10:22 PM

see this article also. http://www.testearly.com/2006/04/05/testing-with-untrusted-https/

Posted by: webguy1 | April 4, 2008 10:44 AM

Daniel R.

HTTPS testing with jWebUnit

Comments

About