Tomcat cipher problems
We encountered an odd problem with a customer, which took some time to get to the bottom of, so we wanted to create this article for any other customers that run into similar problems.
The symptoms of the problem that were encountered included reports of somewhat random failures occurring when fetching grammars that were hosted on an Apache Tomcat Server* over HTTPS. Problems were not encountered when using HTTP only.
The frequency of these failures seemed somewhat random, but was actually happening 1 time in every 256 requests. The user had set their max-age time to 0 (effectively disabling HTTP caching), which should not have affected anything, but was noteworthy nonetheless, since this may have aggravated the issue.
We found that when these grammar fetches were failing, the result was an empty grammar, which subsequently led to the occasional decode failure. The problem we saw was that there wasn't actually a failure in the fetching process, but in the openssl handshake that was taking place. This handshake is needed for any SSL transfer to be made. The nature of the failure turned out to be a bug in the Java code running on the Tomcat server, which only affected certain ciphers when they were used.
This handshake/negotiation is a normal part of any HTTPS exchange, where the client and server must agree on a set of ciphers to use when performing encryption. It turned out that certain ciphers offered by the Tomcat configuration were not working correctly, and thus failing whenever they were selected for use in an exchange.
These problematic ciphers, when used, caused the grammar fetch to fail. We were able to recreate this issue from a simple command line script that was repeatedly calling a curl fetch of the grammar URI
Once we found the reason behind the failures, we were able to dig deeper into the Tomcat configuration and eventually identify the problematic ciphers.
DEV-APP ~ $ rpm -qi java-1.7.0-openjdk
Name : java-1.7.0-openjdk Relocations: (not relocatable)
Version : 1.7.0.9 Vendor: Red Hat, Inc.
Release : 2.3.4.1.el6_3 Build Date: Tue 15 Jan 2013 01:31:33
It appears that this Java problem, described here: http://bugs.java.com/bugdatabase/view_bug.do?bug_id=8014618 may have been a known issue that was fixed in a later release (May 2013) than this installed version (Jan 2013). Updating to the newer version would likely have resolved the problem.
We were also able to identify that if Tomcat was reconfigured to only use the following ciphers, the problems were not encountered:
Good Ciphers
TLS_RSA_WITH_AES_128_CBC_SHA
SSL_RSA_WITH_3DES_EDE_CBC_SHA
Problematic Ciphers
Occasional Failures
TLS_DHE_RSA_WITH_AES_128_CBC_SHA
SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA
100% Failures
TLS_DHE_DSS_WITH_AES_128_CBC_SHA
SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA
Other Servers Having Similar Issues
It is possible that this negotiation handshake could affect other HTTPS servers, so we thought it useful to publish this article in case it proves useful to someone else in the future.
Again, this was an issue with the Web server, not the LumenVox code, however as a result of this, we modified the LumenVox code to help clarify the exact nature of the problem when logging verbosity is increased. In later versions of LumenVox, the following in an example of how this failure would appear in the log files:
Error -25 loading grammar uri: https://172.18.2.15:8844/dev_app/english/collect_ssn.voice.grxml, ErrorString:"Error: Could not open url https://172.18.2.15:8844/dev_app/english/transfer.voice.grxml; cUrl fetch error 35 on url https://172.18.2.15:8844/dev_app/english/transfer.voice.grxml; SSL connect error; error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure
To clarify - the problematic ciphers mentioned above were specific to this particular issue, related to this particular version of Tomcat / Java. We do not want to suggest that these ciphers are in any way 'bad', or should not be used (except perhaps in this one case)
The important takeaway from this article is that there is a possibility that this type of issue could occur on other servers hosting grammar files, so being aware of this possibility may be helpful, although we suspect that this problem is likely to only very rarely occur.
HTTPS Only Issue
Since this problem relates to an SSL handshake to negotiate the encryption algorithm to use for the transfer, this is unlikely to affect any HTTP fetches, or servers only providing HTTP hosting.
Prompt OS Security Patching
We should probably also mention that it is good practice to ensure your systems are maintained to keep up with the latest security patches to protect against new vulnerabilities as they are detected.
*Please note that LumenVox do not directly support Tomcat, nor can offer advise in debugging any tomcat related problems.