Netexpertise

SSL or TLS supported on a JVM can change depending on many things. Here are the many factors it depends on, and how to display which SSL versions are available and enabled on your JVM.

Factors Affecting SSL Support

SSL support depends first on your JDK version. TLS 1.0 and 1.1 are disabled on more and more JDK distributions by default, while TLS 1.2 is pretty standard. TLS 1.3 is supported on JDK 11 and later and JDK8 builds newer than 8u261.

But you can bypass default settings and disable a TLS algorithm in the Java security property file, just called java.security. SSL / TLS versions can be disabled with the jdk.tls.disabledAlgorithms setting.
There’s actually no way to enable explicitly a TLS version in Java: it has to be supported by the JDK distribution and not be in the disabled algorithm list.

You can always force the use of a TLS version, and cipher, in the java command parameters. Check last section.

Check SSL / TLS Versions Programmatically

Supported and enabled TLS versions can be displayed with a very simple piece of Java code. The getProtocols() method from the SSLContext class will help to display supported SSL versions on your JVM.

import java.util.*;
import javax.net.ssl.SSLContext;

public class tls
{

  public static void main (String[] args) throws Exception
  {
    SSLContext context = SSLContext.getInstance("TLS");
    context.init(null, null, null);
    String[] supportedProtocols = context.getDefaultSSLParameters().getProtocols();
    System.out.println(Arrays.toString(supportedProtocols));
  }

}

Execute the following commands to show the JVM version along enabled and supported SSL protocol versions

java -version
echo "Supported TLS:"
javac tls.java
java tls

Some TLS versions on some servers of mine:

Default OpenJDK
openjdk version "1.8.0_302"
OpenJDK Runtime Environment (build 1.8.0_302-b08)
OpenJDK 64-Bit Server VM (build 25.302-b08, mixed mode)
Supported TLS:
[TLSv1.2]

IBM OpenJ9
openjdk version "1.8.0_242"
OpenJDK Runtime Environment (build 1.8.0_242-b08)
Eclipse OpenJ9 VM (build openj9-0.18.1, JRE 1.8.0 Linux amd64-64-Bit
Compressed References 20200122_511 (JIT enabled, AOT enabled)
OpenJ9   - 51a5857d2
OMR      - 7a1b0239a
JCL      - 8cf8a30581 based on jdk8u242-b08)
Supported TLS:
[TLSv1, TLSv1.1, TLSv1.2]

And TLS versions on a mac:

You have now a list of TLS versions enabled on your JVM that you can fully trust.

Check SSL Versions and Ciphers with the Debug Mode

If you want to go further and see what is actually going on, use the Java debug feature. You will get details on disabled SSL protocols, available ciphers, SSL handshake and so on. It is very verbose, you have been warned! But so useful.
Simply launch your java code with java -Djavax.net.debug=all. A simple program that connects to a Mariadb database with JDBC and SSL enabled would be launched with something like:

java -Djavax.net.debug=all \
     -Djavax.net.ssl.trustStore=keystore.jks \
     -Djavax.net.ssl.trustStorePassword=changeit \
     -cp "mariadb-java-client-2.7.3.jar:." myProgram

This will shows a lot of stuff about SSL protocol and ciphers. Some of the lines below:

System property jdk.tls.server.cipherSuites is set to 'null'
Ignoring disabled cipher suite: TLS_DH_anon_WITH_AES_256_CBC_SHA
[...]
update handshake state: client_hello[1]
upcoming handshake states: server_hello[2]
*** ClientHello, TLSv1.2
Cipher Suites: [TLS_DHE_RSA_WITH_AES_128_GCM_SHA256]
[...]
%% Initialized:  [Session-1, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256]
[...]
0000: 01 00 00 01 0A 49 00 00   02 03 64 65 66 12 69 6E  .....I....def.in
0010: 66 6F 72 6D 61 74 69 6F   6E 5F 73 63 68 65 6D 61  formation_schema
0020: 06 54 41 42 4C 45 53 06   54 41 42 4C 45 53 09 54  .TABLES.TABLES.T
0030: 41 42 4C 45 5F 43 41 54   0C 54 41 42 4C 45 5F 53  ABLE_CAT.TABLE_S
0040: 43 48 45 4D 41 0C 2D 00   00 01 00 00 FD 01 00 00  CHEMA.-.........
0050: 00 00 21 00 00 03 03 64   65 66 00 00 00 0B 54 41  ..!....def....TA
0060: 42 4C 45 5F 53 43 48 45   4D 00 0C 3F 00 00 00 00  BLE_SCHEM..?....
[...]

jdk.tls.server.cipherSuites is set to ‘null’ because it was not overridden.
There’s usually a long list of disabled ciphers since they’re linked to disabled TLS protocols for most of them.
Then you see the client hello that shows the TLS version used for the handshake.
Cipher Suites normally displays a long list of available ciphers on the JVM. There’s just one here because I forced it.
Going further down, we can even see SQL queries sent to the Mariadb server during the connection initialisation.
Oracle provides a good documentation on how to debug SSL/TLS going through these messages.

Forcing TLS Versions and Ciphers

It is possible to force the JVM to use some TLS versions and ciphers on the command line. That’s very handy if you don’t have access to the JVM configuration, or if you’d like special settings for a particular Java program.
This can be done with jdk.tls.client.protocols and jdk.tls.client.cipherSuites settings.

java -Djavax.net.debug=all \
     -Djavax.net.ssl.trustStore=keystore.jks \
     -Djavax.net.ssl.trustStorePassword=changeit \
     -Djdk.tls.client.protocols=TLSv1.2 \
     -Djdk.tls.server.cipherSuites="TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA" \
     -cp "mariadb-java-client-2.7.3.jar:." myProgram

TLS protocols and ciphers can also be specified on the JDBC connection string if you use encryption for the database connection.

String url = "jdbc:mariadb://my_database_server:3306/my_database?"+
             "useSSL=true"+
             "&serverTimezone=UTC"+
             "&enabledSslProtocolSuites=TLSv1.2"+
             "&enabledSSLCipherSuites=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA";

 

Tags: Encryption, SSL

Install cert-manager

Cert-manager comes as a Helm chart with its own custom resources you can install on your Kubernetes cluster. It helps certificates automation, renewal and management. It is a MUST have when you deal with certificate providers who offer APIs that let you automate these processes. On the side, you’d better renew LetsEncrypt certificate automatically since they are valid for a 3 month period.

cert-manager is available on the Jetstack Helm repository, add it to your Helm repository list

helm repo add jetstack https://charts.jetstack.io
helm repo update

Cert-manager runs in its own namespace, so first create it, and install cert-manager helm chart

kubectl create namespace cert-manager
helm install cert-manager \
     --namespace cert-manager jetstack/cert-manager \
     --set installCRDs=true

–set installCRDs=true tells cert-manager to install custom resources such as certificaterequests, certificates or clusterissuers.

LetsEncrypt Cluster issuer

A cluster issuer will contain information about a certificate provider. If you want to get your SSL certificates signed by LetsEncrypt, you will need to apply this yaml file to the Kubernetes cluster:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    email: it@company.xxx
    privateKeySecretRef:
      name: letsencrypt-prod
    server: https://acme-v02.api.letsencrypt.org/directory
    solvers:
    - http01:
        ingress:
          class: public-iks-k8s-nginx

LetsEncrypt belongs to the ACME issuers category, meaning it is trusted by most web browsers. It provides a certificate after checking you are the owner of the domain. The check can be done in 2 ways: either a DNS TXT entry or an HTTP challenge. Kubernetes serves HTTP so most people will go for the HTTP01 challenge. This is defined in the solvers section.

The second important piece of information is the class. cert-manager will look at ingresses whose class matches and will provide them with an SSL certificate. IBM Cloud public ingress class annotation is called public-iks-k8s-nginx, so you need to set it in your cluster issuer configuration. Check your ingress to adapt to your own needs.

Ingress Definition

Now that you have a cluster issuer and cert-manager installed, you need to tell them which ingress they should provide certificates to. This is done with ingress annotations.
Simply set the cluster issuer in the cert-manager.io/cluster-issuer annotation.
As seen before, the kubernetes.io/ingress.class annotation is set to public-iks-k8s-nginx on IKS. Set whatever suits your setup.
Add acme.cert-manager.io/http01-edit-in-place wether you want to create a separate ingress for the HTTP challenge or want it to be part of the existing ingress.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app-ingress
  labels:
    name: app-ingress
  annotations:
    acme.cert-manager.io/http01-edit-in-place: "true"
    cert-manager.io/cluster-issuer: letsencrypt-prod
    kubernetes.io/ingress.class: public-iks-k8s-nginx
spec:
  tls:
  - hosts:
    - www.netexpertise.eu
    secretName: letsencrypt-netexpertise

  rules:
  - host: www.netexpertise.eu
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: app-backend-app
            port:
              number: 80

Renew LetsEncrypt Certificate

Cert-manager will create ingress, service and pod in your own namespace that will provide a web page for the HTTP challenge. They will disappear as soon as LetsEncrypt certificate has been renewed and delivered into the secret defined in secretName.

If something goes wrong, you can check the logs of the different pods in the cert-manager namespace, as well as the certificate resource status. A kubectl describe cert should give all necessary information.

 

Tags: Certificate, IBM, IBMCloud, Kubernetes, Nginx, SSL

I wrote in a previous post about a ProxySQL Helm chart for Kubernetes, in which you can configure the pod from SQL queries. The chart includes an ingress to reach the stats server on port 6080. Here are a few steps to get this working.

Ingress needs to be activated in the Helm Values file and ProxySQL web server also requires the admin-web_enabled admin variable to be set.

ingress:
  enabled: true
  annotations:
    kubernetes.io/ingress.class: nginx
  hosts:
    - host: proxy.domain.lan
      paths:
        - /

sqlqueries: |
  SET admin-web_enabled='true';
  LOAD ADMIN VARIABLES TO RUNTIME;
  -- other SQL queries

If you leave it like this, HTTP connections to the web interface will throw the following error:
Error: received handshake message out of context
ProxySQL web interface accepts HTTPS connections only – even though examples in the documentation use HTTP – and there’s no workaround it.

Ingress has to connect to the backend using the HTTPS protocol. Nginx lets you specify the protocol used with the backend-protocol annotation. You still have the choice to leave ingress access from outside in HTTP, but don’t on a public interface.

nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"

 

Tags: Docker, Kubernetes, Mysql, ProxySQL

ProxySQL is a powerful tool that relays traffic to multiple Mysql backends. Configuration can be set in proxysql.cnf that is loaded when the daemon starts. This is really nice from a Docker perspective. You change the file and redeploy with the new settings. Now, I need a ProxySQL Helm chart to deploy on my Kubernetes cluster.

Official ProxySQL Helm Chart

I tried to use the ProxySQL team helm chart but quickly ran into some issues.
The docker hub image is a bit outdated. The Helm chart does not seem to be maintained anymore while they had done a great job.

The configuration lies in proxysql.cnf. It’d be nice the file content would load from Values.yaml, making it dynamic for each deployment.
My setup uses mainly mysql_query_rules_fast_routing but here’s the problem: it cannot be configured through proxysql.cnf and ProxySQL developers made it clear they will not change this.

Docker Image with Mysql Client

ProxySQL docker comes without Mysql client probably to make the image size smaller. You will need it though to inject SQL queries locally when the pod starts up. The dockerfile is really easy but it forces you to maintain your own image.

FROM proxysql/proxysql:2.2.0

RUN apt update && apt -y install mysql-client && apt clean all

ENTRYPOINT ["proxysql", "-f", "-D", "/var/lib/proxysql"]

I made an image directly available on Dockerhub. It is referenced in the helm chart but you can build your own if you want to host it on a private repository.

mysql_query_rules_fast_routing Rules

I decided to make some changes on the Helm template so it can read SQL rules from the Value file, and load them into ProxySQL dynamically once the pod is up and running.

sqlqueries: |
   --
   SET mysql-have_ssl='true';
   LOAD MYSQL VARIABLES TO RUNTIME;
   SET admin-web_enabled='true';
   LOAD ADMIN VARIABLES TO RUNTIME;
   --
   INSERT INTO mysql_servers (hostgroup_id,hostname) VALUES (0,'db1.domain.lan');
   INSERT INTO mysql_servers (hostgroup_id,hostname) VALUES (1,'db2.domain.lan');
   --
   INSERT INTO mysql_users (username,password,use_ssl,default_hostgroup) VALUES ('proxy','*9EF51D21B4A3E7BC7A58925308F229CF4AEEC9E1',1,0);
   --
   INSERT INTO mysql_query_rules_fast_routing (username,schemaname,destination_hostgroup, comment) VALUES ('proxy','schema1',0,'');
   INSERT INTO mysql_query_rules_fast_routing (username,schemaname,destination_hostgroup, comment) VALUES ('proxy','schema2',1,'');
   --
   LOAD MYSQL USERS TO RUNTIME;
   LOAD MYSQL SERVERS TO RUNTIME;
   LOAD MYSQL QUERY RULES TO RUNTIME;

SQL queries are added to the Values.yaml file, loaded into the configmap and mounted as a file in /docker-entrypoint-initdb.d/sqlqueries. We can then inject SQL queries once ProxySQL daemon is up and running. It is achieved with a poststart command that is launched right after the docker entrypoint’s been executed.

lifecycle:
   postStart:
     exec:
       command: ["/bin/sh", "-c", "sleep 1 && mysql --show-warnings -uadmin -padmin -h127.0.0.1 -P{{ .Values.pod.adminPort }} < /docker-entrypoint-initdb.d/sqlqueries"]

Using the Helm Chart

The helm chart is available on github. Clone the repo, edit your own Values file based on example.yaml and deploy the Helm chart

helm install -n my-release -f myconf.yaml ./proxysql

Read this post if you want to enable and reach the stats server through Kubernetes ingress rules.

Bring your Own Self Signed Cert

Proxysql 2.3.0 is now capable of reloading the SSL/TLS certificate on the fly.
Before that, you’d have to build your own proxysql container to embed the CA and Mysql certificate, making it static.
With this new feature, you just have to create a Kubernetes TLS secret with a ca.crt, tls.cert and key.crt that will be mounted by the ProxySQL Helm chart.

Uncomment the matching set of commands that will replace ProxySQL auto-generated certs with the Mysql SYSTEM instruction.

 

Tags: Docker, Kubernetes, Mysql, ProxySQL

I asked my reseller a quote for some Cisco Catalyst switches as usual, and he tells me most of his clients buy Cisco Small Business switches. He argues they are much cheaper, have similar performance and features. Checking on some forums, I get no real answers but vague statements like it is not a real IOS or they are entry level switches. I decided to get a few and install them on remote sites with less traffic for a long term test.

Cisco Small Business Look Great …

Prices are around 4 times cheaper than Cisco Catalysts’ and they are under a lifetime warranty (until the last date of support). That means 3 spares for the same price…

I won’t talk about performance, they’re all pretty much the same.

Before I validate the order, I also made sure of a few essentials:
I need a CLI, it’s handy to connect over slow unstable links, or copy and paste some configuration lines. As mentioned earlier, it is not IOS since it is a different product.
It also provides HTTP(s), telnet and SSH access. Note that unlike Cisco Catalyst, Cisco Small business switches accept only one public SSH key.

Common features like VLANs (up to 256, check the datasheet), port mirroring, SNMP protocol are also available.

… Even though They Don’t Offer as much as Cisco Catalyst

SSH key authentication was not available then but it is now. It seems you can configure only one though.

When switching a port from access to trunk mode, you cannot set the trunk settings before changing to trunk mode. This is particularly annoying if you are configuring a remote switch over that link.

I noticed there is a lot less debug commands and levels but that was expected.

I ran into a bug and only one over 2 years. Interfaces were reporting a lot of packets in error.
This was fixed through a firmware upgrade. I noticed there was very few firmware releases on the download page.
Should I be worried about it? Not necessarily.
Cisco Small Business have a smaller amount of features, meaning no bug on features they don’t implement.
It could also mean less reactivity to fix existing bugs, but you get what you’re paying for, don’t you?

Then What?

Cisco Small Business are a really good, cheap alternative to Cisco Catalyst for access network, and proved to be reliable over time and better than a lot of other switches out there on the market. All depends on the features you need, like advanced QoS functionalities, or if you want to stack them for example. I ended up with a mix keeping Catalyst on the main sites and Cisco Small Business for smaller remote sites equipped with PoE wifi access points. The price is really attractive especially when you need Power over Ethernet (PoE) functionality.

A solution in between is also possible: Cisco Catalyst with the LAN Lite image. They cost halt the price of Cisco Catalyst LAN Base but have restrictions on QoS, ACL, or the number of VLANs among others. It is the same hardware with IOS but note you cannot upgrade from LAN Lite to LAN Base. Such a pity

 

Tags: Cisco, Cisco Small Business