JVM Garbage Collection and Optimizations

Overview When troubleshooting systems for performance-related issues, memory optimizations are a place that needs a deep analysis of what each system stores in the memory, how long those are stored, and access patterns. This post is to keep a note on the background information and valuable points to note in such an effort, specific to Java-based implementations as a deep understanding of the JVM behaviors is very beneficial in the process.

Java language provides much convenience to the developers by taking care of the memory management to a great extent letting the focus be on the rest of the logic. Still having a good understanding of how Java does this underneath, rationalize several best practices we follow in Java implementations and help design the programs better and think seriously on some aspects that can later lead to memory leaks and system stability in the long run. Java Garbage Collector has a big role in this been responsible for freeing up memory by removing memory gar…

Authorization for a Multi-Cloud System

This is a project design I am currently working on to consume SPIFFE(Secure Production Identity Framework For Everyone) bootstrapped trust and identification to provide authorization in a dynamically scaling, heterogeneous system, inspired by Mr. Prabath Siriwardena from WSO2 and under the supervision of Prof. Gihan Dias from University of Moratuwa. An enterprise system running across multiple clouds, as in the hybrid cloud, is an obvious example that will be benefitted from this. The objective is to open doors for the SPIFFE standard based systems to co-exist with rest of the systems with minimal effort, without compromising on security aspects while having an authorization solution based on SPIFFE.

What is SPIFFE?

In brief, it is a trust bootstrapping and identification framework, submitted as a standard and accepted by CNCF(Cloud Native Computing Foundation)[1]. As of now, this standard has two main implementations as SPIRE and Istio[2], a platform that supports service mesh architecture using SPIFFE for identification aspects. This implementation has taken care of a lot of complexities involved in trust bootstrapping and identification across heterogeneous systems. More details can be read at the spiffe.io site.

Why OAuth2.0?

OAuth 2.0 is currently the most widely used standard in the API security domain, that is used in access delegation and authorization in the workloads world as well. While SPIFFE is an emerging standard as of now, OAuth 2.0 has been there for a while, and we can say most of the enterprise system have adopted it. Hence if we can blend these two standards, we can best of both worlds and additional power with interoperability provided by OAuth 2.0 and dynamic trust bootstrapping and identification capabilities of SPIFFE.

How the Design Works?

Please note the SPIRE server in the below diagram can be any implementation that supports the SPIFFE standard.

- We assume an enterprise system that consists of workloads residing in two clouds, here we have assumed that is AWS and GCP. If we imagine this as a currently running system in GCP with workloads secured based on OAuth 2.0 scopes, the other workloads that are to consume these should come with valid access tokens and relevant scopes.
- The part of the system running in the AWS cloud can be imagined to be designed newly to run as part of a multi-cloud system. It makes use of SPIFFE standard to uniquely identify the workloads across multiple clouds.
- As part of this SPIFFE based trust bootstrapping and identification, each workload receives an X.509 certificate signed by the SPIRE server, bearing their identifier referred as the SPIFFE ID.
        eg.  spiffe://localdomain/us-west/data (This is included as a SAN) [3]
- Here comes OAuth 2.0 into the picture. We depend on the capability of the authorization server to issue an OAuth 2 access token under client credentials grant type. This will be under the MTLS OAuth2.0 specification that is currently in the draft stage[4].

There are few special things happening here,

  • MTLS connection is created based on the SPIRE server signed key pairs of the workload. Hence the authorization server and SPIRE server is assumed to have a pre-established trust.
  • As the workload creates the MTLS connection with the authorization server, it creates an OAuth 2 client dynamically on the fly, generates OAuth2 secrets and issues a token. At this point, the authorization server should do several validations before issuing these. 
  • The certificate needs to be validated first, then the content of it needs to be read along with the SPIFFE ID coming in the SAN.
  • Just looking at the SPIFFE ID and issuing a token will not suffice for the enterprise use case.
  • Hence we are to provide the capability of attaching scopes to these tokens based on a policy defined in authorization server using OPA. (OPA stands for Open Policy Agent, which is much flexible to provide RBAC, ABAC or XACML like complex policies as well.) This policy can consume additionally available data and make decisions.
  • After the validation is complete, the authorization server will issue a self-contained access token, including the scopes, expired time etc. that will be sent to the AWS workload, in order to be submitted when calling GCP workloads.
  • GCP workloads do not require any additional functions here, other than using its existing mechanism to validate the OAuth 2.0 token and derive any useful information that came with it.

Hope this explains the scenario well. I am to name this solution Dvaara, indicating opening more doors and controlled access. :)
We are open for any feedback, suggestions.

If you like to understand more on SPIFFE, [6] might help.

[1] - https://www.cncf.io/blog/2018/03/29/cncf-to-host-the-spiffe-project/
[2] - https://istio.io/docs/concepts/security/#istio-security-vs-spiffe
[3] - A sample SVID -
[4] - https://tools.ietf.org/html/draft-ietf-oauth-mtls-12
[5] - https://www.openpolicyagent.org/docs/comparison-to-other-systems.html
[6] - https://pushpalankajaya.blogspot.com/2019/01/spiffe-in-nutshell.html



  1. According to this approach, do the workload themselves handle OAuth2 token retrieval and validations or a node agent (side car) will take care of it ?. I think it would make more sense to delegate this to a sidecar/node agent since workloads themselves should not be aware of how authentication and authorization takes place in a complex service mesh.

    If you are to use sidecars, then how would scopes will be used / controlled ? A side car will not have a clear idea on what actions the workload is trying to do/invoke and what scopes should be available for the token while retrieving and validating.

    1. Thanks for the raised concern and valuable thoughts.
      Yes really make more sense to handle the OAuth specific details in a sidecar than injecting that logic to workload, keeping the seperation of duties.
      In the deisgn we will consider the case of sidecars, but not limited to. Even a legacy system that supports OAuth which might not necessarily be in the modern architecture and SPIFFE will be exist and co-operate in the system. In my opinion having a sidecar component that can understand OAuth protocol is an step forward.
      Ideal solution I am looking at is selecting the scopes attached to the OAuth token, based on a set of policies defined in the authorization server using OPA, which can be consumed by sidecar as well to let the resources be accessed based on the scopes attached to the token.
      Appreciate your thoughts...

  2. This comment has been removed by the author.

  3. This comment has been removed by a blog administrator.

  4. Thanks Pushpalanka Jayawardhana!!!
    Some thoughts:

    1) Will each replica have different certificates? I mean, replica1 will get a different certificate than replica2?
    2) Why AWS workloads really need to get dinamically a certificate from a SPIFFE server, instead of having its own on deployment time?

    1. Hi.. Thanks for your thoughts. Please find few more details on the same.
      (1) Yes, each replica will have a unique set of key pairs for each. However if we want to treat them as same or different can be governed by how we instruct the SPIRE server to issue SPIFFE IDs. More details at https://pushpalankajaya.blogspot.com/2019/01/spiffe-in-nutshell.html

      (2) The environment we are looking at here is more of a larger scale and dynamic in nature which needs to be well automated (as per the requirements of thousands of APIs serving in systems and with micro services architectures.). It will have less control over how these workloads are started and disappeared, rebooted, fail-overed etc.. Frequent key rotation is another aspect supported by this solution with no manual intervention.

  5. This comment has been removed by a blog administrator.

  6. This comment has been removed by a blog administrator.

  7. This comment has been removed by a blog administrator.

  8. This comment has been removed by a blog administrator.

  9. Hi there,
    Thank you so much for the post you do and also I like your post, Any company which picks up a multi-cloud strategy does not get into vendor lock-in. It even does not stick to finding a distinctive cloud infrastructure.
    Click here for MORE DETAILS......

  10. What is the main advantage of using SPIFFE compare to OAuth/OpenID?

    1. Emm.. I don't think the two protocols are trying to solve the samething. I am not sure if OAuth/OpenID can do the part SPIFFE is doing in boostrapping trust and assigning identities. Please enlighten me if there is some approach.


Post a Comment

Popular posts from this blog

Signing SOAP Messages - Generation of Enveloped XML Signatures

OPA for HTTP Authorization

How to Write a Custom User Store Manager - WSO2 Identity Server 4.5.0