Saturday, May 25, 2019

Apache Tomcat 9 Translation

1. Translation Processing Time

By those who haven't learned and used English a lot, messages can be understood easier and faster if those are in their own language. Apache Tomcat project has the following message in English as an example (ref:

applicationContext.addListener.iae.sclNotAllowed=Once the first ServletContextListener has been called, no more ServletContextListeners may be added.

The message means, as you can figure out in the translation process in your brain in the end, that if any ServletContextListener has ever been invoked then you cannot add a new ServletContextListener any more. This can be translated into Korean like this (ref:

applicationContext.addListener.iae.sclNotAllowed=첫번째 ServletContextListener가 호출되고 나면, 더 이상 ServletContextListener들을 추가할 수 없습니다.

Basically English messages require me to take translation processing time in my brain as the cognitive system of mine has been developed enough already with my mother tongue, Korean. It probably takes less  time for some people, but It takes more time for me. I have to spend more time to get the meaning than native or native-like English speakers do. And it does not happen just once or twice, but it keeps occurring again and again, accumulating it into stack of gaps of several hours, days or months. Message translations in software projects may help avoid or reduce the gaps.
    Of course, the translation should be correct. Long time ago, the books published by the "ㅅ" publisher company in South Korea, were very difficult to understand even if those were translated into Korean. Perhaps someone had enough English skills but had never experienced in software development or had never asked proficient engineers to review the translations before publication. I don't think it happened only in IT field. Whether they were about Economics or Statistics, some books (translated) in Korean were harder to understand. Somethings were out of context, with terminologies that were never used in real practices, with weird combinations of Chinese characters to make up new words, or with unnatural passive voices from  too strict literal translations. So, some people used to try to read the original books in English instead, or some others had to rush in head first, including myself.
    One thing clear to me is that once those are translated into correct words, it saves a lot of time in translation process that many people have to spend in otherwise. The more popular software, the more values of correct translation to people.

2. Apache Tomcat Translation with Korean examples

Since Apache Tomcat 9.0.15, almost every English message has been translated into Korean. If you set the default language of the JVM to Korean (`CATALINA_OPTS=" -Duser.language=ko") like the following example, you can see all the internal information, warning or error messages in Korean. I ran Apache Tomcat simply with `bin/ run` below.

$ export CATALINA_OPTS=" -Duser.language=ko"

$ bin/ run

Using CATALINA_BASE:   /Users/tester/tomcat
Using CATALINA_HOME:   /Users/tester/tomcat
Using CATALINA_TMPDIR: /Users/tester/tomcat/temp
24-Apr-2019 23:51:08.477 정보 [main] org.apache.catalina.startup.VersionLoggerListener.log 서버 버전 이름:        Apache Tomcat/9.0.18-dev24-Apr-2019 23:51:08.481 정보 [main] org.apache.catalina.startup.VersionLoggerListener.log Server 빌드 시각:          Apr 20 2019 19:48:52 UTC
24-Apr-2019 23:51:08.481 정보 [main] org.apache.catalina.startup.VersionLoggerListener.log Server 버전 번호:
24-Apr-2019 23:51:08.481 정보 [main] org.apache.catalina.startup.VersionLoggerListener.log 운영체제 이름:               Mac OS X
24-Apr-2019 23:51:08.481 정보 [main] org.apache.catalina.startup.VersionLoggerListener.log 운영체제 버전:            10.14.4
24-Apr-2019 23:51:08.481 정보 [main] org.apache.catalina.startup.VersionLoggerListener.log 아키텍처:          x86_64
24-Apr-2019 23:51:08.481 정보 [main] org.apache.catalina.startup.VersionLoggerListener.log 자바 홈:             /Library/Java/JavaVirtualMachines/jdk1.8.0_144.jdk/Contents/Home/jre
24-Apr-2019 23:51:08.481 정보 [main] org.apache.catalina.startup.VersionLoggerListener.log JVM 버전:           1.8.0_144-b01
24-Apr-2019 23:51:08.481 정보 [main] org.apache.catalina.startup.VersionLoggerListener.log JVM 벤더:            Oracle Corporation
24-Apr-2019 23:51:08.481 정보 [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_BASE:         /Users/tester/tomcat
24-Apr-2019 23:51:08.481 정보 [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_HOME:         /Users/tester/tomcat
24-Apr-2019 23:51:08.488 정보 [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent 프로덕션 환경들에서 최적의 성능을 제공하는, APR 기반 Apache Tomcat Native 라이브러리가, 다음 java.library.path에서 발견되지 않습니다: [/Users/tester/Library/Java/Extensions:/Library/Java/Extensions:/Network/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java:.]24-Apr-2019 23:51:08.749 정보 [main] org.apache.coyote.AbstractProtocol.init 프로토콜 핸들러 ["http-nio-8080"]을(를) 초기화합니다.
24-Apr-2019 23:51:08.775 정보 [main] org.apache.coyote.AbstractProtocol.init 프로토콜 핸들러 ["ajp-nio-8009"]을(를) 초기화합니다.
24-Apr-2019 23:51:08.778 정보 [main] org.apache.catalina.startup.Catalina.load [526] 밀리초 내에 서버가 초기화되었습니다.
24-Apr-2019 23:51:08.806 정보 [main] org.apache.catalina.core.StandardService.startInternal 서비스 [Catalina]을(를) 시작합니다.
24-Apr-2019 23:51:08.807 정보 [main] org.apache.catalina.core.StandardEngine.startInternal 서버 엔진을 시작합니다: [Apache Tomcat/9.0.18-dev]
24-Apr-2019 23:51:08.814 정보 [main] org.apache.catalina.startup.HostConfig.deployDirectory 웹 애플리케이션 디렉토리 [/Users/tester/tomcat/webapps/docs]을(를) 배치합니다.
24-Apr-2019 23:51:09.052 정보 [main] org.apache.catalina.startup.HostConfig.deployDirectory 웹 애플리케이션 디렉토리 [/Users/tester/tomcat/webapps/docs]에 대한 배치가 [237] 밀리초에 완료되었습니다.
24-Apr-2019 23:51:09.055 정보 [main] org.apache.catalina.startup.HostConfig.deployDirectory 웹 애플리케이션 디렉토리 [/Users/tester/tomcat/webapps/manager]을(를) 배치합니다.
24-Apr-2019 23:51:09.117 정보 [main] org.apache.catalina.startup.HostConfig.deployDirectory 웹 애플리케이션 디렉토리 [/Users/tester/tomcat/webapps/manager]에 대한 배치가 [62] 밀리초에 완료되었습니다.
24-Apr-2019 23:51:09.117 정보 [main] org.apache.catalina.startup.HostConfig.deployDirectory 웹 애플리케이션 디렉토리 [/Users/tester/tomcat/webapps/examples]을(를) 배치합니다.
24-Apr-2019 23:51:09.911 정보 [main] org.apache.catalina.startup.HostConfig.deployDirectory 웹 애플리케이션 디렉토리 [/Users/tester/tomcat/webapps/examples]에 대한 배치가 [793] 밀리초에 완료되었습니다.
24-Apr-2019 23:51:09.911 정보 [main] org.apache.catalina.startup.HostConfig.deployDirectory 웹 애플리케이션 디렉토리 [/Users/tester/tomcat/webapps/ROOT]을(를) 배치합니다.
24-Apr-2019 23:51:09.969 정보 [main] org.apache.catalina.startup.HostConfig.deployDirectory 웹 애플리케이션 디렉토리 [/Users/tester/tomcat/webapps/ROOT]에 대한 배치가 [57] 밀리초에 완료되었습니다.
24-Apr-2019 23:51:09.969 정보 [main] org.apache.catalina.startup.HostConfig.deployDirectory 웹 애플리케이션 디렉토리 [/Users/tester/tomcat/webapps/host-manager]을(를) 배치합니다.
24-Apr-2019 23:51:10.019 정보 [main] org.apache.catalina.startup.HostConfig.deployDirectory 웹 애플리케이션 디렉토리 [/Users/tester/tomcat/webapps/host-manager]에 대한 배치가 [50] 밀리초에 완료되었습니다.
24-Apr-2019 23:51:10.022 정보 [main] org.apache.coyote.AbstractProtocol.start 프로토콜 핸들러 ["http-nio-8080"]을(를) 시작합니다.
24-Apr-2019 23:51:10.029 정보 [main] org.apache.coyote.AbstractProtocol.start 프로토콜 핸들러 ["ajp-nio-8009"]을(를) 시작합니다.
24-Apr-2019 23:51:10.031 정보 [main] org.apache.catalina.startup.Catalina.start 서버가 [1,252] 밀리초 내에 시작되었습니다.

Almost every message is now served in Korean: "... 밀리초 내에 서버가 초기화되었습니다" (meaning "the server was initialized in ... ms"), "웹 애플리케이션 디렉토리" (meaning "Web Application Directory"), "배치가 ... 완료되었습니다" (meaning "Deployment ... completed"), etc.
    A screenshot below was taken on the servlet example page for the HelloWorldExample in the default example web application. You can visit http://localhost:8080/, click on "Examples" menu on the top, click on the "Servlet Examples" link and finally click on the "Hello World" example link.

The HelloWorld servlet example (/examples/servlets/servlet/HelloWorldExample)

The Request Info servlet example ("RequestInfoExample") is served in Korean, too:

The RequestInfo servlet example (/examples/servlets/servlet/RequestInfoExample)

When stopping the Apache Tomcat by entering Control-C in the command line console, messages about the stopping process are served in Korean, too:

25-Apr-2019 00:08:47.580 정보 [Thread-5] org.apache.coyote.AbstractProtocol.pause 프로토콜 핸들러 ["http-nio-8080"]을(를) 일시 정지 중
25-Apr-2019 00:08:47.589 정보 [Thread-5] org.apache.coyote.AbstractProtocol.pause 프로토콜 핸들러 ["ajp-nio-8009"]을(를) 일시 정지 중
25-Apr-2019 00:08:47.595 정보 [Thread-5] org.apache.catalina.core.StandardService.stopInternal 서비스 [Catalina]을(를) 중지시킵니다.
25-Apr-2019 00:08:47.615 정보 [Thread-5] org.apache.coyote.AbstractProtocol.stop 프로토콜 핸들러 ["http-nio-8080"]을(를) 중지시킵니다.
25-Apr-2019 00:08:47.618 정보 [Thread-5] org.apache.coyote.AbstractProtocol.stop 프로토콜 핸들러 ["ajp-nio-8009"]을(를) 중지시킵니다.
25-Apr-2019 00:08:47.619 정보 [Thread-5] org.apache.coyote.AbstractProtocol.destroy 프로토콜 핸들러 ["http-nio-8080"]을(를) 소멸시킵니다.
25-Apr-2019 00:08:47.620 정보 [Thread-5] org.apache.coyote.AbstractProtocol.destroy 프로토콜 핸들러 ["ajp-nio-8009"]을(를) 소멸시킵니다.

3. How Was It Started?

As you may know, The Apache Software Foundataion ( has helped and nurtured great open source software projects and communities based on voluntary contributions. People get involved in the community through mailing lists of the project in which they found their interests. They ask questions or try to give answers to help other people; those who are interested in testing, development or documentation also discuss how to improve the software and process in the mailing lists and report bugs through the bug tracking systems. The community invite people as committers if someone has made quite amount of contributions in various forms such as bug reporting, helping others through mailing lists, providing patches, helping documentation, and so on. The committers makes changes in the source. Furthermore, committers who has shared with the vision of the community may become members of the Project Management Committee (PMC) and participate in decision making process for the project on behalf of the community. This governance model is known as The Apache Way. See for more detail.
    Anyway, the Apache Tomcat Translation initiative was started based on the community culture with voluntary contribution from individuals. On Nov. 12, 2018, Mark Thomas, the long time Apache Tomcat committer and PMC member, contributing a lot to the Apache Software Foundation too, posted the following message in the user mailing list (ref: Long story short, to solve the fundamental problem that many people have met such as it being very hard to find which resource files to patch unless you're an expert of Apache Tomcat project, the PMC of Apache Tomcat initiated a POEditor project (see the screenshot below) to encourage more people to participate in the collective translation contributions, hoping to ship the contributed resources in Apache Tomcat 9 releases.

From: Mark Thomas
Subject: Translation help wanted
Date: 2018/11/12 11:49:51


Apache Tomcat includes some translations for error messages and parts of
the user interface - primarily the Manager web application. We would
like to improve the coverage and quality of these translations.
Accordingly, the Tomcat project has been set up on POEditor, a web-based
service for managing the translation of resource files.

The aim is that anyone who wants to contribute to the translations (it
could be anything from fixing a typo in an existing translation to
adding support for a new language) can create an account and contribute.

If you would like to contribute in this way then the
The Tomcat project can be found here:

Anyone should be able to join up as a contributor. If you are
interested, please sign up and start contributing.

Note: All contributions will be taken as being made under the terms of
the Apache License version 2.

I'm aiming to export the translations on a regular basis to the Tomcat
source code. How regularly will depend on the rate of new/updated
translations but as a minimum, I'm aiming to get any updates into the
next Tomcat 9 release.

If you have any difficulties or questions, please ask here.



To unsubscribe, e-mail:
For additional commands, e-mail:

If you have a look at the message thread, unbelievably many people reacted with positive voluntary willingness. And it didn't take many days to prove to us how great the outcome could be achieved by the community. Here is again Mark Thomas's message just after 9 days. (ref:

From: Mark Thomas
To: Tomcat Users List
Subject: Translations update
Date: 2018/11/21 09:58:15

Hi all,

I wanted to let you know about the amazing progress that is being made
on the Tomcat translations at

In the short time since this effort has started the community has
achieved the following:

- French has increased from 18% to 64% coverage
- Simplified Chinese has been added and has already reached 32% coverage
- Korean has been added and has reached 10% coverage
- German has increased from 2% to 7% coverage
- Brazilian Portuguese has been added and has reached 4% coverage
- Spanish has increased from 42% to 44% coverage

as well as a smaller number of additions and corrections to another 6

A big thank you to everyone who has contributed.

There is still lots to do so if you would like to help out please join
us at:



To unsubscribe, e-mail:
For additional commands, e-mail:

In less than 10 days, the portion of translated messages increased from 18% to 64% for French, 42% to 44% for Spanish, 2% to 7% for German. Even better, new languages were added, which had never been in the project before: 32% for Chinese; 10% for Korean; 4% for Brazillian Portugese.
    As of today, April 28, 2019, while I'm writing this article, more than 99% of messages were translated into Korean, and more than 140 volunteers made more than 3,044 contributions in 17 different languages! And the collective work continues. See the POEditor project homepage for detail:

4. And It Continues

Have you ever seen weird translations in software messages in your IT career? Even if the English messages were translated into your language, haven't you ever seen that some messages give an awkward feeling to you sometimes?
    Sharing the concerns, the Apache Tomcat community suggests that we should try to fix this problem together. The suggestion comes not just as an abstract principle, but very concrete and practical solution: the POEditor project ( It's never difficult; it's really easy to edit. If you don't understand why the message is used there on which context, you can also ask questions through commenting in the POEditor project. You can share ideas together. Committers may give answers to your questions, or you can discourse with other translators, too.

If you want to join in the common experience helping each other in the community, feel free to join the Apache Tomcat POEditor project ( Choose the language you want to translate into.
    Also, to join the users' or developers' mailing lists to ask questions or discuss on anything, see

In Today's World where everything is connected to each other digitally, people, located far from each other geographically with time and language differences, start contributing to the open source projects in which they find their interests. It is like people having been collaborating to build shared reservoirs and planting trees on the dams to protect them in commons for thousands of years. People, who have already experienced, now try to figure out how to encourage other people to participate in more easily. Such easy tools as POEditor help people get involved in easier and better. They know that it becomes easier together and they can achieve more benefits in the community together.

Wednesday, November 14, 2018

Apache Jackrabbit Database Usage Patterns and Options to Reduce Database size

Recently, I wrote about how to externalize version storage to an SFTP server backend to reduce database size: It is kind of similar case to how to keep the binary content in either AWS S3 bucket or virtual file system such as SFTP or WebDAV server as I described before in The only difference is, in high level, the former is about version history database table, VERSION_BUNDLE, whereas the latter is about the binary table, DATASTORE.

I'd like to explain how those tables make a significant impact on database size by showing database usage patterns from several real CMS systems. Also, I'd like to list the benefits by reducing the database size at last.

Pattern 1: Huge DATASTORE table for a Simple Website

In the chart, it shows more than 95% of database is consumed by DATASTORE table which stores only binary content such as images, PDF files, etc, not document or configuration nodes and properties. The project implements a CMS based website serving huge amount of binaries. But business users do not probably edit and publish documents often. It is also possible that they migrate some binary data such as images and PDF files from external sources to CMS in order to serve those through website easily.

If they switch the Apache Jackrabbit DataStore component from the default DbDataStore to either S3DataStore or VFSDataStore, they can save more than 95% of database.

Pattern 2: Big DATASTORE table with Modest Document/Node Updates

This site shows modest amount of document and node content in DEFAULT_BUNDLE table which contains the node bundle data of the default Jackrabbit workspace. It means that business users update and publish content in modest size. But still more than 90% of database is consumed for binary content only in DATASTORE table.

The same story goes. If they switch the Apache Jackrabbit DataStore component from the default DbDataStore to either S3DataStore or VFSDataStore, they can save more than 90% of database.

Pattern 3: More Document Oriented CMS

In this site, the DEFAULT_BUNDLE table is relatively bigger than other sites, taking more than 50% of database. It means that content document updates and publication is very important to business users with their CMS system. Business users probably need to update and (re)publish content more frequently for their websites.

As the default workspace data needs to be queried and accessed frequently in the delivery web applications, there's nothing to do more with the DEFAULT_BUNDLE table.
However, they still have consumed more than 20% of database only for binary content in DATASTORE table, and they have consumed up to 20% of database for version history in VERSION_BUNDLE table.
Therefore, if they switch both DataStore component and FileSystem component of VersionManager to alternatives -- S3DataStore / VFSDataStore and VFSFileSystem -- then they can save more than 40% of database.

Pattern 4: More Versioning or Periodic Content Ingestion to CMS

In this site, more than 55% of database is consumed for version history in VERSION_BUNDLE table, and up to 30% of database is consumed for binary content in DATASTORE table.
There are two possibilities: (a) business users update and publish document very often so that it results in a lot of version history data, or (b) there is a batch job periodically running to import external content into CMS with publishing the updated document after imports.
In either case, if they switch both DataStore component and FileSystem component of VersionManager to alternatives -- S3DataStore / VFSDataStore and VFSFileSystem -- then they can save more than 85% of database.

Benefits by Reducing Database Size

What are the benefits by reducing the repository database size by the way?
Here's my list:
  1. Transparent JCR API
    • As you're switching only Apache Jackrabbit internal components, it doesn't affect applications. You don't need to write or use a plugin to manage binary content in a different storage by yourself. The existing JCR API still works transparently.
    • Indexing still works transparently. If you upload a PDF file, it will be indexed and searchable. However, if you implement a custom solution, you need to take care of it by yourself.
  2. Almost unlimited storage for binaries
    • If you use either S3 bucket or SFTP gateway for Google Cloud Platform or even SFTP server directly, then you can store practically almost unlimited amount of binaries and version history in modern cloud computing world.
  3. Cheaper storage
    • Amazon S3 or SFTP server is a lot cheaper than database option. For example, Amazon RDS is more expensive than S3 storage for binary content.
  4. Faster backup, import, migration
    • Apache Jackrabbit DataStore component allows you to do hot-backup and restoration from the backup files to the backend system at runtime.
  5. Build new environment quickly from production data.
    • As the database is small enough in most cases, you can build a new environment from from other environment's backups more quickly.
  6. Save backup storage
    • If you do nightly backup, weekly backup, etc. and you have to keep those backup files for some period (e.g, 1 year), then you might need to worry about the backup disk storage sometimes. If the database size is small enough, your concerns will be more relieved by taking advantage of S3 backup capabilities.
  7. Encryption at rest
    • If you have sensitive PDF files for example, you might want to take advantage of Encryption at REST provided by Amazon S3 or Linux file system.

Externalizing JCR Version Storage with VFSFileSystem

A while ago, I wrote a blog article, Can't we store huge amount of binary data in JCR?. It was about switching Apache Jackrabbit DataStore from DbDataStore to either S3DataStore or VFSDataStore. Depending on your database usage pattern, it will allow you to save huge amount of database just by switching DataStore component configuration in the repository.xml.

In some cases, the version history data in VERSION_BUNDLE could be as big as DATASTORE table. The following is an excerpt from, explaining what's happening when you (de)publish a document, causing revisions in version history:
Each time a document is published, a copy of the current state of the document is stored as a new version. While this feature enables users to restore any previously published version of their document, it comes at the cost of an ever increasing size of the version history storage.
So if your users update and publish documents regularly, the version history data size will increase proportionally as time goes by, which might cause a big database size at some point. Administrators need to monitor it and they might need to remove old revisions just to reduce the database size.

The same story goes here as we have dealt with binary storage issue in database in my previous blog article. Is there a solution for this? Do we really need to care about database size increases for the version history?

Yes, we have a solution in Apache Jackrabbit: VFSFileSystem.

JackrabbitRepository component uses two distinct internal components: Workspace and VersionManager. (I'm using logical names instead of physical class names such as org.apache.jackrabbit.core.RepositoryImpl.WorkspaceInfo here.) See the diagram below:

Whenever a version needs to be made, the node data is copied to VersionManager, which saves the data in its own FileSystem -- DatabaseFileSystem by default if you use RDBMS persistence for Apache Jackrabbit. That's why the database size should increase by default whenever a version is made.

Now if you switch the internal FileSystem of the VersionManager to VFSFileSystem with SFTP or WebDAV backend, then all the version data, the copies from the Workspace, will be stored in an external file system such as SFTP or WebDAV backend instead.

Switching it to VFSFileSystem for VersionManager is straightforward. See the following snippets from repository.xml configuration:


  <!-- SNIP -->

  <Versioning rootPath="${rep.home}/version">

    <FileSystem class="org.apache.jackrabbit.vfs.ext.fs.VFSFileSystem">
      <param name="config" value="${catalina.base}/conf/" />


    <!-- SNIP -->


  <!-- SNIP -->


Just replace FileSystem element and PersistenceManager element inside the Versioning element to use VFSFileSystem which is configured with a properties file specifying SFTP credentials or private key identity file.
Then it will make Apache Jackrabbit Repository to store all the version history data in the backend SFTP file system instead of database.

Please find a working demo project in my GitHub project at The demo project shows how to use VFSFile system for an SFTP backend system option for version history data as well as binary DataStore option with either VFS file system or AWS S3 bucket backend. Just follow its

Friday, January 12, 2018

Recipe for Migrating Hippo CMS Database from One to Another

Sometimes people want to migrate an existing database of Hippo CMS from one to another. For example, they have been running Hippo CMS on Oracle database, but after a while they started thinking about moving their on-premise system and database to a cloud platform. Sounds like a typical use case and that there must be some solutions already out there, right?

Well, surprisingly many people don't know that Apache Jackrabbit has provided a repository copying (or "backup" or "migration" as they call in the documentation) tool since v1.6, dated first in 2010!

There are some reasons why people don't know about the useful tool:
  • Many people use a vendor specific Apache Jackrabbit repository implementations from a specific project or product, not the Apache Jackrabbit Standalone Server itself. So, even if the Backup and migration feature is well documented in Apache Jackrabbit Standalone Server page, it is hard for them to follow.
  • Each vendor specific implementation with Apache Jackrabbit, such as Hippo CMS  has some tweaks for their own purposes, including extra libraries on top of the default Apache Jackrabbit modules. So, if users don't know which extra libraries to add more by themselves, it can hardly work for them.

That's why I created a 'recipe' project in one of my GitHub repositories:

The recipe introduces a step-by-step guide, with Hippo CMS specific examples. I think it should be helpful to other Apache Jackrabbit derivatives too. Please browse the source.

Last but not least, many thanks to Apache Jackrabbit Standalone Server tool! Cheers!

Tuesday, May 23, 2017

Remoting for Automation via Apache Jackrabbit JCR Webdav Server from Command Lines

Sometimes we need to create, update or even delete data in JCR in an automated way. For example, we need to update some properties on specific configuration nodes just after reseting the database and restarting the server for that specific environment. Or sometimes you need to import some data from XML to a remote JCR just after startup. Obviously you can do these manually through UI. But concerns arise when you need to do these in an automated way through a batch job or script.
I'd like to introduce the Apache Jackrabbit JCR WebDAV Server which provides an advanced remoting feature and how you can take advantage of the feature in an automated way like executing it in command lines.

Apache Jackrabbit JCR WebDAV Server

Apache Jackrabbit JCR WebDAV Server was basically designed to support remote JCR API calls via underlying WebDAV protocol. You can create, read, update or delete data in JCR content repository through JCR WebDAV Server via either a) JCR Client API or b) direct WebDAV requests from the client.

It is really good to be able to use JCR APIs directly from a remote client without having to care of the details of WebDAV/HTTP payloads, which should be really a good topic to cover later, but in this article, I'd like to focus only on the use cases from Command Line Client because it's more related to the "automation" topic of this article.

Command Line Examples through WebDAV/HTTP

I don't want to copy every example again here. Jukka Zitting, a former chairman of Apache Jackrabbit project and incubator PMCs, already explained it with very intuitive examples in one of his great blog articles in the past:
The blog article of Jukka's explains how to create a node, how to read a node, how to update one single-valued property in different types such as date or string, and how to delete a node.

I'd like to just add two more helpful examples below.

Updating Multiple Values Property from Command Lines

If you want to update a multiple-valued property like hipposys:members property in the following example CND, you can't use the example to update single valued property in Jukka's blog article:

[hipposys:group] > nt:base
- hipposys:system (boolean)
- hipposys:members (string) multiple
- hipposys:description (string)

To update multiple-valued property, you need to wrap the values in <values xmlns=''>...</values> element for the data argument. Here's an example curl command to update the property:

  curl --request PUT --header "Content-Type: jcr-value/undefined" \
    --data "<values xmlns=''><value>editor</value><value>john</value><value>jane</value></values>" \
    --user admin:admin \

Importing System View XML file to JCR from Command Lines

This example is basically just a variation of how to create a node shown in Jukka's blog article by using an external system view XML file instead.
Suppose you have the following system view XML file (e.g, editor.xml):

<?xml version="1.0" encoding="UTF-8"?>
<sv:node xmlns:sv="" sv:name="editor">
  <sv:property sv:name="jcr:primaryType" sv:type="Name">
  <sv:property sv:name="hipposys:members" sv:type="String" sv:multiple="true">
  <sv:property sv:name="hipposys:securityprovider" sv:type="String">

You can pipe the standard output of the input file, editor.xml, into a curl command by specifying --data argument as @-, meaning the data must be read from the standard input.

  cat editor.xml | curl -v --request MKCOL --data @- --user admin:admin \

The command can be rewritten to the following instead just as a different way to specify the standard input:

  curl --request MKCOL --data @- --user admin:admin \
    http://localhost:8080/cms/server/default/jcr:root/hippo:configuration/hippo:groups/editor \
    < editor.xml

Or, you can specify the input file directly by prefixing the file path with '@' like the following example instead:

  curl --request MKCOL --data @editor.xml --user admin:admin \

Therefore, if you want to remove an existing /hippo:configuration/hippo:groups/editor node and recreate it from the XML file, you could execute a delete command like the following, followed by one of the create command explained above:

  curl --request DELETE --user admin:admin \

I guess you already grasp the whole idea about how we can take advantage of JCR WebDAV Server for automation in command lines. Basically, you can easily create, read, update or delete any content in JCR from command lines!

How to Enable JCR WebDAV Server in Hippo CMS Repository?

By default, JCR WebDAV Server is not enabled in Hippo CMS Repository, but you can easily install and configure it by following this community forge plugin documentation:
Please let me know if you have any further questions.

Thursday, October 20, 2016

Playing with an Apache Jackrabbit DataStore Migration Tool

A while ago, I posted a blog article (Can't we store huge amount of binary data in JCR?) about why Apache Jackrabbit VFSDataStore or S3DataStore is useful and how to use it when storing huge amount of binary data in JCR. But, we already have many running JCR systems with different DataStores (e.g. DbDataStore). So, we need to be able to migrate an existing DataStore to VFSDataStore or S3DataStore. That's what I wanted to do with a migration tool (
In this article, I'd like to share my experiences in migrating a DbDataStore to VFSDataStore in a real project with the tool.

The Problem

One of my project (based on Hippo CMS) uses DbDataStore which is configured in repository.xml as the default option like the following:

<DataStore class="">
  <param name="url" value="java:comp/env/jdbc/repositoryDS" />
  <param name="driver" value="javax.naming.InitialContext" />
  <param name="databaseType" value="mysql" />
  <param name="minRecordLength" value="1024" />
  <param name="maxConnections" value="5" />
  <param name="copyWhenReading" value="true" />

Basically, I want to replace the DbDataStore with VFSDataStore backed by SFTP server after data migration in the end:

<DataStore class="org.apache.jackrabbit.vfs.ext.ds.VFSDataStore">
  <param name="config" value="${catalina.base}/conf/" />
  <param name="asyncWritePoolSize" value="10" />
  <param name="secret" value="123456"/>
  <param name="minRecordLength" value="1024"/>
  <param name="recLengthCacheSize" value="10000" />

And, should look like the following:

# SFTP base folder URL
baseFolderUri = sftp://tester:secret@localhost/vfsds
# when the identity file (your private key file) is used instead of password
#fso.sftp.identities = /home/tester/.ssh/id_rsa

So, we need to migrate all the data managed by DbDatStore to the SFTP location before switching to VFSDataStore.

Data Migration Steps

First of all, we need to download the latest version of the migration tool from
After uncompressing the downloaded file in a folder, we can build it with `mvn package`, which generates `jackrabbit-datastore-migration-x.x.x.jar` file under the `target` folder.

Second, we need to configure the "source" DataStore and "target" DataStore in a YAML file like the following example (e.g. config/migration-db-to-vfs.yaml):

        root: 'WARN'
        com.github.woonsan.jackrabbit.migration.datastore: 'INFO'

    minWorkers: '10'
    maxWorkers: '10'

        homeDir: 'target/storage-db'
        className: ''
            url: 'jdbc:mysql://localhost:3306/hippodb?autoReconnect=true&characterEncoding=utf8'
            user: 'hippo'
            password: 'hippo'
            driver: 'com.mysql.jdbc.Driver'
            databaseType: 'mysql'
            minRecordLength: '1024'
            maxConnections: '10'
            copyWhenReading: 'true'
            tablePrefix: ''
            schemaObjectPrefix: ''
            schemaCheckEnabled: 'false'

        homeDir: 'target/storage-vfs'
        className: 'org.apache.jackrabbit.vfs.ext.ds.VFSDataStore'
            asyncUploadLimit: '0'
            baseFolderUri: 'sftp://tester:secret@localhost/vfsds'
            minRecordLength: '1024'

As you can see, the "source" DataStore is configured with DbDataStore backed by a MySQL database, and the "target" DataStore is configured with VFSDataStore backed by a SFTP location.
Please note that the configuration style for each DataStore is actually equivalent to how it is set in repository.xml if you compare both configurations.
In addition, the YAML configuration has somethings about logging and thread pool worker counts, too, since logging and multi-threaded workers are important in this kind of batch applications.

Now, it's time to execute the migration tool.
Assuming you have JDBC Driver jar file in lib/ directory (e.g. lib/mysql-connector-java-5.1.38.jar), you can execute the tool like the following:

$ java -Dloader.path="lib/" \
       -jar target/jackrabbit-datastore-migration-0.0.1-SNAPSHOT.jar \

Or, if you know a specific location where the JDBC driver jar file exists, maybe you can run it like this instead:

java -Dloader.path=/home/tester/.m2/repository/mysql/mysql-connector-java/5.1.38/ \
     -jar target/jackrabbit-datastore-migration-0.0.1-SNAPSHOT.jar \

If your configurations are okay and the tool run properly, you will see result logs like the following:

.   ____          _            __ _ _
/\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/  ___)| |_)| | | | | || (_| |  ) ) ) )
'  |____| .__|_| |_|_| |_\__, | / / / /
:: Spring Boot ::        (v1.4.0.RELEASE)

2016-10-17 23:14:44.785  INFO 5071 --- [           main] .w.j.m.d.b.MigrationJobExecutionReporter :
Execution Summary:
Total: 22383, Processed: 22383, Read Success: 22383, Read Fail: 0, Write Success: 22383, Write Fail: 0, Duration: 1887607ms
Details (in CSV format):

2016-10-17 23:14:44.820  INFO 5071 --- [           main] c.g.w.j.migration.datastore.Application  : Started Application in 1892.767 seconds (JVM running for 1893.449)

Spring Boot generates the logging very nicely by default. You can also change the logging configuration. Please see Spring Boot documentation for that.
Anyway, it shows the result, including record sequence number, read/write status, byte size and error information, in CSV format in the end after execution logging lines.

Switching to VFSDataStore and Restart

Once all the binary data is migrated from DbDataStore to VFSDataStore (to SFTP location), we can switch to VFSDataStore by replacing the old <DataStore> element by this in repository.xml:

<DataStore class="org.apache.jackrabbit.vfs.ext.ds.VFSDataStore">
  <param name="config" value="${catalina.base}/conf/" />
  <param name="asyncWritePoolSize" value="10" />
  <param name="secret" value="123456"/>
  <param name="minRecordLength" value="1024"/>
  <param name="recLengthCacheSize" value="10000" />

Restart the server, and now the binary data will be served from SFTP server through the new VFSDataStore component!

Tuesday, August 30, 2016

Can't we store huge amount of binary data in JCR?

Can't we store huge amount of binary data in JCR? If you as a software architect have ever met a question like this (e.g, a requirement to store huge amount of binary data such as PDF files in JCR), maybe you could have had a moment depicting some candidate solutions. What is technically feasible and what's not? What is most appropriate to fulfill all the different quality attributes (such as scalability, performance, security, etc.) with acceptable trade-offs? Furthermore, what is more cost-effective and what's not?

Surprisingly, many people have tried to avoid JCR storage for binary data if the amount is going to be really huge. Instead of using JCR, in many cases, they have tried to implement a custom (UI) module to store binary data directly to a different storage such as SFTP, S3 or WebDAV through specific backend APIs.

It somewhat makes sense to separate binary data store if the amount is going to be really huge. Otherwise, the size of the database used by JCR can grow too much, which makes it harder and more costly to maintain, backup, restore and deploy as time goes by. Also, if your application requires to serve the binary data in a very scalable way, it will be more difficult with keeping everything in single database than separating the binary data store somewhere else.

But there is a big disadvantage with this custom (UI) module approach. If you store a PDF file through a custom (UI) module, you won't be able to search the content through standard JCR Query API any more because JCR (Jackrabbit) is never involved in storing/indexing/retrieving the binary data. If you could use JCR API to store the data, then Apache Jackrabbit could have indexed your binary node automatically and you could have been able to search the content very easily. Being unable to search PDF documents through standard JCR API could be a big disappointment.

Let's face the initial question again: Can't we store huge amount of binary data in JCR?
Actually... yes, we can. We can store huge amount of binary data through JCR in a standard way if you choose a right Apache Jackrabbit DataStore for a different backend such as SFTP, WebDAV or S3. Apache Jackrabbit was designed in a way to be able to plug in a different DataStore, and has provided various DataStore components for various backends. As of Apache Jackrabbit 2.13.2 (released on August, 29, 2016), it supports even Apache Commons VFS based DataStore component which enables to use SFTP and WebDAV as backend storage. That's what I'm going to talk about here.

DataStore Component in Apache Jackrabbit

Before jumping into the details, let me try to explain what DataStore was designed for in Apache Jackrabbit first. Basically, Apache Jackrabbit DataStore was designed to support large binary store for performance, reducing disk usage. Normally all node and property data is stored through PersistenceManager, but for relatively large binaries such as PDF files are stored through DataStore component separately.

DataStore enables:
  • Fast copy (only the identifier is stored by PersistenceManager, in database for example),
  • No blocking in storing and reading,
  • Immutable objects in DataStore,
  • Hot backup support, and
  • All cluster nodes using the same DataStore.
Please see for more detail. Especially, please note that a binary data entry in DataStore is immutable. So, a binary data entry cannot be changed after creation. This makes it a lot easier to support caching, hot backup/restore and clustering. Binary data items that are no longer used will be deleted automatically by the Jackrabbit Garbage collector.

Apache Jackrabbit has several DataStore implementations as shown below:

FileDataStore uses a local file system, DbDataStore uses a relational databases, and S3DataStore uses Amazon S3 as backend. Very interestingly, VFSDataStore uses a virtual file system provided by Apache Commons VFS module.

FileDataStore cannot be used if you don't have a stable shared file system between cluster nodes. DbDataStore has been used by Hippo Repository by default because it can work well in a clustered environment unless the binary data increases extremely too much. S3DataStore and VFSDataStore look more interesting because you can store binary data into an external storage. In the following diagrams, binary data is handled by Jackrabbit through standard JCR APIs, so it has a chance to index even binary data such as PDF files. Jackrabbit invokes S3DataStore or VFSDataStore to store or retrieve binary data and the DataStore component invokes its internal Backend component (S3Backend or VFSBackend) to write/read to/from the backend storage.

One important thing to note is that both S3DataStore and VFSDataStore extend CachingDataStore of Apache Jackrabbit. This gives a big performance benefit because a CachingDataStore caches binary data entries in local file system not to communicate with the backend if unnecessary.

As shown in the preceding diagram, when Jackrabbit needs to retrieve a binary data entry, it invokes DataStore (a CachingDataStore such as S3DataStore or VFSDataStore, in this case) with an identifier. CachingDataStore checks if the binary data entry already exists in its LocalCache first. [R1] If not found there, it invokes its Backend (such as S3Backend or VFSBackend) to read the data from the backend storage such as S3, SFTP, WebDAV, etc. [B1] When reading the data entry, it stores the entry into the LocalCache as well and serve the data back to JackrabbitCachingDataStore keeps the LRU cache, LocalCache, up to 64GB by default in a local folder that can be changed in the configuration. Therefore, it should be very performant when a binary data entry is requested multiple times because it is most likely to be served from the local file cache. Serving a binary data from a local cached file is probably much faster than serving data using DbDataStore since DbDataStore doesn't extend CachingDataStore nor have a local file cache concept at all (yet).

Using VFSDataStore in a Hippo CMS Project

To use VFSDataStore, you have the following properties in the root pom.xml:


    <!--***START temporary override of versions*** -->
    <!-- ***END temporary override of versions*** -->

    <-- SNIP -->


Apache Jackrabbit VFSDataStore is supported since 2.13.2. You also need to add the following dependencies in cms/pom.xml:

    <!-- Adding jackrabbit-vfs-ext -->
        Exclude jackrabbit-api and jackrabbit-jcr-commons since those were pulled
        in by Hippo Repository modules.

    <!-- Required to use SFTP VFS2 File System -->

And, we need to configure VFSDataStore in conf/repository.xml like the following example:


  <!-- SNIP -->

  <DataStore class="org.apache.jackrabbit.vfs.ext.ds.VFSDataStore">
    <param name="config" value="${catalina.base}/conf/" />
    <!-- VFSDataStore specific parameters -->
    <param name="asyncWritePoolSize" value="10" />
      CachingDataStore specific parameters:
        - secret : key to generate a secure reference to a binary.
    <param name="secret" value="123456789"/>
      Other important CachingDataStore parameters with default values, just for information:
        - path : local cache directory path. ${rep.home}/repository/datastore by default.
        - cacheSize : The number of bytes in the cache. 64GB by default.
        - minRecordLength : The minimum size of an object that should be stored in this data store. 16KB by default.
        - recLengthCacheSize : In-memory cache size to hold DataRecord#getLength() against DataIdentifier. One item for 140 bytes approximately.
    <param name="minRecordLength" value="1024"/>
    <param name="recLengthCacheSize" value="10000" />

  <!-- SNIP -->


The VFS connectivity is configured in ${catalina.base}/conf/ like the following for instance:

baseFolderUri = sftp://tester:secret@localhost/vfsds

So, the VFSDataStore uses SFTP backend storage in this specific example as configured in the properties file to store/read binary data in the end.

If you want to see more detailed information, examples and other backend usages such as WebDAV through VFSDataBackend, please visit my demo project here:

Note: Hippo CMS 10.x and 11.0 pull in modules of Apache Jackrabbit 2.10.x at the moment. However, there has not been any significant changes nor incompatible changes in org.apache.jackrabbit:jackrabbit-data and org.apache.jackrabbit:jackrabbit-vfs-ext between Apache Jackrabbit 2.10.x and Apache Jackrabbit 2.13.x. Therefore, it seems no problem to pull in org.apache.jackrabbit:jackrabbit-vfs-ext:jar:2.13.x dependency in cms/pom.xml like the preceding at the moment. But it should be more ideal to match all the versions of Apache Jackrabbit modules some day soon.
Update: Note that Hippo CMS 12.x pulls in Apache Jackrabbit 14.0+. Therefore, you can simply use ${jackrabbit.version} for the dependencies mentioned in this article.

Configuration for S3DataStore

In case you want to use S3DataStore instead, you need the following dependency:

    <!-- Adding jackrabbit-aws-ext -->
      <!-- ${jackrabbit.version} or a specific version like 2.14.0-h2. -->
        Exclude jackrabbit-api and jackrabbit-jcr-commons since those were pulled
        in by Hippo Repository modules.

    <!-- Consider using the latest AWS Java SDK for latest bug fixes. -->

And, we need to configure S3DataStore in conf/repository.xml like the following example (excerpt from


  <!-- SNIP -->

  <DataStore class="">
    <param name="config" value="${catalina.base}/conf/"/>
    <param name="secret" value="123456789"/>
    <param name="minRecordLength " value="16384"/> 
    <param name="cacheSize" value="68719476736"/>
    <param name="cachePurgeTrigFactor" value="0.95d"/>
    <param name="cachePurgeResizeFactor" value="0.85d"/>
    <param name="continueOnAsyncUploadFailure" value="false"/>
    <param name="concurrentUploadsThreads" value="10"/>
    <param name="asyncUploadLimit" value="100"/>
    <param name="uploadRetries" value="3"/>

  <!-- SNIP -->


The AWS S3 connectivity is configured in ${catalina.base}/conf/ in the above example.

Please find an example of in the following and adjust the configuration for your environment:

Comparisons with Different DataStores

DbDataStore (the default DataStore used by most Hippo CMS projects) provides a simple clustering capability based on a centralized database, but it could increase the database size and as a result it could increase maintenance/deployment cost and make it relatively harder to use hot backup/restore if the amount of binary data becomes really huge. Also, because DbDataStore doesn't maintain local file cache for the "immutable" binary data entries, it is relatively less performant when serving binary data, in terms of binary data retrieval from JCR. Maybe you can argue that application is responsible for all the cache controls in order not to burden JCR though.

S3DataStore uses Amazon S3 as backend storage, and VFSDataStore uses a virtual file system provided by Apache Commons VFS module. They obviously help reduce the database size, so system administrators could save time and cost in maintenance or new deployments with these DataStores. They are internal plugged-in components as designed by Apache Jackrabbit, so clients can simply use standard JCR APIs to write/read binary data. More importantly, Jackrabbit is able to index the binary data such as PDF files internally to Lucene index, so clients can make standard JCR queries to retrieve data without having to implement custom code depending on specific backend APIs.

One of the notable differences between S3DataStore and VFSDataStore is, the former requires a cloud-based storage (Amazon S3) which might not be allowed in some highly secured environments, whereas the latter allows to use various and cost-effective backend storages including SFTP and WebDAV that can be deployed wherever they want to have. You can take full advantage of cloud based flexible storage with S3DataStore though.


Apache Jackrabbit VFSDataStore can give a very feasible, cost-effective and secure option in many projects when it is required to host huge amount of binary data in JCR. VFSDataStore enables to use SFTP, WebDAV, etc. as backend storage at a moderate cost, and enables to deploy wherever they want to have. Also, it allows to use standard JCR APIs to read and write binary data, so it should save more development effort and time than implementing a custom (UI) plugin to communicate directly with a specific backend storage.

Other Materials

I have once presented this topic to my colleagues. I'd like to share that with you as well.

Please leave a comment if you have any questions or remarks.