Michael Rose

Software generalist, backend engineer, bit-herder, building distributed systems on the JVM for FullContact.

Modding the JVM for Fun and Profit: Part 1

| Comments

Lately I’ve been curious about JVM internals. I’ve been building systems on the JVM for several years now, and I’ve poked and profiled more than my fair share of services. Increasingly, I find myself wanting to reach inside the JVM and see more. More importantly, I learn by doing and by dropping logging in places it doesn’t belong, and I really want to learn what the inside of the JVM is like and start modifying it.

There are organizational benefits to building your own JVM. For example, the Twitter JVM team ships their own JVM with modified G1GC, heap profiling, performance and bugfixes and all sorts of other changes only hinted at by their talks. Often times, one might wish to customize behavior to better fit your usecase without necessarily needing to inject your jar on the classpath -Xbootclasspath/p to override system classes for your production JVMs.

Every so often, my coworkers and I consume a beer or two too many and kick around statements in the form “I wish the JVM did {X}” or “I wish the JVM didn’t do {X}.” If it was enough beers, we talk about maintaining our own JVM :). We’d have a few goals such as making GC logging happen asynchronously, or allowing one to trigger minor GCs from code (there are some reasons to do this.)

Realistically, it’s probably a bad idea to have your own JVM release unless you’re Twitter or Google. Since we are safely in the realm of enthusiasts, lets build us a JVM.

Goals for part 1 of this journey:

  • Download the JVM source code
  • Build JVM source code
  • Check that our JVM works
  • Make a change (we’ll change some GC logging)
  • Run JVM and observe change

I’m working on Ubuntu 16.04, but this should work on any proper Unix. OSX’s instructions seem a little more finnicky, so I didn’t try. However, I assume it should work with some coaxing.

We’ll start out simple. First off, we need to acquire the JVM source code. Since we’re wanting to make tweaks to existing stable JVMs, we’ll start with the releases “forest.” It’s called a forest, because it’s a set of Mercurial subtrees that comprise the JVM. Get it?

1
2
3
4
sudo apt-get install mercurial
hg clone http://hg.openjdk.java.net/jdk8u/jdk8u
cd jdk8u
sh get_source.sh .

Don’t be alarmed by failures that look like this:

1
2
3
4
5
6
7
8
9
10
WARNING: deploy exited abnormally (255)
WARNING: hotspomake/closed exited abnormally (255)
WARNING: hotsposrc/closed exited abnormally (255)
WARNING: hotspotest/closed exited abnormally (255)
WARNING: install exited abnormally (255)
WARNING: jdmake/closed exited abnormally (255)
WARNING: jdsrc/closed exited abnormally (255)
WARNING: jdtest/closed exited abnormally (255)
WARNING: pubs exited abnormally (255)
WARNING: sponsors exited abnormally (255)

That’s just closed-source stuff. It’s no Oracle JVM, but it’ll do.

Then we use a Mercurial tag to synchronize our tree to the latest public JDK8 release (8u92-b14)

1
hg up jdk8u92-b14

Now we need some dependencies. I’m not sure why we need cups, but I’m sure it’s important.

1
sudo apt-get install build-essential libx11-dev libxext-dev libxrender-dev libxtst-dev libxt-dev libcups2-dev libfreetype6-dev

You might need libX11-dev instead if you’re on vanilla Debian or older Ubuntus. You’ll also want ccache if you plan on recompiling the JVM a few times:

1
sudo apt-get install ccache

Thankfully, the JVM has adopted a fairly simple configure/make model that seems to work pretty well.

1
2
bash ./configure
make all

Wait a while…(make images is faster).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
----- Build times -------
Start 2016-07-30 00:36:41
End   2016-07-30 00:46:03
00:00:21 corba
00:00:14 demos
00:02:03 docs
00:03:12 hotspot
00:00:14 images
00:00:13 jaxp
00:00:17 jaxws
00:02:08 jdk
00:00:30 langtools
00:00:10 nashorn
00:09:22 TOTAL
-------------------------
Finished building OpenJDK for target 'all'

Sweet. So you’ve built a JVM.

Under build/linux-x86_64-normal-server-release/images/j2sdk-image, make has built and collected a full JVM installation for us. You can see that it works by checking the version. This can be tarballed up or turned into a dpkg or rpm.

1
2
3
4
5
$ pushd build/linux-x86_64-normal-server-release/images/j2sdk-image
$ bin/java -version
openjdk version "1.8.0-internal"
OpenJDK Runtime Environment (build 1.8.0-internal-xorlev_2016_07_29_17_09-b00)
OpenJDK 64-Bit Server VM (build 25.71-b00, mixed mode)

Cool.

Now down to business. I’ve never liked that -XX:+PrintGCApplicationStoppedTime outputted values in seconds. We’ll helpfully change this to milliseconds. I have Evan Jones’ mmap-pause repo handy locally, so I’ll use his MakeGarbage.java tool to create some GC activity.

1
2
3
4
5
6
$ build/linux-x86_64-normal-server-release/images/j2sdk-image/bin/javac MakeGarbage.java
$ build/linux-x86_64-normal-server-release/images/j2sdk-image/bin/java -XX:+PrintGCDetails -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCApplicationStoppedTime -Xmx1G -Xms1G MakeGarbage
2016-07-30T00:20:25.354-0600: [GC (Allocation Failure) [PSYoungGen: 346876K->0K(348672K)] 347164K->288K(1048064K), 0.0019772 secs] [Times: user=0.01 sys=0.00, real=0.00 secs]
2016-07-30T00:20:25.357-0600: Total time for which application threads were stopped: 0.0022125 seconds, Stopping threads took: 0.0000186 seconds
2016-07-30T00:20:25.408-0600: [GC (Allocation Failure) [PSYoungGen: 346876K->0K(348672K)] 347164K->288K(1048064K), 0.0012677 secs] [Times: user=0.00 sys=0.01, real=0.01 secs]
2016-07-30T00:20:25.409-0600: Total time for which application threads were stopped: 0.0014163 seconds, Stopping threads took: 0.0000186 seconds

The way the JVM project is laid out, most of the VM code lives under the hotspot/ directory, while the jdk/ directory contains more application-y code, that is, the Java standard library and JNI code to back it. Each tree has both common and {platform,cpu}-specific code, for example hotspot/src/os_cpu/linux_ppc. We’ll be primarily working in hotspot/src/share/vm for this mod.

hotspot/src/share/vm/services/runtimeService.cpp
1
2
3
4
5
6
7
8
9
10
11
// Print the time interval for which the app was stopped
// during the current safepoint operation.
if (PrintGCApplicationStoppedTime) {
  gclog_or_tty->date_stamp(PrintGCDateStamps);
  gclog_or_tty->stamp(PrintGCTimeStamps);
  gclog_or_tty->print_cr("Total time for which application threads "
                         "were stopped: %3.7f seconds, "
                         "Stopping threads took: %3.7f seconds",
                         last_safepoint_time_sec(),
                         _last_safepoint_sync_time_sec);
}

Through some simple grepping, we can find the message in question. We’ll modify the message and properly convert the quantities to milliseconds.

hotspot/src/share/vm/services/runtimeService.cpp
1
2
3
4
5
6
7
8
9
10
11
// Print the time interval for which the app was stopped
// during the current safepoint operation.
if (PrintGCApplicationStoppedTime) {
  gclog_or_tty->date_stamp(PrintGCDateStamps);
  gclog_or_tty->stamp(PrintGCTimeStamps);
  gclog_or_tty->print_cr("Total time for which application threads "
                         "were stopped: %3.7f milliseconds, "
                         "Stopping threads took: %3.7f milliseconds",
                         last_safepoint_time_sec() * 1000.0,
                         _last_safepoint_sync_time_sec * 1000.0);
}

Save that, and now we’ll rebuild our JVM.

hotspot/src/share/vm/services/runtimeService.cpp
1
2
3
4
5
6
$ make all # or make images to skip doc creation
$ build/linux-x86_64-normal-server-release/images/j2sdk-image/bin/java -XX:+PrintGCDetails -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCApplicationStoppedTime -Xmx1G -Xms1G MakeGarbage
2016-07-30T00:25:51.051-0600: [GC (Allocation Failure) [PSYoungGen: 346876K->0K(348672K)] 347172K->296K(1048064K), 0.0014952 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-07-30T00:25:51.052-0600: Total time for which application threads were stopped: 1.6477420 milliseconds, Stopping threads took: 0.0195990 milliseconds
2016-07-30T00:25:51.104-0600: [GC (Allocation Failure) [PSYoungGen: 346876K->0K(348672K)] 347172K->296K(1048064K), 0.0013866 secs] [Times: user=0.01 sys=0.00, real=0.00 secs]
2016-07-30T00:25:51.106-0600: Total time for which application threads were stopped: 1.5480560 milliseconds, Stopping threads took: 0.0185380 milliseconds

Huzzah.

Admittedly, this was a very simple change. In part 2 we’ll explore adding a new runtime method to invoke minor GCs from client applications, and asynchronously emit GC logging to avoid disk writes adding to our GC pauses. Then, we’ll package our JDK as a Debian package such that it can become our system’s default Java. Until next time!

Dropwizard Deployment for the Lazy Using Rake and Net::SSH

| Comments

Today I embarked on automated deployment of one of my basic Dropwizard/Angular.JS projects. This project’s purpose is to extract unstructured data from forum posts and turn them into data directly consumable by an API. It’s split into a four major pieces:

  • Core — reusable representation objects to keep things tidy between phases
  • Extraction — Extracts data from forum posts by executing a large unwieldy JDBC query and iterating over the results, then pushing each into MongoDB
  • Query — Dropwizard/JAX-RS REST API for exposing the data found by the extraction phase
  • Frontend — Angular.JS frontend app that consumes the REST API

As with many projects, they all begin somewhat manually, but today I wanted to automate the deployment of my project. Jenkins is a little heavy weight for a personal project, and I don’t really need a full-blown continuous integration framework. Capistrano is more along the lines I wanted, but is too Rails-flavored to be immediately useful for my needs.

Rake

Rake is simple. Stupid simple. It’s a Ruby DSL for running commands, a step up from a hand-rolled bash script.

Mahout & Dropwizard: Collaborative Filtering & Recommenders

| Comments

Recommenders are in use all throughout the web. Chances are, you’ve interacted with dozens of recommenders systems thought the course of simply finding this article.

Amazon is built on recommender systems.

Amazon Recommendations

Using techniques such as item-based recommenders and itemset generation they can not only find what’s relevant to you, they can determine things related to what you’re browsing and further find items frequently bought with yours (and perhaps provide a combo bonus to shake cobwebs from your wallet).

As with any Machine Learning task, the first step is to define your problem. What are you trying to do? Who is your customer? Is it your Business Intelligence team? Is it the consumer browsing your webapp? Is it your fault detection system attempting to draw relations? Spike detection?

The problem I’m attempting to solve is a somewhat common problem. On a site (which will remain nameless for now) members rate threads containing media. For the sake of simplicity, lets say the threads are albums posted by members along with their reviews of said albums. The members of this site are extremely opinionated, and want good recommendations.