This tutorial is based on macOSX with an M1 chip for Python 3.8/3.9 and Java 11.

Intro.

From my practice, one of the last cases I met was the task when the base application engine was in Java, and I was supposed to add a new Machine Learning feature. The key challenge here was that the ML feature is in Python.

So what are we going to do? What kind of solutions do we apply to this task? One of the working solutions is to run this service in docker. Or we can find another, more effective production-friendly solution: to execute python code directly from Java code via the Jep library (Java Embedded Python).

What is the JEP library — a Pure Voodoo Itself.

Down below are some citations from their FAQ on GitHub :

Jep uses JNI and the CPython API to start up the Python interpreter inside the JVM. When you create an Interpreter instance in Java, a Python interpreter will be created for that Java Interpreter instance and will remain in memory until the Interpreter instance is closed with Interpreter.close().

Due to the need to manage a consistent Python thread state, a thread that creates an Interpreter instance must be reused for all method calls to that Interpreter instance.

And here is a citation regarding the Python module execution:

Jep should work with any pure Python modules. It works with a variety of CPython extensions. Developers have reported that Jep works with NumPy, Scipy, Pandas, TensorFlow, Matplotlib, cvxpy, and more.

The speed performance is a crucial metric, especially if your application deal with BigData. Tests in my research showed that using JEP is nearly pure python execution.

How to set up JEP for your project.

Assuming you have Java (Maven) and Python in your system. In the Java project, in the pom.xml file, you set a dependency: 

<dependency>            
    <groupId>black.ninia</groupId>
    <artifactId>jep</artifactId>
    <version>4.0.3</version>
    <scope>compile</scope>        
</dependency>

_{^{The pom.xml dependency for running Python from Java}}

Run in project terminal:

mvn clean install

or

mvn clean install -DskipTests

You installed JEP in Java!

To install JEP in Python, run:

pip install jep 

This command installs the latest JEP version. You can specify a particular JEP version if you need it. In my example, I use JEP 4.0.3 version.

The version incompatibility is substantial here. The lower the Python version and Jep library version you will take, the more complicated java syntax you will use (and vice versa). But latest JEP versions can readily execute JEP old versions’ syntax.

In your system, it is mandatory to set up an environment variable. For macOSX, it will be like running in the terminal:

export DYLD_LIBRARY_PATH="<your_user>/myenv/lib/python3.8/site-packages/jep"$DYLD_LIBRARY_PATH

And do not forget to add the same line to your ~/.zshrc file! JEP is looking for the libjep.jnilib file on macOS and libjep.so file on Linux.

Besides all, for the Linux system, it will be a different environment variable name:

export LD_LIBRARY_PATH="<your_user_path>/myenv/lib/python3.8/site-packages/jep"$LD_LIBRARY_PATH

You set up the JEP library. Now you can execute Python directly from Java.

The syntax for Python 3.8 macOSX M1 chip.

First, we should define the JEP library path (specifically where Java will search for Python to execute) and initialize the MainInterpreter object:

import jep.*;
import jep.MainInterpreter;

import java.nio.file.Files;
import java.nio.file.Path;
import java.util.ArrayList;


String pythonFolder = System.getenv("DYLD_LIBRARY_PATH");

//define the JEP library path
String jepPath = pythonFolder + "/jep/libjep.jnilib";

if (!Files.exists(Path.of(jepPath))){            
   jepPath = pythonFolder + "/jep/libjep.so";        
}
//initialize the MainInterpreter   
MainInterpreter.setJepLibraryPath(jepPath);

_{^{Example of how to set up a Python Path for Jep library in Java}}

Second — to set up JEP configurations (we should add Python libraries to operate and our project folders). Then, create a SharedInterpreter object. A SharedInterpreter directly will execute out python code or python executive document:

jep.JepConfig jepConf = new JepConfig();   

jepConf.addIncludePaths("/Users/galinablokh/Documents/POCNerClassificationSDK/src/main/java");    

jepConf.addIncludePaths(pythonFolder);

SharedInterpreter.setConfig(jepConf);

SharedInterpreter subInterp = new SharedInterpreter();

^{Example of how to set up the Jep configurations and to create a SharedInterpreter for Python execution}

I have a python document with two simple functions to check JEP work. One function should use the native Python library to search particular documents in the directory. The second function calls the more complicated Python library, Spacy. Make sure you installed in Python the Spacy library with downloaded models before Jep run:

import glob
import os

import spacy

def get_c_path(pathlib, extension):
    path_list = glob.glob(f'{pathlib}/' + f'{extension}')
    return path_list


def run_spacy_nlp(sentence):
    nlp = spacy.load("en_core_web_sm")
    doc = nlp(sentence)
    return [(token.text, token.lemma_, token.pos_, token.tag_,
             token.dep_, token.shape_, token.is_alpha, token.is_stop) for token in doc]

^{Example of simple Python functions for Jep check}

Now, let us run the Python document functions from Java:

// run function from the python_functions.py document
subInterp.eval("import python_functions as p");

subInterp.eval("res_spacy = p.run_spacy_nlp('Apple is looking at buying U.K. startup for $1 billion')");

ArrayList result = (ArrayList) subInterp.getValue("res_spacy");

   for (Object item:result) {
       System.out.println(item.toString());
        }

^{Example of function run from the python_functions.py document (python 3.8)}

The output is:

>>> [Apple, Apple, PROPN, NNP, nsubj, Xxxxx, true, false]
>>> [is, be, AUX, VBZ, aux, xx, true, true]
>>> [looking, look, VERB, VBG, ROOT, xxxx, true, false]
>>> [at, at, ADP, IN, prep, xx, true, true]
>>> [buying, buy, VERB, VBG, pcomp, xxxx, true, false]
>>> [U.K., U.K., PROPN, NNP, dobj, X.X., false, false]
>>> [startup, startup, VERB, VBD, dep, xxxx, true, false]
>>> [for, for, ADP, IN, prep, xxx, true, true]
>>> [$, $, SYM, $, quantmod, $, false, false]
>>> [1, 1, NUM, CD, compound, d, false, false]
>>> [billion, billion, NUM, CD, pobj, xxxx, true, false]

This is awesome! We can call and execute Python directly from Java! But wait, what about the syntax for other Python versions I mentioned at the beginning of the article?

The syntax for Python 3.9 macOSX M1 chip.

I said before the higher Python and Jep versions, the simpler syntax in Java. It is so because, in the latest Jep versions for the Python 3.9+ versions, JEP Developers added an automatic search for the Python path when you do “pip install jep”. Let’s see the syntax below!

// set path for jep executing python3.9
MainInterpreter.setJepLibraryPath(jepPath);
        
// set path for python docs with python script to run
jep.JepConfig jepConf = new JepConfig();
jepConf.addIncludePaths(System.getProperty("user.dir")+"/src/main/java/");
        
//create the interpreter for python executing
Interpreter subInterp = jepConf.createSubInterpreter();

^{Example of how to set the path for jep and create the SubInterpreter for executing python3.9}

Four lines of code, and we have a python3.9 Interpreter in Java! Now you will see how to run the Python functions from the document you’ve already seen.

//import  .py doc with to run
subInterp.eval("import python_functions as p");

// run each function from the .py doc I
subInterp.eval("res_spacy = p.run_spacy_nlp('Apple is looking at buying U.K. startup for $1 billion')");
System.out.println(subInterp.getValue("res_spacy"));
        
//II
subInterp.eval("res_c = p.get_c_path('.idea','*.xml')");
System.out.println(subInterp.getValue("res_c"));

^{Example of how to run python3.9 functions from Java Jep}

The printouts will be the same as for the Python 3.8 version. That’s all folks!!

Conclusions.

In this tutorial, you got clear instructions on how to execute your Python code from any Java application for Python 3.8 and Python 3.9 versions. 

All code examples in the article you can run on macOSX M1 chip and Linux. You only need to set up correctly the Jep library for each system.

Sources.

1. Jep library GitHub 
2. My GitHub project with code examples
3. The original article on medium