Java Repository Layout and Artifacts

[karllessard]

Let's start this issue to add suggestions/comments in regard of the proposed layout and artifacts for the upcoming TF Java repositories, that will be based on this one.

[karllessard]

The current layout propose the following hierarchy:

• /support : Support libraries independent from TensorFlow but that could be used by it
  • /nio : Library for improving support of N-dimensional data structures in Java
• /core : Core artifacts that allows basic usage of TensorFlow in Java
  • /annotation-processor: Internal library used to process annotations in TensorFlow Java bindings
  • /libtensorflow : TensorFlow Java bindings
  • /libtensorflow-jni : TensorFlow native bindings for Java (JNI based right now)
• /frameworks : High-level APIs built on top of the core for a better user experience
  • /keras : Keras-like API for Java
  • /inference : Model pipelines for improved inference
• /starters: POM artifacts for single-dependency inclusion of TensorFlow
  • /tensorflow : Main starter that includes everything, including default variants for the core native layers, i.e. darwin, windows and linux with non-GPU support.

[karllessard]

Renaming the artifacts: The actual names for the main core artifacts are libtensorflow for the java part and libtensorflow_jni for the native one.

We can take this opportunity to give them better names that are more Java-like than C-like.

First, do we keep the groupId as org.tensorflow? or should we switch to something new for the SIG, like org.tensorflow.jvm?

Also, I don't know about the artifacts that would come out of the usage of JavaCPP but for the current implementation, I suggest those new names to begin with:

libtensorflow -> tensorflow-java
libtensorflow_jni -> tensorflow-native

[saudet]

In my opinion, these 2 artifacts should be merged, because neither can be used independently anyway. Moreover, I don't see anything in the build files for Bazel in there that we can't already do with JavaCPP, which can do more, so we only lose in functionality by keeping Bazel to manage JNI-enabled modules there. tensorflow-native, I think that would be a good name for this module yes. We'd get multiple artifacts out of it with different classifiers, making it easy to do CI with it as well, and I would also add a tensorflow-native-platform module that depends on all these artifacts with classifiers, but which would be transitively configurable like this via profiles:
https://github.com/bytedeco/javacpp-presets/wiki/Reducing-the-Number-of-Dependencies

As for the group name, I say we should keep the org.tensorflow group as is. Adding redundant suffixes like "java" or "jvm" just makes names longer. It's pretty obvious that Maven modules are for Java.

[karllessard]

In my opinion, these 2 artifacts should be merged, because neither can be used independently anyway.

I like the idea of keeping those 2 artifacts separated because the Java core just need to be build and deploy once while the native core needs to be rebuild/redeployed for each configuration and OS we want to support. The user project can depend on many versions (classifier) of it without any issue, as there won't be any clash of duplicated Java classes.

Having distinct artifacts allows you to include/exclude manually which platform and features you want from the native layer.

For example, let say you just want support for linux (ubuntu), you can add those individual dependencies:
org.tensorflow:tensorflow-java
org.tensorflow:tensorflow-native:linux-x86_64

On the other hand, if size does not matter for you, you can simply depends on one of the starter artifacts that automatically includes the "common" versions of the library, like it is done now:
org.tensorflow:tensorflow

This will automatically include to your classpath:
org.tensorflow:tensorflow-java
org.tensorflow:tensorflow-native:linux-x86_64
org.tensorflow:tensorflow-native:darwin-x86_64
`org.tensorflow:tensorflow-native:windows-x86_64'

The current library loader logic can take care of selecting the right library at runtime.

We could have another starter call org.tensorflow:tensorflow-gpu to include the same libraries but with GPU enabled.

I also don't have a clear list of what other OS we need to support but we can build the tensorflow-native artifact for any of them or any distribution (e.g. CentOS)

I don't see anything in the build files for Bazel in there that we can't already do with JavaCPP

I don't have a clear idea of what JavaCPP can or cannot do but of course, if we don't need to keep Bazel we'll simply drop it. Maybe you can create a PR with the changes you would like to apply to replace JNI/Bazel with JavaCPP, so I/we can understand better what it implies?

I would also add a tensorflow-native-platform module that depends on all these artifacts with classifiers, but which would be transitively configurable like this via profiles

IMHO, I think simple maven inclusion/exclusion is pretty basic and should fulfill most of our cases, instead of relying on additional build variables. Maybe you can explain a bit more why we would need those with some examples?

As for the group name, I say we should keep the org.tensorflow group as is. Adding redundant suffixes like "java" or "jvm" just makes names longer. It's pretty obvious that Maven modules are for Java.

Hehe, the "jvm" stands for "SIG JVM group" and not "Warning! This is a Java library!". It was mainly to avoid any clash with existing artifacts that came out of the core repository (i.e. org.tensorflow:tensorflow) and might still come out after we migrate our part (need to check but last news was that Android was still using the Bazel Java build for some of their apps and I don't know for how long they'll continue to do it).

But yeah, we can stay positive and think that it is safe to preserve the org.tensorflow group ID, which I agree is better.

[saudet]

For example, let say you just want support for linux (ubuntu), you can add those individual dependencies:
org.tensorflow:tensorflow-java
org.tensorflow:tensorflow-native:linux-x86_64

What I do is exactly the same. The only difference is that it would look like this instead:

org.tensorflow:tensorflow-native
org.tensorflow:tensorflow-native:linux-x86_64

The first contains only Java classes that are the same for all platforms, the second, only native libraries for linux-x86_64 platform: Exactly the same thing as you describe. The difference is that they need to be built together, so there is less chance for them to get out of sync. Also, for users, it makes it clear that they are related, while just adding a "java" suffix is pretty redundant as I pointed out previously.

On the other hand, if size does not matter for you, you can simply depends on one of the starter artifacts that automatically includes the "common" versions of the library, like it is done now:
org.tensorflow:tensorflow

Right, that's the tensorflow-native-platform I was referring to, but the name doesn't really matter, although having "native" in there makes it clear that it's related to the tensorflow-native ones only.

IMHO, I think simple maven inclusion/exclusion is pretty basic and should fulfill most of our cases, instead of relying on additional build variables. Maybe you can explain a bit more why we would need those with some examples?

Well, there are many TF users that also use OpenCV, FFmpeg, Tesseract, etc, but say we don't care about those, and just think about TF. This library also depends on other libraries like CUDA and MKL, among others. They are not mandatory, but they are still useful for performance, so it makes sense to provide builds that use CUDA, MKL, etc. Now, how do users go about getting those dependencies? We could force them to install them manually, but since the redistributables are available as Maven artifacts, it's also possible to use them like this:

        <dependency>
            <groupId>org.bytedeco</groupId>
            <artifactId>mkl-platform-redist</artifactId>
            <version>2019.4-1.5.2-SNAPSHOT</version>
        </dependency>
        <dependency>
            <groupId>org.bytedeco</groupId>
            <artifactId>cuda-platform-redist</artifactId>
            <version>10.1-7.6-1.5.2-SNAPSHOT</version>
        </dependency>

And users can select at runtime the platforms they are interested in to limit the binaries that need to be downloaded. But you're saying they should be forced to add/remove manually at build time dependencies for all their platforms and for all the transitive dependencies? You can't mean that. This is basic functionality that the Java platform needs if it wants to remain relevant in the enterprise in the long term! I'll eventually create a PR here, but it's basically going to look like what I've done before for the JavaCPP Presets and Deeplearning4j. And we've talked about this before too on the mailing list, etc. So I'm getting the impression that you're concerned about something else that you're not mentioning here. Even if I make a PR and get everything working, I feel it's not going to be acceptable for some reason, or am I wrong?

[karllessard]

For example, let say you just want support for linux (ubuntu), you can add those individual dependencies:
org.tensorflow:tensorflow-java
org.tensorflow:tensorflow-native:linux-x86_64

What I do is exactly the same. The only difference is that it would look like this instead:

org.tensorflow:tensorflow-native
org.tensorflow:tensorflow-native:linux-x86_64

The first contains only Java classes that are the same for all platforms, the second, only native libraries for linux-x86_64 platform: Exactly the same thing as you describe. The difference is that they need to be built together, so there is less chance for them to get out of sync.

I like that. Then if the artifact name is not only used to hold the native code, maybe we should rename it to tensorflow-core instead, wdyt?

On the other hand, if size does not matter for you, you can simply depends on one of the starter artifacts that automatically includes the "common" versions of the library, like it is done now:
org.tensorflow:tensorflow

Right, that's the tensorflow-native-platform I was referring to, but the name doesn't really matter, although having "native" in there makes it clear that it's related to the tensorflow-native ones only.

Got it. And that's the thing, they might contain more than the tensorflow-native (or tensorflow-core) later, I guess we can decide at some point that the Keras API should be part of the common starters as well. So in this case, tensorflow or tensorflow-gpu sounds appropriate.

IMHO, I think simple maven inclusion/exclusion is pretty basic and should fulfill most of our cases, instead of relying on additional build variables. Maybe you can explain a bit more why we would need those with some examples?

Well, there are many TF users that also use OpenCV, FFmpeg, Tesseract, etc, but say we don't care about those, and just think about TF. This library also depends on other libraries like CUDA and MKL, among others. They are not mandatory, but they are still useful for performance, so it makes sense to provide builds that use CUDA, MKL, etc. Now, how do users go about getting those dependencies? We could force them to install them manually, but since the redistributables are available as Maven artifacts, it's also possible to use them like this:

        <dependency>
            <groupId>org.bytedeco</groupId>
            <artifactId>mkl-platform-redist</artifactId>
            <version>2019.4-1.5.2-SNAPSHOT</version>
        </dependency>
        <dependency>
            <groupId>org.bytedeco</groupId>
            <artifactId>cuda-platform-redist</artifactId>
            <version>10.1-7.6-1.5.2-SNAPSHOT</version>
        </dependency>

And users can select at runtime the platforms they are interested in to limit the binaries that need to be downloaded. But you're saying they should be forced to add/remove manually at build time dependencies for all their platforms and for all the transitive dependencies? You can't mean that.

Probably not because I don't understand what you mean. I was not referring to external libraries like Cuda or MKL, I think I was just pointing out that the examples I gave to handle the different combinations for the tensorflow-core, using the standard Maven dependency system, should be sufficient for most cases. I thought you wanted to introduce some more advanced/custom configuration variables at build time to achieve this.

This is basic functionality that the Java platform needs if it wants to remain relevant in the enterprise in the long term! I'll eventually create a PR here, but it's basically going to look like what I've done before for the JavaCPP Presets and Deeplearning4j. And we've talked about this before too on the mailing list, etc. So I'm getting the impression that you're concerned about something else that you're not mentioning here. Even if I make a PR and get everything working, I feel it's not going to be acceptable for some reason, or am I wrong?

You are very wrong, indeed :) For sure, I want to understand what you intend to do and so far, it sounds promising.

[Craigacp]

One thing I'd note is that it would be useful to have Tensorflow builds which include both MKL and CUDA support, as then the CPU fallback from GPU code is less painful. I understand that DL4J has completely separate bindings which can cause a bit of a perf cliff if the GPU doesn't have an implementation of the operation. This might be less of an issue in Tensorflow as it's got better buildout, but it's worth keeping in mind.

[jxtps]

Layout looks great, I'd consider renaming /support to /util or /helpers - support makes my mind go in the wrong direction.

Artefacts: it's sounds like @saudet has a clear picture in mind after all his work on JavaCPP - I've only used it via DL4J and some other projects and appreciate the benefits as a user but I'm not deeply familiar with the internal layout. Maybe there is some primer docs (on bytedeco.org?) that the rest of us should read to better understand the more nuanced mechanics? At present I'm not in a position to provide meaningful feedback on this aspect.

[karllessard]

@jxtps : sure, I thought of utils before as well so I guess it's a better pick

[saudet]

I like that. Then if the artifact name is not only used to hold the native code, maybe we should rename it to tensorflow-core instead, wdyt?

That's true, but all the Java code in there is to support the native code, so I still think "native" makes sense. Right now "core" is used for the parent module in the hierarchy, so having one of its submodules also named "core" might be confusing... (I know naming is not an easy business...)

Got it. And that's the thing, they might contain more than the tensorflow-native (or tensorflow-core) later, I guess we can decide at some point that the Keras API should be part of the common starters as well. So in this case, tensorflow or tensorflow-gpu sounds appropriate.

tensorflow could also be some other module that depends on tensorflow-native-platform and others. We don't need to put everything into the least amount of modules either if it makes sense. tensorflow-gpu, obviously yes. Right now, for example, I've got tensorflow-platform, tensorflow-platform-gpu, tensorflow-platform-python, and tensorflow-platform-python-gpu in the presets, but again it's just a question of naming. We're talking about the same thing. "native" is redundant in the case of the presets because that's all there is, but not in the case of a downstream project where I think it makes things clearer to have names with "native" (I like that one), "jni" (but not so much this one, some people might not get it), etc.

Probably not because I don't understand what you mean. I was not referring to external libraries like Cuda or MKL, I think I was just pointing out that the examples I gave to handle the different combinations for the tensorflow-core, using the standard Maven dependency system, should be sufficient for most cases. I thought you wanted to introduce some more advanced/custom configuration variables at build time to achieve this.

It's sort of "advanced" I suppose, but it doesn't prevent us from doing things the "traditional" way. In other words, there's no drawbacks, which is confirmed by having this scheme used in production in some extremely large corporations (that I'm not sure I'm allowed to mention) for many years already.

One thing I'd note is that it would be useful to have Tensorflow builds which include both MKL and CUDA support, as then the CPU fallback from GPU code is less painful. I understand that DL4J has completely separate bindings which can cause a bit of a perf cliff if the GPU doesn't have an implementation of the operation. This might be less of an issue in Tensorflow as it's got better buildout, but it's worth keeping in mind.

As part of the CUDA build, DL4J now also supports MKL and MKL-DNN. I'm not sure why the official builds for TF still don't have MKL and MKL-DNN enabled, but then again, the official builds also don't bundle CUDA or cuDNN either. We have to use Anaconda for that. It might have something to do with getting people on Google's cloud with TPUs and stuff...

Artefacts: it's sounds like @saudet has a clear picture in mind after all his work on JavaCPP - I've only used it via DL4J and some other projects and appreciate the benefits as a user but I'm not deeply familiar with the internal layout. Maybe there is some primer docs (on bytedeco.org?) that the rest of us should read to better understand the more nuanced mechanics? At present I'm not in a position to provide meaningful feedback on this aspect.

There's some documentation about all this on these wiki pages:

• https://github.com/bytedeco/javacpp-presets/wiki/Create-New-Presets
  • I've just realized I need to update that one to explain the "platform" artifact, but it's in the "helloworld" project here: https://github.com/bytedeco/javacpp-presets/tree/helloworld/helloworld
• https://github.com/bytedeco/javacpp-presets/wiki/Reducing-the-Number-of-Dependencies

It's obviously not very extensive though, so I would be happy to clarify anything.

[karllessard]

Thanks @saudet , we seem to be pretty much on the page. I'm still not sure about the core vs native though, as "the Java code in there is to support the native code" is not entirely true, there is higher-level APIs as well, like the operation wrappers. Or maybe we could move those in a separate core artifact... Anyway, we can address this later.

But now I would like to know how you visualize the migration to JavaCPP and what would be the next steps. Are you planning to create a PR? I guess there is also a lot a work involved to replace the JNI layer, which unfortunately contains more logic than it should,. Maybe we can start to request help from the SIG members?

[saudet]

Thanks @saudet , we seem to be pretty much on the page. I'm still not sure about the core vs native though, as "the Java code in there is to support the native code" is not entirely true, there is higher-level APIs as well, like the operation wrappers. Or maybe we could move those in a separate core artifact... Anyway, we can address this later.

I still consider the ops basically as wrappers around the native API, but obviously it's not clear cut. Anyway, it's always best to nail the terminology we're going to use as early as possible though. I don't mind "core" as name, I just mind that the parent module is also named the same...

But now I would like to know how you visualize the migration to JavaCPP and what would be the next steps. Are you planning to create a PR? I guess there is also a lot a work involved to replace the JNI layer, which unfortunately contains more logic than it should,. Maybe we can start to request help from the SIG members?

I think I can manage to refactor as necessary within a day or two of work. For starters, it's pretty much just doing what I've done previously elsewhere so it doesn't make sense for someone else to refigure out everything I had to go through previously. :) I just need to find a bit of time which hasn't been obvious the last week for me...

[karllessard]

@saudet : We’ll probably have to stay in sync then because I’m adding right now the NdArray interface to the Tensors, which have quite an impact on the memory allocation and data initialization actually done in the JNI layer, so I suggest that you start with anything unrelated to this.

I intend to propose soon to the SIG major changes on how we handle tensors and datatypes in general.