Skip to content

Commit 60de314

Browse files
agibsoncccagibsonccc
authored
Merge pull request #1057 from eclipse/ag_M2
Add M2 branch for release testing
2 parents 6d65b1a + bd2f790 commit 60de314

File tree

37 files changed

+480
-739
lines changed

37 files changed

+480
-739
lines changed

‎.github/workflows/example-sanity-check.yml

+4-16
Original file line numberDiff line numberDiff line change
@@ -3,21 +3,9 @@ on:
33
- cron: "0 */12 * * *"
44
workflow_dispatch:
55
jobs:
6-
# Wait for up to a minute for previous run to complete, abort if not done by then
7-
pre-ci:
8-
runs-on: ubuntu-latest
9-
timeout-minutes: 1
10-
steps:
11-
- name: 'Block Concurrent Executions'
12-
uses: softprops/turnstyle@v1
13-
with:
14-
poll-interval-seconds: 10
15-
env:
16-
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
176
#Note: no -pl here because we publish everything from this branch and use this as the basis for all uploads.
18-
linux-arm64:
19-
needs: pre-ci
20-
runs-on: ubuntu-16.04
7+
linux-x86_64:
8+
runs-on: ubuntu-18.04
219
steps:
2210
- uses: AutoModality/action-clean@v1
2311
- name: Cancel Previous Runs
@@ -29,15 +17,15 @@ jobs:
2917
uses: actions/setup-java@v1
3018
with:
3119
java-version: 1.8
32-
- name: Build on linux-arm64
20+
- name: Build on linux-x86_64
3321
shell: bash
3422
env:
3523
DEBIAN_FRONTEND: noninteractive
3624
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
3725
run: |
3826
mvn --version
3927
for folder in `ls`; do
40-
if [[ -d "$folder" ]]; then
28+
if [ -d "$folder" ] && [ "$folder" != "mvn-project-template" ]; then
4129
cd "$folder"
4230
if test -f "pom.xml"; then
4331
mvn clean test

‎NOTICE.txt

+20
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
Eclipse Deeplearning4j
2+
Copyright 2021 Eclipse Deeplearning4j Contributors
3+
4+
This product includes software developed at
5+
The Apache Software Foundation (http://www.apache.org/).
6+
7+
This product includes software developed at
8+
* Skymind Inc (Apache 2.0). Copyright (C) 2015-2018 Skymind Inc .
9+
10+
This product includes software developed at
11+
* Konduit KK (Apache 2.0). Copyright (C) 2020.
12+
13+
14+
This product includes software from the Tensorflow Project (Apache 2.0).
15+
* Copyright (C) 2015-2018 Tensorflow Authors.
16+
17+
# https://github.com/onnx/onnx
18+
19+
This product includes software from the Onnx Project project (Apache 2.0).
20+
* Copyright (C) 2020 Onnx Contributors (https://github.com/onnx/onnx)

‎README.md

+2-2
Original file line numberDiff line numberDiff line change
@@ -24,12 +24,12 @@ We do not monitor the github issues of this repository very often.
2424
The **Eclipse Deeplearning4J** (DL4J) ecosystem is a set of projects intended to support all the needs of a JVM based deep learning application. This means starting with the raw data, loading and preprocessing it from wherever and whatever format it is in to building and tuning a wide variety of simple and complex deep learning networks.
2525

2626
The DL4J stack comprises of:
27-
- **DL4J**: High level API to build MultiLayerNetworks and ComputationGraphs with a variety of layers, including custom ones. Supports importing Keras models from h5, including tf.keras models (as of 1.0.0-M1.1) and also supports distributed training on Apache Spark
27+
- **DL4J**: High level API to build MultiLayerNetworks and ComputationGraphs with a variety of layers, including custom ones. Supports importing Keras models from h5, including tf.keras models (as of 1.0.0-M2) and also supports distributed training on Apache Spark
2828
- **ND4J**: General purpose linear algebra library with over 500 mathematical, linear algebra and deep learning operations. ND4J is based on the highly-optimized C++ codebase LibND4J that provides CPU (AVX2/512) and GPU (CUDA) support and acceleration by libraries such as OpenBLAS, OneDNN (MKL-DNN), cuDNN, cuBLAS, etc
2929
- **SameDiff** : Part of the ND4J library, SameDiff is our automatic differentiation / deep learning framework. SameDiff uses a graph-based (define then run) approach, similar to TensorFlow graph mode. Eager graph (TensorFlow 2.x eager/PyTorch) graph execution is planned. SameDiff supports importing TensorFlow frozen model format .pb (protobuf) models. Import for ONNX, TensorFlow SavedModel and Keras models are planned. Deeplearning4j also has full SameDiff support for easily writing custom layers and loss functions.
3030
- **DataVec**: ETL for machine learning data in a wide variety of formats and files (HDFS, Spark, Images, Video, Audio, CSV, Excel etc)
3131
- **Arbiter**: Library for hyperparameter search
32-
- **LibND4J** : C++ library that underpins everything. For more information on how the JVM acceses native arrays and operations refer to [JavaCPP](https://github.com/bytedeco/javacpp)
32+
- **LibND4J** : C++ library that underpins everything. For more information on how the JVM accesses native arrays and operations refer to [JavaCPP](https://github.com/bytedeco/javacpp)
3333

3434
All projects in the DL4J ecosystem support Windows, Linux and macOS. Hardware support includes CUDA GPUs (10.0, 10.1, 10.2 except OSX), x86 CPU (x86_64, avx2, avx512), ARM CPU (arm, arm64, armhf) and PowerPC (ppc64le).
3535

‎android-examples/app/build.gradle

+1-1
Original file line numberDiff line numberDiff line change
@@ -48,7 +48,7 @@ android {
4848
}
4949

5050
dependencies {
51-
def dl4jVersion = '1.0.0-M1.1'
51+
def dl4jVersion = '1.0.0-M2'
5252
def openblasVersion = '0.3.10-1.5.5'
5353
def opencvVersion = '4.5.1-1.5.5'
5454
def leptonicaVersion = '1.80.0-1.5.5'

‎data-pipeline-examples/pom.xml

+31-3
Original file line numberDiff line numberDiff line change
@@ -27,12 +27,12 @@ information regarding copyright ownership.
2727

2828
<groupId>org.deeplearning4j</groupId>
2929
<artifactId>data-pipeline-examples</artifactId>
30-
<version>1.0.0-M1.1</version>
30+
<version>1.0.0-M2</version>
3131
<name>Building a data pipeline prior to modeling</name>
3232
<description>Loading raw data and processing it before training</description>
3333

3434
<properties>
35-
<dl4j-master.version>1.0.0-M1.1</dl4j-master.version>
35+
<dl4j-master.version>1.0.0-M2</dl4j-master.version>
3636
<!-- Change the nd4j.backend property to nd4j-cuda-X-platform to use CUDA GPUs -->
3737
<!-- <nd4j.backend>nd4j-cuda-10.2-platform</nd4j.backend> -->
3838
<nd4j.backend>nd4j-native</nd4j.backend>
@@ -42,7 +42,7 @@ information regarding copyright ownership.
4242
<exec-maven-plugin.version>1.4.0</exec-maven-plugin.version>
4343
<maven-shade-plugin.version>2.4.3</maven-shade-plugin.version>
4444
<logback.version>1.1.7</logback.version>
45-
<scala.binary.version>2.11</scala.binary.version>
45+
<scala.binary.version>2.12</scala.binary.version>
4646
<spark.version>2.4.8</spark.version>
4747
<tablesaw.version>0.38.5</tablesaw.version>
4848
<!-- Note: tablesaw needs a newer version of guava -->
@@ -62,6 +62,27 @@ information regarding copyright ownership.
6262
<updatePolicy>daily</updatePolicy> <!-- Optional, update daily -->
6363
</snapshots>
6464
</repository>
65+
<repository>
66+
<id>M2</id>
67+
<url>https://s01.oss.sonatype.org/content/repositories/orgdeeplearning4j-1019</url>
68+
<releases>
69+
<enabled>true</enabled>
70+
</releases>
71+
</repository>
72+
<repository>
73+
<id>M2-1</id>
74+
<url>https://s01.oss.sonatype.org/content/repositories/orgdeeplearning4j-1032</url>
75+
<releases>
76+
<enabled>true</enabled>
77+
</releases>
78+
</repository>
79+
<repository>
80+
<id>M2-2</id>
81+
<url>https://s01.oss.sonatype.org/content/repositories/orgdeeplearning4j-1034</url>
82+
<releases>
83+
<enabled>true</enabled>
84+
</releases>
85+
</repository>
6586
</repositories>
6687

6788

@@ -76,6 +97,13 @@ information regarding copyright ownership.
7697
</dependencyManagement>
7798

7899
<dependencies>
100+
<dependency>
101+
<groupId>org.deeplearning4j</groupId>
102+
<artifactId>resources</artifactId>
103+
<version>${dl4j-master.version}</version>
104+
</dependency>
105+
106+
79107
<dependency>
80108
<groupId>org.nd4j</groupId>
81109
<artifactId>${nd4j.backend}</artifactId>

‎dl4j-distributed-training-examples/pom.xml

+31-4
Original file line numberDiff line numberDiff line change
@@ -27,18 +27,18 @@ information regarding copyright ownership.
2727

2828
<groupId>org.deeplearning4j</groupId>
2929
<artifactId>dl4j-distributed-training-examples</artifactId>
30-
<version>1.0.0-M1.1</version>
30+
<version>1.0.0-M2</version>
3131
<name>Introduction to Distributed Training with DL4J</name>
3232
<description>A set of examples introducing distributed training with the DL4J framework</description>
3333

3434
<properties>
35-
<dl4j-master.version>1.0.0-M1.1</dl4j-master.version>
35+
<dl4j-master.version>1.0.0-M2</dl4j-master.version>
3636
<!-- Change the nd4j.backend property to nd4j-cuda-X-platform to use CUDA GPUs -->
3737
<!-- <nd4j.backend>nd4j-cuda-10.2-platform</nd4j.backend> -->
3838
<nd4j.backend>nd4j-native</nd4j.backend>
3939
<java.version>1.8</java.version>
4040
<shadedClassifier>bin</shadedClassifier>
41-
<scala.binary.version>2.11</scala.binary.version>
41+
<scala.binary.version>2.12</scala.binary.version>
4242
<maven-compiler-plugin.version>3.8.1</maven-compiler-plugin.version>
4343
<maven.minimum.version>3.3.1</maven.minimum.version>
4444
<exec-maven-plugin.version>1.4.0</exec-maven-plugin.version>
@@ -49,7 +49,7 @@ information regarding copyright ownership.
4949
<jcommander.version>1.27</jcommander.version>
5050
<azure.hadoop.version>2.7.4</azure.hadoop.version>
5151
<azure.storage.version>2.0.0</azure.storage.version>
52-
<spark.version>2.1.0</spark.version>
52+
<spark.version>2.4.8</spark.version>
5353
<aws.sdk.version>1.11.109</aws.sdk.version>
5454
<jackson.version>2.5.1</jackson.version>
5555
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
@@ -68,11 +68,38 @@ information regarding copyright ownership.
6868
<updatePolicy>daily</updatePolicy> <!-- Optional, update daily -->
6969
</snapshots>
7070
</repository>
71+
<repository>
72+
<id>M2</id>
73+
<url>https://s01.oss.sonatype.org/content/repositories/orgdeeplearning4j-1019</url>
74+
<releases>
75+
<enabled>true</enabled>
76+
</releases>
77+
</repository>
78+
<repository>
79+
<id>M2-1</id>
80+
<url>https://s01.oss.sonatype.org/content/repositories/orgdeeplearning4j-1032</url>
81+
<releases>
82+
<enabled>true</enabled>
83+
</releases>
84+
</repository>
85+
<repository>
86+
<id>M2-2</id>
87+
<url>https://s01.oss.sonatype.org/content/repositories/orgdeeplearning4j-1034</url>
88+
<releases>
89+
<enabled>true</enabled>
90+
</releases>
91+
</repository>
7192
</repositories>
7293

7394

7495

7596
<dependencies>
97+
<dependency>
98+
<groupId>org.deeplearning4j</groupId>
99+
<artifactId>resources</artifactId>
100+
<version>${dl4j-master.version}</version>
101+
</dependency>
102+
76103
<dependency>
77104
<groupId>org.apache.spark</groupId>
78105
<artifactId>spark-core_${scala.binary.version}</artifactId>

‎dl4j-distributed-training-examples/scripts/patentExamplePreproc.sh

+1-1
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,7 @@ MINIBATCH=32 #Minibatch size for preprocessed
3636

3737
#Other variables. Don't modify these
3838
SCRIPTDIR=$(dirname "$0")
39-
JARFILE=${SCRIPTDIR}/../target/dl4j-spark-1.0.0-M1.1-bin.jar
39+
JARFILE=${SCRIPTDIR}/../target/dl4j-spark-1.0.0-M2-bin.jar
4040
AZURE_ACCT=fs.azure.account.key.${AZURE_STORAGE_ACCT}.blob.core.windows.net
4141

4242

‎dl4j-distributed-training-examples/scripts/patentExampleTrain.sh

+1-1
Original file line numberDiff line numberDiff line change
@@ -44,7 +44,7 @@ AERON_BUFFER=33554432
4444

4545
#Other variables. Don't modify these
4646
SCRIPTDIR=$(dirname "$0")
47-
JARFILE=${SCRIPTDIR}/../target/dl4j-spark-1.0.0-M1.1-bin.jar
47+
JARFILE=${SCRIPTDIR}/../target/dl4j-spark-1.0.0-M2-bin.jar
4848
AZURE_ACCT=fs.azure.account.key.${AZURE_STORAGE_ACCT}.blob.core.windows.net
4949

5050

‎dl4j-distributed-training-examples/scripts/tinyImagenetTrain.sh

+1-1
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,7 @@ EXECUTOR_MEMORY=12G
3636
MASTER_PORT=7077 # Port for the spark master. Default is 7077
3737
MINIBATCH=32 # Minibatch size for preprocessed datasets
3838
SCRIPTDIR=$(dirname "$0")
39-
JARFILE=${SCRIPTDIR}/../target/dl4j-spark-1.0.0-M1.1-bin.jar
39+
JARFILE=${SCRIPTDIR}/../target/dl4j-spark-1.0.0-M2-bin.jar
4040

4141
#Memory
4242
OFFHEAP_MEM_JAVACPP=20G # Maximum amount of off-heap memory

‎dl4j-examples/pom.xml

+29-1
Original file line numberDiff line numberDiff line change
@@ -32,7 +32,7 @@ information regarding copyright ownership.
3232
<description>A set of examples introducing the DL4J framework</description>
3333

3434
<properties>
35-
<dl4j-master.version>1.0.0-M1.1</dl4j-master.version>
35+
<dl4j-master.version>1.0.0-M2</dl4j-master.version>
3636
<!-- Change the nd4j.backend property to nd4j-cuda-X-platform to use CUDA GPUs -->
3737
<!-- <nd4j.backend>nd4j-cuda-10.2-platform</nd4j.backend> -->
3838
<nd4j.backend>nd4j-native</nd4j.backend>
@@ -64,6 +64,27 @@ information regarding copyright ownership.
6464
<updatePolicy>daily</updatePolicy> <!-- Optional, update daily -->
6565
</snapshots>
6666
</repository>
67+
<repository>
68+
<id>M2</id>
69+
<url>https://s01.oss.sonatype.org/content/repositories/orgdeeplearning4j-1019</url>
70+
<releases>
71+
<enabled>true</enabled>
72+
</releases>
73+
</repository>
74+
<repository>
75+
<id>M2-1</id>
76+
<url>https://s01.oss.sonatype.org/content/repositories/orgdeeplearning4j-1032</url>
77+
<releases>
78+
<enabled>true</enabled>
79+
</releases>
80+
</repository>
81+
<repository>
82+
<id>M2-2</id>
83+
<url>https://s01.oss.sonatype.org/content/repositories/orgdeeplearning4j-1034</url>
84+
<releases>
85+
<enabled>true</enabled>
86+
</releases>
87+
</repository>
6788
</repositories>
6889

6990
<dependencies>
@@ -99,6 +120,13 @@ information regarding copyright ownership.
99120
<artifactId>deeplearning4j-core</artifactId>
100121
<version>${dl4j-master.version}</version>
101122
</dependency>
123+
124+
<dependency>
125+
<groupId>org.deeplearning4j</groupId>
126+
<artifactId>resources</artifactId>
127+
<version>${dl4j-master.version}</version>
128+
</dependency>
129+
102130
<dependency>
103131
<groupId>org.deeplearning4j</groupId>
104132
<artifactId>deeplearning4j-ui</artifactId>

‎dl4j-examples/src/main/java/org/deeplearning4j/examples/advanced/features/transferlearning/editlastlayer/presave/FitFromFeaturized.java

+1-3
Original file line numberDiff line numberDiff line change
@@ -23,8 +23,6 @@
2323
import org.deeplearning4j.nn.conf.distribution.NormalDistribution;
2424
import org.deeplearning4j.nn.conf.layers.OutputLayer;
2525
import org.deeplearning4j.nn.graph.ComputationGraph;
26-
import org.deeplearning4j.nn.modelimport.keras.exceptions.InvalidKerasConfigurationException;
27-
import org.deeplearning4j.nn.modelimport.keras.exceptions.UnsupportedKerasConfigurationException;
2826
import org.deeplearning4j.nn.transferlearning.FineTuneConfiguration;
2927
import org.deeplearning4j.nn.transferlearning.TransferLearning;
3028
import org.deeplearning4j.nn.transferlearning.TransferLearningHelper;
@@ -61,7 +59,7 @@ public class FitFromFeaturized {
6159
protected static final int numClasses = 5;
6260
protected static final int nEpochs = 3;
6361

64-
public static void main(String [] args) throws IOException, InvalidKerasConfigurationException, UnsupportedKerasConfigurationException {
62+
public static void main(String [] args) throws IOException {
6563

6664
//Import vgg
6765
//Note that the model imported does not have an output layer (check printed summary)

‎dl4j-examples/src/main/java/org/deeplearning4j/examples/advanced/features/transferlearning/iterators/FlowerDataSetIteratorFeaturized.java

+2-4
Original file line numberDiff line numberDiff line change
@@ -20,8 +20,6 @@
2020
package org.deeplearning4j.examples.advanced.features.transferlearning.iterators;
2121

2222
import org.deeplearning4j.examples.advanced.features.transferlearning.editlastlayer.presave.FeaturizedPreSave;
23-
import org.deeplearning4j.nn.modelimport.keras.exceptions.InvalidKerasConfigurationException;
24-
import org.deeplearning4j.nn.modelimport.keras.exceptions.UnsupportedKerasConfigurationException;
2523
import org.nd4j.linalg.dataset.AsyncDataSetIterator;
2624
import org.nd4j.linalg.dataset.ExistingMiniBatchDataSetIterator;
2725
import org.nd4j.linalg.dataset.api.iterator.DataSetIterator;
@@ -39,7 +37,7 @@ public class FlowerDataSetIteratorFeaturized {
3937

4038
private static String featureExtractorLayer = FeaturizedPreSave.featurizeExtractionLayer;
4139

42-
public static DataSetIterator trainIterator() throws UnsupportedKerasConfigurationException, IOException, InvalidKerasConfigurationException {
40+
public static DataSetIterator trainIterator() throws IOException {
4341
runFeaturize();
4442
DataSetIterator existingTrainingData = new ExistingMiniBatchDataSetIterator(new File("trainFolder"),"flowers-"+featureExtractorLayer+"-train-%d.bin");
4543
DataSetIterator asyncTrainIter = new AsyncDataSetIterator(existingTrainingData);
@@ -51,7 +49,7 @@ public static DataSetIterator testIterator() {
5149
return asyncTestIter;
5250
}
5351

54-
private static void runFeaturize() throws InvalidKerasConfigurationException, IOException, UnsupportedKerasConfigurationException {
52+
private static void runFeaturize() throws IOException {
5553
File trainDir = new File("trainFolder","flowers-"+featureExtractorLayer+"-train-0.bin");
5654
if (!trainDir.isFile()) {
5755
log.info("\n\tFEATURIZED DATA NOT FOUND. \n\t\tRUNNING \"FeaturizedPreSave\" first to do presave of featurized data");

0 commit comments

Comments
 (0)