commit aa4909edda563601024eeb6fe0cef6ca0a0bfba6
parent 306bd1e7e000e882c583cd347abaf140d507492f
Author: David Freifeld <freifeld.david@gmail.com>
Date: Wed, 5 Aug 2020 18:00:42 -0700
Update to reflect future plans
Diffstat:
2 files changed, 24 insertions(+), 35 deletions(-)
diff --git a/readme.md b/readme.md
@@ -3,16 +3,28 @@
<!-- Jacobian -->
<!-- Created by David Freifeld -->
- <!-- Copyright © 2020 David Freifeld. All rights reserved. -->
## About
-Jacobian is a work-in-progress machine learning library written in C++ designed to run as fast as possible while still being simple to use. Jacobian is accessible via Python and enables you to write models that train faster in the same amount of code. As of now, Jacobian supports feedforward neural networks and has partial support for convolutional neural networks. ***Note that as Jacobian is a work-in-progress, the latest commit is and will be largely unstable until convolutional networks and multiclass classification are fully implemented.***
+Jacobian is a work-in-progress machine learning library written in C++ designed to run as fast as possible while still being simple to use. Jacobian is accessible via Python and enables you to write models that train faster with the same amount of code. As of now, Jacobian supports feedforward neural networks and has partial support for convolutional neural networks. ***Note that as Jacobian is a work-in-progress, the latest commit is and will be largely unstable until this README specifies otherwise.***
-## Benchmark Info
+## The Chopping Block
+What's next for Jacobian?
-One of the tradeoffs of Jacobian is that as of now it doesn't train nearly as close to perfection as other available libraries and doesn't maximize accuracy as much (with the benefit being the added speed. Here's a graph of Jacobian's model metrics over epochs on a simple task (banknote dataset with batch size 16) as compared to other libraries.
+1. **Full AVX support.** This will make time-consuming parts of the code like activation functions (most notably softmax) and coefficient-wise multiplications 2-4x faster. This will only work on processors supporting AVX instructions.
+
+2. **Usable CNNs.** CNNs have been supported for a bit now, but in their current state have not been very practical or stable. Fixes and additions to the CNN backprop, proper pooling layer support, and tensors will enable CNNs to behave more like their stable ANN brethren. Expect small CNN-oriented features as well.
+
+3. **Parallelism.** Running multiple networks in parallel was once supported but has since stopped working. This feature plus the ability to paralellize a singular network will finally allow Jacobian to live up to its original purpose of being a parallel ML library.
+
+**The farther future** may hold initial implementations of RNN-like architectures, GPU support/optimizations, transfer learning, and autodifferentiation a la PyTorch.
+
+**Some smaller features** coming up may include a proper Python install via `pip`, more complex gradient descent optimizers, proper sanity/gradient checks, and more in-depth benchmarks.
+
+## Benchmarks
+
+One of the tradeoffs of Jacobian is that as of now it doesn't train nearly as close to perfection as other available libraries nor does it maximize accuracy as much (with the benefit being the added speed). Here's a graph of Jacobian's model metrics over epochs on a simple task (banknote dataset with batch size 16) as compared to other libraries.
@@ -61,12 +73,12 @@ Next, call `initialize()` to initialize the network's weights.
Finally, train your network for one epoch with `train()`. Training one epoch at a time allows you control over the accuracy reporting (with functions like `get_cost()`, `get_accuracy()`, `get_val_cost()`, and `get_val_accuracy()`) and also allows effective use of services like W&B.
### Examples
-In the `/scripts` directory there is a example of a neural network being used in conjuction with Weights & Biases, allowing for effective hyperparameter searches and accuracy reporting.
+In the `/scripts` directory there is an example of a neural network being used in conjuction with Weights & Biases, allowing for effective hyperparameter searches and accuracy reporting.
-## Installing
+## Building
### Main Steps
-Note: This is all ideally the process, but it's so confusing that I'm not sure. You're better off manually copying the .so file!
+Note: This is all ideally the process for manual building, but it's so confusing that I'm not sure. You're better off manually copying the .so file!
1. Install the C++ library Eigen.
2. Install the C++ and python ends of the pybind11 library.
3. Run `make` with the configuration of your choosing.
@@ -76,50 +88,28 @@ Note: This is all ideally the process, but it's so confusing that I'm not sure.
### Build Configurations (Building from Source)
-There are 5 build configurations, each one prioritizing program speed more than the last. *Warning!* The higher levels (>fast) are as of now unstable as changes to the code have caused it to not play nicely with MKL or icpc.
+There are 5 build configurations, each one prioritizing program speed more than the last. **NOTE:** These options change very frequently as I experiment, so this describes the features that have remained generally invariant.
#### Level 1: `make`
Simply builds the project with no optimization at all. Use this if you don't want to wait long for the library to compile and don't care too much about speed in the moment.
#### Level 2: `make fast`
- - Builds the project with the O3 optimization setting, essentially the highest optimization configuration without manually passing optimization flags (Ofast can prove slower).
+ - Builds the project with the O3 optimization setting.
Use this if you want some speed but shorter build commands and compile times.
#### Level 3: `make faster`
-Enables a whole slew of extra optimizations, some of which include:
+Enables extra optimizations, some of which include:
- Building the project with O3.
- Optimizing for native architecture.
- - Instructing compiler to fetch data for CPU cache earlier.
- - Unrolling loops.
- - Links with Intel's Math Kernel Library (make sure you have this!). Provides an extra boost to speed.
Use this if you care a lot about speed but are not willing to sacrifice anything but compile time for it.
#### Level 4: `make tradeoffs`
-Builds the project with O3, specific optimization flags, and *more* specific optimization flags that sacrifice things like portability and precision as well as makes assumptions to increase speed even further. Use this if you care about speed more than precision and are willing to make some tradeoffs (and also don't mind longer compile times).
-
Some of the new compiler optimizations include:
- - Using an optimized version of `calloc`.
- - Using Intel-specific optimizations.
- - Allowing low-precision alternatives to operations like sqrt and division.
- Enabling the `-ffast-math` flag.
- - Approximating more complicated functions.
In the future this setting may try to parallelize operations (even if the network is already parallelized with MapReduce).
#### Level 5: `make reckless`
-This option is not implemented as of now.
-
-Planned features include:
- - Adding all the aforementioned compiler options as well as compiler options that are potentially unsafe.
- - Defining the `RECKLESS` macro which will skip anything that is not absolutely necessary in the code (with preprocessor statements like `#ifndef`). Will skip all sort of testing and checking before training.
-
-## The Future
-
-Jacobian is actively in development and the following are things that are planned for the nearish future:
-- More architectures such as Convolutional Neural Networks and RNN-like architectures (LSTM, GRU...).
-- Moving towards a more proper release by emphasizing usability.
-- Further increasing speedups from a conceptual perspective with better algorithms, a implementation perspective with optimized code, and a low-level perspective with hardware optimizations + more compiler work.
-- More advanced capabilities such as the inclusion of gradient descent optimizations.
-- Parallelization, data-oriented design, and more ways of increasing usability+speed.
+- Defines the `RECKLESS` macro which skips checks anywhere in the program.
diff --git a/src/utils.cpp b/src/utils.cpp
@@ -99,8 +99,7 @@ Eigen::MatrixXf avx_product(Eigen::MatrixXf a, Eigen::MatrixXf b)
float arr2[(((b.rows() * b.cols()) % 8) * 8) + 8];
memcpy(arr1, b.data(), sizeof(float)*b.cols()*a.rows());
for (int i = 0; i < (((a.rows() * a.cols()) % 8) * 8) + 8; i++) {
- _mm256_store_ps(arr1, _mm256_mul_ps(_mm256_load_ps(arr1+i*8),
- _mm256_load_ps(arr2+i*8)));
+ _mm256_store_ps(arr1, _mm256_mul_ps(_mm256_load_ps(arr1+i*8), _mm256_load_ps(arr2+i*8)));
}
Eigen::Map<Eigen::MatrixXf> dst (arr1, a.rows(), a.cols());
return dst;
