I’m learning how to use Google’s TensorFlow framework to create artificial neural networks for deep learning with Python from Udemy. I’m using Jupiter notebook to practice and blog my learning progress here.

TensorFlow is an open source software library for numerical computation using data flow graphs. Nodes in the graph represent mathematical operations, while the graph edges represent the multidimensional data arrays (tensors) communicated between them. This architecture allows users to deploy computation to one or more CPUs or GPUs, in a desktop, server, or mobile device with a single API (Application programming interface).

# Install TensorFlow Environment

Download Anaconda Distribution for Python 3.6 Version. Tutorial comes in that page.
For MacOS: Open Terminal, use cd to the target directory
Create the environment file: run conda env create -f tfdl_env.yml
Activate the file: run source activate tfdeeplearning
Now you are in the virtual environment of tfdeeplearning
If you want to get out of this, run source deactivate

Note: How is Anaconda related to Python?
Anaconda is a python and R distribution. It aims to provide everything you need (python wise) for data science “out of the box”.
It includes:

• The core python language
• 100+ python “packages” (libraries)
• Spyder (IDE/editor - like pycharm) and Jupyter
• conda, Anaconda’s own package manager, used for updating Anaconda and packages

Also Anaconda is used majorly for the data science. which manipulates large datasets based on statistical methods. ie. Many statistical packages are already available in anaconda libraries(packages)

Vanilla python installed from python.org comew with standard library is okay, in which case using pip to install manually (which comes with most python dists and you should have it if you downloaded from python.org).
Learn more: Anaconda overview; Python 3 tutorial

# TensorFlow Basic Syntax

Output: 1.3.0

Create a tensor ($\approx$ n-dimension array), the basic one (constant)

Output: tensorflow.python.framework.ops.Tensor

Run this operation inside of a session:

Output: b'HelloWorld'
Note:
Use with is to make sure we don’t close the session until we run a block of code then close the session.
The b character prefix signifies that HelloWorld is a byte string, use result.decode('utf-8') to convert byte str to str.

Output: 30

Numpy Operations:

Note:
Interactive Session is particularly useful for Jupyter Notebook, it allows you to constantly call it through multiple cells.
In the Interactive Session, instead of using sess.run(op), we can use op.eval(), it means “evaluate this operation”, and generates the same result.

Matrix multiplication (common in neural networks)

\begin{align*} \begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix} \times \begin{bmatrix} 10 \\ 100 \end{bmatrix} = \begin{bmatrix} 210 \\ 430 \end{bmatrix} \end{align*}

Note: sess.run(result) can also use result.eval() instead.

# TensorFlow Graphs

Graphs are sets of connected nodes(vertices), and the connections are called edges. In TensorFlow each node is an operation with possible inputs that can supply some outputs. When TenserFlow is started, a default graph is created.

Output:
fixed default graph: <tensorflow.python.framework.ops.Graph object at 0x11f4c92b0>
Dynamic random graph: <tensorflow.python.framework.ops.Graph object at 0x11f707dd8>

Output: True

Output: False

# Variables and Placeholders

Variables and Placeholders are two main types of tensor objects in a Graph. During the optimization process, TensorFlow tunes the parameters of the model. Variables can hold the values of weights and biases throughout the session. Variables need to be initialized. Placeholders are initially empty and are used to feed in the actual training examples. However they do need a declared expected data type (tf.float32) with an optiional shape argument.

Output: array([[ 0.27756679, 0.82726526, 0.80544853, 0.43891859], [ 0.56279469, 0.57444489, 0.82595968, 0.63165414], [ 0.16034544, 0.86095798, 0.74416387, 0.17536163], [ 0.44427669, 0.69035304, 0.55842543, 0.00723565]], dtype=float32)

# TensorFlow Neural Network

Create a neuron that performs a simple linear fit to some 2-D data. Graph of $wx+b = z$

1. Build a Graph
2. Initiate the Session
3. Feed Data In and get Output
4. Add in the cost function in order to train the network to optimize the parameters

Output:
[[ 151.07165527 156.49855042 102.27921295 116.58396149 167.95948792]
[ 135.45622253 82.76316071 141.42784119 124.22093201 71.06043243]
[ 113.30171204 93.09214783 76.06819153 136.43911743 154.42727661]
[ 96.7172699 81.83804321 133.83674622 146.38117981 101.10578918]
[ 122.72680664 105.98292542 59.04463196 67.98310089 72.89292145]]

[[ 5134.64404297 5674.25 283.12432861 1705.47070312 6813.83154297]
[ 4341.8125 1598.26696777 4652.73388672 3756.8293457 988.9463501 ]
[ 3207.8112793 2038.10290527 1052.77416992 4546.98046875 5588.11572266]
[ 1707.37902832 614.02526855 4434.98876953 5356.77734375 2029.85546875]
[ 3714.09838867 2806.64379883 262.76763916 747.19854736 1013.29199219]]

Output: [[ 0.81314439 0.98195159 0.73793817]]

Now I want the neural network to solve $y = mx + b$ with 10 training steps:

# TensorFlow Regression

Regression example with huge dataset:

For real cases, usually the datasets are humongous and we do not use all of them at once. Instead, we use batch method to feed the data batch by batch into the network to save the time. The number of batches depends on the size of the data. Here we feed 1000 batches of data, with each batch has 8 corresponding data points (x data points with corresponding y labels). To make batches useful, we grab 8 random data points via rand_ind = np.random.randint(len(x_data), size = batch_size), then train the optimizer.

# TensorFlow Estimator API

This section is to solve the regression task with TensorFlow estimator API. Wait a minute, what is API? Technically, API stands for Application Programming Interface, but still, what is that? Basically, it is part of the server, that can be distinctively separated from its environment, that receives requests and sends responses. Here is an excellent article from Petr Gazarov explaining API.
There are many other higher levels of APIs (Keras, Layers…). The tf.estimator API has several model types to choose from:

• tf.estimator.LinearClassifier: Constructs a linear classification model
• tf.estimator.LinearRegressor: Constructs a linear regression model
• tf.estimator.DNNClassifier: Constructs a neural network classification model
• tf.estimator.DNNRegressor: Constructs a neural network regression model
• tf.estimator.DNNLinearCombinedRegressor: Constructs a neural network and linear combined regression model

Steps for using the Estimator API:

• Define a list of feature columns
• Create the Estimator Model
• Create a Data Input Function
• Call train, evaluate, and predict methods on the estimator object

Output:
Training data matrics {'loss': 8.7310658, 'average_loss': 1.0913832, 'global_step': 1000} Test data matrics {'loss': 8.6690454, 'average_loss': 1.0836307, 'global_step': 1000}
This is a good way to check if the model is overfitting (very low loss on training data but very high loss on test data). We want the loss of training data and test data are very close to each other.

Output:
[{'predictions': array([ 4.43396044], dtype=float32)}, {'predictions': array([ 5.06833887], dtype=float32)}, {'predictions': array([ 5.7027173], dtype=float32)}, {'predictions': array([ 6.33709526], dtype=float32)}, {'predictions': array([ 6.97147369], dtype=float32)}, {'predictions': array([ 7.60585213], dtype=float32)}, {'predictions': array([ 8.24023056], dtype=float32)}, {'predictions': array([ 8.87460899], dtype=float32)}, {'predictions': array([ 9.50898743], dtype=float32)}, {'predictions': array([ 10.14336586], dtype=float32)}]

# TensorFlow Classification

Use real dataset “Pima Indians Diabetes Dataset”, including both categorical and continuous features, to use tf.estimator switching models – from linear classifier to dense neural network classifier.

## Linear Classifier

Output: Index(['Number_pregnant', 'Glucose_concentration', 'Blood_pressure', 'Triceps', 'Insulin', 'BMI', 'Pedigree', 'Age', 'Class', 'Group'], dtype='object')

The distribution of Age from the histgram shows that most of the people are 20s. And instead of treating this variable as a continuous variable, we can bucket these values together, making boundary for each decade or so. Basically we use bucket system to transform continuous variable into categorial variable.

Create the model!

Output:
{'accuracy': 0.73160172, 'accuracy_baseline': 0.64935064, 'auc': 0.79658431, 'auc_precision_recall': 0.6441071, 'average_loss': 0.52852213, 'global_step': 1000, 'label/mean': 0.35064936, 'loss': 5.0870256, 'prediction/mean': 0.35784912}
The accuracy is 73.16%.

Prediction!

## Dense Neural Network Classifier

Output:
{'accuracy': 0.75757575, 'accuracy_baseline': 0.64935064, 'auc': 0.82814813, 'auc_precision_recall': 0.67539084, 'average_loss': 0.49014509, 'global_step': 1000, 'label/mean': 0.35064936, 'loss': 4.7176466, 'prediction/mean': 0.36287409}

0%