I’m learning how to use Google’s TensorFlow framework to create artificial neural networks for deep learning with Python from Udemy. I’m using Jupiter notebook to practice and blog my learning progress here.

TensorFlow is an open source software library for numerical computation using data flow graphs. Nodes in the graph represent mathematical operations, while the graph edges represent the multidimensional data arrays (tensors) communicated between them. This architecture allows users to deploy computation to one or more CPUs or GPUs, in a desktop, server, or mobile device with a single API (Application programming interface).

# Install TensorFlow Environment

Download Anaconda Distribution for Python 3.6 Version. Tutorial comes in that page.

For MacOS: Open Terminal, use `cd`

to the target directory

Create the environment file: run `conda env create -f tfdl_env.yml`

Activate the file: run `source activate tfdeeplearning`

Now you are in the virtual environment of tfdeeplearning

If you want to get out of this, run `source deactivate`

**Note:** How is Anaconda related to Python?

Anaconda is a python and R distribution. It aims to provide everything you need (python wise) for data science “out of the box”.

It includes:

- The core python language
- 100+ python “packages” (libraries)
- Spyder (IDE/editor - like pycharm) and Jupyter
`conda`

, Anaconda’s own package manager, used for updating Anaconda and packages

Also Anaconda is used majorly for the data science. which manipulates large datasets based on statistical methods. ie. Many statistical packages are already available in anaconda libraries(packages)

Vanilla python installed from python.org comew with standard library is okay, in which case using pip to install manually (which comes with most python dists and you should have it if you downloaded from python.org).

Learn more: Anaconda overview; Python 3 tutorial

# TensorFlow Basic Syntax

1 | import tensorflow as tf |

Output: `1.3.0`

**Create a tensor ($\approx$ n-dimension array), the basic one (constant)**

1 | hello = tf.constant('Hello') |

Output: `tensorflow.python.framework.ops.Tensor`

**Run this operation inside of a session:**

1 | with tf.Session() as sess: |

Output: `b'HelloWorld'`

Note:

Use `with`

is to make sure we don’t close the session until we run a block of code then close the session.

The `b`

character prefix signifies that `HelloWorld`

is a byte string, use `result.decode('utf-8')`

to convert byte str to str.

1 | a = tf.constant(10) |

Output: `30`

**Numpy Operations:**

1 | const = tf.constant(10) # constant operation |

Note:`Interactive Session`

is particularly useful for Jupyter Notebook, it allows you to constantly call it through multiple cells.

In the Interactive Session, instead of using `sess.run(op)`

, we can use `op.eval()`

, it means “evaluate this operation”, and generates the same result.

**Matrix multiplication (common in neural networks)**

1 | a = tf.constant([[1,2],[3,4]]) |

Note: `sess.run(result)`

can also use `result.eval()`

instead.

# TensorFlow Graphs

Graphs are sets of connected *nodes*(vertices), and the connections are called *edges*. In TensorFlow each node is an operation with possible inputs that can supply some outputs. When TenserFlow is started, a default graph is created.

1 | print(tf.get_default_graph()) |

Output:

fixed default graph: `<tensorflow.python.framework.ops.Graph object at 0x11f4c92b0>`

Dynamic random graph: `<tensorflow.python.framework.ops.Graph object at 0x11f707dd8>`

1 | graph_one = tf.Graph() |

Output: `True`

1 | print(graph_one is tf.get_default_graph()) |

Output: `False`

# Variables and Placeholders

*Variables* and *Placeholders* are two main types of tensor objects in a Graph. During the optimization process, TensorFlow tunes the parameters of the model. *Variables* can hold the values of weights and biases throughout the session. ** Variables need to be initialized**.

*Placeholders*are initially empty and are used to feed in the actual training examples. However they do need a declared expected data type (tf.float32) with an optiional shape argument.

1 | ### Variable ### |

Output: ```
array([[ 0.27756679, 0.82726526, 0.80544853, 0.43891859],
[ 0.56279469, 0.57444489, 0.82595968, 0.63165414],
[ 0.16034544, 0.86095798, 0.74416387, 0.17536163],
[ 0.44427669, 0.69035304, 0.55842543, 0.00723565]], dtype=float32)
```

1 | ### Placeholder ### |

# TensorFlow Neural Network

Create a neuron that performs a simple linear fit to some 2-D data. Graph of $wx+b = z$

- Build a Graph
- Initiate the Session
- Feed Data In and get Output
- Add in the cost function in order to train the network to optimize the parameters
1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20import numpy as np

import tensorflow as tf

### Build a simple Graph ###

np.random.seed(101)

tf.set_random_seed(101)

rand_a = np.random.uniform(0,100, (5,5))

rand_b = np.random.uniform(0,100, (5,1))

a = tf.placeholder(tf.float32) # default shape

b = tf.placeholder(tf.float32) # default shape

# Two ways to do the operation:

#tf.add(a, b) | tf.multiply(a, b) | tf.matmul(a, b)

add_op = a + b

mul_op = a * b

with tf.Session() as sess:

add_result = sess.run(add_op, feed_dict = { a:rand_a, b:rand_b})

print(add_result, '\n')

mul_result = sess.run(mul_op, feed_dict = { a:rand_a, b:rand_b})

print(mul_result)

Output:

`[[ 151.07165527 156.49855042 102.27921295 116.58396149 167.95948792]

[ 135.45622253 82.76316071 141.42784119 124.22093201 71.06043243]

[ 113.30171204 93.09214783 76.06819153 136.43911743 154.42727661]

[ 96.7172699 81.83804321 133.83674622 146.38117981 101.10578918]

[ 122.72680664 105.98292542 59.04463196 67.98310089 72.89292145]]

[[ 5134.64404297 5674.25 283.12432861 1705.47070312 6813.83154297]

[ 4341.8125 1598.26696777 4652.73388672 3756.8293457 988.9463501 ]

[ 3207.8112793 2038.10290527 1052.77416992 4546.98046875 5588.11572266]

[ 1707.37902832 614.02526855 4434.98876953 5356.77734375 2029.85546875]

[ 3714.09838867 2806.64379883 262.76763916 747.19854736 1013.29199219]]`

1 | ### Build a Neural Network ### |

Output: `[[ 0.81314439 0.98195159 0.73793817]]`

1 | ### Simple Regression Example ### |

Now I want the neural network to solve $ y = mx + b$ with 10 training steps:

1 | # create two random variables |

# TensorFlow Regression

Regression example with huge dataset:

1 | import numpy as np |

For real cases, usually the datasets are humongous and we do not use all of them at once. Instead, we use batch method to feed the data batch by batch into the network to save the time. The number of batches depends on the size of the data. Here we feed 1000 batches of data, with each batch has 8 corresponding data points (x data points with corresponding y labels). To make batches useful, we grab 8 random data points via `rand_ind = np.random.randint(len(x_data), size = batch_size)`

, then train the optimizer.

1 | batch_size = 8 |

# TensorFlow Estimator API

This section is to solve the regression task with TensorFlow estimator API. Wait a minute, what is API? Technically, API stands for *Application Programming Interface*, but still, what is that? Basically, it is part of the server, that can be distinctively separated from its environment, that receives requests and sends responses. Here is an excellent article from Petr Gazarov explaining **API**.

There are many other higher levels of APIs (Keras, Layers…). The tf.estimator API has several model types to choose from:

- tf.estimator.LinearClassifier: Constructs a linear classification model
- tf.estimator.LinearRegressor: Constructs a linear regression model
- tf.estimator.DNNClassifier: Constructs a neural network classification model
- tf.estimator.DNNRegressor: Constructs a neural network regression model
- tf.estimator.DNNLinearCombinedRegressor: Constructs a neural network and linear combined regression model

Steps for using the Estimator API:

- Define a list of feature columns
- Create the Estimator Model
- Create a Data Input Function
- Call train, evaluate, and predict methods on the estimator object

1 | feat_cols = [tf.feature_column.numeric_column('x', shape = 1)] # set all feature columns |

Output:```
Training data matrics
{'loss': 8.7310658, 'average_loss': 1.0913832, 'global_step': 1000}
Test data matrics
{'loss': 8.6690454, 'average_loss': 1.0836307, 'global_step': 1000}
```

This is a good way to check if the model is overfitting (very low loss on training data but very high loss on test data). We want the loss of training data and test data are very close to each other.

1 | # Get predict value |

Output:```
[{'predictions': array([ 4.43396044], dtype=float32)},
{'predictions': array([ 5.06833887], dtype=float32)},
{'predictions': array([ 5.7027173], dtype=float32)},
{'predictions': array([ 6.33709526], dtype=float32)},
{'predictions': array([ 6.97147369], dtype=float32)},
{'predictions': array([ 7.60585213], dtype=float32)},
{'predictions': array([ 8.24023056], dtype=float32)},
{'predictions': array([ 8.87460899], dtype=float32)},
{'predictions': array([ 9.50898743], dtype=float32)},
{'predictions': array([ 10.14336586], dtype=float32)}]
```

1 | # plot the prediction |

# TensorFlow Classification

Use real dataset “Pima Indians Diabetes Dataset”, including both categorical and continuous features, to use tf.estimator switching models – from linear classifier to dense neural network classifier.

## Linear Classifier

1 | diabetes = pd.read_csv('pima-indians-diabetes.csv') |

1 | diabetes.columns |

Output: ```
Index(['Number_pregnant', 'Glucose_concentration', 'Blood_pressure', 'Triceps',
'Insulin', 'BMI', 'Pedigree', 'Age', 'Class', 'Group'],
dtype='object')
```

1 | # normalize the data except the catagorical part (Age, Class, Group) |

1 | # create feature columns |

The distribution of Age from the histgram shows that most of the people are 20s. And instead of treating this variable as a continuous variable, we can bucket these values together, making boundary for each decade or so. **Basically we use bucket system to transform continuous variable into categorial variable.**

1 | age_bucket = tf.feature_column.bucketized_column(age, boundaries = [20,30,40,50,60,70,80]) |

Create the model!

1 | ## Create the model |

Output:```
{'accuracy': 0.73160172,
'accuracy_baseline': 0.64935064,
'auc': 0.79658431,
'auc_precision_recall': 0.6441071,
'average_loss': 0.52852213,
'global_step': 1000,
'label/mean': 0.35064936,
'loss': 5.0870256,
'prediction/mean': 0.35784912}
```

The accuracy is 73.16%.

Prediction!

1 | # No y value in prediction, put new data set in x |

## Dense Neural Network Classifier

1 | # Create a three-layer neural network, with 10 neurons in each layer |

Output:```
{'accuracy': 0.75757575,
'accuracy_baseline': 0.64935064,
'auc': 0.82814813,
'auc_precision_recall': 0.67539084,
'average_loss': 0.49014509,
'global_step': 1000,
'label/mean': 0.35064936,
'loss': 4.7176466,
'prediction/mean': 0.36287409}
```