Variables¶
tf.Variables are used when a tensor's gradient have to be computed. These are often called parameters of an algorithm, that are updated after each training step.
The gradients are computed by automatic differentiation on a DAG
Consider the example of solving linear regression in one variable iteratively : (other methods of solving include - finding the second derivative to compute the learning rate and get a closed form solution, or by computing pseudo inverse)
Computing gradients:¶
Gradient of output with respect to a variable is the product of gradients along the path connecting these two nodes. If there are more than one path between the nodes (When least square is computed for a batch of inputs), the gradient is the sum of the results on all such paths
Placeholders¶
tf.placeholders are commonly used for inputs to the algorithm
import tensorflow as tf
tf.reset_default_graph()
The value of x is passed when running the session
x = tf.placeholder(tf.float32,name="input")
t = tf.placeholder(tf.float32,name="target")
#takes shape based on feed_dict unless the shape is explicitly specified
#Sometimes, we may want to fix certain dimension and the other dimension to be determined at run time
#placeholder to handle different catch size
batch = tf.placeholder(tf.float32,(None,5))
print(x)
print(t)
print(batch)
Variables¶
Variables have to defined with initial value
A = tf.Variable(<initial_value>, ...)
A = tf.Variable(1.0,tf.float32,name="A")
B = tf.Variable(0.0,tf.float32,name="B")
#Initializing a variable with tensor
tensor = tf.random_uniform((5,4),0,101)
tensor_variable = tf.Variable(tensor)
print(A)
print(B)
print(tensor_variable)
y = A*x + B
E = tf.reduce_sum(tf.square(tf.subtract(y,t)))
#if placeholder is a batch, then y is a vector by pythonbroadcasting rule
!rm tensorboard_logs/*
with tf.Session() as sess:
writer = tf.summary.FileWriter('./tensorboard_logs', sess.graph)
writer.close()
!tensorboard --logdir=tensorboard_logs/
Variable have to be initialized before running the graph
init = tf.global_variables_initializer()
with tf.Session() as sess:
sess.run(init)
...
You can also initialize specific variables with
init = tf.variable_initializer([A])
init = tf.global_variables_initializer()
sess = tf.Session()
sess.run(init)
Set up Optimizer¶
optimizer = tf.train.GradientDescentOptimizer(0.01)
train = optimizer.minimize(E)
print(train)
Train¶
for i in range(20):
sess.run(train,feed_dict={x:1,t:2})
#Get the value of the variables after each iteration
loss,a,b = sess.run([E,A,B],feed_dict={x:1,t:2})
print("Loss, A, B : ", loss,a,b)
Getting the gradients¶
The optimizer.minimize is actually a wrapper of two methods
optimizer.compute_gradients : With this gradient, custom regularizers can be applied
optimizer.apply_gradients
This format is used to manipulate the gradients
optimizer = tf.train.GradientDescentOptimizer(0.01)
grad_A, grad_B = optimizer.compute_gradients(E,[A,B])
print(grad_A[0]) #contains both grad
print(grad_A[1]) # and var
#Manipulate grads if needed
train = optimizer.apply_gradients([grad_A,grad_B])
print(train)
for i in range(20):
sess.run(train,feed_dict={x:1,t:2})
loss,a,b,g_a,g_b = sess.run([E,A,B,grad_A[0],grad_B[0]],feed_dict={x:1,t:2})
print("Loss, A, B, Grad_A, Grad_B : ", loss,a,b,g_a,g_b)
sess.close()
!rm tensorboard_logs/*
with tf.Session() as sess:
writer = tf.summary.FileWriter('./tensorboard_logs', sess.graph)
writer.close()
!tensorboard --logdir=tensorboard_logs/