Distributed Tensorflow in Kubernetes - Revision history

Terry.lu at 02:17, 16 November 2018

2018-11-16T02:17:33Z

Terry.lu at 02:14, 16 November 2018

2018-11-16T02:14:05Z

Terry.lu at 00:41, 16 November 2018

2018-11-16T00:41:04Z

Terry.lu at 10:06, 15 November 2018

2018-11-15T10:06:50Z

Terry.lu: Created page with "== Introduce == Distributed Tensorflow (Clustering) can speed up your training. Distributed tensorflow in kubernates make it easy to: #Add k8s nodes to extend computing capa..."

2018-11-15T08:32:36Z

Created page with "== Introduce == Distributed Tensorflow (Clustering) can speed up your training. Distributed tensorflow in kubernates make it easy to: #Add k8s nodes to extend computing capa..."

New page

== Introduce ==

Distributed Tensorflow (Clustering) can speed up your training. Distributed tensorflow in kubernates make it easy to:

#Add k8s nodes to extend computing capability
#Simplify the work to make a distributed tensorflow

This topic will describe how to make a distributed tensorflow.

== Prerequisite ==

#You must know the basic concept of distributed tensorflow here: [https://www.tensorflow.org/deploy/distributed| Distributed TensorFlow]
#You must know how to write a distributed tensorflow training. Ex: [https://www.tensorflow.org/api_docs/python/tf/estimator/train_and_evaluate| train_and_evaluate]

== Steps ==
[[Category:Editor]]

@@ Line 1: / Line 1: @@
 == Introduce ==
-Distributed Tensorflow (Clustering) can speed up your training. Distributed tensorflow in kubernates make it easy to:
+This topic will describe how to make a distributed tensorflow.
-This topic will describe how to make a distributed tensorflow.
+== TFJob ==
 == Prerequisite ==
 # You must know the basic concept of distributed tensorflow here: [https://www.tensorflow.org/deploy/distributed| Distributed TensorFlow]
-#You must know the basic concept of distributed tensorflow here: [https://www.tensorflow.org/deploy/distributed| Distributed TensorFlow]
+# You must know how to write a distributed tensorflow training. Ex: [https://www.tensorflow.org/api_docs/python/tf/estimator/train_and_evaluate| train_and_evaluate]
 == Steps ==
 . Create(Download) source & Dockerfile [[File:Iris train and eval.zip]] and unzip to the same folder.
 . Create training container, where "ecgwc" is the username in dockerhub and "tf-iris:dist" is the container name
@@ Line 22: / Line 23: @@
 </syntaxhighlight>
-. Check if trainig docker is workable.<syntaxhighlight lang="bash">
+. Check if trainig docker is workable.
 $ docker run --rm ecgwc/tf-iris:dist
 </syntaxhighlight>
-[[File:Dist tf k8s-1.png|RTENOTITLE]]
+. Push docker to dockerHub
 <syntaxhighlight lang="bash">
 $ docker push ecgwc/tf-iris:dist
 </syntaxhighlight>
-. Create(Download) yaml file for distributed tensorflow: [[File:Tf-dist-iris.zip|RTENOTITLE]]
+. Create(Download) yaml file for distributed tensorflow: [[File:Tf-dist-iris.zip]]
-. Deploy yaml to k8s<syntaxhighlight lang="bash">
+. Deploy yaml to k8s
 $ kubectl create -f tf-dist-iris.yaml
 </syntaxhighlight>
-. Check training process
+. Check training status
 == Reference ==
 https://github.com/Azure/kubeflow-labs/tree/master/7-distributed-tensorflow

@@ Line 20: / Line 20: @@
 <syntaxhighlight lang="bash">
 $ docker build -t ecgwc/tf-iris:dist .
-</syntaxhighlight>3. Check if trainig docker is workable.<syntaxhighlight lang="bash">
+</syntaxhighlight>
 $ docker run --rm ecgwc/tf-iris:dist
-</syntaxhighlight>[[File:Dist tf k8s-1.png|RTENOTITLE]] 4. Push docker to dockerHub<syntaxhighlight lang="bash">
+</syntaxhighlight>
 $ docker push ecgwc/tf-iris:dist
-</syntaxhighlight> 5. Create(Download) yaml file for distributed tensorflow: [[File:Tf-dist-iris.zip|RTENOTITLE]]
+</syntaxhighlight>
 . Deploy yaml to k8s
-<syntaxhighlight lang="bash">
+. Create(Download) yaml file for distributed tensorflow: [[File:Tf-dist-iris.zip|RTENOTITLE]]
 $ kubectl create -f tf-dist-iris.yaml
-</syntaxhighlight> 7. Check training process
+</syntaxhighlight>
 [[File:Dist tf k8s-2.png|RTENOTITLE]]

@@ Line 15: / Line 15: @@
 == Steps ==
-. Create(Download) source & Dockerfile [[File:Iris train and eval.zip|RTENOTITLE]] and unzip to the same folder.
+. Create(Download) source & Dockerfile ([[File:Iris train and eval.zip|RTENOTITLE]]) and unzip to the same folder.
 . Create training container, where "ecgwc" is the username in dockerhub and "tf-iris:dist" is the container name
 <syntaxhighlight lang="bash">
 $ docker build -t ecgwc/tf-iris:dist .
-</syntaxhighlight>
+</syntaxhighlight>3. Check if trainig docker is workable.<syntaxhighlight lang="bash">
 $ docker run --rm ecgwc/tf-iris:dist
-</syntaxhighlight>[[File:Dist tf k8s-1.png|RTENOTITLE]]
+</syntaxhighlight>[[File:Dist tf k8s-1.png|RTENOTITLE]] 4. Push docker to dockerHub<syntaxhighlight lang="bash">
 $ docker push ecgwc/tf-iris:dist
-</syntaxhighlight>
+</syntaxhighlight> 5. Create(Download) yaml file for distributed tensorflow: [[File:Tf-dist-iris.zip|RTENOTITLE]]
 . Deploy yaml to k8s
 <syntaxhighlight lang="bash">
 $ kubectl create -f tf-dist-iris.yaml
-</syntaxhighlight>
+</syntaxhighlight> 7. Check training process
 [[File:Dist tf k8s-2.png|RTENOTITLE]]

← Older revision		Revision as of 10:06, 15 November 2018
Line 14:		Line 14:

	== Steps ==		== Steps ==
−	[[~~Category~~:~~Editor~~]]	+
		+	1. Create(Download) source & Dockerfile [[File:Iris train and eval.zip\|RTENOTITLE]] and unzip to the same folder.
		+
		+	2. Create training container, where "ecgwc" is the username in dockerhub and "tf-iris:dist" is the container name
		+	<syntaxhighlight lang="bash">
		+	$ docker build -t ecgwc/tf-iris:dist .
		+	</syntaxhighlight>
		+	3. Check if trainig docker is workable.
		+	<syntaxhighlight lang="bash">
		+	$ docker run --rm ecgwc/tf-iris:dist
		+	</syntaxhighlight>[[File:Dist tf k8s-1.png\|RTENOTITLE]]
		+	4. Push docker to dockerHub
		+	<syntaxhighlight lang="bash">
		+	$ docker push ecgwc/tf-iris:dist
		+	</syntaxhighlight>
		+	5. Create(Download) yaml file for distributed tensorflow: [[File:Tf-dist-iris.zip\|RTENOTITLE]]
		+
		+	6. Deploy yaml to k8s
		+	<syntaxhighlight lang="bash">
		+	$ kubectl create -f tf-dist-iris.yaml
		+	</syntaxhighlight>
		+	7. Check training process
		+
		+	[[File:Dist tf k8s-2.png\|RTENOTITLE]]
		+
		+	== Reference ==
		+
		+	[https://github.com/Azure/kubeflow-labs/tree/master/7-distributed-tensorflow https://github.com/Azure/kubeflow-labs/tree/master/7-distributed-tensorflow]