Inside TensorFlow: tf.data + tf.distribute

Рет қаралды 21,881

Күн бұрын

In this episode of Inside TensorFlow, Software Engineer Jiri Simsa gives us the best practices for tf.data and tf.distribute. Let us know what you think about this presentation in the comments below!
tf.data: Build TensorFlow input pipelines → goo.gle/2VTnnjk
Better performance with the tf.data API → goo.gle/38wyKAy
Distributed training with TensorFlow → goo.gle/39wMWdY
Watch more from Inside TensorFlow Playlist → goo.gle/Inside-TensorFlow
Subscribe to the TensorFlow channel → goo.gle/TensorFlow

Пікірлер: 8

@bapireddy5790 4 жыл бұрын

Great talk on tf.data kudos to the engineers for making these high level api's

@zytriesthings4540 Күн бұрын

My mind just exploded

@prathameshdinkar2966 Жыл бұрын

Is synchronous training with 2 devices is similar to just doubling the batch size on single device?

@haneulkim4902 Жыл бұрын

This is an amazing talk! Thank you. I have two questions. 1. @9:22 I understand that it reads all elements tfrecords files sequentially. but since map function is applied before batch, how many elements are loaded? -> one by one? 2. @27:25 so after gradients from different GPU have been for example averaged it then gets passed back to each GPU to update the model? and this process is repeated?

@tanyachutani7272 3 жыл бұрын

Informative talk

@shaunsun9076 4 жыл бұрын

In fact, a little confused about the pipeline. Sometimes we consider it a kind of inputting data method, but sometimes it shows a way of training acceleration in distributed parallel strategies. How to think about it? Thx

@RobertLugg 4 жыл бұрын

I like to think about the purpose: Read data from somewhere. Perhaps perform some transformation. Make the data available to keras.fit(). The "transform" is going from the original format to the correct format that the model needs. One common example is to convert RGB images, where each pixel is represented by three numbers with a range of 0 to 255. The transform might be a division by 255 which will convert each pixel to be three numbers with a range of 0 and 1. This function is specified within the pipeline (the function .map) calls. Now, when reading and making the data available there is a lot of wasted CPU time. So, instead, while the system is waiting, it does the transform for another batch. It doesn't really accelerate training. However, it makes sure that the data in process isn't delaying the training process. I think the best way to think about it is to call it the data input pipeline. Also during inference, you run data through the same input data pipeline so that the same transforms are applied.