In this comparison post, we will discuss the differences and similarities between two well known and famous neural network modelling AI(s) – PyTorch vs Google TensorFlow. We will first get to know each one of them. We will go through their strengths and weaknesses. Then we will do a one-on-one battle comprising of three rounds. In the first round, we will concentrate on the features and strengths of PyTorch.
In the second round, we will talk about TensorFlow and its strengths and how is PyTorch different from Tensorflow. Finally, I will present a conclusion and an opinion where we will together decide who wins – PyTorch or TensorFlow? or should it end into a draw? The Arena is ready. Dear Readers, you are the audience, start cheering as the competitors are entering the arena and let the match between the two greatest AI begin!
PyTorch is a Python-based observable computing bundle targeted at two circles of readers. It is an advanced version of NumPy which is able to use the power of GPUs. It is a deep learning analysis platform that provides best flexibility and agility (speed). PyTorch is currently managed by Adam Paszke, Sam Gross and Soumith Chintala.
What exactly is PyTorch?
Well to put in the words of its creators,
PyTorch gives GPU Tensors, Dynamic Neural Networks and deep Python integration.
It’s a Python original library and unlike others, it doesn’t work like C-Extensions, with an insignificant framework overhead. It blends with acceleration libraries such as Intel MKL and NVIDIA (CuDNN, NCCL) to maximise agility. PyTorch is similar to NumPy in the way that it manages computations, but has a strong GPU support. Similarly to NumPy, it also has the backend in programming language C, so they are both much faster than native Python libraries.
With some extra code, NumPy could be GPU accelerated, but it doesn’t have this strong GPU support that PyTorch or TensorFlow do. Lastly, PyTorch was specifically developed to introduce GPU functionality in Python.
Introducing Google TensorFlow
TensorFlow is a deep neural network, which learns to accomplish a task through assertive reinforcement and works within layers of nodes (data) to help it decide the precise result. Within Google’s AI organization, the researchers and engineers from the Google Brain team are originally behind TensorFlow, it originates with solid support for machine learning and deep learning and the adjustable numerical calculation core is used across various other scientific domains.
What exactly is TensorFlow?
TensorFlow was not specifically created for Python but written mainly in C++ and CUDA. CUDA is the NVIDIA’s very own language for programming GPUs. It provides functionalities in Google’s Language Go, Java, C++, C, and there is community support for Rust and Haskell. So, with TensorFlow, you are not restricted by Python. Even though the language syntax differs a bit, the concepts are the same.
PyTorch vs Google TensorFlow – The Machine vs Samaritan [Round 1]
Let us first talk about a popular new deep learning framework called PyTorch. The name is inspired by the popular torch deep learning framework which was written in the Lua programming language. Learning Lua is a big barrier to entry if you’re just starting to learn deep learning and it doesn’t offer the modularity necessary to interface with other libraries like a more accessible language would.
So a couple of AI researchers who were inspired by Torch’s programming style decided to implement it in Python calling it PyTorch. They also added a few other really cool features to the mix and we’ll talk about the two main ones. The first key feature of PyTorch is imperative programming. An imperative program performs computation as you type it.
While most python code is imperative, in this NumPy example we write four lines of code to ultimately compute the value for D. When the program executes C equals V times A, it runs the actual computation than in there just like you told it to. In contrast in a symbolic program, there is a clear separation between defining the computation graph and compiling it.
If we were to rewrite the same code symbolically then when C equals E times A is executed no computation occurs at that line. Instead, these operations generate a computation or symbolic graph and then we can convert the graph into a function that can be called at the compile step. So computation happens as the last step in the code. Old styles have their trade-offs, symbolic programs are more efficient since you can safely reuse the memory of your values for in-place computation.
TensorFlow is made to use a symbolic program. Imperative programs are more flexible since Python is most suited for them so you can use native Python features like printing out values in the middle of computation and injecting loops into the computation flow itself.
Download Detailed Curriculum and Get Complimentary access to Orientation Session
Time: 11:00 AM to 12:30 PM (IST/GMT +5:30)
PyTorch vs Google Tensor Flow – Almost Human [Round 2]
The second key feature of PyTorch is dynamic computation graphing as opposed to static computation graphing. In other words, PyTorch is defined by “run”, so at runtime, the system generates the graph structure.
TensorFlow is “define and run” where we define conditions and iterations in the graph structure. It’s like writing the whole program before running it, so the degree of freedom is limited. So in this, we define the computation graph once then we can execute that same graph many times. The great thing about this is that we can optimize the graph at the start. Let’s say in our model we want to use some kind of strategy for distributing the graph across multiple machines.
This kind of computationally expensive optimization can be reduced by reusing the same graph. Static graphs work well for neural networks that are fixed size like feed-forward networks or convolutional networks but for a lot of use cases, it would be useful if the graph structure could change depending on the input data like when using recurrent neural networks.
In this snippet, we’re using TensorFlow to unroll a recurrent network unit forward vectors. To do this we’ll need to use a special TensorFlow function called while loop. We have to use special nodes to represent primitives like loops and conditionals because any control flow statements will run only once when the graph is built. But a cleaner way to do this is to use dynamic graphs instead where the computation graph is built and rebuilt as necessary.
At runtime, the code is more straightforward since we can use the standard “for and if” statements. Any time the amount of work that needs to be done is variable, dynamic graphs are useful. Using dynamic graphs makes debugging really easy since a specific line in our written code is what fails as opposed to something deep under section don’t run.
Let’s build a simple two-layer neural network in PyTorch to get a feel for this impact. We start by importing our framework as well as the auto grab package which will let our network automatically implement back-propagation. Then we’ll define our batch size, input dimension, hidden dimension and output dimension. We will then use those values to help define tensors to hold inputs and outputs wrapping them in variables. We will set required gradients to false since we don’t need to compute gradients with respect to these variables during back-propagation.
The next set of variables will define our weights. We will initialize them as variables as well storing random tensors with the float data type and since we do want to compute gradients with respect to these variables we’ll set the flag to true. We’ll define a learning rate then we can begin our training loop for iterations. During the forward-pass, we can compute the predicted label using operations on our variables.
MM stands for matrix multiply and clamps all the elements in the input range into a range between min and max. Once we’ve matrix multiplied for both sets of weights to compute our prediction we can calculate the difference between them and square the sum of all the squared errors a popular loss function. Before we perform back-propagation we need to manually zero the gradients for both sets of weights since the great buffers have to be manually reset before fresh grades are calculated then we can run back-propagation by simply calling the backward function on our loss.
It will compute the gradient of our loss with respect to all variables we set requires gradient to true for and then we can update our ways using gradient descent and our outputs look great.
PyTorch vs Google TensorFlow – The Conclusion [Final Round]
To sum up, PyTorch offers two really useful features – dynamic computation graphs, an imperative programming dynamic computation graphs which are built and rebuilt as necessary at runtime and imperative programs perform computation as you run them. There is no distinction between defining the computation graph and compiling.
Right now TensorFlow has the best documentation on the web for a machine learning library so it’s still the best way for beginners to start learning and it’s best suited for production use since it was built with distributed computing in mind. But for researchers, it seems like PyTorch has a clear advantage here, a lot of cool new ideas will benefit and rely on the use of dynamic grasp.
TensorFlow is still more widely adopted because it has more capabilities and a better scalability for projects (Google scale large projects). PyTorch is certainly attaining momentum as it is easier to learn but does not have the equivalent integration. It is not optimal for product deployment but is very good for small projects that have short deadlines.
If you want to work in the enterprise, it is likely that an organization may have some customized framework. On your Resume, you will be expected to have experience with Machine learning. In any case, they won’t expect you to have worked with all of them. If you have to choose a single one of them, going for the easier one will not look good. So go with Google TensorFlow or learn both!
Excerpt, Inputs and Image credits:
Video presentation by Siraj Raval (Director at School of AI, a YouTube star and best-selling author)
Quora inputs by Iliya Valchanov (Co-founder of 365datascience)
Towards Data Science | https://towardsdatascience.com/pytorch-vs-tensorflow-spotting-the-difference-25c75777377b