This post will provide some troubleshooting resources if you find yourself with this
UnknownError when training an algorithm:
Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
- Windows 10
- Nvidia RTX GPU
- Python 3.8
- Tensorflow 2.4.1 with GPU support
There was a moment where when running my training of a tensorflow model I got the following error:
UnknownError: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[node cnn/conv2d/Relu (defined at c:...\model_def\cnn.py:35) ]] [Op:__inference_train_function_625528] Function call stack: train_function
There was no warning log, so to troubleshoot I tried to reload the kernel, see if I had another python application that used the GPU and even restarted the computer. The former things did not work in a reliable way. In some cases was that there was another python/tensorflow instance active, but other times this was not the case and a restart worked partially.
Looking back to the error message, cuDNN is one of the Nvidia libraries you want to install in order to use more efficiently make use of your graphics card when running deep learning frameworks (see https://developer.nvidia.com/cudnn)
In the end, I found out that one of the applications I was using to tweak my fan speed looked like it was using cuda cores and thus Tensorflow was not able to allocate all the cuda capacity to it, resulting in the error found above. Closing the application fixed the error.
Let me know if this helps you when facing this issue, or if you have a better insight on the real error cause.
---> 30 self._history = self.model.fit( 31 training_data.train_generator, 32 epochs=epochs, steps_per_epoch=len(training_data.train_generator)-1, ~\AppData\Roaming\Python\Python38\site-packages\tensorflow\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_batch_size, validation_freq, max_queue_size, workers, use_multiprocessing) 1098 _r=1): 1099 callbacks.on_train_batch_begin(step) -> 1100 tmp_logs = self.train_function(iterator) 1101 if data_handler.should_sync: 1102 context.async_wait() ~\AppData\Roaming\Python\Python38\site-packages\tensorflow\python\eager\def_function.py in __call__(self, *args, **kwds) 826 tracing_count = self.experimental_get_tracing_count() 827 with trace.Trace(self._name) as tm: --> 828 result = self._call(*args, **kwds) 829 compiler = "xla" if self._experimental_compile else "nonXla" 830 new_tracing_count = self.experimental_get_tracing_count() ~\AppData\Roaming\Python\Python38\site-packages\tensorflow\python\eager\def_function.py in _call(self, *args, **kwds) 886 # Lifting succeeded, so variables are initialized and we can run the 887 # stateless function. --> 888 return self._stateless_fn(*args, **kwds) 889 else: 890 _, _, _, filtered_flat_args = \ ~\AppData\Roaming\Python\Python38\site-packages\tensorflow\python\eager\function.py in __call__(self, *args, **kwargs) 2940 (graph_function, 2941 filtered_flat_args) = self._maybe_define_function(args, kwargs) -> 2942 return graph_function._call_flat( 2943 filtered_flat_args, captured_inputs=graph_function.captured_inputs) # pylint: disable=protected-access 2944 ~\AppData\Roaming\Python\Python38\site-packages\tensorflow\python\eager\function.py in _call_flat(self, args, captured_inputs, cancellation_manager) 1916 and executing_eagerly): 1917 # No tape is watching; skip to running the function. -> 1918 return self._build_call_outputs(self._inference_function.call( 1919 ctx, args, cancellation_manager=cancellation_manager)) 1920 forward_backward = self._select_forward_and_backward_functions( ~\AppData\Roaming\Python\Python38\site-packages\tensorflow\python\eager\function.py in call(self, ctx, args, cancellation_manager) 553 with _InterpolateFunctionError(self): 554 if cancellation_manager is None: --> 555 outputs = execute.execute( 556 str(self.signature.name), 557 num_outputs=self._num_outputs, ~\AppData\Roaming\Python\Python38\site-packages\tensorflow\python\eager\execute.py in quick_execute(op_name, num_outputs, inputs, attrs, ctx, name) 57 try: 58 ctx.ensure_initialized() ---> 59 tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name, 60 inputs, attrs, num_outputs) 61 except core._NotOkStatusException as e: UnknownError: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[node cnn/conv2d/Relu (defined at c:...\model_def\cnn.py:35) ]] [Op:__inference_train_function_625528] Function call stack: train_function