In the above example run_on_device and assign_func_pointer are the kernel names. This will output (among other things) a list of kernels by name, how many times each one was launched, and what percentage of overall GPU usage each one accounted for. While there are no tools which use macOS as a target environment, NVIDIA is making macOS host versions of these tools that you can launch profiling and debugging sessions on supported target platforms. What you're asking for is provided by the simplest usage of nvprof: NVIDIA CUDA Toolkit 11.0 no longer supports development or running applications on macOS. All buffers used for memcpy are pinned, I am sure. For each kernel, there is 256 threads in a block and 5 blocks in a grid.

You can see none operations in streams are overlapped. I attached the timeline of processing 10 tasks. The problem is that operations in different streams are not overlapping. In case you're not aware, there is both documentation and command-line help available ( nvprof -help). The main pipeline logic is in the following.

Cudalaunch nvprof