Same nvJPEG2000, different numbers: timer boundaries and frames in flight

This is an excerpt from a longer write-up, Fastvideo JPEG2000 vs nvJPEG2000 on NVIDIA RTX 4090, which compares two GPU JPEG2000 codecs at the same compressed file size. The part below is about nvJPEG2000 alone: where the timer starts and stops, and how much the same library gives when it is fed differently. Disclosure: the author works at Fastvideo, which makes the other codec in that comparison. The harness, the scripts and the raw logs are open: github.com/fastvideo/jpeg2000-benchmark. The boundaries in the NVIDIA samples are drawn differently In NVIDIA's open sample set CUDALibrarySamples the measurement boundaries are drawn differently. Without this distinction, the numbers in the full article and those in NVIDIA's publications appear comparable, but they are not. The decoding sample. Frames are processed strictly one at a time: one decoder state, one queue of GPU jobs, and a wait after every frame. The -b option, described as a batch size, groups only the reading of files from disk and does not change how the work is done. The sample measures the running time of a single function, nvjpeg2kDecodeImage . The call is asynchronous: it puts the work into a GPU job queue and returns immediately, so a CPU clock cannot measure it. The sample handles that correctly. The time is taken with a pair of CUDA events on the same queue, before the call and after it, and that is the right way. There is nothing wrong with the measurement of the decoding itself. The question is about the second term. Parsing of the compressed image is done by nvjpeg2kStreamParse . This is Tier-2, which NVIDIA's own documentation calls the first stage of decoding. Its time is measured separately, with a CPU clock, and added to the total. Here is how that is done in nvjpeg2000DecodeSample.cpp (call arguments omitted, everything else verbatim): auto io_start = perfclock::now(); nvjpeg2kStreamParse(…); auto io_end = perfclock::now(); double parse_time = std::chrono::duration_cast (io_end - io_start).count(); … time += static_cast (loopTime / 1000.0); time += parse_time; The duration is converted to whole seconds, not to fractional ones. Anything shorter than a second becomes zero, and parsing a frame takes milliseconds. So time += parse_time always adds exactly zero. Allocating GPU buffers, reading the file and writing the result are not part of the measured time, and the finished frame is never copied back to host memory in the measured loop. This is not a choice of measurement boundary; it is an error in the measurement. What counts as part of the algorithm can be decided in more than one way, and that is a fair argument to have. Here the term is written in the code but is always zero for any frame on any hardware. How much is lost this way can be seen in the stage table in section 8: at decoding, Tier-2 takes from 15 % of the frame time at 2K to 29 % at 4K. Those are fvJPEG2000 shares, because the nvJPEG2000 library does not report time by stage and its own shares are unknown to us. But the stage is the same and runs on the CPU in the same way, so the magnitude is the same. That is why the nvJPEG2000 numbers in the full article do not come from that sample. A separate program was written for nvJPEG2000 (bench/nvj2k_bench-02/nvj2k_bench-02.cpp , section 14). Its timer starts before nvjpeg2kStreamParse and stops after decoding, once the GPU has finished, so Tier-2 is inside the measured time, exactly as it is for fvJPEG2000. Both sides are measured by the same rule; otherwise there is nothing to compare. Hence what this means for the reader. The number printed by the NVIDIA sample and the number in the full article cannot be put side by side: the first shows the time of a part of the algorithm, the second covers the whole of decoding. The first will always look better. The encoding sample. Frames go one at a time there as well. Here the whole per-frame loop is measured, and the copy of the compressed image to host memory (nvjpeg2kEncodeRetrieveBitstream ) is inside it. Loading the source frame onto the GPU stays outside, it is done when the file is read. That is exactly the boundary we use in single-frame mode, and there is nothing wrong with it. Several frames in flight: batching that nvJPEG2000 does not have An important caveat: batching works differently in the two codecs. This has to be said outright, otherwise the same word in the tables would mean two different things. First, about what the notation itself means. 8×2 is eight CPU threads, and in each of them two frames are in flight on the GPU at the same time. There are exactly eight CPU threads at any batch size; they do not double. Something else doubles - the number of jobs the GPU computes at the same moment: not eight, but sixteen. In fvJPEG2000 these two frames go into the codec in a single call: the batch is real, and the codec handles them as one job. This is a standard capability of Fastvideo SDK. nvJPEG2000 has no such call. Not a single function in the library accepts an array of images - only one image per call. So the GPU load is built up differently: each thread creates as many independent codec states and as many CUDA streams as the batch size specifies. The thread submits encoding of the first frame to its first stream, and immediately after it, without waiting for the result, the second frame to the second stream, and only then waits for both. The calls are asynchronous and the streams are independent, so both frames are computed on the GPU at the same time. All of this makes use of standard NVIDIA library and CUDA features: multiple codec states, job queues and asynchronous calls are all standard components of both. There are no workarounds here. The only thing missing from the library is a call that accepts several frames at once, so the order of the calls has to be built by hand. This is also worth saying because it does not work by itself. A program that simply calls nvJPEG2000 one frame per thread - and that is exactly how the NVIDIA samples are built - will get eight simultaneous jobs instead of sixteen, and the result will be lower. How much lower can be seen at one and the same number of threads: on a 2K lossy frame eight threads give the encoder 205 frames per second without the technique and 245 with a batch of two, that is 1.2 times more. For the decoder on that same task the starting point is not measured reliably (section 8), so take the neighboring one: on 4K lossy eight threads give 208 frames per second without the technique and 428 with four frames in flight - twice as much. We still report exactly these values and take them as the best for nvJPEG2000: the comparison must be against the maximum that can be obtained from the library, not against what the standard way of using it gives. The standard NVIDIA samples against our own harness All three columns are the same library doing the encoding - nvJPEG2000 from NVIDIA. Only two things differ: which program calls it, and how that program feeds the frames. On the left, the sample from the CUDALibrarySamples set: the program NVIDIA wrote for its own library and published itself, taken unchanged. In the middle, our own harness calling the same NVIDIA library and processing frames in the same order, one after another; it differs from the left column only in the code around nvJPEG2000 - reading the file, its own timer, the order of the calls. On the right, the same harness and the same NVIDIA library, but with several CPU threads and several frames in flight at once. The middle column tests the measurement program, not the codec. If the code we wrap around the NVIDIA library kept it from running at full speed, the middle column would come out below the left one - and then the right column would prove nothing, because the gain could be put down to the library never having had a fair chance. The NVIDIA sample was measured in both of the file-feeding modes it allows, and the better of its results went into the table. In either mode it processes frames strictly in order: one codec state, one CUDA stream, a synchronization after every step. Both encoders were given the same job, and that is checked rather than assumed. The standard NVIDIA encoder, run with the parameters of section 3.1, produced files of 601,940, 2,966,036, 1,274,517 and 8,964,924 bytes - byte for byte what our own harness produced. The time boundaries in this part are not the ones used in sections 6 and 7 of the full article, and the results differ accordingly. In all three columns the copying between host memory and GPU memory is outside the measured time: at encoding the source frame is already in GPU memory, at decoding the finished frame stays there. Otherwise the columns would not be comparable - the standard NVIDIA sample only counts what happens on the GPU. In section 7 the boundary is different: the time in multithreaded mode is counted from host memory to host memory, with both copies included. Everything on this chart is the same library doing the work - nvJPEG2000 from NVIDIA; only the calling program and the way frames are fed differ. The box on the right is how many times our harness with several frames in flight is faster than the same harness feeding frames one after another. The two halves have different scales: decoding 2K lossy reaches 1575 fps while encoding 4K lossless is at 66 fps, and on a common scale half the chart would be unreadable. The values are the ones in the tables below. Encoding, frames per second; the source frame is already in GPU memory | Frame and mode | NVIDIA sample | Our own harness | Our own harness | |---|---|---|---| | one frame at a time | one frame at a time | several frames at once | | | 2K, lossy | 197 | 199 | 291 | | 2K, lossless | 144 | 148 | 185 | | 4K, lossy | 125 | 128 | 171 | | 4K, lossless | 56 | 57 | 66 | The best combination of thread count and batch size in the right-hand column: 32×1 on 2K lossy and on both 4K frames, 16×2 on 2K lossless. The left and the middle column are 1 to 3 % apart.