Linux – vi / vim abrumpt buffer movements within window

When using :set wrap in vi/vim, the contents of an entire wrapped line are placed in the display buffer upon moving the cursor into a new line, even when navigating through a file by display line (gj/gk) as opposed to real line (j/k). For example, if a line is wrapped over 5 screen lines and is currently outside the display buffer, the buffer will jump by 5 lines when navigating into this line to force the entire line to fit in the screen. Is there a way to disable this behavior? Note that set display+=lastline will allow the display of partial lines when a wrapped line is forced off the screen by navigating away from the line, but this does not resolve the buffer jumps that occur when navigating into a line that was initially outside the buffer.

Clarification: In case the above description is nebulous, the gist is that I would like to be able to scroll smoothly in vim with line wrapping enabled such that lines will be partially displayed off the screen rather than obligatorily being displayed in their entirety.