In Java input streams, bytes are read sequentially and are discarded once they are read. This makes it impossible to seek backward in a stream unless a "mark" was placed at the desired position, in which case it's only possible to seek backward to that position. Not all streams support the "mark" feature, however; such streams cannot seek backward at all.
All Java input streams can seek forward by a specified number of bytes of the raw stream. However, this isn't very useful in audio contexts, especially those involving non-PCM audio (such as pretty much every compressed format), because the number of bits in a sample can vary, and if the audio is encoded in blocks, those blocks will likely have headers.
There's no such thing as absolute seeking in Java streams, nor is there any concept of absolute positions. The only "seek" methods are "jump forward by n bytes" and "jump backward to the mark", the latter of which is supported only by some streams.
And yes, the problem is specific to Vorbis SPI (which, I should state right now, is a plugin that requires no special handling to use, allowing me to use the same code to open OGG, WAV, AIFF, and other sampled-audio files, at least for music). With the audio types that Java natively supports, I can open them in a way that provides mark support (and SuperLemmini is already doing this), but doing the same with Vorbis SPI does not produce mark-supporting streams for some reason. Thus, my only options are to either use two files or write my own code that uses JOrbis, which is surprisingly very difficult to use.