It’s primarily used for audiobooks. Those need to be stored somewhere.
The podcast feature of Audiobookshelf uses the same structure, so the server downloads the file and the client can then find it and either download it to your local device or stream it from your server.

If you don’t want that, I wouldn’t use Audiobookshelf as an intermediary tool. I’d use Pocket Casts, or antennapod or some such.