Commit Graph

359 Commits

Author SHA1 Message Date
Andreas Rheinhardt
c33e56c7a6 avformat/aviobuf: Also return truncated buffer in avio_get_dyn_buf()
Two kinds of errors can happen when working with dynamic buffers:
(Re)allocation errors or truncation errors (one has to truncate the
buffer to a size of INT_MAX because avio_close_dyn_buf() and
avio_get_dyn_buf() both return an int). Right now, avio_get_dyn_buf()
returns an empty buffer in either case. But given that
avio_get_dyn_buf() does not destroy the dynamic buffer, one can return
the buffer in case of truncation and let the user check the error flags
and decide for himself instead of hardcoding a single way to proceed
in case of truncation.

(This actually restores the behaviour from before commit
163bb9ac0af495a5cb95441bdb5c02170440d28c.)

Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
2020-06-11 04:03:38 +02:00
Andreas Rheinhardt
7be9b0bb38 avformat/aviobuf: Return better error codes
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
2020-06-11 03:29:22 +02:00
Andreas Rheinhardt
fa0bc627c5 avformat/aviobuf: Stop restricting dynamic buffer sizes to INT_MAX/2
This has originally been done in 568e18b15e
as a precaution against integer overflows, but it is actually easy to
support the full range of int without overflows.

Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
2020-06-11 03:09:56 +02:00
Andreas Rheinhardt
88d5ae068f avformat/aviobuf: Simplify dyn_buf_write() a bit
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
2020-06-11 02:01:56 +02:00
Andreas Rheinhardt
28a078eded avformat/aviobuf: Don't check for overflow after it happened
If adding two ints overflows, it doesn't matter whether the result will
be stored in an unsigned or not; and checking afterwards does not make it
retroactively defined.

Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
2020-06-11 01:18:54 +02:00
Anton Khirnov
2cbd544519 AVIOContext: switch to child_class_iterate() 2020-06-10 12:36:44 +02:00
Andreas Rheinhardt
781c7a6217 avformat/aviobuf, nutenc: Move ff_puv_v, ff_get_v_length to nutenc.c
and make it static again.

These functions have been moved from nutenc to aviobuf and internal.h
in f8280ff4c0 in order to use them in a
forthcoming patch in utils.c. Said patch never happened, so this commit
moves them back and makes them static, effectively reverting said
commit as well as f8280ff4c0 (which added
the ff-prefix to these functions).

Reviewed-by: Michael Niedermayer <michael@niedermayer.cc>
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
2020-05-05 19:28:28 +02:00
Andreas Rheinhardt
639728f51a avformat/aviobuf: Add function to reset dynamic buffer
Resetting a dynamic buffer means to keep the AVIOContext and the
internal buffer used by the dynamic buffer. This is done in order to
save (re)allocations when one has a workflow where one opens and closes
dynamic buffers in sequence.

Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
2020-03-30 05:50:49 +02:00
Paul B Mahol
d64cbd4fda remove CHAR_MIN/CHAR_MAX usage
It is not needed at all.
2020-03-17 22:46:36 +01:00
Andreas Rheinhardt
6e8e8431e1 avformat/aviobuf: Remove AVIOInternal and one level of indirection
In the Libav commit cae448cf, the opaque of every AVIOContext opened
by ffio_fdopen() (which is used internally by avio_open() and avio_open2())
changed: It was a simple pointer to an URLContext before, but now it was
a structure (namely AVIOInternal) containing a pointer to an URLContext
as its only member. The next commits (namely 8c0ceafb and ec4c4839) added
members to AVIOInternal to allow white-/blacklisting of protocols.

But these two commits were never merged into FFmpeg (they were only
merged as no-ops in 510046c2 and 063b26d3), because FFmpeg chose
a different way to implement this (in 93629735); and so our AVIOInternal
still has exactly one member.

This of course means that it is unnecessary to use AVIOInternal as
opaque as it is just adding a level of indirection (not only pointer
dereference, but also wrapper functions). Therefore this commit
removes AVIOInternal entirely and essentially reverts cae448cf.

Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
Reviewed-by: Paul B Mahol <onemda@gmail.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2020-01-30 18:47:21 +01:00
Andreas Rheinhardt
220846f74f avformat/aviobuf: Honor avio_open[2] documentation
The documentation of both avio_open() as well as avio_open2() states
that on failure, the pointer to an AVIOContext given to this function
(via a pointer to a pointer to an AVIOContext) will be set to NULL. Yet
it didn't happen upon failure of ffurl_open_whitelist() or when allocating
the internal buffer failed. This commit changes this.

Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2020-01-07 02:44:05 +01:00
Andreas Rheinhardt
163bb9ac0a avformat/aviobuf: Avoid allocating buffer when using dynamic buffer
Up until now, using a dynamic buffer entailed at least three
allocations: One for the AVIOContext, one for the AVIOContext's opaque
(which, among other things, contains the small write buffer), and one
for the big buffer that is independently allocated that is returned when
calling avio_close_dyn_buf().

It is possible to avoid the third allocation if one doesn't use a
packetized dynamic buffer, if all the data written so far fit into the
write buffer and if one does not require the actual (big) buffer to have
an indefinite lifetime. This is done by making avio_get_dyn_buf() return
a pointer to the data in the write buffer if nothing has been written to
the main buffer yet. The dynamic buffer will then be freed using
ffio_free_dynamic_buffer (which needed to be modified not to call
avio_close_dyn_buf() internally).

So a typical use-case like:

size = avio_close_dyn_buf(dyn_pb, &buf);
do something with buf
av_free(buf);

can be converted to:

size = avio_get_dyn_buf(dyn_pb, &buf);
do something with buf
ffio_free_dynamic_buffer(&dyn_pb);

In more complex scenarios this can simplify freeing as well, because it
is now clear that freeing always has to be performed via
ffio_free_dynamic_buffer().

Of course, in case this saves an allocation it also saves a memcpy.

Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
2019-11-28 15:20:36 -03:00
Marton Balint
95fa73a2b4 avformat/avio: remove 4k limit from avio_printf
We do this by switching to AVBPrint.

v2: Also set IO context error flag in case of ENOMEM.

Signed-off-by: Marton Balint <cus@passwd.hu>
2019-08-17 18:39:49 +02:00
Marton Balint
a82f8f2f10 avformat/avio: add avio_print_string_array and avio_print
These functions can be used to print a variable number of strings consecutively
to the IO context. Unlike av_bprintf, no temporary buffer is necessary.

Signed-off-by: Marton Balint <cus@passwd.hu>
2019-08-17 18:39:49 +02:00
tomajsjiang
3d1506c630 lavf/avio: add a ffio_realloc_buf API for AVIO buffer realloc
Add new API ffio_realloc_buf for AVIO buffer realloc.

Signed-off-by: Zhongxing Jiang <tomajsjiang@tencent.com>
2019-08-15 09:27:11 +08:00
Nicolas George
6e1a2dc112 lavf/aviobuf: make AVSEEK_SIZE usable from outside. 2019-08-13 15:38:57 +02:00
Jun Zhao
4373bb411c lavf/avio: remove ffio_open2_wrapper function
Remove the function ffio_open2_wrapper, it's not being used anymore.

Signed-off-by: Jun Zhao <barryjzhao@tencent.com>
2019-07-21 10:05:40 +08:00
Michael Niedermayer
0334632d5c avformat/aviobuf: Delay buffer downsizing until asserts are met
Fixes: Assertion failure
Fixes: 15151/clusterfuzz-testcase-minimized-ffmpeg_DEMUXER_fuzzer-5757079496687616
Fixes: 15205/clusterfuzz-testcase-minimized-ffmpeg_DEMUXER_fuzzer-5767573242642432
May fix: Ticket7094

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2019-06-30 17:43:50 +02:00
Carl Eugen Hoyos
e3b339cc92 lavf/aviobuf: Increase Statistics verbosity to AV_LOG_VERBOSE. 2018-06-16 02:37:09 +02:00
Jun Zhao
cdd107b965 lavf/aviobuf: add ff_get_chomp_line
Same as ff_get_line but strip the white-space characters in the
string tail.

Signed-off-by: Jun Zhao <mypopydev@gmail.com>
2018-04-12 16:04:58 +08:00
Marton Balint
dcb2ef2211 avformat/aviobuf: add ff_read_line_to_bprint and ff_read_line_to_bprint_overwrite functions
To be able to read lines longer than a static buffer size.

Signed-off-by: Marton Balint <cus@passwd.hu>
2018-02-24 20:17:04 +01:00
James Almer
aa6280805e avformat/aviobuf: zero initialize the AVIOContext in ffio_init_context()
This makes sure no field is ever used uninitialized.

Reviewed-by: Carl Eugen Hoyos <ceffmpeg@gmail.com>
Reviewed-by: wm4 <nfxjfg@googlemail.com>
Signed-off-by: James Almer <jamrial@gmail.com>
2018-02-14 20:56:51 -03:00
Jeyapal, Karthick
62f63b24bf libavformat/avio: Utility function to return URLContext 2017-11-29 14:31:34 +08:00
Nicolas George
158a79c31d lavf/aviobuf: return EINVAL when reading from a write-only context.
Signed-off-by: Nicolas George <george@nsup.org>
2017-10-29 19:40:52 +02:00
Nicolas George
a606f27f4c lavf/avio: temporarily accept 0 as EOF.
Print a warning to let applicatios fix their use.
After a deprecation period, check with a low-level assert.
Also make the constraint explicit in the doxygen comment.

Signed-off-by: Nicolas George <george@nsup.org>
2017-10-29 19:40:52 +02:00
James Almer
d4d2e9fe4e avformat: Drop deprecated feof() AVIO fuction
Deprecated in 08/2014.
2017-10-21 22:08:08 -03:00
Daniel Kucera
858db4b01f libavformat: not treat 0 as EOF
transfer_func variable passed to retry_transfer_wrapper
are h->prot->url_read and h->prot->url_write functions.
These need to return EOF or other error properly.
In case of returning >= 0, url_read/url_write is retried
until error is returned.

Signed-off-by: Daniel Kucera <daniel.kucera@gmail.com>
2017-10-19 22:07:21 +02:00
wm4
5d76674756 lavf: make avio_read_partial() public
Main use-case is proxying avio through a foreign I/O layer and a custom
AVIO context, without losing latency and performance characteristics.

Signed-off-by: Luca Barbato <lu_zero@gentoo.org>

Merged from Libav commit 173b56218f.
2017-09-01 17:56:33 +02:00
Anton Khirnov
b12e4d3bb8 avio: add a destructor for AVIOContext
Before this commit, AVIOContext is to be freed with a plain av_free(),
which prevents us from adding any deeper structure to it.

(cherry picked from commit 99684f3ae7)
Signed-off-by: James Almer <jamrial@gmail.com>
2017-09-01 01:51:15 -03:00
Vitaly Buka
eca2a49716 avformat/aviobuf: Fix signed integer overflow in avio_seek()
Signed integer overflow is undefined behavior.
Detected with clang and -fsanitize=signed-integer-overflow

Signed-off-by: Vitaly Buka <vitalybuka@google.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2017-08-23 03:17:11 +02:00
Marton Balint
09891c5391 avformat/aviobuf: add support for specifying minimum packet size and marking flush points
Reviewed-by: Michael Niedermayer <michael@niedermayer.cc>
Signed-off-by: Marton Balint <cus@passwd.hu>
2017-06-24 18:51:29 +02:00
Marton Balint
c14fa7a330 avformat/aviobuf: fix flushing write buffers after seeking backward or forward
This patch makes aviobuf work more like traditinal file IO, which is how people
think about it.

For example, in the past, aviobuf only flushed buffers until the current buffer
position, even if more data was written to it previously, and a backward seek
was used to reposition the IO context.

From now, aviobuf will keep track of the written data, so no explicit seek will
be required till the end of the buffer, or till the end of file before flushing.

This fixes at least one regression, fate-vsynth3-flv was broken if
flush_packets option was set to false, an explicit seek was removed in
4e3cc4bdd8.

Also from now on, if a forward seek in the write buffer were to cause a gap
between the already written data and the new file position, a flush will
happen.

The must_flush varable is also removed, which might have caused needless
flushes with multiple seeks whithin the write buffer. Since we know the amount
of data written to it, we will know when to flush.

Signed-off-by: Marton Balint <cus@passwd.hu>
2017-06-24 18:51:29 +02:00
Michael Niedermayer
7383a835e4 avformat/aviobuf: Only downscale the buffer once it has been used
The code mistook the first iteration sometimes as the end

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2017-05-19 21:14:26 +02:00
James Almer
34d7f337c1 Merge commit '3f75e5116b900f1428aa13041fc7d6301bf1988a'
* commit '3f75e5116b900f1428aa13041fc7d6301bf1988a':
  avio: Keep track of the amount of data written

Merged-by: James Almer <jamrial@gmail.com>
2017-04-13 19:49:20 -03:00
James Almer
fc9f14c7de Merge commit '8ea35af7620e4f73f9e8c072e1c0fac9a04ec161'
* commit '8ea35af7620e4f73f9e8c072e1c0fac9a04ec161':
  avio: add a new flag for marking streams seekable by timestamp

Merged-by: James Almer <jamrial@gmail.com>
2017-03-21 17:07:44 -03:00
James Almer
4de591e6fb Merge commit '83548fe894cdb455cc127f754d09905b6d23c173'
* commit '83548fe894cdb455cc127f754d09905b6d23c173':
  lavf: fix usage of AVIOContext.seekable

Merged-by: James Almer <jamrial@gmail.com>
2017-03-21 17:02:30 -03:00
Joel Cunningham
8c8e5d5286 HTTP: improve performance by reducing forward seeks
This commit optimizes HTTP performance by reducing forward seeks, instead
favoring a read-ahead and discard on the current connection (referred to
as a short seek) for seeks that are within a TCP window's worth of data.
This improves performance because with TCP flow control, a window's worth
of data will be in the local socket buffer already or in-flight from the
sender once congestion control on the sender is fully utilizing the window.

Note: this approach doesn't attempt to differentiate from a newly opened
connection which may not be fully utilizing the window due to congestion
control vs one that is. The receiver can't get at this information, so we
assume worst case; that full window is in use (we did advertise it after all)
and that data could be in-flight

The previous behavior of closing the connection, then opening a new
with a new HTTP range value results in a massive amounts of discarded
and re-sent data when large TCP windows are used.  This has been observed
on MacOS/iOS which starts with an initial window of 256KB and grows up to
1MB depending on the bandwidth-product delay.

When seeking within a window's worth of data and we close the connection,
then open a new one within the same window's worth of data, we discard
from the current offset till the end of the window.  Then on the new
connection the server ends up re-sending the previous data from new
offset till the end of old window.

Example (assumes full window utilization):

TCP window size: 64KB
Position: 32KB
Forward seek position: 40KB

      *                      (Next window)
32KB |--------------| 96KB |---------------| 160KB
        *
  40KB |---------------| 104KB

Re-sent amount: 96KB - 40KB = 56KB

For a real world test example, I have MP4 file of ~25MB, which ffplay
only reads ~16MB and performs 177 seeks. With current ffmpeg, this results
in 177 HTTP GETs and ~73MB worth of TCP data communication.  With this
patch, ffmpeg issues 4 HTTP GETs and 3 seeks for a total of ~22MB of TCP data
communication.

To support this feature, the short seek logic in avio_seek() has been
extended to call a function to get the short seek threshold value.  This
callback has been plumbed to the URLProtocol structure, which now has
infrastructure in HTTP and TCP to get the underlying receiver window size
via SO_RCVBUF.  If the underlying URL and protocol don't support returning
a short seek threshold, the default s->short_seek_threshold is used

This feature has been tested on Windows 7 and MacOS/iOS.  Windows support
is slightly complicated by the fact that when TCP window auto-tuning is
enabled, SO_RCVBUF doesn't report the real window size, but it does if
SO_RCVBUF was manually set (disabling auto-tuning). So we can only use
this optimization on Windows in the later case

Signed-off-by: Joel Cunningham <joel.cunningham@me.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2017-02-14 17:00:30 +01:00
softworkz
9488032e10 libavformat/avio: Add avio_get_dyn_buf function
This commit adds the avio_get_dyn_buf function which allows accessing
the
content of a DynBuffer without destroying it.

This is required in matroskaenc for preliminary writing (correct) mkv
headers.

Context for this change is fixing regression bug #5977.

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2017-01-07 13:04:05 +01:00
Luca Barbato
3f75e5116b avio: Keep track of the amount of data written
Make avio_size() work with any write AVIOContext.
2016-12-08 15:58:28 -05:00
Stephan Holljes
d0be0cbebc lavf/aviobuf.c: Adapt avio_accept and avio_handshake to new AVIOContext API
Signed-off-by: Stephan Holljes <klaxa1337@googlemail.com>
2016-10-03 23:28:48 +02:00
Anton Khirnov
8ea35af762 avio: add a new flag for marking streams seekable by timestamp 2016-09-30 16:55:45 +02:00
Anton Khirnov
83548fe894 lavf: fix usage of AVIOContext.seekable
It is supposed to be a flag. The only currently defined value is
AVIO_SEEKABLE_NORMAL, but other ones may be added in the future.
However all the current lavf code treats this field as a bool (mainly
for historical reasons).
Change all those cases to properly check for AVIO_SEEKABLE_NORMAL.
2016-09-30 16:54:33 +02:00
James Almer
155f4e9630 avformat/avio: add a crc callback function for CRC-32 IEEE-LE
Reviewed-by: Nicolas George <george@nsup.org>
Reviewed-by: Michael Niedermayer <michael@niedermayer.cc>
Signed-off-by: James Almer <jamrial@gmail.com>
2016-08-04 16:25:32 -03:00
James Almer
d688f39dc4 cosmetics: fix some misspelled words
Signed-off-by: James Almer <jamrial@gmail.com>
2016-07-17 13:10:27 -03:00
Matthieu Bouron
5d75e46677 Merge commit 'db7968bff4851c2be79b15b2cb2ae747424d2fca'
* commit 'db7968bff4851c2be79b15b2cb2ae747424d2fca':
  avio: Allow custom IO users to get labels for the output bytestream

Merged-by: Matthieu Bouron <matthieu.bouron@stupeflix.com>
2016-06-23 15:59:44 +02:00
Martin Storsjö
db7968bff4 avio: Allow custom IO users to get labels for the output bytestream
This allows callers with avio write callbacks to get the bytestream
positions that correspond to keyframes, suitable for live streaming.

In the simplest form, a caller could expect that a header is written
to the bytestream during the avformat_write_header, and the data
output to the avio context during e.g. av_write_frame corresponds
exactly to the current packet passed in.

When combined with av_interleaved_write_frame, and with muxers that
do buffering (most muxers that do some sort of fragmenting or
clustering), the mapping from input data to bytestream positions
is nontrivial.

This allows callers to get directly information about what part
of the bytestream is what, without having to resort to assumptions
about the muxer behaviour.

One keyframe/fragment/block can still be split into multiple (if
they are larger than the aviocontext buffer), which would call
the callback with e.g. AVIO_DATA_MARKER_SYNC_POINT, followed by
AVIO_DATA_MARKER_UNKNOWN for the second time it is called with
the following data.

Signed-off-by: Martin Storsjö <martin@martin.st>
2016-05-18 10:36:45 +03:00
Derek Buitenhuis
f8e89d8a29 Merge commit 'fab8156b2f30666adabe227b3d7712fd193873b1'
* commit 'fab8156b2f30666adabe227b3d7712fd193873b1':
  avio: Copy URLContext generic options into child URLContexts

Merged-by: Derek Buitenhuis <derek.buitenhuis@gmail.com>
2016-04-21 15:55:09 +01:00
Martin Storsjö
fab8156b2f avio: Copy URLContext generic options into child URLContexts
Since all URLContexts have the same AVOptions, such AVOptions
will be applied on the outermost context only and removed from the
dict, while they probably make sense on all contexts.

This makes sure that rw_timeout gets propagated to the innermost
URLContext (to make sure it gets passed to the tcp protocol, when
opening a http connection for instance).

Alternatively, such matching options would be kept in the dict
and only removed after the ffurl_connect call.

Signed-off-by: Martin Storsjö <martin@martin.st>
2016-03-24 10:34:19 +02:00
Hendrik Leppkes
0d4b8a2c16 hls: read protocol options through the AVIOContext
This reverts commit 9f9ed79d4c.

The hlsopts member was never set anywhere and always NULL, furthermore
the HLS demuxer needs to retrieve the proper options from the underlying
http protocol (cookies, user-agent, etc), so a dummy context won't help.

Instead, use the AVIOContext directly to access the options.
2016-03-16 10:31:36 +01:00
Derek Buitenhuis
93629735d7 avformat: Add a protocol blacklisting API
Signed-off-by: Derek Buitenhuis <derek.buitenhuis@gmail.com>
2016-03-04 16:13:42 +00:00