summaryrefslogtreecommitdiff
path: root/fs/splice.c
AgeCommit message (Collapse)AuthorFilesLines
2006-05-01[PATCH] pipe: enable atomic copying of pipe data to/from user spaceJens Axboe1-2/+2
The pipe ->map() method uses kmap() to virtually map the pages, which is both slow and has known scalability issues on SMP. This patch enables atomic copying of pipe pages, by pre-faulting data and using kmap_atomic() instead. lmbench bw_pipe and lat_pipe measurements agree this is a Good Thing. Here are results from that on a UP machine with highmem (1.5GiB of RAM), running first a UP kernel, SMP kernel, and SMP kernel patched. Vanilla-UP: Pipe bandwidth: 1622.28 MB/sec Pipe bandwidth: 1610.59 MB/sec Pipe bandwidth: 1608.30 MB/sec Pipe latency: 7.3275 microseconds Pipe latency: 7.2995 microseconds Pipe latency: 7.3097 microseconds Vanilla-SMP: Pipe bandwidth: 1382.19 MB/sec Pipe bandwidth: 1317.27 MB/sec Pipe bandwidth: 1355.61 MB/sec Pipe latency: 9.6402 microseconds Pipe latency: 9.6696 microseconds Pipe latency: 9.6153 microseconds Patched-SMP: Pipe bandwidth: 1578.70 MB/sec Pipe bandwidth: 1579.95 MB/sec Pipe bandwidth: 1578.63 MB/sec Pipe latency: 9.1654 microseconds Pipe latency: 9.2266 microseconds Pipe latency: 9.1527 microseconds Signed-off-by: Jens Axboe <axboe@suse.de>
2006-05-01[PATCH] splice: call handle_ra_miss() on failure to lookup pageJens Axboe1-0/+6
Notify the readahead logic of the missing page. Suggested by Oleg Nesterov. Signed-off-by: Jens Axboe <axboe@suse.de>
2006-05-01[PATCH] pipe: introduce ->pin() buffer operationJens Axboe1-61/+30
The ->map() function is really expensive on highmem machines right now, since it has to use the slower kmap() instead of kmap_atomic(). Splice rarely needs to access the virtual address of a page, so it's a waste of time doing it. Introduce ->pin() to take over the responsibility of making sure the page data is valid. ->map() is then reduced to just kmap(). That way we can also share a most of the pipe buffer ops between pipe.c and splice.c Signed-off-by: Jens Axboe <axboe@suse.de>
2006-05-01[PATCH] splice: fix bugs in pipe_to_file()Jens Axboe1-18/+19
Found by Oleg Nesterov <oleg@tv-sign.ru>, fixed by me. - Only allow full pages to go to the page cache. - Check page != buf->page instead of using PIPE_BUF_FLAG_STOLEN. - Remember to clear 'stolen' if add_to_page_cache() fails. And as a cleanup on that: - Make the bottom fall-through logic a little less convoluted. Also make the steal path hold an extra reference to the page, so we don't have to differentiate between stolen and non-stolen at the end. Signed-off-by: Jens Axboe <axboe@suse.de>
2006-04-30[PATCH] splice: fix bugs with stealing regular pipe pagesJens Axboe1-1/+3
- Check that page has suitable count for stealing in the regular pipes. - pipe_to_file() assumes that the page is locked on succesful steal, so do that in the pipe steal hook - Missing unlock_page() in add_to_page_cache() failure. Signed-off-by: Jens Axboe <axboe@suse.de>
2006-04-27[PATCH] splice: make the read-side do batched page lookupsJens Axboe1-30/+65
Use the new find_get_pages_contig() to potentially look up the entire splice range in one single call. This speeds up generic_file_splice_read() quite a bit. Signed-off-by: Jens Axboe <axboe@suse.de>
2006-04-27[PATCH] splice: switch to using page_cache_readahead()Jens Axboe1-2/+2
Avoids doing useless work, when the file is fully cached. Signed-off-by: Jens Axboe <axboe@suse.de>
2006-04-26[PATCH] splice: rearrange moving to/from pipe helpersJens Axboe1-24/+11
We need these for people writing their own ->splice_read/write hooks. Signed-off-by: Jens Axboe <axboe@suse.de>
2006-04-26[PATCH] Add support for the sys_vmsplice syscallJens Axboe1-39/+253
sys_splice() moves data to/from pipes with a file input/output. sys_vmsplice() moves data to a pipe, with the input being a user address range instead. This uses an approach suggested by Linus, where we can hold partial ranges inside the pages[] map. Hopefully this will be useful for network receive support as well. Signed-off-by: Jens Axboe <axboe@suse.de>
2006-04-26[PATCH] splice: fix offset problemsJens Axboe1-19/+27
Make the move_from_pipe() actors return number of bytes processed, then move_from_pipe() can decide more cleverly when to move on to the next buffer. This fixes problems with pipe offset and differing file offset. Signed-off-by: Jens Axboe <axboe@suse.de>
2006-04-26[PATCH] splice: fix min() warningAndrew Morton1-1/+1
Signed-off-by: Andrew Morton <akpm@osdl.org> Signed-off-by: Jens Axboe <axboe@suse.de>
2006-04-20[PATCH] splice: fix smaller sized splice readsJens Axboe1-1/+12
Signed-off-by: Jens Axboe <axboe@suse.de>
2006-04-19[PATCH] splice: fixup writeout path after ->map changesJens Axboe1-19/+30
Since ->map() no longer locks the page, we need to adjust the handling of those pages (and stealing) a little. This now passes full regressions again. Signed-off-by: Jens Axboe <axboe@suse.de>
2006-04-19[PATCH] splice: offset fixesJens Axboe1-15/+30
- We need to adjust *ppos for writes as well. - Copy back modified offset value if one was passed in, similar to what sendfile does. Signed-off-by: Jens Axboe <axboe@suse.de>
2006-04-19[PATCH] tee: link_pipe() must be careful when dropping one of the pipe locksJens Axboe1-4/+14
We need to ensure that we only drop a lock that is ordered last, to avoid ABBA deadlocks with competing processes. Signed-off-by: Jens Axboe <axboe@suse.de>
2006-04-19[PATCH] splice: cleanup the SPLICE_F_NONBLOCK handlingJens Axboe1-14/+16
- generic_file_splice_read() more readable and correct - Don't bail on page allocation with NONBLOCK set, just don't allow direct blocking on IO (eg lock_page). Signed-off-by: Jens Axboe <axboe@suse.de>
2006-04-19[PATCH] splice: close i_size truncate races on readJens Axboe1-6/+37
We need to check i_size after doing a blocking readpage. Signed-off-by: Jens Axboe <axboe@suse.de>
2006-04-11[PATCH] splice: add support for sys_tee()Jens Axboe1-0/+186
Basically an in-kernel implementation of tee, which uses splice and the pipe buffers as an intelligent way to pass data around by reference. Where the user space tee consumes the input and produces a stdout and file output, this syscall merely duplicates the data inside a pipe to another pipe. No data is copied, the output just grabs a reference to the input pipe data. Signed-off-by: Jens Axboe <axboe@suse.de>
2006-04-11[PATCH] splice: pass offset around for ->splice_read() and ->splice_write()Jens Axboe1-42/+44
We need not use ->f_pos as the offset for the file input/output. If the user passed an offset pointer in through sys_splice(), just use that and leave ->f_pos alone. Signed-off-by: Jens Axboe <axboe@suse.de>
2006-04-11[PATCH] splice: comment stylesIngo Molnar1-11/+12
- capitalize consistently - end sentences in one way or another - update comment text to match the implementation Signed-off-by: Ingo Molnar <mingo@elte.hu> Signed-off-by: Jens Axboe <axboe@suse.de>
2006-04-11[PATCH] splice: add Ingo as addition copyright holderJens Axboe1-4/+5
The comment is also somewhat out of date, correct that as well. Signed-off-by: Jens Axboe <axboe@suse.de>
2006-04-11[PATCH] splice: unlikely() optimizationsJens Axboe1-8/+7
Also corrects a few comments. Patch mainly from Ingo, changes by me. Signed-off-by: Ingo Molnar <mingo@elte.hu> Signed-off-by: Jens Axboe <axboe@suse.de>
2006-04-11[PATCH] splice: speedups and optimizationsJens Axboe1-20/+13
- Kill the local variables that cache ->nrbufs, they just take up space. - Only set do_wakeup for a real pipe. This is a big win for direct splicing. - Kill i_mutex lock around ->f_pos update, regular io paths don't do this either. Signed-off-by: Jens Axboe <axboe@suse.de>
2006-04-11[PATCH] splice: speedup __generic_file_splice_readJens Axboe1-11/+63
Using find_get_page() is a lot faster than find_or_create_page(). This gets splice a lot closer to sendfile() for fd -> socket transfers. Signed-off-by: Jens Axboe <axboe@suse.de>
2006-04-11[PATCH] splice: add direct fd <-> fd splicing supportJens Axboe1-18/+130
It's more efficient for sendfile() emulation. Basically we cache an internal private pipe and just use that as the intermediate area for pages. Direct splicing is not available from sys_splice(), it is only meant to be used for sendfile() emulation. Additional patch from Ingo Molnar to avoid the PIPE_BUFFERS loop at exit for the normal fast path. Signed-off-by: Jens Axboe <axboe@suse.de>
2006-04-10[PATCH] splice: add optional input and output offsetsIngo Molnar1-13/+41
add optional input and output offsets to sys_splice(), for seekable file descriptors: asmlinkage long sys_splice(int fd_in, loff_t __user *off_in, int fd_out, loff_t __user *off_out, size_t len, unsigned int flags); semantics are straightforward: f_pos will be updated with the offset provided by user-space, before the splice transfer is about to begin. Providing a NULL offset pointer means the existing f_pos will be used (and updated in situ). Providing an offset for a pipe results in -ESPIPE. Providing an invalid offset pointer results in -EFAULT. Signed-off-by: Ingo Molnar <mingo@elte.hu> Signed-off-by: Jens Axboe <axboe@suse.de>
2006-04-10[PATCH] introduce a "kernel-internal pipe object" abstractionIngo Molnar1-59/+63
separate out the 'internal pipe object' abstraction, and make it usable to splice. This cleans up and fixes several aspects of the internal splice APIs and the pipe code: - pipes: the allocation and freeing of pipe_inode_info is now more symmetric and more streamlined with existing kernel practices. - splice: small micro-optimization: less pointer dereferencing in splice methods Signed-off-by: Ingo Molnar <mingo@elte.hu> Update XFS for the ->splice_read/->splice_write changes. Signed-off-by: Jens Axboe <axboe@suse.de>
2006-04-10[PATCH] splice: be smarter about calling do_page_cache_readahead()Jens Axboe1-2/+5
We don't want to call into the read-ahead logic unless we are at the start of a page, _or_ we have multiple pages to read. Signed-off-by: Jens Axboe <axboe@suse.de>
2006-04-10[PATCH] splice: optimize the splice buffer mappingJens Axboe1-10/+30
We don't really need to lock down the pages, just make sure they are uptodate. Signed-off-by: Jens Axboe <axboe@suse.de>
2006-04-10[PATCH] splice: cleanup __generic_file_splice_read()Jens Axboe1-49/+10
The whole shadow/pages logic got overly complex, and this simpler approach is actually faster in testing. Signed-off-by: Jens Axboe <axboe@suse.de>
2006-04-10[PATCH] splice: only call wake_up_interruptible() when we really have toJens Axboe1-4/+12
__wake_up_common() is pretty heavy in the kernel profiles, this brings it down to a more acceptable level. Signed-off-by: Jens Axboe <axboe@suse.de>
2006-04-10[PATCH] splice: potential !page dereferenceDave Jones1-1/+2
We can get to out: with a NULL page, which we probably don't want to be calling page_cache_release() on. Signed-off-by: Dave Jones <davej@redhat.com> Signed-off-by: Jens Axboe <axboe@suse.de>
2006-04-10[PATCH] splice: mark the io page as accessedJens Axboe1-0/+1
We should do that, since we do the LRU manipulation ourselves now. Suggested by Nick Piggin. Signed-off-by: Jens Axboe <axboe@suse.de>
2006-04-02[PATCH] splice: fix page stealing LRU handling.Jens Axboe1-19/+11
Originally from Nick Piggin, just adapted to the newer branch. You can't check PageLRU without holding zone->lru_lock. The page release code can get away with it only because the page refcount is 0 at that point. Also, you can't reliably remove pages from the LRU unless the refcount is 0. Ever. Signed-off-by: Nick Piggin <nickpiggin@yahoo.com.au> Signed-off-by: Jens Axboe <axboe@suse.de>
2006-04-02[PATCH] splice: page stealing needs to wait_on_page_writeback()Jens Axboe1-0/+9
Thanks to Andrew for the good explanation of why this is so. akpm writes: If a page is under writeback and we remove it from pagecache, it's still going to get written to disk. But the VFS no longer knows about that page, nor that this page is about to modify disk blocks. So there might be scenarios in which those blocks-which-are-about-to-be-written-to get reused for something else. When writeback completes, it'll scribble on those blocks. This won't happen in ext2/ext3-style filesystems in normal mode because the page has buffers and try_to_release_page() will fail. But ext2 in nobh mode doesn't attach buffers at all - it just sticks the page in a BIO, finds some new blocks, points the BIO at those blocks and lets it rip. While that write IO's in flight, someone could truncate the file. Truncate won't block on the writeout because the page isn't in pagecache any more. So truncate will the free the blocks from the file under the page's feet. Then something else can reallocate those blocks. Then write data to them. Now, the original write completes, corrupting the filesystem. Signed-off-by: Jens Axboe <axboe@suse.de>
2006-04-02[PATCH] splice: export generic_splice_sendpageJens Axboe1-2/+5
Forgot that one, thanks Jeff. Also move the other EXPORT_SYMBOL to right below the functions. Signed-off-by: Jens Axboe <axboe@suse.de>
2006-04-02[PATCH] splice: add a SPLICE_F_MORE flagJens Axboe1-2/+3
This lets userspace indicate whether more data will be coming in a subsequent splice call. Signed-off-by: Jens Axboe <axboe@suse.de>
2006-04-02[PATCH] splice: add comments documenting more of the codeJens Axboe1-4/+66
Hopefully this will make Andrew a little more happy. Signed-off-by: Jens Axboe <axboe@suse.de>
2006-04-02[PATCH] splice: improve writeback and clean up page stealingJens Axboe1-16/+48
By cleaning up the writeback logic (killing write_one_page() and the manual set_page_dirty()), we can get rid of ->stolen inside the pipe_buffer and just keep it local in pipe_to_file(). This also adds dirty page balancing logic and O_SYNC handling. Signed-off-by: Jens Axboe <axboe@suse.de>
2006-04-02[PATCH] splice: fix shadow[] filling logicJens Axboe1-3/+3
Clear the entire range, and don't increment pidx or we keep filling the same position again and again. Thanks to KAMEZAWA Hiroyuki. Signed-off-by: Jens Axboe <axboe@suse.de>
2006-04-02splice: add SPLICE_F_NONBLOCK flagLinus Torvalds1-4/+21
It doesn't make the splice itself necessarily nonblocking (because the actual file descriptors that are spliced from/to may block unless they have the O_NONBLOCK flag set), but it makes the splice pipe operations nonblocking. Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-03-30[PATCH] splice exportsJeff Garzik1-0/+4
Woe be unto he who builds their filesystems as modules. Signed-off-by: Jeff Garzik <jeff@garzik.org> [ Obscure quote from the infamous geek bible? ] Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-03-30[PATCH] splice: add support for SPLICE_F_MOVE flagJens Axboe1-37/+84
This enables the caller to migrate pages from one address space page cache to another. In buzz word marketing, you can do zero-copy file copies! Signed-off-by: Jens Axboe <axboe@suse.de> Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-03-30[PATCH] Introduce sys_splice() system callJens Axboe1-0/+612
This adds support for the sys_splice system call. Using a pipe as a transport, it can connect to files or sockets (latter as output only). From the splice.c comments: "splice": joining two ropes together by interweaving their strands. This is the "extended pipe" functionality, where a pipe is used as an arbitrary in-memory buffer. Think of a pipe as a small kernel buffer that you can use to transfer data from one end to the other. The traditional unix read/write is extended with a "splice()" operation that transfers data buffers to or from a pipe buffer. Named by Larry McVoy, original implementation from Linus, extended by Jens to support splicing to files and fixing the initial implementation bugs. Signed-off-by: Jens Axboe <axboe@suse.de> Signed-off-by: Linus Torvalds <torvalds@osdl.org>
generated by cgit v1.2.3 (git 2.25.1) at 2024-11-10 02:54:12 +0000