[3871] in arla-drinkers
Re: is cache-only-prefixes an nnpfs limitation?
daemon@ATHENA.MIT.EDU (Tomas Olsson)
Sun Apr 9 16:00:19 2006
From arla-drinkers-bounces@stacken.kth.se Sun Apr 09 20:00:19 2006
Return-Path: <arla-drinkers-bounces@stacken.kth.se>
Delivered-To: arla-drinkers-mtg@bloom-picayune.mit.edu
Received: (qmail 6249 invoked from network); 9 Apr 2006 20:00:18 -0000
Received: from mx1.kth.se (130.237.32.140)
by charon.mit.edu with SMTP; 9 Apr 2006 20:00:18 -0000
Received: from localhost (localhost.localdomain [127.0.0.1])
by mx1.kth.se (Postfix) with ESMTP id 860CF140612;
Sun, 9 Apr 2006 22:00:17 +0200 (CEST)
Received: from mx1.kth.se ([127.0.0.1])
by localhost (mx1.kth.se [127.0.0.1]) (amavisd-new, port 10024) with LMTP
id 30624-01-86; Sun, 9 Apr 2006 22:00:15 +0200 (CEST)
Received: from tapas.stacken.kth.se (tapas.stacken.kth.se [130.237.234.140])
by mx1.kth.se (Postfix) with ESMTP id 43AAE140534;
Sun, 9 Apr 2006 22:00:15 +0200 (CEST)
Received: from tapas.stacken.kth.se (localhost [127.0.0.1])
by tapas.stacken.kth.se (Postfix) with ESMTP id A9E17534F3;
Sun, 9 Apr 2006 22:00:14 +0200 (CEST)
X-Original-To: arla-drinkers@tapas.stacken.kth.se
Delivered-To: arla-drinkers@tapas.stacken.kth.se
Received: from brev.stacken.kth.se (brev.stacken.kth.se [130.237.234.84])
by tapas.stacken.kth.se (Postfix) with ESMTP id BFFB1534F3
for <arla-drinkers@tapas.stacken.kth.se>;
Sun, 9 Apr 2006 22:00:12 +0200 (CEST)
Received: from mx2.kth.se (mx2.kth.se [130.237.48.98])
by brev.stacken.kth.se (8.12.10/8.12.10) with ESMTP id k39K0Cla029703
for <arla-drinkers@stacken.kth.se>;
Sun, 9 Apr 2006 22:00:12 +0200 (MET DST)
Received: from localhost (localhost.localdomain [127.0.0.1])
by mx2.kth.se (Postfix) with ESMTP id D8620140609
for <arla-drinkers@stacken.kth.se>;
Sun, 9 Apr 2006 22:00:11 +0200 (CEST)
Received: from mx2.kth.se ([127.0.0.1])
by localhost (mx2.kth.se [127.0.0.1]) (amavisd-new,
port 10024) with LMTP
id 17045-01-34; Sun, 9 Apr 2006 22:00:09 +0200 (CEST)
Received: from kashyyyk.ite.kth.se (kashyyyk.ite.kth.se [130.237.31.35])
by mx2.kth.se (Postfix) with ESMTP id C0A6D1405F7;
Sun, 9 Apr 2006 22:00:09 +0200 (CEST)
Received: by kashyyyk.ite.kth.se (Postfix, from userid 18404)
id B8D347C8636; Sun, 9 Apr 2006 22:00:09 +0200 (CEST)
From: Tomas Olsson <tol@stacken.kth.se>
To: Adam Megacz <megacz@cs.berkeley.edu>
Subject: Re: is cache-only-prefixes an nnpfs limitation?
References: <x3mzf1bugb.fsf@nowhere.com> <lsrpsjxtyim.fsf@kashyyyk.ite.kth.se>
<x33bgskjcm.fsf@nowhere.com> <lsrpsjwsbe8.fsf@kashyyyk.ite.kth.se>
<x38xqetym2.fsf@nowhere.com>
Date: 09 Apr 2006 22:00:09 +0200
In-Reply-To: <x38xqetym2.fsf@nowhere.com>
Message-ID: <lsrfykmpliu.fsf@kashyyyk.ite.kth.se>
Lines: 81
User-Agent: Gnus/5.09 (Gnus v5.9.0) Emacs/21.3
MIME-Version: 1.0
X-Virus-Scanned: by amavisd-new at kth.se
X-Spam-Status: No, hits=-4.9 tagged_above=-200.0 required=5.0 tests=BAYES_00
X-Spam-Level:
Cc: arla-drinkers@stacken.kth.se
X-BeenThere: arla-drinkers@stacken.kth.se
X-Mailman-Version: 2.1.6
Precedence: list
List-Id: Arla discussions <arla-drinkers.stacken.kth.se>
List-Unsubscribe: <https://lists.stacken.kth.se/mailman/listinfo/arla-drinkers>,
<mailto:arla-drinkers-request@stacken.kth.se?subject=unsubscribe>
List-Archive: <http://lists.stacken.kth.se/pipermail/arla-drinkers>
List-Post: <mailto:arla-drinkers@stacken.kth.se>
List-Help: <mailto:arla-drinkers-request@stacken.kth.se?subject=help>
List-Subscribe: <https://lists.stacken.kth.se/mailman/listinfo/arla-drinkers>,
<mailto:arla-drinkers-request@stacken.kth.se?subject=subscribe>
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
Sender: arla-drinkers-bounces@stacken.kth.se
Errors-To: arla-drinkers-bounces@stacken.kth.se
X-Virus-Scanned: by amavisd-new at kth.se
Adam Megacz <megacz@cs.berkeley.edu> writes:
> I often store large media files in AFS and seek around in them (ie access
> starting at random locations in the file). Many of these files would fit
> in my disk cache, but moving the entire file over the network or crowding
> (almost) everything else out of my local cache would be bad.
>
True. ID3 tags and pdf indices are popular examples too.
> If I understand correctly, when used with arlad, nnpfs never gets file
> data directly from arlad -- it just asks arlad to place the file data in
> a particular file and then let the kernel know that it's there
>
Yup.
> - Do remote-filesystem file have to map 1:1 exactly to local files?
> Or do remote blocks map to local files? Or do remote blocks map
> to local blocks?
>
Traditionally it's 1:1 for data, plus possibly some lookup sugar. nnpfs
just has one "cache file" handle per fid.
For the current block prototype we use the same idea, but just split data
in fixed size chunks in the simplest possible way and use those cache files
as we use the single file today. The chunk size is configurable but
assumed to be a power of 2. For some reason I'm also assuming a 1:1
mapping between (fid, offset) and cache "block" file.
> - If it is known that a remote file will only be read once, is it
> possible to avoid writing it to the localfs disk?
>
Currently, not really. One could fairly easily add code to fetch data
using the pioctl/wakeup_data buffers, but I'm not sure how efficient that
would be. Once the block handling is in place and stable, one could start
thinking about other ways to store cache data, like anonymous mmap space.
Later.
The HPC folks are almost ready to kill for this (and cacheless writes).
Right now they'll have to resort to linking against arlad to do it.
BTW, what's the current status on ways for the user to communicate such
hints to the fs on various OS:es? I just love Windows in this respect,
they have tons of flags. They even have FILE_OPEN_FOR_FREE_SPACE_QUERY. I
wonder if it's ever used.
> I tried reading ./nnpfs/include/nnpfs/nnpfs_message.h from the block
> branch, and it seems to me that when the userland replies with an
> INSTALLDATA message, it tells the kernel not only what localfs file
> contains the requested block, but also *at what offset* in the localfs
> file the requested remotefs block is located.
>
The install messages aren't connected to the requests, there's a separate
WAKEUP to say "it's done" or signal error. So most messages are standalone
and can be used by the daemon without nnpfs requesting it. It's useful for
readahead of data and stat nodes. INSTALLDATA includes the offset of the
block in the fid (not cache file) and enough information to find the cache
file. For now, we put one block in each cache file for simplicity.
> - Does this mean that the nnpfs kernel module maintains a big
> complex mapping of (localfs_inode,offset)<->(remotefs_vnode,block#)?
>
The offset "is" the block#, I'd say
(cache block handle)<->(fid,offset)
We're not yet clear about how to implement the cache policies (LRU?), but
the bookkeeping looks like it will be scary. Finding the data at offset X
in some file needs to be fast, so we'll keep some map/list/tree in the
nnpfs node or maybe a huge global hash. We need some kind of LRU to do
cache replacement. We need some sort of reverse mapping to update the node
on eviction.
All in wired kernel memory. Yikes. Bright ideas are welcome.
> Lastly, is there a separate nnpfs mailing list, or is this the right
> place to discuss these questions?
>
It's the right place.
If y'all want to split nnpfs discussions to new list, convince me. I just
haven't seen a need yet.
/t
_______________________________________________
Arla-drinkers mailing list
Arla-drinkers@stacken.kth.se
https://lists.stacken.kth.se/mailman/listinfo/arla-drinkers
