From davidlohr.bueso at hp.com  Mon Jun  3 15:04:52 2013
From: davidlohr.bueso at hp.com (Davidlohr Bueso)
Date: Mon, 03 Jun 2013 12:04:52 -0700
Subject: [PATCH v1 00/11] locks: scalability improvements for file locking
In-Reply-To: <1370056054-25449-1-git-send-email-jlayton@redhat.com>
References: <1370056054-25449-1-git-send-email-jlayton@redhat.com>
Message-ID: <1370286292.15671.4.camel@buesod1.americas.hpqcorp.net>

On Fri, 2013-05-31 at 23:07 -0400, Jeff Layton wrote:
> This is not the first attempt at doing this. The conversion to the
> i_lock was originally attempted by Bruce Fields a few years ago. His
> approach was NAK'ed since it involved ripping out the deadlock
> detection. People also really seem to like /proc/locks for debugging, so
> keeping that in is probably worthwhile.

Yep, we need to keep this. FWIW, lslocks(8) relies on /proc/locks.

Thanks,
Davidlohr



From bfields at fieldses.org  Mon Jun  3 17:31:01 2013
From: bfields at fieldses.org (J. Bruce Fields)
Date: Mon, 3 Jun 2013 17:31:01 -0400
Subject: [PATCH v1 00/11] locks: scalability improvements for file locking
In-Reply-To: <1370056054-25449-1-git-send-email-jlayton@redhat.com>
References: <1370056054-25449-1-git-send-email-jlayton@redhat.com>
Message-ID: <20130603213101.GC2109@fieldses.org>

On Fri, May 31, 2013 at 11:07:23PM -0400, Jeff Layton wrote:
> Executive summary (tl;dr version): This patchset represents an overhaul
> of the file locking code with an aim toward improving its scalability
> and making the code a bit easier to understand.

Thanks for working on this, that code could use some love!

> Longer version:
> 
> When the BKL was finally ripped out of the kernel in 2010, the strategy
> taken for the file locking code was to simply turn it into a new
> file_lock_locks spinlock. It was an expedient way to deal with the file
> locking code at the time, but having a giant spinlock around all of this
> code is clearly not great for scalability. Red Hat has bug reports that
> go back into the 2.6.18 era that point to BKL scalability problems in
> the file locking code and the file_lock_lock suffers from the same
> issues.
> 
> This patchset is my first attempt to make this code less dependent on
> global locking. The main change is to switch most of the file locking
> code to be protected by the inode->i_lock instead of the file_lock_lock.
> 
> While that works for most things, there are a couple of global data
> structures (lists in the current code) that need a global lock to
> protect them. So we still need a global lock in order to deal with
> those. The remaining patches are intended to make that global locking
> less painful. The big gain is made by turning the blocked_list into a
> hashtable, which greatly speeds up the deadlock detection code.
> 
> I rolled a couple of small programs in order to test this code. The
> first one just forks off 128 children and has them lock and unlock the
> same file 10k times. Running this under "time" against a file on tmpfs
> gives typical values like this:

What kind of hardware was this?

> 
> Unpatched (3.10-rc3-ish):
> real	0m5.283s
> user	0m0.380s
> sys	0m20.469s
> 
> Patched (same base kernel):
> real	0m5.099s
> user	0m0.478s
> sys	0m19.662s
> 
> ...so there seems to be some modest performance gain in this test. I
> think that's almost entirely due to the change to a hashtable and to
> optimize removing and readding blocked locks to the global lists. Note
> that with this code we have to take two spinlocks instead of just one,
> and that has some performance impact too. So the real peformance gain
> from that hashtable conversion is eaten up to some degree by this.

Might be nice to look at some profiles to confirm all of that.  I'd also
be curious how much variation there was in the results above, as they're
pretty close.

> The next test just forks off a bunch of children that each create their
> own file and then lock and unlock it 20k times. Obviously, the locks in
> this case are uncontended. Running that under "time" typically gives
> these rough numbers.
> 
> Unpatched (3.10-rc3-ish):
> real	0m8.836s
> user	0m1.018s
> sys	0m34.094s
> 
> Patched (same base kernel):
> real	0m4.965s
> user	0m1.043s
> sys	0m18.651s
> 
> In this test, we see the real benefit of moving to the i_lock for most
> of this code. The run time is almost cut in half in this test. With
> these changes locking different inodes needs very little serialization.
> 
> If people know of other file locking performance tests, then I'd be
> happy to try them out too. It's possible that this might make some
> workloads slower, and it would be helpful to know what they are (and
> address them) if so.
> 
> This is not the first attempt at doing this. The conversion to the
> i_lock was originally attempted by Bruce Fields a few years ago. His
> approach was NAK'ed since it involved ripping out the deadlock
> detection. People also really seem to like /proc/locks for debugging, so
> keeping that in is probably worthwhile.

Yes, there's already code that depends on it.

The deadlock detection, though--I still wonder if we could get away with
ripping it out.  Might be worth at least giving an option to configure
it out as a first step.

--b.

> There's more work to be done in this area and this patchset is just a
> start. There's a horrible thundering herd problem when a blocking lock
> is released, for instance. There was also interest in solving the goofy
> "unlock on any close" POSIX lock semantics at this year's LSF. I think
> this patchset will help lay the groundwork for those changes as well.
> 
> Comments and suggestions welcome.
> 
> Jeff Layton (11):
>   cifs: use posix_unblock_lock instead of locks_delete_block
>   locks: make generic_add_lease and generic_delete_lease static
>   locks: comment cleanups and clarifications
>   locks: make "added" in __posix_lock_file a bool
>   locks: encapsulate the fl_link list handling
>   locks: convert to i_lock to protect i_flock list
>   locks: only pull entries off of blocked_list when they are really
>     unblocked
>   locks: convert fl_link to a hlist_node
>   locks: turn the blocked_list into a hashtable
>   locks: add a new "lm_owner_key" lock operation
>   locks: give the blocked_hash its own spinlock
> 
>  Documentation/filesystems/Locking |   27 +++-
>  fs/afs/flock.c                    |    5 +-
>  fs/ceph/locks.c                   |    2 +-
>  fs/ceph/mds_client.c              |    8 +-
>  fs/cifs/cifsfs.c                  |    2 +-
>  fs/cifs/file.c                    |   15 +-
>  fs/gfs2/file.c                    |    2 +-
>  fs/lockd/svclock.c                |   12 ++
>  fs/lockd/svcsubs.c                |   12 +-
>  fs/locks.c                        |  254 +++++++++++++++++++++++++------------
>  fs/nfs/delegation.c               |   11 +-
>  fs/nfs/nfs4state.c                |    8 +-
>  fs/nfsd/nfs4state.c               |    8 +-
>  include/linux/fs.h                |   25 +---
>  14 files changed, 249 insertions(+), 142 deletions(-)
> 


From bfields at fieldses.org  Mon Jun  3 17:53:41 2013
From: bfields at fieldses.org (J. Bruce Fields)
Date: Mon, 3 Jun 2013 17:53:41 -0400
Subject: [PATCH v1 01/11] cifs: use posix_unblock_lock instead of
 locks_delete_block
In-Reply-To: <1370056054-25449-2-git-send-email-jlayton@redhat.com>
References: <1370056054-25449-1-git-send-email-jlayton@redhat.com>
 <1370056054-25449-2-git-send-email-jlayton@redhat.com>
Message-ID: <20130603215341.GD2109@fieldses.org>

On Fri, May 31, 2013 at 11:07:24PM -0400, Jeff Layton wrote:
> commit 66189be74 (CIFS: Fix VFS lock usage for oplocked files) exported
> the locks_delete_block symbol. There's already an exported helper
> function that provides this capability however, so make cifs use that
> instead and turn locks_delete_block back into a static function.
> 
> Note that if fl->fl_next == NULL then this lock has already been through
> locks_delete_block(), so we should be OK to ignore an ENOENT error here
> and simply not retry the lock.

ACK.--b.

> 
> Cc: Pavel Shilovsky <piastryyy at gmail.com>
> Signed-off-by: Jeff Layton <jlayton at redhat.com>
> ---
>  fs/cifs/file.c     |    2 +-
>  fs/locks.c         |    3 +--
>  include/linux/fs.h |    5 -----
>  3 files changed, 2 insertions(+), 8 deletions(-)
> 
> diff --git a/fs/cifs/file.c b/fs/cifs/file.c
> index 48b29d2..44a4f18 100644
> --- a/fs/cifs/file.c
> +++ b/fs/cifs/file.c
> @@ -999,7 +999,7 @@ try_again:
>  		rc = wait_event_interruptible(flock->fl_wait, !flock->fl_next);
>  		if (!rc)
>  			goto try_again;
> -		locks_delete_block(flock);
> +		posix_unblock_lock(file, flock);
>  	}
>  	return rc;
>  }
> diff --git a/fs/locks.c b/fs/locks.c
> index cb424a4..7a02064 100644
> --- a/fs/locks.c
> +++ b/fs/locks.c
> @@ -496,13 +496,12 @@ static void __locks_delete_block(struct file_lock *waiter)
>  
>  /*
>   */
> -void locks_delete_block(struct file_lock *waiter)
> +static void locks_delete_block(struct file_lock *waiter)
>  {
>  	lock_flocks();
>  	__locks_delete_block(waiter);
>  	unlock_flocks();
>  }
> -EXPORT_SYMBOL(locks_delete_block);
>  
>  /* Insert waiter into blocker's block list.
>   * We use a circular list so that processes can be easily woken up in
> diff --git a/include/linux/fs.h b/include/linux/fs.h
> index 43db02e..b9d7816 100644
> --- a/include/linux/fs.h
> +++ b/include/linux/fs.h
> @@ -1006,7 +1006,6 @@ extern int vfs_setlease(struct file *, long, struct file_lock **);
>  extern int lease_modify(struct file_lock **, int);
>  extern int lock_may_read(struct inode *, loff_t start, unsigned long count);
>  extern int lock_may_write(struct inode *, loff_t start, unsigned long count);
> -extern void locks_delete_block(struct file_lock *waiter);
>  extern void lock_flocks(void);
>  extern void unlock_flocks(void);
>  #else /* !CONFIG_FILE_LOCKING */
> @@ -1151,10 +1150,6 @@ static inline int lock_may_write(struct inode *inode, loff_t start,
>  	return 1;
>  }
>  
> -static inline void locks_delete_block(struct file_lock *waiter)
> -{
> -}
> -
>  static inline void lock_flocks(void)
>  {
>  }
> -- 
> 1.7.1
> 


From bfields at fieldses.org  Mon Jun  3 17:53:54 2013
From: bfields at fieldses.org (J. Bruce Fields)
Date: Mon, 3 Jun 2013 17:53:54 -0400
Subject: [PATCH v1 02/11] locks: make generic_add_lease and
 generic_delete_lease static
In-Reply-To: <1370056054-25449-3-git-send-email-jlayton@redhat.com>
References: <1370056054-25449-1-git-send-email-jlayton@redhat.com>
 <1370056054-25449-3-git-send-email-jlayton@redhat.com>
Message-ID: <20130603215353.GE2109@fieldses.org>

On Fri, May 31, 2013 at 11:07:25PM -0400, Jeff Layton wrote:
> Signed-off-by: Jeff Layton <jlayton at redhat.com>

ACK.--b.

> ---
>  fs/locks.c |    4 ++--
>  1 files changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/fs/locks.c b/fs/locks.c
> index 7a02064..e3140b8 100644
> --- a/fs/locks.c
> +++ b/fs/locks.c
> @@ -1337,7 +1337,7 @@ int fcntl_getlease(struct file *filp)
>  	return type;
>  }
>  
> -int generic_add_lease(struct file *filp, long arg, struct file_lock **flp)
> +static int generic_add_lease(struct file *filp, long arg, struct file_lock **flp)
>  {
>  	struct file_lock *fl, **before, **my_before = NULL, *lease;
>  	struct dentry *dentry = filp->f_path.dentry;
> @@ -1402,7 +1402,7 @@ out:
>  	return error;
>  }
>  
> -int generic_delete_lease(struct file *filp, struct file_lock **flp)
> +static int generic_delete_lease(struct file *filp, struct file_lock **flp)
>  {
>  	struct file_lock *fl, **before;
>  	struct dentry *dentry = filp->f_path.dentry;
> -- 
> 1.7.1
> 


From bfields at fieldses.org  Mon Jun  3 18:00:24 2013
From: bfields at fieldses.org (J. Bruce Fields)
Date: Mon, 3 Jun 2013 18:00:24 -0400
Subject: [PATCH v1 03/11] locks: comment cleanups and clarifications
In-Reply-To: <1370056054-25449-4-git-send-email-jlayton@redhat.com>
References: <1370056054-25449-1-git-send-email-jlayton@redhat.com>
 <1370056054-25449-4-git-send-email-jlayton@redhat.com>
Message-ID: <20130603220024.GF2109@fieldses.org>

On Fri, May 31, 2013 at 11:07:26PM -0400, Jeff Layton wrote:
> Signed-off-by: Jeff Layton <jlayton at redhat.com>
> ---
>  fs/locks.c         |   24 +++++++++++++++++++-----
>  include/linux/fs.h |    6 ++++++
>  2 files changed, 25 insertions(+), 5 deletions(-)
> 
> diff --git a/fs/locks.c b/fs/locks.c
> index e3140b8..a7d2253 100644
> --- a/fs/locks.c
> +++ b/fs/locks.c
> @@ -150,6 +150,16 @@ static int target_leasetype(struct file_lock *fl)
>  int leases_enable = 1;
>  int lease_break_time = 45;
>  
> +/*
> + * The i_flock list is ordered by:
> + *
> + * 1) lock type -- FL_LEASEs first, then FL_FLOCK, and finally FL_POSIX
> + * 2) lock owner
> + * 3) lock range start
> + * 4) lock range end
> + *
> + * Obviously, the last two criteria only matter for POSIX locks.
> + */

Thanks, yes, that needs documenting!  Though I wonder if this is the
place people will look for it.

>  #define for_each_lock(inode, lockp) \
>  	for (lockp = &inode->i_flock; *lockp != NULL; lockp = &(*lockp)->fl_next)
>  
> @@ -806,6 +816,11 @@ static int __posix_lock_file(struct inode *inode, struct file_lock *request, str
>  	}
>  
>  	lock_flocks();
> +	/*
> +	 * New lock request. Walk all POSIX locks and look for conflicts. If
> +	 * there are any, either return -EAGAIN or put the request on the
> +	 * blocker's list of waiters.
> +	 */

This though, seems a) not 100% accurate (it could also return EDEADLCK,
for example), b) mostly redundant with respect to the following code.

>  	if (request->fl_type != F_UNLCK) {
>  		for_each_lock(inode, before) {
>  			fl = *before;
> @@ -844,7 +859,7 @@ static int __posix_lock_file(struct inode *inode, struct file_lock *request, str
>  		before = &fl->fl_next;
>  	}
>  
> -	/* Process locks with this owner.  */
> +	/* Process locks with this owner. */
>  	while ((fl = *before) && posix_same_owner(request, fl)) {
>  		/* Detect adjacent or overlapping regions (if same lock type)
>  		 */
> @@ -930,10 +945,9 @@ static int __posix_lock_file(struct inode *inode, struct file_lock *request, str
>  	}
>  
>  	/*
> -	 * The above code only modifies existing locks in case of
> -	 * merging or replacing.  If new lock(s) need to be inserted
> -	 * all modifications are done bellow this, so it's safe yet to
> -	 * bail out.
> +	 * The above code only modifies existing locks in case of merging or
> +	 * replacing. If new lock(s) need to be inserted all modifications are
> +	 * done below this, so it's safe yet to bail out.
>  	 */
>  	error = -ENOLCK; /* "no luck" */
>  	if (right && left == right && !new_fl2)
> diff --git a/include/linux/fs.h b/include/linux/fs.h
> index b9d7816..ae377e9 100644
> --- a/include/linux/fs.h
> +++ b/include/linux/fs.h
> @@ -926,6 +926,12 @@ int locks_in_grace(struct net *);
>  /* that will die - we need it for nfs_lock_info */
>  #include <linux/nfs_fs_i.h>
>  
> +/*
> + * struct file_lock represents a generic "file lock". It's used to represent
> + * POSIX byte range locks, BSD (flock) locks, and leases. It's important to
> + * note that the same struct is used to represent both a request for a lock and
> + * the lock itself, but the same object is never used for both.

Yes, and I do find that confusing.  I wonder if there's a sensible way
to use separate structs for the different uses.

--b.

> + */
>  struct file_lock {
>  	struct file_lock *fl_next;	/* singly linked list for this inode  */
>  	struct list_head fl_link;	/* doubly linked list of all locks */
> -- 
> 1.7.1
> 


From jlayton at redhat.com  Tue Jun  4 06:54:17 2013
From: jlayton at redhat.com (Jeff Layton)
Date: Tue, 4 Jun 2013 06:54:17 -0400
Subject: [PATCH v1 00/11] locks: scalability improvements for file locking
In-Reply-To: <20130603213101.GC2109@fieldses.org>
References: <1370056054-25449-1-git-send-email-jlayton@redhat.com>
 <20130603213101.GC2109@fieldses.org>
Message-ID: <20130604065417.46080a57@tlielax.poochiereds.net>

On Mon, 3 Jun 2013 17:31:01 -0400
"J. Bruce Fields" <bfields at fieldses.org> wrote:

> On Fri, May 31, 2013 at 11:07:23PM -0400, Jeff Layton wrote:
> > Executive summary (tl;dr version): This patchset represents an overhaul
> > of the file locking code with an aim toward improving its scalability
> > and making the code a bit easier to understand.
> 
> Thanks for working on this, that code could use some love!
> 
> > Longer version:
> > 
> > When the BKL was finally ripped out of the kernel in 2010, the strategy
> > taken for the file locking code was to simply turn it into a new
> > file_lock_locks spinlock. It was an expedient way to deal with the file
> > locking code at the time, but having a giant spinlock around all of this
> > code is clearly not great for scalability. Red Hat has bug reports that
> > go back into the 2.6.18 era that point to BKL scalability problems in
> > the file locking code and the file_lock_lock suffers from the same
> > issues.
> > 
> > This patchset is my first attempt to make this code less dependent on
> > global locking. The main change is to switch most of the file locking
> > code to be protected by the inode->i_lock instead of the file_lock_lock.
> > 
> > While that works for most things, there are a couple of global data
> > structures (lists in the current code) that need a global lock to
> > protect them. So we still need a global lock in order to deal with
> > those. The remaining patches are intended to make that global locking
> > less painful. The big gain is made by turning the blocked_list into a
> > hashtable, which greatly speeds up the deadlock detection code.
> > 
> > I rolled a couple of small programs in order to test this code. The
> > first one just forks off 128 children and has them lock and unlock the
> > same file 10k times. Running this under "time" against a file on tmpfs
> > gives typical values like this:
> 
> What kind of hardware was this?
> 

Mostly a KVM guest on Intel hardware. I've done some testing on bare
metal too, and the results are pretty similar.
 
> > 
> > Unpatched (3.10-rc3-ish):
> > real	0m5.283s
> > user	0m0.380s
> > sys	0m20.469s
> > 
> > Patched (same base kernel):
> > real	0m5.099s
> > user	0m0.478s
> > sys	0m19.662s
> > 
> > ...so there seems to be some modest performance gain in this test. I
> > think that's almost entirely due to the change to a hashtable and to
> > optimize removing and readding blocked locks to the global lists. Note
> > that with this code we have to take two spinlocks instead of just one,
> > and that has some performance impact too. So the real peformance gain
> > from that hashtable conversion is eaten up to some degree by this.
> 
> Might be nice to look at some profiles to confirm all of that.  I'd also
> be curious how much variation there was in the results above, as they're
> pretty close.
> 

The above is just a random representative sample. The results are
pretty close when running this test, but I can average up several runs
and present the numbers. I plan to get a bare-metal test box on which
to run some more detailed testing and maybe some profiling this week.

> > The next test just forks off a bunch of children that each create their
> > own file and then lock and unlock it 20k times. Obviously, the locks in
> > this case are uncontended. Running that under "time" typically gives
> > these rough numbers.
> > 
> > Unpatched (3.10-rc3-ish):
> > real	0m8.836s
> > user	0m1.018s
> > sys	0m34.094s
> > 
> > Patched (same base kernel):
> > real	0m4.965s
> > user	0m1.043s
> > sys	0m18.651s
> > 
> > In this test, we see the real benefit of moving to the i_lock for most
> > of this code. The run time is almost cut in half in this test. With
> > these changes locking different inodes needs very little serialization.
> > 
> > If people know of other file locking performance tests, then I'd be
> > happy to try them out too. It's possible that this might make some
> > workloads slower, and it would be helpful to know what they are (and
> > address them) if so.
> > 
> > This is not the first attempt at doing this. The conversion to the
> > i_lock was originally attempted by Bruce Fields a few years ago. His
> > approach was NAK'ed since it involved ripping out the deadlock
> > detection. People also really seem to like /proc/locks for debugging, so
> > keeping that in is probably worthwhile.
> 
> Yes, there's already code that depends on it.
> 
> The deadlock detection, though--I still wonder if we could get away with
> ripping it out.  Might be worth at least giving an option to configure
> it out as a first step.
> 
> --b.
> 


I considered that, and have patches that add such a config option. Some
of the later patches in this set optimize the code that is necessary to
support deadlock detection. In particular, turning the blocked_list
into a hashtable really speeds up the list walking. So much so that I
think the case for compiling it out is less obvious.

Should we add an option for people who really want their locks to
scream? Maybe, but I think it would be easy to add that later,
especially now that the code to handle the blocked_hash is fairly well
encapsulated with this patch.

Thanks for taking a look!

-- Jeff

> > There's more work to be done in this area and this patchset is just a
> > start. There's a horrible thundering herd problem when a blocking lock
> > is released, for instance. There was also interest in solving the goofy
> > "unlock on any close" POSIX lock semantics at this year's LSF. I think
> > this patchset will help lay the groundwork for those changes as well.
> > 
> > Comments and suggestions welcome.
> > 
> > Jeff Layton (11):
> >   cifs: use posix_unblock_lock instead of locks_delete_block
> >   locks: make generic_add_lease and generic_delete_lease static
> >   locks: comment cleanups and clarifications
> >   locks: make "added" in __posix_lock_file a bool
> >   locks: encapsulate the fl_link list handling
> >   locks: convert to i_lock to protect i_flock list
> >   locks: only pull entries off of blocked_list when they are really
> >     unblocked
> >   locks: convert fl_link to a hlist_node
> >   locks: turn the blocked_list into a hashtable
> >   locks: add a new "lm_owner_key" lock operation
> >   locks: give the blocked_hash its own spinlock
> > 
> >  Documentation/filesystems/Locking |   27 +++-
> >  fs/afs/flock.c                    |    5 +-
> >  fs/ceph/locks.c                   |    2 +-
> >  fs/ceph/mds_client.c              |    8 +-
> >  fs/cifs/cifsfs.c                  |    2 +-
> >  fs/cifs/file.c                    |   15 +-
> >  fs/gfs2/file.c                    |    2 +-
> >  fs/lockd/svclock.c                |   12 ++
> >  fs/lockd/svcsubs.c                |   12 +-
> >  fs/locks.c                        |  254 +++++++++++++++++++++++++------------
> >  fs/nfs/delegation.c               |   11 +-
> >  fs/nfs/nfs4state.c                |    8 +-
> >  fs/nfsd/nfs4state.c               |    8 +-
> >  include/linux/fs.h                |   25 +---
> >  14 files changed, 249 insertions(+), 142 deletions(-)
> > 


From jlayton at redhat.com  Tue Jun  4 07:09:02 2013
From: jlayton at redhat.com (Jeff Layton)
Date: Tue, 4 Jun 2013 07:09:02 -0400
Subject: [PATCH v1 03/11] locks: comment cleanups and clarifications
In-Reply-To: <20130603220024.GF2109@fieldses.org>
References: <1370056054-25449-1-git-send-email-jlayton@redhat.com>
 <1370056054-25449-4-git-send-email-jlayton@redhat.com>
 <20130603220024.GF2109@fieldses.org>
Message-ID: <20130604070902.07f0caee@tlielax.poochiereds.net>

On Mon, 3 Jun 2013 18:00:24 -0400
"J. Bruce Fields" <bfields at fieldses.org> wrote:

> On Fri, May 31, 2013 at 11:07:26PM -0400, Jeff Layton wrote:
> > Signed-off-by: Jeff Layton <jlayton at redhat.com>
> > ---
> >  fs/locks.c         |   24 +++++++++++++++++++-----
> >  include/linux/fs.h |    6 ++++++
> >  2 files changed, 25 insertions(+), 5 deletions(-)
> > 
> > diff --git a/fs/locks.c b/fs/locks.c
> > index e3140b8..a7d2253 100644
> > --- a/fs/locks.c
> > +++ b/fs/locks.c
> > @@ -150,6 +150,16 @@ static int target_leasetype(struct file_lock *fl)
> >  int leases_enable = 1;
> >  int lease_break_time = 45;
> >  
> > +/*
> > + * The i_flock list is ordered by:
> > + *
> > + * 1) lock type -- FL_LEASEs first, then FL_FLOCK, and finally FL_POSIX
> > + * 2) lock owner
> > + * 3) lock range start
> > + * 4) lock range end
> > + *
> > + * Obviously, the last two criteria only matter for POSIX locks.
> > + */
> 
> Thanks, yes, that needs documenting!  Though I wonder if this is the
> place people will look for it.
> 

Agreed. If you can think of a better spot to put this, I'd be happy to
move it in the next iteration of this series.

> >  #define for_each_lock(inode, lockp) \
> >  	for (lockp = &inode->i_flock; *lockp != NULL; lockp = &(*lockp)->fl_next)
> >  
> > @@ -806,6 +816,11 @@ static int __posix_lock_file(struct inode *inode, struct file_lock *request, str
> >  	}
> >  
> >  	lock_flocks();
> > +	/*
> > +	 * New lock request. Walk all POSIX locks and look for conflicts. If
> > +	 * there are any, either return -EAGAIN or put the request on the
> > +	 * blocker's list of waiters.
> > +	 */
> 
> This though, seems a) not 100% accurate (it could also return EDEADLCK,
> for example), b) mostly redundant with respect to the following code.
> 

Good point -- I'll fix that up. That one was mostly for my own benefit
while trying to figure out how this code works. It's still probably worth
some more verbose comments here since this certainly wasn't 100%
obvious to me when I first dove into this code.

> >  	if (request->fl_type != F_UNLCK) {
> >  		for_each_lock(inode, before) {
> >  			fl = *before;
> > @@ -844,7 +859,7 @@ static int __posix_lock_file(struct inode *inode, struct file_lock *request, str
> >  		before = &fl->fl_next;
> >  	}
> >  
> > -	/* Process locks with this owner.  */
> > +	/* Process locks with this owner. */
> >  	while ((fl = *before) && posix_same_owner(request, fl)) {
> >  		/* Detect adjacent or overlapping regions (if same lock type)
> >  		 */
> > @@ -930,10 +945,9 @@ static int __posix_lock_file(struct inode *inode, struct file_lock *request, str
> >  	}
> >  
> >  	/*
> > -	 * The above code only modifies existing locks in case of
> > -	 * merging or replacing.  If new lock(s) need to be inserted
> > -	 * all modifications are done bellow this, so it's safe yet to
> > -	 * bail out.
> > +	 * The above code only modifies existing locks in case of merging or
> > +	 * replacing. If new lock(s) need to be inserted all modifications are
> > +	 * done below this, so it's safe yet to bail out.
> >  	 */
> >  	error = -ENOLCK; /* "no luck" */
> >  	if (right && left == right && !new_fl2)
> > diff --git a/include/linux/fs.h b/include/linux/fs.h
> > index b9d7816..ae377e9 100644
> > --- a/include/linux/fs.h
> > +++ b/include/linux/fs.h
> > @@ -926,6 +926,12 @@ int locks_in_grace(struct net *);
> >  /* that will die - we need it for nfs_lock_info */
> >  #include <linux/nfs_fs_i.h>
> >  
> > +/*
> > + * struct file_lock represents a generic "file lock". It's used to represent
> > + * POSIX byte range locks, BSD (flock) locks, and leases. It's important to
> > + * note that the same struct is used to represent both a request for a lock and
> > + * the lock itself, but the same object is never used for both.
> 
> Yes, and I do find that confusing.  I wonder if there's a sensible way
> to use separate structs for the different uses.
> 
> --b.
> 

Yeah. I think that latter point accounts for a lot of the difficulty for
the uninitiated to understand this code.

I considered creating a "struct lock_request" too. If I were designing
this code from scratch I certainly would do so, but it's harder to
justify such a change to the existing code.

It would mean a lot of churn in fs-specific lock routines, and we'd
have to completely revamp things like locks_copy_lock. I just don't see
that giving us a lot of benefit in the near term and I really want this
set to focus on concrete benefits.

We might consider it in the future though. I'll add a FIXME comment
here so we can record the idea for posterity.

Thanks for the comments!

> > + */
> >  struct file_lock {
> >  	struct file_lock *fl_next;	/* singly linked list for this inode  */
> >  	struct list_head fl_link;	/* doubly linked list of all locks */
> > -- 
> > 1.7.1
> > 


-- 
Jeff Layton <jlayton at redhat.com>


From rees at umich.edu  Tue Jun  4 07:56:44 2013
From: rees at umich.edu (Jim Rees)
Date: Tue, 4 Jun 2013 07:56:44 -0400
Subject: [PATCH v1 00/11] locks: scalability improvements for file locking
In-Reply-To: <20130604065417.46080a57@tlielax.poochiereds.net>
References: <1370056054-25449-1-git-send-email-jlayton@redhat.com>
 <20130603213101.GC2109@fieldses.org>
 <20130604065417.46080a57@tlielax.poochiereds.net>
Message-ID: <20130604115644.GA4180@umich.edu>

Jeff Layton wrote:

  > Might be nice to look at some profiles to confirm all of that.  I'd also
  > be curious how much variation there was in the results above, as they're
  > pretty close.
  > 
  
  The above is just a random representative sample. The results are
  pretty close when running this test, but I can average up several runs
  and present the numbers. I plan to get a bare-metal test box on which
  to run some more detailed testing and maybe some profiling this week.

Just contributing more runs into the mean doesn't tell us anything about the
variance. With numbers that close you need the variance to tell whether it's
a significant change.


From jlayton at redhat.com  Tue Jun  4 08:15:17 2013
From: jlayton at redhat.com (Jeff Layton)
Date: Tue, 4 Jun 2013 08:15:17 -0400
Subject: [PATCH v1 00/11] locks: scalability improvements for file locking
In-Reply-To: <20130604115644.GA4180@umich.edu>
References: <1370056054-25449-1-git-send-email-jlayton@redhat.com>
 <20130603213101.GC2109@fieldses.org>
 <20130604065417.46080a57@tlielax.poochiereds.net>
 <20130604115644.GA4180@umich.edu>
Message-ID: <20130604081517.0ad9f983@corrin.poochiereds.net>

On Tue, 4 Jun 2013 07:56:44 -0400
Jim Rees <rees at umich.edu> wrote:

> Jeff Layton wrote:
> 
>   > Might be nice to look at some profiles to confirm all of that.  I'd also
>   > be curious how much variation there was in the results above, as they're
>   > pretty close.
>   > 
>   
>   The above is just a random representative sample. The results are
>   pretty close when running this test, but I can average up several runs
>   and present the numbers. I plan to get a bare-metal test box on which
>   to run some more detailed testing and maybe some profiling this week.
> 
> Just contributing more runs into the mean doesn't tell us anything about the
> variance. With numbers that close you need the variance to tell whether it's
> a significant change.

Thanks. I'll see if I can get some standard deviation numbers here, and
I'll do it on some bare metal to ensure that virtualization doesn't
skew any results.

FWIW, they were all consistently very close to one another when I ran
these tests, and the times were all consistently shorter than the
unpatched kernel.

That said, this test is pretty rough. Doing this with "time" measures
other things that aren't related to locking. So I'll also see if I
can come up with a way to measure the actual locking performance more
accurately too.

-- 
Jeff Layton <jlayton at redhat.com>


From metze at samba.org  Tue Jun  4 10:19:53 2013
From: metze at samba.org (Stefan (metze) Metzmacher)
Date: Tue, 04 Jun 2013 16:19:53 +0200
Subject: [PATCH v1 11/11] locks: give the blocked_hash its own spinlock
In-Reply-To: <1370056054-25449-12-git-send-email-jlayton@redhat.com>
References: <1370056054-25449-1-git-send-email-jlayton@redhat.com>
 <1370056054-25449-12-git-send-email-jlayton@redhat.com>
Message-ID: <51ADF789.70906@samba.org>

Hi Jeff,

> There's no reason we have to protect the blocked_hash and file_lock_list
> with the same spinlock. With the tests I have, breaking it in two gives
> a barely measurable performance benefit, but it seems reasonable to make
> this locking as granular as possible.

as file_lock_{list,lock} is only used for debugging (/proc/locks) after this
change, I guess it would be possible to use RCU instead of a spinlock.

@others: this was the related discussion on IRC
(http://irclog.samba.org/) about this:

16:02 < metze> jlayton: do you have time to discuss your file_lock_lock
changes?
16:02 < jlayton> metze: sure, what's up?
16:03 < jlayton> metze: note that it won't help vl's thundering herd
problems...
16:03 < metze> is it correct that after your last patch file_lock_lock
is only used for /proc/locks?
16:03 < jlayton> well, it's only used to protect the list that is used
for /proc/locks
16:04 < jlayton> it still gets taken whenever a lock is acquired or
released in order to manipulate that list
16:04 < metze> would it be a good idea to use rcu instead of a spin lock?
16:04 < jlayton> I tried using RCU, but it turned out to slow everything
down
16:04 < jlayton> this is not a read-mostly workload unfortunately
16:04 < jlayton> so doing it with mutual exclusion turns out to be faster
16:04 < metze> ok
16:05 < jlayton> I might play around with it again sometime, but I don't
think it really helps. What we need to ensure is
                 that we optimize the code that manipulates that list,
and RCU list manipulations have larger overhead
16:06 < jlayton> metze: that's a good question though so if you want to
ask it on the list, please do
16:06 < jlayton> others will probably be wondering the same thing
16:08 < metze> maybe it's worth a comment in commit message and the code
16:08 < metze> btw, why don't you remove the ' /* Protects the
file_lock_list and the blocked_hash */' comment?

metze

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 261 bytes
Desc: OpenPGP digital signature
URL: <http://lists.infradead.org/pipermail/linux-afs/attachments/20130604/5ec71c69/attachment-0001.sig>

From jlayton at redhat.com  Tue Jun  4 10:39:35 2013
From: jlayton at redhat.com (Jeff Layton)
Date: Tue, 4 Jun 2013 10:39:35 -0400
Subject: [PATCH v1 11/11] locks: give the blocked_hash its own spinlock
In-Reply-To: <51ADF789.70906@samba.org>
References: <1370056054-25449-1-git-send-email-jlayton@redhat.com>
 <1370056054-25449-12-git-send-email-jlayton@redhat.com>
 <51ADF789.70906@samba.org>
Message-ID: <20130604103935.2a6afb29@corrin.poochiereds.net>

On Tue, 04 Jun 2013 16:19:53 +0200
"Stefan (metze) Metzmacher" <metze at samba.org> wrote:

> Hi Jeff,
> 
> > There's no reason we have to protect the blocked_hash and file_lock_list
> > with the same spinlock. With the tests I have, breaking it in two gives
> > a barely measurable performance benefit, but it seems reasonable to make
> > this locking as granular as possible.
> 
> as file_lock_{list,lock} is only used for debugging (/proc/locks) after this
> change, I guess it would be possible to use RCU instead of a spinlock.
> 
> @others: this was the related discussion on IRC
> (http://irclog.samba.org/) about this:
> 
> 16:02 < metze> jlayton: do you have time to discuss your file_lock_lock
> changes?
> 16:02 < jlayton> metze: sure, what's up?
> 16:03 < jlayton> metze: note that it won't help vl's thundering herd
> problems...
> 16:03 < metze> is it correct that after your last patch file_lock_lock
> is only used for /proc/locks?
> 16:03 < jlayton> well, it's only used to protect the list that is used
> for /proc/locks
> 16:04 < jlayton> it still gets taken whenever a lock is acquired or
> released in order to manipulate that list
> 16:04 < metze> would it be a good idea to use rcu instead of a spin lock?
> 16:04 < jlayton> I tried using RCU, but it turned out to slow everything
> down
> 16:04 < jlayton> this is not a read-mostly workload unfortunately
> 16:04 < jlayton> so doing it with mutual exclusion turns out to be faster
> 16:04 < metze> ok
> 16:05 < jlayton> I might play around with it again sometime, but I don't
> think it really helps. What we need to ensure is
>                  that we optimize the code that manipulates that list,
> and RCU list manipulations have larger overhead
> 16:06 < jlayton> metze: that's a good question though so if you want to
> ask it on the list, please do
> 16:06 < jlayton> others will probably be wondering the same thing
> 16:08 < metze> maybe it's worth a comment in commit message and the code


I'm not sure it's worth commenting about RCU in the code. In the future
it may be possible to do this -- who knows. It does seem to have been
consistently slower in my testing here though.

> 16:08 < metze> btw, why don't you remove the ' /* Protects the
> file_lock_list and the blocked_hash */' comment?
> 

Removed in my git tree -- thanks for pointing that out.

-- 
Jeff Layton <jlayton at redhat.com>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-afs/attachments/20130604/62095b59/attachment-0001.sig>

From hch at infradead.org  Tue Jun  4 10:46:40 2013
From: hch at infradead.org (Christoph Hellwig)
Date: Tue, 4 Jun 2013 07:46:40 -0700
Subject: [PATCH v1 11/11] locks: give the blocked_hash its own spinlock
In-Reply-To: <51ADF789.70906@samba.org>
References: <1370056054-25449-1-git-send-email-jlayton@redhat.com>
 <1370056054-25449-12-git-send-email-jlayton@redhat.com>
 <51ADF789.70906@samba.org>
Message-ID: <20130604144640.GA7730@infradead.org>

Having RCU for modification mostly workloads never is a good idea, so
I don't think it makes sense to mention it here.

If you care about the overhead it's worth trying to use per-cpu lists,
though.



From bfields at fieldses.org  Tue Jun  4 10:53:22 2013
From: bfields at fieldses.org (J. Bruce Fields)
Date: Tue, 4 Jun 2013 10:53:22 -0400
Subject: [PATCH v1 11/11] locks: give the blocked_hash its own spinlock
In-Reply-To: <20130604144640.GA7730@infradead.org>
References: <1370056054-25449-1-git-send-email-jlayton@redhat.com>
 <1370056054-25449-12-git-send-email-jlayton@redhat.com>
 <51ADF789.70906@samba.org> <20130604144640.GA7730@infradead.org>
Message-ID: <20130604145322.GH2109@fieldses.org>

On Tue, Jun 04, 2013 at 07:46:40AM -0700, Christoph Hellwig wrote:
> Having RCU for modification mostly workloads never is a good idea, so
> I don't think it makes sense to mention it here.
> 
> If you care about the overhead it's worth trying to use per-cpu lists,
> though.

Yes.  The lock and unlock could happen on different CPU's--but I think
you can make the rule that the lock stays associated with the list it
was first put on, and then it's correct in general and hopefully quick
in the common case.

--b.


From jlayton at redhat.com  Tue Jun  4 10:56:56 2013
From: jlayton at redhat.com (Jeff Layton)
Date: Tue, 4 Jun 2013 10:56:56 -0400
Subject: [PATCH v1 11/11] locks: give the blocked_hash its own spinlock
In-Reply-To: <20130604144640.GA7730@infradead.org>
References: <1370056054-25449-1-git-send-email-jlayton@redhat.com>
 <1370056054-25449-12-git-send-email-jlayton@redhat.com>
 <51ADF789.70906@samba.org> <20130604144640.GA7730@infradead.org>
Message-ID: <20130604105656.18663efa@corrin.poochiereds.net>

On Tue, 4 Jun 2013 07:46:40 -0700
Christoph Hellwig <hch at infradead.org> wrote:

> Having RCU for modification mostly workloads never is a good idea, so
> I don't think it makes sense to mention it here.
> 
> If you care about the overhead it's worth trying to use per-cpu lists,
> though.
> 

Yeah, I looked at those too in an earlier set and it did help some.
Moving to a hashtable for the blocked_list really seemed to help the
most there, but percpu lists with lglocks or something might help a lot
on the file_lock_list.

-- 
Jeff Layton <jlayton at redhat.com>


From jlayton at redhat.com  Tue Jun  4 11:15:02 2013
From: jlayton at redhat.com (Jeff Layton)
Date: Tue, 4 Jun 2013 11:15:02 -0400
Subject: [PATCH v1 11/11] locks: give the blocked_hash its own spinlock
In-Reply-To: <20130604145322.GH2109@fieldses.org>
References: <1370056054-25449-1-git-send-email-jlayton@redhat.com>
 <1370056054-25449-12-git-send-email-jlayton@redhat.com>
 <51ADF789.70906@samba.org> <20130604144640.GA7730@infradead.org>
 <20130604145322.GH2109@fieldses.org>
Message-ID: <20130604111502.6231d010@corrin.poochiereds.net>

On Tue, 4 Jun 2013 10:53:22 -0400
"J. Bruce Fields" <bfields at fieldses.org> wrote:

> On Tue, Jun 04, 2013 at 07:46:40AM -0700, Christoph Hellwig wrote:
> > Having RCU for modification mostly workloads never is a good idea, so
> > I don't think it makes sense to mention it here.
> > 
> > If you care about the overhead it's worth trying to use per-cpu lists,
> > though.
> 
> Yes.  The lock and unlock could happen on different CPU's--but I think
> you can make the rule that the lock stays associated with the list it
> was first put on, and then it's correct in general and hopefully quick
> in the common case.
> 

It's important to distinguish between the blocked_list/hash and the
file_lock_list. Yes, they use the same embedded list_head or hlist_node
in the file_lock, but they have very different characteristics and use
cases.

In the testing I did, having a hashtable for the blocked locks helped a
lot more than a percpu list. The trick with deadlock detection is to
ensure that you don't spend a lot of time walking the lists. Since we
do deadlock detection frequently, we need to optimize for that case.

For the file_lock_list, it might make sense to have percpu hlists with
an lglock however. The thing to note here is that the file_lock_list is
almost never read. Only /proc/locks uses it, so anything we can do to
optimize the list manipulation is probably worth it.

All that said, I'm leery about changing too much of this code too fast.
It's pretty old and poorly understood, so I think we need to be
cautious and take an incremental approach to changing it.

-- 
Jeff Layton <jlayton at redhat.com>


From bfields at fieldses.org  Tue Jun  4 16:17:43 2013
From: bfields at fieldses.org (J. Bruce Fields)
Date: Tue, 4 Jun 2013 16:17:43 -0400
Subject: [PATCH v1 04/11] locks: make "added" in __posix_lock_file a bool
In-Reply-To: <1370056054-25449-5-git-send-email-jlayton@redhat.com>
References: <1370056054-25449-1-git-send-email-jlayton@redhat.com>
 <1370056054-25449-5-git-send-email-jlayton@redhat.com>
Message-ID: <20130604201743.GA15594@fieldses.org>

On Fri, May 31, 2013 at 11:07:27PM -0400, Jeff Layton wrote:
> ...save 3 bytes of stack space.
> 
> Signed-off-by: Jeff Layton <jlayton at redhat.com>

ACK.--b.

> ---
>  fs/locks.c |    9 +++++----
>  1 files changed, 5 insertions(+), 4 deletions(-)
> 
> diff --git a/fs/locks.c b/fs/locks.c
> index a7d2253..cef0e04 100644
> --- a/fs/locks.c
> +++ b/fs/locks.c
> @@ -800,7 +800,8 @@ static int __posix_lock_file(struct inode *inode, struct file_lock *request, str
>  	struct file_lock *left = NULL;
>  	struct file_lock *right = NULL;
>  	struct file_lock **before;
> -	int error, added = 0;
> +	int error;
> +	bool added = false;
>  
>  	/*
>  	 * We may need two file_lock structures for this operation,
> @@ -894,7 +895,7 @@ static int __posix_lock_file(struct inode *inode, struct file_lock *request, str
>  				continue;
>  			}
>  			request = fl;
> -			added = 1;
> +			added = true;
>  		}
>  		else {
>  			/* Processing for different lock types is a bit
> @@ -905,7 +906,7 @@ static int __posix_lock_file(struct inode *inode, struct file_lock *request, str
>  			if (fl->fl_start > request->fl_end)
>  				break;
>  			if (request->fl_type == F_UNLCK)
> -				added = 1;
> +				added = true;
>  			if (fl->fl_start < request->fl_start)
>  				left = fl;
>  			/* If the next lock in the list has a higher end
> @@ -935,7 +936,7 @@ static int __posix_lock_file(struct inode *inode, struct file_lock *request, str
>  				locks_release_private(fl);
>  				locks_copy_private(fl, request);
>  				request = fl;
> -				added = 1;
> +				added = true;
>  			}
>  		}
>  		/* Go on to next lock.
> -- 
> 1.7.1
> 


From bfields at fieldses.org  Tue Jun  4 16:17:59 2013
From: bfields at fieldses.org (J. Bruce Fields)
Date: Tue, 4 Jun 2013 16:17:59 -0400
Subject: [PATCH v1 05/11] locks: encapsulate the fl_link list handling
In-Reply-To: <1370056054-25449-6-git-send-email-jlayton@redhat.com>
References: <1370056054-25449-1-git-send-email-jlayton@redhat.com>
 <1370056054-25449-6-git-send-email-jlayton@redhat.com>
Message-ID: <20130604201759.GB15594@fieldses.org>

On Fri, May 31, 2013 at 11:07:28PM -0400, Jeff Layton wrote:
> Move the fl_link list handling routines into a separate set of helpers.
> Also move the global list handling out of locks_insert_block, and into
> the caller that ends up triggering it as that allows us to eliminate the
> IS_POSIX check there.

ACK.--b.

> 
> Signed-off-by: Jeff Layton <jlayton at redhat.com>
> ---
>  fs/locks.c |   34 +++++++++++++++++++++++++++++-----
>  1 files changed, 29 insertions(+), 5 deletions(-)
> 
> diff --git a/fs/locks.c b/fs/locks.c
> index cef0e04..caca466 100644
> --- a/fs/locks.c
> +++ b/fs/locks.c
> @@ -494,13 +494,38 @@ static int posix_same_owner(struct file_lock *fl1, struct file_lock *fl2)
>  	return fl1->fl_owner == fl2->fl_owner;
>  }
>  
> +/* Remove a blocker or lock from one of the global lists */
> +static inline void
> +locks_insert_global_blocked(struct file_lock *waiter)
> +{
> +	list_add(&waiter->fl_link, &blocked_list);
> +}
> +
> +static inline void
> +locks_delete_global_blocked(struct file_lock *waiter)
> +{
> +	list_del_init(&waiter->fl_link);
> +}
> +
> +static inline void
> +locks_insert_global_locks(struct file_lock *waiter)
> +{
> +	list_add_tail(&waiter->fl_link, &file_lock_list);
> +}
> +
> +static inline void
> +locks_delete_global_locks(struct file_lock *waiter)
> +{
> +	list_del_init(&waiter->fl_link);
> +}
> +
>  /* Remove waiter from blocker's block list.
>   * When blocker ends up pointing to itself then the list is empty.
>   */
>  static void __locks_delete_block(struct file_lock *waiter)
>  {
>  	list_del_init(&waiter->fl_block);
> -	list_del_init(&waiter->fl_link);
> +	locks_delete_global_blocked(waiter);
>  	waiter->fl_next = NULL;
>  }
>  
> @@ -524,8 +549,6 @@ static void locks_insert_block(struct file_lock *blocker,
>  	BUG_ON(!list_empty(&waiter->fl_block));
>  	list_add_tail(&waiter->fl_block, &blocker->fl_block);
>  	waiter->fl_next = blocker;
> -	if (IS_POSIX(blocker))
> -		list_add(&waiter->fl_link, &blocked_list);
>  }
>  
>  /* Wake up processes blocked waiting for blocker.
> @@ -552,7 +575,7 @@ static void locks_wake_up_blocks(struct file_lock *blocker)
>   */
>  static void locks_insert_lock(struct file_lock **pos, struct file_lock *fl)
>  {
> -	list_add(&fl->fl_link, &file_lock_list);
> +	locks_insert_global_locks(fl);
>  
>  	fl->fl_nspid = get_pid(task_tgid(current));
>  
> @@ -573,7 +596,7 @@ static void locks_delete_lock(struct file_lock **thisfl_p)
>  
>  	*thisfl_p = fl->fl_next;
>  	fl->fl_next = NULL;
> -	list_del_init(&fl->fl_link);
> +	locks_delete_global_locks(fl);
>  
>  	if (fl->fl_nspid) {
>  		put_pid(fl->fl_nspid);
> @@ -839,6 +862,7 @@ static int __posix_lock_file(struct inode *inode, struct file_lock *request, str
>  				goto out;
>  			error = FILE_LOCK_DEFERRED;
>  			locks_insert_block(fl, request);
> +			locks_insert_global_blocked(request);
>  			goto out;
>    		}
>    	}
> -- 
> 1.7.1
> 


From bfields at fieldses.org  Tue Jun  4 17:22:08 2013
From: bfields at fieldses.org (J. Bruce Fields)
Date: Tue, 4 Jun 2013 17:22:08 -0400
Subject: [PATCH v1 06/11] locks: convert to i_lock to protect i_flock list
In-Reply-To: <1370056054-25449-7-git-send-email-jlayton@redhat.com>
References: <1370056054-25449-1-git-send-email-jlayton@redhat.com>
 <1370056054-25449-7-git-send-email-jlayton@redhat.com>
Message-ID: <20130604212208.GC15594@fieldses.org>

On Fri, May 31, 2013 at 11:07:29PM -0400, Jeff Layton wrote:
> Having a global lock that protects all of this code is a clear
> scalability problem. Instead of doing that, move most of the code to be
> protected by the i_lock instead.
> 
> The exceptions are the global lists that file_lock->fl_link sits on.
> Those still need a global lock of some sort, so wrap just those accesses
> in the file_lock_lock().
> 
> Note that this requires a small change to the /proc/locks code. Instead
> of walking the fl_block list to look for locks blocked on the current
> lock, we must instead walk the global blocker list and skip any that
> aren't blocked on the current lock. Otherwise, we'd need to take the
> i_lock on each inode as we go and that would create a lock inversion
> problem.

locks_insert lock sets fl_nspid after adding to the global list, so in
theory I think that could leave /proc/locks seeing a garbage fl_nspid
or fl_pid field.

I'm similarly worried about the deadlock detection code using the
fl_next value before it's set but I'm not sure what the consequences
are.

But I think both of those are fixable, and this looks OK otherwise?

Patch-sequencing nit: it'd be nice to have the big-but-boring
lock_flocks->spin_lock(&i_lock) search-and-replace in a separate patch
from the (briefer, but more subtle) deadlock and /proc/locks changes.
Off the top of my head all I can think of is bulk-replace of
lock_flocks() by lock_flocks(inode) (which ignores inode argument), then
change definition of the lock_flocks(inode) in the patch which does the
other stuff, then bulk replace of lock_flocks(inode) by
spin_lock(&inode->i_lock), but maybe there's something simpler.

--b.


> 
> Signed-off-by: Jeff Layton <jlayton at redhat.com>
> Signed-off-by: J. Bruce Fields <bfields at redhat.com>
> ---
>  Documentation/filesystems/Locking |   23 +++++--
>  fs/afs/flock.c                    |    5 +-
>  fs/ceph/locks.c                   |    2 +-
>  fs/ceph/mds_client.c              |    8 +-
>  fs/cifs/cifsfs.c                  |    2 +-
>  fs/cifs/file.c                    |   13 ++--
>  fs/gfs2/file.c                    |    2 +-
>  fs/lockd/svcsubs.c                |   12 ++--
>  fs/locks.c                        |  121 ++++++++++++++++++++-----------------
>  fs/nfs/delegation.c               |   11 ++--
>  fs/nfs/nfs4state.c                |    8 +-
>  fs/nfsd/nfs4state.c               |    8 +-
>  include/linux/fs.h                |   11 ----
>  13 files changed, 119 insertions(+), 107 deletions(-)
> 
> diff --git a/Documentation/filesystems/Locking b/Documentation/filesystems/Locking
> index 0706d32..13f91ab 100644
> --- a/Documentation/filesystems/Locking
> +++ b/Documentation/filesystems/Locking
> @@ -344,7 +344,7 @@ prototypes:
>  
>  
>  locking rules:
> -			file_lock_lock	may block
> +			inode->i_lock	may block
>  fl_copy_lock:		yes		no
>  fl_release_private:	maybe		no
>  
> @@ -357,12 +357,21 @@ prototypes:
>  	int (*lm_change)(struct file_lock **, int);
>  
>  locking rules:
> -			file_lock_lock	may block
> -lm_compare_owner:	yes		no
> -lm_notify:		yes		no
> -lm_grant:		no		no
> -lm_break:		yes		no
> -lm_change		yes		no
> +
> +			inode->i_lock	file_lock_lock	may block
> +lm_compare_owner:	yes		maybe		no
> +lm_notify:		yes		no		no
> +lm_grant:		no		no		no
> +lm_break:		yes		no		no
> +lm_change		yes		no		no
> +
> +	->lm_compare_owner is generally called with *an* inode->i_lock
> +held. It may not be the i_lock of the inode for either file_lock being
> +compared! This is the case with deadlock detection, since the code has
> +to chase down the owners of locks that may be entirely unrelated to the
> +one on which the lock is being acquired. For deadlock detection however,
> +the file_lock_lock is also held. The locks primarily ensure that neither
> +file_lock disappear out from under you while doing the comparison.
>  
>  --------------------------- buffer_head -----------------------------------
>  prototypes:
> diff --git a/fs/afs/flock.c b/fs/afs/flock.c
> index 2497bf3..03fc0d1 100644
> --- a/fs/afs/flock.c
> +++ b/fs/afs/flock.c
> @@ -252,6 +252,7 @@ static void afs_defer_unlock(struct afs_vnode *vnode, struct key *key)
>   */
>  static int afs_do_setlk(struct file *file, struct file_lock *fl)
>  {
> +	struct inode = file_inode(file);
>  	struct afs_vnode *vnode = AFS_FS_I(file->f_mapping->host);
>  	afs_lock_type_t type;
>  	struct key *key = file->private_data;
> @@ -273,7 +274,7 @@ static int afs_do_setlk(struct file *file, struct file_lock *fl)
>  
>  	type = (fl->fl_type == F_RDLCK) ? AFS_LOCK_READ : AFS_LOCK_WRITE;
>  
> -	lock_flocks();
> +	spin_lock(&inode->i_lock);
>  
>  	/* make sure we've got a callback on this file and that our view of the
>  	 * data version is up to date */
> @@ -420,7 +421,7 @@ given_lock:
>  	afs_vnode_fetch_status(vnode, NULL, key);
>  
>  error:
> -	unlock_flocks();
> +	spin_unlock(&inode->i_lock);
>  	_leave(" = %d", ret);
>  	return ret;
>  
> diff --git a/fs/ceph/locks.c b/fs/ceph/locks.c
> index 202dd3d..cd0a664 100644
> --- a/fs/ceph/locks.c
> +++ b/fs/ceph/locks.c
> @@ -194,7 +194,7 @@ void ceph_count_locks(struct inode *inode, int *fcntl_count, int *flock_count)
>   * Encode the flock and fcntl locks for the given inode into the pagelist.
>   * Format is: #fcntl locks, sequential fcntl locks, #flock locks,
>   * sequential flock locks.
> - * Must be called with lock_flocks() already held.
> + * Must be called with inode->i_lock already held.
>   * If we encounter more of a specific lock type than expected,
>   * we return the value 1.
>   */
> diff --git a/fs/ceph/mds_client.c b/fs/ceph/mds_client.c
> index 4f22671..ae621b5 100644
> --- a/fs/ceph/mds_client.c
> +++ b/fs/ceph/mds_client.c
> @@ -2482,13 +2482,13 @@ static int encode_caps_cb(struct inode *inode, struct ceph_cap *cap,
>  
>  		ceph_pagelist_set_cursor(pagelist, &trunc_point);
>  		do {
> -			lock_flocks();
> +			spin_lock(&inode->i_lock);
>  			ceph_count_locks(inode, &num_fcntl_locks,
>  					 &num_flock_locks);
>  			rec.v2.flock_len = (2*sizeof(u32) +
>  					    (num_fcntl_locks+num_flock_locks) *
>  					    sizeof(struct ceph_filelock));
> -			unlock_flocks();
> +			spin_unlock(&inode->i_lock);
>  
>  			/* pre-alloc pagelist */
>  			ceph_pagelist_truncate(pagelist, &trunc_point);
> @@ -2499,12 +2499,12 @@ static int encode_caps_cb(struct inode *inode, struct ceph_cap *cap,
>  
>  			/* encode locks */
>  			if (!err) {
> -				lock_flocks();
> +				spin_lock(&inode->i_lock);
>  				err = ceph_encode_locks(inode,
>  							pagelist,
>  							num_fcntl_locks,
>  							num_flock_locks);
> -				unlock_flocks();
> +				spin_unlock(&inode->i_lock);
>  			}
>  		} while (err == -ENOSPC);
>  	} else {
> diff --git a/fs/cifs/cifsfs.c b/fs/cifs/cifsfs.c
> index 72e4efe..29952d5 100644
> --- a/fs/cifs/cifsfs.c
> +++ b/fs/cifs/cifsfs.c
> @@ -768,7 +768,7 @@ static loff_t cifs_llseek(struct file *file, loff_t offset, int whence)
>  
>  static int cifs_setlease(struct file *file, long arg, struct file_lock **lease)
>  {
> -	/* note that this is called by vfs setlease with lock_flocks held
> +	/* note that this is called by vfs setlease with i_lock held
>  	   to protect *lease from going away */
>  	struct inode *inode = file_inode(file);
>  	struct cifsFileInfo *cfile = file->private_data;
> diff --git a/fs/cifs/file.c b/fs/cifs/file.c
> index 44a4f18..0dd10cd 100644
> --- a/fs/cifs/file.c
> +++ b/fs/cifs/file.c
> @@ -1092,6 +1092,7 @@ struct lock_to_push {
>  static int
>  cifs_push_posix_locks(struct cifsFileInfo *cfile)
>  {
> +	struct inode *inode = cfile->dentry->d_inode;
>  	struct cifs_tcon *tcon = tlink_tcon(cfile->tlink);
>  	struct file_lock *flock, **before;
>  	unsigned int count = 0, i = 0;
> @@ -1102,12 +1103,12 @@ cifs_push_posix_locks(struct cifsFileInfo *cfile)
>  
>  	xid = get_xid();
>  
> -	lock_flocks();
> -	cifs_for_each_lock(cfile->dentry->d_inode, before) {
> +	spin_lock(&inode->i_lock);
> +	cifs_for_each_lock(inode, before) {
>  		if ((*before)->fl_flags & FL_POSIX)
>  			count++;
>  	}
> -	unlock_flocks();
> +	spin_unlock(&inode->i_lock);
>  
>  	INIT_LIST_HEAD(&locks_to_send);
>  
> @@ -1126,8 +1127,8 @@ cifs_push_posix_locks(struct cifsFileInfo *cfile)
>  	}
>  
>  	el = locks_to_send.next;
> -	lock_flocks();
> -	cifs_for_each_lock(cfile->dentry->d_inode, before) {
> +	spin_lock(&inode->i_lock);
> +	cifs_for_each_lock(inode, before) {
>  		flock = *before;
>  		if ((flock->fl_flags & FL_POSIX) == 0)
>  			continue;
> @@ -1152,7 +1153,7 @@ cifs_push_posix_locks(struct cifsFileInfo *cfile)
>  		lck->offset = flock->fl_start;
>  		el = el->next;
>  	}
> -	unlock_flocks();
> +	spin_unlock(&inode->i_lock);
>  
>  	list_for_each_entry_safe(lck, tmp, &locks_to_send, llist) {
>  		int stored_rc;
> diff --git a/fs/gfs2/file.c b/fs/gfs2/file.c
> index acd1676..9e634e0 100644
> --- a/fs/gfs2/file.c
> +++ b/fs/gfs2/file.c
> @@ -889,7 +889,7 @@ out_uninit:
>   * cluster; until we do, disable leases (by just returning -EINVAL),
>   * unless the administrator has requested purely local locking.
>   *
> - * Locking: called under lock_flocks
> + * Locking: called under i_lock
>   *
>   * Returns: errno
>   */
> diff --git a/fs/lockd/svcsubs.c b/fs/lockd/svcsubs.c
> index 97e8741..dc5c759 100644
> --- a/fs/lockd/svcsubs.c
> +++ b/fs/lockd/svcsubs.c
> @@ -169,7 +169,7 @@ nlm_traverse_locks(struct nlm_host *host, struct nlm_file *file,
>  
>  again:
>  	file->f_locks = 0;
> -	lock_flocks(); /* protects i_flock list */
> +	spin_lock(&inode->i_lock);
>  	for (fl = inode->i_flock; fl; fl = fl->fl_next) {
>  		if (fl->fl_lmops != &nlmsvc_lock_operations)
>  			continue;
> @@ -181,7 +181,7 @@ again:
>  		if (match(lockhost, host)) {
>  			struct file_lock lock = *fl;
>  
> -			unlock_flocks();
> +			spin_unlock(&inode->i_lock);
>  			lock.fl_type  = F_UNLCK;
>  			lock.fl_start = 0;
>  			lock.fl_end   = OFFSET_MAX;
> @@ -193,7 +193,7 @@ again:
>  			goto again;
>  		}
>  	}
> -	unlock_flocks();
> +	spin_unlock(&inode->i_lock);
>  
>  	return 0;
>  }
> @@ -228,14 +228,14 @@ nlm_file_inuse(struct nlm_file *file)
>  	if (file->f_count || !list_empty(&file->f_blocks) || file->f_shares)
>  		return 1;
>  
> -	lock_flocks();
> +	spin_lock(&inode->i_lock);
>  	for (fl = inode->i_flock; fl; fl = fl->fl_next) {
>  		if (fl->fl_lmops == &nlmsvc_lock_operations) {
> -			unlock_flocks();
> +			spin_unlock(&inode->i_lock);
>  			return 1;
>  		}
>  	}
> -	unlock_flocks();
> +	spin_unlock(&inode->i_lock);
>  	file->f_locks = 0;
>  	return 0;
>  }
> diff --git a/fs/locks.c b/fs/locks.c
> index caca466..055c06c 100644
> --- a/fs/locks.c
> +++ b/fs/locks.c
> @@ -165,22 +165,9 @@ int lease_break_time = 45;
>  
>  static LIST_HEAD(file_lock_list);
>  static LIST_HEAD(blocked_list);
> -static DEFINE_SPINLOCK(file_lock_lock);
> -
> -/*
> - * Protects the two list heads above, plus the inode->i_flock list
> - */
> -void lock_flocks(void)
> -{
> -	spin_lock(&file_lock_lock);
> -}
> -EXPORT_SYMBOL_GPL(lock_flocks);
>  
> -void unlock_flocks(void)
> -{
> -	spin_unlock(&file_lock_lock);
> -}
> -EXPORT_SYMBOL_GPL(unlock_flocks);
> +/* Protects the two list heads above */
> +static DEFINE_SPINLOCK(file_lock_lock);
>  
>  static struct kmem_cache *filelock_cache __read_mostly;
>  
> @@ -498,25 +485,33 @@ static int posix_same_owner(struct file_lock *fl1, struct file_lock *fl2)
>  static inline void
>  locks_insert_global_blocked(struct file_lock *waiter)
>  {
> +	spin_lock(&file_lock_lock);
>  	list_add(&waiter->fl_link, &blocked_list);
> +	spin_unlock(&file_lock_lock);
>  }
>  
>  static inline void
>  locks_delete_global_blocked(struct file_lock *waiter)
>  {
> +	spin_lock(&file_lock_lock);
>  	list_del_init(&waiter->fl_link);
> +	spin_unlock(&file_lock_lock);
>  }
>  
>  static inline void
>  locks_insert_global_locks(struct file_lock *waiter)
>  {
> +	spin_lock(&file_lock_lock);
>  	list_add_tail(&waiter->fl_link, &file_lock_list);
> +	spin_unlock(&file_lock_lock);
>  }
>  
>  static inline void
>  locks_delete_global_locks(struct file_lock *waiter)
>  {
> +	spin_lock(&file_lock_lock);
>  	list_del_init(&waiter->fl_link);
> +	spin_unlock(&file_lock_lock);
>  }
>  
>  /* Remove waiter from blocker's block list.
> @@ -533,9 +528,11 @@ static void __locks_delete_block(struct file_lock *waiter)
>   */
>  static void locks_delete_block(struct file_lock *waiter)
>  {
> -	lock_flocks();
> +	struct inode *inode = file_inode(waiter->fl_file);
> +
> +	spin_lock(&inode->i_lock);
>  	__locks_delete_block(waiter);
> -	unlock_flocks();
> +	spin_unlock(&inode->i_lock);
>  }
>  
>  /* Insert waiter into blocker's block list.
> @@ -657,8 +654,9 @@ void
>  posix_test_lock(struct file *filp, struct file_lock *fl)
>  {
>  	struct file_lock *cfl;
> +	struct inode *inode = file_inode(filp);
>  
> -	lock_flocks();
> +	spin_lock(&inode->i_lock);
>  	for (cfl = file_inode(filp)->i_flock; cfl; cfl = cfl->fl_next) {
>  		if (!IS_POSIX(cfl))
>  			continue;
> @@ -671,7 +669,7 @@ posix_test_lock(struct file *filp, struct file_lock *fl)
>  			fl->fl_pid = pid_vnr(cfl->fl_nspid);
>  	} else
>  		fl->fl_type = F_UNLCK;
> -	unlock_flocks();
> +	spin_unlock(&inode->i_lock);
>  	return;
>  }
>  EXPORT_SYMBOL(posix_test_lock);
> @@ -719,14 +717,19 @@ static int posix_locks_deadlock(struct file_lock *caller_fl,
>  				struct file_lock *block_fl)
>  {
>  	int i = 0;
> +	int ret = 0;
>  
> +	spin_lock(&file_lock_lock);
>  	while ((block_fl = what_owner_is_waiting_for(block_fl))) {
>  		if (i++ > MAX_DEADLK_ITERATIONS)
> -			return 0;
> -		if (posix_same_owner(caller_fl, block_fl))
> -			return 1;
> +			break;
> +		if (posix_same_owner(caller_fl, block_fl)) {
> +			++ret;
> +			break;
> +		}
>  	}
> -	return 0;
> +	spin_unlock(&file_lock_lock);
> +	return ret;
>  }
>  
>  /* Try to create a FLOCK lock on filp. We always insert new FLOCK locks
> @@ -750,7 +753,7 @@ static int flock_lock_file(struct file *filp, struct file_lock *request)
>  			return -ENOMEM;
>  	}
>  
> -	lock_flocks();
> +	spin_lock(&inode->i_lock);
>  	if (request->fl_flags & FL_ACCESS)
>  		goto find_conflict;
>  
> @@ -780,9 +783,9 @@ static int flock_lock_file(struct file *filp, struct file_lock *request)
>  	 * give it the opportunity to lock the file.
>  	 */
>  	if (found) {
> -		unlock_flocks();
> +		spin_unlock(&inode->i_lock);
>  		cond_resched();
> -		lock_flocks();
> +		spin_lock(&inode->i_lock);
>  	}
>  
>  find_conflict:
> @@ -809,7 +812,7 @@ find_conflict:
>  	error = 0;
>  
>  out:
> -	unlock_flocks();
> +	spin_unlock(&inode->i_lock);
>  	if (new_fl)
>  		locks_free_lock(new_fl);
>  	return error;
> @@ -839,7 +842,7 @@ static int __posix_lock_file(struct inode *inode, struct file_lock *request, str
>  		new_fl2 = locks_alloc_lock();
>  	}
>  
> -	lock_flocks();
> +	spin_lock(&inode->i_lock);
>  	/*
>  	 * New lock request. Walk all POSIX locks and look for conflicts. If
>  	 * there are any, either return -EAGAIN or put the request on the
> @@ -1012,7 +1015,7 @@ static int __posix_lock_file(struct inode *inode, struct file_lock *request, str
>  		locks_wake_up_blocks(left);
>  	}
>   out:
> -	unlock_flocks();
> +	spin_unlock(&inode->i_lock);
>  	/*
>  	 * Free any unused locks.
>  	 */
> @@ -1087,14 +1090,14 @@ int locks_mandatory_locked(struct inode *inode)
>  	/*
>  	 * Search the lock list for this inode for any POSIX locks.
>  	 */
> -	lock_flocks();
> +	spin_lock(&inode->i_lock);
>  	for (fl = inode->i_flock; fl != NULL; fl = fl->fl_next) {
>  		if (!IS_POSIX(fl))
>  			continue;
>  		if (fl->fl_owner != owner)
>  			break;
>  	}
> -	unlock_flocks();
> +	spin_unlock(&inode->i_lock);
>  	return fl ? -EAGAIN : 0;
>  }
>  
> @@ -1237,7 +1240,7 @@ int __break_lease(struct inode *inode, unsigned int mode)
>  	if (IS_ERR(new_fl))
>  		return PTR_ERR(new_fl);
>  
> -	lock_flocks();
> +	spin_lock(&inode->i_lock);
>  
>  	time_out_leases(inode);
>  
> @@ -1287,10 +1290,10 @@ restart:
>  			break_time++;
>  	}
>  	locks_insert_block(flock, new_fl);
> -	unlock_flocks();
> +	spin_unlock(&inode->i_lock);
>  	error = wait_event_interruptible_timeout(new_fl->fl_wait,
>  						!new_fl->fl_next, break_time);
> -	lock_flocks();
> +	spin_lock(&inode->i_lock);
>  	__locks_delete_block(new_fl);
>  	if (error >= 0) {
>  		if (error == 0)
> @@ -1308,7 +1311,7 @@ restart:
>  	}
>  
>  out:
> -	unlock_flocks();
> +	spin_unlock(&inode->i_lock);
>  	locks_free_lock(new_fl);
>  	return error;
>  }
> @@ -1361,9 +1364,10 @@ EXPORT_SYMBOL(lease_get_mtime);
>  int fcntl_getlease(struct file *filp)
>  {
>  	struct file_lock *fl;
> +	struct inode *inode = file_inode(filp);
>  	int type = F_UNLCK;
>  
> -	lock_flocks();
> +	spin_lock(&inode->i_lock);
>  	time_out_leases(file_inode(filp));
>  	for (fl = file_inode(filp)->i_flock; fl && IS_LEASE(fl);
>  			fl = fl->fl_next) {
> @@ -1372,7 +1376,7 @@ int fcntl_getlease(struct file *filp)
>  			break;
>  		}
>  	}
> -	unlock_flocks();
> +	spin_unlock(&inode->i_lock);
>  	return type;
>  }
>  
> @@ -1466,7 +1470,7 @@ static int generic_delete_lease(struct file *filp, struct file_lock **flp)
>   *	The (input) flp->fl_lmops->lm_break function is required
>   *	by break_lease().
>   *
> - *	Called with file_lock_lock held.
> + *	Called with inode->i_lock held.
>   */
>  int generic_setlease(struct file *filp, long arg, struct file_lock **flp)
>  {
> @@ -1535,11 +1539,12 @@ static int __vfs_setlease(struct file *filp, long arg, struct file_lock **lease)
>  
>  int vfs_setlease(struct file *filp, long arg, struct file_lock **lease)
>  {
> +	struct inode *inode = file_inode(filp);
>  	int error;
>  
> -	lock_flocks();
> +	spin_lock(&inode->i_lock);
>  	error = __vfs_setlease(filp, arg, lease);
> -	unlock_flocks();
> +	spin_unlock(&inode->i_lock);
>  
>  	return error;
>  }
> @@ -1557,6 +1562,7 @@ static int do_fcntl_delete_lease(struct file *filp)
>  static int do_fcntl_add_lease(unsigned int fd, struct file *filp, long arg)
>  {
>  	struct file_lock *fl, *ret;
> +	struct inode *inode = file_inode(filp);
>  	struct fasync_struct *new;
>  	int error;
>  
> @@ -1570,10 +1576,10 @@ static int do_fcntl_add_lease(unsigned int fd, struct file *filp, long arg)
>  		return -ENOMEM;
>  	}
>  	ret = fl;
> -	lock_flocks();
> +	spin_lock(&inode->i_lock);
>  	error = __vfs_setlease(filp, arg, &ret);
>  	if (error) {
> -		unlock_flocks();
> +		spin_unlock(&inode->i_lock);
>  		locks_free_lock(fl);
>  		goto out_free_fasync;
>  	}
> @@ -1590,7 +1596,7 @@ static int do_fcntl_add_lease(unsigned int fd, struct file *filp, long arg)
>  		new = NULL;
>  
>  	error = __f_setown(filp, task_pid(current), PIDTYPE_PID, 0);
> -	unlock_flocks();
> +	spin_unlock(&inode->i_lock);
>  
>  out_free_fasync:
>  	if (new)
> @@ -2114,7 +2120,7 @@ void locks_remove_flock(struct file *filp)
>  			fl.fl_ops->fl_release_private(&fl);
>  	}
>  
> -	lock_flocks();
> +	spin_lock(&inode->i_lock);
>  	before = &inode->i_flock;
>  
>  	while ((fl = *before) != NULL) {
> @@ -2132,7 +2138,7 @@ void locks_remove_flock(struct file *filp)
>   		}
>  		before = &fl->fl_next;
>  	}
> -	unlock_flocks();
> +	spin_unlock(&inode->i_lock);
>  }
>  
>  /**
> @@ -2145,14 +2151,15 @@ void locks_remove_flock(struct file *filp)
>  int
>  posix_unblock_lock(struct file *filp, struct file_lock *waiter)
>  {
> +	struct inode *inode = file_inode(filp);
>  	int status = 0;
>  
> -	lock_flocks();
> +	spin_lock(&inode->i_lock);
>  	if (waiter->fl_next)
>  		__locks_delete_block(waiter);
>  	else
>  		status = -ENOENT;
> -	unlock_flocks();
> +	spin_unlock(&inode->i_lock);
>  	return status;
>  }
>  
> @@ -2257,8 +2264,10 @@ static int locks_show(struct seq_file *f, void *v)
>  
>  	lock_get_status(f, fl, *((loff_t *)f->private), "");
>  
> -	list_for_each_entry(bfl, &fl->fl_block, fl_block)
> -		lock_get_status(f, bfl, *((loff_t *)f->private), " ->");
> +	list_for_each_entry(bfl, &blocked_list, fl_link) {
> +		if (bfl->fl_next == fl)
> +			lock_get_status(f, bfl, *((loff_t *)f->private), " ->");
> +	}
>  
>  	return 0;
>  }
> @@ -2267,7 +2276,7 @@ static void *locks_start(struct seq_file *f, loff_t *pos)
>  {
>  	loff_t *p = f->private;
>  
> -	lock_flocks();
> +	spin_lock(&file_lock_lock);
>  	*p = (*pos + 1);
>  	return seq_list_start(&file_lock_list, *pos);
>  }
> @@ -2281,7 +2290,7 @@ static void *locks_next(struct seq_file *f, void *v, loff_t *pos)
>  
>  static void locks_stop(struct seq_file *f, void *v)
>  {
> -	unlock_flocks();
> +	spin_unlock(&file_lock_lock);
>  }
>  
>  static const struct seq_operations locks_seq_operations = {
> @@ -2328,7 +2337,8 @@ int lock_may_read(struct inode *inode, loff_t start, unsigned long len)
>  {
>  	struct file_lock *fl;
>  	int result = 1;
> -	lock_flocks();
> +
> +	spin_lock(&inode->i_lock);
>  	for (fl = inode->i_flock; fl != NULL; fl = fl->fl_next) {
>  		if (IS_POSIX(fl)) {
>  			if (fl->fl_type == F_RDLCK)
> @@ -2345,7 +2355,7 @@ int lock_may_read(struct inode *inode, loff_t start, unsigned long len)
>  		result = 0;
>  		break;
>  	}
> -	unlock_flocks();
> +	spin_unlock(&inode->i_lock);
>  	return result;
>  }
>  
> @@ -2368,7 +2378,8 @@ int lock_may_write(struct inode *inode, loff_t start, unsigned long len)
>  {
>  	struct file_lock *fl;
>  	int result = 1;
> -	lock_flocks();
> +
> +	spin_lock(&inode->i_lock);
>  	for (fl = inode->i_flock; fl != NULL; fl = fl->fl_next) {
>  		if (IS_POSIX(fl)) {
>  			if ((fl->fl_end < start) || (fl->fl_start > (start + len)))
> @@ -2383,7 +2394,7 @@ int lock_may_write(struct inode *inode, loff_t start, unsigned long len)
>  		result = 0;
>  		break;
>  	}
> -	unlock_flocks();
> +	spin_unlock(&inode->i_lock);
>  	return result;
>  }
>  
> diff --git a/fs/nfs/delegation.c b/fs/nfs/delegation.c
> index 57db324..43ee7f9 100644
> --- a/fs/nfs/delegation.c
> +++ b/fs/nfs/delegation.c
> @@ -73,20 +73,21 @@ static int nfs_delegation_claim_locks(struct nfs_open_context *ctx, struct nfs4_
>  	if (inode->i_flock == NULL)
>  		goto out;
>  
> -	/* Protect inode->i_flock using the file locks lock */
> -	lock_flocks();
> +	/* Protect inode->i_flock using the i_lock */
> +	spin_lock(&inode->i_lock);
>  	for (fl = inode->i_flock; fl != NULL; fl = fl->fl_next) {
>  		if (!(fl->fl_flags & (FL_POSIX|FL_FLOCK)))
>  			continue;
>  		if (nfs_file_open_context(fl->fl_file) != ctx)
>  			continue;
> -		unlock_flocks();
> +		/* FIXME: safe to drop lock here while walking list? */
> +		spin_unlock(&inode->i_lock);
>  		status = nfs4_lock_delegation_recall(fl, state, stateid);
>  		if (status < 0)
>  			goto out;
> -		lock_flocks();
> +		spin_lock(&inode->i_lock);
>  	}
> -	unlock_flocks();
> +	spin_unlock(&inode->i_lock);
>  out:
>  	return status;
>  }
> diff --git a/fs/nfs/nfs4state.c b/fs/nfs/nfs4state.c
> index 1fab140..ff10b4a 100644
> --- a/fs/nfs/nfs4state.c
> +++ b/fs/nfs/nfs4state.c
> @@ -1373,13 +1373,13 @@ static int nfs4_reclaim_locks(struct nfs4_state *state, const struct nfs4_state_
>  	/* Guard against delegation returns and new lock/unlock calls */
>  	down_write(&nfsi->rwsem);
>  	/* Protect inode->i_flock using the BKL */
> -	lock_flocks();
> +	spin_lock(&inode->i_lock);
>  	for (fl = inode->i_flock; fl != NULL; fl = fl->fl_next) {
>  		if (!(fl->fl_flags & (FL_POSIX|FL_FLOCK)))
>  			continue;
>  		if (nfs_file_open_context(fl->fl_file)->state != state)
>  			continue;
> -		unlock_flocks();
> +		spin_unlock(&inode->i_lock);
>  		status = ops->recover_lock(state, fl);
>  		switch (status) {
>  			case 0:
> @@ -1406,9 +1406,9 @@ static int nfs4_reclaim_locks(struct nfs4_state *state, const struct nfs4_state_
>  				/* kill_proc(fl->fl_pid, SIGLOST, 1); */
>  				status = 0;
>  		}
> -		lock_flocks();
> +		spin_lock(&inode->i_lock);
>  	}
> -	unlock_flocks();
> +	spin_unlock(&inode->i_lock);
>  out:
>  	up_write(&nfsi->rwsem);
>  	return status;
> diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
> index 316ec84..f170518 100644
> --- a/fs/nfsd/nfs4state.c
> +++ b/fs/nfsd/nfs4state.c
> @@ -2645,13 +2645,13 @@ static void nfsd_break_one_deleg(struct nfs4_delegation *dp)
>  
>  	list_add_tail(&dp->dl_recall_lru, &nn->del_recall_lru);
>  
> -	/* only place dl_time is set. protected by lock_flocks*/
> +	/* Only place dl_time is set; protected by i_lock: */
>  	dp->dl_time = get_seconds();
>  
>  	nfsd4_cb_recall(dp);
>  }
>  
> -/* Called from break_lease() with lock_flocks() held. */
> +/* Called from break_lease() with i_lock held. */
>  static void nfsd_break_deleg_cb(struct file_lock *fl)
>  {
>  	struct nfs4_file *fp = (struct nfs4_file *)fl->fl_owner;
> @@ -4520,7 +4520,7 @@ check_for_locks(struct nfs4_file *filp, struct nfs4_lockowner *lowner)
>  	struct inode *inode = filp->fi_inode;
>  	int status = 0;
>  
> -	lock_flocks();
> +	spin_lock(&inode->i_lock);
>  	for (flpp = &inode->i_flock; *flpp != NULL; flpp = &(*flpp)->fl_next) {
>  		if ((*flpp)->fl_owner == (fl_owner_t)lowner) {
>  			status = 1;
> @@ -4528,7 +4528,7 @@ check_for_locks(struct nfs4_file *filp, struct nfs4_lockowner *lowner)
>  		}
>  	}
>  out:
> -	unlock_flocks();
> +	spin_unlock(&inode->i_lock);
>  	return status;
>  }
>  
> diff --git a/include/linux/fs.h b/include/linux/fs.h
> index ae377e9..ccb44ea 100644
> --- a/include/linux/fs.h
> +++ b/include/linux/fs.h
> @@ -1012,8 +1012,6 @@ extern int vfs_setlease(struct file *, long, struct file_lock **);
>  extern int lease_modify(struct file_lock **, int);
>  extern int lock_may_read(struct inode *, loff_t start, unsigned long count);
>  extern int lock_may_write(struct inode *, loff_t start, unsigned long count);
> -extern void lock_flocks(void);
> -extern void unlock_flocks(void);
>  #else /* !CONFIG_FILE_LOCKING */
>  static inline int fcntl_getlk(struct file *file, struct flock __user *user)
>  {
> @@ -1155,15 +1153,6 @@ static inline int lock_may_write(struct inode *inode, loff_t start,
>  {
>  	return 1;
>  }
> -
> -static inline void lock_flocks(void)
> -{
> -}
> -
> -static inline void unlock_flocks(void)
> -{
> -}
> -
>  #endif /* !CONFIG_FILE_LOCKING */
>  
>  
> -- 
> 1.7.1
> 


From bfields at fieldses.org  Tue Jun  4 17:58:39 2013
From: bfields at fieldses.org (J. Bruce Fields)
Date: Tue, 4 Jun 2013 17:58:39 -0400
Subject: [PATCH v1 07/11] locks: only pull entries off of blocked_list
 when they are really unblocked
In-Reply-To: <1370056054-25449-8-git-send-email-jlayton@redhat.com>
References: <1370056054-25449-1-git-send-email-jlayton@redhat.com>
 <1370056054-25449-8-git-send-email-jlayton@redhat.com>
Message-ID: <20130604215839.GD15594@fieldses.org>

On Fri, May 31, 2013 at 11:07:30PM -0400, Jeff Layton wrote:
> Currently, when there is a lot of lock contention the kernel spends an
> inordinate amount of time taking blocked locks off of the global
> blocked_list and then putting them right back on again. When all of this
> code was protected by a single lock, then it didn't matter much, but now
> it means a lot of file_lock_lock thrashing.
> 
> Optimize this a bit by deferring the removal from the blocked_list until
> we're either applying or cancelling the lock. By doing this, and using a
> lockless list_empty check, we can avoid taking the file_lock_lock in
> many cases.
> 
> Because the fl_link check is lockless, we must ensure that only the task
> that "owns" the request manipulates the fl_link. Also, with this change,
> it's possible that we'll see an entry on the blocked_list that has a
> NULL fl_next pointer. In that event, just ignore it and continue walking
> the list.

OK, that sounds safe as in it shouldn't crash, but does the deadlock
detection still work, or can it miss loops?

Those locks that are temporarily NULL would previously not have been on
the list at all, OK, but...  I'm having trouble reasoning about how this
works now.

Previously a single lock was held interrupted across
posix_locks_deadlock and locks_insert_block() which guaranteed we
shouldn't be adding a loop, is that still true?

--b.

> 
> Signed-off-by: Jeff Layton <jlayton at redhat.com>
> ---
>  fs/locks.c |   29 +++++++++++++++++++++++------
>  1 files changed, 23 insertions(+), 6 deletions(-)
> 
> diff --git a/fs/locks.c b/fs/locks.c
> index 055c06c..fc35b9e 100644
> --- a/fs/locks.c
> +++ b/fs/locks.c
> @@ -520,7 +520,6 @@ locks_delete_global_locks(struct file_lock *waiter)
>  static void __locks_delete_block(struct file_lock *waiter)
>  {
>  	list_del_init(&waiter->fl_block);
> -	locks_delete_global_blocked(waiter);
>  	waiter->fl_next = NULL;
>  }
>  
> @@ -704,13 +703,16 @@ EXPORT_SYMBOL(posix_test_lock);
>  /* Find a lock that the owner of the given block_fl is blocking on. */
>  static struct file_lock *what_owner_is_waiting_for(struct file_lock *block_fl)
>  {
> -	struct file_lock *fl;
> +	struct file_lock *fl, *ret = NULL;
>  
>  	list_for_each_entry(fl, &blocked_list, fl_link) {
> -		if (posix_same_owner(fl, block_fl))
> -			return fl->fl_next;
> +		if (posix_same_owner(fl, block_fl)) {
> +			ret = fl->fl_next;
> +			if (likely(ret))
> +				break;
> +		}
>  	}
> -	return NULL;
> +	return ret;
>  }
>  
>  static int posix_locks_deadlock(struct file_lock *caller_fl,
> @@ -865,7 +867,8 @@ static int __posix_lock_file(struct inode *inode, struct file_lock *request, str
>  				goto out;
>  			error = FILE_LOCK_DEFERRED;
>  			locks_insert_block(fl, request);
> -			locks_insert_global_blocked(request);
> +			if (list_empty(&request->fl_link))
> +				locks_insert_global_blocked(request);
>  			goto out;
>    		}
>    	}
> @@ -876,6 +879,16 @@ static int __posix_lock_file(struct inode *inode, struct file_lock *request, str
>  		goto out;
>  
>  	/*
> +	 * Now that we know the request is no longer blocked, we can take it
> +	 * off the global list. Some callers send down partially initialized
> +	 * requests, so we only do this if FL_SLEEP is set. Also, avoid taking
> +	 * the lock if the list is empty, as that indicates a request that
> +	 * never blocked.
> +	 */
> +	if ((request->fl_flags & FL_SLEEP) && !list_empty(&request->fl_link))
> +		locks_delete_global_blocked(request);
> +
> +	/*
>  	 * Find the first old lock with the same owner as the new lock.
>  	 */
>  	
> @@ -1069,6 +1082,7 @@ int posix_lock_file_wait(struct file *filp, struct file_lock *fl)
>  			continue;
>  
>  		locks_delete_block(fl);
> +		locks_delete_global_blocked(fl);
>  		break;
>  	}
>  	return error;
> @@ -1147,6 +1161,7 @@ int locks_mandatory_area(int read_write, struct inode *inode,
>  		}
>  
>  		locks_delete_block(&fl);
> +		locks_delete_global_blocked(&fl);
>  		break;
>  	}
>  
> @@ -1859,6 +1874,7 @@ static int do_lock_file_wait(struct file *filp, unsigned int cmd,
>  			continue;
>  
>  		locks_delete_block(fl);
> +		locks_delete_global_blocked(fl);
>  		break;
>  	}
>  
> @@ -2160,6 +2176,7 @@ posix_unblock_lock(struct file *filp, struct file_lock *waiter)
>  	else
>  		status = -ENOENT;
>  	spin_unlock(&inode->i_lock);
> +	locks_delete_global_blocked(waiter);
>  	return status;
>  }
>  
> -- 
> 1.7.1
> 


From bfields at fieldses.org  Tue Jun  4 17:59:50 2013
From: bfields at fieldses.org (J. Bruce Fields)
Date: Tue, 4 Jun 2013 17:59:50 -0400
Subject: [PATCH v1 08/11] locks: convert fl_link to a hlist_node
In-Reply-To: <1370056054-25449-9-git-send-email-jlayton@redhat.com>
References: <1370056054-25449-1-git-send-email-jlayton@redhat.com>
 <1370056054-25449-9-git-send-email-jlayton@redhat.com>
Message-ID: <20130604215950.GE15594@fieldses.org>

On Fri, May 31, 2013 at 11:07:31PM -0400, Jeff Layton wrote:
> Testing has shown that iterating over the blocked_list for deadlock
> detection turns out to be a bottleneck. In order to alleviate that,
> begin the process of turning it into a hashtable. We start by turning
> the fl_link into a hlist_node and the global lists into hlists. A later
> patch will do the conversion of the blocked_list to a hashtable.

Even simpler would be if we could add a pointer to the (well, a) lock
that a lockowner is blocking on, and then we'd just have to follow a
pointer.  I haven't thought that through, though, perhaps that's hard ot
make work....

--b.

> 
> Signed-off-by: Jeff Layton <jlayton at redhat.com>
> ---
>  fs/locks.c         |   32 ++++++++++++++++----------------
>  include/linux/fs.h |    2 +-
>  2 files changed, 17 insertions(+), 17 deletions(-)
> 
> diff --git a/fs/locks.c b/fs/locks.c
> index fc35b9e..5ed056b 100644
> --- a/fs/locks.c
> +++ b/fs/locks.c
> @@ -163,8 +163,8 @@ int lease_break_time = 45;
>  #define for_each_lock(inode, lockp) \
>  	for (lockp = &inode->i_flock; *lockp != NULL; lockp = &(*lockp)->fl_next)
>  
> -static LIST_HEAD(file_lock_list);
> -static LIST_HEAD(blocked_list);
> +static HLIST_HEAD(file_lock_list);
> +static HLIST_HEAD(blocked_list);
>  
>  /* Protects the two list heads above */
>  static DEFINE_SPINLOCK(file_lock_lock);
> @@ -173,7 +173,7 @@ static struct kmem_cache *filelock_cache __read_mostly;
>  
>  static void locks_init_lock_heads(struct file_lock *fl)
>  {
> -	INIT_LIST_HEAD(&fl->fl_link);
> +	INIT_HLIST_NODE(&fl->fl_link);
>  	INIT_LIST_HEAD(&fl->fl_block);
>  	init_waitqueue_head(&fl->fl_wait);
>  }
> @@ -207,7 +207,7 @@ void locks_free_lock(struct file_lock *fl)
>  {
>  	BUG_ON(waitqueue_active(&fl->fl_wait));
>  	BUG_ON(!list_empty(&fl->fl_block));
> -	BUG_ON(!list_empty(&fl->fl_link));
> +	BUG_ON(!hlist_unhashed(&fl->fl_link));
>  
>  	locks_release_private(fl);
>  	kmem_cache_free(filelock_cache, fl);
> @@ -486,7 +486,7 @@ static inline void
>  locks_insert_global_blocked(struct file_lock *waiter)
>  {
>  	spin_lock(&file_lock_lock);
> -	list_add(&waiter->fl_link, &blocked_list);
> +	hlist_add_head(&waiter->fl_link, &blocked_list);
>  	spin_unlock(&file_lock_lock);
>  }
>  
> @@ -494,7 +494,7 @@ static inline void
>  locks_delete_global_blocked(struct file_lock *waiter)
>  {
>  	spin_lock(&file_lock_lock);
> -	list_del_init(&waiter->fl_link);
> +	hlist_del_init(&waiter->fl_link);
>  	spin_unlock(&file_lock_lock);
>  }
>  
> @@ -502,7 +502,7 @@ static inline void
>  locks_insert_global_locks(struct file_lock *waiter)
>  {
>  	spin_lock(&file_lock_lock);
> -	list_add_tail(&waiter->fl_link, &file_lock_list);
> +	hlist_add_head(&waiter->fl_link, &file_lock_list);
>  	spin_unlock(&file_lock_lock);
>  }
>  
> @@ -510,7 +510,7 @@ static inline void
>  locks_delete_global_locks(struct file_lock *waiter)
>  {
>  	spin_lock(&file_lock_lock);
> -	list_del_init(&waiter->fl_link);
> +	hlist_del_init(&waiter->fl_link);
>  	spin_unlock(&file_lock_lock);
>  }
>  
> @@ -705,7 +705,7 @@ static struct file_lock *what_owner_is_waiting_for(struct file_lock *block_fl)
>  {
>  	struct file_lock *fl, *ret = NULL;
>  
> -	list_for_each_entry(fl, &blocked_list, fl_link) {
> +	hlist_for_each_entry(fl, &blocked_list, fl_link) {
>  		if (posix_same_owner(fl, block_fl)) {
>  			ret = fl->fl_next;
>  			if (likely(ret))
> @@ -867,7 +867,7 @@ static int __posix_lock_file(struct inode *inode, struct file_lock *request, str
>  				goto out;
>  			error = FILE_LOCK_DEFERRED;
>  			locks_insert_block(fl, request);
> -			if (list_empty(&request->fl_link))
> +			if (hlist_unhashed(&request->fl_link))
>  				locks_insert_global_blocked(request);
>  			goto out;
>    		}
> @@ -882,10 +882,10 @@ static int __posix_lock_file(struct inode *inode, struct file_lock *request, str
>  	 * Now that we know the request is no longer blocked, we can take it
>  	 * off the global list. Some callers send down partially initialized
>  	 * requests, so we only do this if FL_SLEEP is set. Also, avoid taking
> -	 * the lock if the list is empty, as that indicates a request that
> +	 * the lock if the hlist is unhashed, as that indicates a request that
>  	 * never blocked.
>  	 */
> -	if ((request->fl_flags & FL_SLEEP) && !list_empty(&request->fl_link))
> +	if ((request->fl_flags & FL_SLEEP) && !hlist_unhashed(&request->fl_link))
>  		locks_delete_global_blocked(request);
>  
>  	/*
> @@ -2277,11 +2277,11 @@ static int locks_show(struct seq_file *f, void *v)
>  {
>  	struct file_lock *fl, *bfl;
>  
> -	fl = list_entry(v, struct file_lock, fl_link);
> +	fl = hlist_entry(v, struct file_lock, fl_link);
>  
>  	lock_get_status(f, fl, *((loff_t *)f->private), "");
>  
> -	list_for_each_entry(bfl, &blocked_list, fl_link) {
> +	hlist_for_each_entry(bfl, &blocked_list, fl_link) {
>  		if (bfl->fl_next == fl)
>  			lock_get_status(f, bfl, *((loff_t *)f->private), " ->");
>  	}
> @@ -2295,14 +2295,14 @@ static void *locks_start(struct seq_file *f, loff_t *pos)
>  
>  	spin_lock(&file_lock_lock);
>  	*p = (*pos + 1);
> -	return seq_list_start(&file_lock_list, *pos);
> +	return seq_hlist_start(&file_lock_list, *pos);
>  }
>  
>  static void *locks_next(struct seq_file *f, void *v, loff_t *pos)
>  {
>  	loff_t *p = f->private;
>  	++*p;
> -	return seq_list_next(v, &file_lock_list, pos);
> +	return seq_hlist_next(v, &file_lock_list, pos);
>  }
>  
>  static void locks_stop(struct seq_file *f, void *v)
> diff --git a/include/linux/fs.h b/include/linux/fs.h
> index ccb44ea..07a009e 100644
> --- a/include/linux/fs.h
> +++ b/include/linux/fs.h
> @@ -934,7 +934,7 @@ int locks_in_grace(struct net *);
>   */
>  struct file_lock {
>  	struct file_lock *fl_next;	/* singly linked list for this inode  */
> -	struct list_head fl_link;	/* doubly linked list of all locks */
> +	struct hlist_node fl_link;	/* node in global lists */
>  	struct list_head fl_block;	/* circular list of blocked processes */
>  	fl_owner_t fl_owner;
>  	unsigned int fl_flags;
> -- 
> 1.7.1
> 


From jlayton at redhat.com  Tue Jun  4 20:46:04 2013
From: jlayton at redhat.com (Jeff Layton)
Date: Tue, 4 Jun 2013 20:46:04 -0400
Subject: [PATCH v1 06/11] locks: convert to i_lock to protect i_flock list
In-Reply-To: <20130604212208.GC15594@fieldses.org>
References: <1370056054-25449-1-git-send-email-jlayton@redhat.com>
 <1370056054-25449-7-git-send-email-jlayton@redhat.com>
 <20130604212208.GC15594@fieldses.org>
Message-ID: <20130604204604.4848f3d9@corrin.poochiereds.net>

On Tue, 4 Jun 2013 17:22:08 -0400
"J. Bruce Fields" <bfields at fieldses.org> wrote:

> On Fri, May 31, 2013 at 11:07:29PM -0400, Jeff Layton wrote:
> > Having a global lock that protects all of this code is a clear
> > scalability problem. Instead of doing that, move most of the code to be
> > protected by the i_lock instead.
> > 
> > The exceptions are the global lists that file_lock->fl_link sits on.
> > Those still need a global lock of some sort, so wrap just those accesses
> > in the file_lock_lock().
> > 
> > Note that this requires a small change to the /proc/locks code. Instead
> > of walking the fl_block list to look for locks blocked on the current
> > lock, we must instead walk the global blocker list and skip any that
> > aren't blocked on the current lock. Otherwise, we'd need to take the
> > i_lock on each inode as we go and that would create a lock inversion
> > problem.
> 
> locks_insert lock sets fl_nspid after adding to the global list, so in
> theory I think that could leave /proc/locks seeing a garbage fl_nspid
> or fl_pid field.
> 

Well spotted. I think we can simply fix that by ensuring that we do the
insert onto the global list as the last thing in locks_insert_lock and
remove it from the global list first thing in locks_delete_lock. I'll
do that and test it out before I send the next iteration.

> I'm similarly worried about the deadlock detection code using the
> fl_next value before it's set but I'm not sure what the consequences
> are.
> 

This I'm less clear on. We already do the insert onto the global
blocked list after locks_insert_block. Am I missing something here?

> But I think both of those are fixable, and this looks OK otherwise?
> 
> Patch-sequencing nit: it'd be nice to have the big-but-boring
> lock_flocks->spin_lock(&i_lock) search-and-replace in a separate patch
> from the (briefer, but more subtle) deadlock and /proc/locks changes.
> Off the top of my head all I can think of is bulk-replace of
> lock_flocks() by lock_flocks(inode) (which ignores inode argument), then
> change definition of the lock_flocks(inode) in the patch which does the
> other stuff, then bulk replace of lock_flocks(inode) by
> spin_lock(&inode->i_lock), but maybe there's something simpler.
> 

Ok, maybe I can do some of these other changes before we switch to the
i_lock, and then do the big boring change after that. I'll see if I can
cook up a reasonable patch for that for the next iteration.

> --b.
> 
> 
> > 
> > Signed-off-by: Jeff Layton <jlayton at redhat.com>
> > Signed-off-by: J. Bruce Fields <bfields at redhat.com>
> > ---
> >  Documentation/filesystems/Locking |   23 +++++--
> >  fs/afs/flock.c                    |    5 +-
> >  fs/ceph/locks.c                   |    2 +-
> >  fs/ceph/mds_client.c              |    8 +-
> >  fs/cifs/cifsfs.c                  |    2 +-
> >  fs/cifs/file.c                    |   13 ++--
> >  fs/gfs2/file.c                    |    2 +-
> >  fs/lockd/svcsubs.c                |   12 ++--
> >  fs/locks.c                        |  121 ++++++++++++++++++++-----------------
> >  fs/nfs/delegation.c               |   11 ++--
> >  fs/nfs/nfs4state.c                |    8 +-
> >  fs/nfsd/nfs4state.c               |    8 +-
> >  include/linux/fs.h                |   11 ----
> >  13 files changed, 119 insertions(+), 107 deletions(-)
> > 
> > diff --git a/Documentation/filesystems/Locking b/Documentation/filesystems/Locking
> > index 0706d32..13f91ab 100644
> > --- a/Documentation/filesystems/Locking
> > +++ b/Documentation/filesystems/Locking
> > @@ -344,7 +344,7 @@ prototypes:
> >  
> >  
> >  locking rules:
> > -			file_lock_lock	may block
> > +			inode->i_lock	may block
> >  fl_copy_lock:		yes		no
> >  fl_release_private:	maybe		no
> >  
> > @@ -357,12 +357,21 @@ prototypes:
> >  	int (*lm_change)(struct file_lock **, int);
> >  
> >  locking rules:
> > -			file_lock_lock	may block
> > -lm_compare_owner:	yes		no
> > -lm_notify:		yes		no
> > -lm_grant:		no		no
> > -lm_break:		yes		no
> > -lm_change		yes		no
> > +
> > +			inode->i_lock	file_lock_lock	may block
> > +lm_compare_owner:	yes		maybe		no
> > +lm_notify:		yes		no		no
> > +lm_grant:		no		no		no
> > +lm_break:		yes		no		no
> > +lm_change		yes		no		no
> > +
> > +	->lm_compare_owner is generally called with *an* inode->i_lock
> > +held. It may not be the i_lock of the inode for either file_lock being
> > +compared! This is the case with deadlock detection, since the code has
> > +to chase down the owners of locks that may be entirely unrelated to the
> > +one on which the lock is being acquired. For deadlock detection however,
> > +the file_lock_lock is also held. The locks primarily ensure that neither
> > +file_lock disappear out from under you while doing the comparison.
> >  
> >  --------------------------- buffer_head -----------------------------------
> >  prototypes:
> > diff --git a/fs/afs/flock.c b/fs/afs/flock.c
> > index 2497bf3..03fc0d1 100644
> > --- a/fs/afs/flock.c
> > +++ b/fs/afs/flock.c
> > @@ -252,6 +252,7 @@ static void afs_defer_unlock(struct afs_vnode *vnode, struct key *key)
> >   */
> >  static int afs_do_setlk(struct file *file, struct file_lock *fl)
> >  {
> > +	struct inode = file_inode(file);
> >  	struct afs_vnode *vnode = AFS_FS_I(file->f_mapping->host);
> >  	afs_lock_type_t type;
> >  	struct key *key = file->private_data;
> > @@ -273,7 +274,7 @@ static int afs_do_setlk(struct file *file, struct file_lock *fl)
> >  
> >  	type = (fl->fl_type == F_RDLCK) ? AFS_LOCK_READ : AFS_LOCK_WRITE;
> >  
> > -	lock_flocks();
> > +	spin_lock(&inode->i_lock);
> >  
> >  	/* make sure we've got a callback on this file and that our view of the
> >  	 * data version is up to date */
> > @@ -420,7 +421,7 @@ given_lock:
> >  	afs_vnode_fetch_status(vnode, NULL, key);
> >  
> >  error:
> > -	unlock_flocks();
> > +	spin_unlock(&inode->i_lock);
> >  	_leave(" = %d", ret);
> >  	return ret;
> >  
> > diff --git a/fs/ceph/locks.c b/fs/ceph/locks.c
> > index 202dd3d..cd0a664 100644
> > --- a/fs/ceph/locks.c
> > +++ b/fs/ceph/locks.c
> > @@ -194,7 +194,7 @@ void ceph_count_locks(struct inode *inode, int *fcntl_count, int *flock_count)
> >   * Encode the flock and fcntl locks for the given inode into the pagelist.
> >   * Format is: #fcntl locks, sequential fcntl locks, #flock locks,
> >   * sequential flock locks.
> > - * Must be called with lock_flocks() already held.
> > + * Must be called with inode->i_lock already held.
> >   * If we encounter more of a specific lock type than expected,
> >   * we return the value 1.
> >   */
> > diff --git a/fs/ceph/mds_client.c b/fs/ceph/mds_client.c
> > index 4f22671..ae621b5 100644
> > --- a/fs/ceph/mds_client.c
> > +++ b/fs/ceph/mds_client.c
> > @@ -2482,13 +2482,13 @@ static int encode_caps_cb(struct inode *inode, struct ceph_cap *cap,
> >  
> >  		ceph_pagelist_set_cursor(pagelist, &trunc_point);
> >  		do {
> > -			lock_flocks();
> > +			spin_lock(&inode->i_lock);
> >  			ceph_count_locks(inode, &num_fcntl_locks,
> >  					 &num_flock_locks);
> >  			rec.v2.flock_len = (2*sizeof(u32) +
> >  					    (num_fcntl_locks+num_flock_locks) *
> >  					    sizeof(struct ceph_filelock));
> > -			unlock_flocks();
> > +			spin_unlock(&inode->i_lock);
> >  
> >  			/* pre-alloc pagelist */
> >  			ceph_pagelist_truncate(pagelist, &trunc_point);
> > @@ -2499,12 +2499,12 @@ static int encode_caps_cb(struct inode *inode, struct ceph_cap *cap,
> >  
> >  			/* encode locks */
> >  			if (!err) {
> > -				lock_flocks();
> > +				spin_lock(&inode->i_lock);
> >  				err = ceph_encode_locks(inode,
> >  							pagelist,
> >  							num_fcntl_locks,
> >  							num_flock_locks);
> > -				unlock_flocks();
> > +				spin_unlock(&inode->i_lock);
> >  			}
> >  		} while (err == -ENOSPC);
> >  	} else {
> > diff --git a/fs/cifs/cifsfs.c b/fs/cifs/cifsfs.c
> > index 72e4efe..29952d5 100644
> > --- a/fs/cifs/cifsfs.c
> > +++ b/fs/cifs/cifsfs.c
> > @@ -768,7 +768,7 @@ static loff_t cifs_llseek(struct file *file, loff_t offset, int whence)
> >  
> >  static int cifs_setlease(struct file *file, long arg, struct file_lock **lease)
> >  {
> > -	/* note that this is called by vfs setlease with lock_flocks held
> > +	/* note that this is called by vfs setlease with i_lock held
> >  	   to protect *lease from going away */
> >  	struct inode *inode = file_inode(file);
> >  	struct cifsFileInfo *cfile = file->private_data;
> > diff --git a/fs/cifs/file.c b/fs/cifs/file.c
> > index 44a4f18..0dd10cd 100644
> > --- a/fs/cifs/file.c
> > +++ b/fs/cifs/file.c
> > @@ -1092,6 +1092,7 @@ struct lock_to_push {
> >  static int
> >  cifs_push_posix_locks(struct cifsFileInfo *cfile)
> >  {
> > +	struct inode *inode = cfile->dentry->d_inode;
> >  	struct cifs_tcon *tcon = tlink_tcon(cfile->tlink);
> >  	struct file_lock *flock, **before;
> >  	unsigned int count = 0, i = 0;
> > @@ -1102,12 +1103,12 @@ cifs_push_posix_locks(struct cifsFileInfo *cfile)
> >  
> >  	xid = get_xid();
> >  
> > -	lock_flocks();
> > -	cifs_for_each_lock(cfile->dentry->d_inode, before) {
> > +	spin_lock(&inode->i_lock);
> > +	cifs_for_each_lock(inode, before) {
> >  		if ((*before)->fl_flags & FL_POSIX)
> >  			count++;
> >  	}
> > -	unlock_flocks();
> > +	spin_unlock(&inode->i_lock);
> >  
> >  	INIT_LIST_HEAD(&locks_to_send);
> >  
> > @@ -1126,8 +1127,8 @@ cifs_push_posix_locks(struct cifsFileInfo *cfile)
> >  	}
> >  
> >  	el = locks_to_send.next;
> > -	lock_flocks();
> > -	cifs_for_each_lock(cfile->dentry->d_inode, before) {
> > +	spin_lock(&inode->i_lock);
> > +	cifs_for_each_lock(inode, before) {
> >  		flock = *before;
> >  		if ((flock->fl_flags & FL_POSIX) == 0)
> >  			continue;
> > @@ -1152,7 +1153,7 @@ cifs_push_posix_locks(struct cifsFileInfo *cfile)
> >  		lck->offset = flock->fl_start;
> >  		el = el->next;
> >  	}
> > -	unlock_flocks();
> > +	spin_unlock(&inode->i_lock);
> >  
> >  	list_for_each_entry_safe(lck, tmp, &locks_to_send, llist) {
> >  		int stored_rc;
> > diff --git a/fs/gfs2/file.c b/fs/gfs2/file.c
> > index acd1676..9e634e0 100644
> > --- a/fs/gfs2/file.c
> > +++ b/fs/gfs2/file.c
> > @@ -889,7 +889,7 @@ out_uninit:
> >   * cluster; until we do, disable leases (by just returning -EINVAL),
> >   * unless the administrator has requested purely local locking.
> >   *
> > - * Locking: called under lock_flocks
> > + * Locking: called under i_lock
> >   *
> >   * Returns: errno
> >   */
> > diff --git a/fs/lockd/svcsubs.c b/fs/lockd/svcsubs.c
> > index 97e8741..dc5c759 100644
> > --- a/fs/lockd/svcsubs.c
> > +++ b/fs/lockd/svcsubs.c
> > @@ -169,7 +169,7 @@ nlm_traverse_locks(struct nlm_host *host, struct nlm_file *file,
> >  
> >  again:
> >  	file->f_locks = 0;
> > -	lock_flocks(); /* protects i_flock list */
> > +	spin_lock(&inode->i_lock);
> >  	for (fl = inode->i_flock; fl; fl = fl->fl_next) {
> >  		if (fl->fl_lmops != &nlmsvc_lock_operations)
> >  			continue;
> > @@ -181,7 +181,7 @@ again:
> >  		if (match(lockhost, host)) {
> >  			struct file_lock lock = *fl;
> >  
> > -			unlock_flocks();
> > +			spin_unlock(&inode->i_lock);
> >  			lock.fl_type  = F_UNLCK;
> >  			lock.fl_start = 0;
> >  			lock.fl_end   = OFFSET_MAX;
> > @@ -193,7 +193,7 @@ again:
> >  			goto again;
> >  		}
> >  	}
> > -	unlock_flocks();
> > +	spin_unlock(&inode->i_lock);
> >  
> >  	return 0;
> >  }
> > @@ -228,14 +228,14 @@ nlm_file_inuse(struct nlm_file *file)
> >  	if (file->f_count || !list_empty(&file->f_blocks) || file->f_shares)
> >  		return 1;
> >  
> > -	lock_flocks();
> > +	spin_lock(&inode->i_lock);
> >  	for (fl = inode->i_flock; fl; fl = fl->fl_next) {
> >  		if (fl->fl_lmops == &nlmsvc_lock_operations) {
> > -			unlock_flocks();
> > +			spin_unlock(&inode->i_lock);
> >  			return 1;
> >  		}
> >  	}
> > -	unlock_flocks();
> > +	spin_unlock(&inode->i_lock);
> >  	file->f_locks = 0;
> >  	return 0;
> >  }
> > diff --git a/fs/locks.c b/fs/locks.c
> > index caca466..055c06c 100644
> > --- a/fs/locks.c
> > +++ b/fs/locks.c
> > @@ -165,22 +165,9 @@ int lease_break_time = 45;
> >  
> >  static LIST_HEAD(file_lock_list);
> >  static LIST_HEAD(blocked_list);
> > -static DEFINE_SPINLOCK(file_lock_lock);
> > -
> > -/*
> > - * Protects the two list heads above, plus the inode->i_flock list
> > - */
> > -void lock_flocks(void)
> > -{
> > -	spin_lock(&file_lock_lock);
> > -}
> > -EXPORT_SYMBOL_GPL(lock_flocks);
> >  
> > -void unlock_flocks(void)
> > -{
> > -	spin_unlock(&file_lock_lock);
> > -}
> > -EXPORT_SYMBOL_GPL(unlock_flocks);
> > +/* Protects the two list heads above */
> > +static DEFINE_SPINLOCK(file_lock_lock);
> >  
> >  static struct kmem_cache *filelock_cache __read_mostly;
> >  
> > @@ -498,25 +485,33 @@ static int posix_same_owner(struct file_lock *fl1, struct file_lock *fl2)
> >  static inline void
> >  locks_insert_global_blocked(struct file_lock *waiter)
> >  {
> > +	spin_lock(&file_lock_lock);
> >  	list_add(&waiter->fl_link, &blocked_list);
> > +	spin_unlock(&file_lock_lock);
> >  }
> >  
> >  static inline void
> >  locks_delete_global_blocked(struct file_lock *waiter)
> >  {
> > +	spin_lock(&file_lock_lock);
> >  	list_del_init(&waiter->fl_link);
> > +	spin_unlock(&file_lock_lock);
> >  }
> >  
> >  static inline void
> >  locks_insert_global_locks(struct file_lock *waiter)
> >  {
> > +	spin_lock(&file_lock_lock);
> >  	list_add_tail(&waiter->fl_link, &file_lock_list);
> > +	spin_unlock(&file_lock_lock);
> >  }
> >  
> >  static inline void
> >  locks_delete_global_locks(struct file_lock *waiter)
> >  {
> > +	spin_lock(&file_lock_lock);
> >  	list_del_init(&waiter->fl_link);
> > +	spin_unlock(&file_lock_lock);
> >  }
> >  
> >  /* Remove waiter from blocker's block list.
> > @@ -533,9 +528,11 @@ static void __locks_delete_block(struct file_lock *waiter)
> >   */
> >  static void locks_delete_block(struct file_lock *waiter)
> >  {
> > -	lock_flocks();
> > +	struct inode *inode = file_inode(waiter->fl_file);
> > +
> > +	spin_lock(&inode->i_lock);
> >  	__locks_delete_block(waiter);
> > -	unlock_flocks();
> > +	spin_unlock(&inode->i_lock);
> >  }
> >  
> >  /* Insert waiter into blocker's block list.
> > @@ -657,8 +654,9 @@ void
> >  posix_test_lock(struct file *filp, struct file_lock *fl)
> >  {
> >  	struct file_lock *cfl;
> > +	struct inode *inode = file_inode(filp);
> >  
> > -	lock_flocks();
> > +	spin_lock(&inode->i_lock);
> >  	for (cfl = file_inode(filp)->i_flock; cfl; cfl = cfl->fl_next) {
> >  		if (!IS_POSIX(cfl))
> >  			continue;
> > @@ -671,7 +669,7 @@ posix_test_lock(struct file *filp, struct file_lock *fl)
> >  			fl->fl_pid = pid_vnr(cfl->fl_nspid);
> >  	} else
> >  		fl->fl_type = F_UNLCK;
> > -	unlock_flocks();
> > +	spin_unlock(&inode->i_lock);
> >  	return;
> >  }
> >  EXPORT_SYMBOL(posix_test_lock);
> > @@ -719,14 +717,19 @@ static int posix_locks_deadlock(struct file_lock *caller_fl,
> >  				struct file_lock *block_fl)
> >  {
> >  	int i = 0;
> > +	int ret = 0;
> >  
> > +	spin_lock(&file_lock_lock);
> >  	while ((block_fl = what_owner_is_waiting_for(block_fl))) {
> >  		if (i++ > MAX_DEADLK_ITERATIONS)
> > -			return 0;
> > -		if (posix_same_owner(caller_fl, block_fl))
> > -			return 1;
> > +			break;
> > +		if (posix_same_owner(caller_fl, block_fl)) {
> > +			++ret;
> > +			break;
> > +		}
> >  	}
> > -	return 0;
> > +	spin_unlock(&file_lock_lock);
> > +	return ret;
> >  }
> >  
> >  /* Try to create a FLOCK lock on filp. We always insert new FLOCK locks
> > @@ -750,7 +753,7 @@ static int flock_lock_file(struct file *filp, struct file_lock *request)
> >  			return -ENOMEM;
> >  	}
> >  
> > -	lock_flocks();
> > +	spin_lock(&inode->i_lock);
> >  	if (request->fl_flags & FL_ACCESS)
> >  		goto find_conflict;
> >  
> > @@ -780,9 +783,9 @@ static int flock_lock_file(struct file *filp, struct file_lock *request)
> >  	 * give it the opportunity to lock the file.
> >  	 */
> >  	if (found) {
> > -		unlock_flocks();
> > +		spin_unlock(&inode->i_lock);
> >  		cond_resched();
> > -		lock_flocks();
> > +		spin_lock(&inode->i_lock);
> >  	}
> >  
> >  find_conflict:
> > @@ -809,7 +812,7 @@ find_conflict:
> >  	error = 0;
> >  
> >  out:
> > -	unlock_flocks();
> > +	spin_unlock(&inode->i_lock);
> >  	if (new_fl)
> >  		locks_free_lock(new_fl);
> >  	return error;
> > @@ -839,7 +842,7 @@ static int __posix_lock_file(struct inode *inode, struct file_lock *request, str
> >  		new_fl2 = locks_alloc_lock();
> >  	}
> >  
> > -	lock_flocks();
> > +	spin_lock(&inode->i_lock);
> >  	/*
> >  	 * New lock request. Walk all POSIX locks and look for conflicts. If
> >  	 * there are any, either return -EAGAIN or put the request on the
> > @@ -1012,7 +1015,7 @@ static int __posix_lock_file(struct inode *inode, struct file_lock *request, str
> >  		locks_wake_up_blocks(left);
> >  	}
> >   out:
> > -	unlock_flocks();
> > +	spin_unlock(&inode->i_lock);
> >  	/*
> >  	 * Free any unused locks.
> >  	 */
> > @@ -1087,14 +1090,14 @@ int locks_mandatory_locked(struct inode *inode)
> >  	/*
> >  	 * Search the lock list for this inode for any POSIX locks.
> >  	 */
> > -	lock_flocks();
> > +	spin_lock(&inode->i_lock);
> >  	for (fl = inode->i_flock; fl != NULL; fl = fl->fl_next) {
> >  		if (!IS_POSIX(fl))
> >  			continue;
> >  		if (fl->fl_owner != owner)
> >  			break;
> >  	}
> > -	unlock_flocks();
> > +	spin_unlock(&inode->i_lock);
> >  	return fl ? -EAGAIN : 0;
> >  }
> >  
> > @@ -1237,7 +1240,7 @@ int __break_lease(struct inode *inode, unsigned int mode)
> >  	if (IS_ERR(new_fl))
> >  		return PTR_ERR(new_fl);
> >  
> > -	lock_flocks();
> > +	spin_lock(&inode->i_lock);
> >  
> >  	time_out_leases(inode);
> >  
> > @@ -1287,10 +1290,10 @@ restart:
> >  			break_time++;
> >  	}
> >  	locks_insert_block(flock, new_fl);
> > -	unlock_flocks();
> > +	spin_unlock(&inode->i_lock);
> >  	error = wait_event_interruptible_timeout(new_fl->fl_wait,
> >  						!new_fl->fl_next, break_time);
> > -	lock_flocks();
> > +	spin_lock(&inode->i_lock);
> >  	__locks_delete_block(new_fl);
> >  	if (error >= 0) {
> >  		if (error == 0)
> > @@ -1308,7 +1311,7 @@ restart:
> >  	}
> >  
> >  out:
> > -	unlock_flocks();
> > +	spin_unlock(&inode->i_lock);
> >  	locks_free_lock(new_fl);
> >  	return error;
> >  }
> > @@ -1361,9 +1364,10 @@ EXPORT_SYMBOL(lease_get_mtime);
> >  int fcntl_getlease(struct file *filp)
> >  {
> >  	struct file_lock *fl;
> > +	struct inode *inode = file_inode(filp);
> >  	int type = F_UNLCK;
> >  
> > -	lock_flocks();
> > +	spin_lock(&inode->i_lock);
> >  	time_out_leases(file_inode(filp));
> >  	for (fl = file_inode(filp)->i_flock; fl && IS_LEASE(fl);
> >  			fl = fl->fl_next) {
> > @@ -1372,7 +1376,7 @@ int fcntl_getlease(struct file *filp)
> >  			break;
> >  		}
> >  	}
> > -	unlock_flocks();
> > +	spin_unlock(&inode->i_lock);
> >  	return type;
> >  }
> >  
> > @@ -1466,7 +1470,7 @@ static int generic_delete_lease(struct file *filp, struct file_lock **flp)
> >   *	The (input) flp->fl_lmops->lm_break function is required
> >   *	by break_lease().
> >   *
> > - *	Called with file_lock_lock held.
> > + *	Called with inode->i_lock held.
> >   */
> >  int generic_setlease(struct file *filp, long arg, struct file_lock **flp)
> >  {
> > @@ -1535,11 +1539,12 @@ static int __vfs_setlease(struct file *filp, long arg, struct file_lock **lease)
> >  
> >  int vfs_setlease(struct file *filp, long arg, struct file_lock **lease)
> >  {
> > +	struct inode *inode = file_inode(filp);
> >  	int error;
> >  
> > -	lock_flocks();
> > +	spin_lock(&inode->i_lock);
> >  	error = __vfs_setlease(filp, arg, lease);
> > -	unlock_flocks();
> > +	spin_unlock(&inode->i_lock);
> >  
> >  	return error;
> >  }
> > @@ -1557,6 +1562,7 @@ static int do_fcntl_delete_lease(struct file *filp)
> >  static int do_fcntl_add_lease(unsigned int fd, struct file *filp, long arg)
> >  {
> >  	struct file_lock *fl, *ret;
> > +	struct inode *inode = file_inode(filp);
> >  	struct fasync_struct *new;
> >  	int error;
> >  
> > @@ -1570,10 +1576,10 @@ static int do_fcntl_add_lease(unsigned int fd, struct file *filp, long arg)
> >  		return -ENOMEM;
> >  	}
> >  	ret = fl;
> > -	lock_flocks();
> > +	spin_lock(&inode->i_lock);
> >  	error = __vfs_setlease(filp, arg, &ret);
> >  	if (error) {
> > -		unlock_flocks();
> > +		spin_unlock(&inode->i_lock);
> >  		locks_free_lock(fl);
> >  		goto out_free_fasync;
> >  	}
> > @@ -1590,7 +1596,7 @@ static int do_fcntl_add_lease(unsigned int fd, struct file *filp, long arg)
> >  		new = NULL;
> >  
> >  	error = __f_setown(filp, task_pid(current), PIDTYPE_PID, 0);
> > -	unlock_flocks();
> > +	spin_unlock(&inode->i_lock);
> >  
> >  out_free_fasync:
> >  	if (new)
> > @@ -2114,7 +2120,7 @@ void locks_remove_flock(struct file *filp)
> >  			fl.fl_ops->fl_release_private(&fl);
> >  	}
> >  
> > -	lock_flocks();
> > +	spin_lock(&inode->i_lock);
> >  	before = &inode->i_flock;
> >  
> >  	while ((fl = *before) != NULL) {
> > @@ -2132,7 +2138,7 @@ void locks_remove_flock(struct file *filp)
> >   		}
> >  		before = &fl->fl_next;
> >  	}
> > -	unlock_flocks();
> > +	spin_unlock(&inode->i_lock);
> >  }
> >  
> >  /**
> > @@ -2145,14 +2151,15 @@ void locks_remove_flock(struct file *filp)
> >  int
> >  posix_unblock_lock(struct file *filp, struct file_lock *waiter)
> >  {
> > +	struct inode *inode = file_inode(filp);
> >  	int status = 0;
> >  
> > -	lock_flocks();
> > +	spin_lock(&inode->i_lock);
> >  	if (waiter->fl_next)
> >  		__locks_delete_block(waiter);
> >  	else
> >  		status = -ENOENT;
> > -	unlock_flocks();
> > +	spin_unlock(&inode->i_lock);
> >  	return status;
> >  }
> >  
> > @@ -2257,8 +2264,10 @@ static int locks_show(struct seq_file *f, void *v)
> >  
> >  	lock_get_status(f, fl, *((loff_t *)f->private), "");
> >  
> > -	list_for_each_entry(bfl, &fl->fl_block, fl_block)
> > -		lock_get_status(f, bfl, *((loff_t *)f->private), " ->");
> > +	list_for_each_entry(bfl, &blocked_list, fl_link) {
> > +		if (bfl->fl_next == fl)
> > +			lock_get_status(f, bfl, *((loff_t *)f->private), " ->");
> > +	}
> >  
> >  	return 0;
> >  }
> > @@ -2267,7 +2276,7 @@ static void *locks_start(struct seq_file *f, loff_t *pos)
> >  {
> >  	loff_t *p = f->private;
> >  
> > -	lock_flocks();
> > +	spin_lock(&file_lock_lock);
> >  	*p = (*pos + 1);
> >  	return seq_list_start(&file_lock_list, *pos);
> >  }
> > @@ -2281,7 +2290,7 @@ static void *locks_next(struct seq_file *f, void *v, loff_t *pos)
> >  
> >  static void locks_stop(struct seq_file *f, void *v)
> >  {
> > -	unlock_flocks();
> > +	spin_unlock(&file_lock_lock);
> >  }
> >  
> >  static const struct seq_operations locks_seq_operations = {
> > @@ -2328,7 +2337,8 @@ int lock_may_read(struct inode *inode, loff_t start, unsigned long len)
> >  {
> >  	struct file_lock *fl;
> >  	int result = 1;
> > -	lock_flocks();
> > +
> > +	spin_lock(&inode->i_lock);
> >  	for (fl = inode->i_flock; fl != NULL; fl = fl->fl_next) {
> >  		if (IS_POSIX(fl)) {
> >  			if (fl->fl_type == F_RDLCK)
> > @@ -2345,7 +2355,7 @@ int lock_may_read(struct inode *inode, loff_t start, unsigned long len)
> >  		result = 0;
> >  		break;
> >  	}
> > -	unlock_flocks();
> > +	spin_unlock(&inode->i_lock);
> >  	return result;
> >  }
> >  
> > @@ -2368,7 +2378,8 @@ int lock_may_write(struct inode *inode, loff_t start, unsigned long len)
> >  {
> >  	struct file_lock *fl;
> >  	int result = 1;
> > -	lock_flocks();
> > +
> > +	spin_lock(&inode->i_lock);
> >  	for (fl = inode->i_flock; fl != NULL; fl = fl->fl_next) {
> >  		if (IS_POSIX(fl)) {
> >  			if ((fl->fl_end < start) || (fl->fl_start > (start + len)))
> > @@ -2383,7 +2394,7 @@ int lock_may_write(struct inode *inode, loff_t start, unsigned long len)
> >  		result = 0;
> >  		break;
> >  	}
> > -	unlock_flocks();
> > +	spin_unlock(&inode->i_lock);
> >  	return result;
> >  }
> >  
> > diff --git a/fs/nfs/delegation.c b/fs/nfs/delegation.c
> > index 57db324..43ee7f9 100644
> > --- a/fs/nfs/delegation.c
> > +++ b/fs/nfs/delegation.c
> > @@ -73,20 +73,21 @@ static int nfs_delegation_claim_locks(struct nfs_open_context *ctx, struct nfs4_
> >  	if (inode->i_flock == NULL)
> >  		goto out;
> >  
> > -	/* Protect inode->i_flock using the file locks lock */
> > -	lock_flocks();
> > +	/* Protect inode->i_flock using the i_lock */
> > +	spin_lock(&inode->i_lock);
> >  	for (fl = inode->i_flock; fl != NULL; fl = fl->fl_next) {
> >  		if (!(fl->fl_flags & (FL_POSIX|FL_FLOCK)))
> >  			continue;
> >  		if (nfs_file_open_context(fl->fl_file) != ctx)
> >  			continue;
> > -		unlock_flocks();
> > +		/* FIXME: safe to drop lock here while walking list? */
> > +		spin_unlock(&inode->i_lock);
> >  		status = nfs4_lock_delegation_recall(fl, state, stateid);
> >  		if (status < 0)
> >  			goto out;
> > -		lock_flocks();
> > +		spin_lock(&inode->i_lock);
> >  	}
> > -	unlock_flocks();
> > +	spin_unlock(&inode->i_lock);
> >  out:
> >  	return status;
> >  }
> > diff --git a/fs/nfs/nfs4state.c b/fs/nfs/nfs4state.c
> > index 1fab140..ff10b4a 100644
> > --- a/fs/nfs/nfs4state.c
> > +++ b/fs/nfs/nfs4state.c
> > @@ -1373,13 +1373,13 @@ static int nfs4_reclaim_locks(struct nfs4_state *state, const struct nfs4_state_
> >  	/* Guard against delegation returns and new lock/unlock calls */
> >  	down_write(&nfsi->rwsem);
> >  	/* Protect inode->i_flock using the BKL */
> > -	lock_flocks();
> > +	spin_lock(&inode->i_lock);
> >  	for (fl = inode->i_flock; fl != NULL; fl = fl->fl_next) {
> >  		if (!(fl->fl_flags & (FL_POSIX|FL_FLOCK)))
> >  			continue;
> >  		if (nfs_file_open_context(fl->fl_file)->state != state)
> >  			continue;
> > -		unlock_flocks();
> > +		spin_unlock(&inode->i_lock);
> >  		status = ops->recover_lock(state, fl);
> >  		switch (status) {
> >  			case 0:
> > @@ -1406,9 +1406,9 @@ static int nfs4_reclaim_locks(struct nfs4_state *state, const struct nfs4_state_
> >  				/* kill_proc(fl->fl_pid, SIGLOST, 1); */
> >  				status = 0;
> >  		}
> > -		lock_flocks();
> > +		spin_lock(&inode->i_lock);
> >  	}
> > -	unlock_flocks();
> > +	spin_unlock(&inode->i_lock);
> >  out:
> >  	up_write(&nfsi->rwsem);
> >  	return status;
> > diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
> > index 316ec84..f170518 100644
> > --- a/fs/nfsd/nfs4state.c
> > +++ b/fs/nfsd/nfs4state.c
> > @@ -2645,13 +2645,13 @@ static void nfsd_break_one_deleg(struct nfs4_delegation *dp)
> >  
> >  	list_add_tail(&dp->dl_recall_lru, &nn->del_recall_lru);
> >  
> > -	/* only place dl_time is set. protected by lock_flocks*/
> > +	/* Only place dl_time is set; protected by i_lock: */
> >  	dp->dl_time = get_seconds();
> >  
> >  	nfsd4_cb_recall(dp);
> >  }
> >  
> > -/* Called from break_lease() with lock_flocks() held. */
> > +/* Called from break_lease() with i_lock held. */
> >  static void nfsd_break_deleg_cb(struct file_lock *fl)
> >  {
> >  	struct nfs4_file *fp = (struct nfs4_file *)fl->fl_owner;
> > @@ -4520,7 +4520,7 @@ check_for_locks(struct nfs4_file *filp, struct nfs4_lockowner *lowner)
> >  	struct inode *inode = filp->fi_inode;
> >  	int status = 0;
> >  
> > -	lock_flocks();
> > +	spin_lock(&inode->i_lock);
> >  	for (flpp = &inode->i_flock; *flpp != NULL; flpp = &(*flpp)->fl_next) {
> >  		if ((*flpp)->fl_owner == (fl_owner_t)lowner) {
> >  			status = 1;
> > @@ -4528,7 +4528,7 @@ check_for_locks(struct nfs4_file *filp, struct nfs4_lockowner *lowner)
> >  		}
> >  	}
> >  out:
> > -	unlock_flocks();
> > +	spin_unlock(&inode->i_lock);
> >  	return status;
> >  }
> >  
> > diff --git a/include/linux/fs.h b/include/linux/fs.h
> > index ae377e9..ccb44ea 100644
> > --- a/include/linux/fs.h
> > +++ b/include/linux/fs.h
> > @@ -1012,8 +1012,6 @@ extern int vfs_setlease(struct file *, long, struct file_lock **);
> >  extern int lease_modify(struct file_lock **, int);
> >  extern int lock_may_read(struct inode *, loff_t start, unsigned long count);
> >  extern int lock_may_write(struct inode *, loff_t start, unsigned long count);
> > -extern void lock_flocks(void);
> > -extern void unlock_flocks(void);
> >  #else /* !CONFIG_FILE_LOCKING */
> >  static inline int fcntl_getlk(struct file *file, struct flock __user *user)
> >  {
> > @@ -1155,15 +1153,6 @@ static inline int lock_may_write(struct inode *inode, loff_t start,
> >  {
> >  	return 1;
> >  }
> > -
> > -static inline void lock_flocks(void)
> > -{
> > -}
> > -
> > -static inline void unlock_flocks(void)
> > -{
> > -}
> > -
> >  #endif /* !CONFIG_FILE_LOCKING */
> >  
> >  
> > -- 
> > 1.7.1
> > 


-- 
Jeff Layton <jlayton at redhat.com>


From jlayton at redhat.com  Wed Jun  5 07:38:22 2013
From: jlayton at redhat.com (Jeff Layton)
Date: Wed, 5 Jun 2013 07:38:22 -0400
Subject: [PATCH v1 07/11] locks: only pull entries off of blocked_list
 when they are really unblocked
In-Reply-To: <20130604215839.GD15594@fieldses.org>
References: <1370056054-25449-1-git-send-email-jlayton@redhat.com>
 <1370056054-25449-8-git-send-email-jlayton@redhat.com>
 <20130604215839.GD15594@fieldses.org>
Message-ID: <20130605073822.4d67c57c@corrin.poochiereds.net>

On Tue, 4 Jun 2013 17:58:39 -0400
"J. Bruce Fields" <bfields at fieldses.org> wrote:

> On Fri, May 31, 2013 at 11:07:30PM -0400, Jeff Layton wrote:
> > Currently, when there is a lot of lock contention the kernel spends an
> > inordinate amount of time taking blocked locks off of the global
> > blocked_list and then putting them right back on again. When all of this
> > code was protected by a single lock, then it didn't matter much, but now
> > it means a lot of file_lock_lock thrashing.
> > 
> > Optimize this a bit by deferring the removal from the blocked_list until
> > we're either applying or cancelling the lock. By doing this, and using a
> > lockless list_empty check, we can avoid taking the file_lock_lock in
> > many cases.
> > 
> > Because the fl_link check is lockless, we must ensure that only the task
> > that "owns" the request manipulates the fl_link. Also, with this change,
> > it's possible that we'll see an entry on the blocked_list that has a
> > NULL fl_next pointer. In that event, just ignore it and continue walking
> > the list.
> 
> OK, that sounds safe as in it shouldn't crash, but does the deadlock
> detection still work, or can it miss loops?
> 
> Those locks that are temporarily NULL would previously not have been on
> the list at all, OK, but...  I'm having trouble reasoning about how this
> works now.
> 
> Previously a single lock was held interrupted across
> posix_locks_deadlock and locks_insert_block() which guaranteed we
> shouldn't be adding a loop, is that still true?
> 
> --b.
> 

I had thought it was when I originally looked at this, but now that I
consider it again I think you may be correct and that there are possible
races here. Since we might end up reblocking behind a different lock
without taking the global spinlock we could flip to blocking behind a
different lock such that a loop is created if you had a complex (>2)
chain of locks.

I think I'm going to have to drop this approach and instead make it so
that the deadlock detection and insertion into the global blocker
list/hash are atomic. Ditto for locks_wake_up_blocks on posix locks and
taking the entries off the list/hash.

Now that I look, I think that approach may actually improve performance
too since we'll be taking the global spinlock less than we would have,
but I'll need to test it out to know for sure.

> > 
> > Signed-off-by: Jeff Layton <jlayton at redhat.com>
> > ---
> >  fs/locks.c |   29 +++++++++++++++++++++++------
> >  1 files changed, 23 insertions(+), 6 deletions(-)
> > 
> > diff --git a/fs/locks.c b/fs/locks.c
> > index 055c06c..fc35b9e 100644
> > --- a/fs/locks.c
> > +++ b/fs/locks.c
> > @@ -520,7 +520,6 @@ locks_delete_global_locks(struct file_lock *waiter)
> >  static void __locks_delete_block(struct file_lock *waiter)
> >  {
> >  	list_del_init(&waiter->fl_block);
> > -	locks_delete_global_blocked(waiter);
> >  	waiter->fl_next = NULL;
> >  }
> >  
> > @@ -704,13 +703,16 @@ EXPORT_SYMBOL(posix_test_lock);
> >  /* Find a lock that the owner of the given block_fl is blocking on. */
> >  static struct file_lock *what_owner_is_waiting_for(struct file_lock *block_fl)
> >  {
> > -	struct file_lock *fl;
> > +	struct file_lock *fl, *ret = NULL;
> >  
> >  	list_for_each_entry(fl, &blocked_list, fl_link) {
> > -		if (posix_same_owner(fl, block_fl))
> > -			return fl->fl_next;
> > +		if (posix_same_owner(fl, block_fl)) {
> > +			ret = fl->fl_next;
> > +			if (likely(ret))
> > +				break;
> > +		}
> >  	}
> > -	return NULL;
> > +	return ret;
> >  }
> >  
> >  static int posix_locks_deadlock(struct file_lock *caller_fl,
> > @@ -865,7 +867,8 @@ static int __posix_lock_file(struct inode *inode, struct file_lock *request, str
> >  				goto out;
> >  			error = FILE_LOCK_DEFERRED;
> >  			locks_insert_block(fl, request);
> > -			locks_insert_global_blocked(request);
> > +			if (list_empty(&request->fl_link))
> > +				locks_insert_global_blocked(request);
> >  			goto out;
> >    		}
> >    	}
> > @@ -876,6 +879,16 @@ static int __posix_lock_file(struct inode *inode, struct file_lock *request, str
> >  		goto out;
> >  
> >  	/*
> > +	 * Now that we know the request is no longer blocked, we can take it
> > +	 * off the global list. Some callers send down partially initialized
> > +	 * requests, so we only do this if FL_SLEEP is set. Also, avoid taking
> > +	 * the lock if the list is empty, as that indicates a request that
> > +	 * never blocked.
> > +	 */
> > +	if ((request->fl_flags & FL_SLEEP) && !list_empty(&request->fl_link))
> > +		locks_delete_global_blocked(request);
> > +
> > +	/*
> >  	 * Find the first old lock with the same owner as the new lock.
> >  	 */
> >  	
> > @@ -1069,6 +1082,7 @@ int posix_lock_file_wait(struct file *filp, struct file_lock *fl)
> >  			continue;
> >  
> >  		locks_delete_block(fl);
> > +		locks_delete_global_blocked(fl);
> >  		break;
> >  	}
> >  	return error;
> > @@ -1147,6 +1161,7 @@ int locks_mandatory_area(int read_write, struct inode *inode,
> >  		}
> >  
> >  		locks_delete_block(&fl);
> > +		locks_delete_global_blocked(&fl);
> >  		break;
> >  	}
> >  
> > @@ -1859,6 +1874,7 @@ static int do_lock_file_wait(struct file *filp, unsigned int cmd,
> >  			continue;
> >  
> >  		locks_delete_block(fl);
> > +		locks_delete_global_blocked(fl);
> >  		break;
> >  	}
> >  
> > @@ -2160,6 +2176,7 @@ posix_unblock_lock(struct file *filp, struct file_lock *waiter)
> >  	else
> >  		status = -ENOENT;
> >  	spin_unlock(&inode->i_lock);
> > +	locks_delete_global_blocked(waiter);
> >  	return status;
> >  }
> >  
> > -- 
> > 1.7.1
> > 


-- 
Jeff Layton <jlayton at redhat.com>


From jlayton at redhat.com  Wed Jun  5 07:43:09 2013
From: jlayton at redhat.com (Jeff Layton)
Date: Wed, 5 Jun 2013 07:43:09 -0400
Subject: [PATCH v1 08/11] locks: convert fl_link to a hlist_node
In-Reply-To: <20130604215950.GE15594@fieldses.org>
References: <1370056054-25449-1-git-send-email-jlayton@redhat.com>
 <1370056054-25449-9-git-send-email-jlayton@redhat.com>
 <20130604215950.GE15594@fieldses.org>
Message-ID: <20130605074309.051ff75f@corrin.poochiereds.net>

On Tue, 4 Jun 2013 17:59:50 -0400
"J. Bruce Fields" <bfields at fieldses.org> wrote:

> On Fri, May 31, 2013 at 11:07:31PM -0400, Jeff Layton wrote:
> > Testing has shown that iterating over the blocked_list for deadlock
> > detection turns out to be a bottleneck. In order to alleviate that,
> > begin the process of turning it into a hashtable. We start by turning
> > the fl_link into a hlist_node and the global lists into hlists. A later
> > patch will do the conversion of the blocked_list to a hashtable.
> 
> Even simpler would be if we could add a pointer to the (well, a) lock
> that a lockowner is blocking on, and then we'd just have to follow a
> pointer.  I haven't thought that through, though, perhaps that's hard ot
> make work....
> 
> --b.
> 

I considered that as well and it makes sense for the simple local
filesystem case where you just track ownership based on fl_owner_t.

But...what about lockd? It considers ownership to be a tuple of the
nlm_host and the pid sent in a lock request. I can't seem to wrap my
brain around how to make such an approach work there. I'll confess
though that I haven't tried *too* hard yet though since I had bigger
problems to sort through at the time. Maybe we can consider that for a
later set?

> > 
> > Signed-off-by: Jeff Layton <jlayton at redhat.com>
> > ---
> >  fs/locks.c         |   32 ++++++++++++++++----------------
> >  include/linux/fs.h |    2 +-
> >  2 files changed, 17 insertions(+), 17 deletions(-)
> > 
> > diff --git a/fs/locks.c b/fs/locks.c
> > index fc35b9e..5ed056b 100644
> > --- a/fs/locks.c
> > +++ b/fs/locks.c
> > @@ -163,8 +163,8 @@ int lease_break_time = 45;
> >  #define for_each_lock(inode, lockp) \
> >  	for (lockp = &inode->i_flock; *lockp != NULL; lockp = &(*lockp)->fl_next)
> >  
> > -static LIST_HEAD(file_lock_list);
> > -static LIST_HEAD(blocked_list);
> > +static HLIST_HEAD(file_lock_list);
> > +static HLIST_HEAD(blocked_list);
> >  
> >  /* Protects the two list heads above */
> >  static DEFINE_SPINLOCK(file_lock_lock);
> > @@ -173,7 +173,7 @@ static struct kmem_cache *filelock_cache __read_mostly;
> >  
> >  static void locks_init_lock_heads(struct file_lock *fl)
> >  {
> > -	INIT_LIST_HEAD(&fl->fl_link);
> > +	INIT_HLIST_NODE(&fl->fl_link);
> >  	INIT_LIST_HEAD(&fl->fl_block);
> >  	init_waitqueue_head(&fl->fl_wait);
> >  }
> > @@ -207,7 +207,7 @@ void locks_free_lock(struct file_lock *fl)
> >  {
> >  	BUG_ON(waitqueue_active(&fl->fl_wait));
> >  	BUG_ON(!list_empty(&fl->fl_block));
> > -	BUG_ON(!list_empty(&fl->fl_link));
> > +	BUG_ON(!hlist_unhashed(&fl->fl_link));
> >  
> >  	locks_release_private(fl);
> >  	kmem_cache_free(filelock_cache, fl);
> > @@ -486,7 +486,7 @@ static inline void
> >  locks_insert_global_blocked(struct file_lock *waiter)
> >  {
> >  	spin_lock(&file_lock_lock);
> > -	list_add(&waiter->fl_link, &blocked_list);
> > +	hlist_add_head(&waiter->fl_link, &blocked_list);
> >  	spin_unlock(&file_lock_lock);
> >  }
> >  
> > @@ -494,7 +494,7 @@ static inline void
> >  locks_delete_global_blocked(struct file_lock *waiter)
> >  {
> >  	spin_lock(&file_lock_lock);
> > -	list_del_init(&waiter->fl_link);
> > +	hlist_del_init(&waiter->fl_link);
> >  	spin_unlock(&file_lock_lock);
> >  }
> >  
> > @@ -502,7 +502,7 @@ static inline void
> >  locks_insert_global_locks(struct file_lock *waiter)
> >  {
> >  	spin_lock(&file_lock_lock);
> > -	list_add_tail(&waiter->fl_link, &file_lock_list);
> > +	hlist_add_head(&waiter->fl_link, &file_lock_list);
> >  	spin_unlock(&file_lock_lock);
> >  }
> >  
> > @@ -510,7 +510,7 @@ static inline void
> >  locks_delete_global_locks(struct file_lock *waiter)
> >  {
> >  	spin_lock(&file_lock_lock);
> > -	list_del_init(&waiter->fl_link);
> > +	hlist_del_init(&waiter->fl_link);
> >  	spin_unlock(&file_lock_lock);
> >  }
> >  
> > @@ -705,7 +705,7 @@ static struct file_lock *what_owner_is_waiting_for(struct file_lock *block_fl)
> >  {
> >  	struct file_lock *fl, *ret = NULL;
> >  
> > -	list_for_each_entry(fl, &blocked_list, fl_link) {
> > +	hlist_for_each_entry(fl, &blocked_list, fl_link) {
> >  		if (posix_same_owner(fl, block_fl)) {
> >  			ret = fl->fl_next;
> >  			if (likely(ret))
> > @@ -867,7 +867,7 @@ static int __posix_lock_file(struct inode *inode, struct file_lock *request, str
> >  				goto out;
> >  			error = FILE_LOCK_DEFERRED;
> >  			locks_insert_block(fl, request);
> > -			if (list_empty(&request->fl_link))
> > +			if (hlist_unhashed(&request->fl_link))
> >  				locks_insert_global_blocked(request);
> >  			goto out;
> >    		}
> > @@ -882,10 +882,10 @@ static int __posix_lock_file(struct inode *inode, struct file_lock *request, str
> >  	 * Now that we know the request is no longer blocked, we can take it
> >  	 * off the global list. Some callers send down partially initialized
> >  	 * requests, so we only do this if FL_SLEEP is set. Also, avoid taking
> > -	 * the lock if the list is empty, as that indicates a request that
> > +	 * the lock if the hlist is unhashed, as that indicates a request that
> >  	 * never blocked.
> >  	 */
> > -	if ((request->fl_flags & FL_SLEEP) && !list_empty(&request->fl_link))
> > +	if ((request->fl_flags & FL_SLEEP) && !hlist_unhashed(&request->fl_link))
> >  		locks_delete_global_blocked(request);
> >  
> >  	/*
> > @@ -2277,11 +2277,11 @@ static int locks_show(struct seq_file *f, void *v)
> >  {
> >  	struct file_lock *fl, *bfl;
> >  
> > -	fl = list_entry(v, struct file_lock, fl_link);
> > +	fl = hlist_entry(v, struct file_lock, fl_link);
> >  
> >  	lock_get_status(f, fl, *((loff_t *)f->private), "");
> >  
> > -	list_for_each_entry(bfl, &blocked_list, fl_link) {
> > +	hlist_for_each_entry(bfl, &blocked_list, fl_link) {
> >  		if (bfl->fl_next == fl)
> >  			lock_get_status(f, bfl, *((loff_t *)f->private), " ->");
> >  	}
> > @@ -2295,14 +2295,14 @@ static void *locks_start(struct seq_file *f, loff_t *pos)
> >  
> >  	spin_lock(&file_lock_lock);
> >  	*p = (*pos + 1);
> > -	return seq_list_start(&file_lock_list, *pos);
> > +	return seq_hlist_start(&file_lock_list, *pos);
> >  }
> >  
> >  static void *locks_next(struct seq_file *f, void *v, loff_t *pos)
> >  {
> >  	loff_t *p = f->private;
> >  	++*p;
> > -	return seq_list_next(v, &file_lock_list, pos);
> > +	return seq_hlist_next(v, &file_lock_list, pos);
> >  }
> >  
> >  static void locks_stop(struct seq_file *f, void *v)
> > diff --git a/include/linux/fs.h b/include/linux/fs.h
> > index ccb44ea..07a009e 100644
> > --- a/include/linux/fs.h
> > +++ b/include/linux/fs.h
> > @@ -934,7 +934,7 @@ int locks_in_grace(struct net *);
> >   */
> >  struct file_lock {
> >  	struct file_lock *fl_next;	/* singly linked list for this inode  */
> > -	struct list_head fl_link;	/* doubly linked list of all locks */
> > +	struct hlist_node fl_link;	/* node in global lists */
> >  	struct list_head fl_block;	/* circular list of blocked processes */
> >  	fl_owner_t fl_owner;
> >  	unsigned int fl_flags;
> > -- 
> > 1.7.1
> > 


-- 
Jeff Layton <jlayton at redhat.com>


From bfields at fieldses.org  Wed Jun  5 08:24:32 2013
From: bfields at fieldses.org (J. Bruce Fields)
Date: Wed, 5 Jun 2013 08:24:32 -0400
Subject: [PATCH v1 07/11] locks: only pull entries off of blocked_list
 when they are really unblocked
In-Reply-To: <20130605073822.4d67c57c@corrin.poochiereds.net>
References: <1370056054-25449-1-git-send-email-jlayton@redhat.com>
 <1370056054-25449-8-git-send-email-jlayton@redhat.com>
 <20130604215839.GD15594@fieldses.org>
 <20130605073822.4d67c57c@corrin.poochiereds.net>
Message-ID: <20130605122432.GA24193@fieldses.org>

On Wed, Jun 05, 2013 at 07:38:22AM -0400, Jeff Layton wrote:
> On Tue, 4 Jun 2013 17:58:39 -0400
> "J. Bruce Fields" <bfields at fieldses.org> wrote:
> 
> > On Fri, May 31, 2013 at 11:07:30PM -0400, Jeff Layton wrote:
> > > Currently, when there is a lot of lock contention the kernel spends an
> > > inordinate amount of time taking blocked locks off of the global
> > > blocked_list and then putting them right back on again. When all of this
> > > code was protected by a single lock, then it didn't matter much, but now
> > > it means a lot of file_lock_lock thrashing.
> > > 
> > > Optimize this a bit by deferring the removal from the blocked_list until
> > > we're either applying or cancelling the lock. By doing this, and using a
> > > lockless list_empty check, we can avoid taking the file_lock_lock in
> > > many cases.
> > > 
> > > Because the fl_link check is lockless, we must ensure that only the task
> > > that "owns" the request manipulates the fl_link. Also, with this change,
> > > it's possible that we'll see an entry on the blocked_list that has a
> > > NULL fl_next pointer. In that event, just ignore it and continue walking
> > > the list.
> > 
> > OK, that sounds safe as in it shouldn't crash, but does the deadlock
> > detection still work, or can it miss loops?
> > 
> > Those locks that are temporarily NULL would previously not have been on
> > the list at all, OK, but...  I'm having trouble reasoning about how this
> > works now.
> > 
> > Previously a single lock was held interrupted across
> > posix_locks_deadlock and locks_insert_block() which guaranteed we
> > shouldn't be adding a loop, is that still true?
> > 
> > --b.
> > 
> 
> I had thought it was when I originally looked at this, but now that I
> consider it again I think you may be correct and that there are possible
> races here. Since we might end up reblocking behind a different lock
> without taking the global spinlock we could flip to blocking behind a
> different lock such that a loop is created if you had a complex (>2)
> chain of locks.
> 
> I think I'm going to have to drop this approach and instead make it so
> that the deadlock detection and insertion into the global blocker
> list/hash are atomic.

Right.  Once you drop the lock you can no longer be sure that what you
learned about the file-lock graph stays true.

> Ditto for locks_wake_up_blocks on posix locks and
> taking the entries off the list/hash.

Here I'm not sure what you mean.

--b.


From jlayton at redhat.com  Wed Jun  5 08:38:59 2013
From: jlayton at redhat.com (Jeff Layton)
Date: Wed, 5 Jun 2013 08:38:59 -0400
Subject: [PATCH v1 07/11] locks: only pull entries off of blocked_list
 when they are really unblocked
In-Reply-To: <20130605122432.GA24193@fieldses.org>
References: <1370056054-25449-1-git-send-email-jlayton@redhat.com>
 <1370056054-25449-8-git-send-email-jlayton@redhat.com>
 <20130604215839.GD15594@fieldses.org>
 <20130605073822.4d67c57c@corrin.poochiereds.net>
 <20130605122432.GA24193@fieldses.org>
Message-ID: <20130605083859.72c855cd@corrin.poochiereds.net>

On Wed, 5 Jun 2013 08:24:32 -0400
"J. Bruce Fields" <bfields at fieldses.org> wrote:

> On Wed, Jun 05, 2013 at 07:38:22AM -0400, Jeff Layton wrote:
> > On Tue, 4 Jun 2013 17:58:39 -0400
> > "J. Bruce Fields" <bfields at fieldses.org> wrote:
> > 
> > > On Fri, May 31, 2013 at 11:07:30PM -0400, Jeff Layton wrote:
> > > > Currently, when there is a lot of lock contention the kernel spends an
> > > > inordinate amount of time taking blocked locks off of the global
> > > > blocked_list and then putting them right back on again. When all of this
> > > > code was protected by a single lock, then it didn't matter much, but now
> > > > it means a lot of file_lock_lock thrashing.
> > > > 
> > > > Optimize this a bit by deferring the removal from the blocked_list until
> > > > we're either applying or cancelling the lock. By doing this, and using a
> > > > lockless list_empty check, we can avoid taking the file_lock_lock in
> > > > many cases.
> > > > 
> > > > Because the fl_link check is lockless, we must ensure that only the task
> > > > that "owns" the request manipulates the fl_link. Also, with this change,
> > > > it's possible that we'll see an entry on the blocked_list that has a
> > > > NULL fl_next pointer. In that event, just ignore it and continue walking
> > > > the list.
> > > 
> > > OK, that sounds safe as in it shouldn't crash, but does the deadlock
> > > detection still work, or can it miss loops?
> > > 
> > > Those locks that are temporarily NULL would previously not have been on
> > > the list at all, OK, but...  I'm having trouble reasoning about how this
> > > works now.
> > > 
> > > Previously a single lock was held interrupted across
> > > posix_locks_deadlock and locks_insert_block() which guaranteed we
> > > shouldn't be adding a loop, is that still true?
> > > 
> > > --b.
> > > 
> > 
> > I had thought it was when I originally looked at this, but now that I
> > consider it again I think you may be correct and that there are possible
> > races here. Since we might end up reblocking behind a different lock
> > without taking the global spinlock we could flip to blocking behind a
> > different lock such that a loop is created if you had a complex (>2)
> > chain of locks.
> > 
> > I think I'm going to have to drop this approach and instead make it so
> > that the deadlock detection and insertion into the global blocker
> > list/hash are atomic.
> 
> Right.  Once you drop the lock you can no longer be sure that what you
> learned about the file-lock graph stays true.
> 
> > Ditto for locks_wake_up_blocks on posix locks and
> > taking the entries off the list/hash.
> 
> Here I'm not sure what you mean.
> 

Basically, I mean that rather than setting the fl_next pointer to NULL
while holding only the inode lock and then ignoring those locks in the
deadlock detection code, we should additionally take the global lock in
locks_wake_up_blocks too and take the blocked locks off the global list
and the i_flock list at the same time.

That actually might not be completely necessary, but it'll make the
logic clearer and easier to understand and probably won't hurt
performance too much. Again, I'll need to do some perf testing to be
sure.

-- 
Jeff Layton <jlayton at redhat.com>


From bfields at fieldses.org  Wed Jun  5 08:46:11 2013
From: bfields at fieldses.org (J. Bruce Fields)
Date: Wed, 5 Jun 2013 08:46:11 -0400
Subject: [PATCH v1 08/11] locks: convert fl_link to a hlist_node
In-Reply-To: <20130605074309.051ff75f@corrin.poochiereds.net>
References: <1370056054-25449-1-git-send-email-jlayton@redhat.com>
 <1370056054-25449-9-git-send-email-jlayton@redhat.com>
 <20130604215950.GE15594@fieldses.org>
 <20130605074309.051ff75f@corrin.poochiereds.net>
Message-ID: <20130605124611.GB24193@fieldses.org>

On Wed, Jun 05, 2013 at 07:43:09AM -0400, Jeff Layton wrote:
> On Tue, 4 Jun 2013 17:59:50 -0400
> "J. Bruce Fields" <bfields at fieldses.org> wrote:
> 
> > On Fri, May 31, 2013 at 11:07:31PM -0400, Jeff Layton wrote:
> > > Testing has shown that iterating over the blocked_list for deadlock
> > > detection turns out to be a bottleneck. In order to alleviate that,
> > > begin the process of turning it into a hashtable. We start by turning
> > > the fl_link into a hlist_node and the global lists into hlists. A later
> > > patch will do the conversion of the blocked_list to a hashtable.
> > 
> > Even simpler would be if we could add a pointer to the (well, a) lock
> > that a lockowner is blocking on, and then we'd just have to follow a
> > pointer.  I haven't thought that through, though, perhaps that's hard ot
> > make work....
> > 
> > --b.
> > 
> 
> I considered that as well and it makes sense for the simple local
> filesystem case where you just track ownership based on fl_owner_t.
> 
> But...what about lockd? It considers ownership to be a tuple of the
> nlm_host and the pid sent in a lock request. I can't seem to wrap my
> brain around how to make such an approach work there.

I wonder if we could do something vaguely like

	struct lock_owner_common {
		struct file_lock *blocker;
	};

	struct nlmsvc_lock_owner {
		struct lock_owner_common owner;
		unsigned int client_pid;
	};

and make fl_owner a (struct lock_owner_common *) and have lockd create
nlmsvc_lock_owners as necessary on the fly.  The lm_compare_owner
callback could then be replaced by a pointer comparison.  I'm not sure
what kind of locking or refcounting might be needed.  But...

> I'll confess though that I haven't tried *too* hard yet

... me neither, so...

> though since I had bigger problems to sort through at the time. Maybe
> we can consider that for a later set?

sounds fine.

--b.


From bfields at fieldses.org  Wed Jun  5 08:59:06 2013
From: bfields at fieldses.org (J. Bruce Fields)
Date: Wed, 5 Jun 2013 08:59:06 -0400
Subject: [PATCH v1 07/11] locks: only pull entries off of blocked_list
 when they are really unblocked
In-Reply-To: <20130605083859.72c855cd@corrin.poochiereds.net>
References: <1370056054-25449-1-git-send-email-jlayton@redhat.com>
 <1370056054-25449-8-git-send-email-jlayton@redhat.com>
 <20130604215839.GD15594@fieldses.org>
 <20130605073822.4d67c57c@corrin.poochiereds.net>
 <20130605122432.GA24193@fieldses.org>
 <20130605083859.72c855cd@corrin.poochiereds.net>
Message-ID: <20130605125906.GC24193@fieldses.org>

On Wed, Jun 05, 2013 at 08:38:59AM -0400, Jeff Layton wrote:
> On Wed, 5 Jun 2013 08:24:32 -0400
> "J. Bruce Fields" <bfields at fieldses.org> wrote:
> 
> > On Wed, Jun 05, 2013 at 07:38:22AM -0400, Jeff Layton wrote:
> > > On Tue, 4 Jun 2013 17:58:39 -0400
> > > "J. Bruce Fields" <bfields at fieldses.org> wrote:
> > > 
> > > > On Fri, May 31, 2013 at 11:07:30PM -0400, Jeff Layton wrote:
> > > > > Currently, when there is a lot of lock contention the kernel spends an
> > > > > inordinate amount of time taking blocked locks off of the global
> > > > > blocked_list and then putting them right back on again. When all of this
> > > > > code was protected by a single lock, then it didn't matter much, but now
> > > > > it means a lot of file_lock_lock thrashing.
> > > > > 
> > > > > Optimize this a bit by deferring the removal from the blocked_list until
> > > > > we're either applying or cancelling the lock. By doing this, and using a
> > > > > lockless list_empty check, we can avoid taking the file_lock_lock in
> > > > > many cases.
> > > > > 
> > > > > Because the fl_link check is lockless, we must ensure that only the task
> > > > > that "owns" the request manipulates the fl_link. Also, with this change,
> > > > > it's possible that we'll see an entry on the blocked_list that has a
> > > > > NULL fl_next pointer. In that event, just ignore it and continue walking
> > > > > the list.
> > > > 
> > > > OK, that sounds safe as in it shouldn't crash, but does the deadlock
> > > > detection still work, or can it miss loops?
> > > > 
> > > > Those locks that are temporarily NULL would previously not have been on
> > > > the list at all, OK, but...  I'm having trouble reasoning about how this
> > > > works now.
> > > > 
> > > > Previously a single lock was held interrupted across
> > > > posix_locks_deadlock and locks_insert_block() which guaranteed we
> > > > shouldn't be adding a loop, is that still true?
> > > > 
> > > > --b.
> > > > 
> > > 
> > > I had thought it was when I originally looked at this, but now that I
> > > consider it again I think you may be correct and that there are possible
> > > races here. Since we might end up reblocking behind a different lock
> > > without taking the global spinlock we could flip to blocking behind a
> > > different lock such that a loop is created if you had a complex (>2)
> > > chain of locks.
> > > 
> > > I think I'm going to have to drop this approach and instead make it so
> > > that the deadlock detection and insertion into the global blocker
> > > list/hash are atomic.
> > 
> > Right.  Once you drop the lock you can no longer be sure that what you
> > learned about the file-lock graph stays true.
> > 
> > > Ditto for locks_wake_up_blocks on posix locks and
> > > taking the entries off the list/hash.
> > 
> > Here I'm not sure what you mean.
> > 
> 
> Basically, I mean that rather than setting the fl_next pointer to NULL
> while holding only the inode lock and then ignoring those locks in the
> deadlock detection code, we should additionally take the global lock in
> locks_wake_up_blocks too and take the blocked locks off the global list
> and the i_flock list at the same time.

OK, thanks, got it.  I have a hard time thinking about that.... But yes
it bothers me that the deadlock detection code could see an out-of-date
value of fl_next, and I can't convince myself that this wouldn't result
in false positives or false negatives.

> That actually might not be completely necessary, but it'll make the
> logic clearer and easier to understand and probably won't hurt
> performance too much. Again, I'll need to do some perf testing to be
> sure.

OK!

--b.


From jlayton at redhat.com  Tue Jun 11 07:09:00 2013
From: jlayton at redhat.com (Jeff Layton)
Date: Tue, 11 Jun 2013 07:09:00 -0400
Subject: [PATCH v2 06/14] locks: don't walk inode->i_flock list in locks_show
In-Reply-To: <1370948948-31784-1-git-send-email-jlayton@redhat.com>
References: <1370948948-31784-1-git-send-email-jlayton@redhat.com>
Message-ID: <1370948948-31784-7-git-send-email-jlayton@redhat.com>

When we convert over to using the i_lock to protect the i_flock list,
that will introduce a potential lock inversion problem in locks_show.
When we want to walk the i_flock list, we'll need to take the i_lock.

Rather than do that, just walk the global blocked_locks list and print
out any that are blocked on the given lock.

Signed-off-by: Jeff Layton <jlayton at redhat.com>
---
 fs/locks.c |    6 ++++--
 1 files changed, 4 insertions(+), 2 deletions(-)

diff --git a/fs/locks.c b/fs/locks.c
index e451d18..3fd27f0 100644
--- a/fs/locks.c
+++ b/fs/locks.c
@@ -2249,8 +2249,10 @@ static int locks_show(struct seq_file *f, void *v)
 
 	lock_get_status(f, fl, *((loff_t *)f->private), "");
 
-	list_for_each_entry(bfl, &fl->fl_block, fl_block)
-		lock_get_status(f, bfl, *((loff_t *)f->private), " ->");
+	list_for_each_entry(bfl, &blocked_list, fl_link) {
+		if (bfl->fl_next == fl)
+			lock_get_status(f, bfl, *((loff_t *)f->private), " ->");
+	}
 
 	return 0;
 }
-- 
1.7.1



From jlayton at redhat.com  Tue Jun 11 07:08:55 2013
From: jlayton at redhat.com (Jeff Layton)
Date: Tue, 11 Jun 2013 07:08:55 -0400
Subject: [PATCH v2 01/14] cifs: use posix_unblock_lock instead of
 locks_delete_block
In-Reply-To: <1370948948-31784-1-git-send-email-jlayton@redhat.com>
References: <1370948948-31784-1-git-send-email-jlayton@redhat.com>
Message-ID: <1370948948-31784-2-git-send-email-jlayton@redhat.com>

commit 66189be74 (CIFS: Fix VFS lock usage for oplocked files) exported
the locks_delete_block symbol. There's already an exported helper
function that provides this capability however, so make cifs use that
instead and turn locks_delete_block back into a static function.

Note that if fl->fl_next == NULL then this lock has already been through
locks_delete_block(), so we should be OK to ignore an ENOENT error here
and simply not retry the lock.

Cc: Pavel Shilovsky <piastryyy at gmail.com>
Signed-off-by: Jeff Layton <jlayton at redhat.com>
Acked-by: J. Bruce Fields <bfields at fieldses.org>
---
 fs/cifs/file.c     |    2 +-
 fs/locks.c         |    3 +--
 include/linux/fs.h |    5 -----
 3 files changed, 2 insertions(+), 8 deletions(-)

diff --git a/fs/cifs/file.c b/fs/cifs/file.c
index 48b29d2..44a4f18 100644
--- a/fs/cifs/file.c
+++ b/fs/cifs/file.c
@@ -999,7 +999,7 @@ try_again:
 		rc = wait_event_interruptible(flock->fl_wait, !flock->fl_next);
 		if (!rc)
 			goto try_again;
-		locks_delete_block(flock);
+		posix_unblock_lock(file, flock);
 	}
 	return rc;
 }
diff --git a/fs/locks.c b/fs/locks.c
index cb424a4..7a02064 100644
--- a/fs/locks.c
+++ b/fs/locks.c
@@ -496,13 +496,12 @@ static void __locks_delete_block(struct file_lock *waiter)
 
 /*
  */
-void locks_delete_block(struct file_lock *waiter)
+static void locks_delete_block(struct file_lock *waiter)
 {
 	lock_flocks();
 	__locks_delete_block(waiter);
 	unlock_flocks();
 }
-EXPORT_SYMBOL(locks_delete_block);
 
 /* Insert waiter into blocker's block list.
  * We use a circular list so that processes can be easily woken up in
diff --git a/include/linux/fs.h b/include/linux/fs.h
index 43db02e..b9d7816 100644
--- a/include/linux/fs.h
+++ b/include/linux/fs.h
@@ -1006,7 +1006,6 @@ extern int vfs_setlease(struct file *, long, struct file_lock **);
 extern int lease_modify(struct file_lock **, int);
 extern int lock_may_read(struct inode *, loff_t start, unsigned long count);
 extern int lock_may_write(struct inode *, loff_t start, unsigned long count);
-extern void locks_delete_block(struct file_lock *waiter);
 extern void lock_flocks(void);
 extern void unlock_flocks(void);
 #else /* !CONFIG_FILE_LOCKING */
@@ -1151,10 +1150,6 @@ static inline int lock_may_write(struct inode *inode, loff_t start,
 	return 1;
 }
 
-static inline void locks_delete_block(struct file_lock *waiter)
-{
-}
-
 static inline void lock_flocks(void)
 {
 }
-- 
1.7.1



From jlayton at redhat.com  Tue Jun 11 07:08:58 2013
From: jlayton at redhat.com (Jeff Layton)
Date: Tue, 11 Jun 2013 07:08:58 -0400
Subject: [PATCH v2 04/14] locks: make "added" in __posix_lock_file a bool
In-Reply-To: <1370948948-31784-1-git-send-email-jlayton@redhat.com>
References: <1370948948-31784-1-git-send-email-jlayton@redhat.com>
Message-ID: <1370948948-31784-5-git-send-email-jlayton@redhat.com>

...save 3 bytes of stack space.

Signed-off-by: Jeff Layton <jlayton at redhat.com>
Acked-by: J. Bruce Fields <bfields at fieldses.org>
---
 fs/locks.c |    9 +++++----
 1 files changed, 5 insertions(+), 4 deletions(-)

diff --git a/fs/locks.c b/fs/locks.c
index 1e6301b..c186649 100644
--- a/fs/locks.c
+++ b/fs/locks.c
@@ -791,7 +791,8 @@ static int __posix_lock_file(struct inode *inode, struct file_lock *request, str
 	struct file_lock *left = NULL;
 	struct file_lock *right = NULL;
 	struct file_lock **before;
-	int error, added = 0;
+	int error;
+	bool added = false;
 
 	/*
 	 * We may need two file_lock structures for this operation,
@@ -885,7 +886,7 @@ static int __posix_lock_file(struct inode *inode, struct file_lock *request, str
 				continue;
 			}
 			request = fl;
-			added = 1;
+			added = true;
 		}
 		else {
 			/* Processing for different lock types is a bit
@@ -896,7 +897,7 @@ static int __posix_lock_file(struct inode *inode, struct file_lock *request, str
 			if (fl->fl_start > request->fl_end)
 				break;
 			if (request->fl_type == F_UNLCK)
-				added = 1;
+				added = true;
 			if (fl->fl_start < request->fl_start)
 				left = fl;
 			/* If the next lock in the list has a higher end
@@ -926,7 +927,7 @@ static int __posix_lock_file(struct inode *inode, struct file_lock *request, str
 				locks_release_private(fl);
 				locks_copy_private(fl, request);
 				request = fl;
-				added = 1;
+				added = true;
 			}
 		}
 		/* Go on to next lock.
-- 
1.7.1



From jlayton at redhat.com  Tue Jun 11 07:08:59 2013
From: jlayton at redhat.com (Jeff Layton)
Date: Tue, 11 Jun 2013 07:08:59 -0400
Subject: [PATCH v2 05/14] locks: encapsulate the fl_link list handling
In-Reply-To: <1370948948-31784-1-git-send-email-jlayton@redhat.com>
References: <1370948948-31784-1-git-send-email-jlayton@redhat.com>
Message-ID: <1370948948-31784-6-git-send-email-jlayton@redhat.com>

Move the fl_link list handling routines into a separate set of helpers.
Also move the global list handling out of locks_insert_block, and into
the caller that ends up triggering it as that allows us to eliminate the
IS_POSIX check there.

Signed-off-by: Jeff Layton <jlayton at redhat.com>
Acked-by: J. Bruce Fields <bfields at fieldses.org>
---
 fs/locks.c |   37 +++++++++++++++++++++++++++++++------
 1 files changed, 31 insertions(+), 6 deletions(-)

diff --git a/fs/locks.c b/fs/locks.c
index c186649..e451d18 100644
--- a/fs/locks.c
+++ b/fs/locks.c
@@ -484,13 +484,38 @@ static int posix_same_owner(struct file_lock *fl1, struct file_lock *fl2)
 	return fl1->fl_owner == fl2->fl_owner;
 }
 
+/* Remove a blocker or lock from one of the global lists */
+static inline void
+locks_insert_global_blocked(struct file_lock *waiter)
+{
+	list_add(&waiter->fl_link, &blocked_list);
+}
+
+static inline void
+locks_delete_global_blocked(struct file_lock *waiter)
+{
+	list_del_init(&waiter->fl_link);
+}
+
+static inline void
+locks_insert_global_locks(struct file_lock *waiter)
+{
+	list_add_tail(&waiter->fl_link, &file_lock_list);
+}
+
+static inline void
+locks_delete_global_locks(struct file_lock *waiter)
+{
+	list_del_init(&waiter->fl_link);
+}
+
 /* Remove waiter from blocker's block list.
  * When blocker ends up pointing to itself then the list is empty.
  */
 static void __locks_delete_block(struct file_lock *waiter)
 {
+	locks_delete_global_blocked(waiter);
 	list_del_init(&waiter->fl_block);
-	list_del_init(&waiter->fl_link);
 	waiter->fl_next = NULL;
 }
 
@@ -514,8 +539,6 @@ static void locks_insert_block(struct file_lock *blocker,
 	BUG_ON(!list_empty(&waiter->fl_block));
 	list_add_tail(&waiter->fl_block, &blocker->fl_block);
 	waiter->fl_next = blocker;
-	if (IS_POSIX(blocker))
-		list_add(&waiter->fl_link, &blocked_list);
 }
 
 /*
@@ -543,13 +566,13 @@ static void locks_wake_up_blocks(struct file_lock *blocker)
  */
 static void locks_insert_lock(struct file_lock **pos, struct file_lock *fl)
 {
-	list_add(&fl->fl_link, &file_lock_list);
-
 	fl->fl_nspid = get_pid(task_tgid(current));
 
 	/* insert into file's list */
 	fl->fl_next = *pos;
 	*pos = fl;
+
+	locks_insert_global_locks(fl);
 }
 
 /*
@@ -562,9 +585,10 @@ static void locks_delete_lock(struct file_lock **thisfl_p)
 {
 	struct file_lock *fl = *thisfl_p;
 
+	locks_delete_global_locks(fl);
+
 	*thisfl_p = fl->fl_next;
 	fl->fl_next = NULL;
-	list_del_init(&fl->fl_link);
 
 	if (fl->fl_nspid) {
 		put_pid(fl->fl_nspid);
@@ -830,6 +854,7 @@ static int __posix_lock_file(struct inode *inode, struct file_lock *request, str
 				goto out;
 			error = FILE_LOCK_DEFERRED;
 			locks_insert_block(fl, request);
+			locks_insert_global_blocked(request);
 			goto out;
   		}
   	}
-- 
1.7.1



From jlayton at redhat.com  Tue Jun 11 07:08:57 2013
From: jlayton at redhat.com (Jeff Layton)
Date: Tue, 11 Jun 2013 07:08:57 -0400
Subject: [PATCH v2 03/14] locks: comment cleanups and clarifications
In-Reply-To: <1370948948-31784-1-git-send-email-jlayton@redhat.com>
References: <1370948948-31784-1-git-send-email-jlayton@redhat.com>
Message-ID: <1370948948-31784-4-git-send-email-jlayton@redhat.com>

Signed-off-by: Jeff Layton <jlayton at redhat.com>
---
 fs/locks.c         |   21 +++++++++++++--------
 include/linux/fs.h |   18 ++++++++++++++++++
 2 files changed, 31 insertions(+), 8 deletions(-)

diff --git a/fs/locks.c b/fs/locks.c
index e3140b8..1e6301b 100644
--- a/fs/locks.c
+++ b/fs/locks.c
@@ -518,9 +518,10 @@ static void locks_insert_block(struct file_lock *blocker,
 		list_add(&waiter->fl_link, &blocked_list);
 }
 
-/* Wake up processes blocked waiting for blocker.
- * If told to wait then schedule the processes until the block list
- * is empty, otherwise empty the block list ourselves.
+/*
+ * Wake up processes blocked waiting for blocker.
+ *
+ * Must be called with the file_lock_lock held!
  */
 static void locks_wake_up_blocks(struct file_lock *blocker)
 {
@@ -806,6 +807,11 @@ static int __posix_lock_file(struct inode *inode, struct file_lock *request, str
 	}
 
 	lock_flocks();
+	/*
+	 * New lock request. Walk all POSIX locks and look for conflicts. If
+	 * there are any, either return error or put the request on the
+	 * blocker's list of waiters and the global blocked_list.
+	 */
 	if (request->fl_type != F_UNLCK) {
 		for_each_lock(inode, before) {
 			fl = *before;
@@ -844,7 +850,7 @@ static int __posix_lock_file(struct inode *inode, struct file_lock *request, str
 		before = &fl->fl_next;
 	}
 
-	/* Process locks with this owner.  */
+	/* Process locks with this owner. */
 	while ((fl = *before) && posix_same_owner(request, fl)) {
 		/* Detect adjacent or overlapping regions (if same lock type)
 		 */
@@ -930,10 +936,9 @@ static int __posix_lock_file(struct inode *inode, struct file_lock *request, str
 	}
 
 	/*
-	 * The above code only modifies existing locks in case of
-	 * merging or replacing.  If new lock(s) need to be inserted
-	 * all modifications are done bellow this, so it's safe yet to
-	 * bail out.
+	 * The above code only modifies existing locks in case of merging or
+	 * replacing. If new lock(s) need to be inserted all modifications are
+	 * done below this, so it's safe yet to bail out.
 	 */
 	error = -ENOLCK; /* "no luck" */
 	if (right && left == right && !new_fl2)
diff --git a/include/linux/fs.h b/include/linux/fs.h
index b9d7816..94105d2 100644
--- a/include/linux/fs.h
+++ b/include/linux/fs.h
@@ -926,6 +926,24 @@ int locks_in_grace(struct net *);
 /* that will die - we need it for nfs_lock_info */
 #include <linux/nfs_fs_i.h>
 
+/*
+ * struct file_lock represents a generic "file lock". It's used to represent
+ * POSIX byte range locks, BSD (flock) locks, and leases. It's important to
+ * note that the same struct is used to represent both a request for a lock and
+ * the lock itself, but the same object is never used for both.
+ *
+ * FIXME: should we create a separate "struct lock_request" to help distinguish
+ * these two uses?
+ *
+ * The i_flock list is ordered by:
+ *
+ * 1) lock type -- FL_LEASEs first, then FL_FLOCK, and finally FL_POSIX
+ * 2) lock owner
+ * 3) lock range start
+ * 4) lock range end
+ *
+ * Obviously, the last two criteria only matter for POSIX locks.
+ */
 struct file_lock {
 	struct file_lock *fl_next;	/* singly linked list for this inode  */
 	struct list_head fl_link;	/* doubly linked list of all locks */
-- 
1.7.1



From jlayton at redhat.com  Tue Jun 11 07:08:56 2013
From: jlayton at redhat.com (Jeff Layton)
Date: Tue, 11 Jun 2013 07:08:56 -0400
Subject: [PATCH v2 02/14] locks: make generic_add_lease and
 generic_delete_lease static
In-Reply-To: <1370948948-31784-1-git-send-email-jlayton@redhat.com>
References: <1370948948-31784-1-git-send-email-jlayton@redhat.com>
Message-ID: <1370948948-31784-3-git-send-email-jlayton@redhat.com>

Signed-off-by: Jeff Layton <jlayton at redhat.com>
Acked-by: J. Bruce Fields <bfields at fieldses.org>
---
 fs/locks.c |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/fs/locks.c b/fs/locks.c
index 7a02064..e3140b8 100644
--- a/fs/locks.c
+++ b/fs/locks.c
@@ -1337,7 +1337,7 @@ int fcntl_getlease(struct file *filp)
 	return type;
 }
 
-int generic_add_lease(struct file *filp, long arg, struct file_lock **flp)
+static int generic_add_lease(struct file *filp, long arg, struct file_lock **flp)
 {
 	struct file_lock *fl, **before, **my_before = NULL, *lease;
 	struct dentry *dentry = filp->f_path.dentry;
@@ -1402,7 +1402,7 @@ out:
 	return error;
 }
 
-int generic_delete_lease(struct file *filp, struct file_lock **flp)
+static int generic_delete_lease(struct file *filp, struct file_lock **flp)
 {
 	struct file_lock *fl, **before;
 	struct dentry *dentry = filp->f_path.dentry;
-- 
1.7.1



From jlayton at redhat.com  Tue Jun 11 07:09:04 2013
From: jlayton at redhat.com (Jeff Layton)
Date: Tue, 11 Jun 2013 07:09:04 -0400
Subject: [PATCH v2 10/14] locks: turn the blocked_list into a hashtable
In-Reply-To: <1370948948-31784-1-git-send-email-jlayton@redhat.com>
References: <1370948948-31784-1-git-send-email-jlayton@redhat.com>
Message-ID: <1370948948-31784-11-git-send-email-jlayton@redhat.com>

Break up the blocked_list into a hashtable, using the fl_owner as a key.
This speeds up searching the hash chains, which is especially significant
for deadlock detection.

Note that the initial implementation assumes that hashing on fl_owner is
sufficient. In most cases it should be, with the notable exception being
server-side lockd, which compares ownership using a tuple of the
nlm_host and the pid sent in the lock request. So, this may degrade to a
single hash bucket when you only have a single NFS client. That will be
addressed in a later patch.

The careful observer may note that this patch leaves the file_lock_list
alone. There's much less of a case for turning the file_lock_list into a
hashtable. The only user of that list is the code that generates
/proc/locks, and it always walks the entire list.

Signed-off-by: Jeff Layton <jlayton at redhat.com>
---
 fs/locks.c |   25 ++++++++++++++++++-------
 1 files changed, 18 insertions(+), 7 deletions(-)

diff --git a/fs/locks.c b/fs/locks.c
index 28959bc..76fb7af 100644
--- a/fs/locks.c
+++ b/fs/locks.c
@@ -126,6 +126,7 @@
 #include <linux/time.h>
 #include <linux/rcupdate.h>
 #include <linux/pid_namespace.h>
+#include <linux/hashtable.h>
 
 #include <asm/uaccess.h>
 
@@ -153,10 +154,19 @@ int lease_break_time = 45;
 #define for_each_lock(inode, lockp) \
 	for (lockp = &inode->i_flock; *lockp != NULL; lockp = &(*lockp)->fl_next)
 
+/*
+ * By breaking up the blocked locks list into a hashtable, we speed up the
+ * deadlock detection.
+ *
+ * FIXME: make this value scale via some heuristic?
+ */
+#define BLOCKED_HASH_BITS	7
+
+static DEFINE_HASHTABLE(blocked_hash, BLOCKED_HASH_BITS);
+
 static HLIST_HEAD(file_lock_list);
-static HLIST_HEAD(blocked_list);
 
-/* Protects the two list heads above */
+/* Protects the file_lock_list and the blocked_hash */
 static DEFINE_SPINLOCK(file_lock_lock);
 
 static struct kmem_cache *filelock_cache __read_mostly;
@@ -475,13 +485,13 @@ static int posix_same_owner(struct file_lock *fl1, struct file_lock *fl2)
 static inline void
 locks_insert_global_blocked(struct file_lock *waiter)
 {
-	hlist_add_head(&waiter->fl_link, &blocked_list);
+	hash_add(blocked_hash, &waiter->fl_link, (unsigned long)waiter->fl_owner);
 }
 
 static inline void
 __locks_delete_global_blocked(struct file_lock *waiter)
 {
-	hlist_del_init(&waiter->fl_link);
+	hash_del(&waiter->fl_link);
 }
 
 static inline void
@@ -729,7 +739,7 @@ static struct file_lock *what_owner_is_waiting_for(struct file_lock *block_fl)
 {
 	struct file_lock *fl;
 
-	hlist_for_each_entry(fl, &blocked_list, fl_link) {
+	hash_for_each_possible(blocked_hash, fl, fl_link, (unsigned long)block_fl->fl_owner) {
 		if (posix_same_owner(fl, block_fl))
 			return fl->fl_next;
 	}
@@ -865,7 +875,7 @@ static int __posix_lock_file(struct inode *inode, struct file_lock *request, str
 	/*
 	 * New lock request. Walk all POSIX locks and look for conflicts. If
 	 * there are any, either return error or put the request on the
-	 * blocker's list of waiters and the global blocked_list.
+	 * blocker's list of waiters and the global blocked_hash.
 	 */
 	if (request->fl_type != F_UNLCK) {
 		for_each_lock(inode, before) {
@@ -2284,13 +2294,14 @@ static void lock_get_status(struct seq_file *f, struct file_lock *fl,
 
 static int locks_show(struct seq_file *f, void *v)
 {
+	int bkt;
 	struct file_lock *fl, *bfl;
 
 	fl = hlist_entry(v, struct file_lock, fl_link);
 
 	lock_get_status(f, fl, *((loff_t *)f->private), "");
 
-	hlist_for_each_entry(bfl, &blocked_list, fl_link) {
+	hash_for_each(blocked_hash, bkt, bfl, fl_link) {
 		if (bfl->fl_next == fl)
 			lock_get_status(f, bfl, *((loff_t *)f->private), " ->");
 	}
-- 
1.7.1



From jlayton at redhat.com  Tue Jun 11 07:09:03 2013
From: jlayton at redhat.com (Jeff Layton)
Date: Tue, 11 Jun 2013 07:09:03 -0400
Subject: [PATCH v2 09/14] locks: convert fl_link to a hlist_node
In-Reply-To: <1370948948-31784-1-git-send-email-jlayton@redhat.com>
References: <1370948948-31784-1-git-send-email-jlayton@redhat.com>
Message-ID: <1370948948-31784-10-git-send-email-jlayton@redhat.com>

Testing has shown that iterating over the blocked_list for deadlock
detection turns out to be a bottleneck. In order to alleviate that,
begin the process of turning it into a hashtable. We start by turning
the fl_link into a hlist_node and the global lists into hlists. A later
patch will do the conversion of the blocked_list to a hashtable.

Signed-off-by: Jeff Layton <jlayton at redhat.com>
---
 fs/locks.c         |   26 +++++++++++++-------------
 include/linux/fs.h |    2 +-
 2 files changed, 14 insertions(+), 14 deletions(-)

diff --git a/fs/locks.c b/fs/locks.c
index b8cd1b1..28959bc 100644
--- a/fs/locks.c
+++ b/fs/locks.c
@@ -153,8 +153,8 @@ int lease_break_time = 45;
 #define for_each_lock(inode, lockp) \
 	for (lockp = &inode->i_flock; *lockp != NULL; lockp = &(*lockp)->fl_next)
 
-static LIST_HEAD(file_lock_list);
-static LIST_HEAD(blocked_list);
+static HLIST_HEAD(file_lock_list);
+static HLIST_HEAD(blocked_list);
 
 /* Protects the two list heads above */
 static DEFINE_SPINLOCK(file_lock_lock);
@@ -163,7 +163,7 @@ static struct kmem_cache *filelock_cache __read_mostly;
 
 static void locks_init_lock_heads(struct file_lock *fl)
 {
-	INIT_LIST_HEAD(&fl->fl_link);
+	INIT_HLIST_NODE(&fl->fl_link);
 	INIT_LIST_HEAD(&fl->fl_block);
 	init_waitqueue_head(&fl->fl_wait);
 }
@@ -197,7 +197,7 @@ void locks_free_lock(struct file_lock *fl)
 {
 	BUG_ON(waitqueue_active(&fl->fl_wait));
 	BUG_ON(!list_empty(&fl->fl_block));
-	BUG_ON(!list_empty(&fl->fl_link));
+	BUG_ON(!hlist_unhashed(&fl->fl_link));
 
 	locks_release_private(fl);
 	kmem_cache_free(filelock_cache, fl);
@@ -475,13 +475,13 @@ static int posix_same_owner(struct file_lock *fl1, struct file_lock *fl2)
 static inline void
 locks_insert_global_blocked(struct file_lock *waiter)
 {
-	list_add(&waiter->fl_link, &blocked_list);
+	hlist_add_head(&waiter->fl_link, &blocked_list);
 }
 
 static inline void
 __locks_delete_global_blocked(struct file_lock *waiter)
 {
-	list_del_init(&waiter->fl_link);
+	hlist_del_init(&waiter->fl_link);
 }
 
 static inline void
@@ -496,7 +496,7 @@ static inline void
 locks_insert_global_locks(struct file_lock *waiter)
 {
 	spin_lock(&file_lock_lock);
-	list_add_tail(&waiter->fl_link, &file_lock_list);
+	hlist_add_head(&waiter->fl_link, &file_lock_list);
 	spin_unlock(&file_lock_lock);
 }
 
@@ -504,7 +504,7 @@ static inline void
 locks_delete_global_locks(struct file_lock *waiter)
 {
 	spin_lock(&file_lock_lock);
-	list_del_init(&waiter->fl_link);
+	hlist_del_init(&waiter->fl_link);
 	spin_unlock(&file_lock_lock);
 }
 
@@ -729,7 +729,7 @@ static struct file_lock *what_owner_is_waiting_for(struct file_lock *block_fl)
 {
 	struct file_lock *fl;
 
-	list_for_each_entry(fl, &blocked_list, fl_link) {
+	hlist_for_each_entry(fl, &blocked_list, fl_link) {
 		if (posix_same_owner(fl, block_fl))
 			return fl->fl_next;
 	}
@@ -2286,11 +2286,11 @@ static int locks_show(struct seq_file *f, void *v)
 {
 	struct file_lock *fl, *bfl;
 
-	fl = list_entry(v, struct file_lock, fl_link);
+	fl = hlist_entry(v, struct file_lock, fl_link);
 
 	lock_get_status(f, fl, *((loff_t *)f->private), "");
 
-	list_for_each_entry(bfl, &blocked_list, fl_link) {
+	hlist_for_each_entry(bfl, &blocked_list, fl_link) {
 		if (bfl->fl_next == fl)
 			lock_get_status(f, bfl, *((loff_t *)f->private), " ->");
 	}
@@ -2304,14 +2304,14 @@ static void *locks_start(struct seq_file *f, loff_t *pos)
 
 	spin_lock(&file_lock_lock);
 	*p = (*pos + 1);
-	return seq_list_start(&file_lock_list, *pos);
+	return seq_hlist_start(&file_lock_list, *pos);
 }
 
 static void *locks_next(struct seq_file *f, void *v, loff_t *pos)
 {
 	loff_t *p = f->private;
 	++*p;
-	return seq_list_next(v, &file_lock_list, pos);
+	return seq_hlist_next(v, &file_lock_list, pos);
 }
 
 static void locks_stop(struct seq_file *f, void *v)
diff --git a/include/linux/fs.h b/include/linux/fs.h
index e2f896d..3b340f7 100644
--- a/include/linux/fs.h
+++ b/include/linux/fs.h
@@ -946,7 +946,7 @@ int locks_in_grace(struct net *);
  */
 struct file_lock {
 	struct file_lock *fl_next;	/* singly linked list for this inode  */
-	struct list_head fl_link;	/* doubly linked list of all locks */
+	struct hlist_node fl_link;	/* node in global lists */
 	struct list_head fl_block;	/* circular list of blocked processes */
 	fl_owner_t fl_owner;
 	unsigned int fl_flags;
-- 
1.7.1



From jlayton at redhat.com  Tue Jun 11 07:08:54 2013
From: jlayton at redhat.com (Jeff Layton)
Date: Tue, 11 Jun 2013 07:08:54 -0400
Subject: [PATCH v2 00/14] locks: scalability improvements for file locking
Message-ID: <1370948948-31784-1-git-send-email-jlayton@redhat.com>

Summary of Significant Changes:
-------------------------------
v2:
- Fix potential races in deadlock detection. Manipulation of global
  blocked_hash and deadlock detection are now atomic. This is a
  little slower than the earlier set, but is provably correct. Also,
  the patch that converts to using the i_lock has been split out from
  most of the other changes. That should make it easier to review, but
  it does leave a potential race in the deadlock detection that is fixed
  up by the following patch. It may make sense to fold patches 7 and 8
  together before merging.

- Add percpu hlists and lglocks for global file_lock_list. This gives
  us some speedup since this list is seldom read.

Abstract (tl;dr version):
-------------------------
This patchset represents an overhaul of the file locking code with an
aim toward improving its scalability and making the code a bit easier to
understand.

Longer version:
---------------
When the BKL was finally ripped out of the kernel in 2010, the strategy
taken for the file locking code was to simply turn it into a new
file_lock_locks spinlock. It was an expedient way to deal with the file
locking code at the time, but having a giant spinlock around all of this
code is clearly not great for scalability. Red Hat has bug reports that
go back into the 2.6.18 era that point to BKL scalability problems in
the file locking code and the file_lock_lock suffers from the same
issues.

This patchset is my first attempt to make this code less dependent on
global locking. The main change is to switch most of the file locking
code to be protected by the inode->i_lock instead of the file_lock_lock.

While that works for most things, there are a couple of global data
structures (lists in the current code) that need a global lock to
protect them. So we still need a global lock in order to deal with
those. The remaining patches are intended to make that global locking
less painful. The big gainis are made by turning the blocked_list into a
hashtable, which greatly speeds up the deadlock detection code.

This is not the first attempt at doing this. The conversion to the
i_lock was originally attempted by Bruce Fields a few years ago. His
approach was NAK'ed since it involved ripping out the deadlock
detection. People also really seem to like /proc/locks for debugging, so
keeping that in is probably worthwhile.

There's more work to be done in this area and this patchset is just a
start. There's a horrible thundering herd problem when a blocking lock
is released, for instance. There was also interest in solving the goofy
"unlock on any close" POSIX lock semantics at this year's LSF. I think
this patchset will help lay the groundwork for those changes as well.

While file locking is not usually considered to be a high-performance
codepath, it *is* an IPC mechanism and I think it behooves us to try to
make it as fast as possible.

I'd like to see this considered for 3.11. Comments and suggestions
welcome.

Performance testing and results:
--------------------------------
In order to measure performance here, I've written some locking
performance tests that I've made available here:

    git://git.samba.org/jlayton/lockperf.git

...there are only 4 tests so far that measure performance of BSD and
POSIX locking. I ran each test 100 times. The first number in each set
of results is the unpatched (baseline) and the second is the patched
results. The arch in both cases was x86_64. I've also done some testing
on i686 and the results are more or less inline with the ones below.
The files were all on tmpfs, to eliminate any storage-related latencies
that might creep in.

Here's a description of each test with their results. I've also included
the standard deviation with each:

posix01: fork off 128 children, and have them all open the same file and
lock and unlock it 10k times (whole file, FL_WRLCK). Measure the time
that it takes to perform all of the locks and unlocks in each process,
and the total them all up. Note that the "mean time" here is not the
time to run the test, but the total time that all the tasks spent
locking and unlocking:

2-way Intel machine, 1G RAM, UMA:
kernel				mean time		std. deviation
-------------------------------------------------------------------------
3.10.0-rc4-00034-g042dd60	780.50511606027		20.0773085693901
3.10.0-rc4-00049-ge0b71e1	420.178537863755	24.4692621642102

32-way AMD Opteron machine, 32G RAM in 4 NUMA nodes:
kernel				mean time		std. deviation
-------------------------------------------------------------------------
3.10.0-rc4-00034-g042dd60	30889.1603525361	267.767126085587
3.10.0-rc4-00049-ge0b71e1	25504.7748006928	336.232893246214

The first thing to note here is how painful this test is as the number
of CPUs scales out, and when NUMA is thrown into the mix. It takes a
staggeringly long time to run on such a machine. The patchset helps this
use-case a lot, but it's still painful. I think these numbers help
illustrate why we need to worry about this as the machines get more
parallel.

posix02: fork off 128 children, and have them each open their own file
and lock and unlock it 20k times. Measure the time that it takes to
complete the locks and unlocks in each process and then total them up:

2-way Intel machine, 1G RAM, UMA:
kernel				mean time		std. deviation
-------------------------------------------------------------------------
3.10.0-rc4-00034-g042dd60	319.86610077757		11.0827165395713
3.10.0-rc4-00049-ge0b71e1	257.24231139548		 8.0772950753018

32-way AMD Opteron machine, 32G RAM in 4 NUMA nodes:
kernel				mean time		std. deviation
-------------------------------------------------------------------------
3.10.0-rc4-00034-g042dd60	1354.73702552958	40.5202787251742
3.10.0-rc4-00049-ge0b71e1	 541.35526474265	12.0492755786782

...so again, multiple CPUs and NUMA take their toll on this workload,
but the results are still universally positive from the patchset.

flock01: fork off 128 children, and have them all open the same file and
lock (LOCK_EX) and unlock it 10k times. Measure the time that it takes
to perform all of the locks and unlocks in each process, and the total
them all up. Note that the "mean time" here is not the time to run the
test, but the total time that all the tasks spent locking and unlocking:

2-way Intel machine, 1G RAM, UMA:
kernel				mean time		std. deviation
-------------------------------------------------------------------------
3.10.0-rc4-00034-g042dd60	704.60957498018		10.6182455520549
3.10.0-rc4-00049-ge0b71e1	729.30782576048		 4.62185698266119

32-way AMD Opteron machine, 32G RAM in 4 NUMA nodes:
kernel				mean time		std. deviation
-------------------------------------------------------------------------
3.10.0-rc4-00034-g042dd60	23617.876013741		221.445828515533
3.10.0-rc4-00049-ge0b71e1	24743.3849476248	340.664907591012

...so here we see some slight slowdown (3-5%). In the case of the UMA
box, it's on the order of the standard deviation, and in a similar test
on a different box, the patched kernel turned out to be faster. Go
figure...

In any case, I'm suspect that this is due to the fact that with this
set, we need to take two spinlocks in order to do the work (the i_lock
and the file_lock_lglock). Since the flock codepath doesn't use the
global blocked_list, it sees no benefit to the conversion of it to a
hash table.

flock02: fork off 128 children, and have them each open their own file
and lock (LOCK_EX) and unlock it 20k times. Measure the time that it
takes to complete the locks and unlocks in each process and then total
them up:

2-way Intel machine, 1G RAM, UMA:
kernel				mean time		std. deviation
-------------------------------------------------------------------------
3.10.0-rc4-00034-g042dd60	241.26319896997		7.69747220750147
3.10.0-rc4-00049-ge0b71e1	167.23062027739		5.1643997156006

32-way AMD Opteron machine, 32G RAM in 4 NUMA nodes:
kernel				mean time		std. deviation
--------------------------------------------------------------------------
3.10.0-rc4-00034-g042dd60	1348.64594042422	39.5186585988528
3.10.0-rc4-00049-ge0b71e1	   8.4762987879		 0.347121187462215

...the speedup here is really dramatic, especially in the NUMA case.
This was so much faster that I had to double check that flock locking
still worked properly afterward.

Jeff Layton (14):
  cifs: use posix_unblock_lock instead of locks_delete_block
  locks: make generic_add_lease and generic_delete_lease static
  locks: comment cleanups and clarifications
  locks: make "added" in __posix_lock_file a bool
  locks: encapsulate the fl_link list handling
  locks: don't walk inode->i_flock list in locks_show
  locks: convert to i_lock to protect i_flock list
  locks: ensure that deadlock detection is atomic with respect to
    blocked_list modification
  locks: convert fl_link to a hlist_node
  locks: turn the blocked_list into a hashtable
  locks: add a new "lm_owner_key" lock operation
  locks: give the blocked_hash its own spinlock
  seq_file: add seq_list_*_percpu helpers
  locks: move file_lock_list to a set of percpu hlist_heads and convert
    file_lock_lock to an lglock

 Documentation/filesystems/Locking |   27 +++-
 fs/afs/flock.c                    |    5 +-
 fs/ceph/locks.c                   |    2 +-
 fs/ceph/mds_client.c              |    8 +-
 fs/cifs/cifsfs.c                  |    2 +-
 fs/cifs/file.c                    |   15 +-
 fs/gfs2/file.c                    |    2 +-
 fs/lockd/svclock.c                |   12 ++
 fs/lockd/svcsubs.c                |   12 +-
 fs/locks.c                        |  306 +++++++++++++++++++++++++------------
 fs/nfs/delegation.c               |   11 +-
 fs/nfs/nfs4state.c                |    8 +-
 fs/nfsd/nfs4state.c               |    8 +-
 fs/seq_file.c                     |   54 +++++++
 include/linux/fs.h                |   38 +++--
 include/linux/seq_file.h          |    6 +
 16 files changed, 362 insertions(+), 154 deletions(-)



From jlayton at redhat.com  Tue Jun 11 07:09:05 2013
From: jlayton at redhat.com (Jeff Layton)
Date: Tue, 11 Jun 2013 07:09:05 -0400
Subject: [PATCH v2 11/14] locks: add a new "lm_owner_key" lock operation
In-Reply-To: <1370948948-31784-1-git-send-email-jlayton@redhat.com>
References: <1370948948-31784-1-git-send-email-jlayton@redhat.com>
Message-ID: <1370948948-31784-12-git-send-email-jlayton@redhat.com>

Currently, the hashing that the locking code uses to add these values
to the blocked_hash is simply calculated using fl_owner field. That's
valid in most cases except for server-side lockd, which validates the
owner of a lock based on fl_owner and fl_pid.

In the case where you have a small number of NFS clients doing a lot
of locking between different processes, you could end up with all
the blocked requests sitting in a very small number of hash buckets.

Add a new lm_owner_key operation to the lock_manager_operations that
will generate an unsigned long to use as the key in the hashtable.
That function is only implemented for server-side lockd, and simply
XORs the fl_owner and fl_pid.

Signed-off-by: Jeff Layton <jlayton at redhat.com>
---
 Documentation/filesystems/Locking |   18 +++++++++++-------
 fs/lockd/svclock.c                |   12 ++++++++++++
 fs/locks.c                        |   12 ++++++++++--
 include/linux/fs.h                |    1 +
 4 files changed, 34 insertions(+), 9 deletions(-)

diff --git a/Documentation/filesystems/Locking b/Documentation/filesystems/Locking
index 13f91ab..ee351ac 100644
--- a/Documentation/filesystems/Locking
+++ b/Documentation/filesystems/Locking
@@ -351,6 +351,7 @@ fl_release_private:	maybe		no
 ----------------------- lock_manager_operations ---------------------------
 prototypes:
 	int (*lm_compare_owner)(struct file_lock *, struct file_lock *);
+	unsigned long (*lm_owner_key)(struct file_lock *);
 	void (*lm_notify)(struct file_lock *);  /* unblock callback */
 	int (*lm_grant)(struct file_lock *, struct file_lock *, int);
 	void (*lm_break)(struct file_lock *); /* break_lease callback */
@@ -360,18 +361,21 @@ locking rules:
 
 			inode->i_lock	file_lock_lock	may block
 lm_compare_owner:	yes		maybe		no
+lm_owner_key		yes		yes		no
 lm_notify:		yes		no		no
 lm_grant:		no		no		no
 lm_break:		yes		no		no
 lm_change		yes		no		no
 
-	->lm_compare_owner is generally called with *an* inode->i_lock
-held. It may not be the i_lock of the inode for either file_lock being
-compared! This is the case with deadlock detection, since the code has
-to chase down the owners of locks that may be entirely unrelated to the
-one on which the lock is being acquired. For deadlock detection however,
-the file_lock_lock is also held. The locks primarily ensure that neither
-file_lock disappear out from under you while doing the comparison.
+	->lm_compare_owner and ->lm_owner_key are generally called with
+*an* inode->i_lock held. It may not be the i_lock of the inode
+associated with either file_lock argument! This is the case with deadlock
+detection, since the code has to chase down the owners of locks that may
+be entirely unrelated to the one on which the lock is being acquired.
+For deadlock detection however, the file_lock_lock is also held. The
+fact that these locks are held ensures that the file_locks do not
+disappear out from under you while doing the comparison or generating an
+owner key.
 
 --------------------------- buffer_head -----------------------------------
 prototypes:
diff --git a/fs/lockd/svclock.c b/fs/lockd/svclock.c
index e703318..ce2cdab 100644
--- a/fs/lockd/svclock.c
+++ b/fs/lockd/svclock.c
@@ -744,8 +744,20 @@ static int nlmsvc_same_owner(struct file_lock *fl1, struct file_lock *fl2)
 	return fl1->fl_owner == fl2->fl_owner && fl1->fl_pid == fl2->fl_pid;
 }
 
+/*
+ * Since NLM uses two "keys" for tracking locks, we need to hash them down
+ * to one for the blocked_hash. Here, we're just xor'ing the host address
+ * with the pid in order to create a key value for picking a hash bucket.
+ */
+static unsigned long
+nlmsvc_owner_key(struct file_lock *fl)
+{
+	return (unsigned long)fl->fl_owner ^ (unsigned long)fl->fl_pid;
+}
+
 const struct lock_manager_operations nlmsvc_lock_operations = {
 	.lm_compare_owner = nlmsvc_same_owner,
+	.lm_owner_key = nlmsvc_owner_key,
 	.lm_notify = nlmsvc_notify_blocked,
 	.lm_grant = nlmsvc_grant_deferred,
 };
diff --git a/fs/locks.c b/fs/locks.c
index 76fb7af..11e7784 100644
--- a/fs/locks.c
+++ b/fs/locks.c
@@ -481,11 +481,19 @@ static int posix_same_owner(struct file_lock *fl1, struct file_lock *fl2)
 	return fl1->fl_owner == fl2->fl_owner;
 }
 
+static unsigned long
+posix_owner_key(struct file_lock *fl)
+{
+	if (fl->fl_lmops && fl->fl_lmops->lm_owner_key)
+		return fl->fl_lmops->lm_owner_key(fl);
+	return (unsigned long)fl->fl_owner;
+}
+
 /* Remove a blocker or lock from one of the global lists */
 static inline void
 locks_insert_global_blocked(struct file_lock *waiter)
 {
-	hash_add(blocked_hash, &waiter->fl_link, (unsigned long)waiter->fl_owner);
+	hash_add(blocked_hash, &waiter->fl_link, posix_owner_key(waiter));
 }
 
 static inline void
@@ -739,7 +747,7 @@ static struct file_lock *what_owner_is_waiting_for(struct file_lock *block_fl)
 {
 	struct file_lock *fl;
 
-	hash_for_each_possible(blocked_hash, fl, fl_link, (unsigned long)block_fl->fl_owner) {
+	hash_for_each_possible(blocked_hash, fl, fl_link, posix_owner_key(block_fl)) {
 		if (posix_same_owner(fl, block_fl))
 			return fl->fl_next;
 	}
diff --git a/include/linux/fs.h b/include/linux/fs.h
index 3b340f7..232a345 100644
--- a/include/linux/fs.h
+++ b/include/linux/fs.h
@@ -908,6 +908,7 @@ struct file_lock_operations {
 
 struct lock_manager_operations {
 	int (*lm_compare_owner)(struct file_lock *, struct file_lock *);
+	unsigned long (*lm_owner_key)(struct file_lock *);
 	void (*lm_notify)(struct file_lock *);	/* unblock callback */
 	int (*lm_grant)(struct file_lock *, struct file_lock *, int);
 	void (*lm_break)(struct file_lock *);
-- 
1.7.1



From jlayton at redhat.com  Tue Jun 11 07:09:07 2013
From: jlayton at redhat.com (Jeff Layton)
Date: Tue, 11 Jun 2013 07:09:07 -0400
Subject: [PATCH v2 13/14] seq_file: add seq_list_*_percpu helpers
In-Reply-To: <1370948948-31784-1-git-send-email-jlayton@redhat.com>
References: <1370948948-31784-1-git-send-email-jlayton@redhat.com>
Message-ID: <1370948948-31784-14-git-send-email-jlayton@redhat.com>

When we convert the file_lock_list to a set of percpu lists, we'll need
a way to iterate over them in order to output /proc/locks info. Add
some seq_list_*_percpu helpers to handle that.

Signed-off-by: Jeff Layton <jlayton at redhat.com>
---
 fs/seq_file.c            |   54 ++++++++++++++++++++++++++++++++++++++++++++++
 include/linux/seq_file.h |    6 +++++
 2 files changed, 60 insertions(+), 0 deletions(-)

diff --git a/fs/seq_file.c b/fs/seq_file.c
index 774c1eb..ce9f97a 100644
--- a/fs/seq_file.c
+++ b/fs/seq_file.c
@@ -921,3 +921,57 @@ struct hlist_node *seq_hlist_next_rcu(void *v,
 		return rcu_dereference(node->next);
 }
 EXPORT_SYMBOL(seq_hlist_next_rcu);
+
+/**
+ * seq_hlist_start_precpu - start an iteration of a hlist
+ * @head: pointer to percpu array of struct hlist_heads
+ * @cpu:  pointer to cpu "cursor"
+ * @pos:  start position of sequence
+ *
+ * Called at seq_file->op->start().
+ */
+struct hlist_node *
+seq_hlist_start_percpu(struct hlist_head __percpu *head, int *cpu, loff_t pos)
+{
+	struct hlist_node *node;
+
+	for_each_possible_cpu(*cpu) {
+		hlist_for_each(node, per_cpu_ptr(head, *cpu)) {
+			if (pos-- == 0)
+				return node;
+		}
+	}
+	return NULL;
+}
+EXPORT_SYMBOL(seq_hlist_start_percpu);
+
+/**
+ * seq_hlist_next_percpu - move to the next position of the hlist
+ * @v:    pointer to current hlist_node
+ * @head: pointer to percpu array of struct hlist_heads
+ * @cpu:  pointer to cpu "cursor"
+ * @pos:  start position of sequence
+ *
+ * Called at seq_file->op->next().
+ */
+struct hlist_node *
+seq_hlist_next_percpu(void *v, struct hlist_head __percpu *head,
+			int *cpu, loff_t *pos)
+{
+	struct hlist_node *node = v;
+
+	++*pos;
+
+	if (node->next)
+		return node->next;
+
+	for (*cpu = cpumask_next(*cpu, cpu_possible_mask); *cpu < nr_cpu_ids;
+	     *cpu = cpumask_next(*cpu, cpu_possible_mask)) {
+		struct hlist_head *bucket = per_cpu_ptr(head, *cpu);
+
+		if (!hlist_empty(bucket))
+			return bucket->first;
+	}
+	return NULL;
+}
+EXPORT_SYMBOL(seq_hlist_next_percpu);
diff --git a/include/linux/seq_file.h b/include/linux/seq_file.h
index 2da29ac..4e32edc 100644
--- a/include/linux/seq_file.h
+++ b/include/linux/seq_file.h
@@ -173,4 +173,10 @@ extern struct hlist_node *seq_hlist_start_head_rcu(struct hlist_head *head,
 extern struct hlist_node *seq_hlist_next_rcu(void *v,
 						   struct hlist_head *head,
 						   loff_t *ppos);
+
+/* Helpers for iterating over per-cpu hlist_head-s in seq_files */
+extern struct hlist_node *seq_hlist_start_percpu(struct hlist_head __percpu *head, int *cpu, loff_t pos);
+
+extern struct hlist_node *seq_hlist_next_percpu(void *v, struct hlist_head __percpu *head, int *cpu, loff_t *pos);
+
 #endif
-- 
1.7.1



From jlayton at redhat.com  Tue Jun 11 07:09:01 2013
From: jlayton at redhat.com (Jeff Layton)
Date: Tue, 11 Jun 2013 07:09:01 -0400
Subject: [PATCH v2 07/14] locks: convert to i_lock to protect i_flock list
In-Reply-To: <1370948948-31784-1-git-send-email-jlayton@redhat.com>
References: <1370948948-31784-1-git-send-email-jlayton@redhat.com>
Message-ID: <1370948948-31784-8-git-send-email-jlayton@redhat.com>

Having a global lock that protects all of this code is a clear
scalability problem. Instead of doing that, move most of the code to be
protected by the i_lock instead.

The exceptions are the global lists that file_lock->fl_link sits on.
Those still need a global lock of some sort, so wrap just those accesses
in the file_lock_lock().

Note that this patch introduces a potential race in the deadlock
detection code which could result in either false positives or missed
file_lock loops. The blocked_list management needs to be done atomically
with the deadlock detection. This will be fixed in a later patch.

Signed-off-by: Jeff Layton <jlayton at redhat.com>
Signed-off-by: J. Bruce Fields <bfields at redhat.com>
---
 Documentation/filesystems/Locking |   23 +++++--
 fs/afs/flock.c                    |    5 +-
 fs/ceph/locks.c                   |    2 +-
 fs/ceph/mds_client.c              |    8 +-
 fs/cifs/cifsfs.c                  |    2 +-
 fs/cifs/file.c                    |   13 ++--
 fs/gfs2/file.c                    |    2 +-
 fs/lockd/svcsubs.c                |   12 ++--
 fs/locks.c                        |  110 ++++++++++++++++++++-----------------
 fs/nfs/delegation.c               |   11 ++--
 fs/nfs/nfs4state.c                |    8 +-
 fs/nfsd/nfs4state.c               |    8 +-
 include/linux/fs.h                |   11 ----
 13 files changed, 113 insertions(+), 102 deletions(-)

diff --git a/Documentation/filesystems/Locking b/Documentation/filesystems/Locking
index 0706d32..13f91ab 100644
--- a/Documentation/filesystems/Locking
+++ b/Documentation/filesystems/Locking
@@ -344,7 +344,7 @@ prototypes:
 
 
 locking rules:
-			file_lock_lock	may block
+			inode->i_lock	may block
 fl_copy_lock:		yes		no
 fl_release_private:	maybe		no
 
@@ -357,12 +357,21 @@ prototypes:
 	int (*lm_change)(struct file_lock **, int);
 
 locking rules:
-			file_lock_lock	may block
-lm_compare_owner:	yes		no
-lm_notify:		yes		no
-lm_grant:		no		no
-lm_break:		yes		no
-lm_change		yes		no
+
+			inode->i_lock	file_lock_lock	may block
+lm_compare_owner:	yes		maybe		no
+lm_notify:		yes		no		no
+lm_grant:		no		no		no
+lm_break:		yes		no		no
+lm_change		yes		no		no
+
+	->lm_compare_owner is generally called with *an* inode->i_lock
+held. It may not be the i_lock of the inode for either file_lock being
+compared! This is the case with deadlock detection, since the code has
+to chase down the owners of locks that may be entirely unrelated to the
+one on which the lock is being acquired. For deadlock detection however,
+the file_lock_lock is also held. The locks primarily ensure that neither
+file_lock disappear out from under you while doing the comparison.
 
 --------------------------- buffer_head -----------------------------------
 prototypes:
diff --git a/fs/afs/flock.c b/fs/afs/flock.c
index 2497bf3..03fc0d1 100644
--- a/fs/afs/flock.c
+++ b/fs/afs/flock.c
@@ -252,6 +252,7 @@ static void afs_defer_unlock(struct afs_vnode *vnode, struct key *key)
  */
 static int afs_do_setlk(struct file *file, struct file_lock *fl)
 {
+	struct inode = file_inode(file);
 	struct afs_vnode *vnode = AFS_FS_I(file->f_mapping->host);
 	afs_lock_type_t type;
 	struct key *key = file->private_data;
@@ -273,7 +274,7 @@ static int afs_do_setlk(struct file *file, struct file_lock *fl)
 
 	type = (fl->fl_type == F_RDLCK) ? AFS_LOCK_READ : AFS_LOCK_WRITE;
 
-	lock_flocks();
+	spin_lock(&inode->i_lock);
 
 	/* make sure we've got a callback on this file and that our view of the
 	 * data version is up to date */
@@ -420,7 +421,7 @@ given_lock:
 	afs_vnode_fetch_status(vnode, NULL, key);
 
 error:
-	unlock_flocks();
+	spin_unlock(&inode->i_lock);
 	_leave(" = %d", ret);
 	return ret;
 
diff --git a/fs/ceph/locks.c b/fs/ceph/locks.c
index 202dd3d..cd0a664 100644
--- a/fs/ceph/locks.c
+++ b/fs/ceph/locks.c
@@ -194,7 +194,7 @@ void ceph_count_locks(struct inode *inode, int *fcntl_count, int *flock_count)
  * Encode the flock and fcntl locks for the given inode into the pagelist.
  * Format is: #fcntl locks, sequential fcntl locks, #flock locks,
  * sequential flock locks.
- * Must be called with lock_flocks() already held.
+ * Must be called with inode->i_lock already held.
  * If we encounter more of a specific lock type than expected,
  * we return the value 1.
  */
diff --git a/fs/ceph/mds_client.c b/fs/ceph/mds_client.c
index 4f22671..ae621b5 100644
--- a/fs/ceph/mds_client.c
+++ b/fs/ceph/mds_client.c
@@ -2482,13 +2482,13 @@ static int encode_caps_cb(struct inode *inode, struct ceph_cap *cap,
 
 		ceph_pagelist_set_cursor(pagelist, &trunc_point);
 		do {
-			lock_flocks();
+			spin_lock(&inode->i_lock);
 			ceph_count_locks(inode, &num_fcntl_locks,
 					 &num_flock_locks);
 			rec.v2.flock_len = (2*sizeof(u32) +
 					    (num_fcntl_locks+num_flock_locks) *
 					    sizeof(struct ceph_filelock));
-			unlock_flocks();
+			spin_unlock(&inode->i_lock);
 
 			/* pre-alloc pagelist */
 			ceph_pagelist_truncate(pagelist, &trunc_point);
@@ -2499,12 +2499,12 @@ static int encode_caps_cb(struct inode *inode, struct ceph_cap *cap,
 
 			/* encode locks */
 			if (!err) {
-				lock_flocks();
+				spin_lock(&inode->i_lock);
 				err = ceph_encode_locks(inode,
 							pagelist,
 							num_fcntl_locks,
 							num_flock_locks);
-				unlock_flocks();
+				spin_unlock(&inode->i_lock);
 			}
 		} while (err == -ENOSPC);
 	} else {
diff --git a/fs/cifs/cifsfs.c b/fs/cifs/cifsfs.c
index 3752b9f..e81655c 100644
--- a/fs/cifs/cifsfs.c
+++ b/fs/cifs/cifsfs.c
@@ -765,7 +765,7 @@ static loff_t cifs_llseek(struct file *file, loff_t offset, int whence)
 
 static int cifs_setlease(struct file *file, long arg, struct file_lock **lease)
 {
-	/* note that this is called by vfs setlease with lock_flocks held
+	/* note that this is called by vfs setlease with i_lock held
 	   to protect *lease from going away */
 	struct inode *inode = file_inode(file);
 	struct cifsFileInfo *cfile = file->private_data;
diff --git a/fs/cifs/file.c b/fs/cifs/file.c
index 44a4f18..0dd10cd 100644
--- a/fs/cifs/file.c
+++ b/fs/cifs/file.c
@@ -1092,6 +1092,7 @@ struct lock_to_push {
 static int
 cifs_push_posix_locks(struct cifsFileInfo *cfile)
 {
+	struct inode *inode = cfile->dentry->d_inode;
 	struct cifs_tcon *tcon = tlink_tcon(cfile->tlink);
 	struct file_lock *flock, **before;
 	unsigned int count = 0, i = 0;
@@ -1102,12 +1103,12 @@ cifs_push_posix_locks(struct cifsFileInfo *cfile)
 
 	xid = get_xid();
 
-	lock_flocks();
-	cifs_for_each_lock(cfile->dentry->d_inode, before) {
+	spin_lock(&inode->i_lock);
+	cifs_for_each_lock(inode, before) {
 		if ((*before)->fl_flags & FL_POSIX)
 			count++;
 	}
-	unlock_flocks();
+	spin_unlock(&inode->i_lock);
 
 	INIT_LIST_HEAD(&locks_to_send);
 
@@ -1126,8 +1127,8 @@ cifs_push_posix_locks(struct cifsFileInfo *cfile)
 	}
 
 	el = locks_to_send.next;
-	lock_flocks();
-	cifs_for_each_lock(cfile->dentry->d_inode, before) {
+	spin_lock(&inode->i_lock);
+	cifs_for_each_lock(inode, before) {
 		flock = *before;
 		if ((flock->fl_flags & FL_POSIX) == 0)
 			continue;
@@ -1152,7 +1153,7 @@ cifs_push_posix_locks(struct cifsFileInfo *cfile)
 		lck->offset = flock->fl_start;
 		el = el->next;
 	}
-	unlock_flocks();
+	spin_unlock(&inode->i_lock);
 
 	list_for_each_entry_safe(lck, tmp, &locks_to_send, llist) {
 		int stored_rc;
diff --git a/fs/gfs2/file.c b/fs/gfs2/file.c
index acd1676..9e634e0 100644
--- a/fs/gfs2/file.c
+++ b/fs/gfs2/file.c
@@ -889,7 +889,7 @@ out_uninit:
  * cluster; until we do, disable leases (by just returning -EINVAL),
  * unless the administrator has requested purely local locking.
  *
- * Locking: called under lock_flocks
+ * Locking: called under i_lock
  *
  * Returns: errno
  */
diff --git a/fs/lockd/svcsubs.c b/fs/lockd/svcsubs.c
index 97e8741..dc5c759 100644
--- a/fs/lockd/svcsubs.c
+++ b/fs/lockd/svcsubs.c
@@ -169,7 +169,7 @@ nlm_traverse_locks(struct nlm_host *host, struct nlm_file *file,
 
 again:
 	file->f_locks = 0;
-	lock_flocks(); /* protects i_flock list */
+	spin_lock(&inode->i_lock);
 	for (fl = inode->i_flock; fl; fl = fl->fl_next) {
 		if (fl->fl_lmops != &nlmsvc_lock_operations)
 			continue;
@@ -181,7 +181,7 @@ again:
 		if (match(lockhost, host)) {
 			struct file_lock lock = *fl;
 
-			unlock_flocks();
+			spin_unlock(&inode->i_lock);
 			lock.fl_type  = F_UNLCK;
 			lock.fl_start = 0;
 			lock.fl_end   = OFFSET_MAX;
@@ -193,7 +193,7 @@ again:
 			goto again;
 		}
 	}
-	unlock_flocks();
+	spin_unlock(&inode->i_lock);
 
 	return 0;
 }
@@ -228,14 +228,14 @@ nlm_file_inuse(struct nlm_file *file)
 	if (file->f_count || !list_empty(&file->f_blocks) || file->f_shares)
 		return 1;
 
-	lock_flocks();
+	spin_lock(&inode->i_lock);
 	for (fl = inode->i_flock; fl; fl = fl->fl_next) {
 		if (fl->fl_lmops == &nlmsvc_lock_operations) {
-			unlock_flocks();
+			spin_unlock(&inode->i_lock);
 			return 1;
 		}
 	}
-	unlock_flocks();
+	spin_unlock(&inode->i_lock);
 	file->f_locks = 0;
 	return 0;
 }
diff --git a/fs/locks.c b/fs/locks.c
index 3fd27f0..d7342a3 100644
--- a/fs/locks.c
+++ b/fs/locks.c
@@ -155,22 +155,9 @@ int lease_break_time = 45;
 
 static LIST_HEAD(file_lock_list);
 static LIST_HEAD(blocked_list);
-static DEFINE_SPINLOCK(file_lock_lock);
-
-/*
- * Protects the two list heads above, plus the inode->i_flock list
- */
-void lock_flocks(void)
-{
-	spin_lock(&file_lock_lock);
-}
-EXPORT_SYMBOL_GPL(lock_flocks);
 
-void unlock_flocks(void)
-{
-	spin_unlock(&file_lock_lock);
-}
-EXPORT_SYMBOL_GPL(unlock_flocks);
+/* Protects the two list heads above */
+static DEFINE_SPINLOCK(file_lock_lock);
 
 static struct kmem_cache *filelock_cache __read_mostly;
 
@@ -488,25 +475,33 @@ static int posix_same_owner(struct file_lock *fl1, struct file_lock *fl2)
 static inline void
 locks_insert_global_blocked(struct file_lock *waiter)
 {
+	spin_lock(&file_lock_lock);
 	list_add(&waiter->fl_link, &blocked_list);
+	spin_unlock(&file_lock_lock);
 }
 
 static inline void
 locks_delete_global_blocked(struct file_lock *waiter)
 {
+	spin_lock(&file_lock_lock);
 	list_del_init(&waiter->fl_link);
+	spin_unlock(&file_lock_lock);
 }
 
 static inline void
 locks_insert_global_locks(struct file_lock *waiter)
 {
+	spin_lock(&file_lock_lock);
 	list_add_tail(&waiter->fl_link, &file_lock_list);
+	spin_unlock(&file_lock_lock);
 }
 
 static inline void
 locks_delete_global_locks(struct file_lock *waiter)
 {
+	spin_lock(&file_lock_lock);
 	list_del_init(&waiter->fl_link);
+	spin_unlock(&file_lock_lock);
 }
 
 /* Remove waiter from blocker's block list.
@@ -523,9 +518,11 @@ static void __locks_delete_block(struct file_lock *waiter)
  */
 static void locks_delete_block(struct file_lock *waiter)
 {
-	lock_flocks();
+	struct inode *inode = file_inode(waiter->fl_file);
+
+	spin_lock(&inode->i_lock);
 	__locks_delete_block(waiter);
-	unlock_flocks();
+	spin_unlock(&inode->i_lock);
 }
 
 /* Insert waiter into blocker's block list.
@@ -544,7 +541,7 @@ static void locks_insert_block(struct file_lock *blocker,
 /*
  * Wake up processes blocked waiting for blocker.
  *
- * Must be called with the file_lock_lock held!
+ * Must be called with the inode->i_lock of the blocker held!
  */
 static void locks_wake_up_blocks(struct file_lock *blocker)
 {
@@ -649,8 +646,9 @@ void
 posix_test_lock(struct file *filp, struct file_lock *fl)
 {
 	struct file_lock *cfl;
+	struct inode *inode = file_inode(filp);
 
-	lock_flocks();
+	spin_lock(&inode->i_lock);
 	for (cfl = file_inode(filp)->i_flock; cfl; cfl = cfl->fl_next) {
 		if (!IS_POSIX(cfl))
 			continue;
@@ -663,7 +661,7 @@ posix_test_lock(struct file *filp, struct file_lock *fl)
 			fl->fl_pid = pid_vnr(cfl->fl_nspid);
 	} else
 		fl->fl_type = F_UNLCK;
-	unlock_flocks();
+	spin_unlock(&inode->i_lock);
 	return;
 }
 EXPORT_SYMBOL(posix_test_lock);
@@ -742,7 +740,7 @@ static int flock_lock_file(struct file *filp, struct file_lock *request)
 			return -ENOMEM;
 	}
 
-	lock_flocks();
+	spin_lock(&inode->i_lock);
 	if (request->fl_flags & FL_ACCESS)
 		goto find_conflict;
 
@@ -772,9 +770,9 @@ static int flock_lock_file(struct file *filp, struct file_lock *request)
 	 * give it the opportunity to lock the file.
 	 */
 	if (found) {
-		unlock_flocks();
+		spin_unlock(&inode->i_lock);
 		cond_resched();
-		lock_flocks();
+		spin_lock(&inode->i_lock);
 	}
 
 find_conflict:
@@ -801,7 +799,7 @@ find_conflict:
 	error = 0;
 
 out:
-	unlock_flocks();
+	spin_unlock(&inode->i_lock);
 	if (new_fl)
 		locks_free_lock(new_fl);
 	return error;
@@ -831,7 +829,7 @@ static int __posix_lock_file(struct inode *inode, struct file_lock *request, str
 		new_fl2 = locks_alloc_lock();
 	}
 
-	lock_flocks();
+	spin_lock(&inode->i_lock);
 	/*
 	 * New lock request. Walk all POSIX locks and look for conflicts. If
 	 * there are any, either return error or put the request on the
@@ -850,8 +848,14 @@ static int __posix_lock_file(struct inode *inode, struct file_lock *request, str
 			if (!(request->fl_flags & FL_SLEEP))
 				goto out;
 			error = -EDEADLK;
+			/*
+			 * XXX: potential race here. We should be adding the
+			 * file_lock to the global list before releasing lock.
+			 */
+			spin_lock(&file_lock_lock);
 			if (posix_locks_deadlock(request, fl))
 				goto out;
+			spin_unlock(&file_lock_lock);
 			error = FILE_LOCK_DEFERRED;
 			locks_insert_block(fl, request);
 			locks_insert_global_blocked(request);
@@ -1004,7 +1008,7 @@ static int __posix_lock_file(struct inode *inode, struct file_lock *request, str
 		locks_wake_up_blocks(left);
 	}
  out:
-	unlock_flocks();
+	spin_unlock(&inode->i_lock);
 	/*
 	 * Free any unused locks.
 	 */
@@ -1079,14 +1083,14 @@ int locks_mandatory_locked(struct inode *inode)
 	/*
 	 * Search the lock list for this inode for any POSIX locks.
 	 */
-	lock_flocks();
+	spin_lock(&inode->i_lock);
 	for (fl = inode->i_flock; fl != NULL; fl = fl->fl_next) {
 		if (!IS_POSIX(fl))
 			continue;
 		if (fl->fl_owner != owner)
 			break;
 	}
-	unlock_flocks();
+	spin_unlock(&inode->i_lock);
 	return fl ? -EAGAIN : 0;
 }
 
@@ -1229,7 +1233,7 @@ int __break_lease(struct inode *inode, unsigned int mode)
 	if (IS_ERR(new_fl))
 		return PTR_ERR(new_fl);
 
-	lock_flocks();
+	spin_lock(&inode->i_lock);
 
 	time_out_leases(inode);
 
@@ -1279,10 +1283,10 @@ restart:
 			break_time++;
 	}
 	locks_insert_block(flock, new_fl);
-	unlock_flocks();
+	spin_unlock(&inode->i_lock);
 	error = wait_event_interruptible_timeout(new_fl->fl_wait,
 						!new_fl->fl_next, break_time);
-	lock_flocks();
+	spin_lock(&inode->i_lock);
 	__locks_delete_block(new_fl);
 	if (error >= 0) {
 		if (error == 0)
@@ -1300,7 +1304,7 @@ restart:
 	}
 
 out:
-	unlock_flocks();
+	spin_unlock(&inode->i_lock);
 	locks_free_lock(new_fl);
 	return error;
 }
@@ -1353,9 +1357,10 @@ EXPORT_SYMBOL(lease_get_mtime);
 int fcntl_getlease(struct file *filp)
 {
 	struct file_lock *fl;
+	struct inode *inode = file_inode(filp);
 	int type = F_UNLCK;
 
-	lock_flocks();
+	spin_lock(&inode->i_lock);
 	time_out_leases(file_inode(filp));
 	for (fl = file_inode(filp)->i_flock; fl && IS_LEASE(fl);
 			fl = fl->fl_next) {
@@ -1364,7 +1369,7 @@ int fcntl_getlease(struct file *filp)
 			break;
 		}
 	}
-	unlock_flocks();
+	spin_unlock(&inode->i_lock);
 	return type;
 }
 
@@ -1458,7 +1463,7 @@ static int generic_delete_lease(struct file *filp, struct file_lock **flp)
  *	The (input) flp->fl_lmops->lm_break function is required
  *	by break_lease().
  *
- *	Called with file_lock_lock held.
+ *	Called with inode->i_lock held.
  */
 int generic_setlease(struct file *filp, long arg, struct file_lock **flp)
 {
@@ -1527,11 +1532,12 @@ static int __vfs_setlease(struct file *filp, long arg, struct file_lock **lease)
 
 int vfs_setlease(struct file *filp, long arg, struct file_lock **lease)
 {
+	struct inode *inode = file_inode(filp);
 	int error;
 
-	lock_flocks();
+	spin_lock(&inode->i_lock);
 	error = __vfs_setlease(filp, arg, lease);
-	unlock_flocks();
+	spin_unlock(&inode->i_lock);
 
 	return error;
 }
@@ -1549,6 +1555,7 @@ static int do_fcntl_delete_lease(struct file *filp)
 static int do_fcntl_add_lease(unsigned int fd, struct file *filp, long arg)
 {
 	struct file_lock *fl, *ret;
+	struct inode *inode = file_inode(filp);
 	struct fasync_struct *new;
 	int error;
 
@@ -1562,10 +1569,10 @@ static int do_fcntl_add_lease(unsigned int fd, struct file *filp, long arg)
 		return -ENOMEM;
 	}
 	ret = fl;
-	lock_flocks();
+	spin_lock(&inode->i_lock);
 	error = __vfs_setlease(filp, arg, &ret);
 	if (error) {
-		unlock_flocks();
+		spin_unlock(&inode->i_lock);
 		locks_free_lock(fl);
 		goto out_free_fasync;
 	}
@@ -1582,7 +1589,7 @@ static int do_fcntl_add_lease(unsigned int fd, struct file *filp, long arg)
 		new = NULL;
 
 	error = __f_setown(filp, task_pid(current), PIDTYPE_PID, 0);
-	unlock_flocks();
+	spin_unlock(&inode->i_lock);
 
 out_free_fasync:
 	if (new)
@@ -2106,7 +2113,7 @@ void locks_remove_flock(struct file *filp)
 			fl.fl_ops->fl_release_private(&fl);
 	}
 
-	lock_flocks();
+	spin_lock(&inode->i_lock);
 	before = &inode->i_flock;
 
 	while ((fl = *before) != NULL) {
@@ -2124,7 +2131,7 @@ void locks_remove_flock(struct file *filp)
  		}
 		before = &fl->fl_next;
 	}
-	unlock_flocks();
+	spin_unlock(&inode->i_lock);
 }
 
 /**
@@ -2137,14 +2144,15 @@ void locks_remove_flock(struct file *filp)
 int
 posix_unblock_lock(struct file *filp, struct file_lock *waiter)
 {
+	struct inode *inode = file_inode(filp);
 	int status = 0;
 
-	lock_flocks();
+	spin_lock(&inode->i_lock);
 	if (waiter->fl_next)
 		__locks_delete_block(waiter);
 	else
 		status = -ENOENT;
-	unlock_flocks();
+	spin_unlock(&inode->i_lock);
 	return status;
 }
 
@@ -2261,7 +2269,7 @@ static void *locks_start(struct seq_file *f, loff_t *pos)
 {
 	loff_t *p = f->private;
 
-	lock_flocks();
+	spin_lock(&file_lock_lock);
 	*p = (*pos + 1);
 	return seq_list_start(&file_lock_list, *pos);
 }
@@ -2275,7 +2283,7 @@ static void *locks_next(struct seq_file *f, void *v, loff_t *pos)
 
 static void locks_stop(struct seq_file *f, void *v)
 {
-	unlock_flocks();
+	spin_unlock(&file_lock_lock);
 }
 
 static const struct seq_operations locks_seq_operations = {
@@ -2322,7 +2330,8 @@ int lock_may_read(struct inode *inode, loff_t start, unsigned long len)
 {
 	struct file_lock *fl;
 	int result = 1;
-	lock_flocks();
+
+	spin_lock(&inode->i_lock);
 	for (fl = inode->i_flock; fl != NULL; fl = fl->fl_next) {
 		if (IS_POSIX(fl)) {
 			if (fl->fl_type == F_RDLCK)
@@ -2339,7 +2348,7 @@ int lock_may_read(struct inode *inode, loff_t start, unsigned long len)
 		result = 0;
 		break;
 	}
-	unlock_flocks();
+	spin_unlock(&inode->i_lock);
 	return result;
 }
 
@@ -2362,7 +2371,8 @@ int lock_may_write(struct inode *inode, loff_t start, unsigned long len)
 {
 	struct file_lock *fl;
 	int result = 1;
-	lock_flocks();
+
+	spin_lock(&inode->i_lock);
 	for (fl = inode->i_flock; fl != NULL; fl = fl->fl_next) {
 		if (IS_POSIX(fl)) {
 			if ((fl->fl_end < start) || (fl->fl_start > (start + len)))
@@ -2377,7 +2387,7 @@ int lock_may_write(struct inode *inode, loff_t start, unsigned long len)
 		result = 0;
 		break;
 	}
-	unlock_flocks();
+	spin_unlock(&inode->i_lock);
 	return result;
 }
 
diff --git a/fs/nfs/delegation.c b/fs/nfs/delegation.c
index 57db324..43ee7f9 100644
--- a/fs/nfs/delegation.c
+++ b/fs/nfs/delegation.c
@@ -73,20 +73,21 @@ static int nfs_delegation_claim_locks(struct nfs_open_context *ctx, struct nfs4_
 	if (inode->i_flock == NULL)
 		goto out;
 
-	/* Protect inode->i_flock using the file locks lock */
-	lock_flocks();
+	/* Protect inode->i_flock using the i_lock */
+	spin_lock(&inode->i_lock);
 	for (fl = inode->i_flock; fl != NULL; fl = fl->fl_next) {
 		if (!(fl->fl_flags & (FL_POSIX|FL_FLOCK)))
 			continue;
 		if (nfs_file_open_context(fl->fl_file) != ctx)
 			continue;
-		unlock_flocks();
+		/* FIXME: safe to drop lock here while walking list? */
+		spin_unlock(&inode->i_lock);
 		status = nfs4_lock_delegation_recall(fl, state, stateid);
 		if (status < 0)
 			goto out;
-		lock_flocks();
+		spin_lock(&inode->i_lock);
 	}
-	unlock_flocks();
+	spin_unlock(&inode->i_lock);
 out:
 	return status;
 }
diff --git a/fs/nfs/nfs4state.c b/fs/nfs/nfs4state.c
index 1fab140..ff10b4a 100644
--- a/fs/nfs/nfs4state.c
+++ b/fs/nfs/nfs4state.c
@@ -1373,13 +1373,13 @@ static int nfs4_reclaim_locks(struct nfs4_state *state, const struct nfs4_state_
 	/* Guard against delegation returns and new lock/unlock calls */
 	down_write(&nfsi->rwsem);
 	/* Protect inode->i_flock using the BKL */
-	lock_flocks();
+	spin_lock(&inode->i_lock);
 	for (fl = inode->i_flock; fl != NULL; fl = fl->fl_next) {
 		if (!(fl->fl_flags & (FL_POSIX|FL_FLOCK)))
 			continue;
 		if (nfs_file_open_context(fl->fl_file)->state != state)
 			continue;
-		unlock_flocks();
+		spin_unlock(&inode->i_lock);
 		status = ops->recover_lock(state, fl);
 		switch (status) {
 			case 0:
@@ -1406,9 +1406,9 @@ static int nfs4_reclaim_locks(struct nfs4_state *state, const struct nfs4_state_
 				/* kill_proc(fl->fl_pid, SIGLOST, 1); */
 				status = 0;
 		}
-		lock_flocks();
+		spin_lock(&inode->i_lock);
 	}
-	unlock_flocks();
+	spin_unlock(&inode->i_lock);
 out:
 	up_write(&nfsi->rwsem);
 	return status;
diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
index 316ec84..f170518 100644
--- a/fs/nfsd/nfs4state.c
+++ b/fs/nfsd/nfs4state.c
@@ -2645,13 +2645,13 @@ static void nfsd_break_one_deleg(struct nfs4_delegation *dp)
 
 	list_add_tail(&dp->dl_recall_lru, &nn->del_recall_lru);
 
-	/* only place dl_time is set. protected by lock_flocks*/
+	/* Only place dl_time is set; protected by i_lock: */
 	dp->dl_time = get_seconds();
 
 	nfsd4_cb_recall(dp);
 }
 
-/* Called from break_lease() with lock_flocks() held. */
+/* Called from break_lease() with i_lock held. */
 static void nfsd_break_deleg_cb(struct file_lock *fl)
 {
 	struct nfs4_file *fp = (struct nfs4_file *)fl->fl_owner;
@@ -4520,7 +4520,7 @@ check_for_locks(struct nfs4_file *filp, struct nfs4_lockowner *lowner)
 	struct inode *inode = filp->fi_inode;
 	int status = 0;
 
-	lock_flocks();
+	spin_lock(&inode->i_lock);
 	for (flpp = &inode->i_flock; *flpp != NULL; flpp = &(*flpp)->fl_next) {
 		if ((*flpp)->fl_owner == (fl_owner_t)lowner) {
 			status = 1;
@@ -4528,7 +4528,7 @@ check_for_locks(struct nfs4_file *filp, struct nfs4_lockowner *lowner)
 		}
 	}
 out:
-	unlock_flocks();
+	spin_unlock(&inode->i_lock);
 	return status;
 }
 
diff --git a/include/linux/fs.h b/include/linux/fs.h
index 94105d2..e2f896d 100644
--- a/include/linux/fs.h
+++ b/include/linux/fs.h
@@ -1024,8 +1024,6 @@ extern int vfs_setlease(struct file *, long, struct file_lock **);
 extern int lease_modify(struct file_lock **, int);
 extern int lock_may_read(struct inode *, loff_t start, unsigned long count);
 extern int lock_may_write(struct inode *, loff_t start, unsigned long count);
-extern void lock_flocks(void);
-extern void unlock_flocks(void);
 #else /* !CONFIG_FILE_LOCKING */
 static inline int fcntl_getlk(struct file *file, struct flock __user *user)
 {
@@ -1167,15 +1165,6 @@ static inline int lock_may_write(struct inode *inode, loff_t start,
 {
 	return 1;
 }
-
-static inline void lock_flocks(void)
-{
-}
-
-static inline void unlock_flocks(void)
-{
-}
-
 #endif /* !CONFIG_FILE_LOCKING */
 
 
-- 
1.7.1



From jlayton at redhat.com  Tue Jun 11 07:09:08 2013
From: jlayton at redhat.com (Jeff Layton)
Date: Tue, 11 Jun 2013 07:09:08 -0400
Subject: [PATCH v2 14/14] locks: move file_lock_list to a set of percpu
 hlist_heads and convert file_lock_lock to an lglock
In-Reply-To: <1370948948-31784-1-git-send-email-jlayton@redhat.com>
References: <1370948948-31784-1-git-send-email-jlayton@redhat.com>
Message-ID: <1370948948-31784-15-git-send-email-jlayton@redhat.com>

The file_lock_list is only used for /proc/locks. The vastly common case
is for locks to be put onto the list and come off again, without ever
being traversed.

Help optimize for this use-case by moving to percpu hlist_head-s. At the
same time, we can make the locking less contentious by moving to an
lglock. When iterating over the lists for /proc/locks, we must take the
global lock and then iterate over each CPU's list in turn.

This change necessitates a new fl_link_cpu field to keep track of which
CPU the entry is on. On x86_64 at least, this field is placed within an
existing hole in the struct to avoid growing its real size.

Signed-off-by: Jeff Layton <jlayton at redhat.com>
---
 fs/locks.c         |   57 +++++++++++++++++++++++++++++++++++----------------
 include/linux/fs.h |    1 +
 2 files changed, 40 insertions(+), 18 deletions(-)

diff --git a/fs/locks.c b/fs/locks.c
index 8124fc1..094eb4d 100644
--- a/fs/locks.c
+++ b/fs/locks.c
@@ -127,6 +127,8 @@
 #include <linux/rcupdate.h>
 #include <linux/pid_namespace.h>
 #include <linux/hashtable.h>
+#include <linux/percpu.h>
+#include <linux/lglock.h>
 
 #include <asm/uaccess.h>
 
@@ -165,8 +167,8 @@ int lease_break_time = 45;
 static DEFINE_SPINLOCK(blocked_hash_lock);
 static DEFINE_HASHTABLE(blocked_hash, BLOCKED_HASH_BITS);
 
-static DEFINE_SPINLOCK(file_lock_lock);
-static HLIST_HEAD(file_lock_list);
+DEFINE_STATIC_LGLOCK(file_lock_lglock);
+static DEFINE_PER_CPU(struct hlist_head, file_lock_list);
 
 static struct kmem_cache *filelock_cache __read_mostly;
 
@@ -512,17 +514,21 @@ locks_delete_global_blocked(struct file_lock *waiter)
 static inline void
 locks_insert_global_locks(struct file_lock *waiter)
 {
-	spin_lock(&file_lock_lock);
-	hlist_add_head(&waiter->fl_link, &file_lock_list);
-	spin_unlock(&file_lock_lock);
+	lg_local_lock(&file_lock_lglock);
+	waiter->fl_link_cpu = smp_processor_id();
+	hlist_add_head(&waiter->fl_link, this_cpu_ptr(&file_lock_list));
+	lg_local_unlock(&file_lock_lglock);
 }
 
 static inline void
 locks_delete_global_locks(struct file_lock *waiter)
 {
-	spin_lock(&file_lock_lock);
+	/* avoid taking lock if already unhashed */
+	if (hlist_unhashed(&waiter->fl_link))
+		return;
+	lg_local_lock_cpu(&file_lock_lglock, waiter->fl_link_cpu);
 	hlist_del_init(&waiter->fl_link);
-	spin_unlock(&file_lock_lock);
+	lg_local_unlock_cpu(&file_lock_lglock, waiter->fl_link_cpu);
 }
 
 /* Remove waiter from blocker's block list.
@@ -2228,6 +2234,11 @@ EXPORT_SYMBOL_GPL(vfs_cancel_lock);
 #include <linux/proc_fs.h>
 #include <linux/seq_file.h>
 
+struct locks_iterator {
+	int	li_cpu;
+	loff_t	li_pos;
+};
+
 static void lock_get_status(struct seq_file *f, struct file_lock *fl,
 			    loff_t id, char *pfx)
 {
@@ -2302,16 +2313,17 @@ static void lock_get_status(struct seq_file *f, struct file_lock *fl,
 static int locks_show(struct seq_file *f, void *v)
 {
 	int bkt;
+	struct locks_iterator *iter = f->private;
 	struct file_lock *fl, *bfl;
 
 	fl = hlist_entry(v, struct file_lock, fl_link);
 
-	lock_get_status(f, fl, *((loff_t *)f->private), "");
+	lock_get_status(f, fl, iter->li_pos, "");
 
 	spin_lock(&blocked_hash_lock);
 	hash_for_each(blocked_hash, bkt, bfl, fl_link) {
 		if (bfl->fl_next == fl)
-			lock_get_status(f, bfl, *((loff_t *)f->private), " ->");
+			lock_get_status(f, bfl, iter->li_pos, " ->");
 	}
 	spin_unlock(&blocked_hash_lock);
 
@@ -2320,23 +2332,24 @@ static int locks_show(struct seq_file *f, void *v)
 
 static void *locks_start(struct seq_file *f, loff_t *pos)
 {
-	loff_t *p = f->private;
+	struct locks_iterator *iter = f->private;
 
-	spin_lock(&file_lock_lock);
-	*p = (*pos + 1);
-	return seq_hlist_start(&file_lock_list, *pos);
+	iter->li_pos = *pos + 1;
+	lg_global_lock(&file_lock_lglock);
+	return seq_hlist_start_percpu(&file_lock_list, &iter->li_cpu, *pos);
 }
 
 static void *locks_next(struct seq_file *f, void *v, loff_t *pos)
 {
-	loff_t *p = f->private;
-	++*p;
-	return seq_hlist_next(v, &file_lock_list, pos);
+	struct locks_iterator *iter = f->private;
+
+	++iter->li_pos;
+	return seq_hlist_next_percpu(v, &file_lock_list, &iter->li_cpu, pos);
 }
 
 static void locks_stop(struct seq_file *f, void *v)
 {
-	spin_unlock(&file_lock_lock);
+	lg_global_unlock(&file_lock_lglock);
 }
 
 static const struct seq_operations locks_seq_operations = {
@@ -2348,7 +2361,8 @@ static const struct seq_operations locks_seq_operations = {
 
 static int locks_open(struct inode *inode, struct file *filp)
 {
-	return seq_open_private(filp, &locks_seq_operations, sizeof(loff_t));
+	return seq_open_private(filp, &locks_seq_operations,
+					sizeof(struct locks_iterator));
 }
 
 static const struct file_operations proc_locks_operations = {
@@ -2448,9 +2462,16 @@ EXPORT_SYMBOL(lock_may_write);
 
 static int __init filelock_init(void)
 {
+	int i;
+
 	filelock_cache = kmem_cache_create("file_lock_cache",
 			sizeof(struct file_lock), 0, SLAB_PANIC, NULL);
 
+	lg_lock_init(&file_lock_lglock, "file_lock_lglock");
+
+	for_each_possible_cpu(i)
+		INIT_HLIST_HEAD(per_cpu_ptr(&file_lock_list, i));
+
 	return 0;
 }
 
diff --git a/include/linux/fs.h b/include/linux/fs.h
index 232a345..18e59b8 100644
--- a/include/linux/fs.h
+++ b/include/linux/fs.h
@@ -953,6 +953,7 @@ struct file_lock {
 	unsigned int fl_flags;
 	unsigned char fl_type;
 	unsigned int fl_pid;
+	int fl_link_cpu;		/* what cpu's list is this on? */
 	struct pid *fl_nspid;
 	wait_queue_head_t fl_wait;
 	struct file *fl_file;
-- 
1.7.1



From jlayton at redhat.com  Tue Jun 11 07:09:02 2013
From: jlayton at redhat.com (Jeff Layton)
Date: Tue, 11 Jun 2013 07:09:02 -0400
Subject: [PATCH v2 08/14] locks: ensure that deadlock detection is atomic with
 respect to blocked_list modification
In-Reply-To: <1370948948-31784-1-git-send-email-jlayton@redhat.com>
References: <1370948948-31784-1-git-send-email-jlayton@redhat.com>
Message-ID: <1370948948-31784-9-git-send-email-jlayton@redhat.com>

Sound deadlock detection requires that we hold the file-lock state
steady while checking for them, and also ensure that updates to that
state are atomic with respect to those checks.

For the checking and insertion side, push the acquisition of the
global lock into __posix_lock_file and ensure that checking and update
of the global lists are done without dropping the lock in between.

On the removal side, when waking up blocked POSIX lock waiters, take
the global lock before walking the blocked list and dequeue the waiters
from the global list prior to removal from the i_flock list.

With this, deadlock detection should be race free while we minimize
excessive file_lock_lock thrashing.

Signed-off-by: Jeff Layton <jlayton at redhat.com>
---
 fs/locks.c |   71 +++++++++++++++++++++++++++++++++++++++++++----------------
 1 files changed, 52 insertions(+), 19 deletions(-)

diff --git a/fs/locks.c b/fs/locks.c
index d7342a3..b8cd1b1 100644
--- a/fs/locks.c
+++ b/fs/locks.c
@@ -475,16 +475,20 @@ static int posix_same_owner(struct file_lock *fl1, struct file_lock *fl2)
 static inline void
 locks_insert_global_blocked(struct file_lock *waiter)
 {
-	spin_lock(&file_lock_lock);
 	list_add(&waiter->fl_link, &blocked_list);
-	spin_unlock(&file_lock_lock);
+}
+
+static inline void
+__locks_delete_global_blocked(struct file_lock *waiter)
+{
+	list_del_init(&waiter->fl_link);
 }
 
 static inline void
 locks_delete_global_blocked(struct file_lock *waiter)
 {
 	spin_lock(&file_lock_lock);
-	list_del_init(&waiter->fl_link);
+	__locks_delete_global_blocked(waiter);
 	spin_unlock(&file_lock_lock);
 }
 
@@ -509,7 +513,6 @@ locks_delete_global_locks(struct file_lock *waiter)
  */
 static void __locks_delete_block(struct file_lock *waiter)
 {
-	locks_delete_global_blocked(waiter);
 	list_del_init(&waiter->fl_block);
 	waiter->fl_next = NULL;
 }
@@ -558,6 +561,30 @@ static void locks_wake_up_blocks(struct file_lock *blocker)
 	}
 }
 
+/*
+ * Wake up processes blocked waiting for blocker. In the FL_POSIX case, we must
+ * also take the global file_lock_lock and dequeue it from the global blocked
+ * list as we wake the processes.
+ *
+ * Must be called with the inode->i_lock of the blocker held!
+ */
+static void locks_wake_up_posix_blocks(struct file_lock *blocker)
+{
+	spin_lock(&file_lock_lock);
+	while (!list_empty(&blocker->fl_block)) {
+		struct file_lock *waiter;
+
+		waiter = list_first_entry(&blocker->fl_block,
+				struct file_lock, fl_block);
+		__locks_delete_global_blocked(waiter);
+		__locks_delete_block(waiter);
+		if (waiter->fl_lmops && waiter->fl_lmops->lm_notify)
+			waiter->fl_lmops->lm_notify(waiter);
+		else
+			wake_up(&waiter->fl_wait);
+	}
+	spin_unlock(&file_lock_lock);
+}
 /* Insert file lock fl into an inode's lock list at the position indicated
  * by pos. At the same time add the lock to the global file lock list.
  */
@@ -592,7 +619,11 @@ static void locks_delete_lock(struct file_lock **thisfl_p)
 		fl->fl_nspid = NULL;
 	}
 
-	locks_wake_up_blocks(fl);
+	if (IS_POSIX(fl))
+		locks_wake_up_posix_blocks(fl);
+	else
+		locks_wake_up_blocks(fl);
+
 	locks_free_lock(fl);
 }
 
@@ -705,6 +736,7 @@ static struct file_lock *what_owner_is_waiting_for(struct file_lock *block_fl)
 	return NULL;
 }
 
+/* Must be called with the file_lock_lock held! */
 static int posix_locks_deadlock(struct file_lock *caller_fl,
 				struct file_lock *block_fl)
 {
@@ -848,17 +880,13 @@ static int __posix_lock_file(struct inode *inode, struct file_lock *request, str
 			if (!(request->fl_flags & FL_SLEEP))
 				goto out;
 			error = -EDEADLK;
-			/*
-			 * XXX: potential race here. We should be adding the
-			 * file_lock to the global list before releasing lock.
-			 */
 			spin_lock(&file_lock_lock);
-			if (posix_locks_deadlock(request, fl))
-				goto out;
+			if (likely(!posix_locks_deadlock(request, fl))) {
+				error = FILE_LOCK_DEFERRED;
+				locks_insert_block(fl, request);
+				locks_insert_global_blocked(request);
+			}
 			spin_unlock(&file_lock_lock);
-			error = FILE_LOCK_DEFERRED;
-			locks_insert_block(fl, request);
-			locks_insert_global_blocked(request);
 			goto out;
   		}
   	}
@@ -949,7 +977,7 @@ static int __posix_lock_file(struct inode *inode, struct file_lock *request, str
 				 * as the change in lock type might satisfy
 				 * their needs.
 				 */
-				locks_wake_up_blocks(fl);
+				locks_wake_up_posix_blocks(fl);
 				fl->fl_start = request->fl_start;
 				fl->fl_end = request->fl_end;
 				fl->fl_type = request->fl_type;
@@ -1001,11 +1029,11 @@ static int __posix_lock_file(struct inode *inode, struct file_lock *request, str
 			locks_insert_lock(before, left);
 		}
 		right->fl_start = request->fl_end + 1;
-		locks_wake_up_blocks(right);
+		locks_wake_up_posix_blocks(right);
 	}
 	if (left) {
 		left->fl_end = request->fl_start - 1;
-		locks_wake_up_blocks(left);
+		locks_wake_up_posix_blocks(left);
 	}
  out:
 	spin_unlock(&inode->i_lock);
@@ -1061,6 +1089,7 @@ int posix_lock_file_wait(struct file *filp, struct file_lock *fl)
 		if (!error)
 			continue;
 
+		locks_delete_global_blocked(fl);
 		locks_delete_block(fl);
 		break;
 	}
@@ -1139,6 +1168,7 @@ int locks_mandatory_area(int read_write, struct inode *inode,
 				continue;
 		}
 
+		locks_delete_global_blocked(&fl);
 		locks_delete_block(&fl);
 		break;
 	}
@@ -1851,6 +1881,7 @@ static int do_lock_file_wait(struct file *filp, unsigned int cmd,
 		if (!error)
 			continue;
 
+		locks_delete_global_blocked(fl);
 		locks_delete_block(fl);
 		break;
 	}
@@ -2148,10 +2179,12 @@ posix_unblock_lock(struct file *filp, struct file_lock *waiter)
 	int status = 0;
 
 	spin_lock(&inode->i_lock);
-	if (waiter->fl_next)
+	if (waiter->fl_next) {
+		locks_delete_global_blocked(waiter);
 		__locks_delete_block(waiter);
-	else
+	} else {
 		status = -ENOENT;
+	}
 	spin_unlock(&inode->i_lock);
 	return status;
 }
-- 
1.7.1



From jlayton at redhat.com  Tue Jun 11 07:09:06 2013
From: jlayton at redhat.com (Jeff Layton)
Date: Tue, 11 Jun 2013 07:09:06 -0400
Subject: [PATCH v2 12/14] locks: give the blocked_hash its own spinlock
In-Reply-To: <1370948948-31784-1-git-send-email-jlayton@redhat.com>
References: <1370948948-31784-1-git-send-email-jlayton@redhat.com>
Message-ID: <1370948948-31784-13-git-send-email-jlayton@redhat.com>

There's no reason we have to protect the blocked_hash and file_lock_list
with the same spinlock. With the tests I have, breaking it in two gives
a barely measurable performance benefit, but it seems reasonable to make
this locking as granular as possible.

Signed-off-by: Jeff Layton <jlayton at redhat.com>
---
 Documentation/filesystems/Locking |   16 ++++++++--------
 fs/locks.c                        |   25 +++++++++++++------------
 2 files changed, 21 insertions(+), 20 deletions(-)

diff --git a/Documentation/filesystems/Locking b/Documentation/filesystems/Locking
index ee351ac..8d8d040 100644
--- a/Documentation/filesystems/Locking
+++ b/Documentation/filesystems/Locking
@@ -359,20 +359,20 @@ prototypes:
 
 locking rules:
 
-			inode->i_lock	file_lock_lock	may block
-lm_compare_owner:	yes		maybe		no
-lm_owner_key		yes		yes		no
-lm_notify:		yes		no		no
-lm_grant:		no		no		no
-lm_break:		yes		no		no
-lm_change		yes		no		no
+			inode->i_lock	blocked_hash_lock	may block
+lm_compare_owner:	yes		maybe			no
+lm_owner_key		yes		yes			no
+lm_notify:		yes		no			no
+lm_grant:		no		no			no
+lm_break:		yes		no			no
+lm_change		yes		no			no
 
 	->lm_compare_owner and ->lm_owner_key are generally called with
 *an* inode->i_lock held. It may not be the i_lock of the inode
 associated with either file_lock argument! This is the case with deadlock
 detection, since the code has to chase down the owners of locks that may
 be entirely unrelated to the one on which the lock is being acquired.
-For deadlock detection however, the file_lock_lock is also held. The
+For deadlock detection however, the blocked_hash_lock is also held. The
 fact that these locks are held ensures that the file_locks do not
 disappear out from under you while doing the comparison or generating an
 owner key.
diff --git a/fs/locks.c b/fs/locks.c
index 11e7784..8124fc1 100644
--- a/fs/locks.c
+++ b/fs/locks.c
@@ -162,12 +162,11 @@ int lease_break_time = 45;
  */
 #define BLOCKED_HASH_BITS	7
 
+static DEFINE_SPINLOCK(blocked_hash_lock);
 static DEFINE_HASHTABLE(blocked_hash, BLOCKED_HASH_BITS);
 
-static HLIST_HEAD(file_lock_list);
-
-/* Protects the file_lock_list and the blocked_hash */
 static DEFINE_SPINLOCK(file_lock_lock);
+static HLIST_HEAD(file_lock_list);
 
 static struct kmem_cache *filelock_cache __read_mostly;
 
@@ -505,9 +504,9 @@ __locks_delete_global_blocked(struct file_lock *waiter)
 static inline void
 locks_delete_global_blocked(struct file_lock *waiter)
 {
-	spin_lock(&file_lock_lock);
+	spin_lock(&blocked_hash_lock);
 	__locks_delete_global_blocked(waiter);
-	spin_unlock(&file_lock_lock);
+	spin_unlock(&blocked_hash_lock);
 }
 
 static inline void
@@ -581,14 +580,14 @@ static void locks_wake_up_blocks(struct file_lock *blocker)
 
 /*
  * Wake up processes blocked waiting for blocker. In the FL_POSIX case, we must
- * also take the global file_lock_lock and dequeue it from the global blocked
- * list as we wake the processes.
+ * also take the global blocked_hash_lock and dequeue it from the global
+ * blocked list as we wake the processes.
  *
  * Must be called with the inode->i_lock of the blocker held!
  */
 static void locks_wake_up_posix_blocks(struct file_lock *blocker)
 {
-	spin_lock(&file_lock_lock);
+	spin_lock(&blocked_hash_lock);
 	while (!list_empty(&blocker->fl_block)) {
 		struct file_lock *waiter;
 
@@ -601,7 +600,7 @@ static void locks_wake_up_posix_blocks(struct file_lock *blocker)
 		else
 			wake_up(&waiter->fl_wait);
 	}
-	spin_unlock(&file_lock_lock);
+	spin_unlock(&blocked_hash_lock);
 }
 /* Insert file lock fl into an inode's lock list at the position indicated
  * by pos. At the same time add the lock to the global file lock list.
@@ -754,7 +753,7 @@ static struct file_lock *what_owner_is_waiting_for(struct file_lock *block_fl)
 	return NULL;
 }
 
-/* Must be called with the file_lock_lock held! */
+/* Must be called with the blocked_hash_lock held! */
 static int posix_locks_deadlock(struct file_lock *caller_fl,
 				struct file_lock *block_fl)
 {
@@ -898,13 +897,13 @@ static int __posix_lock_file(struct inode *inode, struct file_lock *request, str
 			if (!(request->fl_flags & FL_SLEEP))
 				goto out;
 			error = -EDEADLK;
-			spin_lock(&file_lock_lock);
+			spin_lock(&blocked_hash_lock);
 			if (likely(!posix_locks_deadlock(request, fl))) {
 				error = FILE_LOCK_DEFERRED;
 				locks_insert_block(fl, request);
 				locks_insert_global_blocked(request);
 			}
-			spin_unlock(&file_lock_lock);
+			spin_unlock(&blocked_hash_lock);
 			goto out;
   		}
   	}
@@ -2309,10 +2308,12 @@ static int locks_show(struct seq_file *f, void *v)
 
 	lock_get_status(f, fl, *((loff_t *)f->private), "");
 
+	spin_lock(&blocked_hash_lock);
 	hash_for_each(blocked_hash, bkt, bfl, fl_link) {
 		if (bfl->fl_next == fl)
 			lock_get_status(f, bfl, *((loff_t *)f->private), " ->");
 	}
+	spin_unlock(&blocked_hash_lock);
 
 	return 0;
 }
-- 
1.7.1



From bfields at fieldses.org  Tue Jun 11 12:04:05 2013
From: bfields at fieldses.org (J. Bruce Fields)
Date: Tue, 11 Jun 2013 12:04:05 -0400
Subject: [PATCH v2 00/14] locks: scalability improvements for file locking
In-Reply-To: <1370948948-31784-1-git-send-email-jlayton@redhat.com>
References: <1370948948-31784-1-git-send-email-jlayton@redhat.com>
Message-ID: <20130611160405.GC8653@fieldses.org>

On Tue, Jun 11, 2013 at 07:08:54AM -0400, Jeff Layton wrote:
> Summary of Significant Changes:
> -------------------------------
> v2:
> - Fix potential races in deadlock detection. Manipulation of global
>   blocked_hash and deadlock detection are now atomic. This is a
>   little slower than the earlier set, but is provably correct. Also,
>   the patch that converts to using the i_lock has been split out from
>   most of the other changes. That should make it easier to review, but
>   it does leave a potential race in the deadlock detection that is fixed
>   up by the following patch. It may make sense to fold patches 7 and 8
>   together before merging.
> 
> - Add percpu hlists and lglocks for global file_lock_list. This gives
>   us some speedup since this list is seldom read.
> 
> Abstract (tl;dr version):
> -------------------------
> This patchset represents an overhaul of the file locking code with an
> aim toward improving its scalability and making the code a bit easier to
> understand.
> 
> Longer version:
> ---------------
> When the BKL was finally ripped out of the kernel in 2010, the strategy
> taken for the file locking code was to simply turn it into a new
> file_lock_locks spinlock. It was an expedient way to deal with the file
> locking code at the time, but having a giant spinlock around all of this
> code is clearly not great for scalability. Red Hat has bug reports that
> go back into the 2.6.18 era that point to BKL scalability problems in
> the file locking code and the file_lock_lock suffers from the same
> issues.
> 
> This patchset is my first attempt to make this code less dependent on
> global locking. The main change is to switch most of the file locking
> code to be protected by the inode->i_lock instead of the file_lock_lock.
> 
> While that works for most things, there are a couple of global data
> structures (lists in the current code) that need a global lock to
> protect them. So we still need a global lock in order to deal with
> those. The remaining patches are intended to make that global locking
> less painful. The big gainis are made by turning the blocked_list into a
> hashtable, which greatly speeds up the deadlock detection code.
> 
> This is not the first attempt at doing this. The conversion to the
> i_lock was originally attempted by Bruce Fields a few years ago. His
> approach was NAK'ed since it involved ripping out the deadlock
> detection. People also really seem to like /proc/locks for debugging, so
> keeping that in is probably worthwhile.
> 
> There's more work to be done in this area and this patchset is just a
> start. There's a horrible thundering herd problem when a blocking lock
> is released, for instance. There was also interest in solving the goofy
> "unlock on any close" POSIX lock semantics at this year's LSF. I think
> this patchset will help lay the groundwork for those changes as well.
> 
> While file locking is not usually considered to be a high-performance
> codepath, it *is* an IPC mechanism and I think it behooves us to try to
> make it as fast as possible.
> 
> I'd like to see this considered for 3.11. Comments and suggestions
> welcome.
> 
> Performance testing and results:
> --------------------------------
> In order to measure performance here, I've written some locking
> performance tests that I've made available here:
> 
>     git://git.samba.org/jlayton/lockperf.git
> 
> ...there are only 4 tests so far that measure performance of BSD and
> POSIX locking. I ran each test 100 times. The first number in each set
> of results is the unpatched (baseline) and the second is the patched
> results. The arch in both cases was x86_64. I've also done some testing
> on i686 and the results are more or less inline with the ones below.
> The files were all on tmpfs, to eliminate any storage-related latencies
> that might creep in.
> 
> Here's a description of each test with their results. I've also included
> the standard deviation with each:
> 
> posix01: fork off 128 children, and have them all open the same file and
> lock and unlock it 10k times (whole file, FL_WRLCK). Measure the time
> that it takes to perform all of the locks and unlocks in each process,
> and the total them all up. Note that the "mean time" here is not the
> time to run the test, but the total time that all the tasks spent
> locking and unlocking:

Why that and not the time to run the test?

Maybe what you're doing is right, but measuring total time seems more
realistic and simpler, and less likely to accumulate errors.

If we're measuring individual calls I think I'd rather see mean and
variance of the time of individual calls instead of of the sum.

It might also be interested to see how this changes as the number of
cores varies.  (Could you do that by pinning the test threads to some
subset of the 32 cores?)

Admittedly the difference seems clear enough already....  But it'd be
interesting to see how we scale with the new code in addition to just
seeing that's in an improvement.

(And, nit: round these numbers a little?:)

> 2-way Intel machine, 1G RAM, UMA:
> kernel				mean time		std. deviation
> -------------------------------------------------------------------------
> 3.10.0-rc4-00034-g042dd60	780.50511606027		20.0773085693901
> 3.10.0-rc4-00049-ge0b71e1	420.178537863755	24.4692621642102
> 
> 32-way AMD Opteron machine, 32G RAM in 4 NUMA nodes:
> kernel				mean time		std. deviation
> -------------------------------------------------------------------------
> 3.10.0-rc4-00034-g042dd60	30889.1603525361	267.767126085587
> 3.10.0-rc4-00049-ge0b71e1	25504.7748006928	336.232893246214
> 
> The first thing to note here is how painful this test is as the number
> of CPUs scales out, and when NUMA is thrown into the mix. It takes a
> staggeringly long time to run on such a machine. The patchset helps this
> use-case a lot, but it's still painful. I think these numbers help
> illustrate why we need to worry about this as the machines get more
> parallel.
> 
> posix02: fork off 128 children, and have them each open their own file
> and lock and unlock it 20k times. Measure the time that it takes to
> complete the locks and unlocks in each process and then total them up:
> 
> 2-way Intel machine, 1G RAM, UMA:
> kernel				mean time		std. deviation
> -------------------------------------------------------------------------
> 3.10.0-rc4-00034-g042dd60	319.86610077757		11.0827165395713
> 3.10.0-rc4-00049-ge0b71e1	257.24231139548		 8.0772950753018
> 
> 32-way AMD Opteron machine, 32G RAM in 4 NUMA nodes:
> kernel				mean time		std. deviation
> -------------------------------------------------------------------------
> 3.10.0-rc4-00034-g042dd60	1354.73702552958	40.5202787251742
> 3.10.0-rc4-00049-ge0b71e1	 541.35526474265	12.0492755786782
> 
> ...so again, multiple CPUs and NUMA take their toll on this workload,
> but the results are still universally positive from the patchset.
> 
> flock01: fork off 128 children, and have them all open the same file and
> lock (LOCK_EX) and unlock it 10k times. Measure the time that it takes
> to perform all of the locks and unlocks in each process, and the total
> them all up. Note that the "mean time" here is not the time to run the
> test, but the total time that all the tasks spent locking and unlocking:
> 
> 2-way Intel machine, 1G RAM, UMA:
> kernel				mean time		std. deviation
> -------------------------------------------------------------------------
> 3.10.0-rc4-00034-g042dd60	704.60957498018		10.6182455520549
> 3.10.0-rc4-00049-ge0b71e1	729.30782576048		 4.62185698266119
> 
> 32-way AMD Opteron machine, 32G RAM in 4 NUMA nodes:
> kernel				mean time		std. deviation
> -------------------------------------------------------------------------
> 3.10.0-rc4-00034-g042dd60	23617.876013741		221.445828515533
> 3.10.0-rc4-00049-ge0b71e1	24743.3849476248	340.664907591012
> 
> ...so here we see some slight slowdown (3-5%). In the case of the UMA
> box, it's on the order of the standard deviation, and in a similar test
> on a different box, the patched kernel turned out to be faster. Go
> figure...
> 
> In any case, I'm suspect that this is due to the fact that with this
> set, we need to take two spinlocks in order to do the work (the i_lock
> and the file_lock_lglock). Since the flock codepath doesn't use the
> global blocked_list, it sees no benefit to the conversion of it to a
> hash table.
> 
> flock02: fork off 128 children, and have them each open their own file
> and lock (LOCK_EX) and unlock it 20k times. Measure the time that it
> takes to complete the locks and unlocks in each process and then total
> them up:
> 
> 2-way Intel machine, 1G RAM, UMA:
> kernel				mean time		std. deviation
> -------------------------------------------------------------------------
> 3.10.0-rc4-00034-g042dd60	241.26319896997		7.69747220750147
> 3.10.0-rc4-00049-ge0b71e1	167.23062027739		5.1643997156006
> 
> 32-way AMD Opteron machine, 32G RAM in 4 NUMA nodes:
> kernel				mean time		std. deviation
> --------------------------------------------------------------------------
> 3.10.0-rc4-00034-g042dd60	1348.64594042422	39.5186585988528
> 3.10.0-rc4-00049-ge0b71e1	   8.4762987879		 0.347121187462215
> 
> ...the speedup here is really dramatic, especially in the NUMA case.
> This was so much faster that I had to double check that flock locking
> still worked properly afterward.

I guess deadlock detection is the most likely explanation of the
difference between the posix and flock cases?

If so that'd make a pretty strong case for eliminating (or somehow
greatly reducing) the deadlock detection.

With /proc/locks using a percpu lock and everything else a per-file
lock, in theory the deadlock detectionis the only thing that should
cause contention between cores, so the test should scale perfectly in
the flock case.  (Might be interesting to confirm that by looking at how
it changes over # of cores.)

--b.

> 
> Jeff Layton (14):
>   cifs: use posix_unblock_lock instead of locks_delete_block
>   locks: make generic_add_lease and generic_delete_lease static
>   locks: comment cleanups and clarifications
>   locks: make "added" in __posix_lock_file a bool
>   locks: encapsulate the fl_link list handling
>   locks: don't walk inode->i_flock list in locks_show
>   locks: convert to i_lock to protect i_flock list
>   locks: ensure that deadlock detection is atomic with respect to
>     blocked_list modification
>   locks: convert fl_link to a hlist_node
>   locks: turn the blocked_list into a hashtable
>   locks: add a new "lm_owner_key" lock operation
>   locks: give the blocked_hash its own spinlock
>   seq_file: add seq_list_*_percpu helpers
>   locks: move file_lock_list to a set of percpu hlist_heads and convert
>     file_lock_lock to an lglock
> 
>  Documentation/filesystems/Locking |   27 +++-
>  fs/afs/flock.c                    |    5 +-
>  fs/ceph/locks.c                   |    2 +-
>  fs/ceph/mds_client.c              |    8 +-
>  fs/cifs/cifsfs.c                  |    2 +-
>  fs/cifs/file.c                    |   15 +-
>  fs/gfs2/file.c                    |    2 +-
>  fs/lockd/svclock.c                |   12 ++
>  fs/lockd/svcsubs.c                |   12 +-
>  fs/locks.c                        |  306 +++++++++++++++++++++++++------------
>  fs/nfs/delegation.c               |   11 +-
>  fs/nfs/nfs4state.c                |    8 +-
>  fs/nfsd/nfs4state.c               |    8 +-
>  fs/seq_file.c                     |   54 +++++++
>  include/linux/fs.h                |   38 +++--
>  include/linux/seq_file.h          |    6 +
>  16 files changed, 362 insertions(+), 154 deletions(-)
> 


From jlayton at redhat.com  Tue Jun 11 12:35:52 2013
From: jlayton at redhat.com (Jeff Layton)
Date: Tue, 11 Jun 2013 12:35:52 -0400
Subject: [PATCH v2 00/14] locks: scalability improvements for file locking
In-Reply-To: <20130611160405.GC8653@fieldses.org>
References: <1370948948-31784-1-git-send-email-jlayton@redhat.com>
 <20130611160405.GC8653@fieldses.org>
Message-ID: <20130611123552.12743dd7@tlielax.poochiereds.net>

On Tue, 11 Jun 2013 12:04:05 -0400
"J. Bruce Fields" <bfields at fieldses.org> wrote:

> On Tue, Jun 11, 2013 at 07:08:54AM -0400, Jeff Layton wrote:
> > Summary of Significant Changes:
> > -------------------------------
> > v2:
> > - Fix potential races in deadlock detection. Manipulation of global
> >   blocked_hash and deadlock detection are now atomic. This is a
> >   little slower than the earlier set, but is provably correct. Also,
> >   the patch that converts to using the i_lock has been split out from
> >   most of the other changes. That should make it easier to review, but
> >   it does leave a potential race in the deadlock detection that is fixed
> >   up by the following patch. It may make sense to fold patches 7 and 8
> >   together before merging.
> > 
> > - Add percpu hlists and lglocks for global file_lock_list. This gives
> >   us some speedup since this list is seldom read.
> > 
> > Abstract (tl;dr version):
> > -------------------------
> > This patchset represents an overhaul of the file locking code with an
> > aim toward improving its scalability and making the code a bit easier to
> > understand.
> > 
> > Longer version:
> > ---------------
> > When the BKL was finally ripped out of the kernel in 2010, the strategy
> > taken for the file locking code was to simply turn it into a new
> > file_lock_locks spinlock. It was an expedient way to deal with the file
> > locking code at the time, but having a giant spinlock around all of this
> > code is clearly not great for scalability. Red Hat has bug reports that
> > go back into the 2.6.18 era that point to BKL scalability problems in
> > the file locking code and the file_lock_lock suffers from the same
> > issues.
> > 
> > This patchset is my first attempt to make this code less dependent on
> > global locking. The main change is to switch most of the file locking
> > code to be protected by the inode->i_lock instead of the file_lock_lock.
> > 
> > While that works for most things, there are a couple of global data
> > structures (lists in the current code) that need a global lock to
> > protect them. So we still need a global lock in order to deal with
> > those. The remaining patches are intended to make that global locking
> > less painful. The big gainis are made by turning the blocked_list into a
> > hashtable, which greatly speeds up the deadlock detection code.
> > 
> > This is not the first attempt at doing this. The conversion to the
> > i_lock was originally attempted by Bruce Fields a few years ago. His
> > approach was NAK'ed since it involved ripping out the deadlock
> > detection. People also really seem to like /proc/locks for debugging, so
> > keeping that in is probably worthwhile.
> > 
> > There's more work to be done in this area and this patchset is just a
> > start. There's a horrible thundering herd problem when a blocking lock
> > is released, for instance. There was also interest in solving the goofy
> > "unlock on any close" POSIX lock semantics at this year's LSF. I think
> > this patchset will help lay the groundwork for those changes as well.
> > 
> > While file locking is not usually considered to be a high-performance
> > codepath, it *is* an IPC mechanism and I think it behooves us to try to
> > make it as fast as possible.
> > 
> > I'd like to see this considered for 3.11. Comments and suggestions
> > welcome.
> > 
> > Performance testing and results:
> > --------------------------------
> > In order to measure performance here, I've written some locking
> > performance tests that I've made available here:
> > 
> >     git://git.samba.org/jlayton/lockperf.git
> > 
> > ...there are only 4 tests so far that measure performance of BSD and
> > POSIX locking. I ran each test 100 times. The first number in each set
> > of results is the unpatched (baseline) and the second is the patched
> > results. The arch in both cases was x86_64. I've also done some testing
> > on i686 and the results are more or less inline with the ones below.
> > The files were all on tmpfs, to eliminate any storage-related latencies
> > that might creep in.
> > 
> > Here's a description of each test with their results. I've also included
> > the standard deviation with each:
> > 
> > posix01: fork off 128 children, and have them all open the same file and
> > lock and unlock it 10k times (whole file, FL_WRLCK). Measure the time
> > that it takes to perform all of the locks and unlocks in each process,
> > and the total them all up. Note that the "mean time" here is not the
> > time to run the test, but the total time that all the tasks spent
> > locking and unlocking:
> 
> Why that and not the time to run the test?
> 

Yeah, the fact that there is time spent both sleeping to wait for the
lock as well as time spent processing when you're awake makes this
difficult to quantify.

I was really trying to eliminate measuring anything *but* the locking
performance. In most cases, the open/close/fork overhead is pretty
insignificant, but by measuring it this way, I know they don't matter.

I did look at the times to run the test and in general, the time to run
the test is smaller when the numbers above are smaller.

> Maybe what you're doing is right, but measuring total time seems more
> realistic and simpler, and less likely to accumulate errors.
> 
> If we're measuring individual calls I think I'd rather see mean and
> variance of the time of individual calls instead of of the sum.
> 
> It might also be interested to see how this changes as the number of
> cores varies.  (Could you do that by pinning the test threads to some
> subset of the 32 cores?)
> 
> Admittedly the difference seems clear enough already....  But it'd be
> interesting to see how we scale with the new code in addition to just
> seeing that's in an improvement.
> 
> (And, nit: round these numbers a little?:)
> 

I'll see...getting these numbers took long enough. I'm not sure I
really want to spend that much of my life running these tests and
collecting results. ;)

If I get the time, I'll do so but I'd also like to start work on some
patches to help the thundering herd problem when a lock is released.
I'm fairly convinced at this point that these patches do help
performance overall.

And yeah, I just cut and pasted the numbers from my simple perl script
to gather these stats, they obviously could use some rounding...

> > 2-way Intel machine, 1G RAM, UMA:
> > kernel				mean time		std. deviation
> > -------------------------------------------------------------------------
> > 3.10.0-rc4-00034-g042dd60	780.50511606027		20.0773085693901
> > 3.10.0-rc4-00049-ge0b71e1	420.178537863755	24.4692621642102
> > 
> > 32-way AMD Opteron machine, 32G RAM in 4 NUMA nodes:
> > kernel				mean time		std. deviation
> > -------------------------------------------------------------------------
> > 3.10.0-rc4-00034-g042dd60	30889.1603525361	267.767126085587
> > 3.10.0-rc4-00049-ge0b71e1	25504.7748006928	336.232893246214
> > 
> > The first thing to note here is how painful this test is as the number
> > of CPUs scales out, and when NUMA is thrown into the mix. It takes a
> > staggeringly long time to run on such a machine. The patchset helps this
> > use-case a lot, but it's still painful. I think these numbers help
> > illustrate why we need to worry about this as the machines get more
> > parallel.
> > 
> > posix02: fork off 128 children, and have them each open their own file
> > and lock and unlock it 20k times. Measure the time that it takes to
> > complete the locks and unlocks in each process and then total them up:
> > 
> > 2-way Intel machine, 1G RAM, UMA:
> > kernel				mean time		std. deviation
> > -------------------------------------------------------------------------
> > 3.10.0-rc4-00034-g042dd60	319.86610077757		11.0827165395713
> > 3.10.0-rc4-00049-ge0b71e1	257.24231139548		 8.0772950753018
> > 
> > 32-way AMD Opteron machine, 32G RAM in 4 NUMA nodes:
> > kernel				mean time		std. deviation
> > -------------------------------------------------------------------------
> > 3.10.0-rc4-00034-g042dd60	1354.73702552958	40.5202787251742
> > 3.10.0-rc4-00049-ge0b71e1	 541.35526474265	12.0492755786782
> > 
> > ...so again, multiple CPUs and NUMA take their toll on this workload,
> > but the results are still universally positive from the patchset.
> > 
> > flock01: fork off 128 children, and have them all open the same file and
> > lock (LOCK_EX) and unlock it 10k times. Measure the time that it takes
> > to perform all of the locks and unlocks in each process, and the total
> > them all up. Note that the "mean time" here is not the time to run the
> > test, but the total time that all the tasks spent locking and unlocking:
> > 
> > 2-way Intel machine, 1G RAM, UMA:
> > kernel				mean time		std. deviation
> > -------------------------------------------------------------------------
> > 3.10.0-rc4-00034-g042dd60	704.60957498018		10.6182455520549
> > 3.10.0-rc4-00049-ge0b71e1	729.30782576048		 4.62185698266119
> > 
> > 32-way AMD Opteron machine, 32G RAM in 4 NUMA nodes:
> > kernel				mean time		std. deviation
> > -------------------------------------------------------------------------
> > 3.10.0-rc4-00034-g042dd60	23617.876013741		221.445828515533
> > 3.10.0-rc4-00049-ge0b71e1	24743.3849476248	340.664907591012
> > 
> > ...so here we see some slight slowdown (3-5%). In the case of the UMA
> > box, it's on the order of the standard deviation, and in a similar test
> > on a different box, the patched kernel turned out to be faster. Go
> > figure...
> > 
> > In any case, I'm suspect that this is due to the fact that with this
> > set, we need to take two spinlocks in order to do the work (the i_lock
> > and the file_lock_lglock). Since the flock codepath doesn't use the
> > global blocked_list, it sees no benefit to the conversion of it to a
> > hash table.
> > 
> > flock02: fork off 128 children, and have them each open their own file
> > and lock (LOCK_EX) and unlock it 20k times. Measure the time that it
> > takes to complete the locks and unlocks in each process and then total
> > them up:
> > 
> > 2-way Intel machine, 1G RAM, UMA:
> > kernel				mean time		std. deviation
> > -------------------------------------------------------------------------
> > 3.10.0-rc4-00034-g042dd60	241.26319896997		7.69747220750147
> > 3.10.0-rc4-00049-ge0b71e1	167.23062027739		5.1643997156006
> > 
> > 32-way AMD Opteron machine, 32G RAM in 4 NUMA nodes:
> > kernel				mean time		std. deviation
> > --------------------------------------------------------------------------
> > 3.10.0-rc4-00034-g042dd60	1348.64594042422	39.5186585988528
> > 3.10.0-rc4-00049-ge0b71e1	   8.4762987879		 0.347121187462215
> > 
> > ...the speedup here is really dramatic, especially in the NUMA case.
> > This was so much faster that I had to double check that flock locking
> > still worked properly afterward.
> 
> I guess deadlock detection is the most likely explanation of the
> difference between the posix and flock cases?
> 
> If so that'd make a pretty strong case for eliminating (or somehow
> greatly reducing) the deadlock detection.
> 

Well yes, the deadlock detection itself and the addition/removal from
the blocked_hash/blocked_list. The conversion to a hashtable help the
deadlock detection itself, but we can't really do much about the list
manipulation.

I'm not opposed to some sort of CONFIG_* option to allow compiling out
the deadlock detection, but I'd probably prefer to leave that for
another set. It might also make sense to have some sort of boot-time
switch to enable it?

For a distro like Fedora or RHEL, I could envision turning off deadlock
detection in "production" kernels and leaving it enabled in -debug
kernels. I have my doubts as to how useful deadlock detection really is
though...

> With /proc/locks using a percpu lock and everything else a per-file
> lock, in theory the deadlock detectionis the only thing that should
> cause contention between cores, so the test should scale perfectly in
> the flock case.  (Might be interesting to confirm that by looking at how
> it changes over # of cores.)

Sure, if you assume that the locks are uncontended. You can still have
contention for the i_lock of course if tasks running on different CPUs
are trying to lock the same file.

Also the file_lock_lock is a percpu lglock, but occasionally tasks
acquire a lock on one CPU and then get migrated to another. At that
point you have to "reach across" to another CPU to take the entry off
the percpu hlists.

But yeah...for the most part these changes make the code have very
little contention, and the flock02 test results seem to really bear
that out.

Thanks for taking a look!
-- 
Jeff Layton <jlayton at redhat.com>


From bfields at fieldses.org  Thu Jun 13 10:41:40 2013
From: bfields at fieldses.org (J. Bruce Fields)
Date: Thu, 13 Jun 2013 10:41:40 -0400
Subject: [PATCH v2 07/14] locks: convert to i_lock to protect i_flock list
In-Reply-To: <1370948948-31784-8-git-send-email-jlayton@redhat.com>
References: <1370948948-31784-1-git-send-email-jlayton@redhat.com>
 <1370948948-31784-8-git-send-email-jlayton@redhat.com>
Message-ID: <20130613144140.GA20235@fieldses.org>

On Tue, Jun 11, 2013 at 07:09:01AM -0400, Jeff Layton wrote:
> Having a global lock that protects all of this code is a clear
> scalability problem. Instead of doing that, move most of the code to be
> protected by the i_lock instead.
> 
> The exceptions are the global lists that file_lock->fl_link sits on.
> Those still need a global lock of some sort, so wrap just those accesses
> in the file_lock_lock().
> 
> Note that this patch introduces a potential race in the deadlock
> detection code which could result in either false positives or missed
> file_lock loops. The blocked_list management needs to be done atomically
> with the deadlock detection. This will be fixed in a later patch.

Actually, the way I think I'd try this:

	1. Introduce a new spinlock global_lock_lists_lock and take it
	   in all the places we need a global lock.
	2. Do the big flock_lock-to-i_lock conversion, deleting flock_lock
	   in the process.

Does that work?

Yeah, more patch-ordering bike-shedding, but...  I think that would be
simple and bisectable and for a confusing long-untouched bit of code
it's worth getting these little steps right if we can.

--b.

> 
> Signed-off-by: Jeff Layton <jlayton at redhat.com>
> Signed-off-by: J. Bruce Fields <bfields at redhat.com>
> ---
>  Documentation/filesystems/Locking |   23 +++++--
>  fs/afs/flock.c                    |    5 +-
>  fs/ceph/locks.c                   |    2 +-
>  fs/ceph/mds_client.c              |    8 +-
>  fs/cifs/cifsfs.c                  |    2 +-
>  fs/cifs/file.c                    |   13 ++--
>  fs/gfs2/file.c                    |    2 +-
>  fs/lockd/svcsubs.c                |   12 ++--
>  fs/locks.c                        |  110 ++++++++++++++++++++-----------------
>  fs/nfs/delegation.c               |   11 ++--
>  fs/nfs/nfs4state.c                |    8 +-
>  fs/nfsd/nfs4state.c               |    8 +-
>  include/linux/fs.h                |   11 ----
>  13 files changed, 113 insertions(+), 102 deletions(-)
> 
> diff --git a/Documentation/filesystems/Locking b/Documentation/filesystems/Locking
> index 0706d32..13f91ab 100644
> --- a/Documentation/filesystems/Locking
> +++ b/Documentation/filesystems/Locking
> @@ -344,7 +344,7 @@ prototypes:
>  
>  
>  locking rules:
> -			file_lock_lock	may block
> +			inode->i_lock	may block
>  fl_copy_lock:		yes		no
>  fl_release_private:	maybe		no
>  
> @@ -357,12 +357,21 @@ prototypes:
>  	int (*lm_change)(struct file_lock **, int);
>  
>  locking rules:
> -			file_lock_lock	may block
> -lm_compare_owner:	yes		no
> -lm_notify:		yes		no
> -lm_grant:		no		no
> -lm_break:		yes		no
> -lm_change		yes		no
> +
> +			inode->i_lock	file_lock_lock	may block
> +lm_compare_owner:	yes		maybe		no
> +lm_notify:		yes		no		no
> +lm_grant:		no		no		no
> +lm_break:		yes		no		no
> +lm_change		yes		no		no
> +
> +	->lm_compare_owner is generally called with *an* inode->i_lock
> +held. It may not be the i_lock of the inode for either file_lock being
> +compared! This is the case with deadlock detection, since the code has
> +to chase down the owners of locks that may be entirely unrelated to the
> +one on which the lock is being acquired. For deadlock detection however,
> +the file_lock_lock is also held. The locks primarily ensure that neither
> +file_lock disappear out from under you while doing the comparison.
>  
>  --------------------------- buffer_head -----------------------------------
>  prototypes:
> diff --git a/fs/afs/flock.c b/fs/afs/flock.c
> index 2497bf3..03fc0d1 100644
> --- a/fs/afs/flock.c
> +++ b/fs/afs/flock.c
> @@ -252,6 +252,7 @@ static void afs_defer_unlock(struct afs_vnode *vnode, struct key *key)
>   */
>  static int afs_do_setlk(struct file *file, struct file_lock *fl)
>  {
> +	struct inode = file_inode(file);
>  	struct afs_vnode *vnode = AFS_FS_I(file->f_mapping->host);
>  	afs_lock_type_t type;
>  	struct key *key = file->private_data;
> @@ -273,7 +274,7 @@ static int afs_do_setlk(struct file *file, struct file_lock *fl)
>  
>  	type = (fl->fl_type == F_RDLCK) ? AFS_LOCK_READ : AFS_LOCK_WRITE;
>  
> -	lock_flocks();
> +	spin_lock(&inode->i_lock);
>  
>  	/* make sure we've got a callback on this file and that our view of the
>  	 * data version is up to date */
> @@ -420,7 +421,7 @@ given_lock:
>  	afs_vnode_fetch_status(vnode, NULL, key);
>  
>  error:
> -	unlock_flocks();
> +	spin_unlock(&inode->i_lock);
>  	_leave(" = %d", ret);
>  	return ret;
>  
> diff --git a/fs/ceph/locks.c b/fs/ceph/locks.c
> index 202dd3d..cd0a664 100644
> --- a/fs/ceph/locks.c
> +++ b/fs/ceph/locks.c
> @@ -194,7 +194,7 @@ void ceph_count_locks(struct inode *inode, int *fcntl_count, int *flock_count)
>   * Encode the flock and fcntl locks for the given inode into the pagelist.
>   * Format is: #fcntl locks, sequential fcntl locks, #flock locks,
>   * sequential flock locks.
> - * Must be called with lock_flocks() already held.
> + * Must be called with inode->i_lock already held.
>   * If we encounter more of a specific lock type than expected,
>   * we return the value 1.
>   */
> diff --git a/fs/ceph/mds_client.c b/fs/ceph/mds_client.c
> index 4f22671..ae621b5 100644
> --- a/fs/ceph/mds_client.c
> +++ b/fs/ceph/mds_client.c
> @@ -2482,13 +2482,13 @@ static int encode_caps_cb(struct inode *inode, struct ceph_cap *cap,
>  
>  		ceph_pagelist_set_cursor(pagelist, &trunc_point);
>  		do {
> -			lock_flocks();
> +			spin_lock(&inode->i_lock);
>  			ceph_count_locks(inode, &num_fcntl_locks,
>  					 &num_flock_locks);
>  			rec.v2.flock_len = (2*sizeof(u32) +
>  					    (num_fcntl_locks+num_flock_locks) *
>  					    sizeof(struct ceph_filelock));
> -			unlock_flocks();
> +			spin_unlock(&inode->i_lock);
>  
>  			/* pre-alloc pagelist */
>  			ceph_pagelist_truncate(pagelist, &trunc_point);
> @@ -2499,12 +2499,12 @@ static int encode_caps_cb(struct inode *inode, struct ceph_cap *cap,
>  
>  			/* encode locks */
>  			if (!err) {
> -				lock_flocks();
> +				spin_lock(&inode->i_lock);
>  				err = ceph_encode_locks(inode,
>  							pagelist,
>  							num_fcntl_locks,
>  							num_flock_locks);
> -				unlock_flocks();
> +				spin_unlock(&inode->i_lock);
>  			}
>  		} while (err == -ENOSPC);
>  	} else {
> diff --git a/fs/cifs/cifsfs.c b/fs/cifs/cifsfs.c
> index 3752b9f..e81655c 100644
> --- a/fs/cifs/cifsfs.c
> +++ b/fs/cifs/cifsfs.c
> @@ -765,7 +765,7 @@ static loff_t cifs_llseek(struct file *file, loff_t offset, int whence)
>  
>  static int cifs_setlease(struct file *file, long arg, struct file_lock **lease)
>  {
> -	/* note that this is called by vfs setlease with lock_flocks held
> +	/* note that this is called by vfs setlease with i_lock held
>  	   to protect *lease from going away */
>  	struct inode *inode = file_inode(file);
>  	struct cifsFileInfo *cfile = file->private_data;
> diff --git a/fs/cifs/file.c b/fs/cifs/file.c
> index 44a4f18..0dd10cd 100644
> --- a/fs/cifs/file.c
> +++ b/fs/cifs/file.c
> @@ -1092,6 +1092,7 @@ struct lock_to_push {
>  static int
>  cifs_push_posix_locks(struct cifsFileInfo *cfile)
>  {
> +	struct inode *inode = cfile->dentry->d_inode;
>  	struct cifs_tcon *tcon = tlink_tcon(cfile->tlink);
>  	struct file_lock *flock, **before;
>  	unsigned int count = 0, i = 0;
> @@ -1102,12 +1103,12 @@ cifs_push_posix_locks(struct cifsFileInfo *cfile)
>  
>  	xid = get_xid();
>  
> -	lock_flocks();
> -	cifs_for_each_lock(cfile->dentry->d_inode, before) {
> +	spin_lock(&inode->i_lock);
> +	cifs_for_each_lock(inode, before) {
>  		if ((*before)->fl_flags & FL_POSIX)
>  			count++;
>  	}
> -	unlock_flocks();
> +	spin_unlock(&inode->i_lock);
>  
>  	INIT_LIST_HEAD(&locks_to_send);
>  
> @@ -1126,8 +1127,8 @@ cifs_push_posix_locks(struct cifsFileInfo *cfile)
>  	}
>  
>  	el = locks_to_send.next;
> -	lock_flocks();
> -	cifs_for_each_lock(cfile->dentry->d_inode, before) {
> +	spin_lock(&inode->i_lock);
> +	cifs_for_each_lock(inode, before) {
>  		flock = *before;
>  		if ((flock->fl_flags & FL_POSIX) == 0)
>  			continue;
> @@ -1152,7 +1153,7 @@ cifs_push_posix_locks(struct cifsFileInfo *cfile)
>  		lck->offset = flock->fl_start;
>  		el = el->next;
>  	}
> -	unlock_flocks();
> +	spin_unlock(&inode->i_lock);
>  
>  	list_for_each_entry_safe(lck, tmp, &locks_to_send, llist) {
>  		int stored_rc;
> diff --git a/fs/gfs2/file.c b/fs/gfs2/file.c
> index acd1676..9e634e0 100644
> --- a/fs/gfs2/file.c
> +++ b/fs/gfs2/file.c
> @@ -889,7 +889,7 @@ out_uninit:
>   * cluster; until we do, disable leases (by just returning -EINVAL),
>   * unless the administrator has requested purely local locking.
>   *
> - * Locking: called under lock_flocks
> + * Locking: called under i_lock
>   *
>   * Returns: errno
>   */
> diff --git a/fs/lockd/svcsubs.c b/fs/lockd/svcsubs.c
> index 97e8741..dc5c759 100644
> --- a/fs/lockd/svcsubs.c
> +++ b/fs/lockd/svcsubs.c
> @@ -169,7 +169,7 @@ nlm_traverse_locks(struct nlm_host *host, struct nlm_file *file,
>  
>  again:
>  	file->f_locks = 0;
> -	lock_flocks(); /* protects i_flock list */
> +	spin_lock(&inode->i_lock);
>  	for (fl = inode->i_flock; fl; fl = fl->fl_next) {
>  		if (fl->fl_lmops != &nlmsvc_lock_operations)
>  			continue;
> @@ -181,7 +181,7 @@ again:
>  		if (match(lockhost, host)) {
>  			struct file_lock lock = *fl;
>  
> -			unlock_flocks();
> +			spin_unlock(&inode->i_lock);
>  			lock.fl_type  = F_UNLCK;
>  			lock.fl_start = 0;
>  			lock.fl_end   = OFFSET_MAX;
> @@ -193,7 +193,7 @@ again:
>  			goto again;
>  		}
>  	}
> -	unlock_flocks();
> +	spin_unlock(&inode->i_lock);
>  
>  	return 0;
>  }
> @@ -228,14 +228,14 @@ nlm_file_inuse(struct nlm_file *file)
>  	if (file->f_count || !list_empty(&file->f_blocks) || file->f_shares)
>  		return 1;
>  
> -	lock_flocks();
> +	spin_lock(&inode->i_lock);
>  	for (fl = inode->i_flock; fl; fl = fl->fl_next) {
>  		if (fl->fl_lmops == &nlmsvc_lock_operations) {
> -			unlock_flocks();
> +			spin_unlock(&inode->i_lock);
>  			return 1;
>  		}
>  	}
> -	unlock_flocks();
> +	spin_unlock(&inode->i_lock);
>  	file->f_locks = 0;
>  	return 0;
>  }
> diff --git a/fs/locks.c b/fs/locks.c
> index 3fd27f0..d7342a3 100644
> --- a/fs/locks.c
> +++ b/fs/locks.c
> @@ -155,22 +155,9 @@ int lease_break_time = 45;
>  
>  static LIST_HEAD(file_lock_list);
>  static LIST_HEAD(blocked_list);
> -static DEFINE_SPINLOCK(file_lock_lock);
> -
> -/*
> - * Protects the two list heads above, plus the inode->i_flock list
> - */
> -void lock_flocks(void)
> -{
> -	spin_lock(&file_lock_lock);
> -}
> -EXPORT_SYMBOL_GPL(lock_flocks);
>  
> -void unlock_flocks(void)
> -{
> -	spin_unlock(&file_lock_lock);
> -}
> -EXPORT_SYMBOL_GPL(unlock_flocks);
> +/* Protects the two list heads above */
> +static DEFINE_SPINLOCK(file_lock_lock);
>  
>  static struct kmem_cache *filelock_cache __read_mostly;
>  
> @@ -488,25 +475,33 @@ static int posix_same_owner(struct file_lock *fl1, struct file_lock *fl2)
>  static inline void
>  locks_insert_global_blocked(struct file_lock *waiter)
>  {
> +	spin_lock(&file_lock_lock);
>  	list_add(&waiter->fl_link, &blocked_list);
> +	spin_unlock(&file_lock_lock);
>  }
>  
>  static inline void
>  locks_delete_global_blocked(struct file_lock *waiter)
>  {
> +	spin_lock(&file_lock_lock);
>  	list_del_init(&waiter->fl_link);
> +	spin_unlock(&file_lock_lock);
>  }
>  
>  static inline void
>  locks_insert_global_locks(struct file_lock *waiter)
>  {
> +	spin_lock(&file_lock_lock);
>  	list_add_tail(&waiter->fl_link, &file_lock_list);
> +	spin_unlock(&file_lock_lock);
>  }
>  
>  static inline void
>  locks_delete_global_locks(struct file_lock *waiter)
>  {
> +	spin_lock(&file_lock_lock);
>  	list_del_init(&waiter->fl_link);
> +	spin_unlock(&file_lock_lock);
>  }
>  
>  /* Remove waiter from blocker's block list.
> @@ -523,9 +518,11 @@ static void __locks_delete_block(struct file_lock *waiter)
>   */
>  static void locks_delete_block(struct file_lock *waiter)
>  {
> -	lock_flocks();
> +	struct inode *inode = file_inode(waiter->fl_file);
> +
> +	spin_lock(&inode->i_lock);
>  	__locks_delete_block(waiter);
> -	unlock_flocks();
> +	spin_unlock(&inode->i_lock);
>  }
>  
>  /* Insert waiter into blocker's block list.
> @@ -544,7 +541,7 @@ static void locks_insert_block(struct file_lock *blocker,
>  /*
>   * Wake up processes blocked waiting for blocker.
>   *
> - * Must be called with the file_lock_lock held!
> + * Must be called with the inode->i_lock of the blocker held!
>   */
>  static void locks_wake_up_blocks(struct file_lock *blocker)
>  {
> @@ -649,8 +646,9 @@ void
>  posix_test_lock(struct file *filp, struct file_lock *fl)
>  {
>  	struct file_lock *cfl;
> +	struct inode *inode = file_inode(filp);
>  
> -	lock_flocks();
> +	spin_lock(&inode->i_lock);
>  	for (cfl = file_inode(filp)->i_flock; cfl; cfl = cfl->fl_next) {
>  		if (!IS_POSIX(cfl))
>  			continue;
> @@ -663,7 +661,7 @@ posix_test_lock(struct file *filp, struct file_lock *fl)
>  			fl->fl_pid = pid_vnr(cfl->fl_nspid);
>  	} else
>  		fl->fl_type = F_UNLCK;
> -	unlock_flocks();
> +	spin_unlock(&inode->i_lock);
>  	return;
>  }
>  EXPORT_SYMBOL(posix_test_lock);
> @@ -742,7 +740,7 @@ static int flock_lock_file(struct file *filp, struct file_lock *request)
>  			return -ENOMEM;
>  	}
>  
> -	lock_flocks();
> +	spin_lock(&inode->i_lock);
>  	if (request->fl_flags & FL_ACCESS)
>  		goto find_conflict;
>  
> @@ -772,9 +770,9 @@ static int flock_lock_file(struct file *filp, struct file_lock *request)
>  	 * give it the opportunity to lock the file.
>  	 */
>  	if (found) {
> -		unlock_flocks();
> +		spin_unlock(&inode->i_lock);
>  		cond_resched();
> -		lock_flocks();
> +		spin_lock(&inode->i_lock);
>  	}
>  
>  find_conflict:
> @@ -801,7 +799,7 @@ find_conflict:
>  	error = 0;
>  
>  out:
> -	unlock_flocks();
> +	spin_unlock(&inode->i_lock);
>  	if (new_fl)
>  		locks_free_lock(new_fl);
>  	return error;
> @@ -831,7 +829,7 @@ static int __posix_lock_file(struct inode *inode, struct file_lock *request, str
>  		new_fl2 = locks_alloc_lock();
>  	}
>  
> -	lock_flocks();
> +	spin_lock(&inode->i_lock);
>  	/*
>  	 * New lock request. Walk all POSIX locks and look for conflicts. If
>  	 * there are any, either return error or put the request on the
> @@ -850,8 +848,14 @@ static int __posix_lock_file(struct inode *inode, struct file_lock *request, str
>  			if (!(request->fl_flags & FL_SLEEP))
>  				goto out;
>  			error = -EDEADLK;
> +			/*
> +			 * XXX: potential race here. We should be adding the
> +			 * file_lock to the global list before releasing lock.
> +			 */
> +			spin_lock(&file_lock_lock);
>  			if (posix_locks_deadlock(request, fl))
>  				goto out;
> +			spin_unlock(&file_lock_lock);
>  			error = FILE_LOCK_DEFERRED;
>  			locks_insert_block(fl, request);
>  			locks_insert_global_blocked(request);
> @@ -1004,7 +1008,7 @@ static int __posix_lock_file(struct inode *inode, struct file_lock *request, str
>  		locks_wake_up_blocks(left);
>  	}
>   out:
> -	unlock_flocks();
> +	spin_unlock(&inode->i_lock);
>  	/*
>  	 * Free any unused locks.
>  	 */
> @@ -1079,14 +1083,14 @@ int locks_mandatory_locked(struct inode *inode)
>  	/*
>  	 * Search the lock list for this inode for any POSIX locks.
>  	 */
> -	lock_flocks();
> +	spin_lock(&inode->i_lock);
>  	for (fl = inode->i_flock; fl != NULL; fl = fl->fl_next) {
>  		if (!IS_POSIX(fl))
>  			continue;
>  		if (fl->fl_owner != owner)
>  			break;
>  	}
> -	unlock_flocks();
> +	spin_unlock(&inode->i_lock);
>  	return fl ? -EAGAIN : 0;
>  }
>  
> @@ -1229,7 +1233,7 @@ int __break_lease(struct inode *inode, unsigned int mode)
>  	if (IS_ERR(new_fl))
>  		return PTR_ERR(new_fl);
>  
> -	lock_flocks();
> +	spin_lock(&inode->i_lock);
>  
>  	time_out_leases(inode);
>  
> @@ -1279,10 +1283,10 @@ restart:
>  			break_time++;
>  	}
>  	locks_insert_block(flock, new_fl);
> -	unlock_flocks();
> +	spin_unlock(&inode->i_lock);
>  	error = wait_event_interruptible_timeout(new_fl->fl_wait,
>  						!new_fl->fl_next, break_time);
> -	lock_flocks();
> +	spin_lock(&inode->i_lock);
>  	__locks_delete_block(new_fl);
>  	if (error >= 0) {
>  		if (error == 0)
> @@ -1300,7 +1304,7 @@ restart:
>  	}
>  
>  out:
> -	unlock_flocks();
> +	spin_unlock(&inode->i_lock);
>  	locks_free_lock(new_fl);
>  	return error;
>  }
> @@ -1353,9 +1357,10 @@ EXPORT_SYMBOL(lease_get_mtime);
>  int fcntl_getlease(struct file *filp)
>  {
>  	struct file_lock *fl;
> +	struct inode *inode = file_inode(filp);
>  	int type = F_UNLCK;
>  
> -	lock_flocks();
> +	spin_lock(&inode->i_lock);
>  	time_out_leases(file_inode(filp));
>  	for (fl = file_inode(filp)->i_flock; fl && IS_LEASE(fl);
>  			fl = fl->fl_next) {
> @@ -1364,7 +1369,7 @@ int fcntl_getlease(struct file *filp)
>  			break;
>  		}
>  	}
> -	unlock_flocks();
> +	spin_unlock(&inode->i_lock);
>  	return type;
>  }
>  
> @@ -1458,7 +1463,7 @@ static int generic_delete_lease(struct file *filp, struct file_lock **flp)
>   *	The (input) flp->fl_lmops->lm_break function is required
>   *	by break_lease().
>   *
> - *	Called with file_lock_lock held.
> + *	Called with inode->i_lock held.
>   */
>  int generic_setlease(struct file *filp, long arg, struct file_lock **flp)
>  {
> @@ -1527,11 +1532,12 @@ static int __vfs_setlease(struct file *filp, long arg, struct file_lock **lease)
>  
>  int vfs_setlease(struct file *filp, long arg, struct file_lock **lease)
>  {
> +	struct inode *inode = file_inode(filp);
>  	int error;
>  
> -	lock_flocks();
> +	spin_lock(&inode->i_lock);
>  	error = __vfs_setlease(filp, arg, lease);
> -	unlock_flocks();
> +	spin_unlock(&inode->i_lock);
>  
>  	return error;
>  }
> @@ -1549,6 +1555,7 @@ static int do_fcntl_delete_lease(struct file *filp)
>  static int do_fcntl_add_lease(unsigned int fd, struct file *filp, long arg)
>  {
>  	struct file_lock *fl, *ret;
> +	struct inode *inode = file_inode(filp);
>  	struct fasync_struct *new;
>  	int error;
>  
> @@ -1562,10 +1569,10 @@ static int do_fcntl_add_lease(unsigned int fd, struct file *filp, long arg)
>  		return -ENOMEM;
>  	}
>  	ret = fl;
> -	lock_flocks();
> +	spin_lock(&inode->i_lock);
>  	error = __vfs_setlease(filp, arg, &ret);
>  	if (error) {
> -		unlock_flocks();
> +		spin_unlock(&inode->i_lock);
>  		locks_free_lock(fl);
>  		goto out_free_fasync;
>  	}
> @@ -1582,7 +1589,7 @@ static int do_fcntl_add_lease(unsigned int fd, struct file *filp, long arg)
>  		new = NULL;
>  
>  	error = __f_setown(filp, task_pid(current), PIDTYPE_PID, 0);
> -	unlock_flocks();
> +	spin_unlock(&inode->i_lock);
>  
>  out_free_fasync:
>  	if (new)
> @@ -2106,7 +2113,7 @@ void locks_remove_flock(struct file *filp)
>  			fl.fl_ops->fl_release_private(&fl);
>  	}
>  
> -	lock_flocks();
> +	spin_lock(&inode->i_lock);
>  	before = &inode->i_flock;
>  
>  	while ((fl = *before) != NULL) {
> @@ -2124,7 +2131,7 @@ void locks_remove_flock(struct file *filp)
>   		}
>  		before = &fl->fl_next;
>  	}
> -	unlock_flocks();
> +	spin_unlock(&inode->i_lock);
>  }
>  
>  /**
> @@ -2137,14 +2144,15 @@ void locks_remove_flock(struct file *filp)
>  int
>  posix_unblock_lock(struct file *filp, struct file_lock *waiter)
>  {
> +	struct inode *inode = file_inode(filp);
>  	int status = 0;
>  
> -	lock_flocks();
> +	spin_lock(&inode->i_lock);
>  	if (waiter->fl_next)
>  		__locks_delete_block(waiter);
>  	else
>  		status = -ENOENT;
> -	unlock_flocks();
> +	spin_unlock(&inode->i_lock);
>  	return status;
>  }
>  
> @@ -2261,7 +2269,7 @@ static void *locks_start(struct seq_file *f, loff_t *pos)
>  {
>  	loff_t *p = f->private;
>  
> -	lock_flocks();
> +	spin_lock(&file_lock_lock);
>  	*p = (*pos + 1);
>  	return seq_list_start(&file_lock_list, *pos);
>  }
> @@ -2275,7 +2283,7 @@ static void *locks_next(struct seq_file *f, void *v, loff_t *pos)
>  
>  static void locks_stop(struct seq_file *f, void *v)
>  {
> -	unlock_flocks();
> +	spin_unlock(&file_lock_lock);
>  }
>  
>  static const struct seq_operations locks_seq_operations = {
> @@ -2322,7 +2330,8 @@ int lock_may_read(struct inode *inode, loff_t start, unsigned long len)
>  {
>  	struct file_lock *fl;
>  	int result = 1;
> -	lock_flocks();
> +
> +	spin_lock(&inode->i_lock);
>  	for (fl = inode->i_flock; fl != NULL; fl = fl->fl_next) {
>  		if (IS_POSIX(fl)) {
>  			if (fl->fl_type == F_RDLCK)
> @@ -2339,7 +2348,7 @@ int lock_may_read(struct inode *inode, loff_t start, unsigned long len)
>  		result = 0;
>  		break;
>  	}
> -	unlock_flocks();
> +	spin_unlock(&inode->i_lock);
>  	return result;
>  }
>  
> @@ -2362,7 +2371,8 @@ int lock_may_write(struct inode *inode, loff_t start, unsigned long len)
>  {
>  	struct file_lock *fl;
>  	int result = 1;
> -	lock_flocks();
> +
> +	spin_lock(&inode->i_lock);
>  	for (fl = inode->i_flock; fl != NULL; fl = fl->fl_next) {
>  		if (IS_POSIX(fl)) {
>  			if ((fl->fl_end < start) || (fl->fl_start > (start + len)))
> @@ -2377,7 +2387,7 @@ int lock_may_write(struct inode *inode, loff_t start, unsigned long len)
>  		result = 0;
>  		break;
>  	}
> -	unlock_flocks();
> +	spin_unlock(&inode->i_lock);
>  	return result;
>  }
>  
> diff --git a/fs/nfs/delegation.c b/fs/nfs/delegation.c
> index 57db324..43ee7f9 100644
> --- a/fs/nfs/delegation.c
> +++ b/fs/nfs/delegation.c
> @@ -73,20 +73,21 @@ static int nfs_delegation_claim_locks(struct nfs_open_context *ctx, struct nfs4_
>  	if (inode->i_flock == NULL)
>  		goto out;
>  
> -	/* Protect inode->i_flock using the file locks lock */
> -	lock_flocks();
> +	/* Protect inode->i_flock using the i_lock */
> +	spin_lock(&inode->i_lock);
>  	for (fl = inode->i_flock; fl != NULL; fl = fl->fl_next) {
>  		if (!(fl->fl_flags & (FL_POSIX|FL_FLOCK)))
>  			continue;
>  		if (nfs_file_open_context(fl->fl_file) != ctx)
>  			continue;
> -		unlock_flocks();
> +		/* FIXME: safe to drop lock here while walking list? */
> +		spin_unlock(&inode->i_lock);
>  		status = nfs4_lock_delegation_recall(fl, state, stateid);
>  		if (status < 0)
>  			goto out;
> -		lock_flocks();
> +		spin_lock(&inode->i_lock);
>  	}
> -	unlock_flocks();
> +	spin_unlock(&inode->i_lock);
>  out:
>  	return status;
>  }
> diff --git a/fs/nfs/nfs4state.c b/fs/nfs/nfs4state.c
> index 1fab140..ff10b4a 100644
> --- a/fs/nfs/nfs4state.c
> +++ b/fs/nfs/nfs4state.c
> @@ -1373,13 +1373,13 @@ static int nfs4_reclaim_locks(struct nfs4_state *state, const struct nfs4_state_
>  	/* Guard against delegation returns and new lock/unlock calls */
>  	down_write(&nfsi->rwsem);
>  	/* Protect inode->i_flock using the BKL */
> -	lock_flocks();
> +	spin_lock(&inode->i_lock);
>  	for (fl = inode->i_flock; fl != NULL; fl = fl->fl_next) {
>  		if (!(fl->fl_flags & (FL_POSIX|FL_FLOCK)))
>  			continue;
>  		if (nfs_file_open_context(fl->fl_file)->state != state)
>  			continue;
> -		unlock_flocks();
> +		spin_unlock(&inode->i_lock);
>  		status = ops->recover_lock(state, fl);
>  		switch (status) {
>  			case 0:
> @@ -1406,9 +1406,9 @@ static int nfs4_reclaim_locks(struct nfs4_state *state, const struct nfs4_state_
>  				/* kill_proc(fl->fl_pid, SIGLOST, 1); */
>  				status = 0;
>  		}
> -		lock_flocks();
> +		spin_lock(&inode->i_lock);
>  	}
> -	unlock_flocks();
> +	spin_unlock(&inode->i_lock);
>  out:
>  	up_write(&nfsi->rwsem);
>  	return status;
> diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
> index 316ec84..f170518 100644
> --- a/fs/nfsd/nfs4state.c
> +++ b/fs/nfsd/nfs4state.c
> @@ -2645,13 +2645,13 @@ static void nfsd_break_one_deleg(struct nfs4_delegation *dp)
>  
>  	list_add_tail(&dp->dl_recall_lru, &nn->del_recall_lru);
>  
> -	/* only place dl_time is set. protected by lock_flocks*/
> +	/* Only place dl_time is set; protected by i_lock: */
>  	dp->dl_time = get_seconds();
>  
>  	nfsd4_cb_recall(dp);
>  }
>  
> -/* Called from break_lease() with lock_flocks() held. */
> +/* Called from break_lease() with i_lock held. */
>  static void nfsd_break_deleg_cb(struct file_lock *fl)
>  {
>  	struct nfs4_file *fp = (struct nfs4_file *)fl->fl_owner;
> @@ -4520,7 +4520,7 @@ check_for_locks(struct nfs4_file *filp, struct nfs4_lockowner *lowner)
>  	struct inode *inode = filp->fi_inode;
>  	int status = 0;
>  
> -	lock_flocks();
> +	spin_lock(&inode->i_lock);
>  	for (flpp = &inode->i_flock; *flpp != NULL; flpp = &(*flpp)->fl_next) {
>  		if ((*flpp)->fl_owner == (fl_owner_t)lowner) {
>  			status = 1;
> @@ -4528,7 +4528,7 @@ check_for_locks(struct nfs4_file *filp, struct nfs4_lockowner *lowner)
>  		}
>  	}
>  out:
> -	unlock_flocks();
> +	spin_unlock(&inode->i_lock);
>  	return status;
>  }
>  
> diff --git a/include/linux/fs.h b/include/linux/fs.h
> index 94105d2..e2f896d 100644
> --- a/include/linux/fs.h
> +++ b/include/linux/fs.h
> @@ -1024,8 +1024,6 @@ extern int vfs_setlease(struct file *, long, struct file_lock **);
>  extern int lease_modify(struct file_lock **, int);
>  extern int lock_may_read(struct inode *, loff_t start, unsigned long count);
>  extern int lock_may_write(struct inode *, loff_t start, unsigned long count);
> -extern void lock_flocks(void);
> -extern void unlock_flocks(void);
>  #else /* !CONFIG_FILE_LOCKING */
>  static inline int fcntl_getlk(struct file *file, struct flock __user *user)
>  {
> @@ -1167,15 +1165,6 @@ static inline int lock_may_write(struct inode *inode, loff_t start,
>  {
>  	return 1;
>  }
> -
> -static inline void lock_flocks(void)
> -{
> -}
> -
> -static inline void unlock_flocks(void)
> -{
> -}
> -
>  #endif /* !CONFIG_FILE_LOCKING */
>  
>  
> -- 
> 1.7.1
> 


From bfields at fieldses.org  Thu Jun 13 10:50:09 2013
From: bfields at fieldses.org (J. Bruce Fields)
Date: Thu, 13 Jun 2013 10:50:09 -0400
Subject: [PATCH v2 10/14] locks: turn the blocked_list into a hashtable
In-Reply-To: <1370948948-31784-11-git-send-email-jlayton@redhat.com>
References: <1370948948-31784-1-git-send-email-jlayton@redhat.com>
 <1370948948-31784-11-git-send-email-jlayton@redhat.com>
Message-ID: <20130613145009.GB20235@fieldses.org>

On Tue, Jun 11, 2013 at 07:09:04AM -0400, Jeff Layton wrote:
> Break up the blocked_list into a hashtable, using the fl_owner as a key.
> This speeds up searching the hash chains, which is especially significant
> for deadlock detection.
> 
> Note that the initial implementation assumes that hashing on fl_owner is
> sufficient. In most cases it should be, with the notable exception being
> server-side lockd, which compares ownership using a tuple of the
> nlm_host and the pid sent in the lock request. So, this may degrade to a
> single hash bucket when you only have a single NFS client. That will be
> addressed in a later patch.
> 
> The careful observer may note that this patch leaves the file_lock_list
> alone. There's much less of a case for turning the file_lock_list into a
> hashtable. The only user of that list is the code that generates
> /proc/locks, and it always walks the entire list.

Makes sense to me, ACK to this and the previous patch.--b.

> 
> Signed-off-by: Jeff Layton <jlayton at redhat.com>
> ---
>  fs/locks.c |   25 ++++++++++++++++++-------
>  1 files changed, 18 insertions(+), 7 deletions(-)
> 
> diff --git a/fs/locks.c b/fs/locks.c
> index 28959bc..76fb7af 100644
> --- a/fs/locks.c
> +++ b/fs/locks.c
> @@ -126,6 +126,7 @@
>  #include <linux/time.h>
>  #include <linux/rcupdate.h>
>  #include <linux/pid_namespace.h>
> +#include <linux/hashtable.h>
>  
>  #include <asm/uaccess.h>
>  
> @@ -153,10 +154,19 @@ int lease_break_time = 45;
>  #define for_each_lock(inode, lockp) \
>  	for (lockp = &inode->i_flock; *lockp != NULL; lockp = &(*lockp)->fl_next)
>  
> +/*
> + * By breaking up the blocked locks list into a hashtable, we speed up the
> + * deadlock detection.
> + *
> + * FIXME: make this value scale via some heuristic?
> + */
> +#define BLOCKED_HASH_BITS	7
> +
> +static DEFINE_HASHTABLE(blocked_hash, BLOCKED_HASH_BITS);
> +
>  static HLIST_HEAD(file_lock_list);
> -static HLIST_HEAD(blocked_list);
>  
> -/* Protects the two list heads above */
> +/* Protects the file_lock_list and the blocked_hash */
>  static DEFINE_SPINLOCK(file_lock_lock);
>  
>  static struct kmem_cache *filelock_cache __read_mostly;
> @@ -475,13 +485,13 @@ static int posix_same_owner(struct file_lock *fl1, struct file_lock *fl2)
>  static inline void
>  locks_insert_global_blocked(struct file_lock *waiter)
>  {
> -	hlist_add_head(&waiter->fl_link, &blocked_list);
> +	hash_add(blocked_hash, &waiter->fl_link, (unsigned long)waiter->fl_owner);
>  }
>  
>  static inline void
>  __locks_delete_global_blocked(struct file_lock *waiter)
>  {
> -	hlist_del_init(&waiter->fl_link);
> +	hash_del(&waiter->fl_link);
>  }
>  
>  static inline void
> @@ -729,7 +739,7 @@ static struct file_lock *what_owner_is_waiting_for(struct file_lock *block_fl)
>  {
>  	struct file_lock *fl;
>  
> -	hlist_for_each_entry(fl, &blocked_list, fl_link) {
> +	hash_for_each_possible(blocked_hash, fl, fl_link, (unsigned long)block_fl->fl_owner) {
>  		if (posix_same_owner(fl, block_fl))
>  			return fl->fl_next;
>  	}
> @@ -865,7 +875,7 @@ static int __posix_lock_file(struct inode *inode, struct file_lock *request, str
>  	/*
>  	 * New lock request. Walk all POSIX locks and look for conflicts. If
>  	 * there are any, either return error or put the request on the
> -	 * blocker's list of waiters and the global blocked_list.
> +	 * blocker's list of waiters and the global blocked_hash.
>  	 */
>  	if (request->fl_type != F_UNLCK) {
>  		for_each_lock(inode, before) {
> @@ -2284,13 +2294,14 @@ static void lock_get_status(struct seq_file *f, struct file_lock *fl,
>  
>  static int locks_show(struct seq_file *f, void *v)
>  {
> +	int bkt;
>  	struct file_lock *fl, *bfl;
>  
>  	fl = hlist_entry(v, struct file_lock, fl_link);
>  
>  	lock_get_status(f, fl, *((loff_t *)f->private), "");
>  
> -	hlist_for_each_entry(bfl, &blocked_list, fl_link) {
> +	hash_for_each(blocked_hash, bkt, bfl, fl_link) {
>  		if (bfl->fl_next == fl)
>  			lock_get_status(f, bfl, *((loff_t *)f->private), " ->");
>  	}
> -- 
> 1.7.1
> 


From bfields at fieldses.org  Thu Jun 13 11:00:33 2013
From: bfields at fieldses.org (J. Bruce Fields)
Date: Thu, 13 Jun 2013 11:00:33 -0400
Subject: [PATCH v2 11/14] locks: add a new "lm_owner_key" lock operation
In-Reply-To: <1370948948-31784-12-git-send-email-jlayton@redhat.com>
References: <1370948948-31784-1-git-send-email-jlayton@redhat.com>
 <1370948948-31784-12-git-send-email-jlayton@redhat.com>
Message-ID: <20130613150033.GA20666@fieldses.org>

On Tue, Jun 11, 2013 at 07:09:05AM -0400, Jeff Layton wrote:
> Currently, the hashing that the locking code uses to add these values
> to the blocked_hash is simply calculated using fl_owner field. That's
> valid in most cases except for server-side lockd, which validates the
> owner of a lock based on fl_owner and fl_pid.
> 
> In the case where you have a small number of NFS clients doing a lot
> of locking between different processes, you could end up with all
> the blocked requests sitting in a very small number of hash buckets.
> 
> Add a new lm_owner_key operation to the lock_manager_operations that
> will generate an unsigned long to use as the key in the hashtable.
> That function is only implemented for server-side lockd, and simply
> XORs the fl_owner and fl_pid.

Like I've said I think we should look into defining a lock_owner struct
that lockd can allocate as necessary so that the lock code can just do a
pointer comparison on struct lock_owner *'s.  But maybe that doesn't
work out and in any case it can be future work, so looks fine, ACK.

--b.

> 
> Signed-off-by: Jeff Layton <jlayton at redhat.com>
> ---
>  Documentation/filesystems/Locking |   18 +++++++++++-------
>  fs/lockd/svclock.c                |   12 ++++++++++++
>  fs/locks.c                        |   12 ++++++++++--
>  include/linux/fs.h                |    1 +
>  4 files changed, 34 insertions(+), 9 deletions(-)
> 
> diff --git a/Documentation/filesystems/Locking b/Documentation/filesystems/Locking
> index 13f91ab..ee351ac 100644
> --- a/Documentation/filesystems/Locking
> +++ b/Documentation/filesystems/Locking
> @@ -351,6 +351,7 @@ fl_release_private:	maybe		no
>  ----------------------- lock_manager_operations ---------------------------
>  prototypes:
>  	int (*lm_compare_owner)(struct file_lock *, struct file_lock *);
> +	unsigned long (*lm_owner_key)(struct file_lock *);
>  	void (*lm_notify)(struct file_lock *);  /* unblock callback */
>  	int (*lm_grant)(struct file_lock *, struct file_lock *, int);
>  	void (*lm_break)(struct file_lock *); /* break_lease callback */
> @@ -360,18 +361,21 @@ locking rules:
>  
>  			inode->i_lock	file_lock_lock	may block
>  lm_compare_owner:	yes		maybe		no
> +lm_owner_key		yes		yes		no
>  lm_notify:		yes		no		no
>  lm_grant:		no		no		no
>  lm_break:		yes		no		no
>  lm_change		yes		no		no
>  
> -	->lm_compare_owner is generally called with *an* inode->i_lock
> -held. It may not be the i_lock of the inode for either file_lock being
> -compared! This is the case with deadlock detection, since the code has
> -to chase down the owners of locks that may be entirely unrelated to the
> -one on which the lock is being acquired. For deadlock detection however,
> -the file_lock_lock is also held. The locks primarily ensure that neither
> -file_lock disappear out from under you while doing the comparison.
> +	->lm_compare_owner and ->lm_owner_key are generally called with
> +*an* inode->i_lock held. It may not be the i_lock of the inode
> +associated with either file_lock argument! This is the case with deadlock
> +detection, since the code has to chase down the owners of locks that may
> +be entirely unrelated to the one on which the lock is being acquired.
> +For deadlock detection however, the file_lock_lock is also held. The
> +fact that these locks are held ensures that the file_locks do not
> +disappear out from under you while doing the comparison or generating an
> +owner key.
>  
>  --------------------------- buffer_head -----------------------------------
>  prototypes:
> diff --git a/fs/lockd/svclock.c b/fs/lockd/svclock.c
> index e703318..ce2cdab 100644
> --- a/fs/lockd/svclock.c
> +++ b/fs/lockd/svclock.c
> @@ -744,8 +744,20 @@ static int nlmsvc_same_owner(struct file_lock *fl1, struct file_lock *fl2)
>  	return fl1->fl_owner == fl2->fl_owner && fl1->fl_pid == fl2->fl_pid;
>  }
>  
> +/*
> + * Since NLM uses two "keys" for tracking locks, we need to hash them down
> + * to one for the blocked_hash. Here, we're just xor'ing the host address
> + * with the pid in order to create a key value for picking a hash bucket.
> + */
> +static unsigned long
> +nlmsvc_owner_key(struct file_lock *fl)
> +{
> +	return (unsigned long)fl->fl_owner ^ (unsigned long)fl->fl_pid;
> +}
> +
>  const struct lock_manager_operations nlmsvc_lock_operations = {
>  	.lm_compare_owner = nlmsvc_same_owner,
> +	.lm_owner_key = nlmsvc_owner_key,
>  	.lm_notify = nlmsvc_notify_blocked,
>  	.lm_grant = nlmsvc_grant_deferred,
>  };
> diff --git a/fs/locks.c b/fs/locks.c
> index 76fb7af..11e7784 100644
> --- a/fs/locks.c
> +++ b/fs/locks.c
> @@ -481,11 +481,19 @@ static int posix_same_owner(struct file_lock *fl1, struct file_lock *fl2)
>  	return fl1->fl_owner == fl2->fl_owner;
>  }
>  
> +static unsigned long
> +posix_owner_key(struct file_lock *fl)
> +{
> +	if (fl->fl_lmops && fl->fl_lmops->lm_owner_key)
> +		return fl->fl_lmops->lm_owner_key(fl);
> +	return (unsigned long)fl->fl_owner;
> +}
> +
>  /* Remove a blocker or lock from one of the global lists */
>  static inline void
>  locks_insert_global_blocked(struct file_lock *waiter)
>  {
> -	hash_add(blocked_hash, &waiter->fl_link, (unsigned long)waiter->fl_owner);
> +	hash_add(blocked_hash, &waiter->fl_link, posix_owner_key(waiter));
>  }
>  
>  static inline void
> @@ -739,7 +747,7 @@ static struct file_lock *what_owner_is_waiting_for(struct file_lock *block_fl)
>  {
>  	struct file_lock *fl;
>  
> -	hash_for_each_possible(blocked_hash, fl, fl_link, (unsigned long)block_fl->fl_owner) {
> +	hash_for_each_possible(blocked_hash, fl, fl_link, posix_owner_key(block_fl)) {
>  		if (posix_same_owner(fl, block_fl))
>  			return fl->fl_next;
>  	}
> diff --git a/include/linux/fs.h b/include/linux/fs.h
> index 3b340f7..232a345 100644
> --- a/include/linux/fs.h
> +++ b/include/linux/fs.h
> @@ -908,6 +908,7 @@ struct file_lock_operations {
>  
>  struct lock_manager_operations {
>  	int (*lm_compare_owner)(struct file_lock *, struct file_lock *);
> +	unsigned long (*lm_owner_key)(struct file_lock *);
>  	void (*lm_notify)(struct file_lock *);	/* unblock callback */
>  	int (*lm_grant)(struct file_lock *, struct file_lock *, int);
>  	void (*lm_break)(struct file_lock *);
> -- 
> 1.7.1
> 


From bfields at fieldses.org  Thu Jun 13 11:02:47 2013
From: bfields at fieldses.org (J. Bruce Fields)
Date: Thu, 13 Jun 2013 11:02:47 -0400
Subject: [PATCH v2 12/14] locks: give the blocked_hash its own spinlock
In-Reply-To: <1370948948-31784-13-git-send-email-jlayton@redhat.com>
References: <1370948948-31784-1-git-send-email-jlayton@redhat.com>
 <1370948948-31784-13-git-send-email-jlayton@redhat.com>
Message-ID: <20130613150247.GB20666@fieldses.org>

On Tue, Jun 11, 2013 at 07:09:06AM -0400, Jeff Layton wrote:
> There's no reason we have to protect the blocked_hash and file_lock_list
> with the same spinlock. With the tests I have, breaking it in two gives
> a barely measurable performance benefit, but it seems reasonable to make
> this locking as granular as possible.

Out of curiosity...  In the typical case when adding/removing a lock,
aren't both lists being modified in rapid succession?

I wonder if it would be better to instead stick with one lock and take
care to acquire it only once to cover both manipulations.

--b.

> 
> Signed-off-by: Jeff Layton <jlayton at redhat.com>
> ---
>  Documentation/filesystems/Locking |   16 ++++++++--------
>  fs/locks.c                        |   25 +++++++++++++------------
>  2 files changed, 21 insertions(+), 20 deletions(-)
> 
> diff --git a/Documentation/filesystems/Locking b/Documentation/filesystems/Locking
> index ee351ac..8d8d040 100644
> --- a/Documentation/filesystems/Locking
> +++ b/Documentation/filesystems/Locking
> @@ -359,20 +359,20 @@ prototypes:
>  
>  locking rules:
>  
> -			inode->i_lock	file_lock_lock	may block
> -lm_compare_owner:	yes		maybe		no
> -lm_owner_key		yes		yes		no
> -lm_notify:		yes		no		no
> -lm_grant:		no		no		no
> -lm_break:		yes		no		no
> -lm_change		yes		no		no
> +			inode->i_lock	blocked_hash_lock	may block
> +lm_compare_owner:	yes		maybe			no
> +lm_owner_key		yes		yes			no
> +lm_notify:		yes		no			no
> +lm_grant:		no		no			no
> +lm_break:		yes		no			no
> +lm_change		yes		no			no
>  
>  	->lm_compare_owner and ->lm_owner_key are generally called with
>  *an* inode->i_lock held. It may not be the i_lock of the inode
>  associated with either file_lock argument! This is the case with deadlock
>  detection, since the code has to chase down the owners of locks that may
>  be entirely unrelated to the one on which the lock is being acquired.
> -For deadlock detection however, the file_lock_lock is also held. The
> +For deadlock detection however, the blocked_hash_lock is also held. The
>  fact that these locks are held ensures that the file_locks do not
>  disappear out from under you while doing the comparison or generating an
>  owner key.
> diff --git a/fs/locks.c b/fs/locks.c
> index 11e7784..8124fc1 100644
> --- a/fs/locks.c
> +++ b/fs/locks.c
> @@ -162,12 +162,11 @@ int lease_break_time = 45;
>   */
>  #define BLOCKED_HASH_BITS	7
>  
> +static DEFINE_SPINLOCK(blocked_hash_lock);
>  static DEFINE_HASHTABLE(blocked_hash, BLOCKED_HASH_BITS);
>  
> -static HLIST_HEAD(file_lock_list);
> -
> -/* Protects the file_lock_list and the blocked_hash */
>  static DEFINE_SPINLOCK(file_lock_lock);
> +static HLIST_HEAD(file_lock_list);
>  
>  static struct kmem_cache *filelock_cache __read_mostly;
>  
> @@ -505,9 +504,9 @@ __locks_delete_global_blocked(struct file_lock *waiter)
>  static inline void
>  locks_delete_global_blocked(struct file_lock *waiter)
>  {
> -	spin_lock(&file_lock_lock);
> +	spin_lock(&blocked_hash_lock);
>  	__locks_delete_global_blocked(waiter);
> -	spin_unlock(&file_lock_lock);
> +	spin_unlock(&blocked_hash_lock);
>  }
>  
>  static inline void
> @@ -581,14 +580,14 @@ static void locks_wake_up_blocks(struct file_lock *blocker)
>  
>  /*
>   * Wake up processes blocked waiting for blocker. In the FL_POSIX case, we must
> - * also take the global file_lock_lock and dequeue it from the global blocked
> - * list as we wake the processes.
> + * also take the global blocked_hash_lock and dequeue it from the global
> + * blocked list as we wake the processes.
>   *
>   * Must be called with the inode->i_lock of the blocker held!
>   */
>  static void locks_wake_up_posix_blocks(struct file_lock *blocker)
>  {
> -	spin_lock(&file_lock_lock);
> +	spin_lock(&blocked_hash_lock);
>  	while (!list_empty(&blocker->fl_block)) {
>  		struct file_lock *waiter;
>  
> @@ -601,7 +600,7 @@ static void locks_wake_up_posix_blocks(struct file_lock *blocker)
>  		else
>  			wake_up(&waiter->fl_wait);
>  	}
> -	spin_unlock(&file_lock_lock);
> +	spin_unlock(&blocked_hash_lock);
>  }
>  /* Insert file lock fl into an inode's lock list at the position indicated
>   * by pos. At the same time add the lock to the global file lock list.
> @@ -754,7 +753,7 @@ static struct file_lock *what_owner_is_waiting_for(struct file_lock *block_fl)
>  	return NULL;
>  }
>  
> -/* Must be called with the file_lock_lock held! */
> +/* Must be called with the blocked_hash_lock held! */
>  static int posix_locks_deadlock(struct file_lock *caller_fl,
>  				struct file_lock *block_fl)
>  {
> @@ -898,13 +897,13 @@ static int __posix_lock_file(struct inode *inode, struct file_lock *request, str
>  			if (!(request->fl_flags & FL_SLEEP))
>  				goto out;
>  			error = -EDEADLK;
> -			spin_lock(&file_lock_lock);
> +			spin_lock(&blocked_hash_lock);
>  			if (likely(!posix_locks_deadlock(request, fl))) {
>  				error = FILE_LOCK_DEFERRED;
>  				locks_insert_block(fl, request);
>  				locks_insert_global_blocked(request);
>  			}
> -			spin_unlock(&file_lock_lock);
> +			spin_unlock(&blocked_hash_lock);
>  			goto out;
>    		}
>    	}
> @@ -2309,10 +2308,12 @@ static int locks_show(struct seq_file *f, void *v)
>  
>  	lock_get_status(f, fl, *((loff_t *)f->private), "");
>  
> +	spin_lock(&blocked_hash_lock);
>  	hash_for_each(blocked_hash, bkt, bfl, fl_link) {
>  		if (bfl->fl_next == fl)
>  			lock_get_status(f, bfl, *((loff_t *)f->private), " ->");
>  	}
> +	spin_unlock(&blocked_hash_lock);
>  
>  	return 0;
>  }
> -- 
> 1.7.1
> 


From jlayton at redhat.com  Thu Jun 13 11:09:33 2013
From: jlayton at redhat.com (Jeff Layton)
Date: Thu, 13 Jun 2013 11:09:33 -0400
Subject: [PATCH v2 07/14] locks: convert to i_lock to protect i_flock list
In-Reply-To: <20130613144140.GA20235@fieldses.org>
References: <1370948948-31784-1-git-send-email-jlayton@redhat.com>
 <1370948948-31784-8-git-send-email-jlayton@redhat.com>
 <20130613144140.GA20235@fieldses.org>
Message-ID: <20130613110933.0e8b17fc@corrin.poochiereds.net>

On Thu, 13 Jun 2013 10:41:40 -0400
"J. Bruce Fields" <bfields at fieldses.org> wrote:

> On Tue, Jun 11, 2013 at 07:09:01AM -0400, Jeff Layton wrote:
> > Having a global lock that protects all of this code is a clear
> > scalability problem. Instead of doing that, move most of the code to be
> > protected by the i_lock instead.
> > 
> > The exceptions are the global lists that file_lock->fl_link sits on.
> > Those still need a global lock of some sort, so wrap just those accesses
> > in the file_lock_lock().
> > 
> > Note that this patch introduces a potential race in the deadlock
> > detection code which could result in either false positives or missed
> > file_lock loops. The blocked_list management needs to be done atomically
> > with the deadlock detection. This will be fixed in a later patch.
> 
> Actually, the way I think I'd try this:
> 
> 	1. Introduce a new spinlock global_lock_lists_lock and take it
> 	   in all the places we need a global lock.
> 	2. Do the big flock_lock-to-i_lock conversion, deleting flock_lock
> 	   in the process.
> 
> Does that work?
> 
> Yeah, more patch-ordering bike-shedding, but...  I think that would be
> simple and bisectable and for a confusing long-untouched bit of code
> it's worth getting these little steps right if we can.
> 
> 

I don't know...that really sounds more like it'll just add to the
churn without making anything simpler. If we're concerned about
bisectability, I think it'd just be best to merge patches 7 and 8 and
call it a day...

> > 
> > Signed-off-by: Jeff Layton <jlayton at redhat.com>
> > Signed-off-by: J. Bruce Fields <bfields at redhat.com>
> > ---
> >  Documentation/filesystems/Locking |   23 +++++--
> >  fs/afs/flock.c                    |    5 +-
> >  fs/ceph/locks.c                   |    2 +-
> >  fs/ceph/mds_client.c              |    8 +-
> >  fs/cifs/cifsfs.c                  |    2 +-
> >  fs/cifs/file.c                    |   13 ++--
> >  fs/gfs2/file.c                    |    2 +-
> >  fs/lockd/svcsubs.c                |   12 ++--
> >  fs/locks.c                        |  110 ++++++++++++++++++++-----------------
> >  fs/nfs/delegation.c               |   11 ++--
> >  fs/nfs/nfs4state.c                |    8 +-
> >  fs/nfsd/nfs4state.c               |    8 +-
> >  include/linux/fs.h                |   11 ----
> >  13 files changed, 113 insertions(+), 102 deletions(-)
> > 
> > diff --git a/Documentation/filesystems/Locking b/Documentation/filesystems/Locking
> > index 0706d32..13f91ab 100644
> > --- a/Documentation/filesystems/Locking
> > +++ b/Documentation/filesystems/Locking
> > @@ -344,7 +344,7 @@ prototypes:
> >  
> >  
> >  locking rules:
> > -			file_lock_lock	may block
> > +			inode->i_lock	may block
> >  fl_copy_lock:		yes		no
> >  fl_release_private:	maybe		no
> >  
> > @@ -357,12 +357,21 @@ prototypes:
> >  	int (*lm_change)(struct file_lock **, int);
> >  
> >  locking rules:
> > -			file_lock_lock	may block
> > -lm_compare_owner:	yes		no
> > -lm_notify:		yes		no
> > -lm_grant:		no		no
> > -lm_break:		yes		no
> > -lm_change		yes		no
> > +
> > +			inode->i_lock	file_lock_lock	may block
> > +lm_compare_owner:	yes		maybe		no
> > +lm_notify:		yes		no		no
> > +lm_grant:		no		no		no
> > +lm_break:		yes		no		no
> > +lm_change		yes		no		no
> > +
> > +	->lm_compare_owner is generally called with *an* inode->i_lock
> > +held. It may not be the i_lock of the inode for either file_lock being
> > +compared! This is the case with deadlock detection, since the code has
> > +to chase down the owners of locks that may be entirely unrelated to the
> > +one on which the lock is being acquired. For deadlock detection however,
> > +the file_lock_lock is also held. The locks primarily ensure that neither
> > +file_lock disappear out from under you while doing the comparison.
> >  
> >  --------------------------- buffer_head -----------------------------------
> >  prototypes:
> > diff --git a/fs/afs/flock.c b/fs/afs/flock.c
> > index 2497bf3..03fc0d1 100644
> > --- a/fs/afs/flock.c
> > +++ b/fs/afs/flock.c
> > @@ -252,6 +252,7 @@ static void afs_defer_unlock(struct afs_vnode *vnode, struct key *key)
> >   */
> >  static int afs_do_setlk(struct file *file, struct file_lock *fl)
> >  {
> > +	struct inode = file_inode(file);
> >  	struct afs_vnode *vnode = AFS_FS_I(file->f_mapping->host);
> >  	afs_lock_type_t type;
> >  	struct key *key = file->private_data;
> > @@ -273,7 +274,7 @@ static int afs_do_setlk(struct file *file, struct file_lock *fl)
> >  
> >  	type = (fl->fl_type == F_RDLCK) ? AFS_LOCK_READ : AFS_LOCK_WRITE;
> >  
> > -	lock_flocks();
> > +	spin_lock(&inode->i_lock);
> >  
> >  	/* make sure we've got a callback on this file and that our view of the
> >  	 * data version is up to date */
> > @@ -420,7 +421,7 @@ given_lock:
> >  	afs_vnode_fetch_status(vnode, NULL, key);
> >  
> >  error:
> > -	unlock_flocks();
> > +	spin_unlock(&inode->i_lock);
> >  	_leave(" = %d", ret);
> >  	return ret;
> >  
> > diff --git a/fs/ceph/locks.c b/fs/ceph/locks.c
> > index 202dd3d..cd0a664 100644
> > --- a/fs/ceph/locks.c
> > +++ b/fs/ceph/locks.c
> > @@ -194,7 +194,7 @@ void ceph_count_locks(struct inode *inode, int *fcntl_count, int *flock_count)
> >   * Encode the flock and fcntl locks for the given inode into the pagelist.
> >   * Format is: #fcntl locks, sequential fcntl locks, #flock locks,
> >   * sequential flock locks.
> > - * Must be called with lock_flocks() already held.
> > + * Must be called with inode->i_lock already held.
> >   * If we encounter more of a specific lock type than expected,
> >   * we return the value 1.
> >   */
> > diff --git a/fs/ceph/mds_client.c b/fs/ceph/mds_client.c
> > index 4f22671..ae621b5 100644
> > --- a/fs/ceph/mds_client.c
> > +++ b/fs/ceph/mds_client.c
> > @@ -2482,13 +2482,13 @@ static int encode_caps_cb(struct inode *inode, struct ceph_cap *cap,
> >  
> >  		ceph_pagelist_set_cursor(pagelist, &trunc_point);
> >  		do {
> > -			lock_flocks();
> > +			spin_lock(&inode->i_lock);
> >  			ceph_count_locks(inode, &num_fcntl_locks,
> >  					 &num_flock_locks);
> >  			rec.v2.flock_len = (2*sizeof(u32) +
> >  					    (num_fcntl_locks+num_flock_locks) *
> >  					    sizeof(struct ceph_filelock));
> > -			unlock_flocks();
> > +			spin_unlock(&inode->i_lock);
> >  
> >  			/* pre-alloc pagelist */
> >  			ceph_pagelist_truncate(pagelist, &trunc_point);
> > @@ -2499,12 +2499,12 @@ static int encode_caps_cb(struct inode *inode, struct ceph_cap *cap,
> >  
> >  			/* encode locks */
> >  			if (!err) {
> > -				lock_flocks();
> > +				spin_lock(&inode->i_lock);
> >  				err = ceph_encode_locks(inode,
> >  							pagelist,
> >  							num_fcntl_locks,
> >  							num_flock_locks);
> > -				unlock_flocks();
> > +				spin_unlock(&inode->i_lock);
> >  			}
> >  		} while (err == -ENOSPC);
> >  	} else {
> > diff --git a/fs/cifs/cifsfs.c b/fs/cifs/cifsfs.c
> > index 3752b9f..e81655c 100644
> > --- a/fs/cifs/cifsfs.c
> > +++ b/fs/cifs/cifsfs.c
> > @@ -765,7 +765,7 @@ static loff_t cifs_llseek(struct file *file, loff_t offset, int whence)
> >  
> >  static int cifs_setlease(struct file *file, long arg, struct file_lock **lease)
> >  {
> > -	/* note that this is called by vfs setlease with lock_flocks held
> > +	/* note that this is called by vfs setlease with i_lock held
> >  	   to protect *lease from going away */
> >  	struct inode *inode = file_inode(file);
> >  	struct cifsFileInfo *cfile = file->private_data;
> > diff --git a/fs/cifs/file.c b/fs/cifs/file.c
> > index 44a4f18..0dd10cd 100644
> > --- a/fs/cifs/file.c
> > +++ b/fs/cifs/file.c
> > @@ -1092,6 +1092,7 @@ struct lock_to_push {
> >  static int
> >  cifs_push_posix_locks(struct cifsFileInfo *cfile)
> >  {
> > +	struct inode *inode = cfile->dentry->d_inode;
> >  	struct cifs_tcon *tcon = tlink_tcon(cfile->tlink);
> >  	struct file_lock *flock, **before;
> >  	unsigned int count = 0, i = 0;
> > @@ -1102,12 +1103,12 @@ cifs_push_posix_locks(struct cifsFileInfo *cfile)
> >  
> >  	xid = get_xid();
> >  
> > -	lock_flocks();
> > -	cifs_for_each_lock(cfile->dentry->d_inode, before) {
> > +	spin_lock(&inode->i_lock);
> > +	cifs_for_each_lock(inode, before) {
> >  		if ((*before)->fl_flags & FL_POSIX)
> >  			count++;
> >  	}
> > -	unlock_flocks();
> > +	spin_unlock(&inode->i_lock);
> >  
> >  	INIT_LIST_HEAD(&locks_to_send);
> >  
> > @@ -1126,8 +1127,8 @@ cifs_push_posix_locks(struct cifsFileInfo *cfile)
> >  	}
> >  
> >  	el = locks_to_send.next;
> > -	lock_flocks();
> > -	cifs_for_each_lock(cfile->dentry->d_inode, before) {
> > +	spin_lock(&inode->i_lock);
> > +	cifs_for_each_lock(inode, before) {
> >  		flock = *before;
> >  		if ((flock->fl_flags & FL_POSIX) == 0)
> >  			continue;
> > @@ -1152,7 +1153,7 @@ cifs_push_posix_locks(struct cifsFileInfo *cfile)
> >  		lck->offset = flock->fl_start;
> >  		el = el->next;
> >  	}
> > -	unlock_flocks();
> > +	spin_unlock(&inode->i_lock);
> >  
> >  	list_for_each_entry_safe(lck, tmp, &locks_to_send, llist) {
> >  		int stored_rc;
> > diff --git a/fs/gfs2/file.c b/fs/gfs2/file.c
> > index acd1676..9e634e0 100644
> > --- a/fs/gfs2/file.c
> > +++ b/fs/gfs2/file.c
> > @@ -889,7 +889,7 @@ out_uninit:
> >   * cluster; until we do, disable leases (by just returning -EINVAL),
> >   * unless the administrator has requested purely local locking.
> >   *
> > - * Locking: called under lock_flocks
> > + * Locking: called under i_lock
> >   *
> >   * Returns: errno
> >   */
> > diff --git a/fs/lockd/svcsubs.c b/fs/lockd/svcsubs.c
> > index 97e8741..dc5c759 100644
> > --- a/fs/lockd/svcsubs.c
> > +++ b/fs/lockd/svcsubs.c
> > @@ -169,7 +169,7 @@ nlm_traverse_locks(struct nlm_host *host, struct nlm_file *file,
> >  
> >  again:
> >  	file->f_locks = 0;
> > -	lock_flocks(); /* protects i_flock list */
> > +	spin_lock(&inode->i_lock);
> >  	for (fl = inode->i_flock; fl; fl = fl->fl_next) {
> >  		if (fl->fl_lmops != &nlmsvc_lock_operations)
> >  			continue;
> > @@ -181,7 +181,7 @@ again:
> >  		if (match(lockhost, host)) {
> >  			struct file_lock lock = *fl;
> >  
> > -			unlock_flocks();
> > +			spin_unlock(&inode->i_lock);
> >  			lock.fl_type  = F_UNLCK;
> >  			lock.fl_start = 0;
> >  			lock.fl_end   = OFFSET_MAX;
> > @@ -193,7 +193,7 @@ again:
> >  			goto again;
> >  		}
> >  	}
> > -	unlock_flocks();
> > +	spin_unlock(&inode->i_lock);
> >  
> >  	return 0;
> >  }
> > @@ -228,14 +228,14 @@ nlm_file_inuse(struct nlm_file *file)
> >  	if (file->f_count || !list_empty(&file->f_blocks) || file->f_shares)
> >  		return 1;
> >  
> > -	lock_flocks();
> > +	spin_lock(&inode->i_lock);
> >  	for (fl = inode->i_flock; fl; fl = fl->fl_next) {
> >  		if (fl->fl_lmops == &nlmsvc_lock_operations) {
> > -			unlock_flocks();
> > +			spin_unlock(&inode->i_lock);
> >  			return 1;
> >  		}
> >  	}
> > -	unlock_flocks();
> > +	spin_unlock(&inode->i_lock);
> >  	file->f_locks = 0;
> >  	return 0;
> >  }
> > diff --git a/fs/locks.c b/fs/locks.c
> > index 3fd27f0..d7342a3 100644
> > --- a/fs/locks.c
> > +++ b/fs/locks.c
> > @@ -155,22 +155,9 @@ int lease_break_time = 45;
> >  
> >  static LIST_HEAD(file_lock_list);
> >  static LIST_HEAD(blocked_list);
> > -static DEFINE_SPINLOCK(file_lock_lock);
> > -
> > -/*
> > - * Protects the two list heads above, plus the inode->i_flock list
> > - */
> > -void lock_flocks(void)
> > -{
> > -	spin_lock(&file_lock_lock);
> > -}
> > -EXPORT_SYMBOL_GPL(lock_flocks);
> >  
> > -void unlock_flocks(void)
> > -{
> > -	spin_unlock(&file_lock_lock);
> > -}
> > -EXPORT_SYMBOL_GPL(unlock_flocks);
> > +/* Protects the two list heads above */
> > +static DEFINE_SPINLOCK(file_lock_lock);
> >  
> >  static struct kmem_cache *filelock_cache __read_mostly;
> >  
> > @@ -488,25 +475,33 @@ static int posix_same_owner(struct file_lock *fl1, struct file_lock *fl2)
> >  static inline void
> >  locks_insert_global_blocked(struct file_lock *waiter)
> >  {
> > +	spin_lock(&file_lock_lock);
> >  	list_add(&waiter->fl_link, &blocked_list);
> > +	spin_unlock(&file_lock_lock);
> >  }
> >  
> >  static inline void
> >  locks_delete_global_blocked(struct file_lock *waiter)
> >  {
> > +	spin_lock(&file_lock_lock);
> >  	list_del_init(&waiter->fl_link);
> > +	spin_unlock(&file_lock_lock);
> >  }
> >  
> >  static inline void
> >  locks_insert_global_locks(struct file_lock *waiter)
> >  {
> > +	spin_lock(&file_lock_lock);
> >  	list_add_tail(&waiter->fl_link, &file_lock_list);
> > +	spin_unlock(&file_lock_lock);
> >  }
> >  
> >  static inline void
> >  locks_delete_global_locks(struct file_lock *waiter)
> >  {
> > +	spin_lock(&file_lock_lock);
> >  	list_del_init(&waiter->fl_link);
> > +	spin_unlock(&file_lock_lock);
> >  }
> >  
> >  /* Remove waiter from blocker's block list.
> > @@ -523,9 +518,11 @@ static void __locks_delete_block(struct file_lock *waiter)
> >   */
> >  static void locks_delete_block(struct file_lock *waiter)
> >  {
> > -	lock_flocks();
> > +	struct inode *inode = file_inode(waiter->fl_file);
> > +
> > +	spin_lock(&inode->i_lock);
> >  	__locks_delete_block(waiter);
> > -	unlock_flocks();
> > +	spin_unlock(&inode->i_lock);
> >  }
> >  
> >  /* Insert waiter into blocker's block list.
> > @@ -544,7 +541,7 @@ static void locks_insert_block(struct file_lock *blocker,
> >  /*
> >   * Wake up processes blocked waiting for blocker.
> >   *
> > - * Must be called with the file_lock_lock held!
> > + * Must be called with the inode->i_lock of the blocker held!
> >   */
> >  static void locks_wake_up_blocks(struct file_lock *blocker)
> >  {
> > @@ -649,8 +646,9 @@ void
> >  posix_test_lock(struct file *filp, struct file_lock *fl)
> >  {
> >  	struct file_lock *cfl;
> > +	struct inode *inode = file_inode(filp);
> >  
> > -	lock_flocks();
> > +	spin_lock(&inode->i_lock);
> >  	for (cfl = file_inode(filp)->i_flock; cfl; cfl = cfl->fl_next) {
> >  		if (!IS_POSIX(cfl))
> >  			continue;
> > @@ -663,7 +661,7 @@ posix_test_lock(struct file *filp, struct file_lock *fl)
> >  			fl->fl_pid = pid_vnr(cfl->fl_nspid);
> >  	} else
> >  		fl->fl_type = F_UNLCK;
> > -	unlock_flocks();
> > +	spin_unlock(&inode->i_lock);
> >  	return;
> >  }
> >  EXPORT_SYMBOL(posix_test_lock);
> > @@ -742,7 +740,7 @@ static int flock_lock_file(struct file *filp, struct file_lock *request)
> >  			return -ENOMEM;
> >  	}
> >  
> > -	lock_flocks();
> > +	spin_lock(&inode->i_lock);
> >  	if (request->fl_flags & FL_ACCESS)
> >  		goto find_conflict;
> >  
> > @@ -772,9 +770,9 @@ static int flock_lock_file(struct file *filp, struct file_lock *request)
> >  	 * give it the opportunity to lock the file.
> >  	 */
> >  	if (found) {
> > -		unlock_flocks();
> > +		spin_unlock(&inode->i_lock);
> >  		cond_resched();
> > -		lock_flocks();
> > +		spin_lock(&inode->i_lock);
> >  	}
> >  
> >  find_conflict:
> > @@ -801,7 +799,7 @@ find_conflict:
> >  	error = 0;
> >  
> >  out:
> > -	unlock_flocks();
> > +	spin_unlock(&inode->i_lock);
> >  	if (new_fl)
> >  		locks_free_lock(new_fl);
> >  	return error;
> > @@ -831,7 +829,7 @@ static int __posix_lock_file(struct inode *inode, struct file_lock *request, str
> >  		new_fl2 = locks_alloc_lock();
> >  	}
> >  
> > -	lock_flocks();
> > +	spin_lock(&inode->i_lock);
> >  	/*
> >  	 * New lock request. Walk all POSIX locks and look for conflicts. If
> >  	 * there are any, either return error or put the request on the
> > @@ -850,8 +848,14 @@ static int __posix_lock_file(struct inode *inode, struct file_lock *request, str
> >  			if (!(request->fl_flags & FL_SLEEP))
> >  				goto out;
> >  			error = -EDEADLK;
> > +			/*
> > +			 * XXX: potential race here. We should be adding the
> > +			 * file_lock to the global list before releasing lock.
> > +			 */
> > +			spin_lock(&file_lock_lock);
> >  			if (posix_locks_deadlock(request, fl))
> >  				goto out;
> > +			spin_unlock(&file_lock_lock);
> >  			error = FILE_LOCK_DEFERRED;
> >  			locks_insert_block(fl, request);
> >  			locks_insert_global_blocked(request);
> > @@ -1004,7 +1008,7 @@ static int __posix_lock_file(struct inode *inode, struct file_lock *request, str
> >  		locks_wake_up_blocks(left);
> >  	}
> >   out:
> > -	unlock_flocks();
> > +	spin_unlock(&inode->i_lock);
> >  	/*
> >  	 * Free any unused locks.
> >  	 */
> > @@ -1079,14 +1083,14 @@ int locks_mandatory_locked(struct inode *inode)
> >  	/*
> >  	 * Search the lock list for this inode for any POSIX locks.
> >  	 */
> > -	lock_flocks();
> > +	spin_lock(&inode->i_lock);
> >  	for (fl = inode->i_flock; fl != NULL; fl = fl->fl_next) {
> >  		if (!IS_POSIX(fl))
> >  			continue;
> >  		if (fl->fl_owner != owner)
> >  			break;
> >  	}
> > -	unlock_flocks();
> > +	spin_unlock(&inode->i_lock);
> >  	return fl ? -EAGAIN : 0;
> >  }
> >  
> > @@ -1229,7 +1233,7 @@ int __break_lease(struct inode *inode, unsigned int mode)
> >  	if (IS_ERR(new_fl))
> >  		return PTR_ERR(new_fl);
> >  
> > -	lock_flocks();
> > +	spin_lock(&inode->i_lock);
> >  
> >  	time_out_leases(inode);
> >  
> > @@ -1279,10 +1283,10 @@ restart:
> >  			break_time++;
> >  	}
> >  	locks_insert_block(flock, new_fl);
> > -	unlock_flocks();
> > +	spin_unlock(&inode->i_lock);
> >  	error = wait_event_interruptible_timeout(new_fl->fl_wait,
> >  						!new_fl->fl_next, break_time);
> > -	lock_flocks();
> > +	spin_lock(&inode->i_lock);
> >  	__locks_delete_block(new_fl);
> >  	if (error >= 0) {
> >  		if (error == 0)
> > @@ -1300,7 +1304,7 @@ restart:
> >  	}
> >  
> >  out:
> > -	unlock_flocks();
> > +	spin_unlock(&inode->i_lock);
> >  	locks_free_lock(new_fl);
> >  	return error;
> >  }
> > @@ -1353,9 +1357,10 @@ EXPORT_SYMBOL(lease_get_mtime);
> >  int fcntl_getlease(struct file *filp)
> >  {
> >  	struct file_lock *fl;
> > +	struct inode *inode = file_inode(filp);
> >  	int type = F_UNLCK;
> >  
> > -	lock_flocks();
> > +	spin_lock(&inode->i_lock);
> >  	time_out_leases(file_inode(filp));
> >  	for (fl = file_inode(filp)->i_flock; fl && IS_LEASE(fl);
> >  			fl = fl->fl_next) {
> > @@ -1364,7 +1369,7 @@ int fcntl_getlease(struct file *filp)
> >  			break;
> >  		}
> >  	}
> > -	unlock_flocks();
> > +	spin_unlock(&inode->i_lock);
> >  	return type;
> >  }
> >  
> > @@ -1458,7 +1463,7 @@ static int generic_delete_lease(struct file *filp, struct file_lock **flp)
> >   *	The (input) flp->fl_lmops->lm_break function is required
> >   *	by break_lease().
> >   *
> > - *	Called with file_lock_lock held.
> > + *	Called with inode->i_lock held.
> >   */
> >  int generic_setlease(struct file *filp, long arg, struct file_lock **flp)
> >  {
> > @@ -1527,11 +1532,12 @@ static int __vfs_setlease(struct file *filp, long arg, struct file_lock **lease)
> >  
> >  int vfs_setlease(struct file *filp, long arg, struct file_lock **lease)
> >  {
> > +	struct inode *inode = file_inode(filp);
> >  	int error;
> >  
> > -	lock_flocks();
> > +	spin_lock(&inode->i_lock);
> >  	error = __vfs_setlease(filp, arg, lease);
> > -	unlock_flocks();
> > +	spin_unlock(&inode->i_lock);
> >  
> >  	return error;
> >  }
> > @@ -1549,6 +1555,7 @@ static int do_fcntl_delete_lease(struct file *filp)
> >  static int do_fcntl_add_lease(unsigned int fd, struct file *filp, long arg)
> >  {
> >  	struct file_lock *fl, *ret;
> > +	struct inode *inode = file_inode(filp);
> >  	struct fasync_struct *new;
> >  	int error;
> >  
> > @@ -1562,10 +1569,10 @@ static int do_fcntl_add_lease(unsigned int fd, struct file *filp, long arg)
> >  		return -ENOMEM;
> >  	}
> >  	ret = fl;
> > -	lock_flocks();
> > +	spin_lock(&inode->i_lock);
> >  	error = __vfs_setlease(filp, arg, &ret);
> >  	if (error) {
> > -		unlock_flocks();
> > +		spin_unlock(&inode->i_lock);
> >  		locks_free_lock(fl);
> >  		goto out_free_fasync;
> >  	}
> > @@ -1582,7 +1589,7 @@ static int do_fcntl_add_lease(unsigned int fd, struct file *filp, long arg)
> >  		new = NULL;
> >  
> >  	error = __f_setown(filp, task_pid(current), PIDTYPE_PID, 0);
> > -	unlock_flocks();
> > +	spin_unlock(&inode->i_lock);
> >  
> >  out_free_fasync:
> >  	if (new)
> > @@ -2106,7 +2113,7 @@ void locks_remove_flock(struct file *filp)
> >  			fl.fl_ops->fl_release_private(&fl);
> >  	}
> >  
> > -	lock_flocks();
> > +	spin_lock(&inode->i_lock);
> >  	before = &inode->i_flock;
> >  
> >  	while ((fl = *before) != NULL) {
> > @@ -2124,7 +2131,7 @@ void locks_remove_flock(struct file *filp)
> >   		}
> >  		before = &fl->fl_next;
> >  	}
> > -	unlock_flocks();
> > +	spin_unlock(&inode->i_lock);
> >  }
> >  
> >  /**
> > @@ -2137,14 +2144,15 @@ void locks_remove_flock(struct file *filp)
> >  int
> >  posix_unblock_lock(struct file *filp, struct file_lock *waiter)
> >  {
> > +	struct inode *inode = file_inode(filp);
> >  	int status = 0;
> >  
> > -	lock_flocks();
> > +	spin_lock(&inode->i_lock);
> >  	if (waiter->fl_next)
> >  		__locks_delete_block(waiter);
> >  	else
> >  		status = -ENOENT;
> > -	unlock_flocks();
> > +	spin_unlock(&inode->i_lock);
> >  	return status;
> >  }
> >  
> > @@ -2261,7 +2269,7 @@ static void *locks_start(struct seq_file *f, loff_t *pos)
> >  {
> >  	loff_t *p = f->private;
> >  
> > -	lock_flocks();
> > +	spin_lock(&file_lock_lock);
> >  	*p = (*pos + 1);
> >  	return seq_list_start(&file_lock_list, *pos);
> >  }
> > @@ -2275,7 +2283,7 @@ static void *locks_next(struct seq_file *f, void *v, loff_t *pos)
> >  
> >  static void locks_stop(struct seq_file *f, void *v)
> >  {
> > -	unlock_flocks();
> > +	spin_unlock(&file_lock_lock);
> >  }
> >  
> >  static const struct seq_operations locks_seq_operations = {
> > @@ -2322,7 +2330,8 @@ int lock_may_read(struct inode *inode, loff_t start, unsigned long len)
> >  {
> >  	struct file_lock *fl;
> >  	int result = 1;
> > -	lock_flocks();
> > +
> > +	spin_lock(&inode->i_lock);
> >  	for (fl = inode->i_flock; fl != NULL; fl = fl->fl_next) {
> >  		if (IS_POSIX(fl)) {
> >  			if (fl->fl_type == F_RDLCK)
> > @@ -2339,7 +2348,7 @@ int lock_may_read(struct inode *inode, loff_t start, unsigned long len)
> >  		result = 0;
> >  		break;
> >  	}
> > -	unlock_flocks();
> > +	spin_unlock(&inode->i_lock);
> >  	return result;
> >  }
> >  
> > @@ -2362,7 +2371,8 @@ int lock_may_write(struct inode *inode, loff_t start, unsigned long len)
> >  {
> >  	struct file_lock *fl;
> >  	int result = 1;
> > -	lock_flocks();
> > +
> > +	spin_lock(&inode->i_lock);
> >  	for (fl = inode->i_flock; fl != NULL; fl = fl->fl_next) {
> >  		if (IS_POSIX(fl)) {
> >  			if ((fl->fl_end < start) || (fl->fl_start > (start + len)))
> > @@ -2377,7 +2387,7 @@ int lock_may_write(struct inode *inode, loff_t start, unsigned long len)
> >  		result = 0;
> >  		break;
> >  	}
> > -	unlock_flocks();
> > +	spin_unlock(&inode->i_lock);
> >  	return result;
> >  }
> >  
> > diff --git a/fs/nfs/delegation.c b/fs/nfs/delegation.c
> > index 57db324..43ee7f9 100644
> > --- a/fs/nfs/delegation.c
> > +++ b/fs/nfs/delegation.c
> > @@ -73,20 +73,21 @@ static int nfs_delegation_claim_locks(struct nfs_open_context *ctx, struct nfs4_
> >  	if (inode->i_flock == NULL)
> >  		goto out;
> >  
> > -	/* Protect inode->i_flock using the file locks lock */
> > -	lock_flocks();
> > +	/* Protect inode->i_flock using the i_lock */
> > +	spin_lock(&inode->i_lock);
> >  	for (fl = inode->i_flock; fl != NULL; fl = fl->fl_next) {
> >  		if (!(fl->fl_flags & (FL_POSIX|FL_FLOCK)))
> >  			continue;
> >  		if (nfs_file_open_context(fl->fl_file) != ctx)
> >  			continue;
> > -		unlock_flocks();
> > +		/* FIXME: safe to drop lock here while walking list? */
> > +		spin_unlock(&inode->i_lock);
> >  		status = nfs4_lock_delegation_recall(fl, state, stateid);
> >  		if (status < 0)
> >  			goto out;
> > -		lock_flocks();
> > +		spin_lock(&inode->i_lock);
> >  	}
> > -	unlock_flocks();
> > +	spin_unlock(&inode->i_lock);
> >  out:
> >  	return status;
> >  }
> > diff --git a/fs/nfs/nfs4state.c b/fs/nfs/nfs4state.c
> > index 1fab140..ff10b4a 100644
> > --- a/fs/nfs/nfs4state.c
> > +++ b/fs/nfs/nfs4state.c
> > @@ -1373,13 +1373,13 @@ static int nfs4_reclaim_locks(struct nfs4_state *state, const struct nfs4_state_
> >  	/* Guard against delegation returns and new lock/unlock calls */
> >  	down_write(&nfsi->rwsem);
> >  	/* Protect inode->i_flock using the BKL */
> > -	lock_flocks();
> > +	spin_lock(&inode->i_lock);
> >  	for (fl = inode->i_flock; fl != NULL; fl = fl->fl_next) {
> >  		if (!(fl->fl_flags & (FL_POSIX|FL_FLOCK)))
> >  			continue;
> >  		if (nfs_file_open_context(fl->fl_file)->state != state)
> >  			continue;
> > -		unlock_flocks();
> > +		spin_unlock(&inode->i_lock);
> >  		status = ops->recover_lock(state, fl);
> >  		switch (status) {
> >  			case 0:
> > @@ -1406,9 +1406,9 @@ static int nfs4_reclaim_locks(struct nfs4_state *state, const struct nfs4_state_
> >  				/* kill_proc(fl->fl_pid, SIGLOST, 1); */
> >  				status = 0;
> >  		}
> > -		lock_flocks();
> > +		spin_lock(&inode->i_lock);
> >  	}
> > -	unlock_flocks();
> > +	spin_unlock(&inode->i_lock);
> >  out:
> >  	up_write(&nfsi->rwsem);
> >  	return status;
> > diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
> > index 316ec84..f170518 100644
> > --- a/fs/nfsd/nfs4state.c
> > +++ b/fs/nfsd/nfs4state.c
> > @@ -2645,13 +2645,13 @@ static void nfsd_break_one_deleg(struct nfs4_delegation *dp)
> >  
> >  	list_add_tail(&dp->dl_recall_lru, &nn->del_recall_lru);
> >  
> > -	/* only place dl_time is set. protected by lock_flocks*/
> > +	/* Only place dl_time is set; protected by i_lock: */
> >  	dp->dl_time = get_seconds();
> >  
> >  	nfsd4_cb_recall(dp);
> >  }
> >  
> > -/* Called from break_lease() with lock_flocks() held. */
> > +/* Called from break_lease() with i_lock held. */
> >  static void nfsd_break_deleg_cb(struct file_lock *fl)
> >  {
> >  	struct nfs4_file *fp = (struct nfs4_file *)fl->fl_owner;
> > @@ -4520,7 +4520,7 @@ check_for_locks(struct nfs4_file *filp, struct nfs4_lockowner *lowner)
> >  	struct inode *inode = filp->fi_inode;
> >  	int status = 0;
> >  
> > -	lock_flocks();
> > +	spin_lock(&inode->i_lock);
> >  	for (flpp = &inode->i_flock; *flpp != NULL; flpp = &(*flpp)->fl_next) {
> >  		if ((*flpp)->fl_owner == (fl_owner_t)lowner) {
> >  			status = 1;
> > @@ -4528,7 +4528,7 @@ check_for_locks(struct nfs4_file *filp, struct nfs4_lockowner *lowner)
> >  		}
> >  	}
> >  out:
> > -	unlock_flocks();
> > +	spin_unlock(&inode->i_lock);
> >  	return status;
> >  }
> >  
> > diff --git a/include/linux/fs.h b/include/linux/fs.h
> > index 94105d2..e2f896d 100644
> > --- a/include/linux/fs.h
> > +++ b/include/linux/fs.h
> > @@ -1024,8 +1024,6 @@ extern int vfs_setlease(struct file *, long, struct file_lock **);
> >  extern int lease_modify(struct file_lock **, int);
> >  extern int lock_may_read(struct inode *, loff_t start, unsigned long count);
> >  extern int lock_may_write(struct inode *, loff_t start, unsigned long count);
> > -extern void lock_flocks(void);
> > -extern void unlock_flocks(void);
> >  #else /* !CONFIG_FILE_LOCKING */
> >  static inline int fcntl_getlk(struct file *file, struct flock __user *user)
> >  {
> > @@ -1167,15 +1165,6 @@ static inline int lock_may_write(struct inode *inode, loff_t start,
> >  {
> >  	return 1;
> >  }
> > -
> > -static inline void lock_flocks(void)
> > -{
> > -}
> > -
> > -static inline void unlock_flocks(void)
> > -{
> > -}
> > -
> >  #endif /* !CONFIG_FILE_LOCKING */
> >  
> >  
> > -- 
> > 1.7.1
> > 


-- 
Jeff Layton <jlayton at redhat.com>


From jlayton at redhat.com  Thu Jun 13 11:18:44 2013
From: jlayton at redhat.com (Jeff Layton)
Date: Thu, 13 Jun 2013 11:18:44 -0400
Subject: [PATCH v2 12/14] locks: give the blocked_hash its own spinlock
In-Reply-To: <20130613150247.GB20666@fieldses.org>
References: <1370948948-31784-1-git-send-email-jlayton@redhat.com>
 <1370948948-31784-13-git-send-email-jlayton@redhat.com>
 <20130613150247.GB20666@fieldses.org>
Message-ID: <20130613111844.59421622@corrin.poochiereds.net>

On Thu, 13 Jun 2013 11:02:47 -0400
"J. Bruce Fields" <bfields at fieldses.org> wrote:

> On Tue, Jun 11, 2013 at 07:09:06AM -0400, Jeff Layton wrote:
> > There's no reason we have to protect the blocked_hash and file_lock_list
> > with the same spinlock. With the tests I have, breaking it in two gives
> > a barely measurable performance benefit, but it seems reasonable to make
> > this locking as granular as possible.
> 
> Out of curiosity...  In the typical case when adding/removing a lock,
> aren't both lists being modified in rapid succession?
> 
> I wonder if it would be better to instead stick with one lock and take
> care to acquire it only once to cover both manipulations.
> 
> --b.
> 

That's not really the case...

Typically, when doing a call into __posix_lock_file with FL_SLEEP set,
we either end up blocking on the lock or acquiring it. In either case,
we'll only end up taking one of the global spinlocks. The reason for
this is that blocker is what dequeues a waiter from the blocked_hash
before waking it up (in locks_wake_up_posix_blocks).

Also, while this patch description doesn't spell it out, we require a
truly global lock for deadlock detection. In a later patch though, I
convert the file_lock_lock to a per-cpu spinlock. So we really do need
to separate the locks here in order to make the per-cpu file_lock_list
worthwhile.

> > 
> > Signed-off-by: Jeff Layton <jlayton at redhat.com>
> > ---
> >  Documentation/filesystems/Locking |   16 ++++++++--------
> >  fs/locks.c                        |   25 +++++++++++++------------
> >  2 files changed, 21 insertions(+), 20 deletions(-)
> > 
> > diff --git a/Documentation/filesystems/Locking b/Documentation/filesystems/Locking
> > index ee351ac..8d8d040 100644
> > --- a/Documentation/filesystems/Locking
> > +++ b/Documentation/filesystems/Locking
> > @@ -359,20 +359,20 @@ prototypes:
> >  
> >  locking rules:
> >  
> > -			inode->i_lock	file_lock_lock	may block
> > -lm_compare_owner:	yes		maybe		no
> > -lm_owner_key		yes		yes		no
> > -lm_notify:		yes		no		no
> > -lm_grant:		no		no		no
> > -lm_break:		yes		no		no
> > -lm_change		yes		no		no
> > +			inode->i_lock	blocked_hash_lock	may block
> > +lm_compare_owner:	yes		maybe			no
> > +lm_owner_key		yes		yes			no
> > +lm_notify:		yes		no			no
> > +lm_grant:		no		no			no
> > +lm_break:		yes		no			no
> > +lm_change		yes		no			no
> >  
> >  	->lm_compare_owner and ->lm_owner_key are generally called with
> >  *an* inode->i_lock held. It may not be the i_lock of the inode
> >  associated with either file_lock argument! This is the case with deadlock
> >  detection, since the code has to chase down the owners of locks that may
> >  be entirely unrelated to the one on which the lock is being acquired.
> > -For deadlock detection however, the file_lock_lock is also held. The
> > +For deadlock detection however, the blocked_hash_lock is also held. The
> >  fact that these locks are held ensures that the file_locks do not
> >  disappear out from under you while doing the comparison or generating an
> >  owner key.
> > diff --git a/fs/locks.c b/fs/locks.c
> > index 11e7784..8124fc1 100644
> > --- a/fs/locks.c
> > +++ b/fs/locks.c
> > @@ -162,12 +162,11 @@ int lease_break_time = 45;
> >   */
> >  #define BLOCKED_HASH_BITS	7
> >  
> > +static DEFINE_SPINLOCK(blocked_hash_lock);
> >  static DEFINE_HASHTABLE(blocked_hash, BLOCKED_HASH_BITS);
> >  
> > -static HLIST_HEAD(file_lock_list);
> > -
> > -/* Protects the file_lock_list and the blocked_hash */
> >  static DEFINE_SPINLOCK(file_lock_lock);
> > +static HLIST_HEAD(file_lock_list);
> >  
> >  static struct kmem_cache *filelock_cache __read_mostly;
> >  
> > @@ -505,9 +504,9 @@ __locks_delete_global_blocked(struct file_lock *waiter)
> >  static inline void
> >  locks_delete_global_blocked(struct file_lock *waiter)
> >  {
> > -	spin_lock(&file_lock_lock);
> > +	spin_lock(&blocked_hash_lock);
> >  	__locks_delete_global_blocked(waiter);
> > -	spin_unlock(&file_lock_lock);
> > +	spin_unlock(&blocked_hash_lock);
> >  }
> >  
> >  static inline void
> > @@ -581,14 +580,14 @@ static void locks_wake_up_blocks(struct file_lock *blocker)
> >  
> >  /*
> >   * Wake up processes blocked waiting for blocker. In the FL_POSIX case, we must
> > - * also take the global file_lock_lock and dequeue it from the global blocked
> > - * list as we wake the processes.
> > + * also take the global blocked_hash_lock and dequeue it from the global
> > + * blocked list as we wake the processes.
> >   *
> >   * Must be called with the inode->i_lock of the blocker held!
> >   */
> >  static void locks_wake_up_posix_blocks(struct file_lock *blocker)
> >  {
> > -	spin_lock(&file_lock_lock);
> > +	spin_lock(&blocked_hash_lock);
> >  	while (!list_empty(&blocker->fl_block)) {
> >  		struct file_lock *waiter;
> >  
> > @@ -601,7 +600,7 @@ static void locks_wake_up_posix_blocks(struct file_lock *blocker)
> >  		else
> >  			wake_up(&waiter->fl_wait);
> >  	}
> > -	spin_unlock(&file_lock_lock);
> > +	spin_unlock(&blocked_hash_lock);
> >  }
> >  /* Insert file lock fl into an inode's lock list at the position indicated
> >   * by pos. At the same time add the lock to the global file lock list.
> > @@ -754,7 +753,7 @@ static struct file_lock *what_owner_is_waiting_for(struct file_lock *block_fl)
> >  	return NULL;
> >  }
> >  
> > -/* Must be called with the file_lock_lock held! */
> > +/* Must be called with the blocked_hash_lock held! */
> >  static int posix_locks_deadlock(struct file_lock *caller_fl,
> >  				struct file_lock *block_fl)
> >  {
> > @@ -898,13 +897,13 @@ static int __posix_lock_file(struct inode *inode, struct file_lock *request, str
> >  			if (!(request->fl_flags & FL_SLEEP))
> >  				goto out;
> >  			error = -EDEADLK;
> > -			spin_lock(&file_lock_lock);
> > +			spin_lock(&blocked_hash_lock);
> >  			if (likely(!posix_locks_deadlock(request, fl))) {
> >  				error = FILE_LOCK_DEFERRED;
> >  				locks_insert_block(fl, request);
> >  				locks_insert_global_blocked(request);
> >  			}
> > -			spin_unlock(&file_lock_lock);
> > +			spin_unlock(&blocked_hash_lock);
> >  			goto out;
> >    		}
> >    	}
> > @@ -2309,10 +2308,12 @@ static int locks_show(struct seq_file *f, void *v)
> >  
> >  	lock_get_status(f, fl, *((loff_t *)f->private), "");
> >  
> > +	spin_lock(&blocked_hash_lock);
> >  	hash_for_each(blocked_hash, bkt, bfl, fl_link) {
> >  		if (bfl->fl_next == fl)
> >  			lock_get_status(f, bfl, *((loff_t *)f->private), " ->");
> >  	}
> > +	spin_unlock(&blocked_hash_lock);
> >  
> >  	return 0;
> >  }
> > -- 
> > 1.7.1
> > 


-- 
Jeff Layton <jlayton at redhat.com>


From bfields at fieldses.org  Thu Jun 13 11:20:36 2013
From: bfields at fieldses.org (J. Bruce Fields)
Date: Thu, 13 Jun 2013 11:20:36 -0400
Subject: [PATCH v2 12/14] locks: give the blocked_hash its own spinlock
In-Reply-To: <20130613111844.59421622@corrin.poochiereds.net>
References: <1370948948-31784-1-git-send-email-jlayton@redhat.com>
 <1370948948-31784-13-git-send-email-jlayton@redhat.com>
 <20130613150247.GB20666@fieldses.org>
 <20130613111844.59421622@corrin.poochiereds.net>
Message-ID: <20130613152036.GC20666@fieldses.org>

On Thu, Jun 13, 2013 at 11:18:44AM -0400, Jeff Layton wrote:
> On Thu, 13 Jun 2013 11:02:47 -0400
> "J. Bruce Fields" <bfields at fieldses.org> wrote:
> 
> > On Tue, Jun 11, 2013 at 07:09:06AM -0400, Jeff Layton wrote:
> > > There's no reason we have to protect the blocked_hash and file_lock_list
> > > with the same spinlock. With the tests I have, breaking it in two gives
> > > a barely measurable performance benefit, but it seems reasonable to make
> > > this locking as granular as possible.
> > 
> > Out of curiosity...  In the typical case when adding/removing a lock,
> > aren't both lists being modified in rapid succession?
> > 
> > I wonder if it would be better to instead stick with one lock and take
> > care to acquire it only once to cover both manipulations.
> > 
> > --b.
> > 
> 
> That's not really the case...
> 
> Typically, when doing a call into __posix_lock_file with FL_SLEEP set,
> we either end up blocking on the lock or acquiring it. In either case,
> we'll only end up taking one of the global spinlocks. The reason for
> this is that blocker is what dequeues a waiter from the blocked_hash
> before waking it up (in locks_wake_up_posix_blocks).
> 
> Also, while this patch description doesn't spell it out, we require a
> truly global lock for deadlock detection. In a later patch though, I
> convert the file_lock_lock to a per-cpu spinlock. So we really do need
> to separate the locks here in order to make the per-cpu file_lock_list
> worthwhile.

Oh, right, got it!

--b.


From bfields at fieldses.org  Thu Jun 13 11:27:39 2013
From: bfields at fieldses.org (J. Bruce Fields)
Date: Thu, 13 Jun 2013 11:27:39 -0400
Subject: [PATCH v2 13/14] seq_file: add seq_list_*_percpu helpers
In-Reply-To: <1370948948-31784-14-git-send-email-jlayton@redhat.com>
References: <1370948948-31784-1-git-send-email-jlayton@redhat.com>
 <1370948948-31784-14-git-send-email-jlayton@redhat.com>
Message-ID: <20130613152739.GD20666@fieldses.org>

On Tue, Jun 11, 2013 at 07:09:07AM -0400, Jeff Layton wrote:
> When we convert the file_lock_list to a set of percpu lists, we'll need
> a way to iterate over them in order to output /proc/locks info. Add
> some seq_list_*_percpu helpers to handle that.
> 
> Signed-off-by: Jeff Layton <jlayton at redhat.com>
> ---
>  fs/seq_file.c            |   54 ++++++++++++++++++++++++++++++++++++++++++++++
>  include/linux/seq_file.h |    6 +++++
>  2 files changed, 60 insertions(+), 0 deletions(-)
> 
> diff --git a/fs/seq_file.c b/fs/seq_file.c
> index 774c1eb..ce9f97a 100644
> --- a/fs/seq_file.c
> +++ b/fs/seq_file.c
> @@ -921,3 +921,57 @@ struct hlist_node *seq_hlist_next_rcu(void *v,
>  		return rcu_dereference(node->next);
>  }
>  EXPORT_SYMBOL(seq_hlist_next_rcu);
> +
> +/**
> + * seq_hlist_start_precpu - start an iteration of a hlist

Should be "of a percpu array of hlists"?

> + * @head: pointer to percpu array of struct hlist_heads
> + * @cpu:  pointer to cpu "cursor"
> + * @pos:  start position of sequence
> + *
> + * Called at seq_file->op->start().
> + */
> +struct hlist_node *
> +seq_hlist_start_percpu(struct hlist_head __percpu *head, int *cpu, loff_t pos)
> +{
> +	struct hlist_node *node;
> +
> +	for_each_possible_cpu(*cpu) {
> +		hlist_for_each(node, per_cpu_ptr(head, *cpu)) {
> +			if (pos-- == 0)
> +				return node;
> +		}
> +	}
> +	return NULL;
> +}
> +EXPORT_SYMBOL(seq_hlist_start_percpu);
> +
> +/**
> + * seq_hlist_next_percpu - move to the next position of the hlist

Ditto?

Just thinking if the doc scripts collected up a bunch of these summaries
into an index somewhere it'd be confusing for these to have some
description as seq_hlist_start/next.

Otherwise, ACK.

--b.

> + * @v:    pointer to current hlist_node
> + * @head: pointer to percpu array of struct hlist_heads
> + * @cpu:  pointer to cpu "cursor"
> + * @pos:  start position of sequence
> + *
> + * Called at seq_file->op->next().
> + */
> +struct hlist_node *
> +seq_hlist_next_percpu(void *v, struct hlist_head __percpu *head,
> +			int *cpu, loff_t *pos)
> +{
> +	struct hlist_node *node = v;
> +
> +	++*pos;
> +
> +	if (node->next)
> +		return node->next;
> +
> +	for (*cpu = cpumask_next(*cpu, cpu_possible_mask); *cpu < nr_cpu_ids;
> +	     *cpu = cpumask_next(*cpu, cpu_possible_mask)) {
> +		struct hlist_head *bucket = per_cpu_ptr(head, *cpu);
> +
> +		if (!hlist_empty(bucket))
> +			return bucket->first;
> +	}
> +	return NULL;
> +}
> +EXPORT_SYMBOL(seq_hlist_next_percpu);
> diff --git a/include/linux/seq_file.h b/include/linux/seq_file.h
> index 2da29ac..4e32edc 100644
> --- a/include/linux/seq_file.h
> +++ b/include/linux/seq_file.h
> @@ -173,4 +173,10 @@ extern struct hlist_node *seq_hlist_start_head_rcu(struct hlist_head *head,
>  extern struct hlist_node *seq_hlist_next_rcu(void *v,
>  						   struct hlist_head *head,
>  						   loff_t *ppos);
> +
> +/* Helpers for iterating over per-cpu hlist_head-s in seq_files */
> +extern struct hlist_node *seq_hlist_start_percpu(struct hlist_head __percpu *head, int *cpu, loff_t pos);
> +
> +extern struct hlist_node *seq_hlist_next_percpu(void *v, struct hlist_head __percpu *head, int *cpu, loff_t *pos);
> +
>  #endif
> -- 
> 1.7.1
> 


From bfields at fieldses.org  Thu Jun 13 11:37:42 2013
From: bfields at fieldses.org (J. Bruce Fields)
Date: Thu, 13 Jun 2013 11:37:42 -0400
Subject: [PATCH v2 14/14] locks: move file_lock_list to a set of percpu
 hlist_heads and convert file_lock_lock to an lglock
In-Reply-To: <1370948948-31784-15-git-send-email-jlayton@redhat.com>
References: <1370948948-31784-1-git-send-email-jlayton@redhat.com>
 <1370948948-31784-15-git-send-email-jlayton@redhat.com>
Message-ID: <20130613153742.GE20666@fieldses.org>

On Tue, Jun 11, 2013 at 07:09:08AM -0400, Jeff Layton wrote:
> The file_lock_list is only used for /proc/locks. The vastly common case
> is for locks to be put onto the list and come off again, without ever
> being traversed.
> 
> Help optimize for this use-case by moving to percpu hlist_head-s. At the
> same time, we can make the locking less contentious by moving to an
> lglock. When iterating over the lists for /proc/locks, we must take the
> global lock and then iterate over each CPU's list in turn.
> 
> This change necessitates a new fl_link_cpu field to keep track of which
> CPU the entry is on. On x86_64 at least, this field is placed within an
> existing hole in the struct to avoid growing its real size.

ACK.--b.

> 
> Signed-off-by: Jeff Layton <jlayton at redhat.com>
> ---
>  fs/locks.c         |   57 +++++++++++++++++++++++++++++++++++----------------
>  include/linux/fs.h |    1 +
>  2 files changed, 40 insertions(+), 18 deletions(-)
> 
> diff --git a/fs/locks.c b/fs/locks.c
> index 8124fc1..094eb4d 100644
> --- a/fs/locks.c
> +++ b/fs/locks.c
> @@ -127,6 +127,8 @@
>  #include <linux/rcupdate.h>
>  #include <linux/pid_namespace.h>
>  #include <linux/hashtable.h>
> +#include <linux/percpu.h>
> +#include <linux/lglock.h>
>  
>  #include <asm/uaccess.h>
>  
> @@ -165,8 +167,8 @@ int lease_break_time = 45;
>  static DEFINE_SPINLOCK(blocked_hash_lock);
>  static DEFINE_HASHTABLE(blocked_hash, BLOCKED_HASH_BITS);
>  
> -static DEFINE_SPINLOCK(file_lock_lock);
> -static HLIST_HEAD(file_lock_list);
> +DEFINE_STATIC_LGLOCK(file_lock_lglock);
> +static DEFINE_PER_CPU(struct hlist_head, file_lock_list);
>  
>  static struct kmem_cache *filelock_cache __read_mostly;
>  
> @@ -512,17 +514,21 @@ locks_delete_global_blocked(struct file_lock *waiter)
>  static inline void
>  locks_insert_global_locks(struct file_lock *waiter)
>  {
> -	spin_lock(&file_lock_lock);
> -	hlist_add_head(&waiter->fl_link, &file_lock_list);
> -	spin_unlock(&file_lock_lock);
> +	lg_local_lock(&file_lock_lglock);
> +	waiter->fl_link_cpu = smp_processor_id();
> +	hlist_add_head(&waiter->fl_link, this_cpu_ptr(&file_lock_list));
> +	lg_local_unlock(&file_lock_lglock);
>  }
>  
>  static inline void
>  locks_delete_global_locks(struct file_lock *waiter)
>  {
> -	spin_lock(&file_lock_lock);
> +	/* avoid taking lock if already unhashed */
> +	if (hlist_unhashed(&waiter->fl_link))
> +		return;
> +	lg_local_lock_cpu(&file_lock_lglock, waiter->fl_link_cpu);
>  	hlist_del_init(&waiter->fl_link);
> -	spin_unlock(&file_lock_lock);
> +	lg_local_unlock_cpu(&file_lock_lglock, waiter->fl_link_cpu);
>  }
>  
>  /* Remove waiter from blocker's block list.
> @@ -2228,6 +2234,11 @@ EXPORT_SYMBOL_GPL(vfs_cancel_lock);
>  #include <linux/proc_fs.h>
>  #include <linux/seq_file.h>
>  
> +struct locks_iterator {
> +	int	li_cpu;
> +	loff_t	li_pos;
> +};
> +
>  static void lock_get_status(struct seq_file *f, struct file_lock *fl,
>  			    loff_t id, char *pfx)
>  {
> @@ -2302,16 +2313,17 @@ static void lock_get_status(struct seq_file *f, struct file_lock *fl,
>  static int locks_show(struct seq_file *f, void *v)
>  {
>  	int bkt;
> +	struct locks_iterator *iter = f->private;
>  	struct file_lock *fl, *bfl;
>  
>  	fl = hlist_entry(v, struct file_lock, fl_link);
>  
> -	lock_get_status(f, fl, *((loff_t *)f->private), "");
> +	lock_get_status(f, fl, iter->li_pos, "");
>  
>  	spin_lock(&blocked_hash_lock);
>  	hash_for_each(blocked_hash, bkt, bfl, fl_link) {
>  		if (bfl->fl_next == fl)
> -			lock_get_status(f, bfl, *((loff_t *)f->private), " ->");
> +			lock_get_status(f, bfl, iter->li_pos, " ->");
>  	}
>  	spin_unlock(&blocked_hash_lock);
>  
> @@ -2320,23 +2332,24 @@ static int locks_show(struct seq_file *f, void *v)
>  
>  static void *locks_start(struct seq_file *f, loff_t *pos)
>  {
> -	loff_t *p = f->private;
> +	struct locks_iterator *iter = f->private;
>  
> -	spin_lock(&file_lock_lock);
> -	*p = (*pos + 1);
> -	return seq_hlist_start(&file_lock_list, *pos);
> +	iter->li_pos = *pos + 1;
> +	lg_global_lock(&file_lock_lglock);
> +	return seq_hlist_start_percpu(&file_lock_list, &iter->li_cpu, *pos);
>  }
>  
>  static void *locks_next(struct seq_file *f, void *v, loff_t *pos)
>  {
> -	loff_t *p = f->private;
> -	++*p;
> -	return seq_hlist_next(v, &file_lock_list, pos);
> +	struct locks_iterator *iter = f->private;
> +
> +	++iter->li_pos;
> +	return seq_hlist_next_percpu(v, &file_lock_list, &iter->li_cpu, pos);
>  }
>  
>  static void locks_stop(struct seq_file *f, void *v)
>  {
> -	spin_unlock(&file_lock_lock);
> +	lg_global_unlock(&file_lock_lglock);
>  }
>  
>  static const struct seq_operations locks_seq_operations = {
> @@ -2348,7 +2361,8 @@ static const struct seq_operations locks_seq_operations = {
>  
>  static int locks_open(struct inode *inode, struct file *filp)
>  {
> -	return seq_open_private(filp, &locks_seq_operations, sizeof(loff_t));
> +	return seq_open_private(filp, &locks_seq_operations,
> +					sizeof(struct locks_iterator));
>  }
>  
>  static const struct file_operations proc_locks_operations = {
> @@ -2448,9 +2462,16 @@ EXPORT_SYMBOL(lock_may_write);
>  
>  static int __init filelock_init(void)
>  {
> +	int i;
> +
>  	filelock_cache = kmem_cache_create("file_lock_cache",
>  			sizeof(struct file_lock), 0, SLAB_PANIC, NULL);
>  
> +	lg_lock_init(&file_lock_lglock, "file_lock_lglock");
> +
> +	for_each_possible_cpu(i)
> +		INIT_HLIST_HEAD(per_cpu_ptr(&file_lock_list, i));
> +
>  	return 0;
>  }
>  
> diff --git a/include/linux/fs.h b/include/linux/fs.h
> index 232a345..18e59b8 100644
> --- a/include/linux/fs.h
> +++ b/include/linux/fs.h
> @@ -953,6 +953,7 @@ struct file_lock {
>  	unsigned int fl_flags;
>  	unsigned char fl_type;
>  	unsigned int fl_pid;
> +	int fl_link_cpu;		/* what cpu's list is this on? */
>  	struct pid *fl_nspid;
>  	wait_queue_head_t fl_wait;
>  	struct file *fl_file;
> -- 
> 1.7.1
> 


From bfields at fieldses.org  Thu Jun 13 15:45:46 2013
From: bfields at fieldses.org (J. Bruce Fields)
Date: Thu, 13 Jun 2013 15:45:46 -0400
Subject: [PATCH v2 06/14] locks: don't walk inode->i_flock list in
 locks_show
In-Reply-To: <1370948948-31784-7-git-send-email-jlayton@redhat.com>
References: <1370948948-31784-1-git-send-email-jlayton@redhat.com>
 <1370948948-31784-7-git-send-email-jlayton@redhat.com>
Message-ID: <20130613194545.GC19218@fieldses.org>

On Tue, Jun 11, 2013 at 07:09:00AM -0400, Jeff Layton wrote:
> When we convert over to using the i_lock to protect the i_flock list,
> that will introduce a potential lock inversion problem in locks_show.
> When we want to walk the i_flock list, we'll need to take the i_lock.
> 
> Rather than do that, just walk the global blocked_locks list and print
> out any that are blocked on the given lock.

I'm OK with this as obviously /proc/locks shouldn't be the common case,
but it still bugs me a bit that we're suddenly making it something like

	O(number of held locks * number of waiters)

where it used to be

	O(number of held lock + number of waiters)

I wonder if there's any solution that's just as easy and avoids scanning
the blocked list each time.

--b.

> 
> Signed-off-by: Jeff Layton <jlayton at redhat.com>
> ---
>  fs/locks.c |    6 ++++--
>  1 files changed, 4 insertions(+), 2 deletions(-)
> 
> diff --git a/fs/locks.c b/fs/locks.c
> index e451d18..3fd27f0 100644
> --- a/fs/locks.c
> +++ b/fs/locks.c
> @@ -2249,8 +2249,10 @@ static int locks_show(struct seq_file *f, void *v)
>  
>  	lock_get_status(f, fl, *((loff_t *)f->private), "");
>  
> -	list_for_each_entry(bfl, &fl->fl_block, fl_block)
> -		lock_get_status(f, bfl, *((loff_t *)f->private), " ->");
> +	list_for_each_entry(bfl, &blocked_list, fl_link) {
> +		if (bfl->fl_next == fl)
> +			lock_get_status(f, bfl, *((loff_t *)f->private), " ->");
> +	}
>  
>  	return 0;
>  }
> -- 
> 1.7.1
> 


From jlayton at redhat.com  Thu Jun 13 16:26:48 2013
From: jlayton at redhat.com (Jeff Layton)
Date: Thu, 13 Jun 2013 16:26:48 -0400
Subject: [PATCH v2 06/14] locks: don't walk inode->i_flock list in
 locks_show
In-Reply-To: <20130613194545.GC19218@fieldses.org>
References: <1370948948-31784-1-git-send-email-jlayton@redhat.com>
 <1370948948-31784-7-git-send-email-jlayton@redhat.com>
 <20130613194545.GC19218@fieldses.org>
Message-ID: <20130613162648.176979bc@tlielax.poochiereds.net>

On Thu, 13 Jun 2013 15:45:46 -0400
"J. Bruce Fields" <bfields at fieldses.org> wrote:

> On Tue, Jun 11, 2013 at 07:09:00AM -0400, Jeff Layton wrote:
> > When we convert over to using the i_lock to protect the i_flock list,
> > that will introduce a potential lock inversion problem in locks_show.
> > When we want to walk the i_flock list, we'll need to take the i_lock.
> > 
> > Rather than do that, just walk the global blocked_locks list and print
> > out any that are blocked on the given lock.
> 
> I'm OK with this as obviously /proc/locks shouldn't be the common case,
> but it still bugs me a bit that we're suddenly making it something like
> 
> 	O(number of held locks * number of waiters)
> 
> where it used to be
> 
> 	O(number of held lock + number of waiters)
> 
> I wonder if there's any solution that's just as easy and avoids scanning
> the blocked list each time.
> 
> --b.
> 
> > 
> > Signed-off-by: Jeff Layton <jlayton at redhat.com>
> > ---
> >  fs/locks.c |    6 ++++--
> >  1 files changed, 4 insertions(+), 2 deletions(-)
> > 
> > diff --git a/fs/locks.c b/fs/locks.c
> > index e451d18..3fd27f0 100644
> > --- a/fs/locks.c
> > +++ b/fs/locks.c
> > @@ -2249,8 +2249,10 @@ static int locks_show(struct seq_file *f, void *v)
> >  
> >  	lock_get_status(f, fl, *((loff_t *)f->private), "");
> >  
> > -	list_for_each_entry(bfl, &fl->fl_block, fl_block)
> > -		lock_get_status(f, bfl, *((loff_t *)f->private), " ->");
> > +	list_for_each_entry(bfl, &blocked_list, fl_link) {
> > +		if (bfl->fl_next == fl)
> > +			lock_get_status(f, bfl, *((loff_t *)f->private), " ->");
> > +	}
> >  
> >  	return 0;
> >  }
> > -- 
> > 1.7.1
> > 

Yeah, it's ugly, but I don't see a real alternative. We could try to
use RCU for this, but that slows down the list manipulation at the cost
of optimizing a rarely read procfile.

Now that I look though...it occurs to me that we have a problem here
anyway. Only blocked POSIX requests go onto that list currently, so
this misses seeing any blocked flock requests.

The only real solution I can think of is to put flock locks into the
blocked_list/blocked_hash too, or maybe giving them a simple hlist to
sit on.

I'll fix that up in the next iteration. It'll probably make flock()
tests run slower, but such is the cost of preserving this procfile...

-- 
Jeff Layton <jlayton at redhat.com>


From jlayton at redhat.com  Sat Jun 15 07:05:35 2013
From: jlayton at redhat.com (Jeff Layton)
Date: Sat, 15 Jun 2013 07:05:35 -0400
Subject: [PATCH v2 06/14] locks: don't walk inode->i_flock list in
 locks_show
In-Reply-To: <51BB040C.3050101@samba.org>
References: <1370948948-31784-1-git-send-email-jlayton@redhat.com>
 <1370948948-31784-7-git-send-email-jlayton@redhat.com>
 <20130613194545.GC19218@fieldses.org>
 <20130613162648.176979bc@tlielax.poochiereds.net>
 <51BB040C.3050101@samba.org>
Message-ID: <20130615070535.6367eed9@tlielax.poochiereds.net>

On Fri, 14 Jun 2013 07:52:44 -0400
Simo <idra at samba.org> wrote:

> On 06/13/2013 04:26 PM, Jeff Layton wrote:
> > The only real solution I can think of is to put flock locks into the
> > blocked_list/blocked_hash too, or maybe giving them a simple hlist to
> > sit on.
> >
> > I'll fix that up in the next iteration. It'll probably make flock()
> > tests run slower, but such is the cost of preserving this procfile...
> 
> How hard would it be to make the procfile stuff optional ?
> So that those that need performance can decide to not use it ?
> Maybe even something that can be disabled at run time ? Not just compile 
> time.
> 

(re-adding back the cc lists...)

It'd be tricky, especially if you want to do it at runtime. The
procfile itself is not a problem per-se. The real problem is the
tracking you have to do in order to eventually present the procfile. So
a boot-time or compile-time switch might be reasonable, but a runtime
switch will probably never really be.

I have a new patchset that I'm testing now though that should address
Bruce's concerns about iterating over that global list. So far, it
seems to be at least as fast as the latest patchset I posted.

It makes the (spin)locking a bit more complex, but hopefully I can
document this well enough that it's not a great concern.

Stay tuned...

-- 
Jeff Layton <jlayton at redhat.com>


From idra at samba.org  Sat Jun 15 11:04:11 2013
From: idra at samba.org (Simo)
Date: Sat, 15 Jun 2013 11:04:11 -0400
Subject: [PATCH v2 06/14] locks: don't walk inode->i_flock list in
 locks_show
In-Reply-To: <20130615070535.6367eed9@tlielax.poochiereds.net>
References: <1370948948-31784-1-git-send-email-jlayton@redhat.com>
 <1370948948-31784-7-git-send-email-jlayton@redhat.com>
 <20130613194545.GC19218@fieldses.org>
 <20130613162648.176979bc@tlielax.poochiereds.net>
 <51BB040C.3050101@samba.org>
 <20130615070535.6367eed9@tlielax.poochiereds.net>
Message-ID: <51BC826B.5060805@samba.org>

On 06/15/2013 07:05 AM, Jeff Layton wrote:
> On Fri, 14 Jun 2013 07:52:44 -0400
> Simo <idra at samba.org> wrote:
>
>> On 06/13/2013 04:26 PM, Jeff Layton wrote:
>>> The only real solution I can think of is to put flock locks into the
>>> blocked_list/blocked_hash too, or maybe giving them a simple hlist to
>>> sit on.
>>>
>>> I'll fix that up in the next iteration. It'll probably make flock()
>>> tests run slower, but such is the cost of preserving this procfile...
>> How hard would it be to make the procfile stuff optional ?
>> So that those that need performance can decide to not use it ?
>> Maybe even something that can be disabled at run time ? Not just compile
>> time.
>>
> (re-adding back the cc lists...)
>
> It'd be tricky, especially if you want to do it at runtime. The
> procfile itself is not a problem per-se. The real problem is the
> tracking you have to do in order to eventually present the procfile. So
> a boot-time or compile-time switch might be reasonable, but a runtime
> switch will probably never really be.

Just to be clear, I meant for a switch to turn it off at runtime, I 
understand very well that it would be way too hard to turn on at 
runtime. But killing the perf problem might be desirable on a system you 
cannot just reboot.

> I have a new patchset that I'm testing now though that should address
> Bruce's concerns about iterating over that global list. So far, it
> seems to be at least as fast as the latest patchset I posted.
>
> It makes the (spin)locking a bit more complex, but hopefully I can
> document this well enough that it's not a great concern.
>
> Stay tuned...

Thanks Jeff,
this is very valuable work.

Simo.



From jlayton at redhat.com  Mon Jun 17 11:13:45 2013
From: jlayton at redhat.com (Jeff Layton)
Date: Mon, 17 Jun 2013 11:13:45 -0400
Subject: [PATCH v3 02/13] locks: make generic_add_lease and
 generic_delete_lease static
In-Reply-To: <1371482036-15958-1-git-send-email-jlayton@redhat.com>
References: <1371482036-15958-1-git-send-email-jlayton@redhat.com>
Message-ID: <1371482036-15958-3-git-send-email-jlayton@redhat.com>

Signed-off-by: Jeff Layton <jlayton at redhat.com>
Acked-by: J. Bruce Fields <bfields at fieldses.org>
---
 fs/locks.c |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/fs/locks.c b/fs/locks.c
index 7a02064..e3140b8 100644
--- a/fs/locks.c
+++ b/fs/locks.c
@@ -1337,7 +1337,7 @@ int fcntl_getlease(struct file *filp)
 	return type;
 }
 
-int generic_add_lease(struct file *filp, long arg, struct file_lock **flp)
+static int generic_add_lease(struct file *filp, long arg, struct file_lock **flp)
 {
 	struct file_lock *fl, **before, **my_before = NULL, *lease;
 	struct dentry *dentry = filp->f_path.dentry;
@@ -1402,7 +1402,7 @@ out:
 	return error;
 }
 
-int generic_delete_lease(struct file *filp, struct file_lock **flp)
+static int generic_delete_lease(struct file *filp, struct file_lock **flp)
 {
 	struct file_lock *fl, **before;
 	struct dentry *dentry = filp->f_path.dentry;
-- 
1.7.1



From jlayton at redhat.com  Mon Jun 17 11:13:46 2013
From: jlayton at redhat.com (Jeff Layton)
Date: Mon, 17 Jun 2013 11:13:46 -0400
Subject: [PATCH v3 03/13] locks: comment cleanups and clarifications
In-Reply-To: <1371482036-15958-1-git-send-email-jlayton@redhat.com>
References: <1371482036-15958-1-git-send-email-jlayton@redhat.com>
Message-ID: <1371482036-15958-4-git-send-email-jlayton@redhat.com>

Signed-off-by: Jeff Layton <jlayton at redhat.com>
---
 fs/locks.c         |   21 +++++++++++++--------
 include/linux/fs.h |   18 ++++++++++++++++++
 2 files changed, 31 insertions(+), 8 deletions(-)

diff --git a/fs/locks.c b/fs/locks.c
index e3140b8..1e6301b 100644
--- a/fs/locks.c
+++ b/fs/locks.c
@@ -518,9 +518,10 @@ static void locks_insert_block(struct file_lock *blocker,
 		list_add(&waiter->fl_link, &blocked_list);
 }
 
-/* Wake up processes blocked waiting for blocker.
- * If told to wait then schedule the processes until the block list
- * is empty, otherwise empty the block list ourselves.
+/*
+ * Wake up processes blocked waiting for blocker.
+ *
+ * Must be called with the file_lock_lock held!
  */
 static void locks_wake_up_blocks(struct file_lock *blocker)
 {
@@ -806,6 +807,11 @@ static int __posix_lock_file(struct inode *inode, struct file_lock *request, str
 	}
 
 	lock_flocks();
+	/*
+	 * New lock request. Walk all POSIX locks and look for conflicts. If
+	 * there are any, either return error or put the request on the
+	 * blocker's list of waiters and the global blocked_list.
+	 */
 	if (request->fl_type != F_UNLCK) {
 		for_each_lock(inode, before) {
 			fl = *before;
@@ -844,7 +850,7 @@ static int __posix_lock_file(struct inode *inode, struct file_lock *request, str
 		before = &fl->fl_next;
 	}
 
-	/* Process locks with this owner.  */
+	/* Process locks with this owner. */
 	while ((fl = *before) && posix_same_owner(request, fl)) {
 		/* Detect adjacent or overlapping regions (if same lock type)
 		 */
@@ -930,10 +936,9 @@ static int __posix_lock_file(struct inode *inode, struct file_lock *request, str
 	}
 
 	/*
-	 * The above code only modifies existing locks in case of
-	 * merging or replacing.  If new lock(s) need to be inserted
-	 * all modifications are done bellow this, so it's safe yet to
-	 * bail out.
+	 * The above code only modifies existing locks in case of merging or
+	 * replacing. If new lock(s) need to be inserted all modifications are
+	 * done below this, so it's safe yet to bail out.
 	 */
 	error = -ENOLCK; /* "no luck" */
 	if (right && left == right && !new_fl2)
diff --git a/include/linux/fs.h b/include/linux/fs.h
index b9d7816..94105d2 100644
--- a/include/linux/fs.h
+++ b/include/linux/fs.h
@@ -926,6 +926,24 @@ int locks_in_grace(struct net *);
 /* that will die - we need it for nfs_lock_info */
 #include <linux/nfs_fs_i.h>
 
+/*
+ * struct file_lock represents a generic "file lock". It's used to represent
+ * POSIX byte range locks, BSD (flock) locks, and leases. It's important to
+ * note that the same struct is used to represent both a request for a lock and
+ * the lock itself, but the same object is never used for both.
+ *
+ * FIXME: should we create a separate "struct lock_request" to help distinguish
+ * these two uses?
+ *
+ * The i_flock list is ordered by:
+ *
+ * 1) lock type -- FL_LEASEs first, then FL_FLOCK, and finally FL_POSIX
+ * 2) lock owner
+ * 3) lock range start
+ * 4) lock range end
+ *
+ * Obviously, the last two criteria only matter for POSIX locks.
+ */
 struct file_lock {
 	struct file_lock *fl_next;	/* singly linked list for this inode  */
 	struct list_head fl_link;	/* doubly linked list of all locks */
-- 
1.7.1



From jlayton at redhat.com  Mon Jun 17 11:13:47 2013
From: jlayton at redhat.com (Jeff Layton)
Date: Mon, 17 Jun 2013 11:13:47 -0400
Subject: [PATCH v3 04/13] locks: make "added" in __posix_lock_file a bool
In-Reply-To: <1371482036-15958-1-git-send-email-jlayton@redhat.com>
References: <1371482036-15958-1-git-send-email-jlayton@redhat.com>
Message-ID: <1371482036-15958-5-git-send-email-jlayton@redhat.com>

...save 3 bytes of stack space.

Signed-off-by: Jeff Layton <jlayton at redhat.com>
Acked-by: J. Bruce Fields <bfields at fieldses.org>
---
 fs/locks.c |    9 +++++----
 1 files changed, 5 insertions(+), 4 deletions(-)

diff --git a/fs/locks.c b/fs/locks.c
index 1e6301b..c186649 100644
--- a/fs/locks.c
+++ b/fs/locks.c
@@ -791,7 +791,8 @@ static int __posix_lock_file(struct inode *inode, struct file_lock *request, str
 	struct file_lock *left = NULL;
 	struct file_lock *right = NULL;
 	struct file_lock **before;
-	int error, added = 0;
+	int error;
+	bool added = false;
 
 	/*
 	 * We may need two file_lock structures for this operation,
@@ -885,7 +886,7 @@ static int __posix_lock_file(struct inode *inode, struct file_lock *request, str
 				continue;
 			}
 			request = fl;
-			added = 1;
+			added = true;
 		}
 		else {
 			/* Processing for different lock types is a bit
@@ -896,7 +897,7 @@ static int __posix_lock_file(struct inode *inode, struct file_lock *request, str
 			if (fl->fl_start > request->fl_end)
 				break;
 			if (request->fl_type == F_UNLCK)
-				added = 1;
+				added = true;
 			if (fl->fl_start < request->fl_start)
 				left = fl;
 			/* If the next lock in the list has a higher end
@@ -926,7 +927,7 @@ static int __posix_lock_file(struct inode *inode, struct file_lock *request, str
 				locks_release_private(fl);
 				locks_copy_private(fl, request);
 				request = fl;
-				added = 1;
+				added = true;
 			}
 		}
 		/* Go on to next lock.
-- 
1.7.1



From jlayton at redhat.com  Mon Jun 17 11:13:44 2013
From: jlayton at redhat.com (Jeff Layton)
Date: Mon, 17 Jun 2013 11:13:44 -0400
Subject: [PATCH v3 01/13] cifs: use posix_unblock_lock instead of
 locks_delete_block
In-Reply-To: <1371482036-15958-1-git-send-email-jlayton@redhat.com>
References: <1371482036-15958-1-git-send-email-jlayton@redhat.com>
Message-ID: <1371482036-15958-2-git-send-email-jlayton@redhat.com>

commit 66189be74 (CIFS: Fix VFS lock usage for oplocked files) exported
the locks_delete_block symbol. There's already an exported helper
function that provides this capability however, so make cifs use that
instead and turn locks_delete_block back into a static function.

Note that if fl->fl_next == NULL then this lock has already been through
locks_delete_block(), so we should be OK to ignore an ENOENT error here
and simply not retry the lock.

Cc: Pavel Shilovsky <piastryyy at gmail.com>
Signed-off-by: Jeff Layton <jlayton at redhat.com>
Acked-by: J. Bruce Fields <bfields at fieldses.org>
---
 fs/cifs/file.c     |    2 +-
 fs/locks.c         |    3 +--
 include/linux/fs.h |    5 -----
 3 files changed, 2 insertions(+), 8 deletions(-)

diff --git a/fs/cifs/file.c b/fs/cifs/file.c
index 48b29d2..44a4f18 100644
--- a/fs/cifs/file.c
+++ b/fs/cifs/file.c
@@ -999,7 +999,7 @@ try_again:
 		rc = wait_event_interruptible(flock->fl_wait, !flock->fl_next);
 		if (!rc)
 			goto try_again;
-		locks_delete_block(flock);
+		posix_unblock_lock(file, flock);
 	}
 	return rc;
 }
diff --git a/fs/locks.c b/fs/locks.c
index cb424a4..7a02064 100644
--- a/fs/locks.c
+++ b/fs/locks.c
@@ -496,13 +496,12 @@ static void __locks_delete_block(struct file_lock *waiter)
 
 /*
  */
-void locks_delete_block(struct file_lock *waiter)
+static void locks_delete_block(struct file_lock *waiter)
 {
 	lock_flocks();
 	__locks_delete_block(waiter);
 	unlock_flocks();
 }
-EXPORT_SYMBOL(locks_delete_block);
 
 /* Insert waiter into blocker's block list.
  * We use a circular list so that processes can be easily woken up in
diff --git a/include/linux/fs.h b/include/linux/fs.h
index 43db02e..b9d7816 100644
--- a/include/linux/fs.h
+++ b/include/linux/fs.h
@@ -1006,7 +1006,6 @@ extern int vfs_setlease(struct file *, long, struct file_lock **);
 extern int lease_modify(struct file_lock **, int);
 extern int lock_may_read(struct inode *, loff_t start, unsigned long count);
 extern int lock_may_write(struct inode *, loff_t start, unsigned long count);
-extern void locks_delete_block(struct file_lock *waiter);
 extern void lock_flocks(void);
 extern void unlock_flocks(void);
 #else /* !CONFIG_FILE_LOCKING */
@@ -1151,10 +1150,6 @@ static inline int lock_may_write(struct inode *inode, loff_t start,
 	return 1;
 }
 
-static inline void locks_delete_block(struct file_lock *waiter)
-{
-}
-
 static inline void lock_flocks(void)
 {
 }
-- 
1.7.1



From jlayton at redhat.com  Mon Jun 17 11:13:48 2013
From: jlayton at redhat.com (Jeff Layton)
Date: Mon, 17 Jun 2013 11:13:48 -0400
Subject: [PATCH v3 05/13] locks: encapsulate the fl_link list handling
In-Reply-To: <1371482036-15958-1-git-send-email-jlayton@redhat.com>
References: <1371482036-15958-1-git-send-email-jlayton@redhat.com>
Message-ID: <1371482036-15958-6-git-send-email-jlayton@redhat.com>

Move the fl_link list handling routines into a separate set of helpers.
Also ensure that locks and requests are always put on global lists
last (after fully initializing them) and are taken off before unintializing
them.

Signed-off-by: Jeff Layton <jlayton at redhat.com>
---
 fs/locks.c |   45 ++++++++++++++++++++++++++++++++++++---------
 1 files changed, 36 insertions(+), 9 deletions(-)

diff --git a/fs/locks.c b/fs/locks.c
index c186649..c0e613f 100644
--- a/fs/locks.c
+++ b/fs/locks.c
@@ -153,13 +153,15 @@ int lease_break_time = 45;
 #define for_each_lock(inode, lockp) \
 	for (lockp = &inode->i_flock; *lockp != NULL; lockp = &(*lockp)->fl_next)
 
+/* The global file_lock_list is only used for displaying /proc/locks. */
 static LIST_HEAD(file_lock_list);
+
+/* The blocked_list is used to find POSIX lock loops for deadlock detection. */
 static LIST_HEAD(blocked_list);
+
+/* Protects the two list heads above, plus the inode->i_flock list */
 static DEFINE_SPINLOCK(file_lock_lock);
 
-/*
- * Protects the two list heads above, plus the inode->i_flock list
- */
 void lock_flocks(void)
 {
 	spin_lock(&file_lock_lock);
@@ -484,13 +486,37 @@ static int posix_same_owner(struct file_lock *fl1, struct file_lock *fl2)
 	return fl1->fl_owner == fl2->fl_owner;
 }
 
+static inline void
+locks_insert_global_locks(struct file_lock *waiter)
+{
+	list_add_tail(&waiter->fl_link, &file_lock_list);
+}
+
+static inline void
+locks_delete_global_locks(struct file_lock *waiter)
+{
+	list_del_init(&waiter->fl_link);
+}
+
+static inline void
+locks_insert_global_blocked(struct file_lock *waiter)
+{
+	list_add(&waiter->fl_link, &blocked_list);
+}
+
+static inline void
+locks_delete_global_blocked(struct file_lock *waiter)
+{
+	list_del_init(&waiter->fl_link);
+}
+
 /* Remove waiter from blocker's block list.
  * When blocker ends up pointing to itself then the list is empty.
  */
 static void __locks_delete_block(struct file_lock *waiter)
 {
+	locks_delete_global_blocked(waiter);
 	list_del_init(&waiter->fl_block);
-	list_del_init(&waiter->fl_link);
 	waiter->fl_next = NULL;
 }
 
@@ -512,10 +538,10 @@ static void locks_insert_block(struct file_lock *blocker,
 			       struct file_lock *waiter)
 {
 	BUG_ON(!list_empty(&waiter->fl_block));
-	list_add_tail(&waiter->fl_block, &blocker->fl_block);
 	waiter->fl_next = blocker;
+	list_add_tail(&waiter->fl_block, &blocker->fl_block);
 	if (IS_POSIX(blocker))
-		list_add(&waiter->fl_link, &blocked_list);
+		locks_insert_global_blocked(request);
 }
 
 /*
@@ -543,13 +569,13 @@ static void locks_wake_up_blocks(struct file_lock *blocker)
  */
 static void locks_insert_lock(struct file_lock **pos, struct file_lock *fl)
 {
-	list_add(&fl->fl_link, &file_lock_list);
-
 	fl->fl_nspid = get_pid(task_tgid(current));
 
 	/* insert into file's list */
 	fl->fl_next = *pos;
 	*pos = fl;
+
+	locks_insert_global_locks(fl);
 }
 
 /*
@@ -562,9 +588,10 @@ static void locks_delete_lock(struct file_lock **thisfl_p)
 {
 	struct file_lock *fl = *thisfl_p;
 
+	locks_delete_global_locks(fl);
+
 	*thisfl_p = fl->fl_next;
 	fl->fl_next = NULL;
-	list_del_init(&fl->fl_link);
 
 	if (fl->fl_nspid) {
 		put_pid(fl->fl_nspid);
-- 
1.7.1



From jlayton at redhat.com  Mon Jun 17 11:13:43 2013
From: jlayton at redhat.com (Jeff Layton)
Date: Mon, 17 Jun 2013 11:13:43 -0400
Subject: [PATCH v3 00/13] locks: scalability improvements for file locking
Message-ID: <1371482036-15958-1-git-send-email-jlayton@redhat.com>

Summary of Significant Changes:
-------------------------------
v3:
- Change spinlock handling to avoid the need to traverse the global
  blocked_hash when doing output of /proc/locks. This means that the
  fl_block list must continue to be protected by a global lock, but
  the fact that the i_lock is also held in most cases means that we
  can avoid taking it in certain situations.

v2:
- Fix potential races in deadlock detection. Manipulation of global
  blocked_hash and deadlock detection are now atomic. This is a
  little slower than the earlier set, but is provably correct. Also,
  the patch that converts to using the i_lock has been split out from
  most of the other changes. That should make it easier to review, but
  it does leave a potential race in the deadlock detection that is fixed
  up by the following patch. It may make sense to fold patches 7 and 8
  together before merging.

- Add percpu hlists and lglocks for global file_lock_list. This gives
  us some speedup since this list is seldom read.

Abstract (tl;dr version):
-------------------------
This patchset represents an overhaul of the file locking code with an
aim toward improving its scalability and making the code a bit easier to
understand.

Longer version:
---------------
When the BKL was finally ripped out of the kernel in 2010, the strategy
taken for the file locking code was to simply turn it into a new
file_lock_locks spinlock. It was an expedient way to deal with the file
locking code at the time, but having a giant spinlock around all of this
code is clearly not great for scalability. Red Hat has bug reports that
go back into the 2.6.18 era that point to BKL scalability problems in
the file locking code and the file_lock_lock suffers from the same
issues.

This patchset is my first attempt to make this code less dependent on
global locking. The main change is to switch most of the file locking
code to be protected by the inode->i_lock instead of the file_lock_lock.

While that works for most things, there are a couple of global data
structures (lists in the current code) that need a global lock to
protect them. So we still need a global lock in order to deal with
those. The remaining patches are intended to make that global locking
less painful. The big gains are made by turning the blocked_list into a
hashtable, which greatly speeds up the deadlock detection code and
making the file_lock_list percpu.

This is not the first attempt at doing this. The conversion to the
i_lock was originally attempted by Bruce Fields a few years ago. His
approach was NAK'ed since it involved ripping out the deadlock
detection. People also really seem to like /proc/locks for debugging, so
keeping that in is probably worthwhile.

There's more work to be done in this area and this patchset is just a
start. There's a horrible thundering herd problem when a blocking lock
is released, for instance. There was also interest in solving the goofy
"unlock on any close" POSIX lock semantics at this year's LSF. I think
this patchset will help lay the groundwork for those changes as well.

While file locking is not usually considered to be a high-performance
codepath, it *is* an IPC mechanism and I think it behooves us to try to
make it as fast as possible.

I'd like to see this considered for 3.11, but some soak time in -next
would be good. Comments and suggestions welcome.

Performance testing and results:
--------------------------------
In order to measure the benefit of this set, I've written some locking
performance tests that I've made available here:

    git://git.samba.org/jlayton/lockperf.git

Here are the results from the same 32-way, 4 NUMA node machine that I
used to generate the v2 patch results. The first number is the mean
time spent in locking for the test. The number in parenthesis is the
standard deviation.

		3.10.0-rc5-00219-ga2648eb	3.10.0-rc5-00231-g7569869
---------------------------------------------------------------------------
flock01		24119.96 (266.08)		24542.51 (254.89)
flock02		 1345.09  (37.37)		    8.60   (0.31)
posix01		31217.14 (320.91)		24899.20 (254.27)
posix02		 1348.60  (36.83)		   12.70   (0.44)

I wasn't able to reserve the exact same smaller machine for testing this
set, but this one is comparable with 4 CPUs and UMA architecture:

		3.10.0-rc5-00219-ga2648eb	3.10.0-rc5-00231-g7569869
---------------------------------------------------------------------------
flock01		1787.51 (11.23)			1797.75  (9.27)
flock02		 314.90	 (8.84)			  34.87  (2.82)
posix01		1843.43 (11.63)			1880.47 (13.47)
posix02		 325.13  (8.53)			  54.09  (4.02)

I think the conclusion we can draw here is that this patchset it roughly
as fast as the previous one. In addition, the posix02 test saw a vast
increase in performance.

I believe that's mostly due to the fact that with this set I added a
patch that allows the code to avoid taking the global blocked_lock_lock
when waking up waiters if there aren't any. With that, the
blocked_lock_lock never has to be taken at all if there's no contention
for the file_lock (as is the case in the posix02 and flock02 tests).

Jeff Layton (13):
  cifs: use posix_unblock_lock instead of locks_delete_block
  locks: make generic_add_lease and generic_delete_lease static
  locks: comment cleanups and clarifications
  locks: make "added" in __posix_lock_file a bool
  locks: encapsulate the fl_link list handling
  locks: protect most of the file_lock handling with i_lock
  locks: avoid taking global lock if possible when waking up blocked
    waiters
  locks: convert fl_link to a hlist_node
  locks: turn the blocked_list into a hashtable
  locks: add a new "lm_owner_key" lock operation
  locks: give the blocked_hash its own spinlock
  seq_file: add seq_list_*_percpu helpers
  locks: move file_lock_list to a set of percpu hlist_heads and convert
    file_lock_lock to an lglock

 Documentation/filesystems/Locking |   27 +++-
 fs/afs/flock.c                    |    5 +-
 fs/ceph/locks.c                   |    2 +-
 fs/ceph/mds_client.c              |    8 +-
 fs/cifs/cifsfs.c                  |    2 +-
 fs/cifs/file.c                    |   15 +-
 fs/gfs2/file.c                    |    2 +-
 fs/lockd/svclock.c                |   12 ++
 fs/lockd/svcsubs.c                |   12 +-
 fs/locks.c                        |  310 +++++++++++++++++++++++++-----------
 fs/nfs/delegation.c               |   10 +-
 fs/nfs/nfs4state.c                |    8 +-
 fs/nfsd/nfs4state.c               |    8 +-
 fs/seq_file.c                     |   54 +++++++
 include/linux/fs.h                |   38 +++--
 include/linux/seq_file.h          |    6 +
 16 files changed, 365 insertions(+), 154 deletions(-)



From jlayton at redhat.com  Mon Jun 17 11:13:51 2013
From: jlayton at redhat.com (Jeff Layton)
Date: Mon, 17 Jun 2013 11:13:51 -0400
Subject: [PATCH v3 08/13] locks: convert fl_link to a hlist_node
In-Reply-To: <1371482036-15958-1-git-send-email-jlayton@redhat.com>
References: <1371482036-15958-1-git-send-email-jlayton@redhat.com>
Message-ID: <1371482036-15958-9-git-send-email-jlayton@redhat.com>

Testing has shown that iterating over the blocked_list for deadlock
detection turns out to be a bottleneck. In order to alleviate that,
begin the process of turning it into a hashtable. We start by turning
the fl_link into a hlist_node and the global lists into hlists. A later
patch will do the conversion of the blocked_list to a hashtable.

Signed-off-by: Jeff Layton <jlayton at redhat.com>
Acked-by: J. Bruce Fields <bfields at fieldses.org>
---
 fs/locks.c         |   24 ++++++++++++------------
 include/linux/fs.h |    2 +-
 2 files changed, 13 insertions(+), 13 deletions(-)

diff --git a/fs/locks.c b/fs/locks.c
index a8f3b33..32826ed 100644
--- a/fs/locks.c
+++ b/fs/locks.c
@@ -157,13 +157,13 @@ int lease_break_time = 45;
  * The global file_lock_list is only used for displaying /proc/locks. Protected
  * by the file_lock_lock.
  */
-static LIST_HEAD(file_lock_list);
+static HLIST_HEAD(file_lock_list);
 
 /*
  * The blocked_list is used to find POSIX lock loops for deadlock detection.
  * Protected by file_lock_lock.
  */
-static LIST_HEAD(blocked_list);
+static HLIST_HEAD(blocked_list);
 
 /* Protects the two list heads above, and fl->fl_block list. */
 static DEFINE_SPINLOCK(file_lock_lock);
@@ -172,7 +172,7 @@ static struct kmem_cache *filelock_cache __read_mostly;
 
 static void locks_init_lock_heads(struct file_lock *fl)
 {
-	INIT_LIST_HEAD(&fl->fl_link);
+	INIT_HLIST_NODE(&fl->fl_link);
 	INIT_LIST_HEAD(&fl->fl_block);
 	init_waitqueue_head(&fl->fl_wait);
 }
@@ -206,7 +206,7 @@ void locks_free_lock(struct file_lock *fl)
 {
 	BUG_ON(waitqueue_active(&fl->fl_wait));
 	BUG_ON(!list_empty(&fl->fl_block));
-	BUG_ON(!list_empty(&fl->fl_link));
+	BUG_ON(!hlist_unhashed(&fl->fl_link));
 
 	locks_release_private(fl);
 	kmem_cache_free(filelock_cache, fl);
@@ -484,7 +484,7 @@ static inline void
 locks_insert_global_locks(struct file_lock *waiter)
 {
 	spin_lock(&file_lock_lock);
-	list_add_tail(&waiter->fl_link, &file_lock_list);
+	hlist_add_head(&waiter->fl_link, &file_lock_list);
 	spin_unlock(&file_lock_lock);
 }
 
@@ -492,20 +492,20 @@ static inline void
 locks_delete_global_locks(struct file_lock *waiter)
 {
 	spin_lock(&file_lock_lock);
-	list_del_init(&waiter->fl_link);
+	hlist_del_init(&waiter->fl_link);
 	spin_unlock(&file_lock_lock);
 }
 
 static inline void
 locks_insert_global_blocked(struct file_lock *waiter)
 {
-	list_add(&waiter->fl_link, &blocked_list);
+	hlist_add_head(&waiter->fl_link, &blocked_list);
 }
 
 static inline void
 locks_delete_global_blocked(struct file_lock *waiter)
 {
-	list_del_init(&waiter->fl_link);
+	hlist_del_init(&waiter->fl_link);
 }
 
 /* Remove waiter from blocker's block list.
@@ -730,7 +730,7 @@ static struct file_lock *what_owner_is_waiting_for(struct file_lock *block_fl)
 {
 	struct file_lock *fl;
 
-	list_for_each_entry(fl, &blocked_list, fl_link) {
+	hlist_for_each_entry(fl, &blocked_list, fl_link) {
 		if (posix_same_owner(fl, block_fl))
 			return fl->fl_next;
 	}
@@ -2285,7 +2285,7 @@ static int locks_show(struct seq_file *f, void *v)
 {
 	struct file_lock *fl, *bfl;
 
-	fl = list_entry(v, struct file_lock, fl_link);
+	fl = hlist_entry(v, struct file_lock, fl_link);
 
 	lock_get_status(f, fl, *((loff_t *)f->private), "");
 
@@ -2301,14 +2301,14 @@ static void *locks_start(struct seq_file *f, loff_t *pos)
 
 	spin_lock(&file_lock_lock);
 	*p = (*pos + 1);
-	return seq_list_start(&file_lock_list, *pos);
+	return seq_hlist_start(&file_lock_list, *pos);
 }
 
 static void *locks_next(struct seq_file *f, void *v, loff_t *pos)
 {
 	loff_t *p = f->private;
 	++*p;
-	return seq_list_next(v, &file_lock_list, pos);
+	return seq_hlist_next(v, &file_lock_list, pos);
 }
 
 static void locks_stop(struct seq_file *f, void *v)
diff --git a/include/linux/fs.h b/include/linux/fs.h
index e2f896d..3b340f7 100644
--- a/include/linux/fs.h
+++ b/include/linux/fs.h
@@ -946,7 +946,7 @@ int locks_in_grace(struct net *);
  */
 struct file_lock {
 	struct file_lock *fl_next;	/* singly linked list for this inode  */
-	struct list_head fl_link;	/* doubly linked list of all locks */
+	struct hlist_node fl_link;	/* node in global lists */
 	struct list_head fl_block;	/* circular list of blocked processes */
 	fl_owner_t fl_owner;
 	unsigned int fl_flags;
-- 
1.7.1



From jlayton at redhat.com  Mon Jun 17 11:13:52 2013
From: jlayton at redhat.com (Jeff Layton)
Date: Mon, 17 Jun 2013 11:13:52 -0400
Subject: [PATCH v3 09/13] locks: turn the blocked_list into a hashtable
In-Reply-To: <1371482036-15958-1-git-send-email-jlayton@redhat.com>
References: <1371482036-15958-1-git-send-email-jlayton@redhat.com>
Message-ID: <1371482036-15958-10-git-send-email-jlayton@redhat.com>

Break up the blocked_list into a hashtable, using the fl_owner as a key.
This speeds up searching the hash chains, which is especially significant
for deadlock detection.

Note that the initial implementation assumes that hashing on fl_owner is
sufficient. In most cases it should be, with the notable exception being
server-side lockd, which compares ownership using a tuple of the
nlm_host and the pid sent in the lock request. So, this may degrade to a
single hash bucket when you only have a single NFS client. That will be
addressed in a later patch.

The careful observer may note that this patch leaves the file_lock_list
alone. There's much less of a case for turning the file_lock_list into a
hashtable. The only user of that list is the code that generates
/proc/locks, and it always walks the entire list.

Signed-off-by: Jeff Layton <jlayton at redhat.com>
Acked-by: J. Bruce Fields <bfields at fieldses.org>
---
 fs/locks.c |   25 +++++++++++++++++--------
 1 files changed, 17 insertions(+), 8 deletions(-)

diff --git a/fs/locks.c b/fs/locks.c
index 32826ed..d93b291 100644
--- a/fs/locks.c
+++ b/fs/locks.c
@@ -126,6 +126,7 @@
 #include <linux/time.h>
 #include <linux/rcupdate.h>
 #include <linux/pid_namespace.h>
+#include <linux/hashtable.h>
 
 #include <asm/uaccess.h>
 
@@ -160,12 +161,20 @@ int lease_break_time = 45;
 static HLIST_HEAD(file_lock_list);
 
 /*
- * The blocked_list is used to find POSIX lock loops for deadlock detection.
- * Protected by file_lock_lock.
+ * The blocked_hash is used to find POSIX lock loops for deadlock detection.
+ * It is protected by file_lock_lock.
+ *
+ * We hash locks by lockowner in order to optimize searching for the lock a
+ * particular lockowner is waiting on.
+ *
+ * FIXME: make this value scale via some heuristic? We generally will want more
+ * buckets when we have more lockowners holding locks, but that's a little
+ * difficult to determine without knowing what the workload will look like.
  */
-static HLIST_HEAD(blocked_list);
+#define BLOCKED_HASH_BITS	7
+static DEFINE_HASHTABLE(blocked_hash, BLOCKED_HASH_BITS);
 
-/* Protects the two list heads above, and fl->fl_block list. */
+/* Protects the file_lock_list, the blocked_hash and fl->fl_block list */
 static DEFINE_SPINLOCK(file_lock_lock);
 
 static struct kmem_cache *filelock_cache __read_mostly;
@@ -499,13 +508,13 @@ locks_delete_global_locks(struct file_lock *waiter)
 static inline void
 locks_insert_global_blocked(struct file_lock *waiter)
 {
-	hlist_add_head(&waiter->fl_link, &blocked_list);
+	hash_add(blocked_hash, &waiter->fl_link, (unsigned long)waiter->fl_owner);
 }
 
 static inline void
 locks_delete_global_blocked(struct file_lock *waiter)
 {
-	hlist_del_init(&waiter->fl_link);
+	hash_del(&waiter->fl_link);
 }
 
 /* Remove waiter from blocker's block list.
@@ -730,7 +739,7 @@ static struct file_lock *what_owner_is_waiting_for(struct file_lock *block_fl)
 {
 	struct file_lock *fl;
 
-	hlist_for_each_entry(fl, &blocked_list, fl_link) {
+	hash_for_each_possible(blocked_hash, fl, fl_link, (unsigned long)block_fl->fl_owner) {
 		if (posix_same_owner(fl, block_fl))
 			return fl->fl_next;
 	}
@@ -866,7 +875,7 @@ static int __posix_lock_file(struct inode *inode, struct file_lock *request, str
 	/*
 	 * New lock request. Walk all POSIX locks and look for conflicts. If
 	 * there are any, either return error or put the request on the
-	 * blocker's list of waiters and the global blocked_list.
+	 * blocker's list of waiters and the global blocked_hash.
 	 */
 	if (request->fl_type != F_UNLCK) {
 		for_each_lock(inode, before) {
-- 
1.7.1



From jlayton at redhat.com  Mon Jun 17 11:13:49 2013
From: jlayton at redhat.com (Jeff Layton)
Date: Mon, 17 Jun 2013 11:13:49 -0400
Subject: [PATCH v3 06/13] locks: protect most of the file_lock handling with
 i_lock
In-Reply-To: <1371482036-15958-1-git-send-email-jlayton@redhat.com>
References: <1371482036-15958-1-git-send-email-jlayton@redhat.com>
Message-ID: <1371482036-15958-7-git-send-email-jlayton@redhat.com>

Having a global lock that protects all of this code is a clear
scalability problem. Instead of doing that, move most of the code to be
protected by the i_lock instead. The exceptions are the global lists
that the ->fl_link sits on, and the ->fl_block list.

->fl_link is what connects these structures to the
global lists, so we must ensure that we hold those locks when iterating
over or updating these lists.

Furthermore, sound deadlock detection requires that we hold the
blocked_list state steady while checking for loops. We also must ensure
that the search and update to the list are atomic.

For the checking and insertion side of the blocked_list, push the
acquisition of the global lock into __posix_lock_file and ensure that
checking and update of the  blocked_list is done without dropping the
lock in between.

On the removal side, when waking up blocked lock waiters, take the
global lock before walking the blocked list and dequeue the waiters from
the global list prior to removal from the fl_block list.

With this, deadlock detection should be race free while we minimize
excessive file_lock_lock thrashing.

Finally, in order to avoid a lock inversion problem when handling
/proc/locks output we must ensure that manipulations of the fl_block
list are also protected by the file_lock_lock.

Signed-off-by: Jeff Layton <jlayton at redhat.com>
---
 Documentation/filesystems/Locking |   21 ++++--
 fs/afs/flock.c                    |    5 +-
 fs/ceph/locks.c                   |    2 +-
 fs/ceph/mds_client.c              |    8 +-
 fs/cifs/cifsfs.c                  |    2 +-
 fs/cifs/file.c                    |   13 ++--
 fs/gfs2/file.c                    |    2 +-
 fs/lockd/svcsubs.c                |   12 ++--
 fs/locks.c                        |  151 ++++++++++++++++++++++---------------
 fs/nfs/delegation.c               |   10 +-
 fs/nfs/nfs4state.c                |    8 +-
 fs/nfsd/nfs4state.c               |    8 +-
 include/linux/fs.h                |   11 ---
 13 files changed, 140 insertions(+), 113 deletions(-)

diff --git a/Documentation/filesystems/Locking b/Documentation/filesystems/Locking
index 0706d32..413685f 100644
--- a/Documentation/filesystems/Locking
+++ b/Documentation/filesystems/Locking
@@ -344,7 +344,7 @@ prototypes:
 
 
 locking rules:
-			file_lock_lock	may block
+			inode->i_lock	may block
 fl_copy_lock:		yes		no
 fl_release_private:	maybe		no
 
@@ -357,12 +357,19 @@ prototypes:
 	int (*lm_change)(struct file_lock **, int);
 
 locking rules:
-			file_lock_lock	may block
-lm_compare_owner:	yes		no
-lm_notify:		yes		no
-lm_grant:		no		no
-lm_break:		yes		no
-lm_change		yes		no
+
+			inode->i_lock	file_lock_lock	may block
+lm_compare_owner:	yes[1]		maybe		no
+lm_notify:		yes		yes		no
+lm_grant:		no		no		no
+lm_break:		yes		no		no
+lm_change		yes		no		no
+
+[1]:	->lm_compare_owner is generally called with *an* inode->i_lock held. It
+may not be the i_lock of the inode for either file_lock being compared! This is
+the case with deadlock detection, since the code has to chase down the owners
+of locks that may be entirely unrelated to the one on which the lock is being
+acquired. When doing a search for deadlocks, the file_lock_lock is also held.
 
 --------------------------- buffer_head -----------------------------------
 prototypes:
diff --git a/fs/afs/flock.c b/fs/afs/flock.c
index 2497bf3..03fc0d1 100644
--- a/fs/afs/flock.c
+++ b/fs/afs/flock.c
@@ -252,6 +252,7 @@ static void afs_defer_unlock(struct afs_vnode *vnode, struct key *key)
  */
 static int afs_do_setlk(struct file *file, struct file_lock *fl)
 {
+	struct inode = file_inode(file);
 	struct afs_vnode *vnode = AFS_FS_I(file->f_mapping->host);
 	afs_lock_type_t type;
 	struct key *key = file->private_data;
@@ -273,7 +274,7 @@ static int afs_do_setlk(struct file *file, struct file_lock *fl)
 
 	type = (fl->fl_type == F_RDLCK) ? AFS_LOCK_READ : AFS_LOCK_WRITE;
 
-	lock_flocks();
+	spin_lock(&inode->i_lock);
 
 	/* make sure we've got a callback on this file and that our view of the
 	 * data version is up to date */
@@ -420,7 +421,7 @@ given_lock:
 	afs_vnode_fetch_status(vnode, NULL, key);
 
 error:
-	unlock_flocks();
+	spin_unlock(&inode->i_lock);
 	_leave(" = %d", ret);
 	return ret;
 
diff --git a/fs/ceph/locks.c b/fs/ceph/locks.c
index ebbf680..690f73f 100644
--- a/fs/ceph/locks.c
+++ b/fs/ceph/locks.c
@@ -192,7 +192,7 @@ void ceph_count_locks(struct inode *inode, int *fcntl_count, int *flock_count)
 
 /**
  * Encode the flock and fcntl locks for the given inode into the ceph_filelock
- * array. Must be called with lock_flocks() already held.
+ * array. Must be called with inode->i_lock already held.
  * If we encounter more of a specific lock type than expected, return -ENOSPC.
  */
 int ceph_encode_locks_to_buffer(struct inode *inode,
diff --git a/fs/ceph/mds_client.c b/fs/ceph/mds_client.c
index 4d29203..74fd289 100644
--- a/fs/ceph/mds_client.c
+++ b/fs/ceph/mds_client.c
@@ -2481,20 +2481,20 @@ static int encode_caps_cb(struct inode *inode, struct ceph_cap *cap,
 		struct ceph_filelock *flocks;
 
 encode_again:
-		lock_flocks();
+		spin_lock(&inode->i_lock);
 		ceph_count_locks(inode, &num_fcntl_locks, &num_flock_locks);
-		unlock_flocks();
+		spin_unlock(&inode->i_lock);
 		flocks = kmalloc((num_fcntl_locks+num_flock_locks) *
 				 sizeof(struct ceph_filelock), GFP_NOFS);
 		if (!flocks) {
 			err = -ENOMEM;
 			goto out_free;
 		}
-		lock_flocks();
+		spin_lock(&inode->i_lock);
 		err = ceph_encode_locks_to_buffer(inode, flocks,
 						  num_fcntl_locks,
 						  num_flock_locks);
-		unlock_flocks();
+		spin_unlock(&inode->i_lock);
 		if (err) {
 			kfree(flocks);
 			if (err == -ENOSPC)
diff --git a/fs/cifs/cifsfs.c b/fs/cifs/cifsfs.c
index 3752b9f..e81655c 100644
--- a/fs/cifs/cifsfs.c
+++ b/fs/cifs/cifsfs.c
@@ -765,7 +765,7 @@ static loff_t cifs_llseek(struct file *file, loff_t offset, int whence)
 
 static int cifs_setlease(struct file *file, long arg, struct file_lock **lease)
 {
-	/* note that this is called by vfs setlease with lock_flocks held
+	/* note that this is called by vfs setlease with i_lock held
 	   to protect *lease from going away */
 	struct inode *inode = file_inode(file);
 	struct cifsFileInfo *cfile = file->private_data;
diff --git a/fs/cifs/file.c b/fs/cifs/file.c
index 44a4f18..0dd10cd 100644
--- a/fs/cifs/file.c
+++ b/fs/cifs/file.c
@@ -1092,6 +1092,7 @@ struct lock_to_push {
 static int
 cifs_push_posix_locks(struct cifsFileInfo *cfile)
 {
+	struct inode *inode = cfile->dentry->d_inode;
 	struct cifs_tcon *tcon = tlink_tcon(cfile->tlink);
 	struct file_lock *flock, **before;
 	unsigned int count = 0, i = 0;
@@ -1102,12 +1103,12 @@ cifs_push_posix_locks(struct cifsFileInfo *cfile)
 
 	xid = get_xid();
 
-	lock_flocks();
-	cifs_for_each_lock(cfile->dentry->d_inode, before) {
+	spin_lock(&inode->i_lock);
+	cifs_for_each_lock(inode, before) {
 		if ((*before)->fl_flags & FL_POSIX)
 			count++;
 	}
-	unlock_flocks();
+	spin_unlock(&inode->i_lock);
 
 	INIT_LIST_HEAD(&locks_to_send);
 
@@ -1126,8 +1127,8 @@ cifs_push_posix_locks(struct cifsFileInfo *cfile)
 	}
 
 	el = locks_to_send.next;
-	lock_flocks();
-	cifs_for_each_lock(cfile->dentry->d_inode, before) {
+	spin_lock(&inode->i_lock);
+	cifs_for_each_lock(inode, before) {
 		flock = *before;
 		if ((flock->fl_flags & FL_POSIX) == 0)
 			continue;
@@ -1152,7 +1153,7 @@ cifs_push_posix_locks(struct cifsFileInfo *cfile)
 		lck->offset = flock->fl_start;
 		el = el->next;
 	}
-	unlock_flocks();
+	spin_unlock(&inode->i_lock);
 
 	list_for_each_entry_safe(lck, tmp, &locks_to_send, llist) {
 		int stored_rc;
diff --git a/fs/gfs2/file.c b/fs/gfs2/file.c
index ad0dc38..2398b41 100644
--- a/fs/gfs2/file.c
+++ b/fs/gfs2/file.c
@@ -896,7 +896,7 @@ out_uninit:
  * cluster; until we do, disable leases (by just returning -EINVAL),
  * unless the administrator has requested purely local locking.
  *
- * Locking: called under lock_flocks
+ * Locking: called under i_lock
  *
  * Returns: errno
  */
diff --git a/fs/lockd/svcsubs.c b/fs/lockd/svcsubs.c
index 97e8741..dc5c759 100644
--- a/fs/lockd/svcsubs.c
+++ b/fs/lockd/svcsubs.c
@@ -169,7 +169,7 @@ nlm_traverse_locks(struct nlm_host *host, struct nlm_file *file,
 
 again:
 	file->f_locks = 0;
-	lock_flocks(); /* protects i_flock list */
+	spin_lock(&inode->i_lock);
 	for (fl = inode->i_flock; fl; fl = fl->fl_next) {
 		if (fl->fl_lmops != &nlmsvc_lock_operations)
 			continue;
@@ -181,7 +181,7 @@ again:
 		if (match(lockhost, host)) {
 			struct file_lock lock = *fl;
 
-			unlock_flocks();
+			spin_unlock(&inode->i_lock);
 			lock.fl_type  = F_UNLCK;
 			lock.fl_start = 0;
 			lock.fl_end   = OFFSET_MAX;
@@ -193,7 +193,7 @@ again:
 			goto again;
 		}
 	}
-	unlock_flocks();
+	spin_unlock(&inode->i_lock);
 
 	return 0;
 }
@@ -228,14 +228,14 @@ nlm_file_inuse(struct nlm_file *file)
 	if (file->f_count || !list_empty(&file->f_blocks) || file->f_shares)
 		return 1;
 
-	lock_flocks();
+	spin_lock(&inode->i_lock);
 	for (fl = inode->i_flock; fl; fl = fl->fl_next) {
 		if (fl->fl_lmops == &nlmsvc_lock_operations) {
-			unlock_flocks();
+			spin_unlock(&inode->i_lock);
 			return 1;
 		}
 	}
-	unlock_flocks();
+	spin_unlock(&inode->i_lock);
 	file->f_locks = 0;
 	return 0;
 }
diff --git a/fs/locks.c b/fs/locks.c
index c0e613f..8f56651 100644
--- a/fs/locks.c
+++ b/fs/locks.c
@@ -153,27 +153,21 @@ int lease_break_time = 45;
 #define for_each_lock(inode, lockp) \
 	for (lockp = &inode->i_flock; *lockp != NULL; lockp = &(*lockp)->fl_next)
 
-/* The global file_lock_list is only used for displaying /proc/locks. */
+/*
+ * The global file_lock_list is only used for displaying /proc/locks. Protected
+ * by the file_lock_lock.
+ */
 static LIST_HEAD(file_lock_list);
 
-/* The blocked_list is used to find POSIX lock loops for deadlock detection. */
+/*
+ * The blocked_list is used to find POSIX lock loops for deadlock detection.
+ * Protected by file_lock_lock.
+ */
 static LIST_HEAD(blocked_list);
 
-/* Protects the two list heads above, plus the inode->i_flock list */
+/* Protects the two list heads above, and fl->fl_block list. */
 static DEFINE_SPINLOCK(file_lock_lock);
 
-void lock_flocks(void)
-{
-	spin_lock(&file_lock_lock);
-}
-EXPORT_SYMBOL_GPL(lock_flocks);
-
-void unlock_flocks(void)
-{
-	spin_unlock(&file_lock_lock);
-}
-EXPORT_SYMBOL_GPL(unlock_flocks);
-
 static struct kmem_cache *filelock_cache __read_mostly;
 
 static void locks_init_lock_heads(struct file_lock *fl)
@@ -489,13 +483,17 @@ static int posix_same_owner(struct file_lock *fl1, struct file_lock *fl2)
 static inline void
 locks_insert_global_locks(struct file_lock *waiter)
 {
+	spin_lock(&file_lock_lock);
 	list_add_tail(&waiter->fl_link, &file_lock_list);
+	spin_unlock(&file_lock_lock);
 }
 
 static inline void
 locks_delete_global_locks(struct file_lock *waiter)
 {
+	spin_lock(&file_lock_lock);
 	list_del_init(&waiter->fl_link);
+	spin_unlock(&file_lock_lock);
 }
 
 static inline void
@@ -512,6 +510,8 @@ locks_delete_global_blocked(struct file_lock *waiter)
 
 /* Remove waiter from blocker's block list.
  * When blocker ends up pointing to itself then the list is empty.
+ *
+ * Must be called with file_lock_lock held.
  */
 static void __locks_delete_block(struct file_lock *waiter)
 {
@@ -520,37 +520,47 @@ static void __locks_delete_block(struct file_lock *waiter)
 	waiter->fl_next = NULL;
 }
 
-/*
- */
 static void locks_delete_block(struct file_lock *waiter)
 {
-	lock_flocks();
+	spin_lock(&file_lock_lock);
 	__locks_delete_block(waiter);
-	unlock_flocks();
+	spin_unlock(&file_lock_lock);
 }
 
 /* Insert waiter into blocker's block list.
  * We use a circular list so that processes can be easily woken up in
  * the order they blocked. The documentation doesn't require this but
  * it seems like the reasonable thing to do.
+ *
+ * Must be called with file_lock_lock held!
  */
-static void locks_insert_block(struct file_lock *blocker, 
-			       struct file_lock *waiter)
+static void __locks_insert_block(struct file_lock *blocker,
+					struct file_lock *waiter)
 {
 	BUG_ON(!list_empty(&waiter->fl_block));
 	waiter->fl_next = blocker;
 	list_add_tail(&waiter->fl_block, &blocker->fl_block);
 	if (IS_POSIX(blocker))
-		locks_insert_global_blocked(request);
+		locks_insert_global_blocked(waiter);
+}
+
+/* Must be called with i_lock held. */
+static void locks_insert_block(struct file_lock *blocker,
+					struct file_lock *waiter)
+{
+	spin_lock(&file_lock_lock);
+	__locks_insert_block(blocker, waiter);
+	spin_unlock(&file_lock_lock);
 }
 
 /*
  * Wake up processes blocked waiting for blocker.
  *
- * Must be called with the file_lock_lock held!
+ * Must be called with the inode->i_lock held!
  */
 static void locks_wake_up_blocks(struct file_lock *blocker)
 {
+	spin_lock(&file_lock_lock);
 	while (!list_empty(&blocker->fl_block)) {
 		struct file_lock *waiter;
 
@@ -562,10 +572,13 @@ static void locks_wake_up_blocks(struct file_lock *blocker)
 		else
 			wake_up(&waiter->fl_wait);
 	}
+	spin_unlock(&file_lock_lock);
 }
 
 /* Insert file lock fl into an inode's lock list at the position indicated
  * by pos. At the same time add the lock to the global file lock list.
+ *
+ * Must be called with the i_lock held!
  */
 static void locks_insert_lock(struct file_lock **pos, struct file_lock *fl)
 {
@@ -583,6 +596,8 @@ static void locks_insert_lock(struct file_lock **pos, struct file_lock *fl)
  * Wake up processes that are blocked waiting for this lock,
  * notify the FS that the lock has been cleared and
  * finally free the lock.
+ *
+ * Must be called with the i_lock held!
  */
 static void locks_delete_lock(struct file_lock **thisfl_p)
 {
@@ -652,8 +667,9 @@ void
 posix_test_lock(struct file *filp, struct file_lock *fl)
 {
 	struct file_lock *cfl;
+	struct inode *inode = file_inode(filp);
 
-	lock_flocks();
+	spin_lock(&inode->i_lock);
 	for (cfl = file_inode(filp)->i_flock; cfl; cfl = cfl->fl_next) {
 		if (!IS_POSIX(cfl))
 			continue;
@@ -666,7 +682,7 @@ posix_test_lock(struct file *filp, struct file_lock *fl)
 			fl->fl_pid = pid_vnr(cfl->fl_nspid);
 	} else
 		fl->fl_type = F_UNLCK;
-	unlock_flocks();
+	spin_unlock(&inode->i_lock);
 	return;
 }
 EXPORT_SYMBOL(posix_test_lock);
@@ -710,6 +726,7 @@ static struct file_lock *what_owner_is_waiting_for(struct file_lock *block_fl)
 	return NULL;
 }
 
+/* Must be called with the file_lock_lock held! */
 static int posix_locks_deadlock(struct file_lock *caller_fl,
 				struct file_lock *block_fl)
 {
@@ -745,7 +762,7 @@ static int flock_lock_file(struct file *filp, struct file_lock *request)
 			return -ENOMEM;
 	}
 
-	lock_flocks();
+	spin_lock(&inode->i_lock);
 	if (request->fl_flags & FL_ACCESS)
 		goto find_conflict;
 
@@ -775,9 +792,9 @@ static int flock_lock_file(struct file *filp, struct file_lock *request)
 	 * give it the opportunity to lock the file.
 	 */
 	if (found) {
-		unlock_flocks();
+		spin_unlock(&inode->i_lock);
 		cond_resched();
-		lock_flocks();
+		spin_lock(&inode->i_lock);
 	}
 
 find_conflict:
@@ -804,7 +821,7 @@ find_conflict:
 	error = 0;
 
 out:
-	unlock_flocks();
+	spin_unlock(&inode->i_lock);
 	if (new_fl)
 		locks_free_lock(new_fl);
 	return error;
@@ -834,7 +851,7 @@ static int __posix_lock_file(struct inode *inode, struct file_lock *request, str
 		new_fl2 = locks_alloc_lock();
 	}
 
-	lock_flocks();
+	spin_lock(&inode->i_lock);
 	/*
 	 * New lock request. Walk all POSIX locks and look for conflicts. If
 	 * there are any, either return error or put the request on the
@@ -852,11 +869,17 @@ static int __posix_lock_file(struct inode *inode, struct file_lock *request, str
 			error = -EAGAIN;
 			if (!(request->fl_flags & FL_SLEEP))
 				goto out;
+			/*
+			 * Deadlock detection and insertion into the blocked
+			 * locks list must be done while holding the same lock!
+			 */
 			error = -EDEADLK;
-			if (posix_locks_deadlock(request, fl))
-				goto out;
-			error = FILE_LOCK_DEFERRED;
-			locks_insert_block(fl, request);
+			spin_lock(&file_lock_lock);
+			if (likely(!posix_locks_deadlock(request, fl))) {
+				error = FILE_LOCK_DEFERRED;
+				__locks_insert_block(fl, request);
+			}
+			spin_unlock(&file_lock_lock);
 			goto out;
   		}
   	}
@@ -1006,7 +1029,7 @@ static int __posix_lock_file(struct inode *inode, struct file_lock *request, str
 		locks_wake_up_blocks(left);
 	}
  out:
-	unlock_flocks();
+	spin_unlock(&inode->i_lock);
 	/*
 	 * Free any unused locks.
 	 */
@@ -1081,14 +1104,14 @@ int locks_mandatory_locked(struct inode *inode)
 	/*
 	 * Search the lock list for this inode for any POSIX locks.
 	 */
-	lock_flocks();
+	spin_lock(&inode->i_lock);
 	for (fl = inode->i_flock; fl != NULL; fl = fl->fl_next) {
 		if (!IS_POSIX(fl))
 			continue;
 		if (fl->fl_owner != owner)
 			break;
 	}
-	unlock_flocks();
+	spin_unlock(&inode->i_lock);
 	return fl ? -EAGAIN : 0;
 }
 
@@ -1231,7 +1254,7 @@ int __break_lease(struct inode *inode, unsigned int mode)
 	if (IS_ERR(new_fl))
 		return PTR_ERR(new_fl);
 
-	lock_flocks();
+	spin_lock(&inode->i_lock);
 
 	time_out_leases(inode);
 
@@ -1281,11 +1304,11 @@ restart:
 			break_time++;
 	}
 	locks_insert_block(flock, new_fl);
-	unlock_flocks();
+	spin_unlock(&inode->i_lock);
 	error = wait_event_interruptible_timeout(new_fl->fl_wait,
 						!new_fl->fl_next, break_time);
-	lock_flocks();
-	__locks_delete_block(new_fl);
+	spin_lock(&inode->i_lock);
+	locks_delete_block(new_fl);
 	if (error >= 0) {
 		if (error == 0)
 			time_out_leases(inode);
@@ -1302,7 +1325,7 @@ restart:
 	}
 
 out:
-	unlock_flocks();
+	spin_unlock(&inode->i_lock);
 	locks_free_lock(new_fl);
 	return error;
 }
@@ -1355,9 +1378,10 @@ EXPORT_SYMBOL(lease_get_mtime);
 int fcntl_getlease(struct file *filp)
 {
 	struct file_lock *fl;
+	struct inode *inode = file_inode(filp);
 	int type = F_UNLCK;
 
-	lock_flocks();
+	spin_lock(&inode->i_lock);
 	time_out_leases(file_inode(filp));
 	for (fl = file_inode(filp)->i_flock; fl && IS_LEASE(fl);
 			fl = fl->fl_next) {
@@ -1366,7 +1390,7 @@ int fcntl_getlease(struct file *filp)
 			break;
 		}
 	}
-	unlock_flocks();
+	spin_unlock(&inode->i_lock);
 	return type;
 }
 
@@ -1460,7 +1484,7 @@ static int generic_delete_lease(struct file *filp, struct file_lock **flp)
  *	The (input) flp->fl_lmops->lm_break function is required
  *	by break_lease().
  *
- *	Called with file_lock_lock held.
+ *	Called with inode->i_lock held.
  */
 int generic_setlease(struct file *filp, long arg, struct file_lock **flp)
 {
@@ -1529,11 +1553,12 @@ static int __vfs_setlease(struct file *filp, long arg, struct file_lock **lease)
 
 int vfs_setlease(struct file *filp, long arg, struct file_lock **lease)
 {
+	struct inode *inode = file_inode(filp);
 	int error;
 
-	lock_flocks();
+	spin_lock(&inode->i_lock);
 	error = __vfs_setlease(filp, arg, lease);
-	unlock_flocks();
+	spin_unlock(&inode->i_lock);
 
 	return error;
 }
@@ -1551,6 +1576,7 @@ static int do_fcntl_delete_lease(struct file *filp)
 static int do_fcntl_add_lease(unsigned int fd, struct file *filp, long arg)
 {
 	struct file_lock *fl, *ret;
+	struct inode *inode = file_inode(filp);
 	struct fasync_struct *new;
 	int error;
 
@@ -1564,10 +1590,10 @@ static int do_fcntl_add_lease(unsigned int fd, struct file *filp, long arg)
 		return -ENOMEM;
 	}
 	ret = fl;
-	lock_flocks();
+	spin_lock(&inode->i_lock);
 	error = __vfs_setlease(filp, arg, &ret);
 	if (error) {
-		unlock_flocks();
+		spin_unlock(&inode->i_lock);
 		locks_free_lock(fl);
 		goto out_free_fasync;
 	}
@@ -1584,7 +1610,7 @@ static int do_fcntl_add_lease(unsigned int fd, struct file *filp, long arg)
 		new = NULL;
 
 	error = __f_setown(filp, task_pid(current), PIDTYPE_PID, 0);
-	unlock_flocks();
+	spin_unlock(&inode->i_lock);
 
 out_free_fasync:
 	if (new)
@@ -2108,7 +2134,7 @@ void locks_remove_flock(struct file *filp)
 			fl.fl_ops->fl_release_private(&fl);
 	}
 
-	lock_flocks();
+	spin_lock(&inode->i_lock);
 	before = &inode->i_flock;
 
 	while ((fl = *before) != NULL) {
@@ -2126,7 +2152,7 @@ void locks_remove_flock(struct file *filp)
  		}
 		before = &fl->fl_next;
 	}
-	unlock_flocks();
+	spin_unlock(&inode->i_lock);
 }
 
 /**
@@ -2139,14 +2165,15 @@ void locks_remove_flock(struct file *filp)
 int
 posix_unblock_lock(struct file *filp, struct file_lock *waiter)
 {
+	struct inode *inode = file_inode(filp);
 	int status = 0;
 
-	lock_flocks();
+	spin_lock(&inode->i_lock);
 	if (waiter->fl_next)
-		__locks_delete_block(waiter);
+		locks_delete_block(waiter);
 	else
 		status = -ENOENT;
-	unlock_flocks();
+	spin_unlock(&inode->i_lock);
 	return status;
 }
 
@@ -2261,7 +2288,7 @@ static void *locks_start(struct seq_file *f, loff_t *pos)
 {
 	loff_t *p = f->private;
 
-	lock_flocks();
+	spin_lock(&file_lock_lock);
 	*p = (*pos + 1);
 	return seq_list_start(&file_lock_list, *pos);
 }
@@ -2275,7 +2302,7 @@ static void *locks_next(struct seq_file *f, void *v, loff_t *pos)
 
 static void locks_stop(struct seq_file *f, void *v)
 {
-	unlock_flocks();
+	spin_unlock(&file_lock_lock);
 }
 
 static const struct seq_operations locks_seq_operations = {
@@ -2322,7 +2349,8 @@ int lock_may_read(struct inode *inode, loff_t start, unsigned long len)
 {
 	struct file_lock *fl;
 	int result = 1;
-	lock_flocks();
+
+	spin_lock(&inode->i_lock);
 	for (fl = inode->i_flock; fl != NULL; fl = fl->fl_next) {
 		if (IS_POSIX(fl)) {
 			if (fl->fl_type == F_RDLCK)
@@ -2339,7 +2367,7 @@ int lock_may_read(struct inode *inode, loff_t start, unsigned long len)
 		result = 0;
 		break;
 	}
-	unlock_flocks();
+	spin_unlock(&inode->i_lock);
 	return result;
 }
 
@@ -2362,7 +2390,8 @@ int lock_may_write(struct inode *inode, loff_t start, unsigned long len)
 {
 	struct file_lock *fl;
 	int result = 1;
-	lock_flocks();
+
+	spin_lock(&inode->i_lock);
 	for (fl = inode->i_flock; fl != NULL; fl = fl->fl_next) {
 		if (IS_POSIX(fl)) {
 			if ((fl->fl_end < start) || (fl->fl_start > (start + len)))
@@ -2377,7 +2406,7 @@ int lock_may_write(struct inode *inode, loff_t start, unsigned long len)
 		result = 0;
 		break;
 	}
-	unlock_flocks();
+	spin_unlock(&inode->i_lock);
 	return result;
 }
 
diff --git a/fs/nfs/delegation.c b/fs/nfs/delegation.c
index 57db324..7ec4814 100644
--- a/fs/nfs/delegation.c
+++ b/fs/nfs/delegation.c
@@ -73,20 +73,20 @@ static int nfs_delegation_claim_locks(struct nfs_open_context *ctx, struct nfs4_
 	if (inode->i_flock == NULL)
 		goto out;
 
-	/* Protect inode->i_flock using the file locks lock */
-	lock_flocks();
+	/* Protect inode->i_flock using the i_lock */
+	spin_lock(&inode->i_lock);
 	for (fl = inode->i_flock; fl != NULL; fl = fl->fl_next) {
 		if (!(fl->fl_flags & (FL_POSIX|FL_FLOCK)))
 			continue;
 		if (nfs_file_open_context(fl->fl_file) != ctx)
 			continue;
-		unlock_flocks();
+		spin_unlock(&inode->i_lock);
 		status = nfs4_lock_delegation_recall(fl, state, stateid);
 		if (status < 0)
 			goto out;
-		lock_flocks();
+		spin_lock(&inode->i_lock);
 	}
-	unlock_flocks();
+	spin_unlock(&inode->i_lock);
 out:
 	return status;
 }
diff --git a/fs/nfs/nfs4state.c b/fs/nfs/nfs4state.c
index 1fab140..ff10b4a 100644
--- a/fs/nfs/nfs4state.c
+++ b/fs/nfs/nfs4state.c
@@ -1373,13 +1373,13 @@ static int nfs4_reclaim_locks(struct nfs4_state *state, const struct nfs4_state_
 	/* Guard against delegation returns and new lock/unlock calls */
 	down_write(&nfsi->rwsem);
 	/* Protect inode->i_flock using the BKL */
-	lock_flocks();
+	spin_lock(&inode->i_lock);
 	for (fl = inode->i_flock; fl != NULL; fl = fl->fl_next) {
 		if (!(fl->fl_flags & (FL_POSIX|FL_FLOCK)))
 			continue;
 		if (nfs_file_open_context(fl->fl_file)->state != state)
 			continue;
-		unlock_flocks();
+		spin_unlock(&inode->i_lock);
 		status = ops->recover_lock(state, fl);
 		switch (status) {
 			case 0:
@@ -1406,9 +1406,9 @@ static int nfs4_reclaim_locks(struct nfs4_state *state, const struct nfs4_state_
 				/* kill_proc(fl->fl_pid, SIGLOST, 1); */
 				status = 0;
 		}
-		lock_flocks();
+		spin_lock(&inode->i_lock);
 	}
-	unlock_flocks();
+	spin_unlock(&inode->i_lock);
 out:
 	up_write(&nfsi->rwsem);
 	return status;
diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
index 316ec84..f170518 100644
--- a/fs/nfsd/nfs4state.c
+++ b/fs/nfsd/nfs4state.c
@@ -2645,13 +2645,13 @@ static void nfsd_break_one_deleg(struct nfs4_delegation *dp)
 
 	list_add_tail(&dp->dl_recall_lru, &nn->del_recall_lru);
 
-	/* only place dl_time is set. protected by lock_flocks*/
+	/* Only place dl_time is set; protected by i_lock: */
 	dp->dl_time = get_seconds();
 
 	nfsd4_cb_recall(dp);
 }
 
-/* Called from break_lease() with lock_flocks() held. */
+/* Called from break_lease() with i_lock held. */
 static void nfsd_break_deleg_cb(struct file_lock *fl)
 {
 	struct nfs4_file *fp = (struct nfs4_file *)fl->fl_owner;
@@ -4520,7 +4520,7 @@ check_for_locks(struct nfs4_file *filp, struct nfs4_lockowner *lowner)
 	struct inode *inode = filp->fi_inode;
 	int status = 0;
 
-	lock_flocks();
+	spin_lock(&inode->i_lock);
 	for (flpp = &inode->i_flock; *flpp != NULL; flpp = &(*flpp)->fl_next) {
 		if ((*flpp)->fl_owner == (fl_owner_t)lowner) {
 			status = 1;
@@ -4528,7 +4528,7 @@ check_for_locks(struct nfs4_file *filp, struct nfs4_lockowner *lowner)
 		}
 	}
 out:
-	unlock_flocks();
+	spin_unlock(&inode->i_lock);
 	return status;
 }
 
diff --git a/include/linux/fs.h b/include/linux/fs.h
index 94105d2..e2f896d 100644
--- a/include/linux/fs.h
+++ b/include/linux/fs.h
@@ -1024,8 +1024,6 @@ extern int vfs_setlease(struct file *, long, struct file_lock **);
 extern int lease_modify(struct file_lock **, int);
 extern int lock_may_read(struct inode *, loff_t start, unsigned long count);
 extern int lock_may_write(struct inode *, loff_t start, unsigned long count);
-extern void lock_flocks(void);
-extern void unlock_flocks(void);
 #else /* !CONFIG_FILE_LOCKING */
 static inline int fcntl_getlk(struct file *file, struct flock __user *user)
 {
@@ -1167,15 +1165,6 @@ static inline int lock_may_write(struct inode *inode, loff_t start,
 {
 	return 1;
 }
-
-static inline void lock_flocks(void)
-{
-}
-
-static inline void unlock_flocks(void)
-{
-}
-
 #endif /* !CONFIG_FILE_LOCKING */
 
 
-- 
1.7.1



From jlayton at redhat.com  Mon Jun 17 11:13:50 2013
From: jlayton at redhat.com (Jeff Layton)
Date: Mon, 17 Jun 2013 11:13:50 -0400
Subject: [PATCH v3 07/13] locks: avoid taking global lock if possible when
 waking up blocked waiters
In-Reply-To: <1371482036-15958-1-git-send-email-jlayton@redhat.com>
References: <1371482036-15958-1-git-send-email-jlayton@redhat.com>
Message-ID: <1371482036-15958-8-git-send-email-jlayton@redhat.com>

Since we always hold the i_lock when inserting a new waiter onto the
fl_block list, we can avoid taking the global lock at all if we find
that it's empty when we go to wake up blocked waiters.

Signed-off-by: Jeff Layton <jlayton at redhat.com>
---
 fs/locks.c |   17 ++++++++++++++---
 1 files changed, 14 insertions(+), 3 deletions(-)

diff --git a/fs/locks.c b/fs/locks.c
index 8f56651..a8f3b33 100644
--- a/fs/locks.c
+++ b/fs/locks.c
@@ -532,7 +532,10 @@ static void locks_delete_block(struct file_lock *waiter)
  * the order they blocked. The documentation doesn't require this but
  * it seems like the reasonable thing to do.
  *
- * Must be called with file_lock_lock held!
+ * Must be called with both the i_lock and file_lock_lock held. The fl_block
+ * list itself is protected by the file_lock_list, but by ensuring that the
+ * i_lock is also held on insertions we can avoid taking the file_lock_lock
+ * in some cases when we see that the fl_block list is empty.
  */
 static void __locks_insert_block(struct file_lock *blocker,
 					struct file_lock *waiter)
@@ -560,8 +563,16 @@ static void locks_insert_block(struct file_lock *blocker,
  */
 static void locks_wake_up_blocks(struct file_lock *blocker)
 {
+	/*
+	 * Avoid taking global lock if list is empty. This is safe since new
+	 * blocked requests are only added to the list under the i_lock, and
+	 * the i_lock is always held here.
+	 */
+	if (list_empty(&blocker->fl_block))
+		return;
+
 	spin_lock(&file_lock_lock);
-	while (!list_empty(&blocker->fl_block)) {
+	do {
 		struct file_lock *waiter;
 
 		waiter = list_first_entry(&blocker->fl_block,
@@ -571,7 +582,7 @@ static void locks_wake_up_blocks(struct file_lock *blocker)
 			waiter->fl_lmops->lm_notify(waiter);
 		else
 			wake_up(&waiter->fl_wait);
-	}
+	} while (!list_empty(&blocker->fl_block));
 	spin_unlock(&file_lock_lock);
 }
 
-- 
1.7.1



From jlayton at redhat.com  Mon Jun 17 11:13:55 2013
From: jlayton at redhat.com (Jeff Layton)
Date: Mon, 17 Jun 2013 11:13:55 -0400
Subject: [PATCH v3 12/13] seq_file: add seq_list_*_percpu helpers
In-Reply-To: <1371482036-15958-1-git-send-email-jlayton@redhat.com>
References: <1371482036-15958-1-git-send-email-jlayton@redhat.com>
Message-ID: <1371482036-15958-13-git-send-email-jlayton@redhat.com>

When we convert the file_lock_list to a set of percpu lists, we'll need
a way to iterate over them in order to output /proc/locks info. Add
some seq_list_*_percpu helpers to handle that.

Signed-off-by: Jeff Layton <jlayton at redhat.com>
Acked-by: J. Bruce Fields <bfields at fieldses.org>
---
 fs/seq_file.c            |   54 ++++++++++++++++++++++++++++++++++++++++++++++
 include/linux/seq_file.h |    6 +++++
 2 files changed, 60 insertions(+), 0 deletions(-)

diff --git a/fs/seq_file.c b/fs/seq_file.c
index 774c1eb..3135c25 100644
--- a/fs/seq_file.c
+++ b/fs/seq_file.c
@@ -921,3 +921,57 @@ struct hlist_node *seq_hlist_next_rcu(void *v,
 		return rcu_dereference(node->next);
 }
 EXPORT_SYMBOL(seq_hlist_next_rcu);
+
+/**
+ * seq_hlist_start_precpu - start an iteration of a percpu hlist array
+ * @head: pointer to percpu array of struct hlist_heads
+ * @cpu:  pointer to cpu "cursor"
+ * @pos:  start position of sequence
+ *
+ * Called at seq_file->op->start().
+ */
+struct hlist_node *
+seq_hlist_start_percpu(struct hlist_head __percpu *head, int *cpu, loff_t pos)
+{
+	struct hlist_node *node;
+
+	for_each_possible_cpu(*cpu) {
+		hlist_for_each(node, per_cpu_ptr(head, *cpu)) {
+			if (pos-- == 0)
+				return node;
+		}
+	}
+	return NULL;
+}
+EXPORT_SYMBOL(seq_hlist_start_percpu);
+
+/**
+ * seq_hlist_next_percpu - move to the next position of the percpu hlist array
+ * @v:    pointer to current hlist_node
+ * @head: pointer to percpu array of struct hlist_heads
+ * @cpu:  pointer to cpu "cursor"
+ * @pos:  start position of sequence
+ *
+ * Called at seq_file->op->next().
+ */
+struct hlist_node *
+seq_hlist_next_percpu(void *v, struct hlist_head __percpu *head,
+			int *cpu, loff_t *pos)
+{
+	struct hlist_node *node = v;
+
+	++*pos;
+
+	if (node->next)
+		return node->next;
+
+	for (*cpu = cpumask_next(*cpu, cpu_possible_mask); *cpu < nr_cpu_ids;
+	     *cpu = cpumask_next(*cpu, cpu_possible_mask)) {
+		struct hlist_head *bucket = per_cpu_ptr(head, *cpu);
+
+		if (!hlist_empty(bucket))
+			return bucket->first;
+	}
+	return NULL;
+}
+EXPORT_SYMBOL(seq_hlist_next_percpu);
diff --git a/include/linux/seq_file.h b/include/linux/seq_file.h
index 2da29ac..4e32edc 100644
--- a/include/linux/seq_file.h
+++ b/include/linux/seq_file.h
@@ -173,4 +173,10 @@ extern struct hlist_node *seq_hlist_start_head_rcu(struct hlist_head *head,
 extern struct hlist_node *seq_hlist_next_rcu(void *v,
 						   struct hlist_head *head,
 						   loff_t *ppos);
+
+/* Helpers for iterating over per-cpu hlist_head-s in seq_files */
+extern struct hlist_node *seq_hlist_start_percpu(struct hlist_head __percpu *head, int *cpu, loff_t pos);
+
+extern struct hlist_node *seq_hlist_next_percpu(void *v, struct hlist_head __percpu *head, int *cpu, loff_t *pos);
+
 #endif
-- 
1.7.1



From jlayton at redhat.com  Mon Jun 17 11:13:53 2013
From: jlayton at redhat.com (Jeff Layton)
Date: Mon, 17 Jun 2013 11:13:53 -0400
Subject: [PATCH v3 10/13] locks: add a new "lm_owner_key" lock operation
In-Reply-To: <1371482036-15958-1-git-send-email-jlayton@redhat.com>
References: <1371482036-15958-1-git-send-email-jlayton@redhat.com>
Message-ID: <1371482036-15958-11-git-send-email-jlayton@redhat.com>

Currently, the hashing that the locking code uses to add these values
to the blocked_hash is simply calculated using fl_owner field. That's
valid in most cases except for server-side lockd, which validates the
owner of a lock based on fl_owner and fl_pid.

In the case where you have a small number of NFS clients doing a lot
of locking between different processes, you could end up with all
the blocked requests sitting in a very small number of hash buckets.

Add a new lm_owner_key operation to the lock_manager_operations that
will generate an unsigned long to use as the key in the hashtable.
That function is only implemented for server-side lockd, and simply
XORs the fl_owner and fl_pid.

Signed-off-by: Jeff Layton <jlayton at redhat.com>
Acked-by: J. Bruce Fields <bfields at fieldses.org>
---
 Documentation/filesystems/Locking |   16 +++++++++++-----
 fs/lockd/svclock.c                |   12 ++++++++++++
 fs/locks.c                        |   12 ++++++++++--
 include/linux/fs.h                |    1 +
 4 files changed, 34 insertions(+), 7 deletions(-)

diff --git a/Documentation/filesystems/Locking b/Documentation/filesystems/Locking
index 413685f..dfeb01b 100644
--- a/Documentation/filesystems/Locking
+++ b/Documentation/filesystems/Locking
@@ -351,6 +351,7 @@ fl_release_private:	maybe		no
 ----------------------- lock_manager_operations ---------------------------
 prototypes:
 	int (*lm_compare_owner)(struct file_lock *, struct file_lock *);
+	unsigned long (*lm_owner_key)(struct file_lock *);
 	void (*lm_notify)(struct file_lock *);  /* unblock callback */
 	int (*lm_grant)(struct file_lock *, struct file_lock *, int);
 	void (*lm_break)(struct file_lock *); /* break_lease callback */
@@ -360,16 +361,21 @@ locking rules:
 
 			inode->i_lock	file_lock_lock	may block
 lm_compare_owner:	yes[1]		maybe		no
+lm_owner_key		yes[1]		yes		no
 lm_notify:		yes		yes		no
 lm_grant:		no		no		no
 lm_break:		yes		no		no
 lm_change		yes		no		no
 
-[1]:	->lm_compare_owner is generally called with *an* inode->i_lock held. It
-may not be the i_lock of the inode for either file_lock being compared! This is
-the case with deadlock detection, since the code has to chase down the owners
-of locks that may be entirely unrelated to the one on which the lock is being
-acquired. When doing a search for deadlocks, the file_lock_lock is also held.
+[1]:	->lm_compare_owner and ->lm_owner_key are generally called with
+*an* inode->i_lock held. It may not be the i_lock of the inode
+associated with either file_lock argument! This is the case with deadlock
+detection, since the code has to chase down the owners of locks that may
+be entirely unrelated to the one on which the lock is being acquired.
+For deadlock detection however, the file_lock_lock is also held. The
+fact that these locks are held ensures that the file_locks do not
+disappear out from under you while doing the comparison or generating an
+owner key.
 
 --------------------------- buffer_head -----------------------------------
 prototypes:
diff --git a/fs/lockd/svclock.c b/fs/lockd/svclock.c
index e703318..ce2cdab 100644
--- a/fs/lockd/svclock.c
+++ b/fs/lockd/svclock.c
@@ -744,8 +744,20 @@ static int nlmsvc_same_owner(struct file_lock *fl1, struct file_lock *fl2)
 	return fl1->fl_owner == fl2->fl_owner && fl1->fl_pid == fl2->fl_pid;
 }
 
+/*
+ * Since NLM uses two "keys" for tracking locks, we need to hash them down
+ * to one for the blocked_hash. Here, we're just xor'ing the host address
+ * with the pid in order to create a key value for picking a hash bucket.
+ */
+static unsigned long
+nlmsvc_owner_key(struct file_lock *fl)
+{
+	return (unsigned long)fl->fl_owner ^ (unsigned long)fl->fl_pid;
+}
+
 const struct lock_manager_operations nlmsvc_lock_operations = {
 	.lm_compare_owner = nlmsvc_same_owner,
+	.lm_owner_key = nlmsvc_owner_key,
 	.lm_notify = nlmsvc_notify_blocked,
 	.lm_grant = nlmsvc_grant_deferred,
 };
diff --git a/fs/locks.c b/fs/locks.c
index d93b291..55f3af7 100644
--- a/fs/locks.c
+++ b/fs/locks.c
@@ -505,10 +505,18 @@ locks_delete_global_locks(struct file_lock *waiter)
 	spin_unlock(&file_lock_lock);
 }
 
+static unsigned long
+posix_owner_key(struct file_lock *fl)
+{
+	if (fl->fl_lmops && fl->fl_lmops->lm_owner_key)
+		return fl->fl_lmops->lm_owner_key(fl);
+	return (unsigned long)fl->fl_owner;
+}
+
 static inline void
 locks_insert_global_blocked(struct file_lock *waiter)
 {
-	hash_add(blocked_hash, &waiter->fl_link, (unsigned long)waiter->fl_owner);
+	hash_add(blocked_hash, &waiter->fl_link, posix_owner_key(waiter));
 }
 
 static inline void
@@ -739,7 +747,7 @@ static struct file_lock *what_owner_is_waiting_for(struct file_lock *block_fl)
 {
 	struct file_lock *fl;
 
-	hash_for_each_possible(blocked_hash, fl, fl_link, (unsigned long)block_fl->fl_owner) {
+	hash_for_each_possible(blocked_hash, fl, fl_link, posix_owner_key(block_fl)) {
 		if (posix_same_owner(fl, block_fl))
 			return fl->fl_next;
 	}
diff --git a/include/linux/fs.h b/include/linux/fs.h
index 3b340f7..232a345 100644
--- a/include/linux/fs.h
+++ b/include/linux/fs.h
@@ -908,6 +908,7 @@ struct file_lock_operations {
 
 struct lock_manager_operations {
 	int (*lm_compare_owner)(struct file_lock *, struct file_lock *);
+	unsigned long (*lm_owner_key)(struct file_lock *);
 	void (*lm_notify)(struct file_lock *);	/* unblock callback */
 	int (*lm_grant)(struct file_lock *, struct file_lock *, int);
 	void (*lm_break)(struct file_lock *);
-- 
1.7.1



From jlayton at redhat.com  Mon Jun 17 11:13:54 2013
From: jlayton at redhat.com (Jeff Layton)
Date: Mon, 17 Jun 2013 11:13:54 -0400
Subject: [PATCH v3 11/13] locks: give the blocked_hash its own spinlock
In-Reply-To: <1371482036-15958-1-git-send-email-jlayton@redhat.com>
References: <1371482036-15958-1-git-send-email-jlayton@redhat.com>
Message-ID: <1371482036-15958-12-git-send-email-jlayton@redhat.com>

There's no reason we have to protect the blocked_hash and file_lock_list
with the same spinlock. With the tests I have, breaking it in two gives
a barely measurable performance benefit, but it seems reasonable to make
this locking as granular as possible.

Signed-off-by: Jeff Layton <jlayton at redhat.com>
---
 Documentation/filesystems/Locking |   16 ++++++++--------
 fs/locks.c                        |   33 ++++++++++++++++++---------------
 2 files changed, 26 insertions(+), 23 deletions(-)

diff --git a/Documentation/filesystems/Locking b/Documentation/filesystems/Locking
index dfeb01b..cf04448 100644
--- a/Documentation/filesystems/Locking
+++ b/Documentation/filesystems/Locking
@@ -359,20 +359,20 @@ prototypes:
 
 locking rules:
 
-			inode->i_lock	file_lock_lock	may block
-lm_compare_owner:	yes[1]		maybe		no
-lm_owner_key		yes[1]		yes		no
-lm_notify:		yes		yes		no
-lm_grant:		no		no		no
-lm_break:		yes		no		no
-lm_change		yes		no		no
+			inode->i_lock	blocked_lock_lock	may block
+lm_compare_owner:	yes[1]		maybe			no
+lm_owner_key		yes[1]		yes			no
+lm_notify:		yes		yes			no
+lm_grant:		no		no			no
+lm_break:		yes		no			no
+lm_change		yes		no			no
 
 [1]:	->lm_compare_owner and ->lm_owner_key are generally called with
 *an* inode->i_lock held. It may not be the i_lock of the inode
 associated with either file_lock argument! This is the case with deadlock
 detection, since the code has to chase down the owners of locks that may
 be entirely unrelated to the one on which the lock is being acquired.
-For deadlock detection however, the file_lock_lock is also held. The
+For deadlock detection however, the blocked_lock_lock is also held. The
 fact that these locks are held ensures that the file_locks do not
 disappear out from under you while doing the comparison or generating an
 owner key.
diff --git a/fs/locks.c b/fs/locks.c
index 55f3af7..5db80c7 100644
--- a/fs/locks.c
+++ b/fs/locks.c
@@ -159,10 +159,11 @@ int lease_break_time = 45;
  * by the file_lock_lock.
  */
 static HLIST_HEAD(file_lock_list);
+static DEFINE_SPINLOCK(file_lock_lock);
 
 /*
  * The blocked_hash is used to find POSIX lock loops for deadlock detection.
- * It is protected by file_lock_lock.
+ * It is protected by blocked_lock_lock.
  *
  * We hash locks by lockowner in order to optimize searching for the lock a
  * particular lockowner is waiting on.
@@ -174,8 +175,8 @@ static HLIST_HEAD(file_lock_list);
 #define BLOCKED_HASH_BITS	7
 static DEFINE_HASHTABLE(blocked_hash, BLOCKED_HASH_BITS);
 
-/* Protects the file_lock_list, the blocked_hash and fl->fl_block list */
-static DEFINE_SPINLOCK(file_lock_lock);
+/* protects blocked_hash and fl->fl_block list */
+static DEFINE_SPINLOCK(blocked_lock_lock);
 
 static struct kmem_cache *filelock_cache __read_mostly;
 
@@ -528,7 +529,7 @@ locks_delete_global_blocked(struct file_lock *waiter)
 /* Remove waiter from blocker's block list.
  * When blocker ends up pointing to itself then the list is empty.
  *
- * Must be called with file_lock_lock held.
+ * Must be called with blocked_lock_lock held.
  */
 static void __locks_delete_block(struct file_lock *waiter)
 {
@@ -539,9 +540,9 @@ static void __locks_delete_block(struct file_lock *waiter)
 
 static void locks_delete_block(struct file_lock *waiter)
 {
-	spin_lock(&file_lock_lock);
+	spin_lock(&blocked_lock_lock);
 	__locks_delete_block(waiter);
-	spin_unlock(&file_lock_lock);
+	spin_unlock(&blocked_lock_lock);
 }
 
 /* Insert waiter into blocker's block list.
@@ -549,9 +550,9 @@ static void locks_delete_block(struct file_lock *waiter)
  * the order they blocked. The documentation doesn't require this but
  * it seems like the reasonable thing to do.
  *
- * Must be called with both the i_lock and file_lock_lock held. The fl_block
+ * Must be called with both the i_lock and blocked_lock_lock held. The fl_block
  * list itself is protected by the file_lock_list, but by ensuring that the
- * i_lock is also held on insertions we can avoid taking the file_lock_lock
+ * i_lock is also held on insertions we can avoid taking the blocked_lock_lock
  * in some cases when we see that the fl_block list is empty.
  */
 static void __locks_insert_block(struct file_lock *blocker,
@@ -568,9 +569,9 @@ static void __locks_insert_block(struct file_lock *blocker,
 static void locks_insert_block(struct file_lock *blocker,
 					struct file_lock *waiter)
 {
-	spin_lock(&file_lock_lock);
+	spin_lock(&blocked_lock_lock);
 	__locks_insert_block(blocker, waiter);
-	spin_unlock(&file_lock_lock);
+	spin_unlock(&blocked_lock_lock);
 }
 
 /*
@@ -588,7 +589,7 @@ static void locks_wake_up_blocks(struct file_lock *blocker)
 	if (list_empty(&blocker->fl_block))
 		return;
 
-	spin_lock(&file_lock_lock);
+	spin_lock(&blocked_lock_lock);
 	do {
 		struct file_lock *waiter;
 
@@ -600,7 +601,7 @@ static void locks_wake_up_blocks(struct file_lock *blocker)
 		else
 			wake_up(&waiter->fl_wait);
 	} while (!list_empty(&blocker->fl_block));
-	spin_unlock(&file_lock_lock);
+	spin_unlock(&blocked_lock_lock);
 }
 
 /* Insert file lock fl into an inode's lock list at the position indicated
@@ -754,7 +755,7 @@ static struct file_lock *what_owner_is_waiting_for(struct file_lock *block_fl)
 	return NULL;
 }
 
-/* Must be called with the file_lock_lock held! */
+/* Must be called with the blocked_lock_lock held! */
 static int posix_locks_deadlock(struct file_lock *caller_fl,
 				struct file_lock *block_fl)
 {
@@ -902,12 +903,12 @@ static int __posix_lock_file(struct inode *inode, struct file_lock *request, str
 			 * locks list must be done while holding the same lock!
 			 */
 			error = -EDEADLK;
-			spin_lock(&file_lock_lock);
+			spin_lock(&blocked_lock_lock);
 			if (likely(!posix_locks_deadlock(request, fl))) {
 				error = FILE_LOCK_DEFERRED;
 				__locks_insert_block(fl, request);
 			}
-			spin_unlock(&file_lock_lock);
+			spin_unlock(&blocked_lock_lock);
 			goto out;
   		}
   	}
@@ -2317,6 +2318,7 @@ static void *locks_start(struct seq_file *f, loff_t *pos)
 	loff_t *p = f->private;
 
 	spin_lock(&file_lock_lock);
+	spin_lock(&blocked_lock_lock);
 	*p = (*pos + 1);
 	return seq_hlist_start(&file_lock_list, *pos);
 }
@@ -2330,6 +2332,7 @@ static void *locks_next(struct seq_file *f, void *v, loff_t *pos)
 
 static void locks_stop(struct seq_file *f, void *v)
 {
+	spin_unlock(&blocked_lock_lock);
 	spin_unlock(&file_lock_lock);
 }
 
-- 
1.7.1



From jlayton at redhat.com  Mon Jun 17 11:13:56 2013
From: jlayton at redhat.com (Jeff Layton)
Date: Mon, 17 Jun 2013 11:13:56 -0400
Subject: [PATCH v3 13/13] locks: move file_lock_list to a set of percpu
 hlist_heads and convert file_lock_lock to an lglock
In-Reply-To: <1371482036-15958-1-git-send-email-jlayton@redhat.com>
References: <1371482036-15958-1-git-send-email-jlayton@redhat.com>
Message-ID: <1371482036-15958-14-git-send-email-jlayton@redhat.com>

The file_lock_list is only used for /proc/locks. The vastly common case
is for locks to be put onto the list and come off again, without ever
being traversed.

Help optimize for this use-case by moving to percpu hlist_head-s. At the
same time, we can make the locking less contentious by moving to an
lglock. When iterating over the lists for /proc/locks, we must take the
global lock and then iterate over each CPU's list in turn.

This change necessitates a new fl_link_cpu field to keep track of which
CPU the entry is on. On x86_64 at least, this field is placed within an
existing hole in the struct to avoid growing the size.

Signed-off-by: Jeff Layton <jlayton at redhat.com>
Acked-by: J. Bruce Fields <bfields at fieldses.org>
---
 fs/locks.c         |   68 ++++++++++++++++++++++++++++++++++++---------------
 include/linux/fs.h |    1 +
 2 files changed, 49 insertions(+), 20 deletions(-)

diff --git a/fs/locks.c b/fs/locks.c
index 5db80c7..be9ca57 100644
--- a/fs/locks.c
+++ b/fs/locks.c
@@ -127,6 +127,8 @@
 #include <linux/rcupdate.h>
 #include <linux/pid_namespace.h>
 #include <linux/hashtable.h>
+#include <linux/percpu.h>
+#include <linux/lglock.h>
 
 #include <asm/uaccess.h>
 
@@ -155,11 +157,12 @@ int lease_break_time = 45;
 	for (lockp = &inode->i_flock; *lockp != NULL; lockp = &(*lockp)->fl_next)
 
 /*
- * The global file_lock_list is only used for displaying /proc/locks. Protected
- * by the file_lock_lock.
+ * The global file_lock_list is only used for displaying /proc/locks, so we
+ * keep a list on each CPU, with each list protected by its own spinlock via
+ * the file_lock_lglock.
  */
-static HLIST_HEAD(file_lock_list);
-static DEFINE_SPINLOCK(file_lock_lock);
+DEFINE_STATIC_LGLOCK(file_lock_lglock);
+static DEFINE_PER_CPU(struct hlist_head, file_lock_list);
 
 /*
  * The blocked_hash is used to find POSIX lock loops for deadlock detection.
@@ -490,20 +493,30 @@ static int posix_same_owner(struct file_lock *fl1, struct file_lock *fl2)
 	return fl1->fl_owner == fl2->fl_owner;
 }
 
+/* Must be called with the i_lock held! */
 static inline void
 locks_insert_global_locks(struct file_lock *waiter)
 {
-	spin_lock(&file_lock_lock);
-	hlist_add_head(&waiter->fl_link, &file_lock_list);
-	spin_unlock(&file_lock_lock);
+	lg_local_lock(&file_lock_lglock);
+	waiter->fl_link_cpu = smp_processor_id();
+	hlist_add_head(&waiter->fl_link, this_cpu_ptr(&file_lock_list));
+	lg_local_unlock(&file_lock_lglock);
 }
 
+/* Must be called with the i_lock held! */
 static inline void
 locks_delete_global_locks(struct file_lock *waiter)
 {
-	spin_lock(&file_lock_lock);
+	/*
+	 * Avoid taking lock if already unhashed. This is safe since this check
+	 * is done while holding the i_lock, and new insertions into the list
+	 * also require that it be held.
+	 */
+	if (hlist_unhashed(&waiter->fl_link))
+		return;
+	lg_local_lock_cpu(&file_lock_lglock, waiter->fl_link_cpu);
 	hlist_del_init(&waiter->fl_link);
-	spin_unlock(&file_lock_lock);
+	lg_local_unlock_cpu(&file_lock_lglock, waiter->fl_link_cpu);
 }
 
 static unsigned long
@@ -2228,6 +2241,11 @@ EXPORT_SYMBOL_GPL(vfs_cancel_lock);
 #include <linux/proc_fs.h>
 #include <linux/seq_file.h>
 
+struct locks_iterator {
+	int	li_cpu;
+	loff_t	li_pos;
+};
+
 static void lock_get_status(struct seq_file *f, struct file_lock *fl,
 			    loff_t id, char *pfx)
 {
@@ -2301,39 +2319,41 @@ static void lock_get_status(struct seq_file *f, struct file_lock *fl,
 
 static int locks_show(struct seq_file *f, void *v)
 {
+	struct locks_iterator *iter = f->private;
 	struct file_lock *fl, *bfl;
 
 	fl = hlist_entry(v, struct file_lock, fl_link);
 
-	lock_get_status(f, fl, *((loff_t *)f->private), "");
+	lock_get_status(f, fl, iter->li_pos, "");
 
 	list_for_each_entry(bfl, &fl->fl_block, fl_block)
-		lock_get_status(f, bfl, *((loff_t *)f->private), " ->");
+		lock_get_status(f, bfl, iter->li_pos, " ->");
 
 	return 0;
 }
 
 static void *locks_start(struct seq_file *f, loff_t *pos)
 {
-	loff_t *p = f->private;
+	struct locks_iterator *iter = f->private;
 
-	spin_lock(&file_lock_lock);
+	iter->li_pos = *pos + 1;
+	lg_global_lock(&file_lock_lglock);
 	spin_lock(&blocked_lock_lock);
-	*p = (*pos + 1);
-	return seq_hlist_start(&file_lock_list, *pos);
+	return seq_hlist_start_percpu(&file_lock_list, &iter->li_cpu, *pos);
 }
 
 static void *locks_next(struct seq_file *f, void *v, loff_t *pos)
 {
-	loff_t *p = f->private;
-	++*p;
-	return seq_hlist_next(v, &file_lock_list, pos);
+	struct locks_iterator *iter = f->private;
+
+	++iter->li_pos;
+	return seq_hlist_next_percpu(v, &file_lock_list, &iter->li_cpu, pos);
 }
 
 static void locks_stop(struct seq_file *f, void *v)
 {
 	spin_unlock(&blocked_lock_lock);
-	spin_unlock(&file_lock_lock);
+	lg_global_unlock(&file_lock_lglock);
 }
 
 static const struct seq_operations locks_seq_operations = {
@@ -2345,7 +2365,8 @@ static const struct seq_operations locks_seq_operations = {
 
 static int locks_open(struct inode *inode, struct file *filp)
 {
-	return seq_open_private(filp, &locks_seq_operations, sizeof(loff_t));
+	return seq_open_private(filp, &locks_seq_operations,
+					sizeof(struct locks_iterator));
 }
 
 static const struct file_operations proc_locks_operations = {
@@ -2445,9 +2466,16 @@ EXPORT_SYMBOL(lock_may_write);
 
 static int __init filelock_init(void)
 {
+	int i;
+
 	filelock_cache = kmem_cache_create("file_lock_cache",
 			sizeof(struct file_lock), 0, SLAB_PANIC, NULL);
 
+	lg_lock_init(&file_lock_lglock, "file_lock_lglock");
+
+	for_each_possible_cpu(i)
+		INIT_HLIST_HEAD(per_cpu_ptr(&file_lock_list, i));
+
 	return 0;
 }
 
diff --git a/include/linux/fs.h b/include/linux/fs.h
index 232a345..18e59b8 100644
--- a/include/linux/fs.h
+++ b/include/linux/fs.h
@@ -953,6 +953,7 @@ struct file_lock {
 	unsigned int fl_flags;
 	unsigned char fl_type;
 	unsigned int fl_pid;
+	int fl_link_cpu;		/* what cpu's list is this on? */
 	struct pid *fl_nspid;
 	wait_queue_head_t fl_wait;
 	struct file *fl_file;
-- 
1.7.1



From jlayton at redhat.com  Mon Jun 17 11:46:09 2013
From: jlayton at redhat.com (Jeff Layton)
Date: Mon, 17 Jun 2013 11:46:09 -0400
Subject: [PATCH v3 06/13] locks: protect most of the file_lock handling
 with i_lock
In-Reply-To: <1371482036-15958-7-git-send-email-jlayton@redhat.com>
References: <1371482036-15958-1-git-send-email-jlayton@redhat.com>
 <1371482036-15958-7-git-send-email-jlayton@redhat.com>
Message-ID: <20130617114609.708a336b@tlielax.poochiereds.net>

On Mon, 17 Jun 2013 11:13:49 -0400
Jeff Layton <jlayton at redhat.com> wrote:

> Having a global lock that protects all of this code is a clear
> scalability problem. Instead of doing that, move most of the code to be
> protected by the i_lock instead. The exceptions are the global lists
> that the ->fl_link sits on, and the ->fl_block list.
> 
> ->fl_link is what connects these structures to the
> global lists, so we must ensure that we hold those locks when iterating
> over or updating these lists.
> 
> Furthermore, sound deadlock detection requires that we hold the
> blocked_list state steady while checking for loops. We also must ensure
> that the search and update to the list are atomic.
> 
> For the checking and insertion side of the blocked_list, push the
> acquisition of the global lock into __posix_lock_file and ensure that
> checking and update of the  blocked_list is done without dropping the
> lock in between.
> 
> On the removal side, when waking up blocked lock waiters, take the
> global lock before walking the blocked list and dequeue the waiters from
> the global list prior to removal from the fl_block list.
> 
> With this, deadlock detection should be race free while we minimize
> excessive file_lock_lock thrashing.
> 
> Finally, in order to avoid a lock inversion problem when handling
> /proc/locks output we must ensure that manipulations of the fl_block
> list are also protected by the file_lock_lock.
> 
> Signed-off-by: Jeff Layton <jlayton at redhat.com>
> ---
>  Documentation/filesystems/Locking |   21 ++++--
>  fs/afs/flock.c                    |    5 +-
>  fs/ceph/locks.c                   |    2 +-
>  fs/ceph/mds_client.c              |    8 +-
>  fs/cifs/cifsfs.c                  |    2 +-
>  fs/cifs/file.c                    |   13 ++--
>  fs/gfs2/file.c                    |    2 +-
>  fs/lockd/svcsubs.c                |   12 ++--
>  fs/locks.c                        |  151 ++++++++++++++++++++++---------------
>  fs/nfs/delegation.c               |   10 +-
>  fs/nfs/nfs4state.c                |    8 +-
>  fs/nfsd/nfs4state.c               |    8 +-
>  include/linux/fs.h                |   11 ---
>  13 files changed, 140 insertions(+), 113 deletions(-)
> 

[...]

> @@ -1231,7 +1254,7 @@ int __break_lease(struct inode *inode, unsigned int mode)
>  	if (IS_ERR(new_fl))
>  		return PTR_ERR(new_fl);
>  
> -	lock_flocks();
> +	spin_lock(&inode->i_lock);
>  
>  	time_out_leases(inode);
>  
> @@ -1281,11 +1304,11 @@ restart:
>  			break_time++;
>  	}
>  	locks_insert_block(flock, new_fl);
> -	unlock_flocks();
> +	spin_unlock(&inode->i_lock);
>  	error = wait_event_interruptible_timeout(new_fl->fl_wait,
>  						!new_fl->fl_next, break_time);
> -	lock_flocks();
> -	__locks_delete_block(new_fl);
> +	spin_lock(&inode->i_lock);
> +	locks_delete_block(new_fl);

Doh -- bug here. This should not have been changed to
locks_delete_block(). My apologies.

>  	if (error >= 0) {
>  		if (error == 0)
>  			time_out_leases(inode);

[...]

>  posix_unblock_lock(struct file *filp, struct file_lock *waiter)
>  {
> +	struct inode *inode = file_inode(filp);
>  	int status = 0;
>  
> -	lock_flocks();
> +	spin_lock(&inode->i_lock);
>  	if (waiter->fl_next)
> -		__locks_delete_block(waiter);
> +		locks_delete_block(waiter);


Ditto here...

>  	else
>  		status = -ENOENT;
> -	unlock_flocks();
> +	spin_unlock(&inode->i_lock);
>  	return status;
>  }
>  

-- 
Jeff Layton <jlayton at redhat.com>


From jlayton at redhat.com  Mon Jun 17 11:49:38 2013
From: jlayton at redhat.com (Jeff Layton)
Date: Mon, 17 Jun 2013 11:49:38 -0400
Subject: [PATCH v3 06/13] locks: protect most of the file_lock handling
 with i_lock
In-Reply-To: <20130617114609.708a336b@tlielax.poochiereds.net>
References: <1371482036-15958-1-git-send-email-jlayton@redhat.com>
 <1371482036-15958-7-git-send-email-jlayton@redhat.com>
 <20130617114609.708a336b@tlielax.poochiereds.net>
Message-ID: <20130617114938.447a8d81@tlielax.poochiereds.net>

On Mon, 17 Jun 2013 11:46:09 -0400
Jeff Layton <jlayton at redhat.com> wrote:

> On Mon, 17 Jun 2013 11:13:49 -0400
> Jeff Layton <jlayton at redhat.com> wrote:
> 
> > Having a global lock that protects all of this code is a clear
> > scalability problem. Instead of doing that, move most of the code to be
> > protected by the i_lock instead. The exceptions are the global lists
> > that the ->fl_link sits on, and the ->fl_block list.
> > 
> > ->fl_link is what connects these structures to the
> > global lists, so we must ensure that we hold those locks when iterating
> > over or updating these lists.
> > 
> > Furthermore, sound deadlock detection requires that we hold the
> > blocked_list state steady while checking for loops. We also must ensure
> > that the search and update to the list are atomic.
> > 
> > For the checking and insertion side of the blocked_list, push the
> > acquisition of the global lock into __posix_lock_file and ensure that
> > checking and update of the  blocked_list is done without dropping the
> > lock in between.
> > 
> > On the removal side, when waking up blocked lock waiters, take the
> > global lock before walking the blocked list and dequeue the waiters from
> > the global list prior to removal from the fl_block list.
> > 
> > With this, deadlock detection should be race free while we minimize
> > excessive file_lock_lock thrashing.
> > 
> > Finally, in order to avoid a lock inversion problem when handling
> > /proc/locks output we must ensure that manipulations of the fl_block
> > list are also protected by the file_lock_lock.
> > 
> > Signed-off-by: Jeff Layton <jlayton at redhat.com>
> > ---
> >  Documentation/filesystems/Locking |   21 ++++--
> >  fs/afs/flock.c                    |    5 +-
> >  fs/ceph/locks.c                   |    2 +-
> >  fs/ceph/mds_client.c              |    8 +-
> >  fs/cifs/cifsfs.c                  |    2 +-
> >  fs/cifs/file.c                    |   13 ++--
> >  fs/gfs2/file.c                    |    2 +-
> >  fs/lockd/svcsubs.c                |   12 ++--
> >  fs/locks.c                        |  151 ++++++++++++++++++++++---------------
> >  fs/nfs/delegation.c               |   10 +-
> >  fs/nfs/nfs4state.c                |    8 +-
> >  fs/nfsd/nfs4state.c               |    8 +-
> >  include/linux/fs.h                |   11 ---
> >  13 files changed, 140 insertions(+), 113 deletions(-)
> > 
> 
> [...]
> 
> > @@ -1231,7 +1254,7 @@ int __break_lease(struct inode *inode, unsigned int mode)
> >  	if (IS_ERR(new_fl))
> >  		return PTR_ERR(new_fl);
> >  
> > -	lock_flocks();
> > +	spin_lock(&inode->i_lock);
> >  
> >  	time_out_leases(inode);
> >  
> > @@ -1281,11 +1304,11 @@ restart:
> >  			break_time++;
> >  	}
> >  	locks_insert_block(flock, new_fl);
> > -	unlock_flocks();
> > +	spin_unlock(&inode->i_lock);
> >  	error = wait_event_interruptible_timeout(new_fl->fl_wait,
> >  						!new_fl->fl_next, break_time);
> > -	lock_flocks();
> > -	__locks_delete_block(new_fl);
> > +	spin_lock(&inode->i_lock);
> > +	locks_delete_block(new_fl);
> 
> Doh -- bug here. This should not have been changed to
> locks_delete_block(). My apologies.
> 
> >  	if (error >= 0) {
> >  		if (error == 0)
> >  			time_out_leases(inode);
> 
> [...]
> 
> >  posix_unblock_lock(struct file *filp, struct file_lock *waiter)
> >  {
> > +	struct inode *inode = file_inode(filp);
> >  	int status = 0;
> >  
> > -	lock_flocks();
> > +	spin_lock(&inode->i_lock);
> >  	if (waiter->fl_next)
> > -		__locks_delete_block(waiter);
> > +		locks_delete_block(waiter);
> 
> 
> Ditto here...
> 
> >  	else
> >  		status = -ENOENT;
> > -	unlock_flocks();
> > +	spin_unlock(&inode->i_lock);
> >  	return status;
> >  }
> >  
> 

Bah, scratch that -- this patch is actually fine. We hold the i_lock
here and locks_delete_block takes the file_lock_lock, which is correct.
There is a potential race in patch 7 though. I'll reply to that patch
to point it out in a minute.

-- 
Jeff Layton <jlayton at redhat.com>


From jlayton at redhat.com  Mon Jun 17 11:55:08 2013
From: jlayton at redhat.com (Jeff Layton)
Date: Mon, 17 Jun 2013 11:55:08 -0400
Subject: [PATCH v3 07/13] locks: avoid taking global lock if possible
 when waking up blocked waiters
In-Reply-To: <1371482036-15958-8-git-send-email-jlayton@redhat.com>
References: <1371482036-15958-1-git-send-email-jlayton@redhat.com>
 <1371482036-15958-8-git-send-email-jlayton@redhat.com>
Message-ID: <20130617115508.4db5e5f9@tlielax.poochiereds.net>

On Mon, 17 Jun 2013 11:13:50 -0400
Jeff Layton <jlayton at redhat.com> wrote:

> Since we always hold the i_lock when inserting a new waiter onto the
> fl_block list, we can avoid taking the global lock at all if we find
> that it's empty when we go to wake up blocked waiters.
> 
> Signed-off-by: Jeff Layton <jlayton at redhat.com>
> ---
>  fs/locks.c |   17 ++++++++++++++---
>  1 files changed, 14 insertions(+), 3 deletions(-)
> 
> diff --git a/fs/locks.c b/fs/locks.c
> index 8f56651..a8f3b33 100644
> --- a/fs/locks.c
> +++ b/fs/locks.c
> @@ -532,7 +532,10 @@ static void locks_delete_block(struct file_lock *waiter)
>   * the order they blocked. The documentation doesn't require this but
>   * it seems like the reasonable thing to do.
>   *
> - * Must be called with file_lock_lock held!
> + * Must be called with both the i_lock and file_lock_lock held. The fl_block
> + * list itself is protected by the file_lock_list, but by ensuring that the
> + * i_lock is also held on insertions we can avoid taking the file_lock_lock
> + * in some cases when we see that the fl_block list is empty.
>   */
>  static void __locks_insert_block(struct file_lock *blocker,
>  					struct file_lock *waiter)
> @@ -560,8 +563,16 @@ static void locks_insert_block(struct file_lock *blocker,
>   */
>  static void locks_wake_up_blocks(struct file_lock *blocker)
>  {
> +	/*
> +	 * Avoid taking global lock if list is empty. This is safe since new
> +	 * blocked requests are only added to the list under the i_lock, and
> +	 * the i_lock is always held here.
> +	 */
> +	if (list_empty(&blocker->fl_block))
> +		return;
> +


Ok, potential race here. We hold the i_lock when we check list_empty()
above, but it's possible for the fl_block list to become empty between
that check and when we take the spinlock below. locks_delete_block does
not require that you hold the i_lock, and some callers don't hold it.

This is trivially fixable by just keeping this as a while() loop. We'll
do the list_empty() check twice in that case, but that shouldn't change
the performance here much.

I'll fix that in my tree and it'll be in the next resend. Sorry for the
noise...

>  	spin_lock(&file_lock_lock);
> -	while (!list_empty(&blocker->fl_block)) {
> +	do {

>  		struct file_lock *waiter;
>  
>  		waiter = list_first_entry(&blocker->fl_block,
> @@ -571,7 +582,7 @@ static void locks_wake_up_blocks(struct file_lock *blocker)
>  			waiter->fl_lmops->lm_notify(waiter);
>  		else
>  			wake_up(&waiter->fl_wait);
> -	}
> +	} while (!list_empty(&blocker->fl_block));
>  	spin_unlock(&file_lock_lock);
>  }
>  


-- 
Jeff Layton <jlayton at redhat.com>


From jlayton at redhat.com  Tue Jun 18 13:43:46 2013
From: jlayton at redhat.com (Jeff Layton)
Date: Tue, 18 Jun 2013 13:43:46 -0400
Subject: [PATCH v3 06/13] locks: protect most of the file_lock handling
 with i_lock
In-Reply-To: <20130617114938.447a8d81@tlielax.poochiereds.net>
References: <1371482036-15958-1-git-send-email-jlayton@redhat.com>
 <1371482036-15958-7-git-send-email-jlayton@redhat.com>
 <20130617114609.708a336b@tlielax.poochiereds.net>
 <20130617114938.447a8d81@tlielax.poochiereds.net>
Message-ID: <20130618134346.79edf0a7@tlielax.poochiereds.net>

On Mon, 17 Jun 2013 11:49:38 -0400
Jeff Layton <jlayton at redhat.com> wrote:

> On Mon, 17 Jun 2013 11:46:09 -0400
> Jeff Layton <jlayton at redhat.com> wrote:
> 
> > On Mon, 17 Jun 2013 11:13:49 -0400
> > Jeff Layton <jlayton at redhat.com> wrote:
> > 
> > > Having a global lock that protects all of this code is a clear
> > > scalability problem. Instead of doing that, move most of the code to be
> > > protected by the i_lock instead. The exceptions are the global lists
> > > that the ->fl_link sits on, and the ->fl_block list.
> > > 
> > > ->fl_link is what connects these structures to the
> > > global lists, so we must ensure that we hold those locks when iterating
> > > over or updating these lists.
> > > 
> > > Furthermore, sound deadlock detection requires that we hold the
> > > blocked_list state steady while checking for loops. We also must ensure
> > > that the search and update to the list are atomic.
> > > 
> > > For the checking and insertion side of the blocked_list, push the
> > > acquisition of the global lock into __posix_lock_file and ensure that
> > > checking and update of the  blocked_list is done without dropping the
> > > lock in between.
> > > 
> > > On the removal side, when waking up blocked lock waiters, take the
> > > global lock before walking the blocked list and dequeue the waiters from
> > > the global list prior to removal from the fl_block list.
> > > 
> > > With this, deadlock detection should be race free while we minimize
> > > excessive file_lock_lock thrashing.
> > > 
> > > Finally, in order to avoid a lock inversion problem when handling
> > > /proc/locks output we must ensure that manipulations of the fl_block
> > > list are also protected by the file_lock_lock.
> > > 
> > > Signed-off-by: Jeff Layton <jlayton at redhat.com>
> > > ---
> > >  Documentation/filesystems/Locking |   21 ++++--
> > >  fs/afs/flock.c                    |    5 +-
> > >  fs/ceph/locks.c                   |    2 +-
> > >  fs/ceph/mds_client.c              |    8 +-
> > >  fs/cifs/cifsfs.c                  |    2 +-
> > >  fs/cifs/file.c                    |   13 ++--
> > >  fs/gfs2/file.c                    |    2 +-
> > >  fs/lockd/svcsubs.c                |   12 ++--
> > >  fs/locks.c                        |  151 ++++++++++++++++++++++---------------
> > >  fs/nfs/delegation.c               |   10 +-
> > >  fs/nfs/nfs4state.c                |    8 +-
> > >  fs/nfsd/nfs4state.c               |    8 +-
> > >  include/linux/fs.h                |   11 ---
> > >  13 files changed, 140 insertions(+), 113 deletions(-)
> > > 
> > 
> > [...]
> > 
> > > @@ -1231,7 +1254,7 @@ int __break_lease(struct inode *inode, unsigned int mode)
> > >  	if (IS_ERR(new_fl))
> > >  		return PTR_ERR(new_fl);
> > >  
> > > -	lock_flocks();
> > > +	spin_lock(&inode->i_lock);
> > >  
> > >  	time_out_leases(inode);
> > >  
> > > @@ -1281,11 +1304,11 @@ restart:
> > >  			break_time++;
> > >  	}
> > >  	locks_insert_block(flock, new_fl);
> > > -	unlock_flocks();
> > > +	spin_unlock(&inode->i_lock);
> > >  	error = wait_event_interruptible_timeout(new_fl->fl_wait,
> > >  						!new_fl->fl_next, break_time);
> > > -	lock_flocks();
> > > -	__locks_delete_block(new_fl);
> > > +	spin_lock(&inode->i_lock);
> > > +	locks_delete_block(new_fl);
> > 
> > Doh -- bug here. This should not have been changed to
> > locks_delete_block(). My apologies.
> > 
> > >  	if (error >= 0) {
> > >  		if (error == 0)
> > >  			time_out_leases(inode);
> > 
> > [...]
> > 
> > >  posix_unblock_lock(struct file *filp, struct file_lock *waiter)
> > >  {
> > > +	struct inode *inode = file_inode(filp);
> > >  	int status = 0;
> > >  
> > > -	lock_flocks();
> > > +	spin_lock(&inode->i_lock);
> > >  	if (waiter->fl_next)
> > > -		__locks_delete_block(waiter);
> > > +		locks_delete_block(waiter);
> > 
> > 
> > Ditto here...
> > 
> > >  	else
> > >  		status = -ENOENT;
> > > -	unlock_flocks();
> > > +	spin_unlock(&inode->i_lock);
> > >  	return status;
> > >  }
> > >  
> > 
> 
> Bah, scratch that -- this patch is actually fine. We hold the i_lock
> here and locks_delete_block takes the file_lock_lock, which is correct.
> There is a potential race in patch 7 though. I'll reply to that patch
> to point it out in a minute.
> 

...and scratch that again...

There is a bug in the posix_unblock_lock delta above. We don't need to
hold the i_lock there, but we should probably be taking the
file_lock_lock before checking fl_next.

In truth, I have a hard time seeing how that race would manifest itself
in practice, but that's how the existing code works so it's probably
best to preserve that as closely as possible.

Fixed in my git repo and the next spin of this set will have it.

Sorry again for the noise... ;)

-- 
Jeff Layton <jlayton at redhat.com>


From jlayton at redhat.com  Fri Jun 21 08:58:10 2013
From: jlayton at redhat.com (Jeff Layton)
Date: Fri, 21 Jun 2013 08:58:10 -0400
Subject: [PATCH v4 02/14] cifs: use posix_unblock_lock instead of
 locks_delete_block
In-Reply-To: <1371819502-26363-1-git-send-email-jlayton@redhat.com>
References: <1371819502-26363-1-git-send-email-jlayton@redhat.com>
Message-ID: <1371819502-26363-3-git-send-email-jlayton@redhat.com>

commit 66189be74 (CIFS: Fix VFS lock usage for oplocked files) exported
the locks_delete_block symbol. There's already an exported helper
function that provides this capability however, so make cifs use that
instead and turn locks_delete_block back into a static function.

Note that if fl->fl_next == NULL then this lock has already been through
locks_delete_block(), so we should be OK to ignore an ENOENT error here
and simply not retry the lock.

Cc: Pavel Shilovsky <piastryyy at gmail.com>
Signed-off-by: Jeff Layton <jlayton at redhat.com>
Acked-by: J. Bruce Fields <bfields at fieldses.org>
---
 fs/cifs/file.c     |    2 +-
 fs/locks.c         |    3 +--
 include/linux/fs.h |    5 -----
 3 files changed, 2 insertions(+), 8 deletions(-)

diff --git a/fs/cifs/file.c b/fs/cifs/file.c
index 48b29d2..1686e40 100644
--- a/fs/cifs/file.c
+++ b/fs/cifs/file.c
@@ -999,7 +999,7 @@ try_again:
 		rc = wait_event_interruptible(flock->fl_wait, !flock->fl_next);
 		if (!rc)
 			goto try_again;
-		locks_delete_block(flock);
+		posix_unblock_lock(flock);
 	}
 	return rc;
 }
diff --git a/fs/locks.c b/fs/locks.c
index 72fb2b7..d732e22 100644
--- a/fs/locks.c
+++ b/fs/locks.c
@@ -496,13 +496,12 @@ static void __locks_delete_block(struct file_lock *waiter)
 
 /*
  */
-void locks_delete_block(struct file_lock *waiter)
+static void locks_delete_block(struct file_lock *waiter)
 {
 	lock_flocks();
 	__locks_delete_block(waiter);
 	unlock_flocks();
 }
-EXPORT_SYMBOL(locks_delete_block);
 
 /* Insert waiter into blocker's block list.
  * We use a circular list so that processes can be easily woken up in
diff --git a/include/linux/fs.h b/include/linux/fs.h
index 9f160c6..cabc08a 100644
--- a/include/linux/fs.h
+++ b/include/linux/fs.h
@@ -1006,7 +1006,6 @@ extern int vfs_setlease(struct file *, long, struct file_lock **);
 extern int lease_modify(struct file_lock **, int);
 extern int lock_may_read(struct inode *, loff_t start, unsigned long count);
 extern int lock_may_write(struct inode *, loff_t start, unsigned long count);
-extern void locks_delete_block(struct file_lock *waiter);
 extern void lock_flocks(void);
 extern void unlock_flocks(void);
 #else /* !CONFIG_FILE_LOCKING */
@@ -1150,10 +1149,6 @@ static inline int lock_may_write(struct inode *inode, loff_t start,
 	return 1;
 }
 
-static inline void locks_delete_block(struct file_lock *waiter)
-{
-}
-
 static inline void lock_flocks(void)
 {
 }
-- 
1.7.1



From jlayton at redhat.com  Fri Jun 21 08:58:09 2013
From: jlayton at redhat.com (Jeff Layton)
Date: Fri, 21 Jun 2013 08:58:09 -0400
Subject: [PATCH v4 01/14] locks: drop the unused filp argument to
 posix_unblock_lock
In-Reply-To: <1371819502-26363-1-git-send-email-jlayton@redhat.com>
References: <1371819502-26363-1-git-send-email-jlayton@redhat.com>
Message-ID: <1371819502-26363-2-git-send-email-jlayton@redhat.com>

Signed-off-by: Jeff Layton <jlayton at redhat.com>
---
 fs/lockd/svclock.c |    2 +-
 fs/locks.c         |    4 +---
 include/linux/fs.h |    5 ++---
 3 files changed, 4 insertions(+), 7 deletions(-)

diff --git a/fs/lockd/svclock.c b/fs/lockd/svclock.c
index e703318..a469098 100644
--- a/fs/lockd/svclock.c
+++ b/fs/lockd/svclock.c
@@ -276,7 +276,7 @@ static int nlmsvc_unlink_block(struct nlm_block *block)
 	dprintk("lockd: unlinking block %p...\n", block);
 
 	/* Remove block from list */
-	status = posix_unblock_lock(block->b_file->f_file, &block->b_call->a_args.lock.fl);
+	status = posix_unblock_lock(&block->b_call->a_args.lock.fl);
 	nlmsvc_remove_block(block);
 	return status;
 }
diff --git a/fs/locks.c b/fs/locks.c
index cb424a4..72fb2b7 100644
--- a/fs/locks.c
+++ b/fs/locks.c
@@ -2099,13 +2099,12 @@ void locks_remove_flock(struct file *filp)
 
 /**
  *	posix_unblock_lock - stop waiting for a file lock
- *      @filp:   how the file was opened
  *	@waiter: the lock which was waiting
  *
  *	lockd needs to block waiting for locks.
  */
 int
-posix_unblock_lock(struct file *filp, struct file_lock *waiter)
+posix_unblock_lock(struct file_lock *waiter)
 {
 	int status = 0;
 
@@ -2117,7 +2116,6 @@ posix_unblock_lock(struct file *filp, struct file_lock *waiter)
 	unlock_flocks();
 	return status;
 }
-
 EXPORT_SYMBOL(posix_unblock_lock);
 
 /**
diff --git a/include/linux/fs.h b/include/linux/fs.h
index 43db02e..9f160c6 100644
--- a/include/linux/fs.h
+++ b/include/linux/fs.h
@@ -994,7 +994,7 @@ extern void locks_release_private(struct file_lock *);
 extern void posix_test_lock(struct file *, struct file_lock *);
 extern int posix_lock_file(struct file *, struct file_lock *, struct file_lock *);
 extern int posix_lock_file_wait(struct file *, struct file_lock *);
-extern int posix_unblock_lock(struct file *, struct file_lock *);
+extern int posix_unblock_lock(struct file_lock *);
 extern int vfs_test_lock(struct file *, struct file_lock *);
 extern int vfs_lock_file(struct file *, unsigned int, struct file_lock *, struct file_lock *);
 extern int vfs_cancel_lock(struct file *filp, struct file_lock *fl);
@@ -1084,8 +1084,7 @@ static inline int posix_lock_file_wait(struct file *filp, struct file_lock *fl)
 	return -ENOLCK;
 }
 
-static inline int posix_unblock_lock(struct file *filp,
-				     struct file_lock *waiter)
+static inline int posix_unblock_lock(struct file_lock *waiter)
 {
 	return -ENOENT;
 }
-- 
1.7.1



From jlayton at redhat.com  Fri Jun 21 08:58:14 2013
From: jlayton at redhat.com (Jeff Layton)
Date: Fri, 21 Jun 2013 08:58:14 -0400
Subject: [PATCH v4 06/14] locks: encapsulate the fl_link list handling
In-Reply-To: <1371819502-26363-1-git-send-email-jlayton@redhat.com>
References: <1371819502-26363-1-git-send-email-jlayton@redhat.com>
Message-ID: <1371819502-26363-7-git-send-email-jlayton@redhat.com>

Move the fl_link list handling routines into a separate set of helpers.
Also ensure that locks and requests are always put on global lists
last (after fully initializing them) and are taken off before unintializing
them.

Signed-off-by: Jeff Layton <jlayton at redhat.com>
---
 fs/locks.c |   45 ++++++++++++++++++++++++++++++++++++---------
 1 files changed, 36 insertions(+), 9 deletions(-)

diff --git a/fs/locks.c b/fs/locks.c
index 1d6cb28..89d898b 100644
--- a/fs/locks.c
+++ b/fs/locks.c
@@ -153,13 +153,15 @@ int lease_break_time = 45;
 #define for_each_lock(inode, lockp) \
 	for (lockp = &inode->i_flock; *lockp != NULL; lockp = &(*lockp)->fl_next)
 
+/* The global file_lock_list is only used for displaying /proc/locks. */
 static LIST_HEAD(file_lock_list);
+
+/* The blocked_list is used to find POSIX lock loops for deadlock detection. */
 static LIST_HEAD(blocked_list);
+
+/* Protects the two list heads above, plus the inode->i_flock list */
 static DEFINE_SPINLOCK(file_lock_lock);
 
-/*
- * Protects the two list heads above, plus the inode->i_flock list
- */
 void lock_flocks(void)
 {
 	spin_lock(&file_lock_lock);
@@ -484,13 +486,37 @@ static int posix_same_owner(struct file_lock *fl1, struct file_lock *fl2)
 	return fl1->fl_owner == fl2->fl_owner;
 }
 
+static inline void
+locks_insert_global_locks(struct file_lock *fl)
+{
+	list_add_tail(&fl->fl_link, &file_lock_list);
+}
+
+static inline void
+locks_delete_global_locks(struct file_lock *fl)
+{
+	list_del_init(&fl->fl_link);
+}
+
+static inline void
+locks_insert_global_blocked(struct file_lock *waiter)
+{
+	list_add(&waiter->fl_link, &blocked_list);
+}
+
+static inline void
+locks_delete_global_blocked(struct file_lock *waiter)
+{
+	list_del_init(&waiter->fl_link);
+}
+
 /* Remove waiter from blocker's block list.
  * When blocker ends up pointing to itself then the list is empty.
  */
 static void __locks_delete_block(struct file_lock *waiter)
 {
+	locks_delete_global_blocked(waiter);
 	list_del_init(&waiter->fl_block);
-	list_del_init(&waiter->fl_link);
 	waiter->fl_next = NULL;
 }
 
@@ -512,10 +538,10 @@ static void locks_insert_block(struct file_lock *blocker,
 			       struct file_lock *waiter)
 {
 	BUG_ON(!list_empty(&waiter->fl_block));
-	list_add_tail(&waiter->fl_block, &blocker->fl_block);
 	waiter->fl_next = blocker;
+	list_add_tail(&waiter->fl_block, &blocker->fl_block);
 	if (IS_POSIX(blocker))
-		list_add(&waiter->fl_link, &blocked_list);
+		locks_insert_global_blocked(request);
 }
 
 /*
@@ -543,13 +569,13 @@ static void locks_wake_up_blocks(struct file_lock *blocker)
  */
 static void locks_insert_lock(struct file_lock **pos, struct file_lock *fl)
 {
-	list_add(&fl->fl_link, &file_lock_list);
-
 	fl->fl_nspid = get_pid(task_tgid(current));
 
 	/* insert into file's list */
 	fl->fl_next = *pos;
 	*pos = fl;
+
+	locks_insert_global_locks(fl);
 }
 
 /*
@@ -562,9 +588,10 @@ static void locks_delete_lock(struct file_lock **thisfl_p)
 {
 	struct file_lock *fl = *thisfl_p;
 
+	locks_delete_global_locks(fl);
+
 	*thisfl_p = fl->fl_next;
 	fl->fl_next = NULL;
-	list_del_init(&fl->fl_link);
 
 	if (fl->fl_nspid) {
 		put_pid(fl->fl_nspid);
-- 
1.7.1



From jlayton at redhat.com  Fri Jun 21 08:58:13 2013
From: jlayton at redhat.com (Jeff Layton)
Date: Fri, 21 Jun 2013 08:58:13 -0400
Subject: [PATCH v4 05/14] locks: make "added" in __posix_lock_file a bool
In-Reply-To: <1371819502-26363-1-git-send-email-jlayton@redhat.com>
References: <1371819502-26363-1-git-send-email-jlayton@redhat.com>
Message-ID: <1371819502-26363-6-git-send-email-jlayton@redhat.com>

...save 3 bytes of stack space.

Signed-off-by: Jeff Layton <jlayton at redhat.com>
Acked-by: J. Bruce Fields <bfields at fieldses.org>
---
 fs/locks.c |    9 +++++----
 1 files changed, 5 insertions(+), 4 deletions(-)

diff --git a/fs/locks.c b/fs/locks.c
index ddeab49..1d6cb28 100644
--- a/fs/locks.c
+++ b/fs/locks.c
@@ -791,7 +791,8 @@ static int __posix_lock_file(struct inode *inode, struct file_lock *request, str
 	struct file_lock *left = NULL;
 	struct file_lock *right = NULL;
 	struct file_lock **before;
-	int error, added = 0;
+	int error;
+	bool added = false;
 
 	/*
 	 * We may need two file_lock structures for this operation,
@@ -885,7 +886,7 @@ static int __posix_lock_file(struct inode *inode, struct file_lock *request, str
 				continue;
 			}
 			request = fl;
-			added = 1;
+			added = true;
 		}
 		else {
 			/* Processing for different lock types is a bit
@@ -896,7 +897,7 @@ static int __posix_lock_file(struct inode *inode, struct file_lock *request, str
 			if (fl->fl_start > request->fl_end)
 				break;
 			if (request->fl_type == F_UNLCK)
-				added = 1;
+				added = true;
 			if (fl->fl_start < request->fl_start)
 				left = fl;
 			/* If the next lock in the list has a higher end
@@ -926,7 +927,7 @@ static int __posix_lock_file(struct inode *inode, struct file_lock *request, str
 				locks_release_private(fl);
 				locks_copy_private(fl, request);
 				request = fl;
-				added = 1;
+				added = true;
 			}
 		}
 		/* Go on to next lock.
-- 
1.7.1



From jlayton at redhat.com  Fri Jun 21 08:58:08 2013
From: jlayton at redhat.com (Jeff Layton)
Date: Fri, 21 Jun 2013 08:58:08 -0400
Subject: [PATCH v4 00/14] locks: scalability improvements for file locking
Message-ID: <1371819502-26363-1-git-send-email-jlayton@redhat.com>

This is the fourth iteration of this patchset, at this point, I think
it's probably ready for merge. There are a few small cleanups in this
set, but it's almost identical functionally to the v3 set. The cover
letter below is basically equivalent to the one I sent in the v3 set
as well. There was no measurable performance difference with this
set and that one, so I've left the results in there as-is.

Summary of Significant Changes:
-------------------------------
v4:
- eliminate unused argument to posix_unblock_lock
- fix potential race in locks_wake_up_blocks
- more comment cleanups and clarifications

v3:
- Change spinlock handling to avoid the need to traverse the global
  blocked_hash when doing output of /proc/locks. This means that the
  fl_block list must continue to be protected by a global lock, but
  the fact that the i_lock is also held in most cases means that we
  can avoid taking it in certain situations.

v2:
- Fix potential races in deadlock detection. Manipulation of global
  blocked_hash and deadlock detection are now atomic. This is a
  little slower than the earlier set, but is provably correct. Also,
  the patch that converts to using the i_lock has been split out from
  most of the other changes. That should make it easier to review, but
  it does leave a potential race in the deadlock detection that is fixed
  up by the following patch. It may make sense to fold patches 7 and 8
  together before merging.

- Add percpu hlists and lglocks for global file_lock_list. This gives
  us some speedup since this list is seldom read.

Abstract (tl;dr version):
-------------------------
This patchset represents an overhaul of the file locking code with an
aim toward improving its scalability and making the code a bit easier to
understand.

Longer version:
---------------
When the BKL was finally ripped out of the kernel in 2010, the strategy
taken for the file locking code was to simply turn it into a new
file_lock_locks spinlock. It was an expedient way to deal with the file
locking code at the time, but having a giant spinlock around all of this
code is clearly not great for scalability. Red Hat has bug reports that
go back into the 2.6.18 era that point to BKL scalability problems in
the file locking code and the file_lock_lock suffers from the same
issues.

This patchset is my first attempt to make this code less dependent on
global locking. The main change is to switch most of the file locking
code to be protected by the inode->i_lock instead of the file_lock_lock.

While that works for most things, there are a couple of global data
structures (lists in the current code) that need a global lock to
protect them. So we still need a global lock in order to deal with
those. The remaining patches are intended to make that global locking
less painful. The big gains are made by turning the blocked_list into a
hashtable, which greatly speeds up the deadlock detection code and
making the file_lock_list percpu.

This is not the first attempt at doing this. The conversion to the
i_lock was originally attempted by Bruce Fields a few years ago. His
approach was NAK'ed since it involved ripping out the deadlock
detection. People also really seem to like /proc/locks for debugging, so
keeping that in is probably worthwhile.

There's more work to be done in this area and this patchset is just a
start. There's a horrible thundering herd problem when a blocking lock
is released, for instance. There was also interest in solving the goofy
"unlock on any close" POSIX lock semantics at this year's LSF. I think
this patchset will help lay the groundwork for those changes as well.

While file locking is not usually considered to be a high-performance
codepath, it *is* an IPC mechanism and I think it behooves us to try to
make it as fast as possible.

I'd like to see this considered for 3.11, but some soak time in -next
would be good. Comments and suggestions welcome.

Performance testing and results:
--------------------------------
In order to measure the benefit of this set, I've written some locking
performance tests that I've made available here:

    git://git.samba.org/jlayton/lockperf.git

Here are the results from the same 32-way, 4 NUMA node machine that I
used to generate the v2 patch results. The first number is the mean
time spent in locking for the test. The number in parenthesis is the
standard deviation.

		3.10.0-rc5-00219-ga2648eb	3.10.0-rc5-00231-g7569869
---------------------------------------------------------------------------
flock01		24119.96 (266.08)		24542.51 (254.89)
flock02		 1345.09  (37.37)		    8.60   (0.31)
posix01		31217.14 (320.91)		24899.20 (254.27)
posix02		 1348.60  (36.83)		   12.70   (0.44)

I wasn't able to reserve the exact same smaller machine for testing this
set, but this one is comparable with 4 CPUs and UMA architecture:

		3.10.0-rc5-00219-ga2648eb	3.10.0-rc5-00231-g7569869
---------------------------------------------------------------------------
flock01		1787.51 (11.23)			1797.75  (9.27)
flock02		 314.90	 (8.84)			  34.87  (2.82)
posix01		1843.43 (11.63)			1880.47 (13.47)
posix02		 325.13  (8.53)			  54.09  (4.02)

I think the conclusion we can draw here is that this patchset it roughly
as fast as the previous one. In addition, the posix02 test saw a vast
increase in performance.

I believe that's mostly due to the fact that with this set I added a
patch that allows the code to avoid taking the global blocked_lock_lock
when waking up waiters if there aren't any. With that, the
blocked_lock_lock never has to be taken at all if there's no contention
for the file_lock (as is the case in the posix02 and flock02 tests).

Jeff Layton (14):
  locks: drop the unused filp argument to posix_unblock_lock
  cifs: use posix_unblock_lock instead of locks_delete_block
  locks: make generic_add_lease and generic_delete_lease static
  locks: comment cleanups and clarifications
  locks: make "added" in __posix_lock_file a bool
  locks: encapsulate the fl_link list handling
  locks: protect most of the file_lock handling with i_lock
  locks: avoid taking global lock if possible when waking up blocked
    waiters
  locks: convert fl_link to a hlist_node
  locks: turn the blocked_list into a hashtable
  locks: add a new "lm_owner_key" lock operation
  locks: give the blocked_hash its own spinlock
  seq_file: add seq_list_*_percpu helpers
  locks: move file_lock_list to a set of percpu hlist_heads and convert
    file_lock_lock to an lglock

 Documentation/filesystems/Locking |   27 +++-
 fs/afs/flock.c                    |    5 +-
 fs/ceph/locks.c                   |    2 +-
 fs/ceph/mds_client.c              |    8 +-
 fs/cifs/cifsfs.c                  |    2 +-
 fs/cifs/file.c                    |   15 +-
 fs/gfs2/file.c                    |    2 +-
 fs/lockd/svclock.c                |   14 ++-
 fs/lockd/svcsubs.c                |   12 +-
 fs/locks.c                        |  326 ++++++++++++++++++++++++++-----------
 fs/nfs/delegation.c               |   10 +-
 fs/nfs/nfs4state.c                |    8 +-
 fs/nfsd/nfs4state.c               |    8 +-
 fs/seq_file.c                     |   54 ++++++
 include/linux/fs.h                |   43 +++---
 include/linux/seq_file.h          |    6 +
 16 files changed, 384 insertions(+), 158 deletions(-)



From jlayton at redhat.com  Fri Jun 21 08:58:11 2013
From: jlayton at redhat.com (Jeff Layton)
Date: Fri, 21 Jun 2013 08:58:11 -0400
Subject: [PATCH v4 03/14] locks: make generic_add_lease and
 generic_delete_lease static
In-Reply-To: <1371819502-26363-1-git-send-email-jlayton@redhat.com>
References: <1371819502-26363-1-git-send-email-jlayton@redhat.com>
Message-ID: <1371819502-26363-4-git-send-email-jlayton@redhat.com>

Signed-off-by: Jeff Layton <jlayton at redhat.com>
Acked-by: J. Bruce Fields <bfields at fieldses.org>
---
 fs/locks.c |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/fs/locks.c b/fs/locks.c
index d732e22..804bb9e 100644
--- a/fs/locks.c
+++ b/fs/locks.c
@@ -1337,7 +1337,7 @@ int fcntl_getlease(struct file *filp)
 	return type;
 }
 
-int generic_add_lease(struct file *filp, long arg, struct file_lock **flp)
+static int generic_add_lease(struct file *filp, long arg, struct file_lock **flp)
 {
 	struct file_lock *fl, **before, **my_before = NULL, *lease;
 	struct dentry *dentry = filp->f_path.dentry;
@@ -1402,7 +1402,7 @@ out:
 	return error;
 }
 
-int generic_delete_lease(struct file *filp, struct file_lock **flp)
+static int generic_delete_lease(struct file *filp, struct file_lock **flp)
 {
 	struct file_lock *fl, **before;
 	struct dentry *dentry = filp->f_path.dentry;
-- 
1.7.1



From jlayton at redhat.com  Fri Jun 21 08:58:16 2013
From: jlayton at redhat.com (Jeff Layton)
Date: Fri, 21 Jun 2013 08:58:16 -0400
Subject: [PATCH v4 08/14] locks: avoid taking global lock if possible when
 waking up blocked waiters
In-Reply-To: <1371819502-26363-1-git-send-email-jlayton@redhat.com>
References: <1371819502-26363-1-git-send-email-jlayton@redhat.com>
Message-ID: <1371819502-26363-9-git-send-email-jlayton@redhat.com>

Since we always hold the i_lock when inserting a new waiter onto the
fl_block list, we can avoid taking the global lock at all if we find
that it's empty when we go to wake up blocked waiters.

Signed-off-by: Jeff Layton <jlayton at redhat.com>
---
 fs/locks.c |   15 ++++++++++++++-
 1 files changed, 14 insertions(+), 1 deletions(-)

diff --git a/fs/locks.c b/fs/locks.c
index ce302d4..84e269f 100644
--- a/fs/locks.c
+++ b/fs/locks.c
@@ -548,7 +548,10 @@ static void locks_delete_block(struct file_lock *waiter)
  * the order they blocked. The documentation doesn't require this but
  * it seems like the reasonable thing to do.
  *
- * Must be called with file_lock_lock held!
+ * Must be called with both the i_lock and file_lock_lock held. The fl_block
+ * list itself is protected by the file_lock_list, but by ensuring that the
+ * i_lock is also held on insertions we can avoid taking the file_lock_lock
+ * in some cases when we see that the fl_block list is empty.
  */
 static void __locks_insert_block(struct file_lock *blocker,
 					struct file_lock *waiter)
@@ -576,6 +579,16 @@ static void locks_insert_block(struct file_lock *blocker,
  */
 static void locks_wake_up_blocks(struct file_lock *blocker)
 {
+	/*
+	 * Avoid taking global lock if list is empty. This is safe since new
+	 * blocked requests are only added to the list under the i_lock, and
+	 * the i_lock is always held here. Note that removal from the fl_block
+	 * list does not require the i_lock, so we must recheck list_empty()
+	 * after acquiring the file_lock_lock.
+	 */
+	if (list_empty(&blocker->fl_block))
+		return;
+
 	spin_lock(&file_lock_lock);
 	while (!list_empty(&blocker->fl_block)) {
 		struct file_lock *waiter;
-- 
1.7.1



From jlayton at redhat.com  Fri Jun 21 08:58:12 2013
From: jlayton at redhat.com (Jeff Layton)
Date: Fri, 21 Jun 2013 08:58:12 -0400
Subject: [PATCH v4 04/14] locks: comment cleanups and clarifications
In-Reply-To: <1371819502-26363-1-git-send-email-jlayton@redhat.com>
References: <1371819502-26363-1-git-send-email-jlayton@redhat.com>
Message-ID: <1371819502-26363-5-git-send-email-jlayton@redhat.com>

Signed-off-by: Jeff Layton <jlayton at redhat.com>
---
 fs/locks.c         |   21 +++++++++++++--------
 include/linux/fs.h |   18 ++++++++++++++++++
 2 files changed, 31 insertions(+), 8 deletions(-)

diff --git a/fs/locks.c b/fs/locks.c
index 804bb9e..ddeab49 100644
--- a/fs/locks.c
+++ b/fs/locks.c
@@ -518,9 +518,10 @@ static void locks_insert_block(struct file_lock *blocker,
 		list_add(&waiter->fl_link, &blocked_list);
 }
 
-/* Wake up processes blocked waiting for blocker.
- * If told to wait then schedule the processes until the block list
- * is empty, otherwise empty the block list ourselves.
+/*
+ * Wake up processes blocked waiting for blocker.
+ *
+ * Must be called with the file_lock_lock held!
  */
 static void locks_wake_up_blocks(struct file_lock *blocker)
 {
@@ -806,6 +807,11 @@ static int __posix_lock_file(struct inode *inode, struct file_lock *request, str
 	}
 
 	lock_flocks();
+	/*
+	 * New lock request. Walk all POSIX locks and look for conflicts. If
+	 * there are any, either return error or put the request on the
+	 * blocker's list of waiters and the global blocked_list.
+	 */
 	if (request->fl_type != F_UNLCK) {
 		for_each_lock(inode, before) {
 			fl = *before;
@@ -844,7 +850,7 @@ static int __posix_lock_file(struct inode *inode, struct file_lock *request, str
 		before = &fl->fl_next;
 	}
 
-	/* Process locks with this owner.  */
+	/* Process locks with this owner. */
 	while ((fl = *before) && posix_same_owner(request, fl)) {
 		/* Detect adjacent or overlapping regions (if same lock type)
 		 */
@@ -930,10 +936,9 @@ static int __posix_lock_file(struct inode *inode, struct file_lock *request, str
 	}
 
 	/*
-	 * The above code only modifies existing locks in case of
-	 * merging or replacing.  If new lock(s) need to be inserted
-	 * all modifications are done bellow this, so it's safe yet to
-	 * bail out.
+	 * The above code only modifies existing locks in case of merging or
+	 * replacing. If new lock(s) need to be inserted all modifications are
+	 * done below this, so it's safe yet to bail out.
 	 */
 	error = -ENOLCK; /* "no luck" */
 	if (right && left == right && !new_fl2)
diff --git a/include/linux/fs.h b/include/linux/fs.h
index cabc08a..cba0fb5 100644
--- a/include/linux/fs.h
+++ b/include/linux/fs.h
@@ -926,6 +926,24 @@ int locks_in_grace(struct net *);
 /* that will die - we need it for nfs_lock_info */
 #include <linux/nfs_fs_i.h>
 
+/*
+ * struct file_lock represents a generic "file lock". It's used to represent
+ * POSIX byte range locks, BSD (flock) locks, and leases. It's important to
+ * note that the same struct is used to represent both a request for a lock and
+ * the lock itself, but the same object is never used for both.
+ *
+ * FIXME: should we create a separate "struct lock_request" to help distinguish
+ * these two uses?
+ *
+ * The i_flock list is ordered by:
+ *
+ * 1) lock type -- FL_LEASEs first, then FL_FLOCK, and finally FL_POSIX
+ * 2) lock owner
+ * 3) lock range start
+ * 4) lock range end
+ *
+ * Obviously, the last two criteria only matter for POSIX locks.
+ */
 struct file_lock {
 	struct file_lock *fl_next;	/* singly linked list for this inode  */
 	struct list_head fl_link;	/* doubly linked list of all locks */
-- 
1.7.1



From jlayton at redhat.com  Fri Jun 21 08:58:15 2013
From: jlayton at redhat.com (Jeff Layton)
Date: Fri, 21 Jun 2013 08:58:15 -0400
Subject: [PATCH v4 07/14] locks: protect most of the file_lock handling with
 i_lock
In-Reply-To: <1371819502-26363-1-git-send-email-jlayton@redhat.com>
References: <1371819502-26363-1-git-send-email-jlayton@redhat.com>
Message-ID: <1371819502-26363-8-git-send-email-jlayton@redhat.com>

Having a global lock that protects all of this code is a clear
scalability problem. Instead of doing that, move most of the code to be
protected by the i_lock instead. The exceptions are the global lists
that the ->fl_link sits on, and the ->fl_block list.

->fl_link is what connects these structures to the
global lists, so we must ensure that we hold those locks when iterating
over or updating these lists.

Furthermore, sound deadlock detection requires that we hold the
blocked_list state steady while checking for loops. We also must ensure
that the search and update to the list are atomic.

For the checking and insertion side of the blocked_list, push the
acquisition of the global lock into __posix_lock_file and ensure that
checking and update of the  blocked_list is done without dropping the
lock in between.

On the removal side, when waking up blocked lock waiters, take the
global lock before walking the blocked list and dequeue the waiters from
the global list prior to removal from the fl_block list.

With this, deadlock detection should be race free while we minimize
excessive file_lock_lock thrashing.

Finally, in order to avoid a lock inversion problem when handling
/proc/locks output we must ensure that manipulations of the fl_block
list are also protected by the file_lock_lock.

Signed-off-by: Jeff Layton <jlayton at redhat.com>
---
 Documentation/filesystems/Locking |   21 +++--
 fs/afs/flock.c                    |    5 +-
 fs/ceph/locks.c                   |    2 +-
 fs/ceph/mds_client.c              |    8 +-
 fs/cifs/cifsfs.c                  |    2 +-
 fs/cifs/file.c                    |   13 ++--
 fs/gfs2/file.c                    |    2 +-
 fs/lockd/svcsubs.c                |   12 ++--
 fs/locks.c                        |  164 +++++++++++++++++++++++--------------
 fs/nfs/delegation.c               |   10 +-
 fs/nfs/nfs4state.c                |    8 +-
 fs/nfsd/nfs4state.c               |    8 +-
 include/linux/fs.h                |   11 ---
 13 files changed, 154 insertions(+), 112 deletions(-)

diff --git a/Documentation/filesystems/Locking b/Documentation/filesystems/Locking
index 0706d32..413685f 100644
--- a/Documentation/filesystems/Locking
+++ b/Documentation/filesystems/Locking
@@ -344,7 +344,7 @@ prototypes:
 
 
 locking rules:
-			file_lock_lock	may block
+			inode->i_lock	may block
 fl_copy_lock:		yes		no
 fl_release_private:	maybe		no
 
@@ -357,12 +357,19 @@ prototypes:
 	int (*lm_change)(struct file_lock **, int);
 
 locking rules:
-			file_lock_lock	may block
-lm_compare_owner:	yes		no
-lm_notify:		yes		no
-lm_grant:		no		no
-lm_break:		yes		no
-lm_change		yes		no
+
+			inode->i_lock	file_lock_lock	may block
+lm_compare_owner:	yes[1]		maybe		no
+lm_notify:		yes		yes		no
+lm_grant:		no		no		no
+lm_break:		yes		no		no
+lm_change		yes		no		no
+
+[1]:	->lm_compare_owner is generally called with *an* inode->i_lock held. It
+may not be the i_lock of the inode for either file_lock being compared! This is
+the case with deadlock detection, since the code has to chase down the owners
+of locks that may be entirely unrelated to the one on which the lock is being
+acquired. When doing a search for deadlocks, the file_lock_lock is also held.
 
 --------------------------- buffer_head -----------------------------------
 prototypes:
diff --git a/fs/afs/flock.c b/fs/afs/flock.c
index 2497bf3..03fc0d1 100644
--- a/fs/afs/flock.c
+++ b/fs/afs/flock.c
@@ -252,6 +252,7 @@ static void afs_defer_unlock(struct afs_vnode *vnode, struct key *key)
  */
 static int afs_do_setlk(struct file *file, struct file_lock *fl)
 {
+	struct inode = file_inode(file);
 	struct afs_vnode *vnode = AFS_FS_I(file->f_mapping->host);
 	afs_lock_type_t type;
 	struct key *key = file->private_data;
@@ -273,7 +274,7 @@ static int afs_do_setlk(struct file *file, struct file_lock *fl)
 
 	type = (fl->fl_type == F_RDLCK) ? AFS_LOCK_READ : AFS_LOCK_WRITE;
 
-	lock_flocks();
+	spin_lock(&inode->i_lock);
 
 	/* make sure we've got a callback on this file and that our view of the
 	 * data version is up to date */
@@ -420,7 +421,7 @@ given_lock:
 	afs_vnode_fetch_status(vnode, NULL, key);
 
 error:
-	unlock_flocks();
+	spin_unlock(&inode->i_lock);
 	_leave(" = %d", ret);
 	return ret;
 
diff --git a/fs/ceph/locks.c b/fs/ceph/locks.c
index ebbf680..690f73f 100644
--- a/fs/ceph/locks.c
+++ b/fs/ceph/locks.c
@@ -192,7 +192,7 @@ void ceph_count_locks(struct inode *inode, int *fcntl_count, int *flock_count)
 
 /**
  * Encode the flock and fcntl locks for the given inode into the ceph_filelock
- * array. Must be called with lock_flocks() already held.
+ * array. Must be called with inode->i_lock already held.
  * If we encounter more of a specific lock type than expected, return -ENOSPC.
  */
 int ceph_encode_locks_to_buffer(struct inode *inode,
diff --git a/fs/ceph/mds_client.c b/fs/ceph/mds_client.c
index 4d29203..74fd289 100644
--- a/fs/ceph/mds_client.c
+++ b/fs/ceph/mds_client.c
@@ -2481,20 +2481,20 @@ static int encode_caps_cb(struct inode *inode, struct ceph_cap *cap,
 		struct ceph_filelock *flocks;
 
 encode_again:
-		lock_flocks();
+		spin_lock(&inode->i_lock);
 		ceph_count_locks(inode, &num_fcntl_locks, &num_flock_locks);
-		unlock_flocks();
+		spin_unlock(&inode->i_lock);
 		flocks = kmalloc((num_fcntl_locks+num_flock_locks) *
 				 sizeof(struct ceph_filelock), GFP_NOFS);
 		if (!flocks) {
 			err = -ENOMEM;
 			goto out_free;
 		}
-		lock_flocks();
+		spin_lock(&inode->i_lock);
 		err = ceph_encode_locks_to_buffer(inode, flocks,
 						  num_fcntl_locks,
 						  num_flock_locks);
-		unlock_flocks();
+		spin_unlock(&inode->i_lock);
 		if (err) {
 			kfree(flocks);
 			if (err == -ENOSPC)
diff --git a/fs/cifs/cifsfs.c b/fs/cifs/cifsfs.c
index 3752b9f..e81655c 100644
--- a/fs/cifs/cifsfs.c
+++ b/fs/cifs/cifsfs.c
@@ -765,7 +765,7 @@ static loff_t cifs_llseek(struct file *file, loff_t offset, int whence)
 
 static int cifs_setlease(struct file *file, long arg, struct file_lock **lease)
 {
-	/* note that this is called by vfs setlease with lock_flocks held
+	/* note that this is called by vfs setlease with i_lock held
 	   to protect *lease from going away */
 	struct inode *inode = file_inode(file);
 	struct cifsFileInfo *cfile = file->private_data;
diff --git a/fs/cifs/file.c b/fs/cifs/file.c
index 1686e40..0630710 100644
--- a/fs/cifs/file.c
+++ b/fs/cifs/file.c
@@ -1092,6 +1092,7 @@ struct lock_to_push {
 static int
 cifs_push_posix_locks(struct cifsFileInfo *cfile)
 {
+	struct inode *inode = cfile->dentry->d_inode;
 	struct cifs_tcon *tcon = tlink_tcon(cfile->tlink);
 	struct file_lock *flock, **before;
 	unsigned int count = 0, i = 0;
@@ -1102,12 +1103,12 @@ cifs_push_posix_locks(struct cifsFileInfo *cfile)
 
 	xid = get_xid();
 
-	lock_flocks();
-	cifs_for_each_lock(cfile->dentry->d_inode, before) {
+	spin_lock(&inode->i_lock);
+	cifs_for_each_lock(inode, before) {
 		if ((*before)->fl_flags & FL_POSIX)
 			count++;
 	}
-	unlock_flocks();
+	spin_unlock(&inode->i_lock);
 
 	INIT_LIST_HEAD(&locks_to_send);
 
@@ -1126,8 +1127,8 @@ cifs_push_posix_locks(struct cifsFileInfo *cfile)
 	}
 
 	el = locks_to_send.next;
-	lock_flocks();
-	cifs_for_each_lock(cfile->dentry->d_inode, before) {
+	spin_lock(&inode->i_lock);
+	cifs_for_each_lock(inode, before) {
 		flock = *before;
 		if ((flock->fl_flags & FL_POSIX) == 0)
 			continue;
@@ -1152,7 +1153,7 @@ cifs_push_posix_locks(struct cifsFileInfo *cfile)
 		lck->offset = flock->fl_start;
 		el = el->next;
 	}
-	unlock_flocks();
+	spin_unlock(&inode->i_lock);
 
 	list_for_each_entry_safe(lck, tmp, &locks_to_send, llist) {
 		int stored_rc;
diff --git a/fs/gfs2/file.c b/fs/gfs2/file.c
index ad0dc38..2398b41 100644
--- a/fs/gfs2/file.c
+++ b/fs/gfs2/file.c
@@ -896,7 +896,7 @@ out_uninit:
  * cluster; until we do, disable leases (by just returning -EINVAL),
  * unless the administrator has requested purely local locking.
  *
- * Locking: called under lock_flocks
+ * Locking: called under i_lock
  *
  * Returns: errno
  */
diff --git a/fs/lockd/svcsubs.c b/fs/lockd/svcsubs.c
index 97e8741..dc5c759 100644
--- a/fs/lockd/svcsubs.c
+++ b/fs/lockd/svcsubs.c
@@ -169,7 +169,7 @@ nlm_traverse_locks(struct nlm_host *host, struct nlm_file *file,
 
 again:
 	file->f_locks = 0;
-	lock_flocks(); /* protects i_flock list */
+	spin_lock(&inode->i_lock);
 	for (fl = inode->i_flock; fl; fl = fl->fl_next) {
 		if (fl->fl_lmops != &nlmsvc_lock_operations)
 			continue;
@@ -181,7 +181,7 @@ again:
 		if (match(lockhost, host)) {
 			struct file_lock lock = *fl;
 
-			unlock_flocks();
+			spin_unlock(&inode->i_lock);
 			lock.fl_type  = F_UNLCK;
 			lock.fl_start = 0;
 			lock.fl_end   = OFFSET_MAX;
@@ -193,7 +193,7 @@ again:
 			goto again;
 		}
 	}
-	unlock_flocks();
+	spin_unlock(&inode->i_lock);
 
 	return 0;
 }
@@ -228,14 +228,14 @@ nlm_file_inuse(struct nlm_file *file)
 	if (file->f_count || !list_empty(&file->f_blocks) || file->f_shares)
 		return 1;
 
-	lock_flocks();
+	spin_lock(&inode->i_lock);
 	for (fl = inode->i_flock; fl; fl = fl->fl_next) {
 		if (fl->fl_lmops == &nlmsvc_lock_operations) {
-			unlock_flocks();
+			spin_unlock(&inode->i_lock);
 			return 1;
 		}
 	}
-	unlock_flocks();
+	spin_unlock(&inode->i_lock);
 	file->f_locks = 0;
 	return 0;
 }
diff --git a/fs/locks.c b/fs/locks.c
index 89d898b..ce302d4 100644
--- a/fs/locks.c
+++ b/fs/locks.c
@@ -153,27 +153,37 @@ int lease_break_time = 45;
 #define for_each_lock(inode, lockp) \
 	for (lockp = &inode->i_flock; *lockp != NULL; lockp = &(*lockp)->fl_next)
 
-/* The global file_lock_list is only used for displaying /proc/locks. */
+/*
+ * The global file_lock_list is only used for displaying /proc/locks. Protected
+ * by the file_lock_lock.
+ */
 static LIST_HEAD(file_lock_list);
 
-/* The blocked_list is used to find POSIX lock loops for deadlock detection. */
+/*
+ * The blocked_list is used to find POSIX lock loops for deadlock detection.
+ * Protected by file_lock_lock.
+ */
 static LIST_HEAD(blocked_list);
 
-/* Protects the two list heads above, plus the inode->i_flock list */
+/*
+ * This lock protects the blocked_list, and the file_lock_list. Generally, if
+ * you're accessing one of those lists, you want to be holding this lock.
+ *
+ * In addition, it also protects the fl->fl_block list, and the fl->fl_next
+ * pointer for file_lock structures that are acting as lock requests (in
+ * contrast to those that are acting as records of acquired locks).
+ *
+ * Note that when we acquire this lock in order to change the above fields,
+ * we often hold the i_lock as well. In certain cases, when reading the fields
+ * protected by this lock, we can skip acquiring it iff we already hold the
+ * i_lock.
+ *
+ * In particular, adding an entry to the fl_block list requires that you hold
+ * both the i_lock and the blocked_lock_lock (acquired in that order). Deleting
+ * an entry from the list however only requires the file_lock_lock.
+ */
 static DEFINE_SPINLOCK(file_lock_lock);
 
-void lock_flocks(void)
-{
-	spin_lock(&file_lock_lock);
-}
-EXPORT_SYMBOL_GPL(lock_flocks);
-
-void unlock_flocks(void)
-{
-	spin_unlock(&file_lock_lock);
-}
-EXPORT_SYMBOL_GPL(unlock_flocks);
-
 static struct kmem_cache *filelock_cache __read_mostly;
 
 static void locks_init_lock_heads(struct file_lock *fl)
@@ -489,13 +499,17 @@ static int posix_same_owner(struct file_lock *fl1, struct file_lock *fl2)
 static inline void
 locks_insert_global_locks(struct file_lock *fl)
 {
+	spin_lock(&file_lock_lock);
 	list_add_tail(&fl->fl_link, &file_lock_list);
+	spin_unlock(&file_lock_lock);
 }
 
 static inline void
 locks_delete_global_locks(struct file_lock *fl)
 {
+	spin_lock(&file_lock_lock);
 	list_del_init(&fl->fl_link);
+	spin_unlock(&file_lock_lock);
 }
 
 static inline void
@@ -512,6 +526,8 @@ locks_delete_global_blocked(struct file_lock *waiter)
 
 /* Remove waiter from blocker's block list.
  * When blocker ends up pointing to itself then the list is empty.
+ *
+ * Must be called with file_lock_lock held.
  */
 static void __locks_delete_block(struct file_lock *waiter)
 {
@@ -520,37 +536,47 @@ static void __locks_delete_block(struct file_lock *waiter)
 	waiter->fl_next = NULL;
 }
 
-/*
- */
 static void locks_delete_block(struct file_lock *waiter)
 {
-	lock_flocks();
+	spin_lock(&file_lock_lock);
 	__locks_delete_block(waiter);
-	unlock_flocks();
+	spin_unlock(&file_lock_lock);
 }
 
 /* Insert waiter into blocker's block list.
  * We use a circular list so that processes can be easily woken up in
  * the order they blocked. The documentation doesn't require this but
  * it seems like the reasonable thing to do.
+ *
+ * Must be called with file_lock_lock held!
  */
-static void locks_insert_block(struct file_lock *blocker, 
-			       struct file_lock *waiter)
+static void __locks_insert_block(struct file_lock *blocker,
+					struct file_lock *waiter)
 {
 	BUG_ON(!list_empty(&waiter->fl_block));
 	waiter->fl_next = blocker;
 	list_add_tail(&waiter->fl_block, &blocker->fl_block);
 	if (IS_POSIX(blocker))
-		locks_insert_global_blocked(request);
+		locks_insert_global_blocked(waiter);
+}
+
+/* Must be called with i_lock held. */
+static void locks_insert_block(struct file_lock *blocker,
+					struct file_lock *waiter)
+{
+	spin_lock(&file_lock_lock);
+	__locks_insert_block(blocker, waiter);
+	spin_unlock(&file_lock_lock);
 }
 
 /*
  * Wake up processes blocked waiting for blocker.
  *
- * Must be called with the file_lock_lock held!
+ * Must be called with the inode->i_lock held!
  */
 static void locks_wake_up_blocks(struct file_lock *blocker)
 {
+	spin_lock(&file_lock_lock);
 	while (!list_empty(&blocker->fl_block)) {
 		struct file_lock *waiter;
 
@@ -562,10 +588,13 @@ static void locks_wake_up_blocks(struct file_lock *blocker)
 		else
 			wake_up(&waiter->fl_wait);
 	}
+	spin_unlock(&file_lock_lock);
 }
 
 /* Insert file lock fl into an inode's lock list at the position indicated
  * by pos. At the same time add the lock to the global file lock list.
+ *
+ * Must be called with the i_lock held!
  */
 static void locks_insert_lock(struct file_lock **pos, struct file_lock *fl)
 {
@@ -583,6 +612,8 @@ static void locks_insert_lock(struct file_lock **pos, struct file_lock *fl)
  * Wake up processes that are blocked waiting for this lock,
  * notify the FS that the lock has been cleared and
  * finally free the lock.
+ *
+ * Must be called with the i_lock held!
  */
 static void locks_delete_lock(struct file_lock **thisfl_p)
 {
@@ -652,8 +683,9 @@ void
 posix_test_lock(struct file *filp, struct file_lock *fl)
 {
 	struct file_lock *cfl;
+	struct inode *inode = file_inode(filp);
 
-	lock_flocks();
+	spin_lock(&inode->i_lock);
 	for (cfl = file_inode(filp)->i_flock; cfl; cfl = cfl->fl_next) {
 		if (!IS_POSIX(cfl))
 			continue;
@@ -666,7 +698,7 @@ posix_test_lock(struct file *filp, struct file_lock *fl)
 			fl->fl_pid = pid_vnr(cfl->fl_nspid);
 	} else
 		fl->fl_type = F_UNLCK;
-	unlock_flocks();
+	spin_unlock(&inode->i_lock);
 	return;
 }
 EXPORT_SYMBOL(posix_test_lock);
@@ -710,6 +742,7 @@ static struct file_lock *what_owner_is_waiting_for(struct file_lock *block_fl)
 	return NULL;
 }
 
+/* Must be called with the file_lock_lock held! */
 static int posix_locks_deadlock(struct file_lock *caller_fl,
 				struct file_lock *block_fl)
 {
@@ -745,7 +778,7 @@ static int flock_lock_file(struct file *filp, struct file_lock *request)
 			return -ENOMEM;
 	}
 
-	lock_flocks();
+	spin_lock(&inode->i_lock);
 	if (request->fl_flags & FL_ACCESS)
 		goto find_conflict;
 
@@ -775,9 +808,9 @@ static int flock_lock_file(struct file *filp, struct file_lock *request)
 	 * give it the opportunity to lock the file.
 	 */
 	if (found) {
-		unlock_flocks();
+		spin_unlock(&inode->i_lock);
 		cond_resched();
-		lock_flocks();
+		spin_lock(&inode->i_lock);
 	}
 
 find_conflict:
@@ -804,7 +837,7 @@ find_conflict:
 	error = 0;
 
 out:
-	unlock_flocks();
+	spin_unlock(&inode->i_lock);
 	if (new_fl)
 		locks_free_lock(new_fl);
 	return error;
@@ -834,7 +867,7 @@ static int __posix_lock_file(struct inode *inode, struct file_lock *request, str
 		new_fl2 = locks_alloc_lock();
 	}
 
-	lock_flocks();
+	spin_lock(&inode->i_lock);
 	/*
 	 * New lock request. Walk all POSIX locks and look for conflicts. If
 	 * there are any, either return error or put the request on the
@@ -852,11 +885,17 @@ static int __posix_lock_file(struct inode *inode, struct file_lock *request, str
 			error = -EAGAIN;
 			if (!(request->fl_flags & FL_SLEEP))
 				goto out;
+			/*
+			 * Deadlock detection and insertion into the blocked
+			 * locks list must be done while holding the same lock!
+			 */
 			error = -EDEADLK;
-			if (posix_locks_deadlock(request, fl))
-				goto out;
-			error = FILE_LOCK_DEFERRED;
-			locks_insert_block(fl, request);
+			spin_lock(&file_lock_lock);
+			if (likely(!posix_locks_deadlock(request, fl))) {
+				error = FILE_LOCK_DEFERRED;
+				__locks_insert_block(fl, request);
+			}
+			spin_unlock(&file_lock_lock);
 			goto out;
   		}
   	}
@@ -1006,7 +1045,7 @@ static int __posix_lock_file(struct inode *inode, struct file_lock *request, str
 		locks_wake_up_blocks(left);
 	}
  out:
-	unlock_flocks();
+	spin_unlock(&inode->i_lock);
 	/*
 	 * Free any unused locks.
 	 */
@@ -1081,14 +1120,14 @@ int locks_mandatory_locked(struct inode *inode)
 	/*
 	 * Search the lock list for this inode for any POSIX locks.
 	 */
-	lock_flocks();
+	spin_lock(&inode->i_lock);
 	for (fl = inode->i_flock; fl != NULL; fl = fl->fl_next) {
 		if (!IS_POSIX(fl))
 			continue;
 		if (fl->fl_owner != owner)
 			break;
 	}
-	unlock_flocks();
+	spin_unlock(&inode->i_lock);
 	return fl ? -EAGAIN : 0;
 }
 
@@ -1231,7 +1270,7 @@ int __break_lease(struct inode *inode, unsigned int mode)
 	if (IS_ERR(new_fl))
 		return PTR_ERR(new_fl);
 
-	lock_flocks();
+	spin_lock(&inode->i_lock);
 
 	time_out_leases(inode);
 
@@ -1281,11 +1320,11 @@ restart:
 			break_time++;
 	}
 	locks_insert_block(flock, new_fl);
-	unlock_flocks();
+	spin_unlock(&inode->i_lock);
 	error = wait_event_interruptible_timeout(new_fl->fl_wait,
 						!new_fl->fl_next, break_time);
-	lock_flocks();
-	__locks_delete_block(new_fl);
+	spin_lock(&inode->i_lock);
+	locks_delete_block(new_fl);
 	if (error >= 0) {
 		if (error == 0)
 			time_out_leases(inode);
@@ -1302,7 +1341,7 @@ restart:
 	}
 
 out:
-	unlock_flocks();
+	spin_unlock(&inode->i_lock);
 	locks_free_lock(new_fl);
 	return error;
 }
@@ -1355,9 +1394,10 @@ EXPORT_SYMBOL(lease_get_mtime);
 int fcntl_getlease(struct file *filp)
 {
 	struct file_lock *fl;
+	struct inode *inode = file_inode(filp);
 	int type = F_UNLCK;
 
-	lock_flocks();
+	spin_lock(&inode->i_lock);
 	time_out_leases(file_inode(filp));
 	for (fl = file_inode(filp)->i_flock; fl && IS_LEASE(fl);
 			fl = fl->fl_next) {
@@ -1366,7 +1406,7 @@ int fcntl_getlease(struct file *filp)
 			break;
 		}
 	}
-	unlock_flocks();
+	spin_unlock(&inode->i_lock);
 	return type;
 }
 
@@ -1460,7 +1500,7 @@ static int generic_delete_lease(struct file *filp, struct file_lock **flp)
  *	The (input) flp->fl_lmops->lm_break function is required
  *	by break_lease().
  *
- *	Called with file_lock_lock held.
+ *	Called with inode->i_lock held.
  */
 int generic_setlease(struct file *filp, long arg, struct file_lock **flp)
 {
@@ -1529,11 +1569,12 @@ static int __vfs_setlease(struct file *filp, long arg, struct file_lock **lease)
 
 int vfs_setlease(struct file *filp, long arg, struct file_lock **lease)
 {
+	struct inode *inode = file_inode(filp);
 	int error;
 
-	lock_flocks();
+	spin_lock(&inode->i_lock);
 	error = __vfs_setlease(filp, arg, lease);
-	unlock_flocks();
+	spin_unlock(&inode->i_lock);
 
 	return error;
 }
@@ -1551,6 +1592,7 @@ static int do_fcntl_delete_lease(struct file *filp)
 static int do_fcntl_add_lease(unsigned int fd, struct file *filp, long arg)
 {
 	struct file_lock *fl, *ret;
+	struct inode *inode = file_inode(filp);
 	struct fasync_struct *new;
 	int error;
 
@@ -1564,10 +1606,10 @@ static int do_fcntl_add_lease(unsigned int fd, struct file *filp, long arg)
 		return -ENOMEM;
 	}
 	ret = fl;
-	lock_flocks();
+	spin_lock(&inode->i_lock);
 	error = __vfs_setlease(filp, arg, &ret);
 	if (error) {
-		unlock_flocks();
+		spin_unlock(&inode->i_lock);
 		locks_free_lock(fl);
 		goto out_free_fasync;
 	}
@@ -1584,7 +1626,7 @@ static int do_fcntl_add_lease(unsigned int fd, struct file *filp, long arg)
 		new = NULL;
 
 	error = __f_setown(filp, task_pid(current), PIDTYPE_PID, 0);
-	unlock_flocks();
+	spin_unlock(&inode->i_lock);
 
 out_free_fasync:
 	if (new)
@@ -2108,7 +2150,7 @@ void locks_remove_flock(struct file *filp)
 			fl.fl_ops->fl_release_private(&fl);
 	}
 
-	lock_flocks();
+	spin_lock(&inode->i_lock);
 	before = &inode->i_flock;
 
 	while ((fl = *before) != NULL) {
@@ -2126,7 +2168,7 @@ void locks_remove_flock(struct file *filp)
  		}
 		before = &fl->fl_next;
 	}
-	unlock_flocks();
+	spin_unlock(&inode->i_lock);
 }
 
 /**
@@ -2140,12 +2182,12 @@ posix_unblock_lock(struct file_lock *waiter)
 {
 	int status = 0;
 
-	lock_flocks();
+	spin_lock(&file_lock_lock);
 	if (waiter->fl_next)
 		__locks_delete_block(waiter);
 	else
 		status = -ENOENT;
-	unlock_flocks();
+	spin_unlock(&file_lock_lock);
 	return status;
 }
 EXPORT_SYMBOL(posix_unblock_lock);
@@ -2259,7 +2301,7 @@ static void *locks_start(struct seq_file *f, loff_t *pos)
 {
 	loff_t *p = f->private;
 
-	lock_flocks();
+	spin_lock(&file_lock_lock);
 	*p = (*pos + 1);
 	return seq_list_start(&file_lock_list, *pos);
 }
@@ -2273,7 +2315,7 @@ static void *locks_next(struct seq_file *f, void *v, loff_t *pos)
 
 static void locks_stop(struct seq_file *f, void *v)
 {
-	unlock_flocks();
+	spin_unlock(&file_lock_lock);
 }
 
 static const struct seq_operations locks_seq_operations = {
@@ -2320,7 +2362,8 @@ int lock_may_read(struct inode *inode, loff_t start, unsigned long len)
 {
 	struct file_lock *fl;
 	int result = 1;
-	lock_flocks();
+
+	spin_lock(&inode->i_lock);
 	for (fl = inode->i_flock; fl != NULL; fl = fl->fl_next) {
 		if (IS_POSIX(fl)) {
 			if (fl->fl_type == F_RDLCK)
@@ -2337,7 +2380,7 @@ int lock_may_read(struct inode *inode, loff_t start, unsigned long len)
 		result = 0;
 		break;
 	}
-	unlock_flocks();
+	spin_unlock(&inode->i_lock);
 	return result;
 }
 
@@ -2360,7 +2403,8 @@ int lock_may_write(struct inode *inode, loff_t start, unsigned long len)
 {
 	struct file_lock *fl;
 	int result = 1;
-	lock_flocks();
+
+	spin_lock(&inode->i_lock);
 	for (fl = inode->i_flock; fl != NULL; fl = fl->fl_next) {
 		if (IS_POSIX(fl)) {
 			if ((fl->fl_end < start) || (fl->fl_start > (start + len)))
@@ -2375,7 +2419,7 @@ int lock_may_write(struct inode *inode, loff_t start, unsigned long len)
 		result = 0;
 		break;
 	}
-	unlock_flocks();
+	spin_unlock(&inode->i_lock);
 	return result;
 }
 
diff --git a/fs/nfs/delegation.c b/fs/nfs/delegation.c
index 57db324..7ec4814 100644
--- a/fs/nfs/delegation.c
+++ b/fs/nfs/delegation.c
@@ -73,20 +73,20 @@ static int nfs_delegation_claim_locks(struct nfs_open_context *ctx, struct nfs4_
 	if (inode->i_flock == NULL)
 		goto out;
 
-	/* Protect inode->i_flock using the file locks lock */
-	lock_flocks();
+	/* Protect inode->i_flock using the i_lock */
+	spin_lock(&inode->i_lock);
 	for (fl = inode->i_flock; fl != NULL; fl = fl->fl_next) {
 		if (!(fl->fl_flags & (FL_POSIX|FL_FLOCK)))
 			continue;
 		if (nfs_file_open_context(fl->fl_file) != ctx)
 			continue;
-		unlock_flocks();
+		spin_unlock(&inode->i_lock);
 		status = nfs4_lock_delegation_recall(fl, state, stateid);
 		if (status < 0)
 			goto out;
-		lock_flocks();
+		spin_lock(&inode->i_lock);
 	}
-	unlock_flocks();
+	spin_unlock(&inode->i_lock);
 out:
 	return status;
 }
diff --git a/fs/nfs/nfs4state.c b/fs/nfs/nfs4state.c
index 1fab140..ff10b4a 100644
--- a/fs/nfs/nfs4state.c
+++ b/fs/nfs/nfs4state.c
@@ -1373,13 +1373,13 @@ static int nfs4_reclaim_locks(struct nfs4_state *state, const struct nfs4_state_
 	/* Guard against delegation returns and new lock/unlock calls */
 	down_write(&nfsi->rwsem);
 	/* Protect inode->i_flock using the BKL */
-	lock_flocks();
+	spin_lock(&inode->i_lock);
 	for (fl = inode->i_flock; fl != NULL; fl = fl->fl_next) {
 		if (!(fl->fl_flags & (FL_POSIX|FL_FLOCK)))
 			continue;
 		if (nfs_file_open_context(fl->fl_file)->state != state)
 			continue;
-		unlock_flocks();
+		spin_unlock(&inode->i_lock);
 		status = ops->recover_lock(state, fl);
 		switch (status) {
 			case 0:
@@ -1406,9 +1406,9 @@ static int nfs4_reclaim_locks(struct nfs4_state *state, const struct nfs4_state_
 				/* kill_proc(fl->fl_pid, SIGLOST, 1); */
 				status = 0;
 		}
-		lock_flocks();
+		spin_lock(&inode->i_lock);
 	}
-	unlock_flocks();
+	spin_unlock(&inode->i_lock);
 out:
 	up_write(&nfsi->rwsem);
 	return status;
diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
index 316ec84..f170518 100644
--- a/fs/nfsd/nfs4state.c
+++ b/fs/nfsd/nfs4state.c
@@ -2645,13 +2645,13 @@ static void nfsd_break_one_deleg(struct nfs4_delegation *dp)
 
 	list_add_tail(&dp->dl_recall_lru, &nn->del_recall_lru);
 
-	/* only place dl_time is set. protected by lock_flocks*/
+	/* Only place dl_time is set; protected by i_lock: */
 	dp->dl_time = get_seconds();
 
 	nfsd4_cb_recall(dp);
 }
 
-/* Called from break_lease() with lock_flocks() held. */
+/* Called from break_lease() with i_lock held. */
 static void nfsd_break_deleg_cb(struct file_lock *fl)
 {
 	struct nfs4_file *fp = (struct nfs4_file *)fl->fl_owner;
@@ -4520,7 +4520,7 @@ check_for_locks(struct nfs4_file *filp, struct nfs4_lockowner *lowner)
 	struct inode *inode = filp->fi_inode;
 	int status = 0;
 
-	lock_flocks();
+	spin_lock(&inode->i_lock);
 	for (flpp = &inode->i_flock; *flpp != NULL; flpp = &(*flpp)->fl_next) {
 		if ((*flpp)->fl_owner == (fl_owner_t)lowner) {
 			status = 1;
@@ -4528,7 +4528,7 @@ check_for_locks(struct nfs4_file *filp, struct nfs4_lockowner *lowner)
 		}
 	}
 out:
-	unlock_flocks();
+	spin_unlock(&inode->i_lock);
 	return status;
 }
 
diff --git a/include/linux/fs.h b/include/linux/fs.h
index cba0fb5..805320b 100644
--- a/include/linux/fs.h
+++ b/include/linux/fs.h
@@ -1024,8 +1024,6 @@ extern int vfs_setlease(struct file *, long, struct file_lock **);
 extern int lease_modify(struct file_lock **, int);
 extern int lock_may_read(struct inode *, loff_t start, unsigned long count);
 extern int lock_may_write(struct inode *, loff_t start, unsigned long count);
-extern void lock_flocks(void);
-extern void unlock_flocks(void);
 #else /* !CONFIG_FILE_LOCKING */
 static inline int fcntl_getlk(struct file *file, struct flock __user *user)
 {
@@ -1166,15 +1164,6 @@ static inline int lock_may_write(struct inode *inode, loff_t start,
 {
 	return 1;
 }
-
-static inline void lock_flocks(void)
-{
-}
-
-static inline void unlock_flocks(void)
-{
-}
-
 #endif /* !CONFIG_FILE_LOCKING */
 
 
-- 
1.7.1



From jlayton at redhat.com  Fri Jun 21 08:58:21 2013
From: jlayton at redhat.com (Jeff Layton)
Date: Fri, 21 Jun 2013 08:58:21 -0400
Subject: [PATCH v4 13/14] seq_file: add seq_list_*_percpu helpers
In-Reply-To: <1371819502-26363-1-git-send-email-jlayton@redhat.com>
References: <1371819502-26363-1-git-send-email-jlayton@redhat.com>
Message-ID: <1371819502-26363-14-git-send-email-jlayton@redhat.com>

When we convert the file_lock_list to a set of percpu lists, we'll need
a way to iterate over them in order to output /proc/locks info. Add
some seq_list_*_percpu helpers to handle that.

Signed-off-by: Jeff Layton <jlayton at redhat.com>
Acked-by: J. Bruce Fields <bfields at fieldses.org>
---
 fs/seq_file.c            |   54 ++++++++++++++++++++++++++++++++++++++++++++++
 include/linux/seq_file.h |    6 +++++
 2 files changed, 60 insertions(+), 0 deletions(-)

diff --git a/fs/seq_file.c b/fs/seq_file.c
index 774c1eb..3135c25 100644
--- a/fs/seq_file.c
+++ b/fs/seq_file.c
@@ -921,3 +921,57 @@ struct hlist_node *seq_hlist_next_rcu(void *v,
 		return rcu_dereference(node->next);
 }
 EXPORT_SYMBOL(seq_hlist_next_rcu);
+
+/**
+ * seq_hlist_start_precpu - start an iteration of a percpu hlist array
+ * @head: pointer to percpu array of struct hlist_heads
+ * @cpu:  pointer to cpu "cursor"
+ * @pos:  start position of sequence
+ *
+ * Called at seq_file->op->start().
+ */
+struct hlist_node *
+seq_hlist_start_percpu(struct hlist_head __percpu *head, int *cpu, loff_t pos)
+{
+	struct hlist_node *node;
+
+	for_each_possible_cpu(*cpu) {
+		hlist_for_each(node, per_cpu_ptr(head, *cpu)) {
+			if (pos-- == 0)
+				return node;
+		}
+	}
+	return NULL;
+}
+EXPORT_SYMBOL(seq_hlist_start_percpu);
+
+/**
+ * seq_hlist_next_percpu - move to the next position of the percpu hlist array
+ * @v:    pointer to current hlist_node
+ * @head: pointer to percpu array of struct hlist_heads
+ * @cpu:  pointer to cpu "cursor"
+ * @pos:  start position of sequence
+ *
+ * Called at seq_file->op->next().
+ */
+struct hlist_node *
+seq_hlist_next_percpu(void *v, struct hlist_head __percpu *head,
+			int *cpu, loff_t *pos)
+{
+	struct hlist_node *node = v;
+
+	++*pos;
+
+	if (node->next)
+		return node->next;
+
+	for (*cpu = cpumask_next(*cpu, cpu_possible_mask); *cpu < nr_cpu_ids;
+	     *cpu = cpumask_next(*cpu, cpu_possible_mask)) {
+		struct hlist_head *bucket = per_cpu_ptr(head, *cpu);
+
+		if (!hlist_empty(bucket))
+			return bucket->first;
+	}
+	return NULL;
+}
+EXPORT_SYMBOL(seq_hlist_next_percpu);
diff --git a/include/linux/seq_file.h b/include/linux/seq_file.h
index 2da29ac..4e32edc 100644
--- a/include/linux/seq_file.h
+++ b/include/linux/seq_file.h
@@ -173,4 +173,10 @@ extern struct hlist_node *seq_hlist_start_head_rcu(struct hlist_head *head,
 extern struct hlist_node *seq_hlist_next_rcu(void *v,
 						   struct hlist_head *head,
 						   loff_t *ppos);
+
+/* Helpers for iterating over per-cpu hlist_head-s in seq_files */
+extern struct hlist_node *seq_hlist_start_percpu(struct hlist_head __percpu *head, int *cpu, loff_t pos);
+
+extern struct hlist_node *seq_hlist_next_percpu(void *v, struct hlist_head __percpu *head, int *cpu, loff_t *pos);
+
 #endif
-- 
1.7.1



From jlayton at redhat.com  Fri Jun 21 08:58:18 2013
From: jlayton at redhat.com (Jeff Layton)
Date: Fri, 21 Jun 2013 08:58:18 -0400
Subject: [PATCH v4 10/14] locks: turn the blocked_list into a hashtable
In-Reply-To: <1371819502-26363-1-git-send-email-jlayton@redhat.com>
References: <1371819502-26363-1-git-send-email-jlayton@redhat.com>
Message-ID: <1371819502-26363-11-git-send-email-jlayton@redhat.com>

Break up the blocked_list into a hashtable, using the fl_owner as a key.
This speeds up searching the hash chains, which is especially significant
for deadlock detection.

Note that the initial implementation assumes that hashing on fl_owner is
sufficient. In most cases it should be, with the notable exception being
server-side lockd, which compares ownership using a tuple of the
nlm_host and the pid sent in the lock request. So, this may degrade to a
single hash bucket when you only have a single NFS client. That will be
addressed in a later patch.

The careful observer may note that this patch leaves the file_lock_list
alone. There's much less of a case for turning the file_lock_list into a
hashtable. The only user of that list is the code that generates
/proc/locks, and it always walks the entire list.

Signed-off-by: Jeff Layton <jlayton at redhat.com>
Acked-by: J. Bruce Fields <bfields at fieldses.org>
---
 fs/locks.c |   25 +++++++++++++++++--------
 1 files changed, 17 insertions(+), 8 deletions(-)

diff --git a/fs/locks.c b/fs/locks.c
index 941b714..71d847c 100644
--- a/fs/locks.c
+++ b/fs/locks.c
@@ -126,6 +126,7 @@
 #include <linux/time.h>
 #include <linux/rcupdate.h>
 #include <linux/pid_namespace.h>
+#include <linux/hashtable.h>
 
 #include <asm/uaccess.h>
 
@@ -160,13 +161,21 @@ int lease_break_time = 45;
 static HLIST_HEAD(file_lock_list);
 
 /*
- * The blocked_list is used to find POSIX lock loops for deadlock detection.
- * Protected by file_lock_lock.
+ * The blocked_hash is used to find POSIX lock loops for deadlock detection.
+ * It is protected by file_lock_lock.
+ *
+ * We hash locks by lockowner in order to optimize searching for the lock a
+ * particular lockowner is waiting on.
+ *
+ * FIXME: make this value scale via some heuristic? We generally will want more
+ * buckets when we have more lockowners holding locks, but that's a little
+ * difficult to determine without knowing what the workload will look like.
  */
-static HLIST_HEAD(blocked_list);
+#define BLOCKED_HASH_BITS	7
+static DEFINE_HASHTABLE(blocked_hash, BLOCKED_HASH_BITS);
 
 /*
- * This lock protects the blocked_list, and the file_lock_list. Generally, if
+ * This lock protects the blocked_hash and the file_lock_list. Generally, if
  * you're accessing one of those lists, you want to be holding this lock.
  *
  * In addition, it also protects the fl->fl_block list, and the fl->fl_next
@@ -515,13 +524,13 @@ locks_delete_global_locks(struct file_lock *fl)
 static inline void
 locks_insert_global_blocked(struct file_lock *waiter)
 {
-	hlist_add_head(&waiter->fl_link, &blocked_list);
+	hash_add(blocked_hash, &waiter->fl_link, (unsigned long)waiter->fl_owner);
 }
 
 static inline void
 locks_delete_global_blocked(struct file_lock *waiter)
 {
-	hlist_del_init(&waiter->fl_link);
+	hash_del(&waiter->fl_link);
 }
 
 /* Remove waiter from blocker's block list.
@@ -748,7 +757,7 @@ static struct file_lock *what_owner_is_waiting_for(struct file_lock *block_fl)
 {
 	struct file_lock *fl;
 
-	hlist_for_each_entry(fl, &blocked_list, fl_link) {
+	hash_for_each_possible(blocked_hash, fl, fl_link, (unsigned long)block_fl->fl_owner) {
 		if (posix_same_owner(fl, block_fl))
 			return fl->fl_next;
 	}
@@ -884,7 +893,7 @@ static int __posix_lock_file(struct inode *inode, struct file_lock *request, str
 	/*
 	 * New lock request. Walk all POSIX locks and look for conflicts. If
 	 * there are any, either return error or put the request on the
-	 * blocker's list of waiters and the global blocked_list.
+	 * blocker's list of waiters and the global blocked_hash.
 	 */
 	if (request->fl_type != F_UNLCK) {
 		for_each_lock(inode, before) {
-- 
1.7.1



From jlayton at redhat.com  Fri Jun 21 08:58:17 2013
From: jlayton at redhat.com (Jeff Layton)
Date: Fri, 21 Jun 2013 08:58:17 -0400
Subject: [PATCH v4 09/14] locks: convert fl_link to a hlist_node
In-Reply-To: <1371819502-26363-1-git-send-email-jlayton@redhat.com>
References: <1371819502-26363-1-git-send-email-jlayton@redhat.com>
Message-ID: <1371819502-26363-10-git-send-email-jlayton@redhat.com>

Testing has shown that iterating over the blocked_list for deadlock
detection turns out to be a bottleneck. In order to alleviate that,
begin the process of turning it into a hashtable. We start by turning
the fl_link into a hlist_node and the global lists into hlists. A later
patch will do the conversion of the blocked_list to a hashtable.

Signed-off-by: Jeff Layton <jlayton at redhat.com>
Acked-by: J. Bruce Fields <bfields at fieldses.org>
---
 fs/locks.c         |   24 ++++++++++++------------
 include/linux/fs.h |    2 +-
 2 files changed, 13 insertions(+), 13 deletions(-)

diff --git a/fs/locks.c b/fs/locks.c
index 84e269f..941b714 100644
--- a/fs/locks.c
+++ b/fs/locks.c
@@ -157,13 +157,13 @@ int lease_break_time = 45;
  * The global file_lock_list is only used for displaying /proc/locks. Protected
  * by the file_lock_lock.
  */
-static LIST_HEAD(file_lock_list);
+static HLIST_HEAD(file_lock_list);
 
 /*
  * The blocked_list is used to find POSIX lock loops for deadlock detection.
  * Protected by file_lock_lock.
  */
-static LIST_HEAD(blocked_list);
+static HLIST_HEAD(blocked_list);
 
 /*
  * This lock protects the blocked_list, and the file_lock_list. Generally, if
@@ -188,7 +188,7 @@ static struct kmem_cache *filelock_cache __read_mostly;
 
 static void locks_init_lock_heads(struct file_lock *fl)
 {
-	INIT_LIST_HEAD(&fl->fl_link);
+	INIT_HLIST_NODE(&fl->fl_link);
 	INIT_LIST_HEAD(&fl->fl_block);
 	init_waitqueue_head(&fl->fl_wait);
 }
@@ -222,7 +222,7 @@ void locks_free_lock(struct file_lock *fl)
 {
 	BUG_ON(waitqueue_active(&fl->fl_wait));
 	BUG_ON(!list_empty(&fl->fl_block));
-	BUG_ON(!list_empty(&fl->fl_link));
+	BUG_ON(!hlist_unhashed(&fl->fl_link));
 
 	locks_release_private(fl);
 	kmem_cache_free(filelock_cache, fl);
@@ -500,7 +500,7 @@ static inline void
 locks_insert_global_locks(struct file_lock *fl)
 {
 	spin_lock(&file_lock_lock);
-	list_add_tail(&fl->fl_link, &file_lock_list);
+	hlist_add_head(&fl->fl_link, &file_lock_list);
 	spin_unlock(&file_lock_lock);
 }
 
@@ -508,20 +508,20 @@ static inline void
 locks_delete_global_locks(struct file_lock *fl)
 {
 	spin_lock(&file_lock_lock);
-	list_del_init(&fl->fl_link);
+	hlist_del_init(&fl->fl_link);
 	spin_unlock(&file_lock_lock);
 }
 
 static inline void
 locks_insert_global_blocked(struct file_lock *waiter)
 {
-	list_add(&waiter->fl_link, &blocked_list);
+	hlist_add_head(&waiter->fl_link, &blocked_list);
 }
 
 static inline void
 locks_delete_global_blocked(struct file_lock *waiter)
 {
-	list_del_init(&waiter->fl_link);
+	hlist_del_init(&waiter->fl_link);
 }
 
 /* Remove waiter from blocker's block list.
@@ -748,7 +748,7 @@ static struct file_lock *what_owner_is_waiting_for(struct file_lock *block_fl)
 {
 	struct file_lock *fl;
 
-	list_for_each_entry(fl, &blocked_list, fl_link) {
+	hlist_for_each_entry(fl, &blocked_list, fl_link) {
 		if (posix_same_owner(fl, block_fl))
 			return fl->fl_next;
 	}
@@ -2300,7 +2300,7 @@ static int locks_show(struct seq_file *f, void *v)
 {
 	struct file_lock *fl, *bfl;
 
-	fl = list_entry(v, struct file_lock, fl_link);
+	fl = hlist_entry(v, struct file_lock, fl_link);
 
 	lock_get_status(f, fl, *((loff_t *)f->private), "");
 
@@ -2316,14 +2316,14 @@ static void *locks_start(struct seq_file *f, loff_t *pos)
 
 	spin_lock(&file_lock_lock);
 	*p = (*pos + 1);
-	return seq_list_start(&file_lock_list, *pos);
+	return seq_hlist_start(&file_lock_list, *pos);
 }
 
 static void *locks_next(struct seq_file *f, void *v, loff_t *pos)
 {
 	loff_t *p = f->private;
 	++*p;
-	return seq_list_next(v, &file_lock_list, pos);
+	return seq_hlist_next(v, &file_lock_list, pos);
 }
 
 static void locks_stop(struct seq_file *f, void *v)
diff --git a/include/linux/fs.h b/include/linux/fs.h
index 805320b..85ef56a 100644
--- a/include/linux/fs.h
+++ b/include/linux/fs.h
@@ -946,7 +946,7 @@ int locks_in_grace(struct net *);
  */
 struct file_lock {
 	struct file_lock *fl_next;	/* singly linked list for this inode  */
-	struct list_head fl_link;	/* doubly linked list of all locks */
+	struct hlist_node fl_link;	/* node in global lists */
 	struct list_head fl_block;	/* circular list of blocked processes */
 	fl_owner_t fl_owner;
 	unsigned int fl_flags;
-- 
1.7.1



From jlayton at redhat.com  Fri Jun 21 08:58:19 2013
From: jlayton at redhat.com (Jeff Layton)
Date: Fri, 21 Jun 2013 08:58:19 -0400
Subject: [PATCH v4 11/14] locks: add a new "lm_owner_key" lock operation
In-Reply-To: <1371819502-26363-1-git-send-email-jlayton@redhat.com>
References: <1371819502-26363-1-git-send-email-jlayton@redhat.com>
Message-ID: <1371819502-26363-12-git-send-email-jlayton@redhat.com>

Currently, the hashing that the locking code uses to add these values
to the blocked_hash is simply calculated using fl_owner field. That's
valid in most cases except for server-side lockd, which validates the
owner of a lock based on fl_owner and fl_pid.

In the case where you have a small number of NFS clients doing a lot
of locking between different processes, you could end up with all
the blocked requests sitting in a very small number of hash buckets.

Add a new lm_owner_key operation to the lock_manager_operations that
will generate an unsigned long to use as the key in the hashtable.
That function is only implemented for server-side lockd, and simply
XORs the fl_owner and fl_pid.

Signed-off-by: Jeff Layton <jlayton at redhat.com>
Acked-by: J. Bruce Fields <bfields at fieldses.org>
---
 Documentation/filesystems/Locking |   16 +++++++++++-----
 fs/lockd/svclock.c                |   12 ++++++++++++
 fs/locks.c                        |   12 ++++++++++--
 include/linux/fs.h                |    1 +
 4 files changed, 34 insertions(+), 7 deletions(-)

diff --git a/Documentation/filesystems/Locking b/Documentation/filesystems/Locking
index 413685f..dfeb01b 100644
--- a/Documentation/filesystems/Locking
+++ b/Documentation/filesystems/Locking
@@ -351,6 +351,7 @@ fl_release_private:	maybe		no
 ----------------------- lock_manager_operations ---------------------------
 prototypes:
 	int (*lm_compare_owner)(struct file_lock *, struct file_lock *);
+	unsigned long (*lm_owner_key)(struct file_lock *);
 	void (*lm_notify)(struct file_lock *);  /* unblock callback */
 	int (*lm_grant)(struct file_lock *, struct file_lock *, int);
 	void (*lm_break)(struct file_lock *); /* break_lease callback */
@@ -360,16 +361,21 @@ locking rules:
 
 			inode->i_lock	file_lock_lock	may block
 lm_compare_owner:	yes[1]		maybe		no
+lm_owner_key		yes[1]		yes		no
 lm_notify:		yes		yes		no
 lm_grant:		no		no		no
 lm_break:		yes		no		no
 lm_change		yes		no		no
 
-[1]:	->lm_compare_owner is generally called with *an* inode->i_lock held. It
-may not be the i_lock of the inode for either file_lock being compared! This is
-the case with deadlock detection, since the code has to chase down the owners
-of locks that may be entirely unrelated to the one on which the lock is being
-acquired. When doing a search for deadlocks, the file_lock_lock is also held.
+[1]:	->lm_compare_owner and ->lm_owner_key are generally called with
+*an* inode->i_lock held. It may not be the i_lock of the inode
+associated with either file_lock argument! This is the case with deadlock
+detection, since the code has to chase down the owners of locks that may
+be entirely unrelated to the one on which the lock is being acquired.
+For deadlock detection however, the file_lock_lock is also held. The
+fact that these locks are held ensures that the file_locks do not
+disappear out from under you while doing the comparison or generating an
+owner key.
 
 --------------------------- buffer_head -----------------------------------
 prototypes:
diff --git a/fs/lockd/svclock.c b/fs/lockd/svclock.c
index a469098..067778b 100644
--- a/fs/lockd/svclock.c
+++ b/fs/lockd/svclock.c
@@ -744,8 +744,20 @@ static int nlmsvc_same_owner(struct file_lock *fl1, struct file_lock *fl2)
 	return fl1->fl_owner == fl2->fl_owner && fl1->fl_pid == fl2->fl_pid;
 }
 
+/*
+ * Since NLM uses two "keys" for tracking locks, we need to hash them down
+ * to one for the blocked_hash. Here, we're just xor'ing the host address
+ * with the pid in order to create a key value for picking a hash bucket.
+ */
+static unsigned long
+nlmsvc_owner_key(struct file_lock *fl)
+{
+	return (unsigned long)fl->fl_owner ^ (unsigned long)fl->fl_pid;
+}
+
 const struct lock_manager_operations nlmsvc_lock_operations = {
 	.lm_compare_owner = nlmsvc_same_owner,
+	.lm_owner_key = nlmsvc_owner_key,
 	.lm_notify = nlmsvc_notify_blocked,
 	.lm_grant = nlmsvc_grant_deferred,
 };
diff --git a/fs/locks.c b/fs/locks.c
index 71d847c..6242e0b 100644
--- a/fs/locks.c
+++ b/fs/locks.c
@@ -521,10 +521,18 @@ locks_delete_global_locks(struct file_lock *fl)
 	spin_unlock(&file_lock_lock);
 }
 
+static unsigned long
+posix_owner_key(struct file_lock *fl)
+{
+	if (fl->fl_lmops && fl->fl_lmops->lm_owner_key)
+		return fl->fl_lmops->lm_owner_key(fl);
+	return (unsigned long)fl->fl_owner;
+}
+
 static inline void
 locks_insert_global_blocked(struct file_lock *waiter)
 {
-	hash_add(blocked_hash, &waiter->fl_link, (unsigned long)waiter->fl_owner);
+	hash_add(blocked_hash, &waiter->fl_link, posix_owner_key(waiter));
 }
 
 static inline void
@@ -757,7 +765,7 @@ static struct file_lock *what_owner_is_waiting_for(struct file_lock *block_fl)
 {
 	struct file_lock *fl;
 
-	hash_for_each_possible(blocked_hash, fl, fl_link, (unsigned long)block_fl->fl_owner) {
+	hash_for_each_possible(blocked_hash, fl, fl_link, posix_owner_key(block_fl)) {
 		if (posix_same_owner(fl, block_fl))
 			return fl->fl_next;
 	}
diff --git a/include/linux/fs.h b/include/linux/fs.h
index 85ef56a..e42e04f 100644
--- a/include/linux/fs.h
+++ b/include/linux/fs.h
@@ -908,6 +908,7 @@ struct file_lock_operations {
 
 struct lock_manager_operations {
 	int (*lm_compare_owner)(struct file_lock *, struct file_lock *);
+	unsigned long (*lm_owner_key)(struct file_lock *);
 	void (*lm_notify)(struct file_lock *);	/* unblock callback */
 	int (*lm_grant)(struct file_lock *, struct file_lock *, int);
 	void (*lm_break)(struct file_lock *);
-- 
1.7.1



From jlayton at redhat.com  Fri Jun 21 08:58:22 2013
From: jlayton at redhat.com (Jeff Layton)
Date: Fri, 21 Jun 2013 08:58:22 -0400
Subject: [PATCH v4 14/14] locks: move file_lock_list to a set of percpu
 hlist_heads and convert file_lock_lock to an lglock
In-Reply-To: <1371819502-26363-1-git-send-email-jlayton@redhat.com>
References: <1371819502-26363-1-git-send-email-jlayton@redhat.com>
Message-ID: <1371819502-26363-15-git-send-email-jlayton@redhat.com>

The file_lock_list is only used for /proc/locks. The vastly common case
is for locks to be put onto the list and come off again, without ever
being traversed.

Help optimize for this use-case by moving to percpu hlist_head-s. At the
same time, we can make the locking less contentious by moving to an
lglock. When iterating over the lists for /proc/locks, we must take the
global lock and then iterate over each CPU's list in turn.

This change necessitates a new fl_link_cpu field to keep track of which
CPU the entry is on. On x86_64 at least, this field is placed within an
existing hole in the struct to avoid growing the size.

Signed-off-by: Jeff Layton <jlayton at redhat.com>
Acked-by: J. Bruce Fields <bfields at fieldses.org>
---
 fs/locks.c         |   69 +++++++++++++++++++++++++++++++++++++---------------
 include/linux/fs.h |    1 +
 2 files changed, 50 insertions(+), 20 deletions(-)

diff --git a/fs/locks.c b/fs/locks.c
index 04e2c1f..0f544cd 100644
--- a/fs/locks.c
+++ b/fs/locks.c
@@ -127,6 +127,8 @@
 #include <linux/rcupdate.h>
 #include <linux/pid_namespace.h>
 #include <linux/hashtable.h>
+#include <linux/percpu.h>
+#include <linux/lglock.h>
 
 #include <asm/uaccess.h>
 
@@ -155,11 +157,13 @@ int lease_break_time = 45;
 	for (lockp = &inode->i_flock; *lockp != NULL; lockp = &(*lockp)->fl_next)
 
 /*
- * The global file_lock_list is only used for displaying /proc/locks. Protected
- * by the file_lock_lock.
+ * The global file_lock_list is only used for displaying /proc/locks, so we
+ * keep a list on each CPU, with each list protected by its own spinlock via
+ * the file_lock_lglock. Note that alterations to the list also require that
+ * the relevant i_lock is held.
  */
-static HLIST_HEAD(file_lock_list);
-static DEFINE_SPINLOCK(file_lock_lock);
+DEFINE_STATIC_LGLOCK(file_lock_lglock);
+static DEFINE_PER_CPU(struct hlist_head, file_lock_list);
 
 /*
  * The blocked_hash is used to find POSIX lock loops for deadlock detection.
@@ -506,20 +510,30 @@ static int posix_same_owner(struct file_lock *fl1, struct file_lock *fl2)
 	return fl1->fl_owner == fl2->fl_owner;
 }
 
+/* Must be called with the i_lock held! */
 static inline void
 locks_insert_global_locks(struct file_lock *fl)
 {
-	spin_lock(&file_lock_lock);
-	hlist_add_head(&fl->fl_link, &file_lock_list);
-	spin_unlock(&file_lock_lock);
+	lg_local_lock(&file_lock_lglock);
+	fl->fl_link_cpu = smp_processor_id();
+	hlist_add_head(&fl->fl_link, this_cpu_ptr(&file_lock_list));
+	lg_local_unlock(&file_lock_lglock);
 }
 
+/* Must be called with the i_lock held! */
 static inline void
 locks_delete_global_locks(struct file_lock *fl)
 {
-	spin_lock(&file_lock_lock);
+	/*
+	 * Avoid taking lock if already unhashed. This is safe since this check
+	 * is done while holding the i_lock, and new insertions into the list
+	 * also require that it be held.
+	 */
+	if (hlist_unhashed(&fl->fl_link))
+		return;
+	lg_local_lock_cpu(&file_lock_lglock, fl->fl_link_cpu);
 	hlist_del_init(&fl->fl_link);
-	spin_unlock(&file_lock_lock);
+	lg_local_unlock_cpu(&file_lock_lglock, fl->fl_link_cpu);
 }
 
 static unsigned long
@@ -2243,6 +2257,11 @@ EXPORT_SYMBOL_GPL(vfs_cancel_lock);
 #include <linux/proc_fs.h>
 #include <linux/seq_file.h>
 
+struct locks_iterator {
+	int	li_cpu;
+	loff_t	li_pos;
+};
+
 static void lock_get_status(struct seq_file *f, struct file_lock *fl,
 			    loff_t id, char *pfx)
 {
@@ -2316,39 +2335,41 @@ static void lock_get_status(struct seq_file *f, struct file_lock *fl,
 
 static int locks_show(struct seq_file *f, void *v)
 {
+	struct locks_iterator *iter = f->private;
 	struct file_lock *fl, *bfl;
 
 	fl = hlist_entry(v, struct file_lock, fl_link);
 
-	lock_get_status(f, fl, *((loff_t *)f->private), "");
+	lock_get_status(f, fl, iter->li_pos, "");
 
 	list_for_each_entry(bfl, &fl->fl_block, fl_block)
-		lock_get_status(f, bfl, *((loff_t *)f->private), " ->");
+		lock_get_status(f, bfl, iter->li_pos, " ->");
 
 	return 0;
 }
 
 static void *locks_start(struct seq_file *f, loff_t *pos)
 {
-	loff_t *p = f->private;
+	struct locks_iterator *iter = f->private;
 
-	spin_lock(&file_lock_lock);
+	iter->li_pos = *pos + 1;
+	lg_global_lock(&file_lock_lglock);
 	spin_lock(&blocked_lock_lock);
-	*p = (*pos + 1);
-	return seq_hlist_start(&file_lock_list, *pos);
+	return seq_hlist_start_percpu(&file_lock_list, &iter->li_cpu, *pos);
 }
 
 static void *locks_next(struct seq_file *f, void *v, loff_t *pos)
 {
-	loff_t *p = f->private;
-	++*p;
-	return seq_hlist_next(v, &file_lock_list, pos);
+	struct locks_iterator *iter = f->private;
+
+	++iter->li_pos;
+	return seq_hlist_next_percpu(v, &file_lock_list, &iter->li_cpu, pos);
 }
 
 static void locks_stop(struct seq_file *f, void *v)
 {
 	spin_unlock(&blocked_lock_lock);
-	spin_unlock(&file_lock_lock);
+	lg_global_unlock(&file_lock_lglock);
 }
 
 static const struct seq_operations locks_seq_operations = {
@@ -2360,7 +2381,8 @@ static const struct seq_operations locks_seq_operations = {
 
 static int locks_open(struct inode *inode, struct file *filp)
 {
-	return seq_open_private(filp, &locks_seq_operations, sizeof(loff_t));
+	return seq_open_private(filp, &locks_seq_operations,
+					sizeof(struct locks_iterator));
 }
 
 static const struct file_operations proc_locks_operations = {
@@ -2460,9 +2482,16 @@ EXPORT_SYMBOL(lock_may_write);
 
 static int __init filelock_init(void)
 {
+	int i;
+
 	filelock_cache = kmem_cache_create("file_lock_cache",
 			sizeof(struct file_lock), 0, SLAB_PANIC, NULL);
 
+	lg_lock_init(&file_lock_lglock, "file_lock_lglock");
+
+	for_each_possible_cpu(i)
+		INIT_HLIST_HEAD(per_cpu_ptr(&file_lock_list, i));
+
 	return 0;
 }
 
diff --git a/include/linux/fs.h b/include/linux/fs.h
index e42e04f..039f2ae 100644
--- a/include/linux/fs.h
+++ b/include/linux/fs.h
@@ -953,6 +953,7 @@ struct file_lock {
 	unsigned int fl_flags;
 	unsigned char fl_type;
 	unsigned int fl_pid;
+	int fl_link_cpu;		/* what cpu's list is this on? */
 	struct pid *fl_nspid;
 	wait_queue_head_t fl_wait;
 	struct file *fl_file;
-- 
1.7.1



From jlayton at redhat.com  Fri Jun 21 08:58:20 2013
From: jlayton at redhat.com (Jeff Layton)
Date: Fri, 21 Jun 2013 08:58:20 -0400
Subject: [PATCH v4 12/14] locks: give the blocked_hash its own spinlock
In-Reply-To: <1371819502-26363-1-git-send-email-jlayton@redhat.com>
References: <1371819502-26363-1-git-send-email-jlayton@redhat.com>
Message-ID: <1371819502-26363-13-git-send-email-jlayton@redhat.com>

There's no reason we have to protect the blocked_hash and file_lock_list
with the same spinlock. With the tests I have, breaking it in two gives
a barely measurable performance benefit, but it seems reasonable to make
this locking as granular as possible.

Signed-off-by: Jeff Layton <jlayton at redhat.com>
---
 Documentation/filesystems/Locking |   16 +++++++-------
 fs/locks.c                        |   41 +++++++++++++++++++-----------------
 2 files changed, 30 insertions(+), 27 deletions(-)

diff --git a/Documentation/filesystems/Locking b/Documentation/filesystems/Locking
index dfeb01b..cf04448 100644
--- a/Documentation/filesystems/Locking
+++ b/Documentation/filesystems/Locking
@@ -359,20 +359,20 @@ prototypes:
 
 locking rules:
 
-			inode->i_lock	file_lock_lock	may block
-lm_compare_owner:	yes[1]		maybe		no
-lm_owner_key		yes[1]		yes		no
-lm_notify:		yes		yes		no
-lm_grant:		no		no		no
-lm_break:		yes		no		no
-lm_change		yes		no		no
+			inode->i_lock	blocked_lock_lock	may block
+lm_compare_owner:	yes[1]		maybe			no
+lm_owner_key		yes[1]		yes			no
+lm_notify:		yes		yes			no
+lm_grant:		no		no			no
+lm_break:		yes		no			no
+lm_change		yes		no			no
 
 [1]:	->lm_compare_owner and ->lm_owner_key are generally called with
 *an* inode->i_lock held. It may not be the i_lock of the inode
 associated with either file_lock argument! This is the case with deadlock
 detection, since the code has to chase down the owners of locks that may
 be entirely unrelated to the one on which the lock is being acquired.
-For deadlock detection however, the file_lock_lock is also held. The
+For deadlock detection however, the blocked_lock_lock is also held. The
 fact that these locks are held ensures that the file_locks do not
 disappear out from under you while doing the comparison or generating an
 owner key.
diff --git a/fs/locks.c b/fs/locks.c
index 6242e0b..04e2c1f 100644
--- a/fs/locks.c
+++ b/fs/locks.c
@@ -159,10 +159,11 @@ int lease_break_time = 45;
  * by the file_lock_lock.
  */
 static HLIST_HEAD(file_lock_list);
+static DEFINE_SPINLOCK(file_lock_lock);
 
 /*
  * The blocked_hash is used to find POSIX lock loops for deadlock detection.
- * It is protected by file_lock_lock.
+ * It is protected by blocked_lock_lock.
  *
  * We hash locks by lockowner in order to optimize searching for the lock a
  * particular lockowner is waiting on.
@@ -175,8 +176,8 @@ static HLIST_HEAD(file_lock_list);
 static DEFINE_HASHTABLE(blocked_hash, BLOCKED_HASH_BITS);
 
 /*
- * This lock protects the blocked_hash and the file_lock_list. Generally, if
- * you're accessing one of those lists, you want to be holding this lock.
+ * This lock protects the blocked_hash. Generally, if you're accessing it, you
+ * want to be holding this lock.
  *
  * In addition, it also protects the fl->fl_block list, and the fl->fl_next
  * pointer for file_lock structures that are acting as lock requests (in
@@ -191,7 +192,7 @@ static DEFINE_HASHTABLE(blocked_hash, BLOCKED_HASH_BITS);
  * both the i_lock and the blocked_lock_lock (acquired in that order). Deleting
  * an entry from the list however only requires the file_lock_lock.
  */
-static DEFINE_SPINLOCK(file_lock_lock);
+static DEFINE_SPINLOCK(blocked_lock_lock);
 
 static struct kmem_cache *filelock_cache __read_mostly;
 
@@ -544,7 +545,7 @@ locks_delete_global_blocked(struct file_lock *waiter)
 /* Remove waiter from blocker's block list.
  * When blocker ends up pointing to itself then the list is empty.
  *
- * Must be called with file_lock_lock held.
+ * Must be called with blocked_lock_lock held.
  */
 static void __locks_delete_block(struct file_lock *waiter)
 {
@@ -555,9 +556,9 @@ static void __locks_delete_block(struct file_lock *waiter)
 
 static void locks_delete_block(struct file_lock *waiter)
 {
-	spin_lock(&file_lock_lock);
+	spin_lock(&blocked_lock_lock);
 	__locks_delete_block(waiter);
-	spin_unlock(&file_lock_lock);
+	spin_unlock(&blocked_lock_lock);
 }
 
 /* Insert waiter into blocker's block list.
@@ -565,9 +566,9 @@ static void locks_delete_block(struct file_lock *waiter)
  * the order they blocked. The documentation doesn't require this but
  * it seems like the reasonable thing to do.
  *
- * Must be called with both the i_lock and file_lock_lock held. The fl_block
+ * Must be called with both the i_lock and blocked_lock_lock held. The fl_block
  * list itself is protected by the file_lock_list, but by ensuring that the
- * i_lock is also held on insertions we can avoid taking the file_lock_lock
+ * i_lock is also held on insertions we can avoid taking the blocked_lock_lock
  * in some cases when we see that the fl_block list is empty.
  */
 static void __locks_insert_block(struct file_lock *blocker,
@@ -584,9 +585,9 @@ static void __locks_insert_block(struct file_lock *blocker,
 static void locks_insert_block(struct file_lock *blocker,
 					struct file_lock *waiter)
 {
-	spin_lock(&file_lock_lock);
+	spin_lock(&blocked_lock_lock);
 	__locks_insert_block(blocker, waiter);
-	spin_unlock(&file_lock_lock);
+	spin_unlock(&blocked_lock_lock);
 }
 
 /*
@@ -601,12 +602,12 @@ static void locks_wake_up_blocks(struct file_lock *blocker)
 	 * blocked requests are only added to the list under the i_lock, and
 	 * the i_lock is always held here. Note that removal from the fl_block
 	 * list does not require the i_lock, so we must recheck list_empty()
-	 * after acquiring the file_lock_lock.
+	 * after acquiring the blocked_lock_lock.
 	 */
 	if (list_empty(&blocker->fl_block))
 		return;
 
-	spin_lock(&file_lock_lock);
+	spin_lock(&blocked_lock_lock);
 	while (!list_empty(&blocker->fl_block)) {
 		struct file_lock *waiter;
 
@@ -618,7 +619,7 @@ static void locks_wake_up_blocks(struct file_lock *blocker)
 		else
 			wake_up(&waiter->fl_wait);
 	}
-	spin_unlock(&file_lock_lock);
+	spin_unlock(&blocked_lock_lock);
 }
 
 /* Insert file lock fl into an inode's lock list at the position indicated
@@ -772,7 +773,7 @@ static struct file_lock *what_owner_is_waiting_for(struct file_lock *block_fl)
 	return NULL;
 }
 
-/* Must be called with the file_lock_lock held! */
+/* Must be called with the blocked_lock_lock held! */
 static int posix_locks_deadlock(struct file_lock *caller_fl,
 				struct file_lock *block_fl)
 {
@@ -920,12 +921,12 @@ static int __posix_lock_file(struct inode *inode, struct file_lock *request, str
 			 * locks list must be done while holding the same lock!
 			 */
 			error = -EDEADLK;
-			spin_lock(&file_lock_lock);
+			spin_lock(&blocked_lock_lock);
 			if (likely(!posix_locks_deadlock(request, fl))) {
 				error = FILE_LOCK_DEFERRED;
 				__locks_insert_block(fl, request);
 			}
-			spin_unlock(&file_lock_lock);
+			spin_unlock(&blocked_lock_lock);
 			goto out;
   		}
   	}
@@ -2212,12 +2213,12 @@ posix_unblock_lock(struct file_lock *waiter)
 {
 	int status = 0;
 
-	spin_lock(&file_lock_lock);
+	spin_lock(&blocked_lock_lock);
 	if (waiter->fl_next)
 		__locks_delete_block(waiter);
 	else
 		status = -ENOENT;
-	spin_unlock(&file_lock_lock);
+	spin_unlock(&blocked_lock_lock);
 	return status;
 }
 EXPORT_SYMBOL(posix_unblock_lock);
@@ -2332,6 +2333,7 @@ static void *locks_start(struct seq_file *f, loff_t *pos)
 	loff_t *p = f->private;
 
 	spin_lock(&file_lock_lock);
+	spin_lock(&blocked_lock_lock);
 	*p = (*pos + 1);
 	return seq_hlist_start(&file_lock_list, *pos);
 }
@@ -2345,6 +2347,7 @@ static void *locks_next(struct seq_file *f, void *v, loff_t *pos)
 
 static void locks_stop(struct seq_file *f, void *v)
 {
+	spin_unlock(&blocked_lock_lock);
 	spin_unlock(&file_lock_lock);
 }
 
-- 
1.7.1



From sfr at canb.auug.org.au  Mon Jun 24 21:37:04 2013
From: sfr at canb.auug.org.au (Stephen Rothwell)
Date: Tue, 25 Jun 2013 11:37:04 +1000
Subject: [PATCH v4 06/14] locks: encapsulate the fl_link list handling
In-Reply-To: <1371819502-26363-7-git-send-email-jlayton@redhat.com>
References: <1371819502-26363-1-git-send-email-jlayton@redhat.com>
 <1371819502-26363-7-git-send-email-jlayton@redhat.com>
Message-ID: <20130625113704.89a686a55dcec684e5e99434@canb.auug.org.au>

Hi Jeff,

Thanks for doing all this work!

Trivial comments below.

On Fri, 21 Jun 2013 08:58:14 -0400 Jeff Layton <jlayton at redhat.com> wrote:
>
> +static inline void
> +locks_insert_global_locks(struct file_lock *fl)
> +{
> +	list_add_tail(&fl->fl_link, &file_lock_list);
> +}

We generally do not use "inline" in C files any more and leave it to the
compiler to do that.  Also, without the "inline" these function headers
should all be able to fit on single lines like the others here i.e.

static void locks_insert_global_locks(struct file_lock *fl)

-- 
Cheers,
Stephen Rothwell                    sfr at canb.auug.org.au
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-afs/attachments/20130625/7161b155/attachment.sig>

From jlayton at redhat.com  Tue Jun 25 06:32:13 2013
From: jlayton at redhat.com (Jeff Layton)
Date: Tue, 25 Jun 2013 06:32:13 -0400
Subject: [PATCH v4 06/14] locks: encapsulate the fl_link list handling
In-Reply-To: <20130625113704.89a686a55dcec684e5e99434@canb.auug.org.au>
References: <1371819502-26363-1-git-send-email-jlayton@redhat.com>
 <1371819502-26363-7-git-send-email-jlayton@redhat.com>
 <20130625113704.89a686a55dcec684e5e99434@canb.auug.org.au>
Message-ID: <20130625063213.615d8a83@corrin.poochiereds.net>

On Tue, 25 Jun 2013 11:37:04 +1000
Stephen Rothwell <sfr at canb.auug.org.au> wrote:

> Hi Jeff,
> 
> Thanks for doing all this work!
> 
> Trivial comments below.
> 
> On Fri, 21 Jun 2013 08:58:14 -0400 Jeff Layton <jlayton at redhat.com> wrote:
> >
> > +static inline void
> > +locks_insert_global_locks(struct file_lock *fl)
> > +{
> > +	list_add_tail(&fl->fl_link, &file_lock_list);
> > +}
> 
> We generally do not use "inline" in C files any more and leave it to the
> compiler to do that.  Also, without the "inline" these function headers
> should all be able to fit on single lines like the others here i.e.
> 
> static void locks_insert_global_locks(struct file_lock *fl)
> 

Thanks for helping review.

Usually that makes sense, but doesn't the compiler generally determine
that by counting the number of call sites? In this case, we'll have
several call sites and it probably wouldn't inline the function. That
makes this a little less efficient since we'll have to jump to this
routine, do the list_add_tail and then jump back.

That said, I'm not opposed to doing that since these routines grow a
bit in size later and we'll only do this when a lock is acquired or
dropped. I think Al has already merged most of this set into his
for-next branch though. Perhaps I can do a patch on top of that set
that removes the inline keywords from those functions?

-- 
Jeff Layton <jlayton at redhat.com>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-afs/attachments/20130625/a9492a07/attachment.sig>

