[Resend - this time with a comma in the addresses, not a dot] Lennert Buytenhek <buytenh@wantstofly.org> wrote: > [ background: On ARM, SMP synchronisation does need barriers but device > synchronisation does not. The question is that given this, whether > mb() and friends can be NOPs on ARM or not (i.e. whether mb() is > supposed to sync against other CPUs or not, or whether only smp_mb() > can be used for this.) ] Hmmmm... I see your problem. I think the right way to deal with this is to get rid of mb(), rmb(), wmb() and read_barrier_depends() and replace them with io_mb(), io_rmb(), ... I think that there are only two places you should be using explicit memory barriers: (1) To control inter-CPU effects on an SMP system. (2) To control CPU vs device effects. > On Thu, Mar 22, 2007 at 04:17:44PM +0000, Catalin Marinas wrote: > > > Is the requirement for mb() to act correctly in the SMP case as well? > > That's what the docs seem to suggest. A couple of snippets from > memory-barriers.txt: > > [1] A write memory barrier gives a guarantee that all the STORE operations > specified before the barrier will appear to happen before all the STORE > operations specified after the barrier with respect to the other > components of the system. > > [2] A read barrier is a data dependency barrier plus a guarantee that all the > LOAD operations specified before the barrier will appear to happen before > all the LOAD operations specified after the barrier with respect to the > other components of the system. > > [3] TYPE MANDATORY SMP CONDITIONAL > =============== ======================= =========================== > GENERAL mb() smp_mb() > WRITE wmb() smp_wmb() > READ rmb() smp_rmb() > DATA DEPENDENCY read_barrier_depends() smp_read_barrier_depends() > > [4] Mandatory barriers should not be used to control SMP effects, > since mandatory barriers unnecessarily impose overhead on UP > systems. > > Note the wording of 'other components of the system' in [1] and [2] -- > the way I read it, this includes devices as well as other CPUs. Yes, but I suppose which "other components" may depend on the class of barrier used. > [4] says that mandatory barriers (i.e. from [3]: mb(), wmb(), rmb(), > read_barrier_depends()) SHOULD not be used to control SMP effects, but > it does not say that they MUST not. As it stands, mb() is a superset of smp_mb(), and rmb() of smp_rmb(), etc., so, yes, currently, mb() implies smp_mb(). However, mb() shouldn't be used if smb_mb() is sufficient as that may impact performance on a UP system. Really, mb() should only be used with respect to I/O. > > The memory-barriers.txt doc says that smp_* must be used for the SMP > > case. > > The exact wording is: > > [!] Note that SMP memory barriers _must_ be used to control the > ordering of references to shared memory on SMP systems, though > the use of locking instead is sufficient. > > This can IMHO be interpreted in two ways: > 1. If you want to control ordering of references to shared memory on > SMP systems, you must use SMP memory barriers and not any other kind > of memory barrier. If the shared memory is purely an inter-CPU effect, yes. If the shared memory is actually a device with side effects, then I/O safe memory barriers are required - mb() and co. Note that there must _also_ be safety wrt to other CPUs in the system, as other CPUs may also try to access the device. > 2. If you want to control ordering of references to shared memory on > SMP systems, you must use memory barriers, and the SMP memory barrier > is the most appropriate barrier type to use. You may use locking instead to control inter-CPU effects. Locks imply one-way permeable SMP-class memory barriers. > I'm thinking that [2] is what was intended. [1] doesn't seem consistent > with the rest of the document, but if [1] _is_ what is what was intended, > we're off the hook and mb() and friends can be NOPs on ARM. (But it'd > probably still need a thorough audit... :-/ ) I think the best way to do an audit would be to make mb() and co. deprecated, pending obsolete, and to replace them with io_mb() and co. That way people would have to eyeball any usages of mb() and co. > > This means that if code uses mb() to control SMP sharing, it is broken. > > I'm not so sure. If it's _purely_ to control inter-CPU SMP sharing, then yes, it's broken. It must use either a lock or an smp_*mb() barrier. Of course, Linus may disagree... David - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/

Lennert Buytenhek <buytenh@wantstofly.org> wrote: > Does everybody agree on these semantics, though? At least David seems > to think that mb/rmb/wmb aren't required to order normal memory accesses > against each other.. Ummm... I've just realised that your statement here is ambiguous. When you say "aren't required to", do you mean "aren't necessary to" or do you mean "don't have to"? Isn't English a fun language? Anyway, what I meant is that mb() and co. as they stand _must_ do everything smp_mb() and co do respectively, _in_ _addition_ to other side effects. mb() implies smp_mb() rmb() implies smp_rmb() wmb() implies smp_wmb() ... David - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/

Benjamin Herrenschmidt <benh@kernel.crashing.org> wrote: > Hrm... I'm not sure I like the io_* name, I think it's even more > confusing, people will never know when to use what ... I'd've thought it more obvious, but given there are several types of I/O, some of which might require different barriering to others, I can see your point. However, I think mb() unadorned is also confusing. > Maybe we should dig out again my attempt at properly defining semantics > of IO accessors and related barriers and extend it to include CPU vs. > DMA barriers. That could be useful. David - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/