[PATCH v11 20/27] iommu/exynos: allow having multiple System MMUs for a master H/W

[Tomasz Figa]

On 21.03.2014 06:21, Cho KyongHo wrote:
> On Thu, 20 Mar 2014 11:54:58 +0100, Tomasz Figa wrote:
>> On 20.03.2014 11:22, Cho KyongHo wrote:
>>> On Wed, 19 Mar 2014 16:14:57 +0100, Tomasz Figa wrote:
>>>> On 19.03.2014 14:20, Tomasz Figa wrote:
>>>>> On 19.03.2014 01:39, Cho KyongHo wrote:
>>>>>> On Tue, 18 Mar 2014 15:26:48 +0100, Tomasz Figa wrote:
>>>>>>>
>>>>>>>
>>>>>>> On 18.03.2014 14:01, Cho KyongHo wrote:
>>>>>>>> On Fri, 14 Mar 2014 17:12:03 +0100, Tomasz Figa wrote:
>>>>>>>>> Hi KyongHo,
>>>>>>>>>
>>>>>>>>> On 14.03.2014 06:10, Cho KyongHo wrote:
>>>>>>>>>> Some master device descriptor like fimc-is which is an abstraction
>>>>>>>>>> of very complex H/W may have multiple System MMUs. For those devices,
>>>>>>>>>> the design of the link between System MMU and its master H/W is
>>>>>>>>>> needed
>>>>>>>>>> to be reconsidered.
>>>>>>>>>>
>>>>>>>>>> A link structure, sysmmu_list_data is introduced that provides a link
>>>>>>>>>> to master H/W and that has a pointer to the device descriptor of a
>>>>>>>>>> System MMU. Given a device descriptor of a master H/W, it is possible
>>>>>>>>>> to traverse all System MMUs that must be controlled along with the
>>>>>>>>>> master H/W.
>>>>>>>>>
>>>>>>>>> NAK.
>>>>>>>>>
>>>>>>>>> A device driver should handle particular hardware instances
>>>>>>>>> separately,
>>>>>>>>> without abstracting a virtual hardware instance consisting of multiple
>>>>>>>>> physical ones.
>>>>>>>>>
>>>>>>>>> If such abstraction is needed, it should be done above the
>>>>>>>>> exynos-iommu
>>>>>>>>> driver, e.g. by something like iommu-composite driver that would
>>>>>>>>> aggregate several IOMMUs. Keep in mind that such IOMMUs in a group
>>>>>>>>> could
>>>>>>>>> be different, e.g. different Exynos SysMMU versions or even completely
>>>>>>>>> different IPs handled by different drivers.
>>>>>>>>>
>>>>>>>>> Still, I don't think there is a real need for such abstraction.
>>>>>>>>> Instead,
>>>>>>>>> related drivers shall be fixed to properly handle multiple memory
>>>>>>>>> masters and their IOMMUs.
>>>>>>>>>
>>>>>>>>
>>>>>>>> G2D, Scalers and FIMD of Exynos5420 has 2 System MMUs while aother
>>>>>>>> SoC like
>>>>>>>> Exynos5250 does not.
>>>>>>>>
>>>>>>>> I don't understand why you are negative to this approach.
>>>>>>>> This is the simplest than the others.
>>>>>>>>
>>>>>>>> Let me show you an example.
>>>>>>>> FIMC-IS driver just controls MCU in FIMC-IS subsystem and the
>>>>>>>> firmware of
>>>>>>>> the MCU controls all other peripherals in the subsystem. Each
>>>>>>>> peripherals
>>>>>>>> have their own System MMU. Moreover, the configuration of the
>>>>>>>> peripherals
>>>>>>>> varies according to the SoCs.
>>>>>>>>
>>>>>>>> If System MMU driver accepts multiple masters, everything is done in
>>>>>>>> DT.
>>>>>>>> But I worry that it is not easy if System MMU driver does not support
>>>>>>>> multiple masters.
>>>>>>>
>>>>>>> I believe I have stated enough reasons why this kind of implementation
>>>>>>> is bad. I'm not going to waste time repeating myself.
>>>>>>>
>>>>>>> Your concerns presented above are valid, however they are not related to
>>>>>>> what is wrong with this patch. I have given you two proper ways to
>>>>>>> handle this, none should be forced upon particular IOMMU master drivers
>>>>>>> - their authors should have the chance to select the method that works
>>>>>>> best for them.
>>>>>>>
>>>>>>
>>>>>> I don't still understand why you think this patch is wrong.
>>>>>> I think this is the best way not to think for all the driver developers
>>>>>> about other things than their business logic.
>>>>>
>>>>> I agree, but one of the ways I proposed (an iommu-composite layer above
>>>>> the IOMMU low level drivers) doesn't add any extra responsibility of
>>>>> driver developers.
>>>>>
>>>>> Moreover, it's this kind of business logic in low level drivers that is
>>>>> adding more responsibility, because it introduces additional complexity
>>>>> and makes the driver harder to read, maintain and extend in future.
>>>>>
>>>>>>
>>>>>> This does not hurt anyone and I think this is good enough.
>>>>>>
>>>>>
>>>>> Well, it is barely good enough. It is a good practice to make a low
>>>>> level driver handle a single device instance and this is how Linux
>>>>> driver model is designed.
>>>>>
>>>>> Moreover, a single device tree node _must_ represent a single hardware
>>>>> block, so you can't group multiple SysMMUs into a single device tree node.
>>>>>
>>>>
>>>> OK, you add nodes for single SysMMUs devices which is fine, sorry. I was
>>>> under impression that one kernel device (struct device) corresponds to
>>>> multiple SysMMUs, but this was before your patches, sorry. So one issue
>>>> less, but it's still not good.
>>>>
>>>
>>> Ok. Understood why you have mentioned such.
>>>
>>>>> Furthermore, if you force grouping of SysMMUs into a single virtual one,
>>>>> you enforce using the same address space for all masters of some
>>>>> particular hardware blocks, while potentially driver developers would
>>>>> like to separate them.
>>>>
>>>> Probably some clarification is needed. Your other patch adds:
>>>>
>>>> 	sysmmu_fimd0w04: sysmmu at 14640000 {
>>>> 		compatible = "samsung,sysmmu-v3.3";
>>>> 		reg = <0x14640000 0x1000>;
>>>> 		interrupt-parent = <&combiner>;
>>>> 		interrupts = <3 2>;
>>>> 		clock-names = "sysmmu", "master";
>>>> 		clocks = <&clock 422>, <&clock 421>;
>>>> 		samsung,power-domain = <&disp_pd>;
>>>> 		mmu-masters = <&fimd>;
>>>> 	};
>>>>
>>>> 	sysmmu_fimd0w123: sysmmu at 14680000 {
>>>> 		compatible = "samsung,sysmmu-v3.3";
>>>> 		reg = <0x14680000 0x1000>;
>>>> 		interrupt-parent = <&combiner>;
>>>> 		interrupts = <3 0>;
>>>> 		clock-names = "sysmmu", "master";
>>>> 		clocks = <&clock 423>, <&clock 421>;
>>>> 		samsung,power-domain = <&disp_pd>;
>>>> 		mmu-masters = <&fimd>;
>>>> 	};
>>>>
>>>>    From such description, in future FIMD driver won't be able to determine
>>>> which SysMMU is used for windows 0 and 4 and which for windows 1, 2 and
>>>> 3. However it would be desirable to handle both SysMMUs separately,
>>>> allowing each SysMMU to have only mappings for frame buffers needed by
>>>> respective FIMD windows.
>>>>
>>>
>>> If it is required to map frame buffers for the System MMU of a specific window,
>>> you can specify different phandles to mmu-masters of sysmmu_fimd0w04 and
>>> sysmmu_0w123.
>>>
>>> However, I think it is more convenient that all windows of a FIMD share the
>>> same virtual address space because
>>>    - Exynos5250: FIMD has one System MMU
>>>    - Exynos5420: FIMD has 2 System MMUs for Window 0,4 and 1,2,3
>>>    - Another SoC which is not ready for upstreaming: FIMD has 2 System MMUs for window 0,1 and 2,3,4
>>> (I also discouraged when I found a new Soc has different H/W bus topology :)
>>>
>>> For this reason, I prefer allowing a single master to have multiple System MMU.
>>>
>>
>> Well, it sure can be more convenient from programming point of view, but
>> there might be certain security-related aspects that would prefer more
>> fine-granular control over memory accesses of IP blocks.
>>
>>>>> A good interface design should not enforce any particular implementation
>>>>> and this is what we should stick to in this case as well.
>>>>>
>>>>>> If you want to provide another layer between master device and system mmu
>>>>>> as you mentioned, you do that. This patch does not restrict it.
>>>>>
>>>>> It does, as mentioned above. With a device tree listing multiple SysMMUs
>>>>> as one, you can't separate them.
>>>>
>>>> What I mean is that based on DT description, drivers should be able to
>>>> control SysMMUs separately if they want to.
>>>>
>>>
>>> As I mentioned above, drivers can control every System MMU separately.
>>
>> You are missing one point here - device tree stability. Once you specify
>> the same phandle for both masters of FIMD (or any other multi-master IP
>> block) and distribute such device tree, FIMD (or any other) driver will
>> be able to support only shared address space mode on such board.
>
> Nothing is prepared for the relationship between FIMD and System MMU.
> I can remove all 'mmu-masters' property from the nodes of System MMU until
> master drivers define their way of the relationship control.
> But I think that the current driver must work with the smallest effort.
> If the driver developer feels that the separate address space for each
> System MMUs, they can do that with a bit modification to DT.

And so I proposed the scheme below. The stream ID parameter could be 
ignored at the moment, but in future the driver should be extended to 
support it properly.

>> Probably the preferred solution would be to reuse stream ID mechanism of
>> ARM System MMU bindings and make such multi master devices specify
>> #stream-id-cells = <1> and have IDs properly assigned for each of their
>> masters. This would be the best choice for consistency reasons, as
>> existing bindings would be reused, without reinventing the wheel.
>
> Actually, the issue is not that a System MMU has multiple masters but
> a master has multiple System MMUs.

I mean "master" in the standard bus terminology, e.g. part of the IP 
that can issue bus requests. From this point of view, such FIMD variant 
has two bus masters, each with its own SysMMU. This relation should be 
included in DT.

Best regards,
Tomasz