NVMe APST high latency power states being skipped

[Kai-Heng Feng]

On Wed, May 24, 2017 at 6:09 AM,  <Mario.Limonciello at dell.com> wrote:
[snipped]
>
> I know Kai Heng has looked at a /lot/ of disks. I've got stats from a few
> of them, but there are many more that I haven't seen.

Not really, others I've seen have rather low latency. We have the same
high latency ones.

> Perhaps Chris or Kai Heng might be able to provide a better parameter
> to base off from other experience.

A quick summary: we need at least 61000 to make all of them be able to
enters PS3,
1100000 for PS4.

I'll do some performance testing on the 1100000 latency one.

Is there anyway to observe the power state transition in NVMe?

[snipped]

> I think separate from the effort of getting the default right this makes sense.
> To me the most important default should be getting the disk into at least
> the first non-operational state even if latency is bad.
>
> Then provide the ability to block that non-operational state or go into
> other non-operational states that would be otherwise blocked due to latency
> by user code.

We can add this to TLP by greping the PS3 latencies out of `nvme
id-ctrl` and do some math, but it will be ugly.

>
>>
>> >
>> > Kai Heng can comment more on the testing they've done and the performance
>> > impact, but I understand that by tweaking those knobs they've been able to
>> > get all these disks into at least PS3 and saved a lot of power.
>> >
>> > We could go work with the TLP project  or power top guys and have them
>> > go and tweak the various sysfs knobs to make more of these disks work,
>> > but I would rather the kernel had good defaults across this collection of disks.
>>
>> Agreed.

Other than TLP/powertop, we should make this easy to work with
something like thermald.
NVMe is quite hot. It can be quite useful to let thermald controls the
max available power state directly via sysfs knob. Fanless devices
will benefit a lot from this.