[LEDE-DEV] [PATCH 2/3] procd: Don't use syslog before its initialization

[Michal Sojka]

Hi David,

I understand what you're saying and you are right that there can be a
lot of problems (e.g. hangs) due logging. Some comments below.

On Sat, Mar 11 2017, David Lang wrote:
> On Sat, 11 Mar 2017, Michal Sojka wrote:
>
>> On Sat, Mar 11 2017, John Crispin wrote:
>>> On 11/03/17 01:48, Michal Sojka wrote:
>>>> When procd starts a rcS script, it captures its stdout and stderr and
>>>> logs them to syslog. It could happen (and I don't know exactly why),
>>>> that a rcS scripts produces some output before openlog() is called in
>>>> procd. The result is that all subsequent logs from procd are logged
>>>> with ident and PID of some other service that happens to be started at
>>>> that time. I also have no clue why this is happening (bug in musl?),
>>>> but it is very confusing when one tries to debug boot problems.
>>>
>>> is there a way to reliably reproduce this issue ?
>>
>> I can reproduce it reliably with handful of custom packages. I'll try to
>> prepare a more simpler setup where this can be seen.
>
> This sounds like a problem with procd, capturing stdout and stderr and feeding 
> them to syslog could cause a serious problem if the error messages are from the 
> syslog daemon.
>
> it sounds like they are trying to emulate systemd, and that's not always a good 
> thing :-)
>
> sending a bunch of things to syslog before the system is up can also run into 
> problems if the syslog daemon is configured to send the logs to a remote server, 
> and can't reach it yet (all sorts of possible reasons for that).
>
> all sorts of programs spit various garbaage out to stdout or stderr that is 
> meaningless/debug stuff that really should not be turned on for production use, 
> but was missed, this can cause a lot of log messages (and stderr messages in 
> particular tend to be very badly formed)
>
> this sort of thing in the init system needs safeguards, and those need to be 
> thought about.
>
> possibly try to deliver the log to syslog, but throw them away if the attempt 
> blocks

This would require not using syslog(3), but writing a non-blocking
variant of it. That's certainly possible, but I don't know how portable
it should such be for LEDE.

> possibly try to detect that it's starting the syslog daemon and skip delivering 
> logs
>
> poosibly have the startup script for the logging daemon set some lock to turn on 
> this functionality (and leave it up to that script to figure out when to do 
> that)

But the script does not know when the logging daemon is ready. Wouldn't
it be better for the logging daemon to notify procd when it is ready and
procd would start logging via it after that?

> and it should be configurable so that people can turn on/off stdout and/or 
> stderr capturing either per-script and/or globally

Now it's configurable for services, but not for scripts that start the
services.

-Michal