[X-Unix] syslogd problems

[Kevin Stevens]

On Wed, 9 Jun 2004, William H. Magill wrote:

> It depends.
> Apple does not simply implement ALL of FreeBSD. Nor is it "current." I
> believe Darwin is now "based" on 4.5, but I'd have to go through the
> Darwin website and read up.

Of course if Apple just put the correct man pages on the system in the
first place, we wouldn't have to try to decide which alternate
documentation to use.  ;)
>
> > Maybe.  But by "unresponsive" I mean that the mouse will move around
> > the screen, but won't activate/bring forward different running windows
> > (though the window contents, for example 'top', keep refreshing).
> > That doesn't seem very disk-ish.  How would I check, some incantation
> > of vmstat or iostat?
>
> Sounds more like a memory subsystem issue.

I'm very much a casual user (I'm a network guy), without enough insight to
do much more than run top and be confused by the output.  Typically on
this 1.5GB system I show very little memory free, but a lot inactive,
which makes sense.  Here's a typical shot:

Processes:  71 total, 2 running, 69 sleeping... 140 threads 10:30:16 Load
Avg:  0.06, 0.03, 0.00 CPU usage:  0.5% user, 0.4% sys, 99.0% idle
SharedLibs: num = 109, resident = 24.3M code, 2.53M data, 6.50M LinkEdit
MemRegions: num = 3586, resident = 45.3M + 10.7M private, 46.5M shared
PhysMem:  101M wired, 151M active, 1.20G inactive, 1.45G used, 55.3M free
VM: 4.84G + 74.3M 28638(0) pageins, 12307(0) pageouts

What I don't understand is why, if I load a largish program from this
point, I get pageouts instead of simply deallocating some of that vast
pool of inactive memory (it actually does that, I've watched it with some
slow growth apps like pine sorting a large newsgroup).  Or why I need 3+
GB of swap in the first place, or why after allocating all that "base"
swap it needs to add another 74MB swap file.  But like I said, I'm not a
memory management expert by any means, and I'm willing to accept that all
is for the best in the best of all Apple-designed worlds.  ;)

> problems. Virtual memory is not as much a "done deal" as it seems to
> be. It means different things to different people, and it is
> implemented differently, as well as with different assumptions by
> virtually every OS.

Agree, there was a recent discussion on slashdot that basically
illustrated your conclusion.

>  From comments I've read and some empirical information, it appears that
> Darwin attempts to keep "maximum stuff for a program in physical
> memory" if at all possible. Possibly with minimal garbage collection.
>
> Frequently, the "expectations" that the user has about how a program
> works is quite different from those of the original programmer.

Agreed!  For example, I expect to be able to log a small number of
messages from an external source without my dual processor 1.5GB machine
locking up.  (ducking)

> That is to say, if there is only one program running, and it has a
> large data space (like syslogd would after a time running) it could
> completely fill all of physical memory and all of swap, even though
> much or most of the space is occupied by data which will not be
> referenced again... typical of a logging situation. This means that for
> a new program to launch, the memory subsystem first has to do
> significant garbage collection and cleanup.

Understand.  All I can report is that the "top" statistics don't seem to
change much in that scenario:  I don't see a lot of stuck processes, I
don't see any app tying up memory, the various memory pools seem to be
allocated in about the same way.

> One "user expectation" about syslogd is that it is a very limited
> program in its resource demands. However, since it is written to be a
> "dynamic" program, which will not suddenly fail because the volume of
> data for its buffers exceeds what the programmer decided was "a
> reasonable amount," that is undoubtedly an "unreasonable" assumption.

Ok... but it's running all the time ANYWAY.  Normal mail and news logging
is massively greater than the few alarm messages I get from my firewall,
and they have never had any impact on the system whatever.

> I suspect that the only thing which will tell you what is happening is
> something like the old unix Profiler application, which requirers that
> the hooks be compiled in.  If Darwin implemented System Accounting, it
> might be possible to find out fairly painlessly what any given
> application was actually doing and how it was interacting with the
> operating system. The problem is that such things generate sufficient
> overhead that they are eliminated from virtually all modern operating
> system implementations to enhance performance. (Not to mention the fact
> that their usage is more than slightly arcane.) Consequently one has to
> find and install them and hope that they actually do integrate
> correctly.

I've just enough expertise to flounder through a "truss" log and figure
out what file an app is looking for that it can't find.  I suspect that's
not enough.  ;)

> In the end, suspect you will discover that you are not seeing a bug,
> but rather "unexpected, but correct" behavior.

I understand the distinction you're making, but not sure what to do with
it.

> One clue would be ... if you play with the system in its sluggish
> state, does its performance improve over time? ... where time is a
> parameter defined in the same sense that the timers in SMTP are
> defined. (Which are NOT as short as most people assume them to be.)

Can't really say.  The system is not perceived as sluggish, it is
perceived as totally unresponsive, so I've not played with it for more
than a few minutes before doing a cold start.  I can say that once it
enters this state, it doesn't recover over a period of several days, since
that's how long it took me to get someone over to reset the box when I was
out of town and decided to fiddle with syslogd in the first place.  Power
cycle is the only way I know to resolve the situation.  Followed by a
reboot, BTW, as one REALLY, REALLY, BAD THING about OS X 10.3x is that the
system will not come up reliably after a power cycle.  I don't mean
unreliable as in fdisk gets stuck and you have to manually intervene, I
mean unreliable as in it appears to come up properly, after an expected
interval of "checking local disks", but is not in fact running critical
services such as mail, DNS, or remote access (ssh).  You HAVE to do a
normal reboot after a dirty start in order to ensure proper operation.
This is a problem if you aren't physically at the box, because you can't
log in remotely to do it.

My assumption is that the rc dependencies aren't bulletproof, but whatever
the cause,it's a very, very bad thing.

But I digress.  Thanks very much for your insight!

KeS