There's a bug with the Mac OS X Server 10.5.7 Apple Filing Protocol (AFP or "Mac filesharing") process where (perhaps under certain conditions) it fails to restart logging after an arbitrary period of time as specified in /Library/Preferences/com.apple.AppleFileServer.plist (which is typically edited via the Server Admin.app tool).
This bug appears to persist in Mac OS X Server 10.5.8.
For example, if you set AFP to rotate logs every 14 days, the system should compress and roll over logs located in /Library/Logs/AppleFileService every-other week. However that appears not to happen. It will begin to log after the AFP service is restarted, though that's a drastic action.
I presume that /usr/sbin/AppleFileServer handles it owns logs, rather than /usr/sbin/newsyslog, which deftly takes care of old-school logs like ftp.log, system.log and ipfw.log. It's initiated every midnight by the the launchd item /System/Library/LaunchDaemons/com.apple.newsyslog.plist. (It's worth noting that this is not the same as asl, or Apple System Log.)
Anyway, here are three workaround ideas to consider (certainly there are more).
You could just set the "Archive every [x] day(s)" option in Server Admin --> AFP --> Settings --> Logging to where x equals something like 180 — but with a lot of filesharing activity and with logging set to record many types of events, your AFP log could grow to an undesirable size.
Another workaround would be to issue a HUP to AppleFileServer. This will not affect currently connected users, but will restart logging. This could be automatically initiated periodically by your own launchd item.
My solution was to set my preference for log rotation to a very high number of days, then use newsyslog to roll over the AppleFileServerAccess and Error logs. You can edit this yourself by modifying /etc/newsyslog.conf. Here's an example of two lines to include:
/Library/Logs/AppleFileService/AppleFileServiceError.log 644 30 512 * J /private/var/run/AppleFileServer.pid
Here, I've told newsyslog to roll the AFP logs by indicating the full path, set the permissions to 644, keep 30 compressed logs, initiate the action when the logs become 512KB or more, and to use bzip2 as the compression method. The last value is the location of the pid; we need this so newsyslog can properly handle the process. Failure to include this will prevent AFP from restarting logging. And like a HUP, it does not affect logged-in users. A space is needed between each of the values; I haven't had the need to load/unload the newsyslog launchd item, either. Finally, you can tell newsyslog to run at will by the "sudo newsyslog -Fv" command. This is good to do so you can see if you have a syntax error.
See the man pages for newsyslog and newsyslog.conf for more info. (Incidentally, this has been entered as a bug at Apple's radar site; a fix is expected to be forthcoming, though perhaps not until Snow Leopard.)