From kolya@MIT.EDU Wed Jan  5 11:41:53 2000
Received: from MIT.EDU (SOUTH-STATION-ANNEX.MIT.EDU [18.72.1.2])
	by apollo.backplane.com (8.9.3/8.9.1) with SMTP id LAA48642
	for <dillon@apollo.backplane.com>; Wed, 5 Jan 2000 11:41:52 -0800 (PST)
	(envelope-from kolya@MIT.EDU)
Received: from GRAND-CENTRAL-STATION.MIT.EDU by MIT.EDU with SMTP
	id AA10743; Wed, 5 Jan 00 14:41:30 EST
Received: from melbourne-city-street.MIT.EDU (MELBOURNE-CITY-STREET.MIT.EDU [18.69.0.45])
	by grand-central-station.MIT.EDU (8.9.2/8.9.2) with ESMTP id OAA02492;
	Wed, 5 Jan 2000 14:41:48 -0500 (EST)
Received: from pepsi.mit.edu (PEPSI.MIT.EDU [18.239.2.240])
	by melbourne-city-street.MIT.EDU (8.9.3/8.9.2) with ESMTP id OAA20450;
	Wed, 5 Jan 2000 14:41:47 -0500 (EST)
Received: (from kolya@localhost) by pepsi.mit.edu (8.9.3)
	id OAA00536; Wed, 5 Jan 2000 14:41:47 -0500 (EST)
Message-Id: <200001051941.OAA00536@pepsi.mit.edu>
To: Matthew Dillon <dillon@apollo.backplane.com>
Cc: kolya@MIT.EDU
Subject: Re: Patches to Diablo (1.25) 
In-Reply-To: Your message of "Wed, 05 Jan 2000 10:28:13 PST."
             <200001051828.KAA47723@apollo.backplane.com> 
Date: Wed, 05 Jan 2000 14:41:47 -0500
From: Nickolai Zeldovich <kolya@MIT.EDU>

Hi Matt,

>     Heya Nickolai, I am about to put up diablo 1.26 on my 
>     site ( http://www.backplane.com/diablo/ ), which has all your submitted
>     changes in it!

Looks great, thanks. Actually, I was wondering if you might have missed
some additional patches I sent to you afterwards? FWIW, they had msg-id's:

    <199912010059.TAA10893@pepsi.mit.edu>
    <199912050912.EAA01103@pepsi.mit.edu>
    <199912161655.LAA149117@m4-035-15.mit.edu>

I've attached those three messages at the bottom, in mostly-mbox format.

Thanks again!

-- Nickolai Zeldovich.

--------
>From kolya@MIT.EDU Tue Nov 30 19:59:28 1999
Return-Path: <kolya@MIT.EDU>
Received: from grand-central-station.MIT.EDU by po12.mit.edu (8.9.2/4.7) id AAA09924; Wed, 1 Dec 1999 00:59:30 GMT
Received: from melbourne-city-street.MIT.EDU (MELBOURNE-CITY-STREET.MIT.EDU [18.69.0.45])
	by grand-central-station.MIT.EDU (8.9.2/8.9.2) with ESMTP id TAA22171;
	Tue, 30 Nov 1999 19:59:29 -0500 (EST)
Received: from pepsi.mit.edu (PEPSI.MIT.EDU [18.239.2.240])
	by melbourne-city-street.MIT.EDU (8.9.3/8.9.2) with ESMTP id TAA15244;
	Tue, 30 Nov 1999 19:59:28 -0500 (EST)
Received: (from kolya@localhost) by pepsi.mit.edu (8.9.3)
	id TAA10893; Tue, 30 Nov 1999 19:59:28 -0500 (EST)
Message-Id: <199912010059.TAA10893@pepsi.mit.edu>
To: Matthew Dillon <dillon@apollo.backplane.com>
cc: kolya@MIT.EDU
Subject: Re: Patches to Diablo (1.25) 
In-reply-to: Your message of "Sat, 20 Nov 1999 10:14:08 PST."
             <199911201814.KAA12425@apollo.backplane.com> 
Date: Tue, 30 Nov 1999 19:59:28 -0500
From: Nickolai Zeldovich <kolya@MIT.EDU>

Hi Matt,

While running dreaderd, I've found two more things that I think might
be worth fixing:

 -- The overview FD cache wasn't being flushed for feed-only forks,
    and thus kept certain groups from being properly resized upon
    expire. I've added the appropriate call to FlushOverCache for
    feed-only forks; this is my fault, I forgot about this case in
    my previous patch.

 -- Stale clients were being accumulated (ACTV in dreaderd.status but
    no TCP connection) because a file descriptor which corresponds to
    a closed TCP connection is not considered writeable by select(),
    at least on Solaris 2.6. I've changed the select() loop to poll
    all fd's for read, and close the connection if recv() returns 0.
    I'm not too sure about the efficiency here, but so far it seems
    to be working OK.

If you have any suggestions for a better impementation, particularly
for the second case, I'd be very much interested in hearing them.

Here's the actual diff (applies on top of my previous patch):

--- dreaderd/reader.c	1999/11/29 05:22:10	1.1
+++ dreaderd/reader.c	1999/12/01 00:11:18	1.4
@@ -90,7 +90,8 @@
 	struct timeval tv;
 	fd_set rfds = RFds;
 	fd_set wfds = WFds;
-	int i;
+	fd_set read_only_to_find_eof_fds;
+	int i, sel_r;
 
 	/*
 	 *  Get next scheduled timeout, no more then 2 seconds
@@ -107,9 +108,24 @@
 	    NWriteServAct, NWriteServers
 	);
 
-	select(MaxFds, &rfds, &wfds, NULL, &tv);
+	FD_ZERO(&read_only_to_find_eof_fds);
+	for (i = 0; i < MaxFds; ++i) {
+	    if (FD_ISSET(i, &wfds) && (!(FD_ISSET(i, &rfds)))) {
+		FD_SET(i, &rfds);
+		FD_SET(i, &read_only_to_find_eof_fds);
+	    }
+	}
+
+	sel_r = select(MaxFds, &rfds, &wfds, NULL, &tv);
 	gettimeofday(&CurTime, NULL);
 
+	if(sel_r < 0)
+	    logit(LOG_CRIT,
+		  "select error: %s (rfds=0x%x, wfds=0x%x)",
+		  strerror(errno),
+		  rfds,
+		  wfds);
+
 	/*
 	 * select is critical, don't make unnecessary system calls.  Only
 	 * test the time every 10 selects (20 seconds worst case), and
@@ -118,6 +134,8 @@
 	 * THREAD_POST threads.
 	 *
 	 * We do not startup spool/post servers for feed-only forks
+	 *
+	 * However, flush overview cache even for feed-only forks.
 	 */
 
 	if (FeedOnlyServer <= 0) {
@@ -157,6 +175,58 @@
 		}
 		ltime = t;
 		counter = 0;
+	    }
+	} else {
+	    /*
+	     * For a feed-only server, we only flush the overview FD
+	     * cache every 5 minutes, and with a greater granularity.
+	     * It should cycle much faster than that normally, and this
+	     * is to prevent idle feed-only forks from keeping locks.
+	     */
+
+	    if (++counter == 50) {
+		time_t t = CurTime.tv_sec;
+		if (ltime) {
+		    ftime += t - ltime;
+		}
+
+		if (ftime < -5 || ftime >= 300) {
+		    FlushOverCache();
+		    ftime = 0;
+		}
+	    }
+	}
+
+	for (i = 0; i < MaxFds; ++i) {
+	    if (FD_ISSET(i, &rfds) && FD_ISSET(i, &read_only_to_find_eof_fds)) {
+		char junk_byte;
+		int ret_val;
+		/*
+		 * This FD is not marked for reading, but select() claims
+		 * it has something to say. We don't actually want to read
+		 * from it, but we do want to close it if the associated
+		 * connection is dead.
+		 */
+		FD_CLR(i, &rfds);
+
+		/* Use recv() with MSG_PEEK to see if it's closed.
+		 * We shouldn't block because we're O_NONBLOCK.
+		 */
+		ret_val = recv(i, &junk_byte, 1, MSG_PEEK);
+
+		/* If ret_val is zero, this means the socket is closed.
+		 * Blast it. Otherwise, ignore it.
+		 */
+		if(ret_val == 0) {
+		    ForkDesc *desc;
+		    if((desc = FindThread(i, -1)) != NULL) {
+			Connection *conn = desc->d_Data;
+			if(conn) {
+			    DeleteConnection(conn);
+			}
+			DelThread(desc);
+		    }
+		}
 	    }
 	}

>From kolya@MIT.EDU Sun Dec 05 04:12:33 1999
Return-Path: <kolya@MIT.EDU>
Received: from grand-central-station.MIT.EDU by po12.mit.edu (8.9.2/4.7) id JAA25578; Sun, 5 Dec 1999 09:12:36 GMT
Received: from melbourne-city-street.MIT.EDU (MELBOURNE-CITY-STREET.MIT.EDU [18.69.0.45])
	by grand-central-station.MIT.EDU (8.9.2/8.9.2) with ESMTP id EAA04202;
	Sun, 5 Dec 1999 04:12:35 -0500 (EST)
Received: from pepsi.mit.edu (PEPSI.MIT.EDU [18.239.2.240])
	by melbourne-city-street.MIT.EDU (8.9.3/8.9.2) with ESMTP id EAA02578;
	Sun, 5 Dec 1999 04:12:34 -0500 (EST)
Received: (from kolya@localhost) by pepsi.mit.edu (8.9.3)
	id EAA01103; Sun, 5 Dec 1999 04:12:34 -0500 (EST)
Message-Id: <199912050912.EAA01103@pepsi.mit.edu>
To: dillon@backplane.com
cc: kolya@MIT.EDU
Subject: Re: Patches to Diablo (1.25) 
In-reply-to: Your message of "Tue, 30 Nov 1999 19:59:28 EST."
             <199912010059.TAA10893@pepsi.mit.edu> 
Date: Sun, 05 Dec 1999 04:12:33 -0500
From: Nickolai Zeldovich <kolya@MIT.EDU>

Eep, looks like my previous patch introduced a bug which causes invalid
pointer dereferencing if the client disconnects in the middle of a server
request on its behalf. Sorry for fragmenting them so much. Here's one way
to fix this:

--- dreaderd/reader.c	1999/12/01 00:11:18	1.4
+++ dreaderd/reader.c	1999/12/04 22:26:32
@@ -221,10 +221,22 @@
 		    ForkDesc *desc;
 		    if((desc = FindThread(i, -1)) != NULL) {
 			Connection *conn = desc->d_Data;
+			int still_in_use = 0;
+
 			if(conn) {
-			    DeleteConnection(conn);
+			    /*
+			     * If this client has an outstanding server request,
+			     * avoid killing it, because this conn is referenced
+			     * elsewhere.
+			     */
+			    if(conn->co_SReq != NULL)
+				still_in_use = 1;
+
+			    if(!still_in_use)
+				DeleteConnection(conn);
 			}
-			DelThread(desc);
+			if(!still_in_use)
+			    DelThread(desc);
 		    }
 		}
 	    }
 
>From kolya@MIT.EDU Thu Dec 16 11:55:17 1999
Return-Path: <kolya@MIT.EDU>
Received: from grand-central-station.MIT.EDU by po12.mit.edu (8.9.2/4.7) id QAA23330; Thu, 16 Dec 1999 16:55:23 GMT
Received: from melbourne-city-street.MIT.EDU (MELBOURNE-CITY-STREET.MIT.EDU [18.69.0.45])
	by grand-central-station.MIT.EDU (8.9.2/8.9.2) with ESMTP id LAA18854;
	Thu, 16 Dec 1999 11:55:22 -0500 (EST)
Received: from m4-035-15.mit.edu (M4-035-15.MIT.EDU [18.53.0.245])
	by melbourne-city-street.MIT.EDU (8.9.3/8.9.2) with ESMTP id LAA16829;
	Thu, 16 Dec 1999 11:55:18 -0500 (EST)
Received: (from kolya@localhost) by m4-035-15.mit.edu (8.9.3)
	id LAA149117; Thu, 16 Dec 1999 11:55:17 -0500 (EST)
Message-Id: <199912161655.LAA149117@m4-035-15.mit.edu>
To: dillon@backplane.com
cc: kolya@MIT.EDU
Subject: More diablo patches
Date: Thu, 16 Dec 1999 11:55:17 -0500
From: Nickolai Zeldovich <kolya@MIT.EDU>

Hi Matt,

Here are some additional modifications I've made to diablo-1.25-REL,
and found quite useful in my situation; perhaps you could integrate
these into the next release, if you think these would be useful to
others? Some description of what these diffs intend to do:

 -- Parallelizing dexpireover:
	Add an option to make dexpireover fork N times and have each
	fork process 1/Nth of the overview information. This gives a
	noticeable speedup when running dexpireover with -e (history
	based expiration): on our reader Ultra60, we have two 10kRPM
	disks striped together for dhistory + overview (not spool),
	and 'dexpireover -e' takes ~16 hours; doing it in parallel
	with 4 forks cuts the time down to about 6 hours.

	[ As a side note, running dexpireover with -o, using the
	  dexpover.dat file generated by dexpire, takes 10 min ]

 -- Generating Lines: headers in headfeed mode:
	Added an option to dspoolout, and a corresponding switch to
	dnewslink, to generate Lines: headers if one is not already
	present, in headfeed mode. Allows always having a meaningful
	line count in overview.

 -- No-initial-response bug in dnewslink:
	If dnewslink connects to a remote server, and doesn't get any
	reply, it hangs forever try to read from the socket. Set the
	KillFd before trying to read initial response from remote.

Incidentally, did you get a chance to glance over any of my previous
diffs yet?

TIA,

-- Nickolai Zeldovich.


*** util/dexpireover.c	1999/11/29 06:04:37	1.2
--- util/dexpireover.c	1999/12/06 07:34:28	1.9
***************
*** 7,13 ****
   *    for specific rights granted.
   *
   * dexpireover [-a] [-v[N]] [-w grpwildcard] [-f dactive.kp] [-NB] [-n] [-O[n]]
!  *		[-s] [-R] [-e] [-o]
   *
   * 	In this incarnation, dexpireover cleans up overview information as
   *	specified in the 'x' fields in dexpire.ctl (see the sample dexpire.ctl)
--- 7,13 ----
   *    for specific rights granted.
   *
   * dexpireover [-a] [-v[N]] [-w grpwildcard] [-f dactive.kp] [-NB] [-n] [-O[n]]
!  *             [-s] [-R] [-e] [-o] [-pN]
   *
   * 	In this incarnation, dexpireover cleans up overview information as
   *	specified in the 'x' fields in dexpire.ctl (see the sample dexpire.ctl)
***************
*** 21,28 ****
   *	dexpireover to adjust expirations based on remote server retentions.
   *	It does not do this yet either.
   *
!  * Modifications by Nickolai Zeldovich to allow spool-based expiration
!  * (ExpireBySpool and ExpireFromFile)
   */
  
  #include <dreaderd/defs.h>
--- 21,32 ----
   *	dexpireover to adjust expirations based on remote server retentions.
   *	It does not do this yet either.
   *
!  *	Modifications by Nickolai Zeldovich to allow spool-based expiration
!  *	(ExpireBySpool and ExpireFromFile)
!  *
!  *	Specifying the -pN option will fork N dexpireover processes and
!  *	perform the expiration process in parallel. This is useful to speed
!  *	up ExpireBySpool where dhistory lookups take a long time.
   */
  
  #include <dreaderd/defs.h>
***************
*** 62,67 ****
--- 66,76 ----
  #define DEXPOVER_READ_BUFFER_SIZE	4096
  #define DEXPOVER_HASH_SIZE		32768
  
+ /* Since we only look at the first char of the first-level directory,
+  * we do not support more than 16 forks.
+  */
+ #define	MAX_PAR_COUNT			16
+ 
  /*
   * These aren't really buckets, they're parts of a bucket
   */
***************
*** 90,95 ****
--- 99,105 ----
  int nearestPower(int n);
  void ReadDExpOverList(void);
  int expOverListCheckExpired(hash_t *hv);
+ int hexCharToInt(char c);
  
  int UpdateBegArtNoOpt = 0;
  int UpdateCTSOpt = 0;
***************
*** 105,110 ****
--- 115,121 ----
  int OldGroups = 0;
  int UseExpireBySpool = 0;
  int UseExpireFromFile = 0;
+ int ParallelCount = 0;
  char *Wild;
  
  bucket_t *dexpover_msgid_hash;
***************
*** 115,120 ****
--- 126,133 ----
      int i;
      int freeSpaceTarget = 0;
      char *dbfile = NULL;
+     int ParallelIdx = 0;
+     int ParallelPid[MAX_PAR_COUNT];
  
      LoadDiabloConfig(ac, av);
  
***************
*** 164,169 ****
--- 177,191 ----
  	case 'o':
  	    UseExpireFromFile = 1;
  	    break;
+ 	case 'p':
+ 	    if (*ptr)
+ 		ParallelCount = strtol(ptr, NULL, 0);
+ 	    else
+ 		ParallelCount = 1;
+ 	    if (ParallelCount > MAX_PAR_COUNT)
+ 		ParallelCount = MAX_PAR_COUNT;
+ 	    /* Note that a parcount of 1 doesn't do anything useful. */
+ 	    break;
  	case 'R':
  	    RewriteDataOpt = 1;
  	    break;
***************
*** 195,200 ****
--- 217,268 ----
      }
  
      /*
+      * Read in the list of expired msgid hashes, if we are using it
+      */
+ 
+     if (UseExpireFromFile)
+ 	ReadDExpOverList();
+ 
+     /*
+      * fork off parallel copies of dexpireover at this point
+      */
+ 
+     if (ParallelCount) {
+ 	char *stdout_buffer;
+ 
+ 	for (ParallelIdx=0; ParallelIdx < ParallelCount; ParallelIdx++) {
+ 	    int pid;
+ 
+ 	    pid = fork();
+ 	    if(pid == 0)
+ 		break;
+ 	    ParallelPid[ParallelIdx] = pid;
+ 	}
+ 
+ 	stdout_buffer = (char *)malloc(BUFSIZ);
+ 	setvbuf(stdout, stdout_buffer, _IOLBF, BUFSIZ);
+ 
+ 	if (ParallelIdx == ParallelCount) {
+ 	    pid_t pid;
+ 	    int remaining = ParallelCount;
+ 
+ 	    while (remaining) {
+ 		while (remaining && ((pid = wait3(NULL, 0, NULL)) > 0)) {
+ 		    for (i=0; i<ParallelCount; i++)
+ 			if(ParallelPid[i] == pid) {
+ 			    ParallelPid[i] = 0;
+ 			    --remaining;
+ 			}
+ 		}
+ 	    }
+ 
+ 	    printf("Parallelizing dexpireover (%d forks) finished.\n",
+ 	       ParallelCount);
+ 	    exit(0);
+ 	}
+     }
+ 
+     /*
       * Open active file database
       */
  
***************
*** 222,234 ****
  	HistoryOpen(NULL, 0);
  
      /*
-      * Read in the list of expired msgid hashes, if we are using it
-      */
- 
-     if (UseExpireFromFile)
- 	ReadDExpOverList();
- 
-     /*
       * scan dactive.kp
       */
  
--- 290,295 ----
***************
*** 297,303 ****
  	    while ((den = readdir(dir)) != NULL) {
  		if (strlen(den->d_name) == 2 && 
  		    isalnum(den->d_name[0]) &&
! 		    isalnum(den->d_name[1])
  		) {
  		    DIR *dir2;
  
--- 358,370 ----
  	    while ((den = readdir(dir)) != NULL) {
  		if (strlen(den->d_name) == 2 && 
  		    isalnum(den->d_name[0]) &&
! 		    isalnum(den->d_name[1]) &&
! 		    /* We explicitly use the first char, because overview
! 		     * sizes appear to be not evenly distributed wrt second
! 		     * char.
! 		     */
! 		    (ParallelCount ? ((hexCharToInt(den->d_name[0]) % ParallelCount) ==
! 				      ParallelIdx) : 1)
  		) {
  		    DIR *dir2;
  
***************
*** 1420,1426 ****
      }
  
      fclose(DExpOverList);
!     remove(path);
  }
  
  int
--- 1487,1524 ----
      }
  
      fclose(DExpOverList);
! }
! 
! int
! hexCharToInt(char c)
! {
!     return
! 	(c == '0') ? 0 :
! 	(c == '1') ? 1 :
! 	(c == '2') ? 2 :
! 	(c == '3') ? 3 :
! 	(c == '4') ? 4 :
! 	(c == '5') ? 5 :
! 	(c == '6') ? 6 :
! 	(c == '7') ? 7 :
! 	(c == '8') ? 8 :
! 	(c == '9') ? 9 :
! 
! 	(c == 'a') ? 10 :
! 	(c == 'b') ? 11 :
! 	(c == 'c') ? 12 :
! 	(c == 'd') ? 13 :
! 	(c == 'e') ? 14 :
! 	(c == 'f') ? 15 :
! 
! 	(c == 'A') ? 10 :
! 	(c == 'B') ? 11 :
! 	(c == 'C') ? 12 :
! 	(c == 'D') ? 13 :
! 	(c == 'E') ? 14 :
! 	(c == 'F') ? 15 :
! 
! 	-1;
  }
  
  int
*** man/dexpireover.8	1999/12/05 18:04:17	1.1
--- man/dexpireover.8	1999/12/05 18:51:24	1.4
***************
*** 1,4 ****
! .\" $Revision: 1.1 $
  .TH DEXPIREOVER 8
  .SH NAME
  dexpireover \- Diablo program to expire overview files for diablo reader side
--- 1,4 ----
! .\" $Revision: 1.4 $
  .TH DEXPIREOVER 8
  .SH NAME
  dexpireover \- Diablo program to expire overview files for diablo reader side
***************
*** 41,46 ****
--- 41,49 ----
  [
  .B \-O[n]
  ]
+ [
+ .B \-pN
+ ]
  
  .SH DESCRIPTION
  .IR DExpireOver
***************
*** 161,166 ****
--- 164,181 ----
  .PP
  When dexpireover deletes a newsgroup, it does not delete the overview data
  for that group until the next time dexpireover is run.
+ .PP
+ .B \-pN
+ .PP
+ Spawn N working dexpireover processes, and one master process which
+ waits for the parcount children to exit. Useful for speeding up the
+ \-e option, when your dhistory disk(s) can support multiple dhistory
+ lookups at one time without slowing down linearly. Twice the number
+ of disks on your striped /news partition is probably not an
+ unreasonable number to use.
+ .PP
+ Using a value of N greater than 16 is not currently supported. Values
+ other than powers of 2 are likely to generate uneven load across forks.
  .PP
  .SH INTERRUPTING
  Generally speaking, you should not interrupt a running dexpireover.  Doing
*** util/dnewslink.c	1999/12/04 06:06:05	1.2
--- util/dnewslink.c	1999/12/16 06:48:59	1.7
***************
*** 187,192 ****
--- 187,193 ----
  int NoCheckOpt = 0;
  int RxBufSize = 0;
  int HeaderOnlyFeed = 0;
+ int GenLinesHeader = 0;
  struct stat CurSt;
  
  Stream StreamAry[MAXSTREAM];
***************
*** 318,323 ****
--- 319,327 ----
  	case 'H':
  	    HeaderOnlyFeed = 1;
  	    break;
+ 	case 'L':
+ 	    GenLinesHeader = 1;
+ 	    break;
  	case 'h':
  	    HostName = (*ptr) ? ptr : av[++i];
  	    break;
***************
*** 1022,1027 ****
--- 1026,1036 ----
  	clearCommandBuffer();
  	cfd = fd;
  
+ 	/* Set KillFd here to avoid deadlock when the remote server fails
+ 	 * to respond after initial connection
+ 	 */
+ 	KillFd = cfd;
+ 
  	switch(commandResponse(cfd, &ptr, NULL)) {
  	case OK_CANPOST:	/* innd	*/
  	case OK_NOPOST:		/* nntpd may still allow news transfers */
***************
*** 1030,1035 ****
--- 1039,1045 ----
  	    dl_logit("connect: %s\n", (ptr ? ptr : "(unknown error)"));
  	    stprintf("%s %s connect: err resp", HostName, CurrentBatchFile + CBFIndex);
  	    clearCommandBuffer();
+ 	    KillFd = -1;
  	    close(cfd);
  	    cfd = -1;
  	    fd = -1;
***************
*** 1075,1080 ****
--- 1085,1091 ----
  		dl_logit("mode headfeed failed: %s\n", (ptr ? ptr : "<Unexpected EOF>"));
  		stprintf("%s %s connect: mode headfeed failed", HostName, CurrentBatchFile + CBFIndex);
  		clearCommandBuffer();
+ 		KillFd = -1;
  		close(cfd);
  		cfd = -1;
  		sprintf(LastErrBuf, "mode headfeed command failed");
***************
*** 1196,1201 ****
--- 1207,1213 ----
      int r = -1;
      int multiArtFile = 0;
      int wasControl = 0;
+     int haveLines = 0;
  
      HeaderSize = 0;
  
***************
*** 1243,1248 ****
--- 1255,1262 ----
  
  	    if (i - b > 8 && strncasecmp(ptr + b, "Control:", 8) == 0)
  		wasControl = 1;
+ 	    if (i - b > 6 && strncasecmp(ptr + b, "Lines:", 6) == 0)
+ 		haveLines = 1;
  
  	    /*
  	     * skip newline. if i > fsize, we hit the end of the file without
***************
*** 1260,1272 ****
  	     */
  	    if (inHeaders && i - b == 1) {
  		/*
  		 * add Bytes: header, dreaderd needs it!  header feeds 
  		 * *require* it, but non-header-only feeds can calculate it 
  		 * themselves.
  		 */
  		if (HeaderOnlyFeed) {
  		    char tmp[32];
! 		    sprintf(tmp, "Bytes: %d\r\n", size);
  		    commandWrite(cfd, tmp, strlen(tmp));
  		    HeaderSize += strlen(tmp);
  		}
--- 1274,1309 ----
  	     */
  	    if (inHeaders && i - b == 1) {
  		/*
+ 		 * add Lines: header, if requested. note that we can only
+ 		 * do so if we are going to otherwise discard the rest of
+ 		 * the article (HeaderOnlyFeed and not control).
+ 		 */
+ 		if(HeaderOnlyFeed && GenLinesHeader &&
+ 		   haveLines == 0 && wasControl == 0) {
+ 		    int lineCount = 0;
+ 		    char tmp[32];
+ 
+ 		    while (i < size) {
+ 			while (i < size && ptr[i] != '\n')
+ 			    ++i;
+ 			if (i < size && ptr[i] == '\n')
+ 			    ++lineCount;
+ 			++i;
+ 		    }
+ 
+ 		    snprintf(tmp, sizeof(tmp), "Lines: %d\r\n", lineCount);
+ 		    commandWrite(cfd, tmp, strlen(tmp));
+ 		    HeaderSize += strlen(tmp);
+ 		}
+ 
+ 		/*
  		 * add Bytes: header, dreaderd needs it!  header feeds 
  		 * *require* it, but non-header-only feeds can calculate it 
  		 * themselves.
  		 */
  		if (HeaderOnlyFeed) {
  		    char tmp[32];
! 		    snprintf(tmp, sizeof(tmp), "Bytes: %d\r\n", size);
  		    commandWrite(cfd, tmp, strlen(tmp));
  		    HeaderSize += strlen(tmp);
  		}
*** util/dspoolout.c	1999/12/15 06:25:30	1.1
--- util/dspoolout.c	1999/12/16 16:35:07	1.3
***************
*** 18,23 ****
--- 18,24 ----
  #define SF_NOBATCH	0x0004
  #define SF_HEADFEED	0x0008
  #define SF_NOCHECK	0x0010
+ #define SF_GENLINES	0x0020
  
  int MaxRun = 2;		/* default MaxRun			       */
  int MinFlushSecs = 0;	/* minimum time between flshs if queue not caught up */
***************
*** 179,184 ****
--- 180,187 ----
  				flags |= SF_REALTIME;
  			    } else if (strcmp(maxqStr, "nobatch") == 0) {
  				flags |= SF_NOBATCH;
+ 			    } else if (strcmp(maxqStr, "genlines") == 0) {
+ 				flags |= SF_GENLINES;
  			    } else if (strncmp(maxqStr, "bind=", 5) == 0) {
  				obip = maxqStr + 5;
  			    } else if (maxqStr[0] == 'q') {
***************
*** 481,486 ****
--- 484,490 ----
  				((flags & SF_NOSTREAM) ? "-i" : "-nop"),
  				((flags & SF_HEADFEED) ? "-H" : "-nop"),
  				((flags & SF_NOCHECK) ? "-I" : "-nop"),
+ 				((flags & SF_GENLINES) ? "-L" : "-nop"),
  				outboundIpStr,
  				txBufSizeStr,
  				rxBufSizeStr,
***************
*** 552,557 ****
--- 556,562 ----
  				((flags & SF_NOSTREAM) ? "-i" : "-nop"),
  				((flags & SF_HEADFEED) ? "-H" : "-nop"),
  				((flags & SF_NOCHECK) ? "-I" : "-nop"),
+ 				((flags & SF_GENLINES) ? "-L" : "-nop"),
  				outboundIpStr,
  				txBufSizeStr,
  				rxBufSizeStr,
*** man/dnewslink.8	1999/12/15 06:44:09	1.1
--- man/dnewslink.8	1999/12/15 06:47:45	1.2
***************
*** 1,4 ****
! .\" $Revision: 1.1 $
  .TH DNEWSLINK 8
  .SH NAME
  dnewslink \- Diablo program to feed news to remote systems
--- 1,4 ----
! .\" $Revision: 1.2 $
  .TH DNEWSLINK 8
  .SH NAME
  dnewslink \- Diablo program to feed news to remote systems
***************
*** 24,29 ****
--- 24,30 ----
  .B \-N 
  numSeq
  .B \-H
+ .B \-L
  .B \-h 
  remote-host
  .B \-w 
***************
*** 164,169 ****
--- 165,178 ----
  'mode headfeed' as a safety feature to prevent a header-only feed from being
  sent to a normal feed and propogating outward.  Beyond that, the header-only
  feed uses the same ihave/check/takethis protocols that a normal feed uses.
+ 
+ .PP
+ .B \-L
+ .PP
+ This option will enable generation of Lines: headers in headfeed mode
+ (see above); the reason for this option is to always have a meaningful
+ lines count in overview. Specifying the 'genlines' option in dnntpspool.ctl
+ will make dspoolout pass \-L to dnewslink.
  
  .PP
  .B \-h remote-host

