From rminnich at gmail.com Tue May 2 03:25:33 2017 From: rminnich at gmail.com (ron minnich) Date: Mon, 01 May 2017 17:25:33 +0000 Subject: [TUHS] dmr note on BSD's sins Message-ID: OK, I recall a note dmr wrote probably in the late 70s/early 80s when folks at UCB had (iirc) extended the symbol name size in C programs to essentially unlimited. This followed on (iirc) file names going beyond 14 characters. The rough outline was that dmr was calling out the revisions for being too general, and the phrase "BSD sins" sticks in my head (sins as a verb). I'm reminded of this by something that happened with some interns recently, as they wanted to make something immensely complex to cover a case that basically never happened. I was trying to point out that you can go overboard on that sort of thing, and it would have been nice to have such a quote handy -- anyone else remember it? ron -------------- next part -------------- An HTML attachment was scrubbed... URL: From toby at telegraphics.com.au Tue May 2 04:09:07 2017 From: toby at telegraphics.com.au (Toby Thain) Date: Mon, 1 May 2017 14:09:07 -0400 Subject: [TUHS] dmr note on BSD's sins In-Reply-To: References: Message-ID: <1189db4c-a1f5-6d03-45ed-81f1a2c51685@telegraphics.com.au> On 2017-05-01 1:25 PM, ron minnich wrote: > OK, I recall a note dmr wrote probably in the late 70s/early 80s when > folks at UCB had (iirc) extended the symbol name size in C programs to > essentially unlimited. This followed on (iirc) file names going beyond > 14 characters. > > The rough outline was that dmr was calling out the revisions for being > too general, and the phrase "BSD sins" sticks in my head (sins as a verb). > > I'm reminded of this by something that happened with some interns > recently, as they wanted to make something immensely complex to cover a > case that basically never happened. I was trying to point out that you > can go overboard on that sort of thing, and it would have been nice to > have such a quote handy -- anyone else remember it? Not what you're looking for, but in recent times we'd just say YAGNI - You Ain't Gonna Need It. --T > > ron From scj at yaccman.com Wed May 3 02:11:52 2017 From: scj at yaccman.com (Steve Johnson) Date: Tue, 02 May 2017 09:11:52 -0700 Subject: [TUHS] dmr note on BSD's sins In-Reply-To: <1189db4c-a1f5-6d03-45ed-81f1a2c51685@telegraphics.com.au> Message-ID: I recall a paper Dennis wrote (maybe more like a note) that was titled      echo -c considered harmful (I think it was -c).   It decried the tendency, now completely out of control, for everybody and their dog to piddle on perfectly good code just because it's "open". Let's face it, programming languages today are more like shells -- they rise or fall on the quality of their libraries, and the expression of algorithms (or even simplicity or efficiency) are secondary.  And operating systems seem to exist mainly to host window systems that vie for how deeply and obscurely they can hide vital options, and pride themselves that striking any key at any time, no matter how inadvertent, will cause some unexpected and unwanted action. I'm reminded of a friend of mine who took a math course, and said that the professor "proved theorems by induction on the number of lemmas". And yes, I'm sounding like a grumpy old fart this morning... Steve ----- Original Message ----- From: "Toby Thain" To: Cc: Sent:Mon, 1 May 2017 14:09:07 -0400 Subject:Re: [TUHS] dmr note on BSD's sins On 2017-05-01 1:25 PM, ron minnich wrote: > OK, I recall a note dmr wrote probably in the late 70s/early 80s when > folks at UCB had (iirc) extended the symbol name size in C programs to > essentially unlimited. This followed on (iirc) file names going beyond > 14 characters. > > The rough outline was that dmr was calling out the revisions for being > too general, and the phrase "BSD sins" sticks in my head (sins as a verb). > > I'm reminded of this by something that happened with some interns > recently, as they wanted to make something immensely complex to cover a > case that basically never happened. I was trying to point out that you > can go overboard on that sort of thing, and it would have been nice to > have such a quote handy -- anyone else remember it? Not what you're looking for, but in recent times we'd just say YAGNI - You Ain't Gonna Need It. --T > > ron -------------- next part -------------- An HTML attachment was scrubbed... URL: From fair-tuhs at netbsd.org Wed May 3 02:39:23 2017 From: fair-tuhs at netbsd.org (Erik E. Fair) Date: Tue, 02 May 2017 09:39:23 -0700 Subject: [TUHS] dmr note on BSD's sins In-Reply-To: References: <1189db4c-a1f5-6d03-45ed-81f1a2c51685@telegraphics.com.au> Message-ID: <3980.1493743163@cesium.clock.org> There is also Rob Pike's USENIX paper "UNIX Style, or cat -v Considered Harmful" http://harmful.cat-v.org/cat-v/ which definitely called out UCB CSRG "sins" in the view of Pike & Kernighan. Perhaps this is what you're thinking of? Erik Fair From rminnich at gmail.com Wed May 3 02:43:43 2017 From: rminnich at gmail.com (ron minnich) Date: Tue, 02 May 2017 16:43:43 +0000 Subject: [TUHS] dmr note on BSD's sins In-Reply-To: <3980.1493743163@cesium.clock.org> References: <1189db4c-a1f5-6d03-45ed-81f1a2c51685@telegraphics.com.au> <3980.1493743163@cesium.clock.org> Message-ID: well, my memory is dmr. But it was a long time ago ... On Tue, May 2, 2017 at 9:39 AM Erik E. Fair wrote: > There is also Rob Pike's USENIX paper "UNIX Style, or cat -v Considered > Harmful" > > http://harmful.cat-v.org/cat-v/ > > which definitely called out UCB CSRG "sins" in the view of Pike & > Kernighan. Perhaps this is what you're thinking of? > > Erik Fair > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dds at aueb.gr Wed May 3 06:28:25 2017 From: dds at aueb.gr (Diomidis Spinellis) Date: Tue, 2 May 2017 23:28:25 +0300 Subject: [TUHS] dmr note on BSD's sins In-Reply-To: References: Message-ID: On 02/05/2017 19:11, Steve Johnson wrote: > I recall a paper Dennis wrote (maybe more like a note) that was titled > echo -c considered harmful > (I think it was -c). It decried the tendency, now completely out of > control, for everybody and their dog to piddle on perfectly good code > just because it's "open". There's definitely Rob Pike's talk "UNIX Style, or cat -v Considered Harmful", which he delivered at the 1983 Usenix Association Conference and Software Tools USers Group Summer Conference. Unfortunately, I can't find it online. It's interesting that the talk's date is now closer to the birth of Unix than to the present. Diomidis From bigatojj at gmail.com Wed May 3 06:34:03 2017 From: bigatojj at gmail.com (=?UTF-8?Q?Daniel_Camol=C3=AAs?=) Date: Tue, 2 May 2017 17:34:03 -0300 Subject: [TUHS] dmr note on BSD's sins In-Reply-To: References: Message-ID: There you go: http://harmful.cat-v.org/cat-v/ Em 2 de mai de 2017 17:29, "Diomidis Spinellis" escreveu: On 02/05/2017 19:11, Steve Johnson wrote: > I recall a paper Dennis wrote (maybe more like a note) that was titled > echo -c considered harmful > (I think it was -c). It decried the tendency, now completely out of > control, for everybody and their dog to piddle on perfectly good code > just because it's "open". > There's definitely Rob Pike's talk "UNIX Style, or cat -v Considered Harmful", which he delivered at the 1983 Usenix Association Conference and Software Tools USers Group Summer Conference. Unfortunately, I can't find it online. It's interesting that the talk's date is now closer to the birth of Unix than to the present. Diomidis -------------- next part -------------- An HTML attachment was scrubbed... URL: From scj at yaccman.com Wed May 3 06:34:38 2017 From: scj at yaccman.com (Steve Johnson) Date: Tue, 02 May 2017 13:34:38 -0700 Subject: [TUHS] dmr note on BSD's sins In-Reply-To: Message-ID: Thanks for the correction -- that is indeed the paper I was thinking of... Steve ----- Original Message ----- From: "Diomidis Spinellis" To:"Steve Johnson" , "Toby Thain" , Cc: Sent:Tue, 2 May 2017 23:28:25 +0300 Subject:Re: [TUHS] dmr note on BSD's sins On 02/05/2017 19:11, Steve Johnson wrote: > I recall a paper Dennis wrote (maybe more like a note) that was titled > echo -c considered harmful > (I think it was -c). It decried the tendency, now completely out of > control, for everybody and their dog to piddle on perfectly good code > just because it's "open". There's definitely Rob Pike's talk "UNIX Style, or cat -v Considered Harmful", which he delivered at the 1983 Usenix Association Conference and Software Tools USers Group Summer Conference. Unfortunately, I can't find it online. It's interesting that the talk's date is now closer to the birth of Unix than to the present. Diomidis -------------- next part -------------- An HTML attachment was scrubbed... URL: From akosela at andykosela.com Wed May 3 07:22:36 2017 From: akosela at andykosela.com (Andy Kosela) Date: Tue, 2 May 2017 16:22:36 -0500 Subject: [TUHS] dmr note on BSD's sins In-Reply-To: References: Message-ID: On Tuesday, May 2, 2017, Diomidis Spinellis wrote: > On 02/05/2017 19:11, Steve Johnson wrote: > >> I recall a paper Dennis wrote (maybe more like a note) that was titled >> echo -c considered harmful >> (I think it was -c). It decried the tendency, now completely out of >> control, for everybody and their dog to piddle on perfectly good code >> just because it's "open". >> > > There's definitely Rob Pike's talk "UNIX Style, or cat -v Considered > Harmful", which he delivered at the 1983 Usenix Association Conference and > Software Tools USers Group Summer Conference. Unfortunately, I can't find > it online. It's interesting that the talk's date is now closer to the > birth of Unix than to the present. > > This talk is today as legendary as Plato's "On the Good" lecture[1]. [1] https://www.jstor.org/stable/4182081 --Andy -------------- next part -------------- An HTML attachment was scrubbed... URL: From doug at cs.dartmouth.edu Wed May 3 10:52:51 2017 From: doug at cs.dartmouth.edu (Doug McIlroy) Date: Tue, 02 May 2017 20:52:51 -0400 Subject: [TUHS] dmr note on BSD's sins Message-ID: <201705030052.v430qp2K004255@coolidge.cs.Dartmouth.EDU> With Steve's eloquent grump and cat -v on the table, I can't help re-citing the peerless cardinal sin of less --help | wc Doug From dave at horsfall.org Wed May 3 10:59:13 2017 From: dave at horsfall.org (Dave Horsfall) Date: Wed, 3 May 2017 10:59:13 +1000 (EST) Subject: [TUHS] dmr note on BSD's sins In-Reply-To: <201705030052.v430qp2K004255@coolidge.cs.Dartmouth.EDU> References: <201705030052.v430qp2K004255@coolidge.cs.Dartmouth.EDU> Message-ID: On Tue, 2 May 2017, Doug McIlroy wrote: > With Steve's eloquent grump and cat -v on the table, I can't help > re-citing the peerless cardinal sin of > less --help | wc Speaking of "cat", what really drives me nuts is "cat file | cmd"... What's wrong with "cmd < file" (or to really confuse newbies, "< file cmd")? -- Dave Horsfall DTM (VK2KFU) "Those who don't understand security will suffer." From crossd at gmail.com Wed May 3 22:54:19 2017 From: crossd at gmail.com (Dan Cross) Date: Wed, 3 May 2017 08:54:19 -0400 Subject: [TUHS] dmr note on BSD's sins In-Reply-To: References: <201705030052.v430qp2K004255@coolidge.cs.Dartmouth.EDU> Message-ID: On Tue, May 2, 2017 at 8:59 PM, Dave Horsfall wrote: > On Tue, 2 May 2017, Doug McIlroy wrote: > > With Steve's eloquent grump and cat -v on the table, I can't help > > re-citing the peerless cardinal sin of > > less --help | wc > > Speaking of "cat", what really drives me nuts is "cat file | cmd"... > > What's wrong with "cmd < file" (or to really confuse newbies, "< file > cmd")? Ooo! Ooo! Ooo! I've actually got something for this.... First of all, there's nothing strictly speaking *wrong* with 'cmd < file' and cat 'cat file | cmd' is definitely overused, often unintentionally and out of ignorance. However, 'cmd From krewat at kilonet.net Wed May 3 23:09:52 2017 From: krewat at kilonet.net (Arthur Krewat) Date: Wed, 3 May 2017 09:09:52 -0400 Subject: [TUHS] dmr note on BSD's sins In-Reply-To: References: <201705030052.v430qp2K004255@coolidge.cs.Dartmouth.EDU> Message-ID: <58cf42d9-b153-7831-5012-16e5544faf2f@kilonet.net> Not to mention, you can cat multiple files - as in concatenate :) On 5/3/2017 8:54 AM, Dan Cross wrote: > On Tue, May 2, 2017 at 8:59 PM, Dave Horsfall > wrote: > > On Tue, 2 May 2017, Doug McIlroy wrote: > > With Steve's eloquent grump and cat -v on the table, I can't help > > re-citing the peerless cardinal sin of > > less --help | wc > > Speaking of "cat", what really drives me nuts is "cat file | cmd"... > > What's wrong with "cmd < file" (or to really confuse newbies, "< file > cmd")? > > > Ooo! Ooo! Ooo! I've actually got something for this.... > > First of all, there's nothing strictly speaking *wrong* with 'cmd < > file' and cat 'cat file | cmd' is definitely overused, often > unintentionally and out of ignorance. > > However, 'cmd command; `cat` can be useful when the file parameter may be optional. > E.g., 'cat "$@" | ...'. Now some folks will immediately respond by > saying, "many commands will read from stdin if a filename is not > presented on the command line, so why not, 'cmd "$@"'?" And that's > certainly a valid question, to which I would answer that the semantics > of a command sometimes subtly change when presented with one or more > filenames as argument (e.g. 'grep'), so using `cat` may suppress that > behavior if desired. "But `grep` has the `-h` option to tell it not to > print the filename!" Yes, but `grep` is just *one command* and not > *all* of them do. The point being that 'cat' in a pipeline has it's > place, even if that place is rarely the place we see it. > > Another, related use to cover up one of the more odious of recent > design decisions in Unix-like systems is to use `cat` at the *end* of > the pipeline. Some programs change behavior if they know that they are > writing into a tty; one can suppress that if one terminates the > pipeline in `cat`. This is surely a case of mis-using a feature to > mask a bug, but it's often useful regardless. > > - Dan C. > From cym224 at gmail.com Wed May 3 23:41:11 2017 From: cym224 at gmail.com (Nemo) Date: Wed, 3 May 2017 09:41:11 -0400 Subject: [TUHS] dmr note on BSD's sins Message-ID: On 3 May 2017 at 09:09, Arthur Krewat wrote: > Not to mention, you can cat multiple files - as in concatenate :) Along these lines, who said "Cat went to Berkely, came back waving flags." N. From michael at kjorling.se Wed May 3 23:46:32 2017 From: michael at kjorling.se (Michael =?utf-8?B?S2rDtnJsaW5n?=) Date: Wed, 3 May 2017 13:46:32 +0000 Subject: [TUHS] dmr note on BSD's sins In-Reply-To: References: <201705030052.v430qp2K004255@coolidge.cs.Dartmouth.EDU> Message-ID: <20170503134632.GF26584@yeono.kjorling.se> On 3 May 2017 10:59 +1000, from dave at horsfall.org (Dave Horsfall): > What's wrong with "cmd < file" (or to really confuse newbies, "< file > cmd")? If you want to _really_ confuse a newbie, stick an "file" _in the middle of_ a long command, and watch hilarity ensue. Or the "" that someone here mentioned, if you are so inclined... -- Michael Kjörling • https://michael.kjorling.se • michael at kjorling.se “People who think they know everything really annoy those of us who know we don’t.” (Bjarne Stroustrup) From clemc at ccc.com Thu May 4 00:57:07 2017 From: clemc at ccc.com (Clem Cole) Date: Wed, 3 May 2017 10:57:07 -0400 Subject: [TUHS] dmr note on BSD's sins In-Reply-To: References: <201705030052.v430qp2K004255@coolidge.cs.Dartmouth.EDU> Message-ID: below... On Tue, May 2, 2017 at 8:59 PM, Dave Horsfall wrote: > On Tue, 2 May 2017, Doug McIlroy wrote: > > > With Steve's eloquent grump and cat -v on the table, I can't help > > re-citing the peerless cardinal sin of > > less --help | wc > To Doug -- touché > Speaking of "cat", what really drives me nuts is "cat file | cmd"... > > What's wrong with "cmd < file" (or to really confuse newbies, "< file > cmd")? Not to be argumentative, but Dave, please be careful. I agree that idiom is misused, but there are times when I think it is appropriate, which is one of the reason why I love the Unix guiding principle of not "forcing" the user to do one thing or another because we know better, but letting education and good 'taste' be the guide. ​I think Justice Potter Stewart describe​d pornography of knowing it when he sees it. IMO: there are time when using cat is in better taste that a file redirection. For instance, I can think of a few examples when I need to direct input from multiple files, using a more complex 'wye' style pipe line or a redirection from backward quotes, that the cat foo sequence seems like its easier to understand/debug. [Maybe it shows my inner schizophrenia and heard voices from multiple places that I kinda like the 'wye' command]. Clem -------------- next part -------------- An HTML attachment was scrubbed... URL: From arnold at skeeve.com Thu May 4 01:34:28 2017 From: arnold at skeeve.com (arnold at skeeve.com) Date: Wed, 03 May 2017 09:34:28 -0600 Subject: [TUHS] dmr note on BSD's sins In-Reply-To: References: <201705030052.v430qp2K004255@coolidge.cs.Dartmouth.EDU> Message-ID: <201705031534.v43FYSCY028025@freefriends.org> > [Maybe it shows my inner > schizophrenia and heard voices from multiple places that I kinda like the > 'wye' command]. Um, what is the 'wye' command? Never heard of it... Thanks, Arnold From clemc at ccc.com Thu May 4 01:44:38 2017 From: clemc at ccc.com (Clem Cole) Date: Wed, 3 May 2017 11:44:38 -0400 Subject: [TUHS] dmr note on BSD's sins In-Reply-To: <201705031534.v43FYSCY028025@freefriends.org> References: <201705030052.v430qp2K004255@coolidge.cs.Dartmouth.EDU> <201705031534.v43FYSCY028025@freefriends.org> Message-ID: Early line at time version of paste On Wed, May 3, 2017 at 11:34 AM, wrote: > > [Maybe it shows my inner > > schizophrenia and heard voices from multiple places that I kinda like the > > 'wye' command]. > > Um, what is the 'wye' command? Never heard of it... > > Thanks, > > Arnold > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave at horsfall.org Thu May 4 08:18:56 2017 From: dave at horsfall.org (Dave Horsfall) Date: Thu, 4 May 2017 08:18:56 +1000 (EST) Subject: [TUHS] dmr note on BSD's sins In-Reply-To: References: Message-ID: On Wed, 3 May 2017, Nemo wrote: > Along these lines, who said "Cat went to Berkely, came back waving > flags." How did the expression go? LSD and BSD were both produced at Berkeley; this is thought not to be a coincidence. -- Dave Horsfall DTM (VK2KFU) "Those who don't understand security will suffer." From tfb at tfeb.org Fri May 5 00:59:22 2017 From: tfb at tfeb.org (Tim Bradshaw) Date: Thu, 4 May 2017 15:59:22 +0100 Subject: [TUHS] dmr note on BSD's sins In-Reply-To: References: Message-ID: <93E87A49-523F-4BCE-9D89-E62351410A8C@tfeb.org> On 3 May 2017, at 14:41, Nemo wrote: > > Along these lines, who said "Cat went to Berkely, came back waving flags." And anyone who lived through the SunOS 4 -- SunOS 5 transition will know that some of those Berkeley flags (not specifically for cat, but almost certainly including those for cat) were really quite useful. I remember spending really a long time finding and building BSD/GNU versions of utilities which actually had the options you needed on early SunOS 5 machines: later on Sun themselves put some of them back. From lm at mcvoy.com Fri May 5 09:14:36 2017 From: lm at mcvoy.com (Larry McVoy) Date: Thu, 4 May 2017 16:14:36 -0700 Subject: [TUHS] dmr note on BSD's sins In-Reply-To: <93E87A49-523F-4BCE-9D89-E62351410A8C@tfeb.org> References: <93E87A49-523F-4BCE-9D89-E62351410A8C@tfeb.org> Message-ID: <20170504231436.GC2932@mcvoy.com> On Thu, May 04, 2017 at 03:59:22PM +0100, Tim Bradshaw wrote: > On 3 May 2017, at 14:41, Nemo wrote: > > > > Along these lines, who said "Cat went to Berkely, came back waving flags." > > And anyone who lived through the SunOS 4 -- SunOS 5 transition will know that some of those Berkeley flags (not specifically for cat, but almost certainly including those for cat) were really quite useful. I remember spending really a long time finding and building BSD/GNU versions of utilities which actually had the options you needed on early SunOS 5 machines: later on Sun themselves put some of them back. As someone who was at Sun, loved SunOS 4 and hated Solaris, yup. Even the diehard Solaris people like Bryan Cantrill say that they stick /opt/gnu/bin in their path first. It boggles my mind that Sun was so strict on being compat with SVR4 and they knew that everyone hated that. -- --- Larry McVoy lm at mcvoy.com http://www.mcvoy.com/lm From doug at cs.dartmouth.edu Fri May 5 23:06:56 2017 From: doug at cs.dartmouth.edu (Doug McIlroy) Date: Fri, 05 May 2017 09:06:56 -0400 Subject: [TUHS] [Groff] ASCII Minus Sign in man Pages Message-ID: <201705051306.v45D6uQr006114@coolidge.cs.Dartmouth.EDU> > I believe I was the last person to modify the Linux man page macros. > Their current maintainer is not the kind of groff expert to whom it > would occur to modify them; it would work as well to ask me questions Question #1. Which tmac file to they use? If it's not in the groff package, where can it be found? Doug From random832 at fastmail.com Sat May 6 00:01:55 2017 From: random832 at fastmail.com (Random832) Date: Fri, 05 May 2017 10:01:55 -0400 Subject: [TUHS] dmr note on BSD's sins In-Reply-To: <20170504231436.GC2932@mcvoy.com> References: <93E87A49-523F-4BCE-9D89-E62351410A8C@tfeb.org> <20170504231436.GC2932@mcvoy.com> Message-ID: <1493992915.3765984.966912328.38F55A2B@webmail.messagingengine.com> On Thu, May 4, 2017, at 19:14, Larry McVoy wrote: > On Thu, May 04, 2017 at 03:59:22PM +0100, Tim Bradshaw wrote: > > On 3 May 2017, at 14:41, Nemo wrote: > > > > > > Along these lines, who said "Cat went to Berkely, came back waving flags." > > > > And anyone who lived through the SunOS 4 -- SunOS 5 transition will know that some of those Berkeley flags (not specifically for cat, but almost certainly including those for cat) were really quite useful. I remember spending really a long time finding and building BSD/GNU versions of utilities which actually had the options you needed on early SunOS 5 machines: later on Sun themselves put some of them back. I mean, there's a legitimate argument that some of them would have been just as useful or more as separate utilities than as flags. But to some extent the fully generalized version seems questionable. I mean, to apply the design philosophy to, say, ls -t (nevermind that this particular flag has in fact existed since Unix V1, it's exactly the *kind* of thing that should prompt someone who is consistently applying Pike's idea of the unix philosophy to say "sorting should be the sort tool"), you would have to have A) a basic "ls" command which does all the filesystem-accessing work and prints timestamps and all other relevant output in a format that can be sorted lexicographically, B) run it through sort(1), C) a command [or maybe an awk script] that will pare it back down to the usual human-readable formats (names only, ls -l, or maybe a timestamp-and-name format, with human-friendly timestamps). And then D) another tool to sort it into columns. And you'd have to type in all of those every time you want a directory listing, or maybe have it packaged up in a script (which would then need a name, and unique names for every combination of behaviors you might want to use). At that point, who gets to decide that ls -t is useful enough to be "proper unix" but cat -v is "some ridiculous idea from berkeley"? Anyway, on the original subject of unlimited symbol name and filename lengths - ultimately there's a tradeoff - your symbols or filenames probably don't have to get very large before it makes more sense to use a variable-length format to save space, and the smallest reasonable limit for a variable-length format is 255. From clemc at ccc.com Sat May 6 01:20:02 2017 From: clemc at ccc.com (Clem Cole) Date: Fri, 5 May 2017 11:20:02 -0400 Subject: [TUHS] Discuss of style and design of computer programs from a user stand point [was dmr note on BSD's sins] Message-ID: On Thu, May 4, 2017 at 7:14 PM, Larry McVoy wrote: > some of those Berkeley flags (not specifically for cat, but almost > certainly including those for cat) were really quite useful. ​Amen!!! I think that this is key point. What is in good taste or good style? Doug's distain for the results of: less --help | wc is found in bad design, a poor user interface, little fore thought, *etc.* Most of what is there has been added over Eric Scheinbrood's original more(1) I do not believe are used that often, but some of them are of course, and you can not tell who uses what!! So how do you decide what get rid of? How do you learn what people really need -- IMO: much of that is * experience *and this thing we call 'good taste.' As opposed to what I think happened with less(1) and many others similar programs (programmers peeing on the code because they code and the source was available -- I can add this feature to it and I think it is cool. As opposed to asking, what do really get and not having a 'master builder' arbitrating or vetting things). The problem we have is that we don't yet have a way of defining good taste. One might suggest that it takes years of civilization and also that tastes do change over time. Pike's minimalistic view (which I think is taken to the extreme in the joke about automobile dashboard on Brian Kernighan's car) sets the bar to one end and probably one of the reason why UNIX had a bad reputation, certainly by non-computer scientists, when it first appeared as being difficult. Another extreme is systems that put in you a box and never let you do anything but what we told you do; which I find just as frighten and frustration builds there when I use them. Clearly systems that are so noisy that can not find what really want, or need is another dimension of the same bad design. So what to do? [more in a minute...] Larry is right. Many of the 'features' added to UNIX (and Linux) over time have been and *are useful*. Small and simple as it was (and I really admire Ken, Dennis and the Team for its creation), but in 2017 I really don't want to run the Sixth Edition for my day to day work anymore - which I happily did in 1977. But the problem is, as we got 'unlimited address space' of 32 and 64 bits, and more room for more 'useful' things, we also got a great deal of rubbish and waste. I am interpreting Doug's point about less --help | wc is that is that are so many thorns and weeds, it hard to see to see the flowers in the garden. I'd like to observe that is that most college courses in CS I have seen talk about how to construct a programs, algorithms, structures - i.e. the mechanics of some operation. But this discussion is about the human element. What we feel is good or bad and how it related to how to use the program. I think about my friends that have degrees in literature, art and architecture. In all cases, they spend a lot of time examining past examples, of good and bad - and thinking and discussing what makes them so. I'm actually happy to see it was Professor McIlroy that is one of the folks taking a stand on current craziness. I think this is only going to get better if a new crop of students that have been trained in 'good taste.' So, I wonder do any of the schools like Darthmouth and the like teach courses that study 'style' and taste in CS. Yes, it is a young field but we have been around long enough that we do a body of work good and bad to consider. I think there is a book or two and a few lectures in there somewhere. Thoughts? Clem -------------- next part -------------- An HTML attachment was scrubbed... URL: From bakul at bitblocks.com Sat May 6 01:37:03 2017 From: bakul at bitblocks.com (Bakul Shah) Date: Fri, 5 May 2017 08:37:03 -0700 Subject: [TUHS] Discuss of style and design of computer programs from a user stand point [was dmr note on BSD's sins] In-Reply-To: References: Message-ID: <1C6D6D3E-058F-4743-8DA0-17CEC09B64E9@bitblocks.com> I think the key issue is not as much minimalism as composability. BSD often prioritized convenience over composability. Each command doing one thing well and doing line oriented makes them more composability. You can always package up convenient combinations in a script. Plan9 has lc which prints like unix ls -C but it is an rc script. Trying to achieve composability can result in leaner systems. > On May 5, 2017, at 8:20 AM, Clem Cole wrote: > > >> On Thu, May 4, 2017 at 7:14 PM, Larry McVoy wrote: >> some of those Berkeley flags (not specifically for cat, but almost certainly including those for cat) were really quite useful. > > ​Amen!!! I think that this is key point. What is in good taste or good style? Doug's distain for the results of: > less --help | wc > is found in bad design, a poor user interface, little fore thought, etc. Most of what is there has been added over Eric Scheinbrood's original more(1) I do not believe are used that often, but some of them are of course, and you can not tell who uses what!! So how do you decide what get rid of? > > How do you learn what people really need -- IMO: much of that is experience and this thing we call 'good taste.' As opposed to what I think happened with less(1) and many others similar programs (programmers peeing on the code because they code and the source was available -- I can add this feature to it and I think it is cool. As opposed to asking, what do really get and not having a 'master builder' arbitrating or vetting things). > > The problem we have is that we don't yet have a way of defining good taste. One might suggest that it takes years of civilization and also that tastes do change over time. Pike's minimalistic view (which I think is taken to the extreme in the joke about automobile dashboard on Brian Kernighan's car) sets the bar to one end and probably one of the reason why UNIX had a bad reputation, certainly by non-computer scientists, when it first appeared as being difficult. Another extreme is systems that put in you a box and never let you do anything but what we told you do; which I find just as frighten and frustration builds there when I use them. Clearly systems that are so noisy that can not find what really want, or need is another dimension of the same bad design. So what to do? [more in a minute...] > > Larry is right. Many of the 'features' added to UNIX (and Linux) over time have been and are useful. Small and simple as it was (and I really admire Ken, Dennis and the Team for its creation), but in 2017 I really don't want to run the Sixth Edition for my day to day work anymore - which I happily did in 1977. But the problem is, as we got 'unlimited address space' of 32 and 64 bits, and more room for more 'useful' things, we also got a great deal of rubbish and waste. I am interpreting Doug's point about less --help | wc is that is that are so many thorns and weeds, it hard to see to see the flowers in the garden. > > I'd like to observe that is that most college courses in CS I have seen talk about how to construct a programs, algorithms, structures - i.e. the mechanics of some operation. But this discussion is about the human element. What we feel is good or bad and how it related to how to use the program. > > I think about my friends that have degrees in literature, art and architecture. In all cases, they spend a lot of time examining past examples, of good and bad - and thinking and discussing what makes them so. I'm actually happy to see it was Professor McIlroy that is one of the folks taking a stand on current craziness. I think this is only going to get better if a new crop of students that have been trained in 'good taste.' So, I wonder do any of the schools like Darthmouth and the like teach courses that study 'style' and taste in CS. Yes, it is a young field but we have been around long enough that we do a body of work good and bad to consider. > > I think there is a book or two and a few lectures in there somewhere. Thoughts? > > Clem > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tfb at tfeb.org Sat May 6 03:33:19 2017 From: tfb at tfeb.org (Tim Bradshaw) Date: Fri, 5 May 2017 18:33:19 +0100 Subject: [TUHS] dmr note on BSD's sins In-Reply-To: <20170504231436.GC2932@mcvoy.com> References: <93E87A49-523F-4BCE-9D89-E62351410A8C@tfeb.org> <20170504231436.GC2932@mcvoy.com> Message-ID: <6A6951FC-D4C8-48AC-A926-BBE27E9334F7@tfeb.org> I asked my wife (who is a longer-term Sun person than me and also went through this) if my memory was correct (I now keep elaborate and deeply unpublishable notes but I didn't then) and she immediately said 'echo -n': this was presumably either in /(usr/)?bin/echo in BSD/SunOS 4 or in the /bin/sh builtin, and not in the versions for SunOS 5 with resulting horrible failures of scripts all over the place. I think I can sort of see why the serious-standards-compliance-even-when-the-standard-is-horrible thing might have seemed like it would appeal to the kind of organisations Sun wanted to sell to (ones where people wear suits), but it also drove their traditional userbase to Linux/*BSD and this did not help them in the long term. (Also: do the standards say 'thou shall have no other options but these', because if they do they are terrible standards). > On 5 May 2017, at 00:14, Larry McVoy wrote: > >> On Thu, May 04, 2017 at 03:59:22PM +0100, Tim Bradshaw wrote: >>> On 3 May 2017, at 14:41, Nemo wrote: >>> >>> Along these lines, who said "Cat went to Berkely, came back waving flags." >> >> And anyone who lived through the SunOS 4 -- SunOS 5 transition will know that some of those Berkeley flags (not specifically for cat, but almost certainly including those for cat) were really quite useful. I remember spending really a long time finding and building BSD/GNU versions of utilities which actually had the options you needed on early SunOS 5 machines: later on Sun themselves put some of them back. > > As someone who was at Sun, loved SunOS 4 and hated Solaris, yup. Even the > diehard Solaris people like Bryan Cantrill say that they stick /opt/gnu/bin > in their path first. > > It boggles my mind that Sun was so strict on being compat with SVR4 and > they knew that everyone hated that. > -- > --- > Larry McVoy lm at mcvoy.com http://www.mcvoy.com/lm From a.phillip.garcia at gmail.com Sat May 6 04:56:06 2017 From: a.phillip.garcia at gmail.com (A. P. Garcia) Date: Fri, 5 May 2017 13:56:06 -0500 Subject: [TUHS] dmr note on BSD's sins In-Reply-To: <1493992915.3765984.966912328.38F55A2B@webmail.messagingengine.com> References: <93E87A49-523F-4BCE-9D89-E62351410A8C@tfeb.org> <20170504231436.GC2932@mcvoy.com> <1493992915.3765984.966912328.38F55A2B@webmail.messagingengine.com> Message-ID: On Fri, May 5, 2017 at 9:01 AM, Random832 wrote: > On Thu, May 4, 2017, at 19:14, Larry McVoy wrote: >> On Thu, May 04, 2017 at 03:59:22PM +0100, Tim Bradshaw wrote: >> > On 3 May 2017, at 14:41, Nemo wrote: >> > > >> > > Along these lines, who said "Cat went to Berkely, came back waving flags." >> > >> > And anyone who lived through the SunOS 4 -- SunOS 5 transition will know that some of those Berkeley flags (not specifically for cat, but almost certainly including those for cat) were really quite useful. I remember spending really a long time finding and building BSD/GNU versions of utilities which actually had the options you needed on early SunOS 5 machines: later on Sun themselves put some of them back. > > I mean, there's a legitimate argument that some of them would have been > just as useful or more as separate utilities than as flags. But to some > extent the fully generalized version seems questionable. > > I mean, to apply the design philosophy to, say, ls -t (nevermind that > this particular flag has in fact existed since Unix V1, it's exactly the > *kind* of thing that should prompt someone who is consistently applying > Pike's idea of the unix philosophy to say "sorting should be the sort > tool"), you would have to have A) a basic "ls" command which does all > the filesystem-accessing work and prints timestamps and all other > relevant output in a format that can be sorted lexicographically, B) run > it through sort(1), C) a command [or maybe an awk script] that will pare > it back down to the usual human-readable formats (names only, ls -l, or > maybe a timestamp-and-name format, with human-friendly timestamps). And > then D) another tool to sort it into columns. have you been using powershell lately? it makes things like this rather enjoyable. yes, powershell is open source, and yes, it's been ported to linux, but i think unix needs its own home-grown "object oriented shell". From doug at cs.dartmouth.edu Sat May 6 12:02:21 2017 From: doug at cs.dartmouth.edu (Doug McIlroy) Date: Fri, 05 May 2017 22:02:21 -0400 Subject: [TUHS] Discuss of style and design of computer programs from a user stand point [was dmr note on BSD's sins] In-Reply-To: References: Message-ID: <201705060202.v4622L1J013430@coolidge.cs.Dartmouth.EDU> Clem wrote, "So, I wonder do any of the schools like Darthmouth and the like teach courses that study 'style' and taste in CS." Some time around 1970, Bob Rosen instituted a program-reading course at the University of Iowa, by overt analogy to the study of literature. Style was certainly a central concern; I'm not sure about the deeper matter of taste. I'm not aware of other examples, except perhaps for a collaboration between Don Kunth and a teachaer of (I think) creative writing--a course whose content I can't recall. Doug From noel.hunt at gmail.com Sat May 6 12:16:53 2017 From: noel.hunt at gmail.com (Noel Hunt) Date: Sat, 6 May 2017 12:16:53 +1000 Subject: [TUHS] Discuss of style and design of computer programs from a user stand point [was dmr note on BSD's sins] In-Reply-To: <1C6D6D3E-058F-4743-8DA0-17CEC09B64E9@bitblocks.com> References: <1C6D6D3E-058F-4743-8DA0-17CEC09B64E9@bitblocks.com> Message-ID: What you call 'composability' is nothing more than the 'software tools' approach. I recall reading an interview with Dennis Ritchie, but can no longer find it, where he talks about this approach versus the monolithic program model, as for example demonstrated by 'perl', which does everything. Dennis remarked that the 'tools' approach required careful thought (which programs to use, how to connect them with pipes) and so was more difficult to use, hence the popularity of 'perl'. I'd be delighted if someone can point me to where this interview might be. P.S. A most beautiful example of this approach is from Doug McIlroy in his critique of Donald Knuth's solution to a problem posed by Jon Bentley in his 'Programming Pearls' column: Read a file of text, determine the n most frequently used words, and print out a sorted list of those words along with their frequencies. Donald Knuth wrote a long, beautifully crafted program in some pseudo-code. Doug McIlroy provided an alternative solution: tr -cs A-Za-z '\n' | tr A-Z a-z | sort | uniq -c | sort -rn | sed ${1}q This is real genius. On Sat, May 6, 2017 at 1:37 AM, Bakul Shah wrote: > I think the key issue is not as much minimalism as composability. BSD > often prioritized convenience over composability. Each command doing one > thing well and doing line oriented makes them more composability. You can > always package up convenient combinations in a script. Plan9 has lc which > prints like unix ls -C but it is an rc script. Trying to achieve > composability can result in leaner systems. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From toby at telegraphics.com.au Sat May 6 12:40:12 2017 From: toby at telegraphics.com.au (Toby Thain) Date: Fri, 5 May 2017 22:40:12 -0400 Subject: [TUHS] Discuss of style and design of computer programs from a user stand point [was dmr note on BSD's sins] In-Reply-To: References: <1C6D6D3E-058F-4743-8DA0-17CEC09B64E9@bitblocks.com> Message-ID: <8b552139-8e5e-a32f-fd47-6a863642ecab@telegraphics.com.au> On 2017-05-05 10:16 PM, Noel Hunt wrote: > ... > Donald Knuth wrote a long, beautifully crafted program in some > pseudo-code. Doug McIlroy provided an alternative solution: > Much as I love Professor Knuth, he is still doing this. He always was a low level coder*, and his programs are longer and more complicated than the high level equivalents, by the expected factors. (If anyone doubts it, note that he adopted C after Pascal, and many of his programs are linked from his web pages.) No doubt he has solid rationalisations for taking this approach for his purposes, but of course it's not ideal for all situations. --Toby * - Many of us will remember we were expected to understand TAOCP through MIX. And he started out like this: http://www.catonmat.net/blog/donald-knuths-first-computer/ From scj at yaccman.com Sat May 6 15:33:02 2017 From: scj at yaccman.com (Steve Johnson) Date: Fri, 05 May 2017 22:33:02 -0700 Subject: [TUHS] Discuss of style and design of computer programs from a user stand point [was dmr note on BSD's sins] In-Reply-To: <201705060202.v4622L1J013430@coolidge.cs.Dartmouth.EDU> Message-ID: <5a2d6cc957c2efcd968f35aa5557c7a0e309dd27@webmail.yaccman.com> In 1973, I spent 9 months at the University of Waterloo, home of Watfor.   I taught a course in "Advanced Application Techniques" -- the language was FORTRAN, and the techniques included such things as using libraries and handling errors in input data... When I left for Canada, there had been a lot of discussion about programming style at the labs, and so in my first lecture I told the students that I would award points for style, and briefly talked about what that was and how I would assess the style, and I did so throughout the course. What amazed me was that some students seemed to instantly get the idea, and became quite good at producing excellent, clean programs, while others were violently allergic to the whole notion that anything "subjective" should stand between them and their grade in the class.  I remember several interchanges with students, often angry, who had a lot of difficulty understanding why their 35-line solution to a problem should get a lower grade than a 15-line version (excluding comment lines). in fairness, I didn't (and don't) know any better way to learn style than doing a lot of reading, a lot of writing, and getting criticized constructively by those with more experience.  For me, a lot of what I learned was from Stan Brown at the Labs, who read piles of my (atrocious) FORTRAN code and repeatedly pointed out that when you wrote a program, you were not just communicating with the computer, but also other humans (including yourself) who would read (and perhaps modify) the code in the future... Steve ----- Original Message ----- From: "Doug McIlroy" To:, , Cc: Sent:Fri, 05 May 2017 22:02:21 -0400 Subject:Re: [TUHS] Discuss of style and design of computer programs from a user stand point [was dmr note on BSD's sins] Clem wrote, "So, I wonder do any of the schools like Darthmouth and the like teach courses that study 'style' and taste in CS." Some time around 1970, Bob Rosen instituted a program-reading course at the University of Iowa, by overt analogy to the study of literature. Style was certainly a central concern; I'm not sure about the deeper matter of taste. I'm not aware of other examples, except perhaps for a collaboration between Don Kunth and a teachaer of (I think) creative writing--a course whose content I can't recall. Doug -------------- next part -------------- An HTML attachment was scrubbed... URL: From bakul at bitblocks.com Sat May 6 16:07:27 2017 From: bakul at bitblocks.com (Bakul Shah) Date: Fri, 05 May 2017 23:07:27 -0700 Subject: [TUHS] Discuss of style and design of computer programs from a user stand point [was dmr note on BSD's sins] In-Reply-To: Your message of "Sat, 06 May 2017 12:16:53 +1000." References: <1C6D6D3E-058F-4743-8DA0-17CEC09B64E9@bitblocks.com> Message-ID: <20170506060727.A00C0124AEFE@mail.bitblocks.com> On Sat, 06 May 2017 12:16:53 +1000 Noel Hunt wrote: > > What you call 'composability' is nothing more than the 'software > tools' approach. Composable components can have a much tigher coupling. vi, rsync, git and gdb are s/w tools but they are not composable in the sense of sed, grep, uniq, awk, head, tail, tr, sort, ls, find etc. > I recall reading an interview with Dennis Ritchie, but can no > longer find it, where he talks about this approach versus the > monolithic program model, as for example demonstrated by > 'perl', which does everything. Dennis remarked that the > 'tools' approach required careful thought (which programs to > use, how to connect them with pipes) and so was more difficult > to use, hence the popularity of 'perl'. This is the "modularity" approach, where one program does one thing well rather than doing everything (e.g IDE). But moudlarity does not necessarily give you composability. Also, it is not enough to have composable components; you need a language/environment in which they can be used. > A most beautiful example of this approach is from Doug McIlroy > in his critique of Donald Knuth's solution to a problem posed > by Jon Bentley in his 'Programming Pearls' column: > > Read a file of text, determine the n most frequently used > words, and print out a sorted list of those words along with > their frequencies. > > Donald Knuth wrote a long, beautifully crafted program in some > pseudo-code. Doug McIlroy provided an alternative solution: > > tr -cs A-Za-z '\n' | > tr A-Z a-z | > sort | > uniq -c | > sort -rn | > sed ${1}q > > This is real genius. Indeed. Note that a similar approach was implemented by Iverson in APL. In fact a shell pipeline has a lot in common with array programming! Iverson in essence built a whole workshop with a set of simple but powerful standard tools (functions and higher order functions). The same has been carried on in follow on array languages such as j and k and q languages. Many of these array functions are directly applicable to stream processing. In contrast unix shells provide a very shallow glue layer. > On Sat, May 6, 2017 at 1:37 AM, Bakul Shah wrote: > > > I think the key issue is not as much minimalism as composability. BSD > > often prioritized convenience over composability. Each command doing one > > thing well and doing line oriented makes them more composability. You can > > always package up convenient combinations in a script. Plan9 has lc which > > prints like unix ls -C but it is an rc script. Trying to achieve > > composability can result in leaner systems. From michael at kjorling.se Sat May 6 19:18:57 2017 From: michael at kjorling.se (Michael =?utf-8?B?S2rDtnJsaW5n?=) Date: Sat, 6 May 2017 09:18:57 +0000 Subject: [TUHS] Discuss of style and design of computer programs from a user stand point In-Reply-To: <5a2d6cc957c2efcd968f35aa5557c7a0e309dd27@webmail.yaccman.com> References: <201705060202.v4622L1J013430@coolidge.cs.Dartmouth.EDU> <5a2d6cc957c2efcd968f35aa5557c7a0e309dd27@webmail.yaccman.com> Message-ID: <20170506091857.GE12539@yeono.kjorling.se> On 5 May 2017 22:33 -0700, from scj at yaccman.com (Steve Johnson): > For me, a lot of what > I learned was from Stan Brown at the Labs, who read piles of my > (atrocious) FORTRAN code and repeatedly pointed out that when you > wrote a program, you were not just communicating with the computer, > but also other humans (including yourself) who would read (and perhaps > modify) the code in the future... I would actually take that one step further: When you are writing code, you are _first and foremost_ communicating with whatever human will need to read or modify the code later. That human might be you, a colleague, or the violent psychopath who knows both where you live and where your little kids go to school (might as well be you). You should strive to write the code accordingly, _even if_ the odds of the threat ever materializing are slim at most. Style matters a lot, there. Sure, some code really needs to be heavily optimized, and might end up obscure for it. In operating systems, interrupt routines are probably one of the major candidates, but first stage bootloaders need their fair share of it too for other reasons. Tight loops or code that needs to be able to operate on lots of data quickly are other examples. But _most code isn't like that_ and can allow for some extra verbosity and some performance penalty if that means making it significantly easier to read and modify. That's not to say code should be verbose _just because_, but if you don't need ultimate performance, I say aim for readability and modifyability first; processing and memory usage second. You can always go back and optimize bits that turn out to need it. And of course, I find Doug McIlroy's shell snippet mentioned by Noel interesting, _because I did something very similar not all that long ago_; in my case, stringing together a bunch of shell tools to get a list of all URLs from an e-mail message for further processing. (Something like a composable _urlview_; if urlview had even had a "dump to stdout" option, that would have done all I needed it for.) Works like a charm, and isn't even terribly complicated. It could probably be done in a single line of perl or awk instead of piping through a grand total of six commands, but that's okay. And of course, stringing together a bunch of commands was the first approach that occured to me... -- Michael Kjörling • https://michael.kjorling.se • michael at kjorling.se “People who think they know everything really annoy those of us who know we don’t.” (Bjarne Stroustrup) From cym224 at gmail.com Sat May 6 23:09:27 2017 From: cym224 at gmail.com (Nemo) Date: Sat, 6 May 2017 09:09:27 -0400 Subject: [TUHS] Discuss of style and design of computer programs from a user stand point In-Reply-To: <20170506091857.GE12539@yeono.kjorling.se> References: <201705060202.v4622L1J013430@coolidge.cs.Dartmouth.EDU> <5a2d6cc957c2efcd968f35aa5557c7a0e309dd27@webmail.yaccman.com> <20170506091857.GE12539@yeono.kjorling.se> Message-ID: On 06/05/2017, Michael Kjörling wrote: > I would actually take that one step further: When you are writing > code, you are _first and foremost_ communicating with whatever human > will need to read or modify the code later. Something similar appears in the front of almost every Wirth book (though sometimes honoured in the breach). That being said, there are times when the code is transformed by optimization into something very difficult to read. (We used noweb to document the transformation.) N. From doug at cs.dartmouth.edu Sat May 6 23:30:58 2017 From: doug at cs.dartmouth.edu (Doug McIlroy) Date: Sat, 06 May 2017 09:30:58 -0400 Subject: [TUHS] Discuss of style and design of computer programs from a user stand point [was dmr note on BSD's sins] Message-ID: <201705061330.v46DUwlH020977@coolidge.cs.Dartmouth.EDU> > tr -cs A-Za-z '\n' | > tr A-Z a-z | > sort | > uniq -c | > sort -rn | > sed ${1}q > > This is real genius. Not genius. Experience. In the Bentley/Knuth/McIlroy paper I said, "[Old] Unix hands know instinctively how to solve this one in a jiffy." While that is certainly true, the script was informed by my having written "spell", which itself was an elaboration of a model pioneered by Steve Johnson. By 1986, when BKM was published, the lore was baked in: word-processing scripts in a similar vein were stock in trade. A very early exercise of this sort was Dennis Ritchie's enumeration of anagrams in the unabridged Merriam-Webster. Since the word list barely fit on the tiny disk of the time, the job entailed unimaginable marshalling of resources. I was mightily impressed then, and still am. Doug From michael at kjorling.se Sat May 6 23:44:55 2017 From: michael at kjorling.se (Michael =?utf-8?B?S2rDtnJsaW5n?=) Date: Sat, 6 May 2017 13:44:55 +0000 Subject: [TUHS] Discuss of style and design of computer programs from a user stand point In-Reply-To: References: <201705060202.v4622L1J013430@coolidge.cs.Dartmouth.EDU> <5a2d6cc957c2efcd968f35aa5557c7a0e309dd27@webmail.yaccman.com> <20170506091857.GE12539@yeono.kjorling.se> Message-ID: <20170506134455.GG12539@yeono.kjorling.se> On 6 May 2017 09:09 -0400, from cym224 at gmail.com (Nemo): >> I would actually take that one step further: When you are writing >> code, you are _first and foremost_ communicating with whatever human >> will need to read or modify the code later. > > ..., there are > times when the code is transformed by optimization into something very > difficult to read. (We used noweb to document the transformation.) True, optimized code can be difficult to read. But very little code really _needs_ to be optimized to that level, and when you come across it, you will generally know -- at least after running the application through a profiler. (I have to admit we do have it easy these days.) I've been working on optimizing some SQL code recently -- I know, that's not exactly operating system kernel C or assembler -- and it's amazing at times how the code can end up being both easier to understand _as well as_ faster to execute, if you just put some thought into what the code really needs to do. And not just minor speedups either, but wallclock runtime reductions on the order of _two to four nines_. I think there's been only one time in the last several years when I have really _needed_ to optimize code to the point that its workings were not obvious. -- Michael Kjörling • https://michael.kjorling.se • michael at kjorling.se “People who think they know everything really annoy those of us who know we don’t.” (Bjarne Stroustrup) From toby at telegraphics.com.au Sun May 7 00:16:03 2017 From: toby at telegraphics.com.au (Toby Thain) Date: Sat, 6 May 2017 10:16:03 -0400 Subject: [TUHS] The Elements of Programming Style (book) - was Re: Discuss of style and design of computer programs In-Reply-To: <5a2d6cc957c2efcd968f35aa5557c7a0e309dd27@webmail.yaccman.com> References: <5a2d6cc957c2efcd968f35aa5557c7a0e309dd27@webmail.yaccman.com> Message-ID: On 2017-05-06 1:33 AM, Steve Johnson wrote: > In 1973, I spent 9 months at the University of Waterloo, home of Watfor. > I taught a course in "Advanced Application Techniques" -- the language > was FORTRAN, and the techniques included such things as using libraries > and handling errors in input data... > ... > in fairness, I didn't (and don't) know any better way to learn style > than doing a lot of reading, a lot of writing, and getting criticized > constructively by those with more experience. For me, a lot of what I > learned was from Stan Brown at the Labs, who read piles of my > (atrocious) FORTRAN code and repeatedly pointed out that when you wrote > a program, you were not just communicating with the computer, but also > other humans (including yourself) who would read (and perhaps modify) > the code in the future... As a longtime maintenance programmer: I could not agree more. Thankfully this idea is fairly well accepted today. Although, I am working on a project right now that makes me think the author simply never considered that anyone else might ever read or maintain it. In this context, one of the timeless classics is The Elements of Programming Style (Kernighan and Plauger). It had a huge effect on my C when I read it years ago and I still try to apply its lessons every day in any language. --Toby > > Steve > > > > ----- Original Message ----- > From: > "Doug McIlroy" > > To: > , , > Cc: > > Sent: > Fri, 05 May 2017 22:02:21 -0400 > Subject: > Re: [TUHS] Discuss of style and design of computer programs from a > user stand point [was dmr note on BSD's sins] > > > Clem wrote, "So, I wonder do any of the schools like Darthmouth and > the like > teach courses that study 'style' and taste in CS." > > Some time around 1970, Bob Rosen instituted a program-reading > course at the University of Iowa, by overt analogy to the > study of literature. Style was certainly a central concern; > I'm not sure about the deeper matter of taste. I'm not aware > of other examples, except perhaps for a collaboration between > Don Kunth and a teachaer of (I think) creative writing--a > course whose content I can't recall. > > Doug > From lm at mcvoy.com Sun May 7 00:40:11 2017 From: lm at mcvoy.com (Larry McVoy) Date: Sat, 6 May 2017 07:40:11 -0700 Subject: [TUHS] Discuss of style and design of computer programs from a user stand point In-Reply-To: <20170506091857.GE12539@yeono.kjorling.se> References: <201705060202.v4622L1J013430@coolidge.cs.Dartmouth.EDU> <5a2d6cc957c2efcd968f35aa5557c7a0e309dd27@webmail.yaccman.com> <20170506091857.GE12539@yeono.kjorling.se> Message-ID: <20170506144011.GF28787@mcvoy.com> On Sat, May 06, 2017 at 09:18:57AM +0000, Michael Kj??rling wrote: > On 5 May 2017 22:33 -0700, from scj at yaccman.com (Steve Johnson): > > For me, a lot of what > > I learned was from Stan Brown at the Labs, who read piles of my > > (atrocious) FORTRAN code and repeatedly pointed out that when you > > wrote a program, you were not just communicating with the computer, > > but also other humans (including yourself) who would read (and perhaps > > modify) the code in the future... > > I would actually take that one step further: When you are writing > code, you are _first and foremost_ communicating with whatever human > will need to read or modify the code later. That human might be you, a > colleague, or the violent psychopath who knows both where you live and > where your little kids go to school (might as well be you). When I run engineering teams the point I make is if you wrote the code more than 6 months ago it might as well be someone else's code. So write it in a way that someone else can debug / extend it. I also made the point that "clever" sucks. At least most of the time. All "clever" means is "hard to understand". The brief joy one gets from clever code is stomped on by the frustration one gets from having to fix it. When they wouldn't listen to that I'd say "I'm the dumbest guy here, right? And I'm the boss. And I debug could. The path to a big bonus does not go through clever code". That seemed to work. Personally, I find code that is clean, straightforward, obvious to be beautiful. The clever stuff usually strikes an odd note, not a good one. --lm From corey at lod.com Sun May 7 01:09:13 2017 From: corey at lod.com (Corey Lindsly) Date: Sat, 6 May 2017 08:09:13 -0700 (PDT) Subject: [TUHS] Discuss of style and design of computer programs from a In-Reply-To: <20170506144011.GF28787@mcvoy.com> Message-ID: <20170506150913.57571411A@lod.com> > Personally, I find code that is clean, straightforward, obvious to be > beautiful. The clever stuff usually strikes an odd note, not a good > one. > > --lm I am not a programmer. Almost four decades ago, my first computer was a TRS80 Model 1 with 16KB RAM. I spent one month disassembling and stepping through the Z80 code for the resident Microsoft BASIC interpreter. The entire thing fit in a 12KB PROM so it was originally written in assembly and tightly optimized. It was fascinating and extremely instructive. All these years later, I could probably still slap together a Z80 program if I needed to. Anyway, I reached one point in the assembly code that I simply could not understand. It seemed like a mistake, and I went through it again and again until I finally realized what it was doing. There was a branch/loop that jumped to the middle of a multi-byte machine instruction, so that branch had to be disassembled and stepped separately until it "synced" up with the other branch again. Maybe this is standard practice in programming (I don't know) but at the time I thought, what kind of evil genius devised this to save a few bytes of memory? --corey From michael at kjorling.se Sun May 7 01:20:53 2017 From: michael at kjorling.se (Michael =?utf-8?B?S2rDtnJsaW5n?=) Date: Sat, 6 May 2017 15:20:53 +0000 Subject: [TUHS] Discuss of style and design of computer programs from a In-Reply-To: <20170506150913.57571411A@lod.com> References: <20170506144011.GF28787@mcvoy.com> <20170506150913.57571411A@lod.com> Message-ID: <20170506152053.GI12539@yeono.kjorling.se> On 6 May 2017 08:09 -0700, from corey at lod.com (Corey Lindsly): > Anyway, I reached one point in the assembly code that I simply could not > understand. It seemed like a mistake, and I went through it again and > again until I finally realized what it was doing. There was a branch/loop > that jumped to the middle of a multi-byte machine instruction, so that > branch had to be disassembled and stepped separately until it "synced" up > with the other branch again. Maybe this is standard practice in > programming (I don't know) but at the time I thought, what kind of evil > genius devised this to save a few bytes of memory? IIRC, that _was_ a common trick at least on machines of that class. It did have the potential to save a few bytes, yes (more if the instructions were such that you'd get some _other, desired_, behavior by jumping into the middle of one with some specific state), but it also foiled lots of disassemblers: Simply disassembling a binary from start to finish would yield nonsense in those locations, as you experienced. It thus basically forced you to single-step those instructions to figure out what was going on from the binary. I'm pretty sure it works on every architecture with variable-length instructions and arbitrary jump capability, as long as you have control over the specific machine instructions generated (such as if you are programming in assembler). Of course, it _is_ also a total nightmare to maintain such code. I would absolutely not say that doing something like that is standard practice in modern programming. Even in microcontrollers, where program and data memory can be scarce even today, I would argue that the costs would not outweigh the benefits by a long shot. -- Michael Kjörling • https://michael.kjorling.se • michael at kjorling.se “People who think they know everything really annoy those of us who know we don’t.” (Bjarne Stroustrup) From rminnich at gmail.com Sun May 7 01:23:23 2017 From: rminnich at gmail.com (ron minnich) Date: Sat, 06 May 2017 15:23:23 +0000 Subject: [TUHS] Discuss of style and design of computer programs from a In-Reply-To: <20170506150913.57571411A@lod.com> References: <20170506144011.GF28787@mcvoy.com> <20170506150913.57571411A@lod.com> Message-ID: On Sat, May 6, 2017 at 8:09 AM Corey Lindsly wrote: > There was a branch/loop > that jumped to the middle of a multi-byte machine instruction, so that > branch had to be disassembled and stepped separately until it "synced" up > with the other branch again. Maybe this is standard practice in > programming (I don't know) but at the time I thought, what kind of evil > genius devised this to save a few bytes of memory? > > This was extremely common back then. I had a friend who worked on a gas chromatograph project, names redacted here. It had a very advanced idea, a thermal printer. It would print a banner when it started. Getting that print to work, in the ROM they had, was a nightmare that involved all these tricks. When the "A" version came out, they asked my friend to have it print the A after the product number. His response: "NO". There was no way he could ever pick apart the crazy code that had printed out the startup banner so he could add an "A". The startup banner remained the same. Executing code as data in the early startup was also common in those days, and modifying that data and then rerunning it happened all the time. This is why the things like the Therac 25 happened ... https://en.wikipedia.org/wiki/Therac-25 Note the reference to "Cargo coding", reusing code you don't understand. In modern terms we call this software reuse and it's taught at all the best uni's. Google a package, pull it down, compile it in, done. Lest you think things are better now, Linux uses self modifying code to optimize certain critical operations, and at one talk I heard the speaker say that he'd like to put more self modifying code into Linux, "because it's fun". Oh boy. ron -------------- next part -------------- An HTML attachment was scrubbed... URL: From lm at mcvoy.com Sun May 7 01:24:11 2017 From: lm at mcvoy.com (Larry McVoy) Date: Sat, 6 May 2017 08:24:11 -0700 Subject: [TUHS] Discuss of style and design of computer programs from a In-Reply-To: <20170506152053.GI12539@yeono.kjorling.se> References: <20170506144011.GF28787@mcvoy.com> <20170506150913.57571411A@lod.com> <20170506152053.GI12539@yeono.kjorling.se> Message-ID: <20170506152411.GJ28787@mcvoy.com> On Sat, May 06, 2017 at 03:20:53PM +0000, Michael Kj??rling wrote: > I would absolutely not say that doing something like that is standard > practice in modern programming. Even in microcontrollers, where > program and data memory can be scarce even today, I would argue that > the costs would not outweigh the benefits by a long shot. It strikes as being similar to Duff's device (1). Which is a niche thing but I still use that from time to time. Not to save memory, just because as a C programmer it seems pretty natural to do it. --lm (1) https://en.wikipedia.org/wiki/Duff's_device From michael at kjorling.se Sun May 7 01:44:48 2017 From: michael at kjorling.se (Michael =?utf-8?B?S2rDtnJsaW5n?=) Date: Sat, 6 May 2017 15:44:48 +0000 Subject: [TUHS] Discuss of style and design of computer programs from a In-Reply-To: References: <20170506144011.GF28787@mcvoy.com> <20170506150913.57571411A@lod.com> Message-ID: <20170506154447.GJ12539@yeono.kjorling.se> On 6 May 2017 15:23 +0000, from rminnich at gmail.com (ron minnich): > This is why the things like the Therac 25 happened ... > https://en.wikipedia.org/wiki/Therac-25 Or the Ariane 5 flight 501, which according to Wikipedia cost more than 370 million dollars (and resulted in the total loss of the spacecraft). I would say a few million or even a few tens of millions of dollars to double-check the software, or even write new software specifically designed for the Ariane 5 rather than reuse software designed for the Ariane 4 outside of that software's design limits, might not have been a bad way to spend money there. And that was in 1996. Hardly that long ago. https://en.wikipedia.org/wiki/Ariane_5#Notable_launches https://en.wikipedia.org/wiki/Cluster_%28spacecraft%29 -- Michael Kjörling • https://michael.kjorling.se • michael at kjorling.se “People who think they know everything really annoy those of us who know we don’t.” (Bjarne Stroustrup) From michael at kjorling.se Sun May 7 01:51:24 2017 From: michael at kjorling.se (Michael =?utf-8?B?S2rDtnJsaW5n?=) Date: Sat, 6 May 2017 15:51:24 +0000 Subject: [TUHS] Discuss of style and design of computer programs from a In-Reply-To: <20170506152411.GJ28787@mcvoy.com> References: <20170506144011.GF28787@mcvoy.com> <20170506150913.57571411A@lod.com> <20170506152053.GI12539@yeono.kjorling.se> <20170506152411.GJ28787@mcvoy.com> Message-ID: <20170506155124.GK12539@yeono.kjorling.se> On 6 May 2017 08:24 -0700, from lm at mcvoy.com (Larry McVoy): >> I would absolutely not say that doing something like that is standard >> practice in modern programming. Even in microcontrollers, where >> program and data memory can be scarce even today, I would argue that >> the costs would not outweigh the benefits by a long shot. > > It strikes as being similar to Duff's device (1). Which is a niche thing > but I still use that from time to time. Not to save memory, just because > as a C programmer it seems pretty natural to do it. > > --lm > > (1) https://en.wikipedia.org/wiki/Duff's_device I disagree; loop unrolling and jumping to the beginning of some instruction inside that unrolled loop is not at all the same thing as jumping _into the middle of a machine language instruction_. -- Michael Kjörling • https://michael.kjorling.se • michael at kjorling.se “People who think they know everything really annoy those of us who know we don’t.” (Bjarne Stroustrup) From lm at mcvoy.com Sun May 7 01:53:44 2017 From: lm at mcvoy.com (Larry McVoy) Date: Sat, 6 May 2017 08:53:44 -0700 Subject: [TUHS] Discuss of style and design of computer programs from a In-Reply-To: <20170506155124.GK12539@yeono.kjorling.se> References: <20170506144011.GF28787@mcvoy.com> <20170506150913.57571411A@lod.com> <20170506152053.GI12539@yeono.kjorling.se> <20170506152411.GJ28787@mcvoy.com> <20170506155124.GK12539@yeono.kjorling.se> Message-ID: <20170506155344.GK28787@mcvoy.com> On Sat, May 06, 2017 at 03:51:24PM +0000, Michael Kj??rling wrote: > On 6 May 2017 08:24 -0700, from lm at mcvoy.com (Larry McVoy): > >> I would absolutely not say that doing something like that is standard > >> practice in modern programming. Even in microcontrollers, where > >> program and data memory can be scarce even today, I would argue that > >> the costs would not outweigh the benefits by a long shot. > > > > It strikes as being similar to Duff's device (1). Which is a niche thing > > but I still use that from time to time. Not to save memory, just because > > as a C programmer it seems pretty natural to do it. > > > > --lm > > > > (1) https://en.wikipedia.org/wiki/Duff's_device > > I disagree; loop unrolling and jumping to the beginning of some > instruction inside that unrolled loop is not at all the same thing as > jumping _into the middle of a machine language instruction_. That's fine, I feel no need to argue about it. Seemed similar to me but I'm not the sharpest tool in the shed :) From khm at sciops.net Sun May 7 02:40:02 2017 From: khm at sciops.net (Kurt H Maier) Date: Sat, 6 May 2017 09:40:02 -0700 Subject: [TUHS] Discuss of style and design of computer programs from a user stand point In-Reply-To: <20170506091857.GE12539@yeono.kjorling.se> References: <201705060202.v4622L1J013430@coolidge.cs.Dartmouth.EDU> <5a2d6cc957c2efcd968f35aa5557c7a0e309dd27@webmail.yaccman.com> <20170506091857.GE12539@yeono.kjorling.se> Message-ID: <20170506164002.GD38580@wopr> On Sat, May 06, 2017 at 09:18:57AM +0000, Michael Kjörling wrote: > > I would actually take that one step further: When you are writing > code, you are _first and foremost_ communicating with whatever human > will need to read or modify the code later. This can be, and is, taken too far. It is currently fashionable in the Ruby world to erect testing gantries that look like describe 'service' do it 'is listening' do expect(port 443).to be_listening end end and then drag in a couple hundred thousand lines of extra code to make this cuteness approximate instructions for the computer. It's all well and good to write for the person who comes behind you, but at some point it will break, and the closer you've flown to this sun the worse it will be for the poor sod who has to pick up the pieces. I'd take durable code I have to work to understand over fragile artwork any day. khm From david at kdbarto.org Sun May 7 03:52:11 2017 From: david at kdbarto.org (David) Date: Sat, 6 May 2017 10:52:11 -0700 Subject: [TUHS] Discuss of style and design of computer programs from a user stand point In-Reply-To: References: Message-ID: > From: Michael Kjörling > To: tuhs at minnie.tuhs.org > Subject: Re: [TUHS] Discuss of style and design of computer programs > from a user stand point > Message-ID: <20170506091857.GE12539 at yeono.kjorling.se> > Content-Type: text/plain; charset=utf-8 > > I would actually take that one step further: When you are writing > code, you are _first and foremost_ communicating with whatever human > will need to read or modify the code later. That human might be you, a > colleague, or the violent psychopath who knows both where you live and > where your little kids go to school (might as well be you). You should > strive to write the code accordingly, _even if_ the odds of the threat > ever materializing are slim at most. Style matters a lot, there. > Interesting, I was going to say about the same thing about the violent psychopath who has to maintain your code after you leave. When I lectured at UCSD or was giving talks on style for ViaSat I always said the same thing: Whatever you write, the fellow who is going to wind up maintaining it is a known axe killer, now released from prison, completely reformed. He learned computer programming on MS/DOS 3.1 and a slightly broken version of Pascal. He will be given your home phone number and address so if he has any questions about the code you wrote he can get in contact with you. This always got a few chuckles. I then pointed out that whenever anyone gets code that someone else wrote, the recipient always thinks that they can ‘clean up’ what is there because the original author clearly doesn’t understand what proper code looks like. Over time, I’ve learned that everyone has a style when writing code, just like handwriting and given enough time, I can spot who the author of a block of code is just from the indenting, placement of ( and ) around a statement and other small traits. What makes good code is the ability to convey the meaning of the algorithm from the original author to all those who come after. Sometimes even the most unusual code can be quite clear, while the most cleanly formatted and commented code can be opaque to all. David From dave at horsfall.org Sun May 7 04:43:33 2017 From: dave at horsfall.org (Dave Horsfall) Date: Sun, 7 May 2017 04:43:33 +1000 (EST) Subject: [TUHS] Discuss of style and design of computer programs from a user stand point In-Reply-To: <20170506144011.GF28787@mcvoy.com> References: <201705060202.v4622L1J013430@coolidge.cs.Dartmouth.EDU> <5a2d6cc957c2efcd968f35aa5557c7a0e309dd27@webmail.yaccman.com> <20170506091857.GE12539@yeono.kjorling.se> <20170506144011.GF28787@mcvoy.com> Message-ID: On Sat, 6 May 2017, Larry McVoy wrote: > When I run engineering teams the point I make is if you wrote the code > more than 6 months ago it might as well be someone else's code. So > write it in a way that someone else can debug / extend it. A principle I've tried to follow over the years is "write code as though the next person to maintain it is an axe-wielding murderer who knows where you live". > I also made the point that "clever" sucks. At least most of the time. > All "clever" means is "hard to understand". The brief joy one gets from > clever code is stomped on by the frustration one gets from having to fix > it. Apropos the above, I do confess to writing "clever code" from time to time (PERL is a beautiful language for such sins)... Of course, in the olden days it was the APL "one-liners" :-) -- Dave Horsfall DTM (VK2KFU) "Those who don't understand security will suffer." From bakul at bitblocks.com Sun May 7 05:50:49 2017 From: bakul at bitblocks.com (Bakul Shah) Date: Sat, 06 May 2017 12:50:49 -0700 Subject: [TUHS] Discuss of style and design of computer programs from a user stand point In-Reply-To: Your message of "Sun, 07 May 2017 04:43:33 +1000." References: <201705060202.v4622L1J013430@coolidge.cs.Dartmouth.EDU> <5a2d6cc957c2efcd968f35aa5557c7a0e309dd27@webmail.yaccman.com> <20170506091857.GE12539@yeono.kjorling.se> <20170506144011.GF28787@mcvoy.com> Message-ID: <20170506195049.8FBEA124AEA4@mail.bitblocks.com> On Sun, 07 May 2017 04:43:33 +1000 Dave Horsfall wrote: > > > I also made the point that "clever" sucks. At least most of the time. > > All "clever" means is "hard to understand". The brief joy one gets from > > clever code is stomped on by the frustration one gets from having to fix > > it. > Apropos the above, I do confess to writing "clever code" from time to time > (PERL is a beautiful language for such sins)... > > Of course, in the olden days it was the APL "one-liners" :-) On the other hand, you have to grok only one line of code instead reading pages of Java/C++ with surprises lurking everywhere :-) Since you need one line to do the work of many many lines of c++/Java code, you should expect to spend more time on that one line! The same is true with shell one liners as well. We don't (usually) consider constructs such as the following "clever" because we expect its reader to have a certain level of code literacy and context (or an ability to gain such). tr -cs A-Za-z '\n' | tr A-Z a-z | sort | uniq -c | sort -rn | sed ${1}q From usotsuki at buric.co Sun May 7 06:00:52 2017 From: usotsuki at buric.co (Steve Nickolas) Date: Sat, 6 May 2017 16:00:52 -0400 (EDT) Subject: [TUHS] Discuss of style and design of computer programs from a In-Reply-To: <20170506152053.GI12539@yeono.kjorling.se> References: <20170506144011.GF28787@mcvoy.com> <20170506150913.57571411A@lod.com> <20170506152053.GI12539@yeono.kjorling.se> Message-ID: On Sat, 6 May 2017, Michael Kjörling wrote: > On 6 May 2017 08:09 -0700, from corey at lod.com (Corey Lindsly): >> Anyway, I reached one point in the assembly code that I simply could not >> understand. It seemed like a mistake, and I went through it again and >> again until I finally realized what it was doing. There was a branch/loop >> that jumped to the middle of a multi-byte machine instruction, so that >> branch had to be disassembled and stepped separately until it "synced" up >> with the other branch again. Maybe this is standard practice in >> programming (I don't know) but at the time I thought, what kind of evil >> genius devised this to save a few bytes of memory? > > IIRC, that _was_ a common trick at least on machines of that class. It > did have the potential to save a few bytes, yes (more if the > instructions were such that you'd get some _other, desired_, behavior > by jumping into the middle of one with some specific state), but it > also foiled lots of disassemblers: Simply disassembling a binary from > start to finish would yield nonsense in those locations, as you > experienced. It thus basically forced you to single-step those > instructions to figure out what was going on from the binary. > > I'm pretty sure it works on every architecture with variable-length > instructions and arbitrary jump capability, as long as you have > control over the specific machine instructions generated (such as if > you are programming in assembler). Of course, it _is_ also a total > nightmare to maintain such code. > > I would absolutely not say that doing something like that is standard > practice in modern programming. Even in microcontrollers, where > program and data memory can be scarce even today, I would argue that > the costs would not outweigh the benefits by a long shot. In 6502 code, it's not uncommon to do something like foo1: lda #$00 .byte $2C ; 3-byte BIT foo2: lda #$01 . . . to save a byte (and probably still done for the few who write in ASM). The "2C" operand would cause it to disassemble as something like... 1000- LDA #$00 1002- BIT $01A9 which is the route you'd go down if you called "foo1". Apart diddling a few CPU flags, and an unneeded read on $01A9, harmless. (Most 6502 programmers would probably see a strange BIT instruction as an attempt to do this.) It's probably not a good idea to still do this unless you're really REALLY crunched for space. -uso. From michael at kjorling.se Sun May 7 07:45:19 2017 From: michael at kjorling.se (Michael =?utf-8?B?S2rDtnJsaW5n?=) Date: Sat, 6 May 2017 21:45:19 +0000 Subject: [TUHS] Discuss of style and design of computer programs from a In-Reply-To: References: <20170506144011.GF28787@mcvoy.com> <20170506150913.57571411A@lod.com> <20170506152053.GI12539@yeono.kjorling.se> Message-ID: <20170506214519.GN12539@yeono.kjorling.se> On 6 May 2017 16:00 -0400, from usotsuki at buric.co (Steve Nickolas): > In 6502 code, it's not uncommon to do something like > > foo1: lda #$00 > .byte $2C ; 3-byte BIT > foo2: lda #$01 > . > . > . > > to save a byte (and probably still done for the few who write in > ASM). The "2C" operand would cause it to disassemble as something > like... > > 1000- LDA #$00 > 1002- BIT $01A9 > > which is the route you'd go down if you called "foo1". Apart > diddling a few CPU flags, and an unneeded read on $01A9, harmless. You could do something quite similar on the 8086, which I am somewhat more familiar with. For example, in some kind of pseudo 8086 assembler, with $ denoting the value of the instruction pointer at the beginning of the current instruction: jmp short $+3 int 90h ; hex: cd 90 would almost certainly decompile to the above, if the decompiler doesn't barf when it can't create a jump target label mid-instruction, but it would certainly _execute_ as jmp short l1 db 0cdh ; dummy byte l1: nop ; hex: 90 resulting in a jump over one byte followed by a no-op instruction. Of course, invoking interrupt 90h would be a perfectly legal thing to do, assuming that your interrupt tables are set up correctly. If you want to mess with someone trying to figure out what the code is doing, write a (nonsense or meaningful) value to interrupt vector 90h before you do that. For example, assuming that it isn't being used, you could point interrupt 90h at the reboot jump location, which IIRC on the IBM PC would mean point it at FFFFh:FFF0h or absolute address FFFF0h. Anyone trying to actually execute the INT 90h instruction would see their computer reboot, but anyone actually executing the code would see little more than a CPU cache flush due to the jump. Or, something slightly more "useful" jmp short $+3 int 0 ; hex: cd 00 db 0 jz somewhere which, again IIRC, would clear the accumulator (AX) register because 0000 hex is XOR AX,AX, which as a side effect sets the zero flag because the result is zero, and then executes a conditional jump if the zero flag is set (which it will be). In other words, it _executes_ as jmp short l1 db 0cdh ; dummy byte l1: xor ax,ax ; hex: 00 00 jz somewhere but a naiive decompiler will see the CDh byte after the jump and take that as the first byte of the two-byte interrupt instruction. I don't know what 00 followed by the first byte of the JZ instruction would be, but probably a register/register XOR with a register store. Make it a far jump and you have a few extra bytes to play with, to befuddle anyone trying to figure out just what on Earth your code is really doing. The above, of course, is really just scratching the surface. With how many variable-length instructions the x86 has, with careful selection of opcodes and possibly use of undocumented but functionally valid combinations, I wouldn't be the least bit surprised if it's technically possible to write a 8086 program that does one thing when started normally, and something utterly different but still useful when started at an offset that results in execution beginning in the middle of an instruction. Bonus points if the first thing that program does is jump into itself _at that offset_. Now, the _work involved in doing so_, never mind maintaining it... Of course, I wouldn't do anything like the above in any real-world code base meant to run on modern systems unless obfuscation really _was_ a design goal. Honest. If I was tasked to write something that really needed to do something non-trivial in 16 KiB RAM on an original IBM 5150 PC or clone? I'd probably spend quite a bit of time in front of the whiteboard and with reference manuals before writing even one line of code... -- Michael Kjörling • https://michael.kjorling.se • michael at kjorling.se “People who think they know everything really annoy those of us who know we don’t.” (Bjarne Stroustrup) From scj at yaccman.com Sun May 7 08:11:32 2017 From: scj at yaccman.com (Steve Johnson) Date: Sat, 06 May 2017 15:11:32 -0700 Subject: [TUHS] Discuss of style and design of computer programs from a user stand point [was dmr note on BSD's sins] In-Reply-To: <20170506060727.A00C0124AEFE@mail.bitblocks.com> Message-ID: <5abeeef795d15ae5b9a1eaca3bcb4f684ecb1c4d@webmail.yaccman.com>  few more comments about style and optimization:First, the thing that make optimized code hard to understand is that (at least when a compiler does it) the optimized program goes through a different set of states than the user expected from the source code.  When a loop is unrolled, or invariant expressions moved out of a loop, or indexing is turned into pointer manipulation, if you don't know what's happening its mysterious.  But once you'v seen this a few times, it's only a momentary hitch...Also, one lesson I learned from Unix is that if you can make a program 10x faster it becomes a qualitatively different experience.  The first version of Yacc I wrote, using the textbook algorithms, took 20 minutes to process a 30-rule grammar.  On the shared PDP-11, when I ran Yacc everyone else's jobs slowed to a crawl, and people were heard to mutter "*%&#!  Johnson's running Yacc again!"   A couple of years later, Yacc produced parsers faster than the C compiler could compile them.Consider what using Google would be like if it took 7 seconds rather than .7 seconds to respond.  Probably still useful, but a different user experience.We had many decades to be spoiled as far as program performance was concerned.  As our tools got more bloated, faster processors and more memory hid the true effect from most of us.  Now we have to rethink everything to make it parallel.  And complexity is enemy number one when making something parallel... Steve -------------- next part -------------- An HTML attachment was scrubbed... URL: From lm at mcvoy.com Sun May 7 09:35:56 2017 From: lm at mcvoy.com (Larry McVoy) Date: Sat, 6 May 2017 16:35:56 -0700 Subject: [TUHS] Discuss of style and design of computer programs from a user stand point [was dmr note on BSD's sins] In-Reply-To: <5abeeef795d15ae5b9a1eaca3bcb4f684ecb1c4d@webmail.yaccman.com> References: <20170506060727.A00C0124AEFE@mail.bitblocks.com> <5abeeef795d15ae5b9a1eaca3bcb4f684ecb1c4d@webmail.yaccman.com> Message-ID: <20170506233556.GG9278@mcvoy.com> On Sat, May 06, 2017 at 03:11:32PM -0700, Steve Johnson wrote: > And complexity is enemy number one when making something parallel... +1 Couldn't agree more, for multiple reasons. -- --- Larry McVoy lm at mcvoy.com http://www.mcvoy.com/lm From cym224 at gmail.com Sun May 7 10:51:05 2017 From: cym224 at gmail.com (Nemo) Date: Sat, 6 May 2017 20:51:05 -0400 Subject: [TUHS] Discuss of style and design of computer programs from a Message-ID: On 6 May 2017 at 11:23, ron minnich wrote (in part): [...] > Lest you think things are better now, Linux uses self modifying code to > optimize certain critical operations, and at one talk I heard the speaker > say that he'd like to put more self modifying code into Linux, "because it's > fun". Oh boy. Fun, indeed! Even self-modifying chips are being touted -- Yikes! N. From imp at bsdimp.com Sun May 7 11:15:43 2017 From: imp at bsdimp.com (Warner Losh) Date: Sat, 6 May 2017 19:15:43 -0600 Subject: [TUHS] Discuss of style and design of computer programs from a user stand point In-Reply-To: <20170506195049.8FBEA124AEA4@mail.bitblocks.com> References: <201705060202.v4622L1J013430@coolidge.cs.Dartmouth.EDU> <5a2d6cc957c2efcd968f35aa5557c7a0e309dd27@webmail.yaccman.com> <20170506091857.GE12539@yeono.kjorling.se> <20170506144011.GF28787@mcvoy.com> <20170506195049.8FBEA124AEA4@mail.bitblocks.com> Message-ID: On Sat, May 6, 2017 at 1:50 PM, Bakul Shah wrote: > tr -cs A-Za-z '\n' | tr A-Z a-z | sort | uniq -c | sort -rn | sed ${1}q The cool thing about this thread is that I learned two things: what tr -s does, and the Nq does for sed... Sadly, this doesn't work so well for text that isn't ASCII-7 english... Warner From noel.hunt at gmail.com Sun May 7 11:42:31 2017 From: noel.hunt at gmail.com (Noel Hunt) Date: Sun, 7 May 2017 11:42:31 +1000 Subject: [TUHS] Discuss of style and design of computer programs from a user stand point In-Reply-To: References: <201705060202.v4622L1J013430@coolidge.cs.Dartmouth.EDU> <5a2d6cc957c2efcd968f35aa5557c7a0e309dd27@webmail.yaccman.com> <20170506091857.GE12539@yeono.kjorling.se> <20170506144011.GF28787@mcvoy.com> <20170506195049.8FBEA124AEA4@mail.bitblocks.com> Message-ID: I was about to suggest using the Plan9 port utilities of the same name but it seems 'uniq' is not coded to handle Runes (aka utf-8). I don't imagine it would be hard to re-write it to handle utf-8. On Sun, May 7, 2017 at 11:15 AM, Warner Losh wrote: > On Sat, May 6, 2017 at 1:50 PM, Bakul Shah wrote: > > tr -cs A-Za-z '\n' | tr A-Z a-z | sort | uniq -c | sort -rn | sed ${1}q > > The cool thing about this thread is that I learned two things: what tr > -s does, and the Nq does for sed... > > Sadly, this doesn't work so well for text that isn't ASCII-7 english... > > Warner > -------------- next part -------------- An HTML attachment was scrubbed... URL: From crossd at gmail.com Sun May 7 14:06:48 2017 From: crossd at gmail.com (Dan Cross) Date: Sun, 7 May 2017 00:06:48 -0400 Subject: [TUHS] Discuss of style and design of computer programs from a user stand point [was dmr note on BSD's sins] In-Reply-To: <20170506060727.A00C0124AEFE@mail.bitblocks.com> References: <1C6D6D3E-058F-4743-8DA0-17CEC09B64E9@bitblocks.com> <20170506060727.A00C0124AEFE@mail.bitblocks.com> Message-ID: On Sat, May 6, 2017 at 2:07 AM, Bakul Shah wrote: > Note that a similar approach was implemented by Iverson in > APL. In fact a shell pipeline has a lot in common with array > programming! > > Iverson in essence built a whole workshop with a set of simple > but powerful standard tools (functions and higher order > functions). The same has been carried on in follow on array > languages such as j and k and q languages. Many of these > array functions are directly applicable to stream processing. > In contrast unix shells provide a very shallow glue layer. Interesting analogy. I always think that function composition in lazy languages like Haskell reminds me of shell pipelines. I know that Doug swims in those waters; perhaps he will correct me if he feels that's a poor analogy? Carrying it forward a bit, however, I'm always amazed (and a little appalled) at how the kids always think that they invented Rock n' Roll. Ideas are continually recycled, but it's the rare recycler indeed who seeks out the prior art on rediscovering one of the good ones. "Look! You can filter and transform this data!" "...but I've been doing that at the shell nearly every day for the past quarter of a century...." "...this one goes to 11." - Dan C. -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at sk2.org Sun May 7 17:42:27 2017 From: steve at sk2.org (Stephen Kitt) Date: Sun, 7 May 2017 09:42:27 +0200 Subject: [TUHS] Discuss of style and design of computer programs from a In-Reply-To: <20170506214519.GN12539@yeono.kjorling.se> References: <20170506144011.GF28787@mcvoy.com> <20170506150913.57571411A@lod.com> <20170506152053.GI12539@yeono.kjorling.se> <20170506214519.GN12539@yeono.kjorling.se> Message-ID: <20170507094227.4af8e074@heffalump.sk2.org> On Sat, 6 May 2017 21:45:19 +0000, Michael Kjörling wrote: > On 6 May 2017 16:00 -0400, from usotsuki at buric.co (Steve Nickolas): > > In 6502 code, it's not uncommon to do something like > > > > foo1: lda #$00 > > .byte $2C ; 3-byte BIT > > foo2: lda #$01 > > . > > . > > . > > > > to save a byte (and probably still done for the few who write in > > ASM). The "2C" operand would cause it to disassemble as something > > like... > > > > 1000- LDA #$00 > > 1002- BIT $01A9 > > > > which is the route you'd go down if you called "foo1". Apart > > diddling a few CPU flags, and an unneeded read on $01A9, harmless. > > You could do something quite similar on the 8086, which I am somewhat > more familiar with. [...] One possible equivalent 16-bit x86 is foo1: mov al, 0 db 035h ; XOR AX, imm16 foo2: mov al, 1 which implements a branchless fall-through. (But you’d probably just use SALC...) > Or, something slightly more "useful" > > jmp short $+3 > int 0 ; hex: cd 00 > db 0 > jz somewhere > > which, again IIRC, would clear the accumulator (AX) register because > 0000 hex is XOR AX,AX, According to my disassembler 0000 is "add byte ptr [bx+si], al"; "xor ax, ax" is either 31c0 or 33c0. > which as a side effect sets the zero flag > because the result is zero, and then executes a conditional jump if > the zero flag is set (which it will be). In other words, it _executes_ > as > > jmp short l1 > db 0cdh ; dummy byte > l1: xor ax,ax ; hex: 00 00 > jz somewhere > > but a naiive decompiler will see the CDh byte after the jump and take > that as the first byte of the two-byte interrupt instruction. I don't > know what 00 followed by the first byte of the JZ instruction would > be, but probably a register/register XOR with a register store. Make > it a far jump and you have a few extra bytes to play with, to befuddle > anyone trying to figure out just what on Earth your code is really > doing. My memories of disassembling x86 code in the early 90s at least is that, because of all this, disassemblers were pretty good at restarting streams from jump destinations, so it never really was an issue. You need to get creative with single-stepping mode and self-decrypting code to really annoy people trying to understand your code ;-). > The above, of course, is really just scratching the surface. With how > many variable-length instructions the x86 has, with careful selection > of opcodes and possibly use of undocumented but functionally valid > combinations, I wouldn't be the least bit surprised if it's > technically possible to write a 8086 program that does one thing when > started normally, and something utterly different but still useful > when started at an offset that results in execution beginning in the > middle of an instruction. Bonus points if the first thing that program > does is jump into itself _at that offset_. Now, the _work involved in > doing so_, never mind maintaining it... I have seen some short sections of code which do this. The x86 ISA is so rich that you can make it adapt to many different constraints — one I rather liked is assembling code which is valid ASCII (amaze your friends by typing in a .COM using COPY CON!). There’s a C compiler which does that on GitHub somewhere. > Of course, I wouldn't do anything like the above in any real-world > code base meant to run on modern systems unless obfuscation really > _was_ a design goal. Honest. If I was tasked to write something that > really needed to do something non-trivial in 16 KiB RAM on an original > IBM 5150 PC or clone? I'd probably spend quite a bit of time in front > of the whiteboard and with reference manuals before writing even one > line of code... Yes, and combine that with assemblers which output all possible encodings of a given instruction! Regards, Stephen From michael at kjorling.se Sun May 7 23:49:43 2017 From: michael at kjorling.se (Michael =?utf-8?B?S2rDtnJsaW5n?=) Date: Sun, 7 May 2017 13:49:43 +0000 Subject: [TUHS] Discuss of style and design of computer programs from a user stand point In-Reply-To: References: <1C6D6D3E-058F-4743-8DA0-17CEC09B64E9@bitblocks.com> <20170506060727.A00C0124AEFE@mail.bitblocks.com> Message-ID: <20170507134943.GR12539@yeono.kjorling.se> On 7 May 2017 00:06 -0400, from crossd at gmail.com (Dan Cross): > "Look! You can > filter and transform this data!" "...but I've been doing that at the shell > nearly every day for the past quarter of a century...." "...this one goes > to 11." "Look! This code can execute code that it creates itself!" "LISP has been around for a while." -- Michael Kjörling • https://michael.kjorling.se • michael at kjorling.se “People who think they know everything really annoy those of us who know we don’t.” (Bjarne Stroustrup) From michael at kjorling.se Sun May 7 23:54:07 2017 From: michael at kjorling.se (Michael =?utf-8?B?S2rDtnJsaW5n?=) Date: Sun, 7 May 2017 13:54:07 +0000 Subject: [TUHS] Discuss of style and design of computer programs from a user stand point In-Reply-To: References: <201705060202.v4622L1J013430@coolidge.cs.Dartmouth.EDU> <5a2d6cc957c2efcd968f35aa5557c7a0e309dd27@webmail.yaccman.com> <20170506091857.GE12539@yeono.kjorling.se> <20170506144011.GF28787@mcvoy.com> <20170506195049.8FBEA124AEA4@mail.bitblocks.com> Message-ID: <20170507135407.GS12539@yeono.kjorling.se> On 7 May 2017 11:42 +1000, from noel.hunt at gmail.com (Noel Hunt): > I don't imagine it would be hard to re-write [uniq] to > handle utf-8. It does look like at least GNU coreutils 8.13 uniq is broken in that regard, which frankly surprised me. That version isn't _that_ old. $ uniq --version uniq (GNU coreutils) 8.13 Copyright (C) 2011 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later . This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Written by Richard M. Stallman and David MacKenzie. $ ( echo $'\u1234' ; echo $'\u2345' ; echo $'\u1234' ) ሴ ⍅ ሴ $ ( echo $'\u1234' ; echo $'\u2345' ; echo $'\u1234' ) | uniq ሴ $ -- Michael Kjörling • https://michael.kjorling.se • michael at kjorling.se “People who think they know everything really annoy those of us who know we don’t.” (Bjarne Stroustrup) From arnold at skeeve.com Mon May 8 00:58:24 2017 From: arnold at skeeve.com (arnold at skeeve.com) Date: Sun, 07 May 2017 08:58:24 -0600 Subject: [TUHS] Discuss of style and design of computer programs from a user stand point In-Reply-To: <20170507135407.GS12539@yeono.kjorling.se> References: <201705060202.v4622L1J013430@coolidge.cs.Dartmouth.EDU> <5a2d6cc957c2efcd968f35aa5557c7a0e309dd27@webmail.yaccman.com> <20170506091857.GE12539@yeono.kjorling.se> <20170506144011.GF28787@mcvoy.com> <20170506195049.8FBEA124AEA4@mail.bitblocks.com> <20170507135407.GS12539@yeono.kjorling.se> Message-ID: <201705071458.v47EwOtp014162@freefriends.org> Michael Kjörling wrote: > On 7 May 2017 11:42 +1000, from noel.hunt at gmail.com (Noel Hunt): > > I don't imagine it would be hard to re-write [uniq] to > > handle utf-8. > > It does look like at least GNU coreutils 8.13 uniq is broken in that > regard, which frankly surprised me. That version isn't _that_ old. Are your LC_* env variables set correctly? If you file a bug report, it should get fixed. Thanks, Arnold From imp at bsdimp.com Mon May 8 01:13:11 2017 From: imp at bsdimp.com (Warner Losh) Date: Sun, 7 May 2017 09:13:11 -0600 Subject: [TUHS] Discuss of style and design of computer programs from a user stand point In-Reply-To: References: <201705060202.v4622L1J013430@coolidge.cs.Dartmouth.EDU> <5a2d6cc957c2efcd968f35aa5557c7a0e309dd27@webmail.yaccman.com> <20170506091857.GE12539@yeono.kjorling.se> <20170506144011.GF28787@mcvoy.com> <20170506195049.8FBEA124AEA4@mail.bitblocks.com> Message-ID: On Sat, May 6, 2017 at 7:42 PM, Noel Hunt wrote: > I was about to suggest using the Plan9 port utilities of the > same name but it seems 'uniq' is not coded to handle Runes > (aka utf-8). I don't imagine it would be hard to re-write it to > handle utf-8. I guess I should have been clearer on what wouldn't work. It can't possibly work for Japanese and Chinese where words aren't separated by whitespace. Would cause problems in hybrid languages where words can be composed of logograms and sonograms (say Japanese which often use a few Kanji with hiragana endings that then run into hiragana particles or other grammar elements). Can't work without modification (using class names) for Cyrillic because there's no A or Z in words there. Won't work in any language that has a discontiguous set of letters, which includes many western european languages since all the accented or otherwise decorated letters aren't in the range A-Z. So whether or not the underlying tools can handle UTF-8 encoding, there are problems with the original. If you used: tr -cs "[:alpha:]" '\n' | tr "[:upper:]" "[:lower:]" | sort | uniq -c | sort -rn | sed ${1}q you'd still have issues with languages that don't use word separators, or write non-alphabetically. Warner > On Sun, May 7, 2017 at 11:15 AM, Warner Losh wrote: >> >> On Sat, May 6, 2017 at 1:50 PM, Bakul Shah wrote: >> > tr -cs A-Za-z '\n' | tr A-Z a-z | sort | uniq -c | sort -rn | sed ${1}q >> >> The cool thing about this thread is that I learned two things: what tr >> -s does, and the Nq does for sed... >> >> Sadly, this doesn't work so well for text that isn't ASCII-7 english... >> >> Warner > > From michael at kjorling.se Mon May 8 02:33:08 2017 From: michael at kjorling.se (Michael =?utf-8?B?S2rDtnJsaW5n?=) Date: Sun, 7 May 2017 16:33:08 +0000 Subject: [TUHS] Discuss of style and design of computer programs from a user stand point In-Reply-To: <201705071458.v47EwOtp014162@freefriends.org> References: <201705060202.v4622L1J013430@coolidge.cs.Dartmouth.EDU> <5a2d6cc957c2efcd968f35aa5557c7a0e309dd27@webmail.yaccman.com> <20170506091857.GE12539@yeono.kjorling.se> <20170506144011.GF28787@mcvoy.com> <20170506195049.8FBEA124AEA4@mail.bitblocks.com> <20170507135407.GS12539@yeono.kjorling.se> <201705071458.v47EwOtp014162@freefriends.org> Message-ID: <20170507163308.GT12539@yeono.kjorling.se> On 7 May 2017 08:58 -0600, from arnold at skeeve.com: >>> I don't imagine it would be hard to re-write [uniq] to handle utf-8. >> >> It does look like at least GNU coreutils 8.13 uniq is broken in that >> regard, which frankly surprised me. That version isn't _that_ old. > > Are your LC_* env variables set correctly? I believe so (I don't recall seeing any other UTF-8-related weirdness for a very long time), but I would want to verify the behavior on a system that doesn't have a gazillion customizations accumulated over years _and_ has the most recent version of coreutils before I file an actual bug report. (It's not like Debian is ever bleeding edge, to begin with.) I'll see if I spin up a VM or three one of these days to check. -- Michael Kjörling • https://michael.kjorling.se • michael at kjorling.se “People who think they know everything really annoy those of us who know we don’t.” (Bjarne Stroustrup) From doug at cs.dartmouth.edu Mon May 8 04:19:02 2017 From: doug at cs.dartmouth.edu (Doug McIlroy) Date: Sun, 07 May 2017 14:19:02 -0400 Subject: [TUHS] Discuss of style and design of computer programs from a user standpoint Message-ID: <201705071819.v47IJ24H002013@coolidge.cs.Dartmouth.EDU> > tr -cs A-Za-z '\n' | > tr A-Z a-z | > sort | > uniq -c | > sort -rn | > sed ${1}q > > This is real genius. Not genius. Experience. In the Bentley/Knuth/McIlroy paper I said, "[Old] Unix hands know instinctively how to solve this one in a jiffy." While that is certainly true, the script was informed by my having written "spell", which itself was an elaboration of a model pioneered by Steve Johnson. By 1986, when BKM was published, the lore was baked in: word-processing scripts in a similar vein were stock in trade. A very early exercise of this sort was Dennis Ritchie's enumeration of anagrams in the unabridged Merriam-Webster. Since the word list barely fit on the tiny disk of the time, the job entailed unimaginable marshalling of resources. I was mightily impressed then, and still am. Doug From ches at cheswick.com Mon May 8 04:50:52 2017 From: ches at cheswick.com (William Cheswick) Date: Sun, 7 May 2017 14:50:52 -0400 Subject: [TUHS] Discuss of style and design of computer programs from a user standpoint In-Reply-To: <201705071819.v47IJ24H002013@coolidge.cs.Dartmouth.EDU> References: <201705071819.v47IJ24H002013@coolidge.cs.Dartmouth.EDU> Message-ID: <5D559802-6D6E-4668-9CA5-141B071BC535@cheswick.com> > On 7May 2017, at 2:19 PM, Doug McIlroy wrote: > > Not genius. Experience. It’s the pipe that was genius. When I first heard the idea, I thought it was preposterous. My programs’ outputs were designed for the line printer, with carriage control characters. How could this idea be useful? These tools are powerful--I co-founded a company out of network probe software mostly designed around filters. And it scales beautifully to multi-core computing. I had movie visualization software suddenly nearly double in speed one day. They had doubled the number of CPUs in the ferric cluster. My favorite filter: the unsort (or scramble) command. Honeyman’s “idiom” command: sort | uniq -c | sort -rn From lm at mcvoy.com Mon May 8 08:03:47 2017 From: lm at mcvoy.com (Larry McVoy) Date: Sun, 7 May 2017 15:03:47 -0700 Subject: [TUHS] Discuss of style and design of computer programs from a user standpoint In-Reply-To: <5D559802-6D6E-4668-9CA5-141B071BC535@cheswick.com> References: <201705071819.v47IJ24H002013@coolidge.cs.Dartmouth.EDU> <5D559802-6D6E-4668-9CA5-141B071BC535@cheswick.com> Message-ID: <20170507220347.GQ9278@mcvoy.com> On Sun, May 07, 2017 at 02:50:52PM -0400, William Cheswick wrote: > My favorite filter: the unsort (or scramble) command. > sort | uniq -c | sort -rn I use that one all the time. Super handy. -- --- Larry McVoy lm at mcvoy.com http://www.mcvoy.com/lm From dot at dotat.at Mon May 8 23:39:36 2017 From: dot at dotat.at (Tony Finch) Date: Mon, 8 May 2017 14:39:36 +0100 Subject: [TUHS] Discuss of style and design of computer programs from a In-Reply-To: References: Message-ID: Nemo wrote: > On 6 May 2017 at 11:23, ron minnich wrote (in part): > [...] > > Lest you think things are better now, Linux uses self modifying code to > > optimize certain critical operations, and at one talk I heard the speaker > > say that he'd like to put more self modifying code into Linux, "because it's > > fun". Oh boy. > > Fun, indeed! Even self-modifying chips are being touted -- Yikes! You reminded me of these comments on a bug in NVidia's Tegra "Project Denver" dynamic JIT firmware: https://twitter.com/FioraAeterna/status/855445075341398017 > > small brain: bug in your code > big brain: bug in the compiler > cosmic brain: bug in the cpu's on-chip recompiler > https://github.com/golang/go/issues/19809#issuecomment-290804472 https://twitter.com/eqe/status/855533948931252224 > > This happened with TransMeta back in the day, and now with Tegra. I > wonder if NVidia has a update deployment strategy... (Marginally topical relevance is that Linus Torvalds worked for Transmeta) Tony. -- f.anthony.n.finch http://dotat.at/ - I xn--zr8h punycode Lundy, Fastnet, Irish Sea, Shannon, Rockall, Malin, South Hebrides: Easterly or northeasterly 4 or 5, occasionally 6 at first, becoming variable 3 at times later. Slight or moderate. Fair. Good. From scj at yaccman.com Tue May 9 02:21:47 2017 From: scj at yaccman.com (Steve Johnson) Date: Mon, 08 May 2017 09:21:47 -0700 Subject: [TUHS] Discuss of style and design of computer programs from a In-Reply-To: Message-ID: <0aab8f20d7cbcc18f830870cb361db844c229d28@webmail.yaccman.com> That's a very interesting bug, and Fiora is a genius... I too worked at Transmeta, although our recompiler was not on chip... We designed the chip originally based on Spec benchmarks.  This turned out to be a mistake because Windows doesn't look a bit like Spec. For one thing, when booting Windows, 50% of all the instructions were executed once only.  When we found all of these instructions and JITted their basic blocks, it ran like a turtle.  The solution was to put in an X86 interpreter, and only JIT when the instruction was executed 5 times.  The resulting performance was more than acceptable.  But there was a problem demonstrating this with a number of benchmarks, namely those that * Ran a small problem and timed it. * Based on that, made a bigger problem that was scaled by #1 -- the slower the small problem, the smaller the big problem. * Added 1 and 2 to get the score. The small problem often ran mostly in the interpreter.  So the "big" problem was really pretty small.  That ran fast, but overall, #1 dominated. If we actually ran the big problem from the getgo, we looked pretty decent. Among the interesting facts we learned:  Windows executed a billion instructions between detecting a reason to crash and putting up the blue screen... We also ran into problems with self modifying code.  To their credit, Microsoft quickly fixed the things we pointed out (most in 3rd-party drivers). Transmeta actually had some of the most interesting technology I've ever worked with as a compiler writer, including a checkpoint/restart in hardware that let the JIT do out-of-order execution and then do it over serially if a fault appeared. Steve ----- Original Message ----- From: "Tony Finch" To:"Nemo" Cc:"TUHS main list" Sent:Mon, 8 May 2017 14:39:36 +0100 Subject:Re: [TUHS] Discuss of style and design of computer programs from a Nemo wrote: > On 6 May 2017 at 11:23, ron minnich wrote (in part): > [...] > > Lest you think things are better now, Linux uses self modifying code to > > optimize certain critical operations, and at one talk I heard the speaker > > say that he'd like to put more self modifying code into Linux, "because it's > > fun". Oh boy. > > Fun, indeed! Even self-modifying chips are being touted -- Yikes! You reminded me of these comments on a bug in NVidia's Tegra "Project Denver" dynamic JIT firmware: https://twitter.com/FioraAeterna/status/855445075341398017 > > small brain: bug in your code > big brain: bug in the compiler > cosmic brain: bug in the cpu's on-chip recompiler > https://github.com/golang/go/issues/19809#issuecomment-290804472 https://twitter.com/eqe/status/855533948931252224 > > This happened with TransMeta back in the day, and now with Tegra. I > wonder if NVidia has a update deployment strategy... (Marginally topical relevance is that Linus Torvalds worked for Transmeta) Tony. -- f.anthony.n.finch http://dotat.at/ - I xn--zr8h punycode Lundy, Fastnet, Irish Sea, Shannon, Rockall, Malin, South Hebrides: Easterly or northeasterly 4 or 5, occasionally 6 at first, becoming variable 3 at times later. Slight or moderate. Fair. Good. -------------- next part -------------- An HTML attachment was scrubbed... URL: From crossd at gmail.com Tue May 9 03:01:33 2017 From: crossd at gmail.com (Dan Cross) Date: Mon, 8 May 2017 13:01:33 -0400 Subject: [TUHS] Discuss of style and design of computer programs from a In-Reply-To: <0aab8f20d7cbcc18f830870cb361db844c229d28@webmail.yaccman.com> References: <0aab8f20d7cbcc18f830870cb361db844c229d28@webmail.yaccman.com> Message-ID: On Mon, May 8, 2017 at 12:21 PM, Steve Johnson wrote: > That's a very interesting bug, and Fiora is a genius... > That bug actually came from Dave Chase at Google. He mentioned it to me in passing when he found it, "Hey, want to hear about this crazy hardware bug we found?" I remember just sort of looking at him with this, "you've got to be kidding me..." expression. - Dan C. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dds at aueb.gr Thu May 11 00:08:15 2017 From: dds at aueb.gr (Diomidis Spinellis) Date: Wed, 10 May 2017 17:08:15 +0300 Subject: [TUHS] The evolution of Unix facilities and architecture Message-ID: <583620cb-6856-b9ee-a5ee-32885c09362b@aueb.gr> I've made available on GitHub a series of tables showing the evolution of Unix facilities (as documented in the man pages) over the system's lifetime [1] and two diagrams where I attempted to draw the corresponding architecture [2]. I've also documented the process in a short blog post [3]. I'd welcome any suggestions for corrections and improvements you may have, particularly for the architecture diagrams. [1] https://dspinellis.github.io/unix-history-man/ [2] https://dspinellis.github.io/unix-architecture/ [3] https://www.spinellis.gr/blog/20170510/ Cheers, Diomidis From steffen at sdaoden.eu Thu May 11 00:38:13 2017 From: steffen at sdaoden.eu (Steffen Nurpmeso) Date: Wed, 10 May 2017 16:38:13 +0200 Subject: [TUHS] The evolution of Unix facilities and architecture In-Reply-To: <583620cb-6856-b9ee-a5ee-32885c09362b@aueb.gr> References: <583620cb-6856-b9ee-a5ee-32885c09362b@aueb.gr> Message-ID: <20170510143813.YrQ5R%steffen@sdaoden.eu> Yes, hi, Diomidis Spinellis wrote: |I've made available on GitHub a series of tables showing the evolution |of Unix facilities (as documented in the man pages) over the system's |lifetime [1] and two diagrams where I attempted to draw the |corresponding architecture [2]. I've also documented the process in a |short blog post [3]. I'd welcome any suggestions for corrections and |improvements you may have, particularly for the architecture diagrams. | |[1] https://dspinellis.github.io/unix-history-man/ i am confident groff was part of (Free)BSD even before it was released. That is, i have used groff with FreeBSD 4.9 (?) base systems onwards, and i know from the CSRG history that they worked to get there. (It is not part of the repo, though, it is only deduced. I still don't have the McKusick CD's, shame on me.) |[2] https://dspinellis.github.io/unix-architecture/ |[3] https://www.spinellis.gr/blog/20170510/ --steffen | |Ralph says i must not use signatures which spread the light! From erik at ono-sendai.com Thu May 11 09:09:09 2017 From: erik at ono-sendai.com (Erik Berls) Date: Wed, 10 May 2017 16:09:09 -0700 Subject: [TUHS] The evolution of Unix facilities and architecture In-Reply-To: <20170510143813.YrQ5R%steffen@sdaoden.eu> References: <583620cb-6856-b9ee-a5ee-32885c09362b@aueb.gr> <20170510143813.YrQ5R%steffen@sdaoden.eu> Message-ID: groff came into NetBSD (of which, FreeBSD 1.0 incorporated a pre-release NetBSD 0.8 tarball), with the 386bsd + patchkit initial import.  Sun Mar 21 09:45:37 1993 UTC, by cgd.  It was upgraded to groff release 1.08 about 4 months later, by jtc. Source: NetBSD’s cvsweb -----Original Message----- From: Steffen Nurpmeso Reply: tuhs at minnie.tuhs.org Date: May 10, 2017 at 2:16:16 PM To: Diomidis Spinellis Cc: tuhs at minnie.tuhs.org Subject:  Re: [TUHS] The evolution of Unix facilities and architecture > Yes, hi, > > Diomidis Spinellis wrote: > |I've made available on GitHub a series of tables showing the evolution > |of Unix facilities (as documented in the man pages) over the system's > |lifetime [1] and two diagrams where I attempted to draw the > |corresponding architecture [2]. I've also documented the process in a > |short blog post [3]. I'd welcome any suggestions for corrections and > |improvements you may have, particularly for the architecture diagrams. > | > |[1] https://dspinellis.github.io/unix-history-man/ > > i am confident groff was part of (Free)BSD even before it was > released. That is, i have used groff with FreeBSD 4.9 (?) base > systems onwards, and i know from the CSRG history that they worked > to get there. (It is not part of the repo, though, it is only > deduced. I still don't have the McKusick CD's, shame on me.) > > |[2] https://dspinellis.github.io/unix-architecture/ > |[3] https://www.spinellis.gr/blog/20170510/ > > --steffen > | > |Ralph says i must not use signatures which spread the light! > -- Erik Berls From clemc at ccc.com Thu May 11 10:49:37 2017 From: clemc at ccc.com (Clem Cole) Date: Wed, 10 May 2017 20:49:37 -0400 Subject: [TUHS] The evolution of Unix facilities and architecture In-Reply-To: <583620cb-6856-b9ee-a5ee-32885c09362b@aueb.gr> References: <583620cb-6856-b9ee-a5ee-32885c09362b@aueb.gr> Message-ID: On Wed, May 10, 2017 at 10:08 AM, Diomidis Spinellis wrote: > I've made available on GitHub a series of tables showing the evolution of > Unix facilities (as documented in the man pages) over the system's lifetime > [1] and two diagrams where I attempted to draw the corresponding > architecture [2]. I've also documented the process in a short blog post > [3]. I'd welcome any suggestions for corrections and improvements you may > have, particularly for the architecture diagrams. > > [1] https://dspinellis.github.io/unix-history-man/ ​fsck(8) is not a BSD program. It is a CMU program originally, although Michigan can sort of claim it also. Ted Kowalski is (was) the primary author of fsck. He started writing an earlier idea for it at Michigan, which came from something like it he had seen on MTS, I believe called "Scavenger" and IIRC we had also had TSS called 'Vulture' - which cleaned up the 'carrion' after a disk crash (MTS and TSS are brothers for the IBM 360/67 and a number of early UNIX hacker also cut their teeth). Anyway, fsck, scavenger and vulture were all of the same idea. The primary work was done on the CMU 11/34 EE Digital lab system, in first floor of Hammershag Hall in the mid-1970s. Note, I had a >>very<< small hand in fsck, as Ted was teaching me about C at the time (you can blame me for the upper case error messages - that how MTS and TSS worked in those days). If you look at edits and style it's clearly Ted's code. BTW: Ted took the sources to fsck back to the labs when he finished it, and the program was first released via the Summit streams but I can not tell which one [I think PWB 2.0 was the first 'official' AT&T version - aps his old office mate at Summit, might know]. I believe one of the AT&T features was support for RP06, which took 'swapping' and the temporary paging file stuff on the PDP-11. That was not need at CMU because we did not have disk that large on our PDP-11s. Note it was was originally written for the for the 6th edition FS (which I hope I have still have the sources in my files) as well as later for the 7th edition (or as Ted would called it UNIX/TS - which we had had a CMU shortly after he had upgraded us - we ran a hybrid system until the 1979 when V7 was finally released). I believe Noel recovered a copy in his files recently. As I have said previously, fsck migrated to a number of sites independently - via mag tape most likely; Noel has pointed out that MIT had it in the late 1970s also, probably brought back from BTL by one of their summer students. That said, it went more main stream via the BSD 4.x tapes when Joy passed it on, but all of that pre-dates BSD 4.x. How UCB (Joy) got it is unknown, although I have also pointed out that Ted was Joy's housemate at Michigan when they were undergrads, and Ted quite likely sent him a tape or someone like Ken or any number of other BTL folks could have brought it with him when they were there.​ ​Clem​ -------------- next part -------------- An HTML attachment was scrubbed... URL: From steffen at sdaoden.eu Thu May 11 22:40:02 2017 From: steffen at sdaoden.eu (Steffen Nurpmeso) Date: Thu, 11 May 2017 14:40:02 +0200 Subject: [TUHS] The evolution of Unix facilities and architecture In-Reply-To: References: <583620cb-6856-b9ee-a5ee-32885c09362b@aueb.gr> <20170510143813.YrQ5R%steffen@sdaoden.eu> Message-ID: <20170511124002.kJmQX%steffen@sdaoden.eu> Erik Berls wrote: |groff came into NetBSD (of which, FreeBSD 1.0 incorporated a |pre-release NetBSD 0.8 tarball), with the 386bsd + patchkit initial |import.  Sun Mar 21 09:45:37 1993 UTC, by cgd.  It was upgraded to |groff release 1.08 about 4 months later, by jtc. | |Source: NetBSD’s cvsweb So come, ey, Mr. Spinellis has a devilish red focus on FreeBSD. No other BSD in sight. For the git clone of FreeBSD that i have i see groff-1.09 import as a series of committs starting with [b4b083cfbe2] around 1995-01-17. But i better should not have looked, back then perl was still part of the base system, and today Lua is missing, too. P.S.: depite the fact that my VServer (still) runs Alpine happily. At least there is awk, i currently have a mail-server rotation period of ~12 hours because of otherwise valid nonsense connections that blacklistd is not covering, and isn't it absurd that i need to parse log files to recollect state that the server has readily available. But why complain and not coding something better, yes. Ciao. --steffen | |Ralph says i must not use signatures which spread the light! From jnc at mercury.lcs.mit.edu Fri May 12 00:07:29 2017 From: jnc at mercury.lcs.mit.edu (Noel Chiappa) Date: Thu, 11 May 2017 10:07:29 -0400 (EDT) Subject: [TUHS] The evolution of Unix facilities and architecture Message-ID: <20170511140729.2262B18C09A@mercury.lcs.mit.edu> > From: Clem Cole > it was was originally written for the for the 6th edition FS (which I > hope I have still have the sources in my files) ... > I believe Noel recovered a copy in his files recently. Well, I have _something_. It's called 'fcheck', not 'fsck', but it looks like what we're talking about - maybe it was originally named, or renamed, to be in the same series as {d,i,n}check? But it does have the upper-case error messages... :-) Anyway, here it is: http://ana-3.lcs.mit.edu/~jnc/tech/unix/s1/fcheck.c http://ana-3.lcs.mit.edu/~jnc/tech/unix/man8/fcheck.8 Interestingly, the man page for it makes reference to a 'check' command, which I didn't recall at all; here it is: http://ana-3.lcs.mit.edu/~jnc/tech/unix/s1/check.c http://ana-3.lcs.mit.edu/~jnc/tech/unix/man8/check.8 for those who are interested. > Noel has pointed out that MIT had it in the late 1970s also, probably > brought back from BTL by one of their summer students. I think most of the Unix stuff we got from Bell (e.g. the OS, which is clearly PWB1, not V6) came from someone who was in a Scout unit there in high school, of all bizarre connections! ISTR this came the same way, but maybe I'm wrong. It definitely arrived later than the OS - we'd be using icheck/dcheck for quite a while before it arrived - so maybe it was another channel? The only thing that for sure (that I recall) that didn't come this way was Emacs. Since the author had been a grad student in our group at MIT, I think you all can guess how we got that! Noel From lm at mcvoy.com Fri May 12 00:21:42 2017 From: lm at mcvoy.com (Larry McVoy) Date: Thu, 11 May 2017 07:21:42 -0700 Subject: [TUHS] The evolution of Unix facilities and architecture In-Reply-To: <20170511140729.2262B18C09A@mercury.lcs.mit.edu> References: <20170511140729.2262B18C09A@mercury.lcs.mit.edu> Message-ID: <20170511142142.GA4341@mcvoy.com> On Thu, May 11, 2017 at 10:07:29AM -0400, Noel Chiappa wrote: > http://ana-3.lcs.mit.edu/~jnc/tech/unix/s1/fcheck.c Is this style of declarations common? char *bbit, *abbit, *state, *lc, pathname[200], *pp, *name, sflag, nflag, yflag, ; unsigned dsize, fmin, fmax ; I've not seen that before, if it's fairly unique then we might be able to figure out who wrote this stuff (or did I miss that and we know already?) --lm From clemc at ccc.com Fri May 12 02:15:09 2017 From: clemc at ccc.com (Clem Cole) Date: Thu, 11 May 2017 12:15:09 -0400 Subject: [TUHS] The evolution of Unix facilities and architecture In-Reply-To: <20170511140729.2262B18C09A@mercury.lcs.mit.edu> References: <20170511140729.2262B18C09A@mercury.lcs.mit.edu> Message-ID: On Thu, May 11, 2017 at 10:07 AM, Noel Chiappa wrote: > > From: Clem Cole > > > it was was originally written for the for the 6th edition FS (which I > > hope I have still have the sources in my files) ... > > I believe Noel recovered a copy in his files recently. > > Well, I have _something_. It's called 'fcheck', not 'fsck', but it looks > like > what we're talking about - maybe it was originally named, or renamed, to > be in > the same series as {d,i,n}check? But it does have the upper-case error > messages... :-) Anyway, here it is: > > http://ana-3.lcs.mit.edu/~jnc/tech/unix/s1/fcheck.c > http://ana-3.lcs.mit.edu/~jnc/tech/unix/man8/fcheck.8 fcheck ---> aka fsick -- aka fsck -- that's it. > > > Interestingly, the man page for it makes reference to a 'check' command, > which > I didn't recall at all; here it is: > > http://ana-3.lcs.mit.edu/~jnc/tech/unix/s1/check.c > http://ana-3.lcs.mit.edu/~jnc/tech/unix/man8/check.8 > > for those who are interested. > > > > Noel has pointed out that MIT had it in the late 1970s also, probably > > brought back from BTL by one of their summer students. > > I think most of the Unix stuff we got from Bell (e.g. the OS, which is > clearly > PWB1, not V6) came from someone who was in a Scout unit there in high > school, > Jon Stienhart maybe??? He & Paul Rubinfield were in that scout group years ago and were both long time UNIX hackers, but I've forgotten where Stienhart did his undergrad. of all bizarre connections! ISTR this came the same way, but maybe I'm > wrong. > It definitely arrived later than the OS - we'd be using icheck/dcheck for > quite a while before it arrived - so maybe it was another channel? > ​ This is Ted's code and my error messages. > The only thing that for sure (that I recall) that didn't come this way was > Emacs. Since the author had been a grad student in our group at MIT, I > think > you all can guess how we got that! > > Noel > > ​Clem​ -------------- next part -------------- An HTML attachment was scrubbed... URL: From clemc at ccc.com Fri May 12 02:17:20 2017 From: clemc at ccc.com (Clem Cole) Date: Thu, 11 May 2017 12:17:20 -0400 Subject: [TUHS] The evolution of Unix facilities and architecture In-Reply-To: <20170511142142.GA4341@mcvoy.com> References: <20170511140729.2262B18C09A@mercury.lcs.mit.edu> <20170511142142.GA4341@mcvoy.com> Message-ID: Ted certainly did that a lot. (It drove me nuts. I hated it and argued a bit about it.) One of the reasons I hated C when I first learned it. Clem On Thu, May 11, 2017 at 10:21 AM, Larry McVoy wrote: > On Thu, May 11, 2017 at 10:07:29AM -0400, Noel Chiappa wrote: > > http://ana-3.lcs.mit.edu/~jnc/tech/unix/s1/fcheck.c > > Is this style of declarations common? > > char > *bbit, > *abbit, > *state, > *lc, > pathname[200], > *pp, > *name, > sflag, > nflag, > yflag, > ; > > unsigned > dsize, > fmin, > fmax > ; > > I've not seen that before, if it's fairly unique then we might be able to > figure out who wrote this stuff (or did I miss that and we know already?) > > --lm > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From imp at bsdimp.com Fri May 12 02:52:55 2017 From: imp at bsdimp.com (Warner Losh) Date: Thu, 11 May 2017 10:52:55 -0600 Subject: [TUHS] The evolution of Unix facilities and architecture In-Reply-To: References: <20170511140729.2262B18C09A@mercury.lcs.mit.edu> Message-ID: On Thu, May 11, 2017 at 10:15 AM, Clem Cole wrote: > > > > > > > On Thu, May 11, 2017 at 10:07 AM, Noel Chiappa > wrote: >> >> > From: Clem Cole >> >> > it was was originally written for the for the 6th edition FS (which >> I >> > hope I have still have the sources in my files) ... >> > I believe Noel recovered a copy in his files recently. >> >> Well, I have _something_. It's called 'fcheck', not 'fsck', but it looks >> like >> what we're talking about - maybe it was originally named, or renamed, to >> be in >> the same series as {d,i,n}check? But it does have the upper-case error >> messages... :-) Anyway, here it is: >> >> http://ana-3.lcs.mit.edu/~jnc/tech/unix/s1/fcheck.c >> http://ana-3.lcs.mit.edu/~jnc/tech/unix/man8/fcheck.8 > > > fcheck ---> aka fsick -- aka fsck -- that's it. There's a dcheck.c in the TUHS v7 sources. How's that related? Warner >> Interestingly, the man page for it makes reference to a 'check' command, >> which >> I didn't recall at all; here it is: >> >> http://ana-3.lcs.mit.edu/~jnc/tech/unix/s1/check.c >> http://ana-3.lcs.mit.edu/~jnc/tech/unix/man8/check.8 >> >> for those who are interested. >> >> >> > Noel has pointed out that MIT had it in the late 1970s also, >> probably >> > brought back from BTL by one of their summer students. >> >> I think most of the Unix stuff we got from Bell (e.g. the OS, which is >> clearly >> PWB1, not V6) came from someone who was in a Scout unit there in high >> school, > > Jon Stienhart maybe??? He & Paul Rubinfield were in that scout group years > ago and were both long time UNIX hackers, but I've forgotten where Stienhart > did his undergrad. > >> of all bizarre connections! ISTR this came the same way, but maybe I'm >> wrong. >> It definitely arrived later than the OS - we'd be using icheck/dcheck for >> quite a while before it arrived - so maybe it was another channel? > > This is Ted's code and my error messages. > >> >> The only thing that for sure (that I recall) that didn't come this way was >> Emacs. Since the author had been a grad student in our group at MIT, I >> think >> you all can guess how we got that! >> >> Noel >> > Clem > From jnc at mercury.lcs.mit.edu Fri May 12 03:08:03 2017 From: jnc at mercury.lcs.mit.edu (Noel Chiappa) Date: Thu, 11 May 2017 13:08:03 -0400 (EDT) Subject: [TUHS] The evolution of Unix facilities and architecture Message-ID: <20170511170803.1F46518C09A@mercury.lcs.mit.edu> > From: Warner Losh > There's a dcheck.c in the TUHS v7 sources. How's that related? That was one of the earlier tools - not sure how far back it goes, but it's in V6, but not V5. It consistency checks the directory tree. Another tool, 'icheck', consistency checks file blocks and the free list. Noel From michael at kjorling.se Fri May 12 03:11:00 2017 From: michael at kjorling.se (Michael =?utf-8?B?S2rDtnJsaW5n?=) Date: Thu, 11 May 2017 17:11:00 +0000 Subject: [TUHS] The evolution of Unix facilities and architecture In-Reply-To: References: <20170511140729.2262B18C09A@mercury.lcs.mit.edu> <20170511142142.GA4341@mcvoy.com> Message-ID: <20170511171100.GA9980@yeono.kjorling.se> On 11 May 2017 12:17 -0400, from clemc at ccc.com (Clem Cole): > On Thu, May 11, 2017 at 10:21 AM, Larry McVoy wrote: >> Is this style of declarations common? >> >> char >> *bbit, >> *abbit, >> *state, >> *lc, >> pathname[200], >> /.../ > > Ted certainly did that a lot. > (It drove me nuts. I hated it and argued a bit about it.) One of the > reasons I hated C when I first learned it. On the flip side, it certainly does beat `char* x, y, z[100];` or `FILE* fpsrc, fpdst;`. I wonder how many aspiring C programmers have been tripped up by constructs like those? It's perfectly reasonable _once you know about it_, but if you don't, then, well... It's even more fun if your system doesn't have memory protection. No, I'm not speaking from experience; what made you think that I did? ;-) -- Michael Kjörling • https://michael.kjorling.se • michael at kjorling.se “People who think they know everything really annoy those of us who know we don’t.” (Bjarne Stroustrup) From clemc at ccc.com Fri May 12 03:12:20 2017 From: clemc at ccc.com (Clem Cole) Date: Thu, 11 May 2017 13:12:20 -0400 Subject: [TUHS] The evolution of Unix facilities and architecture In-Reply-To: References: <20170511140729.2262B18C09A@mercury.lcs.mit.edu> Message-ID: On Thu, May 11, 2017 at 12:52 PM, Warner Losh wrote: > On Thu, May 11, 2017 at 10:15 AM, Clem Cole wrote: > > > > > > > > > > > > > > On Thu, May 11, 2017 at 10:07 AM, Noel Chiappa > > wrote: > >> > >> > From: Clem Cole > >> > >> > it was was originally written for the for the 6th edition FS > (which > >> I > >> > hope I have still have the sources in my files) ... > >> > I believe Noel recovered a copy in his files recently. > >> > >> Well, I have _something_. It's called 'fcheck', not 'fsck', but it looks > >> like > >> what we're talking about - maybe it was originally named, or renamed, to > >> be in > >> the same series as {d,i,n}check? But it does have the upper-case error > >> messages... :-) Anyway, here it is: > >> > >> http://ana-3.lcs.mit.edu/~jnc/tech/unix/s1/fcheck.c > >> http://ana-3.lcs.mit.edu/~jnc/tech/unix/man8/fcheck.8 > > > > > > fcheck ---> aka fsick -- aka fsck -- that's it. > > There's a dcheck.c in the TUHS v7 sources. How's that related? > > ​Directory CHECK - was a pass down the upper level pathname structure of the FS. In fact it was the model for fsck. Ted had me steer at it. One of the passes in pretty much pulled from that code directly. The problem was that originally there were a couple of tools to put things back together, but until Ted wrote fsck there was not one single tool that pretty much did what you wanted and got it right most of the time. Clem​ > Warner > > > >> Interestingly, the man page for it makes reference to a 'check' command, > >> which > >> I didn't recall at all; here it is: > >> > >> http://ana-3.lcs.mit.edu/~jnc/tech/unix/s1/check.c > >> http://ana-3.lcs.mit.edu/~jnc/tech/unix/man8/check.8 > >> > >> for those who are interested. > >> > >> > >> > Noel has pointed out that MIT had it in the late 1970s also, > >> probably > >> > brought back from BTL by one of their summer students. > >> > >> I think most of the Unix stuff we got from Bell (e.g. the OS, which is > >> clearly > >> PWB1, not V6) came from someone who was in a Scout unit there in high > >> school, > > > > Jon Stienhart maybe??? He & Paul Rubinfield were in that scout group > years > > ago and were both long time UNIX hackers, but I've forgotten where > Stienhart > > did his undergrad. > > > >> of all bizarre connections! ISTR this came the same way, but maybe I'm > >> wrong. > >> It definitely arrived later than the OS - we'd be using icheck/dcheck > for > >> quite a while before it arrived - so maybe it was another channel? > > > > This is Ted's code and my error messages. > > > >> > >> The only thing that for sure (that I recall) that didn't come this way > was > >> Emacs. Since the author had been a grad student in our group at MIT, I > >> think > >> you all can guess how we got that! > >> > >> Noel > >> > > Clem > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ron at ronnatalie.com Fri May 12 06:37:29 2017 From: ron at ronnatalie.com (Ron Natalie) Date: Thu, 11 May 2017 16:37:29 -0400 Subject: [TUHS] The evolution of Unix facilities and architecture In-Reply-To: References: <20170511140729.2262B18C09A@mercury.lcs.mit.edu> Message-ID: <013b01d2ca96$6901b370$3b051a50$@ronnatalie.com> I remember the pre-fsck days. It was part of my test to become an operator at the UNIX site at JHU that I could run the various manual checks. The V6 file system wasn’t exactly stable during crashes (lousy database behavior), so there was almost certainly something to clean up. The first thing we’d run was icheck. This runs down the superblock freelist and all the allocated blocks in the inodes. If there were missing blocks (not in a file or the free list), you could use icheck –s to rebuild it. Similarly, if you had duplicated allocations in the freelist or between the freelist and a single file. Anything more complicated required some clever patching (typically, we’d just mount readonly, copy the files, and then blow them away with clri). Then you’d run dcheck. As mentioned dcheck walks the directory path from the top of the disk counting inode references that it reconciles with the link count in the inode. Occasionally we’d end up with a 0-0 inode (no directory entires, but allocated…typically this is caused by people removing a file while it is still open, a regular practice of some programs for their /tmp files.). clri again blew these away. Clri wrote zeros all over the inode. This had the effect of wiping out the file, but it was dangerous if you got the i-number wrong. We replaced it with “clrm” which just cleared the allocated bit, a lot easy to reverse. If you really had a mess of a file system, you might get a piece of the directory tree broken off from a path to the root. Or you’d have an inode that icheck reported dups. ncheck would try to reconcile an inumber into an absolute path. After a while a program called fsdb came around that allowed you to poke at the various file system structures. We didn’t use it much because by the time we had it, fsck was fast on its heals. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave at horsfall.org Fri May 12 07:34:12 2017 From: dave at horsfall.org (Dave Horsfall) Date: Fri, 12 May 2017 07:34:12 +1000 (EST) Subject: [TUHS] The evolution of Unix facilities and architecture In-Reply-To: <20170511170803.1F46518C09A@mercury.lcs.mit.edu> References: <20170511170803.1F46518C09A@mercury.lcs.mit.edu> Message-ID: On Thu, 11 May 2017, Noel Chiappa wrote: > Another tool, 'icheck', consistency checks file blocks and the free > list. Named so because it checked the inode list (for which the sledge-hammer fix was "clri"). Somewhere in the AUUGN archives is a paper I wrote detailing the proper use of "[ind]check" and "clri". And then along came "fsdb", wherein you could edit the inode directly... -- Dave Horsfall DTM (VK2KFU) "Those who don't understand security will suffer." From dave at horsfall.org Fri May 12 07:44:32 2017 From: dave at horsfall.org (Dave Horsfall) Date: Fri, 12 May 2017 07:44:32 +1000 (EST) Subject: [TUHS] The evolution of Unix facilities and architecture In-Reply-To: <20170511171100.GA9980@yeono.kjorling.se> References: <20170511140729.2262B18C09A@mercury.lcs.mit.edu> <20170511142142.GA4341@mcvoy.com> <20170511171100.GA9980@yeono.kjorling.se> Message-ID: On Thu, 11 May 2017, Michael Kjörling wrote: > On the flip side, it certainly does beat `char* x, y, z[100];` or `FILE* > fpsrc, fpdst;`. I wonder how many aspiring C programmers have been > tripped up by constructs like those? It's perfectly reasonable _once you > know about it_, but if you don't, then, well... Am I the only one here who thinks that e.g. a char pointer should be "char* cp1, cp2" instead of "char *cp1, *cp2"? I.e. the fundamental type is "char*", not "char", and to this day I still write: char* cp1; char* cp2; etc, which IMHO makes it clear (which is every programmer's duty). I used to write that way in a previous life, and the boss didn't complain. -- Dave Horsfall DTM (VK2KFU) "Those who don't understand security will suffer." From ron at ronnatalie.com Fri May 12 07:49:31 2017 From: ron at ronnatalie.com (Ron Natalie) Date: Thu, 11 May 2017 17:49:31 -0400 Subject: [TUHS] C declarations. Message-ID: <015401d2caa0$79762650$6c6272f0$@ronnatalie.com> Bjarne agrees with you. He put the * (and the &) with the type name to emphasize it is part of the type. This works fine as long as you only use one declaration per statement. The problem with that is that * doesn't really bind to the type name. It binds to the variable. char* cp1, cp2; // cp1 is pointer to char, cp2 is just a char. I always found it confusing that the * is used to indicate an pointer here, where as when you want to change an lvalue to a pointer, you use &. But if we're going to gripe about the evolution of C. My biggest gripe is when they fixed structs to be real types, they didn't also do so for arrays. Arrays and their degeneration to poitners is one of the biggest annoyances in C. > Am I the only one here who thinks that e.g. a char pointer should be > "char* cp1, cp2" instead of "char *cp1, *cp2"? I.e. the fundamental type is "char*", not "char", and to this day I still write: From krewat at kilonet.net Fri May 12 08:01:36 2017 From: krewat at kilonet.net (Arthur Krewat) Date: Thu, 11 May 2017 18:01:36 -0400 Subject: [TUHS] C declarations. In-Reply-To: <015401d2caa0$79762650$6c6272f0$@ronnatalie.com> References: <015401d2caa0$79762650$6c6272f0$@ronnatalie.com> Message-ID: But wouldn't that imply you didn't need to use the * when referencing the actual value? For example, if you do this: char *p; char c; The correct way to get the character it's pointing to dereference like this: c=*p; If the type was char* (not char) Wouldn't that imply I wouldn't need the * to dereference the pointer? On 5/11/2017 5:49 PM, Ron Natalie wrote: > Bjarne agrees with you. He put the * (and the &) with the type name to emphasize it is part of the type. > This works fine as long as you only use one declaration per statement. > > The problem with that is that * doesn't really bind to the type name. It binds to the variable. > > char* cp1, cp2; // cp1 is pointer to char, cp2 is just a char. > > I always found it confusing that the * is used to indicate an pointer here, where as when you want to change an lvalue to a pointer, you use &. > > But if we're going to gripe about the evolution of C. My biggest gripe is when they fixed structs to be real types, they didn't also do so for arrays. > Arrays and their degeneration to poitners is one of the biggest annoyances in C. > >> Am I the only one here who thinks that e.g. a char pointer should be >> "char* cp1, cp2" instead of "char *cp1, *cp2"? I.e. the fundamental type is "char*", not "char", and to this day I still write: > > > From imp at bsdimp.com Fri May 12 08:06:45 2017 From: imp at bsdimp.com (Warner Losh) Date: Thu, 11 May 2017 16:06:45 -0600 Subject: [TUHS] The evolution of Unix facilities and architecture In-Reply-To: References: <20170511140729.2262B18C09A@mercury.lcs.mit.edu> <20170511142142.GA4341@mcvoy.com> <20170511171100.GA9980@yeono.kjorling.se> Message-ID: On Thu, May 11, 2017 at 3:44 PM, Dave Horsfall wrote: > On Thu, 11 May 2017, Michael Kjörling wrote: > >> On the flip side, it certainly does beat `char* x, y, z[100];` or `FILE* >> fpsrc, fpdst;`. I wonder how many aspiring C programmers have been >> tripped up by constructs like those? It's perfectly reasonable _once you >> know about it_, but if you don't, then, well... > > Am I the only one here who thinks that e.g. a char pointer should be > "char* cp1, cp2" instead of "char *cp1, *cp2"? I.e. the fundamental type > is "char*", not "char", and to this day I still write: > > char* cp1; > char* cp2; > > etc, which IMHO makes it clear (which is every programmer's duty). I used > to write that way in a previous life, and the boss didn't complain. I've encountered several people with that world view, so you aren't alone. I take a contrary view. Since C doesn't behave that way, it encourages people to think that char* cp1, cp2 is equivalent to what you wrote, which it's not. * is a modifier of char rather than char * being a fundamental type. Been burned too many times by it I guess over the years. Warner From davida at pobox.com Fri May 12 08:03:28 2017 From: davida at pobox.com (David Arnold) Date: Fri, 12 May 2017 08:03:28 +1000 Subject: [TUHS] C declarations. In-Reply-To: <015401d2caa0$79762650$6c6272f0$@ronnatalie.com> References: <015401d2caa0$79762650$6c6272f0$@ronnatalie.com> Message-ID: <68E8DC0A-D1B8-4FF0-AD26-ACDC57E308AF@pobox.com> > On 12 May 2017, at 07:49, Ron Natalie wrote: > > <…> > But if we're going to gripe about the evolution of C. My biggest gripe is when they fixed structs to be real types, they didn't also do so for arrays. > Arrays and their degeneration to poitners is one of the biggest annoyances in C. And, conversely (perversely?) one of its greatest joys. d From lm at mcvoy.com Fri May 12 08:25:47 2017 From: lm at mcvoy.com (Larry McVoy) Date: Thu, 11 May 2017 15:25:47 -0700 Subject: [TUHS] The evolution of Unix facilities and architecture In-Reply-To: <013b01d2ca96$6901b370$3b051a50$@ronnatalie.com> References: <20170511140729.2262B18C09A@mercury.lcs.mit.edu> <013b01d2ca96$6901b370$3b051a50$@ronnatalie.com> Message-ID: <20170511222547.GJ4341@mcvoy.com> This is one place where I think Linux kicked Unix's ass. And I am not really sure how they did it, I have an idea but am not positive. Unix file systems up through UFS as shipped by Sun, were all vulnerable to what I call the power out test. Untar some big tarball and power off the machine in the middle of it. Reboot. Hilarity ensues (not). You were dropped into some stand alone shell after fsck threw up its hands and it was up to you to fix it. Dozens and dozens of errors. It was almost always faster to go to backups because figuring that stuff out, file by file (which I have done more than once), gets you to the point that your run "fsck -y" and go poke at lost+found when fsck is done, realize that there is no hope, and reach for backups. Try the same thing with Linux. The file system will come back, starting with, I believe, ext2. My belief is that Linux orders writes such that while you may lose data (as in, a process created a file, the OS said it was OK, but that file will not be in the file system after a crash), but the rest of the file system will be consistent. I think it's as if you powered off the machine a few seconds earlier than you actually did, some stuff is in flight and until they can write stuff out in the proper order you may lose data on a hard reset. But it doesn't leave your file system in a mess and it's not the brute force slow way that DOS does it. I copied Ted, who had his fingers deep in that code, maybe he can correct me where I got it wrong. Details aside, I think this is a place where Linux moved the state of the art significantly forward. There are other places but this one is a big deal IMHO, maybe the biggest deal. --lm On Thu, May 11, 2017 at 04:37:29PM -0400, Ron Natalie wrote: > I remember the pre-fsck days. It was part of my test to become an operator at the UNIX site at JHU that I could run the various manual checks. > > The V6 file system wasn???t exactly stable during crashes (lousy database behavior), so there was almost certainly something to clean up. > > > > The first thing we???d run was icheck. This runs down the superblock freelist and all the allocated blocks in the inodes. If there were missing blocks (not in a file or the free list), you could use icheck ???s > > to rebuild it. Similarly, if you had duplicated allocations in the freelist or between the freelist and a single file. Anything more complicated required some clever patching (typically, we???d just mount readonly, copy the files, and then blow them away with clri). > > > > Then you???d run dcheck. As mentioned dcheck walks the directory path from the top of the disk counting inode references that it reconciles with the link count in the inode. Occasionally we???d end up with a 0-0 inode (no directory entires, but allocated???typically this is caused by people removing a file while it is still open, a regular practice of some programs for their /tmp files.). clri again blew these away. > > > > Clri wrote zeros all over the inode. This had the effect of wiping out the file, but it was dangerous if you got the i-number wrong. We replaced it with ???clrm??? which just cleared the allocated bit, a lot easy to reverse. > > > > If you really had a mess of a file system, you might get a piece of the directory tree broken off from a path to the root. Or you???d have an inode that icheck reported dups. ncheck would try to reconcile an inumber into an absolute path. > > > > After a while a program called fsdb came around that allowed you to poke at the various file system structures. We didn???t use it much because by the time we had it, fsck was fast on its heals. > -- --- Larry McVoy lm at mcvoy.com http://www.mcvoy.com/lm From ron at ronnatalie.com Fri May 12 08:30:07 2017 From: ron at ronnatalie.com (Ron Natalie) Date: Thu, 11 May 2017 18:30:07 -0400 Subject: [TUHS] The evolution of Unix facilities and architecture In-Reply-To: <20170511222547.GJ4341@mcvoy.com> References: <20170511140729.2262B18C09A@mercury.lcs.mit.edu> <013b01d2ca96$6901b370$3b051a50$@ronnatalie.com> <20170511222547.GJ4341@mcvoy.com> Message-ID: <015901d2caa6$2594b410$70be1c30$@ronnatalie.com> I got a phone call one day from one of the operators that "fsck was stuck in a loop." I thought maybe it was wedged or it was just printing out continuously. I got there and find that Fsck had repaired the root partition and rebooted the machine and then come up and decided the root needed fixing again and then rebooted... The operator didn't mention that the "fsck loop" involved a reboot every time through. The other one I got was that "fsck has been printing for an hour." After getting them to read me one of the errors I suggested they scroll back the printout and find the first message. RP01 OFFLINE. Now, go over to the disk drive and look in the glass top and tell me...is it spinning? No? Now put the run/stop switch to stop. Now put it to Run. Now reboot the machine and DON'T LEAN ON THE FRONT OF THE DRIVE AGAIN. (This one RP06 if you leaned on the lid it would spin down the disk). From lm at mcvoy.com Fri May 12 08:32:32 2017 From: lm at mcvoy.com (Larry McVoy) Date: Thu, 11 May 2017 15:32:32 -0700 Subject: [TUHS] C declarations. In-Reply-To: <68E8DC0A-D1B8-4FF0-AD26-ACDC57E308AF@pobox.com> References: <015401d2caa0$79762650$6c6272f0$@ronnatalie.com> <68E8DC0A-D1B8-4FF0-AD26-ACDC57E308AF@pobox.com> Message-ID: <20170511223232.GM4341@mcvoy.com> On Fri, May 12, 2017 at 08:03:28AM +1000, David Arnold wrote: > > On 12 May 2017, at 07:49, Ron Natalie wrote: > > > > > > But if we're going to gripe about the evolution of C. My biggest gripe is when they fixed structs to be real types, they didn't also do so for arrays. > > Arrays and their degeneration to poitners is one of the biggest annoyances in C. > > And, conversely (perversely?) one of its greatest joys. I dunno if it is one of its greatest joys but pointers in C have always made sense to me. I'm curious as to what is busted about arrays in C? To me they just seemed like a way to define how to look at a wad of memory and they seem to work for me. About the only thing I don't like about them is that there is no late binding as to the size, Ada has late binding and I thought it could be useful (I only know because Rob Netzer and I wrote an Ada compiler for CS736 at UW-Madison that did a lot of Ada but exceptions and late binding we did not do). --lm From ron at ronnatalie.com Fri May 12 08:41:47 2017 From: ron at ronnatalie.com (Ron Natalie) Date: Thu, 11 May 2017 18:41:47 -0400 Subject: [TUHS] C declarations. In-Reply-To: <20170511223232.GM4341@mcvoy.com> References: <015401d2caa0$79762650$6c6272f0$@ronnatalie.com> <68E8DC0A-D1B8-4FF0-AD26-ACDC57E308AF@pobox.com> <20170511223232.GM4341@mcvoy.com> Message-ID: <015b01d2caa7$c658c020$530a4060$@ronnatalie.com> > I'm curious as to what is busted about arrays in C? To me they just seemed like a way to define how to look at a wad of memory and they seem to work for me. The problem is they don't work like every other type. Somewhere around the phototypesetter release or V7, they fixed structs to be assignable/passable to functions. They didn't do that for arrays. They retained their half-assed quasipointer status. It would have been easy to bite the bullet back in those days before people codified the inane behavior. For example: struct foo { int a; int b; }; struct foo x; x.a = 1; x.b = 1; struct foo y; y = x; // works fine. struct foo myfunc(struct foo arg) { arg.a = 5; return 5; } y = myfunc(x); // does not modify x Now look at arrays: char x[4] = { 1, 2, 3, 4}; char y[4]; y = x ; // illegal void myfunc(char arg[4]) { arg[0] = 1000; } myfunc(x); // CHANGES X! If had my way, y = x and passing and returning arrays by value would work just like every other C type. From dave at horsfall.org Fri May 12 09:44:40 2017 From: dave at horsfall.org (Dave Horsfall) Date: Fri, 12 May 2017 09:44:40 +1000 (EST) Subject: [TUHS] C declarations. In-Reply-To: References: <015401d2caa0$79762650$6c6272f0$@ronnatalie.com> Message-ID: On Thu, 11 May 2017, Arthur Krewat wrote: > But wouldn't that imply you didn't need to use the * when referencing > the actual value? > > For example, if you do this: > > char *p; > char c; > > The correct way to get the character it's pointing to dereference like > this: > > c=*p; > > If the type was char* (not char) > > Wouldn't that imply I wouldn't need the * to dereference the pointer? No, because you'd be assigning the contents i.e. the address (or however references are implemented on a given architecture) to a character; whether the fundamental type is "char*" or ("mostly char but it's really a pointer") is the same. Then again, I learned ALGOLW (and PASCAL) before I learned C, and it has a "REFERENCE" type... -- Dave Horsfall DTM (VK2KFU) "Those who don't understand security will suffer." From dave at horsfall.org Fri May 12 09:47:01 2017 From: dave at horsfall.org (Dave Horsfall) Date: Fri, 12 May 2017 09:47:01 +1000 (EST) Subject: [TUHS] The evolution of Unix facilities and architecture In-Reply-To: <20170511222547.GJ4341@mcvoy.com> References: <20170511140729.2262B18C09A@mercury.lcs.mit.edu> <013b01d2ca96$6901b370$3b051a50$@ronnatalie.com> <20170511222547.GJ4341@mcvoy.com> Message-ID: On Thu, 11 May 2017, Larry McVoy wrote: [...] > Try the same thing with Linux. The file system will come back, starting > with, I believe, ext2. That's a journalled FS, isn't it? In which case the transactions get replayed. > My belief is that Linux orders writes such that while you may lose data > (as in, a process created a file, the OS said it was OK, but that file > will not be in the file system after a crash), but the rest of the file > system will be consistent. I think it's as if you powered off the > machine a few seconds earlier than you actually did, some stuff is in > flight and until they can write stuff out in the proper order you may > lose data on a hard reset. And FreeBSD (at least) has been doing ordered writes for quite some time. -- Dave Horsfall DTM (VK2KFU) "Those who don't understand security will suffer." From ron at ronnatalie.com Fri May 12 09:48:27 2017 From: ron at ronnatalie.com (Ron Natalie) Date: Thu, 11 May 2017 19:48:27 -0400 Subject: [TUHS] The evolution of Unix facilities and architecture In-Reply-To: References: <20170511140729.2262B18C09A@mercury.lcs.mit.edu> <013b01d2ca96$6901b370$3b051a50$@ronnatalie.com> <20170511222547.GJ4341@mcvoy.com> Message-ID: <017001d2cab1$16bcf4b0$4436de10$@ronnatalie.com> Ordered writes go back to the original BSD fast file system, no? I seem to recall that when we switched from our V6/V7 disks, the filesystem got a lot more stable in crashes. -----Original Message----- From: TUHS [mailto:tuhs-bounces at minnie.tuhs.org] On Behalf Of Dave Horsfall Sent: Thursday, May 11, 2017 7:47 PM To: The Eunuchs Hysterical Society Subject: Re: [TUHS] The evolution of Unix facilities and architecture On Thu, 11 May 2017, Larry McVoy wrote: [...] > Try the same thing with Linux. The file system will come back, > starting with, I believe, ext2. That's a journalled FS, isn't it? In which case the transactions get replayed. > My belief is that Linux orders writes such that while you may lose > data (as in, a process created a file, the OS said it was OK, but that > file will not be in the file system after a crash), but the rest of > the file system will be consistent. I think it's as if you powered > off the machine a few seconds earlier than you actually did, some > stuff is in flight and until they can write stuff out in the proper > order you may lose data on a hard reset. And FreeBSD (at least) has been doing ordered writes for quite some time. -- Dave Horsfall DTM (VK2KFU) "Those who don't understand security will suffer." From bakul at bitblocks.com Fri May 12 10:15:20 2017 From: bakul at bitblocks.com (Bakul Shah) Date: Thu, 11 May 2017 17:15:20 -0700 Subject: [TUHS] C declarations. In-Reply-To: Your message of "Thu, 11 May 2017 15:32:32 PDT." <20170511223232.GM4341@mcvoy.com> References: <015401d2caa0$79762650$6c6272f0$@ronnatalie.com> <68E8DC0A-D1B8-4FF0-AD26-ACDC57E308AF@pobox.com> <20170511223232.GM4341@mcvoy.com> Message-ID: <20170512001520.38000124AEA4@mail.bitblocks.com> On Thu, 11 May 2017 15:32:32 PDT Larry McVoy wrote: > > I dunno if it is one of its greatest joys but pointers in C have always > made sense to me. > > I'm curious as to what is busted about arrays in C? To me they just > seemed like a way to define how to look at a wad of memory and they > seem to work for me. About the only thing I don't like about them is > that there is no late binding as to the size, Ada has late binding and > I thought it could be useful (I only know because Rob Netzer and I > wrote an Ada compiler for CS736 at UW-Madison that did a lot of Ada > but exceptions and late binding we did not do). Coming from a Pascal background I really liked the terseness of C. But theere were three things that bothered me. a) K&R style argument declarations. This got fixed in ANSI C. C declarations were messy but that did not bother me much. b) No nested procedures. Gnu C had added them but the implementation was a bit screwy and most people didn't care in any case so there was no hope of this getting fixed. Partly because most people used it as a portable assembly language! c) arrays were not first class objects. Given "T v[N];" v[i] is of type T and i+v or v+i is of type T* -- this is perfectly well defined and fine. What is not fine is that an array is a second class object. Thus you can not do for example, int v[5]; struct foo { int w[5]; int x; } y; ... y.w = v; You can not pass a whole array to a function (or return one) without enclosing it in a struct. You can not take an address of an array, only an element of it. There is no way to declare or pass a subarray. A function that operates on a arbitray sized multi dim. array can be written but can get messy. One big impact of current array behavior is that the onus to do boundary condition checks is on the programmer and can't be done by the compiler. Just as Pascal got "conformant arrays" in its 1990 standard, arrays could've been made first class (but without "ref" parameters code can look messy). For instance, consider the folloowing: double a[5,20], b[20,10], c[5,10]; int err = mat_multiply(c, a, b); int mat_mult(ref double c[int cx, cy], a[int ax, ay], b[int bx, by]) { if (cx != ax || cy != by || ay != bx) return -1; ... return 0; } This can be implemented without much trouble and allows full boundary condition checking. Currently the user will have to manually pass {a,b,c}{x,y} parameters and he would have to either create auxiliary vectors to point to each row or have the function to do all the index arithmetic explicitly (and the compiler can't check if your code has no boundary condition bugs). A slightly more difficult situation arises when you want to pass sub arrays. For instance, mat_mutl(c[5,0:9], a[5,10:19], b[0:9,0:5]) Here you will have to somehow pass stride and offset (or use iliffe vectors or some such). From lm at mcvoy.com Fri May 12 10:16:26 2017 From: lm at mcvoy.com (Larry McVoy) Date: Thu, 11 May 2017 17:16:26 -0700 Subject: [TUHS] The evolution of Unix facilities and architecture In-Reply-To: References: <20170511140729.2262B18C09A@mercury.lcs.mit.edu> <013b01d2ca96$6901b370$3b051a50$@ronnatalie.com> <20170511222547.GJ4341@mcvoy.com> Message-ID: <20170512001626.GQ4341@mcvoy.com> On Fri, May 12, 2017 at 09:47:01AM +1000, Dave Horsfall wrote: > On Thu, 11 May 2017, Larry McVoy wrote: > > [...] > > > Try the same thing with Linux. The file system will come back, starting > > with, I believe, ext2. > > That's a journalled FS, isn't it? In which case the transactions get > replayed. My memory is ext2 is not journaled, I think that happened in ext3. Or maybe it was an option on ext2? Either way, I think ext2 did the right thing without the journal. From lm at mcvoy.com Fri May 12 10:21:12 2017 From: lm at mcvoy.com (Larry McVoy) Date: Thu, 11 May 2017 17:21:12 -0700 Subject: [TUHS] The evolution of Unix facilities and architecture In-Reply-To: <017001d2cab1$16bcf4b0$4436de10$@ronnatalie.com> References: <20170511140729.2262B18C09A@mercury.lcs.mit.edu> <013b01d2ca96$6901b370$3b051a50$@ronnatalie.com> <20170511222547.GJ4341@mcvoy.com> <017001d2cab1$16bcf4b0$4436de10$@ronnatalie.com> Message-ID: <20170512002112.GR4341@mcvoy.com> Yeah, I get ordered writes, I taught a CS course at Stanford and I made my students learn all about them. I'm a UFS guy, so far as I know I'm the last guy to push UFS/FFS forward (which is sort of sad). The Linux stuff is better. It just is. And we should all respect that, I know we sit around and love on ancient Unix, and believe me, I love that stuff it changed the world, but we should respect people who have moved it past what Unix did. And I think Linux moved the file system past what Unix did. --lm On Thu, May 11, 2017 at 07:48:27PM -0400, Ron Natalie wrote: > Ordered writes go back to the original BSD fast file system, no? I seem > to recall that when we switched from our V6/V7 disks, > the filesystem got a lot more stable in crashes. > > -----Original Message----- > From: TUHS [mailto:tuhs-bounces at minnie.tuhs.org] On Behalf Of Dave Horsfall > Sent: Thursday, May 11, 2017 7:47 PM > To: The Eunuchs Hysterical Society > Subject: Re: [TUHS] The evolution of Unix facilities and architecture > > On Thu, 11 May 2017, Larry McVoy wrote: > > [...] > > > Try the same thing with Linux. The file system will come back, > > starting with, I believe, ext2. > > That's a journalled FS, isn't it? In which case the transactions get > replayed. > > > My belief is that Linux orders writes such that while you may lose > > data (as in, a process created a file, the OS said it was OK, but that > > file will not be in the file system after a crash), but the rest of > > the file system will be consistent. I think it's as if you powered > > off the machine a few seconds earlier than you actually did, some > > stuff is in flight and until they can write stuff out in the proper > > order you may lose data on a hard reset. > > And FreeBSD (at least) has been doing ordered writes for quite some time. > > -- > Dave Horsfall DTM (VK2KFU) "Those who don't understand security will > suffer." -- --- Larry McVoy lm at mcvoy.com http://www.mcvoy.com/lm From toby at telegraphics.com.au Fri May 12 11:05:55 2017 From: toby at telegraphics.com.au (Toby Thain) Date: Thu, 11 May 2017 21:05:55 -0400 Subject: [TUHS] The evolution of Unix facilities and architecture In-Reply-To: References: <20170511140729.2262B18C09A@mercury.lcs.mit.edu> <013b01d2ca96$6901b370$3b051a50$@ronnatalie.com> <20170511222547.GJ4341@mcvoy.com> Message-ID: <1873b923-6f5e-83a2-753d-6d7a2f8b580a@telegraphics.com.au> On 2017-05-11 7:47 PM, Dave Horsfall wrote: > On Thu, 11 May 2017, Larry McVoy wrote: > > [...] > >> Try the same thing with Linux. The file system will come back, starting >> with, I believe, ext2. > > That's a journalled FS, isn't it? In which case the transactions get > replayed. No, I think ext3fs was the first version that was journaled. So was reiserfs. With pull-plug tests I could get ext3fs to toss cookies but not reiserfs. Now of course the state of the art is copy-on-write, like ZFS. --Toby > >> My belief is that Linux orders writes such that while you may lose data >> (as in, a process created a file, the OS said it was OK, but that file >> will not be in the file system after a crash), but the rest of the file >> system will be consistent. I think it's as if you powered off the >> machine a few seconds earlier than you actually did, some stuff is in >> flight and until they can write stuff out in the proper order you may >> lose data on a hard reset. > > And FreeBSD (at least) has been doing ordered writes for quite some time. > From wes.parish at paradise.net.nz Fri May 12 11:41:51 2017 From: wes.parish at paradise.net.nz (Wesley Parish) Date: Fri, 12 May 2017 13:41:51 +1200 (NZST) Subject: [TUHS] The evolution of Unix facilities and architecture In-Reply-To: <20170512001626.GQ4341@mcvoy.com> References: <20170511140729.2262B18C09A@mercury.lcs.mit.edu> <013b01d2ca96$6901b370$3b051a50$@ronnatalie.com> <20170511222547.GJ4341@mcvoy.com> <20170512001626.GQ4341@mcvoy.com> Message-ID: <1494553311.591512df1d8a4@www.paradise.net.nz> Consistent with what I remember of running fsck on Slackware in the 90s after unscheduled shutdowns. I longed for the time I'd get back when ext3 was incorporated into Linux and ext2 was relegated to legacy. As far as I can remember ext2 was never journaled. I remembered getting quite excited reading about the log-structured file system in the O'Reilly 4.4BSD-Lite CD. That made so much sense to me. Quoting Larry McVoy : > On Fri, May 12, 2017 at 09:47:01AM +1000, Dave Horsfall wrote: > > On Thu, 11 May 2017, Larry McVoy wrote: > > > > [...] > > > > > Try the same thing with Linux. The file system will come back, > starting > > > with, I believe, ext2. > > > > That's a journalled FS, isn't it? In which case the transactions get > > replayed. > > My memory is ext2 is not journaled, I think that happened in ext3. Or > maybe it was an option on ext2? Either way, I think ext2 did the right > thing without the journal. > "I have supposed that he who buys a Method means to learn it." - Ferdinand Sor, Method for Guitar "A verbal contract isn't worth the paper it's written on." -- Samuel Goldwyn From theopav at gmail.com Fri May 12 12:41:51 2017 From: theopav at gmail.com (Theo Pavlidis) Date: Thu, 11 May 2017 22:41:51 -0400 Subject: [TUHS] C declarations. In-Reply-To: <20170512001520.38000124AEA4@mail.bitblocks.com> References: <015401d2caa0$79762650$6c6272f0$@ronnatalie.com> <68E8DC0A-D1B8-4FF0-AD26-ACDC57E308AF@pobox.com> <20170511223232.GM4341@mcvoy.com> <20170512001520.38000124AEA4@mail.bitblocks.com> Message-ID: I tend to stay away from using arrays as arguments of procedures, too many surprises. Instead I use structures. For example: typedef struct { ......... xteam NoSo[MAXB], EsWe[MAXB]; } xboard; ............ xboard TB; ............ ScoreBoard(..., &TB, ...); Of course, in my old age I only do simple game programming ​ Theo Pavlidis ( ​Member of 1127: 1980-86) ​ On Thu, May 11, 2017 at 8:15 PM, Bakul Shah wrote: > On Thu, 11 May 2017 15:32:32 PDT Larry McVoy wrote: > > > > I dunno if it is one of its greatest joys but pointers in C have always > > made sense to me. > > > > I'm curious as to what is busted about arrays in C? To me they just > > seemed like a way to define how to look at a wad of memory and they > > seem to work for me. About the only thing I don't like about them is > > that there is no late binding as to the size, Ada has late binding and > > I thought it could be useful (I only know because Rob Netzer and I > > wrote an Ada compiler for CS736 at UW-Madison that did a lot of Ada > > but exceptions and late binding we did not do). > > Coming from a Pascal background I really liked the terseness > of C. But theere were three things that bothered me. > > a) K&R style argument declarations. This got fixed in ANSI C. > C declarations were messy but that did not bother me much. > > b) No nested procedures. Gnu C had added them but the > implementation was a bit screwy and most people didn't care > in any case so there was no hope of this getting fixed. > Partly because most people used it as a portable assembly > language! > > c) arrays were not first class objects. Given "T v[N];" v[i] > is of type T and i+v or v+i is of type T* -- this is > perfectly well defined and fine. What is not fine is that > an array is a second class object. Thus you can not do > for example, > > int v[5]; > struct foo { > int w[5]; > int x; > } y; > ... > y.w = v; > > You can not pass a whole array to a function (or return > one) without enclosing it in a struct. You can not take an > address of an array, only an element of it. There is no way > to declare or pass a subarray. A function that operates on a > arbitray sized multi dim. array can be written but can get > messy. > > One big impact of current array behavior is that the onus > to do boundary condition checks is on the programmer and > can't be done by the compiler. > > Just as Pascal got "conformant arrays" in its 1990 standard, > arrays could've been made first class (but without "ref" > parameters code can look messy). For instance, consider the > folloowing: > > double a[5,20], b[20,10], c[5,10]; > int err = mat_multiply(c, a, b); > > int mat_mult(ref double c[int cx, cy], a[int ax, ay], b[int bx, by]) { > if (cx != ax || cy != by || ay != bx) return -1; > ... > return 0; > } > > This can be implemented without much trouble and allows > full boundary condition checking. Currently the user will > have to manually pass {a,b,c}{x,y} parameters and he would > have to either create auxiliary vectors to point to each row > or have the function to do all the index arithmetic explicitly > (and the compiler can't check if your code has no boundary > condition bugs). > > A slightly more difficult situation arises when you want to > pass sub arrays. For instance, > > mat_mutl(c[5,0:9], a[5,10:19], b[0:9,0:5]) > > Here you will have to somehow pass stride and offset (or use > iliffe vectors or some such). > -------------- next part -------------- An HTML attachment was scrubbed... URL: From imp at bsdimp.com Fri May 12 12:42:54 2017 From: imp at bsdimp.com (Warner Losh) Date: Thu, 11 May 2017 20:42:54 -0600 Subject: [TUHS] The evolution of Unix facilities and architecture In-Reply-To: <20170512002112.GR4341@mcvoy.com> References: <20170511140729.2262B18C09A@mercury.lcs.mit.edu> <013b01d2ca96$6901b370$3b051a50$@ronnatalie.com> <20170511222547.GJ4341@mcvoy.com> <017001d2cab1$16bcf4b0$4436de10$@ronnatalie.com> <20170512002112.GR4341@mcvoy.com> Message-ID: On Thu, May 11, 2017 at 6:21 PM, Larry McVoy wrote: > Yeah, I get ordered writes, I taught a CS course at Stanford and I made > my students learn all about them. I'm a UFS guy, so far as I know I'm > the last guy to push UFS/FFS forward (which is sort of sad). > > The Linux stuff is better. It just is. And we should all respect that, > I know we sit around and love on ancient Unix, and believe me, I love > that stuff it changed the world, but we should respect people who have > moved it past what Unix did. And I think Linux moved the file system > past what Unix did. The Linux stuff was incrementally better in that it didn't fry the filesystem, but instead would flush large amounts of changes if there was no synchronous write before the power failed. This was progress. However, it got its speed by doing async writes even of the metadata, and that lead to the famous story where Linus was able to recover an entire kernel he'd just accidentally rm -rf'd by hitting reset on his machine before the syncer ran. Better than a scrambled filesystem? Sure. But consistent and robust? Not so much. Plus, driver bugs could still wreck havoc. Soon after things like soft-updates came along in the BSD world which solved the scrambled filesystem problems (for the most part), then SU Journalling to make it robust. On the Linux side, ext2fs begat ext3 and then ext4fs came along and added journalling. ZFS came into being. As did btrfs and many others. Some on Linux, some on Solaris, some on BSD. The competition between them all has helped to make them all better. I've been running a v7 derivative (Venix) on my Rainbow lately. There's issues with the motherboard that causes many insta-panics (it turns off the interrupts due to something bad on the mobo --- a new mobo doesn't do that). I've yet to lose anything, but the performance isn't that great. So it's not an 'all the time' sort of thing. Warner > --lm > > On Thu, May 11, 2017 at 07:48:27PM -0400, Ron Natalie wrote: >> Ordered writes go back to the original BSD fast file system, no? I seem >> to recall that when we switched from our V6/V7 disks, >> the filesystem got a lot more stable in crashes. >> >> -----Original Message----- >> From: TUHS [mailto:tuhs-bounces at minnie.tuhs.org] On Behalf Of Dave Horsfall >> Sent: Thursday, May 11, 2017 7:47 PM >> To: The Eunuchs Hysterical Society >> Subject: Re: [TUHS] The evolution of Unix facilities and architecture >> >> On Thu, 11 May 2017, Larry McVoy wrote: >> >> [...] >> >> > Try the same thing with Linux. The file system will come back, >> > starting with, I believe, ext2. >> >> That's a journalled FS, isn't it? In which case the transactions get >> replayed. >> >> > My belief is that Linux orders writes such that while you may lose >> > data (as in, a process created a file, the OS said it was OK, but that >> > file will not be in the file system after a crash), but the rest of >> > the file system will be consistent. I think it's as if you powered >> > off the machine a few seconds earlier than you actually did, some >> > stuff is in flight and until they can write stuff out in the proper >> > order you may lose data on a hard reset. >> >> And FreeBSD (at least) has been doing ordered writes for quite some time. >> >> -- >> Dave Horsfall DTM (VK2KFU) "Those who don't understand security will >> suffer." > > -- > --- > Larry McVoy lm at mcvoy.com http://www.mcvoy.com/lm From hellwig.geisse at mni.thm.de Fri May 12 16:24:32 2017 From: hellwig.geisse at mni.thm.de (Hellwig Geisse) Date: Fri, 12 May 2017 08:24:32 +0200 Subject: [TUHS] The evolution of Unix facilities and architecture In-Reply-To: References: <20170511140729.2262B18C09A@mercury.lcs.mit.edu> <20170511142142.GA4341@mcvoy.com> <20170511171100.GA9980@yeono.kjorling.se> Message-ID: <1494570272.2218.180.camel@mni.thm.de> On Fr, 2017-05-12 at 07:44 +1000, Dave Horsfall wrote: > > Am I the only one here who thinks that e.g. a char pointer should be  > "char* cp1, cp2" instead of "char *cp1, *cp2"?  I.e. the fundamental > type is "char*", not "char", and to this day I still write: > >     char* cp1; >     char* cp2; > > etc, which IMHO makes it clear (which is every programmer's duty). > I used  to write that way in a previous life, and the boss didn't > complain. This view does not work well with more complicated declarations like "void (*p)(int)". What is the "fundamental type" here? One could argue that the real culprit is the list construction, which does not mix well with C declarations. Hellwig From harald at skogtun.org Fri May 12 18:15:49 2017 From: harald at skogtun.org (Harald Arnesen) Date: Fri, 12 May 2017 10:15:49 +0200 Subject: [TUHS] The evolution of Unix facilities and architecture In-Reply-To: References: <20170511140729.2262B18C09A@mercury.lcs.mit.edu> <013b01d2ca96$6901b370$3b051a50$@ronnatalie.com> <20170511222547.GJ4341@mcvoy.com> Message-ID: <59156F35.2040508@skogtun.org> Dave Horsfall [2017-05-12 01:47]: > On Thu, 11 May 2017, Larry McVoy wrote: >> Try the same thing with Linux. The file system will come back, starting >> with, I believe, ext2. > > That's a journalled FS, isn't it? In which case the transactions get > replayed. No, journalling started with ext3. However, ext2 was (and is) a very solid fs, which survived many power-outs and crashes when I used it back in the 90s. -- Hilsen Harald From michael at kjorling.se Fri May 12 18:17:13 2017 From: michael at kjorling.se (Michael =?utf-8?B?S2rDtnJsaW5n?=) Date: Fri, 12 May 2017 08:17:13 +0000 Subject: [TUHS] The evolution of Unix facilities and architecture In-Reply-To: <1873b923-6f5e-83a2-753d-6d7a2f8b580a@telegraphics.com.au> References: <20170511140729.2262B18C09A@mercury.lcs.mit.edu> <013b01d2ca96$6901b370$3b051a50$@ronnatalie.com> <20170511222547.GJ4341@mcvoy.com> <1873b923-6f5e-83a2-753d-6d7a2f8b580a@telegraphics.com.au> Message-ID: <20170512081713.GD7265@yeono.kjorling.se> On 11 May 2017 21:05 -0400, from toby at telegraphics.com.au (Toby Thain): >>> ext2 >> >> That's a journalled FS, isn't it? In which case the transactions get >> replayed. > > No, I think ext3fs was the first version that was journaled. Correct. ext2 doesn't have journaling, but if you do `mkfs.ext2 -j` on Linux it creates a _journaled ext2_ A.K.A. _ext3_ file system. With the resulting selling point of ext3 being mainly _much_ shorter fsck times. > So was reiserfs. With pull-plug tests I could get ext3fs to toss > cookies but not reiserfs. ReiserFS seems to be one of those where peoples' experiences really differ. I've had massive crashes involving reiserfs myself, but don't think I have ever actually lost any significant amounts of data to ext2/3. Maybe it was just bad luck, but I have been bit sufficiently badly by it to relegate it to the scrap heap of history. At least at the time lack of good recovery tools didn't help (but then again if ZFS breaks sufficiently that the pool doesn't import, you're pretty hosed, too). Backups, backups. These days, for me, it's pretty much all ZFS where I have any say in it, and in addition even my home desktop is on a UPS (with automated, controlled shutdown when approaching battery depletion), so unplanned shutdowns are far less likely to happen. About the only way that can realistically happen is either a kernel crash, or the UPS misreporting (overestimating) the remaining battery time, resulting in a hard shutdown before or during the shutdown process. Neither is impossible, of course. -- Michael Kjörling • https://michael.kjorling.se • michael at kjorling.se “People who think they know everything really annoy those of us who know we don’t.” (Bjarne Stroustrup) From tfb at tfeb.org Fri May 12 23:56:59 2017 From: tfb at tfeb.org (Tim Bradshaw) Date: Fri, 12 May 2017 14:56:59 +0100 Subject: [TUHS] The evolution of Unix facilities and architecture In-Reply-To: <20170512081713.GD7265@yeono.kjorling.se> References: <20170511140729.2262B18C09A@mercury.lcs.mit.edu> <013b01d2ca96$6901b370$3b051a50$@ronnatalie.com> <20170511222547.GJ4341@mcvoy.com> <1873b923-6f5e-83a2-753d-6d7a2f8b580a@telegraphics.com.au> <20170512081713.GD7265@yeono.kjorling.se> Message-ID: <0121AE71-2930-40AB-ABFD-552E207AF979@tfeb.org> On 12 May 2017, at 09:17, Michael Kjörling wrote: > > These days, for me, it's pretty much all ZFS One of ZFS's particularly lovely features was that there was no offline filesystem checker at all. So if you have a filesystem (pool, whatever) which you think something bad might have happened to, you check it *by mounting it*, where the checker runs *in the kernel, so any serious error in the code means a panic, if you're lucky and something worse if you're not. When I found out about this I thought seriously of shorting Sun's stock (if I knew how to do that). I would have made money. As it was we stuck with logged UFS which, by 2007 or so was seriously bulletproof. --tim From michael at kjorling.se Sat May 13 00:22:46 2017 From: michael at kjorling.se (Michael =?utf-8?B?S2rDtnJsaW5n?=) Date: Fri, 12 May 2017 14:22:46 +0000 Subject: [TUHS] The evolution of Unix facilities and architecture In-Reply-To: <0121AE71-2930-40AB-ABFD-552E207AF979@tfeb.org> References: <20170511140729.2262B18C09A@mercury.lcs.mit.edu> <013b01d2ca96$6901b370$3b051a50$@ronnatalie.com> <20170511222547.GJ4341@mcvoy.com> <1873b923-6f5e-83a2-753d-6d7a2f8b580a@telegraphics.com.au> <20170512081713.GD7265@yeono.kjorling.se> <0121AE71-2930-40AB-ABFD-552E207AF979@tfeb.org> Message-ID: <20170512142246.GH7265@yeono.kjorling.se> On 12 May 2017 14:56 +0100, from tfb at tfeb.org (Tim Bradshaw): > So if you have a filesystem (pool, whatever) which you think > something bad might have happened to, you check it *by mounting it*, > where the checker runs *in the kernel, Easy peasy! No need to remember obscure fsck parameters; zpool import is all you need. Irony aside, I didn't say it was perfect. -- Michael Kjörling • https://michael.kjorling.se • michael at kjorling.se “People who think they know everything really annoy those of us who know we don’t.” (Bjarne Stroustrup) From lm at mcvoy.com Sat May 13 00:30:47 2017 From: lm at mcvoy.com (Larry McVoy) Date: Fri, 12 May 2017 07:30:47 -0700 Subject: [TUHS] The evolution of Unix facilities and architecture In-Reply-To: <0121AE71-2930-40AB-ABFD-552E207AF979@tfeb.org> References: <20170511140729.2262B18C09A@mercury.lcs.mit.edu> <013b01d2ca96$6901b370$3b051a50$@ronnatalie.com> <20170511222547.GJ4341@mcvoy.com> <1873b923-6f5e-83a2-753d-6d7a2f8b580a@telegraphics.com.au> <20170512081713.GD7265@yeono.kjorling.se> <0121AE71-2930-40AB-ABFD-552E207AF979@tfeb.org> Message-ID: <20170512143047.GV4341@mcvoy.com> On Fri, May 12, 2017 at 02:56:59PM +0100, Tim Bradshaw wrote: > When I found out about this I thought seriously of shorting Sun's > stock (if I knew how to do that). I would have made money. As it was > we stuck with logged UFS which, by 2007 or so was seriously bulletproof. Wait, someone added logging to UFS? Is there a writeup of that anywhere? That would stomp all over my claim that nobody has hacked on UFS since I did (which would be fine with me, I liked UFS, be cool if someone moved it forward). (pause while I google) Yep, someone did. I'd like to know who. I found this: http://www.oracle.com/technetwork/systems/linux/fs-performance-149840.pdf Can anyone confirm those results? That would be the first I've heard of Solaris being faster than Linux. If that's true has Linux tried to implement the same sort of logging? --lm P.S. I realize this isn't ancient Unix so I could move this to the linux-kernel mailing list. Though maybe it is appropriate, it's tech from the 1990's - is that ancient enough? From richard at inf.ed.ac.uk Sat May 13 00:04:45 2017 From: richard at inf.ed.ac.uk (Richard Tobin) Date: Fri, 12 May 2017 15:04:45 +0100 (BST) Subject: [TUHS] C declarations. In-Reply-To: Ron Natalie's message of Thu, 11 May 2017 17:49:31 -0400 Message-ID: <20170512140445.EA5DD126E33E@macaroni.inf.ed.ac.uk> > The problem with that is that * doesn't really bind to the type name. > It binds to the variable. > > char* cp1, cp2; // cp1 is pointer to char, cp2 is just a char. > > I always found it confusing that the * is used to indicate an pointer > here, where as when you want to change an lvalue to a pointer, you use > &. The way to read it is that you are declaring *cp1 as a char. -- Richard -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. From tfb at tfeb.org Sat May 13 01:11:23 2017 From: tfb at tfeb.org (Tim Bradshaw) Date: Fri, 12 May 2017 16:11:23 +0100 Subject: [TUHS] The evolution of Unix facilities and architecture In-Reply-To: <20170512143047.GV4341@mcvoy.com> References: <20170511140729.2262B18C09A@mercury.lcs.mit.edu> <013b01d2ca96$6901b370$3b051a50$@ronnatalie.com> <20170511222547.GJ4341@mcvoy.com> <1873b923-6f5e-83a2-753d-6d7a2f8b580a@telegraphics.com.au> <20170512081713.GD7265@yeono.kjorling.se> <0121AE71-2930-40AB-ABFD-552E207AF979@tfeb.org> <20170512143047.GV4341@mcvoy.com> Message-ID: <276AEA9A-1584-4D52-AEA1-18725991D93E@tfeb.org> On 12 May 2017, at 15:30, Larry McVoy wrote: > > Wait, someone added logging to UFS? Oh, yes. I forget when it came in (Solaris 2.5?). It's been the default (ie you need to turn it off in vfstab rather than turn it on) for some time, perhaps since Solaris 9? > > Yep, someone did. I'd like to know who. I found this: > > http://www.oracle.com/technetwork/systems/linux/fs-performance-149840.pdf > > Can anyone confirm those results? I can't confirm them, but I can confirm that a lot of rubbish has been talked about filesystem performance by people with various agendas. I had an argument (well over 10 years ago now) with someone who claimed that ext2 (I guess, might have been ext3 by then) was just way faster than UFS for various operations (lots of file creattion/deletion I think). It was ... if you left logging off. If you turned logging on, not so much. --tim From jnc at mercury.lcs.mit.edu Sat May 13 01:12:56 2017 From: jnc at mercury.lcs.mit.edu (Noel Chiappa) Date: Fri, 12 May 2017 11:12:56 -0400 (EDT) Subject: [TUHS] The evolution of Unix facilities and architecture Message-ID: <20170512151256.B27CE18C09A@mercury.lcs.mit.edu> > From: "Ron Natalie" > Ordered writes go back to the original BSD fast file system, no? I seem > to recall that when we switched from our V6/V7 disks, the filesystem got > a lot more stable in crashes. I had a vague memory of reading about that, so I looked in the canonical FFS paper (McKusick et al, "A Fast File System for UNIX" [1984)]) but found no mention of it. I did find a paper about 'fsck' (McKusick, Kowalski, "Fsck: The UNIX File System Check Program") which talks (in Section 2.5. "Updates to the file system") about how "problem[s] with asynchronous inode updates can be avoided by doing all inode deallocations synchronously", but it's not clear if they're talking about something that was actually done, or just saying (hypothetically) that that's how one would fix it. Is is possible that the changes to the file system (e.g. the way free blocks were kept) made it more crash-proof? Noel From clemc at ccc.com Sat May 13 01:17:02 2017 From: clemc at ccc.com (Clem Cole) Date: Fri, 12 May 2017 11:17:02 -0400 Subject: [TUHS] The evolution of Unix facilities and architecture In-Reply-To: <20170512151256.B27CE18C09A@mercury.lcs.mit.edu> References: <20170512151256.B27CE18C09A@mercury.lcs.mit.edu> Message-ID: George Gobble of Purdue did the FS work to V7/4.1 to fix the FS corruption issues. That was taken back by Kirk (wnj) and incorporated in 4.1A. It may have been before USENIX was creating proceedings. I'll have to look on my shelf at home or maybe ask George. Clem On Fri, May 12, 2017 at 11:12 AM, Noel Chiappa wrote: > > From: "Ron Natalie" > > > Ordered writes go back to the original BSD fast file system, no? I > seem > > to recall that when we switched from our V6/V7 disks, the filesystem > got > > a lot more stable in crashes. > > I had a vague memory of reading about that, so I looked in the canonical > FFS > paper (McKusick et al, "A Fast File System for UNIX" [1984)]) but found no > mention of it. > > I did find a paper about 'fsck' (McKusick, Kowalski, "Fsck: The UNIX File > System Check Program") which talks (in Section 2.5. "Updates to the file > system") about how "problem[s] with asynchronous inode updates can be > avoided > by doing all inode deallocations synchronously", but it's not clear if > they're > talking about something that was actually done, or just saying > (hypothetically) that that's how one would fix it. > > Is is possible that the changes to the file system (e.g. the way free > blocks > were kept) made it more crash-proof? > > Noel > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From clemc at ccc.com Sat May 13 01:18:53 2017 From: clemc at ccc.com (Clem Cole) Date: Fri, 12 May 2017 11:18:53 -0400 Subject: [TUHS] The evolution of Unix facilities and architecture In-Reply-To: References: <20170512151256.B27CE18C09A@mercury.lcs.mit.edu> Message-ID: I should have said -- it was not hypothetical -- George implemented it and published the code and we all picked it up, On Fri, May 12, 2017 at 11:17 AM, Clem Cole wrote: > George Gobble of Purdue did the FS work to V7/4.1 to fix the FS corruption > issues. That was taken back by Kirk (wnj) and incorporated in 4.1A. It > may have been before USENIX was creating proceedings. I'll have to look > on my shelf at home or maybe ask George. > > Clem > > On Fri, May 12, 2017 at 11:12 AM, Noel Chiappa > wrote: > >> > From: "Ron Natalie" >> >> > Ordered writes go back to the original BSD fast file system, no? I >> seem >> > to recall that when we switched from our V6/V7 disks, the >> filesystem got >> > a lot more stable in crashes. >> >> I had a vague memory of reading about that, so I looked in the canonical >> FFS >> paper (McKusick et al, "A Fast File System for UNIX" [1984)]) but found no >> mention of it. >> >> I did find a paper about 'fsck' (McKusick, Kowalski, "Fsck: The UNIX File >> System Check Program") which talks (in Section 2.5. "Updates to the file >> system") about how "problem[s] with asynchronous inode updates can be >> avoided >> by doing all inode deallocations synchronously", but it's not clear if >> they're >> talking about something that was actually done, or just saying >> (hypothetically) that that's how one would fix it. >> >> Is is possible that the changes to the file system (e.g. the way free >> blocks >> were kept) made it more crash-proof? >> >> Noel >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From clemc at ccc.com Sat May 13 01:46:08 2017 From: clemc at ccc.com (Clem Cole) Date: Fri, 12 May 2017 11:46:08 -0400 Subject: [TUHS] The evolution of Unix facilities and architecture In-Reply-To: References: <20170512151256.B27CE18C09A@mercury.lcs.mit.edu> Message-ID: I just did a small amount of hunting. The oldest printed USENIX Proceeding seems to be 1983 [which was the one Rob gave 'cat -v considered harmful' - although only the abstract is in it]. George did the ordered writes work earlier as I was still at Tektronix, because I remember getting a tape from him a putting the changes into our V7 system. If we hunt around for a 'Purdue-EE' distribution circa '79-'81 we should be able to find it. BTW: as a piece of History for Diomidis on that same tape is fix for one of the first '0-day' UNIX exploits I can remember. I'll see if I can find it and identify it for you. That would be a good piece of history to call out. The story is this ... George was very upset when he found it. But this was during the time when UNIX was fighting a bit for it's life in the press as not being a 'real' OS. DEC and IBM making claims that it was a toy, *etc*. So most of the the hacker community took it pretty seriously. It is funny, today we would react in the opposite manner., But, there was a big 'hush-hush' meeting at a Summer USENIX that was very exclusive to be invited too. We were in a private conference room, the door was locked etc. I remember that Dennis was there, Joy was there. Ron's old friend Mike must have been in it. I think a couple of the Rand folks. Anyway - it was an issue with profile(2) -- surprise, surprise. Pretty easy fix. We all took the code back and promised to get patches out ASAP and not tell any one about it. Clem On Fri, May 12, 2017 at 11:18 AM, Clem Cole wrote: > I should have said -- it was not hypothetical -- George implemented it > and published the code and we all picked it up, > > On Fri, May 12, 2017 at 11:17 AM, Clem Cole wrote: > >> George Gobble of Purdue did the FS work to V7/4.1 to fix the FS >> corruption issues. That was taken back by Kirk (wnj) and incorporated in >> 4.1A. It may have been before USENIX was creating proceedings. I'll >> have to look on my shelf at home or maybe ask George. >> >> Clem >> >> On Fri, May 12, 2017 at 11:12 AM, Noel Chiappa >> wrote: >> >>> > From: "Ron Natalie" >>> >>> > Ordered writes go back to the original BSD fast file system, no? >>> I seem >>> > to recall that when we switched from our V6/V7 disks, the >>> filesystem got >>> > a lot more stable in crashes. >>> >>> I had a vague memory of reading about that, so I looked in the canonical >>> FFS >>> paper (McKusick et al, "A Fast File System for UNIX" [1984)]) but found >>> no >>> mention of it. >>> >>> I did find a paper about 'fsck' (McKusick, Kowalski, "Fsck: The UNIX File >>> System Check Program") which talks (in Section 2.5. "Updates to the file >>> system") about how "problem[s] with asynchronous inode updates can be >>> avoided >>> by doing all inode deallocations synchronously", but it's not clear if >>> they're >>> talking about something that was actually done, or just saying >>> (hypothetically) that that's how one would fix it. >>> >>> Is is possible that the changes to the file system (e.g. the way free >>> blocks >>> were kept) made it more crash-proof? >>> >>> Noel >>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From chet.ramey at case.edu Sat May 13 01:52:50 2017 From: chet.ramey at case.edu (Chet Ramey) Date: Fri, 12 May 2017 11:52:50 -0400 Subject: [TUHS] The evolution of Unix facilities and architecture In-Reply-To: <20170512143047.GV4341@mcvoy.com> References: <20170511140729.2262B18C09A@mercury.lcs.mit.edu> <013b01d2ca96$6901b370$3b051a50$@ronnatalie.com> <20170511222547.GJ4341@mcvoy.com> <1873b923-6f5e-83a2-753d-6d7a2f8b580a@telegraphics.com.au> <20170512081713.GD7265@yeono.kjorling.se> <0121AE71-2930-40AB-ABFD-552E207AF979@tfeb.org> <20170512143047.GV4341@mcvoy.com> Message-ID: <3aa087e2-9305-3d5b-123e-dc2d7833baf2@case.edu> On 5/12/17 10:30 AM, Larry McVoy wrote: > On Fri, May 12, 2017 at 02:56:59PM +0100, Tim Bradshaw wrote: >> When I found out about this I thought seriously of shorting Sun's >> stock (if I knew how to do that). I would have made money. As it was >> we stuck with logged UFS which, by 2007 or so was seriously bulletproof. > > Wait, someone added logging to UFS? Is there a writeup of that anywhere? You could look at the soft updates paper from 1999 for Kirk's perspective. https://www.usenix.org/legacy/event/usenix99/full_papers/mckusick/mckusick.pdf There was a paper about journaled soft updates, too: https://www.mckusick.com/softdep/suj.pdf Chet -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRU chet at case.edu http://cnswww.cns.cwru.edu/~chet/ From imp at bsdimp.com Sat May 13 02:21:16 2017 From: imp at bsdimp.com (Warner Losh) Date: Fri, 12 May 2017 10:21:16 -0600 Subject: [TUHS] The evolution of Unix facilities and architecture In-Reply-To: <3aa087e2-9305-3d5b-123e-dc2d7833baf2@case.edu> References: <20170511140729.2262B18C09A@mercury.lcs.mit.edu> <013b01d2ca96$6901b370$3b051a50$@ronnatalie.com> <20170511222547.GJ4341@mcvoy.com> <1873b923-6f5e-83a2-753d-6d7a2f8b580a@telegraphics.com.au> <20170512081713.GD7265@yeono.kjorling.se> <0121AE71-2930-40AB-ABFD-552E207AF979@tfeb.org> <20170512143047.GV4341@mcvoy.com> <3aa087e2-9305-3d5b-123e-dc2d7833baf2@case.edu> Message-ID: On Fri, May 12, 2017 at 9:52 AM, Chet Ramey wrote: > On 5/12/17 10:30 AM, Larry McVoy wrote: >> On Fri, May 12, 2017 at 02:56:59PM +0100, Tim Bradshaw wrote: >>> When I found out about this I thought seriously of shorting Sun's >>> stock (if I knew how to do that). I would have made money. As it was >>> we stuck with logged UFS which, by 2007 or so was seriously bulletproof. >> >> Wait, someone added logging to UFS? Is there a writeup of that anywhere? > > You could look at the soft updates paper from 1999 for Kirk's perspective. > > https://www.usenix.org/legacy/event/usenix99/full_papers/mckusick/mckusick.pdf > > There was a paper about journaled soft updates, too: > > https://www.mckusick.com/softdep/suj.pdf There's also Margo's paper on LFS, which added logging to UFS, though by the time it was over it wasn't recognizable. https://www.usenix.org/publications/library/proceedings/sd93/seltzer.pdf The Journaled updates, though, is that someone added an intent log to UFS + SoftUpdates. This was used to replay the last few operations when doing fsck coming back up after a crash. There's also something called gjournal, which adds a different type of journaling to UFS, but the less said about that train-wreck the better. Warner From doug at cs.dartmouth.edu Sat May 13 04:43:14 2017 From: doug at cs.dartmouth.edu (Doug McIlroy) Date: Fri, 12 May 2017 14:43:14 -0400 Subject: [TUHS] The evolution of Unix facilities and architecture Message-ID: <201705121843.v4CIhE68027515@coolidge.cs.Dartmouth.EDU> > We all took the code back and promised to get patches out ASAP and not tell any one about it. Fascinating. Chnages were installed frequently in the Unix lab, mostly at night without fanfare. But an actual zero-day should have been big enough news for me to have heard about. I'm pretty sure I didn't; Dennis evidently kept his counsel. Doug From crossd at gmail.com Sat May 13 04:56:37 2017 From: crossd at gmail.com (Dan Cross) Date: Fri, 12 May 2017 14:56:37 -0400 Subject: [TUHS] The evolution of Unix facilities and architecture In-Reply-To: <201705121843.v4CIhE68027515@coolidge.cs.Dartmouth.EDU> References: <201705121843.v4CIhE68027515@coolidge.cs.Dartmouth.EDU> Message-ID: On Fri, May 12, 2017 at 2:43 PM, Doug McIlroy wrote: > > > We all took the code back and promised to get patches out ASAP and not > tell any one about it. > > Fascinating. Chnages were installed frequently in the Unix lab, mostly > at night without fanfare. But an actual zero-day should have been big > enough news for me to have heard about. I'm pretty sure I didn't; Dennis > evidently kept his counsel. I wonder if such a thing would have been treated the same way within Bell Labs as outside? Presumably you didn't have to worry about hordes of undergraduates picking over your systems looking for ways to get root access. Or, indeed, undergraduates doing anything on your systems, save for the occasional intern or precocious child of an employee. For that matter, this raises a question: what was the attitude towards root access within the labs? Was it constrained to the anointed few or did a large-ish number of people have it? Anyway, I could well imagine a scenario where Dennis comes back but thinks fairly little of it and makes vague mention of a fairly serious bug but gives it little more thought than any other fairly serious bug. It's patched and folks go on with their lives, since it's much less likely to be the source of irritation in a corporate search department than it would be in, say, a university. - Dan C. -------------- next part -------------- An HTML attachment was scrubbed... URL: From clemc at ccc.com Sat May 13 05:43:00 2017 From: clemc at ccc.com (Clem Cole) Date: Fri, 12 May 2017 15:43:00 -0400 Subject: [TUHS] The evolution of Unix facilities and architecture In-Reply-To: References: <201705121843.v4CIhE68027515@coolidge.cs.Dartmouth.EDU> Message-ID: Could be. The scare was that the anti-UNIX folks would get wind of it and it would used in the fight as to why VMS was 'better.' The CS Research community has not yet made the switch off the 36 bit world to Vaxen, so the Arpanet community is still pretty much PDP-10 central; but it was also right around the time when DARPA was defunding the PDP-10; had chosen the VAX but was arguing VMS vs UNIX. I don't thing CSRG had been funded as a group yet. Joy might have done his 'fast vax' paper to show that UNIX was just as good as VMS, but that work might be on the horizon. Certainly all of 4.1a/b/c, 4.2, 4.3, NET-x was years away. The point is that you didn't (yet) have a mass of students on the systems 'in the field', but some folks had that as a vision (and want it to be that way and are scared it something bad happens 'in the press' - it would cause a set back. At the that time, think a couple of Universities are >>starting<< to use UNIX for general CS classes/teaching (Purdue & UCB being two of them), maybe Michigan and U of I, but I think CMU and Stanford are still using PDP-20's [not sure about MIT] (where Princeton and UCLA I think were still IBM shops for undergrads). So the whole reason to keep it quiet @ the USENOX conference was because it was felt at the time, the folks in that room were the primary people hacking the kernel and if we all took the couple of lines of fix back to our shops, the problem was solved. It sort of blows my mind if Doug never knew about it, in hind sight it seems George got his wish!! Clem On Fri, May 12, 2017 at 2:56 PM, Dan Cross wrote: > On Fri, May 12, 2017 at 2:43 PM, Doug McIlroy > wrote: >> >> > We all took the code back and promised to get patches out ASAP and not >> tell any one about it. >> >> Fascinating. Chnages were installed frequently in the Unix lab, mostly >> at night without fanfare. But an actual zero-day should have been big >> enough news for me to have heard about. I'm pretty sure I didn't; Dennis >> evidently kept his counsel. > > > I wonder if such a thing would have been treated the same way within Bell > Labs as outside? > > Presumably you didn't have to worry about hordes of undergraduates picking > over your systems looking for ways to get root access. Or, indeed, > undergraduates doing anything on your systems, save for the occasional > intern or precocious child of an employee. For that matter, this raises a > question: what was the attitude towards root access within the labs? Was it > constrained to the anointed few or did a large-ish number of people have it? > > Anyway, I could well imagine a scenario where Dennis comes back but thinks > fairly little of it and makes vague mention of a fairly serious bug but > gives it little more thought than any other fairly serious bug. It's > patched and folks go on with their lives, since it's much less likely to be > the source of irritation in a corporate search department than it would be > in, say, a university. > > - Dan C. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From clemc at ccc.com Sat May 13 06:06:34 2017 From: clemc at ccc.com (Clem Cole) Date: Fri, 12 May 2017 16:06:34 -0400 Subject: [TUHS] The evolution of Unix facilities and architecture In-Reply-To: References: <201705121843.v4CIhE68027515@coolidge.cs.Dartmouth.EDU> Message-ID: BTW: As I think more about it, I believe that it is probable that George found it on the PDP-11 first (V7) not yet BSD 3.0/4.0 (altlhough I've forgotten the date or which USENIX we had the meeting). I'm not sure why, but I remember putting the fix into a multiple kernels due to specifics of the different MMUs). Because of that, memory I'm going to date it as probably 1981, because I would have had 3 different kernels to play with: PDP-11, Vax and Glaser and I were writing Magix an OS for the Tek Magonlia, a 68000 based workstation being build in Tek Labs. But I'm sure if we go hunting through different people's records around that time, changes to the profile kernel code for security will show up. As I said, George originally found it and we all grabbed his fix... so I would expect to see a lot of updated kernels with a similar changes around the same time; but all put in by different people - but I fear, it was also earlier enough that this was before SCCS was widely being used; so tracking it can only be done by looking at distribution tapes from those times. Clem On Fri, May 12, 2017 at 3:43 PM, Clem Cole wrote: > Could be. The scare was that the anti-UNIX folks would get wind of it > and it would used in the fight as to why VMS was 'better.' The CS Research > community has not yet made the switch off the 36 bit world to Vaxen, so the > Arpanet community is still pretty much PDP-10 central; but it was also > right around the time when DARPA was defunding the PDP-10; had chosen the > VAX but was arguing VMS vs UNIX. > > I don't thing CSRG had been funded as a group yet. Joy might have done his > 'fast vax' paper to show that UNIX was just as good as VMS, but that work > might be on the horizon. Certainly all of 4.1a/b/c, 4.2, 4.3, NET-x was > years away. > > The point is that you didn't (yet) have a mass of students on the systems > 'in the field', but some folks had that as a vision (and want it to be that > way and are scared it something bad happens 'in the press' - it would cause > a set back. > > At the that time, think a couple of Universities are >>starting<< to use > UNIX for general CS classes/teaching (Purdue & UCB being two of them), > maybe Michigan and U of I, but I think CMU and Stanford are still using > PDP-20's [not sure about MIT] (where Princeton and UCLA I think were still > IBM shops for undergrads). > > So the whole reason to keep it quiet @ the USENOX conference was because > it was felt at the time, the folks in that room were the primary people > hacking the kernel and if we all took the couple of lines of fix back to > our shops, the problem was solved. > > It sort of blows my mind if Doug never knew about it, in hind sight it > seems George got his wish!! > > Clem > > On Fri, May 12, 2017 at 2:56 PM, Dan Cross wrote: > >> On Fri, May 12, 2017 at 2:43 PM, Doug McIlroy >> wrote: >>> >>> > We all took the code back and promised to get patches out ASAP and >>> not tell any one about it. >>> >>> Fascinating. Chnages were installed frequently in the Unix lab, mostly >>> at night without fanfare. But an actual zero-day should have been big >>> enough news for me to have heard about. I'm pretty sure I didn't; Dennis >>> evidently kept his counsel. >> >> >> I wonder if such a thing would have been treated the same way within Bell >> Labs as outside? >> >> Presumably you didn't have to worry about hordes of undergraduates >> picking over your systems looking for ways to get root access. Or, indeed, >> undergraduates doing anything on your systems, save for the occasional >> intern or precocious child of an employee. For that matter, this raises a >> question: what was the attitude towards root access within the labs? Was it >> constrained to the anointed few or did a large-ish number of people have it? >> >> Anyway, I could well imagine a scenario where Dennis comes back but >> thinks fairly little of it and makes vague mention of a fairly serious bug >> but gives it little more thought than any other fairly serious bug. It's >> patched and folks go on with their lives, since it's much less likely to be >> the source of irritation in a corporate search department than it would be >> in, say, a university. >> >> - Dan C. >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From reed at reedmedia.net Sat May 13 06:40:03 2017 From: reed at reedmedia.net (Jeremy C. Reed) Date: Fri, 12 May 2017 15:40:03 -0500 (CDT) Subject: [TUHS] The evolution of Unix facilities and architecture In-Reply-To: References: <201705121843.v4CIhE68027515@coolidge.cs.Dartmouth.EDU> Message-ID: profil code somewhere in here? https://github.com/weiss/original-bsd/commits/master/sys/kern/kern_clock.c?after=b44636d7febc9dcf553118bd320571864188351d+104 that has the sccs history back to April 1980 for src/sys/sys/clock.c From dave at horsfall.org Sat May 13 07:12:14 2017 From: dave at horsfall.org (Dave Horsfall) Date: Sat, 13 May 2017 07:12:14 +1000 (EST) Subject: [TUHS] The evolution of Unix facilities and architecture In-Reply-To: <1494570272.2218.180.camel@mni.thm.de> References: <20170511140729.2262B18C09A@mercury.lcs.mit.edu> <20170511142142.GA4341@mcvoy.com> <20170511171100.GA9980@yeono.kjorling.se> <1494570272.2218.180.camel@mni.thm.de> Message-ID: On Fri, 12 May 2017, Hellwig Geisse wrote: > >     char* cp1; > >     char* cp2; > > > > etc, which IMHO makes it clear (which is every programmer's duty). I > > used  to write that way in a previous life, and the boss didn't > > complain. > > This view does not work well with more complicated declarations like > "void (*p)(int)". What is the "fundamental type" here? One could argue > that the real culprit is the list construction, which does not mix well > with C declarations. Let's see: aneurin% cdecl Type `help' or `?' for help explain void (*p)(int) declare p as pointer to function (int) returning void So the "fundamental" type (if there was such a thing) would be a pointer to a function, I guess i.e. don't treat it as anything else. -- Dave Horsfall DTM (VK2KFU) "Those who don't understand security will suffer." From ron at ronnatalie.com Sat May 13 07:29:16 2017 From: ron at ronnatalie.com (Ron Natalie) Date: Fri, 12 May 2017 17:29:16 -0400 Subject: [TUHS] The evolution of Unix facilities and architecture In-Reply-To: References: <201705121843.v4CIhE68027515@coolidge.cs.Dartmouth.EDU> Message-ID: <022a01d2cb66$cfa7f780$6ef7e680$@ronnatalie.com> Allegedly at one point Dennis Mumaugh over at the NSA had a list of UNIX security bugs that became classified. Mike Muuss (and the rest of us BRLers) decided not to request a copy because we figured we’d be better off not being constrained by knowing some bug we were disseminating information on (to other system adminsitrators) was classified. We did have a little informal meeting with Dennis in the hall and ran through our list of known issues at the time. My favorite isn’t so much a UNIX bug but a PDP-11 hardware bug. It’s just that I think it may be impossible on a DEC operating system to create a program that manifests it (you have to fill your entire address space with SPL instructions). -------------- next part -------------- An HTML attachment was scrubbed... URL: From clemc at ccc.com Sat May 13 07:29:20 2017 From: clemc at ccc.com (Clem Cole) Date: Fri, 12 May 2017 17:29:20 -0400 Subject: [TUHS] The evolution of Unix facilities and architecture In-Reply-To: References: <201705121843.v4CIhE68027515@coolidge.cs.Dartmouth.EDU> Message-ID: Excuse me - I said -- profil - I intended to say ptrace(2).... long day -- aging brain bits. http://man.cat-v.org/unix_7th/2/ptrace The debugger system call that allows you to a processed memory. On Fri, May 12, 2017 at 4:40 PM, Jeremy C. Reed wrote: > profil code somewhere in here? > https://github.com/weiss/original-bsd/commits/master/ > sys/kern/kern_clock.c?after=b44636d7febc9dcf553118bd320571864188351d+104 > > that has the sccs history back to April 1980 for src/sys/sys/clock.c > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hellwig.geisse at mni.thm.de Sat May 13 09:25:42 2017 From: hellwig.geisse at mni.thm.de (Hellwig Geisse) Date: Sat, 13 May 2017 01:25:42 +0200 Subject: [TUHS] The evolution of Unix facilities and architecture In-Reply-To: References: <20170511140729.2262B18C09A@mercury.lcs.mit.edu> <20170511142142.GA4341@mcvoy.com> <20170511171100.GA9980@yeono.kjorling.se> <1494570272.2218.180.camel@mni.thm.de> Message-ID: <1494631542.2218.215.camel@mni.thm.de> On Sa, 2017-05-13 at 07:12 +1000, Dave Horsfall wrote: >  > Let's see: > >     aneurin% cdecl >     Type `help' or `?' for help >     explain void (*p)(int) >     declare p as pointer to function (int) returning void > > So the "fundamental" type (if there was such a thing) would be a > pointer to a function, I guess i.e. don't treat it as anything else. > Yes, of course. What I was aiming at: If you try to declare two of these variables, neither "void (*p,q)(int)" nor "void (*(p,q))(int)" is allowed, so you cannot use the "fundamental type" to declare more than one variable of this type in a single declaration list (as you had suggested with "char* cp1, cp2"). "void (*p)(int), (*q)(int)" in contrast is legal, but I wouldn't call "void" the fundamental type in these declarations. Thus my statement "list construction (in declarations) and C declarations don't mix well" - IMO one of the difficulties in reading/writing C declarations, and the starting point of this discussion. Hellwig From jnc at mercury.lcs.mit.edu Sat May 13 09:30:12 2017 From: jnc at mercury.lcs.mit.edu (Noel Chiappa) Date: Fri, 12 May 2017 19:30:12 -0400 (EDT) Subject: [TUHS] The evolution of Unix facilities and architecture Message-ID: <20170512233012.7B9DB18C099@mercury.lcs.mit.edu> > From: Clem Cole > I said -- profil - I intended to say ptrace(2) Is that the one where running an SUID program under the debugger allowed one to patch the in-core image of said program? If so, I have a story, and a puzzle, about that. A couple of us, including Jim Gettys (later of X-windows fame) were on out way out to dinner one evening (I don't recall when, alas, but I didn't meet him until '80 or so), and he mentioned this horrible Unix security bug that had just been found. All he would tell me about it (IIRC) was that it involved ptrace. So, over dinner (without the source) I figured out what it had to be: patching SUID programs. So I asked him if that was what it was, and I don't recall his exact answer, but I vaguely recall he hemmed and hawed in a way that let me know I'd worked it out. So when we got back from dinner, I looked at the source to our system to see if I was right, and.... it had already been fixed! Here's the code: if (xp->x_count!=1 || xp->x_iptr->i_mode&ISVTX) goto error; Now, we'd been running that system since '77 (when I joined CSR), without any changes to that part of the OS, so I'm pretty sure this fix pre-dates your story? So when I saw your email about this, I wondered 'did that bug get fixed at MIT when some undergrad used it to break in' (I _think_ ca. '77 is when they switched from an OS called Delphi on the -11/45 used for the undergrad CS programming course - I _think_ they switched that machine from Delphi to Unix), or did it come with PWB1? (Like I said, that system was mostly PWB1.) So I just looked in the PWB1 sources, and... there it is, the _exact_ same fix. So we must have got it from PWB1. So now the question is: did the PWB guys find and fix this, and forget to tell the research guys? Or did they tell them, and the research guys blew them off? Or what? Noel From dave at horsfall.org Sat May 13 09:38:49 2017 From: dave at horsfall.org (Dave Horsfall) Date: Sat, 13 May 2017 09:38:49 +1000 (EST) Subject: [TUHS] The evolution of Unix facilities and architecture In-Reply-To: <20170512233012.7B9DB18C099@mercury.lcs.mit.edu> References: <20170512233012.7B9DB18C099@mercury.lcs.mit.edu> Message-ID: On Fri, 12 May 2017, Noel Chiappa wrote: > So when we got back from dinner, I looked at the source to our system to see > if I was right, and.... it had already been fixed! Here's the code: > > if (xp->x_count!=1 || xp->x_iptr->i_mode&ISVTX) > goto error; Err, isn't that the sticky bit, not the setuid bit? -- Dave Horsfall DTM (VK2KFU) "Those who don't understand security will suffer." From random832 at fastmail.com Sat May 13 09:52:49 2017 From: random832 at fastmail.com (Random832) Date: Fri, 12 May 2017 19:52:49 -0400 Subject: [TUHS] The evolution of Unix facilities and architecture In-Reply-To: References: <20170512233012.7B9DB18C099@mercury.lcs.mit.edu> Message-ID: <1494633169.1599957.975050752.2FB9B2DA@webmail.messagingengine.com> On Fri, May 12, 2017, at 19:38, Dave Horsfall wrote: > On Fri, 12 May 2017, Noel Chiappa wrote: > > > So when we got back from dinner, I looked at the source to our system to see > > if I was right, and.... it had already been fixed! Here's the code: > > > > if (xp->x_count!=1 || xp->x_iptr->i_mode&ISVTX) > > goto error; > > Err, isn't that the sticky bit, not the setuid bit? The sticky bit makes it keep the image in memory when there are no processes using it. I assume x_count is determining whether there are processes using it. So, taken together, these checks are "is there or might there be in the future a process, other than the one being debugged, using this exact copy of the image rather than loading it from the disk". The next line is "xp->x_iptr->i_flag &= ~ITEXT", which I assume prevents the image from being reused for other processes started while this one is running. I am looking at 7th edition in the UnixTree site, the whole fix is: /* * If text, must assure exclusive use */ if (xp = u.u_procp->p_textp) { if (xp->x_count!=1 || xp->x_iptr->i_mode&ISVTX) goto error; xp->x_iptr->i_flag &= ~ITEXT; } The equivalent section to the one this appears in in 6th edition doesn't have the fix, and the comment claims, doesn't work at all: /* write user I (for now, always an error) */ case 4: if (suiword(ipc.ip_addr, 0) < 0) goto error; suiword(ipc.ip_addr, ipc.ip_data); break; This is clearly PDP-11 specific, maybe a similar bug reappeared with demand-paged virtual memory. From clemc at ccc.com Sat May 13 10:22:23 2017 From: clemc at ccc.com (Clem Cole) Date: Fri, 12 May 2017 20:22:23 -0400 Subject: [TUHS] The evolution of Unix facilities and architecture In-Reply-To: <20170512233012.7B9DB18C099@mercury.lcs.mit.edu> References: <20170512233012.7B9DB18C099@mercury.lcs.mit.edu> Message-ID: Interesting... I don't remember Gettys being at the meeting (I would get to know Jim a few years later when he was at Princeton before he came back to MIT to work on X) and he's a been a friend of mine for a number of years (actually lives in the next town over). I do not remember all the details of the bug at this point, to many beers ago; but yes the jist of the issue was being able to write to user memory with ptraced process with SUID being involved. The only thing that worries me about your response is I thought remembered that MMU was somehow involved. Just turning off SUID was not the only part of the solution. I do remember that the bug was in the Research kernel at the time and Dennis had not known about it until that meeting so if PWB had it fixed, that's an example of something that did not go back, which I would find surprising. I suspect MIT found and fixed it independently, but it never got passed it back for whatever reason. We should try to look in the PWB 1.0 kernel. Clem On Fri, May 12, 2017 at 7:30 PM, Noel Chiappa wrote: > > From: Clem Cole > > > I said -- profil - I intended to say ptrace(2) > > Is that the one where running an SUID program under the debugger allowed > one > to patch the in-core image of said program? > > If so, I have a story, and a puzzle, about that. > > > A couple of us, including Jim Gettys (later of X-windows fame) were on out > way > out to dinner one evening (I don't recall when, alas, but I didn't meet him > until '80 or so), and he mentioned this horrible Unix security bug that had > just been found. All he would tell me about it (IIRC) was that it involved > ptrace. > > So, over dinner (without the source) I figured out what it had to be: > patching SUID programs. So I asked him if that was what it was, and I don't > recall his exact answer, but I vaguely recall he hemmed and hawed in a way > that let me know I'd worked it out. > > So when we got back from dinner, I looked at the source to our system to > see > if I was right, and.... it had already been fixed! Here's the code: > > if (xp->x_count!=1 || xp->x_iptr->i_mode&ISVTX) > goto error; > > Now, we'd been running that system since '77 (when I joined CSR), without > any > changes to that part of the OS, so I'm pretty sure this fix pre-dates your > story? > > So when I saw your email about this, I wondered 'did that bug get fixed at > MIT when some undergrad used it to break in' (I _think_ ca. '77 is when > they > switched from an OS called Delphi on the -11/45 used for the undergrad CS > programming course - I _think_ they switched that machine from Delphi to > Unix), or did it come with PWB1? (Like I said, that system was mostly > PWB1.) > > So I just looked in the PWB1 sources, and... there it is, the _exact_ same > fix. So we must have got it from PWB1. > > So now the question is: did the PWB guys find and fix this, and forget to > tell the research guys? Or did they tell them, and the research guys blew > them off? Or what? > > Noel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From clemc at ccc.com Sat May 13 10:23:55 2017 From: clemc at ccc.com (Clem Cole) Date: Fri, 12 May 2017 20:23:55 -0400 Subject: [TUHS] The evolution of Unix facilities and architecture In-Reply-To: References: <20170512233012.7B9DB18C099@mercury.lcs.mit.edu> Message-ID: On Fri, May 12, 2017 at 8:22 PM, Clem Cole wrote: > We should try to look in the PWB 1.0 kernel. > ​As you said, you found it in the PWB1.0 sources... which is really interesting.​ -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave at horsfall.org Sat May 13 10:26:12 2017 From: dave at horsfall.org (Dave Horsfall) Date: Sat, 13 May 2017 10:26:12 +1000 (EST) Subject: [TUHS] The evolution of Unix facilities and architecture In-Reply-To: <1494633169.1599957.975050752.2FB9B2DA@webmail.messagingengine.com> References: <20170512233012.7B9DB18C099@mercury.lcs.mit.edu> <1494633169.1599957.975050752.2FB9B2DA@webmail.messagingengine.com> Message-ID: On Fri, 12 May 2017, Random832 wrote: > > > if (xp->x_count!=1 || xp->x_iptr->i_mode&ISVTX) > > > goto error; > > > > Err, isn't that the sticky bit, not the setuid bit? > > The sticky bit makes it keep the image in memory when there are no > processes using it. I assume x_count is determining whether there are > processes using it. So, taken together, these checks are "is there or > might there be in the future a process, other than the one being > debugged, using this exact copy of the image rather than loading it from > the disk". I know that, but the discussion was about the SUID bit, and the ability to modify the in-core image of a set-uid program being run... -- Dave Horsfall DTM (VK2KFU) "Those who don't understand security will suffer." From jnc at mercury.lcs.mit.edu Sat May 13 10:44:23 2017 From: jnc at mercury.lcs.mit.edu (Noel Chiappa) Date: Fri, 12 May 2017 20:44:23 -0400 (EDT) Subject: [TUHS] The evolution of Unix facilities and architecture Message-ID: <20170513004423.F3A9C18C099@mercury.lcs.mit.edu> > From: Dave Horsfall > Err, isn't that the sticky bit, not the setuid bit? Oh, right you are. I just looked in the code for ptrace(), and assumed that was it. The fix is _actually_ in sys1$exec() (in V6) and sys1$getxfile() (in PWB1 and the MIT system: /* * set SUID/SGID protections, if no tracing */ if ((u.u_procp->p_flag&STRC)==0) { if(ip->i_mode&ISUID) if(u.u_uid != 0) { u.u_uid = ip->i_uid; u.u_procp->p_uid = ip->i_uid; } The thing is, this code is identical in V6, PWB1, and MIT system!? So now I'm wondering - was this really the bug? Or was there some bug in ptrace I don't see, which was the actual bug that's being discussed here. Because is sure looks like this would prevent the exploitation that I described (start an SUID program under the debugger, then patch the code). Or perhaps somehow this fix was broken by some other feature,, and that introduced the exploit? Noel From random832 at fastmail.com Sat May 13 10:48:35 2017 From: random832 at fastmail.com (Random832) Date: Fri, 12 May 2017 20:48:35 -0400 Subject: [TUHS] The evolution of Unix facilities and architecture In-Reply-To: References: <20170512233012.7B9DB18C099@mercury.lcs.mit.edu> <1494633169.1599957.975050752.2FB9B2DA@webmail.messagingengine.com> Message-ID: <1494636515.1613066.975082816.77F24338@webmail.messagingengine.com> On Fri, May 12, 2017, at 20:26, Dave Horsfall wrote: > On Fri, 12 May 2017, Random832 wrote: > > > > > if (xp->x_count!=1 || xp->x_iptr->i_mode&ISVTX) > > > > goto error; > > > > > > Err, isn't that the sticky bit, not the setuid bit? > > > > The sticky bit makes it keep the image in memory when there are no > > processes using it. I assume x_count is determining whether there are > > processes using it. So, taken together, these checks are "is there or > > might there be in the future a process, other than the one being > > debugged, using this exact copy of the image rather than loading it from > > the disk". > > I know that, but the discussion was about the SUID bit, and the ability > to > modify the in-core image of a set-uid program being run... It seems to me that this check is central to being able to (or not) modify the in-core image of any process at all other than the one being traced (say, by attaching to a SUID program that has already dropped privileges, and making changes that will affect the next time it is run). From random832 at fastmail.com Sat May 13 10:51:43 2017 From: random832 at fastmail.com (Random832) Date: Fri, 12 May 2017 20:51:43 -0400 Subject: [TUHS] The evolution of Unix facilities and architecture In-Reply-To: <20170513004423.F3A9C18C099@mercury.lcs.mit.edu> References: <20170513004423.F3A9C18C099@mercury.lcs.mit.edu> Message-ID: <1494636703.1615694.975085000.13626EAC@webmail.messagingengine.com> On Fri, May 12, 2017, at 20:44, Noel Chiappa wrote: > So now I'm wondering - was this really the bug? Or was there some > bug in ptrace I don't see, which was the actual bug that's being > discussed here. Ah. There's the other piece. You start the SUID program under the debugger, and rather than kicking off the debugger, it simply starts it non-suid. *However*, in the presence of shared text (either of the two cases being checked for in the other place), you can make changes to the text image (e.g. put whatever code you want at the entry point), which will be reused the *next* time it is started *without* the debugger. From dave at horsfall.org Sat May 13 10:55:34 2017 From: dave at horsfall.org (Dave Horsfall) Date: Sat, 13 May 2017 10:55:34 +1000 (EST) Subject: [TUHS] The evolution of Unix facilities and architecture In-Reply-To: <1494636703.1615694.975085000.13626EAC@webmail.messagingengine.com> References: <20170513004423.F3A9C18C099@mercury.lcs.mit.edu> <1494636703.1615694.975085000.13626EAC@webmail.messagingengine.com> Message-ID: On Fri, 12 May 2017, Random832 wrote: > Ah. There's the other piece. You start the SUID program under the > debugger, and rather than kicking off the debugger, it simply starts it > non-suid. *However*, in the presence of shared text (either of the two > cases being checked for in the other place), you can make changes to the > text image (e.g. put whatever code you want at the entry point), which > will be reused the *next* time it is started *without* the debugger. Cripes! I think you're right... If so, well done! -- Dave Horsfall DTM (VK2KFU) "Those who don't understand security will suffer." From torek at torek.net Sat May 13 11:17:02 2017 From: torek at torek.net (Chris Torek) Date: Fri, 12 May 2017 18:17:02 -0700 Subject: [TUHS] The evolution of Unix facilities and architecture In-Reply-To: Your message of "Fri, 12 May 2017 20:51:43 -0400." <1494636703.1615694.975085000.13626EAC@webmail.messagingengine.com> Message-ID: <201705130117.v4D1H26I087121@elf.torek.net> >Ah. There's the other piece. You start the SUID program under >the debugger, and rather than kicking off the debugger, it simply >starts it non-suid. *However*, in the presence of shared text >(either of the two cases being checked for in the other place), >you can make changes to the text image (e.g. put whatever code >you want at the entry point), which will be reused the *next* >time it is started *without* the debugger. Right. Some of this was not a problem with demand-paged files (4.xBSD) since you would not share swap images. But there's more, or rather, *was* more, once people added PT_ATTACH and new system calls or behavior ... specifically, the setreuid / setregid calls from 4.2BSD or the saved setuid behavior in System V. This is one I personally touched. Suppose a process starts out setuid or setgid. This means it has alternative privileges (maybe super-user, maybe not). With these it can do things like open some files or transit some directories. Afterward, using setreuid() and setregid() in 4.2BSD, the process can swap its real and effective IDs, or give up its effective UID or GID entirely, to give up its privileges. So some program could, for instance, chdir past a "lock" directory -- this was the MDQS trick -- and now exist in a tree that it had no access to, or open a file with secrets, that it no longer has permission to open. Once a process had given up any special privileges, though, it could be ptraced again (via PT_ATTACH). Now you can swap file descriptor variables around in it, or otherwise make it do bad things with the privileges it gained while it was setuid. On SysV, with "saved setuid", it is even worse: you can make the process *regain* privileges and do whatever you want. The fix we used was an extra bit in the process flags: "process has had or used privileges". Once set (cleared only on exec()), the process could no longer be ptraced except by root. Processes did -- and still do -- have to be careful about files they may leave open, or other entities that survive exec(). (One should also consider mmap() and shared regions.) Chris From lm at mcvoy.com Sat May 13 11:24:15 2017 From: lm at mcvoy.com (Larry McVoy) Date: Fri, 12 May 2017 18:24:15 -0700 Subject: [TUHS] C declarations. In-Reply-To: <015b01d2caa7$c658c020$530a4060$@ronnatalie.com> References: <015401d2caa0$79762650$6c6272f0$@ronnatalie.com> <68E8DC0A-D1B8-4FF0-AD26-ACDC57E308AF@pobox.com> <20170511223232.GM4341@mcvoy.com> <015b01d2caa7$c658c020$530a4060$@ronnatalie.com> Message-ID: <20170513012415.GZ4341@mcvoy.com> On Thu, May 11, 2017 at 06:41:47PM -0400, Ron Natalie wrote: > If had my way, y = x and passing and returning arrays by value would work > just like every other C type. Maybe, just maybe, now that makes sense. But even now that would be a perf hit unless you added some magical copy on write semantics like Tcl has so it can have pass by value semantics but pass by reference performance. Back then, I think the perf hit would have been so bad everyone would be passing arrays as a reference anyway. -- --- Larry McVoy lm at mcvoy.com http://www.mcvoy.com/lm From jnc at mercury.lcs.mit.edu Sat May 13 11:25:30 2017 From: jnc at mercury.lcs.mit.edu (Noel Chiappa) Date: Fri, 12 May 2017 21:25:30 -0400 (EDT) Subject: [TUHS] The evolution of Unix facilities and architecture Message-ID: <20170513012530.EF62518C099@mercury.lcs.mit.edu> > From: Random832 > It seems to me that this check is central to being able to (or not) > modify the in-core image of any process at all other than the one being > traced (say, by attaching to a SUID program that has already dropped > privileges, and making changes that will affect the next time it is > run). Right, good catch: if you have a program that was _both_ sticky and SUID, when the system is idle (so the text copy in the swap area won't get recycled), call up a copy under the debugger, patch it, exit (leaving the patched copy), and then re-run it without the debugger. I'd have to check the handling of patched sticky pure texts - to see if they are retained or not. {Checks code.} Well, the code to do with pure texts is _very_ different between V6 and PWB1. The exact approach above might not work in V6, because the modified (in-core) copy of pure texts are simply deleted when the last user exits them. But it might be possible for a slight variant to work; leave the copy under the debugger (which will prevent the in-core copy from being discarded), and then run it again without the debugger. That might do it. Under PWB1, I'm not sure if any variant would work (very complicated, and I'm fading). There's an extra flag bit, XWRIT, which is set when a pure text is written into; when the last user stops using the in-code pure text, the modified text is written to swap. (It lools like the in-core copy is always discarded when the last user stops using it.) But the check for sticky would probably stop a sticky pure-text being modified? But maybe the approach that seems like it would work under V6 (leave the patched, debugger copy running, and start a new instance) looks like it should work here too. So maybe the sticky thing is irrelevant? On both V6 and PWB1, it just needs a pure text which is SETUID: start under the debugger, patch, leave running, and start a _new_ copy, which will run the patched version as the SUID user. Noel From ron at ronnatalie.com Sat May 13 12:45:02 2017 From: ron at ronnatalie.com (Ron Natalie) Date: Fri, 12 May 2017 22:45:02 -0400 Subject: [TUHS] C declarations. In-Reply-To: <20170513012415.GZ4341@mcvoy.com> References: <015401d2caa0$79762650$6c6272f0$@ronnatalie.com> <68E8DC0A-D1B8-4FF0-AD26-ACDC57E308AF@pobox.com> <20170511223232.GM4341@mcvoy.com> <015b01d2caa7$c658c020$530a4060$@ronnatalie.com> <20170513012415.GZ4341@mcvoy.com> Message-ID: <025701d2cb92$ec8182a0$c58487e0$@ronnatalie.com> There's no performance "hit" because it doesn't work now. There's really no difference performance or computational complexity wise for char x[4], y[4]; x = y; and struct { char a[4]; } x, y; x = y; About the only thing it would have broken (and would still break today), is the fact that function parameters that are defined to be arrays, really pass pointers. char x[4]; void foo(char a[4]); foo(x); would be costlier than it is now doing the pass by value. Of course you could always fudge if you wanted to pass the pointer by actually doing it that way... char x[4] void foo(char *a); foo(x); -----Original Message----- From: Larry McVoy [mailto:lm at mcvoy.com] Sent: Friday, May 12, 2017 9:24 PM To: Ron Natalie Cc: 'Larry McVoy'; 'David Arnold'; 'The Eunuchs Hysterical Society' Subject: Re: [TUHS] C declarations. On Thu, May 11, 2017 at 06:41:47PM -0400, Ron Natalie wrote: > If had my way, y = x and passing and returning arrays by value would > work just like every other C type. Maybe, just maybe, now that makes sense. But even now that would be a perf hit unless you added some magical copy on write semantics like Tcl has so it can have pass by value semantics but pass by reference performance. Back then, I think the perf hit would have been so bad everyone would be passing arrays as a reference anyway. -- --- Larry McVoy lm at mcvoy.com http://www.mcvoy.com/lm From michael at kjorling.se Sat May 13 22:20:50 2017 From: michael at kjorling.se (Michael =?utf-8?B?S2rDtnJsaW5n?=) Date: Sat, 13 May 2017 12:20:50 +0000 Subject: [TUHS] C declarations. In-Reply-To: <025701d2cb92$ec8182a0$c58487e0$@ronnatalie.com> References: <015401d2caa0$79762650$6c6272f0$@ronnatalie.com> <68E8DC0A-D1B8-4FF0-AD26-ACDC57E308AF@pobox.com> <20170511223232.GM4341@mcvoy.com> <015b01d2caa7$c658c020$530a4060$@ronnatalie.com> <20170513012415.GZ4341@mcvoy.com> <025701d2cb92$ec8182a0$c58487e0$@ronnatalie.com> Message-ID: <20170513122050.GF9980@yeono.kjorling.se> On 12 May 2017 22:45 -0400, from ron at ronnatalie.com (Ron Natalie): > About the only thing it would have broken (and would still break today), is > the fact that function parameters that are defined to be arrays, really pass > pointers. > > char x[4]; > void foo(char a[4]); > foo(x); > > would be costlier than it is now doing the pass by value. Correct me if I am wrong, but _pass by value_ as opposed to _pass by reference_ requires making a copy, no? That's the whole point, to allow the callee to poke at the value it is given at will. At the very least, it would require making a copy when a part of the passed value is modified. Where is that memory going to come from, when even malloc() isn't a part of the language but rather a part of the standard library? Where is all that going to come from if you pass a large array on a memory-constrained system of specs common back in the days when C was designed, especially one that lacks virtual memory support? What if (yes, somewhat contrived example) we had pass by value semantics, and foo() is implemented as void foo(char a[4]) { a[8] = 123; } What exactly should happen? The function delcaration says we are interested in four char, but the function uses far more than that. You could of course invoke the almighty term _undefined behavior_, but you really have to say _something_. And of course, nothing stops us from calling that function as char x[100]; foo(x); To reliably prevent that, it seems to me you'd need to at least amend arrays to know their own size, especially if you don't want the compiler to throw its hands in the air as soon as it sees a malloc() (and recognizes it). Now consider instead something like struct foo_t { char a[4]; char b[10]; }; union foo_u { char[14] x; struct foo_t s; }; void foo(foo_u u) { u.s.a[8] = 123; } (Please excuse any syntax errors there; unions have never been my strong side.) While I agree that _you shouldn't be doing that in the first place_, the fact remains that doing it is not _disallowed_ by the language. It could be disallowed by the language if the array knows its size, but to reliably disallow it, you'd need to be able to detect it at compile time. That would seem to significantly increase the complexity of the compiler, which of course we wouldn't want on a low-spec system. (I have a strong suspicion that my microwave oven has more computing power and memory than some of the systems C was designed to run on...) > Of course you > could always fudge if you wanted to pass the pointer by actually doing it > that way... > char x[4] > void foo(char *a); > foo(x); ...in which case you need to know from somewhere else whether foo() wants a zero-terminated char array or not. Not impossible -- we already have a similar situation with, say, memcmp() and strncmp() -- but unless you're willing to introduce some additional syntax, it seems to me that you lose expressiveness in the language by lumping the two cases together. All the above said, _please don't get me wrong_. Pass by value semantics is a really nice feature to have, and I have been known to curse at certain languages more than once where the normal behavior is pass by reference and pass by value requires a good deal of extra code. But for potentially large (or even undefined-size) types, pass by value comes at a significant cost in terms of memory, and even the copying itself. Copy-on-write solves some of that but requires elaborate bookkeeping instead, to know what goes where (it's at least one additional layer of indirection), and risks fragmentation. For simplicity and predictability, it's hard to beat pass by reference. -- Michael Kjörling • https://michael.kjorling.se • michael at kjorling.se “People who think they know everything really annoy those of us who know we don’t.” (Bjarne Stroustrup) From tfb at tfeb.org Sat May 13 22:35:49 2017 From: tfb at tfeb.org (Tim Bradshaw) Date: Sat, 13 May 2017 13:35:49 +0100 Subject: [TUHS] C declarations. In-Reply-To: <20170513122050.GF9980@yeono.kjorling.se> References: <015401d2caa0$79762650$6c6272f0$@ronnatalie.com> <68E8DC0A-D1B8-4FF0-AD26-ACDC57E308AF@pobox.com> <20170511223232.GM4341@mcvoy.com> <015b01d2caa7$c658c020$530a4060$@ronnatalie.com> <20170513012415.GZ4341@mcvoy.com> <025701d2cb92$ec8182a0$c58487e0$@ronnatalie.com> <20170513122050.GF9980@yeono.kjorling.se> Message-ID: <0CF82AC1-E835-4C06-813F-D9EFD2C12290@tfeb.org> On 13 May 2017, at 13:20, Michael Kjörling wrote: > > Correct me if I am wrong, but _pass by value_ as opposed to _pass by > reference_ requires making a copy, no? That's the whole point, to > allow the callee to poke at the value it is given at will. At the very > least, it would require making a copy when a part of the passed value > is modified. Where is that memory going to come from, when even > malloc() isn't a part of the language but rather a part of the > standard library? Where is all that going to come from if you pass a > large array on a memory-constrained system of specs common back in the > days when C was designed, especially one that lacks virtual memory > support? Are there languages that copy arrays in function calls defaultly? Perhaps Fortran has some convention that allows that but I doubt it gets used very much, because it would be insane in most cases: COMPUTE_MEAN_TEMPERATURE(ATMOS) is really *not* going to work very well if it involves copying the ATMOS array. From michael at kjorling.se Sat May 13 22:42:47 2017 From: michael at kjorling.se (Michael =?utf-8?B?S2rDtnJsaW5n?=) Date: Sat, 13 May 2017 12:42:47 +0000 Subject: [TUHS] C declarations. In-Reply-To: <0CF82AC1-E835-4C06-813F-D9EFD2C12290@tfeb.org> References: <015401d2caa0$79762650$6c6272f0$@ronnatalie.com> <68E8DC0A-D1B8-4FF0-AD26-ACDC57E308AF@pobox.com> <20170511223232.GM4341@mcvoy.com> <015b01d2caa7$c658c020$530a4060$@ronnatalie.com> <20170513012415.GZ4341@mcvoy.com> <025701d2cb92$ec8182a0$c58487e0$@ronnatalie.com> <20170513122050.GF9980@yeono.kjorling.se> <0CF82AC1-E835-4C06-813F-D9EFD2C12290@tfeb.org> Message-ID: <20170513124247.GG9980@yeono.kjorling.se> On 13 May 2017 13:35 +0100, from tfb at tfeb.org (Tim Bradshaw): > Are there languages that copy arrays in function calls defaultly? > Perhaps Fortran has some convention that allows that but I doubt it > gets used very much, because it would be insane in most cases: > COMPUTE_MEAN_TEMPERATURE(ATMOS) is really *not* going to work very > well if it involves copying the ATMOS array. I'm not completely sure about arrays, but at least Java has pass by reference in some cases where you might expect pass by value. IIRC function return values is a prime example. -- Michael Kjörling • https://michael.kjorling.se • michael at kjorling.se “People who think they know everything really annoy those of us who know we don’t.” (Bjarne Stroustrup) From hellwig.geisse at mni.thm.de Sat May 13 23:46:00 2017 From: hellwig.geisse at mni.thm.de (Hellwig Geisse) Date: Sat, 13 May 2017 15:46:00 +0200 Subject: [TUHS] C declarations. In-Reply-To: <0CF82AC1-E835-4C06-813F-D9EFD2C12290@tfeb.org> References: <015401d2caa0$79762650$6c6272f0$@ronnatalie.com> <68E8DC0A-D1B8-4FF0-AD26-ACDC57E308AF@pobox.com> <20170511223232.GM4341@mcvoy.com> <015b01d2caa7$c658c020$530a4060$@ronnatalie.com> <20170513012415.GZ4341@mcvoy.com> <025701d2cb92$ec8182a0$c58487e0$@ronnatalie.com> <20170513122050.GF9980@yeono.kjorling.se> <0CF82AC1-E835-4C06-813F-D9EFD2C12290@tfeb.org> Message-ID: <1494683160.2218.223.camel@mni.thm.de> On Sa, 2017-05-13 at 13:35 +0100, Tim Bradshaw wrote: > Are there languages that copy arrays in > function calls defaultly? Yes, Pascal is an example. If you do *not* want an argument to be copied, its corresponding parameter name must be preceded by the keyword "var". The "User Manual and Report" explicitly warns about the inefficiency incurred when calling functions with parameters of structured types (arrays or records) without "var". Hellwig From steve at quintile.net Sun May 14 01:25:16 2017 From: steve at quintile.net (Steve Simon) Date: Sat, 13 May 2017 16:25:16 +0100 Subject: [TUHS] The evolution of Unix facilities and architecture In-Reply-To: <1494636703.1615694.975085000.13626EAC@webmail.messagingengine.com> References: <20170513004423.F3A9C18C099@mercury.lcs.mit.edu> <1494636703.1615694.975085000.13626EAC@webmail.messagingengine.com> Message-ID: <8B3FFD9C-FA5F-434F-A18E-FF59215C6DDD@quintile.net> hi, this is (IMHO) a rather subtle bug, the ones i remember where rather simpler. is it ok to discuss ancient security holes or is that still bad manners? -Steve From steve at sk2.org Sun May 14 01:36:48 2017 From: steve at sk2.org (Stephen Kitt) Date: Sat, 13 May 2017 17:36:48 +0200 Subject: [TUHS] C declarations. In-Reply-To: <20170513124247.GG9980@yeono.kjorling.se> References: <015401d2caa0$79762650$6c6272f0$@ronnatalie.com> <68E8DC0A-D1B8-4FF0-AD26-ACDC57E308AF@pobox.com> <20170511223232.GM4341@mcvoy.com> <015b01d2caa7$c658c020$530a4060$@ronnatalie.com> <20170513012415.GZ4341@mcvoy.com> <025701d2cb92$ec8182a0$c58487e0$@ronnatalie.com> <20170513122050.GF9980@yeono.kjorling.se> <0CF82AC1-E835-4C06-813F-D9EFD2C12290@tfeb.org> <20170513124247.GG9980@yeono.kjorling.se> Message-ID: <20170513173648.2900e4cd@heffalump.sk2.org> On Sat, 13 May 2017 12:42:47 +0000, Michael Kjörling wrote: > On 13 May 2017 13:35 +0100, from tfb at tfeb.org (Tim Bradshaw): > > Are there languages that copy arrays in function calls defaultly? > > Perhaps Fortran has some convention that allows that but I doubt it > > gets used very much, because it would be insane in most cases: > > COMPUTE_MEAN_TEMPERATURE(ATMOS) is really *not* going to work very > > well if it involves copying the ATMOS array. > > I'm not completely sure about arrays, but at least Java has pass by > reference in some cases where you might expect pass by value. IIRC > function return values is a prime example. Technically (and this is serious nit-picking), Java is always pass-by-value, but the value of an array or object *variable* is the reference to the array or object, not the array or object itself. So the behaviour is pass-by-reference for arrays and objects (although that’s a simplification too, and unhelpful in some instances). The same applies to return values: primitive types are returned by value (copied), arrays and objects are returned via their reference (which is copied too). Regards, Stephen From clemc at ccc.com Sun May 14 02:55:29 2017 From: clemc at ccc.com (Clem Cole) Date: Sat, 13 May 2017 12:55:29 -0400 Subject: [TUHS] The evolution of Unix facilities and architecture In-Reply-To: <8B3FFD9C-FA5F-434F-A18E-FF59215C6DDD@quintile.net> References: <20170513004423.F3A9C18C099@mercury.lcs.mit.edu> <1494636703.1615694.975085000.13626EAC@webmail.messagingengine.com> <8B3FFD9C-FA5F-434F-A18E-FF59215C6DDD@quintile.net> Message-ID: On Sat, May 13, 2017 at 11:25 AM, Steve Simon wrote: > hi, > > this is (IMHO) a rather subtle bug, > the ones i remember where rather simpler. is it ok to discuss ancient > security holes or is that still bad manners? > ​Speaking for myself..... I clearly don't think it is bad manners​ as this stage - I brought it up!E It was a different time when that occurred. Today, I think *the general security community*** pretty lives by the rules of if you find something, notify the folks that fix it as quickly as possible and try to get a patch out and figure out how to get that patch out. Then make damned sure the whole is well documented and published so: a) do we can test for it in the wild, b) make sure it does not happen again. It actually has always impressed me at how good UNIX was (is) when you really get down to it. IMHO, was less the 'thousand eyeballs'' and more the 'eye balls that all of cared, could do something about it and most importantly actually understood' the 'calculus' of the different problems were want made UNIX secure and as good if not better than many 'commercial' systems than its contemporaries. *i.e.* the UNIX schemes used sensible human based security processes/mechanisms combined with basic math & physics ( technology if you will) - as the higher order bits, not being secret or obscure to protect. Were there mistakes, yup. But frankly, VMS had as many if not more and some of them were far, far worse. IBM's OS were considered good, but their were documented exploits in the news there too. Clem ** I note 'security community' because not all firm buy into this behavior. I speak for myself. In the last few weeks my own employer (Intel) recent has been mixed up in a bit over played issue with server chips sets, AMT and Winders [its not my area/group etc but as I under the issue, the bug does not seem to effect UNIX flavors nor systems that do not use AMT - which is a server thingy]. Some outside of Intel people are have complained that folks that own the bug @ my employer has been less that forth coming. I'll not defend nor comment because it's not mine to comment on, other than to state I personally take an attitude of trying to say a much as I can and when I am in a position for my job I will and do. -------------- next part -------------- An HTML attachment was scrubbed... URL: From pechter at gmail.com Sun May 14 03:19:42 2017 From: pechter at gmail.com (William Pechter) Date: Sat, 13 May 2017 13:19:42 -0400 Subject: [TUHS] The evolution of Unix facilities and architecture In-Reply-To: References: <20170513004423.F3A9C18C099@mercury.lcs.mit.edu> <1494636703.1615694.975085000.13626EAC@webmail.messagingengine.com> <8B3FFD9C-FA5F-434F-A18E-FF59215C6DDD@quintile.net> Message-ID: <1038e644-cef1-9424-f6fa-288941033bff@gmail.com> Clem Cole wrote: > > On Sat, May 13, 2017 at 11:25 AM, Steve Simon > wrote: > > hi, > > this is (IMHO) a rather subtle bug, > the ones i remember where rather simpler. is it ok to discuss > ancient security holes or is that still bad manners? > > ​Speaking for myself..... I clearly don't think it is bad manners​ > as this stage - I brought it up!E > It was a different time when that occurred. Today, I think /the > general security community**/ pretty lives by the rules of if you find > something, notify the folks that fix it as quickly as possible and try > to get a patch out and figure out how to get that patch out. Then > make damned sure the whole is well documented and published so: a) do > we can test for it in the wild, b) make sure it does not happen again. > > It actually has always impressed me at how good UNIX was (is) when you > really get down to it. IMHO, was less the 'thousand eyeballs'' and > more the 'eye balls that all of cared, could do something about it and > most importantly actually understood' the 'calculus' of the different > problems were want made UNIX secure and as good if not better than > many 'commercial' systems than its contemporaries. /i.e./ the UNIX > schemes used sensible human based security > processes/mechanisms combined with basic math & physics ( technology > if you will) - as the higher order bits, not being secret or obscure > to protect. > The problem is once you got past "One true Unix" you were left hoping the vendor fixed their bugs. I saw somethings on Solaris and HPUX which were pretty much as bad as Windows. The thing about Research Unix was that the underlying security structure was well designed. VMS wasn't too bad either. The problem was the stuff layered on it. When VMS went to 3.6 or so a friend of mine was almost fired by DEC for randomly testing boxes looking to see DEC's internal boxes weren't running System/Manager, Field/Service and UETP/UETP User/password combinations. DEC had just implemented new security features and alerts and Mitnick had just recently penetrated them (IIRC). Next thing you know corporate security was all over my buddy who was just killing time on night shift temporarily covering someone's vacation time. It was interesting to see the SysV security enhanced Unix from AT&T at Pyramid -- who was migrating to SVR4 from their BSD/SysV hybrid. ACL's, split root/system and security mgr stuff which had been added to get VMS to C and B2... Some of these things had me wondering if any commercial sites would implement two sign-ins to authorize special root-type actions on an os. | Were there mistakes, yup. But frankly, VMS had as many if not more and some of them were far, far worse. IBM's OS were considered good, but their were documented exploits in the news there too. The loginout.exe one was bad. Were there any structural ones past v3.6? > > Clem > > > ** I note 'security community' because not all firm buy into this > behavior. I speak for myself. In the last few weeks my own > employer (Intel) recent has been mixed up in a bit over played issue > with server chips sets, AMT and Winders [its not my area/group etc but > as I under the issue, the bug does not seem to effect UNIX flavors nor > systems that do not use AMT - which is a server thingy]. Some > outside of Intel people are have complained that folks that own the > bug @ my employer has been less that forth coming. I'll not defend > nor comment because it's not mine to comment on, other than to state I > personally take an attitude of trying to say a much as I can and when > I am in a position for my job I will and do. > Actually... I'd think AMT is an automated remote IT Management thing rather than a server thing, since it exists on all the business Thinkpads from my T61's Core 2's up to the T420 i5. They couldn't be considered servers except they do support Samba and NFS and ssh. They also dual boot which is a major part of the risk. Sorry for the pedantic add... but I just remediated my 5 laptops for crap that should've been fixed with new vendor software -- but they can't be bothered. -- Digital had it then. Don't you wish you could buy it now! pechter-at-gmail.com http://xkcd.com/705/ From random832 at fastmail.com Sun May 14 05:05:24 2017 From: random832 at fastmail.com (Random832) Date: Sat, 13 May 2017 15:05:24 -0400 Subject: [TUHS] C declarations. In-Reply-To: <20170513122050.GF9980@yeono.kjorling.se> References: <015401d2caa0$79762650$6c6272f0$@ronnatalie.com> <68E8DC0A-D1B8-4FF0-AD26-ACDC57E308AF@pobox.com> <20170511223232.GM4341@mcvoy.com> <015b01d2caa7$c658c020$530a4060$@ronnatalie.com> <20170513012415.GZ4341@mcvoy.com> <025701d2cb92$ec8182a0$c58487e0$@ronnatalie.com> <20170513122050.GF9980@yeono.kjorling.se> Message-ID: <1494702324.1901172.975588504.1BA76452@webmail.messagingengine.com> On Sat, May 13, 2017, at 08:20, Michael Kjörling wrote: > Correct me if I am wrong, but _pass by value_ as opposed to _pass by > reference_ requires making a copy, no? That's the whole point, to > allow the callee to poke at the value it is given at will. If that were the issue, the callee could simply automatically make the copy if-and-only-if it needs it. The reason that it can't be done without making a copy is, rather, that there might be some *other* reference in play, through which the original array could be modified (thus improperly changing what the callee sees). From random832 at fastmail.com Sun May 14 05:08:12 2017 From: random832 at fastmail.com (Random832) Date: Sat, 13 May 2017 15:08:12 -0400 Subject: [TUHS] C declarations. In-Reply-To: <0CF82AC1-E835-4C06-813F-D9EFD2C12290@tfeb.org> References: <015401d2caa0$79762650$6c6272f0$@ronnatalie.com> <68E8DC0A-D1B8-4FF0-AD26-ACDC57E308AF@pobox.com> <20170511223232.GM4341@mcvoy.com> <015b01d2caa7$c658c020$530a4060$@ronnatalie.com> <20170513012415.GZ4341@mcvoy.com> <025701d2cb92$ec8182a0$c58487e0$@ronnatalie.com> <20170513122050.GF9980@yeono.kjorling.se> <0CF82AC1-E835-4C06-813F-D9EFD2C12290@tfeb.org> Message-ID: <1494702492.1902154.975591160.507FCE4C@webmail.messagingengine.com> On Sat, May 13, 2017, at 08:35, Tim Bradshaw wrote: > Are there languages that copy arrays in function calls defaultly? > Perhaps Fortran has some convention that allows that but I doubt it gets > used very much, because it would be insane in most cases: > COMPUTE_MEAN_TEMPERATURE(ATMOS) is really *not* going to work very well > if it involves copying the ATMOS array. Fortran, for the record, passes nearly everything by reference - to call a fortran function from C (or to make a C function callable from fortran) you've got to mangle the name and change almost everything to pointers. From dave at horsfall.org Sun May 14 09:01:41 2017 From: dave at horsfall.org (Dave Horsfall) Date: Sun, 14 May 2017 09:01:41 +1000 (EST) Subject: [TUHS] The evolution of Unix facilities and architecture In-Reply-To: <8B3FFD9C-FA5F-434F-A18E-FF59215C6DDD@quintile.net> References: <20170513004423.F3A9C18C099@mercury.lcs.mit.edu> <1494636703.1615694.975085000.13626EAC@webmail.messagingengine.com> <8B3FFD9C-FA5F-434F-A18E-FF59215C6DDD@quintile.net> Message-ID: On Sat, 13 May 2017, Steve Simon wrote: > this is (IMHO) a rather subtle bug, the ones i remember where rather > simpler. is it ok to discuss ancient security holes or is that still bad > manners? Bring it on :-) Any systems still vulnerable (and exposed to the Internet) deserves all that they get. For example, I think I was the first one to mention the SPL bug here (note that I didn't find it); I'm still trying to find that little program which I think was published in a Usenix newsletter and exploited by, err, yours truly... -- Dave Horsfall DTM (VK2KFU) "Those who don't understand security will suffer." From richard at inf.ed.ac.uk Sun May 14 09:11:21 2017 From: richard at inf.ed.ac.uk (Richard Tobin) Date: Sun, 14 May 2017 00:11:21 +0100 (BST) Subject: [TUHS] C declarations. In-Reply-To: Random832's message of Sat, 13 May 2017 15:08:12 -0400 Message-ID: <20170513231121.0DB3A1274787@macaroni.inf.ed.ac.uk> > Fortran, for the record, passes nearly everything by reference Sort of. The Fortran 77 standard imposes restrictions that appear to be intended to allow the implementation to pass by value-and-result (i.e. values are copied in, and copied back at return). In particular it disallows aliasing that would allow you to distinguish between the two methods: If a subprogram reference causes a dummy argument in the referenced subprogram to become associated with another dummy argument in the referenced subprogram, neither dummy argument may become defined during execution of that subprogram. http://www.fortran.com/F77_std/rjcnf-15.html#sh-15.9.3.6 -- Richard -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. From dave at horsfall.org Sun May 14 09:21:09 2017 From: dave at horsfall.org (Dave Horsfall) Date: Sun, 14 May 2017 09:21:09 +1000 (EST) Subject: [TUHS] C declarations. In-Reply-To: <1494702492.1902154.975591160.507FCE4C@webmail.messagingengine.com> References: <015401d2caa0$79762650$6c6272f0$@ronnatalie.com> <68E8DC0A-D1B8-4FF0-AD26-ACDC57E308AF@pobox.com> <20170511223232.GM4341@mcvoy.com> <015b01d2caa7$c658c020$530a4060$@ronnatalie.com> <20170513012415.GZ4341@mcvoy.com> <025701d2cb92$ec8182a0$c58487e0$@ronnatalie.com> <20170513122050.GF9980@yeono.kjorling.se> <0CF82AC1-E835-4C06-813F-D9EFD2C12290@tfeb.org> <1494702492.1902154.975591160.507FCE4C@webmail.messagingengine.com> Message-ID: On Sat, 13 May 2017, Random832 wrote: > Fortran, for the record, passes nearly everything by reference [...] And if you passed a constant and the local argument was modified, so was the (single instance of) constant... Actually, wasn't FORTRAN value-result i.e. all args were copied straight back again, or am I confusing that with ALGOL? Blimey, it's been over 40 years since I last submitted a card deck. -- Dave Horsfall DTM (VK2KFU) "Those who don't understand security will suffer." From dave at horsfall.org Sun May 14 09:34:58 2017 From: dave at horsfall.org (Dave Horsfall) Date: Sun, 14 May 2017 09:34:58 +1000 (EST) Subject: [TUHS] Old Unix vulnerabilities Message-ID: OK, I'll kick it off. A beauty in V6 (and possibly V7) was discovered by the kiddies in Elec Eng; by sending a signal with an appropriately-crafted negative value (as determined from inspecting ) you could overwrite u.u_uid with zero... Needless to say I scrambled to fix that one on my 11/40 network! -- Dave Horsfall DTM (VK2KFU) "Those who don't understand security will suffer." From stewart at serissa.com Sun May 14 11:59:57 2017 From: stewart at serissa.com (Lawrence Stewart) Date: Sat, 13 May 2017 21:59:57 -0400 Subject: [TUHS] C declarations. In-Reply-To: <20170513124247.GG9980@yeono.kjorling.se> References: <015401d2caa0$79762650$6c6272f0$@ronnatalie.com> <68E8DC0A-D1B8-4FF0-AD26-ACDC57E308AF@pobox.com> <20170511223232.GM4341@mcvoy.com> <015b01d2caa7$c658c020$530a4060$@ronnatalie.com> <20170513012415.GZ4341@mcvoy.com> <025701d2cb92$ec8182a0$c58487e0$@ronnatalie.com> <20170513122050.GF9980@yeono.kjorling.se> <0CF82AC1-E835-4C06-813F-D9EFD2C12290@tfeb.org> <20170513124247.GG9980@yeono.kjorling.se> Message-ID: <57639684-050B-4995-859F-876CAFE4BE9C@serissa.com> I am not troubled at all by C’s conflation of arrays and pointers. That has always seemed natural - it is what is really going on under the covers.* I’m the sort of person who fiddles with the source code until the compiler generates the code I want. C is great for this. Not always, but C compilers usually do the straightforward and obvious thing and generate the code you would write if you had to do it in assembler, only the compiler keeps track of the stupid details like frame offsets and inventing symbol names for branch targets. I’ve written substantive things in Java, Modula-3, common lisp, python, and so forth, but when I write performance sensitive code, I do it in C** I tend to work on communications code and when you are counting cycles in some HPC library, you write in C but think of every load, store, and shift. My affinity for the language began the same way it probably did for a lot of people here: when I realized that *i++ was exactly a PDP11 addressing mode. When the assistant rector at my church had a meeting about sacred texts, I brought in K&R (“the New Testament” second edition) and my prized copies of the BSTJ Unix issues. -L PS * - But I have never been able to remember the syntax for function pointers. I always “man qsort” to refresh my memory. ** - If the code uses multidimensional arrays or subarrays, then maybe FORTRAN, because really x[i*stride+j] gets really old. From dave at horsfall.org Sun May 14 12:23:38 2017 From: dave at horsfall.org (Dave Horsfall) Date: Sun, 14 May 2017 12:23:38 +1000 (EST) Subject: [TUHS] C declarations. In-Reply-To: <57639684-050B-4995-859F-876CAFE4BE9C@serissa.com> References: <015401d2caa0$79762650$6c6272f0$@ronnatalie.com> <68E8DC0A-D1B8-4FF0-AD26-ACDC57E308AF@pobox.com> <20170511223232.GM4341@mcvoy.com> <015b01d2caa7$c658c020$530a4060$@ronnatalie.com> <20170513012415.GZ4341@mcvoy.com> <025701d2cb92$ec8182a0$c58487e0$@ronnatalie.com> <20170513122050.GF9980@yeono.kjorling.se> <0CF82AC1-E835-4C06-813F-D9EFD2C12290@tfeb.org> <20170513124247.GG9980@yeono.kjorling.se> <57639684-050B-4995-859F-876CAFE4BE9C@serissa.com> Message-ID: On Sat, 13 May 2017, Lawrence Stewart wrote: > * - But I have never been able to remember the syntax for function > pointers. I always “man qsort” to refresh my memory. "cdecl" is your friend: % cdecl Type `help' or `?' for help declare foo as pointer to function (int, double) returning void void (*foo)(int , double ) -- Dave Horsfall DTM (VK2KFU) "Those who don't understand security will suffer." From bakul at bitblocks.com Sun May 14 14:24:11 2017 From: bakul at bitblocks.com (Bakul Shah) Date: Sat, 13 May 2017 21:24:11 -0700 Subject: [TUHS] C declarations. In-Reply-To: Your message of "Sat, 13 May 2017 21:59:57 EDT." <57639684-050B-4995-859F-876CAFE4BE9C@serissa.com> References: <015401d2caa0$79762650$6c6272f0$@ronnatalie.com> <68E8DC0A-D1B8-4FF0-AD26-ACDC57E308AF@pobox.com> <20170511223232.GM4341@mcvoy.com> <015b01d2caa7$c658c020$530a4060$@ronnatalie.com> <20170513012415.GZ4341@mcvoy.com> <025701d2cb92$ec8182a0$c58487e0$@ronnatalie.com> <20170513122050.GF9980@yeono.kjorling.se> <0CF82AC1-E835-4C06-813F-D9EFD2C12290@tfeb.org> <20170513124247.GG9980@yeono.kjorling.se> <57639684-050B-4995-859F-876CAFE4BE9C@serissa.com> Message-ID: <20170514042411.93CCB124AEA4@mail.bitblocks.com> On Sat, 13 May 2017 21:59:57 EDT Lawrence Stewart wrote: > > * - But I have never been able to remember the syntax for function pointers. > I always "man qsort" to refresh my memory. The way I remember: given *x[] or *x(), x /sticks/ to the /right/ first. Thus int *x[]; // x is array of ptr to int int *x(); // x is a function returning ptr to int If you don't want a var to stick to its right, separate using using parentheses. int (*x)[]; // x is a ptr to array of ints int (*x)(); // x is a ptr to function returning int int *(*x)[]; // x is a ptr to array of ptrs to int This sort of also works for multiple variables in one declaration. Given int *x, y; // * /sticks/ to the right (x) first, so not available for y. Now if they'd allowed parenthesizing the type, as in (int *)x, y; we would see that both x & y are of type int *. That would've also allowed declaring multiple vars in one declaration where the previous rule applies! (int*)(*f,*g)(); Part of the confusion is * is tacked on at the front while [] & () at the back of a variable. Someone (Chandy?) had proposed unifying this syntax but it didn't go anywhere. I think the author used @ in a suffix place though I don't recall any other details. But if you put the var all the way to the right and read a declaration right to left, it works: int*[] x // x is a array of ptr to ints int() f // f is a function returning it int*()* f // f is a ptr to function returning ptr to int int*()* f,g // f & g are ptrs to function returning ptr to int This is only slightly weird if you are used to the current syntax but you can easily retrain yourself. IMHO, where evolution of C, Scheme and most programming languages goes wrong is doing evolutionary design by committee. Most users grasp what is easier to use vs what is hard but they don't have the aesthetic sense or imagination or training to make things simple. And in large groups popularity or force of personality or some other irrelevant attribute wins not aesthetics. Language evolution IMHO must be done by benevolent dictatorship or a guild of like minded people who have worked together for a long time! Go seems to have adapted that style.... From tytso at mit.edu Sun May 14 14:30:24 2017 From: tytso at mit.edu (Theodore Ts'o) Date: Sun, 14 May 2017 00:30:24 -0400 Subject: [TUHS] The evolution of Unix facilities and architecture In-Reply-To: <20170511222547.GJ4341@mcvoy.com> References: <20170511140729.2262B18C09A@mercury.lcs.mit.edu> <013b01d2ca96$6901b370$3b051a50$@ronnatalie.com> <20170511222547.GJ4341@mcvoy.com> Message-ID: <20170514043024.pezg5eqslpcqsd5m@thunk.org> On Thu, May 11, 2017 at 03:25:47PM -0700, Larry McVoy wrote: > This is one place where I think Linux kicked Unix's ass. And I am not > really sure how they did it, I have an idea but am not positive. Unix > file systems up through UFS as shipped by Sun, were all vulnerable to > what I call the power out test. Untar some big tarball and power off > the machine in the middle of it. Reboot. Hilarity ensues (not). > > You were dropped into some stand alone shell after fsck threw up its > hands and it was up to you to fix it. Dozens and dozens of errors. > It was almost always faster to go to backups because figuring that > stuff out, file by file (which I have done more than once), gets you > to the point that your run "fsck -y" and go poke at lost+found when > fsck is done, realize that there is no hope, and reach for backups. > > Try the same thing with Linux. The file system will come back, starting > with, I believe, ext2. > > My belief is that Linux orders writes such that while you may lose data > (as in, a process created a file, the OS said it was OK, but that file > will not be in the file system after a crash), but the rest of the file > system will be consistent. I think it's as if you powered off the > machine a few seconds earlier than you actually did, some stuff is in > flight and until they can write stuff out in the proper order you may > lose data on a hard reset. So the story is a bit complicated here, and may be an example of "worse is better" --- which is ironically one of those things which is used as an explanation for why BSD/Unix won ever though the Lisp was technically superior[1] --- but in this case, it's Linux that did something "dirty", and BSD that did something that was supposed to be the "better" solution. [1] https://www.jwz.org/doc/worse-is-better.html So first let's talk about ext2 (which indeed, does not have file system journalling; that came in ext3). The BSD Fast File System goes to a huge amount of effort to make sure that writes are sent to the disk in exactly the right order so that fsck can actually fix things. This requires that the disk not reorder writes (e.g., write caching is disabled or in write-through mode). Linux, in ext2, didn't bother with trying to get the write order correct at all. None. Nada. Zip. Writes would go out in whatever order dictated by the elevator scheduler, and so on a power failure or a kernel crash, the order in which metadata writes would be sent to the disk was completely unconstrained. Sounds horrible, right? In many ways, it was. And I lost count of how often NetBSD and FreeBSD users would talk about how primitive and horrible ext2 was in comparison to FFS, which had all of this excellent engineering work to make sure writes happened in the correct order such that fsck was guaranteed to always be able to fix things. So why did Linux get away with it? When I wrote the fsck for ext2, I knew that anything can and would happen, so it was implemented so that it was extremely paranoid about not ever losing any data. And if there was a chance that an expert could recover the data, e2fsck would stop and ask the system administrator to take a look. In the case that the user ran with fsck -y, the default was drop files into lost+found, where as in some cases with the FFS fsck, it "knew" that in a particular case, the order in which writes were staged out the right thing to do was to let the unlink complete, so it would let the refcount go to zero, or stay at zero. The other thing that we did in Linux is that I made sure we had a highly functional "debugfs" tool. This tool served two purposes. The first was it made it very easy for me to creat a regression test suite for fsck. As far as I know, none of the other major file systems at the time had an fsck with a regression test suite --- and I was religious about adding tests as I added functionality, and as I fixed bugs. The debugfs tool made it easy for me to create test case file systems that was corrupted in various interesting ways. The other use of debugfs was that it made it easy for experts to do file system recovery after a crash, if there was some really precious file that they needed to try to recover. So this is why this is a great example of "worse is better". In Linux, ext2 was ***incredibly*** sloppy about how it handled write ordering --- it didn't do anything at all. But as a consequence we developed tools that were extremely good to compensate, and in practice, it was extremely rare (although it did happen on occasion) that files would get lost or the file systme could end up in a state where fsck would not be able to recover without manual intervention by a system administrator using debugfs. But the other thing to note here is that in the PC era, most disk drives ran with write caching enabled, with writeback caching so that the hard drive could do its own elevator shceduling. So having a file system that very carefully scheduled writes to make sure they happened in the write order didn't help you a *bit* unless you configured your hard drive to disable writeback caching --- at which point you would take a massive speed hit. This is ultimately also one weaknesses of Soft Updates --- it requires that you disable writeback caching, since it works by letting the OS control the order in which writes hit stable storage. With journalling you don't have to do that; but the tradeoff is that when you do a journal commit, you need typically two cache flush operations. (Or a cache flush followed by a FUA write of the commit block, if the disk supports FUA.) There is another example of how Linux embraced the "worse is better" philosophy in ext3, and that has to do with how we do journalling. The sophisticated way to do journalling is to do logical journalling. This is where what you write in the in journal is "set bit XXX in the allocation bitmap", or "update the mtime to YYYY". And in this way, you can batch multiple file system operations into a single block written to the journal. Solaris/UFS and Irix uses this much more sophisticated form of journalling. (Actually, older versions of Solaris did use volume-level journalling, which is basically what ext3/ext4 uses, but they upgraded to the much more "right", more advanced thing, which is logical journalling.) Ext3 uses phyiscal, or volume-level journalling. This journalling works on the block level --- so if we flip a bit in an allocation bitmap, we log the entire 4k block to the journal. By default, we only do a journal commit every five seconds (unless an fsync happens first), so there could be multiple changes to a single inode table blocks that can be batched together, but it's still true that for a given metadata-heavy workload, a file system which uses logical journalling will tend require many fewer blocks written to the journal than a file system such as ext3/ext4 which uses physical block journalling. Why did Linux get away with it? Number one, most workloads aren't really modify metadata all _that_ intensively, and 12k of sequential writes versus 32k of sequential writes doesn't actually take that much more time. Secondly, Ted's law of PC-class hardware ("most PC-class hardware is crap") comes into play, and turns physical journalling into an advantage. PC class hardware tends not to have power fail interrupts, and when power drops, and the voltage levels on the power rails start drooping, DRAM tends to go insane and starts returning garbage long before the DMA engine and the hard drive stops functioning. So if your system is doing logical journalling, after the file system commits a transaction, it will start writing the inode table block to the permanent location on disk. If at that point you get a power drop, garbage can get written to the inode table block, and if the file system is using logical journalling, on reboot the mtime field can get updated from the logical journal --- but the rest of the inode table block is still garbage. In contrast, since ext3 was using physical block journalling, even if various metadata blocks get corrupted due to writes from failing DRAM during a power drop, when we replay the journal, this will restore the entire metadata block, and Things Just Worse. I have talked to an XFS engineer from SGI, and this was definitely a thing which SGI discovered the hard way. After they discovered this problem, they added extra capacitors to the power supply, added a power fail interrupt, and taught Irix so that when the power fail interrupt was triggered, it would frantically cancel DMA transfers in order to avoid this problem. I do not know how many of the other Legacy Unix systems figured out this failure mode --- and I can't claim that we were brilliant enough to design a system to avoid this problem. It just so happened that the brute-force design that we chose was very well suited for crappy (but way cheaper than a Sun Fire E10k :-) PC-class hardware. > I copied Ted, who had his fingers deep in that code, maybe he can correct > me where I got it wrong. Details aside, I think this is a place where > Linux moved the state of the art significantly forward. There are other > places but this one is a big deal IMHO, maybe the biggest deal. So I'm not really sure we can claim to have "moved the state of the art". There certainly wasn't any brilliant computer science innovations here. That sort of thing is more like Soft Updates, of which Valerie Aurora (formerly Henson) once wrote, "I've read this paper at least 15 times, and each time I when get to page 7, I'm feeling pretty good and thinking, "Yeah, okay, I must be smarter now than the last time I read this because I'm getting it this time," - and then I turn to page 8 and my head explodes." --- https://lwn.net/Articles/339337/ I will be the first to admit that with ext2/ext3/ext4, especially in the early days, it was much more about brute force engineering, and regression testing, and much less about "moving the state of the art". Certainly those of us who were working on Linux weren't trying to get papers published in peer reviewed journals or conferences! (And I've always thought that Greg Ganger was _way_ smarter than I. :-) And if the Lisp Machine hackers looked down on BSD, and complained that BSD adopted the "Worse is Better" philosophy, while Lisp strived for the true, elegant, Correct technical solution, it's perhaps especially interesting to consider that if anything, Linux was an even more radical example of the "Worse is Better" philosophy. Cheers, - Ted P.S. There is yet another example of "Worse is Better" in how Linux had PCMCIA support several years before FreeBSD/NetBSD. However, if you ejected a PCMCiA card in a Linux system, there was a chance (in practice it worked out to be about in 1 in 5 times for a WiFI card, in my experience) that the system would crash. The *BSD's took a good 2-3 years longer to get PCMCIA support, but when they did, it was rock solid. Of course, if you are a laptop user, and are happy to keep your 802.11 PCMCIA card permanently installed, guess which OS you were likely to prefer --- "sloppy but works, mostly", or "it'll get there eventually, and will be rock solid when it does, but zip, nada, right now"? > > --lm > > On Thu, May 11, 2017 at 04:37:29PM -0400, Ron Natalie wrote: > > I remember the pre-fsck days. It was part of my test to become an operator at the UNIX site at JHU that I could run the various manual checks. > > > > The V6 file system wasn???t exactly stable during crashes (lousy database behavior), so there was almost certainly something to clean up. > > > > > > > > The first thing we???d run was icheck. This runs down the superblock freelist and all the allocated blocks in the inodes. If there were missing blocks (not in a file or the free list), you could use icheck ???s > > > > to rebuild it. Similarly, if you had duplicated allocations in the freelist or between the freelist and a single file. Anything more complicated required some clever patching (typically, we???d just mount readonly, copy the files, and then blow them away with clri). > > > > > > > > Then you???d run dcheck. As mentioned dcheck walks the directory path from the top of the disk counting inode references that it reconciles with the link count in the inode. Occasionally we???d end up with a 0-0 inode (no directory entires, but allocated???typically this is caused by people removing a file while it is still open, a regular practice of some programs for their /tmp files.). clri again blew these away. > > > > > > > > Clri wrote zeros all over the inode. This had the effect of wiping out the file, but it was dangerous if you got the i-number wrong. We replaced it with ???clrm??? which just cleared the allocated bit, a lot easy to reverse. > > > > > > > > If you really had a mess of a file system, you might get a piece of the directory tree broken off from a path to the root. Or you???d have an inode that icheck reported dups. ncheck would try to reconcile an inumber into an absolute path. > > > > > > > > After a while a program called fsdb came around that allowed you to poke at the various file system structures. We didn???t use it much because by the time we had it, fsck was fast on its heals. > > > > -- > --- > Larry McVoy lm at mcvoy.com http://www.mcvoy.com/lm From random832 at fastmail.com Sun May 14 15:52:30 2017 From: random832 at fastmail.com (Random832) Date: Sun, 14 May 2017 01:52:30 -0400 Subject: [TUHS] Old Unix vulnerabilities In-Reply-To: References: Message-ID: <1494741150.709655.975854064.7B3C6781@webmail.messagingengine.com> On Sat, May 13, 2017, at 19:34, Dave Horsfall wrote: > OK, I'll kick it off. > > A beauty in V6 (and possibly V7) was discovered by the kiddies in Elec > Eng; by sending a signal with an appropriately-crafted negative value (as > determined from inspecting ) you could overwrite u.u_uid with > zero... Needless to say I scrambled to fix that one on my 11/40 network! V7 fixes it by changing the if(sig >= NSIG) in psignal to cast it to unsigned. Kill still doesn't validate its parameter until SysIII and 4BSD. PWB1 does not have the fix. From random832 at fastmail.com Sun May 14 16:11:07 2017 From: random832 at fastmail.com (Random832) Date: Sun, 14 May 2017 02:11:07 -0400 Subject: [TUHS] Old Unix vulnerabilities In-Reply-To: References: Message-ID: <1494742267.712158.975859880.2F405BBE@webmail.messagingengine.com> On Sat, May 13, 2017, at 19:34, Dave Horsfall wrote: > OK, I'll kick it off. Oh, and since we're doing this... I independently noticed a buffer overflow vulnerability in mkdir, and later discovered that someone else had actually published a working exploit for the same bug... in 2004. http://archive.cert.uni-stuttgart.de/bugtraq/2004/06/msg00035.html From scj at yaccman.com Sun May 14 16:12:45 2017 From: scj at yaccman.com (Steve Johnson) Date: Sat, 13 May 2017 23:12:45 -0700 Subject: [TUHS] C declarations. In-Reply-To: <20170514042411.93CCB124AEA4@mail.bitblocks.com> Message-ID: I can't resist jumping in here.   My main point is that system coding is a rather different beast than most application programming.  An operating system must be able to handle memory allocation requests from a user without knowing any details of what is requested.  Dennis explicitly left out a heap and garbage collection code in C because he didn't want OS code to be burdened with it.  Similarly with a "real" string type (with concatenation, etc.). An example that speaks to me is that of creating a symbol table.  This isn't kernel programming, but consider what is needed: * A structure that holds the individual table entries * A tree of lookup tables that reflects the scoping rules of the language -- several tables might need to be searched to find an entry * A function that looks up a name in a scope, returns the entry if it's defined, and adds a new entry in the appropriate scope if it isn't there yet. Statistically a majority of the calls do not change the symbol table, but some do, and you can't tell in advance.  And in the same way, many of the possible scopes where a variable might reside need not be searched. So how much of this table should we copy into the lookup function?  The answer seems obvious to me -- none of it... Before Unix, Dennis and I and several other people worked on a symbolic algebra system written in FORTRAN.  Dennis wrote a dynamic storage allocator, which was a real feat in FORTRAN, complete with garbage collector.  The goal was to have the program work on the 6 major manufacturers' FORTRANs.  We got it up and running on many of the systems, but did not have easy access to OS 360.   Finally, we were granted the third shift on the OS 360, and ran our program (that had become rather solid by this time...).   It was a disaster -- the brogram ran for a few second and then the OS dies and could not be restarted (turns out the core dump area allotted on the disc was not big enough to hold our program).   The problem turned out to be that the IBM compiler did copy in/copy out argument passing, while all the other compilers passed by reference.  If you passed in an argument  that was dynamically allocated, and the system did a garbage collection, all the elements that were still live were moved around but kept their contents intact... until the function returned, and tried to copy the changed value out _where that block used to be at the time of the call._ It took us about 6 months to rewrite the system so we didn't do that -- we had to allocate everything that would be needed first, then do the work, and then free things... None of us were fans of copy in/copy out after that experience... Steve ----- Original Message ----- From: "Bakul Shah" To:"Lawrence Stewart" Cc:"tuhs" Sent:Sat, 13 May 2017 21:24:11 -0700 Subject:Re: [TUHS] C declarations. On Sat, 13 May 2017 21:59:57 EDT Lawrence Stewart wrote: > > * - But I have never been able to remember the syntax for function pointers. > I always "man qsort" to refresh my memory. The way I remember: given *x[] or *x(), x /sticks/ to the /right/ first. Thus int *x[]; // x is array of ptr to int int *x(); // x is a function returning ptr to int If you don't want a var to stick to its right, separate using using parentheses. int (*x)[]; // x is a ptr to array of ints int (*x)(); // x is a ptr to function returning int int *(*x)[]; // x is a ptr to array of ptrs to int This sort of also works for multiple variables in one declaration. Given int *x, y; // * /sticks/ to the right (x) first, so not available for y. Now if they'd allowed parenthesizing the type, as in (int *)x, y; we would see that both x & y are of type int *. That would've also allowed declaring multiple vars in one declaration where the previous rule applies! (int*)(*f,*g)(); Part of the confusion is * is tacked on at the front while [] & () at the back of a variable. Someone (Chandy?) had proposed unifying this syntax but it didn't go anywhere. I think the author used @ in a suffix place though I don't recall any other details. But if you put the var all the way to the right and read a declaration right to left, it works: int*[] x // x is a array of ptr to ints int() f // f is a function returning it int*()* f // f is a ptr to function returning ptr to int int*()* f,g // f & g are ptrs to function returning ptr to int This is only slightly weird if you are used to the current syntax but you can easily retrain yourself. IMHO, where evolution of C, Scheme and most programming languages goes wrong is doing evolutionary design by committee. Most users grasp what is easier to use vs what is hard but they don't have the aesthetic sense or imagination or training to make things simple. And in large groups popularity or force of personality or some other irrelevant attribute wins not aesthetics. Language evolution IMHO must be done by benevolent dictatorship or a guild of like minded people who have worked together for a long time! Go seems to have adapted that style.... -------------- next part -------------- An HTML attachment was scrubbed... URL: From bakul at bitblocks.com Sun May 14 16:48:20 2017 From: bakul at bitblocks.com (Bakul Shah) Date: Sat, 13 May 2017 23:48:20 -0700 Subject: [TUHS] C declarations. In-Reply-To: References: Message-ID: <6D3DCC45-B2C6-4374-83EE-F745C0AF7F36@bitblocks.com> So the problem was your assumptions didn't match IBM's! Most likely the FORTRAN standard /allowed/ pass by value/result instead of pass by ref. None of the changes I suggested in my previous email (that focused on arrays in C) would have required a GC or heap. It would've increased programmer convenience and reduced bugs though! > On May 13, 2017, at 11:12 PM, Steve Johnson wrote: > > I can't resist jumping in here. My main point is that system coding is a rather different beast than most application programming. An operating system must be able to handle memory allocation requests from a user without knowing any details of what is requested. Dennis explicitly left out a heap and garbage collection code in C because he didn't want OS code to be burdened with it. Similarly with a "real" string type (with concatenation, etc.). > > An example that speaks to me is that of creating a symbol table. This isn't kernel programming, but consider what is needed: > A structure that holds the individual table entries > A tree of lookup tables that reflects the scoping rules of the language -- several tables might need to be searched to find an entry > A function that looks up a name in a scope, returns the entry if it's defined, and adds a new entry in the appropriate scope if it isn't there yet. > Statistically a majority of the calls do not change the symbol table, but some do, and you can't tell in advance. And in the same way, many of the possible scopes where a variable might reside need not be searched. > > So how much of this table should we copy into the lookup function? The answer seems obvious to me -- none of it... > > Before Unix, Dennis and I and several other people worked on a symbolic algebra system written in FORTRAN. Dennis wrote a dynamic storage allocator, which was a real feat in FORTRAN, complete with garbage collector. The goal was to have the program work on the 6 major manufacturers' FORTRANs. We got it up and running on many of the systems, but did not have easy access to OS 360. Finally, we were granted the third shift on the OS 360, and ran our program (that had become rather solid by this time...). It was a disaster -- the brogram ran for a few second and then the OS dies and could not be restarted (turns out the core dump area allotted on the disc was not big enough to hold our program). > > The problem turned out to be that the IBM compiler did copy in/copy out argument passing, while all the other compilers passed by reference. If you passed in an argument that was dynamically allocated, and the system did a garbage collection, all the elements that were still live were moved around but kept their contents intact... until the function returned, and tried to copy the changed value out where that block used to be at the time of the call. > > It took us about 6 months to rewrite the system so we didn't do that -- we had to allocate everything that would be needed first, then do the work, and then free things... > > None of us were fans of copy in/copy out after that experience... > > Steve > > > ----- Original Message ----- > From: "Bakul Shah" > To:"Lawrence Stewart" > Cc:"tuhs" > Sent:Sat, 13 May 2017 21:24:11 -0700 > Subject:Re: [TUHS] C declarations. > > > On Sat, 13 May 2017 21:59:57 EDT Lawrence Stewart wrote: > > > > * - But I have never been able to remember the syntax for function pointers. > I always "man qsort" to refresh my memory. > > The way I remember: > > given *x[] or *x(), x /sticks/ to the /right/ first. Thus > > int *x[]; // x is array of ptr to int > int *x(); // x is a function returning ptr to int > > If you don't want a var to stick to its right, separate using > using parentheses. > > int (*x)[]; // x is a ptr to array of ints > int (*x)(); // x is a ptr to function returning int > int *(*x)[]; // x is a ptr to array of ptrs to int > > This sort of also works for multiple variables in one declaration. Given > > int *x, y; // * /sticks/ to the right (x) first, so not available for y. > > Now if they'd allowed parenthesizing the type, as in > > (int *)x, y; > > we would see that both x & y are of type int *. That would've also allowed > declaring multiple vars in one declaration where the previous rule applies! > > (int*)(*f,*g)(); > > Part of the confusion is * is tacked on at the front while [] > & () at the back of a variable. Someone (Chandy?) had > proposed unifying this syntax but it didn't go anywhere. I > think the author used @ in a suffix place though I don't > recall any other details. But if you put the var all the way > to the right and read a declaration right to left, it works: > > int*[] x // x is a array of ptr to ints > int() f // f is a function returning it > int*()* f // f is a ptr to function returning ptr to int > int*()* f,g // f & g are ptrs to function returning ptr to int > > This is only slightly weird if you are used to the current syntax > but you can easily retrain yourself. > > IMHO, where evolution of C, Scheme and most programming > languages goes wrong is doing evolutionary design by > committee. Most users grasp what is easier to use vs what is > hard but they don't have the aesthetic sense or imagination or > training to make things simple. And in large groups popularity > or force of personality or some other irrelevant attribute > wins not aesthetics. Language evolution IMHO must be done by > benevolent dictatorship or a guild of like minded people who > have worked together for a long time! Go seems to have > adapted that style.... -------------- next part -------------- An HTML attachment was scrubbed... URL: From dfawcus+lists-tuhs at employees.org Sun May 14 22:55:43 2017 From: dfawcus+lists-tuhs at employees.org (Derek Fawcus) Date: Sun, 14 May 2017 13:55:43 +0100 Subject: [TUHS] The evolution of Unix facilities and architecture In-Reply-To: <1038e644-cef1-9424-f6fa-288941033bff@gmail.com> References: <20170513004423.F3A9C18C099@mercury.lcs.mit.edu> <1494636703.1615694.975085000.13626EAC@webmail.messagingengine.com> <8B3FFD9C-FA5F-434F-A18E-FF59215C6DDD@quintile.net> <1038e644-cef1-9424-f6fa-288941033bff@gmail.com> Message-ID: <20170514125543.GA81508@cowbell.employees.org> On Sat, May 13, 2017 at 01:19:42PM -0400, William Pechter wrote: > When VMS went to 3.6 or so a friend of mine was almost fired by DEC for > randomly testing boxes looking > to see DEC's internal boxes weren't running System/Manager, > Field/Service and UETP/UETP User/password > combinations. Those default account combinations were still being used to gain access to VMS systems in the '87-'89 time frame; although user/password was less interesting by itself, being an unpriviledged account. DF From dfawcus+lists-tuhs at employees.org Sun May 14 23:14:44 2017 From: dfawcus+lists-tuhs at employees.org (Derek Fawcus) Date: Sun, 14 May 2017 14:14:44 +0100 Subject: [TUHS] C declarations. In-Reply-To: <1494702324.1901172.975588504.1BA76452@webmail.messagingengine.com> References: <015401d2caa0$79762650$6c6272f0$@ronnatalie.com> <68E8DC0A-D1B8-4FF0-AD26-ACDC57E308AF@pobox.com> <20170511223232.GM4341@mcvoy.com> <015b01d2caa7$c658c020$530a4060$@ronnatalie.com> <20170513012415.GZ4341@mcvoy.com> <025701d2cb92$ec8182a0$c58487e0$@ronnatalie.com> <20170513122050.GF9980@yeono.kjorling.se> <1494702324.1901172.975588504.1BA76452@webmail.messagingengine.com> Message-ID: <20170514131444.GB81508@cowbell.employees.org> On Sat, May 13, 2017 at 03:05:24PM -0400, Random832 wrote: > On Sat, May 13, 2017, at 08:20, Michael Kjörling wrote: > > Correct me if I am wrong, but _pass by value_ as opposed to _pass by > > reference_ requires making a copy, no? That's the whole point, to > > allow the callee to poke at the value it is given at will. > > If that were the issue, the callee could simply automatically make the > copy if-and-only-if it needs it. I recall reading a proposed ABI for C on MIPS which would have done something similar for passing structs by value, namely pass a pointer but allocate local storage space in the callee. If the callee ever assigned to the passed struct, it was to copy the actual values, then update its pointer. I don't think that ABI was ever implemented. DF From doug at cs.dartmouth.edu Mon May 15 00:11:34 2017 From: doug at cs.dartmouth.edu (Doug McIlroy) Date: Sun, 14 May 2017 10:11:34 -0400 Subject: [TUHS] C declarations. Message-ID: <201705141411.v4EEBYjv021607@coolidge.cs.Dartmouth.EDU> > Are there languages that copy arrays in function calls defaultly? > Pascal is an example. Pascal's var convention, where the distinction between value and reference is made once and for all for each argument of each function, is sound. The flexibility of PL/I, where the distinction is made at every call (parenthesize the name to pass an array by value) is finicky, though utterly general. > Where is all that [memory] going to come from if you pass a > large array on a memory-constrained system of specs common back in the > days when C was designed Amusingly, under the customary linkage method in the even earlier days when Fortran was designed, pass-by-reference entailed a big overhead that could easily dominate pass-by-value for small arrays. [In the beginning, when CPUs had only one register, subroutine preambles plugged the reference into every mention of that variable throughout the body of the subroutine. This convention persisted in Fortran, which was designed for a machine with three index registered. Since reference variables were sometimes necessary (think of swap(a,b) for example) they were made standard.] Doug From cym224 at gmail.com Mon May 15 00:48:49 2017 From: cym224 at gmail.com (Nemo) Date: Sun, 14 May 2017 10:48:49 -0400 Subject: [TUHS] C declarations. In-Reply-To: References: <015401d2caa0$79762650$6c6272f0$@ronnatalie.com> <68E8DC0A-D1B8-4FF0-AD26-ACDC57E308AF@pobox.com> <20170511223232.GM4341@mcvoy.com> <015b01d2caa7$c658c020$530a4060$@ronnatalie.com> <20170513012415.GZ4341@mcvoy.com> <025701d2cb92$ec8182a0$c58487e0$@ronnatalie.com> <20170513122050.GF9980@yeono.kjorling.se> <0CF82AC1-E835-4C06-813F-D9EFD2C12290@tfeb.org> <1494702492.1902154.975591160.507FCE4C@webmail.messagingengine.com> Message-ID: On 13 May 2017 at 19:21, Dave Horsfall wrote: > On Sat, 13 May 2017, Random832 wrote: > >> Fortran, for the record, passes nearly everything by reference [...] > > And if you passed a constant and the local argument was modified, so was > the (single instance of) constant... Probably depends on the implementation. VAX-FORTRAN passed by reference (but not on the stack). Off-topic: I recall a working function that crashed when fed a constant string. Spelunking revealed that someone had replaced a letter-by-letter case-insensitive comparison with a "faster version" (written in VAX assembler, no less) that converted both arguments to upper case and then block-compared. N. From usotsuki at buric.co Mon May 15 00:58:18 2017 From: usotsuki at buric.co (Steve Nickolas) Date: Sun, 14 May 2017 10:58:18 -0400 (EDT) Subject: [TUHS] C declarations. In-Reply-To: <201705141411.v4EEBYjv021607@coolidge.cs.Dartmouth.EDU> References: <201705141411.v4EEBYjv021607@coolidge.cs.Dartmouth.EDU> Message-ID: On Sun, 14 May 2017, Doug McIlroy wrote: >> Are there languages that copy arrays in function calls defaultly? > >> Pascal is an example. > > Pascal's var convention, where the distinction between value > and reference is made once and for all for each argument of > each function, is sound. The flexibility of PL/I, where the > distinction is made at every call (parenthesize the name to > pass an array by value) is finicky, though utterly general. QuickBasic uses the ( ) trick too. >> Where is all that [memory] going to come from if you pass a >> large array on a memory-constrained system of specs common back in the >> days when C was designed > > Amusingly, under the customary linkage method in the even earlier > days when Fortran was designed, pass-by-reference entailed a big > overhead that could easily dominate pass-by-value for small arrays. > > [In the beginning, when CPUs had only one register, subroutine > preambles plugged the reference into every mention of that variable > throughout the body of the subroutine. This convention persisted > in Fortran, which was designed for a machine with three index > registered. Since reference variables were sometimes necessary > (think of swap(a,b) for example) they were made standard.] > > Doug > Three! O_O That's a lot, if you're used to the 6502 and its mere *two* index registers. -uso. From clemc at ccc.com Mon May 15 03:40:48 2017 From: clemc at ccc.com (Clem Cole) Date: Sun, 14 May 2017 13:40:48 -0400 Subject: [TUHS] The evolution of Unix facilities and architecture In-Reply-To: <20170514043024.pezg5eqslpcqsd5m@thunk.org> References: <20170511140729.2262B18C09A@mercury.lcs.mit.edu> <013b01d2ca96$6901b370$3b051a50$@ronnatalie.com> <20170511222547.GJ4341@mcvoy.com> <20170514043024.pezg5eqslpcqsd5m@thunk.org> Message-ID: Ted -- thank you -- excellent write up. Love it and I could not agree more!! Your 'worse is better' is the same idea as what is 'good enough,' an argument I used to have at DEC, where being 'prefect' cost years and in the end - and we lost because of it. FWIW: I put it slightly differently, make sure you pick a couple of things that matter, and nail them... be the best on those >>few<< items but then the rest needs to be 'good enough' and in time you can make those parts better. But if you wait for all parts to be great or worse 'perfect' - it doesn't matter -- this says an ex-Alpha guy now working in INTEL*64 -- sigh... BTW: Not to quibble, but you might also remember, traditional UNIX took the same path as you did. Ken's v6/v7 FS were not the great about write ordering either. Remember FFS is replacing Ken's work 15 years down the road. Kirk did not do all the careful ordered write stuff until after George Gobble taught us all how to, which was a few years later. When Kirk implemented FFS, it was after the Purdue patches had been released to make Unix's original FS more 'reliable' - and yes - Ted (Kowalski) and my version of the original fsck was not nearly as careful as you were year later. But again, we were a huge step forward from what had been at the time. That said -- your stuff, as Larry has pointed, was rock solid and in practice 'just worked.' Certainly post ext3, I do not have memory of losing any real user data on any of Linux boxes, and an error I made has definitely be the cause of them to crashing over the years ;-) Also, WRT to making the HW properly, and at lot of PC HW being trash -- yup - which comes back to the what is good enough issue. DEC did the same thing and for long time in as SGI in making rock solid HW. Heck DEC, somewhat cornered the SCSI disk business for the mid-range and upper end rack world in the 90s'. Like Sgi's and Sun's OSses of the day, Tru64 has very good DMA controllers under the covers that were hardened and lab tested for corner cases. In fact, it is one of the reasons why while Tru64 could detect an Adaptec controller @ boot up and actually use it (I had on my workstation), it was not officially in the 'SPD' as supported device because the HW failed as you described and TruCluster's in particular could not make a proper DLM do the failure modes of the Apaptec (now as I used to point out to the marketing and HW weenies, none sane person was going to put a $150 SCSI controller in their $.5M TruCluster system - so we could have allowed it, just not allowed it some configs - made it clear in the SPD -- Adaptec on in configs XXX). As a result, the issue places out this way... an then DEC's VPs used to say you could not make & sell an Alpha for under $5K (one person in particular who I will leave nameless - those who were there - all know who I refer). My last act before I left DEC/Compaq for Paceline, was to make the $1K alpha using an $799 (end user) Compaq system with the K7 on the motherboard swapped with an EV6 and some mechanical shims, Adaptec SCSI BT (I still have the motherboard @ home, and the EV6 in on my desk at Intel). It was built using PC parts - case, power supply et al. The key was the Alpha @5K was a better physically built system than the $799 based PC -- but who cared ... your closing para WRT to WiFi and PCMCIA summed up the issue pretty well. Clem On Sun, May 14, 2017 at 12:30 AM, Theodore Ts'o wrote: > On Thu, May 11, 2017 at 03:25:47PM -0700, Larry McVoy wrote: > > This is one place where I think Linux kicked Unix's ass. And I am not > > really sure how they did it, I have an idea but am not positive. Unix > > file systems up through UFS as shipped by Sun, were all vulnerable to > > what I call the power out test. Untar some big tarball and power off > > the machine in the middle of it. Reboot. Hilarity ensues (not). > > > > You were dropped into some stand alone shell after fsck threw up its > > hands and it was up to you to fix it. Dozens and dozens of errors. > > It was almost always faster to go to backups because figuring that > > stuff out, file by file (which I have done more than once), gets you > > to the point that your run "fsck -y" and go poke at lost+found when > > fsck is done, realize that there is no hope, and reach for backups. > > > > Try the same thing with Linux. The file system will come back, starting > > with, I believe, ext2. > > > > My belief is that Linux orders writes such that while you may lose data > > (as in, a process created a file, the OS said it was OK, but that file > > will not be in the file system after a crash), but the rest of the file > > system will be consistent. I think it's as if you powered off the > > machine a few seconds earlier than you actually did, some stuff is in > > flight and until they can write stuff out in the proper order you may > > lose data on a hard reset. > > So the story is a bit complicated here, and may be an example of > "worse is better" --- which is ironically one of those things which is > used as an explanation for why BSD/Unix won ever though the Lisp was > technically superior[1] --- but in this case, it's Linux that did > something "dirty", and BSD that did something that was supposed to be > the "better" solution. > > [1] https://www.jwz.org/doc/worse-is-better.html > > So first let's talk about ext2 (which indeed, does not have file > system journalling; that came in ext3). The BSD Fast File System goes > to a huge amount of effort to make sure that writes are sent to the > disk in exactly the right order so that fsck can actually fix things. > This requires that the disk not reorder writes (e.g., write caching is > disabled or in write-through mode). Linux, in ext2, didn't bother > with trying to get the write order correct at all. None. Nada. Zip. > Writes would go out in whatever order dictated by the elevator > scheduler, and so on a power failure or a kernel crash, the order in > which metadata writes would be sent to the disk was completely > unconstrained. > > Sounds horrible, right? In many ways, it was. And I lost count of > how often NetBSD and FreeBSD users would talk about how primitive and > horrible ext2 was in comparison to FFS, which had all of this > excellent engineering work to make sure writes happened in the correct > order such that fsck was guaranteed to always be able to fix things. > > So why did Linux get away with it? When I wrote the fsck for ext2, I > knew that anything can and would happen, so it was implemented so that > it was extremely paranoid about not ever losing any data. And if > there was a chance that an expert could recover the data, e2fsck would > stop and ask the system administrator to take a look. In the case > that the user ran with fsck -y, the default was drop files into > lost+found, where as in some cases with the FFS fsck, it "knew" that > in a particular case, the order in which writes were staged out the > right thing to do was to let the unlink complete, so it would let the > refcount go to zero, or stay at zero. > > The other thing that we did in Linux is that I made sure we had a > highly functional "debugfs" tool. This tool served two purposes. The > first was it made it very easy for me to creat a regression test suite > for fsck. As far as I know, none of the other major file systems at > the time had an fsck with a regression test suite --- and I was > religious about adding tests as I added functionality, and as I fixed > bugs. The debugfs tool made it easy for me to create test case file > systems that was corrupted in various interesting ways. The other use > of debugfs was that it made it easy for experts to do file system > recovery after a crash, if there was some really precious file that > they needed to try to recover. > > So this is why this is a great example of "worse is better". In Linux, > ext2 was ***incredibly*** sloppy about how it handled write ordering > --- it didn't do anything at all. But as a consequence we developed > tools that were extremely good to compensate, and in practice, it was > extremely rare (although it did happen on occasion) that files would > get lost or the file systme could end up in a state where fsck would > not be able to recover without manual intervention by a system > administrator using debugfs. > > But the other thing to note here is that in the PC era, most disk > drives ran with write caching enabled, with writeback caching so that > the hard drive could do its own elevator shceduling. So having a file > system that very carefully scheduled writes to make sure they happened > in the write order didn't help you a *bit* unless you configured your > hard drive to disable writeback caching --- at which point you would > take a massive speed hit. > > This is ultimately also one weaknesses of Soft Updates --- it requires > that you disable writeback caching, since it works by letting the OS > control the order in which writes hit stable storage. With > journalling you don't have to do that; but the tradeoff is that when > you do a journal commit, you need typically two cache flush > operations. (Or a cache flush followed by a FUA write of the commit > block, if the disk supports FUA.) > > > > There is another example of how Linux embraced the "worse is better" > philosophy in ext3, and that has to do with how we do journalling. > The sophisticated way to do journalling is to do logical journalling. > This is where what you write in the in journal is "set bit XXX in the > allocation bitmap", or "update the mtime to YYYY". And in this way, > you can batch multiple file system operations into a single block > written to the journal. Solaris/UFS and Irix uses this much more > sophisticated form of journalling. (Actually, older versions of > Solaris did use volume-level journalling, which is basically what > ext3/ext4 uses, but they upgraded to the much more "right", more > advanced thing, which is logical journalling.) > > Ext3 uses phyiscal, or volume-level journalling. This journalling > works on the block level --- so if we flip a bit in an allocation > bitmap, we log the entire 4k block to the journal. By default, we > only do a journal commit every five seconds (unless an fsync happens > first), so there could be multiple changes to a single inode table > blocks that can be batched together, but it's still true that for a > given metadata-heavy workload, a file system which uses logical > journalling will tend require many fewer blocks written to the journal > than a file system such as ext3/ext4 which uses physical block > journalling. > > Why did Linux get away with it? Number one, most workloads aren't > really modify metadata all _that_ intensively, and 12k of sequential > writes versus 32k of sequential writes doesn't actually take that much > more time. Secondly, Ted's law of PC-class hardware ("most PC-class > hardware is crap") comes into play, and turns physical journalling > into an advantage. PC class hardware tends not to have power fail > interrupts, and when power drops, and the voltage levels on the power > rails start drooping, DRAM tends to go insane and starts returning > garbage long before the DMA engine and the hard drive stops > functioning. > > So if your system is doing logical journalling, after the file system > commits a transaction, it will start writing the inode table block to > the permanent location on disk. If at that point you get a power > drop, garbage can get written to the inode table block, and if the > file system is using logical journalling, on reboot the mtime field > can get updated from the logical journal --- but the rest of the inode > table block is still garbage. > > In contrast, since ext3 was using physical block journalling, even if > various metadata blocks get corrupted due to writes from failing DRAM > during a power drop, when we replay the journal, this will restore the > entire metadata block, and Things Just Worse. > > I have talked to an XFS engineer from SGI, and this was definitely a > thing which SGI discovered the hard way. After they discovered this > problem, they added extra capacitors to the power supply, added a > power fail interrupt, and taught Irix so that when the power fail > interrupt was triggered, it would frantically cancel DMA transfers in > order to avoid this problem. I do not know how many of the other > Legacy Unix systems figured out this failure mode --- and I can't > claim that we were brilliant enough to design a system to avoid this > problem. It just so happened that the brute-force design that we > chose was very well suited for crappy (but way cheaper than a Sun Fire > E10k :-) PC-class hardware. > > > I copied Ted, who had his fingers deep in that code, maybe he can correct > > me where I got it wrong. Details aside, I think this is a place where > > Linux moved the state of the art significantly forward. There are other > > places but this one is a big deal IMHO, maybe the biggest deal. > > So I'm not really sure we can claim to have "moved the state of the > art". There certainly wasn't any brilliant computer science > innovations here. That sort of thing is more like Soft Updates, of > which Valerie Aurora (formerly Henson) once wrote, > > "I've read this paper at least 15 times, and each time I when get > to page 7, I'm feeling pretty good and thinking, "Yeah, okay, I > must be smarter now than the last time I read this because I'm > getting it this time," - and then I turn to page 8 and my head > explodes." --- https://lwn.net/Articles/339337/ > > I will be the first to admit that with ext2/ext3/ext4, especially in > the early days, it was much more about brute force engineering, and > regression testing, and much less about "moving the state of the art". > Certainly those of us who were working on Linux weren't trying to get > papers published in peer reviewed journals or conferences! (And I've > always thought that Greg Ganger was _way_ smarter than I. :-) > > And if the Lisp Machine hackers looked down on BSD, and complained > that BSD adopted the "Worse is Better" philosophy, while Lisp strived > for the true, elegant, Correct technical solution, it's perhaps > especially interesting to consider that if anything, Linux was an even > more radical example of the "Worse is Better" philosophy. > > Cheers, > > - Ted > > P.S. There is yet another example of "Worse is Better" in how Linux > had PCMCIA support several years before FreeBSD/NetBSD. However, if > you ejected a PCMCiA card in a Linux system, there was a chance (in > practice it worked out to be about in 1 in 5 times for a WiFI card, in > my experience) that the system would crash. The *BSD's took a good > 2-3 years longer to get PCMCIA support, but when they did, it was rock > solid. Of course, if you are a laptop user, and are happy to keep > your 802.11 PCMCIA card permanently installed, guess which OS you were > likely to prefer --- "sloppy but works, mostly", or "it'll get there > eventually, and will be rock solid when it does, but zip, nada, right now"? > > > > > > --lm > > > > On Thu, May 11, 2017 at 04:37:29PM -0400, Ron Natalie wrote: > > > I remember the pre-fsck days. It was part of my test to become an > operator at the UNIX site at JHU that I could run the various manual checks. > > > > > > The V6 file system wasn???t exactly stable during crashes (lousy > database behavior), so there was almost certainly something to clean up. > > > > > > > > > > > > The first thing we???d run was icheck. This runs down the superblock > freelist and all the allocated blocks in the inodes. If there were > missing blocks (not in a file or the free list), you could use icheck ???s > > > > > > to rebuild it. Similarly, if you had duplicated allocations in the > freelist or between the freelist and a single file. Anything more > complicated required some clever patching (typically, we???d just mount > readonly, copy the files, and then blow them away with clri). > > > > > > > > > > > > Then you???d run dcheck. As mentioned dcheck walks the directory > path from the top of the disk counting inode references that it reconciles > with the link count in the inode. Occasionally we???d end up with a 0-0 > inode (no directory entires, but allocated???typically this is caused by > people removing a file while it is still open, a regular practice of some > programs for their /tmp files.). clri again blew these away. > > > > > > > > > > > > Clri wrote zeros all over the inode. This had the effect of wiping > out the file, but it was dangerous if you got the i-number wrong. We > replaced it with ???clrm??? which just cleared the allocated bit, a lot > easy to reverse. > > > > > > > > > > > > If you really had a mess of a file system, you might get a piece of > the directory tree broken off from a path to the root. Or you???d have an > inode that icheck reported dups. ncheck would try to reconcile an inumber > into an absolute path. > > > > > > > > > > > > After a while a program called fsdb came around that allowed you to > poke at the various file system structures. We didn???t use it much > because by the time we had it, fsck was fast on its heals. > > > > > > > -- > > --- > > Larry McVoy lm at mcvoy.com > http://www.mcvoy.com/lm > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jnc at mercury.lcs.mit.edu Mon May 15 07:44:19 2017 From: jnc at mercury.lcs.mit.edu (Noel Chiappa) Date: Sun, 14 May 2017 17:44:19 -0400 (EDT) Subject: [TUHS] The evolution of Unix facilities and architecture Message-ID: <20170514214419.EFC1618C0A2@mercury.lcs.mit.edu> > From: Random832 > Ah. There's the other piece. You start the SUID program under the > debugger, and ... it simply starts it non-suid. *However*, in the > presence of shared text ... you can make changes to the text image > ... which will be reused the *next* time it is started *without* the > debugger. So I actually tried to do this (on a V6 system running on an emulator), after whipping up a tiny test program (which prints "1", and the real and current UIDs): the plan was to patch it to print a different number. However, after a variety of stubbed toes and hiccups (gory details below, if anyone cares), including a semi-interesting issue with the debugger and pure texts), I'm punting: when trying to set a breakpoint in a pure text, I get the error message "Can't set breakpoint", which sort of correlates with the comment in the V6 sig$ptrace(): "write user I (for now, always an error)". So it's not at all clear that the technique we thought would work would, in fact, work - unless people weren't using a stock V6 system, but rather one that had been tweaked to e.g. allow use of debuggers on pure-text programs (including split I+D). It's interesting to speculate on what the 'right' fix would be, if somehow the techique above did work. The 'simple' fix, on systems with a PWB1-line XWRIT flag, would be to ignore SETUID bits when doing an exec() of a pure text that had been modified. But probably 'the' right fix would be to give someone debugging a pure-text program their own private copy of the text. (This would also prevent people who try to run the program from hitting breakpoints while it's being debugged. :-) But anyway, it's clear that back when, when I thought I'd found the bug, I clearly hadn't - which is why when I looked into the source, it looked like it had been 'already' been fixed. (And why Jim G hemmed and hawed...) But I'm kind of curious about that mod in PWB1 that writes a modified pure text back to the swap area when the last process using it exits. What was the thinking behind that? What's the value to allowing someone to patch the in-core pure text, and then save those patches? And there's also the 'other people who try and run a program beind debugged are going to hit breakpoints' issue, if you do allow writing into pure texts... Noel -------- For the gory details: to start with, attempting to run a pure-text program (whether SUID or not) under the debugger produced a "Can't execute {program-name} Process terminated." error message. 'cdb' is printing this error message just after the call to exec() (if that fails, and returns). I modified it to print the error number when that happens, and it's ETXTBSY. I had a quick look at the V6 source, to see if I could see what the problem is, and it seems to be be (in sys1$exec()): if(u.u_arg[1]!=0 && (ip->i_flag&ITEXT)==0 && ip->i_count!=1) { u.u_error = ETXTBSY; goto bad; } What that code does is a little obscure; I'm not sure I understand it. The first term checks to see if the size of the text segment is non-zero (which it is not, in both 0407 and 0410 files). The second is, I think, looking to see if the inode is marked as being in use for a pure text (which it isn't, until later in exec()). The third checks to make sure nobody else is using the file. So I guess this prevents exec() of a file which is already open, and not for a pure text. (Why this is the Right Thing is not instantly clear to me...) Anyway, the reason this fails under 'cdb' is that the debugger already has it open (to be able to read the code). So I munged the debugger to close it before doing the exec(), and then the error went away. Then I ran into a long series of issues, the details of which are not at all interesting, connected with the fact that the version of 'cdb' I was using (one I got off a Tim Shoppa modified V6 disk) doesn't correspond to either of the sources I have for 'cdb'. When I switched to the latest source (so I could fix the issue above), it had some bug where it wouldn't work unless there was a 'core' file. But eventually I kludged it enough to get the 'can't set breakpoints' message, at which point I threw in the towel. From dave at horsfall.org Mon May 15 08:12:04 2017 From: dave at horsfall.org (Dave Horsfall) Date: Mon, 15 May 2017 08:12:04 +1000 (EST) Subject: [TUHS] The evolution of Unix facilities and architecture In-Reply-To: <20170514125543.GA81508@cowbell.employees.org> References: <20170513004423.F3A9C18C099@mercury.lcs.mit.edu> <1494636703.1615694.975085000.13626EAC@webmail.messagingengine.com> <8B3FFD9C-FA5F-434F-A18E-FF59215C6DDD@quintile.net> <1038e644-cef1-9424-f6fa-288941033bff@gmail.com> <20170514125543.GA81508@cowbell.employees.org> Message-ID: On Sun, 14 May 2017, Derek Fawcus wrote: > > to see DEC's internal boxes weren't running System/Manager, > > Field/Service and UETP/UETP User/password combinations. > > Those default account combinations were still being used to gain access > to VMS systems in the '87-'89 time frame; although user/password was > less interesting by itself, being an unpriviledged account. Wasn't there also Guest/Guest as well? Admittedly it would also be pretty boring, but nonetheless still a toe-hold. -- Dave Horsfall DTM (VK2KFU) "Those who don't understand security will suffer." From ron at ronnatalie.com Mon May 15 09:06:47 2017 From: ron at ronnatalie.com (Ron Natalie) Date: Sun, 14 May 2017 19:06:47 -0400 Subject: [TUHS] C declarations. In-Reply-To: <6D3DCC45-B2C6-4374-83EE-F745C0AF7F36@bitblocks.com> References: <6D3DCC45-B2C6-4374-83EE-F745C0AF7F36@bitblocks.com> Message-ID: <030d01d2cd06$c44e2840$4cea78c0$@ronnatalie.com> You all are missing the point as to what the cost of passing arrays by value or what other languages do. C is implicitly a pass/return by value language for EVERY SINGLE POSSIBLE DATATYPE WITH THE EXCEPTION OF ARRAYS. Arrays are special for no reason other than it was thought to costly to do it any otherway. In fact, had it not been kludged up into a pseudo-pass-by-reference, nobody would have probably been the wiser. It was too hard to pass/return structs by value either in the initial compilers, but they fixed it to work exactly that way. So we have this array kludge that works differently than everything else. Further, due to the kludgosity, it's not even exactly type safe. -------------- next part -------------- An HTML attachment was scrubbed... URL: From krewat at kilonet.net Mon May 15 09:34:01 2017 From: krewat at kilonet.net (Arthur Krewat) Date: Sun, 14 May 2017 19:34:01 -0400 Subject: [TUHS] C declarations. In-Reply-To: <030d01d2cd06$c44e2840$4cea78c0$@ronnatalie.com> References: <6D3DCC45-B2C6-4374-83EE-F745C0AF7F36@bitblocks.com> <030d01d2cd06$c44e2840$4cea78c0$@ronnatalie.com> Message-ID: <013240e6-cc66-12c7-325d-a1edf4107726@kilonet.net> On 5/14/2017 7:06 PM, Ron Natalie wrote: > array kludge Is it really a kludge? It's a pointer. int array[128]; Want to reference the array in a memcpy? Do: memcpy(destination, array, sizeof(array)); Want to reference an element in the array? array[element #] Want to pass (a reference to) said array to a function? function(array) Not sure what the kludge is. I think C is more consistent when it comes to arrays than a lot of other languages. At least with C you KNOW it's a pointer to the original. -------------- next part -------------- An HTML attachment was scrubbed... URL: From crossd at gmail.com Mon May 15 10:14:47 2017 From: crossd at gmail.com (Dan Cross) Date: Sun, 14 May 2017 20:14:47 -0400 Subject: [TUHS] C declarations. In-Reply-To: <013240e6-cc66-12c7-325d-a1edf4107726@kilonet.net> References: <6D3DCC45-B2C6-4374-83EE-F745C0AF7F36@bitblocks.com> <030d01d2cd06$c44e2840$4cea78c0$@ronnatalie.com> <013240e6-cc66-12c7-325d-a1edf4107726@kilonet.net> Message-ID: On Sun, May 14, 2017 at 7:34 PM, Arthur Krewat wrote: > On 5/14/2017 7:06 PM, Ron Natalie wrote: > > array kludge > > Is it really a kludge? > Yes. It's a pointer. > Of course it is! But that doesn't mean that it SHOULD be, which is the point. int array[128]; > > Want to reference the array in a memcpy? Do: memcpy(destination, array, > sizeof(array)); > > Want to reference an element in the array? array[element #] > > Want to pass (a reference to) said array to a function? function(array) > > Not sure what the kludge is. > The kludge is that it works different than everything else for no really good reason. I think C is more consistent when it comes to arrays than a lot of other > languages. At least with C you KNOW it's a pointer to the original. > Sure! But if someone had told you, back when you first learned C, that it was an array and had value semantics, would you have thought THAT was strange? "If you want a pointer to the array, use the address operator on the array and pass that; that'll behave like a 'foo*' pointer for an array of foo. Most people do this for efficiency." The point is that it's a thought exercise. It's not that folks don't understand how C works now or the semantics of array-decay-to-pointer; it's just that folks can also imagine a world where things had been done differently. That world doesn't exist, but we can still imagine it. - Dan C. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ron at ronnatalie.com Mon May 15 10:23:31 2017 From: ron at ronnatalie.com (Ron Natalie) Date: Sun, 14 May 2017 20:23:31 -0400 Subject: [TUHS] C declarations. In-Reply-To: References: <6D3DCC45-B2C6-4374-83EE-F745C0AF7F36@bitblocks.com> <030d01d2cd06$c44e2840$4cea78c0$@ronnatalie.com> <013240e6-cc66-12c7-325d-a1edf4107726@kilonet.net> Message-ID: <032701d2cd11$7be3f200$73abd600$@ronnatalie.com> It's a pointer. Ø Of course it is! But that doesn't mean that it SHOULD be, which is the point. Actually, it is **Not a pointer**, it gets passed as a pointer, but it is an array on both ends. The implicit conversion of the array name to a pointer to its first item is a different issue. This bothers me less, but it’s still hokie in my opinion. -------------- next part -------------- An HTML attachment was scrubbed... URL: From lm at mcvoy.com Mon May 15 10:40:19 2017 From: lm at mcvoy.com (Larry McVoy) Date: Sun, 14 May 2017 17:40:19 -0700 Subject: [TUHS] C declarations. In-Reply-To: References: <6D3DCC45-B2C6-4374-83EE-F745C0AF7F36@bitblocks.com> <030d01d2cd06$c44e2840$4cea78c0$@ronnatalie.com> <013240e6-cc66-12c7-325d-a1edf4107726@kilonet.net> Message-ID: <20170515004019.GJ29107@mcvoy.com> On Sun, May 14, 2017 at 08:14:47PM -0400, Dan Cross wrote: > The kludge is that it works different than everything else for no really > good reason. I'd be curious what Steve thinks about all this, I know he weighed in a bit, but does he think that Dennis / Brian/ Ken regret this design choice? Personally, I can easily see why they did it. C doesn't really have a lot of behind-the-scenes magic, it can easily be viewed as a pleasant way to do portable code that sits directly on the hardware, no frameworks, no garbage collector, no magic. It's predictable in ways that higher level languages are not. That's one of the things I like about C, I can reason about it. But I suspect it was more basic than that. They were running on 16 bit pretty slow machines and having a pass by value model for arrays just didn't make any sense. *Everyone* would pass by reference so why bother having a pass by value model? If you really, really wanted that you could get it int salaries[100]; int payroll = add_them_up(malloc_and_copy(salaries, sizeof(salaries)); But any sane person in that day and age would go "why on God's green earth would you do that? It's slow. Just don't change the salaries array and you'll be fine". Think about all the str* stuff - you really want to malloc and free that? It would have killed performance. Personally I think they looked at it, said "everyone will use references for arrays anyway, let's make them references by default". I get that it is not clean and pure and doesn't match how other types work, I do see that, but I also see that their choice made a ton of sense at the time. And maybe still does. It's not the same as the other types because the other types are, typically, small. Arrays can be big, really big. --lm From cym224 at gmail.com Mon May 15 11:24:25 2017 From: cym224 at gmail.com (Nemo) Date: Sun, 14 May 2017 21:24:25 -0400 Subject: [TUHS] The evolution of Unix facilities and architecture In-Reply-To: References: <20170513004423.F3A9C18C099@mercury.lcs.mit.edu> <1494636703.1615694.975085000.13626EAC@webmail.messagingengine.com> <8B3FFD9C-FA5F-434F-A18E-FF59215C6DDD@quintile.net> <1038e644-cef1-9424-f6fa-288941033bff@gmail.com> <20170514125543.GA81508@cowbell.employees.org> Message-ID: On 14 May 2017 at 18:12, Dave Horsfall wrote: > On Sun, 14 May 2017, Derek Fawcus wrote: > >> > to see DEC's internal boxes weren't running System/Manager, >> > Field/Service and UETP/UETP User/password combinations. >> >> Those default account combinations were still being used to gain access >> to VMS systems in the '87-'89 time frame; although user/password was >> less interesting by itself, being an unpriviledged account. > > Wasn't there also Guest/Guest as well? Admittedly it would also be pretty > boring, but nonetheless still a toe-hold. I worked in a VAX shop once where a DEC FSE came by (on the wrong day with the sysadmin out) and was rather upset that the default account passwords had been changed. N. From nevin at eviloverlord.com Mon May 15 12:00:40 2017 From: nevin at eviloverlord.com (Nevin Liber) Date: Sun, 14 May 2017 21:00:40 -0500 Subject: [TUHS] C declarations. In-Reply-To: <20170515004019.GJ29107@mcvoy.com> References: <6D3DCC45-B2C6-4374-83EE-F745C0AF7F36@bitblocks.com> <030d01d2cd06$c44e2840$4cea78c0$@ronnatalie.com> <013240e6-cc66-12c7-325d-a1edf4107726@kilonet.net> <20170515004019.GJ29107@mcvoy.com> Message-ID: On Sun, May 14, 2017 at 7:40 PM, Larry McVoy wrote: > I get that it is not clean and pure and doesn't match how other types work, > I do see that, but I also see that their choice made a ton of sense at the > time. And maybe still does. It's not the same as the other types because > the other types are, typically, small. Arrays can be big, really big. > IIRC, you couldn't pass or return structs either. Also, no function prototypes, so passing data of different sizes would have been a bit more difficult. -- Nevin ":-)" Liber +1-847-691-1404 -------------- next part -------------- An HTML attachment was scrubbed... URL: From random832 at fastmail.com Mon May 15 13:43:39 2017 From: random832 at fastmail.com (Random832) Date: Sun, 14 May 2017 23:43:39 -0400 Subject: [TUHS] C declarations. In-Reply-To: <032701d2cd11$7be3f200$73abd600$@ronnatalie.com> References: <6D3DCC45-B2C6-4374-83EE-F745C0AF7F36@bitblocks.com> <030d01d2cd06$c44e2840$4cea78c0$@ronnatalie.com> <013240e6-cc66-12c7-325d-a1edf4107726@kilonet.net> <032701d2cd11$7be3f200$73abd600$@ronnatalie.com> Message-ID: <1494819819.1845291.976532104.41B8F245@webmail.messagingengine.com> On Sun, May 14, 2017, at 20:23, Ron Natalie wrote: > Actually, it is **Not a pointer**, it gets passed as a pointer, but it is > an array on both ends. I'm not sure what you mean by "on both ends". It is a pointer in the function. sizeof(dst) is 2, not 256, even if it were declared as int dst[128]. On Sun, May 14, 2017, at 19:34, Arthur Krewat wrote: > Is it really a kludge? It's a pointer. > > int array[128]; > > Want to reference the array in a memcpy? Do: memcpy(destination, array, > sizeof(array)); The argument is that that should have to be memcpy(&destination, &array, sizeof(array)), the same as if you were copying between two structs. From dave at horsfall.org Mon May 15 14:35:34 2017 From: dave at horsfall.org (Dave Horsfall) Date: Mon, 15 May 2017 14:35:34 +1000 (EST) Subject: [TUHS] C declarations. In-Reply-To: <20170514042411.93CCB124AEA4@mail.bitblocks.com> References: <015401d2caa0$79762650$6c6272f0$@ronnatalie.com> <68E8DC0A-D1B8-4FF0-AD26-ACDC57E308AF@pobox.com> <20170511223232.GM4341@mcvoy.com> <015b01d2caa7$c658c020$530a4060$@ronnatalie.com> <20170513012415.GZ4341@mcvoy.com> <025701d2cb92$ec8182a0$c58487e0$@ronnatalie.com> <20170513122050.GF9980@yeono.kjorling.se> <0CF82AC1-E835-4C06-813F-D9EFD2C12290@tfeb.org> <20170513124247.GG9980@yeono.kjorling.se> <57639684-050B-4995-859F-876CAFE4BE9C@serissa.com> <20170514042411.93CCB124AEA4@mail.bitblocks.com> Message-ID: On Sat, 13 May 2017, Bakul Shah wrote: > int*()* f // f is a ptr to function returning ptr to int Nope. aneurin% cdecl Type `help' or `?' for help explain int*()* f syntax error declare f as ptr to function returning ptr to int int *(*f)() (Sigh; "cdecl" really needs to print a "> " prompt or something.) -- Dave Horsfall DTM (VK2KFU) "Those who don't understand security will suffer." From bakul at bitblocks.com Mon May 15 14:54:16 2017 From: bakul at bitblocks.com (Bakul Shah) Date: Sun, 14 May 2017 21:54:16 -0700 Subject: [TUHS] C declarations. In-Reply-To: Your message of "Mon, 15 May 2017 14:35:34 +1000." References: <015401d2caa0$79762650$6c6272f0$@ronnatalie.com> <68E8DC0A-D1B8-4FF0-AD26-ACDC57E308AF@pobox.com> <20170511223232.GM4341@mcvoy.com> <015b01d2caa7$c658c020$530a4060$@ronnatalie.com> <20170513012415.GZ4341@mcvoy.com> <025701d2cb92$ec8182a0$c58487e0$@ronnatalie.com> <20170513122050.GF9980@yeono.kjorling.se> <0CF82AC1-E835-4C06-813F-D9EFD2C12290@tfeb.org> <20170513124247.GG9980@yeono.kjorling.se> <57639684-050B-4995-859F-876CAFE4BE9C@serissa.com> <20170514042411.93CCB124AEA4@mail.bitblocks.com> Message-ID: <20170515045416.AF1CF124AEA4@mail.bitblocks.com> On Mon, 15 May 2017 14:35:34 +1000 Dave Horsfall wrote: > On Sat, 13 May 2017, Bakul Shah wrote: > > > int*()* f // f is a ptr to function returning ptr to int > > Nope. > > aneurin% cdecl > Type `help' or `?' for help > explain int*()* f > syntax error > declare f as ptr to function returning ptr to int > int *(*f)() > > (Sigh; "cdecl" really needs to print a "> " prompt or something.) Sorry, I was less than clear. I was just doing a thought experiment about an alternate declaration syntax. Requoting from my email: But if you put the var all the way to the right and read a declaration right to left, it works: Basically suggesting that if all type machinery *preceded* an object declaration (including for functions), we don't need any parentheses and declarations are very easy to understand -- you just read a declaration from right to left. So no need for cdel! Too late for this, of course! From dave at horsfall.org Mon May 15 15:01:58 2017 From: dave at horsfall.org (Dave Horsfall) Date: Mon, 15 May 2017 15:01:58 +1000 (EST) Subject: [TUHS] C declarations. In-Reply-To: <20170515045416.AF1CF124AEA4@mail.bitblocks.com> References: <015401d2caa0$79762650$6c6272f0$@ronnatalie.com> <68E8DC0A-D1B8-4FF0-AD26-ACDC57E308AF@pobox.com> <20170511223232.GM4341@mcvoy.com> <015b01d2caa7$c658c020$530a4060$@ronnatalie.com> <20170513012415.GZ4341@mcvoy.com> <025701d2cb92$ec8182a0$c58487e0$@ronnatalie.com> <20170513122050.GF9980@yeono.kjorling.se> <0CF82AC1-E835-4C06-813F-D9EFD2C12290@tfeb.org> <20170513124247.GG9980@yeono.kjorling.se> <57639684-050B-4995-859F-876CAFE4BE9C@serissa.com> <20170514042411.93CCB124AEA4@mail.bitblocks.com> <20170515045416.AF1CF124AEA4@mail.bitblocks.com> Message-ID: On Sun, 14 May 2017, Bakul Shah wrote: > > Nope. > > > > aneurin% cdecl > > Type `help' or `?' for help > > explain int*()* f > > syntax error > > declare f as ptr to function returning ptr to int > > int *(*f)() > > Sorry, I was less than clear. I was just doing a thought > experiment about an alternate declaration syntax. Requoting > from my email: Sorry - entirely my fault (which I'd realised just *after* I'd sent it, of course). Yes, your idea is a lot better. -- Dave Horsfall DTM (VK2KFU) "Those who don't understand security will suffer." From tfb at tfeb.org Mon May 15 16:46:21 2017 From: tfb at tfeb.org (Tim Bradshaw) Date: Mon, 15 May 2017 07:46:21 +0100 Subject: [TUHS] C declarations. In-Reply-To: <20170513231121.0DB3A1274787@macaroni.inf.ed.ac.uk> References: <20170513231121.0DB3A1274787@macaroni.inf.ed.ac.uk> Message-ID: On 14 May 2017, at 00:11, Richard Tobin wrote: > > If a subprogram reference causes a dummy argument in the referenced > subprogram to become associated with another dummy argument in the > referenced subprogram, neither dummy argument may become defined > during execution of that subprogram. I think the purpose of that is to allow compilers to assume that if you pass two array arguments (say) to a routine then they are really different chunks of memory, so it can optimize things (ie it does not need to worry about the various hazards that happen if they are secretly the same memory). From dot at dotat.at Mon May 15 19:46:08 2017 From: dot at dotat.at (Tony Finch) Date: Mon, 15 May 2017 10:46:08 +0100 Subject: [TUHS] Old Unix vulnerabilities In-Reply-To: <1494741150.709655.975854064.7B3C6781@webmail.messagingengine.com> References: <1494741150.709655.975854064.7B3C6781@webmail.messagingengine.com> Message-ID: Random832 wrote: > On Sat, May 13, 2017, at 19:34, Dave Horsfall wrote: > > > > A beauty in V6 (and possibly V7) was discovered by the kiddies in Elec > > Eng; by sending a signal with an appropriately-crafted negative value (as > > determined from inspecting ) you could overwrite u.u_uid with > > zero... Needless to say I scrambled to fix that one on my 11/40 network! > > V7 fixes it by changing the if(sig >= NSIG) in psignal to cast it to > unsigned. Even without that check V7 wouldn't be vulnerable. In V6, the vulnerability occurs in psig() when the signal action is reset: http://minnie.tuhs.org/cgi-bin/utree.pl?file=V6/usr/sys/ken/sig.c rp = u.u_procp; n = rp->p_sig; rp->p_sig = 0; if((p=u.u_signal[n]) != 0) { u.u_error = 0; if(n != SIGINS && n != SIGTRC) u.u_signal[n] = 0; /* if n < 0 this can overwrite u.u_uid */ In V7, instead of a single pending signal, there is a bitmap of pending signals, so the corresponding code is, http://minnie.tuhs.org/cgi-bin/utree.pl?file=V7/usr/sys/sys/sig.c n = fsig(rp); if (n==0) return; rp->p_sig &= ~(1<<(n-1)); if((p=u.u_signal[n]) != 0) { u.u_error = 0; if(n != SIGINS && n != SIGTRC) u.u_signal[n] = 0; /* always within the array bounds */ Tony. -- f.anthony.n.finch http://dotat.at/ - I xn--zr8h punycode Viking, North Utsire, South Utsire, Northeast Forties: Variable becoming southeasterly 3 or 4, increasing 5 to 7, perhaps gale 8 later. Slight or moderate becoming moderate or rough later. Fog patches, rain later. Moderate, occasionally very poor. From dot at dotat.at Mon May 15 20:21:17 2017 From: dot at dotat.at (Tony Finch) Date: Mon, 15 May 2017 11:21:17 +0100 Subject: [TUHS] C declarations. In-Reply-To: <20170515004019.GJ29107@mcvoy.com> References: <6D3DCC45-B2C6-4374-83EE-F745C0AF7F36@bitblocks.com> <030d01d2cd06$c44e2840$4cea78c0$@ronnatalie.com> <013240e6-cc66-12c7-325d-a1edf4107726@kilonet.net> <20170515004019.GJ29107@mcvoy.com> Message-ID: Larry McVoy wrote: > On Sun, May 14, 2017 at 08:14:47PM -0400, Dan Cross wrote: > > The kludge is that it works different than everything else for no really > > good reason. > > I'd be curious what Steve thinks about all this, I know he weighed in > a bit, but does he think that Dennis / Brian/ Ken regret this design choice? I wonder how much of it was due to evolving in stages from BCPL, where everything is a word, and structs and arrays are just pointers (with maybe an allocation of space for the pointer to point to). And C couldn't pass structs by value for quite a long time. Tony. -- f.anthony.n.finch http://dotat.at/ - I xn--zr8h punycode Fair Isle, Faeroes: Southeast 5 or 6, increasing 7 or gale 8 for a time. Moderate or rough, occasionally very rough in Faeroes. Fair then rain with fog patches. Good, becoming moderate or very poor. From michael at kjorling.se Mon May 15 22:58:40 2017 From: michael at kjorling.se (Michael =?utf-8?B?S2rDtnJsaW5n?=) Date: Mon, 15 May 2017 12:58:40 +0000 Subject: [TUHS] C declarations. In-Reply-To: References: <20170511223232.GM4341@mcvoy.com> <015b01d2caa7$c658c020$530a4060$@ronnatalie.com> <20170513012415.GZ4341@mcvoy.com> <025701d2cb92$ec8182a0$c58487e0$@ronnatalie.com> <20170513122050.GF9980@yeono.kjorling.se> <0CF82AC1-E835-4C06-813F-D9EFD2C12290@tfeb.org> <20170513124247.GG9980@yeono.kjorling.se> <57639684-050B-4995-859F-876CAFE4BE9C@serissa.com> <20170514042411.93CCB124AEA4@mail.bitblocks.com> Message-ID: <20170515125840.GA14426@yeono.kjorling.se> On 15 May 2017 14:35 +1000, from dave at horsfall.org (Dave Horsfall): > (Sigh; "cdecl" really needs to print a "> " prompt or something.) It does print "cdecl> " as a prompt on my system. $ cdecl Type `help' or `?' for help cdecl> explain int*()* f syntax error cdecl> declare f as ptr to function returning ptr to int int *(*f)() cdecl> ^D $ However, running it with -q turns this off: $ cdecl -q declare f as ptr to function returning ptr to int int *(*f)() ^D $ The man page for mine says 2.5, Jan 1996, and the Debian package version says 2.5-11. 1996 in 2017 is old even for Debian. Are you sure that you don't have a stray alias somewhere, or something that might be causing it to misdetect as not running in a terminal? -- Michael Kjörling • https://michael.kjorling.se • michael at kjorling.se “People who think they know everything really annoy those of us who know we don’t.” (Bjarne Stroustrup) From dave at horsfall.org Tue May 16 02:58:24 2017 From: dave at horsfall.org (Dave Horsfall) Date: Tue, 16 May 2017 02:58:24 +1000 (EST) Subject: [TUHS] C declarations. In-Reply-To: <20170515125840.GA14426@yeono.kjorling.se> References: <20170511223232.GM4341@mcvoy.com> <015b01d2caa7$c658c020$530a4060$@ronnatalie.com> <20170513012415.GZ4341@mcvoy.com> <025701d2cb92$ec8182a0$c58487e0$@ronnatalie.com> <20170513122050.GF9980@yeono.kjorling.se> <0CF82AC1-E835-4C06-813F-D9EFD2C12290@tfeb.org> <20170513124247.GG9980@yeono.kjorling.se> <57639684-050B-4995-859F-876CAFE4BE9C@serissa.com> <20170514042411.93CCB124AEA4@mail.bitblocks.com> <20170515125840.GA14426@yeono.kjorling.se> Message-ID: On Mon, 15 May 2017, Michael Kjörling wrote: > It does print "cdecl> " as a prompt on my system. [...] > The man page for mine says 2.5, Jan 1996, and the Debian package > version says 2.5-11. 1996 in 2017 is old even for Debian. I've got 2.5 as well (on FreeBSD 10.3), and "set prompt" has no effect. I even rebuilt it in case there was a later version. > Are you sure that you don't have a stray alias somewhere, or something > that might be causing it to misdetect as not running in a terminal? aneurin% cdecl Type `help' or `?' for help set prompt set options [...] Current set values are: noprompt interactive WTF? I'll fire up GDB and have a poke around (once I'm properly awake; I only got up for a glass of water, and couldn't help but check my email). -- Dave Horsfall DTM (VK2KFU) "Those who don't understand security will suffer." From scj at yaccman.com Tue May 16 04:47:51 2017 From: scj at yaccman.com (Steve Johnson) Date: Mon, 15 May 2017 11:47:51 -0700 Subject: [TUHS] C declarations. In-Reply-To: <030d01d2cd06$c44e2840$4cea78c0$@ronnatalie.com> Message-ID: Some interesting comments:     "You all are missing the point as to what the cost of passing arrays by value or what other languages do" I don't think so.  To me the issues is that the model of what it means to compute has changed since the punch-card days.  When you submitted a card deck in the early days, you had to include both the function definition and the data--the function was compiled, the data was read, and, for the most part there were no significant side effects (just a printout, and maybe some stuff on mag tape). This was a model that had served mathematics well for centuries, and it was very easy to understand.  Functional programming people still like it a lot... However, with the introduction of permanent file systems, a new paradigm came into being.  Now, interactions with the computer looked more like database transactions:  Load your program, change a few lines, put it back, and then call 'make'.  Trying to describe this with a purely functional model leads to absurdities like:      file_system = edit( file_system, file_selector, editing_commands ); In fact, the editing commands can change files, create new ones, and even delete files.  There is no reasonable way to handle any realistic file systems with this model (let alone the Internet!) In C's early days, we were just getting into the new world.  Call by value for arrays would have been expensive or impossible on the machine with just a few kilobytes of memory for program + data.  So we didn't do it. Structures were initially handled like arrays, but the compiler chose to make a local copy when passed a structure pointer.  This copy was, at one time, in static memory, which caused some problems.  Later, it went on the stack.  It wasn't much used... This changed when the Blit terminal project was in place.  It was just too attractive on a 68000 to write     struct pt = {  int x;  int y }        /* when int was 16-bits */ and I made PCC pass small structures like this in registers, like other arguments.  I seem to remember a dramatic speedup (2X or so) from doing this... "(did) Dennis / Brian/ Ken regret this design choice? Not that I recall.  Of course, we all had bugs in this area.  But I think the lack of subscript range checking was a more serious problem than using pointers in the first place.  And, indeed, for a few of the pioneers, BCPL had done exactly the same thing.  Steve -------------- next part -------------- An HTML attachment was scrubbed... URL: From bakul at bitblocks.com Tue May 16 05:00:13 2017 From: bakul at bitblocks.com (Bakul Shah) Date: Mon, 15 May 2017 12:00:13 -0700 Subject: [TUHS] cdecl (Re: C declarations. In-Reply-To: References: <20170511223232.GM4341@mcvoy.com> <015b01d2caa7$c658c020$530a4060$@ronnatalie.com> <20170513012415.GZ4341@mcvoy.com> <025701d2cb92$ec8182a0$c58487e0$@ronnatalie.com> <20170513122050.GF9980@yeono.kjorling.se> <0CF82AC1-E835-4C06-813F-D9EFD2C12290@tfeb.org> <20170513124247.GG9980@yeono.kjorling.se> <57639684-050B-4995-859F-876CAFE4BE9C@serissa.com> <20170514042411.93CCB124AEA4@mail.bitblocks.com> <20170515125840.GA14426@yeono.kjorling.se> Message-ID: <056F4192-940A-48E1-9478-46016F7FD26A@bitblocks.com> pkg upgrade cdecl 2.5 didn't show a prompt but 3.1.3 does (though the prompt is in /green/. Sigh...) > On May 15, 2017, at 9:58 AM, Dave Horsfall wrote: > >> On Mon, 15 May 2017, Michael Kjörling wrote: >> >> It does print "cdecl> " as a prompt on my system. > [...] >> The man page for mine says 2.5, Jan 1996, and the Debian package >> version says 2.5-11. 1996 in 2017 is old even for Debian. > > I've got 2.5 as well (on FreeBSD 10.3), and "set prompt" has no effect. > I even rebuilt it in case there was a later version. > >> Are you sure that you don't have a stray alias somewhere, or something >> that might be causing it to misdetect as not running in a terminal? > > aneurin% cdecl > Type `help' or `?' for help > set prompt > set options > [...] > Current set values are: > noprompt > interactive > > WTF? > > I'll fire up GDB and have a poke around (once I'm properly awake; I only > got up for a glass of water, and couldn't help but check my email). > > > -- > Dave Horsfall DTM (VK2KFU) "Those who don't understand security will suffer." From bakul at bitblocks.com Tue May 16 05:54:25 2017 From: bakul at bitblocks.com (Bakul Shah) Date: Mon, 15 May 2017 12:54:25 -0700 Subject: [TUHS] C declarations. In-Reply-To: References: Message-ID: <27EF4DFC-5972-424F-BB22-0B2BD1593A45@bitblocks.com> Pascal was designed in 1967-69 and its report came out in 1970. It is of similar complexity (the first self compiling compiler was about 4500 lines, IIRC). It had n dimensional fixed size arrays and structures (records). Allowing any parameter of any type to be a ref param meant you didn't have to copy if you didn't want to. And there is no hidden magic, no GC. I guess it wasn't much known in the US then so not much scope for stealing ideas.... I guess the difference was that Wirth started with Algol W while Ritchie stated with B and BCPL. Unlike pascal C had the address of (&) operator so C could've only allow passing pointers to them. This would've been less "magic" and much cleaner. And from there a ref parameter is just a small step! To me your earlier comment made more sense - C was seen as a system programming language not an application programming one. For it passing strings and memory blocks was enough. Earlier I had wondered if dmr et al had a more Fortran like array model in mind and punted on it as it would complicate the language. > On May 15, 2017, at 11:47 AM, Steve Johnson wrote: > > > Some interesting comments: > > "You all are missing the point as to what the cost of passing arrays by value or what other languages do" > > I don't think so. To me the issues is that the model of what it means to compute has changed since the punch-card days. When you submitted a card deck in the early days, you had to include both the function definition and the data--the function was compiled, the data was read, and, for the most part there were no significant side effects (just a printout, and maybe some stuff on mag tape). > > This was a model that had served mathematics well for centuries, and it was very easy to understand. Functional programming people still like it a lot... > > However, with the introduction of permanent file systems, a new paradigm came into being. Now, interactions with the computer looked more like database transactions: Load your program, change a few lines, put it back, and then call 'make'. Trying to describe this with a purely functional model leads to absurdities like: > > file_system = edit( file_system, file_selector, editing_commands ); > > In fact, the editing commands can change files, create new ones, and even delete files. There is no reasonable way to handle any realistic file systems with this model (let alone the Internet!) > > In C's early days, we were just getting into the new world. Call by value for arrays would have been expensive or impossible on the machine with just a few kilobytes of memory for program + data. So we didn't do it. > > Structures were initially handled like arrays, but the compiler chose to make a local copy when passed a structure pointer. This copy was, at one time, in static memory, which caused some problems. Later, it went on the stack. It wasn't much used... > > This changed when the Blit terminal project was in place. It was just too attractive on a 68000 to write > > struct pt = { int x; int y } /* when int was 16-bits */ > > and I made PCC pass small structures like this in registers, like other arguments. I seem to remember a dramatic speedup (2X or so) from doing this... > > > "(did) Dennis / Brian/ Ken regret this design choice? > > Not that I recall. Of course, we all had bugs in this area. But I think the lack of subscript range checking was a more serious problem than using pointers in the first place. And, indeed, for a few of the pioneers, BCPL had done exactly the same thing. > > > > Steve > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave at horsfall.org Tue May 16 08:52:43 2017 From: dave at horsfall.org (Dave Horsfall) Date: Tue, 16 May 2017 08:52:43 +1000 (EST) Subject: [TUHS] cdecl (Re: C declarations. In-Reply-To: <056F4192-940A-48E1-9478-46016F7FD26A@bitblocks.com> References: <20170511223232.GM4341@mcvoy.com> <015b01d2caa7$c658c020$530a4060$@ronnatalie.com> <20170513012415.GZ4341@mcvoy.com> <025701d2cb92$ec8182a0$c58487e0$@ronnatalie.com> <20170513122050.GF9980@yeono.kjorling.se> <0CF82AC1-E835-4C06-813F-D9EFD2C12290@tfeb.org> <20170513124247.GG9980@yeono.kjorling.se> <57639684-050B-4995-859F-876CAFE4BE9C@serissa.com> <20170514042411.93CCB124AEA4@mail.bitblocks.com> <20170515125840.GA14426@yeono.kjorling.se> <056F4192-940A-48E1-9478-46016F7FD26A@bitblocks.com> Message-ID: On Mon, 15 May 2017, Bakul Shah wrote: > pkg upgrade cdecl Didn't work; all my packages are up to date, so I guess you're running FreeBSD 11 or something? > 2.5 didn't show a prompt but 3.1.3 does (though the prompt is in > /green/. Sigh...) I'll wait for 3.1.3 to be ported then, although I might bash ahead with GDB anyway. -- Dave Horsfall DTM (VK2KFU) "Those who don't understand security will suffer." From scj at yaccman.com Tue May 16 04:00:36 2017 From: scj at yaccman.com (Steve Johnson) Date: Mon, 15 May 2017 11:00:36 -0700 Subject: [TUHS] The evolution of Unix facilities and architecture In-Reply-To: Message-ID: Early in the Unix days, a DEC repairperson showed up to do "preventive maintenance" and managed to clobber the nascent file system.   Turns out DEC didn't have any permanent file systems on machines that small... ----- Original Message ----- From: "Nemo" To:"The Eunuchs Hysterical Society" Cc: Sent:Sun, 14 May 2017 21:24:25 -0400 Subject:Re: [TUHS] The evolution of Unix facilities and architecture On 14 May 2017 at 18:12, Dave Horsfall wrote: > On Sun, 14 May 2017, Derek Fawcus wrote: > >> > to see DEC's internal boxes weren't running System/Manager, >> > Field/Service and UETP/UETP User/password combinations. >> >> Those default account combinations were still being used to gain access >> to VMS systems in the '87-'89 time frame; although user/password was >> less interesting by itself, being an unpriviledged account. > > Wasn't there also Guest/Guest as well? Admittedly it would also be pretty > boring, but nonetheless still a toe-hold. I worked in a VAX shop once where a DEC FSE came by (on the wrong day with the sysadmin out) and was rather upset that the default account passwords had been changed. N. -------------- next part -------------- An HTML attachment was scrubbed... URL: From gdmr at inf.ed.ac.uk Tue May 16 17:25:20 2017 From: gdmr at inf.ed.ac.uk (George Ross) Date: Tue, 16 May 2017 08:25:20 +0100 Subject: [TUHS] C declarations. In-Reply-To: Your message of "Mon, 15 May 2017 11:47:51 PDT." Message-ID: <201705160725.v4G7PKiA009291@farg.inf.ed.ac.uk> > ...  When you > submitted a card deck in the early days, you had to include both the > function definition and the data--the function was compiled, the data > was read, and, for the most part there were no significant side > effects (just a printout, and maybe some stuff on mag tape). In the late 1970s I had a summer job which involved writing FORTRAN programs to analyse card decks. This was on some IBM machine, which had a combi card punch/reader. On one run the program loaded and compiled, and the first two data cards went through. Then the machine crashed. After a bit of head-scratching, I realised that it was because the punch station came immediately before the read station, and there was a small typo in the program. It processed the first data card, but rather than logging to the printer it overpunched the second card instead resulting in an invalid hole combination. That card then went into the read station, and the machine didn't like it... -- George D M Ross MSc PhD CEng MBCS CITP, University of Edinburgh, School of Informatics, 10 Crichton Street, Edinburgh, Scotland, EH8 9AB Mail: gdmr at inf.ed.ac.uk Voice: 0131 650 5147 Fax: 0131 650 6899 PGP: 1024D/AD758CC5 B91E D430 1E0D 5883 EF6A 426C B676 5C2B AD75 8CC5 The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. From jnc at mercury.lcs.mit.edu Tue May 16 23:20:34 2017 From: jnc at mercury.lcs.mit.edu (Noel Chiappa) Date: Tue, 16 May 2017 09:20:34 -0400 (EDT) Subject: [TUHS] The evolution of Unix facilities and architecture Message-ID: <20170516132034.E998318C089@mercury.lcs.mit.edu> > From: "Steve Johnson" > a DEC repairperson showed up to do "preventive maintenance" and managed > to clobber the nascent file system. > Turns out DEC didn't have any permanent file systems on machines that > small... A related story (possibly a different version of this one) which I read (can't remember where, now) was that he trashed the contents of the RS04 fixed-head hard disk, because on DEC OS's, those were only used for swapping. Noel From clemc at ccc.com Tue May 16 23:46:58 2017 From: clemc at ccc.com (Clem Cole) Date: Tue, 16 May 2017 09:46:58 -0400 Subject: [TUHS] The evolution of Unix facilities and architecture In-Reply-To: <20170516132034.E998318C089@mercury.lcs.mit.edu> References: <20170516132034.E998318C089@mercury.lcs.mit.edu> Message-ID: I believe it is a true story... The issue was that any early version of the FE tools was set to use the RS04 as a temporary disk when running the system exercisers if it found one. The author of that error is a friend of mine (and would later become a UNIX guy at Masscomp). He said to me about it once. he had checked with the DEC OS teams and thought it was an 'ok' because when they did it, none of the DEC OS used the RS04 for permanent storage - the device had been designed to be a swapping device. >From what I understand, the issue was actually short lived, but widely known in the UNIX community. He told me it the accident only occurred at one site that he knew about (AT&T) and they made a quick change to have it ask before it wiped it out and that verion of the tools was release to the field quickly. But he was personally wise to UNIX from them on (and it later years would come to love become a UNIX a user although I don't think he ever gave up the EDT macros we wrote for EMACS for him and the other ex-VMS folks). Clem On Tue, May 16, 2017 at 9:20 AM, Noel Chiappa wrote: > > From: "Steve Johnson" > > > a DEC repairperson showed up to do "preventive maintenance" and > managed > > to clobber the nascent file system. > > Turns out DEC didn't have any permanent file systems on machines that > > small... > > A related story (possibly a different version of this one) which I read > (can't > remember where, now) was that he trashed the contents of the RS04 > fixed-head > hard disk, because on DEC OS's, those were only used for swapping. > > Noel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ron at ronnatalie.com Wed May 17 08:33:50 2017 From: ron at ronnatalie.com (Ron Natalie) Date: Tue, 16 May 2017 18:33:50 -0400 Subject: [TUHS] The evolution of Unix facilities and architecture In-Reply-To: References: Message-ID: <009101d2ce94$7de8dbe0$79ba93a0$@ronnatalie.com> Our biggest UNIX vs. DEC OS problem was that UNIX set the system clock in GMT whereas the DEC OSes of the day used local time. We got used to the time being 4/5 hours off after a DEC CE was there. We actually contracted our maintenance out for the entire company. We ended up going with GE. We couldn’t convince the guys from DEC who bid that when we said there were certain critical components of the site that needed 24-hour response, that this meant, 24 hours in a row, not three successive 8 hour days. We often were stuck using a DEC CE for initial system setup or warranty work. We had a particularly bad one who would blow power supplies, screw up running systems, etc. She was just simply incompetent. The culmination of this was when she managed somehow to put herself across the AC line of a VAX over in one of our external buildings and ended up being taken a way in an ambulance. Working off hours, we’d often set up the new machines and run diagnostic checks on them ahead of the CE showing up. I got a testy CE show up and tell me that he “didn’t need me checking up on his work.” (Hell, was the COTR and customer). I told him I only had one word to say about that . He beat a quick retreat saying that Nancy was a different story. I was driving to work one morning at Christmas time, and one of our local radio stations was soliciting people to send in their sob stories about how bad a year they had and they would be given a special gift. One story went on for a few minutes, and I hadn’t caught on until it got to the electrical shock at work part and I knew it was Nancy’s story. Our standard joke was that the way you could tell a DEC CE with a flat tire was that he had to change all four before he found the problem. Amusingly, working for the feds had some other interesting fiascos. I got an amusing message from the security and facilities people one day. I had to tell our CE. ME: Bill, I can’t let you in the machine room anymore. BILL: Why not ME: You’re a fire hazard. BILL: How so? ME: You have soldering irons. Of course, I was able to prevail on them that we’d keep an eye on the CE and stand by with fire suppression if we let him do his job. The machine room in my building had no automatic halon system which was popular in those days. What we had was a lot of large halon hand extinguishers. The post fire department came out and set pan fires behind our building and let us practice putting them out with the halon. I can’t imagine what the costs to the federal budget and the ozone layer were on that little activity. Of course, I brought my turnout gear as I was a firefighter and paramedic at the time. This led to another interesting call from the front office. SEC: You need to attend a CPR class. ME: I’ve already had one this year. I’m a state certified paramedic. I go through recurrent training every month. SEC: Well it is a requirement that you have a CPR card. ME: Why am I the only one in my office that this is a requirement for? SEC: It’s your job classification. ME: Because I’m an electrical engineer and the other guys are computer scientists? SEC: Yes. ME: Why? SEC: Because you work with electricity. ME: I work with digital logic. Five volts. Further, even if I was going to shock myself into cardiac arrest, I can’t do CPR on myself. You should make everybody else take CPR. -------------- next part -------------- An HTML attachment was scrubbed... URL: From krewat at kilonet.net Wed May 17 09:13:03 2017 From: krewat at kilonet.net (Arthur Krewat) Date: Tue, 16 May 2017 19:13:03 -0400 Subject: [TUHS] The evolution of Unix facilities and architecture In-Reply-To: <009101d2ce94$7de8dbe0$79ba93a0$@ronnatalie.com> References: <009101d2ce94$7de8dbe0$79ba93a0$@ronnatalie.com> Message-ID: <14366930-f60d-023d-6591-6f45df26ff78@kilonet.net> On 5/16/2017 6:33 PM, Ron Natalie wrote: > Our standard joke was that the way you could tell a DEC CE with a flat > tire was that he had to change all four before he found the problem. Conversely, at BOCES LIRICS in Dix Hills, NY, we had a field-service tech who seemed to know every single wire wrap in the PDP-10's. First the KA10 they had before I got there (but worked on in high school), and then the KS10's. Maybe the VAX/PDP-11 crew were different :) On the other hand, he left in his desk a complete list of KLINIK line phone numbers (modems connected in parallel with the CTY), and PASSWORDS, to just about every PDP-10 in the tri-state area (NY/NJ/CT). From ron at ronnatalie.com Wed May 17 09:18:29 2017 From: ron at ronnatalie.com (Ron Natalie) Date: Tue, 16 May 2017 19:18:29 -0400 Subject: [TUHS] The evolution of Unix facilities and architecture In-Reply-To: <14366930-f60d-023d-6591-6f45df26ff78@kilonet.net> References: <009101d2ce94$7de8dbe0$79ba93a0$@ronnatalie.com> <14366930-f60d-023d-6591-6f45df26ff78@kilonet.net> Message-ID: <00e401d2ce9a$bacbf5a0$3063e0e0$@ronnatalie.com> The wirewrap comment reminds me of another catastrophe. BRL had one of the few operational Denelcor HEP supercomputers. Amusingly the thing was front ended with an 11/34 and had an io processor with 32 Unibuses connected to it. While we were still shaking the thing out at the factory (Mike Muuss shot his mouth off that we could get UNIX to run on the thing, and it turned out nobody had a better idea). We'd come in at night (while the Denelcor employees had the machine in the day). I'd regularly read the log about what happened that day. Well, one day it read that someone had neatened up one of the system backplanes by taking the wires that were overly long and shortening them. Now this thing was all built out of 10800 ECL and those wires were "overly long" and stuffed into the backplane because the length was needed to control the signal propagation times. I told Mike I had a bad feeling about this and the machine might not be usable. From tim.newsham at gmail.com Fri May 19 03:32:07 2017 From: tim.newsham at gmail.com (Tim Newsham) Date: Thu, 18 May 2017 07:32:07 -1000 Subject: [TUHS] Old Unix vulnerabilities In-Reply-To: References: Message-ID: Here are some previously reported ones: v1: http://minnie.tuhs.org/pipermail/unix-jun72/2008-May/000126.html http://minnie.tuhs.org/pipermail/unix-jun72/2008-May/000250.html v7: http://seclists.org/bugtraq/2004/Jun/37 On Sat, May 13, 2017 at 1:34 PM, Dave Horsfall wrote: > OK, I'll kick it off. > > A beauty in V6 (and possibly V7) was discovered by the kiddies in Elec > Eng; by sending a signal with an appropriately-crafted negative value (as > determined from inspecting ) you could overwrite u.u_uid with > zero... Needless to say I scrambled to fix that one on my 11/40 network! > > -- > Dave Horsfall DTM (VK2KFU) "Those who don't understand security will > suffer." > -- Tim Newsham | www.thenewsh.com/~newsham | @newshtwit | thenewsh.blogspot.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at kdbarto.org Sat May 20 00:31:54 2017 From: david at kdbarto.org (David) Date: Fri, 19 May 2017 07:31:54 -0700 Subject: [TUHS] The evolution of Unix facilities and architecture In-Reply-To: References: Message-ID: At Celerity we were porting Unix to a new NCR chipset for our washing machine sized Workstation. We had a VAX 750 as the development box and we cross compiled to the NCR box. We contracted out the 750 maintenance to a 3rd party and had no problems for a couple of years. Then one day I came in to work to find the VAX happy consuming power and doing nothing. Unix wasn’t running and nothing I could do would bring it back. After about 2 hours I got my boss and we contacted the maintenance company. They guy they sent did much what I’d done and then went around the back. He pushed on the backplane of the machine and Lo, it started working. He then removed the pressure and it failed quite immediately. Turns out the backplane had a broken trace in it. We had done no board swaps in many months and the room had had no A/C faults of any kind. The company got a new backplane and had it installed in 2 days. Being 3rd party we couldn’t get it replaced any quicker. After that it worked like a champ. Celerity eventually became part of Sun as Sun Supercomputer. David From jnc at mercury.lcs.mit.edu Sat May 20 01:15:18 2017 From: jnc at mercury.lcs.mit.edu (Noel Chiappa) Date: Fri, 19 May 2017 11:15:18 -0400 (EDT) Subject: [TUHS] Unix with TCP/IP for small PDP-11s Message-ID: <20170519151518.8800D18C09D@mercury.lcs.mit.edu> I'm trying to find out if there is any existing Unix for the PDP-11 which supports TCP/IP on /40-/34-23 class machines (i.e. non-I+D machines)? Does 2.9 BSD with TCP/IP (assuming such a thing exists) fit on those machines? (I know 2.9 does run on them, but I don't know about the TCP/IP part.) The reason I ask is that MIT did a TCP/IP for V6 which would run on them (only incoming packet de-mux is in the kernel - the TCP is in with the application, in the user process), which has recently been recovered from a backup tape. I'm trying to figure out if there is any use for it, as it would take some work to make it usable (I'd have to write device drivers for available Ethernet cards, and adapt an ARP implementation for it). Noel From henry.r.bent at gmail.com Sat May 20 02:29:54 2017 From: henry.r.bent at gmail.com (Henry Bent) Date: Fri, 19 May 2017 12:29:54 -0400 Subject: [TUHS] Unix with TCP/IP for small PDP-11s In-Reply-To: <20170519151518.8800D18C09D@mercury.lcs.mit.edu> References: <20170519151518.8800D18C09D@mercury.lcs.mit.edu> Message-ID: There is a later patch release of 2.9BSD that includes TCP/IP support but I have not had any luck getting a compiled kernel to boot. My recommendation would be Ultrix 3.1 which does run on the smaller machines. I think it is possible that 2.10BSD could be made to work but I have never had any success booting it in SIMH. If anyone has booted it or has any suggestions, I would be interested to hear it. -Henry On May 19, 2017 11:15 AM, "Noel Chiappa" wrote: I'm trying to find out if there is any existing Unix for the PDP-11 which supports TCP/IP on /40-/34-23 class machines (i.e. non-I+D machines)? Does 2.9 BSD with TCP/IP (assuming such a thing exists) fit on those machines? (I know 2.9 does run on them, but I don't know about the TCP/IP part.) The reason I ask is that MIT did a TCP/IP for V6 which would run on them (only incoming packet de-mux is in the kernel - the TCP is in with the application, in the user process), which has recently been recovered from a backup tape. I'm trying to figure out if there is any use for it, as it would take some work to make it usable (I'd have to write device drivers for available Ethernet cards, and adapt an ARP implementation for it). Noel -------------- next part -------------- An HTML attachment was scrubbed... URL: From clemc at ccc.com Sat May 20 03:39:41 2017 From: clemc at ccc.com (Clem Cole) Date: Fri, 19 May 2017 13:39:41 -0400 Subject: [TUHS] Unix with TCP/IP for small PDP-11s In-Reply-To: <20170519151518.8800D18C09D@mercury.lcs.mit.edu> References: <20170519151518.8800D18C09D@mercury.lcs.mit.edu> Message-ID: My quick answer is yes, this sounds cool. >From that time, that two attempts to network 40 class systems that I know about were: - As I have mentioned in the past, in the late 1970s CMU had developed an 8085 later 8086 based Multibus board that would eventually begat the Cisco AGS, called the CMU distributed Front-End. It was an attempt to get IP connected to the 11/40 class systems, by splicing them into the 11's with an DR-11B or C (I've forgotten which). But the primary protocol processing was done on the FE and not in the UNIX kernel. Phil Karn might remember what happened to that stuff, I do not believe I still have it. - Shortly thereafter @ Tektronix had 3Com's UNET. I was primarily running on the 11/70 and that was what we used to develop against the Cyber and VMS IP/TCP's we wrote. We also had it on an 11/44 but both of those are 45 class. We did try to run on an 11/60 - which is 40 class as you asked, and we did get it work as I recall after a fashion. But we did not push it very hard;as we had built a front end that in the same vein as the CMU device, called the NIBB - Network Interface Black Box, which was a Z80 that did the same sort of thing and the need was not there. I was under the impression the 2.X folks may have tried to do it later with all the thunk code. But that was all after my work. Fred Canter at DEC might have tried something in later versions of Ultrix/PDP11, we should look at the sources in his stream had see what he did too. Clem On Fri, May 19, 2017 at 11:15 AM, Noel Chiappa wrote: > I'm trying to find out if there is any existing Unix for the PDP-11 > which supports TCP/IP on /40-/34-23 class machines (i.e. non-I+D > machines)? > > Does 2.9 BSD with TCP/IP (assuming such a thing exists) fit on those > machines? (I know 2.9 does run on them, but I don't know about the TCP/IP > part.) > > > The reason I ask is that MIT did a TCP/IP for V6 which would run on them > (only incoming packet de-mux is in the kernel - the TCP is in with the > application, in the user process), which has recently been recovered from a > backup tape. > > I'm trying to figure out if there is any use for it, as it would take some > work to make it usable (I'd have to write device drivers for available > Ethernet cards, and adapt an ARP implementation for it). > > Noel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ron at ronnatalie.com Sat May 20 05:04:09 2017 From: ron at ronnatalie.com (Ron Natalie) Date: Fri, 19 May 2017 15:04:09 -0400 Subject: [TUHS] Unix with TCP/IP for small PDP-11s In-Reply-To: References: <20170519151518.8800D18C09D@mercury.lcs.mit.edu> Message-ID: <014301d2d0d2$b2b8f0e0$182ad2a0$@ronnatalie.com> To my knowledge no. We had already gone to code overlays on the 11/34’s just to get the pre-TCP unix working. When we needed yet another segment to map mbufs (which we could do on the 11/70), there just weren’t any left. It was then that I scarfed up all the 11/23, 24, 34’s that were available and turned them into routers. -------------- next part -------------- An HTML attachment was scrubbed... URL: From imp at bsdimp.com Sat May 20 10:15:19 2017 From: imp at bsdimp.com (Warner Losh) Date: Fri, 19 May 2017 18:15:19 -0600 Subject: [TUHS] Unix with TCP/IP for small PDP-11s In-Reply-To: <014301d2d0d2$b2b8f0e0$182ad2a0$@ronnatalie.com> References: <20170519151518.8800D18C09D@mercury.lcs.mit.edu> <014301d2d0d2$b2b8f0e0$182ad2a0$@ronnatalie.com> Message-ID: There's a copy of ultrix-11, which is v7 based with some 4.1BSD additions, including sockets and a TCP/IP stack. Don't know if it runs on the smaller PDP-11, but it looks like it might. The sources are in the TUHS archives. Seems like the best low-end v7ish kernel with TCP/IP around, but I've not used it extensively. Warner On Fri, May 19, 2017 at 1:04 PM, Ron Natalie wrote: > To my knowledge no. We had already gone to code overlays on the 11/34’s > just to get the pre-TCP unix working. When we needed yet another segment > to map mbufs (which we could do on the 11/70), there just weren’t any > left. It was then that I scarfed up all the 11/23, 24, 34’s that were > available and turned them into routers. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From imp at bsdimp.com Sat May 20 10:21:01 2017 From: imp at bsdimp.com (Warner Losh) Date: Fri, 19 May 2017 18:21:01 -0600 Subject: [TUHS] Unix with TCP/IP for small PDP-11s In-Reply-To: References: <20170519151518.8800D18C09D@mercury.lcs.mit.edu> <014301d2d0d2$b2b8f0e0$182ad2a0$@ronnatalie.com> Message-ID: https://ia601901.us.archive.org/10/items/bitsavers_decpdp11ulLTRIX112.0SPDSep84_870730/AE-X370C-TC_ULTRIX-11_2.0SPD_Sep84.pdf Looks like it requires MMU, but not split I/D space as it lists the following as compatible: M11, 11/23+, 11/24, 11/34, 11/40 and 11/60. It does require 256kb of memory. See table 2, page 6 for details. Warner On Fri, May 19, 2017 at 6:15 PM, Warner Losh wrote: > There's a copy of ultrix-11, which is v7 based with some 4.1BSD additions, > including sockets and a TCP/IP stack. Don't know if it runs on the smaller > PDP-11, but it looks like it might. The sources are in the TUHS archives. > Seems like the best low-end v7ish kernel with TCP/IP around, but I've not > used it extensively. > > Warner > > On Fri, May 19, 2017 at 1:04 PM, Ron Natalie wrote: > >> To my knowledge no. We had already gone to code overlays on the 11/34’s >> just to get the pre-TCP unix working. When we needed yet another segment >> to map mbufs (which we could do on the 11/70), there just weren’t any >> left. It was then that I scarfed up all the 11/23, 24, 34’s that were >> available and turned them into routers. >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bqt at update.uu.se Sat May 20 20:46:34 2017 From: bqt at update.uu.se (Johnny Billquist) Date: Sat, 20 May 2017 12:46:34 +0200 Subject: [TUHS] Unix with TCP/IP for small PDP-11s In-Reply-To: References: Message-ID: On 2017-05-20 04:00, Warner Losh wrote: > https://ia601901.us.archive.org/10/items/bitsavers_decpdp11ulLTRIX112.0SPDSep84_870730/AE-X370C-TC_ULTRIX-11_2.0SPD_Sep84.pdf > > Looks like it requires MMU, but not split I/D space as it lists the > following as compatible: M11, 11/23+, 11/24, 11/34, 11/40 and 11/60. It > does require 256kb of memory. See table 2, page 6 for details. Uh...? Where do you see that there is any TCP/IP support in Ultrix-11? If any was done by someone else, there is no saying that it would be usable on a machine without split I/D. To be honest, I've never seen any mention of TCP/IP on any machine without split I/D space. I guess it could be done, but it would be a rather big headache... Johnny -- Johnny Billquist || "I'm on a bus || on a psychedelic trip email: bqt at softjar.se || Reading murder books pdp is alive! || tryin' to stay hip" - B. Idol From henry.r.bent at gmail.com Sun May 21 03:39:26 2017 From: henry.r.bent at gmail.com (Henry Bent) Date: Sat, 20 May 2017 13:39:26 -0400 Subject: [TUHS] Unix with TCP/IP for small PDP-11s In-Reply-To: References: Message-ID: There is absolutely TCP/IP in Ultrix-11. This is an emulated 11/23+: -- hrb11# uname -a ULTRIX-11 hrb11 3 0 PDP-11/23 hrb11# ifconfig qe0 qe0: 192.168.1.199 netmask ffffff00 flags=263 broadcast: 192.168.1.255 hrb11# netstat -rn Routing tables Destination Gateway Flags Refcnt Use Interface 74.42.148.138 192.168.1.1 UGH 0 0 qe0 192.168.1 192.168.1.199 U 1 0 qe0 default 192.168.1.1 UG 0 0 qe0 192.168.5 192.168.1.1 UG 0 0 qe0 hrb11# ftp 192.168.1.63 Connected to 192.168.1.63. 220 aelfric FTP server (Version 6.4/OpenBSD/Linux-ftpd-0.17) ready. Name (192.168.1.63:root): ... -- It has trouble talking to the outside world, probably not a big surprise, but works just fine on my local network. I am working on getting it installed in a real 11/23 but the Ultrix RL02 driver doesn't seem to like my Dilog emulation board. -Henry On 20 May 2017 at 06:46, Johnny Billquist wrote: > On 2017-05-20 04:00, Warner Losh wrote: > > https://ia601901.us.archive.org/10/items/bitsavers_decpdp11u >> lLTRIX112.0SPDSep84_870730/AE-X370C-TC_ULTRIX-11_2.0SPD_Sep84.pdf >> >> Looks like it requires MMU, but not split I/D space as it lists the >> following as compatible: M11, 11/23+, 11/24, 11/34, 11/40 and 11/60. It >> does require 256kb of memory. See table 2, page 6 for details. >> > > > Uh...? Where do you see that there is any TCP/IP support in Ultrix-11? If > any was done by someone else, there is no saying that it would be usable on > a machine without split I/D. To be honest, I've never seen any mention of > TCP/IP on any machine without split I/D space. I guess it could be done, > but it would be a rather big headache... > > Johnny > > -- > Johnny Billquist || "I'm on a bus > || on a psychedelic trip > email: bqt at softjar.se || Reading murder books > pdp is alive! || tryin' to stay hip" - B. Idol > -------------- next part -------------- An HTML attachment was scrubbed... URL: From imp at bsdimp.com Sun May 21 04:18:30 2017 From: imp at bsdimp.com (Warner Losh) Date: Sat, 20 May 2017 12:18:30 -0600 Subject: [TUHS] Unix with TCP/IP for small PDP-11s In-Reply-To: References: Message-ID: On Sat, May 20, 2017 at 4:46 AM, Johnny Billquist wrote: > On 2017-05-20 04:00, Warner Losh wrote: > > https://ia601901.us.archive.org/10/items/bitsavers_decpdp11u >> lLTRIX112.0SPDSep84_870730/AE-X370C-TC_ULTRIX-11_2.0SPD_Sep84.pdf >> >> Looks like it requires MMU, but not split I/D space as it lists the >> following as compatible: M11, 11/23+, 11/24, 11/34, 11/40 and 11/60. It >> does require 256kb of memory. See table 2, page 6 for details. >> > > > Uh...? Where do you see that there is any TCP/IP support in Ultrix-11? If > any was done by someone else, there is no saying that it would be usable on > a machine without split I/D. To be honest, I've never seen any mention of > TCP/IP on any machine without split I/D space. I guess it could be done, > but it would be a rather big headache... I read the sources to see the TCP/IP support was there (that's the bit about adding Berkeley Sockets). I see nowhere that it's excluded for the non I/D machines, but haven't tried it first hand. I got interested not because of the PDP-11, but because I have an old Rainbow that recently started running Venix (v7-based version) and was trolling around for some way to do TCP/IP to it (though w/o readily available ethernet cards, I'm not sure it is a viable project). Warner -------------- next part -------------- An HTML attachment was scrubbed... URL: From arnold at skeeve.com Sun May 21 05:05:59 2017 From: arnold at skeeve.com (arnold at skeeve.com) Date: Sat, 20 May 2017 13:05:59 -0600 Subject: [TUHS] Unix with TCP/IP for small PDP-11s In-Reply-To: References: Message-ID: <201705201905.v4KJ5xcq005292@freefriends.org> Warner Losh wrote: > I read the sources to see the TCP/IP support was there (that's the bit > about adding Berkeley Sockets). I see nowhere that it's excluded for the > non I/D machines, but haven't tried it first hand. I got interested not > because of the PDP-11, but because I have an old Rainbow that recently > started running Venix (v7-based version) and was trolling around for some > way to do TCP/IP to it (though w/o readily available ethernet cards, I'm > not sure it is a viable project). Boy is the memory going. What was the TCP/IP implementation people ran on DOS to do connections over serial lines? Could that be found and revived for such a system? Thanks, Arnold From imp at bsdimp.com Sun May 21 06:29:20 2017 From: imp at bsdimp.com (Warner Losh) Date: Sat, 20 May 2017 14:29:20 -0600 Subject: [TUHS] Unix with TCP/IP for small PDP-11s In-Reply-To: <201705201905.v4KJ5xcq005292@freefriends.org> References: <201705201905.v4KJ5xcq005292@freefriends.org> Message-ID: On Sat, May 20, 2017 at 1:05 PM, wrote: > Warner Losh wrote: > > > I read the sources to see the TCP/IP support was there (that's the bit > > about adding Berkeley Sockets). I see nowhere that it's excluded for the > > non I/D machines, but haven't tried it first hand. I got interested not > > because of the PDP-11, but because I have an old Rainbow that recently > > started running Venix (v7-based version) and was trolling around for some > > way to do TCP/IP to it (though w/o readily available ethernet cards, I'm > > not sure it is a viable project). > > Boy is the memory going. What was the TCP/IP implementation people > ran on DOS to do connections over serial lines? Could that be found > and revived for such a system? > That would be mTCP with CSLIP. And under DOS people have that running already, but there's limitations and it's not designed for a Unix kernel.... Warner -------------- next part -------------- An HTML attachment was scrubbed... URL: From pechter at gmail.com Sun May 21 06:42:31 2017 From: pechter at gmail.com (William Pechter) Date: Sat, 20 May 2017 16:42:31 -0400 Subject: [TUHS] Fwd: Re: Unix with TCP/IP for small PDP-11s Message-ID: <31e7f50d-2bd7-450b-8a25-1818d8dfbb85.maildroid@localhost> -----Original Message----- From: pechter at gmail.com To: arnold at skeeve.com Sent: Sat, 20 May 2017 16:41 Subject: Re: [TUHS] Unix with TCP/IP for small PDP-11s Missed the reply all on the phone. Phil Karn had KA9Q in the 80s... It is mentioned on Wikipedia... Don't know much more. PPP might be better than slip. Bill -----Original Message----- From: arnold at skeeve.com To: pechter at gmail.com Sent: Sat, 20 May 2017 16:12 Subject: Re: [TUHS] Unix with TCP/IP for small PDP-11s Yes! Do you want to follow up to the list please? Thanks, Arnold William Pechter wrote: > KA9Q sound right? > > -----Original Message----- > From: arnold at skeeve.com > To: imp at bsdimp.com, bqt at update.uu.se > Cc: tuhs at minnie.tuhs.org > Sent: Sat, 20 May 2017 15:06 > Subject: Re: [TUHS] Unix with TCP/IP for small PDP-11s > > Warner Losh wrote: > > > I read the sources to see the TCP/IP support was there (that's the bit > > about adding Berkeley Sockets). I see nowhere that it's excluded for the > > non I/D machines, but haven't tried it first hand. I got interested not > > because of the PDP-11, but because I have an old Rainbow that recently > > started running Venix (v7-based version) and was trolling around for some > > way to do TCP/IP to it (though w/o readily available ethernet cards, I'm > > not sure it is a viable project). > > Boy is the memory going. What was the TCP/IP implementation people > ran on DOS to do connections over serial lines? Could that be found > and revived for such a system? > > Thanks, > > Arnold From henry.r.bent at gmail.com Sun May 21 06:44:43 2017 From: henry.r.bent at gmail.com (Henry Bent) Date: Sat, 20 May 2017 16:44:43 -0400 Subject: [TUHS] Unix with TCP/IP for small PDP-11s In-Reply-To: References: Message-ID: That was Ultrix-11 3.1, which I believe was the final version. Full source is available on tuhs.org. -Henry On 20 May 2017 at 16:40, Johnny Billquist wrote: > On 2017-05-20 19:39, Henry Bent wrote: > >> There is absolutely TCP/IP in Ultrix-11. This is an emulated 11/23+: >> > > Cool. But obviously not the version the SPD was for then. Which version of > Ultrix-11? > > > Johnny > > -- > Johnny Billquist || "I'm on a bus > || on a psychedelic trip > email: bqt at softjar.se || Reading murder books > pdp is alive! || tryin' to stay hip" - B. Idol > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bqt at update.uu.se Sun May 21 06:40:03 2017 From: bqt at update.uu.se (Johnny Billquist) Date: Sat, 20 May 2017 22:40:03 +0200 Subject: [TUHS] Unix with TCP/IP for small PDP-11s In-Reply-To: References: Message-ID: On 2017-05-20 19:39, Henry Bent wrote: > There is absolutely TCP/IP in Ultrix-11. This is an emulated 11/23+: Cool. But obviously not the version the SPD was for then. Which version of Ultrix-11? Johnny -- Johnny Billquist || "I'm on a bus || on a psychedelic trip email: bqt at softjar.se || Reading murder books pdp is alive! || tryin' to stay hip" - B. Idol From bqt at update.uu.se Sun May 21 06:40:49 2017 From: bqt at update.uu.se (Johnny Billquist) Date: Sat, 20 May 2017 22:40:49 +0200 Subject: [TUHS] Unix with TCP/IP for small PDP-11s In-Reply-To: References: Message-ID: On 2017-05-20 20:18, Warner Losh wrote: > > > On Sat, May 20, 2017 at 4:46 AM, Johnny Billquist > wrote: > > On 2017-05-20 04:00, Warner Losh > wrote: > > https://ia601901.us.archive.org/10/items/bitsavers_decpdp11ulLTRIX112.0SPDSep84_870730/AE-X370C-TC_ULTRIX-11_2.0SPD_Sep84.pdf > > > Looks like it requires MMU, but not split I/D space as it lists the > following as compatible: M11, 11/23+, 11/24, 11/34, 11/40 and > 11/60. It > does require 256kb of memory. See table 2, page 6 for details. > > > > Uh...? Where do you see that there is any TCP/IP support in > Ultrix-11? If any was done by someone else, there is no saying that > it would be usable on a machine without split I/D. To be honest, > I've never seen any mention of TCP/IP on any machine without split > I/D space. I guess it could be done, but it would be a rather big > headache... > > > I read the sources to see the TCP/IP support was there (that's the bit > about adding Berkeley Sockets). I see nowhere that it's excluded for the > non I/D machines, but haven't tried it first hand. I got interested not > because of the PDP-11, but because I have an old Rainbow that recently > started running Venix (v7-based version) and was trolling around for > some way to do TCP/IP to it (though w/o readily available ethernet > cards, I'm not sure it is a viable project). You do know that the Rainbow is not a PDP-11, right? Johnny -- Johnny Billquist || "I'm on a bus || on a psychedelic trip email: bqt at softjar.se || Reading murder books pdp is alive! || tryin' to stay hip" - B. Idol From imp at bsdimp.com Sun May 21 07:05:29 2017 From: imp at bsdimp.com (Warner Losh) Date: Sat, 20 May 2017 15:05:29 -0600 Subject: [TUHS] Unix with TCP/IP for small PDP-11s In-Reply-To: References: Message-ID: On Sat, May 20, 2017 at 2:40 PM, Johnny Billquist wrote: > On 2017-05-20 20:18, Warner Losh wrote: > >> >> >> On Sat, May 20, 2017 at 4:46 AM, Johnny Billquist > > wrote: >> >> On 2017-05-20 04:00, Warner Losh > > wrote: >> >> https://ia601901.us.archive.org/10/items/bitsavers_decpdp11u >> lLTRIX112.0SPDSep84_870730/AE-X370C-TC_ULTRIX-11_2.0SPD_Sep84.pdf >> > ulLTRIX112.0SPDSep84_870730/AE-X370C-TC_ULTRIX-11_2.0SPD_Sep84.pdf> >> >> Looks like it requires MMU, but not split I/D space as it lists >> the >> following as compatible: M11, 11/23+, 11/24, 11/34, 11/40 and >> 11/60. It >> does require 256kb of memory. See table 2, page 6 for details. >> >> >> >> Uh...? Where do you see that there is any TCP/IP support in >> Ultrix-11? If any was done by someone else, there is no saying that >> it would be usable on a machine without split I/D. To be honest, >> I've never seen any mention of TCP/IP on any machine without split >> I/D space. I guess it could be done, but it would be a rather big >> headache... >> >> >> I read the sources to see the TCP/IP support was there (that's the bit >> about adding Berkeley Sockets). I see nowhere that it's excluded for the >> non I/D machines, but haven't tried it first hand. I got interested not >> because of the PDP-11, but because I have an old Rainbow that recently >> started running Venix (v7-based version) and was trolling around for >> some way to do TCP/IP to it (though w/o readily available ethernet >> cards, I'm not sure it is a viable project). >> > > You do know that the Rainbow is not a PDP-11, right? Only since '84 when I bought mine... They are both small memory boxes, though, and something that can run in a 256kB machine should be able to run on a 892kB machine... I have bits of mch.c/low.s from Venix that I can leverage for the port to the Ultrix kernel to the 8088 that's inside the Rainbow it would be running on. Warner -------------- next part -------------- An HTML attachment was scrubbed... URL: From bqt at update.uu.se Sun May 21 07:34:30 2017 From: bqt at update.uu.se (Johnny Billquist) Date: Sat, 20 May 2017 23:34:30 +0200 Subject: [TUHS] Unix with TCP/IP for small PDP-11s In-Reply-To: References: Message-ID: <8f3662bc-f4f9-74c3-d648-15969230eacc@update.uu.se> On 2017-05-20 23:05, Warner Losh wrote: > > > On Sat, May 20, 2017 at 2:40 PM, Johnny Billquist > wrote: > > On 2017-05-20 20:18, Warner Losh wrote: > > I read the sources to see the TCP/IP support was there (that's > the bit > about adding Berkeley Sockets). I see nowhere that it's excluded > for the > non I/D machines, but haven't tried it first hand. I got > interested not > because of the PDP-11, but because I have an old Rainbow that > recently > started running Venix (v7-based version) and was trolling around for > some way to do TCP/IP to it (though w/o readily available ethernet > cards, I'm not sure it is a viable project). > > > You do know that the Rainbow is not a PDP-11, right? > > > Only since '84 when I bought mine... They are both small memory boxes, > though, and something that can run in a 256kB machine should be able to > run on a 892kB machine... I have bits of mch.c/low.s from Venix that I > can leverage for the port to the Ultrix kernel to the 8088 that's inside > the Rainbow it would be running on. Good. Just wanted to make sure you wouldn't get surprised/confused later. :-) No offense intended. That said, I suspect it is not that easy to move over, as this probably ties very closely in to how the hardware works, and more specifically the MMU. Thus, an 8088 have to do things rather differently than a PDP-11. Johnny -- Johnny Billquist || "I'm on a bus || on a psychedelic trip email: bqt at softjar.se || Reading murder books pdp is alive! || tryin' to stay hip" - B. Idol From davida at pobox.com Sun May 21 07:41:31 2017 From: davida at pobox.com (David Arnold) Date: Sun, 21 May 2017 07:41:31 +1000 Subject: [TUHS] Unix with TCP/IP for small PDP-11s In-Reply-To: <201705201905.v4KJ5xcq005292@freefriends.org> References: <201705201905.v4KJ5xcq005292@freefriends.org> Message-ID: > On 21 May 2017, at 05:05, arnold at skeeve.com wrote: > > Warner Losh wrote: > >> I read the sources to see the TCP/IP support was there (that's the bit >> about adding Berkeley Sockets). I see nowhere that it's excluded for the >> non I/D machines, but haven't tried it first hand. I got interested not >> because of the PDP-11, but because I have an old Rainbow that recently >> started running Venix (v7-based version) and was trolling around for some >> way to do TCP/IP to it (though w/o readily available ethernet cards, I'm >> not sure it is a viable project). > > Boy is the memory going. What was the TCP/IP implementation people > ran on DOS to do connections over serial lines? Could that be found > and revived for such a system? ka9q? d From fair-tuhs at netbsd.org Sun May 21 07:59:06 2017 From: fair-tuhs at netbsd.org (Erik E. Fair) Date: Sat, 20 May 2017 14:59:06 -0700 Subject: [TUHS] Unix with TCP/IP for small PDP-11s In-Reply-To: References: <201705201905.v4KJ5xcq005292@freefriends.org> Message-ID: <17526.1495317546@cesium.clock.org> Also MIT PC/IP: https://en.wikipedia.org/wiki/FTP_Software Erik Fair From random832 at fastmail.com Sun May 21 15:13:43 2017 From: random832 at fastmail.com (Random832) Date: Sun, 21 May 2017 01:13:43 -0400 Subject: [TUHS] Unix with TCP/IP for small PDP-11s In-Reply-To: <8f3662bc-f4f9-74c3-d648-15969230eacc@update.uu.se> References: <8f3662bc-f4f9-74c3-d648-15969230eacc@update.uu.se> Message-ID: <1495343623.794127.983422680.4D5F43AD@webmail.messagingengine.com> On Sat, May 20, 2017, at 17:34, Johnny Billquist wrote: > That said, I suspect it is not that easy to move over, as this probably > ties very closely in to how the hardware works, and more specifically > the MMU. Thus, an 8088 have to do things rather differently than a > PDP-11. Also, x86 is totally "split I/D" by any relevant definition, and has a 20-bit physical address space. From ron at ronnatalie.com Sun May 21 21:04:02 2017 From: ron at ronnatalie.com (Ron Natalie) Date: Sun, 21 May 2017 07:04:02 -0400 Subject: [TUHS] Unix with TCP/IP for small PDP-11s In-Reply-To: <1495343623.794127.983422680.4D5F43AD@webmail.messagingengine.com> References: <8f3662bc-f4f9-74c3-d648-15969230eacc@update.uu.se> <1495343623.794127.983422680.4D5F43AD@webmail.messagingengine.com> Message-ID: <020201d2d221$f55e08e0$e01a1aa0$@ronnatalie.com> > Also, x86 is totally "split I/D" by any relevant definition, and has a 20-bit physical address space. The physical address space isn't the issue. The PDP-11's (well the later ones) had 22 bits of address space. Even the amount of virtual address space wasn't the problem. The problem was really the lack of segments The PDP11, only had 8 for code and 8 for the combined stack/data (on the split I/D) machine. While we were able to "thunk" in a code overlay system without any real hardware support, there's no way to do that practically for data. As I stated earlier, using the BSD networking code, we ran out of the combined code/data registers when it came to having ot have one to map the mbufs. I supposed you could have added more magic (and perhaps th is is what the later Ultrix did), to carefully unmap one of the data segements to get to the network data, but boy, we never figured out a good way to do that when we tried. We retired all but the 11/70's (which we had three of) at the time as the Vaxes and other machines were beginning to roll in. In fact, I did eventually also turn one of the 11/70's into a router using my LOS (little operating system...no time for sharing, uniprocessor system) gateway OS. Each driver ran in it's own little program so it was a lot easier. From dave at horsfall.org Mon May 22 02:16:18 2017 From: dave at horsfall.org (Dave Horsfall) Date: Mon, 22 May 2017 02:16:18 +1000 (EST) Subject: [TUHS] Unix with TCP/IP for small PDP-11s In-Reply-To: References: <20170519151518.8800D18C09D@mercury.lcs.mit.edu> Message-ID: On Fri, 19 May 2017, Clem Cole wrote: > We did try to run on an 11/60 - which is 40 class as you asked, and we > did get it work as I recall after a fashion. Ah, the mighty 11/60; boy, but that takes me back. We bought one to replace our ageing /40, and I was fascinated by its octal display (yes, I wrote an idle loop to shift its digits back and forth; there should be an article somewhere in the AUUG archives). Story time: DEC were flogging the /60s real cheap here in Oz, because a deal with a major publisher (Murdoch? Packer? Who knows?) fell through at the last moment, and they had an entire warehouse to shift. Trivia: the thing was micro-codeable, and I had dreams of implementing the Unix call/return sequences as single instructions, but I never did get a round tuit. -- Dave Horsfall DTM (VK2KFU) "Those who don't understand security will suffer." From clemc at ccc.com Mon May 22 05:58:25 2017 From: clemc at ccc.com (Clem Cole) Date: Sun, 21 May 2017 15:58:25 -0400 Subject: [TUHS] Unix with TCP/IP for small PDP-11s In-Reply-To: References: <20170519151518.8800D18C09D@mercury.lcs.mit.edu> Message-ID: On Sun, May 21, 2017 at 12:16 PM, Dave Horsfall wrote: > Story time: DEC were flogging the /60s real cheap here in Oz, because a > ​ ​ > deal with a major publisher (Murdoch? Packer? Who knows?) fell through > ​ ​ > at the last moment, and they had an entire warehouse to shift. > ​It was world wide. The /60 was the fastest machine from release to traditional product in DEC's history.​ > > Trivia: the thing was micro-codeable, and I had dreams of implementing the > ​ ​ > Unix call/return sequences as single instructions, but I never did get a > ​ ​ > round tuit. > ​Indeed. @ Tek we had both the DEC tools and the CMU tools and the 11/40e microcode from CMU (and the compiler that used csav/cret - probably still do somewhere). One problem was the microcode compiler from DEC ran on RSX and CMU's ran on TOPS-10 of which I had neither. ​ The /60 and /40 had similar but different base microengines and it was going to take some work reimplementing the CMU extens for the 60. I've forgotten the details of what changed in the microengine, but something was hugely different and enough that the CMU microcode would have been a model, but would not work even if I could manage to get the tools together. So, it was also on my todo list, but we got an 11/70 shortly after I pulled together the pieces and we never got to it. What I can tell you from the CMU experience is that it bought a small amount of speed up, but the big thing code size reduction for some programs and on 40 class machine -- anything like that helped. But... the problem was it made the binaries not portable. We had to be very careful. The EE production machines (which were 11/34 and later /34As) binaries of course could run on the CS systems, but the CS machines with the /40e microcode, compiler et al, created binaries that core dumped on the 34. Worse yet, was accidently mixing libraries. Ted and I talked about hacking the EE kernel to support a UUI like the FPs did could take the CS binaries and run them. But that project never occurred - which was probably a shame. Instead, Danny Klein and I got really good at purging any tape or disk that passed through a CS system for anything binary on it. Clem -------------- next part -------------- An HTML attachment was scrubbed... URL: From ron at ronnatalie.com Mon May 22 06:57:32 2017 From: ron at ronnatalie.com (Ron Natalie) Date: Sun, 21 May 2017 16:57:32 -0400 Subject: [TUHS] Unix with TCP/IP for small PDP-11s In-Reply-To: References: <20170519151518.8800D18C09D@mercury.lcs.mit.edu> Message-ID: <024601d2d274$de4fb290$9aef17b0$@ronnatalie.com> The 11/60’s main claim to fame if I recall was a writable control store. Don’t know anybody who actually used that feature. -------------- next part -------------- An HTML attachment was scrubbed... URL: From clemc at ccc.com Mon May 22 07:26:42 2017 From: clemc at ccc.com (Clem Cole) Date: Sun, 21 May 2017 17:26:42 -0400 Subject: [TUHS] Unix with TCP/IP for small PDP-11s In-Reply-To: <024601d2d274$de4fb290$9aef17b0$@ronnatalie.com> References: <20170519151518.8800D18C09D@mercury.lcs.mit.edu> <024601d2d274$de4fb290$9aef17b0$@ronnatalie.com> Message-ID: Actually, the 11/60's main claim to fame was it supposed to be a 'commercial' PDP-11 and was built for the small business market. The WCS was a side effect. It was to built to run RSTS and RSX and had a commercial instruction set exten *etc.*. Somebody had written a 'dentist office' package for it and a 'car dealership package' IIRC. And was physically packaged a tad differently than the other 11's as it was trying to be marketing to places that might want to show it off instead of hiding it in a computer room. The main competition at the time was the Burrough 1700 and the IBM System 38 and DEC was trying to find a system to compete in the market. Now, the B1700 is a very cool system and could swap its microcode depending on the language running (it is also bit addressable). As I understand it from talking to the 11/60s product manager many years later at a party, one of the ideas was that the PDP 11/60 WCS an attempt to be able offer features that might be needed to compete with it. The the real truth was that issue in the original 11's had stored the ucode in PROM and if an ECO went out the CPU boards had be swapped, so the chips could be swapped at the factory. Plus a number of places, like MIT and CMU had been banging on DEC to make the ucoding of the system an available option (CMU just built it's own the 11/40e that triple drip made for them). Anyway, as the price of SRAM had fallen dramatically from when the 11 was introduced, and the 11/60 was storing the primary ucode it, not PROM like the other 45 and 40 class systems; WCS became a possible feature to expose. But, as I said, the problem was that the WCS tools were not very portable and the microengine was (as I remember) not well enough documented. Which is why I don't think many people tried. Also the 11/60 also had a unique feature we called the HCU instruction - Halt and Confuse Microcode. Unlike the 11/34 which had an Intel 8008 that ran front (programmers) console, the 11/60's microcode loop read the LED's and keypad. But ... if the ucode took a dive, the only recourse was turn off the system - as it would stop responding to anything (just like PC years later ;-) We never found what sequence caused it; but when I was debugging the original 11/60 port of V6, I would run into a lot ;-) IIRC - we realized on these pages that Dave Horsfall in OZ and I in Oregon must have been doing similar work around the same time. FWIW, my changes made it back to AT&T via Ted and I believe eventually went out in the System III release. That said, I suspect if we were to compare our patches I bet they are pretty similar. Clem On Sun, May 21, 2017 at 4:57 PM, Ron Natalie wrote: > The 11/60’s main claim to fame if I recall was a writable control store. > Don’t know anybody who actually used that feature. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pechter at gmail.com Mon May 22 07:46:15 2017 From: pechter at gmail.com (William Pechter) Date: Sun, 21 May 2017 17:46:15 -0400 Subject: [TUHS] Unix with TCP/IP for small PDP-11s In-Reply-To: <024601d2d274$de4fb290$9aef17b0$@ronnatalie.com> References: <20170519151518.8800D18C09D@mercury.lcs.mit.edu> <024601d2d274$de4fb290$9aef17b0$@ronnatalie.com> Message-ID: I believe Reuters turned them into faster PDP8s via Microcode changes... -----Original Message----- From: Ron Natalie To: 'Clem Cole' , 'Dave Horsfall' Cc: 'The Eunuchs Hysterical Society' Sent: Sun, 21 May 2017 16:58 Subject: Re: [TUHS] Unix with TCP/IP for small PDP-11s The 11/60’s main claim to fame if I recall was a writable control store. Don’t know anybody who actually used that feature. From bqt at update.uu.se Mon May 22 08:42:35 2017 From: bqt at update.uu.se (Johnny Billquist) Date: Mon, 22 May 2017 00:42:35 +0200 Subject: [TUHS] 11/60 (was: Unix with TCP/IP for small PDP-11s) In-Reply-To: References: Message-ID: <34efc530-b7be-ba8b-1762-756e60915ce5@update.uu.se> On 2017-05-21 23:27, Clem Cole wrote: > Actually, the 11/60's main claim to fame was it supposed to be a > 'commercial' PDP-11 and was built for the small business market. The WCS > was a side effect. As you mentioned in another mail, it's main claim to fame was probably the short time it actually existed in the market. DEC seriously did some things wrong on it, such as only 18-bit address, and no split I/D space, at a time when pretty much any other PDP-11 was going there. But it was also one of a couple of -11 models where you had the ability to write your own microcode. But I've never heard of anyone who had the WCS option. (But at one point, I was playing with four 11/60 machines in a computer club.) I don't remember if it was you or someone else who said that there were several microcode bugs in the 11/60. It ran the DEC OSes, but there were some issues with Unix. I seem to also remember seeing some special code in the RSX kernel for some 11/60 oddity, but I would have to search through the code to remember exactly what that was about. > It was to built to run RSTS and RSX and had a commercial instruction set > exten *etc.*. Somebody had written a 'dentist office' package for it and a > 'car dealership package' IIRC. And was physically packaged a tad > differently than the other 11's as it was trying to be marketing to places > that might want to show it off instead of hiding it in a computer room. Uh? I have not seen any plans ever mentioning CIS for the 11/60, but I guess it's possible it was considered at some point. But I can assure you no CIS ever existed for the 11/60. The only machines that ever had the CIS option was the 11/44 and 11/23,11/24. But I have no idea how many machines ever actually had the option installed. One nice detail of the 11/60 is that it had the FPP build in. But there was also an optional hardware FPP accelerator available. Johnny -- Johnny Billquist || "I'm on a bus || on a psychedelic trip email: bqt at softjar.se || Reading murder books pdp is alive! || tryin' to stay hip" - B. Idol From wkt at tuhs.org Mon May 22 11:15:05 2017 From: wkt at tuhs.org (Warren Toomey) Date: Mon, 22 May 2017 11:15:05 +1000 Subject: [TUHS] KA9Q In-Reply-To: <31e7f50d-2bd7-450b-8a25-1818d8dfbb85.maildroid@localhost> References: <31e7f50d-2bd7-450b-8a25-1818d8dfbb85.maildroid@localhost> Message-ID: <20170522011505.GA8612@minnie.tuhs.org> On Sat, May 20, 2017 at 04:42:31PM -0400, William Pechter wrote: > Missed the reply all on the phone. Phil Karn had KA9Q in the 80s. Minnie ran an FTP service on KA9Q on MS-DOS to start with. She had a 30M RLL drive with a WD8003E NIC. I'm so glad she isn't still doing that :) Warren (yes, off topic, mea culpa) From b4 at gewt.net Mon May 22 12:44:58 2017 From: b4 at gewt.net (Cory Smelosky) Date: Sun, 21 May 2017 19:44:58 -0700 Subject: [TUHS] KA9Q In-Reply-To: <20170522011505.GA8612@minnie.tuhs.org> References: <31e7f50d-2bd7-450b-8a25-1818d8dfbb85.maildroid@localhost> <20170522011505.GA8612@minnie.tuhs.org> Message-ID: <00B1142B-9DC4-4DC8-ABAC-ECD7F21F09E2@gewt.net> I keep meaning to try similar for SysVR2 Sent from my iPhone > On May 21, 2017, at 18:15, Warren Toomey wrote: > >> On Sat, May 20, 2017 at 04:42:31PM -0400, William Pechter wrote: >> Missed the reply all on the phone. Phil Karn had KA9Q in the 80s. > > Minnie ran an FTP service on KA9Q on MS-DOS to start with. She had a 30M RLL > drive with a WD8003E NIC. I'm so glad she isn't still doing that :) > > Warren > > (yes, off topic, mea culpa) From dave at horsfall.org Mon May 22 16:00:33 2017 From: dave at horsfall.org (Dave Horsfall) Date: Mon, 22 May 2017 16:00:33 +1000 (EST) Subject: [TUHS] KA9Q In-Reply-To: <20170522011505.GA8612@minnie.tuhs.org> References: <31e7f50d-2bd7-450b-8a25-1818d8dfbb85.maildroid@localhost> <20170522011505.GA8612@minnie.tuhs.org> Message-ID: On Mon, 22 May 2017, Warren Toomey wrote: > Minnie ran an FTP service on KA9Q on MS-DOS to start with. She had a 30M > RLL drive with a WD8003E NIC. I'm so glad she isn't still doing that :) Ahh... I remember Minnie well; those were the days... -- Dave Horsfall DTM (VK2KFU) "Those who don't understand security will suffer." From grog at lemis.com Mon May 22 16:50:10 2017 From: grog at lemis.com (Greg 'groggy' Lehey) Date: Mon, 22 May 2017 16:50:10 +1000 Subject: [TUHS] KA9Q In-Reply-To: References: <31e7f50d-2bd7-450b-8a25-1818d8dfbb85.maildroid@localhost> <20170522011505.GA8612@minnie.tuhs.org> Message-ID: <20170522065010.GC79144@eureka.lemis.com> On Monday, 22 May 2017 at 16:00:33 +1000, Dave Horsfall wrote: > Received: from minnie.tuhs.org (minnie.tuhs.org [45.79.103.53]) > by www.lemis.com (Postfix) with ESMTP id 3D8211B72804 > > On Mon, 22 May 2017, Warren Toomey wrote: > >> Minnie ran an FTP service on KA9Q on MS-DOS to start with. She had a 30M >> RLL drive with a WD8003E NIC. I'm so glad she isn't still doing that :) > > Ahh... I remember Minnie well; those were the days... They still are, for a certain incarnation of minnie. Greg -- Sent from my desktop computer. Finger grog at lemis.com for PGP public key. See complete headers for address and phone numbers. This message is digitally signed. If your Microsoft mail program reports problems, please read http://lemis.com/broken-MUA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 163 bytes Desc: not available URL: From pnr at planet.nl Mon May 22 19:28:19 2017 From: pnr at planet.nl (Paul Ruizendaal) Date: Mon, 22 May 2017 11:28:19 +0200 Subject: [TUHS] Unix with TCP/IP for small PDP-11s Message-ID: <35238F37-6324-4D40-A6C0-7957F2E22964@planet.nl> I had somehow convinced myself that Ultrix-11 needed split I/D, but indeed it does not: # file unix unix: (0450) pure overlay executable not stripped # size unix 14784+(8192,8000,8064,8000,8064,8128,8000,7808,7936,7936,7680,7360,1344)+3524+13500 = 31808b = 076100b (111296 total text) With only 16KB of permanent kernel there will be a lot of overlay switching. I'm not entirely sure why bss could not be 1KB smaller, enabling 8KB more of permanent kernel. The loss of performance from 2 disk buffers less really outweighed less overlay switching? If I understand correctly, the network code continuously switches around segment 5 to access the right mbuf. According to the notes in the TUHS archive (http://www.tuhs.org/Archive/Distributions/DEC/Ultrix-11/Fred-Ultrix3/setup-3.1.txt), running Ultrix-11 with networking on a 11/40 class machine is borderline workable: "I have personally tested it on a 23+, 53 and 83. I know it runs fine on the 73. The smaller machines (34, 40 etc) should work akin to the 23, meaning using overlays and be very tight on RAM for the drivers. TCP/IP is a biiiiig load for those systems!" There are two other routes to TCP/IP on a PDP11 without split I/D: - 3COM's TCP/IP package (initially an overlay over V7, soon after also over 2BSD); I believe the source to this is lost. - DCEC's adaptation of the Wingfield TCP/IP library, designed to work with V6. It is mostly a user space daemon, but requires some kernel enhancements. The Wingfield code is in the TUHS archive, but that version has a modified V6 kernel that also supports NCP networking and requires split I/D. If used with a minimally enhanced V6 kernel, it would easily fit in 64KB, without overlays. Note that these last two options have very different API's and would not be so easy to work with. Paul From clemc at ccc.com Tue May 23 00:09:30 2017 From: clemc at ccc.com (Clem Cole) Date: Mon, 22 May 2017 10:09:30 -0400 Subject: [TUHS] Unix with TCP/IP for small PDP-11s In-Reply-To: <35238F37-6324-4D40-A6C0-7957F2E22964@planet.nl> References: <35238F37-6324-4D40-A6C0-7957F2E22964@planet.nl> Message-ID: On Mon, May 22, 2017 at 5:28 AM, Paul Ruizendaal wrote: > There are two other routes to TCP/IP on a PDP11 without split I/D: > - 3COM's TCP/IP package (initially an overlay over V7, soon after also > over 2BSD); > ​It's called UNET​ > I believe the source to this is lost. > ​So far that is true, there are a couple of leads including a couple of unread tapes in my own basement. The original authors and I have been in touch. Assuming we can find and recover it, there is still an open issue of copyright.​ > ​.....​ > > > Note that these last two options have very different API's > ​Indeed...​ > and would not be so easy to work with. > ​Mumble ... it just means post-Joy sockets code does not 'just work.' UNET uses the Rand interface, which is similar to the interface MIT used for Chaos and Chesson used for the ARPAnet code at UofI, so there are programs that talk to it. We had mail, ftp and telnet working just fine.​ ​ Back in the day and IIRC we were playing with a port of the MIT SUPDUP code but I do not remember if that ever worked on the small address machines.​ -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at quintile.net Tue May 23 00:51:22 2017 From: steve at quintile.net (Steve Simon) Date: Mon, 22 May 2017 15:51:22 +0100 Subject: [TUHS] Unix with TCP/IP for small PDP-11s In-Reply-To: References: <35238F37-6324-4D40-A6C0-7957F2E22964@planet.nl> Message-ID: <80EBBEAA-5774-4594-B69E-00FB1EE0F101@quintile.net> hi Clem, do you have pointers to any documentation on the rand/MIT network API? -Steve > On 22 May 2017, at 15:09, Clem Cole wrote: > > >> On Mon, May 22, 2017 at 5:28 AM, Paul Ruizendaal wrote: >> There are two other routes to TCP/IP on a PDP11 without split I/D: >> - 3COM's TCP/IP package (initially an overlay over V7, soon after also over 2BSD); > ​It's called UNET​ > > > >> I believe the source to this is lost. > ​So far that is true, there are a couple of leads including a couple of unread tapes in my own basement. The original authors and I have been in touch. Assuming we can find and recover it, there is still an open issue of copyright.​ > > >> ​.....​ >> >> >> Note that these last two options have very different API's > ​Indeed...​ > > > >> and would not be so easy to work with. > ​Mumble ... it just means post-Joy sockets code does not 'just work.' UNET uses the Rand interface, which is similar to the interface MIT used for Chaos and Chesson used for the ARPAnet code at UofI, so there are programs that talk to it. We had mail, ftp and telnet working just fine.​ ​ Back in the day and IIRC we were playing with a port of the MIT SUPDUP code but I do not remember if that ever worked on the small address machines.​ > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From clemc at ccc.com Tue May 23 02:29:09 2017 From: clemc at ccc.com (Clem Cole) Date: Mon, 22 May 2017 12:29:09 -0400 Subject: [TUHS] Unix with TCP/IP for small PDP-11s In-Reply-To: <80EBBEAA-5774-4594-B69E-00FB1EE0F101@quintile.net> References: <35238F37-6324-4D40-A6C0-7957F2E22964@planet.nl> <80EBBEAA-5774-4594-B69E-00FB1EE0F101@quintile.net> Message-ID: The Rand API is documented in a couple of papers that have been discussed here previously. I'll have to do a little googling to find them. The ChaosNET docs are the best MIT docs. The API pretty much worked this way... there was one simple kernel hook. A small mod was done to the old nami routine (what modern UNIX calls 'lookup'), as a path was being parsed, the rest of the path was left in place made available to other routines. Thus you could open: /dev/tcp/xxx and the string xxx was available in call that was implementing tcp and was then able to be passed xxx as a parameter. Hence, the traditional open/close/read/write was all that was needed. Joy codified a new API that was supposed to be more network centric and map to different network protocols - the thinking being that open/close/read/write were insufficient semantics for network operations. Also, the only other issue at the time was that BSD's select(2) did not exist, and the UNIX I/O were 100% synchronous. So some other mechanism (also discussed here) needed to be created to avoid blocking in the application. There were a couple of different schemes with V7's multiplexor call was created by Chesson for DataKit (which was similar to was used in the UofI code Arpanet code). Rand, UNET & Chaos had something else that gave the save async function, who's name I've forgotten at the moment, but I believe Noel posted the code for same in the last year from one of the MIT kernels - we had used it at CMU that we had gotten from Rand. On Mon, May 22, 2017 at 10:51 AM, Steve Simon wrote: > hi Clem, > > do you have pointers to any documentation on the rand/MIT network API? > > -Steve > > > > > On 22 May 2017, at 15:09, Clem Cole wrote: > > > On Mon, May 22, 2017 at 5:28 AM, Paul Ruizendaal wrote: > >> There are two other routes to TCP/IP on a PDP11 without split I/D: >> - 3COM's TCP/IP package (initially an overlay over V7, soon after also >> over 2BSD); >> > ​It's called UNET​ > > > > >> I believe the source to this is lost. >> > ​So far that is true, there are a couple of leads including a couple of > unread tapes in my own basement. The original authors and I have been in > touch. Assuming we can find and recover it, there is still an open issue > of copyright.​ > > > > >> ​.....​ >> >> >> Note that these last two options have very different API's >> > ​Indeed...​ > > > > >> and would not be so easy to work with. >> > ​Mumble ... it just means post-Joy sockets code does not 'just work.' > UNET uses the Rand interface, which is similar to the interface MIT used > for Chaos and Chesson used for the ARPAnet code at UofI, so there are > programs that talk to it. We had mail, ftp and telnet working just fine.​ > > ​ Back in the day and IIRC we were playing with a port of the MIT SUPDUP > code but I do not remember if that ever worked on the small address > machines.​ > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ron at ronnatalie.com Tue May 23 02:35:34 2017 From: ron at ronnatalie.com (Ron Natalie) Date: Mon, 22 May 2017 12:35:34 -0400 Subject: [TUHS] Unix with TCP/IP for small PDP-11s In-Reply-To: References: <35238F37-6324-4D40-A6C0-7957F2E22964@planet.nl> <80EBBEAA-5774-4594-B69E-00FB1EE0F101@quintile.net> Message-ID: <003701d2d319$70919d40$51b4d7c0$@ronnatalie.com> Gak, the days before select. I remember writing some terminal programs where I had to fork and do the synchronous read in each direction separately. -------------- next part -------------- An HTML attachment was scrubbed... URL: From pnr at planet.nl Tue May 23 08:07:34 2017 From: pnr at planet.nl (Paul Ruizendaal) Date: Tue, 23 May 2017 00:07:34 +0200 Subject: [TUHS] Unix with TCP/IP for small PDP-11s In-Reply-To: References: <35238F37-6324-4D40-A6C0-7957F2E22964@planet.nl> <80EBBEAA-5774-4594-B69E-00FB1EE0F101@quintile.net> Message-ID: <830AF76B-3E3C-4C76-9ED6-959AC38AF6EE@planet.nl> Clem, You've got me confused. > The API pretty much worked this way... there was one simple kernel hook. A small mod was done to the old nami routine (what modern UNIX calls 'lookup'), as a path was being parsed, the rest of the path was left in place made available to other routines. The UoI Arpanet code did not use the partial path parsing that you describe. The namei() routine is unchanged (see http://www.tuhs.org/cgi-bin/utree.pl?file=SRI-NOSC/ken/nami.c). The code recognizes a network open, because the open() is of a special character file with major number 255; the minor number would be the remote address (1974 arpanet used 8 bit network addresses - 6 bits IMP, 2 bits host). If the 'mode' parameter to open() is 0, 1 or 2, then a NCP/telnet connection would be made to that host. If the mode parameter was anything else, it was a pointer to a data block with the full specification of the connection to be made. See: http://www.tuhs.org/cgi-bin/utree.pl?file=SRI-NOSC/ken/sys2.c - open1() http://www.tuhs.org/cgi-bin/utree.pl?file=SRI-NOSC/ncpk/nopcls.c - netopen() The data block contained all the parameters that are set via calls to connect/bind/listen in the BSD sockets API. > There were a couple of different schemes with V7's multiplexor call was created by Chesson for DataKit (which was similar to was used in the UofI code Arpanet code). You've lost me here. Paul On 22 May 2017, at 18:29 , Clem Cole wrote: > The Rand API is documented in a couple of papers that have been discussed here previously. I'll have to do a little googling to find them. The ChaosNET docs are the best MIT docs. > > > The API pretty much worked this way... there was one simple kernel hook. A small mod was done to the old nami routine (what modern UNIX calls 'lookup'), as a path was being parsed, the rest of the path was left in place made available to other routines. > > Thus you could open: /dev/tcp/xxx and the string xxx was available in call that was implementing tcp and was then able to be passed xxx as a parameter. Hence, the traditional open/close/read/write was all that was needed. > > Joy codified a new API that was supposed to be more network centric and map to different network protocols - the thinking being that open/close/read/write were insufficient semantics for network operations. > > Also, the only other issue at the time was that BSD's select(2) did not exist, and the UNIX I/O were 100% synchronous. So some other mechanism (also discussed here) needed to be created to avoid blocking in the application. There were a couple of different schemes with V7's multiplexor call was created by Chesson for DataKit (which was similar to was used in the UofI code Arpanet code). Rand, UNET & Chaos had something else that gave the save async function, who's name I've forgotten at the moment, but I believe Noel posted the code for same in the last year from one of the MIT kernels - we had used it at CMU that we had gotten from Rand. > > On Mon, May 22, 2017 at 10:51 AM, Steve Simon wrote: > hi Clem, > > do you have pointers to any documentation on the rand/MIT network API? > > -Steve From clemc at ccc.com Tue May 23 09:25:34 2017 From: clemc at ccc.com (Clem Cole) Date: Mon, 22 May 2017 19:25:34 -0400 Subject: [TUHS] Unix with TCP/IP for small PDP-11s In-Reply-To: <830AF76B-3E3C-4C76-9ED6-959AC38AF6EE@planet.nl> References: <35238F37-6324-4D40-A6C0-7957F2E22964@planet.nl> <80EBBEAA-5774-4594-B69E-00FB1EE0F101@quintile.net> <830AF76B-3E3C-4C76-9ED6-959AC38AF6EE@planet.nl> Message-ID: We'll have to do some digging. And having too been part of too many, but similar of different implementations. The Chaos code used the partial parsing as did a few others. IIRC: That's the same trick Apollo used with Domain too. I had thought I had remember that Chesson and the UofI folks used the same trick with the Arpanet code, but I have not looked at it in years and clearly as you point out he did not. As for DataKit et al.. Greg Chesson was a grad student at UoI. The UoI folks did the original V6 Arpanet for UNIX code and Greg was the one of the primary developers same. What I do not remember is who came first, the Rand networking work for the UoI work. Rand did the the ports (later called named pipes) and a few other things pretty early. But again all those dates sort of mix together in the early middle 70s. USENIX was not yet publishing proceedings so it's hard to keep straight. It pretty much just email and memories of those of us that were sharing things at the time. Anyway, when Greg graduated, he was hired by Ken when he finished and developed DataKit at Bell labs. One of the pieces of datakit that was released as part of V7 was Greg's mpx(2) code - which was the multiplexer. The legend is that on is way to UCB for his sabbatical, Ken was checking in Greg and the infamous v6 patches tape was left behind at UoI. Greg used to deny all knowledge of same, but smiled all the time he said it. Clem On Mon, May 22, 2017 at 6:07 PM, Paul Ruizendaal wrote: > Clem, > > You've got me confused. > > > The API pretty much worked this way... there was one simple kernel > hook. A small mod was done to the old nami routine (what modern UNIX > calls 'lookup'), as a path was being parsed, the rest of the path was left > in place made available to other routines. > > The UoI Arpanet code did not use the partial path parsing that you > describe. > The namei() routine is unchanged (see http://www.tuhs.org/cgi-bin/ > utree.pl?file=SRI-NOSC/ken/nami.c). > > The code recognizes a network open, because the open() is of a special > character file with major number 255; > the minor number would be the remote address (1974 arpanet used 8 bit > network addresses - 6 bits IMP, 2 bits host). > If the 'mode' parameter to open() is 0, 1 or 2, then a NCP/telnet > connection would be made to that host. If the mode parameter was anything > else, it was a pointer to a data block with the full specification of the > connection to be made. See: > http://www.tuhs.org/cgi-bin/utree.pl?file=SRI-NOSC/ken/sys2.c - open1() > http://www.tuhs.org/cgi-bin/utree.pl?file=SRI-NOSC/ncpk/nopcls.c - > netopen() > > The data block contained all the parameters that are set via calls to > connect/bind/listen in the BSD sockets API. > > > There were a couple of different schemes with V7's multiplexor call was > created by Chesson for DataKit (which was similar to was used in the UofI > code Arpanet code). > > You've lost me here. > > Paul > > On 22 May 2017, at 18:29 , Clem Cole wrote: > > > The Rand API is documented in a couple of papers that have been > discussed here previously. I'll have to do a little googling to find > them. The ChaosNET docs are the best MIT docs. > > > > > > The API pretty much worked this way... there was one simple kernel > hook. A small mod was done to the old nami routine (what modern UNIX > calls 'lookup'), as a path was being parsed, the rest of the path was left > in place made available to other routines. > > > > Thus you could open: /dev/tcp/xxx and the string xxx was available in > call that was implementing tcp and was then able to be passed xxx as a > parameter. Hence, the traditional open/close/read/write was all that was > needed. > > > > Joy codified a new API that was supposed to be more network centric and > map to different network protocols - the thinking being that > open/close/read/write were insufficient semantics for network operations. > > > > Also, the only other issue at the time was that BSD's select(2) did not > exist, and the UNIX I/O were 100% synchronous. So some other mechanism > (also discussed here) needed to be created to avoid blocking in the > application. There were a couple of different schemes with V7's > multiplexor call was created by Chesson for DataKit (which was similar to > was used in the UofI code Arpanet code). Rand, UNET & Chaos had something > else that gave the save async function, who's name I've forgotten at the > moment, but I believe Noel posted the code for same in the last year from > one of the MIT kernels - we had used it at CMU that we had gotten from Rand. > > > > On Mon, May 22, 2017 at 10:51 AM, Steve Simon > wrote: > > hi Clem, > > > > do you have pointers to any documentation on the rand/MIT network API? > > > > -Steve > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pnr at planet.nl Tue May 23 10:36:06 2017 From: pnr at planet.nl (Paul Ruizendaal) Date: Tue, 23 May 2017 02:36:06 +0200 Subject: [TUHS] Unix with TCP/IP for small PDP-11s In-Reply-To: References: <35238F37-6324-4D40-A6C0-7957F2E22964@planet.nl> <80EBBEAA-5774-4594-B69E-00FB1EE0F101@quintile.net> <830AF76B-3E3C-4C76-9ED6-959AC38AF6EE@planet.nl> Message-ID: On 23 May 2017, at 1:25 , Clem Cole wrote: > As for DataKit et al.. Greg Chesson was a grad student at UoI. The UoI folks did the original V6 Arpanet for UNIX code and Greg was the one of the primary developers same. What I do not remember is who came first, the Rand networking work for the UoI work. Rand did the the ports (later called named pipes) and a few other things pretty early. But again all those dates sort of mix together in the early middle 70s. USENIX was not yet publishing proceedings so it's hard to keep straight. It pretty much just email and memories of those of us that were sharing things at the time. These dates I can fill in: - UoI Arpanet Unix was initially done between December 1974 and March 1975, building on the experience gained with the earlier ANTS I and II projects. - Rand ports were done in 1977, under contract to the air force (report dated June '77). > Anyway, when Greg graduated, he was hired by Ken when he finished and developed DataKit at Bell labs. One of the pieces of datakit that was released as part of V7 was Greg's mpx(2) code - which was the multiplexer. Interesting. I always thought that mpx files were driven by the work on the Blit terminal, the link to networking is new to me. I'm looking into the history of Spider and early Datakit. Sandy Fraser was kind enough to send me the 1974 report on Spider and it already mentions actual usage for remote printing, remote login and a central 'file store'. Spider was an interesting bit of technology, essentially it linked a dozen or so computers over a 1.5Mb/s shared link (a daisy chained twisted pair cable) to a central router/switch. There was only ever one Spider router, but the design allowed for multiple routers to be connected over T1 long distance lines. Does anybody know of surviving v5/v6/v7 code for Spider networking (e.g. the 'tiu' device driver, the 'nfs' file transfer package, etc.)? From jnc at mercury.lcs.mit.edu Tue May 23 11:14:52 2017 From: jnc at mercury.lcs.mit.edu (Noel Chiappa) Date: Mon, 22 May 2017 21:14:52 -0400 (EDT) Subject: [TUHS] Unix with TCP/IP for small PDP-11s Message-ID: <20170523011452.99DD518C095@mercury.lcs.mit.edu> > From: Paul Ruizendaal > There are two other routes to TCP/IP on a PDP11 without split I/D: > ... > DCEC's adaptation of the Wingfield TCP/IP library, designed to work > with V6. It is mostly a user space daemon, but requires some kernel > enhancements. I wonder what the performance would be like, since the TCP is in a user process (a different one from the application), i.e. there's a process switch every time the application goes to send or receive data. This wouldn't have been such an issue when the code was written, since ARPANet-type networks were not very fast, but with a better network, it would have been limiting. > From: Steve Simon > do you have pointers to any documentation on the rand/MIT network API? There was no 'MIT' network API. He was talking about the CHAOSNet API. The TCP/IP done in the CSR group at MIT used a totally different API. The various Unix systems at MIT were pretty well out of touch with each other, and did not exchange code. The only exceptions were the DSSR (later RTS) and CSR groups in Tech Sq, who used pretty much the same system. Noel From jnc at mercury.lcs.mit.edu Tue May 23 11:33:41 2017 From: jnc at mercury.lcs.mit.edu (Noel Chiappa) Date: Mon, 22 May 2017 21:33:41 -0400 (EDT) Subject: [TUHS] Unix with TCP/IP for small PDP-11s Message-ID: <20170523013341.CAA4118C095@mercury.lcs.mit.edu> > From: Clem Cole > So some other mechanism (also discussed here) needed to be created to > avoid blocking in the application. > ... > Rand, UNET & Chaos had something else that gave the save async function, > who's name I've forgotten at the moment I don't think the RAND code had the non-blocking stuff; AFAICR, all it had was named pipes (effectively). Jack Haverty at BBN defined and implemented two new calls (IIRC, 'capac()' and 'await()') to do non-blocking I/O. The documentation for that is in the 'BBN' branch at TUHS: http://minnie.tuhs.org/cgi-bin/utree.pl?file=BBN-V6/doc/ipc/await http://minnie.tuhs.org/cgi-bin/utree.pl?file=BBN-V6/doc/ipc/ipc My memory might be incorrect, but I don't think it was asynchronous (i.e. a process issued a read() or write(), and that returned right away, before the I/O was actually done, and the system notified the process later when the I/O actually completed). I actually did implement asyn I/O for an early LAN device driver - and just to make it fun, the device was a DMA device, and we didn't want the overhead of a copy, so the DMA was direct to buffers in the process - i.e. 'raw' I/O. So that required some major system tweaks, to keep the process from being swapped out - or moved around - while the I/O was pending. > I believe Noel posted the code for same in the last year from one of the > MIT kernels I found it on the dump of an MIT machine, but it was never run on any machine at MIT - we just had the source in case we had any use fot it. Noel From pnr at planet.nl Tue May 23 21:35:02 2017 From: pnr at planet.nl (Paul Ruizendaal) Date: Tue, 23 May 2017 13:35:02 +0200 Subject: [TUHS] Unix with TCP/IP for small PDP-11s Message-ID: <12C21350-E009-46A8-B50F-76D2AA6A8CF7@planet.nl> >> There are two other routes to TCP/IP on a PDP11 without split I/D: >> ... >> DCEC's adaptation of the Wingfield TCP/IP library, designed to work >> with V6. It is mostly a user space daemon, but requires some kernel >> enhancements. > > I wonder what the performance would be like, since the TCP is in a user > process (a different one from the application), i.e. there's a process switch > every time the application goes to send or receive data. This wouldn't have > been such an issue when the code was written, since ARPANet-type networks > were not very fast, but with a better network, it would have been limiting. IEN98 (http://www.rfc-editor.org/rfc/ien/ien98.txt, page 2) has the answer: about 10kb/s. The DCEC version used shared memory instead of rand ports and was claimed to be a bit more performant, but I have no number. I'd be surprised if it was twice as fast, so perhaps 15kb/s. Paul From jnc at mercury.lcs.mit.edu Tue May 23 23:43:51 2017 From: jnc at mercury.lcs.mit.edu (Noel Chiappa) Date: Tue, 23 May 2017 09:43:51 -0400 (EDT) Subject: [TUHS] Unix with TCP/IP for small PDP-11s Message-ID: <20170523134351.3AFE118C09B@mercury.lcs.mit.edu> > From: Paul Ruizendaal > I'm looking into the history of Spider and early Datakit. Sandy Fraser > was kind enough to send me the 1974 report on Spider Is that online? If not, any chances you can make it so? > Does anybody know of surviving v5/v6/v7 code for Spider networking (e.g. > the 'tiu' device driver, the 'nfs' file transfer package, etc.)? You're in luck. To start with, check out: http://minnie.tuhs.org/cgi-bin/utree.pl?file=SRI-NOSC/dmr/oldstuff which contains the drivers tiu.c, mpx.c - I'm not sure what other files there are part of it? I'm not at all clear how this stuff got there - someone at Bell must have just dumped the contents of the 'dmr' directory, and sent it all off? The PWB1-based MIT systems also have a lot of the Spider software (although it was never used). It's a slightly different version than the one above: 'diff' shows that 'tiu.c' is almost identical, but mpx.c has more significant differences. It also contains man pages, and sources for some (?) user programs; I have the source and manpage for 'nfs'. What other names should I be looking for? (The man page for 'nfs' doesn't list any other commands.) I'll put them up momentarily. In the meantime, I'll append the 'tiu' man page. There isn't one for mpx, alas. Noel -------- .th TIU IV 10/28/73 .sh NAME tiu \*- Spider interface .sh DESCRIPTION Spider is a fast digital switching network. .it Tiu is a directory which contains files each referring to a Spider control or data channel. The file /dev/tiu/d\fIn\fR refers to data channel \fIn;\fR likewise /dev/tiu/c\fIn\fR refers to control channel \fIn\fR. .s3 The precise nature of the UNIX interface is specified elsewhere. .sh FILES /dev/tiu/d?, /dev/tiu/c? .sh BUGS From pnr at planet.nl Wed May 24 19:20:23 2017 From: pnr at planet.nl (Paul Ruizendaal) Date: Wed, 24 May 2017 11:20:23 +0200 Subject: [TUHS] Unix with TCP/IP for small PDP-11s In-Reply-To: References: Message-ID: <104E0596-1A31-4F8D-BB0C-C50EA1868EED@planet.nl> > >> I'm looking into the history of Spider and early Datakit. Sandy Fraser >> was kind enough to send me the 1974 report on Spider > > Is that online? If not, any chances you can make it so? It is a paper copy, but I can make a scan for you. > which contains the drivers tiu.c, mpx.c - I'm not sure what other files there > are part of it? I think tiu.c might be all. The TIU ("terminal access unit") was the network card, so to speak (actually some 5 boards in a rack) and did a lot of the heavy lifting. From the tiu.c file I understand that a DR11-B parallel I/O card was used on the PDP side to connect to the TIU, and that access was structured as a block device driver. > I'm not at all clear how this stuff got there - someone at Bell must have just > dumped the contents of the 'dmr' directory, and sent it all off? Looks like it. > The PWB1-based MIT systems also have a lot of the Spider software (although it > was never used). It's a slightly different version than the one above: 'diff' > shows that 'tiu.c' is almost identical, but mpx.c has more significant > differences. > > It also contains man pages, and sources for some (?) user programs; I have the > source and manpage for 'nfs'. What other names should I be looking for? (The > man page for 'nfs' doesn't list any other commands.) I'll put them up > momentarily. I think that in the lifespan of Spider (1972-1978) there were 3 main network programs (basing myself on McIlroy's Unix Reader): - 'nfs' an FTP-like program to copy files to/from a central File Store. I'm not sure whether the File Store was a Unix machine or something else. - 'ufs' not sure what it was, but I think a telnet-like facility - 'npr' a network printing program A little surprising, but no reference to a Spider mail program in that document. > In the meantime, I'll append the 'tiu' man page. Thanks! It is from October 1973, which sounds right for Spider. I guess this code is the first networking on Unix, predating the UoI work by about 18 months. From jnc at mercury.lcs.mit.edu Thu May 25 01:21:15 2017 From: jnc at mercury.lcs.mit.edu (Noel Chiappa) Date: Wed, 24 May 2017 11:21:15 -0400 (EDT) Subject: [TUHS] Unix with TCP/IP for small PDP-11s Message-ID: <20170524152115.C90CE18C09B@mercury.lcs.mit.edu> > From: Paul Ruizendaal >>> the 1974 report on Spider >> Is that online? If not, any chances you can make it so? > It is a paper copy, but I can make a scan for you. That makes it sounds like it might not be possible to put it online? What's the exact title, so I can look and see if it's already online? I'm pretty sure I've got a hardcopy of some Spider thing, but it would probably take me a while to find it... ;-) > I think that in the lifespan of Spider (1972-1978) there were 3 main > network programs (basing myself on McIlroy's Unix Reader): > - 'nfs' an FTP-like program ... > - 'ufs' not sure what it was, but I think a telnet-like facility > - 'npr' a network printing program OK, the only one I have is 'nfs'. Here's the source, and man page: http://ana-3.lcs.mit.edu/~jnc/tech/unix/s2/nfs.a http://ana-3.lcs.mit.edu/~jnc/tech/unix/man6/nfs.6 Noel From reed at reedmedia.net Thu May 25 03:19:05 2017 From: reed at reedmedia.net (Jeremy C. Reed) Date: Wed, 24 May 2017 12:19:05 -0500 (CDT) Subject: [TUHS] Unix with TCP/IP for small PDP-11s In-Reply-To: <20170524152115.C90CE18C09B@mercury.lcs.mit.edu> References: <20170524152115.C90CE18C09B@mercury.lcs.mit.edu> Message-ID: On Wed, 24 May 2017, Noel Chiappa wrote: > > From: Paul Ruizendaal > > >>> the 1974 report on Spider > > >> Is that online? If not, any chances you can make it so? > > > It is a paper copy, but I can make a scan for you. > > That makes it sounds like it might not be possible to put it online? > What's the exact title, so I can look and see if it's already online? > I'm pretty sure I've got a hardcopy of some Spider thing, but it would > probably take me a while to find it... ;-) Maybe it is: %T Spider \(em An Experimental Data Communications System %Z ctr127 %A A. G. Fraser %J Proc. IEEE Conf. on Communications %P 21F %O IEEE Cat. No. 74CH0859-9-CSCB. %D June 1974 (as seen in usr/dict/papers/Rv7man) From cubexyz at gmail.com Thu May 25 10:23:07 2017 From: cubexyz at gmail.com (Mark Longridge) Date: Wed, 24 May 2017 20:23:07 -0400 Subject: [TUHS] can't protect files from deletion v5,v6, or v7 Message-ID: Ok, I just did an experiment with the rm command and the results surprised me. On Unix v5 logged in as root I created a small test file then did chmod 444 on it. Unfortunately it appears that mere users can still rm the file and also directories are not safe from the rmdir command (even directories set to mode 444). This seems to be the case for v6 and v7 as well. To be fair rm will prompt the user with: test1: 0100444 mode but the user only has to type y and hit enter and the file is toast. Is there no way to completely protect files from being deleted? Mark From chris at groessler.org Thu May 25 10:28:13 2017 From: chris at groessler.org (Christian Groessler) Date: Thu, 25 May 2017 02:28:13 +0200 Subject: [TUHS] can't protect files from deletion v5,v6, or v7 In-Reply-To: References: Message-ID: On 05/25/17 02:23, Mark Longridge wrote: > Is there no way to completely protect files from being deleted? There is a way, write-protect the directory. What else? regards, chris From crossd at gmail.com Thu May 25 10:48:38 2017 From: crossd at gmail.com (Dan Cross) Date: Wed, 24 May 2017 20:48:38 -0400 Subject: [TUHS] can't protect files from deletion v5,v6, or v7 In-Reply-To: References: Message-ID: On Wed, May 24, 2017 at 8:23 PM, Mark Longridge wrote: > Ok, I just did an experiment with the rm command and the results surprised > me. > > On Unix v5 logged in as root I created a small test file then did > chmod 444 on it. Unfortunately it appears that mere users can still rm > the file and also directories are not safe from the rmdir command > (even directories set to mode 444). > > This seems to be the case for v6 and v7 as well. > > To be fair rm will prompt the user with: test1: 0100444 mode > but the user only has to type y and hit enter and the file is toast. > > Is there no way to completely protect files from being deleted? Yes, these are the normal semantics, even on modern systems. If you want to prevent a user from removing a file, remove the user's write permission to the directory containing the file. Recall that a "file" in the removable sense is really a directory entry that points to the inode that represents the real file; to remove that, one must modify the directory to remove the directory entry. The permissions on the file itself don't matter since removal isn't an operation on the contents of the file; the only thing it does to the actual file is update the link count in the inode. - Dan C. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ron at ronnatalie.com Thu May 25 10:57:00 2017 From: ron at ronnatalie.com (Ron Natalie) Date: Wed, 24 May 2017 20:57:00 -0400 Subject: [TUHS] can't protect files from deletion v5,v6, or v7 In-Reply-To: References: Message-ID: <021401d2d4f1$d1b89ef0$7529dcd0$@ronnatalie.com> You have to understand that you can’t directly delete a file in UNIX. It’s never worked that way. What you can do is remove the directory reference (in UNIX terms, a link) to the file. When the link count goes to zero, the inode (which embodies the storage and permissions, etc..) of the file then gets freed up (along with the data blocks comprising the file). The inode permissions of the file have NEVER had any bearing on whether it can be unlinked or not. The permission to unlink is that of the directory that contains it. If the directory is writable, the file can be unlinked from that directory. If that’s the last link, the file goes away. RM in many versions checks to see if the file is read only and asks if you are sure, but that is an artificial safety net done by the rm program (which is not present in the unlink system call itself). -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave at horsfall.org Thu May 25 11:36:02 2017 From: dave at horsfall.org (Dave Horsfall) Date: Thu, 25 May 2017 11:36:02 +1000 (EST) Subject: [TUHS] can't protect files from deletion v5,v6, or v7 In-Reply-To: References: Message-ID: On Wed, 24 May 2017, Mark Longridge wrote: > Ok, I just did an experiment with the rm command and the results > surprised me. [...] It's always been the case that you need write permission on the parent directory; after all, it's what you are actually updating. > Is there no way to completely protect files from being deleted? Don't make the parent directory writable... And if your OS supports file attributes I believe there's an "immutable" flag. -- Dave Horsfall DTM (VK2KFU) "Those who don't understand security will suffer." From pnr at planet.nl Fri May 26 05:31:03 2017 From: pnr at planet.nl (Paul Ruizendaal) Date: Thu, 25 May 2017 21:31:03 +0200 Subject: [TUHS] Early Unix Spider network In-Reply-To: References: Message-ID: >> It is a paper copy, but I can make a scan for you. > > That makes it sounds like it might not be possible to put it online? > What's the exact title, so I can look and see if it's already online? > I'm pretty sure I've got a hardcopy of some Spider thing, but it would > probably take me a while to find it... ;-) The report I have is: "SPIDER-a data communication experiment", Tech Report 23 , Bell Lab, 1974. I did not find it online, but it may be out there somewhere. I think it can be public now, but doing some checks. > OK, the only one I have is 'nfs'. Here's the source, and man page: > > http://ana-3.lcs.mit.edu/~jnc/tech/unix/s2/nfs.a > http://ana-3.lcs.mit.edu/~jnc/tech/unix/man6/nfs.6 Many thanks! There is some puzzling stuff in there that I'd like to figure out, but that is easier to discuss once the report is online. Also, it turns out that the TIU driver was in Warren's repo all along: http://minnie.tuhs.org/cgi-bin/utree.pl?file=V4/nsys/dmr/tdir/tiu.c http://minnie.tuhs.org/cgi-bin/utree.pl?file=V4/man/man4/tiu.4 It's fun to read that the V4 man page says "The precise nature of the UNIX interface has not been defined yet." and Noel's version says: "The precise nature of the UNIX interface is defined elsewhere." (yet the dates are the same!). Some things are surprising (to me, at least): First of all, opening a connection to the File Store is a single open on data channel 1: http://chiselapp.com/user/pnr/repository/Spider/artifact/854a591c0e7a3a54?ln=26 I would expected the code to first have sent a connection request to the switch on control channel 1. Perhaps the File Store was an integral part of the switch/router (a Tempo minicomputer ftp://bitsavers.informatik.uni-stuttgart.de/pdf/tempoComputers/TEMPO-1_ad_Nov69.pdf) with channel 1 functionality hardwired. Next, the code has a hackish form of non-blocking I/O: http://chiselapp.com/user/pnr/repository/Spider/artifact/55ee75831bd98d6c?ln=35,37 I'm puzzled about the alarm() sys call. That did not exist in 1973 -- or did it only exist in Bell Labs private builds? The code calls snstat(), for instance here: http://chiselapp.com/user/pnr/repository/Spider/artifact/55ee75831bd98d6c?ln=38 That seems to be a sys call to here: http://chiselapp.com/user/pnr/repository/Spider/artifact/2c7d65073a7cb0a5?ln=474,524 Could that be the tiu sys call (#45) in the sysent.c table for V4-V6? From pnr at planet.nl Sat May 27 05:08:23 2017 From: pnr at planet.nl (Paul Ruizendaal) Date: Fri, 26 May 2017 21:08:23 +0200 Subject: [TUHS] Early Unix Spider network Message-ID: <29939564-F3EC-461E-BA74-BBD4AB8FAA50@planet.nl> Progressed a little further: [1] The 'ufs' command was a variation on the 'nfs' command. The man page that Noel provided for nfs includes the paragraph: "There is a command /usr/usg/tom/ufs which transfers files to the USG Unix systems. The option letter 7 for the 11/70 or 4 for the 11/45 should be used. Otherwise 'ufs' is similar to 'nfs'." This means there must have been a Unix based File Store (server). Does anybody have a suggestion who 'tom' at USG might have been? [2] The V5 man pages in the archive have a man page for 'npr', in section VI. It says: NAME npr - print file on Spider line-printer SYNOPSIS npr file … DESCRIPTION Npr prints files on the line, printer in the Spider room, sending them over the Spider loop. If there are no arguments, the standard input is read and submitted. Thus npr may be used as a filter. FILES /dev/tiu/d2 tiu to loop It suggests that the printer was hooked up to the Spider switch and that channel 2 was hardcoded to it. [3] Upon closer inspection, the tiu.c driver is a character mode device, the use of disk buffers and a strategy() routine had me confused. It is just a reflection of the fact that it uses DMA hardware. The code for tiu.c in NSYS/V4 is rather different from the code in the SRI-NOSC tree: thinking on how to select channels seems to have changed in between these two versions. [4] Also I found the below post that mentions the snstat() call: http://minnie.tuhs.org/pipermail/tuhs/2015-December/006286.html The object code for snstat() is in libc.a in the dmr's V5 image. Reconstructed, the source code is here: http://chiselapp.com/user/pnr/repository/Spider/artifact/a93175746bd9f94f In short, snstat() is a modified stty call, an evolution in the direction of the later ioctl() system call. No progress as yet on the early history of 'alarm()'. Paul From pnr at planet.nl Sat May 27 21:20:39 2017 From: pnr at planet.nl (Paul Ruizendaal) Date: Sat, 27 May 2017 13:20:39 +0200 Subject: [TUHS] Early Unix Spider network / alarm() Message-ID: Below what I've been able to find about alarm(): [1] Oldest alarm() code I can find is in PWB1, dated July 1977: http://minnie.tuhs.org/cgi-bin/utree.pl?file=PWB1/sys/sys/os/sys4.c http://minnie.tuhs.org/cgi-bin/utree.pl?file=PWB1/sys/sys/os/clock.c Either alarm existed in V5 and V6, and was removed from distributions (which seems unlikely to me), or is was added after V6 was released, perhaps soon after. In the latter case the 'nfs' code that we have must be later than 1974 (even though the man page is dated that way). It could be from the 2nd half of 1975. [2] Interestingly, the idea to implement sleep() in terms of alarm() seems to originate in UoI network unix: http://minnie.tuhs.org/cgi-bin/utree.pl?file=SRI-NOSC/ken/sys2.c -- sslep() This occurs in Oct 1977. In V7 this idea is taken to user space and sleep() is no longer a system call. [3] The UoI code has an instance of alarm() being used to break out of a potentially stalled network call, so that usage seems to have established itself early on. From jnc at mercury.lcs.mit.edu Sat May 27 23:52:59 2017 From: jnc at mercury.lcs.mit.edu (Noel Chiappa) Date: Sat, 27 May 2017 09:52:59 -0400 (EDT) Subject: [TUHS] Early Unix Spider network Message-ID: <20170527135259.2EB4418C0A1@mercury.lcs.mit.edu> > From: Paul Ruizendaal >>> The report I have is: "SPIDER-a data communication experiment" >>> ... >>> I think it can be public now, but doing some checks. OK, that would be great to have online. I _think_ the hardcopy I have (somewhere! :-) is that report, but my memory should not be trusted. The people working on TCP/IP did know of the Spider work (like they knew of the Cambridge ring work), but it didn't really have any impact; it was a totally different direction than the one we were going in. >>> it turns out that the TIU driver was in Warren's repo all along: V4?! Wow. I'd have never guessed it went that far back. >>> The code calls snstat() >> The object code for snstat() is in libc.a in the dmr's V5 image. >> Reconstructed, the source code is here: >> ... >> In short, snstat() is a modified stty call Yes, I looked and found the original source, appended below. >>> Could that be the tiu sys call (#45) in the sysent.c table for V4-V6? I wonder if we'll ever be able to find a copy of the kernel code for that tiu() system call. And I wonder what it did? > [1] Oldest alarm() code I can find is in PWB1 > ... > Either alarm existed in V5 and V6 .. or is was added after V6 was > released, perhaps soon after. In the latter case the 'nfs' code that we > have must be later than 1974 Remember, that source came from the MIT system, which is a modified PWB1. So it's not surprising it's using PWB1 system calls. Noel -------- / C interface to spider status call .globl _snstat .globl cerror _snstat: mov r5,-(sp) mov sp,r5 mov 4(r5),r0 mov 6(r5),0f mov 8(r5),0f+2 sys stty; 0f bec 1f jmp cerror 1: clr r0 mov (sp)+,r5 rts pc .data 0: .=.+6 From pnr at planet.nl Mon May 29 19:54:48 2017 From: pnr at planet.nl (Paul Ruizendaal) Date: Mon, 29 May 2017 11:54:48 +0200 Subject: [TUHS] Early Unix Spider network Message-ID: > The people working on TCP/IP did know of the Spider work (like they knew of > the Cambridge ring work), but it didn't really have any impact; it was a > totally different direction than the one we were going in. I'm aware of that, and I think it was the same the other way around. My interest is tracing how the networking API of Unix developed in the very early days, and that's were there is a link. When I asked a few months back why Bell Labs did not jump onto the work done at UoI, Doug observed that the lab's focus was on Datakit and that triggered my interest. >>>> it turns out that the TIU driver was in Warren's repo all along: > > V4?! Wow. I'd have never guessed it went that far back. My current understanding is that Spider development began in 1969 and that it was first operational in 1972. By '73/'74 it connected a dozen computers at Murray Hill and Unix had gained basic network programs. From Sandy Fraser's "Origins of ATM" video lecture I understand that the Spider learnings included that using a mini to simulate a switch/router was too slow and too costly, and that doing flow control inside the network induced avoidable complexity (I guess Fraser/Cerf/Pouzin all learned that lesson around the same time). The follow-on, custom designed Datakit switch was to correct these issues. Work started in 1974 and I guess that prototypes may have been available around 1978 (when Spider was apparently switched off at Murray Hill). By 1981 a multi-site Datakit network connected various Bell labs and by 1983 Datakit was introduced as a commercial service. As to the Spider network API, it currently seems that it was relatively simple: it exposed the switch as a group of character mode devices, with the user program responsible for doing all protocol work. Interestingly, Spider used a high speed DMA based I/O board (DR11-B), whereas the Datakit switch was apparently connected to a low speed polled I/O board (DR11-C). I did not find the Datakit device driver(s) in the V7 source tree (only a few references in tty.h), so it is hard to be sure of anything. However, it seems that in V7 the Datakit switch was used as "a fancy modem" so to speak, supporting the uucp software stack. There is source for a Datakit driver in the V8 tree, but I currently have no time to study that (and perhaps it is beyond my scope anyway). All input and corrections much appreciated.