Date: Fri, 5 Jan 90 15:58:53 -0500 From: jag@decvax.dec.com (John A. Gallant UEG) Message-Id: <9001052058.AA14281@abyss.zk3.dec.com> To: 2501752@mcimail.com, adrienne@oliveb.olivetti.com, darryl@ism780c.isc.com, decwrl!attctc!texbell!dell!alan@decvax.dec.com, decwrl!pacbell!cfcl!rdm@decvax.dec.com, egt@hprnd.hp.com, jag@decvax.dec.com, jerry.armstrong@clemson.ncr.com, jlohmeye@entec.wichita.ncr.com, jrj@aviary.kodak.com, lawlor@iwtio.att.com, olson@sgi.com, philip.bair@dayton.ncr.com, rita@paramus.sony.com, snively@sun.com, srn@hicom.hitachi.com, steve.betts@dayton.ncr.com, tjw@aviary.kodak.com, ts@cup.portal.com, weiner@xylogics.com Subject: re: Darryl's Unix OSD document [Boy we are going to run out of insert char soon.] darryl@ism780c.isc.com writes: #jag@decvax.dec.com writes: #"darryl@ism780c.isc.com writes: #"> 6.5 UNIX_Operating_System #">[...] #"> At kernel configuration and link time, a static table of #"> included peripheral drivers and SIMs is created and linked #"> into the configuration driver. #" A static table is definitly needed for the configuration/XPT/SIM points. #"[...] # #I'd prefer to see that the table be kept simple, but if that is not possible, #I'd like to have the simple part of it mandatory and let vendors specify #optional extras. They own the configuration driver and XPT anyhow. I would also like to see the table kept simple, I am a firm believer in the KISS method, now if I could just convince some of our hardware people of this :-(. We need to think of what will be necessary to be in this table. These points would only be used between the conf/xpt/sim routine for non-normal operation needes, I would concider anything not CCB related "non-normal". Would it be useful to have some entry points from the peripheral drivers also ? I can list a few possiblities: For the XPT: xpt_init() or xpt_setup() xpt_ioctl() ? perhaps a fall thought point from conf xpt_cmd() command operations For the SIM: sim_init() or sim_setup() sim_ioctl() a fall through from conf also I can realize that these will need to be contained in structures of function pointers. This will allow multiple SIMs etal to be contain without problems of name space. I think that the configuration driver will need to have its init() routine defined here also. In my BSD environment the XPT would need to know who to call when it gets it's initial CCB request and nothing is set up yet. #"> 6.5.1 Initialization #"> #"> At system startup, the configuration driver's init function #"> is run from the ioinit array. ....................... #"[...] #Pardon me. I must admit to having little knowledge of the internals of BSD, #and I'm sure that my Sys V experience is showing where it was not intended. #Because of this, perhaps it would help if you could give a brief overview #of the init time actions in a BSD kernel; to make it fair, here's a quick #description of Sys V: # #After booting, the kernel enters main(). Interrupts are disabled at this #point. When the kernel was linked, two tables of routines were created: #io_init[] and io_start[]. (These are constructed mechanically by idconfig, #which looks into the mdevice file for each driver and determines if it has #an init and/or start routine, and builds the names as "prefix"init and #"prefix"start, from the "prefix" specified in mdevice.) The entries are #in mdevice file order, but I don't think that there's any guarantee that #will always be true. While interrupts are off, each entry in the io_init[] #array is called. Then interrupts are turned on, some copyright stuff is #printed, and each routine in io_start[] is called. After this point, normal #operation is expected. Unfortunatly I have a limited understanding of it also, I'm an old Sys V hacker. But here goes: There is this magic file called a configuration file it defines all the types of hardware, system parameters, and optional stuff found in the particular system. This file is used when the kernel is being built to define a conf table for all the hardware controllers and devices, and what device drivers to include. At boot time/sys configuration time this config table is used to call the drivers to see if the devices in the config file are really out there. Each device driver contains a trio of routine that are called during the bootup/sys conf time. They are drv_probe(), drv_slave(), and drv_attach(). The probe routine is used to determine if the controller is really there and depending on the author of the driver, determine what devices are off of the controller. The probe routine is called first and I believe that if the driver is in the kernel, it's probe routine will be called. The probe routine comes back with a Yea/Nea for the controller. If a controller is found the configuration code in the kernel using the conf table calls the slave routine for each of the possible devices off of the controller. And if the slave routine returns a Yea the attach routine is finially called to attach the device to the driver, whew. Above the driver the configuration code in the kernel is also filling in various data structurs with device information out of the conf table and results from the probe/slave/attach routines. There are two problems with this, there is no corralation with the lines in the config file and the sequence of probe calls. Also there is not a way to specially initialize a subsystem that does not fall into the device driver catagory, outside of special calls within the configuration code of the kernel. I think that this type of a change should be out of the domain of our CAM project. There could be an ISV that has a binary licence and no access to inside the kernel. Also with Ultrix, I'm not sure about other BSDs, interrupts are enabled during this device driver sequence. However is fairly easy to block out interrupts during critical code. #"> ............................. It calls across the CAM #"> interface, through the XPT, to each SIM in turn, allowing #"> them to initialize. The configuration driver obtains #"> information about each SIM, HBA, and target device detected #"> and maintains a table of these devices. Note that #"> interrupts are disabled during the ioinit phase. #" I would expect them to be turned off during all the setup and init phases. # #Yes. Also, we should reword these paragraphs so that io_init is not used. #We need to come up with some terminology that satisfies both Sys V and BSD. #Can you suggest something, John? I have no problem with CAM_init, that is if Clint hasn't already trademarked it :-). #"> After all of the SIMs have initialized and the configuration #"> table has been filled in, the configuration driver invokes #"> the initialization routines for each peripheral driver. #"> This allows them to determine what devices are known and #"> make appropriate memory allocations and resource requests of #"> the XPT. #" This is were the I have the biggest problem. With BSD based systems #"the configuration magic sets up a number of data structures that are #"used by a lot of the io/fs utilities. Would this mean that these BSD based #"systems would have to minic what the conf code is doing in the #"configuration/XPT init handlers ? # #I don't see that it is necessary since the configurator will have all the #knowledge that was necessary to construct the driver/SIM table. The configuration code fills in a group of data structures that are not "owned" by the device driver. These data structures are used by different kernel I/O utilities, like iostat, and are filled with information that I and the CAM configuraion driver writer do not want to have to recreate. If the configuraion driver used it's own init array to call all of the different peripheral drivers it would have to take responsibility to correctly setup these structures. Between the conf/XPT/SIM the misc CAM data structures would belong in their domain and I would expect them to be setup from the initialization process. #" I think that what we had originaly discussed back in November(?), is #"[...] #I'm not sure I understand exactly what distinction you are trying to make #here. The two proposals look very similar in functionality. I chose #mine because I wanted to make the configuration driver available as a #separate entity that would be available to do dynamic loading and unloading #of drivers. Perhaps it is my Sys V myopia again, but I had intended that #none of the other drivers or SIM modules would be called out of the ioinit[] #array; instead, by giving responsibility for the whole subsystem's #initialization to the configuration driver, we can also gain some control #and order the initialization process, if that is useful. For example, when #performing an installation on a PC, I might want to ignore most devices #and concentrate on my root disk; I could use the configuration driver to #control this. I can't make that seperation and have all the SCSI peripheral drivers as seperate from all the other device drivers in the system. I can have the conf/XPT/SIM be seperate and under the control of the configuration driver. But because I have to have the peripheral drivers seperate, I need a way to force/make sure that the CAM subsystem is initalized incase one of the peripheral drivers is called before the configuration driver. #"> 6.5.2 Accessing_the_XPT #"> [...] #"> 6.5.2.2 From_the_SIM #"> #"> The SIMs obtain requests from the XPT as they are passed #"> across from the peripheral driver, via a routine included in #"> the SIM's configuration information. The field in the #"> configuration table is declared as #"> void (* SIMaction)(CCB *). The XPT does not modify ccbs or #"> cdbs. The XPT must intercept those ccbs which must be #"> redirected to the configuration driver (i.e., Path Inquiry, #"> Get Device Type, and Set Device Type). #" What worries me is that we have spent a lot of work making sure that the #"SIMs do not have to worry about what is contained in a CDB. Now we have the #"XPT scanning each CCB for particular commands that need to be routed to #"another driver, even though the configuration driver "works for the XPT". #"Unfortunatly I don't have another way to handle this problem. I do however #"have an possible way to make the interception of CCBs a little simpler. #"What about having the commands that are particular to the configuration driver/ #"XPT be within a range of values or have a particular bit, msb, set to signify #"special routing is needed. #" #" This may be too much work considering that we only have about 7 commands, #"but what if the list gets much larger ? How many cascading if() statements #"become too many ? Or how many angels can dance on the head of a pin .. ? Oops #"sorry I became carried away :-). # #42. ;-) Surely a table of routing codes could reduce the cost to O(1) #in checking for CAM commands vs. SCSI commands. Oops I forgot about using a lookup table, just as long as the codes are kept contigious. It's a good thing I don't do character terminal drivers :-). #"> 6.5.3 Callback_on_Completion #"> #"> The callback field in the ccb is a structure that is #"> platform specific, but always contains at least a callback #"> function pointer, named cbfcnp, and declared as #"> void (*cbfcnp)(CCB *). It is allowed for the cbfcnp to #"> contain NULL, in which case no callback operation is #"> performed. #" #" Do we want to look at what the SIM does after the CCB is complete? I would #"expect that it would handle the next CCB on its queue. But what if its queue #"is empty, do we want it to call back to the XPT to allow another SIM to do #"something ? Or would it be better to have each SIM only handle its particular #"queue, and have the other SIMs on interrupts handle theirs? Could we get into #"a dead lock condition where everybody is waiting for an event to start off #"their queue ? # #This is an important point. For good performance, the SIM must give up #control quickly after completing a CCB; otherwise it is possible to #hog the system for one HBA over others. As far as deadlock goes, the #SIM is responsible for watching the timeout value for each CDB. This may #require it to set a callout routine for the minimum timeout value in its #queue. I think that a SIM will have to kick off the next queued request, so that an interrupt can be pending. However I should not hang around to watch/wait for that request to complete. I think that request sence should be handles the same way, the SIM must not spin, there may be other SIMS that have interrupts that are pending on their own scsi bus. [I had better stop, I promised myself to finish this before 4:00 EST. I will see you all on Monday.] John A. Gallant jag@decvax.dec.com Senior Software Engineer - Ultrix Engineering Group Digitial Equipment Corp. (603) 881-2472 "A beautiful theory, killed by a nasty, ugly, little fact." Thomas H. Huxly