Device Drivers Lab

The Wayback Machine - http://web.archive.org/web/20260913191910/https://web.eecs.utk.edu/~smarz1/courses/cosc562/drivers.html References Document Link PCI / PCIe https://web.eecs.utk.edu/~smarz1/docs/PCI_30.pdf Virtio https://web.eecs.utk.edu/~smarz1/docs/virtio-v1.3.pdf Introduction Review the Submission section first so you know what you have to create and submit. There are two levels of host systems that drive these devices. The first layer is the PCI layer, and the second is the VirtIO layer. Each of these devices is a virtual device using the VirtIO protocol. This is why we drove the UART and RTC in the last lab, and we will drive all of the VirtIO devices in this lab. The initialization of each device will be done in handoff. The handoff starts with PCI to VirtIO and then VirtIO to the underlying device. Finally, you will log all of the devices under the global kernel data device registry. Since you have a heap, you can use the alloc library. Peripheral Component Interconnect (PCI) Your main device bus is PCI, which has a comprehensive configuration and access mechanism called the "enhanced configuration access mechanism" or ECAM. It was enhanced for PCI express. PCI-X is PCI "extended" and is not PCI express. PCI express is shortened to PCIe. PCI-X is just an enhancement over the original PCI using the same basic parallel architecture. PCI express is a serial, point-to-point architecture. The starting point is to find the ECAM, which comes from ACPI from the MCFG table. This tells you the ECAM physical address as well as the starting and ending bus numbers. These will be important when we loop through the busses to find devices we can drive. PCI Address Structure The PCI physical memory addresses are broken down as follows. Address Configuration Space A[(20+n-1):20] Bus number (n = \$\log_{2}(\text{end_bus_number}+1)=\[1..8]\$) A[19:15] Device number A[14:12] Function number A[11:8] Extended register number A[7:2] Register number A[1:0] Byte enables The bus numbers are given to you in the MCFG ACPI table. Do not exceed the end bus number. Recall that it is inclusive in the MCFG table. The PCI ECAM address can be a 32-bit or 64-bit address. Only the lower 28 bits contribute to the bus, device, function, and register numbers. The upper bits locate the ECAM MMIO space. Example of 0x3076b000 31 28 27 20 19 15 14 12 11 8 7 2 1 0 0011 [00000111] [01101] [ 011 ] [ 0000 ] [0000 00] 0 0 [ Bus # ] [Dev #] [Func # ] [Ext Reg # ] [ Reg # ] As you can see above, each device has \$2^{16}\$ = 65,536 bytes of space in ECAM, and each bus has \$2^{20}\$ = 1,048,576 bytes (1 MiB). Calculating ECAM Offset /// Calculate the ECAM offset given a bus and slot number const fn ecam_offset(bus: u8, slot: u8) -> usize { let bus = bus as usize; let slot = slot as usize; debug_assert!((slot >> 5) == 0, "Slot (device) is more than 5 bits"); // We don't need to assert bus since it's 8 bits to start with. (bus << 20) | (slot << 15) } ECAM Structure (PCI § 6.1) The ECAM device index is encoded in the MMIO address with the bus and slot encoded to find the top of the ECAM for that device. There are three things you are looking for at the very beginning. First, look for the vendorid. If this is 0xFFFF, then it is telling you that there is nothing connected here. Otherwise, a VirtIO device will have a vendor id of 0x1AF4. The device id will be the particular VirtIO device. Device IDs are encoded starting with 0x1040 plus the virtIO device number. You are looking at the header_type. A type 0 header is a device whereas type 1 is a bridge. Luckily, EDK2 has already setup all of the bridges for us, so we only care about type 0 devices. ECAM Header You can see above the "Header Type" field, which is either 0 (device) or 1 (bridge) as well as the 16-bit Device ID and Vendor ID fields. ECAM in Rust Rust does not like unions (even though it still has them) as well as structures without a specific size. We will write methods to locate data instead and do type conversions. We will see that we will need to do this again in VirtIO. Just like a lot of the ACPI tables, we can wrap the ECAM in a structure. The first thing we need to check is the header type, since type 0 has different fields than type 1. However, both types are the same size since the ECAM address is packed with the bus, slot, and register numbers. The first 16 bytes of the ECAM gives us all of the information we need for: (1) determining whether a device is actually connected (check vendorid == 0xFFFF) and (2) the layout of the rest of the structure (based on header_type). Common ECAM Structure 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35/// The first section of ECAM that is common for both type-0 and type-1 PCI devices. #[repr(C)] pub struct CommonEcam { pub vendor_id: u16, pub device_id: u16, pub command_reg: u16, pub status_reg: u16, pub revision_id: u8, pub class_code: [u8; 3], pub cache_line_size: u8, pub latency_timer: u8, pub header_type: PciType, pub bist: u8, // built-in self test } impl CommonEcam { /// Wrap a PCI ECAM base (from ACPI) at the given bus and slot pub fn new(base: VirtualAddress, bus: u8, slot: u8) -> &'static mut Self { let bus = bus as usize; let slot = slot as usize; let ptr = base.add(ecam_offset(bus, slot)).as_mut_ptr(); unsafe { ptr.as_mut_unchecked() } } /// Get the PCI type (bridge or device) pub fn get_type(&self) -> PciType { self.header_type } } #[derive(Debug, Copy, Clone, Eq, PartialEq, Ord, PartialOrd)] #[repr(u8)] pub enum PciType { Device = 0, Bridge = 1, } Capabilities (PCI § 6.7) Capabilities are device-specific, but they are written in PCI ECAM space. This is a linked list with a capability ID and a next pointer. Also, the capability ID determines the size of the capability. A device only has capabilities if bit index 4 of the status register is 1. Otherwise, there are no capabilities, and reading these fields is an error. Recall that the status register (bits [6:7]) is in the ECAM header for the particular device as well as the command register (bits [4:5]). Common Capability Structure 1 2 3 4 5 6 7 8 9/// # The first 2 bytes of a capability. /// /// Capabilities may expand the size, but the first two fields /// are always id and next, respectively. #[repr(C)] pub struct Capability { pub id: u8, // Capability identifier pub next: u8, // Byte offset to the next capability } The capabilities are where we will find the configuration space for the virtual IO devices. The id gives you the structure of the capability itself, the Capability structure above is just the common first two fields of all capabilities. The next field is a byte offset from the start of the capability list to find the next capability. It is not a memory pointer (it’s only 8 bits), but you can think of it as a next pointer in a linked list. The capabilities are not necessarily in ascending order, so you have to be able to walk the capability list to find all capabilities. MSI-X Capability (PCI § 6.8.2) When configuring message signaled interrupts, we have to turn off the legacy interrupt, which is bit index 3 in the command register. If this is set to 1, it will interrupt the normal way, otherwise, if it is set to 0, it expects MSI/MSI-X to signal interrupts. The MSI-X capability’s ID is 0x11 whereas the older MSI is 0x05. All responses from your PCI devices will be signaled using MSI-X. MSI-X Initial State (PCI § 6.8.2.3) MSI-X is initially disabled at boot time, and you will be required to enable it by: (1) disabling interrupts via the command register for this particular device and (2) enabling MSI-X by setting bit index 15 of the message control register. Virtual IO (virtio) VirtIO devices are paravirtualized devices. Since paravirtualized devices know they are virtual, they are easier to configure, and they are more efficient to drive. All virtio devices have the same request/response system, but there are subtle differences between the bus. Since all of our virtio devices will be connected via PCI, we will use the PCI bus for discovery and configuration. VirtIO Capabilities (Virtio § 4.1.4) If the capability id is 0x09, then this is a vendor specific capability. There are five defined vendor specific capabilities for VirtIO. When you know you have a virtio device (vendorid = 0x1AF4) and a vendor-specific capability (id = 0x09), then the capability will take the following structure (including the id and next pointer). Virtio-specific Capability 1 2 3 4 5 6 7 8 9 10 11#[repr(C)] pub struct VirtioPciCapability { pub id: u8, // PCI_CAP_ID_VNDR (0x09) pub next: u8, // Same as next pointer pub cap_length: u8, // length of this capability including id, next pub cfg_type: u8, // VirtIO configuration type (1 through 5). pub bar: u8, // BAR index. For 64-bit BARs, this is the low base of the BAR. pub padding: [u8; 3], // Padding to a full DWORD pub offset: u32, // Offset within the BAR to find the data. pub length: u32, // Length of the structure in the BAR address. } The configuration type may be one of the following. cfg_type Types 1 2 3 4 5 6 7 8 9 10/// Common configuration for all VirtIO devices pub const COMMON_CFG: u8 = 1; /// Notification configuration pub const NOTIFY_CFG: u8 = 2; /// Interrupt service routine (ISR) configuration pub const ISR_CFG: u8 = 3; /// Device-specific configuration (different for all VirtIO devices) pub const DEVICE_CFG: u8 = 4; /// Alternate PCI configuration access pub const PCI_CFG: u8 = 5; Common Configuration (cfg_type = 1) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34/// Vector value used to disable MSI for queue const MSI_NO_VECTOR: u16 = 0xffff; #[repr(C)] pub VirtioPciCommonCfg { // Negotiation fields pub device_feature_select: u32, // Index to which feature bits to select pub device_feature: u32, // Features device offers to us (read-only) pub driver_feature_select: u32, // Index to which feature bits to select pub driver_feature: u32, //