Muli Ben-Yehuda's journal

February 25, 2006

Filed under: Uncategorized — Muli Ben-Yehuda @ 4:51 PM

Regardless of how many hours I spend working on any given day (many), weekends included, there’s something sad about driving to the lab on Saturday. But hey, I can’t debug Xen without a serial console, and Xen over serial console using Serial Over Lan on my x366 now works, so all is well with the world and I’m glad I made the trip.

(for the curious, the default Xen console is com1,vga, whereas SOL uses com2 on this server, so one needs to pass Xen “console=com2 com2=19200”).

And now, to the gym.

February 21, 2006

the happy happy joy joy dance

Filed under: Uncategorized — Muli Ben-Yehuda @ 10:49 AM

Calgary is working.

Sometime near 4AM on Sunday night I got half dynamic mappings working. Half dynamic mappings are still mapping from the physical address in the PCI address space to the same physical address in the system memory space (like static mappings), but only map those addresses the driver requests from the DMA API (unlike static mappings which map everything). The name is courtesy of Olof Johansson, who did a large chunk of the PPC IOMMU work (Hi Olof :-)).

Once I had half dynamic mode working, I started experimenting with simple transformations on the addresses. When even those worked, which showed that none of the drivers are bypassing the DMA API, I was in a real quandary – why do they work, and fully dynamic, which is just a different transformation on the address, doesn’t?

At this point in time my machine’s remote console dropped off the network and I went to sleep.

On Monday morning, inspiration hit: fully dynamic really is just another transformation on the address – but it’s the only one that I tried that was *0 based*! I immediately made a trivial change to my brain dead TCE allocator: instead of starting to allocate from address 0, start from 0x7000000 (arbitrary, I have 2GB of memory in the victim machine). When this worked, I knew I hit the jackpot.

Investigation later showed that 1MB was the cut-off point. As long as I allocated TCEs above 1MB, everything was fine. 1MB… there’s something about this address… BIOS? and indeed further investigation showed that the region between 640KB and 1MB is special and requires special handling. Now my allocator simply starts from 1MB and all is well with the world.

Next dual steps: moving the guts of Calgary into Xen so that we can use it for isolation and preparing it for submission to the mainline kernel.

Can you tell I’m happy? 🙂

February 18, 2006

Filed under: Uncategorized — Muli Ben-Yehuda @ 1:23 PM

I think I’ll take a break from Calgary today as it’s such a beautiful day outside. I verify that all DMA allocations (dma_alloc_coherent, dma_map_single, dma_map_sg) actually have valid TCE entries, and still either the SAS controller or the USB OHCI controller kill the machine on boot. Either I have a bug in both the allocation and verification or these devices are DMA’ing without the drivers having gone through the DMA API first. We’ll see.

February 17, 2006

Calgary running with translation enabled

Filed under: Uncategorized — Muli Ben-Yehuda @ 2:31 PM

Calgary is now running with translation enabled in the lab on our x366 servers. We’ve been working on this for a while now, bug after bug, but yesterday I had a fit of inspiration, fixed the last one, and now it boots all the way to userspace, with SAS, NICs and USB all DMA’ing fine(!!!). The last bug was particularly stupid – we were writing a 64 bit quantity into the 64 bit TCE Table Address register – with a 32 bit call (writel instead of writeq). Argh!

I’ll detail the more interesting bugs we had for All Code Sucks, eventually. At the moment Calgary is running with static mappings (the identity translation – TCE ‘x’ maps physical address ‘x’, and all physical addresses have corresponding TCEs), and the next step is dynamic mappings. Which I’ve worked on until 2 AM last night. It doesn’t work yet. But it will. Real Soon Now 🙂

OLS 2006

Filed under: Uncategorized — Muli Ben-Yehuda @ 2:21 PM
Date: Wed, 15 Feb 2006 17:14:27 -0600 (CST)
Subject: Proposal Utilizing IOMMUs for Virtualization in Linux and Xen

Your paper proposal Utilizing IOMMUs for Virtualization in Linux and Xen
has been accepted by the 2006 Linux Symposium Committee.

February 14, 2006

Filed under: Uncategorized — Muli Ben-Yehuda @ 10:00 PM

I have been feeling elated and depressed in short cycles lately, depending on where things I have no control over are standing at any given point in time. Trying to break out of the circle is doable. A good first step would be to get off my ass and go to the gym.

I spent the weekend hacking and thought we achieved the next major milestone on the HW bringup. Turned out that my code had the world’s stupidest bug and we’re still not quite there yet (I was initializing the same PHB 8 times instead of initializing the 8 PHBs…). So depressing!

My todo list grows daily. I wish I could become a reclusive kung-fu fighting monk for a few months.

pictorial goodness

Filed under: Uncategorized — Muli Ben-Yehuda @ 9:48 PM

My new wallpaper, from our visit to West Point last year:


The runner up:


February 8, 2006

Filed under: Uncategorized — Muli Ben-Yehuda @ 11:32 PM

So much to do and so little time!

AMD’s IOMMU spec is now available.

February 1, 2006

OLS papers submitted and Calgary speaks!

Filed under: Uncategorized — Muli Ben-Yehuda @ 11:27 AM

Two OLS papers submitted: “Utilizing IOMMUs for Virtualization in
Linux and Xen” and “Smarter Linux Support for Smart IO
Devices”. Hopefully at least one will manage to squeak in by the
program committee 🙂

[muli] Calgary said "Daddy" today for the first time
[muli] of course it pronounced it 0xfda00000, but that's fine too

Create a free website or blog at