This can be done as follows:

1. Launch Disk Utility, and create a new disk image (File > New > Blank Disk Image…).
   I used the following options:
   • Name: Case Insensitive
   • Size: 100 GB
   • Format: Mac OS Extended (Journaled)
   • Partitions: Hard Disk
   • Image Format: sparse bundle disk image
2. In your new case-insensitive volume (mine is named “Case Insensitive”, so all paths used in this document will refer to it as such), create the following directories:
   • /Applications
   • /Home
   • /Home/Library/Application Support/Steam
   • /Home/Documents/Steam Content

   NOTE: These paths are suitable for a single user. I chose to use /Users/dustin/…

3. Download Steam and put Steam.app in /Volumes/Case Insensitive/Applications
4. Symlink your Steam directories to the case-insensitive volume. In Terminal:
   

   ln -s /Volumes/Case\ Insensitive/Home/Library/Application\ Support/Steam ~/Library/Application\ Support/Steam
   ln -s /Volumes/Case\ Insensitive/Home/Documents/Steam\ Content ~/Documents/Steam\ Content

5. Create /volumes (lowercase) and symlink /Volumes/Case Insensitive into it as “case insensitive”. This is to get around the intentional lowercasing Steam does of all paths it comes into contact with.
   

   mkdir /volumes
   ln -s /Volumes/Case\ Insensitive /volumes/case\ insensitive

6. Launch Steam!

If that’s too much work, use this shell script. Save it to your home directory (or wherever, really), launch Terminal, and issue the following commands:

chmod +x ~/steamcase.sh
~/steamcase.sh 100g

Substitute 100g with your desired size. This script creates a sparse bundle disk image, so you don’t need to worry about having all 100 GiB (or whatever size you choose) available up-front.
Then proceed to install Steam.app to /Volumes/Case Insensitive/Applications. Run and enjoy!

Honestly, Valve, this shouldn’t even be necessary.

From the factory, a Mac ships with a case-insensitive filesystem. This seems to be an Apple simplification, as all UNIX-styled operating systems in recent history (for at least two decades, I imagine) have had case-sensitive filesystems. Apple must not want their users to be confused when files with the same name but different cases can coexist. However, I digress. To turn on case sensitivity system-wide on your Mac, you need to format your boot volume and reinstall the operating system.

This seems like a lot of work for some piddly little capital letters, no? However, the point of this article is not “why people choose case sensitivity,” it is “why won’t vendors support it?”

My argument boils down to a handful of simple points:

1. Many operating systems in existence today use case-sensitive filesystems, by default. In fact, some even mandate the use of such filesystems.
2. Supporting case-sensitive filesystems automatically grants support for case-insensitive filesystems. (Force your filenames to lowercase, refer to them as if they were lowercase; if a case-insensitive filesystem comes along and they are in uppercase, what does it matter? You’re not sensitive to case and the OS will fulfill your request.)
3. Companies lose nothing except for bug reports (seems like something you’d want to lose?) for supporting case-sensitive filesystems. Supporting your paying customers should be a top priority.

Valve (the creators of the Steam content delivery system) and Blizzard (of StarCraft fame) are by no means the first companies to ignore this small but important segment of their market. Adobe is notorious for outright rejecting case-sensitivity (to the point where the installers for some of their products will not only fail to install on a case-sensitive volume, but will not let you install on another volume that is case-insensitive); some might even call them the poster child of caseism.

There are various workarounds to applications failing to work on case-sensitive volumes; herein are listed a few, in increasing order of complexity-

• Whine and complain on your blog.
• Simply don’t use the application, boycott the publisher until they stop being idiots. This solution, while being the easiest, is not always the most practical. Then again, none of these are particularly practical.
• In the case of a simple “drag and drop into Applications” install, create a disk image containing a case-insensitive filesystem, and move all your caseist applications into it. This works for StarCraft II and, to some extent, Steam.
• Create a horrible web of symbolic links into a case-sensitive volume.
• In more extreme cases, you might have to resort to installing a separate copy of OS X on a case-insensitive volume. This is practically required for any Adobe application (or at least, was, before CS5. I do not know if they have fixed this. Their previous comments left me with little hope that they would.) They have not fixed this, and do not intend to.
• The last option, which I choose to ignore because it’s patently absurd, is reinstalling your whole system on a case-insensitive volume.

As you can see, none of these solutions are particularly desirable; they are all one of: ridiculously cumbersome (install another copy of OS X?!), mildly cumbersome (mounting another volume is no big deal, really, but it’s still annoying), or downright impossible (say you need Adobe products in your line of work, you can’t go boycotting their software.) These tactics are met with mixed results.

• StarCraft II fails to patch itself on a case-sensitive volume. Moving it to a case-insensitive volume fixes this problem, and the game works fine. NOTE: This shouldn’t really matter. StarCraft (like most other Blizzard games) stores its data in container files that abstract filesystem access.
• Steam presents a warning: Steam requires /Applications/Steam.app/Contents/MacOS to be in a case-insensitive filesystem. Fixing this by moving it into a case-insensitive disk image simply yields another warning: Steam requires /Users/dustin/Library/Application Support/Steam to be in a case-insensitive filesystem. This is simply not possible for me. I refuse to reinstall everything I have here, and I refuse to dual-boot OS X just to play games. It is worth noting that Steam’s OS X beta used to work on case-sensitive filesystems, albeit in a broken manner. My game data was stored in /users/dustin/documents/steam content. It takes calculation to do this. This is intentional. The operating system provides the path of my home directory as /Users/dustin, and Steam EXPLICITLY lowercased it. What is the logic behind that? It seems like you could simply support case-sensitivity by requesting default paths from the operating system, in this case, and not modifying them. This is an even more concerning problem when you take into account the hints regarding a Steam client for Linux (which would absolutely require case sensitivity).
• Adobe refuses to invest the time in fixing case issues. Thinking Digitally has a bit of information on this (w.r.t. Adobe CS3.) There is a huge discussion (which, after a while, starts to fall on deaf ears) at the Adobe Blog, here; just search for “sensitive”.

For the record, making a horrible web of symlinks works to fix Steam, but why should I have to do that?! That’s absurd!
More absurd is that it insists in writing data to /volumes/case insensitive… My case-insensitive volume is at a case-sensitive location. Ugh.

I would love for somebody else to shed some light on this issue. I would petition, but it wouldn’t get me anywhere. I would contact support, but hey, why would multi-million-dollar companies listen to one dude with a blog?
Why is it so difficult to be case-sensitive?

With the latest release of their desktop operating system, OS X, Apple made great improvements to the system’s speed and application load times (supposedly, I’ve heard mixed reviews of Snow Leopard.) iPhoneOS 3.1 brings in these new enhancements, further streamlining the software on their mobile devices.

The single most impressive, noticeable change I’d like to discuss today is library caching.

Boots of Speed

Does it make sense to load 200+ individual files from various locations on disk? Arguably. This is a pretty standard situation. But wouldn’t it be better if you could, say, intelligently combine those 200+ files (as they do not change often) into one larger file, stored in a single contiguous region on-disk?

This is, in effect, what Apple has done with the dynamic libraries on the iPhone. libSystem, UIKit, Foundation, and hundreds more libraries and frameworks are combined into one large (~92MB for 3.1 on the armv6) flat archive. This archive is loaded into memory (or, more likely, mapped as-needed) at 0×30000000 (all libraries in iPhoneOS seemed to start beyond 0×30000000 anyway; regardless, this location itself matters very little.)

/System/Library/Caches/com.apple.dyld/dyld_shared_cache_armv6 contains every shared dynamic library in iPhoneOS, and dyld no longer needs to hunt down individual libraries and map them into memory when they’re required. The device starts faster and applications launch faster. Huge success… or is it?

Precipice of Doom

This does, of course, cost a lot of disk space. 92MB is a lot in a root partition of ~500MB – so, how did Apple save the rest of the space? Why, they simply deleted the originals! That’s right, there are no (well, there might be five) shared libraries in the filesystem. They’re all horded into a single cache file!

At first glance, this doesn’t seem like a very bad thing. iPhone users rarely peek into the internals of the device, and iPhone Developers have their SDK to fall back on. The SDK contains every library in original format, compiled for ARM.

But, what if you compile on the device? iphone-gcc in Cydia allows you to do so! Or, at least, it used to. There are no longer any shared objects to link against; you can create all the intermediate object files you want, you’ll just never be able to link them into a proper executable.

What if you compile using the open toolchain? Similar situation. The open toolchain is typically built using a copy of the iPhone filesystem to supply needed libraries and datafiles. Of course, you could just copy the libraries from the SDK. What if there’s something you can’t find there? What if there’s something that’s only found in this huge cache file?

Note: I was previously unaware that the SDK actually included the private frameworks as well as the public ones, which does greatly diminish the use of this tool. Still, there ARE binaries that would be in the filesystem that might be of interest to hackers that are not available via the SDK. Lacking the private frameworks would have been bad, as developers using them would have NO way to link their binaries, and would have to do everything regarding them at runtime. Fortunately, that’s not the case.

Cache Withdrawal

I’ve spent the better part of two days writing a utility that will recreate a filesystem’s worth of libraries given a cache file. There are a few drawbacks, but the created binaries have been proven linkable and, for the most part, class-dump-able.

The cache file is of a pretty simple format, with a few “gotcha”‘s. It begins with a header describing what it’s for (cache version, processor) and how many libraries it contains, dyld’s offset in memory, an offset to a code signature, etc., and then moves on to a list of library addresses. Each library’s information is stored in 4 64-bit integers – the load address (which is also the offset in the cache file plus 0×30000000), two unknowns, and the offset of this entry’s filename.

Each library can, for the most part, be extracted as-is: Seek to its offset, figure out its size, and dump that into a separate file. The “gotcha” of this part is, of course, that the cache file is a little more complex than “a bunch of libraries stuck end-to-end”. The application that creates the cache seems to separate the code, data, and linker information (already in separate segments, __TEXT, __DATA and __LINKEDIT) intelligently. It also COMBINES __LINKEDIT for every library in the cache.

The cache, thus, looks something like this:

Header, Library Info, Lib1, Lib2, Lib3, …, Data1, Data2, Data3, …, __LINKEDIT, Code Signature

This is very intelligent and very efficient – until you want to extract the libraries. __LINKEDIT, as stated before, has been combined for every library. So, to extract library 1, you’d need Lib1 + Data1 + __LINKEDIT. The same goes for all other libraries in the file. The only problem with this is that __LINKEDIT alone is ~20MiB in size. Each library requires it.

When all is said and done, the set of 200+ extracted libraries comes out to 4.6 GB. Every library contains a whole bunch of duplicated data which is absolutely necessary for its proper function. (This could be rectified if I intelligently recombined the symbol tables, but that’s a whole lot of Mach-O sorcery I don’t feel like getting into right now.)

The Tools of the Trade

Update: the code is here. The explanation is, however, missing.
There is an updated fork (by KennyTM~) here.

Just a brief post to let you all know what I’ve been up to.

Full post here.

I named it Marcus and crafted a custom installation of Gentoo (as I do for every new computer) for it. I also decided to slipstream a gPXE (Enhanced PXE Network Boot) image into the BIOS. All seemed well, and I flashed it.

Reboot time came. I tried to enter the setup screen to configure my new toy!

For some reason, it didn’t work.
It also decided that it wouldn’t boot from the hard drive… at all.
All it would do was drop into gPXE, repeatedly. After gPXE gave up, it would simply fault and stop responding, shutting the monitor off with it. The keyboard barely worked (it took some prodding, and the control key consistently got stuck starting gPXE’s shell).

“Well, crap. I just ruined my nice new motherboard.”

Fortunately, gPXE is powerful, and there were a few options. I first decided to use it to boot Linux, which worked. I also created another gPXE-laden BIOS image, “properly” (I thought) this time. I booted Linux and flashed the new image.

Same thing. Except this time, It wouldn’t boot Linux.

There’s a great DOS tool called UNIFLASH. It will flash practically ANY BIOS chip, within reason. If I could get it to load an operating system, I could try to flash a good BIOS. That was the theory, at least.

No protected-mode operating systems more complex than Cipher worked. DOS with Mini-Windows 98 didn’t even boot (over the network). Everything just led to a black screen. I was using Memdisk and a disk image (memdisk overrides int13h, tricking the system into treating a block of memory like a hard disk). Everything looked bleak.

With the help of AndyTim in #etherboot (Freenode), I experimented with everything I had. Tried to Chainload the hard drive, and that failed too. Tried to Chainload the hard drive with PXELINUX and THEN chain.c32. That failed too.

Back to Memdisk. Booting into DOS with Memdisk and then loading a BIOS flasher seemed to work. Unfortunately, doing anything with the flasher (even VIA’s official flash utility) caused the computer to hang. I was starting to think that I couldn’t fix this thing.

On top of that, UNIFLASH, the most advanced BIOS flashing tool known to man, did not support my chipset. It did, however, support a very similar one. I dove into the land of PASCAL (to which I hope to never return) and added my system to the list; I recompiled UNIFLASH in DOSBox, which proved difficult as well.

I also learned about SAN booting from AndyTim; treating another network computer hosting an image as a hard disk. This proved to be the ONLY thing that worked. I SAN-booted a DOS image, and my new UNIFLASH worked like a charm.

Believe it or not, this is the abridged version of this tale. The moral? gPXE is amazing, but maybe not in the BIOS of a Pico-ITX system.

I would have been out a computer were it not for the patient help of AndyTim. Thanks!

DUSTIN

How now! An external hard drive?

SEAGATE

Clatters to the floor

[Within enclosure] O, I am slain!

Well, it happened something like that…

I was about to attach my external enclosure (containing a Seagate 120GB 2.5″ SATA hard drive) to my other laptop, when the cord came up short and the drive slid off my desk.

To the floor three feet below…

While it was running.

Hard Drives are rated to withstand falls from a few feet, but only when they’re not running. When they are running, however, the heads smash into the platters, spinning at 5,400 RPM.

So I’m running a fsck on the drive, and it finds a bad block (about 2.3 million 4096-byte blocks into the disk). As I went to pull my laptop and hard drive into my lap, and the hard drive slides off the desk again. In the same place, and landing the same way.

By this point, I’ve started to give up. Fsck tells me that there is a bad block in the same place, and I check the SMART statistics: 0 Reallocated Sectors.

I write data to the bad sectors to force them to reallocate. This goes on for quite a while (with the badblocks tool, among other things such as creating as many 10GB files filled with zeroes as I could (I got to 6 files of various sizes before the drive began buzzing) and I get up to about 150 reallocated sectors (Or around 90 as a calibrated hard drive value; for the record, the failure threshold is 36. The drive started at 100.).

I decide to back up the data (only one of the actual files on the drive is corrupt; all the damage is in unallocated space) and delete some things I have elsewhere. 156 reallocated sectors.

After backing up the data I decided that it would be best to do a destructive write test. Writing to bad blocks causes the drive’s firmware to reallocate them on a different section of the physical disk.

By the middle of the test (700,000 4096-byte blocks, starting at the first corrupted region), the drive began buzzing again, and when I finally gave up, it had reallocated about 2,100 sectors, and the calibrated value was down to 44.

At this point, I gave up.

Rest in peace, Seagate. You have been with me since I got my laptop. Thanks for all the fish.

On another note, I’m now in the market for a new 2.5″ SATA drive.

UPDATE: The drive is a lost cause. I’m not writing anything to it, and it’s reallocating sectors on its own.

Powered-on time: 305 days, 6 hours. Rest in Peace.

Hilights

• Wireless worked immediately; during setup, no less! (screenshot)
• All my hardware was detected and drivers automatically downloaded from Windows Update, without my intervention. (screenshot 1: All drivers installed; screenshot 2: Installing my SATA card)
• User Account Control is 100% less intrusive, and doesn’t hang while blanking the screen.
• Aero is Beautiful in this release.
• Paint got a complete overhaul (screenshot: menu) and can do something similar to Word’s AutoShapes.
• Progress bars in the taskbar. (screenshot)
• Calculator now has a “Programmer Mode.”
• It’s fast, in both setup and use.

Minor Issues

• Address bar icon in IE8 doesn’t support PNG transparency properly. (screenshot)
• Folders no longer display the size of their contents unless the contents are selected. (screenshot 1: Windows 7; screenshot 2: Windows XP)
• Paint uses the Ribbon Interface (not an issue, just hard to get used to). (screenshot)

While the beta expires in August of 2009, I foresee great things for Windows 7.

Plus, Howettblog® is now completely located on the HowettNET server!

App Store Category List

Last night, I tried to figure out how the iTunes App Store (as accessed from the iPod Touch) worked.
I was able to simulate the app store experience, and here are the few notes I took on the matter.

All replies from the App Store are gzip-compressed.
If any of these steps fails, the connection is terminated.

1. The device queries phobos.apple.com for a “bag” (ix=2), which contains a signature and a signing key.
2. The device sends a non-binary plist (XML property list) of its current applications to a WebObject called “availableSoftwareUpgrades”
3. The app store replies with a list of all the information for those applications. It is up to the iPod itself to determine whether/not there are upgrades.
4. Periodically, the device makes a request to metrics.apple.com (which replies 100 Continue instead of 200 OK), which I believe is for stats tracking.
5. The device reads software categories and loads icons (WebObject viewFeaturedSoftwareCategories)
6. The device loads the contents of a category (WebObject viewGenre)
7. The device loads an application’s information descriptor (WebObject viewSoftware). This for some reason contains the text to be used in the price display, as well as the “INSTALL” or “BUY NOW” text.
8. The device initiated a secure connection to download an application. This is where I had to stop my research, as I couldn’t track this.

Tools used:

• curl (Commandline URL Fetcher)
• Wireshark (packet capturer/analyzer)
• Apache (Web Server, used here to serve fake App Ptore pages)
• A single firewall rule on my router to redirect all traffic coming from the iPod back to my computer (iptables -t nat -A PREROUTING -s ipod -p tcp -j DNAT –to 192.168.254.1. Note: I couldn’t redirect to a computer inside the router’s network, so I had to hook up via WiFi to the router AND via Ethernet to the modem)

Modified App Store Application

Not much useful information into the app loading process was gleaned from this, unfortunately, though I did manage to snap some “neat” screenshots of my meddling.

Applications are signed, though, so even if this was an exploitable vector, the device would need to be jailbroken first, thus making this useless.
Neat nonetheless.

EDIT: Whoa, it looks like I’ve been Reddited! Or whatever the appropriate form of that word is. I want to clarify that I love Gentoo and use it on a daily basis. For all the ~arch dependency hell, it’s still a pretty damn great distribution.