Planet E

ZYNK: The Zuper Zimple ZFS Sync (Replication) Tool

2008-11-17T21:38:00+00:00

I've been working on building better and better ZFS replication tools for use at Joyent, but it often gets complex and frustrating because, although replication in ZFS is very simplistic, managing all the snapshots, retentions, and mountains of error checking and handling, on top of reporting and stats collection, is a nightmare.

So, just to relax I wrote a fun simple replication tool I call "Zynk". It's pathetically simple (read: elegant) and fun. As the comment says, if something breaks, its a pita to clean up, but otherwise should work well when set in motion. The intention is to run from cron every 30-600 seconds or so, but be aware that you should do the first run manually, because thats gonna take some time... the incrementals afterwards should be able to run in less than whatever frequency via cron you set.

#!/bin/bash
## ZYNK: The Zuper Zimple ZFS Sync (Replication) Tool
## Form: zynk local/dataset root@remote.host destination/dataset

# Please note: The reason this is so simple is because there is no error checking, reporting, or cleanup.
#               In the event that something goes wonkey, you'll manually need to fix the snapshots and
#               modify or remote the /var/run/zynk datafile which contains the most recent snapshot name.
# Furthermore, this absolutely relies on the GNU version of 'date' in order to get epoch time
# Before using, make sure you've distributed your SSH key to the remote host and can ssh without password.


if [ ! $3 ] 
then
        echo "Usage: zynk local/dataset root@remote.host destination/dataset"
        echo "WARNING: The destination is the full path for the remote dataset, not the prefix dataset stub."
        exit
fi


DATE=`date +%s`
if [ $DATE == "%s" ]
then
        echo "Must use GNU Date, please install and modify script."
        exit
fi


if [ -e /var/run/zynk ] 
then
        # Datafile is found, creating incr.
        echo "Incremental started at `date`"
        zfs snapshot ${1}@${DATE}
        zfs send  -i  ${1}@`cat /var/run/zynk` ${1}@${DATE} | ssh ${2} zfs recv -F ${3}
        zfs destroy ${1}@`cat /var/run/zynk`
        ssh ${2} zfs destroy ${3}@`cat /var/run/zynk`
        echo ${DATE} > /var/run/zynk
        echo "Incremental complete at `date`"
else 
        # Datafile not found, creating full.
        echo "Full started at `date`"
        zfs snapshot ${1}@${DATE}
        zfs send     ${1}@${DATE} | ssh ${2} zfs recv ${3}
        echo ${DATE} > /var/run/zynk
        echo "Full completed at `date`"
fi

Here it is in action:

root@quadra ~$ rm /var/run/zynk 
root@quadra ~$ ./zynk data/home/tamr root@localhost backup/zynk/tamr
Full started at Mon Nov 17 13:44:28 PST 2008
Full completed at Mon Nov 17 13:44:28 PST 2008

root@quadra ~$ ./zynk data/home/tamr root@localhost backup/zynk/tamr
Incremental started at Mon Nov 17 13:44:58 PST 2008
Incremental complete at Mon Nov 17 13:44:58 PST 2008

root@quadra ~$ ./zynk data/home/tamr root@localhost backup/zynk/tamr
Incremental started at Mon Nov 17 13:45:01 PST 2008
Incremental complete at Mon Nov 17 13:45:02 PST 2008

root@quadra ~$ ./zynk data/home/tamr root@localhost backup/zynk/tamr
Incremental started at Mon Nov 17 13:45:19 PST 2008
Incremental complete at Mon Nov 17 13:45:20 PST 2008

root@quadra ~$ zfs list -r data/home/tamr backup/zynk/tamr 
NAME                          USED  AVAIL  REFER  MOUNTPOINT
backup/zynk/tamr             2.45M   296G  2.45M  /backup/zynk/tamr
backup/zynk/tamr@1226958319      0      -  2.45M  -
data/home/tamr               2.47M   196M  2.45M  /data/home/tamr
data/home/tamr@1226958319        0      -  2.45M  -

Whats important to note is that it only maintains a single snapshot on either source or destination, so you don't consume a bunch of additional space or have to worry about screwing up quotas.

This isn't intended so much as a "real" tool, but something you can play around with and hopefully excite the mind about some new fun applications. Add error checking, add retention, add reporting, re-implement in a new language. Have fun.... drink Zima. :)

For a thorough discussion of ZFS Replication, see my post from a couple weeks ago: Understanding ZFS: Replication, Archive and Backup.

Sun's Tough Choices: Rich Green Books, 6,000 to Loose Job

2008-11-14T20:59:00+00:00

Today we got a press release no one wanted to see: Sun Microsystems Aligns Business with Global Economic Climate and Amplifies Growth Opportunities Across Open Source Platforms... how much more rosey you can make it sound, I don't know. As a result, the stock traded higher today, by 1% at close, but the stock took a beating in after-hours trading and struggled to stay above $4 in early trading.

So lets break it down, for better or worse...

Many of us have known that Sun is still too large, in terms of employee count. Even with repetitive RIF's (reduction in force), the company keeps making aquisitions and the cuts, as deep and painful as they are, haven't been deep enough. What a horrible thing to say.... but there it is.

Rich Green is gone. Whether he resigned for personal reasons or not, I don't know and don't care. There are a great many of us glad to see him gone, but as for particulars, he hasn't really been an outward facing fellow, so most news about him comes by way of rumor which I don't like to subscribe to.

WIthin Systems Software there are now 3 groups:

Application Platform Software, run by EVP Anil Gadre, formerly Chief Marketing Officer. So these guys handle everything on top of the systems software, including the whole Java stack, databases, software integration, and even Sun Ed.
Systems Platforms, run by the EVP John Fowler, undoubtedly the most public facing of all Sun's executives, will handle all the software that makes the hardware go, including Solaris, Virtualization including xVM and Virtualbox, management software, etc.
Cloud Computing & Developer Platforms, run by SVP Dave Douglas, will handle NetBeans, OpenOffice, and Network.com, trying to put Sun in a position to leverage the Cloud and build new avenues of business for the company.

So we see a stack here. I'm curious as to how much of systems, from a hardware production perspective, will stay with John Fowler and who will be stepping in.

As usual, two big questions are floating in the air: a) when will Sun be acquired, and b) when will Jonathan step down. As to the first, I don't think it will happen. The company is a tough one to deal with, and would invariably involve a lot of slicing and dicing. In this tough economic climate I don't think anyone has the time or money to take on the problems of Sun. As to the latter, I like Jonathan and wish him only the greatest success. I do question many decisions he's made, the MySQL acquisition first and foremost, and I'm not happy with how distracted Sun is by itself. Systems, Systems, Systems... make great systems, sell great systems, provide software that makes them better... systems systems systems! The bottom line is, the Sun Board of Directors will make that call, and clearly they haven't felt it was the right decision.

I want to highlight that point. There is a lot of people after Jonathan's head, but the blame for any missteps is on the Board. They are the final authority and they are signing off on it.

The future is unclear, but its time to streamline. I recommend reading my (Joyent) CEO David Young's perspective: In the Business Cycle of Create, Improve, Destroy: What Sun Needs to Do Now, forward looking given that it was written on Nov 6th.

ELC-E 2008 was great!

2008-11-13T23:06:02+00:00

Back from The Netherlands after an amazing Embedded Linux Conference (ELC-E 2008) at Ede and then some good days at Amsterdam.

Damien Hirst: For The Love of God

I really liked ELC-E, talked with lots of guys, my presentation was good as well (I hope so! let’s wait for the videos), you can see the slides at: http://palestras.gustavobarbieri.com.br/efl/elce2008/Rich_GUI_without_pain.pdf

After the main conference we had an extra UPnP/DLNA conference, it was really small, but interesting nonetheless.

Then I traveled to Amsterdam, it was my first time there and I really enjoyed the city, the people, the beer… and the museums, of course. There I could see the awesome piece that shows in this post called “For The Love of God”.

Fishworks Launches in Las Vegas

2008-11-13T20:08:00+00:00

I was honored to be part of the historic FISHworks, now combined with the "Amber Road" hardware to become officially known as Sun Storage 7000 Unified Storage Systems. At the last minute on Friday, Sun PR apparently needed an extra customer walk-on and my friend Jason William's from Digitar put in a good word for me... Monday morning I was in Vagas.

If you didn't see the launch, have a look here. My walk on is 6 minutes into Chapter 4.

Having been running FISHworks for almost 2 years, I'm absolutely ecstatic about this release. Please download the Sun Unified Storage Simulator, which is FISHworks in a can... a VMware image for PC and Mac (apparently Mr. Marsland didn't convince them to use VirtualBox).

I'm going to write about FW a lot more in the next couple days, but I want to make it very clear that while Analytics is sexxy and gets a lot of attention, this is a world class storage system that will rival your existing storage solutions both in terms of performance and ease-of-use, not to mention functionality. Watch the launch videos for an over-view and then start playing with the Sim.

Seed, Intervalometer, School, etc.

2008-11-13T20:03:00+00:00

It’s been a while since I’ve written anything terribly much here, so I’m going to make a nice rambling post spanning various different subjects…

Firstly, Seed! We’re going to spend a lot of time finishing up Seed 0.2.0 during the next day-cycle (whatever that means, these days - last night, I went to sleep at 8PM and got up this morning at 4 to do homework…). We’ll most likely release this weekend. Robb’s changed the core a lot this time around - we get struct support, a much, much better memory footprint, GObject properties (and, probably, signals) from Javascript, Cairo support (it’s not pretty, but it works!), many improved examples and tests. Also, exceptions work in a lot more places now, making it much easier to debug apps.

I’ve also been rewriting Matt’s Lights Off example in Clutter - I’ll upload a video when I get back if I can figure it out - it’s incredibly awesome!

On another note: I got female headers and got the LCD interfaced properly to the PCB; I’ve taken lots of notes for revision B of the PCB, which should be the “final” revision. I’ve also started a Keynote (I’ll post it here when I’m done) that I’m going to present to E-Club closer to the end of the semester, detailing the project from beginning to “end”, as well as what I learned about executing a “project”.

As for school - everything’s wrapping up; last horrific diff.eq. homework next week, last data structures lab next week, etc. I’m currently in the middle of the last project for VisComm, which is where the muffin pictures on Flickr come from. You really need to take a look at this one full-size… it’s… scary!

I’m calling Carol’s landlord later tonight to schedule an apartment tour… yikes!

Update

2008-11-12T22:21:17+00:00

Its been a while, since I’ve written on here, so I guess its time for an update. First off, I finally got a co-op job for this winter, at COM DEV International as a software developer. I’ll be working on control systems for some parts of the James Webb Space Telescope (replacement for the Hubble Telescope), which is the best job I could hope for (although the ones for Xandros looked pretty cool as well). I even need security clearance for it! Anyways, I’ll be moving to Ottawa in a while for the co-op, and although its not Europe its still somewhere more interesting than here.

On an unrelated note, I’ve very slowly been working on ECDB. I think that I have the erasing part working the way I’m happy with, although I haven’t tested it yet (don’t have an eraseable disc). I need to ask pygi about how to simulate this. Anyways, heres a screenshot (there is actually no EWL being used in this picture - all edje and embryo):

Seed 0.1.0 Release

2008-11-08T10:16:22+00:00

So! Robb announced the first release of Seed tonight! It’s been a crazy journey, so far, and we’re just starting!

In any case… I’m hacking together Debian packages for our PPA for version 0.2, which will probably land in a week or so… then it’ll be really easy to get going…

Sun Storage Web Event Launch Nov 10th

2008-11-07T09:58:00+00:00

I always hate it when events sneak up on me, especially interesting storage events... so mark you calender for a live webcast and chat on Monday, November 10 at 3:30 p.m. PT. You'll recall that Jonathan ominously mentioned in his blog recently: "You'll be hearing more about Open Storage at a launch event we're holding on November 10th." So if you read that but weren't sure what it was, this is the event. I'm sure it'll be plastered all over Sun.com on Monday, just make sure you put aside time to watch the proceedings.

Additionally, Sun FORUM 2.0 "Sun's premier annual customer storage event" (that I'd never heard of prior to this year, great PR!) is happening in the Customer Briefing Center of the Menlo Park campus next week. Use the link above to register for the event (free) if your in the Bay Area.

Sun.com Gets a Major Facelift

2008-11-07T08:22:00+00:00

If you don't make it a habit to hit the Sun.com front page frequently, have a look. I've notice some incremental change over the last couple days and tonight its gotten a major facelift. It now pushes offers in your face more prominently (downloads and try-n-buys), integrates a lot more video (most is rehashed stuff that you may or may not have seen), and most interesting breaks down business sectors.

One interesting thing is banners like Transform Your Email with Microsoft Exchange Server 2007. This suggests a major shift in marketing focus; rather than push potential Java Enterprise solution stacks, just giving in to the fact that the business world is standardized on Exchange and sell sell sell.

I'm not suggesting that Sun hasn't appealed to the Microsoft base in the past, we've gradually see Microsoft logo's slip onto more and more presentation slide decks, but this feels like Sun's finally put its arms around the beast in light of these tough economic times.

Frankly, I'd go even further to suggest that this is just the first step toward Sun really shifting into rescue mode. While I currently have no visibility into whats happening in the halls of Sun, the external vibe I'm getting is that there is extreme pressure on the executives to turn this thing around... or else. That might be a "duh" statement, but most of us will agree that Sun's been in a tough position for a while now... something has changed drastically, something very recent, and I'm not sure its the economic downturn/crash, so much as its putting pressure on an already disastrous situation.

I could speculate further, but for now lets just watch and wait.

Understanding ZFS: Replication, Archive and Backup

2008-11-06T21:12:00+00:00

As with other features of ZFS, the traditionally complex is made simple and straight forward. This simplification can coax administrators into a false complacency.

In ZFS, backup, archive, migration... any activity that fundamentally involves the movement of data from one system to another, is a replication activity. I propose that the traditional idea of weekly backups is, in fact, just really slow crappy replication. An HA Cluster replicates every 5 seconds, but your website replicates once a week.... its really the same thing, just via different interval and possibly different tools. So understand that when I say "replication" I refer to all forms of data movement, both intra- and inter- system.

ZFS replication is preformed through the use of two simplistic subcommands: zfs send and zfs recv. These are commands that utilize STDIN and STDOUT.... and why? Pipes my friend, pipes. Rather than bake piles of functionality into these commands, Matt Ahrens and the ZFS team opted to instead make them very simple and utilize the traditional UNIX ideology of connecting things together for something even better.

Lets look at a simple intra-system example of replication. Lets say that I have a workstation with an couple internal disks, perhaps a RAIDZ, who knows, and I then attach a USB or Firewire external drive on which I create a pool called "backups". Lets now migrate a simple dataset from my local "data" pool to my external drives "backups" pool:

root@quadra ~$ zfs list -r data
NAME                USED  AVAIL  REFER  MOUNTPOINT
data                222K   218M    19K  /data
data/home           114K   200M    24K  /data/home
data/home/benr       18K   200M    18K  /data/home/benr
data/home/conradr    18K   200M    18K  /data/home/conradr
data/home/glennr     18K   200M    18K  /data/home/glennr
data/home/novar      18K   200M    18K  /data/home/novar
data/home/tamr       18K   200M    18K  /data/home/tamr
root@quadra ~$ zfs list -r backups
NAME      USED  AVAIL  REFER  MOUNTPOINT
backups  67.5K   218M    18K  /backups

root@quadra ~$ zfs snapshot data/home/benr@001
root@quadra ~$ zfs send data/home/benr@001 | zfs recv -d backups

root@quadra ~$ zfs list -r backups
NAME                    USED  AVAIL  REFER  MOUNTPOINT
backups                 191K   218M    19K  /backups
backups/home            106K   218M    18K  /backups/home
backups/home/benr        88K   218M    88K  /backups/home/benr
backups/home/benr@001      0      -    88K  -

Lets step through this together.

Replication is always based on a static point in time, meaning a snapshot. We create a snapshot of the dataset(s) we want to replicate, in this case the snapshot "001" of benr's home directory. Using the zfs send command we can send that snapshot to STDOUT. Using a UNIX Pipe, that STDOUT gets sent to the STDIN of the zfs recv command, which has been told via the -d backups argument that I want to preserve the dataset name and heirarchy under the "backups" dataset. This could just as easily a "backups/data-pool" dataset under which things are created, like so:

root@quadra ~$ zfs destroy -r backups/home
root@quadra ~$ zfs create backups/data-pool
root@quadra ~$ zfs send data/home/benr@001 | zfs recv -d backups/data-pool
root@quadra ~$ zfs list -r backups
NAME                              USED  AVAIL  REFER  MOUNTPOINT
backups                           217K   218M    20K  /backups
backups/data-pool                 125K   218M    19K  /backups/data-pool
backups/data-pool/home            106K   218M    18K  /backups/data-pool/home
backups/data-pool/home/benr        88K   218M    88K  /backups/data-pool/home/benr
backups/data-pool/home/benr@001      0      -    88K  -

What about incremental? I mean, I'll want to freshen the copy right? This is done by created another snapshot, and then telling zfs send to only actually send the difference between the two:

root@quadra ~$ cp -r /etc/security/* /data/home/benr
root@quadra ~$ zfs snapshot data/home/benr@002
root@quadra ~$ zfs list -r data/home/benr
NAME                 USED  AVAIL  REFER  MOUNTPOINT
data/home/benr       379K   199M   355K  /data/home/benr
data/home/benr@001    24K      -    88K  -
data/home/benr@002      0      -   355K  -

root@quadra ~$ zfs send -i data/home/benr@001 data/home/benr@002 | zfs recv -d backups/data-pool
root@quadra ~$ zfs list -r backups/data-pool
NAME                              USED  AVAIL  REFER  MOUNTPOINT
backups/data-pool                 417K   217M    19K  /backups/data-pool
backups/data-pool/home            398K   217M    19K  /backups/data-pool/home
backups/data-pool/home/benr       379K   217M   355K  /backups/data-pool/home/benr
backups/data-pool/home/benr@001    24K      -    88K  -
backups/data-pool/home/benr@002      0      -   355K  -

So here I used ZFS send/recv almost exactly as before, but this time I tell zfs send about another snapshot from which to create an incremental. Notice that the zfs recv command didn't change at all.

But what if I want to send it to another system? Easy, pipe the data through ssh (or rsh, or whatever) like so:

root@quadra ~$ zfs send data/home/benr@002 | ssh root@thumper.cuddletech.com zfs recv -d backups/data-pool

So thats the basics... but what does this mean? Lets get creative!

Firstly, we can write a script that every 30 seconds creates a new snapshot, and then thanks to pre-shared SSH keys can use SSH like above to recv the data elsewhere. Add a little error checking and presto! A really nice, simplistic replication scheme. Even if you have a lot of data change, if you copy it every 30 seconds its unlikely to build up into huge chunks that will take very long to move. When it comes to data that changes frequently, the key is to move early and often!

Now, say we don't need that, simple backups are fine. We can create a script that creates a new snapshot each day at midnight, named the day of the week. When Wed comes around the old "wed" snapshot is removed and a new one created, and then we way create a simple script that zfs send/recv's the Friday snapshot every weekend. Simple to do, plus we have those daily snapshots to fall back on in a pinch, hopefully keeping us from going out to a remote copy.

So we've used pipes in a simple way, to securly transport our datastream from one system to another. Consider other unique possiblities, such as piping zfs send... into gzip before sending across the network!

Or.... say what you really want is a portable dump of your ZFS dataset(s). Remember that we're outputting a datastream from zfs send... just re-direct STDOUT to a file!

root@quadra ~$ zfs send data/home/benr@002 > /tmp/home-benr.zdump  
root@quadra ~$ ls -lh /tmp/home-benr.zdump
-rw-r--r-- 1 root root 421K Nov  6 15:14 /tmp/home-benr.zdump

Now lets test a restore from this "zdump":

root@quadra ~$ zfs create backups/dump-restore               
root@quadra ~$ cat /tmp/home-benr.zdump | zfs recv -d backups/dump-restore
root@quadra ~$ zfs list -r backups/dump-restore
NAME                                 USED  AVAIL  REFER  MOUNTPOINT
backups/dump-restore                 392K   217M    19K  /backups/dump-restore
backups/dump-restore/home            373K   217M    18K  /backups/dump-restore/home
backups/dump-restore/home/benr       355K   217M   355K  /backups/dump-restore/home/benr
backups/dump-restore/home/benr@002      0      -   355K  -

Works like a charm! Again, we can use pipes for fun here too. Lets say that we really want a dump that is encrypted and compressed!

root@quadra ~$ pktool genkey keystore=file outkey=zdump.key keytype=aes keylen=128
root@quadra ~$ zfs send data/home/benr@002 | gzip | encrypt -a aes -k zdump.key > /tmp/home_benr-AES256GZ.zdump

So we've output a datastream based on a snapshot (002), compressed it, encrypted it with 128bit AES and then dumped to file. We could just as easily dump it to a tape (/dev/rmt/0cbn or something) for archiving purposes.

Finally, what if we want to work on more than just a single snapshot. What if we want to send all the "home" datasets? For some time now (although just now arriving in Solaris 10) we've had recursive flags for both zfs snapshot and zfs send. Lets give it a try:

root@quadra ~$ zfs snapshot -r data/home@nov6
root@quadra ~$ zfs list -r data/home
NAME                     USED  AVAIL  REFER  MOUNTPOINT
data/home                755K   199M    24K  /data/home
data/home@nov6              0      -    24K  -
data/home/benr           379K   199M   355K  /data/home/benr
data/home/benr@001        24K      -    88K  -
data/home/benr@002          0      -   355K  -
data/home/benr@nov6         0      -   355K  -
data/home/conradr         88K   199M    88K  /data/home/conradr
data/home/conradr@nov6      0      -    88K  -
data/home/glennr          88K   199M    88K  /data/home/glennr
data/home/glennr@nov6       0      -    88K  -
data/home/novar           88K   199M    88K  /data/home/novar
data/home/novar@nov6        0      -    88K  -
data/home/tamr            88K   199M    88K  /data/home/tamr
data/home/tamr@nov6         0      -    88K  -

root@quadra ~$ zfs destroy -r backups/home
root@quadra ~$ zfs list -r backups
NAME      USED  AVAIL  REFER  MOUNTPOINT
backups    86K   218M    20K  /backups

root@quadra ~$ zfs send -R data/home@nov6 | zfs recv -d backups

root@quadra ~$ zfs list -r backups
NAME                        USED  AVAIL  REFER  MOUNTPOINT
backups                     902K   217M    18K  /backups
backups/home                755K   199M    24K  /backups/home
backups/home@nov6              0      -    24K  -
backups/home/benr           379K   199M   355K  /backups/home/benr
backups/home/benr@001        24K      -    88K  -
backups/home/benr@002          0      -   355K  -
backups/home/benr@nov6         0      -   355K  -
backups/home/conradr         88K   199M    88K  /backups/home/conradr
backups/home/conradr@nov6      0      -    88K  -
backups/home/glennr          88K   199M    88K  /backups/home/glennr
backups/home/glennr@nov6       0      -    88K  -
backups/home/novar           88K   199M    88K  /backups/home/novar
backups/home/novar@nov6        0      -    88K  -
backups/home/tamr            88K   199M    88K  /backups/home/tamr
backups/home/tamr@nov6         0      -    88K  -

Simple, just snapshot the parent dataset with the -r flag, then send the parent dataset snapshot with the -R flag. Otherwise, its all the same! And, of course, you can combine this with all our other pipe tricks just the same!

And so we see that using a single set of commands, we have simplistic and powerful replication, backup, and archive capabilities. A lot of power unleashed with just a little imagination; thats the ZFS way.

Understanding ZFS: Compression

2008-11-06T09:31:00+00:00

One of the most appealing features ZFS offers is built in compression capabilities. The tradeoffs are self evident, consume additional CPU but conserve disk space. If your running an OLTP database then compression probly isn't for you, however if you are doing bulk data archiving this could be a huge win.

ZFS is built with the realization that in modern systems we typically have large amounts of memory and CPU available, and we should be provided with the means to put those resources to work. Contrast this with the traditional logic that compression slows things down, because we stop and compress the data before flushing it out to disk, which takes time. Consider that in some situations, you may have significantly faster CPU and Memory than you have IO throughput, in which case it may in fact be faster to read and write compressed data because your reducing the quanity of IO through the channel! Compression isn't just about saving disk space... keep an open mind.

The first important point about ZFS Compression is that its granular. Within ZFS we create datasets (some people call them "nested filesystems", but I find that confusing terminology), each of which has inherited properties. One of those properties is compression. Therefore, if we create a "home" dataset which mounts to "/home", and then create a "home/user" dataset for each user, we can do interesting things, such as apply per-user quotas (disk usage limits) or reservations (set aside space) or, in this context, enable, disable or specify differing types of compression. Some users may want compression, others may not, or you may wish all to use it by default. ZFS gives us a wide range of flexible options. Most importantly, if we change our mind at some point we can change the setting and all new data is compressed, the old uncompressed data is still used as expected. This means that changes are no disruptive, however this does mean that if you really want to later conserve all the disk you can you'd need to enable compression and then slosh all the data off and then back in.

So, how do we enable compression? Simple, use the zfs set compression=on some/dataset command. If we then "get all" properties from a dataset we'll see some interesting information. Here is an example (pruned for length) of my home directory:

root@quadra ~$ zfs get all quadra/home/benr
NAME              PROPERTY         VALUE                  SOURCE
quadra/home/benr  type             filesystem             -
quadra/home/benr  creation         Thu Oct  9 11:33 2008  -
quadra/home/benr  used             122G                   -
quadra/home/benr  available        432G                   -
quadra/home/benr  referenced       122G                   -
quadra/home/benr  compressratio    1.19x                  -
quadra/home/benr  mounted          yes                    -
quadra/home/benr  quota            none                   default
quadra/home/benr  reservation      none                   default
quadra/home/benr  recordsize       128K                   default
quadra/home/benr  mountpoint       /quadra/home/benr      default
quadra/home/benr  checksum         on                     default
quadra/home/benr  compression      on                     inherited from quadra/home
...

Here we see that compression is "on", and was inherited automatically from its parent dataset "quadra/home". We can also see the compression ratio above: 1.19x.

But what are our options? Just on or off? Many ZFS properties have simplistic "defaults", in this case "on" means that we use the "lzjb" compression algorithm. We can instead specify the exact algorithm. Current, in fairly modern releases of Nevada/OpenSolaris we have available the default LZJB (a lossless compression algorithm created by Jeff Bonwick, which is extremely fast) and gzip at compression levels 0-9. If you set "compression=gzip" you'll get GZIP level 6 compression, however you can explicitly "set compression=gzip-9". More compression algorithms may be added in the future. (The source is out there, feel free to give us another!)

But how can you see the effect? Did you know the "du" command will show you the on-disk (compressed) size of a file? Lets experiment!

root@quadra ~$ zfs create quadra/test
root@quadra ~$ zfs get compression quadra/test
NAME         PROPERTY     VALUE        SOURCE
quadra/test  compression  off          default

Ok, we have a dataset to play with. I've downloaded Moby Dick and combined into a single text file.

root@quadra test$ ls -lh moby-dick.txt 
-rw-r--r-- 1 root root 1.8M Nov  6 01:38 moby-dick.txt
root@quadra test$ du -h moby-dick.txt 
1.8M    moby-dick.txt
root@quadra test$ head -4 moby-dick.txt 
..  chapter I 2  LOOMINGS >

     Call me Ishmael.  Some years ago--never mind how
long precisely --having little or no money in my purse, and nothing particular

Alright, so here is Moby Disk in text, weighing in at 1.8M uncompressed. Lets not enable compression (LZJB), copy the file and see how much benefit we get:

root@quadra test$ zfs set compression=on quadra/test
root@quadra test$ cp moby-dick.txt moby-dick-lzjb.txt
root@quadra test$ sync

root@quadra test$ ls -lh
total 3.5M
-rw-r--r-- 1 root root 1.8M Nov  6 01:40 moby-dick-lzjb.txt
-rw-r--r-- 1 root root 1.8M Nov  6 01:38 moby-dick.txt
root@quadra test$ du -ah 
1.7M    ./moby-dick-lzjb.txt
1.8M    ./moby-dick.txt
3.5M    .

Nice, we're saving some space. Now lets repeat with gzip.

root@quadra test$ zfs set compression=gzip quadra/test
root@quadra test$ cp moby-dick.txt moby-dick-gzip.txt
root@quadra test$ ls -lh
total 4.6M
-rw-r--r-- 1 root root 1.8M Nov  6 01:44 moby-dick-gzip.txt
-rw-r--r-- 1 root root 1.8M Nov  6 01:40 moby-dick-lzjb.txt
-rw-r--r-- 1 root root 1.8M Nov  6 01:38 moby-dick.txt
root@quadra test$ du -ah 
1.7M    ./moby-dick-lzjb.txt
1.8M    ./moby-dick.txt
1.1M    ./moby-dick-gzip.txt
4.6M    .

Ahhhh. Nice gain there. Remember that this is gzip-6 really, lets crank it up to gzip-9!

root@quadra test$ zfs set compression=gzip-9 quadra/test
root@quadra test$ cp moby-dick.txt moby-dick-gzip9.txt
root@quadra test$ ls -lh
total 4.6M
-rw-r--r-- 1 root root 1.8M Nov  6 01:44 moby-dick-gzip.txt
-rw-r--r-- 1 root root 1.8M Nov  6 01:46 moby-dick-gzip9.txt
-rw-r--r-- 1 root root 1.8M Nov  6 01:40 moby-dick-lzjb.txt
-rw-r--r-- 1 root root 1.8M Nov  6 01:38 moby-dick.txt
root@quadra test$ du -ah 
1.7M    ./moby-dick-lzjb.txt
1.8M    ./moby-dick.txt
1.1M    ./moby-dick-gzip.txt
512     ./moby-dick-gzip9.txt
4.6M    .

Wow! Thats savings. Just to put this in context, I'll test gzip'ing the file like your used to (using tmpfs, not zfs):

root@quadra test$ cd /tmp
root@quadra tmp$ ls -alh moby-dick.txt 
-rw-r--r-- 1 root root 1.8M Nov  6 01:47 moby-dick.txt
root@quadra tmp$ gzip moby-dick.txt 
root@quadra tmp$ ls -alh moby-dick.txt.gz 
-rw-r--r-- 1 root root 1.1M Nov  6 01:47 moby-dick.txt.gz

And so we see here that just gzip'ing the file matches the compression I got with gzip (gzip-6) enabled.

But before you get too excited, remember that this is consuming system CPU time. The more compression you do, the more CPU you'll consume. If this is a dedicated storage system your working on, then consuming a ton of CPU for compression may well be worth it (many appliances have fast CPU's for just this reason), however if your running critical apps and CPU really counts, then notch it down or even turn it off. I highly recommend you dry-run your application workload and then load test it hard to see whether or not that extra CPU will be a problem. Whenever possible, try to determine these things before you deploy, not after.

To follow up that idea, remember that we can set differing compression levels on different datasets. You may want to put your application data on an uncompressed dataset, but store less commonly used data or backups on a separate dataset where you've cranked compression up. Get creative!

ZFS is an amazing technology and compression is certainly one of its big attractions for the common user. Workstation always low on disk? Compression to the rescue, no stupid FUSE or loopback tricks required. :)

A Word Of Warning

At this point I do want to warn you of something. Notice that du displays actual disk consumption, not true file size. Now consider the way in which most admins actually use the command... to total up cumulative file sizes. On a typical file system, "du -sh ." will nicely total up all the files, which would be the same as if I tar'ed up the files and looked at the tarball's filesize. When using compression you can not use "du" in this way because the files are larger than the actual disk usage. So you get into potentially confusing situations like this:

root@quadra test$ ls -alh
total 5.6M
drwxr-xr-x 2 root root    6 Nov  6 01:46 .
drwxr-xr-x 8 root root    8 Nov  6 01:33 ..
-rw-r--r-- 1 root root 1.8M Nov  6 01:44 moby-dick-gzip.txt
-rw-r--r-- 1 root root 1.8M Nov  6 01:46 moby-dick-gzip9.txt
-rw-r--r-- 1 root root 1.8M Nov  6 01:40 moby-dick-lzjb.txt
-rw-r--r-- 1 root root 1.8M Nov  6 01:38 moby-dick.txt
root@quadra test$ du -h .
5.6M    .
root@quadra test$ tar cfv md.tar moby-dick*
moby-dick-gzip.txt
moby-dick-gzip9.txt
moby-dick-lzjb.txt
moby-dick.txt
root@quadra test$ ls -lh md.tar  
-rw-r--r-- 1 root root 7.0M Nov  6 02:11 md.tar

In the real world, this could come to you as a shock if you wanted to rsync a bunch of data, totalled it up using "du" to estimate the bits that need to move, and then got nervous when you moved way more bits than you initially expected because your forgot to take compression into the equitation. So hopefully some of you can learn here not just how ZFS works, but appreciate "du" in a new way as well. :)

Stupid UNIX Tricks: User-to-User Communication

2008-11-05T23:00:00+00:00

Today and interesting post came across Slashdot: (Useful) Stupid Unix Tricks? The post points out other users that were unaware of the write command. This doesn't surprise me, a lot of sysadmins likely don't know about some of these tools... after all, the "write" command isn't exactly self-evidently a tool to communicate with another user.

There are several tools that allow users to communicate with one another through the shell, we'll talk about 3: wall, write, and talk, as well as its friend mesg.

Firstly, the wall command allows a sysadmin as root to message all users on the system. This is used by script such as "shutdown" or "reboot", telling everyone to get off. Use is simple. Either give it standard input (ie: "banner GO AWAY | wall") or running the command, entering your text including line breaks and then completing with ^D. Here is an example:

root@quadra ~$ wall
System Maintaince Tonight!

Please be aware that the system is going down in 20 mins.  Please save your 
work and get off ASAP!

Thank You    
 - SysOps Team
^D

And here is what the users see:

bash-3.2$ Broadcast Message from root (pts/10) on quadra Wed Nov  5 15:14:58...
System Maintaince Tonight!

Please be aware that the system is going down in 20 mins.  Please save your 
work and get off ASAP!

Thank You
 - SysOps Team

Secondly, the write command sends a message directly to a single user. Anyone can use this command. This is an ancient form of in-band instant messaging that is commonly used by sysadmins to communicate, but has fallen out of use being replaced by out of band instant messaging like Jabber, AIM, etc.

To use write, specify the user, and optionally their pty (if they have multiple logins). Then type your message and hit ^D.

benr@quadra ~$ write tamr
Hey, what are you doing editing /etc/user_attr???

The user see's each line of text entered as you hit return, and finally an "EOT" when you ^D:

bash-3.2$ 
        Message from benr on quadra (pts/6) [ Wed Nov  5 15:18:06 ] ...
Hey, what are you doing editing /etc/user_attr???

bash-3.2$

The user can then respond in the same way. This can also be sped up by using pipes or redirection.

The third command is the real-time user communication tool, talk. I say real-time because you can see each character the other user enters... this has lead to many "d00, learn to type!" insult exchanges. In 1994 when my local ISP was just a SunOS system, many of us dialing in used talk as a form of pseudo-chat with other users.

The "talk" tool requires a daemon process. On a default full install of Solaris you'll find it as a disabled service. To use it, simply enable via SMF:

root@quadra ~$ svcadm enable talk
root@quadra ~$ svcs talk
STATE          STIME    FMRI
online         14:52:36 svc:/network/talk:default

Now, initiate a discussion using the form "talk user2". The user will get a message informing them of the request and tell them how to respond, by recipricating user: "talk user1". When established, a session looks like this:

Whats even more fun, is that you can actually talk to users on other systems! Very versital and fun to user. A GNU variant of "talk" is available thats even more powerful and interesting than the classic SysV version on Solaris, allowing you to shell out and demo things to users, etc.

The final command we'll look at is mesg. This command takes the option y or n. It simply determins whether or not other people can but you using write and talk. This is the system equivalent of setting "away".

benr@quadra ~$ mesg n
benr@quadra ~$ mesg
is n

The main reason for this is to avoid messages from annoying users or avoid getting a message while doing something sensitive. For instance, getting a "write" message while editing a file in VI will cause your screen to become a mess and potentially mess up your edits unless you refresh.

So go forth, have fun, and annoy your friends. Maybe next week I'll try to convince you that blogs are still lame compared to the original finger.

Thank You

2008-11-05T06:15:55+00:00

Seed: In Screenshots

2008-11-03T06:06:48+00:00

Here’s a bunch of applications Robb, Matt, and I have written in Seed… the source is all in Gnome SVN…

Lights Off was Matt’s first application - he went from having written no Javascript at all to a nice, working game in a matter of hours one night.

n-oscillator was more or less the first working GTK application; Robb wrote it (who else would be interested in making annoying noises at all hours!?) just to figure out how to get things working… it lets you start an arbitrary number of oscillators and adjust their volume and frequency, and uses GStreamer for output.

calc was my first GTK+seed application. I wrote it on the train on the way to the Gnome Summit, without any documentation… it’s a horrible front-end for eval, nothing else!

mini-browser is my rewrite of Robb’s original WebKit/seed web browser, Spacewhale. It provides a tabbed interface (though, because of a WebKit bug, no GTK/WebKit-based browser can ‘open in new tab/window’ at all…), but not much else… I’m planning on revisiting this at some point, and getting it to a state similar to epiphany-webkit.

repl is where a lot of our testing takes place, since it provides immediate feedback. It’s a rather vital part of an interpreted language, and can be written in very, very little Seed code:

#!/usr/local/bin/seed
 
while(1)
{
	try
	{ Seed.print(eval(Seed.readline("> "))) }
	catch(e)
	{ Seed.print(e.name + " " + e.message) }
}

shader is a cute demo Robb wrote that lets you edit GLSL shaders and apply them to an image, using Clutter to display the image and apply the texture.

ide is a small Seed editor that I’ve been writing, eventually providing syntax checking and (maybe) debugging of Seed apps.

Seed Tutorial

2008-11-03T04:47:46+00:00

I’ve finished my Seed tutorial, which takes you from square one to writing a quick WebKit-based web browser in Seed, while explaining bits of Javascript and GTK along the way.

I tested it out on Matt, and he’s got a working copy of the browser now, so I guess it works…

Formula1 World Champion: Lewis Hamilton

2008-11-03T00:07:00+00:00

Todays Grand Prix of Brazil marked the end of the 2008 Formula 1 World Championship, and on the 2nd to the last corner, fate crowned Lewis Hamilton World Champion.

It was an exciting race. Unlike last year, Lewis was much more controlled and calm... he wasn't fighting battles he didn't need to fight, which cost him the crown last year. However, very much like last year, late in the race he just fell apart and acted like he forgot how to drive... it was sad. Post race interviews reveal that he was just trying to play it safe and didn't realize that when he gave up position to Sebastian Vettel he was in fact giving up the World Championship. He got oh so lucky that Timo Glock lost out big in the 2nd to last corner allowing Lewis an easy pass and the Championship.

Conspiracy theorists will suggest that Glock actually let Lewis by... but I tend to doubt it. He was on dry tires, struggling on the last lap, and that corner becomes a small river in the wet. Furthermore, there is no incentive to letting Lewis win, Toyota could have used the extra point. Lastly, given the level of concentration in the closing laps, I highly doubt he was aware of the championship situation. Post-race interviews showed that when Vettel passed Lewis he had no idea that he was potentially deciding the World Championship... in fact, his tone was such that I think he regretted the pass. Therefore, the only way this thing goes down as a plot was if there was a radio transmission to Glock in the 2nd half of the last lap with information or an order. If there was information, and he wanted to give up the position, its his to give... if there were orders, you can believe Ferrari will sniff it out and we'll hear about it. But I seriously doubt it, as I said, Toyota had nothing to gain either way.

While I'm a Ferrari fan (more to the point, a Räikkönen fan), I'm glad Lewis won. Frankly, I just don't like Massa. And more importantly, I think Lewis winning the title is just plain good for the sport.

While a side note, I'm sure I'm not the only one thinking that if Obama win's the Presidency, 2008 will go down as a year where two prominent positions always assumed to belong to white males have gone to black men.

And, on top of all that, a British driver with a British constructor won... go Britannia!!

zdb: Examining ZFS At Point-Blank Range

2008-11-01T08:13:00+00:00

ZFS is an amazing in its simplicity and beauty, however it is also deceivingly complex. The chance that you'll ever be forced to peer behind the veil is unlikely outside of the storage enthusiast ranks, but as it proliferates more questions will come up regarding its internals. We have been given a tool to assist us investigate the inner workings, zdb, but it is, somewhat intentionally I think, undocumented. Only two others that I know have had the courage to talk about it publicly, Max Bruning who is perhaps the single most authoritative voice regarding ZFS outside of Sun, and Marcelo Leal.

In this post, we'll look only at the basics of ZDB to establish a baseline for its use. Running "zdb -h" will produce a summary of its syntax.

In its most basic form, zdb poolname, several bits of information about our pool will be output, including:

Cached pool configuratino (-C)
Uberblock (-u)
Datasets (-d)
Report stats on zdb's I/O (-s), this is similar to the first interval of zpool iostat

Thus, zdb testpool is the same as zdb -Cuds testpool. Lets look at the output. The pool we'll be using is actually a 256MB pre-allocated file with a single dataset... as simple as it can come.

root@quadra /$ zdb testpool
    version=12
    name='testpool'
    state=0
    txg=182
    pool_guid=1019414024587234776
    hostid=446817667
    hostname='quadra'
    vdev_tree
        type='root'
        id=0
        guid=1019414024587234776
        children[0]
                type='file'
                id=0
                guid=6723707841658505514
                path='/zdev/disk002'
                metaslab_array=23
                metaslab_shift=21
                ashift=9
                asize=263716864
                is_log=0
Uberblock

        magic = 0000000000bab10c
        version = 12
        txg = 184
        guid_sum = 7743121866245740290
        timestamp = 1225486684 UTC = Fri Oct 31 13:58:04 2008

Dataset mos [META], ID 0, cr_txg 4, 87.0K, 49 objects
Dataset testpool/dataset01 [ZPL], ID 30, cr_txg 6, 19.5K, 5 objects
Dataset testpool [ZPL], ID 16, cr_txg 1, 19.0K, 5 objects
                      capacity   operations   bandwidth  ---- errors ----
description          used avail  read write  read write  read write cksum
testpool             139K  250M   638     0  736K     0     0     0     0
  /zdev/disk002      139K  250M   638     0  736K     0     0     0     0

And so we see a variety of useful information, including:

Zpool (On Disk Format) Version Number
State
Host ID & Hostname
GUID (This is that numberic value you use when zpool import doesn't like the name)
Children VDEV's that make up the pool
Uberblock magic number (read that hex value as "uba-bloc", get it, 0bab10c, its funny!)
Timestamp
List of datasets
Summary of IO stats

So this information is interesting, but frankly not terribly useful if you already have the pool imported. This would likely be of more value if you couldn't, or wouldn't, import the pool, but those cases are rare and 99% of the time zpool import will tell you want you want to know even if you don't actually import.

There are 3 arguments that are really the core ones of interest, but fefore we get to them, you absolutely must understand something unique about zdb. ZDB is like a magnifying glass, at default magnification you can see that its tissue, turn up the magnification and you see that it has veins, turn it up again and you see how intricate the system is, crank it up one more time and you can see blood cells themselves. With zdb, each time we repeat an argument we increase the verbosity and thus dig deeper. For instance, zdb -d will list the datasets of a pool, but zdb -dd will output the list of objects within the pool. Thus, when you really zoom in you'll see commands that look really odd like zdb -ddddddddd. This takes a little practice to get the hang of, so please toy around on a small test pool to get the hang of it.

Now, here are summaries of the 3 primary arguments you'll use and how things change as you crank up the verbosity:

- -bb: Outputs a breakdown of space (block) usage for various ZFS object types.
- -bbb: Same as above, but includes breakdown by DMU/SPA level (L0-L6).
- -bbbb: Same as above, but includes line line per object with details about it, including compression, checksum, DVA, object ID, etc.
- -bbbbb...: Same as above.
- -d: Output list of datasets, including ID, cr_txg, size, and number of objects.
- -dd: Output concise list of objects within the dataset, with object id, lsize, asize, type, etc.
- -ddd: Same as dd.
- -dddd: Outputs list of datasets and objects in detail, including objects path (filename), a/c/r/mtime, mode, etc.
- -ddddd: Same as previous, but includes indirect block addresses (DVAs) as well.
- -dddddd....: Same as above.
zdb -R pool:vdev_specifier:offset:size[:flags]: Given a DVA, outputs object contents in hex display format. If given the :r flag it will output in raw binary format. This can be used for manual recovery of files.

So lets play with the first form above, block traversal. This will sweep the blocks of your pool or dataset adding up what it finds and then producing a report of any leakage and how the space breakdown works. This is extremely useful information, but given that it traverses all blocks its going to take a long time depending on how much data you have. On a home box this might take minutes or a couple hours, on a large storage subsystem is could take hours or days. Lets look at both -b and -bb for my simple test pool:

root@quadra ~$ zdb -b testpool

Traversing all blocks to verify nothing leaked ...

        No leaks (block sum matches space maps exactly)

        bp count:              50
        bp logical:        464896        avg:   9297
        bp physical:        40960        avg:    819    compression:  11.35
        bp allocated:      102912        avg:   2058    compression:   4.52
        SPA allocated:     102912       used:  0.04%

root@quadra ~$ zdb -bb testpool

Traversing all blocks to verify nothing leaked ...

        No leaks (block sum matches space maps exactly)

        bp count:              50
        bp logical:        464896        avg:   9297
        bp physical:        40960        avg:    819    compression:  11.35
        bp allocated:      102912        avg:   2058    compression:   4.52
        SPA allocated:     102912       used:  0.04%

Blocks  LSIZE   PSIZE   ASIZE     avg    comp   %Total  Type
     3  12.0K   1.50K   4.50K   1.50K    8.00     4.48  deferred free
     1    512     512   1.50K   1.50K    1.00     1.49  object directory
     1    512     512   1.50K   1.50K    1.00     1.49  object array
     1    16K      1K   3.00K   3.00K   16.00     2.99  packed nvlist
     -      -       -       -       -       -        -  packed nvlist size
     1    16K      1K   3.00K   3.00K   16.00     2.99  bplist
     -      -       -       -       -       -        -  bplist header
     -      -       -       -       -       -        -  SPA space map header
     3  12.0K   1.50K   4.50K   1.50K    8.00     4.48  SPA space map
     -      -       -       -       -       -        -  ZIL intent log
    16   256K   18.0K   40.0K   2.50K   14.22    39.80  DMU dnode
     3  3.00K   1.50K   3.50K   1.17K    2.00     3.48  DMU objset
     -      -       -       -       -       -        -  DSL directory
     4     2K      2K   6.00K   1.50K    1.00     5.97  DSL directory child map
     3  1.50K   1.50K   4.50K   1.50K    1.00     4.48  DSL dataset snap map
     4     2K      2K   6.00K   1.50K    1.00     5.97  DSL props
     -      -       -       -       -       -        -  DSL dataset
     -      -       -       -       -       -        -  ZFS znode
     -      -       -       -       -       -        -  ZFS V0 ACL
     1    512     512     512     512    1.00     0.50  ZFS plain file
     3  1.50K   1.50K   3.00K      1K    1.00     2.99  ZFS directory
     2     1K      1K      2K      1K    1.00     1.99  ZFS master node
     2     1K      1K      2K      1K    1.00     1.99  ZFS delete queue
     -      -       -       -       -       -        -  zvol object
     -      -       -       -       -       -        -  zvol prop
     -      -       -       -       -       -        -  other uint8[]
     -      -       -       -       -       -        -  other uint64[]
     -      -       -       -       -       -        -  other ZAP
     -      -       -       -       -       -        -  persistent error log
     1   128K   4.50K   13.5K   13.5K   28.44    13.43  SPA history
     -      -       -       -       -       -        -  SPA history offsets
     -      -       -       -       -       -        -  Pool properties
     -      -       -       -       -       -        -  DSL permissions
     -      -       -       -       -       -        -  ZFS ACL
     -      -       -       -       -       -        -  ZFS SYSACL
     -      -       -       -       -       -        -  FUID table
     -      -       -       -       -       -        -  FUID table size
     1    512     512   1.50K   1.50K    1.00     1.49  DSL dataset next clones
     -      -       -       -       -       -        -  scrub work queue
    50   454K   40.0K    101K   2.01K   11.35   100.00  Total

Here we can see the "zooming in" effect I described earlier. Here "BP" stands for "Block Pointer". The most common "Type" you'll see is "ZFS plain file", that is, a normal data file like an image or textfile or something... the data you care about.

Moving on to the second form, -d to output datasets and their objects. This is where introspection really occurs. With a simple -d we can see a recursive list of datasets, but as we turn up the verbosity (-dd) we zoom into the objects within the dataset, and then just get more and more detail about those objects.

root@quadra ~$ zdb -d testpool/dataset01
Dataset testpool/dataset01 [ZPL], ID 30, cr_txg 6, 18.5K, 5 objects

root@quadra ~$ zdb -dd testpool/dataset01
Dataset testpool/dataset01 [ZPL], ID 30, cr_txg 6, 18.5K, 5 objects

    Object  lvl   iblk   dblk  lsize  asize  type
         0    7    16K    16K    16K  14.0K  DMU dnode
         1    1    16K    512    512     1K  ZFS master node
         2    1    16K    512    512     1K  ZFS delete queue
         3    1    16K    512    512     1K  ZFS directory
         4    1    16K    512    512    512  ZFS plain file

So lets pause here. We can see the list of objects in my testpool/dataset01 by object id. This is important because we can use those id's to dig deeper on an individual object later. But for now, lets zoom in a little bit more (-dddd) on this dataset.

root@quadra ~$ zdb -dddd testpool/dataset01
Dataset testpool/dataset01 [ZPL], ID 30, cr_txg 6, 18.5K, 5 objects, rootbp [L0 DMU objset] 400L/200P DVA[0]=0:12200:200> DVA[1]=0:3014c00:200> fletcher4 lzjb LE contiguous birth=8 fill=5 cksum=a525c6edf:45d1513a8c8:ef844ac0e80e:22b9de6164dd69

    Object  lvl   iblk   dblk  lsize  asize  type
         0    7    16K    16K    16K  14.0K  DMU dnode

    Object  lvl   iblk   dblk  lsize  asize  type
         1    1    16K    512    512     1K  ZFS master node
        microzap: 512 bytes, 6 entries

                casesensitivity = 0 
                normalization = 0 
                DELETE_QUEUE = 2 
                ROOT = 3 
                VERSION = 3 
                utf8only = 0 

    Object  lvl   iblk   dblk  lsize  asize  type
         2    1    16K    512    512     1K  ZFS delete queue
        microzap: 512 bytes, 0 entries


    Object  lvl   iblk   dblk  lsize  asize  type
         3    1    16K    512    512     1K  ZFS directory
                                 264  bonus  ZFS znode
        path    /
        uid     0
        gid     0
        atime   Fri Oct 31 12:35:30 2008
        mtime   Fri Oct 31 12:35:51 2008
        ctime   Fri Oct 31 12:35:51 2008
        crtime  Fri Oct 31 12:35:30 2008
        gen     6
        mode    40755
        size    3
        parent  3
        links   2
        xattr   0
        rdev    0x0000000000000000
        microzap: 512 bytes, 1 entries

                testfile01 = 4 (type: Regular File)

    Object  lvl   iblk   dblk  lsize  asize  type
         4    1    16K    512    512    512  ZFS plain file
                                 264  bonus  ZFS znode
        path    /testfile01
        uid     0
        gid     0
        atime   Fri Oct 31 12:35:51 2008
        mtime   Fri Oct 31 12:35:51 2008
        ctime   Fri Oct 31 12:35:51 2008
        crtime  Fri Oct 31 12:35:51 2008
        gen     8
        mode    100644
        size    21
        parent  3
        links   1
        xattr   0
        rdev    0x0000000000000000

Now, this output is short because the dataset include only a single file. In the real world this output will be gigantic and should be redirected to a file. When I did this on the dataset containing my home directory the output file was 750MB... its a lot of data.

Look specifically at Object 4, a "ZFS plain file". Notice that I can see that files pathname, uid, gid, a/m/c/crtime, mode, size, etc. This is where things can get really interesting!

In zdb's 3rd form above (-R) we can actually display the contents of a file, however we need its Device Virtual Address (DVA) and size to do so. In order to get that information, we can zoom in using -d little further, but this time just on Object 4:

root@quadra /$ zdb -ddddd testpool/dataset01 4
Dataset testpool/dataset01 [ZPL], ID 30, cr_txg 6, 19.5K, 5 objects, rootbp [L0 DMU objset] 400L/200P DVA[0]=0:172e000:200> DVA[1]=0:460e000:200> fletcher4 lzjb LE contiguous birth=168 fill=5 cksum=a280728d9:448b88156d8:eaa0ad340c25:21f1a0a7d45740

    Object  lvl   iblk   dblk  lsize  asize  type
         4    1    16K    512    512    512  ZFS plain file
                                 264  bonus  ZFS znode
        path    /testfile01
        uid     0
        gid     0
        atime   Fri Oct 31 12:35:51 2008
        mtime   Fri Oct 31 12:35:51 2008
        ctime   Fri Oct 31 12:35:51 2008
        crtime  Fri Oct 31 12:35:51 2008
        gen     8
        mode    100644
        size    21
        parent  3
        links   1
        xattr   0
        rdev    0x0000000000000000
Indirect blocks:
               0 L0 0:11600:200 200L/200P F=1 B=8

                segment [0000000000000000, 0000000000000200) size   512

Now, see that "Indirect block" 0? Following L0 (Level 0) is a tuple: "0:11600:200". This is the DVA and Size, or more specifically it is the triple: vdev:offset:size. We can use this information to request its contents directly.

And so, the -R form can display and individual blocks from a device. To do so, we need to know the pool name, vdev/offset (DVA) and its size. Given what we did above, we now know that, so lets try it:

root@quadra /$ zdb -R testpool:0:11600:200
Found vdev: /zdev/disk002

testpool:0:11600:200
          0 1 2 3 4 5 6 7   8 9 a b c d e f  0123456789abcdef
000000:  2073692073696854  6620747365742061  This is a test f
000010:  0000000a2e656c69  0000000000000000  ile.............
000020:  0000000000000000  0000000000000000  ................
000030:  0000000000000000  0000000000000000  ................
000040:  0000000000000000  0000000000000000  ................
000050:  0000000000000000  0000000000000000  ................
...

w00t! We can read the file contents!

You'll notice in the zdb syntax ("zdb -h") that this syntax above accepts flags as well. We can find these in the ZDB source. The most interesting is the "r" flag which rather than display the data as above, actually dumps the data in raw form to STDERR.

So why is this useful? Try this on for size:

root@quadra /$ rm /testpool/dataset01/testfile01
root@quadra /$ sync;sync
root@quadra /$ zdb -dd testpool/dataset01
Dataset testpool/dataset01 [ZPL], ID 30, cr_txg 6, 18.0K, 4 objects

    Object  lvl   iblk   dblk  lsize  asize  type
         0    7    16K    16K    16K  14.0K  DMU dnode
         1    1    16K    512    512     1K  ZFS master node
         2    1    16K    512    512     1K  ZFS delete queue
         3    1    16K    512    512     1K  ZFS directory

 ....... THE FILE IS REALLY GONE! ..........

root@quadra /$ zdb -R testpool:0:11600:200:r  2> /tmp/output
Found vdev: /zdev/disk002
root@quadra /$ ls -lh /tmp/output 
-rw-r--r-- 1 root root 512 Nov  1 01:54 /tmp/output
root@quadra /$ cat /tmp/output 
This is a test file.

How sweet is that! We delete a file, verify with zdb -dd that it really and truely is gone, and then bring it back out based on its DVA. Super sweet!

Now, before you get overly excited, some things to note... firstly, if you delete a file in the real world you probly don't have its DVA and size already recorded, so your screwed. Also, notice that the origonal file was 21 bytes, but the "recovered" file is 512... its been padded, so if you recovered a file and tried using an MD5 hash or something to verify the content it wouldn't match, even though the data was valid. In other words, the best "undelete" option is snapshots.. they are quick, easy, use them. Using zdb for file recovery isn't practical.

I recently discovered and used this method to deal with a server that suffered extensive corruption as a result of a shitty (Sun Adaptec rebranded STK) RAID controller gone berzerk following a routine disk replacement. I had several "corrupt" files that I could not read or reach, if I tried to do so I'd get a long pause, lots of errors to syslog, and then a "I/O Error" return. Hopeless, this is a "restore from backups" situation. Regardless, I wanted to learn from the experience. Here is an example of the result:

[root@server ~]$ ls -l /xxxxxxxxxxxxxx/images/logo.gif
/xxxxxxxxxxxxxx/images/logo.gif: I/O error

[root@server ~]$  zdb -ddddd pool/xxxxx 181359
Dataset pool/xxx [ZPL], ID 221, cr_txg 1281077, 3.76G, 187142 objects, rootbp [L0 DMU objset] 400L/200P DVA[0]=0:1803024c00:200> DVA[1]=0:45007ade00:200> fletcher4 
lzjb LE contiguous birth=4543000 fill=1
87142 cksum=8cc6b0fec:3a1b508e8c0:c36726aec831:1be1f0eee0e22c

    Object  lvl   iblk   dblk  lsize  asize  type
    181359    1    16K     1K     1K     1K  ZFS plain file
                                 264  bonus  ZFS znode
        path    /xxxxxxxxxxxxxx/images/logo.gif
        atime   Wed Aug 27 07:42:17 2008
        mtime   Wed Apr 16 01:19:06 2008
        ctime   Thu May  1 00:18:34 2008
        crtime  Thu May  1 00:18:34 2008
        gen     1461218       
        mode    100644
        size    691  
        parent  181080
        links   1
        xattr   0
        rdev    0x0000000000000000
Indirect blocks:
               0 L0 0:b043f0c00:400 400L/400P F=1 B=1461218

                segment [0000000000000000, 0000000000000400) size    1K

[root@server ~]$ zdb -R pool:0:b043f0c00:400:r 2> out
Found vdev: /dev/dsk/c0t1d0s0
[root@server ~]$ file out   
out: GIF file, v89

Because real data is involved I had to cover up most of the above, but you can see how the methods we learned above were used to gain a positive result. Normal means of accessing the file failed miserably, but using zdb -R I dumped the file out. As a verification I opened the GIF in an image viewer and sure enough it looks perfect!

This is a lot to digest, but this is about as simple a primer to zdb as your going to find. Hopefully I've given you a solid grasp of the fundamentals so that you can experiment on your own.

Where do you go from here? As noted before, I recommend you now check out the following:

Max Bruning's ZFS On-Disk Format Using mdb and zdb: Video presentation from the OpenSolaris Developer Conference in Prague on June 28, 2008. An absolute must watch for the hardcore ZFS enthusiast. Warning, may cause your head to explode!
Marcelo Leal's 5 Part ZFS Internals Series. Leal has tremendous courage to post these, he's doing tremendous work! Read it!

Good luck and happy zdb'ing.... don't tell Sun. :)

work and pleasure

2008-11-01T00:48:21+00:00

Today I finished integrating some cool code into Evas: box and table. These utility smart objects are now in Evas for good, we can stop replicating those in many projects and people who just want to use them and not a full featured toolkit like ETK or EWL are now free. More importantly: we can now expose these in Edje, making all layout elements dependent on theme, not having to rely on SWALLOW slots!

The integrated code is very flexible, it make use of the recently introduced “size hints” and also postpone heavy calculations to pre-render time with calculate smart callback. Table has three modes: regular, homogeneous based on table size and homogeneous based on largest minimum item size. Box, since it just represent a sequence of items, is more extensible and allows you to specify a layout function, we provide some like vertical, horizontal, stack, homogeneous based on box, homogeneous based on the largest minimum item size, etc… but you can easily write your “snake layout” and use it. If you need more option details than “size hints”, you can extend the class and implement options_* virtuals.

These code were integrated by me, but not totally written. Gustavo Lima, from ProFUSION, wrote the box for their sequence_box.c (it was relicensed to E’s BSD with permission) and Rasterman wrote table for his elementary “toolkit for mobiles” els_table.c.

On the pleasure and work side, I’ll fly to The Netherlands next Tuesday so I can attend ELC-E 2008 where I’ll present a talk about Rich Graphical User Interfaces on mobile systems, covering Evas, Edje and the new kids on the block Elementary and Guarana.

Last but not least, due trip and other stuff to do I’ll not be able to integrate table and box into Edje soon. If you always wanted to help E17 and EFL, now it’s your chance! See my mail to the list and start hacking, I can reply to you by mail and IRC (when I’m online). We will also need Python bindings for those, so patches to python-evas and python-edje are also welcome!

Explore Your ZFS Adaptive Replacement Cache (ARC)

2008-10-29T19:07:00+00:00

Some time ago I wrote a tool which I call arc_summary. It is a kstat based PERL application that runs some calculations and presents you with a pretty report of ZFS ARC. The idea is to help you interpret the data more appropriately. Lets look at some output:

benr@quadra ~$ ./arc_summary.pl 
System Memory:
         Physical RAM:  4083 MB
         Free Memory :  130 MB
         LotsFree:      63 MB

ZFS Tunables (/etc/system):

ARC Size:
         Current Size:             1530 MB (arcsize)
         Target Size (Adaptive):   1555 MB (c)
         Min Size (Hard Limit):    382 MB (zfs_arc_min)
         Max Size (Hard Limit):    3062 MB (zfs_arc_max)

ARC Size Breakdown:
         Most Recently Used Cache Size:          100%   1555 MB (p)
         Most Frequently Used Cache Size:         0%    0 MB (c-p)

ARC Efficency:
         Cache Access Total:             2090556
         Cache Hit Ratio:      71%       1493620        [Defined State for buffer]
         Cache Miss Ratio:     28%       596936         [Undefined State for Buffer]
         REAL Hit Ratio:       70%       1472533        [MRU/MFU Hits Only]

         Data Demand   Efficiency:    98%
         Data Prefetch Efficiency:     1%

        CACHE HITS BY CACHE LIST:
          Anon:                        0%        12854                  [ New Customer, First Cache Hit ]
          Most Recently Used:         55%        825782 (mru)           [ Return Customer ]
          Most Frequently Used:       43%        646751 (mfu)           [ Frequent Customer ]
          Most Recently Used Ghost:    0%        3619 (mru_ghost)       [ Return Customer Evicted, Now Back ]
          Most Frequently Used Ghost:  0%        4614 (mfu_ghost)       [ Frequent Customer Evicted, Now Back ]
        CACHE HITS BY DATA TYPE:
          Demand Data:                60%        900081 
          Prefetch Data:               0%        9547 
          Demand Metadata:            38%        572127 
          Prefetch Metadata:           0%        11865 
        CACHE MISSES BY DATA TYPE:
          Demand Data:                 2%        12318 
          Prefetch Data:              87%        524549 
          Demand Metadata:             9%        55222 
          Prefetch Metadata:           0%        4847 
---------------------------------------------

First notice that you do not need to be root to run the report. I ran this on my personal workstation, an Intel QuadCore with 4GB of RAM.

So looking at the report, first I output some general memory data for you, then output any ZFS tunable you placed in /etc/system (this will not utilize MDB, but may one day).

Then we look at the ARC Sizing data. The "Current Size" is how large the ARC really is, whereas the "Target Size" is a constantly changing number, like a stock price, of what the ARC thinks it should be. I also include the Min and Max hard sizes for ARC, these are tunable.

The ARC Size Breakdown shows the division of the cache between "Most Recently Used" (MRU) and "Most Frequently Used". More on this later. When you see parenthesis in the output those are the variable names, for those using DTrace or getting into the code.

I wrote arc_summary to answer the question "Why is my ARC so damned big!?!" The ARC Efficiency section helps you determine where or not your getting bang for your buck. ZFS ARC stats presented by Kstat provide several breakdowns of cache hits, so this section really runs computation on those values to present them in a sane manner.

In the report above, we have a cache hit ratio of 70%. The "Cache Hit Ratio" and "REAL Hit Ratio" differ in that ZFS considers an Anon buffer as a hit... but its not really, so I remove those for the "REAL" ratio.

We then breakdown Demand vs Prefetch hits. ZFS does aggressively caches data via prefetch; in the example above, only 1% of hits are a result of prefetched data already being in the cache, 98% of it was hit in the cache because it was explicitly asked for. So in this case, pre-fetch isn't helping me.

Perhaps the most interesting information here is the "Cache Hits by Cache List". ARC maintains multiple cache lists, including:

Most Recently Used: Cache hit once.
Most Frequently Used: Cache hit multiple times.
Most Recently Used Ghost: Objects that were in MRU cache, but removed to save space.
Most Frequently Used Ghost: Same but for MFU.

These Ghosts lists are magic. If you get a lot of hits to the ghost lists, it means that ARC is WAY too small and that you desperately need either more RAM or an L2 ARC device (likely, SSD). Please note, if you are considering investing in L2 ARC, check this FIRST.

So by looking at the spread between MRU and MFU hits, we get an idea of how much the cache is rolling over.... 55% of hits are MRU (1 hit), whereas 43% are MFU (multiple hits), so, not bad. The higher MFU is, the better, but this all depends on your workload. Workstations will tend toward MRU, but on one of my servers, for instance, the ARC was 99% MFU.

Looking on we can see what type of requests are hits or misses. Notice that cache hits are 60% demand (explicitly requested) data blocks, and 38% demand metadata.... prefetch isn't helping on this box.

Its all really pretty self explanatory, if its not, let me know and I'll do an ARC expose, but I tried to create the output report to be pretty intuitive.

Please note, this report is cumulative since boot. It should compliment the arcstats tool. arcstat can tell you want is happening, arc_summary can tell you what has been happening. Any serious ZFS deployment should have both of these bad boys around in your toolbag.

On a closing note... I hope this tool helps you appreciate the innovation of the ARC. Designed at IBM (IBM ARC page) and then implemented and improved for ZFS, ARC is elegant, powerful and extremely efficient! I hope my tool can help you better appreciate this amazing innovation in some small way.

Best. Wife. Ever.

2008-10-29T18:37:00+00:00

I'm older... it won't stop. But my wife makes me feel better with kickassness. Behold my "new" mint condition Faber-Castell 2/83N.

To compliment, I just recently bought a copy of Euclid's The Elements. I love mathematics.... I suck at it, but who cares, I enjoy it.

GNOME Summit, Seed, and more!

2008-10-22T04:16:44+00:00

Firstly, I’ve started tagging photos that I’ve used as backgrounds before here.

First-and-a-halfly: posting to WordPress from WebKit nightlies is still broken. Firefox on OS X. Yuk.

Secondly! Last weekend was the GNOME Developer’s Summit; Matt and Robb and I went. We stayed at Matt’s house again, which was excellent and convenient as always! The first day involved a talk from two of the awesome guys working on gobject-introspection (which is also a cornerstone of our project), which cleared up some of my fuzziness about introspection and their implementation (to some extent).

We (Robb) talked to Mark a little bit that day, and explained our plans (which I have yet to detail here, and won’t today!). More about all that later…

Seed release later this week, after the planned gobject-introspection release. Yay for Seed! I’ll write more about it after the release, post the tutorial I’m writing, etc. Pay no attention to the tutorial in SVN; I haven’t gotten a GNOME SVN account yet, so none of my updates are there… we just switched to SVN from a Bazaar repo @ Launchpad, so all of our commit history is missing too… oh well. In any case… all will be fun!

P.S. Intervalometer is waiting on PCBs from BatchPCB, so it might be a while!

P.P.S. Registration soon (the third)! Looking at… Programming Languages, Graph Theory, Models of Computation, Multivar, Engineering Processes, Electronic Circuits (that’ll get me through the end of Junior year CS and a little further… also not taking Typography this semester because apparently the teacher is… not… optimal… and everyone)

P.P.P.S. Robb’s going to be working for litl starting next week. Yay for an NDA with your roommate

Running Illume everywhere!

2008-10-16T23:31:10+00:00

Rasterman’s Gang: Nokia N810, Sharp Zaurus, OpenMoko Freerunner and Palm Treo-650

After Rasterman announced he had “The Gang” running Illume we decided to help him and run it in yet-another platform, the Freescale iMX31:

Some days ago raster already posted video of his virtual keyboard doing correction/prediction and operating on various resolutions, for those that liked my iPhone-like virtual keyboard demo for n800, this one looks better and is for real, check out his videos: 01, 02, 03 and 04

Blast!

2008-10-16T02:19:28+00:00

Hey all,

Im up to my old tricks again, making some neat things for you to tinker with! This time Ive added to the Inspire wallpaper and added some stuff to make scaling easier. What that means to us artists, is that our work stays at the correct aspect when displayed on widescreen or standard displays. This is something that normal pixmap backgrounds obviously can not do, so we might as well use the tools we have!

Remember, this is only a demonstration of what can be done with these layers and parts. There are a multitude of things to play with in edje.

Lets get started! Firstly, fire up GIMP and create a new image at 800 x 600. (This resolution is only used for demo purposes) I created 4 layers. The base is the Blue galaxy swirl thing. This will layer be stretched to fill all the space. The next 2 are the corner graphics on their own separate layer. The last layer on top is the E logo, but you could swap this for anything, or scrap it entirely.

All the layers in Blast. These become edje parts later.

Next, save each layer as a separate image. This can be done by making the target layer the only visible one then saving and repeating the process. You might be able to Save Layer As… in GIMP 2.6, but I haven’t checked it out yet. Once you have all the layers separated, open them again and resize them to save some space. Things like E logo need to be in the dead centre of the image to centre correctly later.

Now comes the super fun code part! Heres the .edc from Blast. Its already commented so read on!

collections {
   group { name: "e/desktop/background";
      //
      // Lets make a wallpaper that will SCALE well!!! Because not everyone
      // has the same screen aspect these days. There are a couple ways to
      // get this to work. In this exmaple it keeps the objects at all the
      // same sizes so that no scaling happens.
      //
      // All min and max sizes are taken from their respective image properties
      //
      images {				// Include all files from the XCF
	 image: "Scaled.png" COMP;
	 image: "LeftBit.png" COMP;
	 image: "RightBit.png" COMP;
	 image: "ELogo.png" COMP;
      }
      parts {
	 part { name: "Scaled";
	    description { state: "default" 0.0;
	       image.normal: "Scaled.png"; // Make it fill all the space
	    }
	 }
	 part { name: "LeftBit";
	    description { state: "default" 0.0;
	       align: 0.0 1.0;		// Align left and at the bottom.
	       min: 334 314;		// Set the min size and...
	       max: 334 314;		// max size so it wont scale.
	       image.normal: "LeftBit.png";
	    }
	 }
	 part { name: "RightBit";
	    description { state: "default" 0.0;
	       align: 1.0 0.0;		// Align Right and at the top.
	       min: 260 284; 		// Set the min size and...
	       max: 260 284;		// max size so it wont scale.
	       image.normal: "RightBit.png";
	    }
	 }
	 part { name: "ELogo";
	    description { state: "default" 0.0;
	       min: 165 224;		// The absence of an align: will
	       max: 165 224;		// make the image centred.
	       image.normal: "ELogo.png";
	    }
	 }
      }
   }
}

Next you need to compile it up using ‘edje_cc blast.edc’ then import the picture from the Wallpaper selector. You should have a well scaled wallpaper that preserves your efforts for all screen sizes.

Here are some links to the wallpaper at a few different sizes for comparison.

1650×1080 (My Desktop)

480×640 (Embedded style display)

Wallpaper Selector

And here is the all important EDJ file for you to take home and study!

Enjoy!

Toma.

TomaBlast layout in GIMP.

A Peek at Ruby Internals

2008-10-14T02:08:20+00:00

The MountainWest Ruby Conf 2008 had an interesting talk on Ruby Internals by Patrick Farley . If you’ve worked with Ruby and are interested in learning more about classes, meta-classes and how all the method dispatch magic works give it a gander.

Well worth the 50 minutes of time.

x11 benchmark on embedded systems

2008-10-13T22:30:28+00:00

Unlike most would say, x11 does quite well. See our benchmarks comparing X11, FB, DirectFB on a Fresscale imx31. There you can see how x11-16 bitch-slaps everything… that’s even better on n8×0 and that’s why Canola runs fast

Blastwave Saga Continues

2008-10-10T23:30:00+00:00

You'll recall my "unofficial" Blastwave update in August. The separation of Phil and Dennis seems to be solidifying a bit. The CSW Project (Phil and co.) are now up and running at OpenCSW.org. Minus the blastwave name and CSS, its more or less the same site and content you had/have at blastwave.org.

OpenCSW.org includes a History page, which tells the CSW side of the breakup, minus the unverified assertions by Dennis Clarke that Phil Brown tried to setup with a company in Europe. To this end, Dennis has, naturally, written a response: A simple response to the OpenCSW crew, a blog entry that is simply a re-post of a CSW Maintainers list thread.

If you are interested in the unfolding soap opera, read and enjoy. If you simply enjoy using CSW/Blastwave software, nothing has yet changed sufficient to cause any reason to notice.

Please note, if you are new to this saga and read through the links above please bear in mind why Solaris 8 support is such a big issue. See, most of the existing Solaris install base is still running Solaris 8 on SPARC, furthermore, because of Solaris binary compatability, anything built on Solaris 8 (Solaris 2.6 for that matter) will run on Solaris 9, Solaris 10, Nevada or OpenSolaris... the only downside is that building software on Solaris 8, due to its age, can be a big more painful than on Solaris 10. The argument then falls into two camps: deal with the pain and be available to more users, or dump older releases to make builds easier and thus hope to attract new maintainers. In a nutshell anyway. Enjoy.

Welcome to the party, Apple. 4 months later…

2008-10-10T17:06:36+00:00

I’m on my way to the GNOME Developer’s Summit, but some news just came down the RSS tubes:

MacBook Pro video flaw

Apparently they’re going to pay us back for having repaired our machines because of the flaw, and they’re fixing the flaw for free from now on out (I presume this means out-of-warranty, which is good for everyone!). They’re also acknowledging that it was still a problem with machines shipping last month (probably even this month), and that NVIDIA supposedly misled them in suggesting that no MacBook Pros were affected…

The Cascading Crash and You

2008-10-09T21:29:00+00:00

I've been surprised how few people are blogging about the current market situations. Its coming up in casual conversation "around the water cooler" and dominates the news channels, but tech outlets aren't jumping on the bandwagon.

Today on CNBC's "Fast Money", following todays almost 700 point drop, bringing the Dow under 9,000, I heard the first mention of the word "crash", or more properly what they called a "cascading crash". IBM perked up shortly but Forbes is saying IBM won't save tech. Sun shares are back to pre-reverse split levels at $5.21 a share, an 8% drop thats in line with most other tech stocks today.

Despite the markets falling apart it seems to me that most "joe 4 packs" (I drink Guinness) are trying to stay calm. The big question that folks are trying to avoid is at what point will see jobs being cut as a direct result of the markets. So far the people that I'm talking to aren't freaking out but seem secretively concerned.

The industry wisdom seems to be that this is yet another situation where using technology as a competitive weapon and means to reduce cost may save the day. So far thats what "they" are saying but we're not seeing the effect quite yet.

As for myself, I've been blessed so far. When we recently bought our house we liquidated most of our equities and now have some to give back. I'm jumping on the Buffet bandwagon and looking for value, given that we have very little now to invest, and yesterday picked up some GE shares at a great price. Even though GM and Ford are in the tank and dropping quickly, it looks like a great value for a long horizon investment.

So drop a comment, as a time capsule to ourselves.... how are you feeling about the current financial crisis and its impact on your livelyhood?

The Cold Weather has Arrived

2008-10-09T04:00:00+00:00

Summer of code is wrapping up. Only thing left is the mentor summit which I'm very excited about. I'm glad I have the opportunity to attend and hope that I can meet some people while I'm out there. It should also be a lot warmer there ;)

All in all the summer was good. My student (ptomaine) did some good work on efm. It is now much better then before and there are many things on his todo list. I also have a few things I'd like to finish when I find the time and motivation.

In other news, work is going alright. I've recently been moved to working on something else which I've never dealt with before so it's been exciting to dig into something new. My previous project is on hold for unknown reasons, but that's fine with me :)

Until next time, good night.