ivoras' Home

My CV

Name: Ivan Voras
Title: dr.sc. (PhD in Computer Engineering)
Currently employed: University of Zagreb, Croatia, Faculty of electrical engineering and computing (Expert associate, Internet services architect)

My CV is available on request. I'm usually available for temporary contracting work in the areas of designing and deploying Unix systems, system-level programming in C & similar jobs, with some experience in development team leadership.

What's cooking for FreeBSD 9?

This page will document changes that will be included in FreeBSD 9, including those that might end up being committed to earlier branches. In other words, it describes differences between 8.0 and 9.0, no matter what happens to the versions in between.

For information on the currently released 8-STABLE branch see What's cooking for FreeBSD 8 page.

Some of the more important low-level changes can be seen in the future release's UPDATING file.

Also useful are the quarterly Status Reports:

Everyone is encouraged to download a snapshot CD image and try all the new features (as well as the old ones). Developers are very interested in bug reports. Note that FreeBSD 9.0 is not released yet and both the snapshots and the default source trees have debugging enabled by default (which results in dramatic slowdowns so don't benchmark them without removing the debugging options).

If you're interested in how FreeBSD gets developed, you're encouraged to read the mailing lists and developer blogs.

Overall system / architectural changes

Userland DTrace

Status: committed to -CURRENT.
Will appear in 9.0: sure
Author: Rui Paulo
Web: announcement, commit message

The kernel parts of the DTrace system diagnostic framework were imported some time ago, but they are now completed with the support for userland tracing, making it usable in general userland software development and system administration. Userland DTrace is already used in some large well known software packages such as PostgreSQL and X.Org.

CLANG / LLVM compiler

Status: Committed to -CURRENT.
Will appear in 9.0: yes
Authors: Roman Divacky, Pawel Worach, Ed Schouten and many others
Web: LLVM, CLANG, FreeBSD wiki, mailing list announcement

As the GCC compiler suite was relicensed under GPLv3 after the 4.2 release, and the GPLv3 is a big dissapointment for some users of BSD systems (mostly commercial users who have no-gplv3-beyond-company-doors policy), having an alternative, non-GPL3 compiler for the base system has become highly desireable. Currently, the overall consensus is that GCC 4.3 will not be imported into the base system (the same goes for other GPLv3 code).

The LLVM and CLANG projects together offer a full BSD-licenesed C/C++ compiler infrastructure that is, performance and feature-wise close to, or better than GCC. The LLVM is the backend and the CLANG is the front-end part of the infrastructure.

Recent development has shown that not only is it possible to start using LLVM+CLANG right away, it is also very stable. The probability of replacing GCC for the base system in the near future is high. LLVM/CLANG will also add benefits to the overall system such as better error reporting, Apple's Grand Central Dispatch system for developing multithreaded applications and possibly JIT compiling some internal structures like firewall rules.

Note that this mostly affects the base system. There is too much third party software (e.g. ports) that depends on GCC to completely replace it.

Update: To make this happen, PathScale has developed and donated the C++ runtime library under the BSD license. PathScale has some other tools and libraries which may in the future help the transition to a BSD-licensed toolkit: the assembler (pathas) and the debugger (pathDB).

Kernel & low level improvements

Large-scale SMP support

Status: Committed to -CURRENT
Will appear in 9.0: sure
Author: Atillio Rao and others
Web: commit message

This work brings in support for large SMP systems, with more than 32 CPUs. Previously, the kernel structures were unable to account for such a large number of CPUs so the newest method implements extensible CPU accounting. This is not an improvement in scalability in itself but is a prerequisite for large-scale SMP work.

USB 3.0 support

Status: Committed to -CURRENT
Will appear in 9.0: sure
Author: Hans Petter Selasky
Web: commit message

The new wave of USB changes improves on the hugely successful USB development released in FreeBSD 8 and brings in support for USB 3.0.

Network kernel core dumps (netdump)

Status: In development
Will appear in 9.0: probably
Author: Attilio Rao and others
Web: announcement

Netdump is a framework that aims for handling kernel coredumps over the TCP/IP suite in order to dump to a separate machine than the running one. That may be used on an interesting number of cases involving disk-less workstations, disk driver debugging or embedded devices.

Initial NUMA support

Status: Committed to -CURRENT
Will appear in 9.0: sure
Author: John Baldwin
Web: Commit message, discussion

As NUMA-like architectures have become almost ubiqutous, even in i386 / amd64 architectures, there are potentially big performance gains to be had in enabling its supports within operating systems. New development aims to adapt the physical page allocator to be NUMA-aware.

Modern event timer infrastructure

Status: Committed to -CURRENT
Will appear in 9.0: sure
Authors: Alexander Motin
Web: Commit message 1 Commit message 2

To better support the many sources of timer ticks present in todays system and to build the foundation for tickless kernel, a new unifying timer infrastructure was created. It currently supports LAPIC, HPETs, i8254, RTC.

Tickless kernel

Status: Committed to -CURRENT
Will appear in 9.0: sure
Authors:Tsuyoshi Ozawa, Alexander Motin
Web: idea, new timer infrastructure

To improve performance in virtual machines and power usage in laptops, the "dynamic tick mode" (also called, a bit inappropriately, "tickless mode") can replace the classic, strictly periodic hardware timer interrupt ticking with one-shot variable-time ticks. This will save some CPU time which would otherwise be spent handling timer interrupts which have no work assigned to them. The tickless mode is enabled by default and can be turned off by setting the kern.eventtimer.periodic to 1.

Networking improvements

High performance ssh

Status: Committed to -CURENT, MFC-ed
Will appear in 9.0: sure
Author: Brooks Davis
Web: commit message

SSH is commonly used for copying files but also for connection tunneling and similar operations which stream a large amount of data over the encrypted channels. The default configuration is somewhat conservative and does not allow the full bandwidth potential of "fat and long" network connections to be achived. The HPN-SSH project has modified the ssh's flow control buffers to achieve much better tranfer rate, and also includes a multithreaded cipher implementation which makes such bandwidth sustainable on the CPU side. The results of the HPN-SSH project (version 13) are now imported into FreeBSD.

More SMP-scalable TCP/IP

Status: Committed to -CURRENT
Will appear in 9.0: sure
Author: Robert Watson
Web: commit message announcement

Improvements to the networking stack introduce better scalability strategies based on the work by Alan Cox and others. With these changes, it is expected that the connections will have more clear CPU affinity, less cache line contention and better use of modern hardware flow detection and handling.

New NFS client and server

Status: Committed to -CURRENT
Will appear in 9.0: sure
Author: Rick Macklem
Web: commit message

The new NFS client and server introduce the support for NFSv4 as their biggest features, with ACL support, byte range locking and delegation support. It should also be easier to maintain and later upgrate do NFSv4.1

Five new TCP congestion algorithms

Status: Partially committed to -CURRENT
Will appear in 9.0: sure
Author: Lawrence Stewart
Web: Commit message

This commit marks the first formal contribution of the "Five New TCP Congestion ontrol Algorithms for FreeBSD" FreeBSD Foundation funded project. More details about the project are available at: http://caia.swin.edu.au/freebsd/5cc/.

SIFTR - Statistical Information for TCP Research

Status: Committed to -CURRENT
Will appear in 9.0: sure
Author: Lawrence Stewart
Web: commit message

SIFTR logs a range of statistics on active TCP connections to a log file, providing the ability to make highly granular measurements of TCP connection state. The tool is aimed at system administrators, developers and researchers.

Storage subsystems' improvements

A move to support 4K drives

Status: committed

Will appear in 9.0: sure

Authors: Kirk McKusick, Alexandar Motin, Andrey V. Elsukov and others

Web: UFS fragment size commit, SATA quirks commit, gpart alignment calculation

FreeBSD's GEOM and file systems have intrinsically supported large (or even arbitrary) sector sizes for a long time, but there is still the issue of detecting them and communicating this information across the layers. Some new development introduced SATA quirks to detect known 4K drives (with the ability for users to set their own quirks on non-detected drives), the gpart(8) utility will calculate the correct alignment or warn on misalignment, and the default fragment / block size for UFS was changed to 4K / 32K.

Generic GEOM IO schedulers

Status: Committed to -CURRENT
Will appear in 9.0: sure
Authors: Luigi Rizzo, Fabio Checconi
Web: commit message

The new framework, integrated with GEOM, allows for multiple disk IO schedulers to be used, if necessary, on different IO providers (e.g. drives). The usage of some IO schedulers can increase responsiveness in certain kinds of IO workloads, for example a mix of sequential and random IO.

HAST - High Availability Storage

Status: Committed to -CURRENT, planned MFC
Will appear in 9:0: sure
Author: Pawel Jakub Dawidek
Web: FreeBSD wiki page

HAST is a userland-based (ggate) implementation of a distributed storage device concept, similar to Linux's DRBD. It allows over-the-network mirroring of any GEOM storage devices in a semi-synchronous way (writes suceed when the data is sent over the wire).

UFS SoftUpdates+Journal (SU+J)

Status: Pending commit to -CURRENT
Will appear in 9.0: sure
Author: Jeff Roberson
Web: discussion

A new feature added to existing UFS SoftUpdates code makes use of a small journal, technically an intent log, to keep track of metadata garbage collection which has upto now been left as a job for (background) fsck after an unclean shutdown. The intent behind this is to eliminate the requirement for fsck or background fsck on file systems with SoftUpdates enabled after unclean shutdown.

In effect, this feature combines the best of both worlds - the very fast operation of SoftUpdates with the removal of the need for fsck characteristic for journalling file systems. This is not a radical change - the well known SoftUpdates mechanism is still in its original form - but it completes the garbage collection step in a different way.

New driver for AHCI SATA drives

Status: Committed to -CURRENT, MFC-ed
Will appear in 9.0: sure
Author: Alexandar Motin & others
Web: commit message

The new driver supports native AHCI via the CAM (common access method for storage) subsystem. AHCI drives are manipulated by camcontrol and support for new features like NCQ and port multipliers has been integrated. Among other features, performance has been significantly increased, port multipliers and hot-plugging are greatly improved.

ATA CAM implementation

Status: Committed to -CURRENT
Will appear in 9.0: sure
Author: Alexandar Motin
Web: commit message

The ATA disk drivers have all been moved to the CAM system, improving some features of them along the way. This makes CAM a very real central point and foundation of disk interfaces and management of (S)ATA, SCSI, USB and Firewire drives. Some SCSI controllers still have drivers outside CAM.

Improvements include: transfer size increase, better support for port multipliers.

Security

Capsicum

Status: Basic functions committed to -CURRENT
Will appear in 9.0: sure
Author: Robert Watson and others
Web: commit message

Capsicum is a framework for security isolation of sensitive processes, which may prove useful in security strengthening the operating system.

AES-XTS encryption mode in kernel

Status: Committed to -CURRENT
Will appear in 9.0: sure
Author: Pawel Jakub Dawidek
Web: AES-XTS in kernel AES-XTS in GELI AES-XTS via AESNI

The XTS block cypher mode is specially suited for encrypting disk drives and other block devices. It avoids some security problems arising with using plain CBC chaining with addressible-sector encryption.

AES with XTS mode is used in GELI and is also supported when implemented via the AES-NI.

NFSv4 ACLs for UFS

Status: Committed to -CURRENT, MFCed
Will appear in 9.0: sure
Author: Edward Tomasz Napierala
Web: commit message

The well known and loved UFS file system has for some time implemented POSIX.1e ACLs (access control lists) in addition to the classic Unix file permissions model. This file permission model greatly enhances the way files can be managed and allows new security models to be implemented. It is also a standard part of the FreeBSD kernel, ready to be used at any time.

However, the POSIX.1e standard apparently never became trully widespread in practice. Through market share domination (but not completly without technical merit) the NTFS (Microsoft Windows file system) ACL security model has become widely popular and implemented, even so that it directly inspired the ACL model in the NFS (Network File System) version 4. The POSIX model is simpler and more Unix-like but the NTFS/NFSv4 model is more expressive.

The two ACL models are incompatible - security parameters set in NFSv4 model cannot always be directly translated to the POSIX model. Due to this and considering that NFSv4 ACLs are already directly implemented in ZFS, the introduction of NFSv4 ACLs in UFS is simply a feature-completness step which makes both file systems similarily usable from NFSv4 clients.

The POSIX model still remains in the implementation, but is mutualy exclusive (at the mount-point level) with the NFSv4 model.

Other changes

The following is a list of smaller and / or more obscure changes that nevertheless deserve a special mention since they will be of interest to certain users:

A new utility called fuser, used to get a list of processes which have one or more files open. It can be used to get a list of processes having an open file on a mount point (stas)
Default UFS block & fragment sizes increased to 32 KiB / 4 KiB (mckusick)
BSD licensed grep (obrien)
Support for ATA/AHCI per-device write cache disable (mav)
Syscall performance regression fixed (jhb)
Resource accounting (a part of resource container work) (trasz)
Improvements to CPU detection (jkim)
Many bug fixes to hastd (pjd)
ZFS updated to version 28 (pjd)
Beginnings of process-kernel shared page infrastructure (kib)
TRIM (BIO_DELETE) support for UFS (kib)
MCA (Machine check architecture) logging enabled by default (jhb)
usbdump kernel facility and tool (weongyo)
New log rotation file system naming scheme with newsyslog, using timestamps (simon) [MFC-ed]
GELI secure suspend/resume functionality (pjd)
TPM driver (takawata et al)
Increased read-ahead and write queue UFS buffers for better NCQ utilization (ivoras)
AESNI (Nehalem CPU instruction) crypto(4) driver (kib)
DAHDI (Digium/Asterix Hardware Device Interface drivers
Resource containers (trasz)
ZFS updated to version 15 and synced with modern performance improvements (mm)
TLB shootdowns optimizations (alc)
Important fixes to tmpfs memory use and stability (kib)
Significant BSNMPD enhancements (syrinx)
Default SYSV semaphore and SHM sizes increased (ivoras)
Jails can be nested / hierarchical (jamie)
Jails / virtual network stack / os-level virtualization improvements (bz)
Import of xz and liblzma into the base system (mm)
EFI boot loader (rpaulo)
The FreeBSD logo integrated into syscon screensaver (jkim)
ZFS performance and stability improvements (kmacy)
Inter-process shared userland semaphores (davidxu)
ZFS updated to version 14 (delphij) [MFC-ed]
Implemented TRIM / ERASE (S)ATA commands for SSDs and similar drives, notifying them of unused disk space so it can be reclaimed (mav)
Updated gmirror (software RAID-1) load balancing (mav) [MFC-ed]
Solaris-like pwait(1) utility (jilles) [MFC-ed], refactored rc.d to use pwait(1).
IPFW becomes VNET-aware (virtualized network stack, i.e. a firewall instance per jail/VNET) and gets more SMP performance. (luigi)
Imported V4L (video 4 linux) support in the Linuxulator (aka Linux compatibility mode) (netchild)
Better support for Machine Check Exceptions (jb)
Reworked utmp system (ed)
Open-source 3D hardware acceleration for Radeon HD cards (nork, rnoland)

As always, all features described here are, or will be, a part of the FreeBSD "base" system, available in every FreeBSD installation without patching or out-of-the-ordinary configuration.

For more information about development of FreeBSD (among other topics), see my blog with daily and miscellaneous information.

My FreeBSD things

I've come to have so many things related to the FreeBSD project that I finally went and organised all into one section on the web. On the downside, the font of the sections' listing is getting smaller and smaller :)

This page lists only the smaller projects, patches, etc. - bigger projects have their own separate pages. See here for more projects.

What's cooking for FreeBSD?

On these pages I've been maintaining a short, informal overview of major features in major FreeBSD releases, but as I have less and less time to do it properly, I've since moved the content for FreeBSD 10 to FreeBSD's wiki server. I encurage anyone who has the time and interest to cotribute to the wiki page.

What's cooking for FreeBSD 10 - was transitioned to the FreeBSD wiki.
What's cooking for FreeBSD 9
What's cooking for FreeBSD 8
What's cooking for FreeBSD 7

There's also my blog which often has FreeBSD-related topics and my old blog.

Miscellaneous

Short tutorial on writing FreeBSD GEOM classes, and on FreeBSD kernel programming in general.

What's cooking for FreeBSD 10?

Hello,

Over the years I have been maintaining an unofficial list of what's going on in the world of FreeBSD development, but I have less and less time to do so.

To help transition to hopefully, someone who has more time to do it, I've moved the FreeBSD 10 list to the FreeBSD wiki page. I'm hereby inviting anyone with enough interest and time to contribute to this wiki page.

Python Blowfish

This is a slightly modified version of the pure python implementation of the Blowfish cipher by Michael Gilfix. This version contains methods to encrypt and decrypt arbitrary strings (of arbitrary length) using the CTR cipher mode. I've added three new methods: initCTR(), encryptCTR() and decryptCTR() to make this happen. It also contains CBC code contributed by Joel Edwards.

Download Blowfish in Python module.

You might also be interested in XXTEA cipher in pure Python.

Usage example:

552     key = 'This is a test key'
553     cipher = Blowfish (key)
554
555     print "Testing encryption:"
556     xl = 123456
557     xr = 654321
558     print "\tPlain text: (%s, %s)" %(xl, xr)
559     cl, cr = cipher.cipher (xl, xr, cipher.ENCRYPT)
560     print "\tCrypted is: (%s, %s)" %(cl, cr)
561     dl, dr = cipher.cipher (cl, cr, cipher.DECRYPT)
562     print "\tUnencrypted is: (%s, %s)" %(dl, dr)
563
564     print "Testing block encrypt:"
565     text = 'testtest'
566     print "\tText:\t\t%s" %text
567     crypted = cipher.encrypt (text)
568     print "\tEncrypted:\t%s" %crypted
569     decrypted = cipher.decrypt (crypted)
570     print "\tDecrypted:\t%s" %decrypted
571
572     print "Testing CTR encrypt:"
573     cipher.initCTR()
574     text = "The quick brown fox jumps over the lazy dog"
575     print "\tText:\t\t", text
576     crypted = cipher.encryptCTR(text)
577     print "\tEncrypted:\t", crypted
578     cipher.initCTR()
579     decrypted = cipher.decryptCTR(crypted)
580     print "\tDecrypted:\t", decrypted

How secure is it? For one, Blowfish itself has never been broken, so from this point of view it is as secure as AES and other newer algorithms. Blowfish is used in many security products like OpenBSD's OpenSSH, OpenSSL and for the password databases of BSD and Linux systems because of its early implementation (it was created in 1993 by Bruce Scheiner). Since it was created so long ago, it is designed to be very efficient, and it's very fast on modern hardware. Of course, an implementation in Python cannot be as fast as an optimized implementation in C.

The CTR mode effectively creates a stream cipher from the Blowfish block cipher, which is perfectly secure for most purposes except when using the same key more than once - which should be avoided.

One way to completely avoid the problem with reusing the key in stream ciphers is to create a random number which is appended to the password but which is stored or transmitted without encryption before the encrypted part of the message. The other side can use the same number with the password to decrypt the protected message.

Update: The implementation now contains a test with some of the standard Blowfish test vectors, which it nominally passes. If by any chance your tests fail (the tests can be run by simply executing blowfish.py in Python), it most likely means that your Python is broken.

What's cooking for FreeBSD 8?

The next major release of FreeBSD, version 8, was intended to be an "evolutional" release with few exciting changes. Of course by now it is obvious this will be another in a series of releases with groundbreaking changes.

This page will document changes that will be included in FreeBSD 8, including those that might end up being committed to earlier branches. In other words, it describes differences between 7.0 and 8.0, no matter what happens to the versions in between.

FreeBSD 8 has been released! For information on the formerly developed 7-STABLE branch see What's cooking for FreeBSD 7 page.

Some of the more important low-level changes can be seen in the future release's UPDATING file.

Also useful are the quarterly Status Reports:

Everyone is encouraged to download a snapshot CD image and try all the new features (as well as the old ones). Developers are very interested in bug reports. Note that FreeBSD 8.0 is not released yet and both the snapshots and the default source trees have debugging enabled by default (which results in dramatic slowdowns so don't benchmark them without removing the debugging options).

If you're interested in how FreeBSD gets developed, you're encouraged to read the mailing lists and developer blogs.

Overall system / architectural changes

INET-less / IPv6-only kernel

Status: under development, mostly works
Will appear in 8.0: probably not
Authors: Bjoern Zeeb and others
Web: mailing list announcement

As IPv6 development and deployment is progressing, at its own pace, there is interest in making it possible to run a FreeBSD system as IPv6-only (instead of the default configuration which is dual-hosted IPv4+IPv6).

Historically, BSD is the progenitor of all TCP/IP implementations and the IPv4 code in FreeBSD was sprawled across the network layers of the kernel, from device drivers to the higher socket layers. A recent initiative aims at fixing the layering violations in preparation to, at first, build a kernel without INET (i.e. IPv4) support, then build an IPv6-only kernel. This change involves large kernel subsystems such as the firewalls, bridging, NFS and others.

CLANG / LLVM compiler

Status: Experimental, but works. Highly motivated.
Will appear in 8.0: no
Authors: Roman Divacky, Pawel Worach, Ed Schouten and others
Web: LLVM, CLANG, FreeBSD wiki, mailing list announcement

Note that this mostly affects the base system. There is too much third party software that depends on GCC to completely replace it.

Parallel port builds

Status: Committed
Will appear in 8.0: sure
Authors: Pav Lucistnik and a cast of thousands
Web: mailing list announcement

The ports infrastructure is the part of the FreeBSD operating system that's responsible for making thousands (actually close to 20,000) of third party packages available to FreeBSD users. It enables everyone to install custom software from either source code (the traditional and preffered way) or from analogous binary packages.

The port infrastructure for source builds has been enhanced to allow parallel builds of individual ports. In the age of multi-core CPUs this means package build times will be drastically decreased. By default, all available logical CPUs will be used.

This enhancement is not tied to the 8.0 release and is available now on all recent versions of FreeBSD. Port dependancy graphs will still be built serially (i.e. only one port at a time will be built, but each individual port will be built in parallel).

Kernel & low level improvements

Better handling of mounted device removals

Status: Committed to -CURRENT, MFCed
Will appear in 8.0: sure
Author: Edward Tomasz Napierala
Web: FreeBSD Foundation Projects

Panics on "hot" removal of devices with file systems mounted from them (the canonical example is the removal of USB flash memory keys while the file system was mounted) were the most commonly reported problem from end-users. New development, funded by the FreeBSD foundation, has solved this issue.

Jails v2

Status: Committed to -CURRENT, MFCed
Will appear in 8.0: sure
Author: Bjoern Zeeb and others
Web: commit message

The jails subsystem has been greatly enhanced and updated to support modern FreeBSD features. In addition to the support for multiple IP addresses per jail (or none), support for IPv6 and SCTP has been implemented, jails can be nested hierarhically and jails can now be restricted to certain CPUs. Jails are especially powerful when combined with ZFS, where system administrators can be allowed to create and manage their own file systems within the jails.

Xen dom-U support

Status: Committed to -CURRENT
Will appear in 8.0: experimental
Authors: Kip Macy, Doug Rabson
Web: wiki page, mailing list

Xen support has been integrated into FreeBSD, allowing it to be used as a 32-bit guest operating system on recent versions of Xen dom0 (not as a host!). A target for 8.0 is to make FreeBSD ready to be used on Amazon EC2. The project needs testing and sponsorship.

New USB stack

Status: Committed to -CURRENT
Will appear in 8.0: sure
Author: Hans Petter Selasky
Web: announcement, SVN message

The USB stack received a significant overhaul and the new code fixes many standing problems. Some of the new features are full support for split transactions, isochronous transactions, removed dependency on Giant (MPSAFE), a new API and many more. See the SVN message for details.

The new USB stack will use old drivers' and kernel modules' names to increase backward compatibility.

MPSAFE TTY

Status: Committed to -CURRENT
Will appear in 8.0: sure
Author: Ed Schouten
Web: wiki page

The TTY layer is the traditional Unix interface to system users, providing them with interactive sessions to run shells, etc. The current TTY layer in FreeBSD is for the most part the traditional BSD TTY, which is integrated with the drivers and other layers in a way that, though efficient, makes it hard to maintain and extend. The initiative to rewrite the TTY layer aims to make it a true abstraction layer, operating on behalf of both sides of TTY. In addition, it will remove the TTY from the Giant lock, which will eliminate problems with lags and skippy user interface behaviour in the console and X.Org.

Kernel memory limit on AMD64 increased

Status: Committed to -CURRENT, MFCed
Will appear in 8.0: sure
Author: Alan Cox
Web: announcement, SVN commit

Some modern features (of which the most notable currently is ZFS) require a large amount of kernel memory (this has nothing to do with traditional disk caches or the amount of memory visible to the system). Up to now, it was only possible to allocate up to 2 GB for kmem_max, which is becoming a bit cramped. This limit has recently been increased to 512 GB. Together with backpressure improvements for the ARC, this will make the users of ZFS happy.

Kernel threads

Status: Committed to -CURRENT
Will appear in 8.0: sure
Author: Julian Elischer
Web: commit message

Kernel threads upto now were actually "heavy weight" processes running in the kernel address space. This change introduces real light weight kernel threads which consume less low-level resources (process locks, memory maps). It also allows better grouping of threads for display purposes.

procstat(1): A process inspection utility

Status: Committed to -CURRENT
Will appear in 8.0: sure
Author: Robert Watson
Web: announcement

procstat combines functionality from the now-deprecated procfs(4) and adds several new functionalities. Some of the data procstat can provide are: process' command line arguments, file descriptor information, stacks of the kernel threads in the process, security credentials information from the process, thread information and virtual memory mappings. This is utility is mostly useful for debugging.

TextDumps: gathering information after kernel panic

Status: Committed to -CURRENT, MFCed
Will appear in 8.0: sure
Author: Robert Watson
Web: Q&A on textdumps

The usual thing that happens after a kernel panic is a kernel memory dump, either full or (in 7.0 and later) a "minidump". The new "textdump" feature doesn't store the actual kernel memory dump, but extracts commonly needed information from it, stores it into a tar archive of text files, and deletes the dump file. This significantly reduces the size requirements of collecting such information, speeds up development, and enables people to collect debugging information after a crash without kernel developer experience.

ULE 3.0: New version of the SMP-optimized scheduler

Status: Committed to -CURRENT
Will appear in 8.0: sure
Author: Jeff Roberson
Web: commit message, commit message, announcement

Evolution of the ULE scheduler resulted in support for fine-grained CPU affinity calculations, taking into account the physical topology of the CPUs (caches, cores, sockets) and much improved support for binding threads to CPUs. This results in additional functionalities (opens up the possibility of assigning individual CPUs to jails) and noticeable performance improvements.

Superpages

Status: Committed to -CURRENT, MFCed
Will appear in 8.0: sure
Author Alan Cox
Web: research paper, current status

Most general-purpose processors provide support for memory pages of large sizes, called superpages. Superpages enable each entry in the translation lookaside buffer (TLB) to map a large physical memory region into a virtual address space. This dramatically increases TLB coverage, reduces TLB misses, and promises performance improvements for many applications. However, supporting superpages poses several challenges to the operating system, in terms of superpage allocation and promotion tradeoffs, fragmentation control, etc. The performance benefits are substantial, often exceeding 30%; these benefits are sustained even under stressful workload scenarios.

While they can be used on most x86 CPUs, benchmarking has shown that their greatest benefits are visible on quad-core and newer CPUs.

DTrace

Status: Committed to -CURRENT (kernel trace only)
Will appear in 8.0: sure
Author: John Birrell
Web: project web page

DTrace is a tool and a language developed by Sun Microsystems to help debugging and profiling operating systems. It can aggregate information from different parts of kernel (userland tracing is not yet implemented) and analyze them in a ways that's meaningful to the user.

Networking improvements

802.11s D3.03 wireless mesh networking

Status: Committed to -CURRENT
Will appear in 8.0: experimental
Author: Rui Paulo
Web: wiki page

A wireless mesh network, sometimes called WMN, is a wireless network using a mesh topology instead of more typical AP-client topology. These networks are often seen as special type of ad-hoc networks since there's no central node that will break connectivity (in contrast with common wireless networks where there's a central Access Point). 802.11s is an amendment to the 802.11-2007 wireless standard that describes how a mesh network should operate on top of the existing 802.11 MAC.

VirtNet / VIMAGE / Imunes / Network stack virtualization

Status: In final stages of development
Will appear in 8.0: sure
Author: Marko Zec
Web: project web page

The network stack virtualization project aims at extending the FreeBSD kernel to maintain multiple independent instances of networking state. This will allow for complete networking independence between jails on a system, including giving each jail its own firewall, virtual network interfaces, rate limiting, routing tables, and IPSEC configuration.

VIMAGE+Jails will be experimental in 8.0; the system might not work as advertised, especially with regards to security.

Multiple routing tables / FIBs

Status: Committed to 8-CURRENT
Will appear in 8.0: sure
Authors: Julian Elischer
Web: proposal

Support for multiple routing tables (forwarding information bases) allows advanced network topologies. The setfib utility can be used to select routing tables on per-process level.

Equal cost multipath routing

Status: Committed to 8-CURRENT
Will appear in 8.0: sure
Authors: Qing Li
Web: commit message

ECMP routing allows for multiple routes to be handled by the kernel, including default routes. It potentially offers substantial increases in bandwidth by load-balancing traffic over multiple paths.

Zero-copy BPF

Status: Committed to 8-CURRENT
Will appear in 8.0: sure
Authors: Robert Watson, Christian S.J. Peron
Web: BSDCan slides

BPF is Berkeley Packet Filter, facility used to capture raw network packets from the lower layers of the network stack according to user-defined filters and forward them to an application, as well as insert raw packets to the network stack.

This improvement to BPF reduces the number of memory copy operations between the kernel and the application which improves performance in some cases.

Kernel NFS locking support

Status: Committed to 8-CURRENT
Will appear in 8.0: sure
Author: Doug Rabson
Web: commit message, announcement

NFS lock manager in kernel improves performance and behaviour of NFS locking (used to synchronize file access on remote machines). New features include multithreaded operation, deadlock detection, and transparent interaction with local file locks on the server.

NFSv4 support

Status: Under development
Will appear in 8.0: sure
Author: Rick Macklem
Web: call for testing

NFSv4 is a major overhaul of the NFS protocol and brings many new features like a stateful protocol, performance improvements and stronger security (ACLs, strong authentication). Until recently, NFSv4 support in FreeBSD was partial (client-only) and somewhat unstable. New development aims to complete this support.

The introduced NFSv4 infrastructure also replaces the old NFSv2 and NFSv3 servers and clients with the new ones.

Storage subsystems' improvements

Experimental new driver for AHCI

Status: Committed to -CURRENT, experimental
Will appear in 8.0: sure
Author: Alexandar Motin, Scott Long & others
Web: commit message

The new driver, present but not enabled by default in 8.0, supports native AHCI via the CAM (common access method for storage) system. AHCI drives are manipulated by camcontrol and support for new features like NCQ has been integrated.

gvinum 2

Status: committed to -CURRENT
Will appear in 8.0: sure
Author: Ulf Lilleengen
Web: commit message, commit message

gvinum is a logical volume manager based on and compatible with vinum, the FreeBSD's long-standing and practically traditional volume manager. Its features include JBOD, RAID 0, RAID 1 and RAID 5 modes of combining storage devices into higher level volumes, and due to the new version's integration with GEOM it can use and be used by other GEOM devices and classes.

Gvinum 2 is significantly restructured version of gvinum and fixes many long-standing problems. The work done on gvinum makes it more usable and production ready, while maintaining compatibility with older versions. Gvinum exists in parallel with other GEOM classes like gmirror, gstripe and others.

GEOM_PART becomes the default slicer

Status: Committed to -CURRENT
Will appear in 8.0: sure
Author: Marcel Moolenaar & others
Web: commit message

GEOM_PART (gpart) is a new GEOM partition class (slicer) and utility that rolls up support for many partitioning formats (MBR, BSD, GPT etc.) into a single code base.

NOTE: Caveat when upgrading! GEOM_PART might interpret existing partition tables (especially if many operating systems are present - multi boot) differently than the previous classes. Your devices might get renamed.

NOTE: Some old utilities like bsdlabel may not work if the kernel doesn't include GEOM_BSD and other old slicer classes. In other words, bsdlabel et al don't work with GEOM_PART.

Boot support for GPT partitions

Status: Committed to -CURRENT
Will appear in 8.0: sure
Author: John Baldwin
Web: commit message

Support for booting from GPT partitions has been committed to -CURRENT. This support includes the boot sector and loader that enable common i386 machines with a generic BIOS to boot from GPT-partitioned drives.

bsdlabel gets extended to 20 partitions

Status: Committed to -CURRENT
Will appear in 8.0: sure
Author: Marcel Moolenaar
Web: commit message

bsdlabel is (finally!) extended to support more than 8 partitions. The new limit of 20 partitions comes from the number of entries that fit in a single sector.

To make use of this change, GEOM_PART needs to be used instead of GEOM_BSD (this is default in 8.0 but will not work with older kernels). Old utilities like "bsdlabel" will not work with GEOM_PART; the new gpart utility must be used instead.

Security

ProPolice SSP (stack-smashing protection)

Status: Committed to -CURRENT
Will appear in 8.0: sure
Author: Jeremie Le Hen

ProPolice helps prevent exploits that use stack-based buffer overflows by setting a random integer (called the "canary") in the stack right before the return address. It is set in the function's prologue and verified during the epilogue; if it has changed, then a buffer overflow has occured and the program commits suicide by killing itself with SIGABRT (or panic() in case it's the kernel). Both userland and kernel may be protected.

Other changes

The following is a list of smaller and / or more obscure changes that nevertheless deserve a special mention since they will be of interest to certain users:

User-controllable CPU/IRQ binding (jhb)
User-controllable CPU-thread binding with support for CPU sets (jeffr)
RDMA (Remote DMA) support for InfiniBand
ZFS updated to at least zpool format 13 (from 6), with many fixed bugs and resolved problems; bringing ZFS out of the "experimental" status (pjd, kip), partially MFC-ed
Increased the maximum number of groups a user may belong to (NGROUPS) to 1024 by default, in a way that allows further extension if needed. (brooks)
Started upgrade of syscons renderer to support UTF-8 (ed)
Imported new version of OpenBSM (rwatson)
Support for more ARM / Xscale platforms
Imported makefs utility from NetBSD (sam)
Support for SYSVSHM segments larger than 2 GB on AMD64 (kib), MFC-ed
Better support for Machine check exceptions
New network ARP code
SMP-targetted improvements to the routing code
Wireless Virtual AP (aka VAP, aka "virtual WiFi") mode (sam)
tcpdump updated to 4.0 (note: new output format) (rpaulo)
Performance improvements, mostly for SMP scalability
Support for the Intel Nehalem / Core i7 platform (jeffr), MFC-ed
New import of ACPICA (jkim)
Support for QLogic 8Gbit FC HBA-s (+target mode) (mjacob)
VirtualBox 3.x support as VM host, with VT hardware extensions, network bridging, etc. (amd64 and i386)
64-bit (amd64) NVIDIA video drivers
KDE 4.3, GNOME 2.28
Google Chromium

As always, all features described here are, or will be, a part of the FreeBSD "base" system, available in every FreeBSD installation without patching or out-of-the-ordinary configuration.

For more information about development of FreeBSD (among other topics), see my blog with daily and miscellaneous information.

Stream cipher experiments

One fundamental division in shared-key (secret key) cryptography is between stream ciphers and block ciphers. Stream ciphers operate on individual bytes, processing input data (plaintext) as it is available, encrypting them and outputting encrypted bytes (ciphertext). On the other hand, block ciphers always process blocks of data in some predefined size (typically 16 or 8 bytes) at once, and output encrypted blocks. Because there is more data (a block) available to these algorithms, there is more maneuvering space for bit shuffling and mixing, making them somewhat easier to prove safe. This property also allows more flexibility to create different variations of block cipher algorithms. All this has resulted in block ciphers being more studied in recent years and definitely more numerous. In order to apply block ciphers on regular data (which is usually larger then a single block and even so often cannot be divided into regular sized blocks), additional chaining and padding algorithms must be used.

Almost all stream ciphers widely used today are "combiner type" ciphers, which operate by generating a stream of pseudo-random bytes (keystream) which depends exclusively on the given key, and which is combined with input data by using the XOR operation. This method is trivial to decode but introduces a significant weakness: if the same keystream is ever generated more than once (which usually boils down to: if any possible key is used more than once), the cipher is trivially broken at the XOR step. This key reuse is on of the major reasons why WiFi "WEP" encryption is useless.

The complete separation of keystream generation and data encryption (the XOR step) seems a bit inelegant so I have experimented a bit with designing stream ciphers which have these steps integrated. In particular, ciphers which maintain some internal state which is modified by the data being encrypted. Thus, generated keystreams are also plaintext-dependant and a few bytes random bytes (nonce) encrypted (and transmitted) at the beginning of the stream completely destroy any correlation between keystreams generated from the same key. The major practical advantage of this approach over simply using some well-known block cipher like AES is that it eliminates the need for block chaining and padding protocols.

I am not a cryptography expert, just an enthusiast, any none of the described ciphers have been reviewed by experts in the field. I have stated my assesments of some of the algorithms but until somone actually proves they are good (or bad), they should not be considered generally safe.

All algorithms currently experimented on can be downloaded here. The source compiles out of the box on FreeBSD, will probably require trivial or no changes to the Makefile to compile on Linux. Uses OpenSSL.

adlersc	One of the fastest ciphers here (~~35 MB/s) but probably completely unsafe. Uses adler32() function for mixing data, which has too little internal state (32 bits) to be of any use. Also, the function's algebraic properties probably make it unsuitable for this task.
md4sc, md5sc, aessc	These slow stream ciphers use the MD4, MD5 or AES algorithms as hash functions around which data mixing and keystream generation are constructed. The MD4 and MD5 variants are actually slower than the AES variant (2-2.5 MB/s vs 8 MB/s). I believe all these ciphers are Safe (with a capital S), even the MD4/MD5 ones because they are used in a way that doesn't give the potential attacker even one whole hashed block to use in analysis.
rc4vsc	This algorithm uses RC4 as the central algorithm (note: RC4 itself isn't broken in WEP) but modifies it with a key and input data dependant "stream advance" (dry run) steps. This makes it at least as safe as a plain RC4 keystream cipher but for the added security depends very much on the initial nonce (I estimate that using it with decent security requires at least 16 bytes with good entropy to be encrypted and transmitted at the stream start). Performance is merely decent at ~~ 30 MB/s. The algorithm could be made to require a shorter nonce but at the expense of speed (which in the safest case comes down to ~~ 1 MB/s).
rc4v2sc	This algorithm uses RC4 for its basis, modified by permutating the permutation array itself depending on the input plaintext. It is probably safe when used with a good (and not necessarily long - 8 bytes should be enough) nonce, but with mediocre performance of ~~ 11 MB/s.
rijsc	This algorithm resembles the AES variant but with a reduced number of Rijndael rounds (2 rounds). Unfortunately the performance is again mediocre (~~ 13 MB/s) at a probably significant risk caused by severely reducing the base cipher rounds. This variant is probably a dead end.

Performance in the above table is given as an estimate on a 2.4 GHz Core 2 CPU.

It has been fun experimenting with these algorithms and while doing so I got some more ideas to try in the future. I am mostly disappointed by the performance I got from the algorithms - I would expect a good stream cipher to clock at least 50 MB/s on the given CPU, but none of the variants came close to this performance.

Again, these ciphers have not been reviewed! Use at your own risk!

Python XXTEA implementation

This is a pure python implementation of the XXTEA encryption algorithm. It's an unconventional algorithm whose major properties are that it requires very little code to implement and that it can encrypt blocks of arbitrary size without the need for chaining modes.

The interface of the XXTEA class is deliberatily compatible to that of the Blowfish class. Unfortunately, XXTEA is a bit more than twice slower than Blowfish. The implementation has been tested to be compatible with the canonical C implementation.

You can download XXTEA for Python here.

FreeBSD under VMWare

The following tips work both under VMWare ESX (2 & 3) and VMWare Server 1.0 and 2.0. See also information about VMWare Server 2 and FreeBSD. I've successfully run dozens VMWare Virtual Machines with FreeBSD 7 and 8 guests on the free ESXi product, which I recommend due to its better performance over VMWare Server.

1. General tips

Don't use a virtual machine for network-heavy workloads. VMware and other full-hardware virtualization environments (MS Virtual PC, QEmu, etc.) introduce a heavy penalty on I/O, especially network I/O. Expect to be able to get only around 30%-40% out of a gigabit interface (which still amounts to ~~40 MB/s). For example, don't use virtual machines for network routers and similar tasks. Also, don't use them for tasks which require exact timing (e.g. multimedia processing, industrial machine control, etc.). These points are actually valid for any combination of virtualization software and guest OS. For a detailed discussion, see this Slashdot thread on jails vs VMWare.

2. Don't use `lnc`

This tip is no longer current, as lnc doesn't exist in FreeBSD 7. It's still valid for FreeBSD 6.x versions.

While it's the default, lnc driver is the worst network driver for your virtual machine. It's GIANT-locked (meaning it doesn't allow for much parallelizm in the OS), and it's actually deprecated and will be dropped in FreeBSD 7. The replacement for lnc is le and it's present at least in FreeBSD 6.2 and newer, but it's not included in the default GENERIC kernel. Thus, you'll have to configure and compile a custom kernel with device lnc replaced with device le. (Just loading the if_le kernel module won't work because the lnc driver present in the kernel at boot time will detect and use the hardware first.)

There's an undocumented configuration option for the virtual machines that enables VMWare Server to emulate Intel E1000 hardware instead of the AMD Lance. To use it, edit the .vmx file and put ethernet0.virtualDev="e1000" in the appropriate place (anywhere). The simulated device also has TSO support (which is usable in FreeBSD 7, though I don't know what performance can be achieved with the simulated hardware). The em driver is faster and not GIANT-locked so it should give the best performance.

3. Reduce `kern.hz`

Kernel's timer frequency ("HZ") in FreeBSD 6.x and above is set to a relatively large value - 1000 Hz. While beneficial on real hardware, High HZ setting has a negative impact on simulated machine's performance because the VM host software spends too much time handling timer interrupts, which causes context switches, cache flushes and other performance-hindering operations.

You can change the HZ setting by adding a line like kern.hz=50 in /boot/loader.conf. You might also want to use very low values for kern.hz like 10, but test first!

4. Disable internal VMWare swapping

Consider disabling VMWare internal memory swapping and make the virtual machine fit in in the physical memory of the host for best performance. Of course, be informed on the impact of VMWare's memory management before you commit on doing this.

5. VMWare Tools not necessary

It would be nice to have VMWare Tools 100% working on FreeBSD but apparently the company doesn't want to support it properly. Currently, the only features VMWare Tools brings to FreeBSD are GUI enhancers like clipboard sharing and automatic mouse focus grab in X11. VMWare Tools are not needed at all for the following things to work: networking, timer, X.Org GUI.

Networking is handled by the le driver or the em driver. These two will work without any special configuration of FreeBSD. To use the em driver, you might need to modify the VM configuration to include ethernet0.virtualDev = "e1000" or a similar appropriate line. To use the VMWare vmxnet driver (which as far as I can see isn't much different than the le driver), you need to build a kernel without the le driver first.

Timer issues can be lessened (never resolved, even with VMware tools) by reducing kern.hz to something like 50 or 100 Hz (in loader.conf), and installing ntpd.

X.Org can use the generic "vmware" display driver which is included in the default X.Org collection of drivers. Mouse, etc. are also handled generically.

The only remaining functionalities are the ability to "shrink" drives and the ability to soft-shutdown guest machines. For the first one there is a substitute if you're running VMWare Server on a Windows host: third party utilities for VMWare that can do the same thing. Soft shutdown is still best handled by proper VMWare Tools

The emulators/open-vm-tools port builds and works fine in 7.x and 8.x on both i386 and AMD64. This port builds working kernel modules: vmmemctl, vmblock and vmhgfs. It also builds vmxnet, but the network is usually better handled by the em driver. As far as I can tell the Open VMWare Tools (those that are working in FreeBSD) are stable and at least there is no downside to using them.

6. SMP

SMP can be useful at least for certain workloads, at least in recent versions of VMWare products. Virtualized IO is slow and it seems slightly slower with SMP so there are no benefits in enabling SMP for IO-driven workloads (either disk, network or something else). On the other hand it will help with CPU-driven workloads. For example, running make buildworld -j2 on a two-CPU machine will make a farily good use of a real hardware 2CPU system, but when running virtualized, IO wait is so pronounced it takes at least -j3 to avoid noticable idle times.

S e a r c h

Conferences

I was present and presented the following papers (or gave named lectures) at various conferences:

BSDCan 2007 Developers' Summit talk about finstall: ivoras_finstall.pdf (BSDCan)
BSDCan 2007 Developers' Summit talk about gvirstor: ivoras_gvirstor.pdf (BSDCan)
Network Distributed File System in User Space tdfs.pdf (ITI 2006.).
A hierarchical file system interface to database-based content management application ferwebdav.pdf (ITI 2006.).
On recording and presentation of measurement data acquired via web services webservicesdata.pdf (MIPRO 2006)
Integracija Jabber sustava u CMS DORS2006_Jabber.pdf (DORS/CLUC 2006.)
A WebDAV view of FER's Content Management System ivoras_ferweb-webdav2.pdf.gz (CUC 2005)
Distribuding a Web-based Content Management system distribferweb_iti.pdf (ITI 2005)
WebDAV u uredskom okru�enju webdav.pdf (DORS 12, 2005)
FreeBSD 5.2 freebsd52.pdf (DORS 11, 2004)
Kako ubrzati PHP program ubrzatiphp.pdf (DORS X, 2003)
FreeBSD 5 - �iji je Unix dulji freebsdlinux.pdf (CLUC V, 2003)

Except if otherwise specified, these documents are for personal use only. No printing, lecturing or distribuding them is allowed, or any other use for commercial purposes is allowed without written consent from the Author.

Osim ako nije druga�ije nazna�eno, dokumenti se smiju koristiti samo za osobne potrebe. Nije dozvoljeno umna�anje, prezentiranje ili kori�tenje u komercijalne svrhe bez pismene dozvole autora.

Small Projects

Introduction

Small Projects are small utility-like programs/scripts/macros created in the course of daily work or college practice.

Python XMLdict parser	Extremely small and "sane" standalone XML parser library written in Python. It probably has tons of unknown border-cases and will choke on invalid data but it's perfect for parsing simple XML documents.
Visual 'Turing Machine' Emulator	A simple Turing Machine emulator written in Python. The main feature is the simple formatting of input files.
Buffering TCP proxy	A simple TCP proxy used to accelerate web servers (and others with similar usage patterns) in high-load environments
HTTP proxy filter	A simple HTTP proxy filter, made to allow only access from specific unix users (checked via identd) to specific web pages (via regexps). Fast & free.
FreeBSD package discoverer	Python program for discovering and copying FreeBSD packages and their dependancies. Use this to generate specific package lists out of general FreeBSD package mirrors.
Java MIDlet fractal generator	A simple Midlet/MIDP1.0 application that displays the Mandelbrot fractal and allows you to examine it with zoom & pan options. You can also alter colors on the generated image for that extra "wow!" effect :) screenshot
TeXthin OpenType or TeXthin TrueType	European Computer Modern font, converted to TTF/OTF for use on Windows and Unix machines. This is based on TeX fonts ecm2488 and similar, found long ago somewhere on the 'net. There are still some issues with them (e.g. hinting in the TTF versions), but they are usable in the current state.

Summer Of Code

Google's "Summer of code" project is designed to introduce students to the world of open source software development by giving them payed tasks to do during the summer.

I was one of the ~400 winners of the grant in SoC 2005 and my project was to make a journaling layer for storage devices (under the GEOM susbsytem in the kernel) for the FreeBSD operating system.

Update: I'm also participating in SoC 2006, on gvirstor. More info will be given later...

It's fair to say that Google didn't have the slightest idea what they will be dealing with when they started the Summer of Code project. People applied from all over the world, with different customs and tax regulations, and the organization of the whole thing was terrible. For various reasons (most of which are understandable, but should have been forseen) not a single deadline set by Google for themselves was honoured, which includes various notifications and (important for me) payment. It's not what people expect from a large and internationally visible company.

On the other hand, all agreements were eventually honoured and the staff was friendly enough (if overworked), it was a very interesting project and overall even I will say it's been a success.

More on the gjournal project can be found on the FreeBSD's wiki.

Billbija

Tekst je nastao jednog sumornog dana na DMIHu u �akovcu ('98?) na 386 laptopu (bez baterije) i WordPerfectu 5.1 (za DOS, naravno), i otuda se, ne znamo ni mi kako, pro�irio po HR webu kao vatra (naravno, bez ispravnih creditsa, kako je i obi�aj u ovim krajevima).

Autori Billbije su: Neven Mrgan (neven at mrgan.com, sada u Americi), Jurica Lovakovi� (jurica at gmx.net) i Ivan Voras (ivoras at gmail.com)

Billbija

Originalni WordPerfect 5.1 dokument sa Billbijom, napisan daleke 1998.

Billbija

U pocetku bijase Word1.0.(rijec)
I Word bijase u Billa.
I Word bijase Bill.
I Word postade 2.0.
I prebivase medju korisnicima.

 Evo rodoslovlja Windowsa, Boga korisnickoga. 
UNIXu se rodi DOS 1.0, DOSu 1.0 DOS 2.0, DOSu 2.0 DOS 2.1. DOSu 2.1
DOS 3.0, DOS 3.1, DOS 3.2, DOS 3.3 i klonovi njegovi, koji ne
bijahu u Billa. DOSu 3.3 rodi se DOS 4.0 i DOS 4.01, a njemu se
rodi DOS 5, blazen u dane svoje.
 Ukupno od UNIXa 1.0 do DOSa 5, 15 koljena.
A DOSu 5.0 rodi se DOS 6.0 i DOS 6.1, DOSu 6.0 DOS 6.22 i njegovi
sinovi.
 Ukupno od DOSa 5.0 do DOSa 6.22, 5 koljena.
 A Windowsima 3.0 rodise se Windowsi 3.1, Windowsima 3.1 Windowsi
3.11, a Windowsima 3.11 Windowsi 95.
 U to vrijeme bijahu u pustinji Windowsi 3.11, a narod govorase da
su oni prorok Billov. A on im rece:"Ja sam glas koji vice iz
kucista: pripravite diskove svoje za onoga koji dolazi."
 U Redmondu zivljase Intel i zarucnica mu Microsoft. I jedne veceri
Microsoftu se ukaza andjeo Billov i rece: "Zdravo Microsofte,
programera pun, Bill s tobom. Blagoslovljen ti medju programerskim
kucama i plod programera tvojih Windows 95." A Microsoft mu
odgovori:"Evo sluzbenika Billovih, neka nam bude po defaultu
tvome."
 U ono vrijeme naredi kralj IBM da se izvrsi popis OSova
kraljevstva njegova. I Microsoft ode u Chicago. I Microsoft bijase
isao od distributera do ditributera, jer Microsoftu bijase vrijeme
uskoro da rodi Windows 95. I ne nadjose smjestaja, osim u jednom
uredu. I Microsoft u uredu, medju cinovnicima i automatima za
sokove, iskompajlira Windows 95.
 A u to vrijeme tri kralja s Istoka bijahu tuda prolazili i
ugledase na nebu reklamu za Windows 95, te se uputise prema uredu.
imena tim trima kraljevima bijahu MacOS, UNIX i CP/M. I donijese
darove: graficko sucelje, multitasking i jednostavnost. Kralj IBM
sazna za toga OSa, te naredi da se pobrisu svi OSovi mladji od
godinu dana. Stoga Microsoft pobjeze od IBMa, jer mu Bill to
navjesti u snu.I brojni OSovi tada igbubise zivot.
 A dodje i vrijeme beta testiranja Windowsa 95, a u to vrijeme na 
Windowsima 3.11 se isprobavase programi. I dodje Windows 95 kod
Windowsa 3.11, a 3.11 rece:"Ja nisam dostojan da ti budem
kompatibilan, a kamoli da s tobom koegzistiram na kompjuteru." Ali
Windows 95 mu je rekao:"Tako mora biti da se ispuni sto je po
reklamama o kompatibilnosti navjesteno." I e�mail stize sa nebesa i
rece:"ovo je OS moj ljubljeni, u njemu mi je sva milina."
 Bijase tada svadba na COMDEXu, a gazda je stalno zatvarao programe
prije pokretanja novih. Pristupi mu Steve Jobs i rece:"zasto
zatvaras programe, koristi multitasking." I vidje Windows 95 da
vise nema GDI resourcesa, pa stvori novi heap za svaki program
posebno. Tada Windows 95 pokrenu i 50 programa odjednom, a staro je
sucelje pretvorio u novo. I  korisnici primijetise da svi 16bitni
programi imaju novo sucelje, i govorase:on je poslan od Billa.
 Pristupi Windowsima jedan covjek i rece:"Moj OLE2.0 bolestan lezi,
a znam da mu ti mozes pomoci." Win95 rece:"pokazi mi kompjuter
svoj." No covjek mu odgovori:"Ja nisam dostojan da ti unidjes pod
moje kuciste, nego samo reci rijec i ozdravit ce OLE moj." A win95
rece:"To je prava vjera, u ovoga covjeka! Idi kuci, ftp�om cu ti
poslati service pack!."
 Jednom je prolazio win95 kroz ekran i vidje da nema mjesta za sve
njegove podanike. Stoga ih okupi, a sa neba se zacuje 
The Microsoft Sound.wav i promijeni se rezolucija bez restartanja 
windowsa. A u narodu govorase:"Pa taj je mocniji i od OS/2!"
 U to vrijeme bijase skupina korisnika koji su koristili
Netscape Navigator.
 Jednoga dana Win95 ljutit upade medju
njih i rece:" U sto ste pretvorili hram oca moga?! U gnijezdo
razvratno! Ja mogu za tri milisekunde srusiti program koji su
Netscapeovci pisali godinama i napisati svoj browser!" I uze
registry editor i pobrisa key�eve Netscapeove.
 A jednoga dana krenuse win95 i aplikacije njegove loviti
dokumente. Tada se podize velik zahtjev za memorijom i aplikacije
upitase win95:"sto da radimo, ucitelju?" A on se ustade, izadje iz
memorije i stade hodati po hard disku. Aplikacije gledase u njega u
cudu kako hoda po disku kao po memoriji. On im rece:"Ta zasto se
bojite? Pridjite disku!" Word97 izadje iz memorije i stade na disk,
ali mu se vjera pokoleba i on poce zauzimati memoriju. A win95
uhvati ga za handle i kaza mu:"Zasto ti slabi vjera, ti nevjerni
softveru?"
 Tada win95 odluci otici u Redmond, te rece svojim sollution
providerima:" Idite u grad i nadjite programera da kreira cool
sucelje za Iexplorera." I sollution provideri to ucinise. Tako
win95 dodje u Redmond, dojahavsi na ISDN vezi. Narod se okupio i
mahao pointerima IRCajuci:"Blagoslovljen onaj koji dolazi u ime
Billovo! Hosana sinu DOSovu! Hosana u visokoj memoriji!" A
konkurenti im govorase:"zasto slavite ovaj OS?" Korisnici
odgovorise:"Kad mi ne bi IRCali, screen saveri bi to ispisivali!"
 Te veceri win95 pozva sve aplikacije i rece im:"Ovo je cool
toolbar, znak saveza medju svim buducim win aplikacijama." I pozove
office assistanta i rece:"Ovo je Office assistant moj, koji se za
vas brine i rad vam olaksava. Njih koristite meni na spomen."
 Takodjer rece aplikacijama:"Jedan od vas ce me veceras izdati." A
Word97 upita:"Zar ja, OSu moj?" win95 rece:"Pa, i ti ponekad, ali
ne veceras. Prvi koji veceras bude licencirao tudju tehnologiju bit
ce taj." A IExplore1.0 ode kod NCSAa i licencira Mosaic.
 Tada win95 pozove sollution providere i rece im:"Idite i sirite
legalni softver svijetom. Kojima otpustite grijehe, otpusteno im
je. Kojima naplatite kaznu, naplaceno im je."
 Kasnije odu na CeBit. Tamo se aplikacije minimiziraju, a win95 im
kaze:"zar niste mogli ni sat vremena ostati u foregroundu?" A ovako
win95 govorase Billu te veceri:"Nek me mimoidje ovaj gorki task,
ali ako je tako treba biti, Bille, neka se vrsi volja tvoja." I
dodje IExplore1.0 sa Netscapeom, Sunom, i drugima te pozove svoj
about box. Tada ovi znadose da je to OS po kojeg su dosli. Tada
aplikacije zauzese memoriju ali im win95 rece:"zar mislis da ja ne
bih mogao sad reci svom ocu da pokrene 1000 ActiveX kontrola i
otjera njih?" IExplore3.0 na to rece:"Nikad te necu ostaviti!" A
win95 odgovori:"Jos ove veceri, prije nego Scheduler pokrene
defrag,  tri ces me puta zanijekati." I odvedose win95. 
 A IExplore3.0 slijedio ga je na putu, te se tri puta izda:za
MacOS, UNIX i Win3.11. Tada Scheduler pokrenu defrag i IExplore,
vidjevsi to, ugasi sve svoje toolbare od zalosti.
 win95 dovedose kod antitrustovskog suda. Tamo ga sud upita:"Jesi
li ti kralj OSova?" win 95 odgovori:"Ti kaza." Tada sud rece:"Perem
ruke od ovoga OSa, moje krivice na njegovom shutdownu nema." Tada
ga izvede pred korisnike i rece:"Evo OSa!" win95 osudise na smrt
brisanjem windows direktorija. Tako odvedose kompjuter sa win95 u
'Golgota d.o.o.' servis da obrisu direktorij. Tamo pored njgea
stajase jos dva kompjutera, lijevo sa OS/2 a desno sa MacOSom. NA
pozadinu desktopa win95 napisase WINRI. Neki sluzbenici kockase se
za njegov OLE. Tada obrisase direktorij.
 Nakon tri dana Bill preko Interneta reinstalira win95, te on
izadje na svjetlo monitora.
 Kasnije se win95 ukaza aplikacijama, ali nevjerni CorelDraw
rece:"ne vjerujem dok ne vidim" Tada polozi ogromnu sliku na
clipboard i uvidje da win95 uistinu uskrsnu.

=== BILLE NAS ===
Bille nas, 
koji jesi u Redmondu,
sveti se poduzece tvoje,
dodji kraljevstvo Tvoje,
budi OS tvoj,
kako na PCu tako i na Macu,
Patch nas svagdasnji daj nam danas,
i otpusti nama tipflere nase
kako i mi otpustamo Bugovima Tvojim,
i ne uvedi nas u Prompt
nego izbavi nas od DOSa.

=== KORISNICKO VJEROVANJE ===
 Vjerujem u jednoga Wina, OSa svemogucega, stvoritelja Taskbara i
Desktopa, svega vidljivoga i nevidljivoga, i u Explorera, sina
njegovoga, svjetlo od svjetla, pravoga OSa od staroga DOSa,
kompajlirana, nestvorena, istobitna s DOSem, po kojem je sve
stvoreno, koji je instaliran radi nas i radi nasega spasenja, koji
je rodjen od djevice Microsofta, mucen pod UNIXom, raspet, umro i
pokopan, u trecoj verziji uskrsnuo od mrtvih, i opet ce doci u
slavi suditi online i offline i njegovoj vezi nece biti kraja. I u
Kernela Svetoga, svetu Mrezu Microsoftovu, opcinstvo svetih,
registraciju programa, uskrsnuce Registrya i GUI vjecni, press any
key to continue.


=== Sedam svetih sakramenata ===
1.Instaliranje win95(krstenje)
2.Pokretanje win95(pricest)
3.Instaliranje Office97( potvrda (krizma) lojalnosti)
4.Registracija softvera(ispovijed)
5.Spajanje na Microsoftov site(zenidba)
6.Unaprijedjenje u Microsoft Qualified Ddealera(sveti red)
7.Shutdown(poslednja pomast)

=== Microsoftove dogme === 
* Windows 95 nastali su bezgrjesnim kompajliranjem (debuggirano)
* Postoji jedinstvo Svetoga Trojstva Billa, Windowsa 95 i Explorera
* Na Sudnji dan Bill ce pozvati k sebi korisnike, s desne strane
sjedit ce oni sto koristise Office, iexplorer, Windows95, a s
lijeve UNIXasi, Netscapovci i Lotusovci i ostala hereza.

U pripremi:
* Kako je Jona proboravio tri dana u utrobi Java Development Kita

Tekst je prenesen izravno iz WordPerfect dokumenta, sa manjim kozmetickim zahvatima. Svi tipfeleri datiraju iz '98. :)

Miscellaneous things

Underscore problem in Croatian layout in X11

...can be solved by adding the following line to /usr/X11R6/lib/X11/xkb/symbols/pc/hr :

    key <AB10> { [ minus, underscore, dead_belowdot, dead_abovedot ] };

Also useful is to reset the behaviour of tilde key:

    key <TLDE>  { [     grave, asciitilde,      notsign,      notsign ] };

Angle brackets (< and >) problem in syscons Croatian ISO keymap

... can be fixed with this patch:

--- hr.iso.kbd.old	Fri Oct 20 19:55:22 2006
+++ hr.iso.kbd	Fri Oct 20 19:59:41 2006
@@ -56,8 +56,8 @@
   048   'b'    'B'    stx    stx    '{'    '{'    stx    stx     C
   049   'n'    'N'    so     so     '}'    '}'    so     so      C
   050   'm'    'M'    cr     cr     167    167    cr     cr      C
-  051   ','    ';'    nop    nop    ','    ';'    nop    nop     O
-  052   '.'    ':'    nop    nop    '.'    ':'    nop    nop     O
+  051   ','    ';'    nop    nop    '<'    ';'    '<'    nop     O
+  052   '.'    ':'    nop    nop    '>'    ':'    '>'    nop     O
   053   '-'    '_'    nop    nop    '-'    '_'    nop    nop     O
   054   rshift rshift rshift rshift rshift rshift rshift rshift  O
   055   '*'    '*'    '*'    '*'    '*'    '*'    '*'    '*'     O

Introduction

My Papers

Here are some papers & other useful articles I wrote. Some of it is made for various curriculums at the college, some for various journals, blogs and magazines and some just for fun.

Note that I'm using the term "papers" loosly, and except where otherwise noted, they are not academic or scientific papers, but rather "articles," "white papers," or just "stuff I wrote."

The main list of papers is here.

Papers

Academic/scientific papers

Title	Abstract	Event
A High Performance Memory Database for Web Application Caches	This paper presents the architecture and characteristics of a memory database intended to be used as a cache engine for web applications. Primary goals of this database are speed and efficiency while running on SMP systems with several CPU cores (four and more). A secondary goal is the support for simple metadata structures associated with cached data that can aid in efficient use of the cache. Due to these goals, some data structures and algorithms normally associated with this field of computing needed to be adapted to the new environment.	MELECON 2008
Web-enabling Cache Daemon for Complex Data	One of the most common basic techniques for improving the performance of web applications is caching frequently accessed data in fast data stores, colloquially known as cache daemons. In this paper we present a cache daemon suitable for storing complex data while maintaining fine-grained control over data storage, retrieval and expiry. Data manipulation in this cache daemon is performed via standard SQL statements so we call it SQLcached. It is a practical, usable solution already implemented in several large web sites.	ITI 2008
A distributed system for data storage and retrieval (in Croatian)	This work studies technologies applied in building distributed peer-to-peer network systems for data storage and retrieval. These systems consist of arbitrary number of network nodes connected on a higher layer and their connectivity is not influenced by physical topologies of networks of which they are a part of - they form "overlay networks." All nodes in a peer-to-peer overlay network have equal functionality (there are no "server nodes"). This equality of nodes presents some unique problems: mutual discovery of active nodes and varied connectivity as nodes join and depart the network. Various solutions for these problems are presented and evaluated and an architecture for a peer-to-peer system is proposed. The proposed system uses TCP/IP protocols for inter-node communication and allows storage and retrieval of data records in the form of dictionaries (mappings). A prototype of the system is implemented in Java programming language, and technical and user documentation is provided in this work.	dipl.ing. thesis
Network Distributed File System in User Space	File systems have traditionally been implemented in the operating system's kernel to ensure maximum possible speed and integration with the rest of the operating system, and this was true even for network file systems such as NFS. However, available CPU power on mainstream architectures continues to increase daily at a rate which is not closely followed by speed of computer network equipment. When considering development of network-distributed file systems today it becomes clear that speed improvements offered by pure kernel-side implementations are no longer significant given the bandwidth and latencies of computer networks. Recent efforts in enabling user-space file system implementations on free / open source Unix-like operating systems have made it possible to create a solution for distributing file system data over computer networks entirely in user-space. In this work we present such a solution - the Trivially Distributed File System.	ITI 2006
A hierarchical file system interface to database-based content management application	When considering the usability of computer applications one, of the most important factors is the interface they provide for data manipulation. Historically, though each application has defined it's own user interface, some types of interfaces have emerged as most applicable for certain types of data. Over time, some applications (or types of applications) have gained enough popularity that they became ubiquitous and well-known to most computer users to the extent that users consider them the norm, and as such, optimal for their daily tasks. In this paper we present an idea and implemented method of exposing data from a web content management system in the form of hierarchical file system, manageable and editable by usual file management and office application tools.	ITI 2006
On recording and presentation of measurement data acquired via web services	Web services have greatly eased exchange of data, but the usage of web services carries its own specific downsides as well as advantages. Most notable of these are latency of transmission and reliability of server equipment. In this work a straightforward solution for distribution and representation of measurement data was created. This solution is comprised of two parts: a back end process which communicates with the data source via a web service protocol based on XML (in Python language), and a front-end module for the FERweb Content management system (in PHP language). These processes communicate asynchronously via the database.	MIPRO 2006
Distributing a Web-based Content Management System - "FERweb"	This work explores various ways of distributing the FERweb CMS system (a web-based Content Management System of the University of Zagreb, Faculty of Electrical engineering and computing) Primary aim of this project is enhancing the performance of the system, within the constraint that the efforts must be based on existing technologies used in the project and with minimal impact on the existing code.	ITI 2005

General papers

Title	Date	Abstract
On implementation of `switch()..case` in gcc	January 2 2005	A simple exploration of how the `switch()..case` construct is compiled into machine code in gcc 3.4
Distribuding Web-based CMS	June 24, 2005	This is the "official" version of the following paper, presented at the ITI 2005 conference.
Web CMS Distribution	May 7, 2004	This work explores various ways of making the FERweb CMS system (a web-based Content Management System of the Faculty of electrical engineering and computing at the University of Zagreb) a "distributed system". Here, the notion of a distributed system is taken quite literally, and stands for executing as many components of the system as possible on as many different computer systems. The only limits here imposed are that of the practical value of such distribution . its end result must be either an increase of total performance delivered to the user of the system, an increase of overall system reliability, or providing distinct new functionality that was previously not possible, and that the efforts must be based on existing technologies used in the project already, with minimal impact on the existing code. Where appropriate and where available resources permit, measurements are taken and reported.
OS Benchmark	April 10, 2004	This article presents the results of benchmarking various operating systems. The benchmarks include synthetic tests (bytebench, ubench, bonnie++) and, more importantly, semi-"real world" tests (pgbench, web CMS). Operating systems benchmarked are various versions of FreeBSD, DragonflyBSD, NetBSD and Linux.
GCC as Optimising Compiler	Feb 17, 2004	This paper aims to present and analyse some of the optimisations that are conducted by the GNU C compiler, in version 3.3.4 (on a i386 FreeBSD system). This is only an informal and introductory paper and is not meant to be comprehensive nor complete, only to give insight at things that are happening ``behind the scenes''.
Description of PHP-RPC protocol	Jan 15, 2004	Because of the need for a truly light-weight RPC mechanism in PHP, a new protocol is formed: PHP-RPC, to be used instead of XML-RPC when communicating to purely PHP applications. This protocol is faster and has a smaller overhead.

Curriculum papers (mostly in Croatian)

Title	Date	Abstract
RDRAM Memorija	Feb 8, 2004	Ovaj rad opisuje osnovni rad Direct Rambus RAM sustava, kakav se jo� uvijek koristi. Na kraju rada je dan kratak opis promjena u novijim ina�icama sustava.
KSE sustav u FreeBSD 5.0	Dec 2002	Cilj ovog rada je prikazati mogu�nosti postizanja vi�edretvenog rada u operacijskom sustavu FreeBSD 5.0, sa posebnim osvrtom na novorazvijeni sustav KSE: Kernel Scheduled Entities koji donosi mnoga pobolj�anja u odnosu na prethodno dostupna rje�enja, posebno u podr�ci za iskori�tavanje vi�e sistemskih procesora (ukoliko su prisutni) i konkurentnog izvr�avanja poziva jezgre operacijskog sustava. Prakti�ni dio rada demonstrira kori�tenje razli�itih postoje�ih metoda postizanja vi�edretvenosti u aplikacijama, te njihove posljedice za aplikaciju i cijeli sustav.

Other (conference presentations, etc.)

Title	Date	Abstract
FreeBSD's finstall	May, 2008	Presentation of finstall's status, given at BSDCan 2008.
FreeBSD's finstall	May, 2007	Presentation of new FreeBSD graphical installer, given at BSDCan 2007.
FreeBSD's GVIRSTOR	May, 2007	Presentation of GEOM_VIRSTOR kernel module for FreeBSD, givent at BSDCan 2007.
WebDAV u uredskom okru�enju	April, 2005	Kori�tenje WebDAV-a u poslovnom okru�enju, primjene, poslu�itelji, klijenti.
FreeBSD 5.2	April, 2004	FreeBSD arhitektura i okru�enje, novosti u 5.2, napredne mogu�nosti uporabe
Kako ubrzati PHP program	Mar, 2003	PHP je spor! Zbog prili�no "labave" sintakse, PHP interpreter mora prolaziti kroz puno vi�e provjera nego za neke druge skriptne jezike. PHP-ov "model programiranja" je �esto neoptimalan s resursima...
Uvod u FreeBSD	Mar, 2003	�iji Unix je dulji? Kratka povijest vremena, BSDL vs GPL, BSD is dead! *BSD, Prednosti FreeBSDa...

There's more!

There's a separate section containing my FreeBSD papers.

FreeBSD papers

Mono Performance

Mono

Recently, the Mono project has released the 1.0 version of its framework. I followed the development closely, because the concept of running .net application on unices is very interesting. You can say what you will about MS and its business policy, but .net framework is really elegant solution, and C# is a nice and clean language (IMO, much cleaner than Java), and it shows it's been designed by a Pascal/Delphi lover :).

As soon as I got the chance, I installed mono on my desktop machine, both on FreeBSD and WinXP (in dual-boot), and started playing with it. The impressions are very good for starts, and it seems to deliver what's promised. The only disappointment is that Windows.Forms support is practically unusable (so no cross-platform GUI applications yet, except for Gtk#), but it should be addressed in Mono 1.2. Except for that, as far as I can tell, the stuff really works.

Mono can be run on FreeBSD (ports/lang/mono) only on a recent -CURRENT system (mine is from 2004-07-10; it won't work on 5.2[.1]-RELEASE), and even now, there are some threading problems: one is detected by the ./configure script (abnormality with a certain garbage collector), and one I stumbled on myself: monodoc cannot be compiled (it seems that the C# compiler blows up on a bug in libpthread). Gtk# (ports/x11-toolkits/gtk-sharp) installs cleanly and most of the demos work (there is an occasional Null pointer reference, but this seems to be because of demo programming error, rather than interpreter fault).

As is the custom, I started writing some hello world programs, and, since I like my solutions to be of maximum available performance, done a simple speed benchmark. The benchmark consists of this C# code (it intentionally mixes floating-point and integer code):

using System;

class Bench {
    public static void Main() {
        double d = 0;
        for (int i = 0; i < 1000000000; i++) {
            d = d + i;
        }
        Console.WriteLine(d);
    }
}

I've run this script on Win32 using MS .NET and Mono runtimes, and on FreeBSD with Mono runtime. Also, I've written equivalent C and Java programs, for comparison. Here are the results (less is better):

Platform	Time/sec.
Win32 MS .NET	2.8
Win32 Mono	15.1
Win32 gcc -O3	3.2
Win32 java -client	9.4
Win32 java -server	11.7
FreeBSD Mono	7.9
FreeBSD gcc -O3	2.7
FreeBSD java -client	9.7
FreeBSD java -server	11.7

Benchmark code & environment (for win32) can be downloaded here.

This is only a simple first-hand comparison, just to get a feel for the performance of various platforms. Gcc on FreeBSD was 3.3, while on Win32 it's 3.2 (MinGW), so this might account for the variation in performace. Other than that, it's surprising to see interpreted (well, JIT compiled) .net code outperform gcc-compiled C code on win32! Although Mono uses JIT (at least reports it does when used with --stats), it's about 5 times slower than the one in MS .net. I used the same executable with Win32 MS .NET, Win32 Mono, and FreeBSD Mono cases, and I don't know how it managed so much better performance on FreeBSD (performance on Linux should be similar, it's probably because of native unix environment). The JIT is used in all .net cases presented above: I tried timing equivalent programs written for purely interpreted platforms: Python, Ruby and Perl (yes, all three...), and they took about half an hour on average to execute.

Startup time was included for all cases (a few test runs of each were made to preload the libraries into the OS cache), so this could be the reason why java (especially with -server) performs so poorly (1.4.2 VM). But then again, .net has similar sized libraries...

This, of couse, is an overly simplistic look on the platforms involved. A more seriuis benchmark would have to involve at least string operations, and I'll do it if find the time. Until then, I can only say I like what I see here!

Network mounting

Today I did something I thought wouldn't work - but it did! I'm running out of space on my FreeBSD partition (the machine dual-boots WinXP), so I thought of a clever way to solve this problem: I booted the FreeBSD partition in vmware in WinXP, created a big empty file, shared the directory the file was in via Network neigbourhood, mounted it in FreeBSD virtual machine (smbfs), and then mounted that big file as a UFS partition! And it worked! Since I'm running unstable (-CURRENT) FreeBSD system, I really though something would crash or misbehave, but no - it went through flawlessly. I just LOVE modern technology! (yes, I can use that image-file when I boot FreeBSD natively, mounting the (msdosfs) partition and proceeding like before, only without the networking part) - And did I mention I was doing all this from a X11 desktop which I'm accessing through a VNC client? :)

Speaking of all this, I often stand in awe about things like that I'm doing like it's perfectly normal and understandable... but look what is involved in such a deceptively simple task:

Operating systems with more power and flexibility at a command of ordinary users than ever in history
VM emulation technology, running on machines that are VERY unsuitable for such things (x86)
Sufficiently powerful hardware, that allows me to run several virtual OS images on my desktop (and my machine is actually quite slow by today's standards...)
Network interchange protocols, that allow me to connect completely disparate systems and work on all of them at the same time, without their differences slowing me down
User interface systems and protocols (meaning command-line as well as GUI), which make all this seamingly easy (several years ago, I'd probably needed to hack kernel to do something like this)

This probably isn't very exciting to a majority of users, but I find it fascinating, even without the virtual machine emulation part.

Also, together with the Mono project, this is really a great example of technology converging to a common point, usability-wise. It may not happen in the next 10 or even 20 years, but in 50... sure.

Projects

This section of the site is about various not-so-small projects I'm making from time to time. Mostly, they are tech demos or learning projects, but have a broader scope than the "Small Projects" page in the main section.

Some of my larger projects are hosted on SourceForge.net:

SQLCached In-memory SQL database for use as a flexible memory cache (C)
PanDAV Python WebDAV server & server library (Python)
TDFS Trivially distributed file system (C)
Velvet System monitor and logger daemon (Python)
TempusWiki A simple Wiki engine (PHP)
Isaac3 Extremely easy to set up image gallery (PHP, Java)
MDCacheD Cache daemon similar to memcached, but supporting SMP and multiple data domains (C)
GGtrace Graphical (X11) I/O request tracer for FreeBSD (C, C++)

See my FreeBSD page for more projects.

New blog

I've created my "official" blog for the time being, where I hope I'll put all the things I write so it's available in one place.

The address is http://ivoras.sharanet.org/blog.

JPHP

JPHP is, simply put, PHP embedded in the Java VM. The main components of this project are JPHP Java class and libjphp.so Java JNI interface library that makes the actual calls to the PHP VM. A simple example for JPHP is:

    JPHP jp = new JPHP();
    jp.simple_exec("echo \"Hello from PHP from Java\\n\";");

The project was intended to be used in a hybrid Java-PHP web application server, but the project was abandoned and thus only basic library and functionality exist. I'm usually available for subcontracting for realizing the full project.

(I chose the name "JPHP" pretty arbitrary - if anyone from the other projects named "JPHP" has objections to it, let me know and I'll change it.)

JPHP	JPHP Library and example.

PanDAV

NOTE: An updated version of PanDAV is available at SourceForge.net here.

PanDAV is a modular WebDAV server written from scratch in Python 2.4. I made it because I wasn't happy with other available software of this type, and to fit certain special requirements:

Highly modular and clean design. Nowadays, writing a WebDAV server that serves static filesystem data is both outdated and unnecessary. The main application of WebDAV and similar technologies is accessing dynamic data; in other words, to serve as a translation layer between database and user software. Thus, initial modularity of the software is of great importance. (Though the source publicly available here only does filesystem access, that's only for demonstration purposes).
Extendability. This actually goes hand-in-hand with previous point, but it deserves a special mention. It's easy to create e.g. custom properties on objects and access them via WebDAV. Each node in the simulated WebDAV "file system" can be of a different type - ordinary file, database object, anything.
Performance. Performance was another major concern in writing PanDAV. Most of the other Python implementation tend to be simplistic and written with stock components without thought for speed. PanDAV features multi-threaded request handling and custom-made XML parsing. Unfortunately, no real performance measurements exist yet :)

You can get a free version of PanDAV here. It's limited to traversing directories, downloading and uploading files and it's been tested with Microsoft Windows XP Network Places client and DAVExplorer Java client.

This work is licensed under a Creative Commons License.

mdcached

mdcached stands for "multi-domain cache daemon" and in its idea is very similar to memcached - by default it even implements the same network protocol. The implementation is made from scratch and has practically no connection to the memcached codebase.

Things that makes it significantly different than memcached are:

It implements multiple "domains" - virtual lists that can hold arbitrary (under the restrictions of the protocol) keys and values. There can be arbitrary number of domains as well as keys in the domains (for values of "arbitrary number" <= 2^32-1). Multiple key-value pairs with the same key can exist, as long as they're in different domains.
The daemon is multithreaded and can thusly take advantage of multiple CPUs in the system (SMP-friendly).

In the default mode, mdcached daemon emulates memcached network protocol. Clients that know they're communicating with mdcached can switch between the original protocol and the "domains" protocol as they see fit (though the "domains" protocol is superset of the original). Speed in general is comparable to memcached, though inserts are somewhat slower. mdcached is written in C, and has been developed & tested on FreeBSD 5.4 but should have no problems running on Linux. If you do encounter incompatibilities, please send patches :)

If anyone's using this, either in testing or productions, please notify me, just so I can keep track of its usage.

You can download it here.

GenWeb.py

This site is maintained by a little Python script I wrote about a year ago. Its only purpose in life is to take a directory tree with templates, spec files and HTML files and apply them mutually to generate a directory tree of HTML documents.

I have no documentation on it at this time, but if anyone's interested, here it is. It's designed to be trivial for usage.

The archive contains a template, a spec file and a HTML page (all from this site) so hopefully interested people will grok its function.

mkraw.py

This is a small utility I wrote to share virtual disks between QEmu and VMWare. It creates a VMWare-compatible .raw files for specified raw disk image files or devices.

You can use it to directly boot raw disk images created with dd or a similar device imager in VMWare. It's been tested in VMWare Server 1.0 but should work in VMWare Workstation 4, or possibly earlier version.

An example scenario is:

Create a disk image with dd or similar utility (hint: try truncate).
You can use/boot the raw disk image in qemu as usual (-hda image_file) ...or...
Use mkraw.py to create .raw file for use with VMWare. Thus you can use the same image in qemu and VMWare (not at the same time, of course)

To use it, create a disk file and run mkraw.py on it. This will create a .raw file for the disk image which you can specify while adding a disk to a virtual machine in VMWare (select "use an existing disk" option, then search for .raw files).

You can download it here.

What's cooking for FreeBSD 7?

The next major release of FreeBSD, version 7, is one of the most significant so far, with amount of new technologies and improvement largest since the introduction of 5.0. Since constantly searching the mailing lists for important changes can be a bit tedious, I've created this (frequently updated) page to list some of the more interesting new things in one place.

FreeBSD 7.0 has been released! I've now started the continuation of this project: What's cooking for FreeBSD 8.

Also useful are the quarterly Status Reports:

If you're interested in how FreeBSD gets developed, you're encouraged to read the mailing lists and developer blogs.

Network stack improvements and cleanup

Even though this document mentions only several people, the effort to improve the network stack and its performance has been carried by many.

New `sendfile()` implementation, improved `sosend()`

Status: Committed to -CURRENT
Will appear in 7.0: sure
Author: Andre Oppermann, Robert Watson
Homepage: http://people.freebsd.org/~andre/, announcement message

While working on TSO support, Andre Oppermann has found several ways to optimize kernel's internal networking support. The new sendfile() implementation sends larger chunks of data at once and improves performance up to 5x when used with TSO and other new enhancements. Improvements to the sosend() and related functions resulted in lowering the CPU consumption of sending side of network interfaces almost three times. Note that these are microbenchmarks and performance improvements in real usage still needs to be quantified.

TSO and LRO support

Status: Committed or ready for -CURRENT
Will appear in 7.0: sure
Author: Andre Oppermann and Andrew Gallatin
Homepage: http://people.freebsd.org/~andre/

The ongoing effort to improve FreeBSD's network performance (especially after the hit taken during transition to SMP) has resulted in the new ability to support TSO (TCP/IP segmentation offload) and LRO (Large Receive Offload) hardware on gigabit and faster cards. Some of the drivers that support TSO include: em, bc, cxgb, ixgbe, msk, mxge, nxge, nfe, re (or in plain words: Intel, Broadcom, NVidia, Realtek and other cards, gigabit or better). LRO support is currently in mxge.

TCP socket buffers auto-sizing

Status: Partially committed to -CURRENT
Will appear in 7.0: sure
Author: Andre Oppermann
Homepage: http://people.freebsd.org/~andre/

FreeBSD has a default 32K send socket buffer. This supports a maximal transfer rate of only slightly more than 2Mbit/s on a 100ms RTT trans- continental link. Or at 200ms just above 1Mbit/s. With TCP send buffer auto scaling and the default values below it supports 20Mbit/s at 100ms and 10Mbit/s at 200ms. Both read and write buffer are auto-sized.

While the support for send buffers auto sizing is committed, patches for receiving side are still under testing.

Rapid Spanning Tree Protocol (802.1w)

Status: Committed to -CURRENT
Will appear in 7.0: sure
Author: Andrew Thompson
Homepage: http://people.freebsd.org/~thompsa/

RSTP provides faster spanning tree convergence. The protocol will exchange information with neighboring switches to quickly transition to forwarding without creating loops. The code will default to RSTP mode but will downgrade any port connected to a legacy STP network so is fully backward compatible.

SCTP (Stream Control Transmission Protocol)

Status: Committed to -CURRENT
Will appear in 7.0: sure
Authors: Randall Stewart, George Neville-Neil
Homepage: http://www.sctp.org/

FreeBSD is the reference implementation for the SCTP.

Like TCP, SCTP provides a reliable transport service, ensuring that data is transported across the network without error and in sequence. Like TCP, SCTP is a session-oriented mechanism, meaning that a relationship is created between the endpoints of an SCTP association prior to data being transmitted, and this relationship is maintained until all data transmission has been successfully completed.

Unlike TCP, SCTP provides a number of functions that are critical for telephony signaling transport, and at the same time can potentially benefit other applications needing transport with additional performance and reliability.

Link aggregation / trunking

Status: committed to -CURRENT
Will appear in 7.0: sure
Author: Reyk Floeter (from OpenBSD)
Manpage: lagg(4)

OpenBSD's trunk(4) was imported to FreeBSD in time to be shipped in FreeBSD 7.0. The trunk interface allows aggregation of multiple network interfaces as one virtual trunk interface for the purpose of providing fault-tolerance and high-speed links. The driver currently supports the trunk protocols failover (the default), fec, lacp, loadbalance, roundrobin, and none.

Improvements to kernel facilities

PMC performance monitoring

Status: Available in -CURRENT, partially available in RELENG_6
Will appear in 7.0: sure
Author: Joseph Koshy
Homepage: http://people.freebsd.org/~jkoshy/projects/perf-measurement

This project implements a kernel module (hwpmc(4)), an application programming interface (pmc(3)) and a few simple applications (pmcstat(8) and pmccontrol(8)) for measuring system performance using event monitoring hardware in modern CPUs.

Some parts (hwpmc, libpmc, pmcstat) were developed even before RELENG_6 was branched and new development goals for 7.x include support for callgraphs and a GUI front end.

Interrupt filtering

Status: Mostly committed to -CURRENT
Will appear in 7.0: sure
Author: Paolo Pisati
Homepage: wiki page

Interrupt filtering is a new method to handle interrupts in FreeBSD that retains backward compatibility with the previous models (FAST and ITHREAD), while improving over them in some aspects. With interrupt filtering, the interrupt handler is divided into 2 parts: the filter (that checks if the actual interrupt belongs to a device) and a private per-handler ithread (that is scheduled in case some blocking work has to be done). The main benefits of this work are:

Feedback from filters (the operating system finally knows what's the state of an event and can react consequently).
Lower latency/overhead for shared interrupt line.
Previous experiments with interrupt filtering showed an increase in performance against the plain ithread model in some cases.
General shrink of the machine dependent code - part of the interrupting handling code was turned into machine independent code.

Linuxulator for Linux 2.6

Status: Committed to -CURRENT
Will appear in 7.0: sure
Authors: Alexander Leidinger, Roman Divacky
Homepage: blog post, cvs commit note

FreeBSD includes support for natively executing Linux binaries. This is done via runtime translation of Linux syscalls to BSD syscalls, with no performance penalty. The facility is colloquially called the "linuxulator".

Linuxulator in -CURRENT has been updated to run binaries made for Linux 2.6.16 (though the default for 7.0 will still be 2.4), and the official Linux environment will be Fedora Core 5.

New scheduler: ULE 2.0 / 3.0

Status: Committed to -CURRENT
Will appear in 7.0: sure
Author: Jeff Roberson
Homepage: CVS file reference, commit message, description

The original SCHED_ULE was under-performing and buggy, so it got reworked. The new scheduler replaces, and has the same name as, SCHED_ULE, but is of a somewhat different architecture. It replaces the double queue mechanism with circular queues, and fixes a lot of other things, but it's still an O(1) scheduler with per-CPU queues.

During SCHED_ULE 2 development there was a brief period where there was a third (or fourth, depending on how you count) scheduler, named SCHED_SMP, forked from SCHED_ULE 2 and heavily optimized for configurations with large number of CPUs (8+). This SCHED_SMP has been renamed and committed as SCHED_ULE. While the new scheduler will really shine for multi-CPU machines, it's now also recommended for single processor systems as it has much better interactive performance (mixing of processes with different requirements for IO vs CPU time). ULE will not be enabled by default for 7.0 but it's an officially recommended performance optimization.

Improved accounting file format

Status: Committed to -CURRENT
Will appear in 7.0: sure
Author: Diomidis Spinellis
Manpage: acct(5)

The accounting record format has been revised to store time values with microsecond precision. This allows the recording of meaningful values for short-running commands on modern fast processors. The adoption of the IEEE 754 float format for storing time and usage values greatly increases their range and precision, and also simplifies the processing of accounting records by third party tools. The new record format and the tools lastcomm(1) and sa(8) maintain backwards compatibility with the original accounting format.

Storage subsystems' improvements

ZFS

Status: Committed to -CURRENT
Will appear in 7.0: sure
Author: Pawel Jakub Dawidek
Homepage: announcement message, commit announcement message

Sun's ZFS is in the process of being ported to FreeBSD, with the intention of offering most (or all) features found in the original implementation. It's integrated with FreeBSD's existing features like UFS and GEOM, thus offering the possibility of creating FreeBSD UFS file systems on ZFS volumes, and using GEOM providers to host ZFS file systems.

ZFS is an advanced file system (actually, a combination of file system and volume manager) with many interesting features built-in: snapshots, copy-on-write, dynamic striping and RAID5, up to 128-bit file system size (limited to 64 bits in practice even in Solaris - there's no 128-bit integer type in standard C language), and globally optimal I/O sorting and aggregation. It's marked EXPERIMENTAL in 7.0-RELEASE.

ZFS is still experimental on FreeBSD, and it's recommented that users get familiar with FreeBSD ZFS documentation before using it. For a more light-hearted introduction see this presentation by Pawel.

tmpfs

Status: Committed to -CURRENT
Will appear in 7.0: sure
Authors: Julio M. Merino Vidal, Rohit Jalan, Howard Su, Glen Leeder
Homepage: TMPFS page on FreeBSD wiki, TMPFS at NetBSD

TMPFS is a memory file system designed to efficiently allocate (and deallocate) memory used for the file system itself, as contrasted to the "usual" way of creating memory file systems by creating memory storage devices ("RAM drives"). It's marked EXPERIMENTAL for 7.0-RELEASE.

gjournal

Status: Committed to -CURRENT
Will appear in 7.0: sure
Author: Pawel Jakub Dawidek
Homepage: http://bsdblogs.droso.org/pjd, announcement message

Gjournal is a GEOM storage class that provides data journaling facilities to any providers (and consumers) the user needs. As a special case it has support in UFS file system code, and in this combination it makes UFS a journaled file system. In itself, gjournal consumes two devices (one for the data, one for the journal) and provides one. Since it takes special care to work well with disk drive hardware caches, it can be used to accelerate and provide reliability in many other uses, such as GELI and GBDE encrypted device providers.

I'm proud to say current gjournal is a continuation of my idea implemented for Google's Summer of Code 2005.

gvirstor

Status: Committed to -CURRENT
Will appear in 7.0: sure
Author: Ivan Voras
Homepage: http://wikitest.freebsd.org/gvirstor

Gvirstor is a GEOM storage class that provides a storage device of arbitrary size in "overcommit" mode (i.e. larger than physically available storage). Providers can be added to the virstor device on-line (while used, e.g. mounted), and removed if unused and at the end of the list of components.

This work was created by me, with Pawel Jakub Dawidek as mentor and sponsored by Google in Summer of Code 2006.

gmultipath

Status: Committed to -CURRENT
Will appear in 7.0: sure
Author: Matt Jacob
Homepage: CVS message

Gmultipath allows failover between multiple devices that represent the same storage device. This is an active/passive{/passive...} arrangement that has no intrinsic internal knowledge of whether devices it is given are truly multipath devices. As such, this is a simplistic approach, but still a useful one. The first of N identical devices (and N *may* be 1!) becomes the active path until a BIO request is failed with EIO or ENXIO. When this occurs, the active disk is ripped away and the next in a list is picked to (retry and) continue with.

New platforms

New platform: ARM architecture

Status: Committed to -CURRENT, MFC-ed to RELENG_6
Will appear in 7.0: sure
Authors: Olivier Houchard, Warner Losh & more
Homepage: http://www.freebsd.org/platforms/arm.html, http://bsdimp.blogspot.com/

Support for ARM embedded architecture has been under development since 6.0, enabling FreeBSD presence in the embedded markets.

The support is now MFC-ed to 6.x and is available in 6.2-RELEASE. It's still under development and will likely support more boards in the future.

New platform: sun4v (Niagara / T1)

Status: Committed to -CURRENT
Will appear in 7.0: probably
Authors: Kip Macy, John Birrell & more
Homepage: CVS announcement

There's still a long way to fully supporting Sun's Niagara/sun4v platform, but progress is slowly being made. Niagara offers advanced features such as eight cores and 32 threads per CPU, and hardware public key cryptography acceleration. Unfortunately, this architecture is not supported out-of-the-box in 7.0.

Security features

Security event auditing

Status: Committed to -CURRENT, MFC-ed to RELENG_6
Will appear in 7.0: sure
Authors: Robert Watson & more
Homepage: http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/audit.html

Event auditing allows the reliable, fine-grained, and configurable logging of a variety of security-relevant system events, including logins, configuration changes, and file and network access. These log records can be invaluable for live system monitoring, intrusion detection, and postmortem analysis. FreeBSD implements Sun's published BSM API and file format, and is interoperable with both Sun's Solaris and Apple's Mac OS X audit implementations.

Audit framework was MFC-ed to RELENG_6 and is available in 6.2-RELEASE.

New privilege separation capabilities

Status: Committed to -CURRENT
Will appear in 7.0: sure
Author: Robert Watson
Homepage: list announcement

This is a framework which can be used together with MAC to creating policies similar to RBAC (as seen in Solaris & others) which allow the root privilege to be separated into several fine grained capabilities such as "can access the network" or "can bypass file system quotas". This is work in progress and no shipped policy modules directly implement all of the functionality yet.

Multimedia features

Hi-def audio

Status: Mostly committed to -CURRENT
Will appear in 7.0: sure
Author: Ariff Abdullah
Homepage: http://people.freebsd.org/~ariff/HDA/

Newly developed driver, snd_hda has been developed to support professional sound equipment and new hardware on the market. HDA hardware is capable of delivering 192 kHz/32 bit quality for two and 96 kHz/32 bit for up to eight channels. Latency has been reduced for many cases.

Related to this, new drivers for envy24(ht) sound hardware is committed to -CURRENT, and multichannel audio support is due to be finished soon.

Userland enhancements

jemalloc

Status: Committed to -CURRENT
Will appear in 7.0: sure
Author: Jason Evans
Homepage: http://people.freebsd.org/~jasone/jemalloc/

The currently used malloc() library, called phkmalloc since its creator is Poul-Henning Kamp, is almost a decade old in its present implementation. It was designed for a time when memory was scarce, the priorities considered in memory allocation were different, and multithreading was still an academic idea. Even so, it's one of the more popular malloc() implementations, used in all BSDs and even some historical Linux distributions.

Because of its inefficiency when used in multithreaded applications running on multiprocessor systems, a new userland memory allocator was created, named jemalloc after Jason Evans, its creator. The improvements in computer speed and memory availability mean that compared to phkmalloc, which only needed to be conservative in memory usage, jemalloc needed to be more sophisticated and account for low-level properties such as CPU cache locality and parallel execution.

The result is an allocator which is optimized for multithreading, using multiple allocation arenas to help concurrency. On single processor systems there's only one arena, while on multi-processor or multi-core systems there are four times as many arenas as there are processors. Allocations are divided into broad classes based on their size and those classes are further subdivided. Benchmarks show that jemalloc does significantly better in multithreaded applications (like MySQL) and for applications that make many small allocations.

Bits & pieces

Authors: many

Here are some additional changes for 7.0 that are not so glamorous or are smaller in scope:

Lots of performance improvements on SMP machines (see MySQL read performance, MySQL write performance and BIND performance graphs.)
Significantly increased scalability on SMP machines, mainly from extraordinary work done by David Xu (the libthr threading library), Jeff Roberson (scheduler, flock locking), Atillio Rao (improved kernel locking performance) and Robert Watson (file descriptor locking, unix sockets locking and more).
Significantly increased network scalability, resulting mostly from switch to direct dispatch of the network stack from netisr. This is especially helpful for 10 Gbit/s NICs and was mainly done by Robert Watson and Kip Macy.
GIANT lock has been pushed further back, and almost all kernel subsystems are now finely locked (e.g. VM, VFS, Net). Some of the recent improvements are: locking the CAM subsystem and many SCSI drivers (by Scott Long), and similar locking work has been done on the NFS client and the Firewire implementation.
iSCSI initiator (iSCSI target is available in ports)
SATA support
Read-only access to XFS file systems
Added support for MSI/MSI-X extensions to PCI
Support for Apple (Mac) hardware is being worked on
pf firewall updated to 4.1
X.Org 7.2 - things like beryl now work if you have the right drivers
gcc 4.2
Implemented symbol versioning for many base OS libraries
libthr becomes the default threading library

Things that didn't make it

Despite plans and best efforts, some things won't make it into FreeBSD 7.0-RELEASE. These are:

SCHED_CORE - Doesn't perform as well as SCHED_ULE2
DTrace - MFCed into 7.2.
Superpages - MFCed into 7.2.

Of course, this much new technology will need much testing before it's ready for use. You can help by installing a snapshot of -CURRENT and running it on as close to your regular load as possible. Disable debugging features (which are enabled by default during development) before benchmarking.

FreeBSD 7 Footshooters

This page will contain a list of (relatively) small annoyances and issues present in FreeBSD 7 (and possibly earlier versions), with the purpose of forewarning and helping others that have stumbled on the same problems as I have. These are nothing special and some of them may be documented elsewhere.

tmpfs and fstab

If you mechanically add "2" to the end of fstab lines (fsck pass#), you'll get a big surprise the next time you reboot the machine if you try to add tmpfs to the list, or change the entry from md-based /tmp to tmpfs. The problem is that there's no fsck for tmpfs so when the machine boots, it will try to check the file system, will fail and stop the boot process. The solution is to simply change the "2" to "0".

SCTP depends on IPv6

You can't build a kernel with SCTP option but without IPV6 option, so remove it together with IPV6.

ZFS manipulation needs writable /boot

Modifying zpools and zfs (e.g. adding devices to zpools) needs to be done with /boot/zfs directory writable. Since /boot is usually on the root file system, the root partition needs to be writable. This is important in single user mode where the root file system is mounted read-only. After mounting it rw, and before starting ZFS with /etc/rc.d/zfs start, hostid must be initialized with /etc/rc.d/hostid start. Failure to have /boot writable while changing zfs structure will result in a panic when ZFS is started "regularily" the next time. This can be fixed by deleting zpool.cache from /boot/cache and performing zfs import. Failure to initilize hostid will result in ZFS file system(s) not being present the next time ZFS is started, which can be fixed by zfs import -f.

Compiler flag -Os might produce bad code

There have been many reports, and my own experience supports them, that using gcc -Os flag (optimize for smaller code) sometimes produces bad code. The symptoms vary wildly over architectures and flag combinations, so that a system that appears to be working fine might have subtle bugs lurking to be discovered in the most inconvenient time. This affects both the kernel and the userland. The compiler in FreeBSD 7.0 is gcc 4.2.1. This might be fixed in the future by importing a newer version of the compiler, but don't assume it's happened if you need the flag.

ZFS is only experimental

ZFS (the file system) is an experimental feature in FreeBSD. Don't even try using it before proper tuning.

My FreeBSD papers

Conference talks

Encrypted iSCSI device tutorial

Suppose you want to use a remote iSCSI device, but you don't exactly trust either the storage or the network in between. Of course, there's a way around it :)

The setup presented here is very simple and will behave like this:

[iSCSI server] -- encrypted data on the server and over the wire -- [iSCSI client]

Note: all these instructions are valid for FreeBSD 7.0 - previous versions are probably missing some parts.

Setting up an iSCSI target

You can skip this section if you already have an iSCSI target (a "target" is where the data is stored, i.e. the "server" node of iSCSI).

1. Install the iscsi-target port.

2. Edit /usr/local/etc/iscsi/targets file and add lines similar to the following:

# NAME          DEVICE          START           LENGTH
extent0         /dev/da2        0               5GB
# NAME          ACCESS          STORAGE         NETMASK
target0         rw              extent0         10.0.0.0/24

These lines should be self-explanatory. If you need more help, see targets(5) or NetBSD's iscsi-target HOWTO.

3. Enable iscsi-target in /etc/rc.conf by adding the following line to it:

iscsi_target_enable="YES"

4. Start the server by running /usr/local/etc/rc.d/iscsi_target start. You should see something like the following outputted to the console:

Starting iscsi_target.
Reading configuration from `/usr/local/etc/iscsi/targets'
target0:rw:10.0.0.0/24
        extent0:/dev/da2:0:5368709120
DISK: 1 logical unit (10485760 blocks, 512 bytes/block), type iscsi fs
DISK: LUN 0: 5120 MB disk storage for "target0"
TARGET: TargetName is iqn.1994-04.org.netbsd.iscsi-target

Setting up the iSCSI initiator

The "initiator" is the client part in iSCSI, and it connects to the server. The following steps should be done on the client system.

1. Edit /etc/iscsi.conf and add the following lines:

target0 { # nickname
        targetaddress        = 10.0.0.102
        targetname           = iqn.1994-04.org.netbsd.iscsi-target:target0
}

2. Load the iscsi_initiator kernel module with:

# kldload iscsi_initiator

Also, add the following line to /etc/loader.conf to load the module on boot:

iscsi_initiator_load="YES"

3. Start the iSCSI session by running:

# iscontrol -n target0

Several lines should be output to the console, which should look like the following:

iscontrol[8516]: running
iscontrol[8516]: (pass3:iscsi0:0:0:0):  tagged openings now 0
iscontrol[8516]: cam_open_btl: no passthrough device found at 1:0:1
iscontrol[8516]: cam_open_btl: no passthrough device found at 1:0:2
iscontrol[8516]: cam_open_btl: no passthrough device found at 1:0:3
iscontrol: supervise starting main loop

More importantly, the kernel log (which you can see with tail /var/log/messages) should now contain something similar to this output:

Jan  4 23:17:08 client kernel: da0 at iscsi0 bus 0 target 0 lun 0
Jan  4 23:17:08 client kernel: da0:  Fixed Direct Access SCSI-3 device

This means the device da0 has been created - this is the local representation of the remote iSCSI drive. Technically, da0 is the GEOM device node for a SCSI-like storage device. All further transformations on it are performed as natural parts of the GEOM framework.

3. Set up GEOM_GELI on the new device:

# geli init /dev/da0

The utility will ask you for a passphrase which will be used to encrypt the data. GEOM_ELI (as is the encryption layer known) has many more options, but the defaults are good enough. It will use AES encryption with sane defaults.

4. Load the GEOM_ELI kernel module:

# kldload geom_eli.ko

Also, add the following to /boot/loader.conf to load the module at boot time:

geom_eli_load="YES"

5. Attach the encrypted device:

# geli attach /dev/da0

Lines similar to the following should appear in the kernel log:

Jan  4 23:33:28 client kernel: GEOM_ELI: Device da0.eli created.
Jan  4 23:33:28 client kernel: GEOM_ELI: Encryption: AES-CBC 128
Jan  4 23:33:28 client kernel: GEOM_ELI:     Crypto: software

The device da0.eli has been created - this is the end-point device that can be used by file systems and for other purposes (swap, etc.).

6. Make the file system and mount it!

# newfs -U -L mydata /dev/da0.eli

A successful run of newfs looks something like this:

/dev/da0.eli: 5120.0MB (10485756 sectors) block size 16384, fragment size 2048
        using 28 cylinder groups of 183.77MB, 11761 blks, 23552 inodes.
        with soft updates
super-block backups (for fsck -b #) at:
 160, 376512, 752864, 1129216, 1505568, 1881920, 2258272, 2634624, 3010976, 3387328,
 3763680, 4140032, 4516384, 4892736, 5269088, 5645440, 6021792, 6398144,
 6774496, 7150848, 7527200, 7903552, 8279904, 8656256, 9032608, 9408960, 9785312, 
 10161664

Since we used a volume label for the file system, observe the following message in the kernel log:

Jan  4 23:38:17 client kernel: GEOM_LABEL: Label for provider da0.eli is ufs/mydata.

Now you can mount the file system:

# mount /dev/ufs/mydata /mydata

And that's it!

There are two points that can't be readily automated right now: the iscontrol step which starts the iSCSI initiator, and the geli requiring a password. The former can be approximated by creating a small shell script that does the step and putting it in /usr/local/etc/rc.d but the second cannot be, since the whole point of having an encrypted storage is that it isn't accessible by unwanted people.

The way this setup works is that the unencrypted data is used by the file system (as it should - you wouldn't be able to use it otherwise) via the da0.eli device. This data is encrypted and the encrypted data is written to da0 device. This is the iSCSI client device and the data is tranferred to the server in its encrypted form. The server and the network never see unencrypted data.

Due to GEOM's modularity (all devices can have any tranformations applied to it; "partition" entries are also devices), other components could be added to the data processing graph, such as journaling (gjournal), caching (gcache), etc. in which case the end-point device name will "grow" suffixes, such as da0.eli.journal. Even RAID levels can be added, though it makes little sense to do it on the client (it's perfectly fine on the server).

Tree macro tutorial

BSD systems include macros for several useful structures and algorithms, including several types of lists and trees. While the lists are easy to use, I always forget the right order of declarations for trees. So here it is, how to use tree macros in FreeBSD (C language):

First, declare the structure that is to be stored in the tree:

    struct mydata {
        RB_ENTRY(mydata) linkage;
        int payload;
    };

Second, declare the comparison function. This function compares two structures in the way similar to strcmp():

    static int mydata_cmp(struct mydata *e1, struct mydata *e2) {
        return e2->payload - e1->payload;
    }

Next, declare the head structure and head entry. Head structure is the struct type of the tree head, and it's the way tree is accessed.

    RB_HEAD(mydata_entries, mydata) head = RB_INITIALIZER(&head);

You're now ready to declare the prototypes for the internal tree structures and the functions themselves:

    RB_PROTOTYPE(mydata_entries, mydata, linkage, mydata_cmp);
    RB_GENERATE(mydata_entries, mydata, linkage, mydata_cmp);

The tree can now be used normally, the way described in the manual:

    struct mydata *data;
    RB_INSERT(mydata_entries, &head, data);

    struct mydata find;
    find.payload = 42;
    data = RB_FIND(mydata_entries, &head, &find);

    RB_FOREACH(data, mydata_entries, &head)
        printf("%d\n", data->payload);

In the examples above:

mydata is the structure to be stored in the tree. It contains some arbitrary payload data but must contain an TREE_ENTRY element.
mydata_entries is the type that contains the tree. It's declared by the RB_HEAD macro.
head is the tree head.

FreeBSD ggtrace

Ggtrace is "GEOM gate tracer", utility to track I/O requests on a storage device on FreeBSD. It uses the ggate facility of FreeBSD to attach to a file or device and produces a device that can be used for any I/O, including hosting filesystems.

I/O requests are presented in the form of a moving histogram that can be used to discern which parts of the storage device are used most often. An important use of ggtrace is to analyze how filesystems arrange and access data on storage devices.

Example graph produced by ggtrace

On the graph above, a filesystem was mounted without Soft-updates and lots of small files were copied to it (the red parts that go from left to right) and then deleted (the red parts that go from right to left). These operations were repeated several times with different timescales in ggtrace.

Several blocks on the device are presented as pixels in a single row. The leftmost pixel represents blocks from the beginning of the device and the rightmost pixel represents ending blocks. The scale is such that all blocks fit in the window. The window itself can be scaled at will.

Red blocks represent writes, blue blocks represent reads. Areas that get both read and write requests have mixed red+blue color components. The intensity of the color represents the intensity of activity: less intense color is used when there is little activity in the area and bright color appears when there are many requests in the area.

The timeframe alloted (timescale) to each row is adjustable at runtime via a slider control.

Ggtrace requires wxgtk26 and png ports to build. The sources can be downloaded here but see the SourceForge site for possibly newer version. Ggtrace is working and usable only on the RELENG_6 branch.

gvirstor

gvirstor is a virtual storage device for FreeBSD sponsored by Google's Summer of Code 2006 program. Here is a short list of features gvirstor provides:

Creates storage devices in "overcommit" mode, larger than physically available storage
Can create devices of arbitrary size, upto 2^63 bytes, with arbitrary chunk (extent) size
Notifies via kernel-syslog mechanism when used-up storage approaches available storage (watermark values for these warnings are controllable with sysctls)
Allows adding new storage devices (components) at runtime, when virstor device is "hot"
Allows removing storage devices (components) at runtime, when they are unused and at the end of list of added components
Updates virstor allocation table (i.e. metadata) synchronously with write requests (all considerations with drive caches still remain, but if the drive doesn't lie, the data is safe)

The gvirstor homepage is located in FreeBSD's wiki. It is considered stable and safe to use, and will be committed to FreeBSD's CVS soon.

gjournal

News: Pawel Jakub Dawidek <pjd[at]freebsd.org> has created a reimplementation of gjournal which should be more stable and can work together with UFS to achieve a journaled file system. The implementation described on this page is officially obsolete.

gjournal is a project I made for Google's Summer of Code in 2005. Its purpose is to provide device-level journaling for disk drives (and similar devices), but the ultimate goal was to sort of provide a stop-gap solution instead of filesystem journaling. It turned out that it couldn't be done this way because of the way the UFS filesystem handles references to deleted-but-open files (and probably other thins). Another feature that was added to gjournal was "Copy-on-write logging" - the ability to log the writes to the device and then allow at an arbitrary point to commit or rollback the changes.

The official project page is here. The code is AFAIK stable, but unoptimized and untested in real life/deployment. It can be downloaded here.

If there's continued interest in gjournal, I'd probably continue development by splitting the journal and COW functions into separate and more easily maintainable codebases. One other, very likely possibility for the COW funcionality is to remake it in userland using ggate (more easily done, faster & more maintainable code.

ggcomp for FreeBSD

GEOM compression layer for FreeBSD 5.3+

Update: Don't use it :) It's badly designed and has horrible performance. I'll probably write a real GEOM class to do it when I find the time.

ggcomp provides block-compression on the GEOM device layer, implemented as a ggate consumer in userland. It registers a ggate device with 8k sector size, which can be used to, e.g. host a UFS filesystem. Sectors written to it will be compressed on the fly using zlib (with the "fastest" compression level) and written to the underlying storage (which can be a device or a regular file). The compression ratio depends on the data being written, but it's actually not bad. Performance, on the other hand, suffers greatly, and you can expect about 10x worse performance on such compressed device, as compared to "regular" non-compressed performance.

It registers a device with "virtual" size and number of sectors that is some (configurable) factor larger than the physical media.

The code is still in development, but at this time it's pretty safe to use, and won't e.g. arbitrarily panic your kernel :). The only caveat concerns the borderline case when the physical device gets full, or no more blocks can be assigned to compressed data. In that case, ggcomp returns EIO to the upper layers.

Download

You can download ggcomp here. It's a source tree with included Makefile (just run 'make' to produce executables). See the included README file for recent news and usage examples.

Predictions of future

It's fun to pretend to predict future events and environments, if only so we can laugh when the predictions turn out to be wrong and much more quaint than the real thing. So, here are my predictions. I don't want to be overly optimistic and put the predictions too near "our time", so let's say they are targeted for about 20 years from now (beginning of 2008.) though some of them may be realized even 10 years from now. Here's the list:

Microsoft will sell cars, or car parts, or fridges. Something "classical".
Dell is the new Apple.
Google is the new Microsoft. Some FutureTech Company is threatening the long and obnoxious rule of Google.
The concept of paying for bandwidth instead of content is relegated to the history books and looks like something from third-world nations in Africa.
Computers without always-online Internet connectivity will be almost useless to the "common man", though there will always be a core of a culture of "old-fashioned" users that want total control over their data (probably evolving from the today's OpenBSD and Crypto nut communities). The "old-fashioned" applications that deal only with local data will stagnate, simply because there's nothing new to invent there (the last real inventions in desktop apps happened in Xerox some decades ago: the mouse and WYSIWYG editors).
A small percentage of "global warming" and other "sky is falling" theories happen. We'll probably lose Venice, but people will continue to live their lives without major interruptions. Possible future "sky is falling" theories include meteor hits and WW3.
USA and Russia are constantly attempting to start another Cold War but nobody cares.

Should I buy a Mac?

A Mac?

Thanks to a friend, I had a chance to try two Apple laptops, a MacBook and a MacBook Pro over the weekend. The reason for this is that I wanted to try a Macintosh system for some time now, but I never had a chance until now. I mean, I saw the commercials, the web sites and the blogs, had a chance to tinker with the occasional demo system but I never used them for more than 5 minutes.

I'd first like to vent some acid about the availability and pricing of Apple computers in this sorry little excuse for a country. It seems there's only one Apple partner/reseller in this country, and the prices they have are very unreasonable. The MacBook costs (when directly converted back to USD) around $2100, and the MacBook pro costs close to twice than that, around $4100. Compare this to regular Apple prices (in US) and you'll see the local prices are doubled! I know Apple computers are traditionally priced up to 25% more than equivalent PC laptops but this is ridiculous.

Hardware

By the feeling of it, both laptops seem to weigh about the same (I didn't measure it with scales), which is suprising for the smaller one. The MacBook is too heavy for its size, especially when compared to similar PC laptops. I don't know who designed the keyboard on the MacBook nor why he designed it like that, but he should be shot for it. Not only is it ugly, but it's also unergonomic in several ways: the keys have an awkward shape and angle (totally parallel with the surface of the laptop), and the space between the keys is too large for comfort. The keyboard on the MacBook Pro looks small for the size of the laptop surface, but the keys are actually large enough. There's plenty of empty space around it could maybe be filled with more buttons or controls (the "function" keys are half-height and double for hardware control keys like audio volume and screen brigthness - they could have been larger and the hardware controls could have been separate). The quality and feeling of it is much better than that of MacBook but it's still behind that of quality PC laptops. The lid of the MacBook Pro is opened by a small narrow "button" which is tedious to push as it's too narrow for my fingers. Overally, both models "feel" more robust than average PC laptop hardware, with the only exception being the CD loader, which emits strangely screeching noises when loading and unloading CDs. The location of the CD/DVD slot drive on MacBook Pro - at the front of the laptop - is a bad joke. The only hardware-related thing I'm impressed with is the magnetic power plug - it's much cooler that I thought it would be.

The screens on both are reasonably good but again not as good as on high end PC laptops (especially Sony's). On the MacBook the colors seems somewhat washed up (noticable while playing DVDs).

The connectors on the MacBook are badly placed - all of them are on the left side of the laptop, placed too close together, which is again unreasonable as there's plenty of empty space on that side alone. The MacBook has only two USB ports, one next to the other, which creates problems if you have bulky USB devices or connectors. MacBook Pro, though much larger, has almost the same number of ports. There are still only two USB ports, but luckily this time placed on both sides. The network port (RJ-45) is on the right hand side now, which is better since most people are right handed. There's no VGA port (there's an adapter), which makes both models unreasonably tedious to use for giving presentations over a projector.

Software

Keyboard layout deserves a special mention. As a person that's been using PCs and PC laptops "forever", I think some of the choices Apple made are simply not reasonable. It's a special problem for me as I'm using non-US layout (Croatian), and while the layout is close enough to that used on PC laptops, there are three big problems with it: the first is that the "Y" and "Z" keys are switched (this is a special local problem here as they have been switched before by the industry in early nineties and people have just gotten used to it - it's a small matter but it creates stupid typos). The second is that the "special" keys or puncuations that have been replaced by local diacrytic characters on the keyboard can be accessed by completely different key sequences than on PC keyboars. For example, to type the "@" character on a PC keyboard, the sequence is AltGR-V, but on a Mac it's Ctrl-Shift-2 (three keys!), which is almost impossible to type with one hand. The "|" character on PC is AltGr-W, and on the Mac it's Alt-Shift-�, which is not consistent with the sequence for "@". The third problem is that the "Enter key" is much too small (most of it is about 1 cm in width). All these problems are related to the local version of the Mac keyboard and probably don't exist for US users on the US keyboard, but are very annoying for people who work with both PCs and Macs over here. As I do a lot of writing (both programming and "normal" text), this is simply a handicap.

After a short time to get used to it, MacOS X (10.4) seems much better than the hardware it runs on. Though it doesn't look as impressive as it does on commercials and demos, it's decent and pleasent to work with. I have the feeling that its graphic capabilities are underused. Maybe it's because of new developments in Linux and Windows, but it looks somewhat simplistic in comparison. I can find four major things I dislike in the UI: the buttons don't have enough visual feedback when they are pressed, it looks like there's a bug that "locks" windows in drag mode even when drag-lock is disable, the mouse pointer controlled by the track pad doesn't continue to move in the same direction when you reach the end of the trackpad with your finger like it does in Windows and Linux, and applications are incisistent in how they look ("brushed metal" vs "light-gray gradient" skin). Overall I'd say these are really minor problems and I think I can get used to them. The Spotlight feature is very neat, but I didn't create nearly enough documents to test it fully.

The overall philosopy of the UI is fine, I'd say the applications I've tried behave more consistently than on Windows or on Linux, so this is a definite good point. Unfortunately, I didn't try XCode as I hoped I could because the keyboard layout is just depressingly hard to use for programming.

I managed to find all applications I'd need in normal use, either as bundled or on the web (the fact that OS X is Unix underneath has helped this a lot). At first I thought I would have objections to the "single-mouse-button" policy, but I got used to it very quickly. I'd still recommend getting a two-button mouse for real work but it's not that bad even with the single button.

Hmmm...

At the end, I decided not to buy any of them. The MacBook Pro is so expensive that it's silly (at local prices), and the MacBook has too many hardware issues (bad placement of ports, keyboard that seems to be designed to make your hands hurt while using it, unusually heavy weight) to be usable. I think I can see myself using a MacBook Pro if, for example, someone bought it for me or I win it at a lottery, and I find a way to fix the keayboard layout issues, but I just don't feel like buying it for my own money. The "wow" effect is practically nonexistant.

Roadrunner and the unfairness of life

I doubt I'm the first one to notice it, but (re)watching the old "Road runner" cartoons it struck me just how unfairly the life treats Coyote (as well as how brutal and cheerfully violent the cartoons are - no way something like that would be made nowadays). Like all kids, I enjoyed watching Wile E. Coyote fail in comical ways, getting kicked, squashed, burned and blown up, laughing all the way.

The whole cartoon is the embodiment of the "Life isn't fair" line. It just isn't fair (to coyote)! Starting from the cartoons' name: everyone knows them as the "Road runner cartoons" despite that 90% of the time we look at the Coyote character, starving and trying to survive by catching the scrawny (and stupid) bird. And it's not like the Coyote is being melodramatic or depressed - life actually isn't fair to him, and everything has turned against him. The laws of physics have been rigged so he will suffer: he's attracting heavy, sharp, hot and explosive objects like a multi-purpose magnet (or a temporary black hole) , the Roadrunner can walk on thin air but that same air will drop him to his doom, explosives will not explode except when he's near them, slings and giant springs have Newton's second and third laws suspended until he's on the path of maximum damage (to himself), and energy sources (batteries and fuel tanks) will only (but always) fail when his vehicle is upside down. The Roadrunner apparently has telekinetic and ESP abilities allowing it to sense his weapons and redirect their fire to the Coyote, and they will hit him no matter where he hides.

The Coyote is like some modern Tantalus, tortured forever by the near presence of his food but never allowed to eat it. Alternatively, maybe he was Hitler in his former life and his karma is just catching up with him. There's never any blood on him or around him - the worst that happens is that his body gets mangled and his fur burned, so he continues to live forever (probably in pain).

On the other hand he sometimes really is stupid, making his experiments under unstable boulders, trusting his life with untested contraptions and, at last, failing to find something else to eat. He can also be considered the embodiment of stupid optimism.

BSDCan 2007 and FreeBSD DevSummit 2007

BSDCan

It's been a very interesting and useful conference, I've learned much from it and met many interesting people. It's no use just listing the talks given on the conference (you can look that up here), but, together with DevSummit, this was one of the best events I've ever been to.

I'll try and think of something more to say...

DevSummit - Day 2

This is the "official" day of DevSummit talks, and has so far proven to be very interesting and instructive. Reports on ZFS and SMP scalability were also uplifting: it's really great to see so much good work being done recently. An interesting piece of information was the result of recent scalability measurements: FreeBSD 4.x run excellent on 1 CPU machines, 5.x scaled to 2 CPUs, 6.x to 4 CPUs, and 7.x will run great on 8 CPU machines (see here).

Another very interesting talk was a self-referential, meta-presentation on giving presentations about FreeBSD by Robert Watson, which also addressed the problem of having too much to say and ttoo little time in a... unique... way :) During the talk he brought up a very interesting point about the way FreeBSD is organized: it's one of the very rare open-source projects that have institutionalized "transition of power" - leadership is reelected every few years and is in no way fixed or messianic.

This was also the day I gave two of my talks, about gvirstor and finstall!

DevSummit - Day 1

Informal sessions were spread across the day, and have proven to be very interesting. I was mostly on the "alternative" track, participating in discussions about the ports tree and storage systems. Judging from the reaction a brief mention of it provoked, it looks like finstall is going to be very popular!

DevSummit - Day 0

The flight was long and the customs check barbaric, but finally I've arrived at University of Ottawa campus to attend the DevSummit and BSDCan events held there. The campus is really nice and big, I really like the architecture.

Since it was late afternoon, the only thing I could do was join the others at the highly regarded introductory beer consuming at local Royal Oak pub.

Matrix as an OS

Since one of my fascinations is operating systems design, implementation and maintenance, ever since I've first seen the Matrix movies I've thought some of the concepts in them can be related to familiar concepts in operating systems:

The Matrix world: a running operating system, with userland (the "common" world, in which people live, and the kernel (the "Matrix" proper). Apparently it's a pretty buggy OS...
People: processes, both kernel processes and user processes. There's a big distinction between normal, "unprivileged" people, and daemons with root privileges - "agents". Root daemons can open privileged ports, kill random processes, manage memory, etc.
Matrix: the kernel. It looks like a message passing kernel, not necessarily a microkernel (though they are some microkernel aspects, such as the abundance of kernel processes, strict separation of duty between them, and the already mentioned message passing). Kernel manages all processes, and performs operations on their behalf (such as keeping them alive, servicing them and recycling them). But there's an apparent security defect: some userland processes can (because of a bug) transfer and execute parts of their programs in the kernel space. Only certain syscalls are affected (the "phones"), and this kind of privilege escalation garbles the userland process' return stack, such that if the process receives a signal, it segfaults and is garbage collected (if you're killed in the Matrix, you're dead for real).
Oracle: the process (task) scheduler. Has all the numbers from process monitoring (resource usage) and knows in advance (broadly) how to schedule them to run to their optimum.
Agents: system monitoring / intrusion detection / prevention system (IDS / IPS) with heuristical operation. Most of them have a kernel part (kernel module) but are basically daemons run with superuser privileges in the userland. They are tasked to find and kill processes which attempt to violate system security.
The trainman: kernel-userland gateway / message passing queue. You've got to go through him if you want to validly pass data between userland and kernel. You also might be stuck in the queue forever.
The Merovingian: networking / IPC stack. It's his business to know everything going on between processes. Has a bug manifesting in occasional input / output data corruption.
Vampires / ghosts: compatibility shims for older API / KPI versions. Their code is rudimentary and, for historical reasons, interfaces with parts of kernel normal processes shouldn't (i.e. they have lots of layering violations).
The Architect: kernel monitoring infrastructure (hypervisor), tasked with monitoring processes, killing those that wedge and restarting those that crashed. Since it's a realtime high-availability OS, the debugging and monitoring infrastructure has the absolute highest priority and is "blessed" to be infallible (thus, to limit the possibility of error, is very limited in its complexity). It's been misconfigured to be overzealous, does availability checking too often, taking too many resources, and so interferes with the normal operation of the operating system.
Keymaster: security / privilege subsystem. It's stable, but unfortunately relies on the VM system and the IPC system which are buggy, and can be exploited by processes to gain more privileges from him.
THE PLOT: There's a design bug between the VM (virtual memory) system, the process management system and the scheduler, manifesting under high system load (lots of processes, high memory pressure). It is a compound error, which results in at least three things:
- Memory pages can get corrupt or missasigned to processes that don't own them. Since kernel and userland share the VM, processes on either side can end up with memory pages from the other, revealing sensitive data and making way for security escalations. Mixing up the VM pages bypasses address space protection between the processes.
- The IPC subsystem, bad as it already is, gets even worse when its data structures get corrupted or the memory load gets so high it deadlocks waiting for buffers.
- The system monitor goes berserk, killing and restarting processes in a loop, unaware that it makes the things worse by building additional memory pressure and process load, eventually greatly helping spread the VM pages corruption between the processes.
Agent Smith: privileged IPC daemon with part of it implemented as a kernel module. It's so closely tied with the kernel module part that it shares data structures with it without sanity checking. Once it was killed by another privileged process, but it was in the middle of a syscall so when the monitor restarted him, the corruption which was already done to its process descriptor resulted in most of its program being executed in the kernel context. It continued to work in this corrupted state for a long time, wedged in a loop, erroneously tagging processes as security breaches and overwriting some of their memory pages with its own.
Neo: Initially a userland network server process, the VM corruption resulted in it being assigned both superuser privileges and high priority (CPU time). Eventually, it got its executable memory pages mixed up with the IDS process Smith, but not the data pages. Before long it also starts killing processes, including Smith and his corrupted copies.
THE ENDING: process Smith eventually tries to kill the scheduler process, but since it's itself scheduled by it, cannot do so reliably. The system gets wedged because the scheduler cannot perform its tasks anymore, including interrupt servicing, but the part of Smith's code in the scheduler's VM image (which is accidentally also the part shared with process Neo) still runs. Since there are only two processes running, they both are trying to kill each other. Meanwhile, since interrupts are no longer being served, the hardware watchdog timer wakes up, inserts a NMI, which wakes up the monitoring system. It decides the system is in a critical state and proceeds to kill all processes, then restarts them to bring the system up again. The End.

Post mortem analysis: There appears to be an inherent flaw in the design of the operating system, especially in the VM, IPC and monitoring subsystems, resulting in a global memory corruption among processes and critical failure of address space protection for a small number of processes.

Recommendation: More fine tuning is needed to settle out the proper process priorities, reduce priority inversions and imbalance. VM system probably needs to be rewritten and IDS system replaced with a less resource intensive version. System monitor needs to be modified not to start extensive operations if the system load is above a threshold.

----
There! An interpretation of The Matrix without involving "free will" in any way.

Ideas

I'm an ideas kind of guy - I tend to have a lot of ideas and no time to implement them. Since ideas are a dime a dozen, and there's no way I can find the time and resources (i.e. money) to make use of most of them, I'll just dump them on this page. Of course, most ideas are garbage, and the same holds for those on this page.

The small print: If anyone actually manages to implement an idea presented here, I'd appreciate I get mentioned in credits (also, I won't refuse money :) ). If anyone wants to implement an idea on this page and really wants me to remove the idea from the page, I can be persuaded (let's trade).

Anyway, here's my idea dump, in no particular order:

Hardware

Oval displays? In the almost-traditional over-the-top "what will Apple do next, designer-vise" discussion with a friend we came up with an "interesting" idea - round / oval displays. Of course, it's a stupid idea. Or is it? Technologically, an OS that has such advanced display features as OS X shouldn't have any problems with maintaining an oval display region / desktop. The one major technological problem could be waste area during production, if the displays are to be "cut out" of a rectangular slab of LCD. That leaves us the UI / human interface problem. Does it have any benefits here (aside from being unusual and sexy given current technology design standards)? The human field of vision is oval, so it could be argued that an oval display is more adjusted to it. Except for blocks of text, I believe that most other design elements can be modified to fit an oval shape.

2x 2.5" drives in a 5.25" bay I want a 5.25" case (internal HDD enclosure) like it's used for holding 3.5" hard drives in a 5.25" bay, but one that holds two 2.5" drives (one above the other), with proper cooling and hot swap ability. I want to do create a RAID1 array from the drives (hardware RAID optional, but forget it if it's not smart enough to handle hot swapping properly). Computers usually have more than enough 5.25" bays so why not use them?

"Auxels" The problem: Star-trek interface (i.e. touch screen) is cool, but the ape descendants using them (e.g. people) like tactile feedback. I think this kind of feedback could be provided by low frequency acoustic vibrations from behind the screen, Since these must be localized to where the user "clicked" with his finger, the audio sources or conducts must be spread across the surface, pixel-like. Hence "auxels" - acoustic pixels.

Software

JavaScript application server Current web development environments as a rule, use a different language at the server side (e.g. PHP) and at the client side (JavaScript). It would be great to use a single language (by default - JavaScript) for both, thus dropping the requirement that programmers need to use two languages and allowing much better client-server integration (i.e. AJAX-y features). This is something I actually intend to implement, if I ever get the free time.

JavaScript application "platform" Creating basic user interfaces in HTML is fairly easy when compared to any other method except visual "drag and drop" design. At the same time, HTTP's model of transactions (do UI on the client side, then POST the data to the server for processing) is easy and simple to understand. Why not create a sort of embeddable server which would allow developers to code for the same processing model (i.e. HTML or XML templates, business logic code, backend code, database) in plain JavaScript and package it all in one desktop application ("exe")? I don't consider desktop applications dead yet.

P2P mp3 streaming Many people within the office listen to MP3 streaming "radio stations", and many of them listen to the same ones. Wouldn't it be great if the clients themselves became sort of repeating stations? Each client listening to a stream could offer the same stream to the other computers in the office, possibly via mDNS or similar discovery protocol. In effect, this would save enormous bandwidth on the Internet.

This could as well be generalized to other types of traffic, such as web pages. BitTorrent is a right step in this direction.

Imaginary RPG Imagine a CRPG which is not much tied to the physical world. In it, events and environment depend on the imagination and the "willpower" of the participants. Let something like Solaris (from Stanislav Lem's book) or Limbo (from Planescape RPG setting) grant participant whatever they wish for, and let it be real. Let people have their wishes granted, and get in conflict with the wishes of other people!

I guess I'm after a sort of game with tangible (to make a pun) duality between the real and the unreal. If a man walks the desert and wishes it, he could perceive the desert as a forest, and as far as he is concerned, it WOULD be a forest. But unless he's powerful enough, a wall in real world will appear as wall in his own reality.

Or maybe a more interesting thing would be "reality by consensus", where actions and the environment up to a point in future are implicitly or explicitly "agreed upon" by the participants, and everyone's wishes have a certain echo in the reality-to-be.

This looks like a primarily non-combat CRPG, but there's space for combat too - I'm thinking of something like the ending combat in the Star Trek novel Spock must die, where the combatants "wish" various (mostly environmental) cataclysms on each other.

SciFi/stories

Occasionally, I just get random ideas that are best classified as SciFi stories' plots :)

Loophole for FTL? Photons have momentum but zero rest mass. What if you try to STOP photons as-is, would you get infinite energy?

A non-linear universe Consider how light spreads through glass: photons are constantly absorbed and released by the atoms in their path. Suppose that releases are not strictly in the same direction, but *always* vary, either constantly (always x deg from the entry point in a certain direction) or randomly. Could lens-optics be possible? Is it happening now? By a very slight amount related to the alpha constant?

Blinded by the sight of God? God said in the Bible: "no man shall see me and live!" - what if God is everywhere and everything, and this is the reason for mortality? Would the sensory deprived from birth live forever?

Perpetual catastrophe Oldish couple looking at sunset over a skyscraper city. So beautiful, so fragile. It's discovered that the galactic core has exploded (with a nod to Larry Niven), and the shock wave is coming just behind the light (or some other random catastrophe - volcanic winter?). Most people are despairing and rowdy, some (the couple is one of them?) are building a room - cubic room with many layers of shielding, to contain a monument (a stone slab) with writings, and cultural treasure around it. The room's shielding is a marvel of technology. Out of boredom, someone sets a satellite/s searching for the room signature and finds there two almost identical sites! When dug out, the second room looks almost destroyed, but in there are writings in it that look like they talk about sudden discovery of "the second one", which are dismally copied again, on the newly finished monument, before it is sealed.

Nagging suspicion "There's something sinister when a accomplished and fruitful writer receives, every week precisely, one of his own books by mail." Each book is neatly wrapped in decorative paper, with a paper card containing the preceding sentence attached with a piece of plain string to the wrapping.

(The writer goes mad and kills the postman :) )

O Fermi where art thou? What is the easiest way for a long-living (immortal even) race to discover, in a galaxy full of stars each of which could have it's own system of planets, where the best mining areas, rich with needed (and expensive) substances are? How about this: bio-engineer an evolution, starting with several thousands of possible life-starting structures, and culminating one day in beings marveling at their intelligence when they discover radio, and seed the galaxy with it. After that, they'd only have to listen for beacons appearing, beckoning from spots across the space.

The late race The cosmic background radiation is low, around 3 kelvins. Imagine how much time does it take for a universe created in a Big Bang to cool down so much! Regarding the Fermi paradox: this universe may be dying - all the civilizations have come and gone, and those who remain would be saddened and depressed looking at the night sky that is mostly black. Our civilization is mostly dark, almost depressed and disinterested - were the youngest civilization happier, sparkling with the enthusiasm of a newborn universe?

Does the Matrix have debuggers? Hutchison Effect (various weird Tesla+Philadelphia experiment+anti-gravity effects) was not repeated by Hutchison since 1980ies - he claims that it simply doesn't work now. What if something has changed since the time he did it (and possibly when Tesla did it) and now? Was a bug fixed in the Matrix?

Zero point energy? Zero point energy of the vacuum has been proven and theoretically calculated. In relativity, energy and mass are almost interchangeable. So zero point energy, which has extremely huge densities, could "produce" enough gravity to visibly bend space. So far, no fiction. But we're not noticing any space-bending in random blocks of vacuum. What if matter and its effects we notice is actually the LACK of (zero-point) energy?

A realistic Matrix Imagine a world heavily interwoven with digital information and presentation-enhancing technology, for example where words in books get highlighted when people touch them, automatically bringing on a view of encyclopedia that defines it, etc. People are happy and always young. Then a reality is revealed: it's a Matrix-like world where people painstakingly save money IRL to have a few hours of virtual happiness, and IRL live in dirty conditions in what are practically sleeping pens not homes. (Hmmm... cf. Gibson's Matrix)

Randomness as a positive evolutionary trait? Some "Dark Enemy" or Force or whatever is preying on all intelligence, it's attracted to intelligent creatures and kills them/turns them into itself/does something horrible. It is fought, defeated and exiled into another dimension/galaxy/some dark place thought uninhabitable. Certainly, no intelligence would arise in such a place. But it does, and eventually is attacked by and defeats the Dark Enemy, by evolving to be as much random as possible, never going in straight lines, always doing multiple actions at the same time, which seem unrelated and purposeless, so it isn't perceived as intelligent behavior. Its actions result in a Rube Goldberg-style "incredible machine" sequences with far-reaching consequences. Then it goes exploring "Our Place"... would we notice before it's too late?

Security by obscurity In ancient books, names of demons and other Scary Things were often deliberately misspelled in order not to provoke their manifestation by accidentally speaking such a name. The practice went on for a long time and at the end no one really knew the right names to speak. What if some loser finds such a book, tries to pronounce a name for fun and accidentally stumbles on the right pronunciation? Instant dark ages - a demon appears, all science suddenly stops working or malfunctions...

Background/setting: Earth in slavery An asteroid is detected to be on collision course with Earth, will be there in some small number of years, like five. Soon after, aliens from somwhere nearby appear and offer to remove it or change its course, in exchange for a big payment. "Nations of Earch" agree and subsequently humans are practically slaves to the aliens, providing them with either workforce or mining the Earth for them. It's a tough problem since the aliens obviously have significantly more powerful technology, and since everything is going to the aliense the economy and technology are stagnating.

Broodings and daydreams

This is just a collection of random thoughts. Don't take them seriously.

ivoras' Home

My CV

What's cooking for FreeBSD 9?

Overall system / architectural changes

Userland DTrace

CLANG / LLVM compiler

Kernel & low level improvements

Large-scale SMP support

USB 3.0 support

Network kernel core dumps (netdump)

Initial NUMA support

Modern event timer infrastructure

Tickless kernel

Networking improvements

High performance ssh

More SMP-scalable TCP/IP

New NFS client and server

Five new TCP congestion algorithms

SIFTR - Statistical Information for TCP Research

Storage subsystems' improvements

A move to support 4K drives

Generic GEOM IO schedulers

HAST - High Availability Storage

UFS SoftUpdates+Journal (SU+J)

New driver for AHCI SATA drives

ATA CAM implementation

Security

Capsicum

AES-XTS encryption mode in kernel

NFSv4 ACLs for UFS

Other changes

My FreeBSD things

What's cooking for FreeBSD?

Miscellaneous

What's cooking for FreeBSD 10?

Python Blowfish

What's cooking for FreeBSD 8?

Overall system / architectural changes

INET-less / IPv6-only kernel

CLANG / LLVM compiler

Parallel port builds

Kernel & low level improvements

Better handling of mounted device removals

Jails v2

Xen dom-U support

New USB stack

MPSAFE TTY

Kernel memory limit on AMD64 increased

Kernel threads

procstat(1): A process inspection utility

TextDumps: gathering information after kernel panic

ULE 3.0: New version of the SMP-optimized scheduler

Superpages

DTrace

Networking improvements

802.11s D3.03 wireless mesh networking

VirtNet / VIMAGE / Imunes / Network stack virtualization

Multiple routing tables / FIBs

Equal cost multipath routing

Zero-copy BPF

Kernel NFS locking support

NFSv4 support

Storage subsystems' improvements

Experimental new driver for AHCI

gvinum 2

GEOM_PART becomes the default slicer

Boot support for GPT partitions

bsdlabel gets extended to 20 partitions

Security

ProPolice SSP (stack-smashing protection)

Other changes

Stream cipher experiments

Python XXTEA implementation

FreeBSD under VMWare

1. General tips

2. Don't use lnc

3. Reduce kern.hz

4. Disable internal VMWare swapping

5. VMWare Tools not necessary

6. SMP

2. Don't use `lnc`

3. Reduce `kern.hz`

New `sendfile()` implementation, improved `sosend()`